Memory improvements (1/3): Introduce new data access layer and schemas #1728

cjao · 2023-07-12T15:54:58Z

I have added the tests to cover my changes.
I have updated the documentation and CHANGELOG accordingly.
I have read the CONTRIBUTING document.

Closes https://github.com/AgnostiqHQ/covalent-staging/issues/707

cjao · 2023-07-12T16:05:38Z

This PR installs plumbing for the series on memory usage improvements. A new data access layer is introduced which shall replace the in-memory Result object as used by the server code in the next PR.

Merging notes

This PR passes all unit tests but functional tests will be broken until the next PR.
In anticipation of wide-ranging changes to _core in Memory improvements (2/3): Migrate core Covalent to new data access layer #1729 , some unit tests {dispatcher,data_manager,execution}_test.py are temporarily renamed to tmp_{dispatcher,data_manager,execution}.test.py
Some artificial commits were introduced to make the PR as self-contained as possible; you can see those in the commit history. Those are reverted in Memory improvements (2/3): Migrate core Covalent to new data access layer #1729 .

codecov · 2023-07-13T10:56:41Z

Codecov Report

Merging #1728 (b487f58) into develop (889bf3e) will increase coverage by 1.34%.
The diff coverage is 95.63%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1728      +/-   ##
===========================================
+ Coverage    80.17%   81.51%   +1.34%     
===========================================
  Files          232      272      +40     
  Lines        10239    12585    +2346     
  Branches       193      193              
===========================================
+ Hits          8209    10259    +2050     
- Misses        1897     2193     +296     
  Partials       133      133

Flag	Coverage Δ
Dispatcher	`90.99% <97.08%> (+4.63%)`	⬆️
Functional_Tests	`?`
SDK	`76.51% <92.30%> (+1.68%)`	⬆️
UI_Backend	`87.38% <57.14%> (-1.04%)`	⬇️
UI_Frontend	`73.43% <ø> (ø)`

covalent/_file_transfer/enums.py

covalent/_file_transfer/file_transfer.py

covalent/_file_transfer/strategies/shutil_strategy.py

covalent/_results_manager/result.py

covalent/_serialize/common.py

covalent/_serialize/electron.py

covalent/_serialize/lattice.py

covalent/_serialize/result.py

covalent/_serialize/transport_graph.py

wjcunningham7

This first review covers everything except the covalent_dispatcher module

covalent/_file_transfer/enums.py

covalent/_serialize/common.py

covalent/_shared_files/defaults.py

covalent/_shared_files/schemas/asset.py

covalent/_shared_files/schemas/common.py

covalent/_shared_files/schemas/electron.py

covalent/executor/executor_plugins/dask.py

covalent/_workflow/transportable_object.py

FyzHsn

Really nice work. This is round 2 of 11 reviews. Left some questions in this review.

covalent/_serialize/electron.py

covalent/_shared_files/schemas/common.py

covalent/_shared_files/schemas/electron.py

covalent/_shared_files/utils.py

covalent/_workflow/electron.py

FyzHsn

This is the final review for this PR. Great work! Left some minor comments and noted that there are some commented-out tests. Either remove them or uncomment them.

covalent_dispatcher/_dal/db_interfaces/electron_utils.py

covalent_dispatcher/_dal/importers/result.py

covalent_dispatcher/_db/write_result_to_db.py

tests/covalent_dispatcher_tests/_core/data_manager_test.py

tests/covalent_dispatcher_tests/_core/dispatcher_test.py

Introduce temporary implementations of `update._node` and `update.lattice_data`. These will be removed once core covalent is transitioned to the new DAL.

Change abs imports to rel imports. Needed to please pip-missing-reqs.

- Improve type hints

…lent into memory-improvements-pr-1

cjao

@kessler-frost One quibble about Result._update_node but otherwise this is quite clean.

covalent_dispatcher/_dal/result.py

covalent/_workflow/transportable_object.py

cjao

Some comments on qelectron_data_exists.

covalent_dispatcher/_dal/importers/electron.py

covalent/_shared_files/schemas/electron.py

covalent/_serialize/electron.py

covalent_dispatcher/_dal/exporters/electron.py

covalent/_workflow/transport.py

cjao mentioned this pull request Jul 12, 2023

Memory improvements (2/3): Migrate core Covalent to new data access layer #1729

Merged

3 tasks

cjao force-pushed the memory-improvements-pr-1 branch 3 times, most recently from 4a828a8 to f403fad Compare July 13, 2023 10:51

cjao force-pushed the memory-improvements-pr-1 branch 3 times, most recently from b099de5 to b47baf2 Compare July 14, 2023 13:34

cjao marked this pull request as ready for review July 14, 2023 15:31

cjao requested a review from a team as a code owner July 14, 2023 15:31

cjao requested review from wjcunningham7, kessler-frost, FyzHsn and AlejandroEsquivel July 14, 2023 15:31

FyzHsn requested changes Jul 17, 2023

View reviewed changes

wjcunningham7 requested changes Jul 20, 2023

View reviewed changes

FyzHsn requested changes Jul 24, 2023

View reviewed changes

FyzHsn requested changes Aug 4, 2023

View reviewed changes

cjao added 10 commits September 24, 2023 08:43

Mem (1/3): introduce new DAL and core Pydantic models

39a1598

Mem (1/3): Fix schemas

9389b3b

Mem (1/3): DAL PR: temporarily redirect core dispatcher tests

23761a7

Mem (1/3): DAL PR: fix tests

69def08

Introduce temporary implementations of `update._node` and `update.lattice_data`. These will be removed once core covalent is transitioned to the new DAL.

Mem (1/3): Fix requirements workflow

f942531

Change abs imports to rel imports. Needed to please pip-missing-reqs.

Mem (1/3): Uncomment boilerplate in disabled unit tests

4693404

Mem (1/3): Add unit test for format_server_url

8bb0adb

Mem (1/3): defer copy_file_locally to next PR

b18db33

Mem (1/3): update changelog

f792432

Mem (1/3): Core DAL improvements

cb8ae1d

- Improve type hints

cjao force-pushed the memory-improvements-pr-1 branch from b47baf2 to cb8ae1d Compare September 25, 2023 13:43

kessler-frost added 3 commits October 3, 2023 19:24

Merge branch 'develop' into memory-improvements-pr-1

801842b

fixing ui backend tests

0109e36

Merge branch 'memory-improvements-pr-1' of github.com:AgnostiqHQ/cova…

35076e8

…lent into memory-improvements-pr-1

santoshkumarradha mentioned this pull request Oct 4, 2023

Reduce memory consumption when running workflows #1544

Closed

kessler-frost added 5 commits October 4, 2023 13:56

addressed most comments, still some left

b1ceb26

implementing final set of suggestions

6912aec

Merge branch 'develop' into memory-improvements-pr-1

f02c68f

updated changelog

acf39f1

fixed changelog

c3be087

kessler-frost marked this pull request as ready for review October 6, 2023 15:40

Prasy12 previously approved these changes Oct 11, 2023

View reviewed changes

kessler-frost previously approved these changes Oct 11, 2023

View reviewed changes

kessler-frost self-requested a review October 11, 2023 05:51

cjao commented Oct 12, 2023

View reviewed changes

covalent_dispatcher/_dal/result.py Outdated Show resolved Hide resolved

cjao commented Oct 12, 2023

View reviewed changes

covalent/_workflow/transportable_object.py Outdated Show resolved Hide resolved

implemented suggestions

1c7aa36

kessler-frost dismissed stale reviews from Prasy12 and themself via 1c7aa36 October 12, 2023 13:38

kessler-frost added 2 commits October 12, 2023 09:38

Merge branch 'develop' into memory-improvements-pr-1

4b3f46d

Merge branch 'develop' into memory-improvements-pr-1

a64a1a6

cjao commented Oct 12, 2023

View reviewed changes

kessler-frost added 4 commits October 13, 2023 03:03

Added qelectron_data_exists to the ElectronMetadata schema

f49860f

Added qelectron_data_exists to the electron metadata

fa73fee

fixed typo

f085136

fixing test

3262dfb

cjao commented Oct 13, 2023

View reviewed changes

covalent/_workflow/transport.py Outdated Show resolved Hide resolved

fixing test

b487f58

kessler-frost approved these changes Oct 13, 2023

View reviewed changes

AlejandroEsquivel merged commit 314e859 into develop Oct 13, 2023
12 checks passed

AlejandroEsquivel deleted the memory-improvements-pr-1 branch October 13, 2023 21:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory improvements (1/3): Introduce new data access layer and schemas #1728

Memory improvements (1/3): Introduce new data access layer and schemas #1728

cjao commented Jul 12, 2023 •

edited

Loading

cjao commented Jul 12, 2023 •

edited

Loading

codecov bot commented Jul 13, 2023 •

edited

Loading

wjcunningham7 left a comment

FyzHsn left a comment

FyzHsn left a comment

cjao left a comment

cjao left a comment

Memory improvements (1/3): Introduce new data access layer and schemas #1728

Memory improvements (1/3): Introduce new data access layer and schemas #1728

Conversation

cjao commented Jul 12, 2023 • edited Loading

cjao commented Jul 12, 2023 • edited Loading

Merging notes

codecov bot commented Jul 13, 2023 • edited Loading

Codecov Report

wjcunningham7 left a comment

Choose a reason for hiding this comment

FyzHsn left a comment

Choose a reason for hiding this comment

FyzHsn left a comment

Choose a reason for hiding this comment

cjao left a comment

Choose a reason for hiding this comment

cjao left a comment

Choose a reason for hiding this comment

cjao commented Jul 12, 2023 •

edited

Loading

cjao commented Jul 12, 2023 •

edited

Loading

codecov bot commented Jul 13, 2023 •

edited

Loading