Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[asdf] Do not copy array data #456

Merged
merged 3 commits into from
Aug 3, 2021

Conversation

marscher
Copy link
Contributor

@marscher marscher commented Jul 30, 2021

Changes

Because asdf does not handle time datatypes (correctly), there was a line of code within the serialization of xarray Variable to convert all ndarray like types from the asdf ndarray wrapper. This effectively prevents the use of the "copy_arrays" flag, since the
data got always copied.

Related Issues

Closes #454, #110

Checks

  • updated CHANGELOG.md
  • updated tests
  • updated doc/
  • update example/tutorial notebooks

@pep8speaks
Copy link

pep8speaks commented Jul 30, 2021

Hello @marscher! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-08-02 17:06:44 UTC

@codecov
Copy link

codecov bot commented Jul 30, 2021

Codecov Report

Merging #456 (0135a18) into master (282ed4a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #456   +/-   ##
=======================================
  Coverage   96.85%   96.85%           
=======================================
  Files          88       88           
  Lines        5595     5599    +4     
=======================================
+ Hits         5419     5423    +4     
  Misses        176      176           
Impacted Files Coverage Δ
weldx/asdf/tags/weldx/core/common_types.py 100.00% <100.00%> (ø)
weldx/asdf/tags/weldx/core/data_array.py 100.00% <100.00%> (ø)
weldx/visualization/k3d_impl.py 79.68% <0.00%> (+0.08%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 282ed4a...0135a18. Read the comment docs.

@CagtayFabry
Copy link
Member

nice, does this work now with copy_arrays?

we should probably dig into the (asdf related) code for more np.array(), np.asarray() etc. calls

@marscher
Copy link
Contributor Author

marscher commented Aug 2, 2021

Our unit tests switch the copy_arrays flag at least. But I doubt that the underlying functionality of mem mapping is really being tested. We can argue that this is already tested extensively upstream.

@CagtayFabry
Copy link
Member

Our unit tests switch the copy_arrays flag at least. But I doubt that the underlying functionality of mem mapping is really being tested. We can argue that this is already tested extensively upstream.

that is certainly not tested, we just use these parameters to catch any errors that might flare up
we would probably have to look into a memory profiler to see the difference

@CagtayFabry CagtayFabry added no-changelog-entry-needed xarray issues related to handling xarray objects labels Aug 2, 2021
@marscher marscher merged commit 46459b0 into BAMWelDX:master Aug 3, 2021
@marscher marscher deleted the do_not_copy_array_data branch August 3, 2021 08:20
@marscher
Copy link
Contributor Author

marscher commented Aug 3, 2021

I got my memory profiler booted up anyways for another PR, so eventually I'll take a look.

@CagtayFabry CagtayFabry added the ASDF everything ASDF related (python + schemas) label Aug 9, 2021
@CagtayFabry CagtayFabry mentioned this pull request Dec 8, 2021
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ASDF everything ASDF related (python + schemas) no-changelog-entry-needed xarray issues related to handling xarray objects
Projects
None yet
Development

Successfully merging this pull request may close these issues.

xarray.Variable deserialization casts NDArrayType (asdf) to np.ndarray
3 participants