Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use faster yaml loader and better delayed loading #16

Merged
merged 1 commit into from
Sep 18, 2024
Merged

use faster yaml loader and better delayed loading #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion bioio_sldy/reader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import pathlib
import typing

import dask
import dask.array as da
import numpy as np
import xarray as xr
Expand Down Expand Up @@ -149,9 +150,14 @@ def _read_delayed(self) -> xr.DataArray:
for timepoint in timepoints:
data_for_timepoint: typing.List[da.Array] = []
for channel in channels:
data = image.get_data(
value = dask.delayed(image.get_data)(
timepoint=timepoint, channel=channel, delayed=True
)
data = da.from_delayed(
value,
shape=(image.sizeZ, image.sizeY, image.sizeX),
dtype=image.dtype,
)
Comment on lines +153 to +160
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were previously loading the whole file one timepoint+channel pair at a time??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YES 😢 at least it was in memmap mode but still.... the test file is pretty big. Each one of these calls had a noticeable hitch of less than a second but with the delayed code they return basically immediately.

data_for_timepoint.append(data)

data_as_list.append(da.stack(data_for_timepoint))
Expand Down
16 changes: 13 additions & 3 deletions bioio_sldy/sldy_image.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ class SldyImage:
_data_paths: typing.Set[pathlib.Path] = set()

@staticmethod
def _yaml_mapping(loader: yaml.Loader, node: yaml.Node, deep: bool = False) -> dict:
def _yaml_mapping(
loader: yaml.CLoader, node: yaml.Node, deep: bool = False
) -> dict:
"""
Static method intended to map key-value pairs found in image
metadata yaml files to Python dictionaries.
Expand Down Expand Up @@ -108,7 +110,7 @@ def _get_yaml_contents(
"""
try:
with fs.open(yaml_path) as f:
return yaml.load(f, Loader=yaml.Loader)
return yaml.load(f, Loader=yaml.CLoader)
except FileNotFoundError:
if is_required:
raise
Expand Down Expand Up @@ -171,7 +173,7 @@ def __init__(
yaml.add_constructor(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG,
SldyImage._yaml_mapping,
yaml.Loader,
yaml.CLoader,
)

self._fs = fs
Expand Down Expand Up @@ -219,6 +221,14 @@ def __init__(
self.timepoints = sorted(self._timepoint_to_data_paths.keys())
self.channels = sorted(self._channel_to_data_paths.keys())

self.sizeT = self._image_record["CImageRecord70"]["mNumTimepoints"]
self.sizeC = self._image_record["CImageRecord70"]["mNumChannels"]
self.sizeZ = self._image_record["CImageRecord70"]["mNumPlanes"]
self.sizeY = self._image_record["CImageRecord70"]["mHeight"]
self.sizeX = self._image_record["CImageRecord70"]["mWidth"]
# TODO check this but are all sldys uint16? bioformats seems to say so
self.dtype = np.dtype(np.uint16)
Comment on lines +224 to +230
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick check: have we found any files that have pixel data <-> metadata conflicts?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you are putting these values into their own attributes because we don't have a system in place for storing pre-read / pre-load dims?

Copy link
Contributor Author

@toloudis toloudis Sep 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I made a ticket in bioio-base about this. Even if the xarray delayed setup is "fast", it would be still faster to return dims from metadata.

We had czi files that had pixel data size different from metadata size but as I recall it was generally because of unexpected termination of the image capture.
I don't know if this happens with other formats.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should just move to standardize this sooner rather than later and create a dims_from_meta attribute or something that Readers can optionally impl.

The docs for the attr should just mention that we have no guarantee that the dims are correct due to "early stopping of image acquisition"


@property
def metadata(self) -> typing.Dict[str, typing.Optional[dict]]:
"""
Expand Down
Loading