use faster yaml loader and better delayed loading

[toloudis]

Link to Relevant Issue

This pull request resolves #15

Description of Changes

1. YAML loading is VERY SLOW. Use faster yaml loader.
2. The way we constructed the delayed load array was also very (unusably) slow. Improve use of delayed loading.

After these changes, most of the time spent is in yaml parsing.

My scratch test script looks like this:

from bioio_base.types import PhysicalPixelSizes
from bioio import BioImage
from bioio.plugins import dump_plugins
from bioio.writers.ome_tiff_writer import OmeTiffWriter
import bioio_sldy
import dask
import numpy
import os
from pathlib import Path

filepath = Path(
    "\\\\allen\\aics\\assay-dev\\MicroscopyData\\John Paul\\2024\\20240305\\20240305_T01_001.sldy"
)
img = BioImage(filepath, reader=bioio_sldy.Reader)
print(img.shape)
print(img.dims)
# now let's extract one timepoint
s = img.get_image_dask_data("CZYX", T=7)
print(s.shape)
t = s.compute()
print(t.shape)

[evamaxfield]

Thanks for finding both the faster YAML load + delayed reads.

I am fine with the new attributes for storing the dims but I think long term we may want to simply add another dims container that stores img.dims_from_meta using the same Dimensions object.

[evamaxfield]

We were previously loading the whole file one timepoint+channel pair at a time??

[toloudis]

YES 😢 at least it was in memmap mode but still.... the test file is pretty big. Each one of these calls had a noticeable hitch of less than a second but with the delayed code they return basically immediately.

[evamaxfield]

Quick check: have we found any files that have pixel data <-> metadata conflicts?

[evamaxfield]

I assume you are putting these values into their own attributes because we don't have a system in place for storing pre-read / pre-load dims?

[toloudis]

Right, I made a ticket in bioio-base about this. Even if the xarray delayed setup is "fast", it would be still faster to return dims from metadata.

We had czi files that had pixel data size different from metadata size but as I recall it was generally because of unexpected termination of the image capture.
I don't know if this happens with other formats.

[evamaxfield]

I think we should just move to standardize this sooner rather than later and create a dims_from_meta attribute or something that Readers can optionally impl.

The docs for the attr should just mention that we have no guarantee that the dims are correct due to "early stopping of image acquisition"