Deepdish library no longer maintained

[RemingtonRohel]

DEPRECATION

package: deepdish

Reason

See this PR and this issue on the deepdish github. Newer numpy installs are going to have problems, so we can update now to avoid any issue.

Replacement

The h5py library is already used in places within pyDARNio, and is supported by the HDF5 group. It is slightly less convenient to use, but much faster than deepdish.

Details

This is a convenient time to change the structure of some Borealis files - namely, fields such as data_descriptors and correlation_descriptors which are lists of utf-32 strings stored as one large array of uint8 type. It's a headache to write and read these fields with h5py, so this is a prime opportunity to refactor the data format to something more user-friendly. There may be some headache in dealing with scan_start_marker. Trying to read in the field with h5py yields:

>>> rec.attrs['scan_start_marker']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/remington/pydarnio-env/lib64/python3.6/site-packages/h5py/_hl/attrs.py", line 61, in __getitem__
    dtype = attr.dtype
  File "h5py/h5a.pyx", line 334, in h5py.h5a.AttrID.dtype.__get__
  File "h5py/h5a.pyx", line 336, in h5py.h5a.AttrID.dtype.__get__
  File "h5py/h5t.pyx", line 422, in h5py.h5t.TypeID.py_dtype
TypeError: No NumPy equivalent for TypeBitfieldID exists

[RemingtonRohel]

Just found out that the bug in the snippet above has been handled by h5py, and the version I was using (3.1.0) was too old. See here for the PR that fixed it. Verified that version 3.8.0 can handle TypeBitfieldID types.

[RemingtonRohel]

Update: Did a bit more digging, based on the date the PR was merged any h5py version >= 3.3.0 should work with TypeBitfieldID

[carleyjmartin]

We can add a version requirement in setup.py (and also do we need to move to a toml at some point, probably it's own issue?) Do we also know if h5py has some dependency version requirements too, I think there was an issue I was having some time ago, but I couldn't update h5py because I couldn't get a newer version of a dependency.

install_requires=['pyyaml', 'numpy',
                      'h5py>=3.3.0', 'pathlib2']

[RemingtonRohel]

@carleyjmartin Are you interested in attempting to replace all deepdish usages with h5py? I have a branch (deepdish_to_h5py) that I've started to make some changes in, but I don't have time this week to get this all done. I'd be happy to provide support though if you are willing to take the lead.

[carleyjmartin]

@carleyjmartin Are you interested in attempting to replace all deepdish usages with h5py? I have a branch (deepdish_to_h5py) that I've started to make some changes in, but I don't have time this week to get this all done. I'd be happy to provide support though if you are willing to take the lead.

Yeah sure, I can get to it this week.