Dealing with non binary files

[yger]

For next release of SpyKING CIRCUS, we are planning to allow users to keep their data as they are, reading/writing directly to some classical file formats (openephys, multi-channels system, NWB, ...). This is because for large scale recordings, data files can be very large, and we would like to avoid an export to raw binary, as this can take time and disk space. The problem is that phy is currently only able to deal with raw binary files, am I right? Or I guess because of kwik format, any HDF5-like structure could easily be supported? How can we design wrappers to tell phy how to properly read chunks of data, then?

[rossant]

That should be doable. We might not have the resources to implement all of these formats ourselves, but we'd be happy to help contributors. I'm planning to update that part of the code in the coming weeks to make the process even easier.

[yger]

Great, I would be happy to contribute, so let me know when you start updating your code. I'm currently doing those wrappers myself for SpyKING CIRCUS, so I guess I could re-use some snippet of codes. At the end, phy will simply need to have specified open/close/get_data methods given file formats, and everything could work the same

[rossant]

Okay, let me know where I can see the code of your wrappers in SpyKING CIRCUS so that I make sure it will be easily portable into phy.

[yger]

Ok, this is work in progress, but this will be in the branch 0.5, folder
circus/datafiles/

The generic file type will be defined in datafile.py, with some key methods
(but for phy, I guess only get_snippet is relevant, to extract waveforms or
time slices), and then I'm now starting various wrappers for datafiles. The
code does not yet look very nice, but this is working, and SpyKING CIRCUS
is now file-type agnostic, such as phy should be

Thanks

2016-09-19 11:22 GMT+02:00 Cyrille Rossant [email protected]:

Okay, let me know where I can see the code of your wrappers in SpyKING
CIRCUS so that I make sure it will be easily portable into phy.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#63 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABmE_4-1HTCZsGeThJ0EVbH4CFog5ddqks5qrlRvgaJpZM4KARjK
.

[WaveformDai]

Any new progress on reading raw data instead of binary file? It will definitely help save lots of disk space!
Thank you for your wonderful tools!

[rossant]

superseded by cortex-lab/phylib#5