You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The command line interface provided in AllenInstitute/deepinterpolation is too unstable to be reliably used for denoising videos in production. Among its deficiencies
Rather than read the input movie into memory once at the start of inference, it reads data from the input HDF5 one chunk at a time, opening and closing a new file handle as needed.
This can cause the job to fail on an OSError if the connection to the NFS system becomes stale while the job is running (as often happens on our SLURM cluster)
Rather than storing output to a cache and periodically flushing it to disk as results build up in memory, the CLI opens a handle to the output file that must remain open for the entire compute job
This, again, can cause failures due to the slow connection between our NFS file system and our SLURM compute notes
It relies on tensorflow's native parallelization, which means that many CPUs remain idle during I/O processes. This could be sped up by reading in all of the data at once, intelligently chunkingt it, and farming the work for each chunk out to a different CPU
This branch of a fork of the deepinterpolation repository implements fixes to these problems
However, beyond all of this, AllenInstitute/deepinterpolation tries to support many use cases (FMRI imaging, ECEPhys traces) that we do not need to support yet. Since, for denoising, all we need to do is take a precomputed tensorflow model and apply it to a 2 Photon movie, I propose that Pika write our own denoising module that can live in ophys_etl_pipelines. This will give us ownership and control over the code and allow us to decouple our pipeline from the still rapid development going on in AllenInstitute/deepinterpolation.
Tasks
Write an ophys_etl_pipeline module to accept the HDF5 stored models solved for by AllenInstitute/deepinterpolation and apply it to a motion corrected 2 Photon movie using tensorflow
Validation
Verify that the results of running inference with deepinterpolation's CLI and the new module are equivalent
The text was updated successfully, but these errors were encountered:
The command line interface provided in AllenInstitute/deepinterpolation is too unstable to be reliably used for denoising videos in production. Among its deficiencies
https://github.com/AllenInstitute/deepinterpolation/blob/master/deepinterpolation/generator_collection.py#L934-L998
This can cause the job to fail on an OSError if the connection to the NFS system becomes stale while the job is running (as often happens on our SLURM cluster)
https://github.com/AllenInstitute/deepinterpolation/blob/master/deepinterpolation/generator_collection.py#L934-L998
This, again, can cause failures due to the slow connection between our NFS file system and our SLURM compute notes
This branch of a fork of the deepinterpolation repository implements fixes to these problems
https://github.com/danielsf/deepinterpolation/commits/danielsf/by_hand_parallelization
However, beyond all of this, AllenInstitute/deepinterpolation tries to support many use cases (FMRI imaging, ECEPhys traces) that we do not need to support yet. Since, for denoising, all we need to do is take a precomputed tensorflow model and apply it to a 2 Photon movie, I propose that Pika write our own denoising module that can live in ophys_etl_pipelines. This will give us ownership and control over the code and allow us to decouple our pipeline from the still rapid development going on in AllenInstitute/deepinterpolation.
Tasks
Validation
The text was updated successfully, but these errors were encountered: