preprocess lf data?

[mariehemelt]

Is it possible to use preprocess_binary_file to preprocess lf files? I attempted it (below, and also with f_high=None). Is this possible and I am not calling it correctly, or is it impossible currently?

filtered_fname = preprocess_binary_file(dp, filt_key='lf', fname=None, target_dp=None, move_orig_data=True, median_subtract=False, f_low=250, f_high=0, order=3, verbose=True)
Preprocessing D:\MarieQD\LFP\1657_220729_g1_t0.imec0.lf.bin...
- filtering in time (between 250 and 0 Hz) forward:True, backward:False.
Traceback (most recent call last):
File "", line 1, in
File "d:\marieqd\npyx\neuropyxels\npyx\inout.py", line 760, in preprocess_binary_file
meta = read_metadata(dp)
File "d:\marieqd\npyx\neuropyxels\npyx\inout.py", line 47, in read_metadata
meta = metadata(dp)
File "d:\marieqd\npyx\neuropyxels\npyx\inout.py", line 199, in metadata
if 'imProbeOpt' in meta_glx["highpass"].keys(): # 3A
KeyError: 'highpass'

[chris-angeloni]

I've been modifying some of the core functions to do this, but I think from your error I think the filt_key you want to use is "lowpass" not "lf" (edit: actually, I'm not sure if this is true, different functions seem to expect different filt_keys, which would be nice to be made more consistant, but your issue is stemming from reading the metadata, so I don't think my code below is helpful. If I had to guess, your .meta files from SpikeGLX are missing or not where they are expected to be?).

Maybe not useful:
I had issues using the builtin functions filter the data... some bandpass filter settings just return NaN for some reason. I modified them slightly to get low-pass filtering only to be an option, which seems to generally work, but the filter settings you used return real data when I tested them (see usage example below).

If it is useful, I wrote (basically a stripped down version of extract_rawChunk) a function for loading data that doesn't preprocess the data at all (which I think is desirable for LFP), see below:

def load_binary_chunk(dp, times, channels=np.arange(384), filt_key='lowpass', scale=True, verbose=False):
    
    dp = Path(dp)
    meta = read_metadata(dp)
    fname = Path(dp)/meta['lowpass']['binary_relative_path'][2:]
    
    fs = meta[filt_key]['sampling_rate']
    Nchans=meta[filt_key]['n_channels_binaryfile']
    bytes_per_sample=2
    
    assert len(times)==2
    assert times[0]>=0
    assert times[1]<meta['recording_length_seconds']
    
    # Format inputs
    ignore_ks_chanfilt = True
    channels=assert_chan_in_dataset(dp, channels, ignore_ks_chanfilt)
    t1, t2 = int(np.round(times[0]*fs)), int(np.round(times[1]*fs))
    
    vmem=dict(psutil.virtual_memory()._asdict())
    chunkSize = int(fs*Nchans*bytes_per_sample*(times[1]-times[0]))
    if verbose:
        print('Used RAM: {0:.1f}% ({1:.2f}GB total).'.format(vmem['used']*100/vmem['total'], vmem['total']/1024/1024/1024))
        print('Chunk size:{0:.3f}MB. Available RAM: {1:.3f}MB.'.format(chunkSize/1024/1024, vmem['available']/1024/1024))
    if chunkSize>0.9*vmem['available']:
        print('WARNING you are trying to load {0:.3f}MB into RAM but have only {1:.3f}MB available.\
              Pick less channels or a smaller time chunk.'.format(chunkSize/1024/1024, vmem['available']/1024/1024))
        return
    
    # Get chunk from binary file
    with open(fname, 'rb') as f_src:
        # each sample for each channel is encoded on 16 bits = 2 bytes: samples*Nchannels*2.
        byte1 = int(t1*Nchans*bytes_per_sample)
        byte2 = int(t2*Nchans*bytes_per_sample)
        bytesRange = byte2-byte1

        f_src.seek(byte1)

        bData = f_src.read(bytesRange)
        
    # Decode binary data
    # channels on axis 0, time on axis 1
    assert len(bData)%2==0
    rc = np.frombuffer(bData, dtype=np.int16) # 16bits decoding
    rc = rc.reshape((int(t2-t1), Nchans)).T
    rc = rc[:-1,:] # remove sync channel
    
    # Scale data
    if scale:
        rc = rc * meta['bit_uV_conv_factor'] # convert into uV
    
    return rc

Usage (load, offset correct, then filter):

# load and plot baseline period
meta = read_metadata(dp)
fs = meta['lowpass']['sampling_rate']
load_times = [4*60,(4+1)*60]
dat = load_binary_chunk(dp,load_times,filt_key='lowpass')
dat = dat - np.median(dat,axis=1)[:,np.newaxis]
plt.plot(dat[100,0:fs])

# add data to GPU array
d_t = np.ascontiguousarray(dat.T)
d_t = cp.asarray(d_t)
d_filt = gpufilter(d_t, fs=fs, fslow=250, fshigh=0, order=3,
             car=False, forward=True, backward=True, ret_numpy=True).T
plt.plot(d_filt[100,0:fs])