Number of recorded datapoints of different channels

[thiagoburghi]

Basic Information:

RTXI Version: 3.0.4
Installed from Live CD?: no

Error Description: Recordings from more than one channel yield different number of datapoints, making it difficult to know where the signals should be aligned.

What was supposed to happen: all channels in the .h5 should have the same number of datapoints

What steps will trigger the error: including more than one channel to be recorded in the Data Recorder Widget, starting a recording, and stopping a recording.

[fusge]

This would be impossible to achieve without slowing down other parts of RTXI to guarantee that there are exactly the same number of points for all recording channels. Can you describe your signal alignment requirements? Is it not possible to align the data points using the time information included with the data in the HDF5 file?

[thiagoburghi]

My bad, I only now realised that we should be using the timestamps in the hdf5 file. I wonder if aligning the first elements of two recorded signals is enough? Here is an example of what I get when I read the .h5 files in Matlab:

data_v = h5read(file,'/Trial1/Synchronous Data/PCIe-6259 OUTPUT 7 Recording Component');
data_iapp = h5read(file,'/Trial1/Synchronous Data/PCIe-6259 OUTPUT 4 Recording Component');

data_v.time(1)
ans = 453781214017385
   
data_iapp.time(1)
ans = 453781213910506

should I just find the indices of the timestamps that are the closest, and then assume that each subsequent sample of v and iapp will be aligned well enough? Also, out of curiosity, what are the units of the timestamps/what do they represent? Many thanks

[fusge]

The time values in the hdf5 file represents the number of nanoseconds since some arbitrary time in the past before RTXI app was opened. It is used to check how far apart in time the data points are from each other and from the start of the trial, and you can identify the start of the trial by finding the lowest time value in any of the datasets for that particular trial.

Whether aligning the data points by index would be enough for your analysis depends on the update frequency you were operating RTXI under. For instance, if you were running RTXI in 1 ms period and your recordings are within a few microseconds then it may be an acceptable approach. It all depends on what you are trying to achieve when aligning the points.

Your example has two datasets whose first value are around 100 microseconds apart. If that difference is too large then you can reduce the latency by eliminating dynamic allocations and making sure to cache values whenever possible. Typically this means avoiding the following operations inside the execute function:

1. new and delete
2. creating a vector or map
3. inserting values and changing the size of your dynamic arrays (push_back, insert)
4. calling qt code

Hope that helps.

[thiagoburghi]

Hi, thanks! It took me a long time but I got back to this. I see that there might be a logistical problem there with saving the datapoints in a synchronized fashion. My usecase is that I would like to have the sampled data every dt time , where dt is the sampling period selected in the control panel. From there I am fitting models to the data generated/collected in RTXI. In fact I've been using the data from the amplifier software, because it gives me the data in the form I want, while in RTXI there's all this other stuff I would need to do to get it to work. I thought there could be a simple fix where the the Data Recorder simply stamps the data with a counter from when we started to record, and the counter multiplies the current sampling period of the control panel. The timestamps logged as of now are adding friction to my pipeline, and I can imagine it would add friction to that of other users as well...

[fusge]

Hi, It may be possible to tell RTXI to save a fixed number of points when stopping the recorder, either by waiting for other channels to read in some data, or by shaving off data points from others. This would guarantee that channels always record the same number of points every time, and this can be exposed as a checkbox to the user. The user can then make their own decisions as to whether this approach is acceptable or not.

Another option is to instead save indices instead of time stamps. This of course would reduce accuracy since calculations do not happen at precisely the time reconstructed from the indices, but again it can be exposed as a checkbox to the user, and they can make their own arrangements.

Which one of these options seems most attractive to your use case?

[thiagoburghi]

Hi, and thanks for considering this feature! I think the second option is probably the most useful. I don't think the lost accuracy will be a big deal for most people trying to fit models to that data, which is my usecase.