How to use Virtual File Drivers with HighFive

[s769]

I want to use the Virtual File Driver for cuFile for GPU Direct I/O. After building the VFD and linking as they say on their page, can I just use HighFive as usual, or is there anything specific I need to do?

[1uc]

I'm not familiar with the feature. The README suggests it's a matter of setting the correct File Access Property List (FAPL). Those are supported in HighFive, you might need to wrap the specific functions you need to set the value of the FAPL, or use the C-API directly to set it. You can use .getId() to obtain a plain hid_t that can be used with the HDF5 C-API. This is helpful when using advanced/unsupported features.

You'll likely want to use read_raw and write_raw because otherwise you risk HighFive making a copy, e.g. for things like std::vector<std::vector<>> (not contiguous) or Eigen::Matrix (column-major).

I don't know of anyone that has tried anything similar with HighFive. It feels like something the authors didn't expected, and it's certainly untested. Please reach out if you notice something isn't working. It seems like a nice feature to support. Therefore, if it's reasonable effort we might try to fix issues. If you notice that it works (with small tweaks) that would also be interesting for us to know.