Quick question on multithreading and distributed

[natgeo-wong]

Can NCDatasets.jl be used with either Multithreading or Distributed.jl?

[natgeo-wong]

After some fiddling around, i think NCDatasets.jl is not thread-safe. I'm getting different results when I'm using a single thread compared to when I'm using multiple threads, meaning that I'm opening different NetCDF files using different threads. I am doing further testing to confirm this.

[Alexander-Barth]

the underlying C netCDF library is indeed not thread-safe

Unidata/netcdf-c#1373
https://www.unidata.ucar.edu/blogs/developer/en/entry/implementing-thread-safe-access-to

In my case, I was able to make it work with a lock. Basically the code looks like:

ds = NCDataset("file.nc","c")
# define all variables
ncsla = defVar(...)

OI_lock = ReentrantLock()

Threads.@threads for n = 1:ntime
    fi = some_computation()
    lock(OI_lock) do
       ncsla[:,:,n] = fi
    end
end

But as long as, the libnetcdf is not thread safe, this is more of a hack than a solution.

[natgeo-wong]

Ahhhhh okay I see! Thank you!! So nothing can be done on the Julia end, I take it, besides this stopgap.

[Alexander-Barth]

This work-around from the rust NetCDF library, would also be possible for NCDatasets:
Unidata/netcdf-c#1373 (comment)

But it is a quite invasive change, has poor performance and it would get obsolet once we have a proper fix (a thread-save libnetcdf)

[natgeo-wong]

Well, I can always get around it or not use multithreaded anyway.

If I open and close the file before using thread loops my code should otherwise be fine right?

[Alexander-Barth]

If I open and close the file before using thread loops my code should otherwise be fine right?

well in addition to that I would also make sure different threads to not read from and write to the file at the same time (reading the upstream issue also reading from a netcdf file can be problematic for libnetcdf; sadly). It my tests, it was also ok if different threads write to the same variable but at different index and not at the same time.