Another potential culprit is appearing: vic_run_veg_lib seems to be wrong.

Another potential culprit: parallelization. I've noticed that, if I insert print statements into anything within the loop over cells in vic_image_run() or any function below that, like vic_run(), the order of grid cells gets all jumbled up. I've attempted to run with a single processor via mpirun -n 1 but maybe I'm doing it wrong? Regardless, running with mpirun -n 1 or omitting it and just running vic_image.exe -g .... by itself both result in jumbled time order of the execution of various print statements, to the point that it becomes very difficult to interpret.

I'm not saying that it's wrong to have parallelization per se; I'm saying that if parallelization is occurring even when I tell it to use a single processor, perhaps that could be causing a mis-match between the i of veg_lib[i] (which is carried indirectly through vic_run_veg_lib) and the i of veg_con[i], such that the veg_class called for by veg_con has veg_lib properties of invalid tiles from other cells. I.e., perhaps we're asking for properties of a veg_class in the current cell, but we're presented with the veg_lib of a different cell for which that class is not present; thus we get the garbage values. This would be consistent with things working OK for the tonic-generated param files, since again, they define the complete veg_lib for all classes in all cells. But it would be getting the right answer for the wrong reason... Perhaps there's an issue with my mpich library on my machine...

Now, the primary suspect is OpenMPI (at least the version on my machine, which might not be the right version). I deleted the line in vic_image_run() immediately before the loop over grid cells, which contains a pragma instructing OpenMPI to run the for loop using multiple threads. Suddenly my simulations ran fine.

The questions now are:

1. Is there something wrong with my OpenMPI? (and if so, how to fix to either handle the pragma correctly or to tell it to ignore the pragma)
2. How easily could this happen to other users? Is it happening at UW but not being detected due to all grid cells containing redundant copies of the veg_lib?
3. Regardless, is it a good idea for vic param files to define veg_lib parameters for zero-area veg tiles? (in terms of space usage, possibility for corrupt values where validation doesn't take place, etc.)

@tbohn -thanks for the report.

• For clarity, the pragma statement is for OpenMP (not OpenMPI) .
• If you put the pragma statement back and set the OMP_NUM_THREADS environment variable to 1, do you still see the buggy behavior?

  export OMP_NUM_THREADS=1
  

• If you compile without OpenMP but with the pragma statement in place, do you see the buggy behavior? For that, you'd need to remove the following line from the makefile.
  

  VIC/vic/drivers/image/Makefile

  Line 91 in 9fae303

  -fopenmp \

• Finally, are you still building against anaconda libraries? I'm suspect of that configuration so it would be good to know if this is repeatable on a different set of libraries.

To answer your questions:

• yes, export OMP_NUM_THREADS=1 fixes the problem.
• yes, commenting out the -fopenmp line fixes the problem.
• yes, I'm still building against the anaconda libraries.

@tbohn - how are you executing VIC? Do you get the odd behavior if you don't use MPI?

Can you run a test with:

export OMP_NUM_THREADS=8  # or some number of threads > 1
./vic_image -g your_param.txt  # omit mpirun 

The above test reproduces the bug.

That test is identical to one of the several ways that I ran vic and encountered the bug (i.e., I ran vic both with and without mpirun), except I wasn't explicitly setting OMP_NUM_THREADS to anything. So this implies that my environment somehow uses multiple threads by default.

I think in order to close this issue we should add a test that checks for bit for bit equivalence between the single core image driver and the openmp enabled image driver. That should probably fall on me but if someone else is interested and gets to it first, go for it.

@jhamman or @tbohn is there any update on this issue? I am running into a similar issue with NA values in my VIC output when running with OpenMPI, VIC5.1.0, and the calibrated VIC parameter file and domain from Currier et al. 2023. I get many NAs throughout the CRB domain.

I have tried a variety of the suggestions in this and other threads and in the VIC image driver documentation page (MX_RCACHE = 2, removing the -fopenmp flag, etc.), as well as trying different forcings, etc. We did not build against any conda libraries.

I am able to run all of the VIC sample datasets without this issue.

The issue presents as either large chunks of the domain as NA immediately:

Or NAs becoming more prevalent over time:

@tbohn thank you for your response!

-David

The NA values in my VIC output were caused by two issues:

1. Not setting any values for OMP_NUM_THREADS before running the image driver with OpenMPI, this caused a phenomenon were only a few cells were NA at the simulation start but the number of NA cells grew over time and seems to be a conflict between OpenMP and OpenMPI. export OMP_NUM_THREADS=1 solved this issue.
2. When generating my VIC forcings, I used MetSim. However, I made my MetSim domain file using the R terra package and doing so imparted something into the VIC forcings which caused large zones of NA cells in the VIC output, even though the VIC forcings looked fine upon inspection and analysis. Creating the MetSim domain with Python code did not cause this issue, even though the MetSim domain from R and Python were essentially identical in terms of mask and elevation. So perhaps something with the drivers used in the R terra package.