Firedrake gives MPI error after install

[andrewlhicks]

Hello,

I recently uninstalled Firedrake, then reinstalled (this was due to the firedrake-update not quite working properly). The install went fine, but if I import anything from Firedrake I get the following message:

Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(60): MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffc619b5038) failed
MPII_Init_thread(205)...: gpu_init failed
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=269078287
:
system msg for write_line failure : Bad file descriptor

firedrake-install.log

I've attached the install log. I'm really confused here as to what's happening, since I had previously installed it and had no issues in the past.

[connorjward]

This is a strange error. A couple things:

• Can you try running make check from inside firedrake/src/petsc (you might need to specify PETSC_ARCH=default)
• Can you see if you can compile and run a simple Hello World C program with mpicc + mpiexec
• Can you share firedrake/src/petsc/configure.log

[andrewlhicks]

Alright, I've done your suggestions. Here is the output from make check, which is filled with errors throughout:
makecheck_output.log

Secondly, I did compile a simple Hello World C program using mpicc, with no issues. Then, I executed it with no issues, and finally I used mpiexec to execute it without issue, and even mpiexec -n 2 just for fun and it behaved exactly as expected.

Finally, here is the configure log:
configure.log

[connorjward]

Hmm... Your PETSc installation is definitely broken. Inside configure.log I'm seeing concerning messages like:

Executing: gcc  -o /tmp/petsc-v7tix4bv/config.setCompilers/conftest    /tmp/petsc-v7tix4bv/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 1
stderr:
/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status

Quick check. You totally removed the previous firedrake directory before running firedrake-install right?
I also notice that your GCC is a little bit old.

@JDBetteridge do you have any insights?

[JDBetteridge]

Your MPICH build, or possibly hwloc build seems to be picking up CUDA in your environment. If you're on a HPC system first unload any CUDA/GPU related modules. Set the following environment variable:

export PETSC_CONFIGURE_OPTIONS='--with-cuda=0 --download-hwloc-configure-arguments="--disable-opencl --disable-cuda"'

and try the install again. Please post back if you're still having issues.

[andrewlhicks]

This does not seem to work. However, there is a solution which does seem to work, though it seems to be not an ideal solution.

After doing firedrake-install as normal, I set the following environmental variable before running any scripts:

export MPIR_CVAR_ENABLE_GPU=0

This seems to work, though it seems there should be a more permanent fix to this.

[JDBetteridge]

I set the following environmental variable before running any scripts:
export MPIR_CVAR_ENABLE_GPU=0

I'm going to mark this as the answer for anyone looking for a solution. Your choices are to rebuild MPICH without GPU support (apparently my flags weren't sufficient, maybe someone else can provide some alternative flags), or to set this environment variable you mentioned.