Si example problems

[alex2shiyu]

Hi Nakib,

Thank you for the brilliant work of both algorithm and software implementation.
I am trying to run the silicon examples. And I encountered some problems, could you please provide some insights?

Version:

• elphbolt: v1.1 (passed all the tests)
• Quantum Espresso: q-e-qe-6.7MaX-Release (compile with intel 2021.7)
• epw: 5.3.1 (the one included in Q-E)
• Replaced two source files of epw following the instructions: YES.
• spglib: 1.14.1 (compile with gcc12.3.0)
• OpenCoarrays: 2.10.0 (compile with gcc12.3.0)
• gcc: 12.3.0

Question

epw

• error 1:

   loadqmesh_serial error : Cannot load fine q points

After I looked into epw tutorials and their examples, it's like I need to assign the following values

   nkf1         = 10
   nkf2         = 10
   nkf3         = 10
 
   nqf1         = 20
   nqf2         = 20
   nqf3         = 20

Then the code can run normally. I want to know whether it's normal to add these six more parameters? (because your tutorial does not mention this)

• error 2:
  I can not find epmatwp1 or si.epmatwp1 file but epmatwp after EPW calculation
  According to EPW release log, it seems like that epmatwp1 has been changed to epmatwp in 5.3.1. Is this correct?

elphbolt

After I resolved the problems with the workaround mentioned above, I ran the elphbolt with

• 60 images
• Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz 64 cores
  But the code does not finish even after 12 hours. I found it got stuck in Calculating electron density of states.... I logged into the compute node and found it is really running. From your README.org, it says it just takes 3 minuts on 2 CPU cores. Why does my calculation takes so long time? Is it because I added the following parameters in EPW stage?

   nkf1         = 10
   nkf2         = 10
   nkf3         = 10
 
   nqf1         = 20
   nqf2         = 20
   nqf3         = 20

I am looking forward to your reply.

Best,
Shiyu

[alex2shiyu]

Hi,
Just an update, when I changed the number of images from 60 to 2, it finished successfully. It's weird.
Though the problem for elphbolt seems to be resolved, I am still looking forward to your reply on the issues regarding to epw.

Thanks!

[alex2shiyu]

For the issue regarding the number of images, It seems I know the reason.

The number of images should be less than or equal to min(#q, #states, #k).

For silicon examples, min(#q, #states, #k) = min(16, 96, 25)=16. Then I tested #image=16 and 17, the former passed while the latter got stuck in one step of the logistics.

[nakib]

I have now fixed this issue. At the moment the fix is in the development branch.

[nakib]

Dear Shiyu,

Thank you for your kind words and your questions. Also, I must mention that I appreciate the fact that you explained the issues in a nice, systematic way.

Regarding the EPW issue:

My understanding is that the nkf* and nqf* are the fine meshes which are needed for calculating quantities that require a summation over these meshes. So as long as you have the following:

elecselfen = .f.
phonselfen = .f.
a2f = .f.

things should work. Are you using the provided epw.in file verbatim? Or did you make some modifications to it?

Secondly, you are right that there was a file name change. What was called <prefix>.epmatwp1 is now called epmatwp. Just rename epmatwp to epmatwp1 when you pass it to elphbolt. I should update my instructions regarding this.

As always, it is best to compare the high symmetry path plots of the electron and phonon bands and the electron-phonon matrix elements between epw and elphbolt. This is a strong check that everything is consistent.

Regarding the issue in elphbolt:

Were you running the tiny example parallelized over 60 images? If yes, could you check from the output that you have 29 irreducible k-points in the energy restricted mesh? Where your calculation got stuck was in the subroutine calculate_el_dos in bz_sums.f90. If you follow the code there, you will see a call to distribute_points. This is called to distribute the 29 points over all the images. It looks like at the present this distribution does not work when the number of points is less than the number of images. I [or a community member like you :)] could look into generalizing distribute_points to handle the case you are throwing at it or make it stop with an error. That said, in realistic calculations, I think one will always have more k and q points than the number of coarray images.

I hope this answers you question. Let me know if anything is unclear.

Best,
Nakib

[alex2shiyu]

Hi Nakib,

Thanks for your prompt and insightful reply.

For the question regarding epw.in file:
Yes, I used the input file from the example without any modification except for the phonon path dvscf_dir and output path outdir. I agree with your explanation and I also checked epw tutorial by myself, so it's really weird.

Thank you for the suggestion for the sanity check. Will do.

For the problem regarding elphbolt:
Yes, I calculated for the tiny example and there are 29 irreducible K points. I agree with what you said. For the production calculations, it's not a problem. I think this problem is well explained and makes sense.

However, I ran into another coarray or mpi-like problem when using a lot of images for dense q-grid of silicon examples.
when I increased the q-mesh from the default qmesh = 6 6 6 to qmesh = 50 50 50 as well as assigned mesh_ref = 3 as you suggested in the README.org, elphbolt ran into some errors just like what we sometimes met in the MPI codes.

• sometimes the error shows like below when I use 64 cores/node * 6 node = 384 cores,

node11.cluster.2352650Exhausted 1048576 MQ irecv request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM2_MQ_RECVREQS_MAX=1048576)

I tried to increase PSM2_MQ_RECVREQS_MAX or PSM2_MEMORY like

#export PSM2_MEMORY=huge
#export PSM2_MQ_RECVREQS_MAX=2097152

but it does not work. It will throw out the same error or directly tell me that it runs out of memory. It's weird because my cluster is very powerful with the specs listed below

• 2 CPUs Intel Xeon Platinum 8358 (2.6 GHz, 32-core)
• 512 GB of RAM (16x 32 GB DIMMs)
• 1 TB NVMe SSD (OS)

• sometimes the error shows like below when I use 64 cores/node * 3 node = 192 cores

 ===================================================================================
 =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
 =   PID 2328069 RUNNING AT node08
 =   EXIT CODE: 9
 =   CLEANING UP REMAINING PROCESSES
 =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
 ===================================================================================
 [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:911): assert (!closed) failed
 [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
 [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event
 srun: error: node08: task 0: Out Of Memory

I guess this means it really ran out of memory. But as I mentioned above, my cluster nodes are equipped with a large RAM, averagely 8 GB per CPU core. I believe you performed the same large calculation before for the silicon (qmesh = 50 50 50 ), right? Do you think the errors are normal? Besides, may I ask a question? What is the largest number of images have you ever tried?
I have many experiences in MPI but not familiar with coarray, so not have many ideas about also not confident in the HPC errors I met with coarray or elphbolt.

Many thanks to you and your suggestions are very helpful!!

Best,
Shiyu

[nakib]

Actually, I have not seen these errors before. Looks like these are all MPI related. I can try to narrow the problem down if you give me more information about your build.

1. At what stage of the elphbolt calculation are you getting these errors? Can you send me the ouput log?
2. Can you send me your job submission script?
3. Did you build OpenCoarrays with OpenMPI or MPICH? The latter has always worked a bit better for me.
4. Can you rerun your calculation asking for 32 images per node? Do you still get the same error?

As for your other question, the largest job I have run is on 9 nodes x 36 cores/node = 324 cores. I had hyperthreading turned off, if this is relevant.

Best,
Nakib

[alex2shiyu]

Actually, I compiled openCoarrays with MVAPICH which I remember is based on MPICH. So I did not doubt the validity. Anyway, I will try MPICH and then see what's going on. If still the same problems, I will provide more detailed information as you suggested.
I appreciate your continuous help.

Best,
Shiyu