Optimal parallelization on small cluster

[kerim371]

Hi,

I started using JUDI with small test cluster on the could.
For that purpose I have one task manager node and two compute nodes.
Each node have (given from lcpu command):

• 2 CPUs
• 1 socket
• 1 core per socket
• 2 threads per socket
  Cluster manager is SLURM.

I think best parallelization would be if I manage to use each of two computing nodes per task (shot) and enble OpenMP. So there would be 2 shot processing at a time and each node uses two CPU to process a shot.

How should look the configuration?
For example I use fwi_example_NLopt.jl and my settings for now are:

ENV["DEVITO_LANGUAGE"]="openmp"  # Enable OpenMP. 
ENV["OMP_NUM_THREADS"]="2"       # Number of OpenMP threads

using Distributed, ClusterManagers
addprocs(SlurmManager(4), partition="debug", t="01:00:00")

but I believe this is not correct as there probably would be 4 workers while OpenMP enabled.
That means 4 shots are processed at the time and each process tries to use OpenMP with 2 cores - not optimal as I have only 4 cores (not 8).

[mloubout]

So in general yes you would have one julia worker per node and then have multi-threading (openmp) for each julia worker on each node. So in your test case you have two nodes. It means you should do addprocs(2, ...) to add two workers (one per node) or with slurm addprocs(SlurmManager(2), ...). The OMP_NUM_THREADS will define the number of core used by each node. You might need to forward the environment as well.

Just remember that addprocs only manages the number of julia workers irrespective for the hardware/available resources, not the number of cores.

If you want good performance, you may wanna use ThreadPinning on each worker for optimal core allocation.

[kerim371]

Thank you for hints.

Yet can't understand how to make it work OpenMP along with Distributed workers.
I use addprocs(SlurmManager(2)) and export DEVITO_LANGUAGE=openmp; export OMP_NUM_THREADS=2.
But when I watch CPU usage of each node I can see that only one CPU of each worker is loaded:

Even if I don't add workers and do the computations on master node with ENV["OMP_NUM_THREADS"]="2" then I also feel like only single CPU is working:

You said something about forward the environment (OMP_NUM_THREADS): could you please explain what you mean?

And thank you for ThreadPinning, I will try it.

lscpu command shows:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    2
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 106
Model name:            Intel Xeon Processor (Icelake)
Stepping:              0
CPU MHz:               1995.312
BogoMIPS:              3990.62
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd rsb_ctxsw ibrs ibpb stibp ibrs_enhanced fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear spec_ctrl intel_stibp arch_capabilities

[mloubout]

How small is your debug problem? If it's a small 2D, the propagator itself probably runs in O(ms) so you probably won't see it

[kerim371]

Yes it is 2D fwi_example_NLopt.jl with added procs at the top of the script and @everywhere function f!(x,grad) at bottom. It uses overthrust_model.h5

[kerim371]

@mloubout I tried to run on master node without workers with OMP enabled and got warnings:

/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c: In function ‘forward’:
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:53:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
     #pragma omp parallel num_threads(nthreads)
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:55:0: warning: ignoring #pragma omp for [-Wunknown-pragmas]
       #pragma omp for schedule(dynamic,1)
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:58:0: warning: ignoring #pragma omp simd [-Wunknown-pragmas]
         #pragma omp simd aligned(damp,m,u:64)
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:70:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
     #pragma omp parallel num_threads(nthreads_nonaffine)
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:73:0: warning: ignoring #pragma omp for [-Wunknown-pragmas]
       #pragma omp for schedule(dynamic,chunk_size)
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:87:0: warning: ignoring #pragma omp atomic [-Wunknown-pragmas]
           #pragma omp atomic update
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:93:0: warning: ignoring #pragma omp atomic [-Wunknown-pragmas]
           #pragma omp atomic update
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:99:0: warning: ignoring #pragma omp atomic [-Wunknown-pragmas]
           #pragma omp atomic update
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:105:0: warning: ignoring #pragma omp atomic [-Wunknown-pragmas]
           #pragma omp atomic update
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:115:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
     #pragma omp parallel num_threads(nthreads_nonaffine)
 ^
/tmp/devito-jitcache-uid1000/c0918bc7e19461f778d36a46f442c60d09b34112.c:118:0: warning: ignoring #pragma omp for [-Wunknown-pragmas]
       #pragma omp for schedule(dynamic,chunk_size)

It seems I have OpenMP library:

[kerim@master-fwi ~]$ /sbin/ldconfig -p | grep gomp
        libgomp.so.1 (libc6,x86-64) => /lib64/libgomp.so.1
[kerim@master-fwi ~]$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I think devito does't pass the right compiler flag.

OS CentOS 7

[mloubout]

Your GCC version is too old you need at the very least 5.0 for openmp, 9 or 10 would be better

[kerim371]

Ok I will do that! Thank you!

[kerim371]

@mloubout hi,

You were right about GCC version.
GCC 4.8.5 is too old to run OpenMP but 10.2 is pretty fine.
The parallelization works.