-
|
Hi, Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 75 replies
-
|
@caoshiyi this depends strongly on the architecture and system topology of the cluster in question. |
Beta Was this translation helpful? Give feedback.
-
|
@hthayko Thanks! My email is [email protected]. |
Beta Was this translation helpful? Give feedback.
-
|
@caoshiyi just want to give you a heads up that we are not affiliated with @hthayko. |
Beta Was this translation helpful? Give feedback.
-
|
Yup - we are a separate entity. Thanks for clarifying @tlubowe! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @hthayko, would it be possible to try this platform? I am trying to get cuQuantum multi node to work on OCI with Slurm. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @acoracle - you are welcome to try out our platform at app.bluequbit.io |
Beta Was this translation helpful? Give feedback.
-
The
This operation involves a nontrivial traversal pattern in the elements of In terms of implementation, there is no overlap unless the number of targets ( |
Beta Was this translation helpful? Give feedback.
-
|
@mtjrider I see. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Let me know if you have other questions. |
Beta Was this translation helpful? Give feedback.
-
|
@tlubowe Regarding this discussion, is there something straightforward that can be done on a cluster with 2 DGX A100 where the cuQuantum Appliance 22.11 container is loaded? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @leofang I tried it with the same container as before and on 2 nodes. This is the output: |
Beta Was this translation helpful? Give feedback.
-
|
@erinaldi It looks like our UCX build is not supported by Perlmutter's Slingshot interconnect, so the OpenMPI library provided in the container will not work across the node boundary. Our team is currently working on plugging in Perlmutter's native Cray-MPICH library to our container, we will keep you posted. @mtjrider for visibility. |
Beta Was this translation helpful? Give feedback.
-
|
Updates on CUDA-aware MPICH soon. I'm sure if you unloaded MPICH modules and used native OpenMPI, it'd probably work(?) This would be a very slow solution though as the implementation wouldn't be GPU aware and our libraries wouldn't support it IUUC. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for isolating the problem @mtjrider and @DmitryLyakh |
Beta Was this translation helpful? Give feedback.
-
|
Hi @erinaldi,
I've escalated with Perlmutter and I'm working with them directly on this. |
Beta Was this translation helpful? Give feedback.
-
|
@mtjrider @tlubowe do you have some experience for running the distributed simulation on the Perlmutter supercomputer cluster? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. We have access, too. |
Beta Was this translation helpful? Give feedback.
-
|
@mtjrider can you share with me some scripts for doing this on Perlmutter? I tried to run it with the shifter container but I am not so sure what's the proper way to launch the jobs so that they can communicate among nodes in the container. My email is [email protected], if discussing through emails can be more convenient for you. Thanks very much! |
Beta Was this translation helpful? Give feedback.
-
|
Having a multi-node container solution on Perlmutter is also relevant for me, so I’d appreciate any hint. |
Beta Was this translation helpful? Give feedback.
-
|
@erinaldi Also, please try adding the following flags to your srun command to see if it solves your problem: |
Beta Was this translation helpful? Give feedback.
-
|
First try with |
Beta Was this translation helpful? Give feedback.
-
|
List of MPI plugins on the node: |
Beta Was this translation helpful? Give feedback.
-
|
run test with specific MPI plugin |
Beta Was this translation helpful? Give feedback.
-
|
This was the trick @DmitryLyakh , selecting the |
Beta Was this translation helpful? Give feedback.
-
|
@erinaldi after much waiting, an update. All of the following on Perlmutter ... EDIT: Be sure that this variable is set: Do the following to pull the image and confirm it was downloaded: It may take a second, but you should see something like this: Once you confirm that you have the correct image (above) pulled, you can use the attached scripts to run a multi-node job using the cuQuantum appliance. You will have to make changes so that the account name, qos, etc. are appropriately set. Once modifications are done, I recommend running |
Beta Was this translation helpful? Give feedback.
-
|
Hi @mtjrider, I tried your procedure and cuQuantum Appliance 23.03 works fine. But when I tried to run it with cuQuantum Appliance 23.10 (simply by replacing each occurrence of The result of I also tried cuQuantum Appliance 24.03 but it failed to download. Is there anything that needs to be configured for cuQuantum Appliance 23.10? |
Beta Was this translation helpful? Give feedback.
-
|
Let me also add here: I have been stuck to using 23.03 on Perlmutter. 23.10 didn’t work and the staff at NERSC said something was broken in the container. Perhaps 24.03 is the lucky one? |
Beta Was this translation helpful? Give feedback.
-
|
@xumingkuan there is not cuQuantum Appliance 24.03. There is only a new 24.03 version of cuQuantum SDK. |
Beta Was this translation helpful? Give feedback.
-
|
I have seen this error now on version 23.10 |
Beta Was this translation helpful? Give feedback.
-
|
The error is not there when I replace 23.10 with 23.06 |
Beta Was this translation helpful? Give feedback.
-
|
@mtjrider What is the best way to run a distributed statevector simulation with cusvaer in the simulator = Aer.get_backend('aer_simulator_statevector')
# create options dictionary to override communicator settings and set precision
options = {
'cusvaer_comm_plugin_type': cusvaer.CommPluginType.MPI_MPICH, # use MPICH MPI
'cusvaer_comm_plugin_soname': 'libmpi.so', # name of the MPI shared library
'precision': 'single', # single precision
'shots': 128
}
simulator.set_options(**options)(in the container, Runs in 21.04753017425537 seconds using 2 MPI processes.
backend: cusvaer_simulator_statevector
Runs in 2.753363847732544 seconds using 4 MPI processes.
backend: cusvaer_simulator_statevectorHowever, if I change the circuit to save the state vector using circuit = read_circuit(f)
circuit.save_statevector()can I access the distributed statevector? E.g. I would normally do this result = simulator.run(circ).result()
psi = result.data(0)["statevector"]and compute amplitudes? Typically with the Born rule state = int(bit_str, 2)
prob_bit_str = abs(psi[state]) ** 2 |
Beta Was this translation helpful? Give feedback.
-
|
@mtjrider is there an example to run on multiple MPI processes and collect certain bit string amplitudes? |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps there is access to this functionality through some qiskit primitives, like in this discussion? #117 |
Beta Was this translation helpful? Give feedback.
-
|
A (very) clunky solution is to label the states on each sub-vector by checking the rank of the process where |
Beta Was this translation helpful? Give feedback.
-
|
I found a better solution by redefining by In case you have a more efficient suggestion I am open to hear it, but for now this seems to be a solution that works for me and, at least on a node up to 4 processes, it gives the correct results. |
Beta Was this translation helpful? Give feedback.
-
Hi @erinaldiq, |
Beta Was this translation helpful? Give feedback.
@erinaldi after much waiting, an update.
All of the following on Perlmutter ...
EDIT: Be sure that this variable is set:
export MPICH_GPU_SUPPORT_ENABLED=1Do the following to pull the image and confirm it was downloaded:
It may take a second, but you should see something like this:
Once you confirm that you have the correct image (above) pulled, you can use the attached scripts to run a multi-node job using the cuQuantum appliance. You will have to make changes so th…