mpi run error on Azure cluster

[hovnatan]

I get the following error when I run statevector simulation job on a 2 node Azure cluster (8 GPU each). Any idea what can be the reason?

Traceback (most recent call last):
  File "/mnt/azureml/cr/j/477144ae8b684b2d9cc2712f0b38493d/exe/wd/ghz_cusvaer.py", line 112, in <module>
    run(args.nbits, args.depth, args.precision, args.nnodes, args.ngpus_per_node)
  File "/mnt/azureml/cr/j/477144ae8b684b2d9cc2712f0b38493d/exe/wd/ghz_cusvaer.py", line 83, in run
    job = simulator.run(circuit, shots=1024)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cuquantum-24.08/lib/python3.11/site-packages/cusvaer/backends/statevector_simulator.py", line 464, in run
    result = self._run_job(job_id, qobj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cuquantum-24.08/lib/python3.11/site-packages/cusvaer/backends/statevector_simulator.py", line 502, in _run_job
    result_list.append(self.run_experiment(experiment))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cuquantum-24.08/lib/python3.11/site-packages/cusvaer/backends/statevector_simulator.py", line 646, in run_experiment
    cusvsim.apply_queued_gates()
  File "cusvaer/backends/cusvaerext.pyx", line 209, in cusvaer.backends.cusvaerext.CusvSimulator.apply_queued_gates
cusvaer.backends.ubackendext.RuntimeError: communicator error
[1728919256.397093] [97994c19bae84409b32617f1f411430c000002:1012 :0]           flush.c:56   UCX  ERROR req 0x563f26c69840: error during flush: Connection reset by remote peer
[97994c19bae84409b32617f1f411430c000002:01012] common_ucx.c:472  Error: ucp_disconnect_nb(0) failed: Connection reset by remote peer
[97994c19bae84409b32617f1f411430c000002:01012] mca: base: close: component self closed
[97994c19bae84409b32617f1f411430c000002:01012] mca: base: close: unloading component self

[ymagchi]

Hi @hovnatan,

Could you please share your compute environments and job script?
The error occurred during MPI communication and it could depend on the targeting machine.

[hovnatan]

It is an Azure managed 2 node, 8 A100 GPU each cluster. The environment ups our docker images from https://hub.docker.com/r/bluequbit/multigpu/tags then tries running job `python ghz_cusvaer.py` in it. It automatically inserts `mpirun -np ...` before the command.

…

On Mon, Oct 14, 2024 at 9:07 PM Takuma Yamaguchi ***@***.***> wrote: Hi @hovnatan <https://github.com/hovnatan>, Could you please share your compute environments and job script? The error occurred during MPI communication and it could depend on the targeting machine. — Reply to this email directly, view it on GitHub <#159 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAJIVU5IC5GGZCVGV2GVWTZ3P26BAVCNFSM6AAAAABP5IJWZSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJTHEZTOMA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hovnatan]

Basic test with this script works fine. Can it be some sort of networking limit we are hitting with simulation transfer sizes?

import time

from mpi4py import MPI

comm = MPI.COMM_WORLD

rank = comm.rank
size = comm.size
name = MPI.Get_processor_name()

shared = (rank + 1) * 5

if rank == 0:
    data = shared
    for dest in range(1, 16):
        comm.send(data, dest=dest)
        print("From rank", name, "we sent", data)
    time.sleep(5)

else:
    data = comm.recv(source=0)
    print("on node", name, "we received:", data)

[ymagchi]

Thank you for checking it. There won't be any limitations for simulation transfer sizes.
Could you please run the simulation with an environment variable UCX_LOG_LEVEL=info to see the communication logs?

[hovnatan]

Thanks for your help @ymagchi . I run the same thing with UCX_LOG_LEVEL=debug . Attached are logs.
std_log_process_0.txt

[ymagchi]

Thank you for the log.
I think some environment variables for UCX have been explicitly provided:

[1729594678.699838] [75287eb6cce04f46bb342eefebf1c571000002:608  :0]          parser.c:2045 UCX  INFO  UCX_* env variables: UCX_NET_DEVICES=eth0 UCX_TLS=tcp

Is it possible to run without these restrictions?

[hovnatan]

Thanks @ymagchi! It turned out Azure ML Cluster was setting UCX_NET_DEVICES=eth0 and UCX_TLS=tcp environment variables (I am not sure about the reason though). In the Python code before importing mpi4py I cleared those variables with

os.environ.pop("UCX_NET_DEVICES", None)
os.environ.pop("UCX_TLS", None)

and everything worked fine. It is a slower than the multinode benchmarks from Nvidia official blog, but we are fine with it for now.