Model does not run correctly on non A100/H100 GPUs

[joshabramson]

A note from us at Google DeepMind:

We have now tested accuracy on V100 and there are serious issues with the output (looks like random noise). Users have reported similar issues with RTX 2060S and RTX Quadro 4000.

For now the only supported and tested devices are A100 and H100.

[aozalevsky]

if you're interested in stats, works fine on

NVIDIA GeForce RTX 3080, Driver Version: 535.183.01

[smg3d]

Could you be specific about what was not working on your side on the V100? How to recognize that there is a problem? Is non-sense structure THE indicator of numerical inaccuracies?

The mentioned post with non-sense structures on Quadro 4000 and RTX 2060S were not done in your docker environment...

Because on my side, predictions look perfect on an old Quadro P3000 6GB for several proteins and ligand complexes (i.e. on a 6 years-old Thinkpad laptop and a mobile GPU with compute < 8.0),. Also works great on RTX-3090.

Other than non-sense structures, what other observation could indicate that we have numerical inaccuracy? Is there a controlled test we could do to identify potential numerical inaccuracy in our setup?

[joshabramson]

The nonsense structure is the indicator of the problem here - output will look almost random. The problem appears related to bfloat16, which is not supported on older GPU. We will continue to investigate next week.

Interesting to know that it does work on some older GPU, thanks for the report. Even if the major issue under investigation here isn't present, please note we have not done any large scale numerical verification of outputs on devices other than A100/H100.

[smg3d]

Thank you for the precision @joshabramson. I will watch for "exploded" structures, and report the specifics if ever it happens on one of my GPUs.

The P3000 definitely does not support natively BF16 (CUDA capability 6.1). I guess it emulates it via float32 compute.

Since it is quite probable that several people will try to run AF3 on their available hardware, here are some details of my setup where it works perfect so far.

Number of tokens (12 runs so far on that GPU) : 167-334 tokens, so largest bucket size tested was 512.

Largest test:
334 tokens (Running model inference for seed 1 took 618.33 seconds.) (yup... slow, but it works... while driving two 4K monitors)

Typical inference speed for < 256 tokens : 150-190 seconds per seed (so typically less than 3 minutes for < 256 tokens)

GPU : Quadro P3000, Pascal architecture, Computer Capability = 6.1 (ThinkPad P71 laptop)

Docker : default setup, NOT using unified memory

run_alphafold.py option : --flash_attention_implementation=xla

nvidia-smi

Sat Nov 16 20:53:47 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro P3000                   Off |   00000000:01:00.0  On |                  N/A |
| N/A   59C    P0             52W /   75W |    5593MiB /   6144MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1672      G   /usr/lib/Xorg                                 519MiB |
|    0   N/A  N/A      1963      G   cinnamon                                      195MiB |
|    0   N/A  N/A      2581      G   ...seed-version=20241115-050104.422000         90MiB |
|    0   N/A  N/A     52689      C   python                                       4750MiB |
+-----------------------------------------------------------------------------------------+

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Sep_12_02:18:05_PDT_2024
Cuda compilation tools, release 12.6, V12.6.77
Build cuda_12.6.r12.6/compiler.34841621_0

deviceQuery

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro P3000"
  CUDA Driver Version / Runtime Version          12.6 / 12.6
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 6032 MBytes (6325141504 bytes)
  (10) Multiprocessors, (128) CUDA Cores/MP:     1280 CUDA Cores
  GPU Max Clock rate:                            1215 MHz (1.22 GHz)
  Memory Clock rate:                             3504 Mhz
  Memory Bus Width:                              192-bit
  L2 Cache Size:                                 1572864 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.6, CUDA Runtime Version = 12.6, NumDevs = 1, Device0 = Quadro P3000
Result = PASS

neofetch

                   -`                    ***@neo
                  .o+`                   -------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Host: 20HKCTO1WW ThinkPad P71 
               `+oooooo:                 Kernel: 6.6.60-1-lts 
               -+oooooo+:                Uptime: 3 hours, 39 mins 
             `/:-:++oooo+:               Packages: 1900 (pacman), 9 (flatpak) 
            `/++++/+++++++:              Shell: bash 5.2.37 
           `/++++++++++++++:             Resolution: 3840x2160, 3840x2160 
          `/+++ooooooooooooo/`           DE: Cinnamon 6.2.9 
         ./ooosssso++osssssso+`          WM: Mutter (Muffin) 
        .oossssso-````/ossssss+`         WM Theme: cinnamon (Adapta) 
       -osssssso.      :ssssssso.        Theme: Adwaita-dark [GTK2/3] 
      :osssssss/        osssso+++.       Icons: Faenza [GTK2/3] 
     /ossssssss/        +ssssooo/-       Terminal: terminator 
   `/ossssso+/:-        -:/+osssso+-     CPU: Intel i7-7820HQ (8) @ 2.900GHz 
  `+sso+:-`                 `.-/+oso:    GPU: NVIDIA Quadro P3000 Mobile 
 `++:.                           `-/+/   Memory: 8323MiB / 64093MiB 
 .`                                 `/

[jurgjn]

We ran the "2PV7" example from the docs on all GPU models available on our cluster with the following results:

gpu	ranking_score	driver_ver	cuda
rtx_2080_ti	-99.68	535.183.06	12.2
rtx_3090	0.67	550.127.05	12.4
rtx_4090	0.67	550.127.05	12.4
titan_rtx	-99.78	550.127.05	12.4
quadro_rtx_6000	-99.74	550.90.07	12.4
v100	-99.78	550.127.05	12.4
a100_pcie_40gb	0.67	550.127.05	12.4
a100_80gb	0.67	550.127.05	12.4

Specifically, a ranking score of -99 corresponds to noise/explosion, and a ranking score of 0.67 corresponds to a visually compelling output structure.

Update (20.11): added driver/cuda versions reported by nvidia-smi.

[Augustin-Zidek]

Thanks @jurgjn, this is incredibly useful information!

These are the GPU capabilities (see https://developer.nvidia.com/cuda-gpus) for the GPUs mentioned:

rtx_2080_ti      7.5  (bad)
rtx_3090         8.6
rtx_4090         8.9
titan_rtx        7.5  (bad)
quadro_rtx_6000  7.5  (bad)
v100             7.0  (bad)
a100_pcie_40gb   8.0
a100_80gb        8.0

Looks like anything with GPU capability < 8.0 produces bad results.

[Augustin-Zidek]

Quick update: I've pushed a08cffd that makes AlphaFold 3 fail if run on a GPU with capability < 8.0 to prevent people getting bad results by surprise.

In case you want to test with old GPUs, just remove the check added in a08cffd.

[lucajovine]

Just to add one more piece of info, I am using a RTX A6000 (capability 8.6) and so far all looks well.

[gwirn]

RTX A5000 (capability 8.6) works well too

[Augustin-Zidek]

Could more people test with capability 6.x?

Based on the result above from @smg3d, it looks that maybe only capability 7.x is broken, while 6.x (and >8.0) might be fine.

I.e. current theory:

6.0 <= Capability < 7.0  Maybe OK? We need more data.
7.0 <= Capability < 8.0  BAD
8.0 <= Capability        OK

[smg3d]

I wonder if it could be a driver effect? I noticed several people are mentioning they are using older driver. Might be useful to know which driver and Cuda @jurgjn was using on his system.

I was using Driver 560.35.03 and Cuda V12.6.77 (Actually just upgraded to driver 565 today).

[georgkempf]

I could now try AF3 on a Quadro P4000 (Pascal) and like @smg3d reported for P3000, on this GPU it works. This test was done with the same driver and cuda versions (565.57.01, cuda_12.5.r12.5) as the tests on RTX 2060S (Turing) and Quadro RTX 4000 (Turing).

[zhanglzu]

V100 also meet "exploded" structures

NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6

[recorderlegend]

Quadro RTX 8000 also got exploding structures

Driver Version 555.42.06, CUDA Version 12.5

[smg3d]

Could more people test with capability 6.x?

Based on the result above from @smg3d, it looks that maybe only capability 7.x is broken, while 6.x (and >8.0) might be fine.

I.e. current theory:

6.0 <= Capability < 7.0  Maybe OK? We need more data.
7.0 <= Capability < 8.0  BAD
8.0 <= Capability        OK

I can confirm that it runs well on P100 (capability 6.0).

So far it has been confirmed that it runs well on the following 6.x Capability:

• P3000
• P4000
• P100

And so far there has been no reports of "exploded structures" on 6.x Capability.

[smg3d]

I think it would be good for users to be able to use AlphaFold3 on Pascal GPUs (without requiring them to modify code). The data on this issue strongly suggest that the "exploded structures" problem does not affect Pascal GPUs (compute capability 6.x).

Moreover, there are still several clusters with P100s, and these often have 0 or very short wait time (compared to the A100s). For example, on one of the Canadian national clusters, AF3 jobs on P100 currently start immediately, whereas jobs on the A100 (on the same cluster) often have 10-30 minutes wait time in the queue. So for a single inference job on small-medium size protein complexes, we get our predictions back much faster with the P100, despite the inference being ~5x slower (358 sec vs 73 sec on the tested dimer).

I tested and submitted a small PR to allow Pascal GPUs to run without raising the error message.