You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My system is AMD 3955W, 128 GB RAM, RTX A4000 16GB, Windows 10 WSL2, Ubuntu 20.04.
When I tried to predict a 156 a.a sequence, I got an error message "jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr".
Below is the log. Can someone help me to troubleshoot it?
(alphafold) whelanlab@CHEM-AG007D:~/alphafold$ exp/run_local.sh R1.fasta
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/haiku/_src/data_structures.py:37: FutureWarning: jax.tree_structure is deprecated, and will be removed in a future release. Use jax.tree_util.tree_structure instead.
PyTreeDef = type(jax.tree_structure(None))
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --output_dir has a non-None defau lt value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --model_names has a non-None defa ult value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --data_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --preset has a non-None default v alue; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniref90_database_path has a no n-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --mgnify_database_path has a non- None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniclust30_database_path has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --bfd_database_path has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --pdb70_database_path has a non-N one default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --template_mmcif_dir has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --max_template_date has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/validators.py:203: UserWarning: Flag --obsolete_pdbs_path has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
I0320 18:45:41.543587 139960428058432 templates.py:880] Using precomputed obsolete pdbs /mnt/d/alphafold/database/pdb_mmcif/obsolete.dat.
I0320 18:45:43.169499 139960428058432 xla_bridge.py:166] Remote TPU is not linked into jax; skipping remote TPU.
I0320 18:45:43.169672 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
I0320 18:45:43.511162 139960428058432 xla_bridge.py:413] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm" . Available platform names are: Interpreter Host CUDA
I0320 18:45:43.511674 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I0320 18:45:43.511768 139960428058432 xla_bridge.py:413] Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_clien t. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.
I0320 18:45:51.545218 139960428058432 run_alphafold.py:293] Have 5 models: ['model_1', 'model_2', 'model_3', 'model_4', 'model_5']
I0320 18:45:51.545373 139960428058432 run_alphafold.py:306] Using random seed 4477881255564421494 for the data pipeline
I0320 18:45:51.553825 139960428058432 pipeline.py:130] query uniref90
I0320 18:45:51.554164 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmpydz_0bbz/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/uniref90/uniref90.fasta"
I0320 18:45:51.564941 139960428058432 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I0320 19:45:55.761877 139960428058432 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 3604.197 seconds
I0320 19:45:55.808635 139960428058432 pipeline.py:141] query mgnify
I0320 19:45:55.809039 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmp5t31w7bf/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/mgnify/mgy_clusters.fa"
I0320 19:45:55.817553 139960428058432 utils.py:36] Started Jackhmmer (mgy_clusters.fa) query
I0320 20:44:21.103180 139960428058432 utils.py:40] Finished Jackhmmer (mgy_clusters.fa) query in 3505.285 seconds
I0320 20:44:21.107893 139960428058432 pipeline.py:153] query mgnify
I0320 20:44:21.108175 139960428058432 hhblits.py:128] Launching subprocess "hhblits -i R1.fasta -cpu 4 -oa3m /tmp/tmpasn145w/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /mnt/d/alphafold/database/bfd/bfd_metaclust_clu_complete_id30_c9 0_final_seq.sorted_opt -d /mnt/d/alphafold/database/uniclust30/uniclust30_2018_08/uniclust30_2018_08"
I0320 20:44:21.117413 139960428058432 utils.py:36] Started HHblits query
I0321 02:22:28.651200 139960428058432 utils.py:40] Finished HHblits query in 20287.534 seconds
I0321 02:22:28.785565 139960428058432 hhsearch.py:76] Launching subprocess "hhsearch -i /tmp/tmp5es22b99/query.a3m -o /tmp/tmp5es22b99/output.hhr -maxseq 1000000 -d /mnt/d/alphafold/database/pdb70/pdb70"
I0321 02:22:28.794022 139960428058432 utils.py:36] Started HHsearch query
I0321 02:54:02.204876 139960428058432 utils.py:40] Finished HHsearch query in 1893.411 seconds
I0321 02:54:02.476578 139960428058432 templates.py:892] Searching for template for: None
W0321 02:54:02.476918 139960428058432 templates.py:139] Template structure not in release dates dict: 1ivz
I0321 02:54:02.477041 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ivz.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SSSSQHFNLNFTITNLPYSQDIAQPSTTKYQQTKRSIENALNQLFRNSSIKSYFSDCQVLAFRSVSNNNNHTGVDSLCNFSPLARRVDRVAIYEEFLRMTHNGTQLLNFTLDRKSVFVDSGP
I0321 02:54:04.145149 139960428058432 templates.py:278] Found an exact template match 1ivz_A.
W0321 02:54:04.168012 139960428058432 templates.py:139] Template structure not in release dates dict: 2e7v
I0321 02:54:04.168263 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2e7v.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: KAYFYHSSFQILNVEYTEALNSPATHEYRTLSERIEAMITDEFRGSSLKSEFIRTHVVKLRKEGTGVVADVVMKFRSSKRNNRKVMKTRIQSVLRRLSSSGNLEIAPSNEI
I0321 02:54:04.227076 139960428058432 templates.py:278] Found an exact template match 2e7v_A.
W0321 02:54:04.230510 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsb
I0321 02:54:04.230618 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsb.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.350127 139960428058432 templates.py:278] Found an exact template match 6bsb_A.
W0321 02:54:04.353108 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsc
I0321 02:54:04.353233 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsc.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.616760 139960428058432 templates.py:278] Found an exact template match 6bsc_A.
W0321 02:54:04.619688 139960428058432 templates.py:139] Template structure not in release dates dict: 2acm
I0321 02:54:04.619805 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2acm.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SFFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:05.585400 139960428058432 templates.py:278] Found an exact template match 2acm_A.
W0321 02:54:05.598183 139960428058432 templates.py:139] Template structure not in release dates dict: 5yl9
I0321 02:54:05.598412 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/5yl9.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: VVEQYNQTILNLTSEISTLENKSAELNYTVQKLQTLIDNI
I0321 02:54:05.712731 139960428058432 templates.py:278] Found an exact template match 5yl9_B.
W0321 02:54:05.715018 139960428058432 templates.py:139] Template structure not in release dates dict: 1gvp
I0321 02:54:05.715114 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1gvp.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFGS
I0321 02:54:05.777992 139960428058432 templates.py:278] Found an exact template match 1gvp_A.
W0321 02:54:05.780884 139960428058432 templates.py:139] Template structure not in release dates dict: 6s8s
I0321 02:54:05.780977 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6s8s.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FDFEGNLALFDKAAVFEEIDTY
I0321 02:54:06.046435 139960428058432 templates.py:278] Found an exact template match 6s8s_B.
W0321 02:54:06.049705 139960428058432 templates.py:139] Template structure not in release dates dict: 1ae3
I0321 02:54:06.049842 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ae3.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFG
I0321 02:54:06.119893 139960428058432 templates.py:278] Found an exact template match 1ae3_A.
W0321 02:54:06.122766 139960428058432 templates.py:139] Template structure not in release dates dict: 4v23
I0321 02:54:06.122855 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/4v23.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: AFKYIKPQSQFIVDLGAY
I0321 02:54:06.287062 139960428058432 templates.py:278] Found an exact template match 4v23_A.
I0321 02:54:06.426746 139960428058432 run_alphafold.py:185] Running model model_1
WARNING:tensorflow:From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0321 02:54:06.770174 139960428058432 deprecation.py:528] From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from te nsorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
2023-03-21 02:54:08.766418: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_G ROWTH environment variable is set. Original config value was 0.
I0321 02:54:09.424343 139960428058432 model.py:132] Running predict with shape(feat) = {'aatype': (4, 156), 'residue_index': (4, 156), 'seq_length': (4,), 'template_aatype': (4, 4, 156), 'template_all_atom_masks': (4, 4, 156, 37), 'template_all_atom_positions': (4, 4, 156, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 156), 'msa_mask': (4, 508, 156), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_m ask': (4, 4), 'template_pseudo_beta': (4, 4, 156, 3), 'template_pseudo_beta_mask': (4, 4, 156), 'atom14_atom_exists': (4, 156, 14), 'residx_atom14_to_atom 37': (4, 156, 14), 'residx_atom37_to_atom14': (4, 156, 37), 'atom37_atom_exists': (4, 156, 37), 'extra_msa': (4, 5120, 156), 'extra_msa_mask': (4, 5120, 1 56), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 156), 'true_msa': (4, 508, 156), 'extra_has_deletion': (4, 5120, 156), 'extra_deletion_value': (4, 5120, 156), 'msa_feat': (4, 508, 156, 49), 'target_feat': (4, 156, 22)}
2023-03-21 02:54:09.995058: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.2.1 bu t source was compiled with: 8.6.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgra de your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configur ation.
2023-03-21 02:54:09.997237: E external/org_tensorflow/tensorflow/compiler/xla/status_macros.cc:54] INTERNAL: RET_CHECK failure (external/org_tensorflow/te nsorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
*** Begin stack trace ***
Traceback (most recent call last):
File "/home/whelanlab/alphafold/run_alphafold.py", line 338, in
app.run(main)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/home/whelanlab/alphafold/run_alphafold.py", line 310, in main
predict_structure(
File "/home/whelanlab/alphafold/run_alphafold.py", line 192, in predict_structure
prediction_result = model_runner.predict(processed_feature_dict)
File "/home/whelanlab/alphafold/alphafold/model/model.py", line 134, in predict
result = self.apply(self.params, jax.random.PRNGKey(0), feat)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/random.py", line 136, in PRNGKey
key = prng.seed_with_impl(impl, seed)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 269, in seed_with_impl
return random_seed(seed, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 581, in random_seed
return random_seed_p.bind(seeds_arr, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 593, in random_seed_impl
base_arr = random_seed_impl_base(seeds, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 598, in random_seed_impl_base
return seed(seeds)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 833, in threefry_seed
lax.shift_right_logical(seed, lax_internal._const(seed, 32)))
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/lax/lax.py", line 513, in shift_right_logical
return shift_right_logical_p.bind(x, y)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 122, in apply_primitive
compiled_fun = xla_primitive_callable(prim, *unsafe_map(arg_spec, args),
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 254, in wrapper
return cached(config._trace_context(), *args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 247, in cached
return f(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 201, in xla_primitive_callable
compiled = _xla_callable_uncached(lu.wrap_init(prim_fun), device, None,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 357, in _xla_callable_uncached
return computation.compile(_allow_propagation_to_outputs=allow_prop).unsafe_call
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3373, in compile
executable = self._compile_unloaded(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3344, in _compile_unloaded
return UnloadedMeshExecutable.from_hlo(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3626, in from_hlo
xla_executable = dispatch.compile_or_get_cached(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1091, in compile_or_get_cached
return backend_compile(backend, serialized_computation, compile_options,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/profiler.py", line 314, in wrapper
return func(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1036, in backend_compile
return backend.compile(built_c, compile_options=options)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
The text was updated successfully, but these errors were encountered:
My system is AMD 3955W, 128 GB RAM, RTX A4000 16GB, Windows 10 WSL2, Ubuntu 20.04.
When I tried to predict a 156 a.a sequence, I got an error message "jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr".
Below is the log. Can someone help me to troubleshoot it?
(alphafold) whelanlab@CHEM-AG007D:~/alphafold$ exp/run_local.sh R1.fasta
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/haiku/_src/data_structures.py:37: FutureWarning: jax.tree_structure is deprecated, and will be removed in a future release. Use jax.tree_util.tree_structure instead.
PyTreeDef = type(jax.tree_structure(None))
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --output_dir has a non-None defau lt value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --model_names has a non-None defa ult value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --data_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --preset has a non-None default v alue; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniref90_database_path has a no n-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --mgnify_database_path has a non- None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniclust30_database_path has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --bfd_database_path has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --pdb70_database_path has a non-N one default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --template_mmcif_dir has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --max_template_date has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/validators.py:203: UserWarning: Flag --obsolete_pdbs_path has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
I0320 18:45:41.543587 139960428058432 templates.py:880] Using precomputed obsolete pdbs /mnt/d/alphafold/database/pdb_mmcif/obsolete.dat.
I0320 18:45:43.169499 139960428058432 xla_bridge.py:166] Remote TPU is not linked into jax; skipping remote TPU.
I0320 18:45:43.169672 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
I0320 18:45:43.511162 139960428058432 xla_bridge.py:413] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm" . Available platform names are: Interpreter Host CUDA
I0320 18:45:43.511674 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I0320 18:45:43.511768 139960428058432 xla_bridge.py:413] Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_clien t. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.
I0320 18:45:51.545218 139960428058432 run_alphafold.py:293] Have 5 models: ['model_1', 'model_2', 'model_3', 'model_4', 'model_5']
I0320 18:45:51.545373 139960428058432 run_alphafold.py:306] Using random seed 4477881255564421494 for the data pipeline
I0320 18:45:51.553825 139960428058432 pipeline.py:130] query uniref90
I0320 18:45:51.554164 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmpydz_0bbz/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/uniref90/uniref90.fasta"
I0320 18:45:51.564941 139960428058432 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I0320 19:45:55.761877 139960428058432 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 3604.197 seconds
I0320 19:45:55.808635 139960428058432 pipeline.py:141] query mgnify
I0320 19:45:55.809039 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmp5t31w7bf/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/mgnify/mgy_clusters.fa"
I0320 19:45:55.817553 139960428058432 utils.py:36] Started Jackhmmer (mgy_clusters.fa) query
I0320 20:44:21.103180 139960428058432 utils.py:40] Finished Jackhmmer (mgy_clusters.fa) query in 3505.285 seconds
I0320 20:44:21.107893 139960428058432 pipeline.py:153] query mgnify
I0320 20:44:21.108175 139960428058432 hhblits.py:128] Launching subprocess "hhblits -i R1.fasta -cpu 4 -oa3m /tmp/tmpasn145w/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /mnt/d/alphafold/database/bfd/bfd_metaclust_clu_complete_id30_c9 0_final_seq.sorted_opt -d /mnt/d/alphafold/database/uniclust30/uniclust30_2018_08/uniclust30_2018_08"
I0320 20:44:21.117413 139960428058432 utils.py:36] Started HHblits query
I0321 02:22:28.651200 139960428058432 utils.py:40] Finished HHblits query in 20287.534 seconds
I0321 02:22:28.785565 139960428058432 hhsearch.py:76] Launching subprocess "hhsearch -i /tmp/tmp5es22b99/query.a3m -o /tmp/tmp5es22b99/output.hhr -maxseq 1000000 -d /mnt/d/alphafold/database/pdb70/pdb70"
I0321 02:22:28.794022 139960428058432 utils.py:36] Started HHsearch query
I0321 02:54:02.204876 139960428058432 utils.py:40] Finished HHsearch query in 1893.411 seconds
I0321 02:54:02.476578 139960428058432 templates.py:892] Searching for template for: None
W0321 02:54:02.476918 139960428058432 templates.py:139] Template structure not in release dates dict: 1ivz
I0321 02:54:02.477041 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ivz.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SSSSQHFNLNFTITNLPYSQDIAQPSTTKYQQTKRSIENALNQLFRNSSIKSYFSDCQVLAFRSVSNNNNHTGVDSLCNFSPLARRVDRVAIYEEFLRMTHNGTQLLNFTLDRKSVFVDSGP
I0321 02:54:04.145149 139960428058432 templates.py:278] Found an exact template match 1ivz_A.
W0321 02:54:04.168012 139960428058432 templates.py:139] Template structure not in release dates dict: 2e7v
I0321 02:54:04.168263 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2e7v.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: KAYFYHSSFQILNVEYTEALNSPATHEYRTLSERIEAMITDEFRGSSLKSEFIRTHVVKLRKEGTGVVADVVMKFRSSKRNNRKVMKTRIQSVLRRLSSSGNLEIAPSNEI
I0321 02:54:04.227076 139960428058432 templates.py:278] Found an exact template match 2e7v_A.
W0321 02:54:04.230510 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsb
I0321 02:54:04.230618 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsb.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.350127 139960428058432 templates.py:278] Found an exact template match 6bsb_A.
W0321 02:54:04.353108 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsc
I0321 02:54:04.353233 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsc.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.616760 139960428058432 templates.py:278] Found an exact template match 6bsc_A.
W0321 02:54:04.619688 139960428058432 templates.py:139] Template structure not in release dates dict: 2acm
I0321 02:54:04.619805 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2acm.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SFFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:05.585400 139960428058432 templates.py:278] Found an exact template match 2acm_A.
W0321 02:54:05.598183 139960428058432 templates.py:139] Template structure not in release dates dict: 5yl9
I0321 02:54:05.598412 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/5yl9.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: VVEQYNQTILNLTSEISTLENKSAELNYTVQKLQTLIDNI
I0321 02:54:05.712731 139960428058432 templates.py:278] Found an exact template match 5yl9_B.
W0321 02:54:05.715018 139960428058432 templates.py:139] Template structure not in release dates dict: 1gvp
I0321 02:54:05.715114 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1gvp.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFGS
I0321 02:54:05.777992 139960428058432 templates.py:278] Found an exact template match 1gvp_A.
W0321 02:54:05.780884 139960428058432 templates.py:139] Template structure not in release dates dict: 6s8s
I0321 02:54:05.780977 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6s8s.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FDFEGNLALFDKAAVFEEIDTY
I0321 02:54:06.046435 139960428058432 templates.py:278] Found an exact template match 6s8s_B.
W0321 02:54:06.049705 139960428058432 templates.py:139] Template structure not in release dates dict: 1ae3
I0321 02:54:06.049842 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ae3.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFG
I0321 02:54:06.119893 139960428058432 templates.py:278] Found an exact template match 1ae3_A.
W0321 02:54:06.122766 139960428058432 templates.py:139] Template structure not in release dates dict: 4v23
I0321 02:54:06.122855 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/4v23.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: AFKYIKPQSQFIVDLGAY
I0321 02:54:06.287062 139960428058432 templates.py:278] Found an exact template match 4v23_A.
I0321 02:54:06.426746 139960428058432 run_alphafold.py:185] Running model model_1
WARNING:tensorflow:From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0321 02:54:06.770174 139960428058432 deprecation.py:528] From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from te nsorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
2023-03-21 02:54:08.766418: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_G ROWTH environment variable is set. Original config value was 0.
I0321 02:54:09.424343 139960428058432 model.py:132] Running predict with shape(feat) = {'aatype': (4, 156), 'residue_index': (4, 156), 'seq_length': (4,), 'template_aatype': (4, 4, 156), 'template_all_atom_masks': (4, 4, 156, 37), 'template_all_atom_positions': (4, 4, 156, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 156), 'msa_mask': (4, 508, 156), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_m ask': (4, 4), 'template_pseudo_beta': (4, 4, 156, 3), 'template_pseudo_beta_mask': (4, 4, 156), 'atom14_atom_exists': (4, 156, 14), 'residx_atom14_to_atom 37': (4, 156, 14), 'residx_atom37_to_atom14': (4, 156, 37), 'atom37_atom_exists': (4, 156, 37), 'extra_msa': (4, 5120, 156), 'extra_msa_mask': (4, 5120, 1 56), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 156), 'true_msa': (4, 508, 156), 'extra_has_deletion': (4, 5120, 156), 'extra_deletion_value': (4, 5120, 156), 'msa_feat': (4, 508, 156, 49), 'target_feat': (4, 156, 22)}
2023-03-21 02:54:09.995058: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.2.1 bu t source was compiled with: 8.6.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgra de your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configur ation.
2023-03-21 02:54:09.997237: E external/org_tensorflow/tensorflow/compiler/xla/status_macros.cc:54] INTERNAL: RET_CHECK failure (external/org_tensorflow/te nsorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
*** Begin stack trace ***
*** End stack trace ***
Traceback (most recent call last):
File "/home/whelanlab/alphafold/run_alphafold.py", line 338, in
app.run(main)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/home/whelanlab/alphafold/run_alphafold.py", line 310, in main
predict_structure(
File "/home/whelanlab/alphafold/run_alphafold.py", line 192, in predict_structure
prediction_result = model_runner.predict(processed_feature_dict)
File "/home/whelanlab/alphafold/alphafold/model/model.py", line 134, in predict
result = self.apply(self.params, jax.random.PRNGKey(0), feat)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/random.py", line 136, in PRNGKey
key = prng.seed_with_impl(impl, seed)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 269, in seed_with_impl
return random_seed(seed, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 581, in random_seed
return random_seed_p.bind(seeds_arr, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 593, in random_seed_impl
base_arr = random_seed_impl_base(seeds, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 598, in random_seed_impl_base
return seed(seeds)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 833, in threefry_seed
lax.shift_right_logical(seed, lax_internal._const(seed, 32)))
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/lax/lax.py", line 513, in shift_right_logical
return shift_right_logical_p.bind(x, y)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 122, in apply_primitive
compiled_fun = xla_primitive_callable(prim, *unsafe_map(arg_spec, args),
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 254, in wrapper
return cached(config._trace_context(), *args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 247, in cached
return f(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 201, in xla_primitive_callable
compiled = _xla_callable_uncached(lu.wrap_init(prim_fun), device, None,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 357, in _xla_callable_uncached
return computation.compile(_allow_propagation_to_outputs=allow_prop).unsafe_call
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3373, in compile
executable = self._compile_unloaded(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3344, in _compile_unloaded
return UnloadedMeshExecutable.from_hlo(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3626, in from_hlo
xla_executable = dispatch.compile_or_get_cached(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1091, in compile_or_get_cached
return backend_compile(backend, serialized_computation, compile_options,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/profiler.py", line 314, in wrapper
return func(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1036, in backend_compile
return backend.compile(built_c, compile_options=options)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
The text was updated successfully, but these errors were encountered: