Original error: UNIMPLEMENTED: DNN library is not found.

[phanxuanphucnd]

The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling transformers.utils.move_cache().
Moving 5 files to the new cache system
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:04<00:00, 1.04it/s]
0%| | 0/5015 [00:00<?, ?it/s]2022-10-13 06:34:12.209025: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:421] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.2.4. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-10-13 06:34:12.962264: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:421] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.2.4. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-10-13 06:34:12.974684: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:421] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.2.4. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
0%| | 0/5015 [00:02<?, ?it/s]
Traceback (most recent call last):
File "run.py", line 24, in
output = model.vqa(image, question)
File "/home/phucpx/EVJVQA/uio/runner.py", line 297, in vqa
generate_image=False, num_decodes=num_decodes)
File "/home/phucpx/EVJVQA/uio/runner.py", line 190, in run
return_all_decodes=True
File "/home/phucpx/EVJVQA/uio/model.py", line 388, in predict_batch_with_aux
mutable=['cache'])
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
return fun(*args, **kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 1247, in apply
)(variables, *args, **kwargs, rngs=rngs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/core/scope.py", line 865, in wrapper
y = fn(root, *args, **kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 1689, in scope_fn
return fn(module.clone(parent=scope), *args, **kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 402, in wrapped_module_method
return self._call_wrapped_method(fun, args, kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 705, in _call_wrapped_method
y = fun(self, *args, **kwargs)
File "/home/phucpx/EVJVQA/uio/network.py", line 1054, in call
image_decoder_tokens = self.discrete_vae.get_codebook_indices(image_decoder_targets, vae_decode) # 0 is the start token.
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 402, in wrapped_module_method
return self._call_wrapped_method(fun, args, kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 705, in _call_wrapped_method
y = fun(self, *args, **kwargs)
File "/home/phucpx/EVJVQA/uio/network.py", line 348, in get_codebook_indices
h = self.encoder(x, training)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 402, in wrapped_module_method
return self._call_wrapped_method(fun, args, kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 705, in _call_wrapped_method
y = fun(self, *args, **kwargs)
File "/home/phucpx/EVJVQA/uio/network.py", line 207, in call
name='conv_in')(x)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 402, in wrapped_module_method
return self._call_wrapped_method(fun, args, kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 705, in _call_wrapped_method
y = fun(self, *args, **kwargs)
File "/home/phucpx/EVJVQA/uio/t5x_layers.py", line 540, in call
precision=self.precision)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/lax/convolution.py", line 165, in conv_general_dilated
preferred_element_type=preferred_element_type)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/core.py", line 328, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/core.py", line 331, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/core.py", line 698, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 113, in apply_primitive
**params)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/util.py", line 222, in wrapper
return cached(config._trace_context(), *args, **kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/util.py", line 215, in cached
return f(*args, **kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 197, in xla_primitive_callable
prim.name, donated_invars, False, *arg_specs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 343, in _xla_callable_uncached
keep_unused, *arg_specs).compile().unsafe_call
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 980, in compile
**self.compile_args)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 1137, in from_xla_computation
host_callbacks)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 1055, in compile_or_get_cached
host_callbacks)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/profiler.py", line 313, in wrapper
return func(*args, **kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/jax/_src/dispatch.py", line 994, in backend_compile
return backend.compile(built_c, compile_options=options)
jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: UNKNOWN: Failed to determine best cudnn convolution algorithm for:
%cudnn-conv = (f32[1,256,256,128]{2,1,3,0}, u8[0]{0}) custom-call(f32[1,256,256,3]{2,1,3,0} %copy, f32[3,3,3,128]{1,0,2,3} %copy.1), window={size=3x3 pad=1_1x1_1}, dim_labels=b01f_01io->b01f, custom_call_target="__cudnn$convForward", metadata={op_name="jit(conv_general_dilated)/jit(main)/conv_general_dilated[window_strides=(1, 1) padding=((1, 1), (1, 1)) lhs_dilation=(1, 1) rhs_dilation=(1, 1) dimension_numbers=ConvDimensionNumbers(lhs_spec=(0, 3, 1, 2), rhs_spec=(3, 2, 0, 1), out_spec=(0, 3, 1, 2)) feature_group_count=1 batch_group_count=1 lhs_shape=(1, 256, 256, 3) rhs_shape=(3, 3, 3, 128) precision=None preferred_element_type=None]" source_file="/home/phucpx/EVJVQA/uio/t5x_layers.py" source_line=540}, backend_config="{"conv_result_scale":1,"activation_mode":"0","side_input_scale":0}"

Original error: UNIMPLEMENTED: DNN library is not found.

To ignore this failure and try to use a fallback algorithm (which may have suboptimal performance), use XLA_FLAGS=--xla_gpu_strict_conv_algorithm_picker=false. Please also file a bug for the root cause of failing autotuning.

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

---

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "run.py", line 24, in
output = model.vqa(image, question)
File "/home/phucpx/EVJVQA/uio/runner.py", line 297, in vqa
generate_image=False, num_decodes=num_decodes)
File "/home/phucpx/EVJVQA/uio/runner.py", line 190, in run
return_all_decodes=True
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 402, in wrapped_module_method
return self._call_wrapped_method(fun, args, kwargs)
File "/home/phucpx/miniconda3/envs/phucpx/lib/python3.7/site-packages/flax/linen/module.py", line 705, in _call_wrapped_method
y = fun(self, *args, **kwargs)
File "/home/phucpx/EVJVQA/uio/model.py", line 388, in predict_batch_with_aux
mutable=['cache'])
File "/home/phucpx/EVJVQA/uio/network.py", line 1054, in call
image_decoder_tokens = self.discrete_vae.get_codebook_indices(image_decoder_targets, vae_decode) # 0 is the start token.
File "/home/phucpx/EVJVQA/uio/network.py", line 348, in get_codebook_indices
h = self.encoder(x, training)
File "/home/phucpx/EVJVQA/uio/network.py", line 207, in call
name='conv_in')(x)
File "/home/phucpx/EVJVQA/uio/t5x_layers.py", line 540, in call
precision=self.precision)
jaxlib.xla_extension.XlaRuntimeError: UNKNOWN: Failed to determine best cudnn convolution algorithm for:
%cudnn-conv = (f32[1,256,256,128]{2,1,3,0}, u8[0]{0}) custom-call(f32[1,256,256,3]{2,1,3,0} %copy, f32[3,3,3,128]{1,0,2,3} %copy.1), window={size=3x3 pad=1_1x1_1}, dim_labels=b01f_01io->b01f, custom_call_target="__cudnn$convForward", metadata={op_name="jit(conv_general_dilated)/jit(main)/conv_general_dilated[window_strides=(1, 1) padding=((1, 1), (1, 1)) lhs_dilation=(1, 1) rhs_dilation=(1, 1) dimension_numbers=ConvDimensionNumbers(lhs_spec=(0, 3, 1, 2), rhs_spec=(3, 2, 0, 1), out_spec=(0, 3, 1, 2)) feature_group_count=1 batch_group_count=1 lhs_shape=(1, 256, 256, 3) rhs_shape=(3, 3, 3, 128) precision=None preferred_element_type=None]" source_file="/home/phucpx/EVJVQA/uio/t5x_layers.py" source_line=540}, backend_config="{"conv_result_scale":1,"activation_mode":"0","side_input_scale":0}"

Original error: UNIMPLEMENTED: DNN library is not found.

o ignore this failure and try to use a fallback algorithm (which may have suboptimal performance), use XLA_FLAGS=--xla_gpu_strict_conv_algorithm_picker=false. Please also file a bug for the root cause of failing autotuning.

Hi there, i got the above error when run with the pretrained model unifedio: xl_100k.bin.
Please, how to fix them?

Thansk all

[chrisc36]

I think that error is caused by the Cudnn library not being installed correctly, it has to be installed into the system manually, see here.