ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE

[Vipermdl]

Hi, I`m facing the problem with training:
(py36) [madongliang@compute-0-5 experiments]$ sh train.sh
Traceback (most recent call last):
Traceback (most recent call last):
File "../mmdetection/tools/train.py", line 8, in
File "../mmdetection/tools/train.py", line 8, in
Traceback (most recent call last):
File "../mmdetection/tools/train.py", line 8, in
Traceback (most recent call last):
File "../mmdetection/tools/train.py", line 8, in
Traceback (most recent call last):
Traceback (most recent call last):
File "../mmdetection/tools/train.py", line 8, in
Traceback (most recent call last):
from mmdet.apis import (train_detector, init_dist, get_root_logger,
from mmdet.apis import (train_detector, init_dist, get_root_logger,
File "../mmdetection/tools/train.py", line 8, in
File "../mmdetection/tools/train.py", line 8, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
from mmdet.apis import (train_detector, init_dist, get_root_logger,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
from mmdet.apis import (train_detector, init_dist, get_root_logger,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
Traceback (most recent call last):
File "../mmdetection/tools/train.py", line 8, in
from mmdet.apis import (train_detector, init_dist, get_root_logger,
from mmdet.apis import (train_detector, init_dist, get_root_logger,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
from mmdet.apis import (train_detector, init_dist, get_root_logger,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
from mmdet.apis import (train_detector, init_dist, get_root_logger,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
from .train import train_detector
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from mmdet.core import (DistOptimizerHook, DistEvalmAPHook,
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .post_processing import * # noqa: F401, F403
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .post_processing import * # noqa: F401, F403
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in
from .bbox_nms import multiclass_nms
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
from .bbox_nms import multiclass_nms
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
from .bbox_nms import multiclass_nms
from .bbox_nms import multiclass_nms
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
from .bbox_nms import multiclass_nms
from .bbox_nms import multiclass_nms
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
from .bbox_nms import multiclass_nms
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
from .bbox_nms import multiclass_nms
from mmdet.ops.nms import nms_wrapper
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
from mmdet.ops.nms import nms_wrapper
from mmdet.ops.nms import nms_wrapper
from mmdet.ops.nms import nms_wrapper
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
from mmdet.ops.nms import nms_wrapper
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
from mmdet.ops.nms import nms_wrapper
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
from mmdet.ops.nms import nms_wrapper
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
from .roi_align import RoIAlign, roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
from mmdet.ops.nms import nms_wrapper
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in
from .roi_align import RoIAlign, roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
from .roi_align import RoIAlign, roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
from .roi_align import RoIAlign, roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
from .roi_align import RoIAlign, roi_align
from .roi_align import RoIAlign, roi_align
from .roi_align import RoIAlign, roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
from .functions.roi_align import roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .roi_align import RoIAlign, roi_align
from .functions.roi_align import roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .functions.roi_align import roi_align
from .functions.roi_align import roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .functions.roi_align import roi_align
from .functions.roi_align import roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .functions.roi_align import roi_align
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .. import roi_align_cuda
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
from .. import roi_align_cuda
from .functions.roi_align import roi_align
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
from .. import roi_align_cuda
from .. import roi_align_cuda
from .. import roi_align_cuda
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in
from .. import roi_align_cuda
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
from .. import roi_align_cuda
from .. import roi_align_cuda
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE
ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE

I`m using CUDA 9.0, pytorch 1.0.1.post2, python 3.6
Everything compiled well during installation.

Thanks!

[libuyu]

According to my experience, this error happens when the cuda tools are not linked properly in the conda environment or the cuda version is not compatible with the code. I used to solve this problem by creating a totally new conda environment.
For more suggestions, you can refer to open-mmlab/mmdetection#385 .

[ruiyuanlu]

Try

import torch

at the very first in your code. If it doesn't work, maybe this comment can help you.

[Vipermdl]

According to my experience, this error happens when the cuda tools are not linked properly in the conda environment or the cuda version is not compatible with the code. I used to solve this problem by creating a totally new conda environment.
For more suggestions, you can refer to open-mmlab/mmdetection#385 .

thanks, this is working.