Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Compile lstm.onnx failed. #441

Open
LeiWang1999 opened this issue Jun 5, 2022 · 1 comment
Open

[BUG] Compile lstm.onnx failed. #441

LeiWang1999 opened this issue Jun 5, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@LeiWang1999
Copy link
Contributor

🐛 Bug

I have a converted lstm model located in /data1/v-leiwang3/benchmark/nnfusion_models/lstm.float32.onnx, which can be inferenced by onnxruntime, but I got a compiled failed issue with nnfusion.

To Reproduce

nnfusion /workspace/v-leiwang3/benchmark/nnfusion_models/lstm.float32.onnx -f onnx -p batch_size:1;seq_length:512 -fwarmup_step=5 -frun_step=100
[WARNING] 2022-06-05T05:12:58z src/contrib/custom_op/custom_op.h 27	$NNFUSION_HOME was not set, use /root/.nnfusion.
[WARNING] 2022-06-05T05:12:58z src/contrib/custom_op/custom_op.h 27	$NNFUSION_HOME was not set, use /root/.nnfusion.

============================================================================
---- Processing '/workspace/v-leiwang3/benchmark/nnfusion_models/lstm.float32.onnx'
============================================================================
[INFO] 2022-06-05T05:12:58z src/nnfusion/frontend/onnx_import/onnx.cpp 54	Optimizing ONNX Graph with External Tool (models/pytorch2onnx/ort_run_frozen.py)
[WARNING] 2022-06-05T05:12:58z src/nnfusion/common/util.cpp 47	$NNFUSION_HOME was not set, use /root/.nnfusion.
ONNX model check passed!
Importing ONNX model into ONNX Runtime...
Execution Providers: ['CUDAExecutionProvider', 'CPUExecutionProvider']
output
[-0.00914649  0.01211212  0.03566384  0.02083438  0.02440321  0.02316982
  0.01730602  0.00079239 -0.01367441 -0.01027629] ...(size= 100 end with -0.010356168 )
[INFO] 2022-06-05T05:13:02z src/nnfusion/frontend/onnx_import/onnx.cpp 40	Import ONNX Graph Size: [256593293]
[INFO] 2022-06-05T05:13:02z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 317	Converting Onnx Graph
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Transpose_6
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Transpose_6, output 45, shape Shape{512, 1, 256}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_15
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_15, output 111, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_16
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_16, output 114, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_25
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_25, output 179, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_26
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_26, output 182, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_35
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_35, output 247, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_36
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_36, output 250, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_45
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_45, output 315, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_46
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_46, output 318, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_55
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_55, output 383, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_56
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_56, output 386, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_65
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_65, output 451, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_66
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_66, output 454, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_75
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_75, output 519, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_76
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_76, output 522, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: LSTM_85
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node LSTM_85, output 587, shape Shape{512, 1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_86
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_86, output 590, shape Shape{512, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Slice_91
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Slice_91, output Slice_91, shape Shape{1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Transpose
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Transpose, output 595, shape Shape{1, 1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Squeeze_92
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Squeeze_92, output 596, shape Shape{1, 1024}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: Gemm_93
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node Gemm_93, output output, shape Shape{1, 100}
[INFO] 2022-06-05T05:13:04z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 514	convert graph done
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/engine.cpp 36	Graph Passes count:27
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/engine.cpp 37	Interpreter Passes count:6
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/pass/graph/batchnorm_inference_folding_pass.cpp 889	batchnorm inference folding Pass starts up for Graph: Graph_1
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/pass/graph/batchnorm_inference_folding_pass.cpp 901	batchnorm inference folding Pass ends for Graph: Graph_1
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[INFO] 2022-06-05T05:13:04z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:input(op type: Parameter, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_15(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_25(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_35(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_45(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_55(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_65(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_75(op type: Lstm, dev type: CUDA_GPU)
[ERROR] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:LSTM_85(op type: Lstm, dev type: CUDA_GPU)
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:input
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Reshape_67 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 45 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_15
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 114 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_25
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 182 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_35
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 250 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_45
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 318 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_55
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 386 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_65
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 454 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_75
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 522 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:LSTM_85
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 590 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 595 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator 596 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Dot_66 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Result_69 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node _linear.bias
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node Reshape_67
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node _linear.weight
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 586
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 582
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 759
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 758
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 757
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 518
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 514
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 739
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 738
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 737
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 450
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 446
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 719
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 718
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 717
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 382
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 378
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 699
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 698
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 697
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 314
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 310
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 679
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 678
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 677
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 246
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 242
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 659
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 658
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 657
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 178
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 174
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 639
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 638
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 637
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 110
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 106
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 619
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 618
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 617
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node input
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 45
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_15
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 114
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_25
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 182
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_35
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 250
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_45
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 318
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_55
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 386
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_65
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 454
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_75
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 522
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node LSTM_85
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 590
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node Slice_91
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 595
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node 596
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node Dot_66
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node Add_68
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 100	processed node Result_69
[INFO] 2022-06-05T05:13:05z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 213	assign thread info-------------------------------
[ERROR] 2022-06-05T05:13:05z src/nnfusion/util/errors.hpp 169	Check failed: '(*gnode)["Kernel_Selection_Result"].is_valid()' at /workspace/v-leiwang3/nnfusion/src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp:926:
Kernel should be selected before this pass:Lstm kernel name : LSTM_15
terminate called after throwing an instance of 'nnfusion::errors::CheckError'
  what():  Check failed: '(*gnode)["Kernel_Selection_Result"].is_valid()' at /workspace/v-leiwang3/nnfusion/src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp:926:
Kernel should be selected before this pass:Lstm kernel name : LSTM_15

some additional debug logs has been added by myself to find the problem, It seems that something stucked at lstm node, any suggestion?
@LeiWang1999 LeiWang1999 added the bug Something isn't working label Jun 5, 2022
@LeiWang1999
Copy link
Contributor Author

LeiWang1999 commented Jun 5, 2022 •
edited
Loading

I think the problem is because nnfusion has no Lstm Op Cuda_GPU DeviceType kernel, because there is no lstm.cpp under folder core/kernels/cuda_gpu/kernels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@LeiWang1999