Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tests] Tests fail on gfx1030 for Fused solver for Fwd Convolution with Residual add, Bias add and then activation function (#2517) #2633

Closed
junliume opened this issue Dec 22, 2023 · 1 comment · Fixed by #2635
Assignees

Comments

@junliume
Copy link
Collaborator

junliume commented Dec 22, 2023

After #2517
Navi21, ROCm 6.0, develop 2b9d9cd, "No supported fusion solvers found", make check fails:

 833/1799 Test  #835: ConvFwdBiasActivAPI/ConvFwdBiasResAddActivTest.ConvFusedAPI/(5,  G:1 N:1 C:4 D:14 H:11 W:1 k:4 z:3 y:3 x:3 pad_z:1 pad_y:1 pad_x:1 stride_z:1 stride_y:1 stride_x:1 dilation_z:1 dilation_y:1 dilation_x:1 conv_mode:0, 1, 1, 8) ........................................................................................................................................................................................***Failed    3.48 sec
Running main() from /opt/rocm/cget/build/tmp-c4f1a42bcee536579a1248d25169f834201170c187fc6d80e8c510a7d28f6f48/googletest-1.14.0/googletest/src/gtest_main.cc
Note: Google Test filter = ConvFwdBiasActivAPI/ConvFwdBiasResAddActivTest.ConvFusedAPI/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ConvFwdBiasActivAPI/ConvFwdBiasResAddActivTest
[ RUN      ] ConvFwdBiasActivAPI/ConvFwdBiasResAddActivTest.ConvFusedAPI/0
MIOpen(HIP): Info [GetFindModeValueImpl] MIOPEN_FIND_MODE = DYNAMIC_HYBRID(5)
PRNG seed: 12345678
MIOpen(HIP): Info [get_device_name] Raw device name: gfx1030
MIOpen(HIP): Info [Handle] stream: 0x2654620, device_id: 0
MIOpen(HIP): Info [AmdRocmMetadataVersionDetect] ROCm MD version AMDHSA_COv3, HIP version 6.0.23494, MIOpen version 3.1.0.2b9d9cdbb
MIOpen(HIP): Info [Measure] RamDb::Prefetch time: 0.037761 ms
MIOpen(HIP): Info [TryLoad] Find-db regenerating.
MIOpen(HIP): Info [AllocateBuffersAndMakeFusionInvokeParams] bias addr: 0x7f1f09a05000, size: 8
MIOpen(HIP): Info [AllocateBuffersAndMakeFusionInvokeParams] weight addr: 0x7f1f09a06000, size: 864
MIOpen(HIP): Info [AllocateBuffersAndMakeFusionInvokeParams] in addr: 0x7f1f09a08000, size: 1232
MIOpen(HIP): Info [AllocateBuffersAndMakeFusionInvokeParams] out addr: 0x7f1f09a09000, size: 1232
MIOpen(HIP): Info [GetPerfDbPathFile] Found exact perf database file
MIOpen(HIP): Info [Compile] No supported fusion solvers found
...
/root/github/MLOpen1/test/gtest/fused_conv_bias_res_add_activ.cpp:185: Failure
Expected equality of these values:
  status
    Which is: 8
  miopenStatusSuccess
    Which is: 0

MIOpen(HIP): Info [PrintVersion] HIPRTC v.9.0
/root/github/MLOpen1/test/gtest/fused_conv_bias_res_add_activ.cpp:130: Failure
Value of: error < threshold
  Actual: false
Expected: true
Error beyond tolerance Error:nan,  Threshold: 0.078125

[  FAILED  ] ConvFwdBiasActivAPI/ConvFwdBiasResAddActivTest.ConvFusedAPI/0, where GetParam() = (5,  G:1 N:1 C:4 D:14 H:11 W:1 k:4 z:3 y:3 x:3 pad_z:1 pad_y:1 pad_x:1 stride_z:1 stride_y:1 stride_x:1 dilation_z:1 dilation_y:1 dilation_x:1 conv_mode:0, 1, 1, 8) (3367 ms)

Originally posted by @atamazov in #2517 (comment)

@junliume
Copy link
Collaborator Author

@atamazov reference to another discussion here:
#2517 (comment)

We should skip the test is detected inapplicable, like...

Basically, not. Otherwise the test won't detect the cases when solver's IsApplicable() is broken due to some reason. All the stuff like

if(!solv.IsApplicable(ctx, problem))
    GTEST_SKIP();

should be removed from the tests, unless clearly explained why it is absolutely necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment