[QUIC] Disable QUIC tests on CLR stress tests #96392

ManickaP · 2024-01-02T06:21:26Z

I'm not sure about disabling the tests completely on Arm32, any opinions @wfurt @rzikm @CarnaViire

ghost · 2024-01-02T06:21:38Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

I'm not sure about disabling the tests completely on Arm32, any opinions @wfurt @rzikm @CarnaViire

Author:	ManickaP
Assignees:	-
Labels:	`area-System.Net.Quic`
Milestone:	-

stephentoub · 2024-01-02T15:28:04Z

src/libraries/System.Net.Quic/tests/FunctionalTests/MsQuicCipherSuitesPolicyTests.cs

@@ -10,6 +10,7 @@ namespace System.Net.Quic.Tests
 {
    [Collection(nameof(DisableParallelization))]
    [ConditionalClass(typeof(QuicTestBase), nameof(QuicTestBase.IsSupported))]
+    [SkipOnCoreClr("MsQuic is timing sensitive and flaky under stress", ~RuntimeTestModes.RegularRun)]


There's no way to make the tests resilient against timing issues? What specifically is the problem that leads to the flakiness?

We suspect #55979 as root cause. There is logic in MsQuic to watch load and start refusing connection if CPU utilization is too high. For normal tests runs we disabled parallel runs to avoid it but the stress may be still be too much.

To start with, we may disable stress only on ARM. It seems like it failed on ARM64 as well but it is hard to tell much as most of the links leads to nowhere now ;(

ManickaP · 2024-01-03T10:21:16Z

I did a bit digging in Kusto:

jit stress re-runs/failures are only on Arm32v7 --> I can narrow down the exclusion
the most re-runs/failures are on libraries-pgo Arm32v7, but the last one was on 22.10.
out of ~4700 re-runs/failures ~4500 are Arm32v7
there's no way to know how much passes we got on Arm32v7 so no idea about the percentage

Query:

AzureDevOpsTests
| where TestName has "System.Net.Quic" and Repository == "dotnet/runtime" and Message has "The connection timed out from inactivity"
| summarize count(), max(RunCompleted) by TestRunName, BuildDefinitionName

CarnaViire · 2024-01-03T10:47:24Z

Thanks @ManickaP.

the most re-runs/failures are on libraries-pgo Arm32v7, but the last one was on 22.10.

I wonder if that was #91757 (it was mitigated around Nov 13)

I think we should disable QUIC tests on jitstress runs on Arm32 and Arm64 (because Arm64 was also reported there) but keep them for other platforms.

I'd vote for not disabling Arm32 completely yet, if we don't see failures in our normal runs since Nov 13...

ManickaP · 2024-01-03T10:50:00Z

So I don't know how to combine SkipOnCoreClr and PlatformDetection in AND manner (i.e. to not exclude all Arm32 test runs). The only way I can see now is to copy the logic from https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.XUnitExtensions/src/CoreClrConfigurationDetection.cs and use ConditionalClass with my own condition for ClrStress run on Arm32.

Is there an easier way @ViktorHofer, @akoeplinger, @BruceForstall? Is this even worth the effort @wfurt @rzikm @CarnaViire?

ManickaP · 2024-01-03T12:30:34Z

because Arm64 was also reported there

Interestingly, there are no such records in Kusto.

BruceForstall · 2024-01-03T17:59:06Z

Is there an easier way...

I'm certainly no expert in libraries test exclusions, but I do believe there is no "AND" functionality, as you've found.

BruceForstall · 2024-01-03T17:59:49Z

btw, thank you for working on this!

wfurt · 2024-01-03T18:13:33Z

because Arm64 was also reported there

Interestingly, there are no such records in Kusto.

there may be other occasional failures on other platforms. I think it would be OK to exclude arm32 since that is was majority and wait. We may need to create new static variable for the condition - especially if we want to be more selective.

ManickaP · 2024-01-04T05:34:34Z

/azp list

azure-pipelines · 2024-01-04T05:34:40Z

CI/CD Pipelines for this repository: runtime-coreclr outerloop runtime-coreclr jitstress runtime-coreclr jitstressregs runtime-coreclr jitstress2-jitstressregs runtime-coreclr gcstress0x3-gcstress0xc runtime-coreclr gcstress-extra runtime-coreclr r2r-extra runtime-coreclr jitstress-isas-x86 runtime-coreclr jitstress-isas-arm runtime-coreclr jitstressregs-x86 runtime-coreclr libraries-jitstressregs runtime-coreclr libraries-jitstress2-jitstressregs runtime-coreclr r2r runtime-coreclr runincontext runtime-coreclr crossgen2 runtime-libraries-coreclr outerloop runtime-libraries-coreclr outerloop-windows runtime-libraries-coreclr outerloop-linux runtime-libraries-coreclr outerloop-osx runtime runtime-libraries enterprise-linux runtime-libraries stress-http runtime-libraries stress-ssl runtime-dev-innerloop runtime-coreclr crossgen2 outerloop coreclr-release-outerloop-nightly runtime-coreclr crossgen2-composite runtime-jit-experimental runtime-coreclr libraries-jitstress dotnet-linker-tests runtime-coreclr ilasm runtime-coreclr crossgen2-composite gcstress runtime-staging runtime-coreclr pgo runtime-coreclr libraries-pgo Antigen runtime-community Fuzzlyn runtime-coreclr superpmi-replay runtime-wasm runtime-coreclr superpmi-diffs runtime-coreclr superpmi-asmdiffs-checked-release runtime-extra-platforms jit-cfg runtime-wasm-perf runtime-llvm runtime-coreclr jitstress-random runtime-coreclr libraries-jitstress-random runtime-android-grpc-client-tests runtime-wasm-libtests runtime-android runtime-androidemulator runtime-ioslike runtime-ioslikesimulator runtime-linuxbionic runtime-maccatalyst runtime-coreclr pgostress runtime-coreclr jitstress-isas-avx512 runtime-tools-tests runtime-libraries-mono outerloop runtime-sanitized runtime-wasm-dbgtests runtime-nativeaot-outerloop

ManickaP · 2024-01-04T05:36:53Z

/azp run runtime-coreclr libraries-jitstress

azure-pipelines · 2024-01-04T05:37:03Z

Azure Pipelines successfully started running 1 pipeline(s).

ManickaP · 2024-01-04T05:40:43Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2024-01-04T05:40:52Z

Azure Pipelines successfully started running 1 pipeline(s).

ManickaP · 2024-01-04T08:09:12Z

I manually checked the AzDO pipeline:

PR (Arm32 stress is skipped, Arm64 stress still ran)
main (baseline)

And failures are unrelated. Can I get any final reviews?

CarnaViire

LGTM, thanks!

akoeplinger · 2024-01-05T23:36:05Z

So I don't know how to combine SkipOnCoreClr and PlatformDetection in AND manner (i.e. to not exclude all Arm32 test runs). The only way I can see now is to copy the logic from dotnet/arcade@main/src/Microsoft.DotNet.XUnitExtensions/src/CoreClrConfigurationDetection.cs and use ConditionalClass with my own condition for ClrStress run on Arm32.

You don't need to copy the logic since CoreClrConfigurationDetection.IsJitStress is public so you can use it in a custom property in PlatformDetection which is used as the condition in [ConditionalClass].

Disable QUIC tests on CLR stress tests

c66f16c

dotnet-issue-labeler bot added the area-System.Net.Quic label Jan 2, 2024

ghost assigned ManickaP Jan 2, 2024

This was referenced Jan 2, 2024

Tracking issue for CI build timeouts #76454

Closed

[mono][tvos] OOM in System.IO.Tests.MemoryStreamTests #92467

Closed

stephentoub reviewed Jan 2, 2024

View reviewed changes

Exclude only Arm32 CLR stress tests

8a6099a

CarnaViire approved these changes Jan 4, 2024

View reviewed changes

ManickaP merged commit baeb298 into dotnet:main Jan 4, 2024
142 of 149 checks passed

ManickaP deleted the mapichov/quic-clrstress-disable branch January 4, 2024 14:57

github-actions bot locked and limited conversation to collaborators Feb 5, 2024

karelz added this to the 9.0.0 milestone May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QUIC] Disable QUIC tests on CLR stress tests #96392

[QUIC] Disable QUIC tests on CLR stress tests #96392

ManickaP commented Jan 2, 2024

ghost commented Jan 2, 2024

stephentoub Jan 2, 2024

wfurt Jan 2, 2024

ManickaP commented Jan 3, 2024 •

edited

Loading

CarnaViire commented Jan 3, 2024

ManickaP commented Jan 3, 2024

ManickaP commented Jan 3, 2024

BruceForstall commented Jan 3, 2024

BruceForstall commented Jan 3, 2024

wfurt commented Jan 3, 2024

ManickaP commented Jan 4, 2024

azure-pipelines bot commented Jan 4, 2024

ManickaP commented Jan 4, 2024

azure-pipelines bot commented Jan 4, 2024

ManickaP commented Jan 4, 2024

azure-pipelines bot commented Jan 4, 2024

ManickaP commented Jan 4, 2024

CarnaViire left a comment

akoeplinger commented Jan 5, 2024

[QUIC] Disable QUIC tests on CLR stress tests #96392

[QUIC] Disable QUIC tests on CLR stress tests #96392

Conversation

ManickaP commented Jan 2, 2024

ghost commented Jan 2, 2024

stephentoub Jan 2, 2024

Choose a reason for hiding this comment

wfurt Jan 2, 2024

Choose a reason for hiding this comment

ManickaP commented Jan 3, 2024 • edited Loading

CarnaViire commented Jan 3, 2024

ManickaP commented Jan 3, 2024

ManickaP commented Jan 3, 2024

BruceForstall commented Jan 3, 2024

BruceForstall commented Jan 3, 2024

wfurt commented Jan 3, 2024

ManickaP commented Jan 4, 2024

azure-pipelines bot commented Jan 4, 2024

ManickaP commented Jan 4, 2024

azure-pipelines bot commented Jan 4, 2024

ManickaP commented Jan 4, 2024

azure-pipelines bot commented Jan 4, 2024

ManickaP commented Jan 4, 2024

CarnaViire left a comment

Choose a reason for hiding this comment

akoeplinger commented Jan 5, 2024

ManickaP commented Jan 3, 2024 •

edited

Loading