restarted. Azure DevOps can't recover from restarts.

[lewing]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
914462	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111310
914414	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111292
914073	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111229
913931	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111290
913887	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111287
913841	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111247
913835	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111245
913813	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111214
913756	dotnet/runtime	browser-wasm linux Release LibraryTests
913679	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111179
913673	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#110471
913581	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110966
913254	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#110966
911851	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111213
913076	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111269
913066	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110471
913017	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
912981	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110780
912074	dotnet/runtime	browser-wasm linux Release LibraryTests	Log	dotnet/runtime#110281
912864	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111263
912683	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111136
912330	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111179
912107	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#105004
911863	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#106309
911665	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111220
911619	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111225
911601	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110780
911590	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#106309
911501	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#105004
911179	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109493
911284	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111179
911210	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111220
911128	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109207
911131	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
911117	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111219
911020	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111215
910542	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111198
910481	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111152
910999	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111213
910970	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111179
910797	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110281
910778	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111208
910720	dotnet/runtime	browser-wasm linux Release LibraryTests
910698	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111050
907708	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110750
910594	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110727
910444	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111086
910423	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111170
910393	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111093
909202	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111152
2615551	dotnet-dotnet	LinuxBionic_NativeAOT_Shortstack_x64
910198	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111093
910180	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111190
909956	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111086
909416	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111074
909303	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111170
909275	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111081
905509	dotnet/runtime	browser-wasm linux Release LibraryTests	Log	dotnet/runtime#110566
908980	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111015
908868	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111130
908349	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111130
908308	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111086
908210	dotnet/runtime	browser-wasm linux Release LibraryTests
907966	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111010
907946	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111110
907774	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111093
907554	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111050
907417	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111118
907397	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107118
907377	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108732
907320	dotnet/runtime	browser-wasm linux Release LibraryTests
907245	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111081
907239	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111110
907031	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111079
907028	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111101
907022	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111093
906845	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#110966
906741	dotnet/runtime	browser-wasm linux Release LibraryTests
906733	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#110966
906566	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111084
906537	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#111081
906407	dotnet/runtime	wasi-wasm windows Release LibraryTests_Smoke		dotnet/runtime#111080
906293	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111073
906252	dotnet/runtime	browser-wasm linux Release LibraryTests
906180	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#110966
906176	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110229
906022	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111067
905963	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109052
905789	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111053
905727	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111050
905542	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111041
905427	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110778
905400	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110229
2611720	dotnet-dotnet	AzureLinux_x64_Cross_ShortStack_Mono_LLVMAOT_arm64
905366	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110591
905289	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111033
905061	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110842
905015	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111024
904774	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111015
904713	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110966
Displaying 100 of 190 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	77	190

[am11]

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

[am11]

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

[steveisok]

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

[ericstj]

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

[snakex64]

This seems to be happening consistently in PR #109320 . Should I just try re-running a third time or is it never gonna work?