MSBuild crashing in the build

[jkoritzinsky]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=410579
Build error leg or test failing: Build / Mono Product Build windows x64 release / Build product
Pull request: #91768

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Child Node \"[0-9]+\" exited prematurely.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
901796	dotnet/runtime	Build product	Log	#110900
899293	dotnet/runtime	Build Tests	Log	#110827
898428	dotnet/runtime	Build managed test components	Log	#110732
2603970	dotnet-runtime	Build product	Log
896498	dotnet/runtime	Build product	Log
885563	dotnet/runtime	Restore and Build Product	Log
885302	dotnet/runtime	Build product	Log	#109493
2594336	dotnet-runtime	Build product	Log
884077	dotnet/runtime	Build product	Log	#110329

Build	Definition	Test	Pull Request
900389	dotnet/runtime	Workloads-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution
896092	dotnet/runtime	NoWebcil-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution	#110731
895918	dotnet/runtime	NoWebcil-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution	#110655
895618	dotnet/runtime	NoWebcil-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution
895127	dotnet/runtime	Workloads-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution	#110676
888092	dotnet/runtime	Workloads-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution	#109069
883887	dotnet/runtime	Workloads-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution	#109069
882445	dotnet/runtime	Workloads-ST-Wasm.Build.Tests.PInvokeTableGeneratorTests.WorkItemExecution	#109069

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	17

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=410579
Error message validated: Child Node "[0-9]+" exited prematurely.
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/19/2023 6:06:29 PM UTC

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=410579
Build error leg or test failing: Build / Mono Product Build windows x64 release / Build product
Pull request: #91768

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Child Node \"[0-9]+\" exited prematurely.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Author:	jkoritzinsky
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure, untriaged, Known Build Error
Milestone:	-

[ViktorHofer]

190 hits per month

cc @rainersigwald @rokonec

[lewing]

reup for 97 hits per month

[MichalStrehovsky]

We now have more useful information in the logs for these. Two of the available logs have this in it:

2024-02-19T21:13:25.8115210Z MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
2024-02-19T21:13:25.8126770Z MSBUILD : error MSB4166: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/MSBuild_pid-50918_4ee39488d5ca4c7d9fb049f19f43c144.failure.txt:
2024-02-19T21:13:25.8128200Z MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 50918:
2024-02-19T21:13:25.8129250Z MSBUILD : error MSB4166: =====================
2024-02-19T21:13:25.8140670Z MSBUILD : error MSB4166: 2/19/2024 9:13:25 PM
2024-02-19T21:13:25.8241810Z MSBUILD : error MSB4166: System.IO.IOException: No buffer space available
2024-02-19T21:13:25.8260730Z MSBUILD : error MSB4166:  ---> System.Net.Sockets.SocketException (55): No buffer space available
2024-02-19T21:13:25.8363710Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8364490Z MSBUILD : error MSB4166:    --- End of inner exception stack trace ---
2024-02-19T21:13:25.8465500Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8569320Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2024-02-19T21:13:25.8670730Z MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
2024-02-19T21:13:25.8773480Z MSBUILD : error MSB4166: ===================

@rainersigwald does this ring any bell?

[rainersigwald]

All of the recent hits seem to be dotnet/msbuild#10342, which should be fixed in the preview7 GA build. Is there a timeline for updating to that? Anything my team can help with?

[rainersigwald]

2024-02-19T21:13:25.8115210Z MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
2024-02-19T21:13:25.8126770Z MSBUILD : error MSB4166: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/MSBuild_pid-50918_4ee39488d5ca4c7d9fb049f19f43c144.failure.txt:
2024-02-19T21:13:25.8128200Z MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 50918:
2024-02-19T21:13:25.8129250Z MSBUILD : error MSB4166: =====================
2024-02-19T21:13:25.8140670Z MSBUILD : error MSB4166: 2/19/2024 9:13:25 PM
2024-02-19T21:13:25.8241810Z MSBUILD : error MSB4166: System.IO.IOException: No buffer space available
2024-02-19T21:13:25.8260730Z MSBUILD : error MSB4166:  ---> System.Net.Sockets.SocketException (55): No buffer space available
2024-02-19T21:13:25.8363710Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8364490Z MSBUILD : error MSB4166:    --- End of inner exception stack trace ---
2024-02-19T21:13:25.8465500Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8569320Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2024-02-19T21:13:25.8670730Z MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
2024-02-19T21:13:25.8773480Z MSBUILD : error MSB4166: ===================

this one I don't recognize, but I haven't seen it in the current links.

[rainersigwald]

Is it possible to have multiple issues with known problems that match a signature with prioritization? I'd like to create one like at Microsoft.Build.BackEnd.Logging.LoggingContext.LogBuildEvent to bucket the dotnet/msbuild#10342 hits so we can use this one for "everything else that causes an MSBuild worker node crash".

[jkoritzinsky]

You can have multiple known issues I think, but I don't think there's a prioritization mechanism.

I'd recommend having a separate issue to bucket the dotnet/msbuild#10342 hits and leave this one as a catch-all to trigger build retries when a node dies (and we can triage specific failure classes into different known issues if they happen with enough frequency).