[Apple][TVOS][release/6.0-staging] Unhandled exception. System.IO.IOException: No space left on device

[richlander]

I am seeing this. Either our tests are doing something wrong, our infra isn't cleaning things up, or someone downloaded too many shows.

Leg: Build tvOSSimulator x64 Release AllSubsets_Mono

   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52
System.IO.IOException: No space left on device : '/Users/runner/runners/3.241.0/_diag/Worker_20240718-051117-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at Microsoft.VisualStudio.Services.Agent.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 81
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at Microsoft.VisualStudio.Services.Agent.Tracing.Error(Exception exception) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/Tracing.cs:line 56
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 66
Unhandled exception. System.IO.IOException: No space left on device : '/Users/runner/runners/3.241.0/_diag/Worker_20240718-051117-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at Microsoft.VisualStudio.Services.Agent.TraceManager.Dispose(Boolean disposing) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/TraceManager.cs:line 66
   at Microsoft.VisualStudio.Services.Agent.TraceManager.Dispose() in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/TraceManager.cs:line 58
   at Microsoft.VisualStudio.Services.Agent.HostContext.Dispose(Boolean disposing) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostContext.cs:line 589
   at Microsoft.VisualStudio.Services.Agent.HostContext.Dispose() in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostContext.cs:line 531
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.Main(String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 24
,##[error]We stopped hearing from agent Azure Pipelines 13. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610
Agent: Azure Pipelines 13
Started: Wed at 10:11 PM
Duration: 1h 54m 35s

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[ivanpovazan]

/cc: @matouskozak

[matouskozak]

I took a look at the current state and I'm not seeing this failure anywhere (last 3 rolling builds).

I see a dip in executed tests between 18-19 of July which could be related to the Azure outage. @richlander do you happen to have a link to the build with this error so I can take a look what happened?

[richlander]

#105063

https://dev.azure.com/dnceng-public/public/_build/results?buildId=745877&view=logs&jobId=cd385208-11d3-5d70-fa38-1c21a65d17f1

[matouskozak]

#105063

https://dev.azure.com/dnceng-public/public/_build/results?buildId=745877&view=logs&jobId=cd385208-11d3-5d70-fa38-1c21a65d17f1

Ah, sorry. I was looking at runtime pipeline and not runtime-staging. Looks like this is a recurring issue on this pipeline. This is a range of commits when the failure started occurring fe40750...4b5fdbc. I wonder if it could be related to the pipeline changes introduced in ac83237.

[richlander]

Sorry for not including the branch name in the title. Will definitely do that next time.