Self-hosted runner (AMD mi250 CI caller) #423
Triggered via workflow run
December 15, 2023 14:05
amyeroberts
amyeroberts
completed
e2b6df7
Status
Failure
Total duration
18m 0s
Artifacts
–
self-push-amd-mi250-caller.yml
on: workflow_run
AMD mi250
/
Check Runner Status
7s
Matrix: AMD mi250 / Check Runners
Matrix: AMD mi250 / Setup
Matrix: AMD mi250 / Model tests
AMD mi250
/
Send results to webhook
26s
Annotations
2 errors and 4 warnings
|
AMD mi250 / Check Runners (single-gpu)
System.IO.IOException: No space left on device : '/opt/github-runners/github-runner4/_diag/Worker_20231215-140612-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/opt/github-runners/github-runner4/_diag/Worker_20231215-140612-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at GitHub.Runner.Common.Tracing.Error(Exception exception)
at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Unhandled exception. System.IO.IOException: No space left on device : '/opt/github-runners/github-runner4/_diag/Worker_20231215-140612-utc.log'
at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at System.Diagnostics.TraceSource.Flush()
at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
at GitHub.Runner.Common.TraceManager.Dispose()
at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
at GitHub.Runner.Common.HostContext.Dispose()
at GitHub.Runner.Worker.Program.Main(String[] args)
|
|
AMD mi250 / Check Runners (multi-gpu)
The job was canceled because "single-gpu" failed.
|
|
AMD mi250 / Check Runners (single-gpu)
You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB
|
|
AMD mi250 / Check Runners (single-gpu)
Delete stale container networks failed, docker network prune fail with exit code 1
|
|
AMD mi250 / Check Runners (multi-gpu)
Runner hf-amd-mi250-ci-2gpu-2 did not respond to a cancelation request with 00:05:00.
|
|
AMD mi250 / Check Runners (multi-gpu)
You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB
|