[FEAT] Add steps to spin up, submit job, and spin down ray clusters

[raunakab]

Overview

• new steps that:
  • spin up
  • submit job
  • spin down ray clusters

Note

If any of the previous steps fail, the "tear-down" step (responsible for tearing down the ray cluster), will still always run. (The only way this tear-down step would not be run is if the workflow is manually cancelled).

[codspeed-hq]

CodSpeed Performance Report

Merging #3403 will degrade performances by 12.24%

_{Comparing feat/run-2 (1d1cc20) with main (31a7abc)}

Summary

⚡ 1 improvements
❌ 1 regressions
✅ 15 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	feat/run-2	Change
❌	test_iter_rows_first_row[100 Small Files]	278.6 ms	317.5 ms	-12.24%
⚡	test_show[100 Small Files]	23.8 ms	15.6 ms	+52.72%

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.13%. Comparing base (5b36bd0) to head (1d1cc20).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3403      +/-   ##
==========================================
- Coverage   77.35%   76.13%   -1.22%     
==========================================
  Files         684      684              
  Lines       83627    85253    +1626     
==========================================
+ Hits        64688    64911     +223     
- Misses      18939    20342    +1403     

see 23 files with indirect coverage changes

---- 🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[jaychia]

Not sure, but might need to check if this background process will get correctly terminated when the workflow finishes.

[raunakab]

Maybe one way to check is by running 2 subsequent jobs on the same machine.

If the background job is not killed after the first WF run terminates, then it will affect the second WF run when it tries to use port 8265.