add /verifyruns cog #36
Conversation
There was a problem hiding this comment.
There is some time we could save in having a single command verify all of modal, nvidia and amd one after the other and letting us know how many of those 3 succeeded. Also as far as the UX goes, why not have verify actually trigger the bot 3 times instead of having a human manually verify in a thread
| This cog provides functionality to verify that either a GitHub Actions or | ||
| Modal run completed successfully by checking for specific message patterns | ||
| in a Discord thread. It supports verification of two types of runs: | ||
| 1. GitHub Actions runs - Identified by "GitHub Action triggered!" message |
There was a problem hiding this comment.
So this seems to test a trigger but we've had issues where this wouldn't have helped like for example when we had timeout issues with both the NVIDIA and AMD runner
|
OK, thanks, Mark. I'll switch I switched this PR to draft for now because I'm in the middle of re-doing this. |
|
Latest changes launch 3 run jobs. Screenshot below. I find the bot's messages like "Created thread GitHub Job (AMD) - 2024-11-29 1… for your GitHub job" a little distracting, given that the threads also show up as separate lines in the UI. Should we remove those "created thread" messages? @msaroufim, I don't know if this addresses your comment from above ("we've had issues where this wouldn't have helped"). If a GitHub workflow times out, then I think the GitHub cog will just return, and the thread won't have the happy path messages in it, so verification will fail. So I think it addresses your comment, but I'm not fully sure.
|
Yeah should remove those in another PR
Yeah i recently implemented timeouts so they should solve that specific problem |
|
Mark, do you think this needs a final message at the end that says "✅ All tests passed"? |
|
Yeah that sounds nice |
Description
Adds a
/verifyruncommand that does the same work as the smoke test but is easier to use.Checklist
Before submitting this PR, ensure the following steps have been completed:
- Run the cluster bot on your server:
- Start a training run by with the slash command
- Wait for the training run to complete.
- Copy the URL for the thread started by the cluster bot in response to
- Click on the 3 dots (
- Select Copy Message Link.
- Using the copied URL, run the smoke test:
- Verify that the smoke test script responds with:
For more information on running a cluster bot on your own server, see/run.You may need to exercise some judgement about the script and GPU type.
your
/runmessage ("Cluster Bot started a thread: ..."):...) to the cluster bot's message.README.md.