Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[batch] Add job groups to the hailtop.batch interface #14610

Open
daniel-goldstein opened this issue Jul 11, 2024 · 0 comments
Open

[batch] Add job groups to the hailtop.batch interface #14610

daniel-goldstein opened this issue Jul 11, 2024 · 0 comments

Comments

@daniel-goldstein
Copy link
Contributor

daniel-goldstein commented Jul 11, 2024

What happened?

Batch now supports organizing jobs into a hierarchy through job groups, but this is so far only used internally by Query on Batch. Users who want to leverage this structure cannot do so through the hailtop.batch interface. All jobs created by e.g. b.new_job() are implicitly assigned to the "root" job group. We should add the following to hailtop.batch:

  • A JobGroup class to the public interface that shares any relevant methods that currently exist on the Batch class like new_{bash|python}_job, wait, etc.
  • A method on Batch to create a new job group and a method on JobGroup to create a child job group
  • A JobGroup should not have a run method, because there is no sound way in the lower-level batch client currently to only submit a subset of staged jobs/groups and it's unclear whether that behavior would ever be desired.

Version

0.2.132

Relevant log output

No response

@daniel-goldstein daniel-goldstein added new-feature needs-triage A brand new issue that needs triaging. batch and removed needs-triage A brand new issue that needs triaging. labels Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant