Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop future allocations when job is deleted #24669

Closed
geekodour opened this issue Dec 14, 2024 · 3 comments
Closed

Stop future allocations when job is deleted #24669

geekodour opened this issue Dec 14, 2024 · 3 comments
Assignees

Comments

@geekodour
Copy link

I am unsure why this is not the default behavior, why doe we not cancel all pending and running allocation if the job itself is stopped and purged.

Eg. I have deleted the main job, i have these orphan allocations running around. Why this this even considered? I cannot think of any usecase that this behaviour would be useful.

image

@geekodour
Copy link
Author

sometimes I get pending values in negatives, i am not sure what's going on.

image

I am not sure if this is a bug or intended behavior etc so not able to get to the root of it

@geekodour
Copy link
Author

image

Also https://developer.hashicorp.com/nomad/api-docs/jobs#list-job-allocations seems to not work with batch job (it seems)

apologies for dumping all of the issues here, I'll create separate issues later.

@tgross
Copy link
Member

tgross commented Dec 16, 2024

@geekodour

I am unsure why this is not the default behavior, why doe we not cancel all pending and running allocation if the job itself is stopped and purged.

This is the default behavior, although for architectural reasons the allocations will be running on the clients briefly (depending on kill timeouts and the like). But I suspect you're seeing inconsistency in the UI. Can you verify the behavior is the same when hitting the API via the CLI?

Re-reading your post here now that I'm somewhere I can read screenshots... the allocations you're seeing pending are the allocations of child jobs. Removing the base job doesn't remove running child jobs because once they've been dispatched they're treated as entirely separate jobs with their own lifecycle. This is what allows the scheduler to handle allocation failures for each child independently.

The nomad job stop command does take a list of jobs though, so you can do something like nomad job status | awk -F' +' '/whatever/{print $1}' | nomad job stop as a workaround to tear down all the child jobs.

@tgross tgross self-assigned this Dec 16, 2024
@tgross tgross closed this as not planned Won't fix, can't repro, duplicate, stale Dec 16, 2024
@github-project-automation github-project-automation bot moved this from Needs Triage to Done in Nomad - Community Issues Triage Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

2 participants