Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization: group actions by callable, create tasks in batches for performance #58

Open
davidshaw-uw opened this issue Dec 21, 2023 · 0 comments

Comments

@davidshaw-uw
Copy link
Contributor

If we conceptualize actions as instances of a given callable, we can group actions by callable to create jobmon tasks in bulk. This might make the task creation faster since we don't have to set upstreams and create metadata for each lone object, but rather in groups.

Idea:

action_map = defaultdict(defaultdict(list)))
for action in self.action_generator():
     action_map[action.name].update(action.kwargs)

for action_name, kwargs in action_map.items():
     task_template = TaskTemplateFactory.get_task_template(action_name)
     task_template.create_tasks(**kwargs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant