Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: Use github actions to cache numba kernels #278

Open
sjperkins opened this issue Jun 15, 2023 · 0 comments · Fixed by #279
Open

Suggestion: Use github actions to cache numba kernels #278

sjperkins opened this issue Jun 15, 2023 · 0 comments · Fixed by #279
Labels
enhancement New feature or request

Comments

@sjperkins
Copy link
Member

Describe the problem that the feature should address

Currently CI is taking an increasing amount of time to run because all numba kernels have to be compiled from scratch.

In conversation with @JSKenyon, it seems most, if not all kernels are decorated with cache=True. This implies that its not necessary to recompile kernels from scratch all the time.

Describe the solution you'd like

Numba respects a NUMBA_CACHE_DIR variable, while github actions has access to per-repository-caches.

A github action cache key can be constructed from shell/environment variables/hashing files and this is used to determine cache hits/misses. e.g. numba-cache-abcdef1234567890. However, I suspect that we don't really want to manage these hits and misses ourselves -- we really just want some path that we can point NUMBA_CACHE_DIR at and let numba handle individual kernel cache hits/misses according to it's internal logic.

If the above follows, then it should be sufficient to use a well structured key (i.e. prefixed with numba-cache-) in conjunction with the actions/cache@v3 restore-keys to restore the most recent numba kernel cache and construct a new one during the current CI run.

This should speed up CI runs to the degree that code changes propagate to all numba code -- a change fundamental to all numba kernels will recompile the entire code base, while a peripheral one might just change one kernel.

@sjperkins sjperkins added the enhancement New feature or request label Jun 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant