You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the problem that the feature should address
Currently CI is taking an increasing amount of time to run because all numba kernels have to be compiled from scratch.
In conversation with @JSKenyon, it seems most, if not all kernels are decorated with cache=True. This implies that its not necessary to recompile kernels from scratch all the time.
A github action cache key can be constructed from shell/environment variables/hashing files and this is used to determine cache hits/misses. e.g. numba-cache-abcdef1234567890. However, I suspect that we don't really want to manage these hits and misses ourselves -- we really just want some path that we can point NUMBA_CACHE_DIR at and let numba handle individual kernel cache hits/misses according to it's internal logic.
If the above follows, then it should be sufficient to use a well structured key (i.e. prefixed with numba-cache-) in conjunction with the actions/cache@v3restore-keys to restore the most recent numba kernel cache and construct a new one during the current CI run.
This should speed up CI runs to the degree that code changes propagate to all numba code -- a change fundamental to all numba kernels will recompile the entire code base, while a peripheral one might just change one kernel.
The text was updated successfully, but these errors were encountered:
Describe the problem that the feature should address
Currently CI is taking an increasing amount of time to run because all numba kernels have to be compiled from scratch.
In conversation with @JSKenyon, it seems most, if not all kernels are decorated with
cache=True
. This implies that its not necessary to recompile kernels from scratch all the time.Describe the solution you'd like
Numba respects a NUMBA_CACHE_DIR variable, while github actions has access to per-repository-caches.
A github action cache key can be constructed from shell/environment variables/hashing files and this is used to determine cache hits/misses. e.g.
numba-cache-abcdef1234567890
. However, I suspect that we don't really want to manage these hits and misses ourselves -- we really just want some path that we can pointNUMBA_CACHE_DIR
at and let numba handle individual kernel cache hits/misses according to it's internal logic.If the above follows, then it should be sufficient to use a well structured key (i.e. prefixed with
numba-cache-
) in conjunction with theactions/cache@v3
restore-keys
to restore the most recent numba kernel cache and construct a new one during the current CI run.This should speed up CI runs to the degree that code changes propagate to all numba code -- a change fundamental to all numba kernels will recompile the entire code base, while a peripheral one might just change one kernel.
The text was updated successfully, but these errors were encountered: