Build Pipeline Is Broken (Some Python Files Don't Run)

[Bloo-dev]

Ever since 2024/07/18 the build pipeline on this repo seems to be broken due to some python files seemingly not executing, causing (for example) the following error:

This has happend before and was solved by specifying a specific ubuntu version, but this no longer seems to work.

@SpecialBuilder32 to provide more details.

[SpecialBuilder32]

So I've been doing some digging, and I think 6510fa8 fixed this particular problem? I want to do some more investigation before closing this issue though.

Previously I had thought this was caused by the versioning.isolated_library plugin just not running suddenly - that is where this added_libs field gets setup. After some digging, I think I had misunderstood the execution order of our beet pipeline, and this plugin should've been configured to run before diving down into each library's individual beet.yaml.

The reason I think it works on some github action runner images (and locally) is due to the ordering of glob results. Locally on windows, this processes in alphabetical order, always running on lib_brewing first, which has no dependencies and so the added_libs field not being set doesn't matter. After building lib_brewing, the field exists in the cache and the next library doesn't throw the KeyError. When we run on the ubuntu-runner, glob returns files in an arbitrary order, so if the first library to be built has any dependencies to include (about a 30% chance) then we run into this problem.

It seems that this arbitrary order doesn't change between different build jobs, so long as the runner image is the same. The order does appear to change when they release a new image. I think this is also related to why our auto-versioning plugin sometimes goes crazy and gives every module a patch increment and then settles down for a month. Somehow this arbitrary file order resolution is affecting the zip file hash. (And I think I need to wait for the next released image to capture the difference on my debug branch)