rules_python broken with Bazel 8 on BCR

[rickeylev]

🐞 bug report

Affected Rule

Unclear, looks like early module-phase startup code

Is this a regression?

Yes, works in 7.x

Description

• https://buildkite.com/bazel/bcr-bazel-compatibility-test/builds/22#0192d96c-707d-4db4-89d9-c41ae4ef3cf1
• Top BCR modules broken with Bazel 8.0.0rc8 bazel-central-registry#3056

(18:10:19) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/1ae3de34028be386c3858fecaf88706d/external/rules_python+/python/private/full_version.bzl:35:13: Traceback (most recent call last):
	File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/1ae3de34028be386c3858fecaf88706d/external/rules_python+/python/private/python.bzl", line 218, column 43, in _python_impl
		full_python_version = full_version(
	File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/1ae3de34028be386c3858fecaf88706d/external/rules_python+/python/private/full_version.bzl", line 35, column 13, in full_version
		fail(
Error in fail: Unknown Python version '3.8', available values are: 3.10,3.9,3.11

🔬 Minimal Reproduction

Unclear, maybe running the bzlmod example under bazel 8?

[meteorcloudy]

I will try to make it easier to set up the environment for reproducing BCR presubmit failures. But for now, yes, you need to run bazel build //... under examples/bzlmod.

rules_python/.bcr/presubmit.yml

Lines 15 to 16 in 7c5e7cf

	bcr_test_module:
	module_path: "examples/bzlmod"

[rickeylev]

Luckily, it doesn't look specific to BCR. In rules_python from head:

cd examples/bzlmod
USE_BAZEL_VERSION=last_rc bazel test ...

Will result in the same error as reported.

[rickeylev]

tldr: I think we need to change our bzlmod example to not restrict python versions. Restricting the versions is tenenable with a small constrained dep footprint, but the example isn't that.

So in the bzlmod example, we intentionally restrict the python versions available to 3.9, 3.10, and 3.11. This is mostly just for demonstration purposes of this version-restriction feature.

The examples works with bazel 7, but fails under bazel 8. So, this would imply there's a bug with bazel 7 that got fixed in bazel 8. But...that seems like a weird bug -- failing to traverse the module graph? That would be hard to go unnoticed.

Hm, it looks like different modules are being used under bazel7 vs bazel 8

Modules and py versions that python.toolchain sees

Bazel 8:

example_bzlmod 3.9
example_bzlmod 3.10
rules_python 3.11
protobuf 3.8
protobuf 3.9
protobuf 3.10
protobuf 3.11
protobuf 3.12
other_module 3.9
other_module 3.11
rules_fuzzing 3.8
rules_fuzzing 3.9
rules_fuzzing 3.10
rules_fuzzing 3.11
rules_fuzzing 3.12

Bazel 7:

example_bzlmod 3.9
example_bzlmod 3.10
rules_python 3.11
other_module 3.9
other_module 3.11

Huh. Weird. example_bzlmod is the root module. other_module is one setup for the test. It's almost as if Bazel 7 was only looking at...things defined in the example's module file? and bazel 8 is now (what appears to be properly) traversing the module graph.

Looking at bazel mod graph output, I think what's happening is bazel 8 is using some newer module versions than bazel 7, and those newer versions are registering additional python versions.

If I bump the protobuf version, I bet it'll fail in 7.x...ah yup.

[meteorcloudy]

Yeah, it might be module updates in https://cs.opensource.google/bazel/bazel/+/master:src/MODULE.tools. We had to include some new modules to support auto load

[rickeylev]

As a side note, I'm thinking the restriction logic we have in the module-phase is overly-eager about its enforcement. The intent is to allow the root module to prevent certain python versions from being used, but I don't think e.g. python 3.8 is actually being used in the build. It's just that e.g. protobuf needs to register something for 3.8 just in case it is used. Instead of failing at the most phase, we could probably let it fail later, at the build phase -- e.g. don't register a 3.8 toolchain, or register a failing toolchain for 3.8, or something like that.