WORKSPACE to MODULE.bazel migration setting default python version

[gfrankliu]

We are migrating from WORKSPACE to MODULE.bazel. Most of modules are now in MODULE.bazel and very few are still in WORKSPACE.

We have rules_python still in WORKSPACE because some of our internal custom python codes/rules/builds still need more time to update.

We use python version 3.10, eg: in WORKSPACE:

python_register_toolchains(
    name = "python_interpreter",
    python_version = "3.10",
)

We notice an issue where our external dependencies from MODULE.bazel are using python 3.11 instead of our default 3.10.

Since we are still using WORKSPACE and can't set python.toolchain(is_default = True, python_version = "3.10") in our root MODULE.bazel, how can we "tell" those external modules (they are already using MODULE.bazel) to use python 3.10?

[gfrankliu]

We are using rules_python 0.40.0, and latest bazel 7: bazel 7.4.1

[aignas]

That might be working as intended - MODULE.bazel dependencies are using the default toolchain set by bzlmod code and it cannot know anything about the WORKSPACE paths.

I am not sure if this is something that can be easily fixed.

At the same time, I am not sure why you can define your toolchain in bzlmod? You should still be able to use pip_parse from WORKSPACE with the toolchain set in MODULE.bazel.

[gfrankliu]

Thanks for the suggestion! I just tried this:

In MODULE.bazel, I added:

bazel_dep(name = "rules_python", version = "0.40.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(is_default = True, python_version = "3.10")
use_repo(python, "python_3_10")

In WORKSPACE, I only have below lines, nothing else:

load("@rules_python//python:pip.bzl", "pip_parse")
pip_parse(
    name = "third_party_lib",
    requirements_lock = "//:requirements.txt",
)

bazel build gives this error:

ERROR: Failed to load Starlark extension '@@rules_python_internal//:rules_python_config.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
 - @@rules_python_internal
This could either mean you have to add the '@@rules_python_internal' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
ERROR: Error computing the main repository mapping: cycles detected during computation of main repo mapping

Since pip.parse isn't 100% matching pip_parse, it breaks some of our internal custom bazel rules. It will take some time for us to update all those custom rules. In the meantime, we would like to migrate non python stuff into MODULE.bazel.

[aignas]

You need the rules_python deps in the WORKSPACE as well as the error message might be hinting you.

With rules_python 1.0 and 7.x bazel, the following WORKSPACE file works:

$ cat WORKSPACE
load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "third_party",
    requirements_lock = "//:requirements.txt",
)

[gfrankliu]

Thanks and that gets me further, but it fails when I pin the version to python 3.10 in WORKSPACE

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")
load("@python_3_10//:defs.bzl", "interpreter")

pip_parse(
    name = "third_party",
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

load("@third_party//:requirements.bzl", "install_deps")

install_deps()

bazel build gives errors:

	File "/home/gfrankliu/.cache/bazel/_bazel_gfrankliu/b71142cf8050563176d73cc4d125110e/external/rules_python~/python/private/pypi/whl_library.bzl", line 178, column 68, in _whl_library_impl
		python_interpreter = pypi_repo_utils.resolve_python_interpreter(
	File "/home/gfrankliu/.cache/bazel/_bazel_gfrankliu/b71142cf8050563176d73cc4d125110e/external/rules_python~/python/private/pypi/pypi_repo_utils.bzl", line 67, column 40, in _resolve_python_interpreter
		python_interpreter = mrctx.path(root_build_bazel).dirname.get_child(python_interpreter_target.name)
Error in path: Unable to load package for @@python_3_10_x86_64-unknown-linux-gnu//:BUILD.bazel: The repository '@@python_3_10_x86_64-unknown-linux-gnu' could not be resolved: Repository '@@python_3_10_x86_64-unknown-linux-gnu' is not defined

In the requirement.bzl _config section, I can see

    "python_interpreter": "python3",
    "python_interpreter_target": "@@python_3_10_x86_64-unknown-linux-gnu//:bin/python3",

The bazel errors says the repo @@python_3_10_x86_64-unknown-linux-gnu doesn't exist. In the bazel cache directory, I do see "external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu" but don't see "external/python_3_10_x86_64-unknown-linux-gnu".

I was using rules_python 0.38 in this test. If I switch to 1.0.0, I get a different error:

		load("@python_3_10//:defs.bzl", "interpreter")
Error: file '@python_3_10//:defs.bzl' does not contain symbol 'interpreter'

[aignas]

This works for me:

$ cat WORKSPACE
load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "third_party",
    requirements_lock = "//:requirements.txt",
    python_interpreter_target = "@python_3_10_host//:python",
)

load("@third_party//:requirements.bzl", "install_deps")

install_deps()

$ cat MODULE.bazel
bazel_dep(name = "rules_python", version = "0.40.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")

python.toolchain(is_default = True, python_version = "3.10")

use_repo(python, "python_3_10", "python_3_10_host")

[aignas]

I have added a PR to add this snippet to our docs as this may be also useful to others: #2510

[gfrankliu]

Thanks @aignas
I was using @python_3_10 and I can see you are using @python_3_10_host in your example. Have you tried @python_3_10 ?

With your hint, I just tried @python_3_10_host , which seems to work. Just curious why @python_3_10 doesn't work.

[aignas]

python_3_10 is something that points to an interpreter that can only be used in the build context. python_3_10_host uses the same repositories but resolves/copies the files in the repository_rule context. The legacy load("@python_3_10_host//:defs.bzl", "interpreter") has been replaced with @python_3_10_host//:python.