pip.parse vendored support

[gfrankliu]

We are currently using WORKSPACE and pip_parse_vendored following the examples in https://github.com/bazelbuild/rules_python/tree/main/examples/pip_parse_vendored

While migrating to MODULE.bazel, there are no similar pip.parse_vendored example. In particular, I am looking for the equivalent of https://github.com/bazelbuild/rules_python/blob/main/examples/pip_parse_vendored/BUILD.bazel#L24-L34

Looking at pip_parse generated @pip_deps_to_be_vendored//:requirements.bzl and pip.parse generated :requirements.bzl , I see the latter misses _config = { ... } and _packages = [ ... ]
Is that a bug?

[gfrankliu]

Here is a quick test example to show the differences in the requirements.bzl

Using WORKSPACE and pip_parse:

# WORKSPACE
python_register_toolchains(
    name = "python_interpreter",
    ignore_root_user_error = True,
    python_version = "3.10.11",
)
load("@python_interpreter//:defs.bzl", "interpreter")
pip_parse(
    name = "test_lib",
    enable_implicit_namespace_pkgs = True,
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

# BUILD
genrule(
    name = "test_bzl",
    srcs = ["@test_lib//:requirements.bzl"],
    outs = ["test.bzl"],
    cmd = "cp $< $@",
)

Using the new MODULE.bazel and pip.parse:

# MODULE.bazel
python.toolchain(
    ignore_root_user_error = True,
    is_default = True,
    python_version = "3.10.11",
)
use_repo(python, "python_3_10_11")
pip.parse(
    hub_name = "test_lib",
    enable_implicit_namespace_pkgs = True,
    python_version = "3.10.11",
    requirements_lock = "//:requirements.txt",
)

# BUILD
genrule(
    name = "test_bzl",
    srcs = ["@test_lib//:requirements.bzl"],
    outs = ["test.bzl"],
    cmd = "cp $< $@",
)

You can see the new test.bzl misses:

load("@rules_python//python/pip_install:pip_repository.bzl", "whl_library")

_packages =
_config = 
def entry_point(pkg, script = None):
def _get_annotation(requirement):
def install_deps(**whl_library_kwargs):

Is this a bug in the new pip.parse?

[aignas]

bzlmod has lockfile mechanism and its own vendoring code. Does it not satisfy your needs?

There are no plans to add vendoring in the same way it was done in WORKSPACE.

[gfrankliu]

We have some internal custom bazel rules that

• uses _config (specifically extra_pip_args from _config) from the old requirement.bzl generated by pip_parse. The new pip.parse generated requirement.bzl misses the whole _config.
• use _packages from the old requirement.bzl generated by pip_parse. The new pip.parse generated requirement.bzl misses _packages

The python modules directory under runfiles also look different, eg: module httplib2 looks like app.runfiles/third_party_lib_httplib2/site-packages/ when using WORKSPACE/pip_parse, but looks like app.runfiles/rules_python~~pip~third_party_lib_310_httplib2/site-packages/ when using bzmod/pip.parse

Is it possible to configure pip.parse to behave more like pip_parse to make the rules_python WORKSPACE to bzlmod migration easier?

[aignas]

It seems that you may be depending on implementation details within rules_python and unfortunately some things just are not possible in bzlmod:

• Having extra_pip_args might be possible, but in general they can be different for each target platform, so supporting that might be tough. What is the use case here?
• What is the use case for _packages?
• The layout is not controlled by us and is imposed by bzlmod itself, so unfortunately that won't be possible.

You can go the other way though to first make pip_parse closer to what pip.parse is. We tried to make it possible to do everything that you can do through the API exposed by pip_parse (i.e. the attributes and the publicly available constants in requirements.bzl) to be possible in pip.parse bzlmod extension.

Since there is nothing to do here, I will migrate this to a discussion.

[gfrankliu]

• I see in my pip_parse generated requirements.bzl, extra_pip_args came from the requirements.in. eg: my requirements.in has

--extra-index-url https://download.pytorch.org/whl/cu117
-f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

and my requirements.bzl has corresponding:

    "extra_pip_args": ["--extra-index-url", "https://download.pytorch.org/whl/cu117", "--find-links", "https://storage.googleapis.com/jax-releases/jax_cuda_releases.html"],

Does that mean supporting this in the pip.parsed requirement.bzl should be a straightforward copy from requirements.in?

• Those custom bazel rules we have were written by someone who's no longer on the team. I am still trying to understand. It seems to get extra_pip_args and _packages to inject into the Dataflow Apache beam python so that Dataflow cloud runner can use pip behind the scene as part of the Dataflow job startup: https://cloud.google.com/dataflow/docs/guides/manage-dependencies#python-define-dependencies
• I guess we can update to match the new directory layout.

[aignas]

Does that mean supporting this in the pip.parsed requirement.bzl should be a straightforward copy from requirements.in?

I don't think it makes sense adding it in there as it may complicate things.

Those custom bazel rules we have were written by someone who's no longer on the team. I am still trying to understand. It seems to get extra_pip_args and _packages to inject into the Dataflow Apache beam python so that Dataflow cloud runner can use pip behind the scene as part of the Dataflow job startup:

You could have a separate build rule that is duplicating the logic on your end. If the data is in the requirements.in file already, you can read the same data yourself. To be honest, I see no reason for rules_python to be involved in here. However, I understand the pain of maintaining/fixing code that you did not write, so I wish you the best of luck with that. :)