Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fixing compile pip requirements when running on an RBE #2479

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ Unreleased changes template.
* Bazel 6 support is dropped and Bazel 7.4.1 is the minimum supported
version, per our Bazel support matrix. Earlier versions are not
tested by CI, so functionality cannot be guaranteed.
* In a suitably configured RBE, `pip_compile` will now be runnable.

{#v0-0-0-fixed}
### Fixed
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@

from python.runfiles import runfiles

# Replace the os.replace function with shutil.copy to work around os.replace not being able to
# Replace the os.replace function with shutil.move to work around os.replace not being able to
# replace or move files across filesystems.
os.replace = shutil.copy
os.replace = shutil.move
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I am looking at the PR that made the change previously (#1053) and I can't see why one wouldn't want the shutil.move. I also can't see looking at the source of shutil.move if it would be problematic, it will also try to fallback to shutil.copy2, so I think the shutil.move is a robust solution here.


# Next, we override the annotation_style_split and annotation_style_line functions to replace the
# backslashes in the paths with forward slashes. This is so that we can have the same requirements
Expand Down
16 changes: 11 additions & 5 deletions python/private/pypi/pip_compile.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ make it possible to have multiple tools inside the `pypi` directory
load("//python:py_binary.bzl", _py_binary = "py_binary")
load("//python:py_test.bzl", _py_test = "py_test")

_TAGS_FOR_REMOTE_EXECUTION = ["no-sandbox", "no-remote-exec", "requires-network"]

def pip_compile(
name,
srcs = None,
Expand All @@ -38,6 +40,7 @@ def pip_compile(
requirements_windows = None,
visibility = ["//visibility:private"],
tags = None,
remoteable = True,
**kwargs):
"""Generates targets for managing pip dependencies with pip-compile.

Expand Down Expand Up @@ -68,6 +71,7 @@ def pip_compile(
extra_args: passed to pip-compile.
extra_deps: extra dependencies passed to pip-compile.
generate_hashes: whether to put hashes in the requirements_txt file.
remoteable: whether of not to add tags that support remote execution.
py_binary: the py_binary rule to be used.
py_test: the py_test rule to be used.
requirements_in: file expressing desired dependencies. Deprecated, use src or srcs instead.
Expand Down Expand Up @@ -141,17 +145,19 @@ def pip_compile(
Label("//python/runfiles:runfiles"),
] + extra_deps

tags = tags or []
tags.append("requires-network")
tags.append("no-remote-exec")
tags.append("no-sandbox")
tags_to_use = []
if tags:
tags_to_use += tags
if remoteable:
tags_to_use += _TAGS_FOR_REMOTE_EXECUTION

attrs = {
"args": args,
"data": data,
"deps": deps,
"main": pip_compile,
"srcs": [pip_compile],
"tags": tags,
"tags": tags_to_use,
"visibility": visibility,
}

Expand Down