Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update py_import macros for the ability to unpack additional wheels in the same folder as the main wheel. #20089

Merged
merged 1 commit into from
Dec 13, 2024

Conversation

copybara-service[bot]
Copy link

@copybara-service copybara-service bot commented Dec 3, 2024

Update py_import macros for the ability to unpack additional wheels in the same folder as the main wheel.

Usage example: provide NVIDIA wheel dependencies for ML wheels that have rpaths pointing to NVIDIA folders. When a user executes pip install tensorflow[and_cuda], NVIDIA wheels are installed together with Tensorflow wheel. To reproduce this behavior in hermetic Python approach, we need to define py_import as follows (provided NVIDIA dependencies are defined in requirements.in and requirements lock files):

    py_import(
        name = "tf_py_import",
        wheel = ":wheel",
        deps = [
            "@pypi_absl_py//:pkg",
            "@pypi_astunparse//:pkg",
            "@pypi_flatbuffers//:pkg",
            "@pypi_gast//:pkg",
            "@pypi_ml_dtypes//:pkg",
            "@pypi_numpy//:pkg",
            "@pypi_opt_einsum//:pkg",
            "@pypi_packaging//:pkg",
            "@pypi_protobuf//:pkg",
            "@pypi_requests//:pkg",
            "@pypi_termcolor//:pkg",
            "@pypi_typing_extensions//:pkg",
            "@pypi_wrapt//:pkg",
        ],
        wheel_deps = [
            "@pypi_nvidia_cublas_cu12//:whl",
            "@pypi_nvidia_cuda_cupti_cu12//:whl",
            "@pypi_nvidia_cuda_nvcc_cu12//:whl",
            "@pypi_nvidia_cuda_nvrtc_cu12//:whl",
            "@pypi_nvidia_cuda_runtime_cu12//:whl",
            "@pypi_nvidia_cudnn_cu12//:whl",
            "@pypi_nvidia_cufft_cu12//:whl",
            "@pypi_nvidia_curand_cu12//:whl",
            "@pypi_nvidia_cusolver_cu12//:whl",
            "@pypi_nvidia_cusparse_cu12//:whl",
            "@pypi_nvidia_nccl_cu12//:whl",
            "@pypi_nvidia_nvjitlink_cu12//:whl",
        ],
    )

@copybara-service copybara-service bot force-pushed the test_702375747 branch 10 times, most recently from a4c172b to cd0b2fc Compare December 10, 2024 19:06
@copybara-service copybara-service bot force-pushed the test_702375747 branch 2 times, most recently from ffdad50 to cc1d57f Compare December 12, 2024 23:52
… in the same folder as the main wheel.

Usage example: provide NVIDIA wheel dependencies for ML wheels that have rpaths pointing to NVIDIA folders. When a user executes `pip install tensorflow[and_cuda]`, NVIDIA wheels are installed together with Tensorflow wheel. To reproduce this behavior in hermetic Python approach, we need to define `py_import` as follows (provided NVIDIA dependencies are defined in `requirements.in` and requirements lock files):

        py_import(
            name = "tf_py_import",
            wheel = ":wheel",
            deps = [
                "@pypi_absl_py//:pkg",
                "@pypi_astunparse//:pkg",
                "@pypi_flatbuffers//:pkg",
                "@pypi_gast//:pkg",
                "@pypi_ml_dtypes//:pkg",
                "@pypi_numpy//:pkg",
                "@pypi_opt_einsum//:pkg",
                "@pypi_packaging//:pkg",
                "@pypi_protobuf//:pkg",
                "@pypi_requests//:pkg",
                "@pypi_termcolor//:pkg",
                "@pypi_typing_extensions//:pkg",
                "@pypi_wrapt//:pkg",
            ],
            wheel_deps = [
                "@pypi_nvidia_cublas_cu12//:whl",
                "@pypi_nvidia_cuda_cupti_cu12//:whl",
                "@pypi_nvidia_cuda_nvcc_cu12//:whl",
                "@pypi_nvidia_cuda_nvrtc_cu12//:whl",
                "@pypi_nvidia_cuda_runtime_cu12//:whl",
                "@pypi_nvidia_cudnn_cu12//:whl",
                "@pypi_nvidia_cufft_cu12//:whl",
                "@pypi_nvidia_curand_cu12//:whl",
                "@pypi_nvidia_cusolver_cu12//:whl",
                "@pypi_nvidia_cusparse_cu12//:whl",
                "@pypi_nvidia_nccl_cu12//:whl",
                "@pypi_nvidia_nvjitlink_cu12//:whl",
            ],
        )

PiperOrigin-RevId: 705666137
@copybara-service copybara-service bot merged commit 30c7eb7 into main Dec 13, 2024
1 check passed
@copybara-service copybara-service bot deleted the test_702375747 branch December 13, 2024 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant