Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Build release python wheels and upload to AWS S3 #3398

Merged
merged 13 commits into from
Nov 22, 2024
33 changes: 33 additions & 0 deletions .github/actions/install/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Install uv, rust, and python
description: Install uv, rust, and python
inputs:
python_version:
description: The version of python to install
required: false
default: '3.9'
runs:
using: composite
steps:
- name: Install rust
shell: bash
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
CARGO_BIN="$HOME/.cargo/bin"
echo 'export PATH="$CARGO_BIN:$PATH"' >> $HOME/.bashrc
echo "$CARGO_BIN" >> $GITHUB_PATH
- name: Install uv
shell: bash
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
UV_BIN="$HOME/.local/bin"
echo 'export PATH="$UV_BIN:$PATH"' >> $HOME/.bashrc
echo "$UV_BIN" >> $GITHUB_PATH
- name: Source changes to add new installs to path
shell: bash
run: |
source $HOME/.bashrc
raunakab marked this conversation as resolved.
Show resolved Hide resolved
- name: Install (and pin) python version ${{ inputs.python_version }}
shell: bash
run: |
uv python install ${{ inputs.python_version }}
uv python pin ${{ inputs.python_version }}
64 changes: 64 additions & 0 deletions .github/workflows/build-commit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Build a Daft commit and store the outputted wheel in AWS S3

on:
pull_request:
workflow_dispatch:
workflow_call:
secrets:
ACTIONS_AWS_ROLE_ARN:
description: The ARN of the AWS role to assume
required: true
outputs:
wheel:
description: The wheel file that was built
value: ${{ jobs.build-commit.outputs.wheel }}

jobs:
build-commit:
runs-on: buildjet-8vcpu-ubuntu-2004
timeout-minutes: 15 # Remove for ssh debugging
permissions:
id-token: write
contents: read
outputs:
wheel: ${{ steps.upload.outputs.wheel }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: us-west-2
role-session-name: build-commit-workflow
role-to-assume: ${{ secrets.ACTIONS_AWS_ROLE_ARN }}
- uses: ./.github/actions/install
- uses: buildjet/cache@v4
with:
path: ~/target
key: ${{ runner.os }}-cargo-deps-${{ hashFiles('**/Cargo.lock') }}
restore-keys: ${{ runner.os }}-cargo-deps-
- name: Build release wheel
run: |
export CARGO_TARGET_DIR=~/target
uv v
source .venv/bin/activate
raunakab marked this conversation as resolved.
Show resolved Hide resolved
uv pip install pip maturin boto3
maturin build --release
- name: Upload wheel to AWS S3
id: upload
run: |
count=$(ls ~/target/wheels/*.whl 2> /dev/null | wc -l)
if [ "$count" -gt 1 ]; then
echo "Found more than 1 wheel"
exit 1
elif [ "$count" -eq 0 ]; then
echo "Found no wheels"
exit 1
fi
for file in ~/target/wheels/*.whl; do
aws s3 cp $file s3://github-actions-artifacts-bucket/builds/${{ github.sha }}/ --acl public-read --no-progress;
file_basename=$(basename $file)
echo "wheel=$file_basename" >> $GITHUB_OUTPUT
echo "Output wheel has been built and stored in S3 at the following location:" >> $GITHUB_STEP_SUMMARY
echo "https://us-west-2.console.aws.amazon.com/s3/buckets/github-actions-artifacts-bucket?prefix=builds/${{ github.sha }}/" >> $GITHUB_STEP_SUMMARY
done