Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(CI): handle remote files in a safer way #2217

Merged
merged 4 commits into from
Sep 27, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions .github/actions/deps-install/action.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • name: Install libudev (Linux)
  • name: Install python3
  • name: Install paramiko pip package

Is there some reason we don't want to lock these as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some reason we don't want to lock these as well?

They are fairly known dependencies and if we lock the versions we might miss the important fixes. It's also helpful to see if we are compatible with their recent releases.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use scp instead of paramiko? I understand windows is an outlier here, and we might continue to use it in that case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear to me what requires libudev. It's reference in the fmt-and-lint workflow only. Is it required for cargo clippy or cargo fmt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use scp instead of paramiko? I understand windows is an outlier here, and we might continue to use it in that case.

We can but we need to rewrite this script.

It's not clear to me what requires libudev. It's reference in the fmt-and-lint workflow only. Is it required for cargo clippy or cargo fmt?

It's probably outdated (since e71e2cf maybe?), let me try to remove it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@onur-ozkan onur-ozkan Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most probably required from the non-default features.

Reverted.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requirement is introduced by [email protected]

Could we check if this is absolutely necessary? Can someone please confirm that we actually use this crate?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we check if this is absolutely necessary? Can someone please confirm that we actually use this crate?

It is necessary for enable-solana feature (and we run --all-features for linting runner). As we don't provide any output (like executable binary) to public from linting runner, there shouldn't be any concern for including libudev in that runner I think.

Original file line number Diff line number Diff line change
Expand Up @@ -12,33 +12,54 @@ inputs:
runs:
using: 'composite'
steps:
- name: Download protoc (Linux)
if: runner.os == 'Linux' && contains(inputs.deps, 'protoc')
uses: ./.github/actions/download-and-verify
with:
url: "https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-linux-x86_64.zip"
output_file: "protoc-25.3-linux-x86_64.zip"
checksum: "f853e691868d0557425ea290bf7ba6384eef2fa9b04c323afab49a770ba9da80"

- name: Install protoc (Linux)
env:
TMP: ${{ inputs.temp || runner.temp }}
if: runner.os == 'Linux' && contains(inputs.deps, 'protoc')
shell: bash
run: |
wget https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-linux-x86_64.zip
unzip protoc-25.3-linux-x86_64 -d "$TMP/protobuf"
unzip protoc-25.3-linux-x86_64.zip -d "$TMP/protobuf"
echo "$TMP/protobuf/bin" >> $GITHUB_PATH

- name: Download protoc (MacOS)
if: runner.os == 'macOS' && contains(inputs.deps, 'protoc')
uses: ./.github/actions/download-and-verify
with:
url: "https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-osx-x86_64.zip"
output_file: "protoc-25.3-osx-x86_64.zip"
checksum: "247e003b8e115405172eacc50bd19825209d85940728e766f0848eee7c80e2a1"

- name: Install protoc (MacOS)
env:
TMP: ${{ inputs.temp || runner.temp }}
if: runner.os == 'macOS' && contains(inputs.deps, 'protoc')
shell: bash
run: |
wget https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-osx-x86_64.zip
unzip protoc-25.3-osx-x86_64.zip -d "$TMP/protobuf"
echo "$TMP/protobuf/bin" >> $GITHUB_PATH


- name: Download protoc (Windows)
uses: ./.github/actions/download-and-verify
with:
url: "https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-win64.zip"
output_file: "protoc-25.3-win64.zip"
checksum: "d6b336b852726364313330631656b7f395dde5b1141b169f5c4b8d43cdf01482"

- name: Install protoc (Windows)
env:
TMP: ${{ inputs.temp || runner.temp }}
if: runner.os == 'Windows' && contains(inputs.deps, 'protoc')
shell: powershell
run: |
Invoke-WebRequest -Uri https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-win64.zip -OutFile protoc-25.3-win64.zip
7z x protoc-25.3-win64.zip -o"$TMP\protobuf"
echo "$TMP\protobuf\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

Expand Down
46 changes: 46 additions & 0 deletions .github/actions/download-and-verify/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: "Download and verify remote files"

runs:
using: "composite"
steps:
- name: Download (Unix)
if: runner.os != 'Windows'
shell: bash
run: curl -L -o ${{ inputs.output_file }} ${{ inputs.url }}

- name: Download (Windows)
if: runner.os == 'Windows'
shell: powershell
run: Invoke-WebRequest -Uri ${{ inputs.url }} -OutFile ${{ inputs.output_file }}

- name: Verify (Unix)
if: runner.os != 'Windows'
shell: bash
run: |
if [[ "$RUNNER_OS" == "macOS" ]]; then
echo "${{ inputs.checksum }} *${{ inputs.output_file }}" | shasum -a 256 -c
else
echo "${{ inputs.checksum }} ${{ inputs.output_file }}" | sha256sum -c
fi

- name: Verify (Windows)
if: runner.os == 'Windows'
shell: powershell
run: |
$expectedChecksum = "${{ inputs.checksum }}"
$actualChecksum = (Get-FileHash -Path "${{ inputs.output_file }}" -Algorithm SHA256).Hash
if ($expectedChecksum -ne $actualChecksum) {
Write-Output "Checksum did not match! Expected: $expectedChecksum, Found: $actualChecksum"
exit 1
}

inputs:
url:
description: "URL of the remote file."
required: true
output_file:
description: "Output path."
required: true
checksum:
description: "Expected checksum of the downloaded file."
required: true
24 changes: 18 additions & 6 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ jobs:

- name: Test
run: |
wget -O - https://raw.githubusercontent.com/KomodoPlatform/komodo/master/zcutil/fetch-params-alt.sh | bash
wget -O - https://raw.githubusercontent.com/KomodoPlatform/komodo/0adeeabdd484ef40539d1275c6a765f5c530ea79/zcutil/fetch-params-alt.sh | bash
cargo test --test 'mm2_tests_main' --no-fail-fast

mac-x86-64-kdf-integration:
Expand Down Expand Up @@ -154,7 +154,7 @@ jobs:

- name: Test
run: |
wget -O - https://raw.githubusercontent.com/KomodoPlatform/komodo/master/zcutil/fetch-params-alt.sh | bash
wget -O - https://raw.githubusercontent.com/KomodoPlatform/komodo/0adeeabdd484ef40539d1275c6a765f5c530ea79/zcutil/fetch-params-alt.sh | bash
cargo test --test 'mm2_tests_main' --no-fail-fast

win-x86-64-kdf-integration:
Expand All @@ -181,10 +181,16 @@ jobs:
- name: Cargo cache
uses: ./.github/actions/cargo-cache

- name: Download wget64
uses: ./.github/actions/download-and-verify
with:
url: "https://github.com/KomodoPlatform/komodo/raw/d456be35acd1f8584e1e4f971aea27bd0644d5c5/zcutil/wget64.exe"
output_file: "wget64.exe"
checksum: "d80719431dc22b0e4a070f61fab982b113a4ed9a6d4cf25e64b5be390dcadb94"

- name: Test
run: |
Invoke-WebRequest -Uri https://github.com/KomodoPlatform/komodo/raw/d456be35acd1f8584e1e4f971aea27bd0644d5c5/zcutil/wget64.exe -OutFile \wget64.exe
Invoke-WebRequest -Uri https://raw.githubusercontent.com/KomodoPlatform/komodo/master/zcutil/fetch-params-alt.bat -OutFile \cmd.bat && \cmd.bat
Invoke-WebRequest -Uri https://raw.githubusercontent.com/KomodoPlatform/komodo/0adeeabdd484ef40539d1275c6a765f5c530ea79/zcutil/fetch-params-alt.bat -OutFile \cmd.bat && \cmd.bat
cargo test --test 'mm2_tests_main' --no-fail-fast

docker-tests:
Expand Down Expand Up @@ -213,7 +219,7 @@ jobs:

- name: Test
run: |
wget -O - https://raw.githubusercontent.com/KomodoPlatform/komodo/v0.8.1/zcutil/fetch-params-alt.sh | bash
wget -O - https://raw.githubusercontent.com/KomodoPlatform/komodo/0adeeabdd484ef40539d1275c6a765f5c530ea79/zcutil/fetch-params-alt.sh | bash
shamardy marked this conversation as resolved.
Show resolved Hide resolved
cargo test --test 'docker_tests_main' --features run-docker-tests --no-fail-fast

wasm:
Expand Down Expand Up @@ -241,11 +247,17 @@ jobs:
- name: Install wasm-pack
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh

- name: Download geckodriver
uses: ./.github/actions/download-and-verify
with:
url: "https://github.com/mozilla/geckodriver/releases/download/v0.32.2/geckodriver-v0.32.2-linux64.tar.gz"
output_file: "geckodriver-v0.32.2-linux64.tar.gz"
checksum: "1eab226bf009599f5aa1d77d9ed4c374e10a03fd848b500be1b32cefd2cbec64"

- name: Install firefox and geckodriver
run: |
sudo apt-get update -y
sudo apt-get install -y firefox
wget https://github.com/mozilla/geckodriver/releases/download/v0.32.2/geckodriver-v0.32.2-linux64.tar.gz
sudo tar -xzvf geckodriver-v0.32.2-linux64.tar.gz -C /bin
sudo chmod +x /bin/geckodriver

Expand Down
Loading