Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor CI workflow #439

Merged
merged 153 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
bfebed8
Remove artifact upload/download
danielhollas Mar 20, 2024
eb7f22a
Run only amd64-base for now
danielhollas Mar 20, 2024
dce9a38
Use docker/build-push-action
danielhollas Mar 20, 2024
50b9f16
Simplify
danielhollas Mar 20, 2024
ce8b97d
Set target, bump action versions
danielhollas Mar 21, 2024
94d05d6
Huh?
danielhollas Mar 21, 2024
4dd865e
Add context
danielhollas Mar 21, 2024
589ee67
One more try
danielhollas Mar 21, 2024
ea59188
Path context
danielhollas Mar 21, 2024
9e6d443
Try bake-action
danielhollas Mar 21, 2024
59259c1
ARM build
danielhollas Apr 8, 2024
056f859
Add check-jsonschema pre-commit
danielhollas Apr 8, 2024
fd36101
Comment out QEMU action
danielhollas Apr 8, 2024
303cc27
Try self-hosted runner
danielhollas Apr 8, 2024
bd5200c
Pass in correct metadata
danielhollas Apr 8, 2024
ff0efa9
Generate bake meta file
danielhollas Apr 8, 2024
076e9b2
Install jq
danielhollas Apr 8, 2024
8f090b7
Ugh
danielhollas Apr 8, 2024
daecefc
Push!
danielhollas Apr 8, 2024
2db11cc
Push?
danielhollas Apr 8, 2024
b9fe7d6
ugh, suspicious
danielhollas Apr 8, 2024
dabc273
Tests?
danielhollas Apr 8, 2024
7dccaf0
Merge branch 'main' into remove-artifacts
danielhollas Apr 8, 2024
7369fc8
fix workflow file
danielhollas Apr 8, 2024
4a5da9a
Fix
danielhollas Apr 8, 2024
7978b1e
Fix?
danielhollas Apr 8, 2024
c256c9c
Try again
danielhollas Apr 8, 2024
05909e9
Try metadatav4
danielhollas Apr 8, 2024
510dca6
Now?
danielhollas Apr 8, 2024
cb77324
Getting closer
danielhollas Apr 8, 2024
df94b22
Hmm?
danielhollas Apr 8, 2024
64a47f8
aiidalab_exec depends on notebook_service
danielhollas Apr 8, 2024
04cba11
Cleanup
danielhollas Apr 8, 2024
9142e30
Update .github/workflows/build.yml
danielhollas Apr 9, 2024
877edbd
Simplify pytest invocation
danielhollas Apr 17, 2024
8913922
Merge branch 'main' into remove-artifacts
danielhollas Apr 17, 2024
f4beeae
Fix?
danielhollas Apr 17, 2024
c3fbf55
Remove unneeded step
danielhollas Apr 17, 2024
d3b3da1
Debug
danielhollas Apr 17, 2024
62b1d77
Update merge-bake-template-target.sh
danielhollas Apr 17, 2024
8332a67
pipefail in GHA
danielhollas Apr 17, 2024
f05761b
Try fixing PYTHON_MINOR_VERSION
danielhollas Apr 17, 2024
5994487
One more try
danielhollas Apr 17, 2024
202128d
Ugh
danielhollas Apr 17, 2024
0305af3
Cleanup
danielhollas Apr 17, 2024
418bb21
Remove new-build version
danielhollas Apr 17, 2024
87cc98b
More robust merge script
danielhollas Apr 17, 2024
75ce6b0
One more thing
danielhollas Apr 17, 2024
84e274b
Probably not
danielhollas Apr 17, 2024
8ceb3de
Hotfix
danielhollas Apr 17, 2024
d2ceb23
Default registry in docker-compose files
danielhollas Apr 17, 2024
711efe1
Revert "Default registry in docker-compose files"
danielhollas Apr 17, 2024
aa675c8
Split integration tests
danielhollas Apr 17, 2024
4072d67
Cleanup
danielhollas Apr 17, 2024
260d261
Reuse setup python env action
danielhollas Apr 17, 2024
9f1a5ad
Simplify
danielhollas Apr 17, 2024
dd51bf5
Build arm64 on GHA using QEMU
danielhollas Apr 17, 2024
19a3ee2
Update build.yml
danielhollas Apr 17, 2024
df2f440
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 17, 2024
ed07958
shell: bash
danielhollas Apr 17, 2024
e8b31d7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 17, 2024
ecbe747
Disable arm64 build
danielhollas Apr 17, 2024
cfab33b
Checkout head commit instead of merge commit
danielhollas Apr 18, 2024
cbfa736
Revert "Checkout head commit instead of merge commit"
danielhollas Apr 18, 2024
eeb1eea
Enable arm64 build
danielhollas Apr 18, 2024
0d68dd5
Try building arm/amd together
danielhollas Apr 18, 2024
58febf1
Test before arm64 build
danielhollas Apr 18, 2024
f728526
Reorder
danielhollas Apr 18, 2024
b61ee7d
Use SHA of the actual commit
danielhollas Apr 18, 2024
da8f350
Test AMR64 full-stack image
danielhollas Apr 18, 2024
2b74605
Fix
danielhollas Apr 18, 2024
30431aa
Delete old files
danielhollas Apr 18, 2024
28c0050
Use digest to identify image
danielhollas Apr 18, 2024
6746037
Try pushing
danielhollas Apr 18, 2024
ce6e5d4
What's in there?
danielhollas Apr 18, 2024
fa4e6e5
Add outputs
danielhollas Apr 18, 2024
887f94a
Maybe?
danielhollas Apr 18, 2024
4934e44
Merge branch 'main' into remove-artifacts
danielhollas Apr 18, 2024
837b11e
Global pytest mark
danielhollas Apr 18, 2024
66ee2e1
Remove missing input
danielhollas Apr 18, 2024
f9c5ef2
Add test.yml
danielhollas Apr 18, 2024
8dee62a
Temporarily skip arm64 build
danielhollas Apr 18, 2024
e341bfd
Remove tests from build
danielhollas Apr 18, 2024
033dff3
update create-dev-env
danielhollas Apr 18, 2024
f4cc6c5
Hook up new test workflow
danielhollas Apr 18, 2024
b63da11
Add integration tests
danielhollas Apr 18, 2024
656f2b2
matrix strategy in main?
danielhollas Apr 18, 2024
ac46f76
Single quote
danielhollas Apr 18, 2024
a0cd2be
Fix tests and customize name
danielhollas Apr 18, 2024
d6c2f54
Extract image names
danielhollas Apr 19, 2024
bcbe78c
Remove create-dev-env
danielhollas Apr 19, 2024
493c72f
Reenable arm64 build
danielhollas Apr 19, 2024
0b1f1bf
increase timeout
danielhollas Apr 19, 2024
004eb16
fix
danielhollas Apr 19, 2024
fb23e00
further timeout increase
danielhollas Apr 19, 2024
4805724
remove precommit from dev requirements
danielhollas Apr 19, 2024
6d403f8
Add note to integration tests
danielhollas Apr 19, 2024
40bdfc9
Timeout after two hours
danielhollas Apr 19, 2024
1e28616
Fix arm tests
danielhollas Apr 19, 2024
a76bbfd
Update login action
danielhollas Apr 19, 2024
23130e2
Update metadata-action
danielhollas Apr 19, 2024
ae5b70c
Example output of merge-bake-template-target.sh
danielhollas Apr 19, 2024
3368377
Initial publish workflow
danielhollas Apr 19, 2024
658a2dd
Disable arm build
danielhollas Apr 19, 2024
b8153c3
More tags for ghcr.io images
danielhollas Apr 19, 2024
6a1b4da
Reword
danielhollas Apr 19, 2024
7a76a9a
Dependabot monthly
danielhollas Apr 19, 2024
2fddb79
Skip arm test
danielhollas Apr 19, 2024
2022330
Fix integration tests
danielhollas Apr 19, 2024
b68e1ff
Skip ARM64 integration test
danielhollas Apr 19, 2024
87ffc4f
Rest of publish workflow
danielhollas Apr 19, 2024
7e3cb1a
Disable dockerhub login for now
danielhollas Apr 19, 2024
366fa34
Move build-essential to lab image, hopefully speed up build a bit
danielhollas Apr 19, 2024
dcda994
Skip erlang install in base-with-services image
danielhollas Apr 19, 2024
8bdf705
Pass secrets to publish workflow
danielhollas Apr 19, 2024
c6ecf2c
Fix build
danielhollas Apr 19, 2024
faebcc0
Inherit secrets
danielhollas Apr 19, 2024
961a6d2
Reenable dockerhub login
danielhollas Apr 19, 2024
eb9ec70
Reenable arm64 build and tests
danielhollas Apr 19, 2024
a161008
Revert "Skip erlang install in base-with-services image"
danielhollas Apr 19, 2024
861b444
Run publish even if arm64 tests fail
danielhollas Apr 20, 2024
7ee4c96
Fix arm64 build
danielhollas Apr 20, 2024
a37f7f5
Enable all targets for publishing
danielhollas Apr 20, 2024
cfc094f
Remove sha tag
danielhollas Apr 20, 2024
69aeae2
Add proper if check for publish workflow
danielhollas Apr 21, 2024
cdbd9a9
Add softprops/action-gh-release
danielhollas Apr 21, 2024
6feb9a5
Remove unneeded pip check
danielhollas Apr 21, 2024
32f2a82
Revert "remove precommit from dev requirements"
danielhollas Apr 21, 2024
30eba26
Pin ubuntu runner version
danielhollas Apr 21, 2024
b9ca67d
Revert "Move build-essential to lab image, hopefully speed up build a…
danielhollas Apr 21, 2024
fad7527
Revert "Remove unneeded pip check"
danielhollas Apr 21, 2024
e538e73
minor cleanup
danielhollas Apr 23, 2024
30eb7c9
Remove bake template (#446)
danielhollas Apr 29, 2024
83b1975
Move all extra apt packages to base, hopefully speeds up build
danielhollas Apr 30, 2024
cf014d9
Move erlang installation to base image
danielhollas Apr 30, 2024
3979fb4
Actually install erlang this time
danielhollas Apr 30, 2024
6a4f8fe
Combine mamba install call for amd64 build in base-with-services
danielhollas Apr 30, 2024
0f647b0
fix1
danielhollas Apr 30, 2024
fb44554
fix2
danielhollas Apr 30, 2024
263b7c6
Update stack/base-with-services/Dockerfile
danielhollas Apr 30, 2024
9190d81
Simplify build workflow (#449)
danielhollas Apr 30, 2024
57f04ac
try without push-by-digest
danielhollas May 8, 2024
1cc9837
Revert "try without push-by-digest"
danielhollas May 8, 2024
8f8a30e
Tweak name
danielhollas May 8, 2024
c555da0
Fix local build
danielhollas May 8, 2024
eb4d201
Fix dodo.py
danielhollas May 8, 2024
aa7f2e5
Merge branch 'main' into remove-artifacts
danielhollas May 9, 2024
85b011e
Do not allow special tags
danielhollas May 9, 2024
d5f8b9a
Run on buildjet temporarily
danielhollas May 9, 2024
569bb8c
Link to GH issue
danielhollas May 9, 2024
00bd176
Shorten main name
danielhollas May 8, 2024
d896289
Fix buildjet setup-python
danielhollas May 9, 2024
251e6ba
Nope, setup-python simply doesn't work with linux/arm64 arch
danielhollas May 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 0 additions & 27 deletions .github/actions/create-dev-env/action.yml

This file was deleted.

31 changes: 0 additions & 31 deletions .github/actions/load-image/action.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: daily
interval: monthly
groups:
gha-dependencies:
patterns:
Expand Down
77 changes: 77 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
name: Build images and upload them to ghcr.io

env:
BUILDKIT_PROGRESS: plain

on:
workflow_call:
inputs:
runsOn:
description: GitHub Actions Runner image
required: true
type: string
platforms:
description: Target platforms for the build (linux/amd64 and/or linux/arm64)
required: true
type: string
outputs:
images:
description: Images identified by digests
value: ${{ jobs.build.outputs.images }}

jobs:
build:
name: ${{ inputs.platforms }}
runs-on: ${{ inputs.runsOn }}
timeout-minutes: 120

outputs:
images: ${{ steps.bake_metadata.outputs.images }}

# Make sure we fail if any command in a piped command sequence fails
defaults:
run:
shell: bash -e -o pipefail {0}

steps:

- name: Checkout Repo ⚡️
uses: actions/checkout@v4

- name: Set up QEMU
if: ${{ inputs.platforms != 'linux/amd64' }}
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to GitHub Container Registry 🔑
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build and upload to ghcr.io 📤
id: build-upload
uses: docker/bake-action@v4
with:
push: true
# Using provenance to disable default attestation so it will build only desired images:
# https://github.com/orgs/community/discussions/45969
provenance: false
set: |
*.platform=${{ inputs.platforms }}
*.output=type=registry,push-by-digest=true,name-canonical=true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why the push-by-digest is used for? From https://docs.docker.com/build/exporters/image-registry/ it says push without name. I assume with using this option, in the build phase the image should not tagged (see also my comment on <image>-meta, which give the tag for the image.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked again, and it is needed, because without it a latest tag will be automatically added since we didn't specify any tags. See e.g. here: https://github.com/aiidalab/aiidalab-docker-stack/pkgs/container/base

image

This is a quite important trick that makes everything work by using digests instead of tags.

danielhollas marked this conversation as resolved.
Show resolved Hide resolved
files: |
docker-bake.hcl
build.json
.github/workflows/env.hcl
Comment on lines +64 to +70
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
set: |
*.platform=${{ inputs.platforms }}
*.output=type=registry,push-by-digest=true,name-canonical=true
files: |
docker-bake.hcl
build.json
.github/workflows/env.hcl
set: |
*.platform=linux/amd64
*.output=type=registry,push-by-digest=true,name-canonical=true
*.cache-from=type=local,src=/tmp/.buildx-cache
*.cache-to=type=local,mode=max,dest=/tmp/.buildx-cache-new
files: |
docker-bake.hcl
build.json
# Temp fix for issue of fast size grow in local caching:
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Could you give it a try on caching? This may further speed the build?

Copy link
Contributor Author

@danielhollas danielhollas May 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting find, thanks! I would like to keep this PR as simple as possible so that we have a simple reference that we can always go back to.
I'd like to try it in a subsequent PR if that is okay with you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!


- name: Set output variables
id: bake_metadata
run: |
.github/workflows/extract-image-names.sh | tee -a "${GITHUB_OUTPUT}"
env:
BAKE_METADATA: ${{ steps.build-upload.outputs.metadata }}
73 changes: 0 additions & 73 deletions .github/workflows/docker-build-test-upload.yml

This file was deleted.

65 changes: 0 additions & 65 deletions .github/workflows/docker-merge-tags.yml

This file was deleted.

94 changes: 0 additions & 94 deletions .github/workflows/docker-push.yml

This file was deleted.

Loading