push image to multiple registry #1079

sarathchandra24 · 2024-03-04T23:45:31Z

Push images to multiple registries; images: fluentbit, fluent-operator

What this PR does / why we need it:

Current system is pushing images to only single registry: Docker Hub;

We want to push images to other registries too; to solve this we are pushing every image to GitHub Container Registry and then we only release image to docker hub if they are changes towards master or tags.

This is a new PR for the requested changes in #1071

Which issue(s) this PR fixes:

Fixes #1068

Does this PR introduced a user-facing change?

Additional documentation, usage docs, etc.:

We are currently defining default image names and tags
kubesphere/fluent-bit:v2.2.2
kubesphere/fluent-bit:v2.2.2-debug

This commit would be changed to tag based versioning; if a new git tag is released, new tag would be used for images.

Images will be pushing to gchr.io/fluent/fluent-operator/{imageName}:{imageVersion}

Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Sarath Chandra Oruganti <[email protected]>

Signed-off-by: Sarath Chandra Oruganti <[email protected]>

sarathchandra24 · 2024-03-07T05:59:44Z

I made a release on my fork and here are the jobs
fluent bit
fluentd
fluent operator

Github packages: I created a tag v17.0.2 and here are the packages;
fluent-bit:v17.0.2
fluent-bit:v17.0.2-debug
fluentd:v17.0.2
fluent-operator:v17.0.2

Docker Hub:
fluent-bit:17.0.2
fluent-bit:17.0.2-debug
fluentd:17.0.2
fluent-operator:17.0.2

sarathchandra24 · 2024-03-07T06:01:52Z

.github/workflows/scan-docker-image-action.yaml

+          uses: hands-lab/dockle-action@v1
+          with:
+            image: ${{ inputs.source_registry }}/${{ inputs.source_image }}
+            exit-code: "0"


Ignoring error current need to change to 1 so that we will stop releasing image to docker hub. need suggestion.

You mean we needn't to sync images to dockerhub in a scan image action?

I think we can continue the scan regardless the scan result.
Let's keep it as 0

sarathchandra24 · 2024-03-07T06:01:56Z

.github/workflows/clone-docker-image-action.yaml

+            docker run --rm  \
+              quay.io/skopeo/stable:latest \
+              copy \
+                --all \


Might have to add --additional-tag "latest" that creates a latest tag while copying currently it is not copying, need suggestion.

it's ok to leave the latest on ghcr only, we can add a note in the repo's readme saying users can fetch latest image from ghcr if they want .

And of course sync the latest to dockerhub is also reasonable

.github/workflows/build-fb-image.yaml

benjaminhuo · 2024-03-08T03:09:22Z

.github/workflows/build-fb-image.yaml

-          password: ${{ secrets.REGISTRY_PASSWORD }}
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}


Do I need to add a GitHub token to this repo's action secret?
But that token cannot start with Github, maybe we can change it to GH_TOKEN ?

As far as I know of I am using GITHUB_TOKEN only which will restrict permissions defined in the actions. I referred to the following document. I allowed the write permissions globally on a repo, and use this clause in actions

permissions: contents: read packages: write

Let's check if this works! if not we can create a secret GH_TOKEN and use it.

all right, looks like GITHUB_TOKEN is created automatically for each action and I need to create one manually

.github/workflows/build-op-image.yaml

benjaminhuo · 2024-03-08T03:27:56Z

@sarathchandra24 Thanks for the great effort, I'm going to add you as maintainer

Signed-off-by: Sarath Chandra Oruganti <[email protected]>

.github/workflows/build-op-image.yaml

* push image to multiple registry Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Sarath Chandra Oruganti <[email protected]> * github actions permissions Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Sarath Chandra Oruganti <[email protected]> * fluentd multi registry image push Signed-off-by: Sarath Chandra Oruganti <[email protected]> * Limit build to PR changes for certain files Signed-off-by: Sarath Chandra Oruganti <[email protected]> --------- Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Paul Smith <[email protected]>

benjaminhuo · 2024-05-23T01:41:05Z

.github/workflows/build-fb-image.yaml

+            type=ref,event=branch
+            type=ref,event=pr
+            type=ref,event=tag
+            type=semver,pattern={{version}}


There is a problem here that the fluent-operator and fluent bit have different version, and in this action it use the tag of fluent-operator to build fluentbit images:

when we released fluent-operator v2.8.0, the fluentbit remains 2.2.2, but fluent-bit:v2.8.0 is built instead

and this is the cause of #1178

The make file should be used to build fluentbit images instead https://github.com/fluent/fluent-operator/blob/v2.8.0/Makefile#L3

cc @sarathchandra24 @wanjunlei @wenchajun @DouglasWillis1

hmm, let me know if my understanding is correct.

If we are releasing a tag suppose v2.8.0, it is building three images fluentd:v2.8.0, fluent-operator:v2.8.0 and fluent-bit:v2.8.0. As per the workflows, all the images are using tags to release new images.

Do we need to separate them; i.e. are these images having different build and release cycles?

If we are releasing a tag suppose v2.8.0, it is building three images fluentd:v2.8.0, fluent-operator:v2.8.0 and fluent-bit:v2.8.0. As per the workflows, all the images are using tags to release new images.

Do we need to separate them; i.e. are these images having different build and release cycles?

@sarathchandra24 Yes, we need to separate them. The tag of this project should only be used to build fluent-operator while the fluentbit and fluentd have separate tags as defined in the makefile:

fluentbit:
https://github.com/fluent/fluent-operator/blob/master/Makefile#L3

fluend:
https://github.com/fluent/fluent-operator/blob/master/Makefile#L5

sarathchandra24 force-pushed the multi-registry-image-push branch from 726270b to 4a9bb83 Compare March 4, 2024 23:47

This was referenced Mar 4, 2024

Push Images to GHCR.io #1071

Closed

kubesphere fluent* images should be published to multiple registries #1068

Closed

sarathchandra24 force-pushed the multi-registry-image-push branch from 0b64928 to f44d5c1 Compare March 6, 2024 05:11

sarathchandra24 and others added 3 commits March 6, 2024 15:36

push image to multiple registry

14c8292

Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Sarath Chandra Oruganti <[email protected]>

github actions permissions

254337a

Signed-off-by: Sarath Chandra Oruganti <[email protected]> Signed-off-by: Sarath Chandra Oruganti <[email protected]>

fluentd multi registry image push

60045e6

Signed-off-by: Sarath Chandra Oruganti <[email protected]>

sarathchandra24 force-pushed the multi-registry-image-push branch from 9e59a2a to 60045e6 Compare March 6, 2024 21:36

benjaminhuo requested review from patrick-stephens, wenchajun and wanjunlei March 7, 2024 03:33

sarathchandra24 commented Mar 7, 2024

View reviewed changes

benjaminhuo requested changes Mar 8, 2024

View reviewed changes

sarathchandra24 and others added 2 commits March 7, 2024 22:32

Limit build to PR changes for certain files

36673e6

Signed-off-by: Sarath Chandra Oruganti <[email protected]>

Merge branch 'fluent:master' into multi-registry-image-push

d7e2841

benjaminhuo approved these changes Mar 8, 2024

View reviewed changes

benjaminhuo merged commit 7f91cc0 into fluent:master Mar 8, 2024
14 of 17 checks passed

benjaminhuo reviewed Mar 8, 2024

View reviewed changes

.github/workflows/build-op-image.yaml Show resolved Hide resolved

sarathchandra24 mentioned this pull request Mar 10, 2024

Fix: attach latest tag to images - DockerHub synchronize with GHCR #1086

Merged

sarathchandra24 mentioned this pull request Mar 15, 2024

bug: Missing build args in fluentd docker image #1094

Closed

benjaminhuo reviewed May 23, 2024

View reviewed changes

sarathchandra24 mentioned this pull request May 27, 2024

Fix release cycles for fluentd and fluentbit images manually. #1183

Merged

benjaminhuo mentioned this pull request Jun 4, 2024

bug: fluentd unable to start with error="fork/exec /usr/local/bundle/bin/fluentd: no such file or directory" #1187

Open

sarathchandra24 self-assigned this Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

push image to multiple registry #1079

push image to multiple registry #1079

sarathchandra24 commented Mar 4, 2024

sarathchandra24 commented Mar 7, 2024

sarathchandra24 Mar 7, 2024

benjaminhuo Mar 8, 2024

benjaminhuo Mar 8, 2024

sarathchandra24 Mar 7, 2024

benjaminhuo Mar 8, 2024 •

edited

Loading

benjaminhuo Mar 8, 2024 •

edited

Loading

sarathchandra24 Mar 8, 2024

benjaminhuo Mar 8, 2024

benjaminhuo commented Mar 8, 2024

benjaminhuo May 23, 2024 •

edited

Loading

benjaminhuo May 23, 2024

sarathchandra24 May 25, 2024 •

edited

Loading

benjaminhuo May 26, 2024 •

edited

Loading

push image to multiple registry #1079

push image to multiple registry #1079

Conversation

sarathchandra24 commented Mar 4, 2024

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduced a user-facing change?

Additional documentation, usage docs, etc.:

sarathchandra24 commented Mar 7, 2024

sarathchandra24 Mar 7, 2024

Choose a reason for hiding this comment

benjaminhuo Mar 8, 2024

Choose a reason for hiding this comment

benjaminhuo Mar 8, 2024

Choose a reason for hiding this comment

sarathchandra24 Mar 7, 2024

Choose a reason for hiding this comment

benjaminhuo Mar 8, 2024 • edited Loading

Choose a reason for hiding this comment

benjaminhuo Mar 8, 2024 • edited Loading

Choose a reason for hiding this comment

sarathchandra24 Mar 8, 2024

Choose a reason for hiding this comment

benjaminhuo Mar 8, 2024

Choose a reason for hiding this comment

benjaminhuo commented Mar 8, 2024

benjaminhuo May 23, 2024 • edited Loading

Choose a reason for hiding this comment

benjaminhuo May 23, 2024

Choose a reason for hiding this comment

sarathchandra24 May 25, 2024 • edited Loading

Choose a reason for hiding this comment

benjaminhuo May 26, 2024 • edited Loading

Choose a reason for hiding this comment

benjaminhuo Mar 8, 2024 •

edited

Loading

benjaminhuo Mar 8, 2024 •

edited

Loading

benjaminhuo May 23, 2024 •

edited

Loading

sarathchandra24 May 25, 2024 •

edited

Loading

benjaminhuo May 26, 2024 •

edited

Loading