Add image sync workflow #5

sd109 · 2023-05-10T09:47:03Z

No description provided.

Also comments and formatting

mkjpryor

We also need to capture the default notebook images in the daskhub and jupyterhub manifests

.github/workflows/sync-images.yaml

skopeo-manifests/skopeo-manifest-kubeflow.yml

- Trigger workflow on push - Format sync command more cleanly - Use GITHUB_WORKSPACE var - Test output capturing for catching sync errors

mkjpryor

This looks good. Last remaining comment is whether these manifests really need the skopeo-manifest- prefix, since they are already in a folder called skopeo-manifests?

skopeo-manifests/kubeflow.yml

Changes addressed

mkjpryor

I hadn't noticed the Kubernetes images in here before, sorry!

mkjpryor · 2023-07-11T08:42:22Z

.github/workflows/sync-images.yaml

+        # TODO: Revert to main branch before merging
+        with:
+          ref: skopeo


@sd109 We just need to do what this says :)

mkjpryor · 2023-07-11T08:47:11Z

kubeflow-azimuth/overlay/kustomization.yaml

+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/tensorboard-controller
+- name: docker.io/kubeflownotebookswg/tensorboards-web-app
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/tensorboards-web-app
+- name: docker.io/kubeflownotebookswg/volumes-web-app
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/volumes-web-app
+- name: docker.io/metacontrollerio/metacontroller
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/metacontrollerio/metacontroller
+- name: gcr.io/arrikto/kubeflow/oidc-authservice
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/arrikto/kubeflow/oidc-authservice
+- name: gcr.io/knative-releases/knative.dev/eventing/cmd/controller
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/eventing/cmd/controller
+- name: gcr.io/knative-releases/knative.dev/eventing/cmd/webhook
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/eventing/cmd/webhook
+- name: gcr.io/knative-releases/knative.dev/net-istio/cmd/controller
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/net-istio/cmd/controller
+- name: gcr.io/knative-releases/knative.dev/net-istio/cmd/webhook
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/net-istio/cmd/webhook
+- name: gcr.io/knative-releases/knative.dev/serving/cmd/activator
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/serving/cmd/activator
+- name: gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler
+- name: gcr.io/knative-releases/knative.dev/serving/cmd/controller
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/serving/cmd/controller
+- name: gcr.io/knative-releases/knative.dev/serving/cmd/domain-mapping-webhook
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/serving/cmd/domain-mapping-webhook
+- name: gcr.io/knative-releases/knative.dev/serving/cmd/domain-mapping
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/serving/cmd/domain-mapping
+- name: gcr.io/knative-releases/knative.dev/serving/cmd/webhook
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/knative-releases/knative.dev/serving/cmd/webhook
+- name: gcr.io/kubebuilder/kube-rbac-proxy
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/kubebuilder/kube-rbac-proxy
+- name: gcr.io/kubebuilder/kube-rbac-proxy
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/kubebuilder/kube-rbac-proxy
+- name: gcr.io/ml-pipeline/api-server
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/api-server
+- name: gcr.io/ml-pipeline/cache-server
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/cache-server
+- name: gcr.io/ml-pipeline/frontend
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/frontend
+- name: gcr.io/ml-pipeline/metadata-envoy
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/metadata-envoy
+- name: gcr.io/ml-pipeline/metadata-writer
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/metadata-writer
+- name: gcr.io/ml-pipeline/minio
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/minio
+- name: gcr.io/ml-pipeline/mysql
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/mysql
+- name: gcr.io/ml-pipeline/persistenceagent
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/persistenceagent
+- name: gcr.io/ml-pipeline/scheduledworkflow
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/scheduledworkflow
+- name: gcr.io/ml-pipeline/viewer-crd-controller
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/viewer-crd-controller
+- name: gcr.io/ml-pipeline/visualization-server
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/visualization-server
+- name: gcr.io/ml-pipeline/workflow-controller
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/ml-pipeline/workflow-controller
+- name: gcr.io/tfx-oss-public/ml_metadata_store_server
+  newName: ghcr.io/stackhpc/azimuth-charts/gcr.io/tfx-oss-public/ml_metadata_store_server
+- name: docker.io/kserve/kserve-controller
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kserve/kserve-controller
+- name: docker.io/kserve/models-web-app
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kserve/models-web-app
+- name: docker.io/kubeflow/training-operator
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflow/training-operator
+- name: docker.io/kubeflownotebookswg/jupyter-pytorch-cuda-full
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/jupyter-pytorch-cuda-full
+- name: docker.io/kubeflownotebookswg/jupyter-pytorch-full
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/jupyter-pytorch-full
+- name: docker.io/kubeflownotebookswg/jupyter-scipy
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/jupyter-scipy
+- name: docker.io/kubeflownotebookswg/jupyter-tensorflow-cuda-full
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/jupyter-tensorflow-cuda-full
+- name: docker.io/kubeflownotebookswg/jupyter-tensorflow-full
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/kubeflownotebookswg/jupyter-tensorflow-full
+- name: mysql
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/library/mysql
+- name: python
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/library/python
+- name: quay.io/jetstack/cert-manager-cainjector
+  newName: ghcr.io/stackhpc/azimuth-charts/quay.io/jetstack/cert-manager-cainjector
+- name: quay.io/jetstack/cert-manager-controller
+  newName: ghcr.io/stackhpc/azimuth-charts/quay.io/jetstack/cert-manager-controller
+- name: quay.io/jetstack/cert-manager-webhook
+  newName: ghcr.io/stackhpc/azimuth-charts/quay.io/jetstack/cert-manager-webhook
+- name: registry.k8s.io/coredns/coredns
+  newName: ghcr.io/stackhpc/azimuth-charts/registry.k8s.io/coredns/coredns
+- name: registry.k8s.io/etcd
+  newName: ghcr.io/stackhpc/azimuth-charts/registry.k8s.io/etcd
+- name: registry.k8s.io/kube-apiserver
+  newName: ghcr.io/stackhpc/azimuth-charts/registry.k8s.io/kube-apiserver
+- name: registry.k8s.io/kube-controller-manager
+  newName: ghcr.io/stackhpc/azimuth-charts/registry.k8s.io/kube-controller-manager
+- name: registry.k8s.io/kube-proxy
+  newName: ghcr.io/stackhpc/azimuth-charts/registry.k8s.io/kube-proxy
+- name: registry.k8s.io/kube-scheduler
+  newName: ghcr.io/stackhpc/azimuth-charts/registry.k8s.io/kube-scheduler
+- name: docker.io/tensorflow/tensorflow
+  newName: ghcr.io/stackhpc/azimuth-charts/docker.io/tensorflow/tensorflow


Some of these images belong to the underlying Kubernetes rather than kubeflow.

We should just grep the generated kubeflow manifests for image: and see which actually belong to kubeflow. Alternatively, we could do a diff before and after installing kubeflow to determine exactly which, but things like kube-apiserver, kube-controller-manager, kube-proxy and kube-scheduler shouldn't be in there.

mkjpryor · 2023-07-11T08:48:03Z

skopeo-manifests/kubeflow.yml

+registry.k8s.io:
+  images:
+    coredns/coredns:
+    - v1.9.3
+    etcd:
+    - 3.5.6-0
+    kube-apiserver:
+    - v1.26.3
+    kube-controller-manager:
+    - v1.26.3
+    kube-proxy:
+    - v1.26.3
+    kube-scheduler:
+    - v1.26.3


Same as above - I'd be surprised if any of these images are actually part of kubeflow.

Scott Davidson added 19 commits May 9, 2023 16:05

Add skopeo sync workflow

c332c8f

Trigger workflow on push

bd8d12f

Fix filenames in test matrix

2e4afcd

Fix filenames in test matrix

d736820

Fix manifest format

73be504

Fix missing '/' separators

a77d8fb

Check skopeo version

47d3179

Test docker option

6c4e798

Use containerised skopeo

5f7d688

Fix container version

c9cfd65

Run sync in container

c156128

Fix path

3ed77fb

Checkout correct branch in workflow

50ca070

Debug test

80473c7

Debug test

f6c940a

Debug test

3021f82

Fix container mount path

17d4d89

Also comments and formatting

Add check for changes

5d18b20

Fix check for changes

e4bf910

sd109 requested a review from mkjpryor May 10, 2023 09:47

mkjpryor requested changes May 10, 2023

View reviewed changes

.github/workflows/sync-images.yaml Outdated Show resolved Hide resolved

.github/workflows/sync-images.yaml Outdated Show resolved Hide resolved

skopeo-manifests/skopeo-manifest-kubeflow.yml Outdated Show resolved Hide resolved

Scott Davidson added 9 commits May 11, 2023 11:14

Address requested changes:

0b93f3f

- Trigger workflow on push - Format sync command more cleanly - Use GITHUB_WORKSPACE var - Test output capturing for catching sync errors

Test output capture

88af872

Fix typo

d9a0a0f

Write to file instead

126b4df

Write stderr to file too

8f06178

Add error checking for sync output

d33dadb

Duplicate sync output with tee

63982ca

Also pipe stderr to tee

d0d9f05

Fix sha256 tag handling

96705ca

Formatting

ad6a682

sd109 requested a review from mkjpryor May 15, 2023 11:47

mkjpryor requested changes May 15, 2023

View reviewed changes

Scott Davidson added 3 commits May 25, 2023 11:58

Shorten manifest names

1de9be4

Update file list

0b639fd

Comments and formatting

354de67

mkjpryor previously requested changes May 26, 2023

View reviewed changes

skopeo-manifests/kubeflow.yml Outdated Show resolved Hide resolved

skopeo-manifests/kubeflow.yml Outdated Show resolved Hide resolved

Scott Davidson added 10 commits May 30, 2023 09:39

Be explicit about defaulting to docker.io

a2a7b1d

Merge branch 'main' into skopeo

6222b28

Update ignores

231f93a

Use stackhpc ghcr container images

d442ec6

Formatting

807ac24

Fix traefik image config

d5da99e

Fix container image paths

29b285a

Fix ghcr image paths

c1a5f32

Formatting

e8c502e

Comments and formatting

295f1ae

sd109 mentioned this pull request May 31, 2023

Add automated checking of skopeo-synced images #6

Open

Scott Davidson added 7 commits May 31, 2023 16:32

Force ghcr images as kustomization instead

65f97ce

Don't hard-code image tag

0b7566a

Ignore vscode dir

7cc3fcd

Remove explicit image tags

4900d5d

Fix python & mysql image names

34aac8b

Handle images defined in config maps

724ef7e

Update comments

3d86b3f

sd109 requested a review from mkjpryor June 15, 2023 16:26

Scott Davidson added 2 commits June 15, 2023 17:27

Fix linux sed usage

60b39ea

Fix linux sed usage properly

6aa158d

mkjpryor requested changes Jul 11, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add image sync workflow #5

Add image sync workflow #5

sd109 commented May 10, 2023

mkjpryor left a comment

mkjpryor left a comment

mkjpryor left a comment

mkjpryor Jul 11, 2023

mkjpryor Jul 11, 2023

mkjpryor Jul 11, 2023

Add image sync workflow #5

Are you sure you want to change the base?

Add image sync workflow #5

Conversation

sd109 commented May 10, 2023

mkjpryor left a comment

Choose a reason for hiding this comment

mkjpryor left a comment

Choose a reason for hiding this comment

mkjpryor left a comment

Choose a reason for hiding this comment

mkjpryor Jul 11, 2023

Choose a reason for hiding this comment

mkjpryor Jul 11, 2023

Choose a reason for hiding this comment

mkjpryor Jul 11, 2023

Choose a reason for hiding this comment