Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(image registry): allow configuring another image registry #479

Merged
merged 7 commits into from
Aug 28, 2024

Conversation

DSchmidtDev
Copy link
Contributor

This PR adds the ability to configure globally another image registry like your personal docker mirror.
If no global registry is set the default one configured in the service itself is used.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

@DSchmidtDev DSchmidtDev marked this pull request as ready for review June 19, 2024 14:00
@DSchmidtDev
Copy link
Contributor Author

@darnaut or @david-leifker would be great if you have time for some review :)
Thanks!

@DSchmidtDev DSchmidtDev force-pushed the feat/global_container_registry branch from 3f8d229 to 376d51c Compare June 20, 2024 17:57
Copy link

This PR is stale. We will close it in 30 days if there is no comment or activity. If you want feedback but not able to get it on github please head to #contribute channel in slack at https://slack.datahubproject.io.

@github-actions github-actions bot added the stale label Jul 22, 2024
@DSchmidtDev
Copy link
Contributor Author

comment to keep it active. Maybe someone please review? :)
@RyanHolstien

@github-actions github-actions bot removed the stale label Aug 1, 2024
@DSchmidtDev DSchmidtDev force-pushed the feat/global_container_registry branch from 90178d0 to 4d5a2e4 Compare August 21, 2024 07:06
@@ -61,7 +61,8 @@ spec:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ required "Global or specific tag is required" (.Values.image.tag | default .Values.global.datahub.version) }}"
{{- $registry := .Values.global.imageRegistry | default .Values.image.registry -}}
Copy link
Contributor

@darnaut darnaut Aug 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add this as helpers (e.g. charts/datahub/templates/_helpers.tpl) instead of duplicating the logic across many templates?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. I added the usage of templates. Unfortunately especially the datahub-upgrade jobs don't use the same mechanisms and structuring like subcharts so I simply duplicated the templates.
One idea would be to re-map the variables like a context but I am not familiar with it yet and do not want to blow up this PR more.
But would be a refactoring to do after that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking better. I provided an example that removes the need to duplicate templates.

@DSchmidtDev DSchmidtDev force-pushed the feat/global_container_registry branch from 54a7fc6 to 4b7c990 Compare August 27, 2024 06:49
@DSchmidtDev DSchmidtDev force-pushed the feat/global_container_registry branch from 471caa1 to f6f10aa Compare August 27, 2024 08:27
@DSchmidtDev DSchmidtDev requested a review from darnaut August 27, 2024 16:12
Create image registry, name and tag for mysql setup job
*/}}
{{- define "mysqlSetupJob.image" -}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to duplicate for each image, just have a generic one like:

{{/*
Create image registry, name and tag for a datahub component 
*/}}
{{- define "datahub.image" -}}
{{- $registry := .imageRegistry | default .image.registry -}}
{{ $registry }}/{{ .image.repository }}:{{ required "Global or specific tag is required" (.image.tag | default .version) -}}
{{- end -}}

and include it as:

image: {{ include "datahub.image" (dict "imageRegistry" .Values.global.imageRegistry "version" .Values.global.datahub.version "image" .Values.image) }}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah very nice. Did not know this trick :)

@@ -123,6 +123,7 @@ datahub-ingestion-cron:
elasticsearchSetupJob:
enabled: true
image:
registry: docker.io
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't it be better to just set global.imageRegistry? This way image.registry would be used for overrides.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought to leave it at the lowest level so every job itself is consistent to use docker.io if someone does not adapt anything. But if one wants to use e.g. a mirror it just needs to be overridden globally.

But you are right, if its set globally it takes precedence over the local config. I may need to adapt the template as well. Which way do you prefer? I can adapt it according to it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the latest commit I adapted the order. What do you think about?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I suspect most users wanting to change the registry would want to do so for all images (e.g. their cluster does not have access to docker-hub/internet), otherwise it's to overwrite a specific image.

@DSchmidtDev DSchmidtDev force-pushed the feat/global_container_registry branch from a75c444 to 98511c7 Compare August 28, 2024 08:08
@darnaut darnaut merged commit 185d126 into acryldata:master Aug 28, 2024
1 check passed
@DSchmidtDev DSchmidtDev deleted the feat/global_container_registry branch August 29, 2024 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants