Skip to content

Commit

Permalink
feat(datahub-ingestion-cron): added support for configuring securityC…
Browse files Browse the repository at this point in the history 
…ontext for all ingestion crons
  • Loading branch information
@upendrao
upendrao committed Oct 3, 2023
1 parent ef63fae commit caafe8f
Show file tree
Hide file tree
Showing 5 changed files with 55 additions and 35 deletions.
4 changes: 2 additions & 2 deletions charts/datahub/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: A Helm chart for LinkedIn DataHub
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.2.188
version: 0.2.189
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: 0.11.0
Expand All @@ -26,7 +26,7 @@ dependencies:
repository: file://./subcharts/datahub-mce-consumer
condition: global.datahub_standalone_consumers_enabled
- name: datahub-ingestion-cron
version: 0.2.136
version: 0.2.137
repository: file://./subcharts/datahub-ingestion-cron
condition: datahub-ingestion-cron.enabled
- name: acryl-datahub-actions
Expand Down
2 changes: 1 addition & 1 deletion charts/datahub/subcharts/datahub-ingestion-cron/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ description: A Helm chart for Kubernetes
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.2.136
version: 0.2.137
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: v0.11.0
66 changes: 34 additions & 32 deletions charts/datahub/subcharts/datahub-ingestion-cron/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,35 +4,37 @@ A Helm chart for datahub's metadata-ingestion framework with kerberos authentica

## Chart Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
| image.repository | string | `"linkedin/datahub-ingestion"` | DataHub Ingestion image repository |
| image.tag | string | `"head"` | DataHub Ingestion image tag |
| imagePullSecrets | array | `[]` (does not add image pull secrets to deployed pods) | Docker registry secret names as an array |
| labels | string | `{}` | Metadata labels to be added to each crawling cron job |
| crons | type | `{}` | A map of crawling parameters per different technology being crawler, the key in the object will be used as the name for the new cron job |
| crons.schedule | string | `"0 0 * * *"` | Cron expression (default is daily at midnight) for crawler jobs |
| crons.recipe | object | `{}` | Recipe configuration to be executed (required) |
| crons.recipe.configmapName | string | `""` | Name of configmap to be mounted containing recipe to be executed |
| crons.recipe.fileName | string | `""` | Name of property within configMap referenced by `recipe.configName` with the concrete recipe definition |
| crons.command | array | `["/bin/sh", "-c", "datahub ingest -c /etc/recipe/<crons.recipe.fileName>"]` | Array of strings denoting the crawling command to be invoked in the cron job. By default it will execute the recipe defined in the `crons.recipe` object. Cron crawling customization is possible by having extra volumes with custom logic to be executed. |
| crons.hostAliases | array | `[]` | host aliases |
| crons.env | object | `{}` | Environment variables to add to the cronjob container |
| crons.envFromSecrets | object | `{}` | Environment variables from secrets to the cronjob container |
| crons.envFromSecrets*.secret | string | | secretKeyRef.name used for environment variable |
| crons.envFromSecrets*.key | string | | secretKeyRef.key used for environment variable |
| crons.extraVolumes | array | `[]` | Additional volumes to add to the pods |
| crons.extraVolumeMounts | array | `[]` | Additional volume mounts to add to the pods |
| crons.extraInitContainers | object | `{}` | Init containers to add to the cronjob container |
| crons.serviceAccountName | string | | Service account name used for the cronjob container |
| crons.podAnnotations | object | `{}` | Annotations to add to the pods |
| crons.restartPolicy | string | `"Always"` | Pod restart policy |
| crons.concurrencyPolicy | string | `"Allow"` | Specifies how to treat concurrent executions of a job |
| crons.failedJobsHistoryLimit | integer | `1` | Number of failed finished jobs to retain |
| crons.successfulJobsHistoryLimit | integer | `3` | Number of successful finished jobs to retain |
| crons.backoffLimit | integer | `6` | Number of retries before marking job failed |
| crons.nodeSelector | object | `{}` | Node labels for pod assignment |
| crons.affinity | object | `{}` | Affinity for pod assignment |
| crons.tolerations | list | `[]` | Tolerations for pod assignment |
| crons.extraSidecars | list | `[]` | Add sidecar containers to the pod |
| Key | Type | Default | Description |
|----------------------------------|---------|------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
| image.repository | string | `"linkedin/datahub-ingestion"` | DataHub Ingestion image repository |
| image.tag | string | `"head"` | DataHub Ingestion image tag |
| imagePullSecrets | array | `[]` (does not add image pull secrets to deployed pods) | Docker registry secret names as an array |
| labels | string | `{}` | Metadata labels to be added to each crawling cron job |
| podSecurityContext | object | `{}` | Pod security context for cron jobs |
| crons | type | `{}` | A map of crawling parameters per different technology being crawler, the key in the object will be used as the name for the new cron job |
| crons.schedule | string | `"0 0 * * *"` | Cron expression (default is daily at midnight) for crawler jobs |
| crons.recipe | object | `{}` | Recipe configuration to be executed (required) |
| crons.recipe.configmapName | string | `""` | Name of configmap to be mounted containing recipe to be executed |
| crons.recipe.fileName | string | `""` | Name of property within configMap referenced by `recipe.configName` with the concrete recipe definition |
| crons.command | array | `["/bin/sh", "-c", "datahub ingest -c /etc/recipe/<crons.recipe.fileName>"]` | Array of strings denoting the crawling command to be invoked in the cron job. By default it will execute the recipe defined in the `crons.recipe` object. Cron crawling customization is possible by having extra volumes with custom logic to be executed. |
| crons.hostAliases | array | `[]` | host aliases |
| crons.env | object | `{}` | Environment variables to add to the cronjob container |
| crons.envFromSecrets | object | `{}` | Environment variables from secrets to the cronjob container |
| crons.envFromSecrets*.secret | string | | secretKeyRef.name used for environment variable |
| crons.envFromSecrets*.key | string | | secretKeyRef.key used for environment variable |
| crons.extraVolumes | array | `[]` | Additional volumes to add to the pods |
| crons.extraVolumeMounts | array | `[]` | Additional volume mounts to add to the pods |
| crons.extraInitContainers | object | `{}` | Init containers to add to the cronjob container |
| crons.serviceAccountName | string | | Service account name used for the cronjob container |
| crons.securityContext | object | `{}` | SecurityContext specific to each crawling cron job |
| crons.podAnnotations | object | `{}` | Annotations to add to the pods |
| crons.restartPolicy | string | `"Always"` | Pod restart policy |
| crons.concurrencyPolicy | string | `"Allow"` | Specifies how to treat concurrent executions of a job |
| crons.failedJobsHistoryLimit | integer | `1` | Number of failed finished jobs to retain |
| crons.successfulJobsHistoryLimit | integer | `3` | Number of successful finished jobs to retain |
| crons.backoffLimit | integer | `6` | Number of retries before marking job failed |
| crons.nodeSelector | object | `{}` | Node labels for pod assignment |
| crons.affinity | object | `{}` | Affinity for pod assignment |
| crons.tolerations | list | `[]` | Tolerations for pod assignment |
| crons.extraSidecars | list | `[]` | Add sidecar containers to the pod |
6 changes: 6 additions & 0 deletions charts/datahub/subcharts/datahub-ingestion-cron/templates/cron.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ spec:
{{- if .serviceAccountName }}
serviceAccountName: {{ .serviceAccountName }}
{{- end }}
securityContext:
{{- toYaml $.Values.podSecurityContext | nindent 12 }}
containers:
- name: {{ $jobName }}-crawler
image: "{{ $.Values.image.repository }}:{{ required "Global or specific tag is required" ($.Values.image.tag | default $.Values.global.datahub.version) }}"
Expand All @@ -48,6 +50,10 @@ spec:
{{- toYaml .extraVolumeMounts | nindent 14 }}
{{- end }}
command: ["/bin/sh", "-c", {{ default $defaultCommand .command }} ]
{{- if .securityContext }}
securityContext:
{{- toYaml .securityContext | nindent 14 }}
{{- end }}
env:
{{- if .env }}
{{- range $key,$value := .env }}
Expand Down
12 changes: 12 additions & 0 deletions charts/datahub/subcharts/datahub-ingestion-cron/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ image:

imagePullSecrets: []

podSecurityContext: {}
# fsGroup: 2000

crons: {}
#### Example data
#hive:
Expand Down Expand Up @@ -62,6 +65,15 @@ crons: {}
##
#serviceAccountName: "my-cron-service"

## If you want to specify container level security
#securityContext:
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000

## Add your own pod annotations.
##
#podAnnotations: {}
Expand Down

0 comments on commit caafe8f

Please sign in to comment.