Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[scalar-manager] Create the Helm chart for Scalar Manager #254

Merged
merged 12 commits into from
Jun 12, 2024
12 changes: 8 additions & 4 deletions charts/scalar-manager/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,17 @@ type: application
version: 2.0.0-SNAPSHOT
appVersion: 2.0.0-SNAPSHOT
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we only have 1.0.0-SNAPSHOT images.
I use 2.0.0-SNAPSHOT here because we will bump the SNAPSHOT images version to 2.0.0 after officially release Scalar Manager.

deprecated: false
icon: https://scalar-labs.com/wp-content/themes/scalar/assets/img/logo_scalar.svg
keywords:
- scalardb
- scalardl
- scalar-manager
- scalar-manager
- scalardb-cluster
- scalardl-ledger
- scalardl-auditor
- scalar-admin-for-kubernetes
home: https://scalar-labs.com/
sources:
- https://github.com/scalar-labs/scalar-manager
- https://github.com/scalar-labs/scalar-manager-api
- https://github.com/scalar-labs/scalar-manager-web
maintainers:
- name: Takanori Yokoyama
email: [email protected]
Expand Down
52 changes: 37 additions & 15 deletions charts/scalar-manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,40 @@ Current chart version is `2.0.0-SNAPSHOT`

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| fullnameOverride | string | `""` | Override the fully qualified app name |
| image.pullPolicy | string | `"IfNotPresent"` | Specify a imagePullPolicy |
| image.repository | string | `"ghcr.io/scalar-labs/scalar-manager"` | Docker image |
| image.tag | string | `""` | Override the image tag whose default is the chart appVersion |
| imagePullSecrets | list | `[{"name":"reg-docker-secrets"}]` | Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace |
| nameOverride | string | `""` | Override the Chart name |
| replicaCount | int | `1` | number of replicas to deploy |
| scalarManager.grafanaUrl | string | `""` | |
| scalarManager.port | int | `5000` | The port that Scalar Manager container exposes |
| scalarManager.refreshInterval | int | `30` | |
| scalarManager.targets | list | `[]` | |
| service.port | int | `8000` | The port that service exposes |
| service.type | string | `"ClusterIP"` | The service type |
| serviceAccount.automountServiceAccountToken | bool | `true` | Specify to mount a service account token or not |
| serviceAccount.serviceAccountName | string | `""` | Name of the existing service account resource |
| api.grafanaKubernetesServiceLabelName | string | `"app.kubernetes.io/name"` | |
| api.grafanaKubernetesServiceLabelValue | string | `"grafana"` | |
| api.grafanaKubernetesServicePortName | string | `"http-web"` | |
| api.helmScalarAdminForKubernetesChartName | string | `"scalar-admin-for-kubernetes"` | |
| api.helmScalarAdminForKubernetesChartVersion | string | `"1.0.0"` | |
| api.helmScalarRepositoryName | string | `"scalar-labs"` | |
| api.helmScalarRepositoryUrl | string | `"https://scalar-labs.github.io/helm-charts"` | |
| api.image.pullPolicy | string | `"IfNotPresent"` | |
| api.image.repository | string | `"ghcr.io/scalar-labs/scalar-manager-api"` | |
| api.image.tag | string | `""` | |
| api.lokiKubernetesServiceLabelName | string | `"app"` | |
| api.lokiKubernetesServiceLabelValue | string | `"loki"` | |
| api.lokiKubernetesServicePortName | string | `"http-metrics"` | |
| api.pausedStateRetentionMaxNumber | string | `"100"` | |
| api.pausedStateRetentionStorage | string | `"configmap"` | |
| api.prometheusKubernetesServiceLabelName | string | `"app"` | |
| api.prometheusKubernetesServiceLabelValue | string | `"kube-prometheus-stack-prometheus"` | |
| api.prometheusKubernetesServicePortName | string | `"http-web"` | |
| fullnameOverride | string | `""` | |
| imagePullSecrets[0].name | string | `"reg-docker-secrets"` | |
| nameOverride | string | `""` | |
| nodeSelector | object | `{}` | |
| podAnnotations | object | `{}` | |
| podLabels | object | `{}` | |
| podSecurityContext | object | `{}` | |
| replicaCount | int | `1` | |
| resources | object | `{}` | |
| securityContext | object | `{}` | |
| service.port | int | `80` | |
| service.type | string | `"LoadBalancer"` | |
| serviceAccount.automount | bool | `true` | |
| serviceAccount.create | bool | `true` | |
| serviceAccount.name | string | `""` | |
| tolerations | list | `[]` | |
| web.image.pullPolicy | string | `"IfNotPresent"` | |
| web.image.repository | string | `"ghcr.io/scalar-labs/scalar-manager-web"` | |
| web.image.tag | string | `""` | |
6 changes: 3 additions & 3 deletions charts/scalar-manager/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ app.kubernetes.io/instance: {{ .Release.Name }}
Create the name of the service account to use
*/}}
{{- define "scalar-manager.serviceAccountName" -}}
{{- if .Values.serviceAccount.serviceAccountName }}
{{- .Values.serviceAccount.serviceAccountName }}
{{- if .Values.serviceAccount.create }}
{{- default (include "scalar-manager.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- print (include "scalar-manager.fullname" .) "-sa" | trunc 63 | trimSuffix "-" }}
{{- default "default" .Values.serviceAccount.name }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I mentioned in another comment, to keep configuration consistency between each Scalar Helm Chart, I think it would be better to use the same implementation as the ScalarDB Cluster chart.
https://github.com/scalar-labs/helm-charts/blob/scalardb-cluster-1.3.2/charts/scalardb-cluster/values.yaml#L270-L274

What do you think?

{{- end }}
{{- end }}
58 changes: 0 additions & 58 deletions charts/scalar-manager/templates/deployment.yaml

This file was deleted.

9 changes: 0 additions & 9 deletions charts/scalar-manager/templates/role.yaml

This file was deleted.

13 changes: 0 additions & 13 deletions charts/scalar-manager/templates/rolebinding.yaml

This file was deleted.

10 changes: 10 additions & 0 deletions charts/scalar-manager/templates/scalar-manager/clusterrole.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question. What is the reason why we use ClusterRole instead of Role?

In other words, does Scalar Manager access the following resources that need the ClusterRole?
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#clusterrole-example

  • cluster-scoped resources (like nodes)
  • non-resource endpoints (like /healthz)
  • namespaced resources (like Pods), across all namespaces

metadata:
name: {{ include "scalar-manager.fullname" . }}
labels:
{{- include "scalar-manager.labels" . | nindent 4 }}
rules:
- apiGroups: ["", "apps", "batch", "rbac.authorization.k8s.io"]
resources: ["pods", "deployments", "services", "namespaces", "configmaps", "secrets", "cronjobs", "serviceaccounts", "roles", "rolebindings", "jobs"]
verbs: ["get", "list", "create", "patch", "delete", "update"]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put all resources and verbs in one rule for convenience.
However, this rule grants too many permissions.
I guess in better practice we should separate them accordingly?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, you should define separate rules for each api group. For example, the current configuration grants permissions to a pod resource in the apps group that does not exist. This can be a problem if this resource is created in the future and should not be granted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your advise.
I made a change according to your suggestion in 3182519

The rules became to

rules:
  - apiGroups: [""]
    resources: ["pods", "services", "namespaces", "configmaps", "secrets", "serviceaccounts"]
    verbs: ["get", "list", "create", "patch", "delete", "update"]
  - apiGroups: ["batch"]
    resources: ["cronjobs", "jobs"]
    verbs: ["get", "list", "create", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["roles", "rolebindings"]
    verbs: ["get", "list", "create", "delete"]

In the core group, for example, this application doesn't delete or update pods. If so, should we separate the resources with proper verbs in general practice?

Like, for example,

rules:
  - apiGroups: [""]
    resources: ["configmaps", "secrets", "serviceaccounts"]
    verbs: ["get", "list", "create", "patch", "delete", "update"]
  - apiGroups: [""]
    resources: ["pods", "services", "namespaces"]
    verbs: ["get", "list"]

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this application doesn't delete or update pods. If so, should we separate the resources with proper verbs in general practice?

Yes, that is correct. Based on the principle of least privilege, unnecessary privileges should not be granted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@superbrothers
Thank you very much.

Sorry, I recalled that we probably need these permissions because the application in this Helm chart (Scalar Manager) uses Helm to install another Helm chart. Allow me to keep them.

Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ include "scalar-manager.fullname" . }}
labels:
{{- include "scalar-manager.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: {{ include "scalar-manager.fullname" . }}
namespace: {{ .Release.Namespace }}
apiGroup: ""
roleRef:
kind: ClusterRole
name: {{ include "scalar-manager.fullname" . }}
apiGroup: rbac.authorization.k8s.io
12 changes: 12 additions & 0 deletions charts/scalar-manager/templates/scalar-manager/configmap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: v1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question. I think this ConfigMap is used for storing cluster and pause information. And, it seems that this chart adds create permission for ConfigMap resources.

Do we need to create this ConfigMap on the helm chart side?

I want to confirm whether Scalar Manager API can create ConfigMap on the Scalar Manager side or not.

(If Scalar Manager can create ConfigMap, I want to know the reason why we need to create this ConfigMap on the helm chart side.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, we will update Scalar Manager API to create ConfigMap by itself and remove this ConfigMap from the helm chart side in the future release.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, we noticed that this ConfigMap overwrites some stored metadata (e.g., managed cluster or pause duration info) when we run the helm upgrade command. So, it would be better to remove this ConfigMap from the helm chart side and create it on the Scalar Manager API side.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We address this issue on the Scalar Manager API side.
https://github.com/scalar-labs/scalar-manager-api/pull/90

kind: ConfigMap
metadata:
name: {{ include "scalar-manager.fullname" . }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: {{ include "scalar-manager.fullname" . }}
name: {{ include "scalar-manager.fullname" . }}-metadata

I think this ConfigMap name scalar-manager might cause some confusion. For example, there might be some users who think this ConfigMap includes the configuration of Scalar Manager itself.

So, I think it would be better to describe what is this ConfigMap used for in the resource name.

What do you think?

namespace: {{ .Release.Namespace }}
labels:
app.kubernetes.io/app: {{ include "scalar-manager.fullname" . }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to add the label app.kubernetes.io/app in scalar-manager.selectorLabels in the _helpers.tpl file.

In that case, we can add app.kubernetes.io/app: scalar-manager label to other resources and it can help us to identify the resources related to Scalar Manager.

Is there any reason why you set app.kubernetes.io/app to this ConfigMap only?
(In other words, is there any reason why you don't set app.kubernetes.io/app label to other resources?)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, we will move this label to _helpers.tpl.

{{- include "scalar-manager.labels" . | nindent 4 }}
data:
managed-clusters: "[]"
paused-states: "[]"
paused-states-updated-at: "0"
101 changes: 101 additions & 0 deletions charts/scalar-manager/templates/scalar-manager/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "scalar-manager.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "scalar-manager.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "scalar-manager.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/scalar-manager/configmap.yaml") . | sha256sum }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
checksum/config: {{ include (print $.Template.BasePath "/scalar-manager/configmap.yaml") . | sha256sum }}

I think this ConfigMap (/scalar-manager/configmap.yaml) is used for storing cluster or pause information. And, we don't update this ConfigMap by using the helm upgrade command directly (it will be updated by Scalar Manager API).

If the above understanding is correct, this checksum/config annotation doesn't work. And, I think we don't need this annotation because we don't need to restart (re-create) pods when the ConfigMap updates.

So, I think we can remove this annotation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, we will remove this annotation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed in another comment, we decided to use application.properties. So, we will keep this annotation to apply the updated application.properties properly when we run the helm upgrade command.

{{- if .Values.podAnnotations }}
{{- toYaml .Values.podAnnotations | nindent 8 }}
{{- end }}
labels:
{{- include "scalar-manager.selectorLabels" . | nindent 8 }}
{{- if .Values.podLabels }}
{{- toYaml .Values.podLabels | nindent 8 }}
{{- end }}
spec:
restartPolicy: Always
serviceAccountName: {{ include "scalar-manager.serviceAccountName" . }}
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
terminationGracePeriodSeconds: 90
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question. How long does Scalar Manager take to stop gracefully?
To consider the value of terminationGracePeriodSeconds, I want to confirm how long it takes.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, we can use the default value. So, we can remove this configuration from here.

containers:
- name: {{ .Chart.Name }}-api
image: "{{ .Values.api.image.repository }}:{{ .Values.api.image.tag | default .Chart.AppVersion }}"
resources:
{{- toYaml .Values.resources | nindent 12 }}
ports:
- containerPort: 8080
imagePullPolicy: {{ .Values.api.image.pullPolicy }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
env:
- name: GRAFANA_KUBERNETES_SERVICE_LABEL_NAME
value: {{ .Values.api.grafanaKubernetesServiceLabelName | quote }}
- name: GRAFANA_KUBERNETES_SERVICE_LABEL_VALUE
value: {{ .Values.api.grafanaKubernetesServiceLabelValue | quote }}
- name: GRAFANA_KUBERNETES_SERVICE_PORT_NAME
value: {{ .Values.api.grafanaKubernetesServicePortName | quote }}
- name: PROMETHEUS_KUBERNETES_SERVICE_LABEL_NAME
value: {{ .Values.api.prometheusKubernetesServiceLabelName | quote }}
- name: PROMETHEUS_KUBERNETES_SERVICE_LABEL_VALUE
value: {{ .Values.api.prometheusKubernetesServiceLabelValue | quote }}
- name: PROMETHEUS_KUBERNETES_SERVICE_PORT_NAME
value: {{ .Values.api.prometheusKubernetesServicePortName | quote }}
- name: LOKI_KUBERNETES_SERVICE_LABEL_NAME
value: {{ .Values.api.lokiKubernetesServiceLabelName | quote }}
- name: LOKI_KUBERNETES_SERVICE_LABEL_VALUE
value: {{ .Values.api.lokiKubernetesServiceLabelValue | quote }}
- name: LOKI_KUBERNETES_SERVICE_PORT_NAME
value: {{ .Values.api.lokiKubernetesServicePortName | quote }}
- name: HELM_SCALAR_REPOSITORY_NAME
value: {{ .Values.api.helmScalarRepositoryName | quote }}
- name: HELM_SCALAR_REPOSITORY_URL
value: {{ .Values.api.helmScalarRepositoryUrl | quote }}
- name: HELM_SCALAR_ADMIN_FOR_KUBERNETES_CHART_NAME
value: {{ .Values.api.helmScalarAdminForKubernetesChartName | quote }}
- name : HELM_SCALAR_ADMIN_FOR_KUBERNETES_CHART_VERSION
value: {{ .Values.api.helmScalarAdminForKubernetesChartVersion | quote }}
- name: CONFIG_MAP_LABEL_NAME
value: "app.kubernetes.io/app"
- name: CONFIG_MAP_LABEL_VALUE
value: {{ include "scalar-manager.fullname" . | quote }}
- name: PAUSED_STATE_RETENTION_STORAGE
value: {{ .Values.api.pausedStateRetentionStorage | quote }}
- name: PAUSED_STATE_RETENTION_MAX_NUMBER
value: {{ .Values.api.pausedStateRetentionMaxNumber | quote }}
- name: {{ .Chart.Name }}-web
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If my remembering is correct, Scalar Manager works as follows.

+-[Kubernetes Cluster A]---+  +-[Kubernetes Cluster B]---+  +-[Kubernetes Cluster C]---+
|                          |  |                          |  |                          |
|  +--------------------+  |  |  +--------------------+  |  |  +--------------------+  |
|  | Scalar products    |  |  |  | Scalar products    |  |  |  | Scalar products    |  |
|  +--------------------+  |  |  +--------------------+  |  |  +--------------------+  |
|                          |  |                          |  |                          |
|  +--------------------+  |  |  +--------------------+  |  |  +--------------------+  |
|  | Scalar Manager API |  |  |  | Scalar Manager API |  |  |  | Scalar Manager API |  |
|  +--------------------+  |  |  +--------------------+  |  |  +--------------------+  |
|            |             |  |            |             |  |            |             |
+------------+-------------+  +------------+-------------+  +------------+-------------+
             |                             |                             |
             |                             |                             |
             |                             |                             |
             +-----------------------------+-----------------------------+
                                           |
                                           |
                                           |
                                 +---------+----------+
                                 | Scalar Manager Web |
                                 +--------------------+

So, we don't need to deploy Scalar Manager API and Scalar Manager Web in the same one pod.

Vice versa, it would be better to deploy Scalar Manager API and Scalar Manager Web separately.

Is my understanding correct? (Sorry, I might miss some Scalar Manager specifications...)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, at this time, we must deploy api and web containers in one pod. It's a specification of the current Scalar Manager.

image: "{{ .Values.web.image.repository }}:{{ .Values.web.image.tag | default .Chart.AppVersion }}"
resources:
{{- toYaml .Values.resources | nindent 12 }}
ports:
- containerPort: 3000
imagePullPolicy: {{ .Values.web.image.pullPolicy }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a confirmation. Is it fine to set the same resources and securityContext to both containers?
For example, if either one of the workloads is heavy, users might want to set resources separately.
So, I want to confirm whether the requirements of resources and securityContext are completely the same or not between Scalar Manager API and Scalar Manager Web.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, we will separate resources configuration to api.resources and web.resources.
And, we keep the securityContext as is.

securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
apiVersion: v1
kind: Service
metadata:
namespace: {{ .Release.Namespace }}
name: {{ include "scalar-manager.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "scalar-manager.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: {{ .Values.scalarManager.port }}
protocol: TCP
name: http
- protocol: TCP
name: web
port: {{ .Values.service.port }}
targetPort: 3000
selector:
{{- include "scalar-manager.selectorLabels" . | nindent 4 }}
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
{{- if not .Values.serviceAccount.serviceAccountName }}
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: {{ .Release.Namespace }}
name: {{ include "scalar-manager.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "scalar-manager.labels" . | nindent 4 }}
{{- end }}
Loading
Loading