Skip to content

Commit

Permalink
add updated helm-project-operator
Browse files Browse the repository at this point in the history
Signed-off-by: Alexandre Lamarre <[email protected]>
  • Loading branch information
alexandreLamarre committed Dec 27, 2024
1 parent b504cc4 commit c35b7b8
Show file tree
Hide file tree
Showing 84 changed files with 7,298 additions and 0 deletions.
11 changes: 11 additions & 0 deletions internal/helm-project-operator/charts/example-chart/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: v2
name: example-chart
description: Example Helm Project Operator chart
version: 0.0.0
appVersion: 0.0.0
annotations:
catalog.cattle.io/certified: rancher
catalog.cattle.io/hidden: "true"
catalog.cattle.io/release-name: example-chart
catalog.cattle.io/os: linux,windows
catalog.cattle.io/permits-os: linux,windows
5 changes: 5 additions & 0 deletions internal/helm-project-operator/charts/example-chart/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# example-chart

This chart is a dummy chart that is deployed on behalf of the default Helm Project Operator.

This chart is primarily intended for testing purposes.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
questions:
- variable: data
label: mydata
description: My Data
type: string
required: true
group: Data
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-map
namespace: {{ .Release.Namespace }}
data:
config: |-
{{ .Values.data | toYaml | indent 4 }}
project-namespaces: |-
{{ .Values.global.cattle.projectNamespaces | toYaml | indent 4 }}
project-id: |-
{{ .Values.global.cattle.projectID | toYaml | indent 4 }}
release-project-id: |-
{{ .Values.global.cattle.releaseProjectID | toYaml | indent 4 }}
project-namespace-selector: |-
{{ .Values.global.cattle.projectNamespaceSelector | toYaml | indent 4 }}
system-default-registry: |-
{{ .Values.global.cattle.systemDefaultRegistry | toYaml | indent 4 }}
cattle-url: |-
{{ .Values.global.cattle.url | toYaml | indent 4 }}
cluster-id: |-
{{ .Values.global.cattle.clusterId | toYaml | indent 4 }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-map-2
namespace: {{ .Release.Namespace }}
data:
config: |-
{{ .Values.data | toYaml | indent 4 }}
project-namespaces: |-
{{ .Values.global.cattle.projectNamespaces | toYaml | indent 4 }}
project-id: |-
{{ .Values.global.cattle.projectID | toYaml | indent 4 }}
release-project-id: |-
{{ .Values.global.cattle.releaseProjectID | toYaml | indent 4 }}
project-namespace-selector: |-
{{ .Values.global.cattle.projectNamespaceSelector | toYaml | indent 4 }}
system-default-registry: |-
{{ .Values.global.cattle.systemDefaultRegistry | toYaml | indent 4 }}
cattle-url: |-
{{ .Values.global.cattle.url | toYaml | indent 4 }}
cluster-id: |-
{{ .Values.global.cattle.clusterId | toYaml | indent 4 }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{{- if and .Values.global.rbac.create .Values.global.rbac.userRoles.create }}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Name }}-admin
namespace: {{ .Release.Namespace }}
labels:
helm.cattle.io/project-helm-chart-role: {{ .Release.Name }}
{{- if .Values.global.rbac.userRoles.aggregateToDefaultRoles }}
helm.cattle.io/project-helm-chart-role-aggregate-from: admin
{{- end }}
rules:
- apiGroups:
- "test.cattle.io"
resources:
- test
resourceNames:
- test
verbs:
- 'test'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Name }}-edit
namespace: {{ .Release.Namespace }}
labels:
helm.cattle.io/project-helm-chart-role: {{ .Release.Name }}
{{- if .Values.global.rbac.userRoles.aggregateToDefaultRoles }}
helm.cattle.io/project-helm-chart-role-aggregate-from: edit
{{- end }}
rules:
- apiGroups:
- "test.cattle.io"
resources:
- test
resourceNames:
- test
verbs:
- 'test'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Name }}-view
namespace: {{ .Release.Namespace }}
labels:
helm.cattle.io/project-helm-chart-role: {{ .Release.Name }}
{{- if .Values.global.rbac.userRoles.aggregateToDefaultRoles }}
helm.cattle.io/project-helm-chart-role-aggregate-from: view
{{- end }}
rules:
- apiGroups:
- "test.cattle.io"
resources:
- test
resourceNames:
- test
verbs:
- 'test'
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: my-dashboard-values
namespace: {{ .Release.Namespace }}
labels:
helm.cattle.io/dashboard-values-configmap: {{ .Release.Name }}
data:
data.json: |-
{{ .Values.data | toJson | indent 4 }}
34 changes: 34 additions & 0 deletions internal/helm-project-operator/charts/example-chart/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
global:
cattle:
clusterID: ""
projectID: ""
projectNamespaces: []
projectNamespaceSelector: {}
releaseProjectID: ""
systemDefaultRegistry: ""
url: ""
rbac:
## Create RBAC resources for ServiceAccounts and users
##
create: true

userRoles:
## Create default user Roles that the Helm Project Operator will automatically create RoleBindings for
##
## How does this work?
##
## The operator will watch for all subjects bound to each Kubernetes default ClusterRole in the project registration namespace
## where the ProjectHelmChart that deployed this chart belongs to; if it observes a subject bound to a particular role in
## the project registration namespace (e.g. edit) and if a Role exists that is deployed by this chart with the label
## 'helm.cattle.io/project-helm-chart-role-aggregate-from': '<role, e.g. edit>', it will automaticaly create a RoleBinding
## in the release namespace binding all such subjects to that Role.
##
## Note: while the default behavior is to use the Kubernetes default ClusterRole, the operator deployment can be configured
## to use a different set of ClusterRoles as the source of truth for admin, edit, and view permissions.
##
create: true
## Add labels to Roles to have the operator pick them up
aggregateToDefaultRoles: true

data:
hello: world
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: v2
name: helm-project-operator
description: Helm Project Operator
version: 0.2.1
appVersion: 0.2.1
annotations:
catalog.cattle.io/certified: rancher
catalog.cattle.io/display-name: Helm Project Operator
catalog.cattle.io/kube-version: '>=1.16.0-0'
catalog.cattle.io/namespace: cattle-helm-system
catalog.cattle.io/permits-os: linux,windows
catalog.cattle.io/provides-gvr: helm.cattle.io.projecthelmchart/v1alpha1
catalog.cattle.io/rancher-version: '>= 2.6.0-0'
catalog.cattle.io/release-name: helm-project-operator
catalog.cattle.io/os: linux,windows
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Helm Project Operator

## How does the operator work?

1. On deploying a Helm Project Operator, users can create ProjectHelmCharts CRs with `spec.helmApiVersion` set to `dummy.cattle.io/v1alpha1` in a **Project Registration Namespace (`cattle-project-<id>`)**.
2. On seeing each ProjectHelmChartCR, the operator will automatically deploy the embedded Helm chart on the Project Owner's behalf in the **Project Release Namespace (`cattle-project-<id>-dummy`)** based on a HelmChart CR and a HelmRelease CR automatically created by the ProjectHelmChart controller in the **Operator / System Namespace**.
3. RBAC will automatically be assigned in the Project Release Namespace to allow users to based on Role created in the Project Release Namespace with a given set of labels; this will be based on RBAC defined on the Project Registration Namespace against the [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) (see below for more information about configuring RBAC).

### What is a Project?

In Helm Project Operator, a Project is a group of namespaces that can be identified by a `metav1.LabelSelector`; by default, the label used to identify projects is `field.cattle.io/projectId`, the label used to identify namespaces that are contained within a given [Rancher](https://rancher.com/) Project.

### What is a ProjectHelmChart?

A ProjectHelmChart is an instance of a (project-scoped) Helm chart deployed on behalf of a user who has permissions to create ProjectHelmChart resources in a Project Registration namespace.

Generally, the best way to think about the ProjectHelmChart model is by comparing it to two other models:
1. Managed Kubernetes providers (EKS, GKE, AKS, etc.): in this model, a user has the ability to say "I want a Kubernetes cluster" but the underlying cloud provider is responsible for provisioning the infrastructure and offering **limited view and access** of the underlying resources created on their behalf; similarly, Helm Project Operator allows a Project Owner to say "I want this Helm chart deployed", but the underlying Operator is responsible for "provisioning" (deploying) the Helm chart and offering **limited view and access** of the underlying Kubernetes resources created on their behalf (based on configuring "least-privilege" Kubernetes RBAC for the Project Owners / Members in the newly created Project Release Namespace).
2. Dynamically-provisioned Persistent Volumes: in this model, a single resource (PersistentVolume) exists that allows you to specify a Storage Class that actually implements provisioning the underlying storage via a Storage Class Provisioner (e.g. Longhorn). Similarly, the ProjectHelmChart exists that allows you to specify a `spec.helmApiVersion` ("storage class") that actually implements deploying the underlying Helm chart via a Helm Project Operator (e.g. [`rancher/prometheus-federator`](https://github.com/rancher/prometheus-federator)).

### Configuring the Helm release created by a ProjectHelmChart

The `spec.values` of this ProjectHelmChart resources will correspond to the `values.yaml` override to be supplied to the underlying Helm chart deployed by the operator on the user's behalf; to see the underlying chart's `values.yaml` spec, either:
- View to the chart's definition located at [`rancher/helm-project-operator` under `charts/example-chart`](https://github.com/rancher/helm-project-operator/blob/main/charts/example-chart) (where the chart version will be tied to the version of this operator)
- Look for the ConfigMap named `dummy.cattle.io.v1alpha1` that is automatically created in each Project Registration Namespace, which will contain both the `values.yaml` and `questions.yaml` that was used to configure the chart (which was embedded directly into the `helm-project-operator` binary).

### Namespaces

All Helm Project Operators have three different classifications of namespaces that the operator looks out for:
1. **Operator / System Namespace**: this is the namespace that the operator is deployed into (e.g. `cattle-helm-system`). This namespace will contain all HelmCharts and HelmReleases for all ProjectHelmCharts watched by this operator. **Only Cluster Admins should have access to this namespace.**
2. **Project Registration Namespace (`cattle-project-<id>`)**: this is the set of namespaces that the operator watches for ProjectHelmCharts within. The RoleBindings and ClusterRoleBindings that apply to this namespace will also be the source of truth for the auto-assigned RBAC created in the Project Release Namespace (see more details below). **Project Owners (admin), Project Members (edit), and Read-Only Members (view) should have access to this namespace**.
> Note: Project Registration Namespaces will be auto-generated by the operator and imported into the Project it is tied to if `.Values.global.cattle.projectLabel` is provided (which is set to `field.cattle.io/projectId` by default); this indicates that a Project Registration Namespace should be created by the operator if at least one namespace is observed with that label. The operator will not let these namespaces be deleted unless either all namespaces with that label are gone (e.g. this is the last namespace in that project, in which case the namespace will be marked with the label `"helm.cattle.io/helm-project-operator-orphaned": "true"`, which signals that it can be deleted) or it is no longer watching that project (because the project ID was provided under `.Values.helmProjectOperator.otherSystemProjectLabelValues`, which serves as a denylist for Projects). These namespaces will also never be auto-deleted to avoid destroying user data; it is recommended that users clean up these namespaces manually if desired on creating or deleting a project
> Note: if `.Values.global.cattle.projectLabel` is not provided, the Operator / System Namespace will also be the Project Registration Namespace
3. **Project Release Namespace (`cattle-project-<id>-dummy`)**: this is the set of namespaces that the operator deploys Helm charts within on behalf of a ProjectHelmChart; the operator will also automatically assign RBAC to Roles created in this namespace by the Helm charts based on bindings found in the Project Registration Namespace. **Only Cluster Admins should have access to this namespace; Project Owners (admin), Project Members (edit), and Read-Only Members (view) will be assigned limited access to this namespace by the deployed Helm Chart and Helm Project Operator.**
> Note: Project Release Namespaces are automatically deployed and imported into the project whose ID is specified under `.Values.helmProjectOperator.projectReleaseNamespaces.labelValue` (which defaults to the value of `.Values.global.cattle.systemProjectId` if not specified) whenever a ProjectHelmChart is specified in a Project Registration Namespace
> Note: Project Release Namespaces follow the same orphaning conventions as Project Registration Namespaces (see note above)
> Note: if `.Values.projectReleaseNamespaces.enabled` is false, the Project Release Namespace will be the same as the Project Registration Namespace
### Helm Resources (HelmChart, HelmRelease)

On deploying a ProjectHelmChart, the Helm Project Operator will automatically create and manage two child custom resources that manage the underlying Helm resources in turn:
- A HelmChart CR (managed via an embedded [k3s-io/helm-contoller](https://github.com/k3s-io/helm-controller) in the operator): this custom resource automatically creates a Job in the same namespace that triggers a `helm install`, `helm upgrade`, or `helm uninstall` depending on the change applied to the HelmChart CR; this CR is automatically updated on changes to the ProjectHelmChart (e.g. modifying the values.yaml) or changes to the underlying Project definition (e.g. adding or removing namespaces from a project).
> **Important Note: If a ProjectHelmChart is not deploying or updating the underlying Project Monitoring Stack for some reason, the Job created by this resource in the Operator / System namespace should be the first place you check to see if there's something wrong with the Helm operation; however, this is generally only accessible by a Cluster Admin.**
- A HelmRelease CR (managed via an embedded [rancher/helm-locker](https://github.com/rancher/helm-locker) in the operator): this custom resource automatically locks a deployed Helm release in place and automatically overwrites updates to underlying resources unless the change happens via a Helm operation (`helm install`, `helm upgrade`, or `helm uninstall` performed by the HelmChart CR).
> Note: HelmRelease CRs emit Kubernetes Events that detect when an underlying Helm release is being modified and locks it back to place; to view these events, you can use `kubectl describe helmrelease <helm-release-name> -n <operator/system-namespace>`; you can also view the logs on this operator to see when changes are detected and which resources were attempted to be modified
Both of these resources are created for all Helm charts in the Operator / System namespaces to avoid escalation of privileges to underprivileged users.

### RBAC

As described in the section on namespaces above, Helm Project Operator expects that Project Owners, Project Members, and other users in the cluster with Project-level permissions (e.g. permissions in a certain set of namespaces identified by a single label selector) have minimal permissions in any namespaces except the Project Registration Namespace (which is imported into the project by default) and those that already comprise their projects. Therefore, in order to allow Project Owners to assign specific chart permissions to other users in their Project namespaces, the Helm Project Operator will automatically watch the following bindings:
- ClusterRoleBindings
- RoleBindings in the Project Release Namespace

On observing a change to one of those types of bindings, the Helm Project Operator will check whether the `roleRef` that the the binding points to matches a ClusterRole with the name provided under `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.admin`, `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.edit`, or `helmProjectOperator.releaseRoleBindings.clusterRoleRefs.view`; by default, these roleRefs correspond will correspond to `admin`, `edit`, and `view` respectively, which are the [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles).

> Note: for Rancher RBAC users, these [default Kubernetes user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) directly correlate to the `Project Owner`, `Project Member`, and `Read-Only` default Project Role Templates.
If the `roleRef` matches, the Helm Project Operator will filter the `subjects` of the binding for all Users and Groups and use that to automatically construct a RoleBinding for each Role in the Project Release Namespace with the same name as the role and the following labels:
- `helm.cattle.io/project-helm-chart-role: {{ .Release.Name }}`
- `helm.cattle.io/project-helm-chart-role-aggregate-from: <admin|edit|view>`

By default, the `example-chart` (the underlying chart deployed by Helm Project Operator) does not create any default roles; however, if a Cluster Admin would like to assign additional permissions to certain users, they can either directly assign RoleBindings in the Project Release Namespace to certain users or created Roles with the above two labels on them to allow Project Owners to control assigning those RBAC roles to users in their Project Registration namespaces.

### Advanced Helm Project Operator Configuration

|Value|Configuration|
|---|---------------------------|
|`valuesOverride`| Allows an Operator to override values that are set on each ProjectHelmChart deployment on an operator-level; user-provided options (specified on the `spec.values` of the ProjectHelmChart) are automatically overridden if operator-level values are provided. For an exmaple, see how the default value overrides `federate.targets` (note: when overriding list values like `federate.targets`, user-provided list values will **not** be concatenated) |
|`projectReleaseNamespaces.labelValues`| The value of the Project that all Project Release Namespaces should be auto-imported into (via label and annotation). Not recommended to be overridden on a Rancher setup. |
|`otherSystemProjectLabelValues`| Other namespaces that the operator should treat as a system namespace that should not be monitored. By default, all namespaces that match `global.cattle.systemProjectId` will not be matched. `kube-system` is explicitly marked as a system namespace as well, regardless of label or annotation. |
|`releaseRoleBindings.aggregate`| Whether to automatically create RBAC resources in Project Release namespaces
|`releaseRoleBindings.clusterRoleRefs.<admin\|edit\|view>`| ClusterRoles to reference to discover subjects to create RoleBindings for in the Project Release Namespace for all corresponding Project Release Roles. See RBAC above for more information |
|`hardenedNamespaces.enabled`| Whether to automatically patch the default ServiceAccount with `automountServiceAccountToken: false` and create a default NetworkPolicy in all managed namespaces in the cluster; the default values ensure that the creation of the namespace does not break a CIS 1.16 hardened scan |
|`hardenedNamespaces.configuration`| The configuration to be supplied to the default ServiceAccount or auto-generated NetworkPolicy on managing a namespace |
|`helmController.enabled`| Whether to enable an embedded k3s-io/helm-controller instance within the Helm Project Operator. Should be disabled for RKE2 clusters since RKE2 clusters already run Helm Controller to manage internal Kubernetes components |
|`helmLocker.enabled`| Whether to enable an embedded rancher/helm-locker instance within the Helm Project Operator. |
Loading

0 comments on commit c35b7b8

Please sign in to comment.