Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use roles from the embedded-cluster config #4133

Merged
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/license.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
pull_request:

env:
TRIVY_VERSION: 0.44.1
TRIVY_VERSION: 0.47.0

name: License scan

Expand All @@ -16,7 +16,9 @@ jobs:
- uses: actions/checkout@v3

- name: Setup Go
uses: actions/setup-go@v4
uses: actions/setup-go@v3
with:
go-version: '^1.20.0'

- name: Install Go deps
run: go mod download
Expand Down
2 changes: 1 addition & 1 deletion deploy/okteto/okteto-v2.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1.3
FROM golang:1.20.5
FROM golang:1.21.3

EXPOSE 2345

Expand Down
2 changes: 1 addition & 1 deletion deploy/okteto/okteto.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1.3
FROM golang:1.20-bookworm as builder
FROM golang:1.21-bookworm as builder

EXPOSE 2345

Expand Down
25 changes: 14 additions & 11 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
module github.com/replicatedhq/kots

go 1.20
go 1.21.0

toolchain go1.21.3
Comment on lines +3 to +5
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this appears to be required as part of importing embedded-cluster-operator

Copy link
Contributor

@cbodonnell cbodonnell Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make it an easier import if there was a separate module for the kinds/apis as we have for kotskinds and kurlkinds? probably not necessary now, but thinking ahead for when the project inevitably grows

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be something to consider certainly


require (
cloud.google.com/go/storage v1.34.1
Expand Down Expand Up @@ -39,14 +41,15 @@ require (
github.com/mholt/archiver/v3 v3.5.1
github.com/mikesmitty/edkey v0.0.0-20170222072505-3356ea4e686a
github.com/mitchellh/hashstructure v1.1.0
github.com/onsi/ginkgo/v2 v2.11.0
github.com/onsi/gomega v1.27.10
github.com/onsi/ginkgo/v2 v2.13.0
github.com/onsi/gomega v1.29.0
github.com/open-policy-agent/opa v0.51.0
github.com/ory/dockertest/v3 v3.10.0
github.com/otiai10/copy v1.9.0
github.com/phayes/freeport v0.0.0-20220201140144-74d24b5ae9f5
github.com/pkg/errors v0.9.1
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
github.com/replicatedhq/embedded-cluster-operator v0.3.0
github.com/replicatedhq/kotskinds v0.0.0-20231004174055-e6676d808a82
github.com/replicatedhq/kurlkinds v1.3.6
github.com/replicatedhq/troubleshoot v0.76.4-0.20231102041618-a7bb9ea31e61
Expand Down Expand Up @@ -74,17 +77,17 @@ require (
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
helm.sh/helm/v3 v3.13.2
k8s.io/api v0.28.2
k8s.io/apimachinery v0.28.2
k8s.io/api v0.28.3
k8s.io/apimachinery v0.28.3
k8s.io/cli-runtime v0.28.2
k8s.io/client-go v0.28.2
k8s.io/client-go v0.28.3
k8s.io/cluster-bootstrap v0.23.6
k8s.io/helm v2.14.3+incompatible
k8s.io/kubelet v0.23.6
k8s.io/metrics v0.28.2
k8s.io/utils v0.0.0-20230505201702-9f6742963106
sigs.k8s.io/application v0.8.3
sigs.k8s.io/controller-runtime v0.16.2
sigs.k8s.io/controller-runtime v0.16.3
sigs.k8s.io/kustomize/api v0.14.0
sigs.k8s.io/kustomize/kyaml v0.14.3
sigs.k8s.io/yaml v1.3.0
Expand Down Expand Up @@ -163,7 +166,7 @@ require (
github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7 // indirect
github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emicklei/go-restful/v3 v3.10.2 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/evanphx/json-patch/v5 v5.6.0 // indirect
Expand Down Expand Up @@ -384,9 +387,9 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
k8s.io/apiextensions-apiserver v0.28.2 // indirect
k8s.io/apiserver v0.28.2 // indirect
k8s.io/component-base v0.28.2 // indirect
k8s.io/apiextensions-apiserver v0.28.3 // indirect
k8s.io/apiserver v0.28.3 // indirect
k8s.io/component-base v0.28.3 // indirect
k8s.io/klog/v2 v2.100.1 // indirect
k8s.io/kube-aggregator v0.19.12 // indirect
k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9 // indirect
Expand Down
91 changes: 71 additions & 20 deletions go.sum

Large diffs are not rendered by default.

6 changes: 2 additions & 4 deletions pkg/embeddedcluster/helmvm_node.go
Original file line number Diff line number Diff line change
Expand Up @@ -143,10 +143,8 @@ func nodeRolesFromLabels(labels map[string]string) []string {

numRolesStr, ok := labels[types.EMBEDDED_CLUSTER_ROLE_LABEL]
if !ok {
// the first node will not initially have a role label, but is a 'controller'
if val, ok := labels["node-role.kubernetes.io/control-plane"]; ok && val == "true" {
return []string{"controller"}
}
// every node will have a label like this, so if it's missing, something is wrong
fmt.Printf("failed to find role label %q", types.EMBEDDED_CLUSTER_ROLE_LABEL)
return nil
}
numRoles, err := strconv.Atoi(strings.TrimPrefix(numRolesStr, "total-"))
Expand Down
32 changes: 14 additions & 18 deletions pkg/embeddedcluster/node_join.go
Original file line number Diff line number Diff line change
Expand Up @@ -241,9 +241,14 @@ func GenerateAddNodeCommand(ctx context.Context, client kubernetes.Interface, to
// GenerateK0sJoinCommand returns the k0s node join command, without the token but with all other required flags
// (including node labels generated from the roles etc)
func GenerateK0sJoinCommand(ctx context.Context, client kubernetes.Interface, roles []string) (string, error) {
controllerRoleName, err := ControllerRoleName(ctx)
if err != nil {
return "", fmt.Errorf("failed to get controller role name: %w", err)
}

k0sRole := "worker"
for _, role := range roles {
if role == "controller" {
if role == controllerRoleName {
k0sRole = "controller"
}
}
Expand All @@ -253,7 +258,7 @@ func GenerateK0sJoinCommand(ctx context.Context, client kubernetes.Interface, ro
cmd = append(cmd, "--enable-worker")
}

labels, err := getRolesNodeLabels(ctx, client, roles)
labels, err := getRolesNodeLabels(ctx, roles)
if err != nil {
return "", fmt.Errorf("failed to get role labels: %w", err)
}
Expand Down Expand Up @@ -304,35 +309,26 @@ func getControllerNodeIP(ctx context.Context, client kubernetes.Interface) (stri
return "", fmt.Errorf("failed to find healthy controller node")
}

func getRolesNodeLabels(ctx context.Context, client kubernetes.Interface, roles []string) (string, error) {
roleLabels := getRoleListLabels(roles)
func getRolesNodeLabels(ctx context.Context, roles []string) (string, error) {
roleListLabels := getRoleListLabels(roles)

for _, role := range roles {
labels, err := getRoleNodeLabels(ctx, client, role)
if err != nil {
return "", fmt.Errorf("failed to get node labels for role %s: %w", role, err)
}
roleLabels = append(roleLabels, labels...)
labels, err := getRoleNodeLabels(ctx, roles)
if err != nil {
return "", fmt.Errorf("failed to get node labels for roles %v: %w", roles, err)
}
roleLabels := append(roleListLabels, labels...)

return strings.Join(roleLabels, ","), nil
}

// TODO: look up role in cluster config, apply additional labels based on role
func getRoleNodeLabels(ctx context.Context, client kubernetes.Interface, role string) ([]string, error) {
toReturn := []string{}

return toReturn, nil
}

// getRoleListLabels returns the labels needed to identify the roles of this node in the future
// one label will be the number of roles, and then deterministic label names will be used to store the role names
func getRoleListLabels(roles []string) []string {
toReturn := []string{}
toReturn = append(toReturn, fmt.Sprintf("%s=total-%d", types.EMBEDDED_CLUSTER_ROLE_LABEL, len(roles)))

for idx, role := range roles {
toReturn = append(toReturn, fmt.Sprintf("%s-%d=%s", types.EMBEDDED_CLUSTER_ROLE_LABEL, idx, role))
toReturn = append(toReturn, fmt.Sprintf("%s-%d=%s", types.EMBEDDED_CLUSTER_ROLE_LABEL, idx, labelify(role)))
}

return toReturn
Expand Down
140 changes: 140 additions & 0 deletions pkg/embeddedcluster/roles.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
package embeddedcluster

import (
"context"
"fmt"
"regexp"
"sort"
"strings"

embeddedclusterv1beta1 "github.com/replicatedhq/embedded-cluster-operator/api/v1beta1"
)

const DEFAULT_CONTROLLER_ROLE_NAME = "controller"

var labelValueRegex = regexp.MustCompile(`[^a-zA-Z0-9-_.]+`)

// GetRoles will get a list of role names
func GetRoles(ctx context.Context) ([]string, error) {
config, err := ClusterConfig(ctx)
if err != nil {
return nil, fmt.Errorf("failed to get cluster config: %w", err)
}

if config == nil {
// use the default nil spec
config = &embeddedclusterv1beta1.ConfigSpec{}
}

// determine role names
roles := []string{}
if config.Controller.Name != "" {
roles = append(roles, config.Controller.Name)
} else {
roles = append(roles, DEFAULT_CONTROLLER_ROLE_NAME)
}

for _, role := range config.Custom {
if role.Name != "" {
roles = append(roles, role.Name)
}
}

return roles, nil
}

// ControllerRoleName determines the name for the 'controller' role
// this might be part of the config, or it might be the default
func ControllerRoleName(ctx context.Context) (string, error) {
conf, err := ClusterConfig(ctx)
if err != nil {
return "", fmt.Errorf("failed to get cluster config: %w", err)
}

if conf != nil && conf.Controller.Name != "" {
return conf.Controller.Name, nil
}
return DEFAULT_CONTROLLER_ROLE_NAME, nil
}

// sort roles by name, but put controller first
func SortRoles(controllerRole string, inputRoles []string) []string {
roles := inputRoles

// determine if the controller role is present
hasController := false
controllerIdx := 0
for idx, role := range roles {
if role == controllerRole {
hasController = true
controllerIdx = idx
break
}
}

// if the controller role is present, remove it
if hasController {
roles = append(roles[:controllerIdx], roles[controllerIdx+1:]...)
}

// sort the roles
sort.Strings(roles)

// if the controller role was present, add it back to the front
if hasController {
roles = append([]string{controllerRole}, roles...)
}

return roles
}

// getRoleNodeLabels looks up roles in the cluster config and determines the additional labels to be applied from that
func getRoleNodeLabels(ctx context.Context, roles []string) ([]string, error) {
config, err := ClusterConfig(ctx)
if err != nil {
return nil, fmt.Errorf("failed to get cluster config: %w", err)
}

return getRoleLabelsImpl(config, roles), nil
}

func labelify(s string) string {
// remove illegal characters
removechars := labelValueRegex.ReplaceAllString(s, "-")
// remove leading dashes
trimmed := strings.TrimPrefix(removechars, "-")
// restrict it to 63 characters
if len(trimmed) > 63 {
trimmed = trimmed[:63]
}
// remove trailing dashes
trimmed = strings.TrimSuffix(trimmed, "-")
return trimmed
}

func getRoleLabelsImpl(config *embeddedclusterv1beta1.ConfigSpec, roles []string) []string {
toReturn := []string{}

if config == nil {
return toReturn
}

for _, role := range roles {
if role == config.Controller.Name {
for k, v := range config.Controller.Labels {
toReturn = append(toReturn, fmt.Sprintf("%s=%s", labelify(k), labelify(v)))
}
}
for _, customRole := range config.Custom {
if role == customRole.Name {
for k, v := range customRole.Labels {
toReturn = append(toReturn, fmt.Sprintf("%s=%s", labelify(k), labelify(v)))
}
}
}
}

sort.Strings(toReturn)

return toReturn
}
Loading
Loading