Skip to content

Commit

Permalink
Docs for data plane registration (#1332)
Browse files Browse the repository at this point in the history
  • Loading branch information
Miles-Garnsey authored May 28, 2024
1 parent 5e45f07 commit da96a4f
Show file tree
Hide file tree
Showing 6 changed files with 42 additions and 98 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ This section needs to be moved elsewhere, probably a dedicated page of its own.
## Connecting to remote clusters
The control plane needs to establish client connections to remote cluster where the data plane runs. Credentials are provided via a [kubeconfig](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) file that is stored in a Secret. That secret is then referenced via a `ClientConfig` custom resource.
A kubeconfig entry for a cluster hosted by a cloud provider with include an auth token for authenticated with the cloud provider. That token expires. If you use one of these kubeconfigs be aware that the operator will not be able to access the remote cluster once that token expires. For this reason it is recommended that you use the [create-clientconfig.sh](https://github.com/k8ssandra/k8ssandra-operator/blob/main/scripts/create-clientconfig.sh) script for configuring a connection to the remote cluster. This script is discussed in more detail in a later section.
A kubeconfig entry for a cluster hosted by a cloud provider with include an auth token for authenticated with the cloud provider. That token expires. If you use one of these kubeconfigs be aware that the operator will not be able to access the remote cluster once that token expires. For this reason it is recommended that you use the [k8ssandra-client](https://github.com/k8ssandra/k8ssandra-client) for configuring a connection to the remote cluster.
-->

## Installing the operator
Expand Down
6 changes: 3 additions & 3 deletions docs/content/en/install/local/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Make sure you have the following installed before going through the related inst
* [kubectl](https://kubernetes.io/docs/tasks/tools/)
and [helm v3+](https://helm.sh/docs/intro/install/) on your preferred OS.

Also, note that the [setup-kind-multicluster.sh](#setup-kind-multicluster) and [create-clientconfig.sh](#create-clientconfig) scripts mentioned below are available in the K8ssandra Operator [GitHub repo](https://github.com/k8ssandra/k8ssandra-operator/tree/main/scripts).
Also, note that the [setup-kind-multicluster.sh](#setup-kind-multicluster) script mentioned below is available in the K8ssandra Operator [GitHub repo](https://github.com/k8ssandra/k8ssandra-operator/tree/main/scripts). The [k8ssandra-client](https://github.com/k8ssandra/k8ssandra-client) tool is also helpful.

#### kind

Expand Down Expand Up @@ -64,9 +64,9 @@ The `make NUM_CLUSTERS=<number> create-kind-multicluster` command, which is show

**Tip:** kind generates a `kubeconfig` with the IP address of the API server set to `localhost` because the cluster is intended for local development. We need a `kubeconfig` with the IP address set to the internal address of the API server. The `setup-kind-mulitcluster.sh` script takes care of this requirement for you.

#### create-clientconfig
#### k8ssandra-client

[create-clientconfig.sh](https://github.com/k8ssandra/k8ssandra-operator/blob/main/scripts/create-clientconfig.sh) is in the k8ssandra-operator repo. This script is used to configure access to remote clusters, as described in subsequent topics.
The k8ssandra-client can be used to register data planes to control planes for multi-cluster use cases. See [here]({{< relref "tasks/data-plane-registration" >}}) for more details.

## Next steps

Expand Down
48 changes: 3 additions & 45 deletions docs/content/en/install/local/multi-cluster-helm/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,9 +263,9 @@ kubectl -n k8ssandra-operator get deployment k8ssandra-operator \
false
```

### Generate and install ClientConfigs
### Register the data plane to the control plane

[create-clientconfig.sh](https://github.com/k8ssandra/k8ssandra-operator/blob/main/scripts/create-clientconfig.sh) lives in the k8ssandra-operator repo. It is used to configure access to remote clusters.
[k8ssandra-client](https://github.com/k8ssandra/k8ssandra-client) is used to register data planes to control planes. It does this by creating a ClientConfig custom resource on the control plane and a secret with a kubeconfig entry providing access to the dataplane.

**Note:** K8ssandra Operator restarts automatically whenever there is a change to a `ClientConfig` (a create, update, or delete operation). This restart is done in order to update connections to remote clusters.

Expand All @@ -275,49 +275,7 @@ First, set the context to `kind-k8ssandra-0`, the control plane cluster.
kubectl config use-context kind-k8ssandra-0
```

Run the create-clientconfig.sh script, once per data plane cluster.

```bash
scripts/create-clientconfig.sh --namespace k8ssandra-operator \
--src-kubeconfig ./build/kind-kubeconfig \
--dest-kubeconfig ./build/kind-kubeconfig \
--src-context kind-k8ssandra-1 \
--dest-context kind-k8ssandra-0 \
--output-dir clientconfig
```

**Output:**

```bash
Creating clientconfig/kubeconfig
Creating secret kind-k8ssandra-1-config
Error from server (NotFound): secrets "kind-k8ssandra-1-config" not found
secret/kind-k8ssandra-1-config created
Creating ClientConfig clientconfig/kind-k8ssandra-1.yaml
clientconfig.config.k8ssandra.io/kind-k8ssandra-1 created
```

Then enter:

```bash
scripts/create-clientconfig.sh --namespace k8ssandra-operator \
--src-kubeconfig ./build/kind-kubeconfig \
--dest-kubeconfig ./build/kind-kubeconfig \
--src-context kind-k8ssandra-2 \
--dest-context kind-k8ssandra-0 \
--output-dir clientconfig
```

**Output:**

```bash
Creating clientconfig/kubeconfig
Creating secret kind-k8ssandra-2-config
Error from server (NotFound): secrets "kind-k8ssandra-2-config" not found
secret/kind-k8ssandra-2-config created
Creating ClientConfig clientconfig/kind-k8ssandra-2.yaml
clientconfig.config.k8ssandra.io/kind-k8ssandra-2 created
```
Run k8ssandra-client as described [here]({{< relref "/tasks/data-plane-registration" >}}).

### Deploy the K8ssandraCluster

Expand Down
19 changes: 4 additions & 15 deletions docs/content/en/install/local/multi-cluster-kustomize/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,12 +175,11 @@ Verify that the `K8SSANDRA_CONTROL_PLANE` environment variable is set to `false`
kubectl -n k8ssandra-operator get deployment k8ssandra-operator -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="K8SSANDRA_CONTROL_PLANE")].value}'
```

### Create a ClientConfig
Now we need to create a `ClientConfig` for the `kind-k8ssandra-1` cluster. We will use the
`create-clientconfig.sh` script which can be found
[here](https://github.com/k8ssandra/k8ssandra-operator/blob/main/scripts/create-clientconfig.sh).
### Register the data plane

Here is a summary of what the script does:
[k8ssandra-client](https://github.com/k8ssandra/k8ssandra-client) is used to register data planes to control planes. It does this by installing a ClientConfig on the control plane. K8ssandra-client data plane registration is described [here]({{< relref "/tasks/data-plane-registration" >}}).

Here is a summary of what the k8ssandra-client's `register` command does':

* Get the k8ssandra-operator service account from the data plane cluster
* Extract the service account token
Expand All @@ -192,16 +191,6 @@ Here is a summary of what the script does:
Create a `ClientConfig` in the `kind-k8ssandra-0` cluster using the service account
token and CA cert from `kind-k8ssandra-1`:

```sh
scripts/create-clientconfig.sh --namespace k8ssandra-operator \
--src-kubeconfig ./build/kind-kubeconfig \
--dest-kubeconfig ./build/kind-kubeconfig \
--src-context kind-k8ssandra-1 \
--dest-context kind-k8ssandra-0 \
--output-dir clientconfig
```
The script stores all of the artifacts that it generates in a directory which is specified with the `--output-dir` option. If not specified, a temp directory is created.

### Restart the control plane

There is a controller in the operator that watches for ClientConfig changes. When it detects a change (create/update/delete), it automatically restarts the operator.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
---
title: "Multi Cluster Connection Management"
title: "Multi Cluster Connectivity"
linkTitle: "Multi Cluster"
description: "Managing connections between multiple Kubernetes clusters in a multi cluster environment."
description: "Multi-cluster connectivity architecture."
---


## Background
K8ssandra Operator is capable of creating and managing a multi-datacenter Cassandra cluster that spans multiple Kubernetes clusters. The operator uses Kubernetes APIs to manage objects in remote clusters; therefore, it needs to create client connections with appropriate permissions to remote clusters.

Expand Down Expand Up @@ -157,9 +158,6 @@ As stated earlier, the operator only processes ClientConfigs at startup. If you

Proceed with caution before deleting a ClientConfig. If there are any K8ssandraClusters that use the kube config provided by the ClientConfig, then the operator won't be able to properly manage them.

#### Creating a ClientConfig
Creating a ClientConfig involves creating the kubeconfig file and secret. This can be error prone if done by hand. Instead use the `create-clientconfig.sh` script which can be found [here](https://github.com/k8ssandra/k8ssandra-operator/blob/main/scripts/create-clientconfig.sh).

The operator should already be installed in the remote cluster for which you want to create a ClientConfig. The operator service account is created when the operator is installed. Creating the ClientConfig requires access to that service account.

Here is a brief summary of what the script does:
Expand All @@ -171,32 +169,4 @@ Here is a brief summary of what the script does:
* Create a secret for the kubeconfig in the control plane custer
* Create a ClientConfig in the control plane cluster that references the secret.

Suppose we have two clusters with context names `kind-k8ssandra-0` and `kind-k8ssandra-1` and the control plane is running
in `kind-k8ssandra-0`. We want to create a ClientConfig for `kind-k8ssandra-1`. This can be accomplished by running the
following:

```sh
./create-clientconfig.sh \
--namespace k8ssandra-operator \
--src-kubeconfig ./kubeconfigs/k8ssandra-1 \
--dest-kubeconfig ./kubeconfigs/k8ssandra-0
```

The above assumes that each cluster has its configuration stored in distinct files. If you have one single kubeconfig
for both clusters, then you can create the ClientConfig as follows:

```sh
./create-clientconfig.sh \
--namespace k8ssandra-operator \
--src-kubeconfig ./kubeconfig \
--dest-kubeconfig ./kubeconfig \
--src-context kind-k8ssandra-1 \
--dest-context kind-k8ssandra-0
```

Here `./kubeconfig` refers to the kubeconfig file containing configurations to access both contexts `kind-k8ssandra-0`
and `kind-k8ssandra-1`.

You can specify the namespace where the secret and ClientConfig are created with the `--namespace` option.

**Note:** Remember to restart the operator in the control plane cluster after creating the ClientConfig.
See [here]({{< relref "/tasks/data-plane-registration" >}}) for more information on the dataplane registration procedure.
27 changes: 27 additions & 0 deletions docs/content/en/tasks/data-plane-registration/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
title: "Data plane registration"
linkTitle: "Data Plane Registration"
description: "Managing connections between multiple Kubernetes clusters in a multi cluster environment."
---

## Background

Details of the multi-cluster connectivity architecture can be found [here]({{< relref "reference/multi-cluster" >}}). Details regarding ClientConfigs, ServiceAccounts, and Kubeconfig secrets are dealt with there.

Proceed with caution before deleting a ClientConfig or the secrets backing it. If there are any K8ssandraClusters that use the kube config provided by the ClientConfig, then the operator won't be able to properly manage them.

#### Creating a ClientConfig and Kubeconfig secret
Registering a data plane to a control plane can be error prone if done by hand. Instead use the `k8ssandra-client` CLI tool which can be found [here](https://github.com/k8ssandra/k8ssandra-client). It can be installed as a kubectl plugin by placing it on your `$PATH` (invoked with `kubectl k8ssandra ...`) or run completely separately (invoked with `./kubectl-k8ssandra ...`).

An example command to register a data plane to a control plane would be `kubectl k8ssandra register --source-context gke_k8ssandra_us-central1-c_registration-2 --dest-context gke_k8ssandra_us-central1-c_registration-1` where the source context (`gke_k8ssandra_us-central1-c_registration-2`) is the data plane (because that's the source of the credentials) and the destination context (`gke_k8ssandra_us-central1-c_registration-1`) is the control plane.

If the operator was installed in a different namespace than `k8ssandra-operator` in either the source or the destination cluster, use the following flags to indicate the actual namespaces:

- `--dest-namespace <namespace>`
- `--source-namespace <namespace>`

A full list of options for this command can be found by using the `--help` flag.

#### Referencing a remote data plane within a K8ssandraCluster

Within the K8ssandraCluster DC list, the k8sContext can be used to reference a remote data plane. When creating ClientConfigs using k8ssandra-client, you will find that you can simply refer to the data plane using the name of the ClientConfig resource. However, if the ClientConfig has been created by hand, the ClientConfig `spec.contextName` and `meta.name` may be different, in which case the `spec.contextName` should be used. We recommend using k8ssandra-client to register data planes due to complexities like these.

0 comments on commit da96a4f

Please sign in to comment.