From 82879390ba36fcc3cfa99e56dda333affd19f4db Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 29 Nov 2024 17:49:06 +0100 Subject: [PATCH] [DPE-5683;DPE-5352;DPE-5354] Documentation on Kraft, Azure and AWS (#264) Co-authored-by: discourse-gatekeeper-docs-bot Co-authored-by: upload-charms-docs-bot --- docs/how-to/h-deploy-aws.md | 199 ++++++++++++++++++++++++++ docs/how-to/h-deploy-azure.md | 253 ++++++++++++++++++++++++++++++++++ docs/how-to/h-kraft-mode.md | 57 ++++++++ docs/index.md | 3 + 4 files changed, 512 insertions(+) create mode 100644 docs/how-to/h-deploy-aws.md create mode 100644 docs/how-to/h-deploy-azure.md create mode 100644 docs/how-to/h-kraft-mode.md diff --git a/docs/how-to/h-deploy-aws.md b/docs/how-to/h-deploy-aws.md new file mode 100644 index 00000000..8c18f96a --- /dev/null +++ b/docs/how-to/h-deploy-aws.md @@ -0,0 +1,199 @@ +# How to deploy on AWS + +[Amazon Web Services](https://aws.amazon.com/) is a popular subsidiary of Amazon that provides on-demand cloud computing platforms on a metered pay-as-you-go basis. Access the AWS web console at [console.aws.amazon.com](https://console.aws.amazon.com/). + +## Summary + +* [Install AWS and Juju tooling](#install-aws-and-juju-tooling) + * [Authenticate](#authenticate) +* [Bootstrap Juju controller on AWS EC2](#bootstrap-juju-controller-on-aws-ec2) +* [Deploy charms](#deploy-charms) +* [Clean up](#clean-up) + +--- + +## Install AWS and Juju tooling + +Install Juju via snap: + +```shell +sudo snap install juju --channel 3.5/stable +``` + +Follow the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) for guidance on how to install the Amazon Web Services CLI. + +To check whether both Juju and AWS CLI are correctly installed, run commands to display their versions: + +```shell +juju version + +aws --version +``` + +[details="Output example"] +```shell +3.5.4-genericlinux-amd64 + +aws-cli/2.13.25 Python/3.11.5 Linux/6.2.0-33-generic exe/x86_64.ubuntu.23 prompt/off +``` +[/details] + +### Authenticate + +[Create an IAM account](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html) or use legacy user access keys and secret key to operate AWS EC2: + +```shell +mkdir -p ~/.aws && cat <<- EOF > ~/.aws/credentials.yaml +credentials: + aws: + NAME_OF_YOUR_CREDENTIAL: + auth-type: access-key + access-key: SECRET_ACCESS_KEY_ID + secret-key: SECRET_ACCESS_KEY_VALUE +EOF +``` + +## Bootstrap Juju controller on AWS EC2 + +Add AWS credentials to Juju: + +```shell +juju add-credential aws -f ~/.aws/credentials.yaml +``` + +Bootstrap Juju controller ([check all supported configuration options](https://juju.is/docs/juju/amazon-ec2)): + +```shell +juju bootstrap aws +``` + +[details="Output example"] +```shell +Creating Juju controller "aws-us-east-1" on aws/us-east-1 +Looking for packaged Juju agent version 3.5.4 for amd64 +Located Juju agent version 3.5.4-ubuntu-amd64 at https://juju-dist-aws.s3.amazonaws.com/agents/agent/3.5.4/juju-3.5.4-linux-amd64.tgz +Launching controller instance(s) on aws/us-east-1... + - i-0f4615983d113166d (arch=amd64 mem=8G cores=2) +Installing Juju agent on bootstrap instance +Waiting for address +Attempting to connect to 54.226.221.6:22 +Attempting to connect to 172.31.20.34:22 +Connected to 54.226.221.6 +Running machine configuration script... +Bootstrap agent now started +Contacting Juju controller at 54.226.221.6 to verify accessibility... + +Bootstrap complete, controller "aws-us-east-1" is now available +Controller machines are in the "controller" model + +Now you can run + juju add-model +to create a new model to deploy workloads. +``` +[/details] + +## Deploy charms + +Create a new Juju model, if needed: + +```shell +juju add-model +``` + +[note type="caution"] +(Optional) Increase the debug level if you are troubleshooting charms: +```shell +juju model-config logging-config='=INFO;unit=DEBUG' +``` +[/note] + +Deploy and integrate Kafka and ZooKeeper: + +```shell +juju deploy zookeeper -n3 --channel 3/stable [--constraints "instance-type="] +juju deploy kafka -n3 --channel 3/stable [--constraints "instance-type="] +juju integrate kafka zookeeper +``` + +[note type="caution"] +The smallest AWS instance types may not provide sufficient resources to host a Kafka broker. We recommend choosing an instance type with a minimum of `8` GB of RAM and `4` CPU cores, such as `m7i.xlarge`. + +For more guidance on sizing production environments, see the [Requirements page](/t/charmed-kafka-reference-requirements/10563). Additional information about AWS instance types is available in the [AWS documentation](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Instances:instanceState=running). +[/note] + +We also recommend to deploy a [Data Integrator](https://charmhub.io/data-integrator) for creating an admin user to manage the content of the Kafka cluster: + +```shell +juju deploy data-integrator admin \ + --config extra-user-roles=admin \ + --config topic-name=admin-topic +``` + +And integrate it with the Kafka application: + +```shell +juju integrate kafka admin +``` + +For more information on Data Integrator and how to use it, please refer to the [how-to manage applications](/t/charmed-kafka-how-to-manage-app/10285) guide. + +## Clean up + +[note type="caution"] +Always clean AWS resources that are no longer necessary! Abandoned resources are tricky to detect and they can become expensive over time. +[/note] + +To list all controllers that have been registered to your local client, use the `juju controllers` command. + +To destroy the Juju controller and remove AWS instance (**Warning**: all your data will be permanently deleted): + +```shell +juju destroy-controller --destroy-all-models --destroy-storage --force +``` + +Should the destroying process take a long time or be seemingly stuck, proceed to delete EC2 resources also manually +via the AWS portal. See [Amazon AWS documentation](https://repost.aws/knowledge-center/terminate-resources-account-closure) for more information +on how to remove active resources no longer needed. + +After destroying the controller, check and manually delete all unnecessary AWS EC2 instances, to show the list of all your EC2 instances run the following command (make sure to use the correct region): + +```shell +aws ec2 describe-instances --region us-east-1 --query "Reservations[].Instances[*].{InstanceType: InstanceType, InstanceId: InstanceId, State: State.Name}" --output table +``` + +[details="Output example"] +```shell +------------------------------------------------------- +| DescribeInstances | ++---------------------+----------------+--------------+ +| InstanceId | InstanceType | State | ++---------------------+----------------+--------------+ +| i-0f374435695ffc54c| m7i.xlarge | terminated | +| i-0e1e8279f6b2a08e0| m7i.xlarge | terminated | +| i-061e0d10d36c8cffe| m7i.xlarge | terminated | +| i-0f4615983d113166d| m7i.xlarge | terminated | ++---------------------+----------------+--------------+ +``` +[/details] + +List your Juju credentials with the `juju credentials` command: + +```shell +... +Client Credentials: +Cloud Credentials +aws NAME_OF_YOUR_CREDENTIAL +... +``` + +Remove AWS EC2 CLI credentials from Juju: + +```shell +juju remove-credential aws NAME_OF_YOUR_CREDENTIAL +``` + +Finally, remove AWS CLI user credentials (to avoid forgetting and leaking): + +```shell +rm -f ~/.aws/credentials.yaml +``` \ No newline at end of file diff --git a/docs/how-to/h-deploy-azure.md b/docs/how-to/h-deploy-azure.md new file mode 100644 index 00000000..bf3ab9a4 --- /dev/null +++ b/docs/how-to/h-deploy-azure.md @@ -0,0 +1,253 @@ +# How to deploy on Azure + +[Azure](https://azure.com/) is the cloud computing platform developed by Microsoft. It has management, access and development of applications and services to individuals, companies, and governments through its global infrastructure. Access the Azure web console at [portal.azure.com](https://portal.azure.com/). + +## Summary + +* [Install Azure and Juju tooling](#install-client-environment) + * [Authenticate](#authenticate) +* [Bootstrap Juju controller on Azure](#bootstrap-juju-controller-on-azure) +* [Deploy charms](#deploy-charms) +* [Clean up](#clean-up) + +--- + +## Install Client Environment + +[note type="caution"] +**WARNING**: The current limitations: +* Only supported starting Juju 3.6 (currently edge) +* Juju cli should be on Azure VM for it to be able to reach cloud metadata endpoint. +* Managed Identity and the Juju resources should be on the same Azure subscription +* The current setup has been tested on Ubuntu 22.04+ +[/note] + +### Juju + +Install Juju via snap: + +```shell +sudo snap install juju --channel 3.6/edge +``` + +Check that the Juju version is correctly installed: + +```shell +juju version +``` + +[details="Output example"] +```shell +3.6-rc1-genericlinux-amd64 +``` +[/details] + +### Azure CLI + +Install the Azure CLI on Linux distributions by following the [Azure CLI's installation guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt). + +Verify that it is correctly installed: + +```shell +az --version +``` + +[details="Output example"] +```shell +azure-cli 2.65.0 + +core 2.65.0 +telemetry 1.1.0 + +Dependencies: +msal 1.31.0 +azure-mgmt-resource 23.1.1 + +Python location '/opt/az/bin/python3' +Extensions directory '/home/deusebio/.azure/cliextensions' + +Python (Linux) 3.11.8 (main, Sep 25 2024, 11:33:44) [GCC 11.4.0] + +Legal docs and information: aka.ms/AzureCliLegal + +Your CLI is up-to-date. +``` +[/details] + +### Authenticate + +[note] +For more information on how to authenticate, refer to the [Juju documentation](/t/the-microsoft-azure-cloud-and-juju/1086) and [a dedicated user guide on how to register Azure on Juju](/t/how-to-use-juju-with-microsoft-azure/15219). +[/note] + +First of all, retrieve your subscription id: + +```shell +az login +``` + +After authenticating via the web browser, you will be shown a list of information and a table with the subscriptions connected to your user, e.g.: + +```shell +No Subscription name Subscription ID Tenant +----- ------------------------------------ ------------------------------------ ------------- +[1] * canonical.com +[2] canonical.com +``` + +In the prompt, select the subscription id you would like to connect the controller to, and store the id +as it will be needed in the next step when bootstrapping the controller. + +### Bootstrap Juju controller on Azure + +First, you need to add a set of credentials to your Juju client: + +```shell +juju add-credentials azure +``` + +This will start a script that will help you set up the credentials, where you will be asked: + +* `credential-name` — a sensible name that will help you identify the credential set, say `` +* `region` — a default region that is most convenient to deploy your controller and applications. Note that credentials are not region-specific +* `auth type` — authentication type. Select `interactive`, which is the recommended way to authenticate to Azure using Juju +* `subscription_id` — the value `` taken in the previous step +* `application_name` — any unique string to avoid collision with other users or applications +* `role-definition-name` — any unique string to avoid collision with other users or applications, and store it as `` + +Next, you will be asked to authenticate the requests via your web browser with the following message: + +```shell +To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code to authenticate. +``` + +In the browser, open the [authentication page](https://microsoft.com/devicelogin) and enter the code `` provided in the output. + +You will be asked to authenticate twice, for allowing the creation of two different resources in Azure. + +You will see a message that the credentials have been added locally: + +```shell +Credential added locally for cloud "azure". +``` + +Once the credentials are correctly added, we can bootstrap a controller: + +```shell +juju bootstrap azure +``` + +## Deploy charms + +Create a new Juju model, if needed: + +```shell +juju add-model +``` + +[note type="caution"] +(Optional) Increase the debug level if you are troubleshooting charms: +```shell +juju model-config logging-config='=INFO;unit=DEBUG' +``` +[/note] + +Deploy and integrate Kafka and ZooKeeper: + +```shell +juju deploy zookeeper -n3 --channel 3/stable [--constraints "instance-type="] +juju deploy kafka -n3 --channel 3/stable [--constraints "instance-type="] +juju integrate kafka zookeeper +``` + +[note type="caution"] +Note that the smallest instance types on Azure may not have enough resources for hosting +a Kafka broker. We recommend selecting an instance type that provides at the very least `8` GB of RAM and `4` cores, e.g. `Standard_A4_v2`. +For more guidance on production environment sizing, see the [Requirements page](/t/charmed-kafka-reference-requirements/10563). +You can find more information about the available instance types in the [Azure documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview). +[/note] + +We also recommend to deploy a [Data Integrator](https://charmhub.io/data-integrator) for creating an admin user to manage the content of the Kafka cluster: + +```shell +juju deploy data-integrator admin --channel edge \ + --config extra-user-roles=admin \ + --config topic-name=admin-topic +``` + +And integrate it with the Kafka application: + +```shell +juju integrate kafka admin +``` + +For more information on Data Integrator and how to use it, please refer to the [how-to manage applications](/t/charmed-kafka-how-to-manage-app/10285) guide. + +## Clean up + +[note type="caution"] +Always clean Azure resources that are no longer necessary! Abandoned resources are tricky to detect and they can become expensive over time. +[/note] + +To list all controllers that have been registered to your local client, use the `juju controllers` command. + +To destroy the Juju controller and remove the Azure instance (Warning: all your data will be permanently removed): + +```shell +juju destroy-controller --destroy-all-models --destroy-storage --force +``` + +Should the destroying process take a long time or be seemingly stuck, proceed to delete VM resources also manually +via the Azure portal. See [Azure documentation](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resources-portal) for more information +on how to remove active resources no longer needed. + +Next, check and manually delete all unnecessary Azure VM instances, to show the list of all your Azure VMs run the following command (make sure to use the correct region): + +```shell +az resource list +``` + +List your Juju credentials with the `juju credentials` command. + +[details="Output example"] +```shell +Client Credentials: +Cloud Credentials +azure NAME_OF_YOUR_CREDENTIAL +``` +[/details] + +Remove Azure CLI credentials from Juju: + +```shell +juju remove-credential azure NAME_OF_YOUR_CREDENTIAL +``` + +After deleting the credentials, the `interactive` process may still leave the role resource and its assignment hanging around. +We recommend checking if these are still present by running: + +```shell +az role definition list --name +``` + +To get the full list, use it without specifying the `--name` argument. + +You can check whether you still have a +role assignment bound to `` registered by using: + +```shell +az role assignment list --role +``` + +If there is an unwanted role left, you can remove the role assignment first and then the role itself with the following commands: + +```shell +az role assignment delete --role +az role definition delete --name +``` + +Finally, log out from Azure CLI: + +```shell +az logout +``` \ No newline at end of file diff --git a/docs/how-to/h-kraft-mode.md b/docs/how-to/h-kraft-mode.md new file mode 100644 index 00000000..4564df19 --- /dev/null +++ b/docs/how-to/h-kraft-mode.md @@ -0,0 +1,57 @@ +# How to set up KRaft mode + +[note type="caution"] +This feature is experimental and not production-ready. UX can change significantly in the upcoming months. +[/note] + +Apache Kafka Raft (KRaft) is a consensus protocol introduced to remove ZooKeeper dependency from a Kafka deployment. +This guide provides step-by-step instructions to configure Kafka in [KRaft mode](https://kafka.apache.org/documentation/#kraft) introduced in [KIP-500](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum). + +## Prerequisites + +Follow the first steps of the [How to deploy Charmed Kafka](https://discourse.charmhub.io/t/charmed-kafka-documentation-how-to-deploy/13261) guide to set up the environment. Stop before deploying Charmed Kafka and continue with the instructions below. + +## Roles setup + +A new **role** has been introduced to the charm, named `controller`. The application with this role assigned becomes a controller for the cluster. Using this role, the charm can be deployed either as a single app or multiple applications with a dedicated controller app. We recommend splitting controller tasks to a dedicated app. + +### Single application deployment + +To deploy Charmed Kafka in KRaft mode as a single application, assign both `controller` and `broker` roles to the application when using the `juju deploy` command: + +```shell +juju deploy kafka --channel 3/edge --config roles="broker,controller" +``` + +Once the unit is shown as `active|idle` in the `juju status` command output, the deployment should be ready to be used. + +### Multiple applications deployment + +To deploy Charmed Kafka in KRaft mode as multiple applications, you need to split roles between applications. +First, deploy the applications: a dedicated cluster controller and a broker cluster with relevant roles: + +```shell +juju deploy kafka --channel 3/edge --config roles="controller" controller +juju deploy kafka --channel 3/edge --config roles="broker" broker +``` + +Finally, integrate the applications with the `peer-cluster-orchestrator` and `peer-cluster` roles: + +```shell +juju integrate broker:peer-cluster-orchestrator controller:peer-cluster +``` + +The deployment is complete when all the units are shown as `active|idle` in the `juju status` command output. + +## (Optional) Create an external admin user + +[note] +This step is normally done using `data-integrator`. However, the current ACL implementation has a bug where internal operations will fail on the integration step with the data integrator. +[/note] + +To create external users, use `juju ssh` to connect to a broker unit and use the `charmed-kafka.configs` bin command to create a new user: + +```shell +juju ssh broker/0 -- sudo su +charmed-kafka.configs --alter --entity-type=users --entity-name= --add-config=SCRAM-SHA-512=[password=] --bootstrap-server :9092 --command-config /var/snap/charmed-kafka/current/etc/kafka/admin.config +``` \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index 1515822b..67f56825 100644 --- a/docs/index.md +++ b/docs/index.md @@ -52,6 +52,8 @@ The Charmed Kafka Operator is free software, distributed under the Apache Softwa 1. [7. Cleanup your environment](tutorial/t-cleanup-environment.md) 1. [How To](how-to) 1. [Deploy](how-to/h-deploy.md) + 1. [Deploy on AWS](how-to/h-deploy-aws.md) + 1. [Deploy on Azure](how-to/h-deploy-azure.md) 1. [Manage units](how-to/h-manage-units.md) 1. [Manage applications](how-to/h-manage-app.md) 1. [Enable encryption](how-to/h-enable-encryption.md) @@ -62,6 +64,7 @@ The Charmed Kafka Operator is free software, distributed under the Apache Softwa 1. [Create mTLS Client Credentials](how-to/h-create-mtls-client-credentials.md) 1. [Enable Oauth through Hydra](how-to/h-enable-oauth.md) 1. [Backup and restore configuration](how-to/h-backup-restore-configuration.md) + 1. [Set up KRaft mode](how-to/h-kraft-mode.md) 1. [Reference](reference) 1. [Release Notes](reference/r-releases) 1. [Revision 156/126](reference/r-releases/r-rev156_126.md)