From 2e0578854f06cc51539cb9c429f73f67b076aeae Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 17 Oct 2024 08:53:28 +0200 Subject: [PATCH] [DPE-5555;DPE-5220;DPE-5530] Security, Hardening and Backup/Restore Documentation (#245) Co-authored-by: discourse-gatekeeper-docs-bot --- docs/explanation/e-cluster-configuration.md | 19 +++ docs/explanation/e-hardening.md | 118 ++++++++++++++++++ docs/explanation/e-security.md | 94 ++++++++++++++ docs/how-to/h-backup-restore-configuration.md | 113 +++++++++++++++++ docs/index.md | 44 ++++++- 5 files changed, 387 insertions(+), 1 deletion(-) create mode 100644 docs/explanation/e-cluster-configuration.md create mode 100644 docs/explanation/e-hardening.md create mode 100644 docs/explanation/e-security.md create mode 100644 docs/how-to/h-backup-restore-configuration.md diff --git a/docs/explanation/e-cluster-configuration.md b/docs/explanation/e-cluster-configuration.md new file mode 100644 index 00000000..c9fc342f --- /dev/null +++ b/docs/explanation/e-cluster-configuration.md @@ -0,0 +1,19 @@ +# Overview of a cluster configuration content + +[Apache Kafka](https://kafka.apache.org) is an open-source distributed event streaming platform that requires an external solution to coordinate and sync metadata between all active brokers. +One of such solutions is [ZooKeeper](https://zookeeper.apache.org). + +Here are some of the responsibilities of ZooKeeper in a Kafka cluster: + +- **Cluster membership**: through regular heartbeats, it keeps tracks of the brokers entering and leaving the cluster, providing an up-to-date list of brokers. +- **Controller election**: one of the Kafka brokers is responsible for managing the leader/follower status for all the partitions. ZooKeeper is used to elect a controller and to make sure there is only one of it. +- **Topic configuration**: each topic can be replicated on multiple partitions. ZooKeeper keeps track of the locations of the partitions and replicas, so that high-availability is still attained when a broker shuts down. Topic-specific configuration overrides (e.g. message retention and size) are also stored in ZooKeeper. +- **Access control and authentication**: ZooKeeper stores access control lists (ACL) for Kafka resources, to ensure only the proper, authorized, users or groups can read or write on each topic. + +The values for the configuration parameters mentioned above are stored in znodes, the hierarchical unit data structure in ZooKeeper. +A znode is represented by its path and can both have data associated with it and children nodes. +ZooKeeper clients interact with its data structure similarly to a remote file system that would be sync-ed between the ZooKeeper units for high availability. +For a Charmed Kafka related to a Charmed ZooKeeper: +- the list of the broker ids of the cluster can be found in `/kafka/brokers/ids` +- the endpoint used to access the broker with id `0` can be found in `/kafka/brokers/ids/0` +- the credentials for the Charmed Kafka users can be found in `/kafka/config/users` \ No newline at end of file diff --git a/docs/explanation/e-hardening.md b/docs/explanation/e-hardening.md new file mode 100644 index 00000000..32e5f31e --- /dev/null +++ b/docs/explanation/e-hardening.md @@ -0,0 +1,118 @@ +# Security Hardening Guide + +This document provides guidance and instructions to achieve +a secure deployment of [Charmed Kafka](https://github.com/canonical/kafka-bundle), including setting up and managing a secure environment. +The document is divided into the following sections: + +1. Environment, outlining the recommendation for deploying a secure environment +2. Applications, outlining the product features that enable a secure deployment of a Kafka cluster +3. Additional resources, providing any further information about security and compliance + +## Environment + +The environment where applications operate can be divided in two components: + +1. Cloud +2. Juju + +### Cloud + +Charmed Kafka can be deployed on top of several clouds and virtualization layers: + +| Cloud | Security guide | +|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| OpenStack | [OpenStack Security Guide](https://docs.openstack.org/security-guide/) | +| AWS | [Best Practices for Security, Identity and Compliance](https://aws.amazon.com/architecture/security-identity-compliance), [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys) | +| Azure | [Azure security best practices and patterns](https://learn.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns), [Managed identities for Azure resource](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/) | + +### Juju + +Juju is the component responsible for orchestrating the entire lifecycle, from deployment to Day 2 operations, of +all applications. Therefore, it is imperative that it is set up securely. Please refer to the Juju documentation for more information on: + +* [Juju security](https://discourse.charmhub.io/t/juju-security/15684) +* [How to harden your deployment](https://juju.is/docs/juju/harden-your-deployment) + +#### Cloud credentials + +When configuring the cloud credentials to be used with Juju, ensure that the users have correct permissions to operate at the required level. +Juju superusers responsible for bootstrapping and managing controllers require elevated permissions to manage several kind of resources, such as +virtual machines, networks, storages, etc. Please refer to the references below for more information on the policies required to be used depending on the cloud. + +| Cloud | Cloud user policies | +|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| OpenStack | N/A | +| AWS | [Juju AWS Permission](https://discourse.charmhub.io/t/juju-aws-permissions/5307), [AWS Instance Profiles](https://discourse.charmhub.io/t/using-aws-instance-profiles-with-juju-2-9/5185), [Juju on AWS](https://juju.is/docs/juju/amazon-ec2) | +| Azure | [Juju Azure Permission](https://juju.is/docs/juju/microsoft-azure), [How to use Juju with Microsoft Azure](https://discourse.charmhub.io/t/how-to-use-juju-with-microsoft-azure/15219) | + +#### Juju users + +It is very important that the different juju users are set up with minimal permission depending on the scope for their operations. +Please refer to the [User access levels](https://juju.is/docs/juju/user-permissions) documentation for more information on the access level and corresponding abilities +that the different users can be granted. + +Juju user credentials must be stored securely and rotated regularly to limit the chances of unauthorized access due to credentials leakage. + +## Applications + +In the following we provide guidance on how to harden your deployment using: + +1. Operating System +2. Kafka and ZooKeeper Security Upgrades +3. Encryption +4. Authentication +5. Monitoring and Auditing + +### Operating System + +Charmed Kafka and Charmed ZooKeeper currently run on top of Ubuntu 22.04. Deploy a [Landscape Client Charm](https://charmhub.io/landscape-client?) in order to +connect the underlying VM to a Landscape User Account to manage security upgrades and integrate Ubuntu Pro subscriptions. + +### Kafka and ZooKeeper Security Upgrades + +Charmed Kafka and Charmed ZooKeeper operators install a pinned revision of the [Charmed Kafka snap](https://snapcraft.io/charmed-kafka) +and [Charmed ZooKeeper snap](https://snapcraft.io/charmed-zookeeper), respectively, in order to provide reproducible and secure environments. +New versions of Charmed Kafka and Charmed ZooKeeper may be released to provide patching of vulnerabilities (CVEs). +It is important to refresh the charm regularly to make sure the workload is as secure as possible. +For more information on how to refresh the charm, see the [how-to upgrade](https://charmhub.io/kafka/docs/h-upgrade) guide. + +### Encryption + +Charmed Kafka must be deployed with encryption enabled. +To do that, you need to relate Kafka and ZooKeeper charms to one of the TLS certificate operator charms. +Please refer to the [Charming Security page](https://charmhub.io/topics/security-with-x-509-certificates) for more information on how to select the right certificate +provider for your use-case. + +For more information on encryption setup, see the [How to enable encryption](https://charmhub.io/kafka/docs/h-enable-encryption) guide. + +### Authentication + +Charmed Kafka supports the following authentication layers: + +1. [SCRAM-based SASL Authentication](/t/charmed-kafka-how-to-manage-app/10285) +2. [certificate-base Authentication (mTLS)](/t/create-mtls-client-credentials/11079) +3. OAuth Authentication using [Hydra](/t/how-to-connect-to-kafka-using-hydra-as-oidc-provider/14610) or [Google](/t/how-to-connect-to-kafka-using-google-as-oidc-provider/14611) + +Each combination of authentication scheme and encryption is associated to the dedicated listener and it maps to a well-defined port. +Please refer to the [listener reference documentation](/t/charmed-kafka-documentation-reference-listeners/13264) for more information. + +### Monitoring and Auditing + +Charmed Kafka provides native integration with the [Canonical Observability Stack (COS)](https://charmhub.io/topics/canonical-observability-stack). +To reduce the blast radius of infrastructure disruptions, the general recommendation is to deploy COS and the observed application into +separate environments, isolated one another. Refer to the [COS production deployments best practices](https://charmhub.io/topics/canonical-observability-stack/reference/best-practices) +for more information. + +Refer to How-To user guide for more information on: +* [how to integrate the Charmed Kafka deployment with COS](/t/charmed-kafka-how-to-enable-monitoring/10283) +* [how to customise the alerting rules and dashboards](/t/charmed-kafka-documentation-how-to-integrate-custom-alerting-rules-and-dashboards/13431) + +External user access to Kafka is logged to the `kafka-authorizer.log` that is pushes to [Loki endpoint](https://charmhub.io/loki-k8s) and exposed via [Grafana](https://charmhub.io/grafana), both components being part of the COS stack. +Access denials are logged at INFO level, whereas allowed accesses are logged at DEBUG level. Depending on the auditing needs, +customize the logging level either for all logs via the [`log_level`](https://charmhub.io/kafka/configurations?channel=3/stable#log_level) config option or +only tune the logging level of the `authorizerAppender` in the `log4j.properties` file. Refer to the Reference documentation, for more information about +the [file system paths](/t/charmed-kafka-documentation-reference-file-system-paths/13262). + +## Additional Resources + +For further information and details on the security and cryptographic specifications used by Charmed Kafka, please refer to the [Security Explanation page](/t/charmed-kafka-documentation-explanation-security/15714). \ No newline at end of file diff --git a/docs/explanation/e-security.md b/docs/explanation/e-security.md new file mode 100644 index 00000000..21b58ec2 --- /dev/null +++ b/docs/explanation/e-security.md @@ -0,0 +1,94 @@ +# Cryptography + +This document describes cryptography used by Charmed Kafka. + +## Resource checksums +Every version of the Charmed Kafka and Charmed ZooKeeper operators install a pinned revision of the Charmed Kafka snap +and Charmed ZooKeeper, respectively, in order to +provide reproducible and secure environments. The [Charmed Kafka snap](https://snapstore.io/charmed-kafka) and [Charmed ZooKeeper snap](https://snapstore.io/charmed-zookeeper) package the +Kafka and ZooKeeper workload together with +a set of dependencies and utilities required by the lifecycle of the operators (see [Charmed Kafka snap contents](https://github.com/canonical/charmed-kafka-snap/blob/3/edge/snap/snapcraft.yaml) and [Charmed ZooKeeper snap contents](https://github.com/canonical/charmed-zookeeper-snap/blob/3/edge/snap/snapcraft.yaml)). +Every artifact bundled into the Charmed Kafka snap and Charmed ZooKeeper snap is verified against their SHA256 or SHA512 checksum after download. + +## Sources verification +Charmed Kafka sources are stored in: + +* GitHub repositories for snaps, rocks and charms +* LaunchPad repositories for the Kafka and ZooKeeper upstream fork used for building their respective distributions + +### LaunchPad +Distributions are built using private repositories only, hosted as part of the [SOSS namespace](https://launchpad.net/soss) to eventually +integrate with Canonical's standard process for fixing CVEs. +Branches associated with releases are mirrored to a public repository, hosted in the [Data Platform namespace](https://launchpad.net/~data-platform) +to also provide the community with the patched source code. + +### GitHub +All Charmed Kafka and Charmed ZooKeeper artifacts are published and released +programmatically using release pipelines implemented via GitHub Actions. +Distributions are published as both GitHub and LaunchPad releases via the [central-uploader repository](https://github.com/canonical/central-uploader), while +charms, snaps and rocks are published using the workflows of their respective repositories. + +All repositories in GitHub are set up with branch protection rules, requiring: + +* new commits to be merged to main branches via Pull-Request with at least 2 approvals from repository maintainers +* new commits to be signed (e.g. using GPG keys) +* developers to sign the [Canonical Contributor License Agreement (CLA)](https://ubuntu.com/legal/contributors) + +## Encryption +The Charmed Kafka operator can be used to deploy a secure Kafka cluster that provides encryption-in-transit capabilities out of the box +for: + +* Interbroker communications +* ZooKeeper connection +* External client connection + +In order to set up secure connection Kafka and ZooKeeper applications need to be integrated with TLS Certificate Provider charms, e.g. +`self-signed-certificates` operator. CSRs are generated for every unit using `tls_certificates_interface` library that uses `cryptography` +python library to create X.509 compatible certificates. The CSR is signed by the TLS Certificate Provider and returned to the units, and +stored in a password-protected Keystore file. The password of the Keystore is stored in Juju secrets starting from revision 168 on Kafka +and revision 130 on ZooKeeper. The relation provides also the certificate for the CA to be loaded in a password-protected Truststore file. + +When encryption is enabled, hostname verification is turned on for client connections, including inter-broker communication. Cipher suite can +be customized by providing a list of allowed cipher suite to be used for external clients and zookeeper connections, using the charm config options +`ssl_cipher_suites` and `zookeeper_ssl_cipher_suites` config options respectively. Please refer to the [reference documentation](https://charmhub.io/kafka/configurations) +for more information. + +Encryption at rest is currently not supported, although it can be provided by the substrate (cloud or on-premises). + +## Authentication +In the Charmed Kafka solution, authentication layers can be enabled for + +1. ZooKeeper connections +2. Kafka inter broker communication +3. Kafka Clients + +### Kafka authentication to ZooKeeper +Authentication to ZooKeeper is based on Simple Authentication and Security Layer (SASL) using digested MD5 hashes of +username and password, and implemented both for client-server (with Kafka) and server-server communication. +Username and passwords are exchanged using peer relations among ZooKeeper units and using normal relations between Kafka and ZooKeeper. +Juju secrets are used for exchanging credentials starting from revision 168 on Kafka and revision 130 on ZooKeeper. + +Username and password for the different users are stored in ZooKeeper servers in a JAAS configuration file in plain format. +Permission on the file is restricted to the root user. + +### Kafka Inter-broker authentication +Authentication among brokers is based on SCRAM-SHA-512 protocol. Username and passwords are exchanged +via peer relations, using Juju secrets from revision 168 on Charmed Kafka. + +Kafka username and password used by brokers to authenticate one another are stored +both in a ZooKeeper zNode and in a JAAS configuration file in the Kafka server in plain format. +The file needs to be readable and +writable by root (as it is created by the charm), and be readable by the `snap_daemon` user running the Kafka server snap commands. + +### Client authentication to Kafka +Clients can authenticate to Kafka using: + +1. username and password exchanged using SCRAM-SHA-512 protocols +2. client certificates or CAs (mTLS) + +When using SCRAM, username and passwords are stored in ZooKeeper to be used by the Kafka processes, +in peer-relation data to be used by the Kafka charm and in external relation to be shared with client applications. +Starting from revision 168 on Kafka, Juju secrets are used for storing the credentials in place of plain unencrypted text. + +When using mTLS, client certificates are loaded into a `tls-certificates` operator and provided to the Kafka charm via the plain-text unencrypted +relation. Certificates are stored in the password-protected Truststore file. \ No newline at end of file diff --git a/docs/how-to/h-backup-restore-configuration.md b/docs/how-to/h-backup-restore-configuration.md new file mode 100644 index 00000000..723d184c --- /dev/null +++ b/docs/how-to/h-backup-restore-configuration.md @@ -0,0 +1,113 @@ +# Configuration backup and restore + +Charmed Kafka's configuration is distributed using [Charmed ZooKeeper](https://charmhub.io/zookeeper?channel=3/stable). +A Charmed ZooKeeper backup can be stored on any S3-compatible storage. +S3 access and configurations are managed with the [`s3-integrator` charm](https://charmhub.io/s3-integrator). + +This guide contains step-by-step instructions on how to deploy and configure the `s3-integrator` charm for [AWS S3](https://aws.amazon.com/s3/), send the configurations to the Charmed ZooKeeper application, and finally manage your Charmed ZooKeeper backups. + +## Configure `s3-integrator` + +First, deploy the `s3-integrator` charm: + +```shell +juju deploy s3-integrator +juju run s3-integrator/leader sync-s3-credentials access-key= secret-key= +``` + +Then, use `juju config` to add your configuration parameters. For example: + +```shell +juju config s3-integrator \ + endpoint="https://s3.us-west-2.amazonaws.com" \ + bucket="zk-backups-bucket-1" \ + path="/zk-backups" \ + region="us-west-2" +``` + +[note] +The only mandatory configuration parameter in the command above is the `bucket`. +[/note] + +### Integrate with Charmed ZooKeeper + +To pass these configurations to Charmed ZooKeeper, integrate the two applications: + +```shell +juju integrate s3-integrator zookeeper +``` + +You can create, list, and restore backups now: + +```shell +juju run zookeeper/leader create-backup +juju run zookeeper/leader list-backups +juju run zookeeper/leader restore backup-id= +``` + +## Create a backup + +Check that Charmed ZooKeeper deployment with configurations set for S3 storage is `active` and `idle` with the `juju status` command. Once it's active, create a backup with the `create-backup` command: + +```shell +juju run zookeeper/leader create-backup +``` + +Charmed ZooKeeper backups created with the command above will always be **full** backups: a copy of *all* the Charmed Kafka configuration will be stored in S3. + +The command will output the ID of the newly created backup: + +``` + Backup created +┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Id ┃ Log-sequence-number ┃ Path ┃ +┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ +│ 2024-09-12T14:32:46Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │ +│ │ │ 4:32:46Z/snapshot │ +└──────────────────────┴─────────────────────┴────────────────────────────────┘ +``` + +## List backups + +To list available backups, run the `list-backups` command: + +```shell +juju run zookeeper/leader list-backups +``` + +This command shows available backups, for example: + +``` + Backups +┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Id ┃ Log-sequence-number ┃ Path ┃ +┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ +│ 2024-09-12T14:32:46Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │ +│ │ │ 4:32:46Z/snapshot │ +│ 2024-09-12T14:32:00Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │ +│ │ │ 4:32:00Z/snapshot │ +│ 2024-09-12T14:26:12Z │ 8589934621 │ zookeeper_backups/2024-09-12T1 │ +│ │ │ 4:26:12Z/snapshot │ +└──────────────────────┴─────────────────────┴────────────────────────────────┘ +``` + +Below is a list of parameters shown for each backup: + +- `Id`: identifier of the backup. +- `Log-Sequence-number`: a database-specific number to identify its state. Learn more about the Zxid on [Apache ZooKeeper documentation](https://zookeeper.apache.org/doc/r3.9.2/zookeeperProgrammers.html#sc_timeInZk). +- `Path`: path of the snapshot file in the S3 repository. + +## Restore a backup + +[note] +This operation puts you at risk of losing unsaved configuration data. +We recommend creating a backup first. +[/note] + +To restore from backup, run the `restore` command and pass the `backup-id` (in the `YYYY-MM-DDTHH:MM:SSZ` format) that is listed in the `list-backups` action output: + +```shell +juju run zookeeper/leader restore backup-id= +``` + +The restore will then proceed. Follow its progress using `juju status`. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index d8ba161d..1515822b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,7 +2,9 @@ Charmed Kafka is an open-source operator that makes it easier to manage Apache Kafka, with built-in support for enterprise features. -Charmed Kafka is built on top of [Juju](https://juju.is/) and reliably simplifies the deployment, scaling, design, and management of Apache Kafka in production. Additionally, you can use the charm to manage your Kafka clusters with automation capabilities. It also offers replication, TLS, password rotation, easy-to-use application integration, and monitoring. +Apache Kafka is a free, open source software project by the Apache Software Foundation. Users can find out more at the [Kafka project page](https://kafka.apache.org). + +Charmed Kafka is built on top of [Juju](https://juju.is/) and reliably simplifies the deployment, scaling, design, and management of [Apache Kafka](https://kafka.apache.org/) in production. Additionally, you can use the charm to manage your Kafka clusters with automation capabilities. It also offers replication, TLS, password rotation, easy-to-use application integration, and monitoring. Charmed Kafka operates Apache Kafka on physical systems, Virtual Machines (VM), and a wide range of cloud and cloud-like environments, including AWS, Azure, OpenStack, and VMware. Charmed Kafka is a solution designed and developed to help ops teams and @@ -32,6 +34,46 @@ Charmed Kafka is a distribution of Apache Kafka. It’s an open-source project t - Explore [Canonical Data Fabric solutions](https://canonical.com/data) - [Contact us]([/t/13107) for all further questions +Apache®, Apache Kafka, Kafka®, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. + ## License The Charmed Kafka Operator is free software, distributed under the Apache Software License, version 2.0. See [LICENSE](https://github.com/canonical/kafka-operator/blob/main/LICENSE) for more information. + +# Contents + +1. [Tutorial](tutorial) + 1. [1. Introduction](tutorial/t-overview.md) + 1. [2. Set up the environment](tutorial/t-setup-environment.md) + 1. [3. Deploy Kafka](tutorial/t-deploy.md) + 1. [4. Integrate with client applications](tutorial/t-relate-kafka.md) + 1. [5. Manage passwords](tutorial/t-manage-passwords.md) + 1. [6. Enable Encryption](tutorial/t-enable-encryption.md) + 1. [7. Cleanup your environment](tutorial/t-cleanup-environment.md) +1. [How To](how-to) + 1. [Deploy](how-to/h-deploy.md) + 1. [Manage units](how-to/h-manage-units.md) + 1. [Manage applications](how-to/h-manage-app.md) + 1. [Enable encryption](how-to/h-enable-encryption.md) + 1. [Upgrade](how-to/h-upgrade.md) + 1. [Enable Monitoring](how-to/h-enable-monitoring.md) + 1. [Integrate alerts and dashboards](how-to/h-integrate-alerts-dashboards.md) + 1. [Migrate a cluster](how-to/h-cluster-migration.md) + 1. [Create mTLS Client Credentials](how-to/h-create-mtls-client-credentials.md) + 1. [Enable Oauth through Hydra](how-to/h-enable-oauth.md) + 1. [Backup and restore configuration](how-to/h-backup-restore-configuration.md) +1. [Reference](reference) + 1. [Release Notes](reference/r-releases) + 1. [Revision 156/126](reference/r-releases/r-rev156_126.md) + 1. [Revision 156/136](reference/r-releases/r-rev156_136.md) + 1. [File System Paths](reference/r-file-system-paths.md) + 1. [Snap Entrypoints](reference/r-snap-entrypoints.md) + 1. [Kafka Listeners](reference/r-listeners.md) + 1. [Statuses](reference/r-statuses.md) + 1. [Requirements](reference/r-requirements.md) + 1. [Performance Tuning](reference/r-performance-tuning.md) + 1. [Contact](reference/r-contacts.md) +1. [Explanation](explanation) + 1. [Security](explanation/e-security.md) + 1. [Hardening Guide](explanation/e-hardening.md) + 1. [Overview of Cluster Configuration](explanation/e-cluster-configuration.md) \ No newline at end of file