[DPE-5555;DPE-5220;DPE-5530] Security, Hardening and Backup/Restore D…

…ocumentation (#245) Co-authored-by: discourse-gatekeeper-docs-bot <[email protected]>
canonical · Oct 17, 2024 · 2e05788 · 2e05788
1 parent 7cfebee
commit 2e05788
Show file tree

Hide file tree

Showing 5 changed files with 387 additions and 1 deletion.
diff --git a/docs/explanation/e-cluster-configuration.md b/docs/explanation/e-cluster-configuration.md
@@ -0,0 +1,19 @@
+# Overview of a cluster configuration content
+
+[Apache Kafka](https://kafka.apache.org) is an open-source distributed event streaming platform that requires an external solution to coordinate and sync metadata between all active brokers.
+One of such solutions is [ZooKeeper](https://zookeeper.apache.org).
+
+Here are some of the responsibilities of ZooKeeper in a Kafka cluster:
+
+- **Cluster membership**: through regular heartbeats, it keeps tracks of the brokers entering and leaving the cluster, providing an up-to-date list of brokers.
+- **Controller election**: one of the Kafka brokers is responsible for managing the leader/follower status for all the partitions. ZooKeeper is used to elect a controller and to make sure there is only one of it.
+- **Topic configuration**: each topic can be replicated on multiple partitions. ZooKeeper keeps track of the locations of the partitions and replicas, so that high-availability is still attained when a broker shuts down. Topic-specific configuration overrides (e.g. message retention and size) are also stored in ZooKeeper.
+- **Access control and authentication**: ZooKeeper stores access control lists (ACL) for Kafka resources, to ensure only the proper, authorized, users or groups can read or write on each topic.
+
+The values for the configuration parameters mentioned above are stored in znodes, the hierarchical unit data structure in ZooKeeper.
+A znode is represented by its path and can both have data associated with it and children nodes.
+ZooKeeper clients interact with its data structure similarly to a remote file system that would be sync-ed between the ZooKeeper units for high availability.
+For a Charmed Kafka related to a Charmed ZooKeeper:
+- the list of the broker ids of the cluster can be found in `/kafka/brokers/ids`
+- the endpoint used to access the broker with id `0` can be found in `/kafka/brokers/ids/0`
+- the credentials for the Charmed Kafka users can be found in `/kafka/config/users`
diff --git a/docs/explanation/e-hardening.md b/docs/explanation/e-hardening.md
@@ -0,0 +1,118 @@
+# Security Hardening Guide
+
+This document provides guidance and instructions to achieve 
+a secure deployment of [Charmed Kafka](https://github.com/canonical/kafka-bundle), including setting up and managing a secure environment.
+The document is divided into the following sections:
+
+1. Environment, outlining the recommendation for deploying a secure environment
+2. Applications, outlining the product features that enable a secure deployment of a Kafka cluster
+3. Additional resources, providing any further information about security and compliance
+
+## Environment
+
+The environment where applications operate can be divided in two components:
+
+1. Cloud
+2. Juju 
+
+### Cloud
+
+Charmed Kafka can be deployed on top of several clouds and virtualization layers: 
+
+| Cloud     | Security guide                                                                                                                                                                                                                                                         |
+|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| OpenStack | [OpenStack Security Guide](https://docs.openstack.org/security-guide/)                                                                                                                                                                                                 |
+| AWS       | [Best Practices for Security, Identity and Compliance](https://aws.amazon.com/architecture/security-identity-compliance), [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys)          | 
+| Azure     | [Azure security best practices and patterns](https://learn.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns), [Managed identities for Azure resource](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/) |
+
+### Juju 
+
+Juju is the component responsible for orchestrating the entire lifecycle, from deployment to Day 2 operations, of 
+all applications. Therefore, it is imperative that it is set up securely. Please refer to the Juju documentation for more information on:
+
+* [Juju security](https://discourse.charmhub.io/t/juju-security/15684)
+* [How to harden your deployment](https://juju.is/docs/juju/harden-your-deployment)
+
+#### Cloud credentials
+
+When configuring the cloud credentials to be used with Juju, ensure that the users have correct permissions to operate at the required level. 
+Juju superusers responsible for bootstrapping and managing controllers require elevated permissions to manage several kind of resources, such as
+virtual machines, networks, storages, etc. Please refer to the references below for more information on the policies required to be used depending on the cloud. 
+
+| Cloud     | Cloud user policies                                                                                                                                                                                                                            |
+|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| OpenStack | N/A                                                                                                                                                                                                                                            |
+| AWS       | [Juju AWS Permission](https://discourse.charmhub.io/t/juju-aws-permissions/5307), [AWS Instance Profiles](https://discourse.charmhub.io/t/using-aws-instance-profiles-with-juju-2-9/5185), [Juju on AWS](https://juju.is/docs/juju/amazon-ec2) | 
+| Azure     | [Juju Azure Permission](https://juju.is/docs/juju/microsoft-azure), [How to use Juju with Microsoft Azure](https://discourse.charmhub.io/t/how-to-use-juju-with-microsoft-azure/15219)                                                         |
+
+#### Juju users
+
+It is very important that the different juju users are set up with minimal permission depending on the scope for their operations. 
+Please refer to the [User access levels](https://juju.is/docs/juju/user-permissions) documentation for more information on the access level and corresponding abilities 
+that the different users can be granted. 
+
+Juju user credentials must be stored securely and rotated regularly to limit the chances of unauthorized access due to credentials leakage.
+
+## Applications
+
+In the following we provide guidance on how to harden your deployment using:
+
+1. Operating System
+2. Kafka and ZooKeeper Security Upgrades
+3. Encryption 
+4. Authentication
+5. Monitoring and Auditing
+
+### Operating System
+
+Charmed Kafka and Charmed ZooKeeper currently run on top of Ubuntu 22.04. Deploy a [Landscape Client Charm](https://charmhub.io/landscape-client?) in order to 
+connect the underlying VM to a Landscape User Account to manage security upgrades and integrate Ubuntu Pro subscriptions. 
+
+### Kafka and ZooKeeper Security Upgrades
+
+Charmed Kafka and Charmed ZooKeeper operators install a pinned revision of the [Charmed Kafka snap](https://snapcraft.io/charmed-kafka)
+and [Charmed ZooKeeper snap](https://snapcraft.io/charmed-zookeeper), respectively, in order to provide reproducible and secure environments. 
+New versions of Charmed Kafka and Charmed ZooKeeper may be released to provide patching of vulnerabilities (CVEs). 
+It is important to refresh the charm regularly to make sure the workload is as secure as possible. 
+For more information on how to refresh the charm, see the [how-to upgrade](https://charmhub.io/kafka/docs/h-upgrade) guide.
+
+### Encryption
+
+Charmed Kafka must be deployed with encryption enabled. 
+To do that, you need to relate Kafka and ZooKeeper charms to one of the TLS certificate operator charms. 
+Please refer to the [Charming Security page](https://charmhub.io/topics/security-with-x-509-certificates) for more information on how to select the right certificate
+provider for your use-case. 
+
+For more information on encryption setup, see the [How to enable encryption](https://charmhub.io/kafka/docs/h-enable-encryption) guide.
+
+### Authentication
+
+Charmed Kafka supports the following authentication layers:
+
+1. [SCRAM-based SASL Authentication](/t/charmed-kafka-how-to-manage-app/10285)
+2. [certificate-base Authentication (mTLS)](/t/create-mtls-client-credentials/11079)
+3. OAuth Authentication using [Hydra](/t/how-to-connect-to-kafka-using-hydra-as-oidc-provider/14610) or [Google](/t/how-to-connect-to-kafka-using-google-as-oidc-provider/14611)
+
+Each combination of authentication scheme and encryption is associated to the dedicated listener and it maps to a well-defined port. 
+Please refer to the [listener reference documentation](/t/charmed-kafka-documentation-reference-listeners/13264) for more information. 
+
+### Monitoring and Auditing
+
+Charmed Kafka provides native integration with the [Canonical Observability Stack (COS)](https://charmhub.io/topics/canonical-observability-stack).
+To reduce the blast radius of infrastructure disruptions, the general recommendation is to deploy COS and the observed application into 
+separate environments, isolated one another. Refer to the [COS production deployments best practices](https://charmhub.io/topics/canonical-observability-stack/reference/best-practices)
+for more information. 
+
+Refer to How-To user guide for more information on:
+* [how to integrate the Charmed Kafka deployment with COS](/t/charmed-kafka-how-to-enable-monitoring/10283)
+* [how to customise the alerting rules and dashboards](/t/charmed-kafka-documentation-how-to-integrate-custom-alerting-rules-and-dashboards/13431)
+
+External user access to Kafka is logged to the `kafka-authorizer.log` that is pushes to [Loki endpoint](https://charmhub.io/loki-k8s) and exposed via [Grafana](https://charmhub.io/grafana), both components being part of the COS stack.
+Access denials are logged at INFO level, whereas allowed accesses are logged at DEBUG level. Depending on the auditing needs, 
+customize the logging level either for all logs via the [`log_level`](https://charmhub.io/kafka/configurations?channel=3/stable#log_level) config option or 
+only tune the logging level of the `authorizerAppender` in the `log4j.properties` file. Refer to the Reference documentation, for more information about 
+the [file system paths](/t/charmed-kafka-documentation-reference-file-system-paths/13262).
+
+## Additional Resources
+
+For further information and details on the security and cryptographic specifications used by Charmed Kafka, please refer to the [Security Explanation page](/t/charmed-kafka-documentation-explanation-security/15714).
diff --git a/docs/explanation/e-security.md b/docs/explanation/e-security.md
@@ -0,0 +1,94 @@
+# Cryptography
+
+This document describes cryptography used by Charmed Kafka.
+
+## Resource checksums
+Every version of the Charmed Kafka and Charmed ZooKeeper operators install a pinned revision of the Charmed Kafka snap
+and Charmed ZooKeeper, respectively, in order to 
+provide reproducible and secure environments. The [Charmed Kafka snap](https://snapstore.io/charmed-kafka) and [Charmed ZooKeeper snap](https://snapstore.io/charmed-zookeeper) package the 
+Kafka and ZooKeeper workload together with 
+a set of dependencies and utilities required by the lifecycle of the operators (see [Charmed Kafka snap contents](https://github.com/canonical/charmed-kafka-snap/blob/3/edge/snap/snapcraft.yaml) and [Charmed ZooKeeper snap contents](https://github.com/canonical/charmed-zookeeper-snap/blob/3/edge/snap/snapcraft.yaml)).
+Every artifact bundled into the Charmed Kafka snap and Charmed ZooKeeper snap is verified against their SHA256 or SHA512 checksum after download. 
+
+## Sources verification
+Charmed Kafka sources are stored in:
+
+* GitHub repositories for snaps, rocks and charms
+* LaunchPad repositories for the Kafka and ZooKeeper upstream fork used for building their respective distributions
+
+### LaunchPad
+Distributions are built using private repositories only, hosted as part of the [SOSS namespace](https://launchpad.net/soss) to eventually
+integrate with Canonical's standard process for fixing CVEs. 
+Branches associated with releases are mirrored to a public repository, hosted in the [Data Platform namespace](https://launchpad.net/~data-platform) 
+to also provide the community with the patched source code. 
+
+### GitHub
+All Charmed Kafka and Charmed ZooKeeper artifacts are published and released 
+programmatically using release pipelines implemented via GitHub Actions. 
+Distributions are published as both GitHub and LaunchPad releases via the [central-uploader repository](https://github.com/canonical/central-uploader), while 
+charms, snaps and rocks are published using the workflows of their respective repositories. 
+
+All repositories in GitHub are set up with branch protection rules, requiring:
+
+* new commits to be merged to main branches via Pull-Request with at least 2 approvals from repository maintainers
+* new commits to be signed (e.g. using GPG keys)
+* developers to sign the [Canonical Contributor License Agreement (CLA)](https://ubuntu.com/legal/contributors)
+
+## Encryption
+The Charmed Kafka operator can be used to deploy a secure Kafka cluster that provides encryption-in-transit capabilities out of the box 
+for:
+
+* Interbroker communications
+* ZooKeeper connection
+* External client connection 
+
+In order to set up secure connection Kafka and ZooKeeper applications need to be integrated with TLS Certificate Provider charms, e.g. 
+`self-signed-certificates` operator. CSRs are generated for every unit using `tls_certificates_interface` library that uses `cryptography` 
+python library to create X.509 compatible certificates. The CSR is signed by the TLS Certificate Provider and returned to the units, and 
+stored in a password-protected Keystore file. The password of the Keystore is stored in Juju secrets starting from revision 168 on Kafka 
+and revision 130 on ZooKeeper. The relation provides also the certificate for the CA to be loaded in a password-protected Truststore file.
+
+When encryption is enabled, hostname verification is turned on for client connections, including inter-broker communication. Cipher suite can 
+be customized by providing a list of allowed cipher suite to be used for external clients and zookeeper connections, using the charm config options
+`ssl_cipher_suites`  and `zookeeper_ssl_cipher_suites` config options respectively. Please refer to the [reference documentation](https://charmhub.io/kafka/configurations)
+for more information. 
+
+Encryption at rest is currently not supported, although it can be provided by the substrate (cloud or on-premises).
+
+## Authentication
+In the Charmed Kafka solution, authentication layers can be enabled for
+
+1. ZooKeeper connections
+2. Kafka inter broker communication 
+3. Kafka Clients
+
+### Kafka authentication to ZooKeeper
+Authentication to ZooKeeper is based on Simple Authentication and Security Layer (SASL) using digested MD5 hashes of
+username and password, and implemented both for client-server (with Kafka) and server-server communication.
+Username and passwords are exchanged using peer relations among ZooKeeper units and using normal relations between Kafka and ZooKeeper.
+Juju secrets are used for exchanging credentials starting from revision 168 on Kafka and revision 130 on ZooKeeper.
+
+Username and password for the different users are stored in ZooKeeper servers in a JAAS configuration file in plain format. 
+Permission on the file is restricted to the root user. 
+
+### Kafka Inter-broker authentication
+Authentication among brokers is based on SCRAM-SHA-512 protocol. Username and passwords are exchanged 
+via peer relations, using Juju secrets from revision 168 on Charmed Kafka.
+
+Kafka username and password used by brokers to authenticate one another are stored 
+both in a ZooKeeper zNode and in a JAAS configuration file in the Kafka server in plain format. 
+The file needs to be readable and
+writable by root (as it is created by the charm), and be readable by the `snap_daemon` user running the Kafka server snap commands.
+
+### Client authentication to Kafka
+Clients can authenticate to Kafka using:
+
+1. username and password exchanged using SCRAM-SHA-512 protocols 
+2. client certificates or CAs (mTLS)
+
+When using SCRAM, username and passwords are stored in ZooKeeper to be used by the Kafka processes, 
+in peer-relation data to be used by the Kafka charm and in external relation to be shared with client applications. 
+Starting from revision 168 on Kafka, Juju secrets are used for storing the credentials in place of plain unencrypted text.
+
+When using mTLS, client certificates are loaded into a `tls-certificates` operator and provided to the Kafka charm via the plain-text unencrypted 
+relation. Certificates are stored in the password-protected Truststore file.