diff --git a/ceph.md b/ceph.md index a021708..5cd158d 100644 --- a/ceph.md +++ b/ceph.md @@ -161,24 +161,23 @@ spec: mountPath: "/etc/ceph" readOnly: true ``` -The `OpenStackDataPlane` can also use `extraMounts`. +The `OpenStackDataPlaneNodeSet` can also use `extraMounts`. ```yaml apiVersion: dataplane.openstack.org/v1beta1 -kind: OpenStackDataPlane +kind: OpenStackDataPlaneNodeSet spec: - roles: - edpm-compute: - nodeTemplate: - extraMounts: - - extraVolType: Ceph - volumes: - - name: ceph - secret: - secretName: ceph-conf-files - mounts: - - name: ceph - mountPath: "/etc/ceph" - readOnly: true + ... + nodeTemplate: + extraMounts: + - extraVolType: Ceph + volumes: + - name: ceph + secret: + secretName: ceph-conf-files + mounts: + - name: ceph + mountPath: "/etc/ceph" + readOnly: true ``` When a CR containing the above is created, an Ansible pod running on OpenShift mounts the files in the Ceph secret @@ -307,14 +306,14 @@ overwrite any custom service with the same name during reconciliation. After the `ConfigMap` and `OpenStackDataPlaneService` services above have been created (e.g. `oc create -f ceph-nova.yaml`), update the -`OpenStackDataPlane` +`OpenStackDataPlaneNodeSet` [EDPM services list](https://openstack-k8s-operators.github.io/dataplane-operator/composable_services) to replace the `nova` service with `nova-custom-ceph` and add the `ceph-client` service. ```yaml apiVersion: dataplane.openstack.org/v1beta1 -kind: OpenStackDataPlane +kind: OpenStackDataPlaneNodeSet spec: ... roles: @@ -495,10 +494,10 @@ spec: ## Full Examples The examples above are focussed on showing how a -single `OpenStackControlPlane` and `OpenStackDataPlane` +single `OpenStackControlPlane` and `OpenStackDataPlaneNodeSet` CR can be modified to include Ceph configuration by adding `extraMounts` and `customServiceConfig`. Links to complete examples are below. - `OpenStackControlPlane`: [core_v1beta1_openstackcontrolplane_network_isolation_ceph.yaml](https://github.com/openstack-k8s-operators/openstack-operator/blob/main/config/samples/core_v1beta1_openstackcontrolplane_network_isolation_ceph.yaml) -- `OpenStackDataPlane`: [dataplane_v1beta1_openstackdataplane_ceph.yaml](https://github.com/openstack-k8s-operators/dataplane-operator/blob/main/config/samples/dataplane_v1beta1_openstackdataplane_ceph.yaml) +- `OpenStackDataPlaneNodeSet`: [dataplane_v1beta1_openstackdataplanenodeset_ceph.yaml](https://github.com/openstack-k8s-operators/dataplane-operator/blob/main/config/samples/dataplane_v1beta1_openstackdataplanenodeset_ceph.yaml) diff --git a/edpm_service_overview.md b/edpm_service_overview.md index fa7677b..405b6dd 100644 --- a/edpm_service_overview.md +++ b/edpm_service_overview.md @@ -71,9 +71,9 @@ ports for the Ceph Monitoring service. dport: [6789, 3300] ``` -When the `run-os` -[composable service](https://openstack-k8s-operators.github.io/dataplane-operator/composable_services/) -runs, it executes the role +When the `configure-os` and `run-os` +[composable services](https://openstack-k8s-operators.github.io/dataplane-operator/composable_services/) +run, they execute the role [edpm_nftables](https://github.com/openstack-k8s-operators/edpm-ansible/tree/main/roles/edpm_nftables) This role reads files in `/var/lib/edpm-config/firewall/` and creates a `edpm-rules.nft` file in `/etc/nftables/` and then @@ -94,11 +94,11 @@ which results in the following output from the NFT command. tcp dport { 3300, 6789 } ct state new counter packets 0 bytes 0 accept comment "110 allow ceph_mon" [root@edpm-compute-0 ~]# ``` -If the service needs to be deployed after the `run-os` service has -run, then the Ansible for that service can directly call the -`edpm_nftables` role to update the files in `/etc/nftables` and -reload the rules. An example of this from the `edpm_libvirt` role -is below. +If the service needs to be deployed after the `configure-os` and +`run-os` services have run, then the Ansible for that service can +directly call the `edpm_nftables` role to update the files in +`/etc/nftables` and reload the rules. An example of this from the +`edpm_libvirt` role is below. ```yaml - name: Copy qemu vnc firewall config diff --git a/hci.md b/hci.md index 7939421..8f07f2e 100644 --- a/hci.md +++ b/hci.md @@ -11,34 +11,30 @@ deployments are also known as Hyperconverged Infrastructure (HCI). 3. Configure OpenStack to use the collocated Ceph server In order to complete the above procedure, the `services` list of the -`OpenStackDataPlane` CR needs to be edited. +`OpenStackDataPlaneNodeSet` CR needs to be edited. -## OpenStackDataPlane services list +## OpenStackDataPlaneNodeSet services list -EDPM nodes can be configured by creating an `OpenStackDataPlane` CRs -(or `OpenStackDataPlaneRole` and `OpenStackDataPlaneNode` CRs) -which the +EDPM nodes can be configured by creating an +`OpenStackDataPlaneNodeSet` CR which the [dataplane-operator](https://openstack-k8s-operators.github.io/dataplane-operator) -will reconcile. These types of CRs have a `services` list like the -following: +will reconcile when an `OpenStackDataPlaneDeployment` CR is created. +These types of CRs have a `services` list like the following: ```yaml apiVersion: dataplane.openstack.org/v1beta1 -kind: OpenStackDataPlane +kind: OpenStackDataPlaneNodeSet spec: ... - roles: - edpm-compute: - ... - services: - - configure-network - - validate-network - - install-os - - configure-os - - run-os - - ovn - - libvirt - - nova + services: + - configure-network + - validate-network + - install-os + - configure-os + - run-os + - ovn + - libvirt + - nova ``` Only the services which are on the list will be configured. @@ -60,35 +56,33 @@ This example also assumes that the EDPM nodes: - Are at least three in number (Ceph clusters must have at least three nodes for redundancy) -Create an `OpenStackDataPlane` CR file, e.g. `dataplane_cr.yaml` to -represent the EDPM nodes. See -[dataplane_v1beta1_openstackdataplane.yaml](https://github.com/openstack-k8s-operators/dataplane-operator/blob/main/config/samples/dataplane_v1beta1_openstackdataplane.yaml) -for an example which you will modify as described in this document. +Create an `OpenStackDataPlaneNodeSet` CR file, +e.g. `dataplane_cr.yaml` to represent the EDPM nodes. See +[dataplane_v1beta1_openstackdataplanenodeset.yaml](https://github.com/openstack-k8s-operators/dataplane-operator/blob/main/config/samples/dataplane_v1beta1_openstackdataplanenodeset.yaml) +for an example to modify as described in this document. Do not yet create the CR in OpenShift as the edits described in the -next sections will be required. +next sections are required. ### Shorten the Service list -Shorten the services list and add the `ceph-hci-pre` service so that -it only has `configure-network`, `validate-network`, and -`ceph-hci-pre`. +Update the `services` list: + +- Add the `ceph-hci-pre` service before the `configure-os` service. +- Remove any services after the `run-os` service for now. ```yaml apiVersion: dataplane.openstack.org/v1beta1 -kind: OpenStackDataPlane +kind: OpenStackDataPlaneNodeSet spec: ... - roles: - edpm-compute: - ... - services: - - configure-network - - validate-network - - install-os - - configure-os - - ceph-hci-pre - - run-os + services: + - configure-network + - validate-network + - install-os + - ceph-hci-pre + - configure-os + - run-os ``` In the example above the services for `ovn`, `libvirt`, and `nova` have been removed. If there are other services besides the one in @@ -101,23 +95,28 @@ edpm-ansible role called `ceph-hci-pre`. This role injects a `ceph-networks.yaml` file into `/var/lib/edpm-config/firewall` so that when the `edpm_nftables` role runs, firewall ports are open for Ceph services. By default the `ceph-networks.yaml` file only -contains directives to open the ports required by the Ceph RBD block -storage service. This is because of the following default Ansible -variable values: +contains directives to open the ports required by the Ceph RBD +(block), RGW (object) and NFS (files) services. This is because of the +following default Ansible variable value: ```yaml edpm_ceph_hci_pre_enabled_services: - ceph_mon - ceph_mgr - ceph_osd + - ceph_rgw + - ceph_nfs + - ceph_rgw_frontend + - ceph_nfs_frontend ``` -If other Ceph services like RGW, CephFS, or Dashboard will be deployed +If other Ceph services, like the Ceph Dashboard, will be deployed on HCI nodes, then add additional services to the enabled services list above. For more informatoin, see the `ceph-hci-pre` role in the [edpm-ansible role documentation](https://openstack-k8s-operators.github.io/edpm-ansible/roles.html). -The `run-os` service is run after `ceph-hci-pre` because it enables -the firewall rules which `ceph-hci-pre` put in place. The `run-os` -service also configures NTP, which is requried by Ceph. +The `configure-os` and `run-os` services are run after `ceph-hci-pre` +because they enable the firewall rules which `ceph-hci-pre` put in +place. The `run-os` service also configures NTP, which is requried by +Ceph. ### Add a Ceph cluster network @@ -441,21 +440,20 @@ and Ceph configuration file. ```yaml apiVersion: dataplane.openstack.org/v1beta1 -kind: OpenStackDataPlane +kind: OpenStackDataPlaneNodeSet spec: - roles: - edpm-compute: - nodeTemplate: - extraMounts: - - extraVolType: Ceph - volumes: - - name: ceph - secret: - secretName: ceph-conf-files - mounts: - - name: ceph - mountPath: "/etc/ceph" - readOnly: true + ... + nodeTemplate: + extraMounts: + - extraVolType: Ceph + volumes: + - name: ceph + secret: + secretName: ceph-conf-files + mounts: + - name: ceph + mountPath: "/etc/ceph" + readOnly: true ``` ### Restore the full services list @@ -467,23 +465,20 @@ the full services list needs to be restored. For example: ```yaml apiVersion: dataplane.openstack.org/v1beta1 -kind: OpenStackDataPlane +kind: OpenStackDataPlaneNodeSet spec: ... - roles: - edpm-compute: - ... - services: - - configure-network - - validate-network - - install-os - - configure-os - - ceph-hci-pre - - run-os - - ceph-client - - ovn - - libvirt - - nova-custom-ceph + services: + - configure-network + - validate-network + - install-os + - configure-os + - ceph-hci-pre + - run-os + - ceph-client + - ovn + - libvirt + - nova-custom-ceph ``` In addition to restoring the default service list, the `ceph-client` service is added after `run-os`. The `ceph-client` service configures