This repository has been archived by the owner on Oct 20, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 34
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
43 changed files
with
2,902 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
36 changes: 36 additions & 0 deletions
36
site/website/versioned_docs/version-v0.6.2/1_concepts/1_introduction.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| --- | ||
| id: 1_introduction | ||
| title: Introduction | ||
| sidebar_label: Introduction | ||
| --- | ||
|
|
||
| The Orange NiFi operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of [Apache NiFi](https://nifi.apache.org/) clusters deployed to K8s. | ||
|
|
||
| ## Overview | ||
|
|
||
| Apache NiFi is an open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic. | ||
| Some of the high-level capabilities and objectives of Apache NiFi include, and some of the main features of the **NiFiKop** are: | ||
|
|
||
| - **Fine grained** node configuration support | ||
| - Graceful rolling upgrade | ||
| - graceful NiFi cluster **scaling** | ||
| - Encrypted communication using SSL | ||
| - the provisioning of secure NiFi clusters | ||
| - Advanced Dataflow and user management via CRD | ||
|
|
||
| Some of the roadmap features : | ||
|
|
||
| - Monitoring via **Prometheus** | ||
| - Automatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins) | ||
| - graceful NiFi cluster **scaling and rebalancing** | ||
|
|
||
| ## Motivation | ||
|
|
||
| At [Orange](https://opensource.orange.com/fr/accueil/) we are building some [Kubernetes operator](https://github.com/Orange-OpenSource?utf8=%E2%9C%93&q=operator&type=&language=), that operate NiFi and Cassandra clusters (among other types) for our business cases. | ||
|
|
||
| There are already some approaches to operating NiFi on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our needs. | ||
|
|
||
| - [Helm chart](https://github.com/cetic/helm-nifi) | ||
| - [Cloudera Nifi Operator](https://blog.cloudera.com/cloudera-flow-management-goes-cloud-native-with-apache-nifi-on-red-hat-openshift-kubernetes-platform/) | ||
|
|
||
| Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator. |
62 changes: 62 additions & 0 deletions
62
site/website/versioned_docs/version-v0.6.2/1_concepts/2_design_principes.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| id: 2_design_principes | ||
| title: Design Principes | ||
| sidebar_label: Design Principes | ||
| --- | ||
|
|
||
| ## Pod level management | ||
|
|
||
| NiFi is a stateful application. The first piece of the puzzle is the Node, which is a simple server capable of createing/forming a cluster with other Nodes. Every Node has his own **unique** configuration which differs slightly from all others. | ||
|
|
||
| All NiFi on Kubernetes setup use [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) to create a NiFi Cluster. Just to quickly recap from the K8s docs: | ||
|
|
||
| >StatefulSet manages the deployment and scaling of a set of Pods, and provide guarantees about their ordering and uniqueness. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains sticky identities for each of its Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that is maintained across any rescheduling. | ||
| How does this looks from the perspective of Apache NiFi ? | ||
|
|
||
| With StatefulSet we get : | ||
| - unique Node IDs generated during Pod startup | ||
| - networking between Nodes with headless services | ||
| - unique Persistent Volumes for Nodes | ||
|
|
||
| Using StatefulSet we **lose** the ability to : | ||
|
|
||
| - modify the configuration of unique Nodes | ||
| - remove a specific Node from a cluster (StatefulSet always removes the most recently created Node) | ||
| - use multiple, different Persistent Volumes for each Node | ||
|
|
||
| The Orange NiFi Operator uses `simple` Pods, ConfigMaps, and PersistentVolumeClaims, instead of StatefulSet (based on the design used by [Banzai Cloud Kafka Operator](https://github.com/banzaicloud/kafka-operator)). | ||
| Using these resources allows us to build an Operator which is better suited to NiFi. | ||
|
|
||
| With the Orange NiFi operator we can: | ||
|
|
||
| - modify the configuration of unique Nodes | ||
| - remove specific Nodes from clusters | ||
| - use multiple Persistent Volumes for each Node | ||
|
|
||
| ## Dataflow Lifecycle management | ||
|
|
||
| The [Dataflow Lifecycle management feature](./3_features.md#dataflow-lifecycle-management-via-crd) introduces 3 new CRDs : | ||
|
|
||
| - **NiFiRegistryClient :** Allowing you to declare a [NiFi registry client](https://nifi.apache.org/docs/nifi-registry-docs/html/getting-started.html#connect-nifi-to-the-registry). | ||
| - **NiFiParameterContext :** Allowing you to create parameter context, with two kinds of parameters, a simple `map[string]string` for non-sensitive parameters and a `list of secrets` which contains sensitive parameters. | ||
| - **NiFiDataflow :** Allowing you to declare a Dataflow based on a `NiFiRegistryClient` and optionally a `ParameterContext`, which will be deployed and managed by the operator on the `targeted NiFi cluster`. | ||
|
|
||
| The following diagram shows the interactions between all the components : | ||
|
|
||
|  | ||
|
|
||
| With each CRD comes a new controller, with a reconcile loop : | ||
|
|
||
| - **NiFiRegistryClient's controller :** | ||
|
|
||
|  | ||
|
|
||
| - **NiFiParameterContext's controller :** | ||
|
|
||
|  | ||
|
|
||
| - **NiFiDataflow's controller :** | ||
|
|
||
|  | ||
|
|
57 changes: 57 additions & 0 deletions
57
site/website/versioned_docs/version-v0.6.2/1_concepts/3_features.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| --- | ||
| id: 3_features | ||
| title: Features | ||
| sidebar_label: Features | ||
| --- | ||
|
|
||
| To highligt some of the features we needed and were not possible with the operators available, please keep reading | ||
|
|
||
| ## Fine Grained Node Config Support | ||
|
|
||
| We needed to be able to react to events in a fine-grained way for each Node - and not in the limited way StatefulSet does (which, for example, removes the most recently created Nodes). Some of the available solutions try to overcome these deficits by placing scripts inside the container to generate configs at runtime (a good example is our [Cassandra Operator](https://github.com/Orange-OpenSource/casskop)), whereas the Orange NiFi operator's configurations are deterministically placed in specific Configmaps. | ||
|
|
||
| ## Graceful NiFi Cluster Scaling | ||
|
|
||
| Apache NiFi is a good candidate to create an operator, because everything is made to orchestrate it through REST Api calls. With this comes automation of actions such as scaling, following all required steps : https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#decommission-nodes. | ||
|
|
||
| ## Graceful Rolling Upgrade | ||
|
|
||
| Operator supports graceful rolling upgrade. It means the operator will check if the cluster is healthy. | ||
|
|
||
| ## Dynamic Configuration Support | ||
|
|
||
| NiFi operates with two type of configs: | ||
|
|
||
| - Read-only | ||
| - PerNode | ||
|
|
||
| Read only config requires node restart to update all the others may be updated dynamically. | ||
| Operator CRD distinguishes these fields, and proceed with the right action. It can be a rolling upgrade, or | ||
| a dynamic reconfiguration. | ||
|
|
||
| ## Dataflow lifecycle management via CRD | ||
|
|
||
| In a cloud native approach, we are looking for important management features, which we have applied to NiFi Dataflow : | ||
|
|
||
| - **Automated deployment :** Based on the NiFi registry, you can describe your `NiFiDataflow` resource that will be deployed and run on the targeted NiFi cluster. | ||
| - **Portability :** On kubernetes everything is a yaml file, so with NiFiKop we give you the ability to describe your clusters but also the `registry clients`, `parameter contexts` and `dataflows` of your NiFi application, so that you can redeploy the same thing in a different namespace or cluster. | ||
| - **State management :** With NiFiKop resources, you can describe what you want, and the operator deals with the NiFi Rest API to make sure the resource stays in sync (even if someone manually makes changes directly on NiFi cluster). | ||
| - **Configurations :** Based on the `Parameter Contexts`, NiFiKop allows you to associate to your `Dataflow` (= your applications) with a different configuration depending on the environment ! | ||
|
|
||
| ## Users and access policies management | ||
|
|
||
| Without the management of users and access policies associated, it was not possible to have a fully automated NiFi cluster setup due to : | ||
|
|
||
| - **Node scaling :** when a new node joins the cluster it needs to have some roles like `proxy user request`, `view data` etc., by managing users and access policies we can easily create a user for this node with the right accesses. | ||
| - **Operator admin rigth :** For the operator to manage efficiently the cluster it needs a lot of rights as `deploying process groups`, `empty the queues` etc., these rights are not available by default when you set a user as [InitialAdmin](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity). Once again by giving the ability to define users and access policies we go through this. | ||
| - **User's access :** as seen just below we need to define the operator as `InitialAdmin`, in this situation there is no more users that can access to the web UI to manually give access to other users. That's why we extend the `InitialAdmin` concept into the operator, giving the ability to define a list of users as admins. | ||
|
|
||
| In addition to these requirements to have a fully automated and managed cluster, we introduced some useful features : | ||
|
|
||
| - **User management :** using `NifiUser` resource, you are able to create (or bind an existing) user in NiFi cluster and apply some access policies that will be managed and continuously synced by the operator. | ||
| - **Group management :** using `NifiUserGroup` resource, you can create groups in NiFi cluster and apply access policies and a list of `NifiUser` that will be managed and continuously synced by the operator. | ||
| - **Default group :** As the definition of `NifiUser` and `NifiUserGroup` resources could be heavy for some simple use cases, we also decided to define two default groups that you can feed with a list of users that will be created and managed by the operator (no kubernetes resources to create) : | ||
| - **Admins :** a group giving access to everything on the NiFi Cluster, | ||
| - **Readers :** a group giving access as viewer on the NiFi Cluster. | ||
|
|
||
| By introducing this feature we are giving you the ability to fully automate your deployment, from the NiFi Cluster to your managed NiFi Dataflow. |
95 changes: 95 additions & 0 deletions
95
site/website/versioned_docs/version-v0.6.2/1_concepts/4_roadmap.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| --- | ||
| id: 4_roadmap | ||
| title: Roadmap | ||
| sidebar_label: Roadmap | ||
| --- | ||
|
|
||
| ## Available | ||
|
|
||
| ### NiFi cluster installation | ||
|
|
||
| | | | | ||
| | --------------------- | --------- | | ||
| | Status | Done | | ||
| | Priority | High | | ||
| | Targeted Start date | Jan 2020 | | ||
|
|
||
| ### Graceful NiFi Cluster Scaling | ||
|
|
||
| | | | | ||
| | --------------------- | --------- | | ||
| | Status | Done | | ||
| | Priority | High | | ||
| | Targeted Start date | Jan 2020 | | ||
|
|
||
| Apache NiFi is a good candidate to create an operator, because everything is made to orchestrate it through REST Api calls. With this comes automation of actions such as scaling, following all required steps : https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#decommission-nodes. | ||
|
|
||
| ### Communication via SSL | ||
|
|
||
| | | | | ||
| | --------------------- | -------- | | ||
| | Status | Done | | ||
| | Priority | High | | ||
| | Targeted Start date | May 2020 | | ||
|
|
||
|
|
||
| The operator fully automates NiFi's SSL support. | ||
| The operator can provision the required secrets and certificates for you, or you can provide your own. | ||
|
|
||
| ### Dataflow lifecycle management via CRD | ||
|
|
||
| | | | | ||
| | --------------------- | --------- | | ||
| | Status | Done | | ||
| | Priority | High | | ||
| | Targeted Start date | Aug 2020 | | ||
|
|
||
| ### Users & access policies management | ||
|
|
||
| | | | | ||
| | --------------------- | ----- | | ||
| | Status | Done| | ||
| | Priority | High | | ||
| | Targeted Start date | November 2020 | | ||
|
|
||
| The operator fully automates NiFi's user and access policies management. | ||
|
|
||
| ## Backlog | ||
|
|
||
| ### Monitoring via Prometheus | ||
|
|
||
| | | | | ||
| | --------------------- | -------- | | ||
| | Status | To Do | | ||
| | Priority | High | | ||
| | Targeted Start date | Oct 2020 | | ||
|
|
||
| The NiFi operator exposes NiFi JMX metrics to Prometheus. | ||
|
|
||
| ### Reacting on Alerts | ||
|
|
||
| | | | | ||
| | --------------------- | ----- | | ||
| | Status | To Do | | ||
| | Priority | Low | | ||
| | Targeted Start date | - | | ||
|
|
||
| The NiFi Operator acts as a **Prometheus Alert Manager**. It receives alerts defined in Prometheus, and creates actions based on Prometheus alert annotations. | ||
|
|
||
| Currently, there are three actions expected : | ||
| - upscale cluster (add a new Node) | ||
| - downscale cluster (remove a Node) | ||
| - add additional disk to a Node | ||
|
|
||
| ### Seamless Istio mesh support | ||
|
|
||
| | | | | ||
| | --------------------- | ----- | | ||
| | Status | To Do | | ||
| | Priority | Low | | ||
| | Targeted Start date | - | | ||
|
|
||
| - Operator allows to use ClusterIP services instead of Headless, which still works better in case of Service meshes. | ||
| - To avoid too early nifi initialization, which might lead to unready sidecar container. The operator will use a small script to | ||
| mitigate this behaviour. All NiFi image can be used the only one requirement is an available **wget** command. | ||
| - To access a NiFi cluster which runs inside the mesh. Operator will supports creating Istio ingress gateways. |
Oops, something went wrong.