Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated structure for CD docs #242

Merged
merged 16 commits into from
Nov 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ help:
yarn-install: .PHONY
yarn

web: yarn ## runs the docs site locally
web: ## runs the docs site locally
yarn dev
6 changes: 4 additions & 2 deletions pages/adding-new-application/background-app-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >-

In this guide we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app.

When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under.
When a user sets up a new Plural workspace in a git repository (we'll call that a _deployment repository_ going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under.
Next, the user can install an application bundle using the `plural bundle install <app_name> <bundle>` CLI command.

> Most applications come with more than one bundle, one for each targeted cloud provider.
Expand All @@ -32,6 +32,7 @@ $ tree -L 1 .
```

The `workspace.yaml` would look like this:

```yaml
apiVersion: plural.sh/v1alpha1
kind: ProjectManifest
Expand All @@ -40,7 +41,7 @@ metadata:
spec:
cluster: pluraldev
bucket: pluraldevsh-tf-state
project: "123456765432"
project: '123456765432'
provider: aws
region: us-east-2
owner:
Expand All @@ -53,6 +54,7 @@ spec:
```

And the `context.yaml` like this. In the `spec.configuration` section you can see the recorded user input for each artifact.

```yaml
apiVersion: plural.sh/v1alpha1
kind: Context
Expand Down
2 changes: 1 addition & 1 deletion pages/adding-new-application/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ description: >-
If you want to contribute your own application to the Plural marketplace we invite you to create and publish your own artifacts.
In this section we guide you through the ins and outs of Plural's application artifacts, how you can build your own with the tools that Plural provides you, and how you can publish them for others to use.

> We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program).
> We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program).
116 changes: 60 additions & 56 deletions pages/adding-new-application/plural-artifact-structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ description: >-
As mentioned in [Background on Application Installations](/adding-new-application/background-app-install),
the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user provided values for that installation.
Some extra configuration files are necessary in the application's artifact for the Plural API to be able to understand:

- the Helm charts and Terraform modules dependencies to run them through its templating engine
- dependencies on other Plural artifacts
- platform specific components and infrastructure configurations
Expand Down Expand Up @@ -83,8 +84,7 @@ $ tree .
└── vendor_images.yaml
```

Let's disect this artifact's structure.

Let's dissect this artifact's structure.

## Helm

Expand All @@ -105,7 +105,8 @@ Plural leverages dependency tracking of applications to achieve a high degree of
Dependencies between artifacts are defined in the recipe files (see below).
Dependencies are also tracked between the Helm charts and Terraform modules of other applications in a dedicated `deps.yaml` file in each chart's or module's directory.
For example, for Dagster's Helm chart you would list required dependencies on:
- the `bootstrap` application's Helm chart

- the `bootstrap` application's Helm chart
- the `ingress-nginx` application's Helm chart
- the `postrges` operator application's Helm chart

Expand All @@ -120,33 +121,33 @@ metadata:
spec:
breaking: true
dependencies:
- type: helm
name: bootstrap
repo: bootstrap
version: '>= 0.5.1'
- type: helm
name: ingress-nginx
repo: ingress-nginx
version: ">= 0.1.2"
- type: helm
name: postgres
repo: postgres
version: ">= 0.1.6"
- type: terraform
name: aws
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: azure
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: gcp
repo: dagster
version: '>= 0.1.0'
optional: true
- type: helm
name: bootstrap
repo: bootstrap
version: '>= 0.5.1'
- type: helm
name: ingress-nginx
repo: ingress-nginx
version: '>= 0.1.2'
- type: helm
name: postgres
repo: postgres
version: '>= 0.1.6'
- type: terraform
name: aws
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: azure
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: gcp
repo: dagster
version: '>= 0.1.0'
optional: true
```

## Terraform
Expand All @@ -157,9 +158,10 @@ Most commonly you'd find object storage alongside their IAM resources, or additi
Sometimes it will also include Kubernetes resources like service accounts, if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner.

> One peculiarity about the Terraform modules is that, at the very least, they need to contain the Kubernetes namespace for your application.
This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step.
> This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step.

Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about dependencies on other modules.

```
apiVersion: plural.sh/v1alpha1
kind: Dependencies
Expand All @@ -176,7 +178,6 @@ spec:
- aws
```


## Plural Files

The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`).
Expand All @@ -185,6 +186,7 @@ On the top-level directory of each artifact you'll also find a`repository.yaml`.
The `repository.yaml` and recipe YAMLs are an integral part of Plural's artifact packaging format.

### `repository.yaml`

```yaml
name: dagster
description: A data orchestration platform for the development, production, and observation of data assets.
Expand All @@ -199,53 +201,55 @@ oauthSettings:
uriFormat: https://{domain}/oauth2/callback
authMethod: POST
tags:
- tag: dag
- tag: data
- tag: data-pipelines
- tag: dag
- tag: data
- tag: data-pipelines
```

The metadata in this file informs the Plural API about the application this artifact envelopes.
It will store its name, category and description, where it can find the icon and docs to display in the marketplace,
the notes template to prompt after installation, as well as links to any upstream git repository or homepage (if applicable).

`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it.

> Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt.
This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart.
> This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart.

The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push` or `plural apply` (see [Publishing](/adding-new-application/publishing)).

### `plural/recipes/dagster-aws.yaml`

```yaml
name: dagster-aws
description: Installs dagster on an aws eks cluster
provider: AWS
primary: true
dependencies:
- repo: bootstrap
name: aws-k8s
- repo: ingress-nginx
name: ingress-nginx-aws
- repo: postgres
name: aws-postgres
- repo: bootstrap
name: aws-k8s
- repo: ingress-nginx
name: ingress-nginx-aws
- repo: postgres
name: aws-postgres
oidcSettings:
uriFormat: https://{domain}/oauth2/callback
authMethod: POST
domainKey: hostname
sections:
- name: dagster
configuration:
- name: dagsterBucket
type: BUCKET
documentation: s3 bucket for storing dagster logs
default: dagster
- name: hostname
type: DOMAIN
documentation: fqdn on which to deploy your dagster instance
items:
- type: TERRAFORM
name: aws
- type: HELM
name: dagster
- name: dagster
configuration:
- name: dagsterBucket
type: BUCKET
documentation: s3 bucket for storing dagster logs
default: dagster
- name: hostname
type: DOMAIN
documentation: fqdn on which to deploy your dagster instance
items:
- type: TERRAFORM
name: aws
- type: HELM
name: dagster
```

The recipe file ties a bundle together, with one dedicated recipe per cloud provider.
Expand Down
7 changes: 5 additions & 2 deletions pages/adding-new-application/publishing.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,17 @@ To publish an artifact you first need to create a publisher with your Plural acc

1. Click on your profile on the bottom left on `app.plural.sh`.
2. Select `create publisher`.
3. Enter your information.
3. Enter your information.
4. Hit save.

## Publish an Artifact with the Plural CLI

When you're done with implementing your artifact and you have created a publisher you're ready to publish it.
When you're done with implementing your artifact and you have created a publisher you're ready to publish it.

1. Navigate to the directory of your artifact in your local copy of the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts).
2. Locate the `Pluralfile`.
3. Enter the name of your publisher as the first argument in the `ATTRIBUTES` line. E.g. if your publisher is called `mypublisher`, it should look like this:

```
REPO dagster
ATTRIBUTES mypublisher repository.yaml
Expand All @@ -29,7 +30,9 @@ TF terraform/*
HELM helm/*
RECIPE plural/recipes/*
```

4. Finally, publish your artifact with the command `plural apply -f Pluralfile`. E.g. for our dagster example

```console
$ plural apply -f Pluralfile
Setting attributes for dagster
Expand Down
17 changes: 17 additions & 0 deletions pages/deployments/addons.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
title: Kubernetes Add-Ons
description: Extend Kubernetes to support load balancing, dns, monitoring and security
---

Kubernetes natively is just a container orchestrator, but it's apis can be easily extended to support many other major concerns within the DevOps space. Some of the more common usecases we currently support are:

- Networking - you need load balancers, dns, and cert management to build a full production service exposed to the outside world. We can deploy all the operators and controllers needed to get this set up in a fully self-service way.
- Monitoring - inject agents from Datadog, New Relic, Prometheus and other providers to provide observability to your fleet
- Security - tools like Trivy and Kubescape can continuously monitor clusters for vulnerabilities and enhance your security posture.
- Cost Monitoring - deploy the kubecost agent to give you coherent cost dashboards for your entire fleet

All of these will be constantly updated and maintained by us, you just need to perform the initial configuration via our wizards.

You can see all the addons we support in the `Add Ons` tab of Plural Deployments, as seen here:

![](/assets/deployments/addons.png)
35 changes: 35 additions & 0 deletions pages/deployments/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Plural CD Architecture
description: Secure, Scalable Pull-Based Architecture
---

## Overview

Plural CD is based on a scalable, secure, agent-based pull architecture. It doesn't require direct access to any of the clusters it deploys to, meaning it can manage workloads in any cloud, on-prem, on the edge, or even on a local laptop running KIND. Further since it doesn't require networking intensive kubernetes watch streams, central access to kubernetes, or relies on single-mastered operator control loops, it should scale to virtually any size kubernetes fleet. We also enhance your kubernetes setup with a auth proxy, allowing you to have full visibility to your workloads without compromising the network security of your setup or require the creation of complex multi-cloud networking setups.

Here's a quick diagram of the setup:

![](/assets/deployments/architecture.png)

## Control Plane

The Control Plane layer is a full-stack service deployable onto any kubernetes cluster you designate as your management cluster. It contains a few main components:

- Horizontally scalable git cache - we should be able to ingest as many git repos as you'd like and auto-shard them throughout your cluster automatically and efficiently.
- Configuration Management - supports re-configurable backends, but allows you to easily parameterize services with information like hostnames, docker image tags, and other secret and non-secret information.
- Auth Proxy - this is a secure bidirectional grpc channel initiated by a deployment agent used to make kubernetes api calls no matter where a cluster may live and give you full dashboarding capabilities from the Plural CD UI.
- Cluster API Providers - Plural CD natively integrates with cluster api and allows you to create and manage new clusters at scale and fork your own kubernetes cluster APIs on top of existing setups for services like EKS, AKS and GKE or on-prem solutions like Rancher

We provide simple installers if you'd like to deploy the control plane layer to a kubernetes cluster already in your fleet, or you can use our own kubernetes setup in the standard plural install flow.

## Deployment Agent

A thin deployment agent is installed onto each cluster and perpetually managed by Plural CD from then on. It will perpetually poll the control plane for new services to apply and if there are any changes to make, apply them into your cluster. It also can do a few other things like:

- establish the bidectional grpc channel for the auth proxy layer (this does require an ingress with websocket support)
- execute integration tests

There were a few design considerations involved in our agent that are worth understanding as well:

- The expectation is the agent is deployed independently at scale, potentially hundreds of times for a large organization, so it must be maximally simple to reduce operational complexity to near 0.
- It is designed to be extensible to arbitrarily many frameworks for defining kubernetes manifests. We aren't actually huge fans of helm or kustomize and want organizations to have the flexibility to ultimately use other toolchains for managing their kubernetes YAML codebase (or ultimately de-YAML themselves)
53 changes: 53 additions & 0 deletions pages/deployments/browser-quickstart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: Plural CD In-Browser Quickstart
description: Deploying your services using the Plural Console UI.
---

## Overview

This guide goes over how to deploy your services with Plural using the Plural Console UI. At the end of this tutorial, you will have:

- Provisioned new clusters, and/or deployed the Plural Deployment Operator on your existing clusters.
- Imported the Git repositories containing your code and manifests.
- Deployed your code onto your clusters of choice.
- Optionally updated any configurations and permissions for the clusters and services.

in under 30 minutes, all within your browser. You can see the process in the video here or follow the instructions step-by-step:

{% embed url="https://www.youtube.com/watch?v=jBlq45ntbBw" aspectRatio="16 / 9" /%}

## Onboard to Plural and install the Plural Console

If you haven't already, you'll need to follow the Plural guide to install Console. You can use the guide for the [in-browser Cloud Shell](/getting-started/cloud-shell-quickstart) or the [CLI](/getting-started/quickstart) to get started.

## Log in to Console and Add Clusters

Once you've logged in to Plural Console, you can click the "Continuous deployment" tab to set up your clusters. By default, you should see the cluster that your Console is deployed on.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino just checking that this is true, they'll always at least have their current console cluster visible here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep


To deploy additional clusters, click the "Create cluster" button in the top right. To spin up a new cluster, fill in the required fields:

![](/assets/deployments/create-cluster.png)

To import an existing cluster, select the "Import existing" toggle and add a name, handle, and any tags. You then need to run the following command on the cluster itself to install the Plural Deploy Operator:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino any additional detail here on where to run this command? I can add this to the BYOK section too if helpful


```
plural cd install --url https://<CONSOLE_URL>/ext/gql --token <TOKEN>
```

Make sure your kubeconfig points to the desired cluster for import. More information is available in the guide for [existing clusters](/deployments/existing-cluster).

Once you've added the required clusters, you should see them populated on the "Clusters" subtab within your Console.

## Import Git Repositories and Deploy services

You'll need to then import the Git repository containing your service and the associated Kubernetes manifests. Click the "Git repositories" subtab, and hit the "Import Git" button in the top right.

![](/assets/deployments/import-git.png)

Add the url of the repo and authorization if necessary, and hit import. Once your service is pullable, you can deploy it to your clusters.

Navigate to the "Services" subtab and click "Deploy service". Add a name and namespace and select the cluster you want to deploy to. You can then select the relevant repository and add any secrets necessary to parameterize the deployment.

Your service should populate initially as `Stale` until your components are Ready, at which point they will flip to `Healthy`.

Congratulations! You've deployed your first service with Plural.
Loading
Loading