Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated structure for CD docs #242

Merged
merged 16 commits into from
Nov 13, 2023
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions pages/adding-new-application/background-app-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >-

In this guide we will lay out how your user provided values tie in with the deployment process as it relates to the configuration and templating of the app.

When a user sets up a new Plural workspace in a git repository (we'll call that a *deployment repository* going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under.
When a user sets up a new Plural workspace in a git repository (we'll call that a _deployment repository_ going forward) a `workspace.yaml` file is created that stores global values for that cluster such as the cloud account and region, the cluster and VPC name and what subdomain all the applications will be hosted under.
Next, the user can install an application bundle using the `plural bundle install <app_name> <bundle>` CLI command.

> Most applications come with more than one bundle, one for each targeted cloud provider.
Expand All @@ -32,6 +32,7 @@ $ tree -L 1 .
```

The `workspace.yaml` would look like this:

```yaml
apiVersion: plural.sh/v1alpha1
kind: ProjectManifest
Expand All @@ -40,7 +41,7 @@ metadata:
spec:
cluster: pluraldev
bucket: pluraldevsh-tf-state
project: "123456765432"
project: '123456765432'
provider: aws
region: us-east-2
owner:
Expand All @@ -53,6 +54,7 @@ spec:
```

And the `context.yaml` like this. In the `spec.configuration` section you can see the recorded user input for each artifact.

```yaml
apiVersion: plural.sh/v1alpha1
kind: Context
Expand Down
2 changes: 1 addition & 1 deletion pages/adding-new-application/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ description: >-
If you want to contribute your own application to the Plural marketplace we invite you to create and publish your own artifacts.
In this section we guide you through the ins and outs of Plural's application artifacts, how you can build your own with the tools that Plural provides you, and how you can publish them for others to use.

> We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program).
> We are focused on properly compensating any contributions to the Plural platform, which includes a meaningful [bounty](https://github.com/pluralsh/plural-artifacts#plural-contributor-program).
114 changes: 59 additions & 55 deletions pages/adding-new-application/plural-artifact-structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ description: >-
As mentioned in [Background on Application Installations](/adding-new-application/background-app-install),
the Plural CLI creates a wrapper Helm chart and Terraform module for each installed application and inputs the user provided values for that installation.
Some extra configuration files are necessary in the application's artifact for the Plural API to be able to understand:

- the Helm charts and Terraform modules dependencies to run them through its templating engine
- dependencies on other Plural artifacts
- platform specific components and infrastructure configurations
Expand Down Expand Up @@ -85,7 +86,6 @@ $ tree .

Let's disect this artifact's structure.
dogmar marked this conversation as resolved.
Show resolved Hide resolved


## Helm

The `helm` directory contains the app's Helm chart as it will be available through the Plural API and used by the Plural CLI to configure and deploy the Kubernetes components into your cluster.
Expand All @@ -105,7 +105,8 @@ Plural leverages dependency tracking of applications to achieve a high degree of
Dependencies between artifacts are defined in the recipe files (see below).
Dependencies are also tracked between the Helm charts and Terraform modules of other applications in a dedicated `deps.yaml` file in each chart's or module's directory.
For example, for Dagster's Helm chart you would list required dependencies on:
- the `bootstrap` application's Helm chart

- the `bootstrap` application's Helm chart
- the `ingress-nginx` application's Helm chart
- the `postrges` operator application's Helm chart

Expand All @@ -120,33 +121,33 @@ metadata:
spec:
breaking: true
dependencies:
- type: helm
name: bootstrap
repo: bootstrap
version: '>= 0.5.1'
- type: helm
name: ingress-nginx
repo: ingress-nginx
version: ">= 0.1.2"
- type: helm
name: postgres
repo: postgres
version: ">= 0.1.6"
- type: terraform
name: aws
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: azure
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: gcp
repo: dagster
version: '>= 0.1.0'
optional: true
- type: helm
name: bootstrap
repo: bootstrap
version: '>= 0.5.1'
- type: helm
name: ingress-nginx
repo: ingress-nginx
version: '>= 0.1.2'
- type: helm
name: postgres
repo: postgres
version: '>= 0.1.6'
- type: terraform
name: aws
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: azure
repo: dagster
version: '>= 0.1.0'
optional: true
- type: terraform
name: gcp
repo: dagster
version: '>= 0.1.0'
optional: true
```

## Terraform
Expand All @@ -157,9 +158,10 @@ Most commonly you'd find object storage alongside their IAM resources, or additi
Sometimes it will also include Kubernetes resources like service accounts, if their deployment cannot be achieved through the artifact's Helm chart in a convenient manner.

> One peculiarity about the Terraform modules is that, at the very least, they need to contain the Kubernetes namespace for your application.
This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step.
> This is because during a `plural deploy` with the Plural CLI the `terraform apply` will always run before the `helm install` step.

Just like the Helm chart, the Terraform modules also contain a `deps.yaml` file that inform the Plural API about dependencies on other modules.

```
apiVersion: plural.sh/v1alpha1
kind: Dependencies
Expand All @@ -176,7 +178,6 @@ spec:
- aws
```


## Plural Files

The `plural` directory contains the artifact's packaging information (`plural/recipes`), metadata (`plural/tags` and `plural/icons`), and application specific instructions (`plural/docs` and `plural/notes.tpl`).
Expand All @@ -185,6 +186,7 @@ On the top-level directory of each artifact you'll also find a`repository.yaml`.
The `repository.yaml` and recipe YAMLs are an integral part of Plural's artifact packaging format.

### `repository.yaml`

```yaml
name: dagster
description: A data orchestration platform for the development, production, and observation of data assets.
Expand All @@ -199,53 +201,55 @@ oauthSettings:
uriFormat: https://{domain}/oauth2/callback
authMethod: POST
tags:
- tag: dag
- tag: data
- tag: data-pipelines
- tag: dag
- tag: data
- tag: data-pipelines
```

The metadata in this file informs the Plural API about the application this artifact envelopes.
It will store its name, category and description, where it can find the icon and docs to display in the marketplace,
the notes template to prompt after installation, as well as links to any upstream git repository or homepage (if applicable).

`oauthSettings` specifies the URI format for the OIDC callback address and informs the Plural API how to setup the OIDC endpoint for your application if you use it.

> Behind the scenes, every `plural bundle install` triggers the OIDC client creation when you answer with `yes` on the OIDC prompt.
This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart.
> This happens, because every client needs to be created before a `plural build` which then inputs the client info into the helm chart.

The `private` flag controls whether the artifact's bundles are published publicly or privately on a `plural push` or `plural apply` (see [Publishing](/adding-new-application/publishing)).

### `plural/recipes/dagster-aws.yaml`

```yaml
name: dagster-aws
description: Installs dagster on an aws eks cluster
provider: AWS
primary: true
dependencies:
- repo: bootstrap
name: aws-k8s
- repo: ingress-nginx
name: ingress-nginx-aws
- repo: postgres
name: aws-postgres
- repo: bootstrap
name: aws-k8s
- repo: ingress-nginx
name: ingress-nginx-aws
- repo: postgres
name: aws-postgres
oidcSettings:
uriFormat: https://{domain}/oauth2/callback
authMethod: POST
domainKey: hostname
sections:
- name: dagster
configuration:
- name: dagsterBucket
type: BUCKET
documentation: s3 bucket for storing dagster logs
default: dagster
- name: hostname
type: DOMAIN
documentation: fqdn on which to deploy your dagster instance
items:
- type: TERRAFORM
name: aws
- type: HELM
name: dagster
- name: dagster
configuration:
- name: dagsterBucket
type: BUCKET
documentation: s3 bucket for storing dagster logs
default: dagster
- name: hostname
type: DOMAIN
documentation: fqdn on which to deploy your dagster instance
items:
- type: TERRAFORM
name: aws
- type: HELM
name: dagster
```

The recipe file ties a bundle together, with one dedicated recipe per cloud provider.
Expand Down
7 changes: 5 additions & 2 deletions pages/adding-new-application/publishing.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,17 @@ To publish an artifact you first need to create a publisher with your Plural acc

1. Click on your profile on the bottom left on `app.plural.sh`.
2. Select `create publisher`.
3. Enter your information.
3. Enter your information.
4. Hit save.

## Publish an Artifact with the Plural CLI

When you're done with implementing your artifact and you have created a publisher you're ready to publish it.
When you're done with implementing your artifact and you have created a publisher you're ready to publish it.

1. Navigate to the directory of your artifact in your local copy of the [Plural artifacts git repository](https://github.com/pluralsh/plural-artifacts).
2. Locate the `Pluralfile`.
3. Enter the name of your publisher as the first argument in the `ATTRIBUTES` line. E.g. if your publisher is called `mypublisher`, it should look like this:

```
REPO dagster
ATTRIBUTES mypublisher repository.yaml
Expand All @@ -29,7 +30,9 @@ TF terraform/*
HELM helm/*
RECIPE plural/recipes/*
```

4. Finally, publish your artifact with the command `plural apply -f Pluralfile`. E.g. for our dagster example

```console
$ plural apply -f Pluralfile
Setting attributes for dagster
Expand Down
53 changes: 53 additions & 0 deletions pages/deployments/browser-quickstart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: Plural CD In-Browser Quickstart
description: Deploying your services using the Plural Console UI.
---

## Overview

This guide goes over how to deploy your services with Plural using the Plural Console UI. At the end of this tutorial, you will have:

- Provisioned new clusters, and/or deployed the Plural Deployment Operator on your existing clusters.
- Imported the Git repositories containing your code and manifests.
- Deployed your code onto your clusters of choice.
- Optionally updated any configurations and permissions for the clusters and services.

in under 30 minutes, all within your browser. You can see the process in the video here or follow the instructions step-by-step:

{% embed url="https://www.youtube.com/watch?v=jBlq45ntbBw" aspectRatio="16 / 9" /%}

## Onboard to Plural and install the Plural Console

If you haven't already, you'll need to follow the Plural guide to install Console. You can use the guide for the [in-browser Cloud Shell](/getting-started/cloud-shell-quickstart) or the [CLI](/getting-started/quickstart) to get started.

## Log in to Console and Add Clusters

Once you've logged in to Plural Console, you can click the "Continuous deployment" tab to set up your clusters. By default, you should see the cluster that your Console is deployed on.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino just checking that this is true, they'll always at least have their current console cluster visible here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep


To deploy additional clusters, click the "Create cluster" button in the top right. To spin up a new cluster, fill in the required fields:

![](/assets/deployments/create-cluster.png)

To import an existing cluster, select the "Import existing" toggle and add a name, handle, and any tags. You then need to run the following command on the cluster itself to install the Plural Deploy Operator:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino any additional detail here on where to run this command? I can add this to the BYOK section too if helpful


```
plural cd install --url https://<CONSOLE_URL>/ext/gql -- token <TOKEN>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino this is just the token from Console right? I can add instructions on how to get it or link to that helper page if so

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will also remove space in front of token

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the ui should tell them exactly what to run, yea

```

Make sure your kubeconfig points to the desired cluster for import.

Once you've added the required clusters, you should see them populated on the "Clusters" subtab within your Console.

## Import Git Repositories and Deploy services

You'll need to then import the Git repository containing your service and the associated Kubernetes manifests. Click the "Git repositories" subtab, and hit the "Import Git" button in the top right.

![](/assets/deployments/import-git.png)

Add the url of the repo and authorization if necessary, and hit import. Once your service is pullable, you can deploy it to your clusters.

Navigate to the "Services" subtab and click "Deploy service". Add a name and namespace and select the cluster you want to deploy to. You can then select the relevant repository and add any secrets necessary to parameterize the deployment.

Your service should populate initially as `Stale` until your components are Ready, at which point they will flip to `Healthy`.

Congratulations! You've deployed your first service with Plural.
4 changes: 4 additions & 0 deletions pages/deployments/cli-quickstart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Plural CD CLI Quickstart
description: Deploying your services using the Plural CLI.
---
4 changes: 4 additions & 0 deletions pages/deployments/cluster-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Configure my Cluster
description: XXX
---
4 changes: 4 additions & 0 deletions pages/deployments/cluster-cost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Optimize Cluster Costs
description: XXX
---
4 changes: 4 additions & 0 deletions pages/deployments/cluster-create.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Create Workload Clusters
description: XXX
---
4 changes: 4 additions & 0 deletions pages/deployments/cluster-destroy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Destroy the Cluster Safely
description: XXX
---
10 changes: 10 additions & 0 deletions pages/deployments/clusters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
title: Cluster Management
description: XXX
---

The Plural team is currently exploring a solution to allow our customers to deploy their in-house services using Plural.

We believe that combining the ability to quickly spin up new clusters, author simple Kubernetes manifests, and control the deployment of code changes from development to production would the process of developing on Kubernetes significantly easier.

If you're interested, you can find more information on our website at [https://www.plural.sh/plural-deployments-early-access](https://www.plural.sh/plural-deployments-early-access).
4 changes: 4 additions & 0 deletions pages/deployments/credentials.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Create Provider Credentials
description: XXX
---
4 changes: 4 additions & 0 deletions pages/deployments/debugging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Debugging
description: XXX
---
4 changes: 4 additions & 0 deletions pages/deployments/deprecations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Managing Deprecations and K8s upgrades
description: XXX
---
28 changes: 28 additions & 0 deletions pages/deployments/git.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: Import Git Repositories
description: How to import a repo containing your service and manifests
---

## Structure

To deploy your service with Plural, configure your repository with a folder containing your Kubernetes manifests that indicates how your service should be deployed. As an example, you can check out the charts folder of the [Plural Deployment Operator](https://github.com/pluralsh/deployment-operator/tree/main/charts/deployment-operator).

## Import from the Browser

To deploy a service, Plural needs to pull the Git repository containing your Kubernetes manifests. Navigate to the "Git repositories" subtab and click the "Import Git" button in the top right corner.

Connect your repo by adding the URL. Plural supports both SSH and basic auth. If your manifests are located in private repo, add your credentials via upload or manual entry.

Plural will then attempt to connect to the repository. If successful, the status will update to `Pullable`, and you can then deploy your service from the "Services" subtab.

To update the repository URL, use the `Update` button inline with the repository row.

## Import from the CLI

To import a repo via the CLI, use the `plural cd repositories create` command. Provide the URL and any authentication required. As an example:

```
plural cd repositories create --url https://github.com/helm/examples.git
```

You should then see your repository show up when calling `plural cd repositories list`.
Loading
Loading