prow

Prow deployment

This directory contains the manifests used for the deployment of the Prow cluster.

Upgrading Prow

The Kubernetes Prow deployment is automatically deployed, but all the other projects like Knative, Istio, and cert-manager do the deployment manually.

Here is the process to upgrade Prow:

1. ⚠️ You must be given the role roles/container.developer on the jetstack-build-infra project. You must be able to run kubectl commands on the github-build-infra cluster.

2. Clone this repo:

   git clone https://github.com/jetstack/testing
   cd testing

3. Pick a build of Prow by running:

   % gcloud container images list-tags gcr.io/k8s-prow/deck | head
   DIGEST       TAGS                                     TIMESTAMP
   96dba717b1f3 latest,latest-root,v20210412-ed35ec0cee  2021-04-12T16:17:11
   255fe5a57fb4 v20210412-176e4b678c                     2021-04-12T15:39:17
   53107953d93e v20210412-f0c722e283                     2021-04-12T14:59:15
   f2eca760c0f9 v20210410-57fae234ba                     2021-04-10T02:55:02

   For example, let us pick the latest one. What we call the "target commit" in the next steps is the commit hash that appears in the image tag:

   v20210412-ed35ec0cee
   #         <-------->
   #        target commit

   In this example, ed35ec0cee is the target commit to which you will be upgrading to (Prow does not have "releases").

4. Find out what is the "current commit" of the current deployment of Prow. This is stored in the file prow/version. For example:

   % cat prow/version
   v20200628-cc1c099dad
   #         <-------->
   #        current commit

   At this point, you know that:

   	image tag	commit
   current version	v20200628-cc1c099dad	cc1c099dad
   target version	v20210412-ed35ec0cee	ed35ec0cee

5. Open ANNOUNCEMENTS.md and look for anything that changed between the current commit and the target commit.

6. Update the file prow/version with your target image tag, and open a PR to jetstack/infra. For example:

   diff --git a/prow/version b/prow/version
   --- a/prow/version
   +++ b/prow/version
   @@ -1 +1 @@
   -v20200628-cc1c099dad
   +v20210412-ed35ec0cee

7. Get the PR merged. Merging the PR will not do anything, we do not do rolling deployments.

8. Pull the latest changes from master. From now on, you must be on the master branch:

   git checkout master
   git pull origin master

9. Make sure you have a context in your KUBECONFIG that is called build-infra (this context name is defined in print-workspace-status.sh. Create the build-infra context with:

   gcloud auth login
   gcloud container clusters get-credentials --project jetstack-build-infra --region europe-west1-b github-build-infra
   kubectl config rename-context gke_jetstack-build-infra_europe-west1-b_github-build-infra build-infra

10. Generate and apply the Prow manifests to the github-build-infra cluster:

    bazel run //prow/cluster:production.apply

Creating new Prowjobs

See documentation for ProwJobs in k/test-infra.

Testing locally

ProwJobs can be tested locally by running the (interactive) ./prow/pj-on-kind.sh script. This script will spin up a local KIND cluster and create a new ProwJob instance for which there will be a Pod created that will be running the actual test.

See documentation in k/test-infra for how the script works.

An example of running pull-cert-manager-upgrade-v1-21 job locally:

1. Remove Bazel presets from job config, so it doesn't look for Bazel cache creds
2. Run ./prow/pj-on-kind.sh pull-cert-manager-upgrade-v1-21
3. Pass some cert-manager PR number when requested. This will be checked out.
4. Pass 'empty' for any storage volumes when requested.
5. Retrieve kubeconfig for the kind cluster kind get kubeconfig --name mkpod and set KUBECONFIG
6. kubectl get pods - to get the name of the pod that is running the test
7. kubectl logs <pod-name> -c test -f stream the logs