If you want more functionality from your Prow instance this guide is for you. It primarily links to other resources that catalogue existing components and features.
Prow has a number of optional cluster components and a suite of plugins for hook that provide all sorts of automation. Check out the README in the prow/cmd directory for a list of cluster components and the README in the prow/plugins directory for information about available plugins.
Some Prow components expose prometheus metrics that can be used for monitoring, alerting, and pretty graphs. You can find details in the README in the prow/metrics directory.
You can easily make your Prow instance automatically update itself when changes
are made to its component's kubernetes resource files. This is achieved with a
postsubmit job that kubectl applys the resource files whenever they are
changed (based on a run_if_changed regexp). In order to kubectl apply to the
cluster, the job will need to supply credentials (e.g. a kubeconfig file or
GCP service account key-file). Since
this job requires priviledged credentials to deploy to the cluster, it is
important that it is run in a separate build cluster that is isolated from all
presubmit jobs. See the
documentation about separate build clusters
for details. An example of such a job can be found
here.
Once you have a postsubmit deploy job, any changes to Prow component files are
automatically applied to the cluster when the changes merge. In order to ensure
that all changes to production are properly approved, you can use OWNERS files
with the approve plugin and Tide.
With the help of the Prow Autobump utility you can easily create commits that update all references to Prow images to the latest image version that has been vetted by the https://prow.k8s.io instance. If your Prow component resource files live in GitHub, this utility can even automatically create/update a Pull Request that includes these changes. This works great when run as a periodic job since it will maintain a single open PR that is periodically updated to reference the most recent upstream version. See the README for details and an example.
Combining a postsubmit deploy job with a periodic job that runs the Prow Autobump utility allows Prow to be updated to the latest version by simply merging the automatically created Pull Request (or letting Tide merge it after it has been approved).
Prow can also automatically upload changes to files that correspond to Kubernetes ConfigMaps. This includes its own config, plugins and job-config config maps. Take a look at the updateconfig plugin and config-bootstrapper for more details. Both of these tools share the updateconfig plugin's plugin configuration. The plugin provides slightly better GitHub integration and is simpler to enable, but the config-bootstrapper is more flexible. It can be run in a postsubmit job to provide config upload on non-GitHub Prow instances or run after custom config generation is executed.
- If you find that your GitHub bot is running low on API tokens consider using
ghproxyto cache requests to GitHub and take advantage of the strange re-validation rules that allow for additional API token savings. - Testgrid provides a highly configurable visual overview of test results and can be configured to send alerts for failing or stale results. Testgrid is in the process of being open sourced, but until it has completely made the switch OSS users will need to use the https://testgrid.k8s.io instance that is managed by the GKE-Engprod team.
- Kind lets you run an entire Kubernetes cluster in a container. This makes it fast and easy for ProwJobs to test anything that runs on Kubernetes (or Kubernetes itself).
- label_sync maintains GitHub labels across orgs and repos based on yaml configuration.
If your Prow instance operates on a lot of GitHub repos or runs lots of jobs you should review the "Scaling Prow" guide for tips and best practices.