Skip to content

Release Process

Jump to bottom
Nicholas openSUSE Software Engineer edited this page Oct 24, 2024 · 19 revisions

Responsibilities

The Release Team is currently responsible for the long-term maintenance of our primary chart repositories (such as rancher/charts and rancher/system charts).

Chart Maintenance

  1. (Existing charts only): Chart owners are responsible for the overall maintenance and PR reviews associated with the charts they own.

    • This includes PRs updating an existing chart and bringing it into the rancher/charts repository if they have a separate upstream repository.

  2. (New charts only): Chart owners are responsible for new charts that must be added to the rancher/charts repository and any upstream repository associated with the chart.

    • A review from a member of the Rancher Charts Team is required for the first version of a new chart only to ensure it complies with all the requirements (eg., the CRD template is correct, required annotations were added, images are mirrored, etc...).

Branch Maintenance

  1. The Release Team sets up new dev-v2.x and release-v2.x branches when a new Rancher minor version update is introduced and forward-port all charts.
  2. The Release Team is responsible for verifying if charts in previous release lines have been forward-ported before a chart release and forward-porting any charts that may need to be added.

Regular and Out-of-band (OOB) Chart Releases

  1. Except for release PRs, no PRs should be made to a release branch (e.g., release-v2.x). All work should be made to an existing dev branch (e.g., dev-v2.x) or a new staging branch in the Rancher organization made for independent work, such as in emergency or out-of-band releases (e.g., dev-v2.7-oob-longhorn, dev-v2.7.1, etc.).
  2. (OOB only) Any team requesting an OOB release for their chart is responsible for creating a GitHub issue stating the chart name, version, and Rancher release line they want to release and assigning it to the Release team. Once the chart has been released, the Release engineer who completed the OOB release is responsible for notifying the requester team of its completion and requesting that the issue be closed.

Assets Lifecycle

Starting on 17/May/2024 there is a new asset lifecycle in place.

We will keep only the relevant assets in the corresponding branch.

This means a cycle of always 3 active branches, with each branch always holding up to 2 previous versions before it.

In a nutshell:

  • release-v2.7 hold chart versions for Rancher: (2.5; 2.6; 2.7)
  • release-v2.8 hold chart versions for Rancher: (2.6; 2.7; 2.8)
  • release-v2.9 hold chart versions for Rancher: (2.7; 2.8; 2.9)
flowchart LR;
  subgraph BigBox[Assets Lifecycle]
    direction LR
    subgraph A[release-v2.7]
        direction LR
        A1[10.1.2] --- A2[100.0.0+up0.0.0] ---  A3[102.0.0+up0.0.0]
    end
    subgraph B[release-v2.8]
        direction LR
        B1[100.0.0+up0.0.0] --- B2[102.0.0+up0.0.0] --- B3[103.0.0+up0.0.0]
    end
    subgraph C[release-v2.9]
        direction LR
        C1[102.0.0+up0.0.0] --- C2[103.0.0+up0.0.0] --- C3[104.0.0+up0.0.0]
    end
    A2 -.-> B1
    A3 -.-> B2
    B2 -.-> C1
    B3 -.-> C2
  end
Loading

New Rancher Charts Release Process

Only for branch lines >= 2.9 (i.e., release-v2.9; release-v2.10; etc...)

Steps to release charts (Out of band Or Batch release)

        Charts
            Workspace:     /<path_to_your_cloned_charts_repository>/charts
            ChartsRepoURL: https://github.com/rancher/charts
            ChartsForkURL: <your_forked_repo_for_charts_URL>
            BranchLines:     [2.10 2.9 2.8]
  • Pre-requisites: charts repository
    • The chart(s) (is/are) already merged on the corresponding dev-v2.* branch
    • You have cloned the git repository of the charts repository locally, which is up-to-date with the remote changes.
      • i.e., `git fetch; git checkout <some_dev_branch>; git pull <some_dev_branch>;
    • Name the remotes in your local git repo to something like:
release-work/charts » git remote -v                   
nicholasSUSE    https://github.com/nicholasSUSE/charts (fetch)
nicholasSUSE    https://github.com/nicholasSUSE/charts (push)
upstream        https://github.com/rancher/charts (fetch)
upstream        https://github.com/rancher/charts (push)
  1. Fetch && Checkout the target release-v2.X branch.
  2. Create a new branch from it, usually using a standard like: (e.g., 1st-batch-release-2.X.Y, oob-release-monitoring, fp-from-dev-2X-release-2X).
  3. Get the list of available charts for releasing:
    • release list charts 2.X
  4. Update the local branch (The auto-completion will help a lot):
    • release update charts 2.9 <chart_name> <version>
    • Repeat this step until updating all charts to be released
  5. Create the Pull Request:
    • release push charts 2.9 --debug
    • Use the debug to inspect what you will push.
    • After the command execution, it will give you the Pull Request URL if successful.
  6. Edit the Pull Request:
  7. Final validations:
  8. Ask for a review. The reviewer must only approve after all jobs are complete.
  9. After having one approval,
    • ONLY AFTER ALL PREVIOUS JOBS ARE COMPLETE: If you proceed and a job fails later, you will have pushed a faulty image to the Prime registry!
    • add the regsync-ready label to the Pull Request.
    • The regsync GitHub action will start to sync the images to the Prime registry.
  10. Wait for all GitHub Actions jobs to be completed again.
  1. Merge! This will release the charts.
  2. Wait for the git mirror to synchronize and propagate the changes. (more info at section: X)
  3. Notify the company of any relevant QA tests.
  4. Forward-port any charts, if needed, to newer branch lines. see: #Forward-porting charts
  5. Update Confluence Table.
Steps needed on specific conditions
  1. Only for the latest active release branch
    • Wait for the completion of git-pages workflow
    • If it fails, there is a warning automation. (troubleshoot info on the notes for the step below)
  2. Only for Rancher Minor Version Updates (i.e,. rancher v2.X to a new rancher v2.X+1)
    • If you created a new branch line and it is the first time you have released it. (e.g., release-v2.11 for release cycle 2.11.0)
    • Update the feature charts https://github.com/rancher/charts/settings/pages to the new release branch. This is where the icons rancher/rancher are loaded from and the main index.yaml feed is loaded for helm CLI like helm repo add rancher-charts charts.rancher.io.
      • It shouldn't affect the rancher-tagged releases, as those are pinned to specific release branches.

Troubleshoot: Once you modify to switch the branch from the prior release branch (i.e., release-v2.6) to the new release branch (i.e., release-v2.7), you may need to file an empty commit PR like https://github.com/rancher/charts/pull/2220 to have Github Pages pick it up and deploy the latest index.yaml. Upon making these changes, ensure the current index matches the contents of charts.rancher.io/index.yaml in the release-v2.7 branch.

Legacy Rancher Charts Release Process

Only for branch lines < 2.9 (i.e., release-v2.8; release-v2.7; release-v2.6; etc...)

The steps below work under the assumption your rancher/charts fork is:

  • clean
  • up to date
  • the remote for rancher/charts is named upstream

Steps to release feature charts

(For releases in 2.8) Make sure that the generated-changes folder for the charts from dev-v2.7 is copied correctly to dev-v2.8 since Mapps has copied it from release-v2.7 and there might be some changes in dev-v2.7 which are yet to be released and are required in dev-v2.8 as well.

  1. Open a PR on GitHub from dev-v2.x to its respective release-v2.x branch and set the title to  [release-v2.x] Charts release for Rancher vX.X.X (Where vX.X.X corresponds to the Rancher version being released)
  2. Get sign-off from QA and Manager to proceed with the release validation
  3. Post a comment in the PR with the release validation checklist
  4. Run validation scripts and fill up the checklist (see Validation Scripts section below for instructions)
  5. Request a review from a rancher/charts committer (anyone who can review)
  6. The reviewer must approve only after all checks have been done and verified. This is because, once approval is given on the PR, the regsync Github Action will start the syncing of images to the Prime registry. Link
  7. After the GitHub Action is completed and the CI check is green, anyone who has permission to merge can merge the PR.
  8. Notify your manager and QA in the rancher release coordination channel (or any relevant channel) that the charts have been released for QA to start their post-charts release Apps & Marketplace validation
  9. Empty the release.yaml in the release-* branch and do the sync up.
  10. Complete post-release cleanup and sync (see section below for post-release cleanup steps)
  11. Forward-port any charts that need to be forward-ported to newer version lines (see the section below for chart forward-porting steps)
  12. Change the branch references in rancher/rancher from dev to release like this PR https://github.com/rancher/rancher/pull/42952
  13. Update the latest versions into the table and also any new net charts and their latest versions in the Matrix table like in the Apps & Marketplace  section of Rancher Manager v2.7.6
  14. Update dates of the Mapps responsible tasks in the release Checklist page of confluence like v2.7.7 Release Coordination Master Page Only for Minor Versions:
  15. Update the feature charts https://github.com/rancher/charts/settings/pages to the new release branch. This is where the icons rancher/rancher are loaded from and the main index.yaml feed is loaded for helm CLI like helm repo add rancher-charts charts.rancher.io. It shouldn't affect the actual rancher-tagged releases as those are pinned to specific release branches.

Note: Once you modify to switch the branch from the prior release branch (i.e. release-v2.6) to the new release branch (i.e. release-v2.7), you may need to file an empty commit PR like https://github.com/rancher/charts/pull/2220 to have Github Pages pick it up and deploy the latest index. yaml. On making these changes, ensure the contents of charts.rancher.io/index.yaml matches the current index.yaml in the release-v2.7 branch. 

Steps to release feature charts in batches

  1. Create a branch off the release-v2.x branch from your forked repository
  2. Cherry-pick the commits from the dev-v2.x branch associated with the chart(s) that need to be released (this should include changes made to the packages, charts, and assets directories)
  3. We need to push our branch in the forked repository to rancher/charts the repository. This is required since we need to create a PR from a branch belonging to rancher/charts for the regsync Github Action to run. 
  4. Get sign-off from QA and Manager to proceed with the release validation
  5. Add the release validation checklist to the PR description
  6. Run validation scripts and fill up the checklist (see Validation Scripts section below for instructions)
  7. Request a review from a rancher/charts committer (anyone who is able to review)
  8. The reviewer must approve only after all checks have been done and verified. This is because once approval is given on the PR, the regsync Github Action will start the syncing of images to the Prime registry. Link
  9. After the GitHub Action is completed and the CI check is green, the PR can be merged by anyone who has permission to merge.
  10. Repeat steps 1 through 10 until all batches have been merged.
  11. Notify your manager and QA in the rancher release coordination the charts have been released in order for QA to start their post charts release Apps & Marketplace validation
  12. Empty the release.yaml in the release-* branch and do the sync up.
  13. Complete post-release cleanup and sync (see section below for post-release cleanup steps)
  14. Forward-port any charts that need to be forward-ported to newer version lines (see section below for chart forward-porting steps)
  15. Change the branch references in rancher/rancher from dev to release like this PR https://github.com/rancher/rancher/pull/42952
  16. Update latest versions into the table and also any new net charts and their latest versions in the Matrix table like in Apps & Marketplace  section of Rancher Manager v2.7.6
  17. Update dates of the Mapps responsible tasks in the release Checklist page of confluence like v2.7.7 Release Coordination Master Page
  18. Only for Minor Versions: Update the feature charts https://github.com/rancher/charts/settings/pages to the new release branch. This is where the icons rancher/rancher are loaded from and the main index.yaml feed is loaded for helm CLI like helm repo add rancher-charts charts.rancher.io. It shouldn't affect the actual rancher-tagged releases as those are pinned to specific release branches.

Note: Once you modify to switch the branch from the prior release branch (i.e. release-v2.6) to the new release branch (i.e. release-v2.7), you may need to file an empty commit PR like https://github.com/rancher/charts/pull/2220 to have Github Pages pick it up and deploy the latest index. yaml. On making these changes, ensure the contents of charts.rancher.io/index.yaml matches the current index.yaml in the release-v2.7 branch. 

Release feature charts Out-of-Band (OOB)

  1. (For releases in 2.8) Make sure that the generated-changes folder for the charts from dev-v2.7 is copied correctly to dev-v2.8 since Mapps has copied it from release-v2.7 and there might some changes in dev-v2.7 which are yet to be released and are required in dev-v2.8 as well.
  2. Make sure to get sign off from the feature chart team QA or owner before starting the release.
  3. Create a branch off the release-v2.x branch from your forked repository
  4. Cherry pick the commits from the dev-v2.x branch associated with the chart(s) that need to be released OOB (this should include changes made to the packages, charts, and assets directories)
  5. We need to push our branch in the forked repository directly to the rancher/charts  repository and then create a Pull Request from the pushed branch to the production branch. This is required since we need to create a PR from a branch belonging to rancher/charts  for the regsync Github Action to run. 
  6. Make sure your upstream points to rancher/charts  and then execute this command git push --set-upstream upstream oob-july . Here oob-july  is the branch your created in step 1.  
  7. Open a PR on GitHub to the release-v2.x branch and set the title to[release-v2.x] OOB Charts release for  and add the issue(s) associated with the OOB release to the description of the PR.
  8. Wait for the CI checks to pass and give the thumb's up to the bot's validation comment.
  9. Run validation scripts and fill up the checklist (see section below for release validation steps)
  10. Request a review from a rancher/charts committer (anyone who is able to review).
  11. After approving the PR, add the "regsync-ready"label to it. This will trigger the prime registry image sync action. Link
  12. After the GitHub Action is completed and the CI check is green, the PR can be merged by anyone who has permission to merge.
  13. Notify the authors of the issues associated with the OOB release that the chart(s) have been released and request for them to close the issue
  14. After the PR is approved and merged, do the sync-up.
  15. Complete post release clean-up and sync-up (see section below: clean-up and sync-up branches post chart release)
  16. Forward-port any charts that need to be forward-ported to newer version lines (see section below for chart forward-porting steps) If the versioning scheme changes, i.e from 2.x.x to 102.x.x, do not forget to notify the respective teams
  17. Update release confluence page.

Clean-up and sync-up branches post chart release

  1. Checkout the corresponding dev branch in your fork.
  2. Remove the charts released from the release.yaml
  3. Merge the upstream release-v2.x branch onto the dev-v2.x branch to bring the merge commit from the release and any other commits back into dev and sync up the branches (the CI validation will fail for new PRs made to dev if this step is not done)
  4. If any of the released charts are not in the dev branch: Run make validate to regenerate release.yaml. The validation is expected to fail and generate a new release.yaml if there are any new charts in dev-v2.x; commit the changes made to release.yaml and the next make validate run should pass. If it fails before a new release.yaml is generated, it is likely that something went wrong during the merge of release into dev.
  5. Open the PR with the changes to the upstream dev-v* branch
  6. After getting the approvals, merge it without the squash option.
    • Important: Do not Squash and Commit
    • Do: Create a merge commit
    • is the only way to create the merge commit, so that the release-v* branch is in sync with its respective dev-v* branch. You will enable under your GH Repo settings the option: Allow merge commits  Execute the merge and disable again the option.

Forward-porting charts

Any charts released in an older version line need to be forward-ported to newer version lines. E.g a chart released in 2.5 needs to be forward-ported to 2.6 and 2.7, and a chart released in 2.6 needs to be forward-ported to 2.7 and so on.

  1. Checkout the dev-v2.x branch where x corresponds to the branch where you want to forward-port a chart into.
  2. Run the following command: make forward-port CHART=rancher-chart VERSION=100.0.0+up1.2.3 BRANCH=dev-v2.x-1 UPSTREAM=upstream 
  3. Commit the output of the command with a comment "make forward-port "
  4. Repeat for all charts that need to be forward-ported including CRD charts as these are done separately
  5. Open a pull request to the corresponding dev-v2.x branch with the forward port commits and ask for a review.
  6. Once the pull request is approved, merge it.

Release validation and validation scripts

Validation scripts are located in the rancher/charts repository in scripts/release-validation here https://github.com/rancher/charts/tree/dev-v2.7/scripts/release-validation

Currently they need to be ran inside the rancher-validation directory for them to work properly, and it is best to direct the output of the scripts to a file for review (except the diff-with-released script since it creates a directory for you)

  1. Run make unzip to regenerate the charts/ from scratch, then git diff to check differences between assets/ and charts/ yields NO differences or innocuous differences (addresses Checkpoint 1 in the pr-template-checklist, and it is best done first as the repository should be in a clean state)
  2. Run the last-released script (./last-released.sh > last-released.log) to address item in the Checkpoint 0 of the pr-template-checklist.
  3. Run the validate-annotations (./validate-annotations.sh > validate-annotations.log) and validate-index (./validate-index.sh > validate-index.log) to address Checkpoint 2 of the pr-template-checklist

System Charts Release

Repository: https://github.com/rancher/system-charts/

Releasing system charts is simpler that other repositories, because it does not use rancher/charts-build-scripts to build, maintain, and release charts (although it follows the same dev-v2.x and release-v2.x branch strategy as other Rancher repositories).

Steps to release system charts

  1. Open a PR on GitHub from dev-v2.x to its respective release-v2.x branch and set the title to [release-v2.x] System Charts release for Rancher vX.X.X (where vX.X.X corresponds to the Rancher version being released)
  2. Validate the following:
    • New images are mirrored
    • New chart versions are exactly one version up over the previous version and none of the versions that has already been released were modified
    • The questions.yaml in the charts to be released have rancher_min_version and rancher_max_version set correctly.

      Note: Generally we release one new chart version or each minor version line of the chart for each Rancher version line and hide the previous version so that only one chart is available per Rancher version. All charts should have a rancher_min_version and rancher_max_version set, except for the latest version in a version line (i.e. If you have a chart with versions 0.1.0 0.1.1, 0.2.0, and 0.2.1 only versions 0.1.1 and 0.2.1 should not have a rancher_max_version set). Examples:

  3. Get sign off from QA to proceed with the release
  4. Merge PR (this step releases the charts)
Clone this wiki locally