Skip to content

Commit

Permalink
ci: release and publishing automation (#786)
Browse files Browse the repository at this point in the history
* Update CONTRIBUTING.md

Co-authored-by: Kyle Peacock <[email protected]>

* run npm audit before publishing
  • Loading branch information
Marcin Nowak-Liebiediew authored Nov 1, 2023
1 parent 6d45a41 commit bb665ce
Show file tree
Hide file tree
Showing 17 changed files with 3,719 additions and 520 deletions.
47 changes: 47 additions & 0 deletions .github/workflows/prepare-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Prepare Release

on:
workflow_dispatch:
inputs:
semverBump:
description: 'Specify next SemVer version (either of: "patch", "minor", "major", "prepatch", "preminor", "premajor", or custom SemVer compatible version (e.g. "0.32.1-beta.1", or "1.0.0"))'
type: string
required: true
default: 'patch'

jobs:
create-release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Cache node modules
id: cache-npm
uses: actions/cache@v3
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- if: ${{ steps.cache-npm.outputs.cache-hit != 'true' }}
name: List the state of node modules
continue-on-error: true
run: npm list
- name: Install node dependencies
run: npm ci
- name: Set up git config
run: |
git config author.email "${{ github.event.sender.id }}+${{ github.event.sender.login }}@users.noreply.github.com"
git config author.name "${{ github.event.sender.login }}"
git config committer.email "41898282+github-actions[bot]@users.noreply.github.com"
git config committer.name "GitHub Actions Bot"
git config user.email "${{ github.event.sender.id }}+${{ github.event.sender.login }}@users.noreply.github.com"
git config user.name "${{ github.event.sender.login }}"
- name: Create release Pull request & GitHub Release
env:
GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}'
run: npm run release ${{ inputs.semverBump }}
72 changes: 72 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
name: Publish and Release

on:
pull_request:
branches:
- main
types:
- closed

jobs:
publish:
if: github.event.pull_request.merged == true && startsWith(github.event.pull_request.head.ref, 'release/')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Extract version from branch name
run: |
BRANCH="${{ github.event.pull_request.head.ref }}"
VERSION="${BRANCH#release/}"
echo "BRANCH=$BRANCH" >> $GITHUB_ENV
echo "VERSION_TAG=v$VERSION" >> $GITHUB_ENV
- name: Mark as Latest Release in GitHub Releases
env:
GH_TOKEN: '${{ secrets.GITHUB_TOKEN }}'
run: |
gh release edit "${{ env.VERSION_TAG }}" --draft=false --prerelease=false --latest=true
- uses: actions/setup-node@v3
with:
node-version: '20.x'
registry-url: 'https://registry.npmjs.org'
- name: Cache node modules
id: cache-npm
uses: actions/cache@v3
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- if: ${{ steps.cache-npm.outputs.cache-hit != 'true' }}
name: List the state of node modules
continue-on-error: true
run: npm list
- run: npm ci
- run: npm audit
- run: npm publish
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

# publish docs
- name: Install dfx
run: dfinity/setup-dfx@main
- name: Regenerate project's documentation
run: npm run make:docs
- name: Add new identity to dfx
run: |
echo ${{ secrets.DFX_IDENTITY_PEM }} > identity.pem
dfx identity import docs-deployer identity.pem
dfx identity use docs-deployer
- name: Deploy docs
run: dfx deploy --network ic

- name: Delete release branch
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: git push origin --delete ${{ env.BRANCH }}
217 changes: 213 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# Contributing to this repository

## Getting Started
# Contribution guidelines

Directions to set up your project are available in the [README](./README.md).

Expand Down Expand Up @@ -34,10 +32,221 @@ This will automatically link your commit to the GitHub issue, and automatically

Please document your changes in the [changelog.html](./docs/generated/changelog.html) file.

### Continuous Integration (CI)
## Formatting

To save time on formatting, we use automated formatting for this repo using prettier. You can either use git pre-commit hooks or run the command `npm exec prettier:format` before submitting your PR to have your changes pass. We check formatting on CI.

We use GitHub Actions for Continuous Integration. As part of this, we have a workflow for releasing the package. This workflow is triggered manually and includes steps for checking out the code, setting up Node.js, installing dependencies, and running the release script.
To trigger the release workflow, go to the "Actions" tab in the GitHub repository, select the "Release" workflow, and click on "Run workflow".

## Continuous Integration (CI)

Changes will have to pass automated tests before they can be merged. If your changes fail the tests, you will have to address the failures and re-run the tests.

We use GitHub Actions for Continuous Integration. As part of this, we have a workflow for releasing the package. This workflow is triggered manually and includes steps for checking out the code, setting up Node.js, installing dependencies, and running the release script.
To trigger the release workflow, go to the "Actions" tab in the GitHub repository, select the "Release" workflow, and click on "Run workflow".

GitHub Actions for this repo are configured in [./.github/workflows](./.github/workflows).

- [conventional-commits.yml](./.github/workflows/.yml) - checks the title of pull requests to ensure they follow a specified format. It is triggered when a pull request is opened, reopened, edited, or synchronized.
- [e2e-tests.yml](./.github/workflows/.yml) - runs end-to-end tests for the project. It is triggered when a pull request is opened, reopened, edited, or synchronized.
- [lint.yml](./.github/workflows/.yml) - checks the code for linting errors. It is triggered when a pull request is opened, reopened, edited, or synchronized.
- [mitm.yml](./.github/workflows/.yml) - sets up a Man-in-the-Middle (MITM) proxy for testing purposes. It is triggered when a pull request is opened, reopened, edited, or synchronized.
- [prepare-release.yml](./.github/workflows/prepare-release.yml) - prepares a release by creating a pull request and a GitHub release. It is triggered manually through a workflow dispatch event. The user needs to specify the next SemVer version as an input when triggering the workflow.
- [prettier.yml](./.github/workflows/.yml) - checks the formatting of the code using Prettier. It is triggered when a pull request is opened, reopened, edited, or synchronized.
- [publish.yml](./.github/workflows/.yml) - used to publish and release a new version of the project. It is triggered when a pull request is closed and merged into the main branch, and the pull request's head branch starts with `release/`.
- [size-limit.yml](./.github/workflows/.yml) - uses the `andresz1/size-limit-action` action to calculate the size of the project.
- [unit-tests.yml](./.github/workflows/.yml) - runs unit tests for the project. It is triggered when a pull request is opened, reopened, edited, or synchronized.

## Reviewing

A member of the team will review your changes. Once the member has reviewed your changes, they will comment on your pull request. If the member has any questions, they will add a comment to your pull request. If the member is happy with your changes, they will merge your pull request.

## Main Branch Conventions

All commits in the master branch should come from squashed GitHub Pull Requests, and those commit messages should follow the [conventionalcommits.org](https://conventionalcommits.org) syntax.

# Release new version and Publish it to NPM

> [!IMPORTANT]
> The 5-line script shared below should give you an overview of what you need to do to execute the process, however, it is **not recommended** to actually run it this way, because the script will approve and merge the PR - something you should do manually!
>
> ```
> gh workflow run "prepare-release.yml" -f "semverBump=major" && sleep 3
> RUN_ID=$(gh run list --workflow=prepare-release.yml --status in_progress --json databaseId --jq '.[0].databaseId')
> gh run watch $RUN_ID
> gh pr review $(gh pr list --json number --jq '.[0].number') --approve
> gh pr merge $(gh pr list --json number --jq '.[0].number')
> # ... and you're done with releasing and publishing!
> ```
## Release process
We utilize the [release-it](https://github.com/release-it/release-it) package to streamline our release process.
Start the process by initiating the GitHub Action Workflow `prepare-release.yml`. This can be done by:
- Navigating to the GitHub web UI and clicking "Run workflow" at https://github.com/smallstepman/agent-js/actions/workflows/prepare-release.yml, or
- Running this command from your console:
```console
gh workflow run "prepare-release.yml" -f "semverBump=major"
```
You can set `semverBump=...` to `prepatch`, `patch`, `preminor`, `minor`, `premajor`, `major`, or any valid SemVer like `0.31.3-beta.2`, `0.32.2-abcdef`, or `0.32.0`. View your workflow progress in the console with this command:
```console
gh run watch $(gh run list --workflow=prepare-release.yml --status in_progress --json databaseId --jq '.[0].databaseId')`.
```

<details>
<summary>
How does it work?
</summary>

The `prepare-release.yml` GitHub Actions workflow checks out the code, sets up Node.js, installs dependencies, and runs the release script via `release-it`. The process, defined in our `package.json` file, includes the following tasks:

- version bump,
- roll version in CHANGELOG.md,
- new release branch creation,
- git tag creation and push,
- suitable GitHub Release summary creation based on commit history,
- and release PR opening.

Looking at the process from perspective of git log, here is how it would look like

```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'default' , 'themeVariables': {
'gitInv0': '#ff0000'
} } }%%
gitGraph:
commit id: "PR #3458"
commit id: "PR #3451"
commit id: "PR #3454" type: HIGHLIGHT
branch release/0.20.0
checkout release/0.20.0
commit id: "Release 0.20.0" tag: "v0.20.0"
checkout main
merge release/0.20.0 id: "chore: release 0.20.0"
commit id: "PR #3453"
commit id: "PR #3460"
commit id: "PR #3455" type: HIGHLIGHT
branch release/0.21.0
checkout release/0.21.0
commit id: "Release 0.21.0" tag: "v0.21.0"
checkout main
merge release/0.21.0 id: "chore: release 0.21.0"
commit id: "etc, etc"
```

The commits with red square icon, indicate the moment when release process was triggered (either by using GitHub Actionr or by running `npm run release`)

</details>

<details>
<summary>
Can I trigger the release process without using GitHub Action?
</summary>

Yes, you can manually initiate the process. To do this, you must first install the GitHub CLI binary on your system and authenticate using `gh auth login`. After login, you can trigger the process using:

```console
GITHUB_TOKEN="$(gh auth token)" npm run release patch # or minor/major/etc
```

</details>

<details>
<summary>
How can I manually perform everything (bypassing `release-it`)?
</summary>

You can execute the following commands:

```console
# Ensure you are on main branch, and there are no uncommited files
VERSION="0.20.0"
ts-node bin/version.ts $VERSION
ts-node bin/roll-changelog.ts $VERSION

git pull
git checkout release/$VERSION 2>/dev/null || git checkout -b release/$VERSION
git merge main
git push --set-upstream origin release/$VERSION

git add .
git commit -m "chore: release 0.20.0"
git tag "v0.20.0"
git push --force-with-lease

RELEASE_URL=$(gh release create v0.20.0 --generate-notes)
gh pr create --base main --title 'chore: release $VERSION' --body 'GitHub Release: $RELEASE_URL\nNPM release: https://www.npmjs.com/package/@dfinity/agent/v/${version}'"
git checkout main
```

</details>

## Publishing to NPM and Document Updates

Once you've initiated a release process, the resulting pull request from the `release/...` branch to the `main` branch needs to be reviewed. Upon merging, it automatically triggers the `publish.yml` workflow, which handles publishing the new version to NPM, along with updating documentation and changelog.

<details>
<summary>
How to manually publish to NPM (without utilizing `publish.yml` workflow)?
</summary>

Perform the following steps to manually publish a package to NPM:

1. Create a branch and execute these commands:
- `git clean -dfx`. This removes all non-tracked files and directories.
- `npm install`. This ensures everything is installed and up-to-date locally.
- `npm run build --workspaces`. This builds all applications and packages.
- `npm run version [patch|major|minor|version]`. This updates the version in each package.
- Manually update the version in the root package.json file.
- `npm install`. This updates the packages' versions in the package-lock.json file.
2. Initiate a new release branch using `git checkout -b release/v<#.#.#>`.
3. Stage your changes with `git add .`.
4. Create a commit including your changes using `git commit -m 'chore: release v<#.#.#>'`.
5. Open a pull request from your fork of the repository.

Once the changes are merged, you can publish to NPM by running:

- `npm run build --workspaces`. Re-building for safety.
- `npm publish --workspaces`. To publish packages to NPM.
- To do this, you will need publishing authorization under our NPM organization. Contact IT if you require access.
- You can include the `--dry-run` flag to verify the versions and packages before actual publishing.

After publishing to NPM, go to https://github.com/dfinity/agent-js/releases/new, select "Draft a new release", enter the new tag version (in `v#.#.#` format), and click "Publish release".

</details>

<details>
<summary>
How to manually publish a new version of the documents (without utilizing `publish.yml` workflow)?
</summary>

1. Start with a fresh clone (or execute `git clean -dfx .`)
2. Run `npm install`
3. Execute `npm run make:docs`
4. Deploy the docs using `dfx deploy --network ic`. Note: You may need to request for permissions as a controller for the wallet that owns the docs.

</details>

# Deprecation

To deprecate a package, follow these steps

- Remove all contents except the package.json, license, and readme
- Add a note to the README saying `**Warning** this package is deprecated`
- Remove unnecessary content, dependencies, and metadata from the package.json
- add a `"deprecation"` tag to the package.json with instructions you want users to follow in migrating
- remove the package as a workspace from the root `package.json`
- the next time that agent-js releases, manually publish a new version of newly deprecated packages by incrementing the patch version and running `npm publish`

So far, the following packages were deprecated:

- @dfinity/ledger-identityhq (#665)
- @dfinity/authentication (#665 & #661)
Loading

0 comments on commit bb665ce

Please sign in to comment.