Upgrading the Interchain Stack

Our Cosmos layer depends on the "Interchain Stack" - a set of packages developed by the Cosmos community, must notable the CometBFT consensus engine, the Cosmos-SDK, and the ibc-go module. We maintain forks of some repositories, but many are used as-is, and we hope to reduce the number of forks we use over time.

Once a dependency is updated, we need to update agoric-sdk to use it. This involves several steps that we don't face in normal development, especially the maintenance of protocol buffer definitions and generated code.

• Preparation
  • Plan the target versions of the various components. Use the latest minor version and patch version unless there is a compatibility issue. Note that for cometbft and cosmos-sdk which are still on a 0.x.y semantic version, the x functions as a major version number.
  • Review the upgrade/migration guides to estimate the work required. Give particular attention to breaking changes to external APIs, CLIs, RPCs, and associated formats.
    • Disseminate these to the larger team.
    • Plan necessary work to adapt Agoric's codebase to these breaking changes.
• CometBFT
  • Confirm that our fork is still needed. (We have just one substantial change that might get upstreamed some day.)
  • Log all actions for later inclusion in the PR description
  • Run make test on previous state of Agoric branch and for the unmodified release to be merged and record all failures. (Some tests are either broken or don't work in our environment.) Re-run failures at least once to check for simple flakes.
  • Create a work branch off of current Agoric and merge the target release into it.
  • Log all conflict resolutions.
  • Run make test, solve any regressions.
  • Create a draft PR to run CI tests. Solve or justify any regressions.
  • Don't merge or tag until we've further validated with cosmos-sdk unit tests.
• Cosmos-SDK
  • Confirm that our fork is still needed. (Will be true for a while.)
  • Run make test on previous state of Agoric branch and for the unmodified release to be merged and record all failures. (Some tests are either broken or don't work in our environment.) Re-run failures at least once to check for simple flakes.
  • Optional: history-less merge of Agoric changes into new release
    • Might be useful when difference between cosmos-sdk releases dwarfs the difference between Agoric branch and its base Cosmos release.
    • Use git diff to create a single-file context diff of last base Cosmos release vs Agoric branch (or last tag - should be tagged).
      • E.g. git diff v0.45.16...v0.45.16-alpha.agoric.3 > diff-0.45.16-alpha.agoric.3
    • (Possibly using a separate Git clone...) Create a working branch off the target release.
    • patch -p1 < context-diff-generated-above
      • specify new name of moved files when possible
      • keep a log of how each conflict is resolved and why
    • save in a "nohistory" branch
    • continue with "post-merge development steps" then come back and do a history-preserving merge
  • History-preserving merge
    • Create a merge of the current and next cosmos releases accepting only the newer release
      • In a working branch off of newer release, git merge -s ours <old-tag>
    • Tag this product and merge it into a working branch off the current tip of Agoric.
    • If you previously did a history-less merge, resolve conflicts in the same way, otherwise keep a log of how each conflict is resolved and why.
  • Post-merge development steps
    • set up go.mod to use local copy of cometbft
    • regenerate products (make proto-gen, go mod tidy)
    • get it to build (make build)
    • get unit tests passing (make test)
    • create a draft PR and get CI tests to pass
    • when CI tests pass, merge and tag cometbft and use permanent tag in go.mod
• Agoric-SDK
  • If there's a matching version of gaia, run upgradegaia.sh to pull in its changes.
  • Update .go files to import new major version.
    • Don't edit generated files. Protobuf generated files will be udpated below.
  • Update go.mod with new versions, if needed.
    • The go mod tidy below will automatically update dependencies which come from import paths.
    • But we need to manually edit some replace directives. Look at the cosmos-sdk and gaia go.mod files.
  • Run go mod tidy
  • Run go mod download
  • Run make proto-update-deps
  • Run make proto-gen
  • Run govulncheck
  • Commit
  • PR
    • add proto:expect-breakage tag if any protos are updated