Feature Request: Implement Versioned JSON-RPC API endpoints

[davecgh]

Currently, the JSON-RPC server is versioned via semver and exposed via the version JSON-RPC method. While this approach works in terms of allowing callers to detect changes, it often leads to compatibility issues and headaches during transition periods while development continues between release versions.

For example, consider when the latest release of dcrd has a major JSON-RPC server version of 6, but then during development leading up the next release, it is bumped to 7. At that point, any consumers, such as dcrwallet, dcrdata, dcrdex, etc, can no longer be used with the current development version of dcrd due to the API incompatibilities. While it is, of course, expected that consumers will necessarily need to be updated for any API changes, the hard switch without any type of deprecation period is not ideal.

I believe a good solution to this is to introduce versioned endpoints to the JSON-RPC API instead of using a single semver for the server. For example, the websocket endpoint might be: wss://127.0.0.1:9109/v2/ws, wss://127.0.0.1:9109/v3/ws, etc. Then, whenever a breaking change to the API is introduced, it is done so under a new endpoint.

In order to avoid having to maintain old endpoints forever, whenever a new endpoint is introduced, the old version should be deprecated but still made available for at least one additional full release cycle of the software. This would mean JSON-RPC endpoints follow the standard approach of supporting the most recent version and the version prior to it and thus ensures consumers have ample time to seamlessly transition to newer endpoints before the older endpoint is removed.

It should be noted that this will require a bit more work when introducing breaking API changes, but it will provide a much more stable API for consumers which I think is important given consumers are really the entire reason for having an RPC API to begin with.

The two main approaches I think would be reasonable are:

• Provide a single versioned endpoint for the entire API; or
• Provide an individual versioned endpoint for each method

Both of these approaches have pros and cons and should be analyzed further in terms cost vs benefit prior to making any concrete decisions. Nevertheless, I wanted to get this documented and get the ball rolling so that it can hopefully land in the dcrd release after the upcoming 1.7.0 release.

[dnldd]

Would like to be assigned to this, should start researching on the the design soon.

[dnldd]

This details the possible design approaches outlined for versioning dcrd's RPC endpoints.

Option One: A single versioned endpoint for the entire API.

Providing a single version endpoint for the entire json-rpc API would mean having at least three handlers: the previous versioned endpoint (...\v1), the explicit current endpoint (...\v2) and the default endpoint which would default to the functionality provided by the current endpoint (...\). With this schema, registered API calls will identify the endpoint they can be reached on, this will be used to resolve whether a caller can access the requested functionality based on the endpoint being called from.

With this versioned approach, consumers of the API can avoid the state of flux the API goes through when a new version is being finalized by explicitly specifying the API version being accessed. This way when the API version specified becomes the previously versioned endpoint there should not be any interruption due to the explicitly specified version still being accessible.

New API calls introduced or bug fixes to existing calls would trigger bumping the API version. The new version will then be in flux until the next dcrd release for it to be finalized. During this period, for existing calls undergoing bug fixes there will be two versions: the existing version and the updated version. These calls will be differentiated by the version appended to their names: eg. handleVerifyMessageV2 and handleVerifyMessageV3, this would also reflect in the associated RPC tests for the call as well as the RPC documentation.

Before API changes:

• V1 - stable, previous version, accessible on /v1
• V2 - stable, current version, accessible on /v2 and /

During API changes:

• V1 - stable, no longer accessible on /v1
• V2 - stable, previous version, accessible on /v2
• V3 - in flux, current version, accessible on /v3 and /

After release:

• V1 - stable, no longer accessible on /v1
• V2 - stable, previous version, accessible on /v2
• V3 - stable, current version, accessible on /v3 and /

Disadvantages.

In the example described above, explicit subscribers of the V1 endpoint will no longer have access to the endpoint after the new release. A workaround would be to still maintain the v1 endpoint and return a deprecated error prompting callers to switch to v2 or v3 endpoints. This will be in place until the next API version update.

Advantages.

This approach to versioning the json-rpc endpoint would most likely be the simplest to implement considering the current design of the RPC server. It should fit with the existing design without changing a lot. Consumers of the
API can also be oblivious to the versioning changes made since / defaults to the current API version. They would only have to specify the version in the API when looking to use specific functionality with a different implementation in the previous API version.

Option Two: An individual versioned endpoint for each method.

Providing an individual versioned endpoint for each RPC method would mean at least a dedicated handler for each method. For example handleVerifyMessage would have /verifymessage followed by the version being called or just
/verifymessage which would default to the current RPC version.

There are over 75 RPC methods currently, this approach will require the same number of handler as a result, in the case of different implementations or updated implementations for the same method we would have two handlers for a method. This could get costly as more methods are added to the RPC with regards to maintenance particularly.

Before API changes:

• handleVerifyMessageV1 - stable, previous version, accessible on /verifymessage/v1
• handleVerifyMessageV2 - stable, current version, accessible on /verifymessage/v2 and /verifymessage

During API changes:

• handleVerifyMessageV1 - stable, no longer accessible on /verifymessage/v1
• handleVerifyMessageV2 - stable, previous version, accessible on /verifymessage/v2
• handleVerifyMessageV3 - in flux, current version, accessible on /verifymessage/v3 and /verifymessage

After release:

• handleVerifyMessageV1 - stable, no longer accessible on /verifymessage/v1
• handleVerifyMessageV2 - stable, previous version, accessible on /verifymessage/v2
• handleVerifyMessageV3 - stable, current version, accessible on /verifymessage/v3 and /verifymessage

Disadvantages.

In the example described above, explicit subscribers of the /verifymessage/v1 endpoint will no longer have access to the endpoint after the new release. As stated for the previous implementation option, a workaround
would be to still maintain the /verifymessage/v1 endpoint and return an error prompting callers to switch to /verifymessage/v2 or /verifymessage/v3 endpoints.

The number of handlers needed to provide versioned endpoints for each RPC grows with each added method. Comparing the approaches outlined, this would be the more difficult to implement considering the work that has been done to resolve incoming messages on a general endpoint would have to be scraped in favour of resolving specific messages on specific endpoints instead.

Callers would also need to update to make the same calls they used to because of the change in endpoint schemas. Might be tedious for some callers to specify the method being targetted for each API call.

Advantages.

The updated endpoint schema explicitly details the targeted method and the version if needed. This improves readability for the caller since the context for the call being made is now part of the endpoint schema.

[dabuchera]

Running exactly into this now. Dcrd running on Master, dcrwallet on latest.
Wallet synchronization stopped: rpcsyncer.Run: advertised API version 7.0.0 incompatible with required version 6.0.0
Any workaround on this?

[chappjc]

Only workaround is to run dcrwallet from master (preferably a release tag e.g. https://github.com/decred/dcrwallet/releases/tag/release-v1.7.0-rc2).
But, the 1.7 release is around the corner and the second release candidate is available now, so perhaps that's your best bet.

[dnldd]

After further discussions in chat with davec, versioning JSON-RPC endpoints individually was the best of the two approaches suggested. #2901 when done will outline the endpoint versioning process.