system_accountNextIndex alternative?

[josepot]

The legacy JSON-RPC API provides the system_accountNextIndex RPC endpoint, which considers the transactions present in the pool.

Additionally, PJS offers a useful shorthand allowing users to pass -1 as the nonce. This signals PJS to obtain the nonce from the system_accountNextIndex endpoint.

Using the new JSON-RPC API, we can utilize the AccountNonceApi_account_nonce runtime call against the most recently seen block to get the latest nonce. However, if there are transactions in the pool, the resulting transaction will eventually become invalid because it won't have the latest nonce.

I am concerned that there is no way to accomplish the same with the new JSON-RPC API. As a library author trying to provide an alternative to PJS, my users are experiencing issues because they must manually track the latest nonce when sending multiple transactions in parallel.

I am seeking advice on the following:

1. Should polkadot-api attempt to provide similar functionality by keeping track of currently broadcasted transactions from this client? This approach seems somewhat hacky and risky.
2. Are there any other options or ideas that I may not have considered?

Any guidance or suggestions would be greatly appreciated.

cc: @tomaka @jsdw @bkchr

[josepot]

The more I think about this, the more I realize that there is no ideal solution to this problem:

• There is no trustless way to get the next nonce from the network. A node can prove it has certain transactions in the pool for a given address, but it cannot prove those are the only transactions it has for that address.

• Since there is no way to get this information trustlessly from the network, it seems that the responsibility for keeping track of the nonce should fall on the signer. However, for the signer to do this, it would also need to handle broadcasting transactions and ensuring that the broadcasted transactions remain valid, which is obviously not a feasible option.

It appears that any solution we choose will be suboptimal in some way.

Thoughts? 🙏

[tomaka]

Using the same account from two different clients at the same time is fundamentally a wrong thing to do.

There's fundamentally no way to work around the fact that if you send a transaction from two places at once at the same time, they will both use the same nonce. It is inherently subject to race conditions. Information takes time to go from point A to point B.
The only way to make that work would be to have some kind of "mutex" or coordinator or something that ensures that the two transactions are generated and sent one after the other. But if you do that, you should use that coordinator to assign the nonces, it will be more robust.

If you send transaction with nonce 1 from A, then wait a bit, then send transaction with nonce 2 from B, it will be work 99% of the time but note that it's also not a strictly robust solution. It is important that the next block producer receives either only transaction 1 or both transactions 1 and 2. If it receives only transaction 2, it will be rejected for bad nonce. Since you don't know what the next block producer is, the only way you can somewhat guarantee that is to send both transactions 1 and 2 from the same client, as the two transactions will be sent to the same nodes.
But even then, it's not strictly guaranteed, as you might have connected to additional peers in-between the two transactions.

Basically, the system is not designed to 100% reliably handle more than one transaction per account per block. If you send more than one transaction per account per block, there's always a possibility (even tiny) for a transaction to be rejected due to bad nonce.

So, to summarize:

• Sending one transaction per account at a time, each time waiting for the previous one to be included in a block: works 100% of the time (but then note that there's the possibility for the block to be reverted and not finalized, which is important because a transaction that gets reverted is not so different from a transaction that gets rejected for bad nonce).
• Sending multiple transactions at a time from the same client, tracking and updating the nonce locally: works 99.9% of the time. It can still fail in case you send the first transaction to some faulty peers, then connect to other peers and send the second transaction to them.
• Sending multiple transactions at a time from multiple clients, with some kind of external system that makes sure that the nonces are different: works I guess 99% of the time. It can fail if the second transaction reaches the block producer before the other, and that the block producer produces a block right at this moment. Might work less often if the network load increases.
• Sending multiple transactions at a time from multiple clients, using the mechanism that PolkadotJS does that you describe in the OP: I guess 97% of the time. It can fail if the transactions are generated at the same time and have the same nonces.

---

The only situation where what PolkadotJS does could be useful is when people open two browser tabs, then they submit a transaction from the first tab, wait one or two seconds, then submit a transaction from the second tab.
It's like having a "mutex" (as I mention above), except that the mutex is the user's brain.

However, given that light clients do not receive transactions, what PolkadotJS does can't work on top of a light client. We specifically don't want light clients to download transactions, since that's very much the definition of a light client: a light client should be O(1) against the number of transactions per block, in other words it should remain light even if there are 1000 transactions or more per block.

[tomaka]

What I suggest you do in my opinion is:

• Provide an "easy to use" function that simply uses the nonce of the best block and tracks the nonce locally. Document that it should be called from one client at a time.
• Provide a more "advanced" function where the API user explicitly passes the nonce by parameter. This way, API users are made aware of this whole nonce thing and need to think about it.

[josepot]

Provide a more "advanced" function where the API user explicitly passes the nonce by parameter. This way, API users are made aware of this whole nonce thing and need to think about it.

We already have this feature, and it has led to some user complaints. They encountered issues when handling the logic for auto-incrementing the nonce, which resulted in race conditions. While these race conditions are easily avoidable, from the users' perspective, they had to implement logic they didn't need before when using PJS. Consequently, they find our current API more complicated and error-prone.

Provide an "easy to use" function that simply uses the nonce of the best block.

We already have this as well. In fact, we also offer an option to create "optimistic" transactions. This means that if you have observed a block that changes the state favorably for your needs, you can create a transaction against that block. The nonce, mortality, and initial validation will occur against that block. If that block gets pruned, your transaction will be reported as "invalid." This behavior is desirable for use cases where the transaction should only be included if the block becomes part of the canonical chain.

and tracks the nonce locally. Document that it should be called from one client at a time.

Initially, I was unsure whether this was worth adding. However, after reading your comments and thinking about it further, I agree that we should include this option. We will implement it, and in the documentation, we will prominently highlight the risks and trade-offs associated with using this feature.

[tomaka]

Consequently, they find our current API more complicated and error-prone.

That's why it's "advanced".