Paginating StorageMap

[amiyatulu]

To get the first 10 element with pass page size of 10

 let my_storage =
    polkadot::storage().profile_validation().citizen_profile_root();

let mut my_iter = client
    .storage()
    .at_latest()
    .await
    .unwrap()
    .iter(my_storage, 10)
    .await
    .unwrap();

    while let Some((key, value)) = my_iter.next().await.unwrap() {
        println!("Key: 0x{}", hex::encode(&key));
        println!("Value: {}", value);
    }

I want to paginate the storage map. How can I get the next 10 element or get range of value based on page number?

[jsdw]

Just FYI, the 10 doesn't control the number of values that are returned from this call; it just determines the number of results to fetch at a time (In the upcoming release this variable is removed and Subxt will handle this all behind the scenes).

So, there is no need to do anything special yourself; calling my_iter.next().await in a loop will return all of the relevant values that exist.

[amiyatulu]

But won't it loop form the beginning? How can I get values from the middle skipping some value?

[jsdw]

Yup; it'll loop from the beginning and handle pagination internally for you at the moment! So one approach to paginating is to hold onto the storate iterator you're handed back from the .iter() fn, and call that as needed.

If you want more control over how you iterate storage keys, then currently you'll have to do that much more manually; see https://docs.rs/subxt/latest/subxt/storage/struct.Storage.html#method.fetch_keys to fetch some keys (with an optional starting key) and then use https://docs.rs/subxt/latest/subxt/storage/struct.Storage.html#method.fetch_raw to try to fetch values for those keys as you go. One downside of this is that you then need to decode the results yourself.

The storage APIs are an area I'd like to improve, and I do agree that it should be possible to iterate from some starting key, to make the higher level APIs more useful.

[jsdw]

I had a look at the new interface, and as it stands it doesn't really support pagination by providing an initial key: https://paritytech.github.io/json-rpc-interface-spec/api/chainHead_unstable_storage.html

What it does support is getting a list of all of the keys, and then asking for specific values for sets of those keys.

What are you trying to achieve? It may be something that we can feed back into that spec.

[amiyatulu]

I want to get values from the middle, in lexicographic order.

e.g. If total count of StorageMap is 100, I can get values and keys from 50 to 60.

Not sure what will be better, in the storage api directly, or can be done with writing a custom runtime rpc function in the substrate itself. It will be more complicated with StorageDoubleMap etc, so custom runtime rpc may be better I think.

In the documentation of fetch_keys it says:

Supports pagination by passing a value to start_key.

fetch_key takes three argument: key, count: [u32], optional start_key

Not sure what's the difference between key and start_key.

[jsdw]

key is the prefix, or root that you want to fetch all of the descendants for. In the case of eg a double map, you could provide either the key that points to the map as a whole, and then you'd fetch all of the keys inside the map, or you could provide eg the concat(root_key + first_key_in_map) to fetch all of the keys underneath first_key_in_map.

start_key is the actual descendant key you want to begin from (in lexicographic order) when fetching all of the descendant keys. You can paginate by setting this to be the last key that you fetched previously.

I don't think that you're able to ask for eg "keys 50 to 60" in some map though (likely because the backend would then have to iterate through the storage to get to get 50 before returning results for you), so instead you have to know which key to start from, which means paginating from the beginning (this is fairly common in other APIs too for the same reason that providing a start item (often called a cursor) is more efficient than having to iterate to the Nth item in a DB).

[amiyatulu]

May be I can number the keys in another StorageMap, and fetch the key from the number, and than value from the key or use offchain indexing.

[jsdw]

FYI I did also open this on the new RPC spec: paritytech/json-rpc-interface-spec#85

The commend by Josep offhand is a good point; the new API, being light client friendly, may not be able to paginate through storage with a startKey at all, and in that case it may be that the only API we can offer here in the long run (Subxt also wants to support these new APIs and be light client friendly) are to iterate all keys/values or whatever in order.

Depending on the number of keys you expect to have and exact use case, it may be then that you would need to just keep the iterator around, or fetch all keys up front and then paginate through the actual values, or something like that.