Performance Issues / Slowness

[cvarjao]

1. Multi Ledger: Having multi ledger causes AFJ to have to check each ledger to find out which one has the needed DID/REVOC_REG_DEF/REVOC_REG_ENTRY/
   1. Mitigation: https://hyperledger.github.io/indy-did-method/
2. Ledger Performance Issue
   1. Person / CANDY-PROD
   2. LSBC / sovrin-main-net
3. Ledger genesis file with dead nodes
4. Ledger Pool Reconciliation
5. Size of the revocation registry
6. Tails file size / speed
7. Mediator
8. Size of Credential
9. Agent (Issuer/Verifier) Performance
10. Fresh/Existing Install

• Check if AFJ/Wallet support timing protocol.

[cvarjao]

I've found some unresponsive nodes in used in the ledgers genesis:

java.net.ConnectException: failed to connect to /185.27.183.66   //SovrinStagingNet
java.net.ConnectException: failed to connect to /147.88.60.137 //SovrinMainNet
java.net.ConnectException: failed to connect to /37.97.232.101 //SovrinMainNet
java.net.ConnectException: failed to connect to /128.131.169.35 //SovrinMainNet
java.net.ConnectException: failed to connect to /138.68.240.143 //SovrinMainNet
java.net.ConnectException: failed to connect to /52.187.10.28 //SovrinMainNet
java.net.ConnectException: failed to connect to /207.224.246.57 //SovrinMainNet
java.net.ConnectException: failed to connect to /34.224.255.108 //SovrinMainNet
java.net.ConnectException: failed to connect to /34.226.105.29 //SovrinMainNet
java.net.ConnectException: failed to connect to /128.130.204.35 //SovrinMainNet

https://github.com/hyperledger/aries-mobile-agent-react-native/blob/main/packages/legacy/core/configs/ledgers/indy/ledgers.json

[swcurran]

What ledgers are the nodes from? We need to contact the ledger owners to get them to up their genesis files.

[cvarjao]

@swcurran , I've updated the list to include the ledger name. It is not an exhaustive list!
This is the second time we start to see unresponsive nodes that has a negative performance impact. Should this ledger discover/genesis be discuss to be more reliable? I am not sure if AFJ or VDR can also do something to discover, block/recover from those dead nodes

[WadeBarnes]

@cvarjao, when was the last time you updated the genesis files and what source did you use?

[swcurran]

@WadeBarnes — something to address with our Sovrin hats on. Looks like we need to update the Genesis Files for the Test and MainNet.

[cvarjao]

@WadeBarnes , this is the last time it got update (Dec 14, 2022):

• chore: combine genesis tx to JSON file openwallet-foundation/bifold-wallet#566

@jleach created that make-blocks.j script pulling from https://raw.githubusercontent.com/hyperledger/indy-node-monitor/main/fetch-validator-status/networks.json

[WadeBarnes]

Thanks. I just compared what you have registered, for Sovrin MainNet, in your file to a fully up to date copy. There are only two entries missing from your copy. I'll look through the IPs you have listed above.

[WadeBarnes]

@WadeBarnes — something to address with our Sovrin hats on. Looks like we need to update the Genesis Files for the Test and MainNet.

They are using an up to date genesis file. Only two entries behind what the nodes report. Something else is afoot.

[WadeBarnes]

The following table contains the results of my analysis. It looks like whatever code is processing the genesis file is not processing it completely or accurately. None of the Sovrin MainNet nodes listed, other than prosovitor should be contacted based on your copy of the genesis file. The code is attempting to contact nodes that either don't exist, or have different IPs.

Node Alias	DID	Status	IP Address	Network	Note
ServerVS	BrVSq23mLDYKJaxb9874MMA1ZudNq6YrUHziuMKjbEeL	Inactive	147.88.60.137	Sovrin MainNet	Removed from Network pool txn 102
thoth	ABje77QDKW2TaP37sA8dfsCkaFopFZpdzNfk2q9pWNsn	Inactive	37.97.232.101	Sovrin MainNet	Removed from Network pool txn 34
danube	476kwEjDj5rxH5ZcmTtgnWqDbAnYJAGGMgX7Sq183VED	Active	128.131.169.35	Sovrin MainNet	Old IP - Current IP (193.46.104.5) assigned pool txn 68
prosovitor	C8W35r9D2eubcrnAjyb4F3PC3vWQS1BHDg7UvDkvdV6Q	Inactive	138.68.240.143	Sovrin MainNet	Removed from Network pool txn 109 (not in your genesis file)
iRespond	3SD8yyJsK7iKYdesQjwuYbBGCPSs1Y9kYJizdwp2Q1zp	Active	52.187.10.28	Sovrin MainNet	Old IP - Current IP (20.184.4.103) assigned pool txn 101
DustStorm	8gGDjbrn6wdq6CEjwoVStjQCEj3r7FCxKrA5d3qqXxjm	Active	207.224.246.57	Sovrin MainNet	Old IP - Current IP (172.110.164.16) assigned pool txn 105
BIGAWSUSEAST1-001	HMJedzRbFkkuijvijASW2HZvQ93ooEVprxvNhqhCJUti	Inactive	34.224.255.108	Sovrin MainNet	Removed from Network pool txn 106
digitalbazaar	D9oXgXC3b6ms3bXxrUu6KqR65TGhmC1eu7SUUanPoF71	Inactive	34.226.105.29	Sovrin MainNet	Removed from Network pool txn 97
danube	476kwEjDj5rxH5ZcmTtgnWqDbAnYJAGGMgX7Sq183VED	Active	128.130.204.35	Sovrin MainNet	Old IP - Current IP (193.46.104.5) assigned pool txn 68

[WadeBarnes]

@WadeBarnes , this is the last time it got update (Dec 14, 2022):

• chore: combine genesis tx to JSON file hyperledger/aries-mobile-agent-react-native#566

@jleach created that make-blocks.j script pulling from https://raw.githubusercontent.com/hyperledger/indy-node-monitor/main/fetch-validator-status/networks.json

Reading the pool transactions directly from the networks themselves is the best way to get an up to date genesis file. That said the URLs listed in that file contain genesis files up-to-date within a few transactions, for example the published genesis file for Sovrin MainNet is only two transactions short of the live system's pool transactions list.

If you want an updated list, start an instance of indy-node-monitor, ./manage start indy-node-monitor, and use the /networks/{network}/pool/transactions API to get the pool transactions from the live network.

That said, having an up-to-date genesis file is not going to help when it's not parsed correctly, which is seemingly the issue identified above.

[WadeBarnes]

I've had a quick look at the Sovrin TestNet genesis file you have registered. It too is only two txns off from the live network, with NECValidator and Entrustient being removed from the network. 185.27.183.66 is associated with Stuard (C4e4rEwPZ4bM341VEL9ysnAgBiMW42RH3UmbriPuzoCb) which was removed from Sovrin TestNet at txn 24. So this is another instance where the code should not be even attempting to contact the node, because the genesis file indicates the node is no longer part of the network.

[swcurran]

That’s very weird — what code could it be using to contact the nodes? Is it that there is actually another cached copy of the genesis file somewhere? I would think that the library code (which I think is Indy VDR) would be doing The Right Thing, if it has the correct genesis file.

[WadeBarnes]

I've compared the Sovrin TestNet and MainNet genesis files to the ones here (https://github.com/hyperledger/aries-mobile-agent-react-native/blob/main/packages/legacy/core/configs/ledgers/indy/ledgers.json) and they match other than the two missing txns for each.

I find it weird too.

[esune]

Is there an ETA to the availability of a release in the App/Play Store with fix(es) targeting this issue? I'm holding-off testing of the new VC-AuthN release since the credentials used for the LSBC dev/test environment are on Sovrin TestNet and there are significant slow-downs/issues currently when presenting them - I will resume as soon as a new version is available.

[esune]

Posting as potentially relevant info (that was already shared internally). My test device is a Pixel 6 Pro on Android 14, I have also tested on Android 13 a couple of weeks ago with same results.

When trying to respond to the proof request generated by https://a2a-dev.apps.silver.devops.gov.bc.ca (this is the proof-configuration, credentials rooted on Sovrin TestNet), the BC Wallet hangs for extremely long spans of time (currently it's been sitting there for several minutes without apparent progress) after scanning the QR code, at the screen that should display which credentials could be presented (see attached screenshot). I do have valid credentials to respond to the proof in my wallet.

Other errors that I have been prompted with when trying to respond to a proof-request are related to the inability to automatically select requested attributes and inability to retrieve revocation status list or credential definition (see attached screenshots).

[WadeBarnes]

For comparison of the analysis I did previously, I've spun up askar based ACA-Py instances v0.10.3 connected to Sovrin StagingNet, and Sovrin MainNet. I've confirmed indy-vdr 0.3.4 is parsing the genesis files correctly, it's identifying and skipping the correct list of non-validator nodes before and after establishing the pool connection, so it's only contacting active validators.