Offchain Indexers unreliably processing data

[ethyi]

The offchain indexer is failing to reliably process offchain json uris.
This is making get_asset calls return NFTs without its offchain data populated, the most important being the 'files' field which usually holds the uri to the NFT's image.

Context

NFT metadata gets first indexed when the metadata account is processed via acc geyser. Note that this is different from the mint account.
The ingester initially populates the DB with the onchain metadata, a reference to the offchain uri, and puts a placeholder "processing" in the metadata field until the offchain indexer populates it.
The offchain indexer is a background task that is asynchronously invoked by the ingester. When a new task is submitted it is added to the "tasks" table, and each row contains data on the id that called it and its status.

The task makes a get request to the uri with a timeout of 3 seconds. If at any point the request returns a "reqwest::Error", the row in the tasks table will detail that a failure happened along with a short message identifying the error. Currently all reqwest::Error are converted to "Error downloading batch files" which is likely what most will receive.

• Each NFT will create its own background task, regardless if many have the same json uri
• The bgtask runner will purge tasks older than 1 hr
• If the task is already present in the database, a new task job will not be submitted even if the metadata account is reprocessed.
  • Owning an FFF is a great way to test this since you can dynamically change the offchain data.
• The timeout for offchain calls is 3 seconds.
• There are a maximum of 3 attempts before the offchain indexer completely drops the data.
• The bgtask runner will rerun tasks for all rows in asset_data that have their metadata as "processing"
• (*on helius) there are more offchain failures than there are successes

Examples

3LBkptXr6N9y3S2eSRTxCega91qHd4Fd6hWyNmBdCzRu

Solutions

• Priority background task runner with more flexibility. Addresses that see lots of activity will use this instead
• Increase attempts, and increase timeout if possible.

[linuskendall]

I took a bit of look at this and a few ideas from my side:

1. We should add a configurable timeout and a configurable number of retries, probably also MAX_TASK_BATCH_SIZE should be configurable too so we could tweak these values accordingly. In this way we can experiment with different values for these and track results (see 2)

2. We should capture the metrics for error rates and compare, right now it looks like it's quite different across different envs. We have one env where it's about 30% at the moment. Is the bgtask_succes/bgtask_error sufficiently finegrained metric or is there another one we should look at ?

3. When it comes to the priority, we could add a priority field to the DB and add a second ordering here: https://github.com/linuskendall/digital-asset-rpc-infrastructure/blob/fix_das_api_frontend_panic/nft_ingester/src/tasks/mod.rs#L149 ? We could also if we wanted to get fancy add the ability for a bgtask runner to only process tasks with priority > N, so you can deploy priority runners in your infra and others. This would require some tooling to interface with the DB and mark certain assets as priority for downloads.

Potential future:

4. Maybe we should have something like a download manager for these downloads? For example I am wondering if we should ratelimit our fetches from a single upstream service.

[austbot]

This is an interesting problem and has to do with the number of available file descriptors, networking capacity and reliability of thr places the json is hosted.

I have a timeout in there and batch size. But they are t configurable now but easy to put in. The arweave json uris seem to have alot of trouble.

It's possible that the http lib we are using doesnt handle many many connections to the same host very well.

[NicolasPennie]

This is an interesting problem and has to do with the number of available file descriptors, networking capacity and reliability of thr places the json is hosted.

I have a timeout in there and batch size. But they are t configurable now but easy to put in. The arweave json uris seem to have alot of trouble.

It's possible that the http lib we are using doesnt handle many many connections to the same host very well.

We would really benefit form setting a TTL on off-chain data. So we don't re-index an offchain until we have passed the TTL duration. This would dramatically decrease the load and especially help with services like arweave or ipfs gateways that rate limit by IP.