Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to query the oasis node #494

Open
Swornatoya opened this issue Dec 1, 2023 · 16 comments
Open

Unable to query the oasis node #494

Swornatoya opened this issue Dec 1, 2023 · 16 comments

Comments

@Swornatoya
Copy link

The node stopped syncing and I had upgraded the oasis node to v2.6.0. When I check the logs of the node it is syncing fine as shown in the below image.
image

When I try to run the below curl query against the node.
curl --location 'http://************/block' \ --header 'Content-Type: application/json' \ --data '{ "network_identifier": { "blockchain": "Oasis", "network": "b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243f30af63c815535" }, "block_identifier": { "index": 16840161 } }'

I am getting response as { "code": 0, "message": "network identifier {\"blockchain\":\"Oasis\",\"network\":\"b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243f30af63c815535\"} is not supported: requestNetwork is not supported", "retriable": false }

Please let me know if the network identifier has been updated.

@pro-wh
Copy link
Collaborator

pro-wh commented Dec 1, 2023

yup, it's been updated. use the new network bb3d748def55bdfb797a2ac53ee6ee141e54cd2ab2dc2375f4a0703a178e6e55

from https://docs.oasis.io/node/mainnet/

@Swornatoya
Copy link
Author

Swornatoya commented Dec 4, 2023

@pro-wh Thanks for your response.

When I try querying the latest block synced using the upgraded node I am getting the response. But if I try to query the older blocks it does not work and I get the below response.

Curl Query: curl --location 'http://10.213.67.23:7332/block' \ --header 'Content-Type: application/json' \ --data '{ "network_identifier": { "blockchain": "Oasis", "network": "bb3d748def55bdfb797a2ac53ee6ee141e54cd2ab2dc2375f4a0703a178e6e55" }, "block_identifier": { "index": 16810161 } }'

Response: { "code": 12, "message": "unable to get block", "retriable": true }

@pro-wh
Copy link
Collaborator

pro-wh commented Dec 4, 2023

that latest synced block index can't be right. the bb3d748def55bdfb797a2ac53ee6ee141e54cd2ab2dc2375f4a0703a178e6e55 mainnet started at block 16817956. that mainnet can't answer about blocks from before it started

@Swornatoya
Copy link
Author

@pro-wh : Then how do we get the data prior to the block 16817956 from our node.

@pro-wh
Copy link
Collaborator

pro-wh commented Dec 5, 2023

run a separate node from the prior network, and run a separate oasis rosetta gateway connected to that node to answer for blocks in that network. it's a lot of trouble, we're aware. sorry about that

@Swornatoya
Copy link
Author

Swornatoya commented Dec 11, 2023

@pro-wh : Tried running the old node with the network b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243f30af63c815535 . But the node is not syncing and when try to check the latest block query it returns the genesis block and also getting the below error in the node.

{"caller":"pex_reactor.go:531","level":"info","module":"tendermint:pex","msg":"No addresses to dial. Falling back to seeds","ts":"2023-12-11T10:58:09.941068528Z"}
{"address":"[email protected]:26656","caller":"switch.go:716","level":"info","module":"tendermint:p2p","msg":"Dialing peer","ts":"2023-12-11T10:58:09.941084628Z"}
{"caller":"pex_reactor.go:632","err":"incompatible: peer is on a different network. Got bb3d748def55bdfb797a2ac53ee6ee141e54cd2ab2dc2375f4, expected b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243","level":"error","module":"tendermint:p2p","msg":"Error dialing seed","seed":"[email protected]:26656","ts":"2023-12-11T10:58:10.065524383Z"}

And when I try to query the block data from the node with the below curl query
curl --location 'http://********/block' \ --header 'Content-Type: application/json' \ --data '{ "network_identifier": { "blockchain": "Oasis", "network": "b11b369e0da5bb230b220127f5e7b242d385ef8c6f54906243f30af63c815535" }, "block_identifier": { "index": 8048956 } }'
Getting the output as : {"code":12,"message":"unable to get block","retriable":true}

@pro-wh
Copy link
Collaborator

pro-wh commented Dec 11, 2023

the seed nodes are now on the new network, so you won't be able to connect to a network using it. you can run a node in "archive" mode, where it won't connect to the p2p network. you'll need to run it with your own data, i.e. from before the wipe in the upgrade steps, as it won't sync data from a network.

@Swornatoya
Copy link
Author

We are running the oasis node using the below docker compose file
image
Even after mapping the volume of either :/data or :/node, we don’t get the volumes at NewDisk23Machine/node-oasis-rosetta/
image
But if you enter inside the container (with docker exec -it), it shows files and data is present inside
image
Ideally all this content should get mapped to host machine volume path as given in compose file. But it doesn’t. Instead, this container data goes to docker storage volumes and sits there. So every time the node container restarts, the volume is lost and node starts from scratch.
And now with the new network id and upgraded node we are unable to get the data before the block 16817956

@pro-wh
Copy link
Collaborator

pro-wh commented Dec 27, 2023

the volume for /data is commented out in the screenshot above

@Swornatoya
Copy link
Author

@pro-wh : Even when we uncomment the volume mapping for /data we don't get the volumes mapped to our node folder

@pro-wh
Copy link
Collaborator

pro-wh commented Jan 2, 2024

could it be a permissions issue? the mounted directory needs to be www-data:www-data (33) and be mode 0700. check the container logs for diagnostics. here's how I got it working locally:

mkdir -p /tmp/rosetta-data/etc
chmod 700 /tmp/rosetta-data
cp config.yml /tmp/rosetta-data/etc
wget -O /tmp/rosetta-data/etc/genesis.json https://github.com/oasisprotocol/mainnet-artifacts/releases/download/2023-11-29/genesis.json
sudo chown -R www-data:www-data /tmp/rosetta-data
docker run -it -v /tmp/rosetta-data:/data rosetta-priv:latest

(edit: chown later for convenience)

then

sudo ls -al /tmp/rosetta-data

shows files

@Swornatoya
Copy link
Author

@pro-wh : The image rosetta-priv:latest that you are using is it the latest and we need to use this image.
As because the image we are currently using is with version v2.6.0 and this we had build using the repo from the github.

@pro-wh
Copy link
Collaborator

pro-wh commented Jan 3, 2024

tried with 2.6.0, works with that too. there are no changes to how the docker image uses that directory between 2.6.0 and master

@Swornatoya
Copy link
Author

@pro-wh : Tried to start the node with the backup that we had to read the block data before 16817956. But getting error as below:
{"Version":"22.2.8-gitc159d0364","caller":"node.go:554","level":"info","module":"oasis-node","msg":"Starting oasis-node","ts":"2024-01-19T12:31:49.083307905Z"} {"caller":"helpers.go:40","level":"info","module":"common/persistent","msg":"All 1 tables opened in 0s","ts":"2024-01-19T12:31:49.092521167Z"} {"caller":"helpers.go:40","level":"info","module":"common/persistent","msg":"Discard stats nextEmptySlot: 0","ts":"2024-01-19T12:31:49.096822896Z"} {"caller":"helpers.go:40","level":"info","module":"common/persistent","msg":"Set nextTxnTs to 17295617","ts":"2024-01-19T12:31:49.097009697Z"} {"caller":"helpers.go:40","level":"info","module":"common/persistent","msg":"Deleting empty file: /data/persistent-store.badger.db/050796.vlog","ts":"2024-01-19T12:31:49.097095498Z"} {"caller":"upgrade.go:171","err":"binary consensus version not compatible: own: 6.0.0, required: 7.0.0","handler":"mainnet-upgrade-2023-11-29","level":"error","log_event":"upgrade/incompatible-binary","module":"upgrade","msg":"incompatible binary version for upgrade","ts":"2024-01-19T12:31:49.097705402Z"} {"caller":"node.go:599","err":"binary consensus version not compatible: own: 6.0.0, required: 7.0.0","level":"error","module":"oasis-node","msg":"failed to initialize upgrade backend","ts":"2024-01-19T12:31:49.097754102Z"} {"caller":"helpers.go:40","level":"info","module":"common/persistent","msg":"Lifetime L0 stalled for: 0s","ts":"2024-01-19T12:31:49.097771802Z"} {"caller":"helpers.go:40","level":"info","module":"common/persistent","msg":"Level 0 [ ]: NumTables: 01. Size: 3.3 MiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel Done","ts":"2024-01-19T12:31:49.103776442Z"}

@pro-wh
Copy link
Collaborator

pro-wh commented Jan 22, 2024

from our team:

did they enable archive mode?

https://docs.oasis.io/node/run-your-node/archive-node#configuration

@pro-wh
Copy link
Collaborator

pro-wh commented Jan 22, 2024

quick correction to the docs:

The setting is now just mode (top level, and not consensus.tendermind.mode as the docs say)

edit: docs updated too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants