Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reconnect websocket on random connection failures #3

Open
guilledk opened this issue Apr 18, 2023 · 2 comments
Open

Reconnect websocket on random connection failures #3

guilledk opened this issue Apr 18, 2023 · 2 comments
Assignees
@guilledk

Comments

@guilledk
Copy link
Collaborator

It's really common for websockets to drop the connection specially after long sessions, we should port over hyperion-history-api's reconnect machinery https://github.com/eosrio/hyperion-history-api/blob/v3.3.7/connections/state-history.ts
@guilledk guilledk added the enhancement New feature or request label Apr 18, 2023
@guilledk guilledk self-assigned this Apr 18, 2023
@guilledk guilledk mentioned this issue Apr 18, 2023
Merged
@guilledk guilledk removed the enhancement New feature or request label Apr 19, 2023
@guilledk
Copy link
Collaborator Author

This was already addressed in #4 and #6 but JT just reported this new one:

[READER] 2 block/s | Blocks: 0 (0.0%) | Actions: 0 | Deltas: 0
2023-11-16T14:07:08.625Z [PID:523520] [info] : 310360890 pushed, at 2 blocks/sec
2023-11-16T14:07:08.828Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:08.828Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:08.828Z [PID:523520] [info] : done.
2023-11-16T14:07:09.326Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:09.326Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:09.327Z [PID:523520] [info] : done.
2023-11-16T14:07:09.625Z [PID:523520] [info] : 310360892 pushed, at 2 blocks/sec
2023-11-16T14:07:09.827Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:09.827Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:09.827Z [PID:523520] [info] : done.
2023-11-16T14:07:10.324Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:10.324Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:10.324Z [PID:523520] [info] : done.
[READER] 2 block/s | Blocks: 0 (0.0%) | Actions: 0 | Deltas: 0
2023-11-16T14:07:10.625Z [PID:523520] [info] : 310360894 pushed, at 2 blocks/sec
2023-11-16T14:07:10.825Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:10.825Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:10.825Z [PID:523520] [info] : done.
2023-11-16T14:07:11.122Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:11.122Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:11.122Z [PID:523520] [info] : done.
2023-11-16T14:07:11.625Z [PID:523520] [info] : 310360896 pushed, at 2 blocks/sec
2023-11-16T14:07:11.791Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:11.791Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:11.791Z [PID:523520] [info] : done.
[READER] 1.5 block/s | Blocks: 0 (0.0%) | Actions: 0 | Deltas: 0
2023-11-16T14:07:12.625Z [PID:523520] [info] : 310360897 pushed, at 1 blocks/sec
2023-11-16T14:07:13.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:14.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:15.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:16.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:17.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:18.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:19.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:20.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:21.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:22.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:23.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:24.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:25.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:26.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:27.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:28.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:29.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:30.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:31.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:32.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:33.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:34.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:35.630Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:36.631Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:37.632Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:38.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:39.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:40.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:41.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:42.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:43.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:44.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:45.634Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:46.634Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:47.634Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:48.635Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:49.635Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:50.636Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:51.636Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:52.638Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:53.638Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:54.639Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:55.639Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:56.639Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:57.640Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:58.641Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:59.641Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:00.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:01.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:02.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:03.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:04.643Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:05.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:06.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:07.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:08.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:09.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:10.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:11.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:12.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:13.645Z [PID:523520] [info] : stall detected... restarting ship reader.
2023-11-16T14:08:13.645Z [PID:523520] [warn] : restarting SHIP reader!...
2023-11-16T14:08:13.646Z [PID:523520] [warn] : reader stopped, waiting 2 seconds to restart.
2023-11-16T14:08:13.646Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:13.647Z [PID:523520] [warn] : SHIP Reader disconnected.
2023-11-16T14:08:14.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:15.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
[READER] Restarting...
2023-11-16T14:08:16.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:17.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:18.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
[READER] Connecting to ws://127.0.0.1:19000...
Connecting to ws://127.0.0.1:19000...
2023-11-16T14:08:18.650Z [PID:523520] [info] : SHIP Reader connected.
[READER] Head block: 310361029
[READER] Requesting blocks from 309853135 to 4294967295
2023-11-16T14:08:19.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:20.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
[READER] Connecting to ws://127.0.0.1:19000...
Connecting to ws://127.0.0.1:19000...
2023-11-16T14:08:20.652Z [PID:523520] [info] : SHIP Reader connected.
DecodingError: Unknown variant idx: 123
    at decodeInner (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:215:27)
    at decodeBinary (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:181:16)
    at Object.abiDecode [as decode] (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:131:20)
    at HyperionSequentialReader.handleShipMessage (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:184:35)
    at file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:140:18
    at WebSocket.<anonymous> (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/state-history.js:27:41)
    at WebSocket.emit (node:events:514:28)
    at Receiver.receiverOnMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/websocket.js:1192:20)
    at Receiver.emit (node:events:514:28)
    at Receiver.dataMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/receiver.js:560:14) {
  ctx: {
    types: {
      bool: [Object],
      string: [Object],
      asset: [Function],
      symbol: [Function],
      symbol_code: [Function],
      block_timestamp_type: [Function],
      bytes: [Function],
      checksum160: [Function],
      checksum256: [Function],
      checksum512: [Function],
      extended_asset: [Function],
      float128: [Function],
      float32: [Function],
      float64: [Function],
      int128: [Function],
      int16: [Function],
      int32: [Function],
      int64: [Function],
      int8: [Function],
      name: [Function],
      public_key: [Function],
      signature: [Function],
      time_point: [Function],
      time_point_sec: [Function],
      uint128: [Function],
      uint16: [Function],
      uint32: [Function],
      uint64: [Function],
      uint8: [Function],
      varint32: [Function],
      varuint32: [Function]
    },
    strictExtensions: false,
    codingPath: [ [Object] ]
  },
  underlyingError: Error: Unknown variant idx: 123
      at decodeInner (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:215:27)
      at decodeBinary (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:181:16)
      at Object.abiDecode [as decode] (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:131:20)
      at HyperionSequentialReader.handleShipMessage (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:184:35)
      at file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:140:18
      at WebSocket.<anonymous> (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/state-history.js:27:41)
      at WebSocket.emit (node:events:514:28)
      at Receiver.receiverOnMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/websocket.js:1192:20)
      at Receiver.emit (node:events:514:28)
      at Receiver.dataMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/receiver.js:560:14)
}
2023-11-16T14:08:21.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:22.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:23.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:24.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:25.646Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec

@guilledk
Copy link
Collaborator Author

guilledk commented Dec 1, 2023

https://pastebin.com/j9Xu6KzU

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@guilledk guilledk

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@guilledk