method handler crashed error occuring after upgrade

[yaanakbr]

Describe the bug

User reported method handler crashed on method eth_getTransactionByHash

response for tx (0xc2cf7b3e7caa73a73014579a1aa6d84a3903d03af7d978c3df524cda07e3cb2e) Request: Unexpected error code (-32603): {"jsonrpc":"2.0","id":0,"error":{"code":-32603,"message":"method handler crashed"}}

we are running against tip:

We make these calls within 2 seconds from the timestamp shown here : https://arbiscan.io/tx/0x707d9b93699d015fbb53c1308fe541e9c332fd7a54d00d11f6a129c5913f1495

To Reproduce

Poll for new transactions and this error is happening pretty frequent since new upgrade:

Looking at our node logs, we can see this is happening within the node:

May 23 14:42:26 nitro[21982]: ERROR[05-23|14:42:26.025] RPC method eth_getTransactionByHash crashed: runtime error: invalid memory address or nil pointer dereference
May 23 14:42:26 nitro[21982]: goroutine 4920491992 [running]:
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/service.go:207 +0x89
May 23 14:42:26 nitro[21982]: panic({0x40dbe00, 0x69e27a0})
May 23 14:42:26 nitro[21982]:         /usr/local/go/src/runtime/panic.go:884 +0x213
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/internal/ethapi.(*TransactionAPI).GetTransactionByHash(0xc015ec3e90, {0x4fc1228, 0xc13461b450}, {0x70, 0x44, 0x32, 0x15, 0xf0, 0x96, 0x2f, ...})
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/internal/ethapi/api.go:1859 +0x141
May 23 14:42:26 nitro[21982]: reflect.Value.call({0xc016770f60?, 0xc0182083b0?, 0x719e23095b38?}, {0x458c42b, 0x4}, {0xc13461b590, 0x3, 0x0?})
May 23 14:42:26 nitro[21982]:         /usr/local/go/src/reflect/value.go:586 +0xb07
May 23 14:42:26 nitro[21982]: reflect.Value.Call({0xc016770f60?, 0xc0182083b0?, 0x16?}, {0xc13461b590?, 0x1?, 0x1?})
May 23 14:42:26 nitro[21982]:         /usr/local/go/src/reflect/value.go:370 +0xbc
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc01820da40, {0x4fc1228?, 0xc13461b450}, {0xc00067c048, 0x18}, {0xc10370c840, 0x1, 0x15e3497?})
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/service.go:213 +0x3c5
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc11ac060f0?, {0x4fc1228?, 0xc13461b450?}, 0xc15c2dca80, 0x1?, {0xc10370c840?, 0x155d5d0?, 0x413d320?})
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:565 +0x45
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc1502f2960, 0xc1168f6a50, 0xc15c2dca80)
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:512 +0x239
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc1502f2960, 0xc1168f6ab0?, 0xc15c2dca80)
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:470 +0x237
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*handler).handleNonBatchCall(0xc1502f2960, 0xc1168f6a50, 0xc15c2dca80)
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:296 +0x1ae
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1.1(0x4fc1228?)
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:269 +0x27
May 23 14:42:26 nitro[21982]: github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:387 +0xc5
May 23 14:42:26 nitro[21982]: created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc
May 23 14:42:26 nitro[21982]:         /workspace/go-ethereum/rpc/handler.go:383 +0x8d
May 23 14:42:26 nitro[21982]: WARN [05-23|14:42:26.025] Served eth_getTransactionByHash          conn=XXX.XXX.XXX.XXX:36952 reqid=1357371 duration="519.189µs" err="method handler crashed"

Additional context

Client Version:

curl http://localhost:8547   -X POST   -H "Content-Type: application/json"   --data '{"jsonrpc":"2.0","method":"web3_clientVersion","params":[],"id":1}'
{"jsonrpc":"2.0","id":1,"result":"nitro/v2.3.4-b4cc111/linux-amd64/go1.20.14"}

[amaurer]

Same issue running the offchainlabs/nitro-node:v2.3.4-b4cc111
Another node in cluster running offchainlabs/nitro-node:v2.3.3-6a1c1a7 , no issue

Both nodes pointing to a Relayer instance running offchainlabs/nitro-node:v2.3.3-6a1c1a7

INFO [05-23|19:52:12.594] Submitted transaction                    hash=0x3959e734156d393ebf43f9ea20aa436b3c503471e07787e751e565385c6e400e from=0xaFa6A78Bb856A23641e001e86dD457c351d7fdCB nonce=13540 recipient=0x9753A981aa128fD8F6f3a7d9dDE7367274555A64 value=0
ERROR[05-23|19:52:12.665] RPC method eth_getTransactionByHash crashed: runtime error: invalid memory address or nil pointer dereference
goroutine 640227 [running]:
github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
	/workspace/go-ethereum/rpc/service.go:207 +0x89
panic({0x40dbe00, 0x69e27a0})
	/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/ethereum/go-ethereum/internal/ethapi.(*TransactionAPI).GetTransactionByHash(0xc0459f4330, {0x4fc1228, 0xc0725e8fa0}, {0x39, 0x59, 0xe7, 0x34, 0x15, 0x6d, 0x39, ...})
	/workspace/go-ethereum/internal/ethapi/api.go:1859 +0x141
reflect.Value.call({0xc00019eea0?, 0xc024b66958?, 0x7b80dcaf6670?}, {0x458c42b, 0x4}, {0xc0725e9040, 0x3, 0x0?})
	/usr/local/go/src/reflect/value.go:586 +0xb07
reflect.Value.Call({0xc00019eea0?, 0xc024b66958?, 0x16?}, {0xc0725e9040?, 0x1?, 0x1?})
	/usr/local/go/src/reflect/value.go:370 +0xbc
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc04475cea0, {0x4fc1228?, 0xc0725e8fa0}, {0xc071e0fab8, 0x18}, {0xc0728ba288, 0x1, 0x15e3497?})
	/workspace/go-ethereum/rpc/service.go:213 +0x3c5
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc0ef86fae0?, {0x4fc1228?, 0xc0725e8fa0?}, 0xc0dc8bd260, 0x1?, {0xc0728ba288?, 0xc013489dc0?, 0xc013489db0?})
	/workspace/go-ethereum/rpc/handler.go:565 +0x45
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc070582c80, 0xc0ecf161b0, 0xc0dc8bd260)
	/workspace/go-ethereum/rpc/handler.go:512 +0x239
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc070582c80, 0xc0dc8bd2d0?, 0xc0dc8bd260)
	/workspace/go-ethereum/rpc/handler.go:470 +0x237
github.com/ethereum/go-ethereum/rpc.(*handler).handleBatch.func4(0xc0ecf161b0)
	/workspace/go-ethereum/rpc/handler.go:227 +0x265
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
	/workspace/go-ethereum/rpc/handler.go:387 +0xc5
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc
	/workspace/go-ethereum/rpc/handler.go:383 +0x8d

WARN [05-23|19:52:12.666] Served eth_getTransactionByHash          conn=10.22.200.6:35904 reqid=20 duration="435µs" err="method handler crashed"

[zekchan]

Same issue offchainlabs/nitro-node:v2.3.4-b4cc111

WARN [05-23|23:10:05.479] Served eth_getTransactionByHash          conn=xxxx:34128 reqid=1 duration=4.980908ms err="method handler crashed"
ERROR[05-23|23:10:05.507] RPC method eth_getTransactionByHash crashed: runtime error: invalid memory address or nil pointer dereference
goroutine 2323408939 [running]:
github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
	/workspace/go-ethereum/rpc/service.go:207 +0x89
panic({0x40dbe00, 0x69e27a0})
	/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/ethereum/go-ethereum/internal/ethapi.(*TransactionAPI).GetTransactionByHash(0xc002930270, {0x4fc1228, 0xc01f9460f0}, {0x6b, 0x28, 0xb7, 0xb0, 0x8, 0xea, 0x58, ...})
	/workspace/go-ethereum/internal/ethapi/api.go:1859 +0x141
reflect.Value.call({0xc00b288000?, 0xc00d7ed948?, 0x7f1d16308178?}, {0x458c42b, 0x4}, {0xc01f9468c0, 0x3, 0x0?})
	/usr/local/go/src/reflect/value.go:586 +0xb07
reflect.Value.Call({0xc00b288000?, 0xc00d7ed948?, 0x16?}, {0xc01f9468c0?, 0x1?, 0x1?})
	/usr/local/go/src/reflect/value.go:370 +0xbc
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc00b298300, {0x4fc1228?, 0xc01f9460f0}, {0xc004fb7578, 0x18}, {0xc11bf8f9f8, 0x1, 0x15e3497?})
	/workspace/go-ethereum/rpc/service.go:213 +0x3c5
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc0b7870c30?, {0x4fc1228?, 0xc01f9460f0?}, 0xc11bfa1a40, 0x1?, {0xc11bf8f9f8?, 0x155d5d0?, 0x413d320?})
	/workspace/go-ethereum/rpc/handler.go:565 +0x45
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc150cf3900, 0xc171725da0, 0xc11bfa1a40)
	/workspace/go-ethereum/rpc/handler.go:512 +0x239
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc150cf3900, 0xc171725e00?, 0xc11bfa1a40)
	/workspace/go-ethereum/rpc/handler.go:470 +0x237
github.com/ethereum/go-ethereum/rpc.(*handler).handleNonBatchCall(0xc150cf3900, 0xc171725da0, 0xc11bfa1a40)
	/workspace/go-ethereum/rpc/handler.go:296 +0x1ae
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1.1(0x4fc1228?)
	/workspace/go-ethereum/rpc/handler.go:269 +0x27
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
	/workspace/go-ethereum/rpc/handler.go:387 +0xc5
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc
	/workspace/go-ethereum/rpc/handler.go:383 +0x8d

[toolchainx]

Came across the same issue

[qn-srikanth]

Seems to also happen when an invalid tx hash is passed :

Eg : 0xca84899dc9e562be229fc3188ee324038F84c5170b866b34c638a892c703ca83.

curl localhost:8547 \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{"method":"eth_getTransactionByHash","params":["0xca84899dc9e562be229fc3188ee324038F84c5170b866b34c638a892c703ca83"],"id":1,"jsonrpc":"2.0"}'  --verbose

• Current result :

 {"jsonrpc":"2.0","id":0,"error":{"code":-32603,"message":"method handler crashed"}}

• Expected result :

{"jsonrpc":"2.0","id":1,"result":null}

---

Version :
{"jsonrpc":"2.0","id":1,"result":"nitro/v2.3.4-b4cc111/linux-amd64/go1.20.14"}

[Tristan-Wilson]

This will be fixed in the next release #2336

[yaanakbr]

This will be fixed in the next release #2336

@Tristan-Wilson -- any ETA on the next release? 😄