[ISSUE] status endpoint crash

[rotarur]

If I make a larger query like this:

graph indexer allocations get --deployment QmYNPk3Mabrx4VmxPKmBevdi11cfVUfTXVtgygG4M4m9tQ

where the SG allocation is very old or the query itself is very abstract and the API needs to perform a larger query, I notice that the status API endpoint is crashing with no logs related.

This command above produces this error:

✖ [GraphQL] Failed to query Epoch Block Oracle Subgraph

then I query the status network

graph indexer status network --network arbitrum-one

and I get this output with an error indicating that (possibly) the JVM crashed (again no logs and no pods restarted)

graph@indexer-cli:~$ graph indexer status network --network arbitrum-one
Registration
┌───────────────────────────────────────┬────────────────────────────────────────────┬─────────────────┬────────────┬─────────────────────────────────────┐
│ url                                   │ address                                    │ protocolNetwork │ registered │ location                            │
├───────────────────────────────────────┼────────────────────────────────────────────┼─────────────────┼────────────┼─────────────────────────────────────┤
│ https://indexer.upgrade.thegraph.com/ │ 0xbdfb5Ee5a2aBF4fc7bB1bD1221067aEf7f9dE491 │ eip155:42161    │ true       │ 89.99997854232788,36.57848596572876 │
└───────────────────────────────────────┴────────────────────────────────────────────┴─────────────────┴────────────┴─────────────────────────────────────┘

Endpoints
┌─────────┬─────────────────┬─────────────────────────────────────────────┬────────┐
│ name    │ protocolNetwork │ url                                         │ status │
├─────────┼─────────────────┼─────────────────────────────────────────────┼────────┤
│ service │ arbitrum-one    │ https://indexer.upgrade.thegraph.com/       │ down   │
├─────────┼─────────────────┼─────────────────────────────────────────────┼────────┤
│ status  │ arbitrum-one    │ https://indexer.upgrade.thegraph.com/status │ up     │
└─────────┴─────────────────┴─────────────────────────────────────────────┴────────┘
The following endpoint tests failed:

https://indexer.upgrade.thegraph.com/
  Test: http get https://indexer.upgrade.thegraph.com/
  Error: fetch failed
  Possible actions:
    - Make sure https://indexer.upgrade.thegraph.com/ can be resolved and reached from this machine
    - Make sure the port of https://indexer.upgrade.thegraph.com/ is set up correctly
    - Make sure the test command returns an HTTP status code < 400

Indexer Deployments
<--- Last few GCs --->

[26273:0x64f1df0]    83686 ms: Mark-Compact (reduce) 4090.9 (4116.2) -> 4090.9 (4109.2) MB, 35.44 / 0.00 ms  (+ 8.5 ms in 1 steps since start of marking, biggest step 8.5 ms, walltime since start of marking 48 ms) (average mu = 0.501, current mu = 0.311) [26273:0x64f1df0]    83747 ms: Mark-Compact (reduce) 4093.2 (4111.5) -> 4093.2 (4111.5) MB, 55.82 / 0.00 ms  (average mu = 0.340, current mu = 0.078) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xca5580 node::Abort() [node]
 2: 0xb781f9  [node]
 3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 5: 0x10dc505  [node]
 6: 0x10f4388 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0x10a8c86 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x1503a16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x7daacfed9ef6
Aborted