RenovateOnPrem: 'JavaScript heap out of memory' error

[JoeJennings]

When RenovateOnPrem (version 2.6.0) is processing a Java component(s) repository that contains 40 pom.xml files that contain over 650+ dependencies a 'JavaScript heap out of memory' error sometimes occurs, about 50% of the time. The linux host RenovateOnPrem is running on has 32GB of memory. When this occurs RenovateOnPrem seems to crash and then restarts. Looking at the host's free memory it averages around 10GB of free memory when Renovate is not processing repositories and around 6GB of free memory when Renovate is processing repositories. With Renovate logging set to 'debug' and observing the logs of multiple occurrences I suspect the issue occurs after Renovate has processed the dependency updates and is creating the pull request updates.

Logs:

2023-01-25T08:18:05.744947282Z DEBUG: Dependency com.revenuecycle.platform:revenuecycle-products-parent is part of group all dependencies (repository=revenuecycle-core/revenuecycle-products-reactor)
2023-01-25T08:18:05.989735251Z DEBUG: 690 flattened updates found: org.apache.maven.plugins:maven-compiler-plugin, ...<removed items here to shorten logs> (repository=revenuecycle-core/revenuecycle-products-reactor)
2023-01-25T08:18:06.110527359Z DEBUG: Returning 1 branch(es) (repository=revenuecycle-core/revenuecycle-products-reactor)
2023-01-25T08:18:45.601636157Z 
2023-01-25T08:18:45.601779345Z <--- Last few GCs --->
2023-01-25T08:18:45.601797046Z 
2023-01-25T08:18:45.601802578Z [6:0x5af06a0] 58144023 ms: Mark-sweep 4032.2 (4135.0) -> 4017.7 (4136.5) MB, 3203.5 / 0.6 ms  (average mu = 0.131, current mu = 0.021) allocation failure scavenge might not succeed
2023-01-25T08:18:45.601819650Z [6:0x5af06a0] 58147219 ms: Mark-sweep 4033.8 (4136.8) -> 4018.9 (4137.8) MB, 3139.3 / 0.7 ms  (average mu = 0.079, current mu = 0.018) allocation failure scavenge might not succeed
2023-01-25T08:18:45.601825163Z 
2023-01-25T08:18:45.601853504Z 
2023-01-25T08:18:45.601867250Z <--- JS stacktrace --->
2023-01-25T08:18:45.601872310Z 
2023-01-25T08:18:45.605680828Z FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-01-25T08:18:45.713979561Z  1: 0xb02960 node::Abort() [node]
2023-01-25T08:18:45.714826663Z  2: 0xa18149 node::FatalError(char const*, char const*) [node]
2023-01-25T08:18:45.715470957Z  3: 0xcdd22e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
2023-01-25T08:18:45.716058699Z  4: 0xcdd5a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
2023-01-25T08:18:45.716720933Z  5: 0xe94c15  [node]
2023-01-25T08:18:45.717378429Z  6: 0xe956f6  [node]
2023-01-25T08:18:45.718036939Z  7: 0xea3c1e  [node]
2023-01-25T08:18:45.718743380Z  8: 0xea4660 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
2023-01-25T08:18:45.719387165Z  9: 0xea75de v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
2023-01-25T08:18:45.720091776Z 10: 0xe68b1a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
2023-01-25T08:18:45.720809123Z 11: 0x11e1886 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
2023-01-25T08:18:45.721595553Z 12: 0x15d54f9  [node]

Logging into bin/bash on the running RenovateOnPrem docker container and checking the JavaScript (node.js) heap space limit it appears to be 4G, which doesn't seem to be enough when Renovate is processing this repository.

node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
4144  

Using the RenovateOnPrem docker-compose.yml and setting the NODE_OPTS environmental variable '--max-old-space-size' to 8GB allows Renovate to process this repository without running into the 'JavaScript heap out of memory' error.

node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
8240 

Questions:

1. Does increasing the heap space feel like an appropriate solution to this issue or are there other alternatives?
2. Are there additional investigation/troubleshooting steps I could perform that would be beneficial to the RenovateOnPrem team regarding this issue?

Observations:

1. RenovateOnPrem is configured to write logs to file, however the logs around when the 'JavaScript heap out of memory' error occurs are not written out to file. I am able to see the logs in the docker logs on the host.
2. Setting the log level to 'trace' while Renovate processes this repository causes a more severe issue that RenovateOnPrem is not able to recover from (restart).

[rarkins]

Increasing heap space seems like a reasonable trade-off. Without the hosted Renovate App we run Renovate on machines with 3.75GB of memory and I haven't noticed many out of memory incidents or people complaining about it.

It would be useful if you are able to get any of the following info:

• Is it significantly increasing forever, i.e. a memory leak?
• Is there any way to know where this memory is being used?
• Do you see the memory return to a baseline after each repo stops?
• Do you see the same type of memory use if you Renovate the same repo using the Renovate CLI?