Debugging "allocation failed: JavaScript heap out of memory"

[godfat]

We're using wrangler to run HTTP Router, and we hit into allocation failed: JavaScript heap out of memory error when we're running end-to-end tests against it.

The error looks like this:

✘ ERROR <--- Last few GCs --->

  [3455:0x106cffaa8000]  1294500 ms: Mark-Compact 1402.0 (1415.2) -> 1393.4 (1414.9) MB, pooled: 4 MB, 235.44 / 0.00 ms  (average mu = 0.908, current mu = 0.625) allocation failure; scavenge might not succeed
  [3455:0x106cffaa8000]  1297045 ms: Mark-Compact 1403.8 (1416.6) -> 1396.1 (1417.5) MB, pooled: 2 MB, 235.76 / 0.00 ms  (average mu = 0.908, current mu = 0.907) allocation failure; scavenge might not succeed

  <--- JS stacktrace --->

  workerd/jsg/setup.c++:38: fatal: V8 fatal error; location = Reached heap limit; message = : allocation failed: JavaScript heap out of memory

A full log can be downloaded at (will expire in 6 days) https://gitlab.com/gitlab-org/gitlab/-/jobs/8365282139/artifacts/external_file/gdk.gitlab-http-router.log

More investigation context can be found at: https://gitlab.com/gitlab-org/gitlab/-/issues/499983

So far we couldn't reproduce it locally, and when I tried to run memory profile on it locally, it only consumed about 2M, and as highest I can observe is about 10M, which was far from we see from the end-to-end tests: https://gitlab.com/gitlab-org/gitlab/-/issues/499983#note_2210491776

I wonder if you might have some tips for debugging this?

Let me know if you need any information that I could provide.

[steve-mt]

I've attached the expiring logs below so we don't loose them 🙇

gdk.gitlab-http-router.log.zip

[irvinebroque]

Try running your Worker with the --remote flag which will run your Worker on Cloudflare. We enforce memory limits in production, but don't have the equivalent limiting behavior in local development. Replying on ticket for more context specific to Gitlab

[godfat]

Do you mean that there's no memory limit locally? What is the error we're seeing then? Does that mean it's actually the machine running out of memory?

I am not sure if running on Cloudflare can help here, because we need to fix the error locally, not on Cloudflare.

[godfat]

I have an update that I tried to pass --log-level debug to wrangler and then the error just disappeared.

The log will grow to double so we cannot just use this as a workaround though. Is there another way to avoid this memory issue? Passing --log-level debug to resolve it seems to suggest that there's some weird bugs internally.