Application crashes on ECS: Request timed out

[jamietanna]

We're running whitesource/renovate:2.6.0 on ECS with 4192 vCPU acllocations and 4192 MB Memory, which is hits a maximum of 80% CPU usage.

We've noticed a handful of ECS crashes - at least in the last week - which AWS notes as:

service ... instance ... port ... is unhealthy in target-group ... due to (reason Request timed out)

On the GitHub side, there’s a lot of We couldn't deliver this payload: timed out that happens just before the application stops responding.

I can’t see any relevant logs displayed, at least at LOG_LEVEL=INFO - would it help to up this to TRACE?

We're trying to work out whether it's down to specific repo(s) that this is happening in, but unfortunately getting a full dump of the Webhook delivery events is a bit slow.

[rarkins]

LOG_LEVEL=debug is likely to be more feasible than trace, so try that first.

When the request times out, is that referring to some type of ECS health check, not one of Renovate's own requests?

[jamietanna]

So according to GitHub, it's that the hook timed out:

I think it's that the connection between CloudFlare -> ALB -> Renovate appears to be timing out.

Looking at the ALB monitor, we're seeing:

But the ALB logs don't seem to show any 4xx 🤔