Running renovate-pro webhook instance separately

[huydinhle]

We want to run an instance of renovate-pro webhook to only process github webhook request for renovate alongside with our current renovate server based on this documentation. https://webcache.googleusercontent.com/search?q=cache:-kIJZDUFF_4J:https://docs.renovatebot.com/pro/advanced/+&cd=1&hl=en&ct=clnk&gl=us

Is it still possible do do this ? and If it is, then is there an official documentation on this? ( We keep getting 404 for the official docs page)

[rarkins]

FYI the docs are in markdown format in this repo, e.g. https://github.com/whitesource/renovate/blob/master/docs/advanced.md

It was never possible to run the webhook instance separately - it's part of a monolithic Node.js application.

[huydinhle]

In the docs mentioned this

For GitHub, this means you may want to have a single server dedicated to processing webhook jobs and then spread the scheduled load amongst other repositories. For the webhook server, you could set an autodiscoverFilter that matches no repos.

What does this mean by the webhook server?

[wyardley]

@rarkins I think the question was whether we could use the workaround mentioned here:
https://github.com/whitesource/renovate/blob/master/docs/advanced.md#horizontal-scaling
that is, use multiple instances of the entire service, but use autodiscoverFilter to force the webhook instance to only process webhook requests.

[rarkins]

OK, I understand the confusion. The webhook and worker logic are bound together, but the idea was:

• Have one server that does no scheduled jobs and processes only webhook jobs
• Have one or more additional servers that do not get sent any webhooks, and only work on scheduled jobs. Use an autoDiscover filter to get them to work on separate subsets of the total list of repos

[wyardley]

@rarkins yep -- that was our goal! Sorry about any confusion. We had found the cached version of the docs, but hadn't found the updated link, so thanks for pointing that out.

Do you think it matters whether they're two containers in a single Kubernetes deployment, or two separate deployments?

[rarkins]

I can't think of any problem with it being in the same deployment

[wyardley]

Great, thanks!

[etiennetremel]

I wasn't able to find documentation on how to disable scheduling of the cron job. If I want to run one instance as webhook only (scheduling disabled), how would I do that?

As mentioned, I have a fake autodiscoverFilter to prevent any match but it seems that the webhook doesn't trigger at all when using this:

autodiscoverFilter: 'no-repo-match',

I could use an hacky way and prevent the cronjob from running, like executing only on the 31st of February:

SCHEDULER_CRON=0 0 5 31 2 ?

[rarkins]

I could use an hacky way and prevent the cronjob from running, like executing only on the 31st of February:

That was the type of thing I had in mind.

the webhook doesn't trigger at all when using this

So the Pro server receives webhooks but ignores them?

[etiennetremel]

I'm not sure about the logic. From what I observed, requests from Github to the Renovate webhook are successful but I only see the following logs with no repositories added to the queue, which I think are part of the scheduler:

 INFO: jobScheduler running
 INFO: Found 1 account(s) installed
       "accountNames": ["orgs"],
       "accountCount": 1
 INFO: Repository count=123
 INFO: Applying autodiscoverFilter
 INFO: None of the discovered projects matched the filter
 INFO: Enqueing repos
       "repositories": []

Which make me wonder if the autodiscoverFilter prevent repositories from being processed/added to the queue.

[rarkins]

Always try to run in debug log mode if you're experimenting. From what I see of the INFO mode, it's what I had expected. i.e. it has access to 123 repositories but the filter (deliberately) means none get enqueued for "scheduled" jobs. I thought what you were aiming for was that when a webhook is received, it processes a job then, right?

[etiennetremel]

Yes that's correct, below is what I'm seeing with debug on and after pushing a commit with an old Terraform module version referenced. I would expect Renovate to pick up the change and run.

DEBUG: Received webhook event (repository=orgs/my-repo)
       "event": "push",
       "action": undefined,
       "installationId": ...
DEBUG: Ignoring unimportant commits to master (repository=orgs/my-repo)

[rarkins]

Seems like the filename wasn't recognized as a package file. The list of patterns for the webhooks is currently static and not as complete as in the main code.

If you update a file like a package.json, Dockerfile, or renovate.json, you hopefully see a different result.

[etiennetremel]

It works as expected when updating the renovate.json file. Terraform files *.tf are probably not supported with renovate/pro:0.18.5. Will patiently wait for the new release. Thanks for your help @rarkins !

[jamietanna]

Just adding a bit of interest to being able to get the Webhook instance running separately.

We're running a single instance for ~2000 repositories which are fairly noisy in terms of Webhook traffic, and are finding that the app is sometimes getting killed by our ECS healthchecking, as the healthcheck endpoint appears to be deprioritised when the scheduled run is occuring.

Being able to split the two out would be ideal.

In the meantime, we're considering running the Webhook-exposing app as one instance, and then a slightly more bulky instance for the scheduled runs.