Don't wait for Phabricator events in the 'events' service, wait through a Taskcluster hook instead

[marco-c]

Taskcluster now supports triggering a webhook through a unique url (with a token).
We could build a new hook that is triggered by each Phabricator diff (same config as currently), creating a task associated to the diff.
That task would run the same workflow as events:

• Clone base repo, using a Taskcluster cache and robustcheckout
• Apply the diff on tip
• Push to try
• Update phabricator harbormaster state for the diff

This would resolve the issues around disk space that we have on Heroku, allowing us to do things like #2004 and #2204.
This would also allow us to make the 'events' service slimmer. If we do both this and #945, we could get rid of the 'events' service altogether.

[La0]

The overall workflow would be :

1. Phabricator triggers a webhook on Taskcluster
   • using a new hook declared in fxci-config repo
   • using Taskcluster recent public webhook trigger
2. The hook triggers a new task with the request from the caller (in payload

{
    firedBy: "triggerHookWithToken",
    taskId: 'IgfFQSAqQwysozyeB7udBw',
    payload: {..}               // API call payload
}

We know from our libmozevent implementation that Phabricator is able to send

• diff ID (that's the only required info)
• target PHID (required to send messages back to Phabricator
• repo PHID
• revision ID

3. From there we can reuse code from libmozevent, without using the Bus (used on events to run continously). Internal wokflow is the one mentionned by @marco-c above:

• clone base repo using TC cache & robustcheckout (implemented in libmozevent)
• apply the patch (also in libmozevent , just requires to create a PhabricatorBuild to match current implem)
• push to try is immediate
• updating Harbormaster state is also available in libmozevent

I suggest we implement this in a new sub-project in this repo, named trigger. It would depend on libmozevent and only implement glue-code to retrieve infos from the Taskcluster payload (or from local CLI arguments to debug/develop)

[La0]

I filed a bugzilla ticket to inquiry about the secret token for a hook : https://bugzilla.mozilla.org/show_bug.cgi?id=1934254

I'll wait for an answer before asking for a new configuration on Phabricator

[La0]

I requested the scope to get the existing hooks token : mozilla-releng/fxci-config#199

We can then trigger the testing hook manually and update existing code to test what Phabricator will send.

[La0]

Code review developers now have the TC scope to retrieve the token, as demonstrated with that code + a TC client with scope hooks:get-trigger-token:project-relman/code-review-*:

from taskcluster.hooks import Hooks

hooks = Hooks({
  'rootUrl': 'https://firefox-ci-tc.services.mozilla.com',
  'credentials': {
    'clientId': '...',
    'accessToken': '...'
  },
})

group_id, hook_id = "project-relman", "code-review-testing"

resp  = hooks.getTriggerToken(group_id, hook_id)
token = resp["token"]

print(hooks.buildUrl('triggerHookWithToken', group_id, hook_id, token, {"data": "test"}))

Then we can start a task through curl:

curl -X POST --header 'Content-Type:  application/json' --data '{"privateData": "test from bastien"}' https://firefox-ci-tc.services.mozilla.com/api/hooks/v1/hooks/project-relman/code-review-testing/trigger/TOKEN

Here is a sample task, crashing of course as it does not have any info on which patch to analyze.

Next steps:

1. request new webhook on Phabricator prod towards our testing hook
2. update bot code to have a trigger mode
3. update our hook to support new arguments from webhook

[La0]

I filed a new bugzilla ticket to request the Phabricator hook : https://bugzilla.mozilla.org/show_bug.cgi?id=1935142

[La0]

• dkl investigated the payload sent by Phabricator webhook
• I made a patch on fxci-config to support that payload when triggering the hook with a token
  • We'll receive the PHID as environment variable PHABRICATOR_OBJECT_PHID

[La0]

The code-review-testing hook has been updated, and I was able to trigger a task using the token AND the payload from dkl: https://firefox-ci-tc.services.mozilla.com/tasks/aO2klyylQXGCcCvWrYTIGg/definition

Next steps:

• implement a new mode in the bot to use these new environment variables
• ask dkl to implement the rule