Adjustments for expired dot token Jenkins jobs

[robrap]

The job https://tools-edx-jenkins.edx.org/job/oauth/job/stage-edx-delete_expired_dot_tokens/ failed, and I'm guessing it will pass on the next run, which is currently configured for a day later.

Here are some adjustments we'd like to make:

• Adjust the 3 oauth Jenkins jobs so arch-bom has permissions to run the job (to clean up after a failure). (SRE request?)
• Adjust in OpsGenie to only report on the 3rd failure, so it has a chance to re-run and fix itself.
  • Adjust the stage job to run hourly during the working day.
    • Note that the 3rd failure will alert at whatever time it fails, thus configuring during working hours.
  • Do we want to configure the 3rd failure alerting for the prod jobs as well? If so, should they only be during working hours?

[jristau1984]

From Grooming discussion:

• Just have SRE grant re-run permissions to Arch-BOM to allow rerunning the job after a failure. This can get a smaller subtask specific to that request, and use the escalate label to get it in SRE board.