Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve celery drift issue #741

Open
1 task
robrap opened this issue Jul 26, 2024 · 3 comments
Open
1 task

Resolve celery drift issue #741

robrap opened this issue Jul 26, 2024 · 3 comments
Assignees
Labels
escalate-to-psre Create a PSRE ticket for this issue waiting-on-sre

Comments

@robrap
Copy link
Contributor

robrap commented Jul 26, 2024

We are seeing large numbers of celery errors with "Substantial drift from celery" warnings:

Important:

  • I added a 99% exclusion rule related to the "missed heartbeat from celery" logs, and provided details in https://2u-internal.atlassian.net/browse/GSRE-1952?focusedCommentId=5146559. If GSRE-1952 fixes the drift issue, and the missed heartbeat turns out to be a separate problem, the missed heartbeat issue should be ticketed separately and not lost with this ticket. The retention rule could be switched to point to the updated ticket as well.
@robrap robrap added this to Arch-BOM Jul 26, 2024
@robrap robrap converted this from a draft issue Jul 26, 2024
@robrap robrap added the escalate-to-psre Create a PSRE ticket for this issue label Jul 26, 2024
Copy link

@jristau1984 jristau1984 moved this to Blocked in Arch-BOM Jul 29, 2024
@jristau1984 jristau1984 moved this from Blocked to Backlog in Arch-BOM Aug 5, 2024
@jristau1984
Copy link

@robrap wants to keep this open to follow up on the GSRE ticket at some point.

@timmc-edx
Copy link
Member

sorry-not-sorry

celery-drift-small

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
escalate-to-psre Create a PSRE ticket for this issue waiting-on-sre
Projects
Status: Backlog
Development

No branches or pull requests

3 participants