Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics may not be emitted for cleanup jobs #3688

Open
acrmp opened this issue Mar 19, 2024 · 0 comments
Open

Metrics may not be emitted for cleanup jobs #3688

acrmp opened this issue Mar 19, 2024 · 0 comments

Comments

@acrmp
Copy link
Member

acrmp commented Mar 19, 2024

Issue

Currently it is possible for the periodic updater not to emit metrics for daily cleanup jobs.

Context

The periodic updater looks at the delayed_jobs table to populate the known jobs and their queue lengths in order to emit metrics.

The metrics are emitted by cloud controller on the api nodes, if no api node sees the temporary insertion of the row in the delayed_jobs table before cc-worker picks it up then the job will not have a metric emitted.

It seems like for similar behaviour with the local worker a change was made to ensure that metrics would always be emitted: 0fddfd4

Steps to Reproduce

  • Deploy Cloud Foundry with a small number of api instances
  • Wait for a cleanup job to run
  • Occasionally observe that job metrics are not emitted

Expected result

Job queue metrics are emitted.

Current result

Job queue metrics are sometimes not emitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant