Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Support exponential/native histograms in Temporal Server/SDKs #6633

Open
gregbrowndev opened this issue Oct 9, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@gregbrowndev
Copy link

gregbrowndev commented Oct 9, 2024

Is your feature request related to a problem? Please describe.

Temporal Server is fairly expensive to monitor in self-hosted environments due to the volume of metric series generated. Observability platforms, such as AWS CloudWatch Metrics, Grafana Cloud, etc. charge per active metric series so the costs quickly add up.

Prometheus has experimental support for native histograms, and stability is improving daily. One of the main advantages of native histograms over Prometheus' classic histograms is that they can store the same data with fewer metric series and higher accuracy/resolution.

The presenter in the YouTube video "Prometheus Native Histograms in Production - Björn Rabenstein, Grafana Labs" at 17:30 states: "bottom line is you get 10x the resolution at half the price". That infographic also shows the number of series is ~16k compared to ~1k for classic vs native histograms in his example, respectively. Because you only need a single series to store the whole histogram (for a given set of labels).

For teams deploying a new Temporal installation, having the option to export exponential histograms would be great, as we can save costs and we since don't have extensive dashboards/alerting/SRE based on the old metric names, we can quickly build out the SRE on the new native histograms.

Describe alternatives you've considered

I'm using Grafana Alloy specifically to scrape the Temporal Server metrics. When I enable the option to scrape native histograms, no native histograms are scraped. The SDK metrics emitted using the OTel config are emitted as classic histograms. I believe this needs to be updated in the Server/SDK code.

Additional context
Add any other context or screenshots about the feature request here.

@gregbrowndev gregbrowndev added the enhancement New feature or request label Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant