Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce over-alerting from HoneyBadger for memcached down #2772

Open
jrochkind opened this issue Oct 28, 2024 · 0 comments
Open

Reduce over-alerting from HoneyBadger for memcached down #2772

jrochkind opened this issue Oct 28, 2024 · 0 comments
Labels
infrastructure related to heroku, AWS, etc deploy/operational infrastructure

Comments

@jrochkind
Copy link
Contributor

We use a memcached server (via a heroku plugin) only for keeping track of request velocity for rate-limiting (with rack-attack).

Sometimes the memcached server goes down for a couple minutes or whatever. We aren't sure why, or how to fix it, and it's annoying, but not actually disastrous to not be metering rate limiting for a couple minutes.

While normally HoneyBadger is pretty good at "collapsing" multiple instances of the same error into one listed error for HB -- for these, I guess there are enough different stack traces, that HB isn't able to collapse them and reports the error to us over and over again, even though to us it's just "right memcached is still down 20 seconds later."

This annoys us and can lead to error fatigue.

We should figure out how to get HoneyBadger to properly collapse these errors, or consider having it ignore them entirely -- since we don't really care about them, but on the other hand if memcached went down and STAYED down forever we'd want to know, so maybe we do want some alerting. Have to think about it.

On customizing HoneyBadger fingerprinting for grouping/collapsing errors: https://docs.honeybadger.io/lib/ruby/getting-started/customizing-error-grouping/

I guess HB is fingerprinting them all separate since they are all for different Rails controller actions, it's like a low-level side thing that goes wrong, so they all get different fingerprints?

@jrochkind jrochkind added the infrastructure related to heroku, AWS, etc deploy/operational infrastructure label Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure related to heroku, AWS, etc deploy/operational infrastructure
Projects
None yet
Development

No branches or pull requests

1 participant