Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stack trace to error handler's HandlePanicFunc #423

Merged
merged 1 commit into from
Jul 4, 2024

Conversation

brandur
Copy link
Contributor

@brandur brandur commented Jul 4, 2024

This one in response to #418 in which although we persist a stack trace
to a job row's errors property, we don't reveal it in a panic handler,
which is quite inconvenient for purposes of logging or other telemetry
(e.g. sending to Sentry).

Here, HandlePanic's signature changes to (trace is added):

HandlePanic(ctx context.Context, job *rivertype.JobRow, panicVal any, trace string) *ErrorHandlerResult

A couple notes on choices:

  • The naming of trace is reused from AttemptError.

  • The value type is string. This is a little non-obvious because Go
    exposes it as a []byte, something I've never quite understood as to
    why, but I did string because for one it's more convenient to use,
    but more importantly, it's the same type on AttemptError.

This is a breaking change, but it seems like being able to get a stack
trace during panic is important enough that it's worth it, and with any
luck there's few enough people using this feature that it won't break
that many people. The fix is quite easy regardless and will easily be
caught by the compiler.

Fixes #418.

@brandur brandur requested a review from bgentry July 4, 2024 05:35
@brandur brandur force-pushed the brandur-panic-trace branch from 3a5c57b to fabdde1 Compare July 4, 2024 05:40
Copy link
Contributor

@bgentry bgentry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM aside from comments. Thanks for jumping on this!

CHANGELOG.md Outdated
@@ -11,6 +11,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- `Config.TestOnly` has been added. It disables various features in the River client like staggered maintenance service start that are useful in production, but may be somewhat harmful in tests because they make start/stop slower. [PR #414](https://github.com/riverqueue/river/pull/414).

### Changed

⚠️ Version 0.8.0 has a small breaking change in `ErrorHandler`. As before, we try never to make breaking changes, but this one was deemed quite important because `ErrorHandler` was fundamentally lacking important functionality.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong version number?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, thanks. Fixed.

job_executor.go Outdated
@@ -234,7 +234,7 @@ func (e *jobExecutor) invokeErrorHandler(ctx context.Context, res *jobExecutorRe

case res.PanicVal != nil:
errorHandlerRes = invokeAndHandlePanic("HandlePanic", func() *ErrorHandlerResult {
return e.ErrorHandler.HandlePanic(ctx, e.JobRow, res.PanicVal)
return e.ErrorHandler.HandlePanic(ctx, e.JobRow, res.PanicVal, string(res.PanicTrace))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we convert this to string once when saving it to the result so that it doesn’t have to be cast here and by pgx when writing to the db?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, makes sense. Fixed.

This one in response to #418 in which although we persist a stack trace
to a job row's errors property, we don't reveal it in a panic handler,
which is quite inconvenient for purposes of logging or other telemetry
(e.g. sending to Sentry).

Here, `HandlePanic`'s signature changes to (`trace` is added):

    HandlePanic(ctx context.Context, job *rivertype.JobRow, panicVal any, trace string) *ErrorHandlerResult

A couple notes on choices:

* The naming of `trace` is reused from `AttemptError`.

* The value type is `string`. This is a little non-obvious because Go
  exposes it as a `[]byte`, something I've never quite understood as to
  why, but I did `string` because for one it's more convenient to use,
  but more importantly, it's the same type on `AttemptError`.

This is a breaking change, but it seems like being able to get a stack
trace during panic is important enough that it's worth it, and with any
luck there's few enough people using this feature that it won't break
that many people. The fix is quite easy regardless and will easily be
caught by the compiler.

Fixes #418.
@brandur brandur force-pushed the brandur-panic-trace branch from fabdde1 to aeb3b97 Compare July 4, 2024 14:26
@brandur brandur merged commit 7899e20 into master Jul 4, 2024
10 checks passed
@brandur brandur deleted the brandur-panic-trace branch July 4, 2024 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

river hides panic stack traces in database
2 participants