Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix subscription stress test in notifier #269

Merged
merged 1 commit into from
Mar 13, 2024
Merged

Conversation

brandur
Copy link
Contributor

@brandur brandur commented Mar 13, 2024

While I was trying to fix another problem in the notifier's subscription
stress test, I ended up introducing another one by using a single wait
group for both the notification sending goroutine and subscriptions
loops which caused the test intermittency described in #268. The test
was ending too soon, and channels would only receive 1-2 messages, and
sometimes zero (which would fail the test).

notifier_test.go:462: Channel  0 contains   2 message(s)
notifier_test.go:462: Channel  1 contains   2 message(s)
notifier_test.go:462: Channel  2 contains   2 message(s)
notifier_test.go:462: Channel  3 contains   2 message(s)
notifier_test.go:462: Channel  4 contains   2 message(s)
notifier_test.go:462: Channel  5 contains   2 message(s)
notifier_test.go:462: Channel  6 contains   2 message(s)
notifier_test.go:462: Channel  7 contains   0 message(s)

Here, fix the problem by putting in different symbols for the
notification send and subscription loops. The proper order of operations
to finish the test are:

  1. Wait for all subscription groups to finish.
  2. Tell the notification send loop to shut down.
  3. Wait for the notification send loop to return (so as not to introduce
    any goroutine leaks like had been present before).

Fixes #268.

While I was trying to fix another problem in the notifier's subscription
stress test, I ended up introducing another one by using a single wait
group for both the notification sending goroutine and subscriptions
loops which caused the test intermittency described in #268. The test
was ending too soon, and channels would only receive 1-2 messages, and
sometimes zero (which would fail the test).

    notifier_test.go:462: Channel  0 contains   2 message(s)
    notifier_test.go:462: Channel  1 contains   2 message(s)
    notifier_test.go:462: Channel  2 contains   2 message(s)
    notifier_test.go:462: Channel  3 contains   2 message(s)
    notifier_test.go:462: Channel  4 contains   2 message(s)
    notifier_test.go:462: Channel  5 contains   2 message(s)
    notifier_test.go:462: Channel  6 contains   2 message(s)
    notifier_test.go:462: Channel  7 contains   0 message(s)

Here, fix the problem by putting in different symbols for the
notification send and subscription loops. The proper order of operations
to finish the test are:

1. Wait for all subscription groups to finish.
2. Tell the notification send loop to shut down.
3. Wait for the notification send loop to return (so as not to introduce
   any goroutine leaks like had been present before).

Fixes #268.
@@ -454,9 +455,9 @@ func TestNotifier(t *testing.T) {
}()
}

close(shutdown)
wg.Wait()
notifier.Stop()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notifier was already stopping in a t.Cleanup above, so this invocation wasn't necessary.

@brandur brandur requested a review from bgentry March 13, 2024 01:18
@brandur
Copy link
Contributor Author

brandur commented Mar 13, 2024

ty.

@brandur brandur merged commit 8d7cfa6 into master Mar 13, 2024
10 checks passed
@brandur brandur deleted the brandur-fix-subscribe-test branch March 13, 2024 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Intermittent test failure: TestNotifier/MultipleSubscribersStress
2 participants