kv/kvclient/kvcoord: TestDistSenderReplicaStall failed

[cockroach-teamcity]

kv/kvclient/kvcoord.TestDistSenderReplicaStall failed on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.RunValues[...].func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) aborted in DistSender
        	            	Wraps: (3) result is ambiguous: context deadline exceeded
        	            	Wraps: (4) secondary error attachment
        	            	  | aborted during DistSender.Send: context deadline exceeded
        	            	  | (1)
        	            	  |   | (opaque error wrapper)
        	            	  |   | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
        	            	  |   | reportable 0:
        	            	  |   |
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:3113
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2192
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1741
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1260
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:224
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1133
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).send
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1116
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:965
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:988
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Inc
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:499
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test.TestDistSenderReplicaStall.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord_test/pkg/kv/kvclient/kvcoord/dist_sender_circuit_breaker_test.go:81
        	            	  |   | github.com/cockroachdb/cockroach/pkg/testutils.RunValues[...].func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23
        	            	  |   | testing.tRunner
        	            	  |   | 	GOROOT/src/testing/testing.go:1689
        	            	  |   | runtime.goexit
        	            	  |   | 	src/runtime/asm_amd64.s:1695
        	            	  | Wraps: (2) aborted during DistSender.Send
        	            	  | Wraps: (3) context deadline exceeded
        	            	  | Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) context.deadlineExceededError
        	            	Wraps: (5) context deadline exceeded
        	            	Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) *kvpb.AmbiguousResultError (4) *secondary.withSecondaryError (5) context.deadlineExceededError
        	Test:       	TestDistSenderReplicaStall/clientTimeout=true
    --- FAIL: TestDistSenderReplicaStall/clientTimeout=true (49.87s)

Parameters:

• attempt=1
• race=true
• run=2
• shard=7

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-44994

[arulajmani]

I'll remove the release blocker label given this is a DistSenderCircuitBreaker test, which we don't turn on by default.