Investigate why MemberReplace failpoint flakes on release-3.4

[serathius]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

In last robustness meeting we identified 3 flakes for memberReplace

• https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-robustness-release34-amd64/1858425623123333120
• https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-robustness-release34-amd64/1856613465200791552
• https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-robustness-release34-arm64/1858782506342420480

All happening on release-3.4 and TestRobustnessExploratory/KubernetesHighTraffic/ClusterOfSize3/MemberReplace test

What did you expect to happen?

Issue not being specific to release-3.4

How can we reproduce it (as minimally and precisely as possible)?

There is no way to select failpoints via test name, but you can modify allFailpoints in test/robustness/failpoint/failpoint.go to leave only MemberReplace

And run it with GO_TEST_FLAGS='-v --run TestRobustnessExploratory/KubernetesHighTraffic/ClusterOfSize3 --count 100 --failfast --timeout 1h' make test-robustness-release-3.4

Anything else we need to know?

No response

Etcd version (please run commands below)

release-3.4 branch

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[joshuazh-x]

I can take a look at this.

[joshuazh-x]

Without PR #11639, MemberList returns local membership configuration without linearizable guarantee. The removed member may show up in the member list response. The issue is fixed in 3.5 and above so it shall be specific to 3.4.

Release 3.4

etcd/etcdserver/api/v3rpc/member.go

Lines 90 to 93 in 435ac80

	func (cs *ClusterServer) MemberList(ctx context.Context, r *pb.MemberListRequest) (*pb.MemberListResponse, error) {
	membs := membersToProtoMembers(cs.cluster.Members())
	return &pb.MemberListResponse{Header: cs.header(), Members: membs}, nil
	}

Release 3.5

etcd/server/etcdserver/api/v3rpc/member.go

Lines 90 to 98 in 601a884

	func (cs *ClusterServer) MemberList(ctx context.Context, r *pb.MemberListRequest) (*pb.MemberListResponse, error) {
	if r.Linearizable {
	if err := cs.server.LinearizableReadNotify(ctx); err != nil {
	return nil, togRPCError(err)
	}
	}
	membs := membersToProtoMembers(cs.cluster.Members())
	return &pb.MemberListResponse{Header: cs.header(), Members: membs}, nil
	}

[ahrtr]

Without PR #11639, MemberList returns local membership configuration without linearizable guarantee. The removed member may show up in the member list response.

Thanks for the analysis. One workaround is to issue a linearizable read request in between for 3.4.

[serathius]

Didn't we want to backport it to v3.4? #11639 (comment)

[ahrtr]

Better not since it changes the proto buffer, also we haven't seen any related production issue in 3.4 so far. But I am not strongly against it as it's a compatible change, see #11639 (comment)