node stuck in pre_vote state

[RoadRunnr]

Describe the bug

Versions:

• ra: 2.7.0
• OTP: 26.2

I have a ra system where:

• node 0 is in pre_vote, at commit index 13,
• node 1 is leader, at commit index 39,
• node 2 is follower, at commit index 39,

The state of node 0 is not changing at all.

when I dump the state of the nodes I see this:

Node 0:

#{membership => voter,last_applied => 13,commit_index => 13,
  commit_latency => 24,
  cluster =>
      #{{ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-0.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
            #{status => normal,next_index => 1,voter_status => #{},
              query_index => 0,match_index => 0,commit_index_sent => 0},
        {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
            #{status => normal,next_index => 1,query_index => 0,
              match_index => 0,commit_index_sent => 0}},
  current_term => 1,
  voted_for =>
      {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-0.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'},
  leader_id => undefined,aux_state => undefined,
  cluster_change_permitted => true,
  cluster_index_term => {9,1},
  pending_consistent_queries => [],
  persisted_last_applied => 13,
  queries_waiting_heartbeats => {[],[]},
  query_index => 1,votes => 1,
  pre_vote_token => #Ref<0.364221713.3290693633.220725>}

Node 1:

#{membership => voter,last_applied => 45,commit_index => 45,
  commit_latency => 25,
  cluster =>
      #{{ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
            #{status => normal,next_index => 1,query_index => 0,
              match_index => 0,commit_index_sent => 0},
        {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-2.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
            #{status => normal,next_index => 46,voter_status => #{},
              query_index => 41097,match_index => 45,
              commit_index_sent => 45}},
  current_term => 1,
  voted_for =>
      {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'},
  leader_id =>
      {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'},
  aux_state => undefined,cluster_change_permitted => true,
  cluster_index_term => {9,1},
  pending_consistent_queries => [],
  persisted_last_applied => 45,
  queries_waiting_heartbeats => {[],[]},
  query_index => 41097,
  pre_vote_token => #Ref<0.3712892406.1947992065.59241>,
  previous_cluster =>
      {0,0,
       #{{ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
             #{status => normal,next_index => 1,query_index => 0,
               match_index => 0,commit_index_sent => 0}}}}

Node 2:

#{membership => voter,last_applied => 45,commit_index => 45,
  commit_latency => 25,
  cluster =>
      #{{ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
            #{status => normal,next_index => 1,query_index => 0,
              match_index => 0,commit_index_sent => 0},
        {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-2.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'} =>
            #{status => normal,next_index => 1,voter_status => #{},
              query_index => 0,match_index => 0,commit_index_sent => 0}},
  current_term => 1,voted_for => undefined,
  leader_id =>
      {ergw_global,'ergw-c-node@smc-pgw-c-4g-pgw-1.smc-pgw-c-4g-pgw.4g-pgw.svc.cluster.local'},
  aux_state => undefined,cluster_change_permitted => true,
  cluster_index_term => {9,1},
  pending_consistent_queries => [],
  persisted_last_applied => 45,
  queries_waiting_heartbeats => {[],[]},
  query_index => 40977}

All 3 nodes have a consitent view for the Erlang node communication

Reproduction steps

I have no idea how to reproduce this.

Expected behavior

Node not stuck at pre_vote

Additional context

No response

[michaelklishin]

If you have no idea how to reproduce this, I'm afraid we have no idea how we can help. We do not guess in this community.

[michaelklishin]

Are there any logs from node 0 that can be shared? Without it we would have to guess what might have tripped the node during its second (third, etc) election cycle. Its index suggests that it could participate in an election once before.

The question is what has changed since then during the following rounds. Right now we have no information that would have shed some light on that. Not even a clue to build a hypothesis around.

[michaelklishin]

Also, since the current state of node 0 is all there is, and it won't affect logs, have you tried restarting node 0? Does it eventually get past pre_vote after a restart?

[RoadRunnr]

Unfortunately, there are no logs. ra uses a log domain and the default log configuration is so that anything that is not in the otp or ssl domain will not be logged. We'll be adjusting the log to get Ra logs should this ever happen again.

A restart of node 0 resolved the situation.

The nodes a running on kubernetes. That means that all of them come up with a black state at about the same moment. I could imagine that this hits some kind of race condition.

[michaelklishin]

@kjnilsson has spotted the smoking gun: node 0 is not a member of the same Raft cluster as nodes 1 and 2 at the time these state "snapshots" were taken.

[RoadRunnr]

has spotted the smoking gun: node 0 is not a member of the same Raft cluster as nodes 1 and 2 at the time these state "snapshots" were taken.

How would that be possible and how can you tell? All nodes are part of the same Erlang cluster and I have no idea how it would be possible to create a state where there are different Raft clusters.

[kjnilsson]

Provide the steps you took to cluster in the first place and any subsequent cluster changes you performed.