Arbiter in a strange condition

[hgshoggins]

Looking for help on a matter that leaves my cluster prone to split-brain. I have volumes spread over 2 bricks + 1 arbiter.
One morning, the arbiter was showing as State: Peer Rejected (Connected). So as suggested here, what I did was, on the arbiter brick:

1. Stop glusterd
2. In /var/lib/glusterd, delete everything except glusterd.info (the UUID file)
3. Start glusterd
4. Probe one of the good peers
5. Restart glusterd, check 'gluster peer status'

gluster peer status shows again as State: Peer in Cluster (Connected). Yay.

Now when I call gluster volume info stockage:

Volume Name: stockage
Type: Replicate
Volume ID: efd9dd36-c4b5-497e-87cf-a76da6577e48
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster-1.oob-servers.fqdn:/data/brick/stockage
Brick2: gluster-2.oob-servers.fqdn:/data/brick/stockage
Brick3: smallarbiter.oob-servers.fqdn:/data/brick/stockage (arbiter)
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

Should be good, right? Except when I call gluster volume heal stockage info, I get the dreaded Transport endpoint is not connected:

Brick gluster-1.oob-servers.fqdn:/data/brick/stockage
/pige/2023-09-05/2023-09-05-12.flac 
/pige/2023-09-05/2023-09-05-13.flac 
/pige/2023-09-05 
/pige/2023-08-06 
/pige/2023-09-05/2023-09-05-14.flac 
/pige/2023-09-05/2023-09-05-15.flac 
/pige/2023-09-05/2023-09-05-16.flac 
/pige/2023-09-05/2023-09-05-17.flac 
Status: Connected
Number of entries: 8

Brick gluster-2.oob-servers.fqdn:/data/brick/stockage
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05/2023-09-05-12.flac 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05/2023-09-05-13.flac 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-08-06 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05/2023-09-05-14.flac 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05/2023-09-05-15.flac 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05/2023-09-05-16.flac 
<gfid:9b60ada5-0cf5-4e71-90a2-41b9fe53f9c4>/2023-09-05/2023-09-05-17.flac 
Status: Connected
Number of entries: 8

Brick smallarbiter.oob-servers.fqdn:/data/brick/stockage
Status: Transport endpoint is not connected
Number of entries: -

And gluster volume status stockage gives a worrying output:

Staging failed on smallarbiter.oob-servers.fqdn. Error: Volume stockage does not exist

The cluster is drifting. Is there a safe way to reattach the arbiter properly and heal everything out?

[hgshoggins]

I have this in the logs:

Version of Cksums mail differ. local cksum = 3956038290, remote cksum = 2874151090 on peer gluster-2.oob-servers.fqdn

So gluster v get all cluster.max-op-version gives:

Option                                   Value                                  
------                                   -----                                  
cluster.max-op-version                   100000                      

but gluster volume set all cluster.op-version 100000 returns:

volume set: failed: Required op-version (100000) should not be equal or lower than current cluster op-version (100000).

It doesn't seem to be a matter of op-version. What could it be, and how to solve it?

[hgshoggins]

Aaah an interesting info is that my rejected peer is running glusterfs-11.0-3.fc38.x86_64 binaries, and the two other still working and talking to each other but unable to heal bricks are running glusterfs-10.4-2.fc37.x86_64. Although they are all on the same op-version, that could be the culprit, right?

[hgshoggins]

After having upgraded my two data nodes to latest 11.0.3, I ended up with a broken cluster with unavailable files on the clients, lots of Transport endpoint is not connected on them, etc.
And then I found this in the gluster-users mailing-list archives, talking about an optimization that was causing clusters to reject certain peers and not updating the volumes' cksums. So I remembered that I had an optimization activated on my volumes. Hey, my cluster is down, why not trying it?
So I disabled cluster.readdir-optimize with:

gluster volume set stockage cluster.readdir-optimize off

... and it did the trick after a restart of the glusterd service.

Now that all my nodes have a corresponding cksum for each held volume, I re-enabled the cluster.readdir-optimize feature and everything runs smooth.