Process to rotate Gluster bricks in 'Replicated' volume type cluster

[tonytheleg]

Hello,
I'm fairly new to Gluster and in working through some issues we had yesterday, I'm wondering if the approach we are taking to rotating bricks out of a cluster is the best approach so I thought I'd pose the question here.

To give some background of our use case:
We are using Gluster as a persistent storage solution for our monitoring stack running on Docker Swarm. We have a 3 node/brick cluster with a single replicated volume. Due to company policy/red tape/yada yada, these servers are ephemeral in that they expire every 60 days so we have to rotate them out. To implement this with Gluster, we have been creating a new brick, adding it to the cluster, than removing an old one, so that we get back to 3 replicas, and don't ever lose a server. This is similar to the steps outlined in Expanding Volumes and Shrinking Volumes except I am using the force flag to remove since the others didn't appear to work for replicated (at least that's what I remember from a few months ago, please feel free to call me out on any wrongness haha)

This week we hit some weirdness with files not healing despite the self-heal daemon running and triggering a full heal. The files did not say they were split brain, and despite getting it fixed using split-brain fix techniques, I can't quite pinpoint where our issue lies that it got in that state to begin with. Is there any suggestions on how we should do these rotations based on our use case? Should we add the new brick, make sure everything heals and then remove the old? Do we need to even wait to remove the old brick once the new one is added? Should we stop any processes or anything? I want to make sure our process can be removed from the "root cause" equation so we can keep poking at it and see if it occurs again.

While on the topic of healing, could you also clarify the heal output for me?

Brick server1:/gfs/test-volume_0
Number of entries: 0

Brick server2:/gfs/test-volume_1
Number of entries: 101
/95.txt
/32.txt
/66.txt

With the above example output, does this essentially say that the listed files on server2 do not match whats on the other servers and they need to be updated or perhaps they are missing? Is there any diagram/flowchart of whats happening between the bricks communication-wise in a replicated volume when healing is occuring? Just wanted to understand the nuts and bolts a bit more.

Thank you in advance!

[tonytheleg]

Based on more digging, I'm gonna go with Red Hat's methods outlined. I'm assuming this is the best approach and will mark this answered unless someone has a different opinion.

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-replacing_hosts#Replacing_a_Host_Machine_with_a_Different_Hostname