two-way Geo-replication

[derWalter]

Hi everyone,

I would like to start a discussion about how a two-way Geo-replication could work
and how this could get implemented.

(And I am interested to see if someone even looks into the discussion section here :) )

Greetings
Walter

[derWalter]

mhhh doesnt seem like some is looking inside of this section :P

[pranithk]

Looking, but not sure how it would work. So looking to see if someone has comments.

[derWalter]

nice, thx for your answer!

so I ll keep looking here 👍

[Shwetha-Acharya]

I would like to understand what is that you mean by two way Geo-replication.

1. Are you looking for one Geo-rep session to do simultaneous replication from location primary to location secondary(and vice versa)?
2. Or is about alternatively using the primary and secondary locations as source and destination? For example: you have a Geo-rep session where primary is at location A and secondary is at location B. Now you perform fail over to promote secondary to primary and use the old primary as the new secondary volume, and do this activity at regular interval of time.
3. Or is it about having two geo-rep sessions, one having the Primary volume at location A and Secondary volume at location B and other geo-rep session having Primary volume at loaction B and its secondary volume at location A?

[derWalter]

Wow, you think of very nice use cases!

But what I meant is perfectly explained
by you in list item #1

Have two geo distant servers, who get accessed locally
by clients and who need to update each other

Use case #2 and #3 are quite good examples,
how to make good use of such a functionality.

At the moment there is no solid solution if

1. latency is a problem
2. bandwidth is a problem
3. (as they are inter depended )both of the above are problems at the same time

[Shwetha-Acharya]

Considering the factors like latency and bandwidth, I think option 2 in my previous comment would be the safest and the best approach, though it includes, the extra step of failover at regular intervals.

First point mentioned by me was an hypothetical example with the current design, while the latter two are achievable as of now.

Tagging @aravindavk @kotreshhr @sunnyku to get their insights as well.

[Shwetha-Acharya]

Also we are ultimately having the replica of primary to secondary with our normal geo-rep session.
This means we are achieving the goal of creating backup of our data, at the geo-graphically seperated areas.

Both the clients on which primary and secondary are mounted are eventually going to have same data. So, I did not quite understand the need of creating one more back up with two-way replication, as we can already handle the data loss scenario with the help of replicated or dispersed volumes link

If you are talking about making changes(eg: creating files and directories) on secondary volume as well, and syncing it back to primary, this is not allowed with the current design. Secondary will be a read-only volume.

If you want to create one more back up of the secondary volume you can simply go with cascaded geo-rep setup link

[Shwetha-Acharya]

as we can already handle the data loss scenario with the help of replicated or dispersed volumes link

a regular replicated volume and the reads based on latency combined together can solve the latency issues as well. Thanks @mykaul for the pointers on reads based on latency.

[aravindavk]

The current Geo-replication uses Rsync to synchronise the data between Primary and Secondary. Geo-replication does the change detection by using Changelogs and directs Rsync to only sync those files. Using this approach will not help for two way replication if a file is changed in two sites then one might overwrite the other one.

I haven't thought about all the corner cases, but thinking loud here(Still need to work on identifying conflicts)

• Do not use rsync or use librsync for single file when required.
• Maintain remote_time as xattr in each file. After each sync update this xattr.
• After running the change detection, if local time is greater than remote_time then add to sync queue.
• Lock is required to prevent changing the local and remote file during the sync.
• Update remote data only if the file is not modified in remote site.
• On conflict, sync as specified in the policies(Latest wins, Bigger file wins etc).

[derWalter]

to me this sounds like a very nice thing to do 👍

[derWalter]

Oh, i totally forgot to bring the actual use case: two office teams, working on the same files at the same time, but on different locations. we are talking 15-20ms latency and 15/5mbit connections.

read/write on both ends, not talking backup here but live systems. I've found nothing for small scale (1-5 interlinked nodes)