addOOO then add feed leads to duplicate msgs

[staltz]

Context: working on ssbc/ssb-replication-scheduler#11 and noticed a duplicate message in the test results.

Problem: If you addOOO a message (e.g. the metafeed/announce msg from someone's main feed) and then replicate the main feed, you will add the metafeed/announce message twice.

Seems like OOO is not safe against these cases, and it doesn't seem like it's OOO's fault, because addOOO appended to the log when there was no message from that feed, the problem is add who appended the whole feed and this means it re-added the OOO msg.

Possible solutions? I think this issue is highly correlated to #395

cc @arj03

[arj03]

@staltz Yep that sounds like that problem. There are some solutions in that thread, we can have a call if that's helpful. Might be good to discuss the different solutions.

[staltz]

Random notes from our video call. Not cleaned up yet.

---

Partial repl => full repl
This happens in tests, but also when a hops 2 peer becomes a hops 1 (because we don't have votes indexed feed, and we need all the content from that peer).

Full repl => partial repl

If a pub wants to fully replicate main feed AND replicate indexed-feed msgs then ssb-ebt needs to be changed so that indexed replication DOESNT do addOOO.

When switching from full to partial, do we delete the main feed and then redownload portions of it via indexed feeds? Or do we delete only the ones that we will not replicate via indexed feeds? The former, please. Much simpler.

TODO: write down all the use cases.

We should have a dedicated subfeed for votes, and I publish my votes on both the main and the subfeed for votes. Whoever I replicate who has a metafeed tree I can assume they have this dedicated subfeed for votes.