Shard message list and messages in a conversation

[dariusc93]

Blocks stored is limited to 1MB by default in ipfs (with it not exceeding 2MB for bitswap). This creates a problem as messages build up over time for the list of cid linking to the encrypted messages. Once the block exceeds the default limit, it will likely error as it would not be able to obtain that block with a higher size, but even if there was a way to increase the block size externally (which at the moment that isnt an option), the issue there is that we wont be able to transmit the block over bitswap as it would exceeds the spec limit, which makes it hard to transfer the block to another node (eg for offline or backup). With this, what we should eventually do is shard the dag containing the list of cids as it begins to increase its size to have it point to an alternative dag containing additional messages. This would allow us to stay below the block limit as messages in a conversation begins to increase, as well as if the message itself (which may require splitting if it exceeds a specific size as the recipient list increases) but be able to keep things linked together when fetching or searching the messages later on.

Scenarios:

• A conversation with 50 recipients that has a single message containing 2048 character message would produce an approx 1.4MB block.

• A conversation with a single recipient (2 recipients in the conversation list) with 4250 messages, produce an approx 935KB block. This exclude encrypted messages since the list contains a small metadata document showing timestamp, message and conversation id, and the cid pointing to the encrypted message).

Note:

1. Though future plans would reduce the data itself, sharding the block would reduce the single block size of the message if recipients will play a part in the encrypted message.
2. A large portion of this has been resolved with Keystore per conversation #171

[dariusc93]

The best way to properly handle this is to keep track of the size of the block and once it comes close to 1MB (or any defined size if we decide to increase or decrease it) to link it to the next block while while referencing the previous block in the new block.

[dariusc93]

Resolved in #437