reindexEncrypted needs to be crash-resistant

[staltz]

Context: ssbc/ssb-box2#22

There could be a feature in ssb-db2's reindexEncrypted where it is crash-resistant. Since reindexEncrypted takes no arguments, this is just a matter of storing a boolean in some file (or just the file existence can be the "true" boolean) and then once ssb-db2 starts up it resumes reindexEncrypted if that file exists. Note also that reindexEncrypted does not need to be atomic. Any subset of encrypted records that got decrypted is some partial progress, and then after app restart, the progress can continue.

Also, another thing I notice is that it seems reindexEncrypted is not very efficient, at least not with jitdb. If there are encrypted records at seq 6, 14, 18, 22, and assume max seq is 100 then it'll reindex jitdb from seq 6–100, then reindex 14–100, then 18–100, then 22–100. It could just pick the minimum (6) and do 6–100 once.

[arj03]

About this:

Also, another thing I notice is that it seems reindexEncrypted is not very efficient, at least not with jitdb. If there are encrypted records at seq 6, 14, 18, 22, and assume max seq is 100 then it'll reindex jitdb from seq 6–100, then reindex 14–100, then 18–100, then 22–100. It could just pick the minimum (6) and do 6–100 once.

This assumes that something comes in and does a query in jitdb between the reindex calls. In jitdb we just reset the indexes so they will be re-created on next query. So if there is nothing concurrent this should only do 1 reindex from 6, right?

[staltz]

Ooooh

[staltz]

Yeah, so let's drop that jitdb optimization idea, you were right.

The "boolean" file in disk to trigger resume of reindexedEncrypted is still going to be relevant to implement.

[staltz]

I got a test written for this, and I got the implementation written correctly, but my implementation isn't passing the tests due to some odd issue with jitdb. Maybe one day we can try having a pair programming debugging.

[arj03]

Okay, maybe friday afternoon @staltz ? I'm a big hung up right now.

[staltz]

Sounds good. This isn't a blocker right now.