feat: update redb to v2 #2120
feat: update redb to v2 #2120
Conversation
Frando
changed the title
| let rtx = old_db.begin_read()?; | ||
| let wtx = new_db.begin_write()?; | ||
|
|
||
| migrate_table!(rtx, wtx, old::AUTHORS_TABLE, new::AUTHORS_TABLE); |
There was a problem hiding this comment.
why not a function?
There was a problem hiding this comment.
I failed to get the quite complex, nested generics right, and the macro doesn't need these.
|
FYI I tried this out locally, works fine except that now my store is stuck on 2.0 :-) |
|
We should probably wait a bit more with merging this in case there are some issues with redb v2. One week or so? |
|
I'm wondering - would this also be a good time to remove old db migrations in iroh-sync, and panic with a message like "You need to update to iroh v0.12 before updating to v0.13"? |
seems reasonable |
| let (key, values) = entry?; | ||
| let key = key.value(); | ||
| if i > 0 && i % 1000 == 0 { | ||
| info!(" {name} {i:>ind$}/{len}"); |
There was a problem hiding this comment.
Dunno how it is supposed to work, but it looks like this
|
I looked at redb issues. Nothing was opened in the last week, and also no closed issues related to the 1.0 -> 2.0 transition. So maybe now is time to merge it? |
## Description `redb` 2.0.0 was released: https://github.com/cberner/redb/releases/tag/v2.0.0 most relevant for us: > 2.0.0 uses a new file format that optimizes len() to be constant time. This means that it is not backwards compatible with 1.x. To upgrade, consider using a pattern like that shown in the [upgrade_v1_to_v2](https://github.com/cberner/redb/blob/222a37f4600588261b0983eebcd074bb69d6e5a0/tests/backward_compatibility.rs#L282-L299) test. and some nice API changes that will simplify the iroh-sync codebase (no more ouroboros for self-refential structs needed!) > * Remove lifetimes from read-only tables > * Remove lifetime from WriteTransaction and ReadTransaction and some further API changes, see the notes. --- * [x] Update iroh-bytes to redb v2 * Did some crude search&replaces to remove now unneeded lifetime bounds. Compiles now, let's see what test says. * [x] Migration for iroh-bytes db * Code adapted from here https://github.com/n0-computer/migrate-bao-store-redb/blob/main/src/main.rs * [x] Update iroh-sync to redb v2 * This will be a bit more involved as iroh-bytes because we hit more API changes, but also great because we can remove `ouroborous` because iterators can now own a range * [x] Migration for iroh-sync db The migrations likely need a dependency to redb v1 to read the old database, so at least for one release cycle we'll have to depend on both redb v2 and v1. ## Notes & open questions This currently includes an extra-safeguard by preserving the old databases for sync and blobs as `docs.redb.backup-redb-v1` and `blobs.db-backup-redb-v1` - do we want this or delete the backups after successfull migration? ## Change checklist - [x] Self-review. - [ ] Documentation updates if relevant. - [ ] Tests if relevant. --------- Co-authored-by: Friedel Ziegelmayer <[email protected]>
Description
redb2.0.0 was released: https://github.com/cberner/redb/releases/tag/v2.0.0most relevant for us:
and some nice API changes that will simplify the iroh-sync codebase (no more ouroboros for self-refential structs needed!)
and some further API changes, see the notes.
ouroborousbecause iterators can now own a rangeThe migrations likely need a dependency to redb v1 to read the old database, so at least for one release cycle we'll have to depend on both redb v2 and v1.
Notes & open questions
This currently includes an extra-safeguard by preserving the old databases for sync and blobs as
docs.redb.backup-redb-v1andblobs.db-backup-redb-v1- do we want this or delete the backups after successfull migration?Change checklist