Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(chain): Implement DB migration #132

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

lukechampine
Copy link
Member

This adds a MigrateDB function that converts a DB from one version to the next. Accordingly, the per-block and per-state versions have been removed.

I've tested this against walletd v0.8.0, though not exhaustively.

Currently there is no live feedback during migration. We could add logging or some other form of callback. We also discussed calling MigrateDB inside NewDBStore; I'm open to that, but it would require changing the function signature if we want logs as well, so we'd need to update a lot of callsites.

chain/migrate.go Outdated Show resolved Hide resolved
chain/migrate.go Show resolved Hide resolved
@lukechampine
Copy link
Member Author

Ok, new approach. Every key is suffixed with a version. (Prefix would also work fine, and might be better for sorting...hmm)
When we migrate, we check the version byte on every key to determine whether it needs updating.
The old DB keys don't have a version byte at all, though, so instead we check the key length. This is feasible because all the keys have a fixed length, unlike values which can vary in size. This is also why the version is on the key, not the value: if it was on the value, then migrating the old DB (which lacks version bytes) would be ambiguous: you wouldn't be able to tell if the byte was actually a version byte, or part of an old DB value.

I tested this manually by migrating a consensus.db, loading it, and reading from it. When I migrate it again, it's a no-op. When migration is interrupted and resumed, the already-migrated keys are skipped as intended.

@lukechampine lukechampine force-pushed the migrate-db branch 3 times, most recently from 80d719a to beda074 Compare December 16, 2024 16:44
// if the db is empty, initialize it; otherwise, check that the genesis
// block is correct
if dbGenesis, ok := dbs.BestIndex(0); !ok {
if version := dbs.bucket(bVersion).getRaw(dbs.vkey(bVersion)); len(version) != 1 {
Copy link
Member

@n8maninger n8maninger Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is currently panicking when trying to migrate a fresh Zen walletd v0.8.0 database: failed to create chain store: panic during database initialization: bucket already exists

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, steps to reproduce? I don't see that error; here's what I did:

  • git checkout v0.8.0 && go install ./cmd/walletd
  • walletd -network zen
  • Wait for a few thousand blocks, then Ctrl-C
  • In a test function, call coreutils.OpenBoltChainDB, then chain.MigrateDB, then chain.NewDBStore

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did basically the same, but checked out walletd master with coreutils replaced. Let me double check that I didn’t screw up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

3 participants