Merge db voters and votes collections

[dsernst]

Work to merge the voters and votes collections, which was lingering technical debt. See #152 (comment).

This will unblock us to properly finish link-via-auth.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
siv	✅ Ready (Inspect)	Visit Preview	Oct 9, 2024 6:24am

[dsernst]

INITIAL NOTES:

We have this new db architecture design...
- merge voters and votes into approved-voters.

• We need the application code for the new design.
  • Update POST /api/add-voters
  • GET /api/check-auth-token
  • Update POST /api/submit-vote
    • anything that reads the current list of votes or voters:
      • GET /api/accepted-votes
      • POST /api/unlock-votes
      • POST /api/invalidates-votes
        • POST /unlock filter out invalidated votes
        • GET /invalidated-votes
        • GET /was-invalidated
    • grep collection('votes')
      • 19 hits (7 are scripts)
    • grep collection('voters')
      • 20 hits (4 are scripts)
    • grep for collection('invalidated_votes')

Migration Script

• We need a migration script to convert the current database data into the new format.
  • We need to write the migration script
  • We need to test the migration script
    • Maybe want backup in case the migration script corrupts data?
  • We need to run the migration script for production data
  • Do we want to provide instructions / migration script completion checks for other people running SIV...?
    • We decided for now this is Nice-to-have, but not Must-have, so not blocking

Is this a SemVer breaking change?

• siv-web (codebase). Current version: 0.1.0 [according to package.json]
  • This would a SemVer breaking change, but because we are in 0.x pre-release, we're technically not really promising that we're documenting these breaking changes right now, anyway.
• SIV Protocol https://docs.siv.org/technical-specifications
  • evaluated this is not a breaking change for this SIV Protocol document.

GAME PLAN:

• backup all the production data - Initialized: Sep 13, 2024, 10:32:06 PM

• update the production data for the new application code shape
  - using DB atomic transactions
  - [x] sketched out high-level overview of this script 2024-09-13-merge-db-voters-and-votes-collections.ts
  - [ ] db system to track running new migration scripts just once
  - [x] overview of what that would look like

• update all the application code

STRATEGY:

1. Migration script to update just a single election for the new shape.
2. Get all the application code working locally for that one election.
3. Update the migration script to be able to work for all elections in the DB.
4. Deploy the new application code.
5. Run the migration script over all the old elections.

Migration script strategy:

1. db.migrations - Track which migration scripts have already been run
   • unique per deployment (e.g. production vs different people's personal dev environments)

title (string)      | run_on (timestamp)
----------------------------------------
2024-09-13-merge... | 2024-11-10

2. run-migrations script (which could run whenever SIV starts up, e.g. post-install hook, or manually triggered by admin)
   grab list from db.migrations table
   grab files in /migrations folder

   Are there any new migration script files that the DB doesn't have a record of running already?
   if so, run them.
   ...if succeeded, store their success time in db.migrations