EVM Relayer DB migrations PoC #13617

krehermann · 2024-06-18T20:19:16Z

Ticket

The point of this PR is to propose a mechanism for DB migrations that are scoped to schema.

The motivation is that (one day) LOOPPs will need to own and manage migrations.

The scope of this example is:

Pattern for templating migrations that need to be resolved and run for multiple instances (think evm relayer per chain)
- supports both go and sql migrations
Isolation of the migrations so that one day the can be moved out of this repo
- see core/store/migrate/plugins for all the feature work
Dynamic schema creation by the resource that needs it ie the evm relayer in this case
- there are some dirty hacks to make that happen in this PoC, but point is to illustrate the moving parts
- in this case creating an evm relayer will run the migrations specific to that chain id
CLI plumbing to manage the lifecycle of the dynamic migrations
- in this limited example we have node db migrate|rollback|status|version --relayer evm -chain-id <id>

> ./chainlink node db status --relayer evm --chain-id 42
time=2024-06-18T16:05:45.964-06:00 level=INFO msg="    Applied At                  Migration"
time=2024-06-18T16:05:45.964-06:00 level=INFO msg="    ======================================="
time=2024-06-18T16:05:45.966-06:00 level=INFO msg="    Pending                  -- 0001_create_schema.tmpl.sql"
time=2024-06-18T16:05:45.967-06:00 level=INFO msg="    Pending                  -- 0002_initial.go"
> ./chainlink node db migrate --relayer evm --chain-id 42
2024-06-18T16:05:55.281-0600 [INFO]  Migrating database: "postgres://chainlink_dev:insecurepassword@localhost:5432/chainlink_development_test?sslmode=disable" cmd/shell_local.go:1008          version=2.13.0@dbd5cf7 
Generated migrations: [/var/folders/91/h_s80sjd1f3267zl7pgdlvd40000gn/T/evm_42/42/0001_create_schema.sql]
time=2024-06-18T16:05:55.321-06:00 level=INFO msg="OK   0001_create_schema.sql (5.81ms)"
time=2024-06-18T16:05:55.358-06:00 level=INFO msg="OK   0002_initial.go (37.12ms)"
time=2024-06-18T16:05:55.358-06:00 level=INFO msg="goose: successfully migrated database to version: 2"
> ./chainlink node db status --relayer evm --chain-id 42
time=2024-06-18T16:06:07.394-06:00 level=INFO msg="    Applied At                  Migration"
time=2024-06-18T16:06:07.394-06:00 level=INFO msg="    ======================================="
time=2024-06-18T16:06:07.398-06:00 level=INFO msg="    Tue Jun 18 22:05:55 2024 -- 0001_create_schema.tmpl.sql"
time=2024-06-18T16:06:07.399-06:00 level=INFO msg="    Tue Jun 18 22:05:55 2024 -- 0002_initial.go"
> ./chainlink node db version --relayer evm --chain-id 42
2024-06-18T16:06:14.150-0600 [INFO]  EVM database version: 2                            cmd/shell_local.go:1081          version=2.13.0@dbd5cf7

Testing infra for the database
- ability to programmatically instantiate the correct db resource (ie the correct schema with the right tables)
  - this is tricky because all of the existing tests assume a properly configured db external to the tests
  - that can't work when the schemas are dynamic
    - if one test use evm relayer chainid=7 and other chainid=11, they have different resource requirements
  - this is solved by refactoring our txdb and introducing an evm-specific helper to instantiate the correct db schema
Proof that something like this can work
- we refactor the evm forwarder to use all of this machinery (this was a arbitrary choice, informed by its relatively simple ORM)
  - update the SQL statements to depend on the runtime schema
  - remove evm_chain_id from the tables
  - update test to use the evm-specific DB helper so that the correct schema is generated for the test

Out of scope

test refactoring require pgtest to be available to the evm helper. this implies we need to put pgtest somewhere common
anything non-evm related. but the pattern ought to work for other relayers and applications
anything beyond the evm forwarder refactor to prove it can work.
clean initialization code in the relayer. db migrations need a sql.DB. the relayer only has a sqlutil.Datasource. we'll need to reconcile this to support dynamic migrations (or move the dynamic migrations up the stack)

Opens:
rollback

the dynamic migration version are decoupled from the core node version. in principle this is a gap because an operator would need to rollback each schema manually

Requires Dependencies

Resolves Dependencies

note: discovered wrongly schema'd eth_txs types

github-actions · 2024-06-18T20:20:28Z

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

#added For any new functionality added.
#breaking_change For any functionality that requires manual action for the node to boot.
#bugfix For bug fixes.
#changed For any change to the existing functionality.
#db_update For any feature that introduces updates to database schema.
#deprecation_notice For any upcoming deprecation functionality.
#internal For changesets that need to be excluded from the final changelog.
#nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
#removed For any functionality/config that is removed.
#updated For any functionality that is updated.
#wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

reductionista · 2024-06-21T21:50:07Z

core/web/evm_forwarders_controller.go

@@ -79,6 +92,13 @@ func (cc *EVMForwardersController) Delete(c *gin.Context) {
 		jsonAPIError(c, http.StatusUnprocessableEntity, err)
 		return
 	}
+	// this is a breaking change, but it's necessary to support multiple chains
+	chainId, err := stringutils.ToInt64(c.Param("chainID"))


I don't know much about the forwarder, but if we didn't support multiple chains before... how did it work, you could only use forwarders on the default chain or something?

no, we were leaking the DB id and expecting user to use that. that DB id was unique b/c all the forwarders were in one table

reductionista · 2024-06-26T18:14:18Z

core/store/migrate/plugins/relayer/evm/testutils/db.go

+var evmInitialState string
+
+// NewEVMDB creates a new EVM database with the given schema and chainId
+func NewDB(t testing.TB, cfg evm.Cfg) *sqlx.DB {


Name of function in comment should match name of function

reductionista · 2024-06-26T18:45:28Z

core/services/relay/evm/evm.go

+	err = evmdb.Migrate(context.Background(), db, evmdb.Cfg{
+		Schema:  "evm_" + chain.ID().String(),
+		ChainID: ubig.New(chain.ID()),
+	})


Have we thought through what happens if the node gets shut down while the relay is migrating? Is completing the migration what we want, or would it be better to abort?

reductionista · 2024-06-26T18:50:38Z

core/store/migrate/plugins/relayer/evm/0002_initial.go

+	goose.AddMigrationContext(upFunc, downFunc)
+	return nil
+}
+*/


TODO: remove if unused

reductionista · 2024-06-26T19:05:51Z

core/store/migrate/plugins/relayer/evm/resolver.go

+	if err != nil {
+		return nil, err
+	}
+	b := make([]byte, 1024*1024)


I'm not sure how to do this cleanly off the top of my head, but I assume once we get past the initial draft stage, we'd want to make this able to handle arbitrarily large migration files. If not, we should at least make it a constant or config option so we can increase it later if needed.

reductionista · 2024-06-26T19:19:25Z

core/store/migrate/plugins/relayer/evm/migrate.go

+}
+*/
+// Migrate migrates a subsystem of the chainlink database.
+// It generates migrations based on the template for the subsystem and applies them to the database.


TODO: update comment to be evm specific (I assume "chainlink database" will no longer be a meaningful concept, right?)

reductionista · 2024-06-26T19:23:30Z

core/store/migrate/plugins/relayer/evm/migrate.go

+			return fmt.Errorf("failed to generate migrations for opt %v: %w", cfg, err)
+		}
+		fmt.Printf("Generated migrations: %v\n", migrations)
+	*/


Getting a little confused about all of the commented code. I see setupPluginMigraitons is commented out, as well as Register0002, then this portion of Migratewhich callsgenerateMigrationsis commented, even thoughgenerateMigrations` is uncommented? Which of these will we end up using, or are they all still WIP?

reductionista · 2024-06-26T19:29:42Z

core/services/chainlink/relayer_factory.go

@@ -72,6 +74,7 @@ func (r *RelayerFactory) NewEVM(ctx context.Context, config EVMFactoryConfig) (m

 		relayerOpts := evmrelay.RelayerOpts{
 			DS:                   ccOpts.DS,
+			DBURL:                &dburl,


Cool, so... we tell all relays to use the databse url in the global toml config (config.AppConfig.Database().URL()) by default, but allow it to be overridden on a per-relay basis by reading the CL_DATABASE environment variable that's set locally on each running relay? I think that makes sense

reductionista · 2024-06-26T19:33:22Z

I don't see any big problems with this approach, seems like some good progress in the right direction. Left some minor comments and questions, but most of them may not be too relevant at this stage as it's such an early draft--I assume more of the details will be filled in as it progresses.

cl-sonarqube-production · 2024-06-27T00:57:15Z

Quality Gate failed

Failed conditions
30.7% Duplication on New Code (required ≤ 10%)
C Reliability Rating on New Code (required ≥ A)
35 New Major Issues (required ≤ 5)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

github-actions · 2024-09-08T00:16:25Z

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

krehermann added 19 commits June 13, 2024 17:55

hacking

7274c52

make resolved, optional migrations succeeded. TODO rollback interleaving

c0e1d00

reorg

2bbfbbd

subsytem migration wip

b5566ef

plugin migration and commit-ordered manifest

8931f16

rename template dir

c48e999

manifest wip

eb3ee09

rollback; optional plugin args to cli

a5a3fd3

clean up internal and isolate manifest

c24b885

real evm

85ea68c

realistic test for parameterized evm schema.

3d9b1f1

note: discovered wrongly schema'd eth_txs types

more testing

5fe7e39

realistic evm migration working in unit test

4000d64

cli plumbing

2cd82c9

hack txdb to run evm migrations on the fly

dad8f5c

initialize evm test db per cfg; refactor txdb to use with it

0bc3f9d

cleanup and comment

c9d923e

rm manifest

472f44c

update forwarder ORM to be schema-specific

051dba7

krehermann temporarily deployed to sdlc June 18, 2024 20:19 — with GitHub Actions Inactive

hack db into NewRelayer for PoC

dbd5cf7

krehermann temporarily deployed to sdlc June 18, 2024 20:56 — with GitHub Actions Inactive

krehermann temporarily deployed to sdlc June 18, 2024 21:25 — with GitHub Actions Inactive

clean up the cli

1715ce5

krehermann force-pushed the bcf-3266/db-migration-poc branch from 6188659 to 1715ce5 Compare June 18, 2024 21:31

krehermann temporarily deployed to sdlc June 18, 2024 21:31 — with GitHub Actions Inactive

krehermann temporarily deployed to sdlc June 18, 2024 21:50 — with GitHub Actions Inactive

krehermann requested review from jmank88 and dimriou June 18, 2024 22:44

krehermann force-pushed the bcf-3266/db-migration-poc branch from 6427ad9 to b73cfaf Compare June 21, 2024 18:10

krehermann temporarily deployed to sdlc June 21, 2024 18:10 — with GitHub Actions Inactive

krehermann force-pushed the bcf-3266/db-migration-poc branch from b73cfaf to 9af599a Compare June 21, 2024 20:29

krehermann temporarily deployed to sdlc June 21, 2024 20:29 — with GitHub Actions Inactive

wrestling with too many connections

76fc0b8

krehermann force-pushed the bcf-3266/db-migration-poc branch from 9af599a to 76fc0b8 Compare June 21, 2024 21:13

krehermann temporarily deployed to sdlc June 21, 2024 21:13 — with GitHub Actions Inactive

reductionista reviewed Jun 21, 2024

View reviewed changes

reductionista reviewed Jun 26, 2024

View reviewed changes

refactor for multiple go migrations and add test fixtures

3d2f239

krehermann temporarily deployed to sdlc June 26, 2024 23:09 — with GitHub Actions Inactive

krehermann had a problem deploying to sdlc June 26, 2024 23:26 — with GitHub Actions Failure

add log_poller_blocks migration and tests

c2b2a28

krehermann force-pushed the bcf-3266/db-migration-poc branch from 977375d to c2b2a28 Compare June 26, 2024 23:35

krehermann temporarily deployed to sdlc June 26, 2024 23:35 — with GitHub Actions Inactive

add log poller filters migrations and tests

8726b51

krehermann temporarily deployed to sdlc June 27, 2024 00:11 — with GitHub Actions Inactive

krehermann added the DO_NOT_MERGE label Jul 9, 2024

krehermann changed the title ~~hacking~~ EVM Relayer DB migrations PoC Jul 9, 2024

github-actions bot added the Stale label Sep 8, 2024

github-actions bot closed this Sep 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVM Relayer DB migrations PoC #13617

EVM Relayer DB migrations PoC #13617

krehermann commented Jun 18, 2024 •

edited

Loading

github-actions bot commented Jun 18, 2024

reductionista Jun 21, 2024

krehermann Jun 26, 2024

reductionista Jun 26, 2024

reductionista Jun 26, 2024

reductionista Jun 26, 2024

reductionista Jun 26, 2024

reductionista Jun 26, 2024

reductionista Jun 26, 2024

reductionista Jun 26, 2024

reductionista commented Jun 26, 2024

cl-sonarqube-production bot commented Jun 27, 2024

github-actions bot commented Sep 8, 2024

EVM Relayer DB migrations PoC #13617

EVM Relayer DB migrations PoC #13617

Conversation

krehermann commented Jun 18, 2024 • edited Loading

Ticket

Requires Dependencies

Resolves Dependencies

github-actions bot commented Jun 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reductionista commented Jun 26, 2024

cl-sonarqube-production bot commented Jun 27, 2024

Quality Gate failed

github-actions bot commented Sep 8, 2024

krehermann commented Jun 18, 2024 •

edited

Loading