Would a tiny ARMI DB be a useful addition for testing?

[opotowsky]

In this PR, @john-science brought up the fact that a small DB could improve testing for CLI endpoints.

But I think having a DB might be beneficial for simplifying/improving many tests, or even adding new ones.

Is this a worthwhile effort? How small can we make this and have it still be useful? Feelings aren't hurt if this is shot down, just didn't want to lose the thought in a merged PR.

@keckler @albeanth @mgjarrett @onufer

[albeanth]

Not sure if this sort of check already exists, but I think it would be useful to include a small database to ensure backwards compatibility with the most recent tag. E.g., on CI for each PR we could run

db = databaseFactory(pathToH5, "r")
with db:
    r = db.load(0, 0, allowMissing=True)

[john-science]

I realized that we have several tests that already have an ARMI DB for testing, they build it on the fly from the test reactor:

armi/armi/bookkeeping/db/tests/test_database3.py

Lines 255 to 267 in 2dab122

	dbi = DatabaseInterface(r, cs)
	dbi.initDB(fName="reloadingDB.h5")
	db = dbi.database
	# populate the db with something
	for cycle, node in (
	(cycle, node) for cycle in range(numCycles) for node in range(numNodes)
	):
	r.p.cycle = cycle
	r.p.timeNode = node
	r.p.cycleLength = 2000
	db.writeToDB(r)
	db.close()

I don't really love the idea of having a static binary file in the repo that can just go stale and bad at any time.

I think whoever wrote all these DB tests had the right idea.

[albeanth]

@john-science could these tests be configured to test backwards compatibility? Given that downstream analyses can include retrieving information off of a database from a previous tagged release, I think it's important to be able to ensure backwards compatibility.

I think the idea of a static binary file going "stale" or "bad" is the whole idea behind the test. If it goes "bad" then we've broken backwards compatibility and either need to 1) provide a conversion/migration script for affected versions, or 2) stop supporting those releases.

What do you think?

[jakehader]

I think it might be interesting to consider serializing / deserializing the database into a text based format for the purposes of testing and doing comparisons of differences.

This would mean that the database interface and database 3, would need to expand to support say a .json format but in theory in could be done. I think the critical issue with database changes within the API is related to parameter changes (adding new ones that don't exist in a previous version or changing names). There may be other hdf5 dependency impacts over time or h5py changes, but those should be infrequent.

If it's the parameter state that is changing between two versions it might be easier to create an application based parameter test that on the first time run would create a catalog of the parameter definition metadata and store that into a file (if enabled/configured). If the file already exists then a test would regenerate the state information for the new code version, compare to an existing version and report pass/fail. If it's the first time an application runs this then it would just create the file and the application owner could decide to track it or not. Maybe not an elegant solution, but maybe lighter weight than storage a full database and trying to put parameter data on it that could become stale.

[john-science]

Various Thoughts

1. Old Migration Scripts

Backwards compatibility in ARMI right now is pretty hindered. We already have some backwards migrations scripts for all versions of the code:

https://github.com/terrapower/armi/tree/main/armi/scripts/migration

But I don't have any example DBs to test those on. Not even OUTSIDE the repo.

2. New Migration Scripts

We have some new migration scripts in-house I considered brining into ARMI, but they are not all general enough to work for a wider audience.

3. Including Migration Scripts in ARMI

I'm happy to have the ARMI migration scripts in the ARMI repo. But the fact that they are INSTALLED and importable as part of ARMI is strange and awkward. We will never have code coverage on them, so why bother?

I would prefer to have them outside the installable armi/ package/directory.

I have even considered making them their own repo, so we can include nice, full DBs with them.

[onufer]

Late to the party. I like the idea. the main reason people don't like this sort of idea is database could be big and make git more bulky. i trust folks to make it small enough.

even if its only 3 assemblies its at least something to do more tests on. it also might help us see backwards compatibly problems sooner.