Introduce Magic Migrate

[schneems]

The goal of magic migrate is to simplify versioned metadata storage.

The problem

Cloud native build packs use toml to store metadata about a layer between builds. We commonly use this to store information like what ruby version was downloaded or a sha of some kind. On the next build we can look at this information to determine if some expensive process (like downloading a binary) can be skipped. Essentially we treat it like a cache key.

If we cannot load (deserialize) the old metadata into the currently requested structure then the default behavior is to clear the cache. This means that either the programmer must be careful to never make backwards incompatible changes to the metadata, or risk triggering a cache invalidation.

Now consider that we cannot guarantee that the cache was generated from the last version of the buildpack. Someone might deploy, then wait several years before deploying again.

The classic Ruby buildpack has this problem. It's cache is unversioned so if a mistake is made in the cache structure or contents in one version, then the fix must be hardcoded and checked on every subsequent deploy of every future version https://github.com/heroku/heroku-buildpack-ruby/blob/453b13983b638d68d9d65ab89d36a2fc18128e4a/lib/language_pack/ruby.rb#L1270-L1332.

Introducing magic migrate

Magic migrate doesn't make these problem go away, instead it makes the problem easier to reason about.

When the schema of the metadata changes, the programmer can introduce a new struct and tell rust how to migrate from one version to the next using either From or TryFrom (if fallible). Then they use the corresponding magic migrate trait to tell rust how to walk this chain backwards. Now when we try to load data from disk it will try to load the latest struc, if it can't it will go to the one before, and so-on. Once it finds the original serialized struct, it converts it forwards, one step at a time until we arrive at the currently desired struct.

Now instead of trying to hold all possible cache state in mind from all possible versions of the code, the programmer only needs to know how to make each conversion one by one.