Failsafe Migrations with Multi-Tenant DBs

[pgruener]

Hi,

Id like to inspect a possible setup with kamal for our application (which is not rails).
After reading the other discussions and docs, I am stuck with two topics, which I dont really recognize in the writings, or I just dont realize them:

1. Realization of a really fail-safe migration
   I just understand, there is a migration (not part of kamal, but illustrated by rails server start), which runs, and if it runs successfully the rollout continues.
   If it fails, the rollout stops.
   Thats what I understand.

But if it fails, the database is not reverted .. so the migration could have been run/changed some tables, which might break the old container image to work well with this new/incomplete schema.
Of course those situations should not happen, when the migration and the application is "done well" and tested, but I am just thinking about murphy.

So the only way to handle such a situation, is, as far as I know, to copy the DB, perform the migration in the new DB, and if that worked, the new containers will be started on the new db, while the old containers can be shutdown, same as the old DB.

This setup brings other limitations such as HA, but this is why I thought, I could get some ideas about such a situation in here. But explicitely for my case, a short downtime is not a big problem.

2. Handle multi-tenancy by multi-db (each (sub-)domain belongs to a customer, which relies to an own DB)
   We have a SaaS application, where each customer has its own domain and correlated database. So the previously described migration process would have to be performed for each single customer. By example 1000 times for 1000 customers. That's why I would not schedule that in a "whole deploy" process, but customer-by-customer, ich with a quick downtime.

If one or two of the migrations are failing, the older images may not have been deleted for the period until a fixed version did the upgrade fine.

Did anybody of you solve a scenario like that with kamal, or am I missing things, or am I just on a "wrong road"?

Would be great to get some insights or opinions.

Thanks alot