Pulumi deployment

[scottyeager]

The idea is to create a mostly automated Pulumi script in Python that can do the following:

• User provides minimal necessary config and fully QSFS is deployed automatically
• User can change their backends set or front end VM node by changing the config and running pulumi up again
• It's possible to recover to a new frontend VM with minimal intervention
• Supports Prometheus, user can optionally supply a config file that gets automatically uploaded to the frontend VM if present (this can be used to forward Prometheus data to a remote system, rather than needing to scrape it from the VM)
  • Basic feature is there
  • Need to add an example config
• Enables end to end testing of various features in a fully automated way (replacing backends, recovering to new VM, etc...)
  • First test is complete but doesn't run consistently
  • Sometimes we get a timeout on gridproxy.grid.tf/ping when redeploying and that ruins the whole test. Maybe can retry a few times in that case?
  • Outputs during the recover step seem kind of inconsistent with no clear cause. This could just be backend instability since we have no reliable way to reach zdbs currently

Work is on: https://github.com/threefoldtech/quantum-storage/tree/pulumi