[EPIC] redesign KMS provider plugin

[cannischan]

• Run design session to determine how to redesign the plugin set up
• Write AC and requirements on this task

Sequence diagram

[romdalf]

Updated based on Design meeting held on June 9th 2022 with @cvlc @mhmxs @vfiftyfive @rovandep

• Multi KMS support only means one at a time
• Main space of Trousseau multi KMS is hybridish clouds, with Trousseau users have to have separate config only for Trousseau and not the entire KMS configuration per platform
• A migration helper would be nice, when user replaces KMS plugin, Trousseau should help migrating old secrets to new ([RFE] KMS migration helper #105)
• KMS plugins moves to separate sidecars ([RFE] KMS provider code to be running as a sidecars #103)
• KMS plugins provide a unix socket with a simple interface ([RFE] KMS provider code to be running as a sidecars #103)
• BYOKP: bring your own KMS plugin, users just mount any self implementation ([RFE] KMS provider code to be running as a sidecars #103)
• Once Vault moved to it's own sidecar, we should put main gRPC service into a sidecar:
  • There is an issue with unix sockets now. If Trousseau dies, but api server has a reference/bind mount to the old socket file, api server stops working. ([RFE] Trousseau pre-flight check for gRPC unix socket presence on crash #102)
  • So if we have an unbreakable simple process which creates unix socket, we can proxy requests to Trousseau. Why it is good? It makes product more robust ([RFE] gRPC code to be running as a sidecar  #104):
    • We have the same unix socket all the time, so crash won't break the entire cluster
    • We can try to reconnect before Kubernetes times out in the same connection
    • We can implement retry, failover, circuit breaker, whatever we want to find a ready instance as quick as possible