Support creating a single k8s cluster across multiple Proxmox clusters

[aitorpazos]

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Proxmox's Corosync quorum mechanism puts a limit on the number of hypervisors that can join a Proxmox cluster. Proxmox staff recommends here to keep them of "reasonable size" (unofficially this seems to be considered around 25 nodes although we run larger clusters ok) and promote using a multi-datacenter approach.

Once #210 is released, it would be great to be able to deploy a single kubernetes cluster across multiple Proxmox clusters. This would allow to support larger kubernetes clusters if needed and be resilient to a Proxmox cluster failure if more than 3 of them are used.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-proxmox version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[mcbenjemaa]

Although this is a vast topic, it would be great if we support it.

How can we select a proxmox cluster for an upcoming machine, and under which criteria?
And do you know how we can manage the maintenance of the machines?
Do you know if we can migrate the machine to other clusters?

[aitorpazos]

How can we select a proxmox cluster for an upcoming machine, and under which criteria?

Could we use a similar mechanism that can be used atm with MachineDeployments referencing ProxmoxMachineTemplate? ie: we would have different MachineDeployments for the nodes in the different Proxmox clusters.
Maybe this could be supported through a new CRD (eg: ProxmoxGroupCluster) as supporting simple and composed use cases in same object may be messy 🤔.

And do you know how we can manage the maintenance of the machines?

As it is done when having multiple MachineDeployment in a given cluster.

Do you know if we can migrate the machine to other clusters?

It is probably ok to not support this.

[mcbenjemaa]

Maintenance is crucial, when someone needs to do various things on those Proxmox nodes, you expect that your workloads keeps running and not be down for the maintenance window, that's why we need a maintenance plan for this topic.

[aitorpazos]

True, but what I was trying to say is that I would expect this to keep working as it currently does within MachineDeployment.