Deleting PostgreSQL cluster leaves orphaned master endpoints

[Starefossen]

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.7.1
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

When deleting a PostgreSQL cluster the master endpoints are updated but not removed. This causes a cascading issue where you try to create the cluster again it will fail.

$ kubectl delete postgresql devops-keycloak-postgres
// some time later
$ kubectl get endpoints                                                                                  
NAME                              ENDPOINTS   AGE                                                                                                             devops-keycloak-postgres          <none>      6m36s                                                                                                           
devops-keycloak-postgres-config   <none>      6m36s                                                                                                           

When you try to create this database again you end up with the following error:

Events:                                                                                                                                                       
  Type     Reason  Age   From               Message                                                                                                           
  ----     ------  ----  ----               -------                                                                                                           
  Normal   Create  117s  postgres-operator  Started creation of new cluster resources                                                                         
  Warning  Create  117s  postgres-operator  could not create cluster: could not create master endpoint: could not create master endpoint: endpoints "devops-keycloak-postgres" already exists

Removing the orchaned endpoints and restarting the postgres-operator pod resolves the problem.

[oliveratprimer]

Also seeing this with v1.6.3

[bergey]

Deleting the cluster also leaves behind:

• Pod Disruption Budget, which blocks recreation
• Deployment for Pooler, which is surprising even if it does not block recreation

registry.opensource.zalan.do/acid/postgres-operator:v1.6.2

Happy to open new issues for one or both of these if consolidating with the endpoints issue is not appropriate.

[FxKu]

When you delete a healthy cluster, there should not be any resources left. Only, when a cluster is never fully synced - can be a new cluster never ending up in Running state or the operator was restarted and the existing cluster is already broken - it happens that there are leftovers.

We have discussions and PRs open to add ownerReferences or finalizers that might solve it. Unfortunately, we did not have the time to experiment enough to consider them as a save option we want. See this #941 for a good overview of the current state of discussions.