[Bug]: PrometheusRule can't be applied until Prometheus is up

[Ghiro94]

What version or versions you have tested?

No response

Which operating systems have you used?

• macOS
• Windows
• Linux

What did you expect to happen?

PrometheusRules gets created after vab apply

What happened instead?

The deployment of PrometheusRule generate timeout errors since it tries to install them before prometheus has been started. To fix this behavior it is required to comment the inclusion of the PrometheusRule, then deploy the monitoring module and finally apply it again with PrometheusRule enabled.

# 1st run with all enabled
...
Creating PrometheusRule: kube-apiserver-rules
Error: deploy of resources failed with error: error applying resource
...
Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://prometheus-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.1.15.29:443: connect: connection refused

# 2nd run with PrometheusRule disabled
Applying the configuration...
Creating Job: hubble-generate-certs
Creating ServiceMonitor: kubelet
Creating ServiceMonitor: prometheus-operator
Creating TLSStore: default
Creating ValidatingWebhookConfiguration: prometheus-operator
Creating ValidatingWebhookConfiguration: cert-manager-webhook

# 3rd run waited until prometheus is up and then enabled PrometheusRule
Applying the configuration...
Creating Job: hubble-generate-certs
Creating PrometheusRule: kube-apiserver-rules
Creating PrometheusRule: kube-prometheus-general-rules
Creating PrometheusRule: kube-prometheus-node-recording-rules
Creating PrometheusRule: kubelet-rules
Creating PrometheusRule: kubernetes-rules
Creating PrometheusRule: kubeserver-availability-rules
Creating PrometheusRule: node-exporters-rules
Creating PrometheusRule: node-rules

Code of Conduct

• I agree to follow this project’s Code of Conduct

[JGiola]

Hi @Ghiro94, can you confirm that if you run vab build the *WebhookConfiguration are not the latest resources in the list for the target cluster?

And just to be sure, if they are not the latest ones, after them they will appear only resources of custom CRDs?

[Ghiro94]

Hi @JGiola, actually the *WebhookConfiguration are the latest resources in the list of the target cluster.

[JGiola]

So in your example the first apply is not the first in absolute in the cluster, but a prometheus-operator instance is already deployed in the cluster correct?

[Ghiro94]

In my example in the 1st run there is no prometheus-operator installed. After I ran vab apply I got the error, then I disabled PrometheusRules and ran again vab apply . After that I enabled again PrometheusRules and i was able to intall them