controlplane pods not scraped in rke2

[mjnagel]

Environment

Device and OS: AWS ec2
App version: 0.33.1 (snapshot)
Kubernetes distro being used: RKE2
Other: Hardened with CIS profile

Steps to reproduce

Deploy uds-core on rke2 cluster.

Expected result

Prometheus is able to scrape all configured targets.

Actual Result

3 controlplane targets show as down, with errors of connection refused:

• etcd
• kube-controller-manager
• kube-scheduler

This may be partially related to network policy restrictions in my environment similar to this issue.

Severity/Priority

Low/Medium - unsure of the effect of these targets being down.

Additional Context

There is a resolved issue and write up in this issue in Big Bang. The TLDR there is that these services are only listening on localhost, so connections from Prometheus get dropped. Unclear if this is RKE2 specific or more common across other distros.