Only run providers that are referenced in the policy

[blakerouse]

What does this PR do?

This updates the Elastic Agent to only run composable providers if they are referenced in the policy.

Why is it important?

This change provides the following benefits:

• Reduces the memory and load of the Elastic Agent, by only running the providers it references.
• No longer requires access to the kubernetes API or resources just to run an Elastic Agent (as long as its not using any ${kubernetes*` references in the policy).
• Fixes an issue where if no providers are running that a set of empty variables is passed to the coordinator.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test (covered by unit tests)

Disruptive User Impact

None. As it still runs all those that are referenced from the user.

How to test this PR locally

Run the elastic-agent container with just the system integration and notice that it doesn't require access to the Kubernetes API any more.

Related issues

• Closes Run only providers that have references in the policy #3609
• Closes Elastic Agent is waiting for initial configuration forever when all providers are disabled #4648

[mergify]

This pull request does not have a backport label. Could you fix it @blakerouse? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

Looks good overall, had some questions about the design and intended control flow.

[elastic-sonarqube]

Quality Gate passed

Issues
5 New issues
1 Fixed issue
0 Accepted issues

Measures
0 Security Hotspots
85.8% Coverage on New Code
4.8% Duplication on New Code

See analysis details on SonarQube