Added waggledance metrics with tags for enabling filtering in datadog

[DhrubajyotiSadhu]

📝 Description: Metrics have been incorporated into Waggle Dance with the inclusion of tags, which will facilitate filtering within Datadog.

🔗 Related Issues

fixes #234

[massdosage]

I have a question at a higher level - how many tags will this add to each metric? Bear in mind that Datadog calculates the cost of custom metrics (more or less) as: "metric name X host X tag" so each tag you add is going to increase your custom metric costs. I'm just checking whether this will cause a cost issue or the cardinality of the tag values will be low enough that this won't be an issue.

[DhrubajyotiSadhu]

I have a question at a higher level - how many tags will this add to each metric? Bear in mind that Datadog calculates the cost of custom metrics (more or less) as: "metric name X host X tag" so each tag you add is going to increase your custom metric costs. I'm just checking whether this will cause a cost issue or the cardinality of the tag values will be low enough that this won't be an issue.

Hi @massdosage, thanks for the question
Our costs will remain consistent since we won't be sending all these metrics to Datadog; we'll utilize EKS annotations to filter them out. Additionally, the Datadog team confirmed yesterday that the expense for having 5 metrics without tags will be equivalent to having 1 metric with 5 tags.