System Metrics Template Results in Very High Cardinality at Scale

[rickspencer3]

Template: linux_system

Describe the problem
When using this template at scale in a dynamic environment, for example Elastic Beanstalk, cardinality can grow dramatically. This is because the host is a tag, and if you have many immutable hosts being created constantly, there will be many thousands of values for the host tag.

Suggestion
A reasonable resolution for this is to store the raw data in a bucket with a short retention period (say, for example, 1 day) but then to downsample the data, dropping the host tag, and storing the downsampled data in a bucket with a longer retention period.