Sampling Best Practices Guide

[bryan-aguilar]

I believe it would be a benefit to have clear documentation for trace sampling best practices. Users who are new to trace instrumentation will usually start by sampling 100% of spans. That is also the default behavior of the ADOT Collector. This can become an issue when trying to migrate their observability to solution to a large scale solution. Users could hit rate limits when exporting to x-ray back end and also face large resource constraints if sampling is not implemented properly.

To better onboard end users to sampling at scale, the best practices guide could have clear documentation or point to existing resources to answer questions like the ones listed below.

What is sampling?
What are the different types of sampling?
When should I sample?
What should I sample?
How do I sample in the OpenTelemetry ecosystem?