Exclude AWS API calls that do not cause errors or take below a certain amount of time

[kmcquade]

Is your feature request related to a problem?
Hi,

We enabled Boto instrumentation in OpenTelemetry but our flame graph is getting littered with a lot of requests. 99% of the time, it is too much information, but sometimes it is helpful to include a span about the AWS API calls. For example, if a call to S3 takes over a certain amount of seconds, or if there's downtime for an AWS API that affected a call.

I don't see anything in the docs (or in the code) for a way to suppress spans for AWS API calls unless (1) there are errors or (2) if the span takes above a certain amount of time. Some of the other packages have more configurability - like where the httplib package allows you to exclude certain URLs. Do you know if it's possible to do that with the botocore/boto/boto3sqs libraries?

Thanks so much in advance.

Describe the solution you'd like
See above

Describe alternatives you've considered
I am a bit new to OpenTelemetry so I am not sure if there's another way. Thought it would be better to ask

[kmcquade]

I think this is something that can be accomplished via filterprocessor? https://docs.honeycomb.io/send-data/opentelemetry/collector/#filtering-span-events-and-other-data

I will experiment and let others know if that works. If anyone else has some ideas, I'd love suggestions.

[emdneto]

Usually, people do that on the collector side. You can use filterprocessor + ottl in the collector or ottl condition in tail sampling processor. If you need to care about this on the app side, you can create a custom SpanProcessor for your use case and implement the logic you want.

[kmcquade]

Hi @emdneto. Here is what I tried in my Lambda Otel collector config:

processors:
  filter:
    traces:
      spanevent:
        - 'exists(attributes["error"]) == true and IsMatch(name, ".*DynamoDB.*") == true'
        - 'exists(attributes["error"]) == true and IsMatch(name, ".*STS.*") == true'
        - 'exists(attributes["error"]) == true and IsMatch(name, ".*S3.*") == true'

I'm following these Honeycomb docs on filtering span events, but I'm getting an Extension.Crash in the Lambda function with a message like this.

"failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'processors': unknown type: \"filter\" for id: \"filter\" (valid values: [])"

I'm using the AWS Otel Lambda layer, as covered here in Honeycomb. This is also what is recommended by other vendors.

It's the latest version, specified in the AWS Otel docs for Lambda: https://aws-otel.github.io/docs/getting-started/lambda/lambda-python.

I raised this in the Honeycomb slack channel. They recommended that we use a more upgraded collector, but the collector version that the AWS Otel extension uses appears to be v0.39.0, which was last released 3 years ago 🤦

I'll raise this as an issue in the aws-otel-collector repository. Thanks for your help.

[kmcquade]

Here is the GitHub issue that I opened, which is the responsibility of the AWS Observability team, for any future viewers of this issue: aws-observability/aws-otel-collector#2750