Viewing timeout / retry metrics

[pH14]

Hello! We're using the S3 client and I was wondering if there's a way to hook into the timeout / retry service layers, or create our own middleware layer to do the same, so that we can bubble up how frequently we see retries happen. We currently set timeouts here, and we'd love to have some insight into when those timeouts are hit

[rcoh]

there currently isn't a way to hook into those layers but you can use a tracing subscriber to capture the tracing events that get fired from those layers

[pH14]

Interesting! That's a neat solution, are there any examples of how to create that subscriber?

[Velfi]

We have a tracing-subscriber example in our dev guide.

[ahrens]

I have a similar problem, I want to observe ErrorKind::ThrottlingError events and make them visible to our application's user. Do you think we should use a tracing subscriber, e.g. on the events generated in aws_smithy_client::retry.rs? I'm imagining that these could change in a future (even non-breaking) release, but it might be OK if we check it every time we update the version of the SDK that we depend on.

FWIW, the closest other solution I could come up with was creating a new struct that implements ClassifyRetry and wraps another implementor, and then use Operation::with_retry_classifier(). However, CustomizableOperation::map_operation() doesn't allow changing the generic type of the classifier (Retry).

[Velfi]

We have a better solution for this coming in a few months. For now, just use the trace logs. I can't think of a reason why we'd change the message before GA, so you should be safe.

[ahrens]

Thanks!

[hzxa21]

We have a better solution for this coming in a few months.

We have encountered the same issue. Look forward to the better solution. This will help improve the observability.

[hzxa21]

We have a better solution for this coming in a few months. For now, just use the trace logs. I can't think of a reason why we'd change the message before GA, so you should be safe.

@Velfi @rcoh I saw that 1.0 is released. Do we now have a better way to monitor timeout and retry?

[ahrens]

@hzxa21 FYI, for my use case of observing Throttling errors, I was able to use the CustomizableOperation::interceptor() to register an interceptor. I'd love to hear if there's a better way, but here's what we're doing:

[derive(Debug)]
struct ErrorLoggingInterceptor<T, E> {
    msg: String,
    rate_limit_errors: Arc<RelaxedCounter>,
    _phantom: PhantomData<(T, E)>,
}
impl<T, E> ErrorLoggingInterceptor<T, E> {
    fn apply<B>(
        msg: &str,
        rate_limit_errors: Arc<RelaxedCounter>,
        op: CustomizableOperation<T, E, B>,
    ) -> CustomizableOperation<T, E, B>
    where
        T: Debug + Send + Sync + 'static,
        E: Error + ProvideErrorMetadata + Debug + Send + Sync + 'static,
    {
        op.interceptor(Self {
            msg: msg.to_owned(),
            rate_limit_errors,
            _phantom: PhantomData,
        })
    }
}
impl<T, E> Intercept for ErrorLoggingInterceptor<T, E>
where
    T: Debug + Send + Sync + 'static,
    E: Error + ProvideErrorMetadata + Debug + Send + Sync + 'static,
{
    fn read_after_attempt(
        &self,
        context: &FinalizerInterceptorContextRef<'_>,
        _runtime_components: &RuntimeComponents,
        _cfg: &mut ConfigBag,
    ) -> std::result::Result<(), BoxError> {
        if let Some(Err(error)) = context.output_or_error() {
            debug!("{}: {error:?}", self.msg);
            if let Some(code) = error
                .as_operation_error()
                .and_then(|e| e.downcast_ref::<E>())
                .and_then(|e| e.meta().code())
            {
                if THROTTLING_ERRORS.contains(&code) {
                    self.rate_limit_errors.inc();
                    debug!("{}: throttling: {code}", self.msg);
                }
            }
        }
        Ok(())
    }

    fn name(&self) -> &'static str {
        "zettaobject::object_access::s3::ErrorLoggingInterceptor"
    }
}

and it's used like so:

                ErrorLoggingInterceptor::apply(
                    &msg,
                    self.rate_limit_errors.clone(),
                    self.client
                        .get_object()
                        .bucket(&self.bucket)
                        .key(&key)
                        .set_range(range_string.clone())
                        .customize(),
                )
                .send()

[jdisanti]

@ahrens, I'm pretty sure that's the approach @Velfi was referring to.