Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling tracing by usage of opentelemetry #128

Draft
wants to merge 2 commits into
base: new-index
Choose a base branch
from

Conversation

rem1-dev
Copy link

@rem1-dev rem1-dev commented Nov 7, 2024

Enabling tracing by usage of opentelemetry

@rem1-dev
Copy link
Author

rem1-dev commented Nov 7, 2024

@shesek this is same as PR #99 but applied to fresh branch created out of new-index. Commits were squashed.

@rem1-dev
Copy link
Author

rem1-dev commented Nov 7, 2024

I'm working on solving those conflict files

@rem1-dev
Copy link
Author

rem1-dev commented Nov 7, 2024

@shesek conflicts resolved, it's ready to be merged.

Cargo.toml Outdated Show resolved Hide resolved
src/new_index/schema.rs Show resolved Hide resolved
src/bin/electrs.rs Outdated Show resolved Hide resolved
@philippem philippem self-requested a review November 13, 2024 21:18
@philippem philippem marked this pull request as draft November 13, 2024 21:19
@philippem
Copy link
Collaborator

@rem1-dev when testing locally we ran into some errors

tempo-1           | level=warn ts=2024-11-13T20:09:28.400374334Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185917 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"
tempo-1           | level=warn ts=2024-11-13T20:09:28.426261953Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185912 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"

which may be related to cardinality or size of the spans.

Possible causes:

- Large Trace Data: The trace contains too many spans or overly large data (such as logs or metadata).
- High Cardinality: Excessive number of unique tags, attributes, or data points being attached to spans, leading to large trace size.
- Excessive Span Detail: Individual spans may contain large payloads or detailed information that increase the trace size.
- Improper Sampling Rate: Sampling configuration might be too high, causing too many traces to be captured, some of which may exceed the size limit.
- Large Payloads in Spans: If spans contain large amounts of data (e.g., large JSON bodies, files, or detailed logs), it could push the trace size over the limit.
- Too Many Traces Collected: An overly aggressive tracing configuration that collects an excessive number of traces, increasing the likelihood of exceeding the size limit.
- Long or Complex Transactions: If a trace represents a very large or complex operation (e.g., a multi-step workflow or a distributed operation), it might naturally grow too large.

@rem1-dev
Copy link
Author

@philippem how can I reproduce this locally?

@philippem
Copy link
Collaborator

@shesek what do you think about adding a new macro to reduce the amount of copy-paste? will require new module

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants