Implement in-memory Service Dependency Graph using Apache Beam

[yurishkuro]

For background, see #5910

Jaeger all-in-one typically runs with in-memory or badger storage that both have a special implementation of Dependencies Storage API where instead of pre-computing and storing the dependencies they just brute-force re-calculate them on demand each time:

• jaeger/plugin/storage/badger/dependencystore/storage.go

  Line 27 in 9a30dfc

  func (s *DependencyStore) GetDependencies(ctx context.Context, endTs time.Time, lookback time.Duration) ([]model.DependencyLink, error) {

• jaeger/plugin/storage/memory/memory.go

  Line 85 in 9a30dfc

  func (st *Store) GetDependencies(ctx context.Context, endTs time.Time, lookback time.Duration) ([]model.DependencyLink, error) {

It's ok for small demos, but:

• if all-in-one is run on a large machine and is allowed to store a lot of traces, this brute-force recalculation could be very slow
• the two implementations are completely independent, and different from the Spark implementation, so we have 3 different copies of the code to maintain

Following on the proposal from RFC #5910, we could re-implement this logic as an in-process streaming component using Apache Beam with direct executor. This will allow us to consolidate the graph building logic across memory and badger storages (in fact extract it from them into an independent component), and in the future we can find a way to adapt it to run in a distributed manner on big data runners without actually changing the business logic.

Some implementation details:

• The logic will be running as a dedicated processor in the OTEL pipeline (similar to adaptive sampling processor)
• The processor will accumulate dependency graph from a stream of traces it receives and periodically write them to dependencies storage (it can obtain the storage from jaeger_storage extension)
• The processor will need to perform a trace aggregation similar to tail sampling processor by grouping spans by trace ID and waiting for a period of inactivity after which the trace can be declared "complete" (i.e. fully received, assembled, and ready for processing). It would be interesting if such functionality can be abstracted out of tail sampling processor. Unlike tail sampler, the aggregator for dependencies does not need to keep full spans in memory, because (a) if it really needs them it can get them from SpanReader, and (b) it can just keep a more lightweight structure of only span IDs, span kinds, and their parent-child relationships - this will be useful once that logic starts running on a big data pipeline.

Steps:

• build a processor performing streaming aggregation for the basic service map (using the logic that already exists in the memory store)
• implement dependencies storage in the memory store (the GetDependency could be controlled by a feature flag allowing switching between current behavior and the new behavior that just reads the data)
• hook-up the streaming processor to write to storage
• verify the behavior via existing integration test for dependencies

[NavinShrinivas]

Hey yuri, this seems interesting. Are you thinking this will be a separate service that the collector forwards the details to? I'm just trying to make sense of this.

Are you in the process of splitting it down into smaller tasks?

[yurishkuro]

It's not a separate service.

[tronda]

The OpenTelemetry Collector Contrib includes the ServiceGraphConnector. This generates metrics based on the trace data which can be used to draw a dependency graph. Having struggled with Jaeger Spark dependency job, the service graph connector sounded appealing to us because the deployment would be much easier since we already have Prometheus available. Are there any architectural issues with using the service graph connector which doesn't fit Jaeger?

[yurishkuro]

@tronda Jaeger is not a metrics database, so in order to use the ServiceGraphConnector the user needs to run another backend. Then the transitive dependency graph is simply not representable in the metrics format, but is much more useful then the p2p graph that ServiceGraphConnector can produce.