diff --git a/.mvn/wrapper/MavenWrapperDownloader.java b/.mvn/wrapper/MavenWrapperDownloader.java index f8cf2608..b04970b1 100755 --- a/.mvn/wrapper/MavenWrapperDownloader.java +++ b/.mvn/wrapper/MavenWrapperDownloader.java @@ -17,9 +17,6 @@ Licensed to the Apache Software Foundation (ASF) under one under the License. */ -import java.net.*; -import java.io.*; -import java.nio.channels.*; import java.util.Properties; public class MavenWrapperDownloader { diff --git a/DESIGN.md b/DESIGN.md index 0e1c6940..c1b6ea96 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -2,95 +2,138 @@ ## Goals -* Provide a fast and reliable storage that enable extensibility via Kafka Topics. +* Provide a fast and reliable storage that enable extensibility via Kafka topics. * Provide full storage functionality via streaming aggregations (e.g., dependency graph). * Create a processing space where additional enrichment can be plugged in into the processing pipeline. * Remove need for additional storage when Kafka is available. +* More focused on supporting processing than storage: traces and dependency links are emitted +downstream to support metrics aggregation. Storage is currently supported but in a single node. -### Zipkin Storage Component +## Kafka Zipkin Storage -A Zipkin Storage component has the following internal parts: +Storage is composed by 3 main components: -* `Builder`: which configures if - - `strictTraceId(boolean strictTraceId)` - - `searchEnabled(boolean searchEnabled)` - - `autocompleteKeys(List keys)` - - `autocompleteTtl(int autocompleteTtl)` - - `autocompleteCardinality(int autocompleteCardinality)` -* `SpanStore`: main component - - `Call>> getTraces(QueryRequest request);` - - `Call> getTrace(String traceId);` - - `Call> getServiceNames();` - - `Call> getSpanNames(String serviceName);` - - `Call> getDependencies(long endTs, long lookback);` -* `SpanConsumer`: which ingest spans - - `Call accept(List spans)` -* `QueryRequest`: which includes - - `String serviceName, spanName;` - - `Map annotationQuery;` - - `Long minDuration, maxDuration;` - - `long endTs, lookback;` - - `int limit;` +- Span Consumer: repartition of collected span batches into individual spans keyed by `traceId` +- Span Aggregation: stream processing of spans into aggregated traces and then into dependency links. +- Span Store: building local state stores to support search and query API. -### Kafka Zipkin Storage +And it is supported by 3 main Kafka topics: -#### `KafkaSpanStore` +- `zipkin-spans`: Topic where list of spans indexed by trace Id are stored. +- `zipkin-trace`: Topic where aggregated traces are stored. +- `zipkin-dependency`: Topic where dependency links per trace are stored. -Span Store is expecting Spans to be stored in topics partitioned by `TraceId`. +### Kafka Span Consumer -> These can be created by Span Consumer, or can be **enriched** by other Stream Processors, outside of -Zipkin Server. +This component processes collected span batches (via HTTP, Kafka, ActiveMQ, etc), +take each element and re-indexed them by `traceId` on the "spans" topic. -Kafka Span Store will need to support different kind of queries: +This component is currently compensating how `KafkaSender` (part of [Zipkin-Reporter](https://github.com/openzipkin/zipkin-reporter-java)) +is reporting spans to Kafka, by grouping spans into batches and sending them to a un-keyed +Kafka topic. +Component source code: [KafkaSpanConsumer.java](storage/src/main/java/zipkin2/storage/kafka/KafkaSpanConsumer.java) -##### Get Service Names/Get Span Names +### Stream Processing -Service name to Span names pairs are indexed by aggregating spans. +#### Span Aggregation -##### Get Trace/Find Traces +"Partitioned" Spans are processed to produced two aggregated streams: `Traces` and `Dependencies`. -When search requests are received, span index is used to search for trace ids. After a list is -retrieved, trace DAG is retrieved from trace state store. +**Traces**: -##### Get Dependencies +Spans are grouped by ID and stored on a local +[Session window](https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/SessionWindows.html), +where the `traceId` becomes the token, and `trace-timeout` (default: 1 minute) +(i.e. period of time without receiving a span with the same session; also known as session inactivity gap +in Kafka Streams) +defines if a trace is still active or not. This is evaluated on the next span received on the stream-- +regardless of incoming `traceId`. If session window is closed, a trace message is emitted to the +traces topic. -After `spans` are aggregated into traces, traces are processed to collect dependencies. -Dependencies changelog are stored in a Kafka topic to be be stored as materialized view on -Zipkin instances. +![Session Windows](https://kafka.apache.org/20/images/streams-session-windows-02.png) -### Stream processors +> Each color represents a trace. The longer `trace timeout` we have, the longer we wait +to close a window and the longer we wait to emit traces downstream for dependency link and additional +aggregations; but also the more consistent the trace aggregation is. +If we choose a smaller gap, then we emit traces faster with the risk of breaking traces into +smaller chunks, and potentially affecting counters downstream. -#### Trace Aggregation Stream Processor +**Dependencies** -This is the main processors that take incoming spans and aggregate them into: +Once `traces` are emitted downstream as part of the initial processing, dependency links are evaluated +on each trace, and emitted the dependencies topic for further metric aggregation. -- Traces -- Dependencies +Kafka Streams topology: -![service aggregation](docs/service-aggregation-stream.png) +![trace aggregation](docs/trace-aggregation-topology.png) -![dependency aggregation](docs/dependency-aggregation-stream.png) +#### Trace Store Stream -#### Store Stream Processors +This component build local stores from state received on `spans` Kafka topic +for traces, service names and autocomplete tags. -Global tables for traces, service names and dependencies to be available on local state. +Kafka Streams source code: [TraceStoreTopologySupplier](storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplier.java) -![trace store](docs/trace-store-stream.png) +Kafka Streams topology: -![service store](docs/service-store-stream.png) +![trace store](docs/trace-store-topology.png) -![dependency store](docs/dependency-store-stream.png) +#### Dependency Store -#### Index Stream Processor +This component build local store from state received on `dependency` Kafka topic. -Custom processor to full-text indexing of traces using Lucene as back-end. +It builds a 1 minute time-window when counts calls and errors. -![span index](docs/span-index-stream.png) +Kafka Streams source code: [DependencyStoreTopologySupplier](storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplier.java) -#### Retention Stream Processor +Kafka Streams topology: -This is the processor that keeps track of trace timestamps for cleanup. +![dependency store](docs/dependency-store-topology.png) -![trace retention](docs/trace-retention-stream.png) \ No newline at end of file +### Kafka Span Store + +This component supports search and query APIs on top of local state stores build by the Store +Kafka Streams component. + +Component source code: [KafkaSpanStore.java](storage/src/main/java/zipkin2/storage/kafka/KafkaSpanStore.java) + +#### Get Service Names/Get Span Names/Get Remote Service Names + +These queries are supported by service names indexed stores built from `spans` Kafka topic. + +Store names: + +- `zipkin-service-names`: key/value store with service name as key and value. +- `zipkin-span-names`: key/value store with service name as key and span names list as value. +- `zipkin-remote-service-names`: key/value store with service name as key and remote service names as value. + +#### Get Trace/Find Traces + +These queries are supported by two key value stores: + +- `zipkin-traces`: indexed by `traceId`, contains span list status received from `spans` Kafka topic. +- `zipkin-traces-by-timestamp`: list of trace IDs indexed by `timestamp`. + +`GetTrace` query is supported by `zipkin-traces` store. +`FindTraces` query is supported by both: When receiving a query request time range is used to get +trace IDs, and then query request is tested on each trace to build a response. + +#### Get Dependencies + +This query is supported 1-minute windowed store from `DependencyStoreStream`. + +When a request is received, time range is used to pick valid windows and join counters. + +Windowed store: + +- `zipkin-dependencies`. + +### Kafka Autocomplete Tags + +#### Get Keys/Get Values + +Supported by a key-value containing list of values valid for `autocompleteKeys`. + +- `zipkin-autocomplete-tags`: key-value store. diff --git a/Dockerfile b/Dockerfile index afcd0d41..57093db8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,34 +12,49 @@ # the License. # -FROM openjdk:8 +FROM alpine -ARG KAFKA_STORAGE_VERSION=0.1.1 +ENV ZIPKIN_REPO https://repo1.maven.org/maven2 +ENV ZIPKIN_VERSION 2.16.1 +ENV KAFKA_STORAGE_VERSION 0.4.1-SNAPSHOT -ENV ZIPKIN_REPO https://jcenter.bintray.com -ENV ZIPKIN_VERSION 2.12.6 -ENV ZIPKIN_LOGGING_LEVEL INFO +WORKDIR /zipkin + +RUN apk add unzip curl --no-cache && \ + curl -SL $ZIPKIN_REPO/io/zipkin/zipkin-server/$ZIPKIN_VERSION/zipkin-server-$ZIPKIN_VERSION-exec.jar > zipkin-server.jar && \ + # don't break when unzip finds an extra header https://github.com/openzipkin/zipkin/issues/1932 + unzip zipkin-server.jar ; \ + rm zipkin-server.jar + +COPY autoconfigure/target/zipkin-autoconfigure-storage-kafka-${KAFKA_STORAGE_VERSION}-module.jar BOOT-INF/lib/kafka-module.jar +RUN unzip -o BOOT-INF/lib/kafka-module.jar lib/* -d BOOT-INF + +FROM gcr.io/distroless/java:11-debug # Use to set heap, trust store or other system properties. ENV JAVA_OPTS -Djava.security.egd=file:/dev/./urandom + +RUN ["/busybox/sh", "-c", "adduser -g '' -D zipkin"] + # Add environment settings for supported storage types +ENV STORAGE_TYPE kafka + +COPY --from=0 /zipkin/ /zipkin/ WORKDIR /zipkin -RUN curl -SL $ZIPKIN_REPO/io/zipkin/java/zipkin-server/$ZIPKIN_VERSION/zipkin-server-${ZIPKIN_VERSION}-exec.jar > zipkin.jar +# TODO haven't found a better way to mount libs from custom storage. issue #28 +#COPY autoconfigure/target/zipkin-autoconfigure-storage-kafka-${KAFKA_STORAGE_VERSION}-module.jar kafka-module.jar +#ENV MODULE_OPTS -Dloader.path='BOOT-INF/lib/kafka-module.jar,BOOT-INF/lib/kafka-module.jar!/lib' -Dspring.profiles.active=kafka +ENV MODULE_OPTS -Dspring.profiles.active=kafka + +RUN ["/busybox/sh", "-c", "ln -s /busybox/* /bin"] -ADD storage/target/zipkin-storage-kafka-${KAFKA_STORAGE_VERSION}.jar zipkin-storage-kafka.jar -ADD autoconfigure/target/zipkin-autoconfigure-storage-kafka-${KAFKA_STORAGE_VERSION}-module.jar zipkin-autoconfigure-storage-kafka.jar +ENV KAFKA_STORAGE_DIR /data +RUN mkdir /data && chown zipkin /data +VOLUME /data -ENV STORAGE_TYPE=kafkastore +USER zipkin -EXPOSE 9410 9411 +EXPOSE 9411 -CMD exec java \ - ${JAVA_OPTS} \ - -Dloader.path='zipkin-storage-kafka.jar,zipkin-autoconfigure-storage-kafka.jar' \ - -Dspring.profiles.active=kafkastore \ - -Dcom.linecorp.armeria.annotatedServiceExceptionVerbosity=all \ - -Dcom.linecorp.armeria.verboseExceptions=true \ - -cp zipkin.jar \ - org.springframework.boot.loader.PropertiesLauncher \ - --logging.level.zipkin2=${ZIPKIN_LOGGING_LEVEL} \ No newline at end of file +ENTRYPOINT ["/busybox/sh", "-c", "exec java ${MODULE_OPTS} ${JAVA_OPTS} -cp . org.springframework.boot.loader.PropertiesLauncher"] diff --git a/Makefile b/Makefile index 2bce60a6..1fec29e7 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,8 @@ all: build OPEN := 'xdg-open' MAVEN := './mvnw' -VERSION := '0.3.3-SNAPSHOT' +VERSION := '0.4.1-SNAPSHOT' +IMAGE_NAME := 'jeqo/zipkin-kafka' .PHONY: run run: build zipkin-local @@ -11,15 +12,24 @@ run: build zipkin-local .PHONY: run-docker run-docker: build docker-build docker-up +.PHONY: kafka-topics +kafka-topics: + docker-compose exec kafka-zookeeper /busybox/sh /kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand \ + --zookeeper localhost:2181 --create --topic zipkin-spans --partitions 1 --replication-factor 1 --if-not-exists + docker-compose exec kafka-zookeeper /busybox/sh /kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand \ + --zookeeper localhost:2181 --create --topic zipkin-trace --partitions 1 --replication-factor 1 --if-not-exists + docker-compose exec kafka-zookeeper /busybox/sh /kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand \ + --zookeeper localhost:2181 --create --topic zipkin-dependency --partitions 1 --replication-factor 1 --if-not-exists + .PHONY: docker-build docker-build: - TAG=${VERSION} \ - docker-compose build + docker build -t ${IMAGE_NAME}:latest . + docker build -t ${IMAGE_NAME}:${VERSION} . .PHONY: docker-push docker-push: docker-build - TAG=${VERSION} \ - docker-compose push + docker push ${IMAGE_NAME}:latest + docker push ${IMAGE_NAME}:${VERSION} .PHONY: docker-up docker-up: @@ -33,7 +43,7 @@ docker-down: .PHONY: docker-kafka-up docker-kafka-up: - docker-compose up -d kafka zookeeper + docker-compose up -d kafka-zookeeper .PHONY: license-header license-header: @@ -49,10 +59,11 @@ test: build .PHONY: zipkin-local zipkin-local: - STORAGE_TYPE=kafkastore \ + STORAGE_TYPE=kafka \ + KAFKA_BOOTSTRAP_SERVERS=localhost:19092 \ java \ - -Dloader.path='storage/target/zipkin-storage-kafka-${VERSION}.jar,autoconfigure/target/zipkin-autoconfigure-storage-kafka-${VERSION}-module.jar' \ - -Dspring.profiles.active=kafkastore \ + -Dloader.path='autoconfigure/target/zipkin-autoconfigure-storage-kafka-${VERSION}-module.jar,autoconfigure/target/zipkin-autoconfigure-storage-kafka-${VERSION}-module.jar!/lib' \ + -Dspring.profiles.active=kafka \ -cp zipkin.jar \ org.springframework.boot.loader.PropertiesLauncher @@ -60,11 +71,23 @@ zipkin-local: get-zipkin: curl -sSL https://zipkin.io/quickstart.sh | bash -s +.PHONY: zipkin-test-multi +zipkin-test-multi: + curl -s https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-lens/testdata/netflix.json | \ + curl -X POST -s localhost:9411/api/v2/spans -H'Content-Type: application/json' -d @- ; \ + ${OPEN} 'http://localhost:9412/zipkin/?lookback=custom&startTs=1' + sleep 61 + curl -s https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-lens/testdata/messaging.json | \ + curl -X POST -s localhost:9411/api/v2/spans -H'Content-Type: application/json' -d @- ; \ + .PHONY: zipkin-test zipkin-test: - curl -s https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-ui/testdata/netflix.json | \ + curl -s https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-lens/testdata/netflix.json | \ curl -X POST -s localhost:9411/api/v2/spans -H'Content-Type: application/json' -d @- ; \ ${OPEN} 'http://localhost:9411/zipkin/?lookback=custom&startTs=1' + sleep 61 + curl -s https://raw.githubusercontent.com/openzipkin/zipkin/master/zipkin-lens/testdata/messaging.json | \ + curl -X POST -s localhost:9411/api/v2/spans -H'Content-Type: application/json' -d @- ; \ .PHONY: release release: diff --git a/README.md b/README.md index aeddf067..7398b18d 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,55 @@ -# Zipkin Storage: Kafka +# Zipkin Storage: Kafka *[EXPERIMENTAL]* [![Build Status](https://www.travis-ci.org/jeqo/zipkin-storage-kafka.svg?branch=master)](https://www.travis-ci.org/jeqo/zipkin-storage-kafka) Kafka-based storage for Zipkin. -> This is in experimentation phase at the moment. - ``` - +----------------------------*zipkin*------------------------------------------- - | +-->( service:span ) - | +-->( span-index ) -( collected-spans )-|->[ span-consumer ] [ aggregation ] [ span-store ]--+-->( traces ) - | | ^ | ^ +-->( dependencies ) - +-------|------------------------|----|---------|------------------------------- - | | | | -----------------------------|------------------------|----|---------|------------------------- - | | | | - | | | | -*kafka* +-->( trace-spans )---// enriching //--+------->( service:span ) - | // sampling // | - +-->( service-span )-// filtering // ---+------->( dependencies ) - | ^ ------------------------------------------------------|----|--------------------------------- - | | -*stream-processors* [ custom processors ]--->( other storages ) - - + +----------------------------*zipkin*---------------------------------------------- + | [ dependency-store ]--->( dependencies ) + | ^ +-->( autocomplete-tags ) +( collected-spans )-|->[ span-consumer ] [ aggregation ] [ trace-store ]--+-->( traces ) + via http, kafka, | | ^ | ^ | +-->( service-names ) + amq, grpc, etc. +-------|--------------------|----|---------|------|------------------------------- + | | | | | +----------------------------|--------------------|----|---------|------|------------------------------- + +-->( spans )--------+----+---------| | + | | | +*kafka* +->( traces ) | + topics | | + +->( dependencies ) + +------------------------------------------------------------------------------------------------------- ``` -- [Design notes](DESIGN.md) +> Spans collected via different transports are partitioned by `traceId` and stored in a "spans" Kafka topic. +Partitioned spans are then aggregated into traces and then into dependency links, both +results are emitted into Kafka topics as well. +These 3 topics are used as source for local stores (Kafka Stream stores) that support Zipkin query and search APIs. -## Configuration +[Design notes](DESIGN.md) -### Storage configurations +[Configuration](autoconfigure/README.md) -| Configuration | Description | Default | -|---------------|-------------|---------| -| `KAFKA_STORE_SPAN_CONSUMER_ENABLED` | Process spans collected by Zipkin server | `true` | -| `KAFKA_STORE_SPAN_STORE_ENABLED` | Aggregate and store Zipkin data | `true` | -| `KAFKA_STORE_BOOTSTRAP_SERVERS` | Kafka bootstrap servers, format: `host:port` | `localhost:9092` | -| `KAFKA_STORE_ENSURE_TOPICS` | Ensure topics are created if don't exist | `true` | -| `KAFKA_STORE_DIRECTORY` | Root path where Zipkin stores tracing data | `/tmp/zipkin` | -| `KAFKA_STORE_COMPRESSION_TYPE` | Compression type used to store data in Kafka topics | `NONE` | -| `KAFKA_STORE_RETENTION_SCAN_FREQUENCY` | Frequency to scan old records, in milliseconds. | `86400000` (1 day) | -| `KAFKA_STORE_RETENTION_MAX_AGE` | Max age of a trace, to recognize old one for retention policies. | `604800000` (7 day) | +## Building -### Topics configuration +To build the project you will need Java 8+. -| Configuration | Description | Default | -| `KAFKA_STORE_SPANS_TOPIC` | Topic where incoming spans are stored. | `zipkin-spans` | -| `KAFKA_STORE_SPANS_TOPIC_PARTITIONS` | Span topic number of partitions. | `1` | -| `KAFKA_STORE_SPANS_TOPIC_REPLICATION_FACTOR` | Span topic replication factor. | `1` | -| `KAFKA_STORE_TRACES_TOPIC` | Topic where aggregated traces are stored. | `zipkin-traces` | -| `KAFKA_STORE_TRACES_TOPIC_PARTITIONS` | Traces topic number of partitions. | `1` | -| `KAFKA_STORE_TRACES_TOPIC_REPLICATION_FACTOR` | Traces topic replication factor. | `1` | -| `KAFKA_STORE_DEPENDENCIES_TOPIC` | Topic where aggregated service dependencies names are stored. | `zipkin-dependencies` | -| `KAFKA_STORE_DEPENDENCIES_TOPIC_PARTITIONS` | Services topic number of partitions. | `1` | -| `KAFKA_STORE_DEPENDENCIES_TOPIC_REPLICATION_FACTOR` | Services topic replication factor. | `1` | +```bash +make build +``` -> Use partitions and replication factor when Topics are created by Zipkin. If topics are created manually -those options are not used. +And testing: -## Get started +```bash +make test +``` -To build the project you will need Java 8+. +If you want to build a docker image: ```bash -make build -make test +make docker-build ``` ### Run locally @@ -79,7 +60,7 @@ To run locally, first you need to get Zipkin binaries: make get-zipkin ``` -By default Zipkin will be waiting for a Kafka broker to be running on `localhost:29092`. If you don't have one, +By default Zipkin will be waiting for a Kafka broker to be running on `localhost:19092`. If you don't have one, this service is available via Docker Compose: ```bash @@ -94,31 +75,48 @@ make run ### Run with Docker -Run: +If you have Docker available, run: ```bash -make run-docker +make run-docker ``` And Docker image will be built and Docker compose will start. +#### Examples + +There are two examples, running Zipkin with kafka as storage: + ++ [Single-node](docker-compose.yml) ++ [Multi-mode](docker-compose-distributed.yml) + ### Testing -To validate storage: +To validate storage make sure that Kafka topics are created so Kafka Stream instances can be +initialized properly: ```bash +make kafka-topics make zipkin-test ``` This will start a browser and check a traces has been registered. -### Examples +It will send another trace after a minute (`trace timeout`) + 1 second to trigger +aggregation and visualize dependency graph. + +If running multi-node docker example, run: + +```bash +make zipkin-test-multi +``` + +![traces](docs/traces.png) -There are two examples, running zipkin with kafka as storage: +![dependencies](docs/dependencies.png) -+ Single-node: `examples/single-node` -+ Multi-mode: `examples/multi-mode` +## Acknowledgments -## Acknowledged +This project is inspired in Adrian Cole's VoltDB storage -This project is inspired in Adrian Cole's +Kafka Streams images are created with \ No newline at end of file diff --git a/autoconfigure/README.md b/autoconfigure/README.md new file mode 100644 index 00000000..e2a88493 --- /dev/null +++ b/autoconfigure/README.md @@ -0,0 +1,19 @@ +# Configuration + +## Broker and topics + +| Configuration | Description | Default | +|---------------|-------------|---------| +| `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | `localhost:9092` | +| `KAFKA_SPANS_TOPIC` | Topic where incoming list of spans are stored. | `zipkin-spans` | +| `KAFKA_TRACE_TOPIC` | Topic where aggregated traces are stored. | `zipkin-trace` | +| `KAFKA_DEPENDENCY_TOPIC` | Topic where aggregated service dependencies names are stored. | `zipkin-dependency` | + +## Storage configurations + +| Configuration | Description | Default | +|---------------|-------------|---------| +| `KAFKA_STORAGE_DIR` | Root path where Zipkin stores tracing data | `/tmp/zipkin-storage-kafka` | +| `KAFKA_STORAGE_TRACE_TIMEOUT` | How long to wait until a trace window is closed (ms). If this config is to small, dependency links won't be caught and metrics may drift. | `600000` (1 minute) | +| `KAFKA_STORAGE_TRACE_TTL` | How long to keep traces stored. | `259200000` (3 days) | +| `KAFKA_STORAGE_DEPENDENCY_TTL` | How long to keep dependencies stored. | `604800000` (1 week) | diff --git a/autoconfigure/pom.xml b/autoconfigure/pom.xml index 45e60cff..41f79a35 100644 --- a/autoconfigure/pom.xml +++ b/autoconfigure/pom.xml @@ -14,7 +14,9 @@ the License. --> - + 4.0.0 io.github.jeqo.zipkin @@ -39,6 +41,19 @@ zipkin-storage-kafka + + + junit + junit + test + 4.12 + + + org.assertj + assertj-core + 3.13.2 + test + org.springframework.boot spring-boot-autoconfigure @@ -55,15 +70,6 @@ - - net.orfjackal.retrolambda - retrolambda-maven-plugin - - - none - - - org.springframework.boot spring-boot-maven-plugin diff --git a/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageAutoConfiguration.java b/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageAutoConfiguration.java index b16f8c94..f2dd8faa 100644 --- a/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageAutoConfiguration.java +++ b/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageAutoConfiguration.java @@ -18,12 +18,11 @@ import org.springframework.boot.context.properties.EnableConfigurationProperties; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; -import org.springframework.scheduling.annotation.EnableScheduling; import zipkin2.storage.StorageComponent; @Configuration @EnableConfigurationProperties(ZipkinKafkaStorageProperties.class) -@ConditionalOnProperty(name = "zipkin.storage.type", havingValue = "kafkastore") +@ConditionalOnProperty(name = "zipkin.storage.type", havingValue = "kafka") @ConditionalOnMissingBean(StorageComponent.class) class ZipkinKafkaStorageAutoConfiguration { diff --git a/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageProperties.java b/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageProperties.java index cb600aed..d52b3170 100644 --- a/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageProperties.java +++ b/autoconfigure/src/main/java/zipkin2/autoconfigure/storage/kafka/ZipkinKafkaStorageProperties.java @@ -15,7 +15,8 @@ import java.io.Serializable; import java.time.Duration; -import org.apache.kafka.common.record.CompressionType; +import java.util.LinkedHashMap; +import java.util.Map; import org.springframework.boot.context.properties.ConfigurationProperties; import zipkin2.storage.kafka.KafkaStorage; @@ -23,90 +24,82 @@ public class ZipkinKafkaStorageProperties implements Serializable { private static final long serialVersionUID = 0L; - private boolean spanConsumerEnabled = true; - private boolean spanStoreEnabled = true; - - private boolean ensureTopics = true; - private String bootstrapServers = "localhost:9092"; - private String compressionType = CompressionType.NONE.name(); - - private Long retentionScanFrequency = Duration.ofDays(1).toMillis(); - private Long retentionMaxAge = Duration.ofDays(7).toMillis(); - private Long traceInactivityGap = Duration.ofMinutes(1).toMillis(); - - private String spansTopic = "zipkin-spans-v1"; - private Integer spansTopicPartitions = 1; - private Short spansTopicReplicationFactor = 1; - private String spanServicesTopic = "zipkin-span-services-v1"; - private Integer spanServicesTopicPartitions = 1; - private Short spanServicesTopicReplicationFactor = 1; - private String servicesTopic = "zipkin-services-v1"; - private Integer servicesTopicPartitions = 1; - private Short servicesTopicReplicationFactor = 1; - private String spanDependenciesTopic = "zipkin-span-dependencies-v1"; - private Integer spanDependenciesTopicPartitions = 1; - private Short spanDependenciesTopicReplicationFactor = 1; - private String dependenciesTopic = "zipkin-dependencies-v1"; - private Integer dependenciesTopicPartitions = 1; - private Short dependenciesTopicReplicationFactor = 1; - - private String storeDirectory = "/tmp/zipkin"; + private Boolean spanConsumerEnabled; - KafkaStorage.Builder toBuilder() { - return KafkaStorage.newBuilder() - .spanConsumerEnabled(spanConsumerEnabled) - .spanStoreEnabled(spanStoreEnabled) - .ensureTopics(ensureTopics) - .bootstrapServers(bootstrapServers) - .compressionType(compressionType) - .retentionMaxAge(Duration.ofMillis(retentionMaxAge)) - .retentionScanFrequency(Duration.ofMillis(retentionScanFrequency)) - .traceInactivityGap(Duration.ofMillis(traceInactivityGap)) - .spansTopic(KafkaStorage.Topic.builder(spansTopic) - .partitions(spansTopicPartitions) - .replicationFactor(spansTopicReplicationFactor) - .build()) - .spanServicesTopic(KafkaStorage.Topic.builder(spanServicesTopic) - .partitions(spanServicesTopicPartitions) - .replicationFactor(spanServicesTopicReplicationFactor) - .build()) - .servicesTopic(KafkaStorage.Topic.builder(servicesTopic) - .partitions(servicesTopicPartitions) - .replicationFactor(servicesTopicReplicationFactor) - .build()) - .spanDependenciesTopic(KafkaStorage.Topic.builder(spanDependenciesTopic) - .partitions(spanDependenciesTopicPartitions) - .replicationFactor(spanDependenciesTopicReplicationFactor) - .build()) - .dependenciesTopic(KafkaStorage.Topic.builder(dependenciesTopic) - .partitions(dependenciesTopicPartitions) - .replicationFactor(dependenciesTopicReplicationFactor) - .build()) - .storeDirectory(storeDirectory); - } - - public boolean isSpanConsumerEnabled() { - return spanConsumerEnabled; - } + private String bootstrapServers; - public void setSpanConsumerEnabled(boolean spanConsumerEnabled) { - this.spanConsumerEnabled = spanConsumerEnabled; - } + private Long traceTtlCheckInterval; + private Long traceTtl; + private Long traceTimeout; - public boolean isSpanStoreEnabled() { - return spanStoreEnabled; - } + private Long dependencyTtl; - public void setSpanStoreEnabled(boolean spanStoreEnabled) { - this.spanStoreEnabled = spanStoreEnabled; - } + private String spansTopic; + private String traceTopic; + private String dependencyTopic; + + private String storeDir; + + private String aggregationStreamAppId; + private String traceStoreStreamAppId; + private String dependencyStoreStreamAppId; + + /** + * Additional Kafka configuration. + */ + private Map adminOverrides = new LinkedHashMap<>(); + private Map producerOverrides = new LinkedHashMap<>(); + private Map aggregationStreamOverrides = new LinkedHashMap<>(); + private Map traceStoreStreamOverrides = new LinkedHashMap<>(); + private Map dependencyStoreStreamOverrides = new LinkedHashMap<>(); - public boolean isEnsureTopics() { - return ensureTopics; + KafkaStorage.Builder toBuilder() { + KafkaStorage.Builder builder = KafkaStorage.newBuilder(); + if (spanConsumerEnabled != null) builder.spanConsumerEnabled(spanConsumerEnabled); + if (bootstrapServers != null) builder.bootstrapServers(bootstrapServers); + if (traceTimeout != null) { + builder.traceTimeout(Duration.ofMillis(traceTimeout)); + } + if (traceTtlCheckInterval != null) { + builder.traceTtlCheckInterval(Duration.ofMillis(traceTtlCheckInterval)); + } + if (traceTtl != null) { + builder.traceTtl(Duration.ofMillis(traceTtl)); + } + if (dependencyTtl != null) { + builder.dependencyTtl(Duration.ofMillis(dependencyTtl)); + } + if (aggregationStreamAppId != null) builder.aggregationStreamAppId(aggregationStreamAppId); + if (traceStoreStreamAppId != null) builder.aggregationStreamAppId(traceStoreStreamAppId); + if (dependencyStoreStreamAppId != null) { + builder.aggregationStreamAppId(dependencyStoreStreamAppId); + } + if (storeDir != null) builder.storeDirectory(storeDir); + if (spansTopic != null) builder.spansTopicName(spansTopic); + if (traceTopic != null) builder.tracesTopicName(traceTopic); + if (dependencyTopic != null) builder.dependenciesTopicName(dependencyTopic); + if (adminOverrides != null) builder.adminOverrides(adminOverrides); + if (producerOverrides != null) builder.producerOverrides(producerOverrides); + if (aggregationStreamOverrides != null) { + builder.aggregationStreamOverrides(aggregationStreamOverrides); + } + if (traceStoreStreamOverrides != null) { + builder.traceStoreStreamOverrides(traceStoreStreamOverrides); + } + if (dependencyStoreStreamOverrides != null) { + builder.dependencyStoreStreamOverrides(dependencyStoreStreamOverrides); + } + if (aggregationStreamAppId != null) builder.aggregationStreamAppId(aggregationStreamAppId); + if (traceStoreStreamAppId != null) builder.traceStoreStreamAppId(traceStoreStreamAppId); + if (dependencyStoreStreamAppId != null) { + builder.dependencyStoreStreamAppId(dependencyStoreStreamAppId); + } + + return builder; } - public void setEnsureTopics(boolean ensureTopics) { - this.ensureTopics = ensureTopics; + public void setSpanConsumerEnabled(boolean spanConsumerEnabled) { + this.spanConsumerEnabled = spanConsumerEnabled; } public String getBootstrapServers() { @@ -117,36 +110,28 @@ public void setBootstrapServers(String bootstrapServers) { this.bootstrapServers = bootstrapServers; } - public String getCompressionType() { - return compressionType; + public Long getTraceTtlCheckInterval() { + return traceTtlCheckInterval; } - public void setCompressionType(String compressionType) { - this.compressionType = compressionType; + public void setTraceTtlCheckInterval(Long traceTtlCheckInterval) { + this.traceTtlCheckInterval = traceTtlCheckInterval; } - public Long getRetentionScanFrequency() { - return retentionScanFrequency; + public Long getTraceTtl() { + return traceTtl; } - public void setRetentionScanFrequency(Long retentionScanFrequency) { - this.retentionScanFrequency = retentionScanFrequency; + public void setTraceTtl(Long traceTtl) { + this.traceTtl = traceTtl; } - public Long getRetentionMaxAge() { - return retentionMaxAge; + public Long getTraceTimeout() { + return traceTimeout; } - public void setRetentionMaxAge(Long retentionMaxAge) { - this.retentionMaxAge = retentionMaxAge; - } - - public Long getTraceInactivityGap() { - return traceInactivityGap; - } - - public void setTraceInactivityGap(Long traceInactivityGap) { - this.traceInactivityGap = traceInactivityGap; + public void setTraceTimeout(Long traceTimeout) { + this.traceTimeout = traceTimeout; } public String getSpansTopic() { @@ -157,124 +142,110 @@ public void setSpansTopic(String spansTopic) { this.spansTopic = spansTopic; } - public Integer getSpansTopicPartitions() { - return spansTopicPartitions; - } - - public void setSpansTopicPartitions(Integer spansTopicPartitions) { - this.spansTopicPartitions = spansTopicPartitions; - } - - public Short getSpansTopicReplicationFactor() { - return spansTopicReplicationFactor; - } - - public void setSpansTopicReplicationFactor(Short spansTopicReplicationFactor) { - this.spansTopicReplicationFactor = spansTopicReplicationFactor; - } - - public String getSpanServicesTopic() { - return spanServicesTopic; + public Boolean getSpanConsumerEnabled() { + return spanConsumerEnabled; } - public void setSpanServicesTopic(String spanServicesTopic) { - this.spanServicesTopic = spanServicesTopic; + public void setSpanConsumerEnabled(Boolean spanConsumerEnabled) { + this.spanConsumerEnabled = spanConsumerEnabled; } - public Integer getSpanServicesTopicPartitions() { - return spanServicesTopicPartitions; + public String getTraceTopic() { + return traceTopic; } - public void setSpanServicesTopicPartitions(Integer spanServicesTopicPartitions) { - this.spanServicesTopicPartitions = spanServicesTopicPartitions; + public void setTraceTopic(String traceTopic) { + this.traceTopic = traceTopic; } - public Short getSpanServicesTopicReplicationFactor() { - return spanServicesTopicReplicationFactor; + public String getDependencyTopic() { + return dependencyTopic; } - public void setSpanServicesTopicReplicationFactor(Short spanServicesTopicReplicationFactor) { - this.spanServicesTopicReplicationFactor = spanServicesTopicReplicationFactor; + public void setDependencyTopic(String dependencyTopic) { + this.dependencyTopic = dependencyTopic; } - public String getServicesTopic() { - return servicesTopic; + public String getStoreDir() { + return storeDir; } - public void setServicesTopic(String servicesTopic) { - this.servicesTopic = servicesTopic; + public void setStoreDir(String storeDir) { + this.storeDir = storeDir; } - public Integer getServicesTopicPartitions() { - return servicesTopicPartitions; + public Long getDependencyTtl() { + return dependencyTtl; } - public void setServicesTopicPartitions(Integer servicesTopicPartitions) { - this.servicesTopicPartitions = servicesTopicPartitions; + public void setDependencyTtl(Long dependencyTtl) { + this.dependencyTtl = dependencyTtl; } - public Short getServicesTopicReplicationFactor() { - return servicesTopicReplicationFactor; + public Map getAdminOverrides() { + return adminOverrides; } - public void setServicesTopicReplicationFactor(Short servicesTopicReplicationFactor) { - this.servicesTopicReplicationFactor = servicesTopicReplicationFactor; + public void setAdminOverrides(Map adminOverrides) { + this.adminOverrides = adminOverrides; } - public String getSpanDependenciesTopic() { - return spanDependenciesTopic; + public Map getProducerOverrides() { + return producerOverrides; } - public void setSpanDependenciesTopic(String spanDependenciesTopic) { - this.spanDependenciesTopic = spanDependenciesTopic; + public void setProducerOverrides(Map producerOverrides) { + this.producerOverrides = producerOverrides; } - public Integer getSpanDependenciesTopicPartitions() { - return spanDependenciesTopicPartitions; + public Map getAggregationStreamOverrides() { + return aggregationStreamOverrides; } - public void setSpanDependenciesTopicPartitions(Integer spanDependenciesTopicPartitions) { - this.spanDependenciesTopicPartitions = spanDependenciesTopicPartitions; + public void setAggregationStreamOverrides( + Map aggregationStreamOverrides) { + this.aggregationStreamOverrides = aggregationStreamOverrides; } - public Short getSpanDependenciesTopicReplicationFactor() { - return spanDependenciesTopicReplicationFactor; + public Map getTraceStoreStreamOverrides() { + return traceStoreStreamOverrides; } - public void setSpanDependenciesTopicReplicationFactor( - Short spanDependenciesTopicReplicationFactor) { - this.spanDependenciesTopicReplicationFactor = spanDependenciesTopicReplicationFactor; + public void setTraceStoreStreamOverrides( + Map traceStoreStreamOverrides) { + this.traceStoreStreamOverrides = traceStoreStreamOverrides; } - public String getDependenciesTopic() { - return dependenciesTopic; + public Map getDependencyStoreStreamOverrides() { + return dependencyStoreStreamOverrides; } - public void setDependenciesTopic(String dependenciesTopic) { - this.dependenciesTopic = dependenciesTopic; + public void setDependencyStoreStreamOverrides( + Map dependencyStoreStreamOverrides) { + this.dependencyStoreStreamOverrides = dependencyStoreStreamOverrides; } - public Integer getDependenciesTopicPartitions() { - return dependenciesTopicPartitions; + public String getAggregationStreamAppId() { + return aggregationStreamAppId; } - public void setDependenciesTopicPartitions(Integer dependenciesTopicPartitions) { - this.dependenciesTopicPartitions = dependenciesTopicPartitions; + public void setAggregationStreamAppId(String aggregationStreamAppId) { + this.aggregationStreamAppId = aggregationStreamAppId; } - public Short getDependenciesTopicReplicationFactor() { - return dependenciesTopicReplicationFactor; + public String getTraceStoreStreamAppId() { + return traceStoreStreamAppId; } - public void setDependenciesTopicReplicationFactor(Short dependenciesTopicReplicationFactor) { - this.dependenciesTopicReplicationFactor = dependenciesTopicReplicationFactor; + public void setTraceStoreStreamAppId(String traceStoreStreamAppId) { + this.traceStoreStreamAppId = traceStoreStreamAppId; } - public String getStoreDirectory() { - return storeDirectory; + public String getDependencyStoreStreamAppId() { + return dependencyStoreStreamAppId; } - public void setStoreDirectory(String storeDirectory) { - this.storeDirectory = storeDirectory; + public void setDependencyStoreStreamAppId(String dependencyStoreStreamAppId) { + this.dependencyStoreStreamAppId = dependencyStoreStreamAppId; } } diff --git a/autoconfigure/src/main/resources/zipkin-server-kafka.yml b/autoconfigure/src/main/resources/zipkin-server-kafka.yml new file mode 100644 index 00000000..c7dbea63 --- /dev/null +++ b/autoconfigure/src/main/resources/zipkin-server-kafka.yml @@ -0,0 +1,22 @@ +# When enabled, this allows shorter env properties (ex -Dspring.profiles.active=kafka) +zipkin: + storage: + kafka: + # Connection to Kafka + bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} + # Kafka topic names + spans-topic: ${KAFKA_SPANS_TOPIC:zipkin-spans} + trace-topic: ${KAFKA_TRACE_TOPIC:zipkin-trace} + dependency-topic: ${KAFKA_DEPENDENCY_TOPIC:zipkin-dependency} + # Kafka Streams configs + aggregation-stream-app-id: ${KAFKA_STORAGE_AGGREGATION_STREAM_APP_ID:zipkin-aggregation} + trace-store-stream-app-id: ${KAFKA_STORAGE_TRACE_STORE_STREAM_APP_ID:zipkin-trace-store} + dependency-store-stream-app-id: ${KAFKA_STORAGE_DEPENDENCY_STORE_STREAM_APP_ID:zipkin-dependency-store} + store-dir: ${KAFKA_STORAGE_DIR:/tmp/zipkin-storage-kafka} + # Kafka Storage flags + span-consumer-enabled: true + # Kafka Storage timing configs + trace-timeout: ${KAFKA_STORAGE_TRACE_TIMEOUT:60000} + trace-ttl: ${KAFKA_STORAGE_TRACE_TTL:259200000} + trace-ttl-check-interval: ${KAFKA_STORAGE_TRACE_TTL_CHECK_INTERVAL:3600000} + dependency-ttl: ${KAFKA_STORAGE_DEPENDENCY_TTL:604800000} \ No newline at end of file diff --git a/autoconfigure/src/main/resources/zipkin-server-kafkastore.yml b/autoconfigure/src/main/resources/zipkin-server-kafkastore.yml deleted file mode 100644 index 463ed41c..00000000 --- a/autoconfigure/src/main/resources/zipkin-server-kafkastore.yml +++ /dev/null @@ -1,26 +0,0 @@ -# When enabled, this allows shorter env properties (ex -Dspring.profiles.active=kafka) -zipkin: - storage: - kafka: - bootstrap-servers: ${KAFKA_STORE_BOOTSTRAP_SERVERS:localhost:9092} - ensure-topics: ${KAFKA_STORE_ENSURE_TOPICS:true} - store-directory: ${KAFKA_STORE_DIRECTORY:/tmp/zipkin} - compression-type: ${KAFKA_STORE_COMPRESSION_TYPE:NONE} - retention-scan-frequency: ${KAFKA_STORE_RETENTION_SCAN_FREQUENCY:86400000} - retention-max-age: ${KAFKA_STORE_RETENTION_MAX_AGE:604800000} - trace-inactivity-gap: ${KAFKA_STORE_TRACE_INACTIVITY_GAP:60000} - spans-topic: ${KAFKA_STORE_SPANS_TOPIC:zipkin-spans-v1} - spans-topic-partitions: ${KAFKA_STORE_SPANS_TOPIC_PARTITIONS:1} - spans-topic-replication-factor: ${KAFKA_STORE_SPANS_TOPIC_REPLICATION_FACTOR:1} - span-services-topic: ${KAFKA_STORE_SPAN_SERVICES_TOPIC:zipkin-span-services-v1} - span-services-topic-partitions: ${KAFKA_STORE_SPAN_SERVICES_TOPIC_PARTITIONS:1} - span-services-topic-replication-factor: ${KAFKA_STORE_SPAN_SERVICES_TOPIC_REPLICATION_FACTOR:1} - services-topic: ${KAFKA_STORE_SERVICES_TOPIC:zipkin-services-v1} - services-topic-partitions: ${KAFKA_STORE_SERVICES_TOPIC_PARTITIONS:1} - services-topic-replication-factor: ${KAFKA_STORE_SERVICES_TOPIC_REPLICATION_FACTOR:1} - span-dependencies-topic: ${KAFKA_STORE_SPAN_DEPENDENCIES_TOPIC:zipkin-span-dependencies-v1} - span-dependencies-topic-partitions: ${KAFKA_STORE_SPAN_DEPENDENCIES_TOPIC_PARTITIONS:1} - span-dependencies-topic-replication-factor: ${KAFKA_STORE_SPAN_DEPENDENCIES_TOPIC_REPLICATION_FACTOR:1} - dependencies-topic: ${KAFKA_STORE_DEPENDENCIES_TOPIC:zipkin-dependencies-v1} - dependencies-topic-partitions: ${KAFKA_STORE_DEPENDENCIES_TOPIC_PARTITIONS:1} - dependencies-topic-replication-factor: ${KAFKA_STORE_DEPENDENCIES_TOPIC_REPLICATION_FACTOR:1} diff --git a/autoconfigure/src/test/java/zipkin2/storage/kafka/ZipkinKafkaStorageAutoConfigurationTest.java b/autoconfigure/src/test/java/zipkin2/storage/kafka/ZipkinKafkaStorageAutoConfigurationTest.java index 06f3c510..841dc135 100644 --- a/autoconfigure/src/test/java/zipkin2/storage/kafka/ZipkinKafkaStorageAutoConfigurationTest.java +++ b/autoconfigure/src/test/java/zipkin2/storage/kafka/ZipkinKafkaStorageAutoConfigurationTest.java @@ -13,8 +13,9 @@ */ package zipkin2.storage.kafka; +import org.apache.kafka.clients.admin.AdminClientConfig; import org.apache.kafka.clients.producer.ProducerConfig; -import org.apache.kafka.common.record.CompressionType; +import org.apache.kafka.streams.StreamsConfig; import org.junit.After; import org.junit.Rule; import org.junit.Test; @@ -55,7 +56,7 @@ public void doesNotProvidesStorageComponent_whenStorageTypeNotKafka() { public void providesStorageComponent_whenStorageTypeKafka() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore" + "zipkin.storage.type:kafka" ).applyTo(context); Access.registerKafka(context); context.refresh(); @@ -67,7 +68,7 @@ public void providesStorageComponent_whenStorageTypeKafka() { public void canOverridesProperty_bootstrapServers() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", + "zipkin.storage.type:kafka", "zipkin.storage.kafka.bootstrap-servers:host1:19092" ).applyTo(context); Access.registerKafka(context); @@ -78,163 +79,124 @@ public void canOverridesProperty_bootstrapServers() { } @Test - public void canOverridesProperty_ensureTopics() { + public void canOverridesProperty_adminConfigs() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.ensure-topics:false" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.admin-overrides.bootstrap.servers:host1:19092" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).ensureTopics).isEqualTo(false); + assertThat(context.getBean(KafkaStorage.class).adminConfig.get( + AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG)).isEqualTo("host1:19092"); } @Test - public void canOverridesProperty_compressionType() { + public void canOverridesProperty_producerConfigs() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.compression-type:SNAPPY" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.producer-overrides.acks:1" ).applyTo(context); Access.registerKafka(context); context.refresh(); assertThat(context.getBean(KafkaStorage.class).producerConfig.get( - ProducerConfig.COMPRESSION_TYPE_CONFIG)).isEqualTo(CompressionType.SNAPPY.name); + ProducerConfig.ACKS_CONFIG)).isEqualTo("1"); } - - @Test - public void canOverridesProperty_storeDirectory() { - context = new AnnotationConfigApplicationContext(); - TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.store-directory:/zipkin" - ).applyTo(context); - Access.registerKafka(context); - context.refresh(); - - assertThat(context.getBean(KafkaStorage.class).storageDirectory).isEqualTo("/zipkin"); - } - @Test - public void canOverridesProperty_spansTopicName() { + public void canOverridesProperty_aggregationStreamConfigs() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.spans-topic:zipkin-spans-1" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.aggregation-stream-overrides.application.id:agg1" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).spansTopic.name).isEqualTo("zipkin-spans-1"); + assertThat(context.getBean(KafkaStorage.class).aggregationStreamConfig.get( + StreamsConfig.APPLICATION_ID_CONFIG)).isEqualTo("agg1"); } @Test - public void canOverridesProperty_spansTopicPartitions() { + public void canOverridesProperty_traceStoreStreamConfigs() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.spans-topic-partitions:2" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.trace-store-stream-overrides.application.id:store1" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).spansTopic.partitions).isEqualTo(2); + assertThat(context.getBean(KafkaStorage.class).traceStoreStreamConfig.get( + StreamsConfig.APPLICATION_ID_CONFIG)).isEqualTo("store1"); } @Test - public void canOverridesProperty_spansTopicReplicationFactor() { + public void canOverridesProperty_dependencyStoreStreamConfigs() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.spans-topic-replication-factor:2" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.dependency-store-stream-overrides.application.id:store1" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).spansTopic.replicationFactor).isEqualTo( - (short) 2); + assertThat(context.getBean(KafkaStorage.class).dependencyStoreStreamConfig.get( + StreamsConfig.APPLICATION_ID_CONFIG)).isEqualTo("store1"); } @Test - public void canOverridesProperty_servicesTopicName() { + public void canOverridesProperty_storeDirectory() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.services-topic:zipkin-services-1" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.store-dir:/zipkin" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).servicesTopic.name).isEqualTo("zipkin-services-1"); + assertThat(context.getBean(KafkaStorage.class).storageDirectory).isEqualTo("/zipkin"); } @Test - public void canOverridesProperty_servicesTopicPartitions() { + public void canOverridesProperty_spansTopicName() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.services-topic-partitions:2" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.spans-topic:zipkin-spans-1" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).servicesTopic.partitions).isEqualTo(2); + assertThat(context.getBean(KafkaStorage.class).spansTopicName).isEqualTo("zipkin-spans-1"); } @Test - public void canOverridesProperty_servicesTopicReplicationFactor() { + public void canOverridesProperty_tracesTopicName() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.services-topic-replication-factor:2" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.trace-topic:zipkin-traces-1" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).servicesTopic.replicationFactor).isEqualTo( - (short) 2); + assertThat(context.getBean(KafkaStorage.class).traceTopicName).isEqualTo("zipkin-traces-1"); } @Test public void canOverridesProperty_dependenciesTopicName() { context = new AnnotationConfigApplicationContext(); TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.dependencies-topic:zipkin-dependencies-1" + "zipkin.storage.type:kafka", + "zipkin.storage.kafka.dependency-topic:zipkin-dependencies-1" ).applyTo(context); Access.registerKafka(context); context.refresh(); - assertThat(context.getBean(KafkaStorage.class).dependenciesTopic.name).isEqualTo( + assertThat(context.getBean(KafkaStorage.class).dependencyTopicName).isEqualTo( "zipkin-dependencies-1"); } - - @Test - public void canOverridesProperty_dependenciesTopicPartitions() { - context = new AnnotationConfigApplicationContext(); - TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.dependencies-topic-partitions:2" - ).applyTo(context); - Access.registerKafka(context); - context.refresh(); - - assertThat(context.getBean(KafkaStorage.class).dependenciesTopic.partitions).isEqualTo(2); - } - - @Test - public void canOverridesProperty_dependenciesTopicReplicationFactor() { - context = new AnnotationConfigApplicationContext(); - TestPropertyValues.of( - "zipkin.storage.type:kafkastore", - "zipkin.storage.kafka.dependencies-topic-replication-factor:2" - ).applyTo(context); - Access.registerKafka(context); - context.refresh(); - - assertThat(context.getBean(KafkaStorage.class).dependenciesTopic.replicationFactor).isEqualTo( - (short) 2); - } } \ No newline at end of file diff --git a/docker-compose-distributed.yml b/docker-compose-distributed.yml new file mode 100644 index 00000000..b5c4ad31 --- /dev/null +++ b/docker-compose-distributed.yml @@ -0,0 +1,47 @@ +# +# Copyright 2019 jeqo +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except +# in compliance with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. +# + +--- +version: '3' +services: + kafka-zookeeper: + image: openzipkin/zipkin-kafka + container_name: kafka-zookeeper + ports: + - 2181:2181 + - 9092:9092 + - 19092:19092 + zipkin-aggregation: + image: jeqo/zipkin-kafka + container_name: zipkin-aggregation + ports: + - 9411:9411 + environment: + KAFKA_BOOTSTRAP_SERVERS: kafka-zookeeper:9092 + KAFKA_STORAGE_SPAN_CONSUMER_ENABLED: 'true' + QUERY_ENABLED: 'false' + zipkin-storage: + image: jeqo/zipkin-kafka + container_name: zipkin-storage + ports: + - 9412:9411 + environment: + KAFKA_BOOTSTRAP_SERVERS: kafka-zookeeper:9092 + KAFKA_STORAGE_SPAN_CONSUMER_ENABLED: 'false' + KAFKA_STORAGE_TRACES_RETENTION_PERIOD: -1 + KAFKA_STORAGE_TRACES_INACTIVITY_GAP: 5000 + volumes: + - zipkin:/data +volumes: + zipkin: \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index d47cf69f..99bc853c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -15,47 +15,24 @@ --- version: '3' services: - zookeeper: - image: confluentinc/cp-zookeeper:5.1.0 + kafka-zookeeper: + image: openzipkin/zipkin-kafka + container_name: kafka-zookeeper ports: - - 2181:2181 - environment: - ZOOKEEPER_CLIENT_PORT: 2181 - ZOOKEEPER_TICK_TIME: 2000 - kafka: - image: confluentinc/cp-kafka:5.1.0 - ports: - - 9092:9092 - - 29092:29092 - environment: - KAFKA_BROKER_ID: 1 - KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 - depends_on: - - zookeeper - kafka-manager: - image: syscomiddleware/kafka-manager:1.3.3 - ports: - - 19000:9000 - environment: - ZK_HOSTS: zookeeper:2181 + - 2181:2181 + - 9092:9092 + - 19092:19092 zipkin: - build: - context: . - args: - KAFKA_STORAGE_VERSION: ${TAG} - image: jeqo/zipkin-kafka:${TAG} + image: jeqo/zipkin-kafka + container_name: zipkin ports: - 9411:9411 environment: - KAFKA_STORE_BOOTSTRAP_SERVERS: kafka:9092 -# KAFKA_STORE_DIRECTORY: /zipkin/data - KAFKA_STORE_COMPRESSION_TYPE: SNAPPY - ZIPKIN_LOGGING_LEVEL: DEBUG -# volumes: -# - zipkin:/zipkin/data -#volumes: -# zipkin: \ No newline at end of file + KAFKA_BOOTSTRAP_SERVERS: kafka-zookeeper:9092 + KAFKA_STORAGE_DIR: /data + KAFKA_STORAGE_TRACES_RETENTION_PERIOD: -1 + KAFKA_STORAGE_TRACES_INACTIVITY_GAP: 5000 + volumes: + - zipkin:/data +volumes: + zipkin: \ No newline at end of file diff --git a/docs/dependencies.png b/docs/dependencies.png new file mode 100644 index 00000000..013c894e Binary files /dev/null and b/docs/dependencies.png differ diff --git a/docs/dependency-aggregation-stream.png b/docs/dependency-aggregation-stream.png deleted file mode 100644 index f12c0f78..00000000 Binary files a/docs/dependency-aggregation-stream.png and /dev/null differ diff --git a/docs/dependency-aggregation-stream.puml b/docs/dependency-aggregation-stream.puml deleted file mode 100644 index 356de8bf..00000000 --- a/docs/dependency-aggregation-stream.puml +++ /dev/null @@ -1,30 +0,0 @@ -@startuml -digraph G { - graph [labelloc=top,label="kafka-streams topology",fontname="Verdana",fontsize=12]; - edge [fontname="Verdana",fontsize=9,labelfontname="Verdana",labelfontsize=9]; - node [fontname="Verdana",fontsize=9,shape=record]; -subgraph cluster_c3 { -label = "Sub-Topology: 0"; - c0 [label="Topic: zipkin-span-dependencies-v1"] - // null - c0 -> c4 []; - c4 [label="Source: KSTREAM-SOURCE-0000000000"] - c5 [label="Processor: KSTREAM-REDUCE-0000000002"] - // null - c4 -> c5 []; - // null - c5 -> c2 []; - c6 [label="Processor: KTABLE-TOSTREAM-0000000003"] - // null - c5 -> c6 []; - c7 [label="Sink: KSTREAM-SINK-0000000004"] - // null - c6 -> c7 []; - // null - c7 -> c1 []; -} - c0 [label="Topic: zipkin-span-dependencies-v1", shape=cds] - c1 [label="Topic: zipkin-dependencies-v1", shape=cds] - c2 [label="Store: KSTREAM-REDUCE-STATE-STORE-0000000001", shape=box3d] -} -@enduml \ No newline at end of file diff --git a/docs/dependency-store-stream.png b/docs/dependency-store-stream.png deleted file mode 100644 index bf417124..00000000 Binary files a/docs/dependency-store-stream.png and /dev/null differ diff --git a/docs/dependency-store-stream.puml b/docs/dependency-store-stream.puml deleted file mode 100644 index 1a2392d4..00000000 --- a/docs/dependency-store-stream.puml +++ /dev/null @@ -1,17 +0,0 @@ -@startuml -digraph G { - graph [labelloc=top,label="kafka-streams topology",fontname="Verdana",fontsize=12]; - edge [fontname="Verdana",fontsize=9,labelfontname="Verdana",labelfontsize=9]; - node [fontname="Verdana",fontsize=9,shape=record]; - c0 [label="Processor: KTABLE-SOURCE-0000000001"] - // null - c1 -> c0 []; - c1 [label="Source: KSTREAM-SOURCE-0000000000"] - c2 [label="Topic: zipkin-dependencies-v1"] - // null - c2 -> c1 []; - c3 [label="Global Store: 0", shape=box3d] - // null - c0 -> c3 []; -} -@enduml diff --git a/docs/dependency-store-topology.png b/docs/dependency-store-topology.png new file mode 100644 index 00000000..7e1fe8ad Binary files /dev/null and b/docs/dependency-store-topology.png differ diff --git a/docs/service-aggregation-stream.png b/docs/service-aggregation-stream.png deleted file mode 100644 index aa59b22f..00000000 Binary files a/docs/service-aggregation-stream.png and /dev/null differ diff --git a/docs/service-aggregation-stream.puml b/docs/service-aggregation-stream.puml deleted file mode 100644 index d614277f..00000000 --- a/docs/service-aggregation-stream.puml +++ /dev/null @@ -1,33 +0,0 @@ -@startuml -digraph G { - graph [labelloc=top,label="kafka-streams topology",fontname="Verdana",fontsize=12]; - edge [fontname="Verdana",fontsize=9,labelfontname="Verdana",labelfontsize=9]; - node [fontname="Verdana",fontsize=9,shape=record]; -subgraph cluster_c3 { -label = "Sub-Topology: 0"; - c0 [label="Topic: zipkin-span-services-v1"] - // null - c0 -> c4 []; - c4 [label="Source: KSTREAM-SOURCE-0000000000"] - c5 [label="Processor: KSTREAM-MAPVALUES-0000000001"] - // null - c4 -> c5 []; - c6 [label="Processor: KSTREAM-AGGREGATE-0000000002"] - // null - c5 -> c6 []; - // null - c6 -> c2 []; - c7 [label="Processor: KTABLE-TOSTREAM-0000000003"] - // null - c6 -> c7 []; - c8 [label="Sink: KSTREAM-SINK-0000000004"] - // null - c7 -> c8 []; - // null - c8 -> c1 []; -} - c0 [label="Topic: zipkin-span-services-v1", shape=cds] - c1 [label="Topic: zipkin-services-v1", shape=cds] - c2 [label="Store: zipkin-services-v1", shape=box3d] -} -@enduml \ No newline at end of file diff --git a/docs/service-store-stream.png b/docs/service-store-stream.png deleted file mode 100644 index 15928b27..00000000 Binary files a/docs/service-store-stream.png and /dev/null differ diff --git a/docs/service-store-stream.puml b/docs/service-store-stream.puml deleted file mode 100644 index 883a8418..00000000 --- a/docs/service-store-stream.puml +++ /dev/null @@ -1,17 +0,0 @@ -@startuml -digraph G { - graph [labelloc=top,label="kafka-streams topology",fontname="Verdana",fontsize=12]; - edge [fontname="Verdana",fontsize=9,labelfontname="Verdana",labelfontsize=9]; - node [fontname="Verdana",fontsize=9,shape=record]; - c0 [label="Processor: KTABLE-SOURCE-0000000001"] - // null - c1 -> c0 []; - c1 [label="Source: KSTREAM-SOURCE-0000000000"] - c2 [label="Topic: zipkin-services-v1"] - // null - c2 -> c1 []; - c3 [label="Global Store: 0", shape=box3d] - // null - c0 -> c3 []; -} -@enduml \ No newline at end of file diff --git a/docs/trace-aggregation-topology.png b/docs/trace-aggregation-topology.png new file mode 100644 index 00000000..9af8a574 Binary files /dev/null and b/docs/trace-aggregation-topology.png differ diff --git a/docs/trace-store-stream.png b/docs/trace-store-stream.png deleted file mode 100644 index a16c80e0..00000000 Binary files a/docs/trace-store-stream.png and /dev/null differ diff --git a/docs/trace-store-stream.puml b/docs/trace-store-stream.puml deleted file mode 100644 index 662ff458..00000000 --- a/docs/trace-store-stream.puml +++ /dev/null @@ -1,17 +0,0 @@ -@startuml -digraph G { - graph [labelloc=top,label="kafka-streams topology",fontname="Verdana",fontsize=12]; - edge [fontname="Verdana",fontsize=9,labelfontname="Verdana",labelfontsize=9]; - node [fontname="Verdana",fontsize=9,shape=record]; - c0 [label="Processor: KTABLE-SOURCE-0000000001"] - // null - c1 -> c0 []; - c1 [label="Source: KSTREAM-SOURCE-0000000000"] - c2 [label="Topic: zipkin-span-v1"] - // null - c2 -> c1 []; - c3 [label="Global Store: 0", shape=box3d] - // null - c0 -> c3 []; -} -@enduml \ No newline at end of file diff --git a/docs/trace-store-topology.png b/docs/trace-store-topology.png new file mode 100644 index 00000000..c341c170 Binary files /dev/null and b/docs/trace-store-topology.png differ diff --git a/docs/traces.png b/docs/traces.png new file mode 100644 index 00000000..268ade08 Binary files /dev/null and b/docs/traces.png differ diff --git a/examples/multi-mode/docker-compose.yml b/examples/multi-mode/docker-compose.yml deleted file mode 100644 index 32ec7f77..00000000 --- a/examples/multi-mode/docker-compose.yml +++ /dev/null @@ -1,61 +0,0 @@ -# -# Copyright 2019 jeqo -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except -# in compliance with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software distributed under the License -# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express -# or implied. See the License for the specific language governing permissions and limitations under -# the License. -# - ---- -version: '3' -services: - zookeeper: - image: confluentinc/cp-zookeeper:5.1.0 - ports: - - 2181:2181 - environment: - ZOOKEEPER_CLIENT_PORT: 2181 - ZOOKEEPER_TICK_TIME: 2000 - kafka: - image: confluentinc/cp-kafka:5.1.0 - ports: - - 9092:9092 - - 29092:29092 - environment: - KAFKA_BROKER_ID: 1 - KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 - depends_on: - - zookeeper - zipkin-ingestion: - image: jeqo/zipkin-kafka:0.3.1-SNAPSHOT - ports: - - 9411:9411 - environment: - KAFKA_STORE_BOOTSTRAP_SERVERS: kafka:9092 - KAFKA_STORE_COMPRESSION_TYPE: SNAPPY - KAFKA_STORE_SPAN_CONSUMER_ENABLED: 'true' - KAFKA_STORE_SPAN_STORE_ENABLED: 'false' - zipkin-store: - image: jeqo/zipkin-kafka:0.3.1-SNAPSHOT - ports: - - 9412:9411 - environment: - KAFKA_STORE_BOOTSTRAP_SERVERS: kafka:9092 - KAFKA_STORE_COMPRESSION_TYPE: SNAPPY - KAFKA_STORE_SPAN_CONSUMER_ENABLED: 'false' - KAFKA_STORE_SPAN_STORE_ENABLED: 'true' - KAFKA_STORE_DIRECTORY: /zipkin/data - volumes: - - zipkin:/zipkin/data -volumes: - zipkin: \ No newline at end of file diff --git a/examples/single-node/docker-compose.yml b/examples/single-node/docker-compose.yml deleted file mode 100644 index c9c3c77e..00000000 --- a/examples/single-node/docker-compose.yml +++ /dev/null @@ -1,50 +0,0 @@ -# -# Copyright 2019 jeqo -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except -# in compliance with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software distributed under the License -# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express -# or implied. See the License for the specific language governing permissions and limitations under -# the License. -# - ---- -version: '3' -services: - zookeeper: - image: confluentinc/cp-zookeeper:5.1.0 - ports: - - 2181:2181 - environment: - ZOOKEEPER_CLIENT_PORT: 2181 - ZOOKEEPER_TICK_TIME: 2000 - kafka: - image: confluentinc/cp-kafka:5.1.0 - ports: - - 9092:9092 - - 29092:29092 - environment: - KAFKA_BROKER_ID: 1 - KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT - KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 - depends_on: - - zookeeper - zipkin: - image: jeqo/zipkin-kafka:0.3.0 - ports: - - 9411:9411 - environment: - KAFKA_STORE_BOOTSTRAP_SERVERS: kafka:9092 - KAFKA_STORE_DIRECTORY: /zipkin/data - KAFKA_STORE_COMPRESSION_TYPE: SNAPPY - volumes: - - zipkin:/zipkin/data -volumes: - zipkin: \ No newline at end of file diff --git a/pom.xml b/pom.xml index 3f615641..2d43e875 100644 --- a/pom.xml +++ b/pom.xml @@ -14,7 +14,9 @@ the License. --> - + 4.0.0 io.github.jeqo.zipkin @@ -37,18 +39,19 @@ UTF-8 - 1.6 - java16 + 1.8 + java18 ${project.basedir} - 2.12.7 - 2.1.3.RELEASE + 2.16.1 + 2.1.7.RELEASE 2.11.1 + 2.10.1 - + 2.3.2 3.0 @@ -91,12 +94,6 @@ zipkin-storage-kafka ${project.version} - - - - - - @@ -106,27 +103,6 @@ zipkin ${zipkin.version} - - - io.zipkin.zipkin2 - zipkin - test-jar - ${zipkin.version} - test - - - - junit - junit - test - 4.12 - - - org.assertj - assertj-core - 3.11.1 - test - @@ -180,24 +156,6 @@ - - net.orfjackal.retrolambda - retrolambda-maven-plugin - 2.5.6 - - - - process-main - - - ${main.java.version} - true - true - - - - - maven-jar-plugin 3.1.0 @@ -252,6 +210,8 @@ **/*.md src/test/resources/** src/main/resources/** + **/*.puml + Makefile true diff --git a/storage/pom.xml b/storage/pom.xml index ea7706a8..b1af0403 100644 --- a/storage/pom.xml +++ b/storage/pom.xml @@ -14,7 +14,9 @@ the License. --> - + 4.0.0 io.github.jeqo.zipkin @@ -27,8 +29,7 @@ ${project.basedir}/.. - 2.1.1 - 7.7.0 + 2.3.0 @@ -44,21 +45,17 @@ ${kafka.version} - org.apache.lucene - lucene-core - ${lucene.version} + io.zipkin.reporter2 + zipkin-sender-kafka + ${zipkin-sender-kafka.version} - - org.apache.lucene - lucene-grouping - ${lucene.version} - - - org.apache.lucene - lucene-queryparser - ${lucene.version} - + + org.junit.jupiter + junit-jupiter + 5.5.1 + test + org.apache.kafka kafka-streams-test-utils @@ -85,10 +82,16 @@ 0.1.3 test + + org.testcontainers + junit-jupiter + 1.12.0 + test + org.testcontainers kafka - 1.10.6 + 1.12.0 test diff --git a/storage/src/main/java/zipkin2/storage/kafka/KafkaAutocompleteTags.java b/storage/src/main/java/zipkin2/storage/kafka/KafkaAutocompleteTags.java new file mode 100644 index 00000000..a89bb513 --- /dev/null +++ b/storage/src/main/java/zipkin2/storage/kafka/KafkaAutocompleteTags.java @@ -0,0 +1,95 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka; + +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import org.apache.kafka.streams.KafkaStreams; +import org.apache.kafka.streams.state.QueryableStoreTypes; +import org.apache.kafka.streams.state.ReadOnlyKeyValueStore; +import zipkin2.Call; +import zipkin2.storage.AutocompleteTags; +import zipkin2.storage.kafka.internal.KafkaStreamsStoreCall; +import zipkin2.storage.kafka.streams.TraceStoreTopologySupplier; + +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.AUTOCOMPLETE_TAGS_STORE_NAME; + +/** + * Autocomplete tags query component based on Kafka Streams local store built by {@link + * TraceStoreTopologySupplier} + * + * These stores are currently supporting only single instance as there is not mechanism implemented + * for scatter gather data from different instances. + */ +public class KafkaAutocompleteTags implements AutocompleteTags { + final KafkaStreams traceStoreStream; + + KafkaAutocompleteTags(KafkaStorage storage) { + traceStoreStream = storage.getTraceStoreStream(); + } + + @Override public Call> getKeys() { + ReadOnlyKeyValueStore> autocompleteTagsStore = + traceStoreStream.store(AUTOCOMPLETE_TAGS_STORE_NAME, + QueryableStoreTypes.keyValueStore()); + return new GetKeysCall(autocompleteTagsStore); + } + + @Override public Call> getValues(String key) { + ReadOnlyKeyValueStore> autocompleteTagsStore = + traceStoreStream.store(AUTOCOMPLETE_TAGS_STORE_NAME, + QueryableStoreTypes.keyValueStore()); + return new GetValuesCall(autocompleteTagsStore, key); + } + + static class GetKeysCall extends KafkaStreamsStoreCall> { + final ReadOnlyKeyValueStore> autocompleteTagsStore; + + GetKeysCall(ReadOnlyKeyValueStore> autocompleteTagsStore) { + this.autocompleteTagsStore = autocompleteTagsStore; + } + + @Override protected List query() { + List keys = new ArrayList<>(); + autocompleteTagsStore.all().forEachRemaining(keyValue -> keys.add(keyValue.key)); + return keys; + } + + @Override public Call> clone() { + return new GetKeysCall(autocompleteTagsStore); + } + } + + static class GetValuesCall extends KafkaStreamsStoreCall> { + final ReadOnlyKeyValueStore> autocompleteTagsStore; + final String key; + + GetValuesCall( + ReadOnlyKeyValueStore> autocompleteTagsStore, String key) { + this.autocompleteTagsStore = autocompleteTagsStore; + this.key = key; + } + + @Override protected List query() { + Set valuesSet = autocompleteTagsStore.get(key); + if (valuesSet == null) return new ArrayList<>(); + return new ArrayList<>(valuesSet); + } + + @Override public Call> clone() { + return new GetValuesCall(autocompleteTagsStore, key); + } + } +} diff --git a/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanConsumer.java b/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanConsumer.java index aff1669e..3e7f4ae4 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanConsumer.java +++ b/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanConsumer.java @@ -15,44 +15,35 @@ import java.io.IOException; import java.util.ArrayList; -import java.util.HashSet; +import java.util.Collections; import java.util.List; -import java.util.Map; -import java.util.Set; import org.apache.kafka.clients.producer.Producer; import org.apache.kafka.clients.producer.ProducerRecord; -import org.apache.kafka.common.serialization.StringSerializer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; +import org.apache.kafka.clients.producer.RecordMetadata; import zipkin2.Call; import zipkin2.Callback; import zipkin2.Span; import zipkin2.codec.SpanBytesEncoder; import zipkin2.internal.AggregateCall; +import zipkin2.reporter.AwaitableCallback; +import zipkin2.reporter.kafka.KafkaSender; import zipkin2.storage.SpanConsumer; /** - * Collected Spans processor. + * Span Consumer to compensate current {@link KafkaSender} distribution of span batched without key. * - * Spans are partitioned by trace ID to enabled downstream processing of spans as part of a trace. + * This component split batch into individual spans keyed by trace ID to enabled downstream + * processing of spans as part of a trace. */ public class KafkaSpanConsumer implements SpanConsumer { // Topic names final String spansTopicName; - final String spanServicesTopicName; // Kafka producers final Producer producer; - final StringSerializer stringSerializer; - // In-memory map of ServiceNames:SpanNames - final Map> serviceSpanMap; - KafkaSpanConsumer(KafkaStorage storage) { - spansTopicName = storage.spansTopic.name; - spanServicesTopicName = storage.spanServicesTopic.name; + spansTopicName = storage.spansTopicName; producer = storage.getProducer(); - stringSerializer = new StringSerializer(); - serviceSpanMap = storage.serviceSpanMap; } @Override @@ -62,26 +53,13 @@ public Call accept(List spans) { // Collect traceId:spans for (Span span : spans) { String key = span.traceId(); - byte[] value = SpanBytesEncoder.PROTO3.encode(span); + byte[] value = SpanBytesEncoder.PROTO3.encodeList(Collections.singletonList(span)); calls.add(KafkaProducerCall.create(producer, spansTopicName, key, value)); - // Check if new spanNames are in place - Set spanNames = serviceSpanMap.getOrDefault(span.localServiceName(), new HashSet<>()); - if (!spanNames.contains(span.name())) { - spanNames.add(span.name()); - serviceSpanMap.put(span.localServiceName(), spanNames); - calls.add(KafkaProducerCall.create( - producer, - spanServicesTopicName, - span.localServiceName(), - stringSerializer.serialize(spanServicesTopicName, span.name()))); - } } return AggregateCall.newVoidCall(calls); } static class KafkaProducerCall extends Call.Base { - static final Logger LOG = LoggerFactory.getLogger(KafkaProducerCall.class); - final Producer kafkaProducer; final String topic; final String key; @@ -100,40 +78,49 @@ static class KafkaProducerCall extends Call.Base { static Call create( Producer producer, - String topicName, + String topic, String key, byte[] value) { - return new KafkaProducerCall(producer, topicName, key, value); + return new KafkaProducerCall(producer, topic, key, value); } @Override + @SuppressWarnings("FutureReturnValueIgnored") protected Void doExecute() throws IOException { - try { - ProducerRecord producerRecord = new ProducerRecord<>(topic, key, value); - kafkaProducer.send(producerRecord); - return null; - } catch (Exception e) { - LOG.error("Error sending span to Kafka", e); - throw new IOException(e); - } + AwaitableCallback callback = new AwaitableCallback(); + kafkaProducer.send(new ProducerRecord<>(topic, key, value), new CallbackAdapter(callback)); + callback.await(); + return null; } @Override @SuppressWarnings("FutureReturnValueIgnored") protected void doEnqueue(Callback callback) { - ProducerRecord producerRecord = new ProducerRecord<>(topic, key, value); - kafkaProducer.send(producerRecord, (recordMetadata, e) -> { - if (e == null) { - callback.onSuccess(null); - } else { - LOG.error("Error sending span to Kafka", e); - callback.onError(e); - } - }); + kafkaProducer.send(new ProducerRecord<>(topic, key, value), new CallbackAdapter(callback)); } @Override public Call clone() { return new KafkaProducerCall(kafkaProducer, topic, key, value); } + + static final class CallbackAdapter implements org.apache.kafka.clients.producer.Callback { + final Callback delegate; + + CallbackAdapter(Callback delegate) { + this.delegate = delegate; + } + + @Override public void onCompletion(RecordMetadata metadata, Exception exception) { + if (exception == null) { + delegate.onSuccess(null); + } else { + delegate.onError(exception); + } + } + + @Override public String toString() { + return delegate.toString(); + } + } } } diff --git a/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanStore.java b/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanStore.java index e73945b0..fe3def2b 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanStore.java +++ b/storage/src/main/java/zipkin2/storage/kafka/KafkaSpanStore.java @@ -13,200 +13,205 @@ */ package zipkin2.storage.kafka; -import java.io.IOException; +import java.time.Instant; import java.util.ArrayList; -import java.util.HashMap; +import java.util.Collections; import java.util.List; -import java.util.Map; import java.util.Set; import org.apache.kafka.streams.KafkaStreams; +import org.apache.kafka.streams.state.KeyValueIterator; import org.apache.kafka.streams.state.QueryableStoreTypes; import org.apache.kafka.streams.state.ReadOnlyKeyValueStore; +import org.apache.kafka.streams.state.ReadOnlyWindowStore; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import zipkin2.Call; -import zipkin2.Callback; import zipkin2.DependencyLink; import zipkin2.Span; +import zipkin2.internal.DependencyLinker; import zipkin2.storage.QueryRequest; +import zipkin2.storage.ServiceAndSpanNames; import zipkin2.storage.SpanStore; -import zipkin2.storage.kafka.index.SpanIndexService; +import zipkin2.storage.kafka.internal.KafkaStreamsStoreCall; +import zipkin2.storage.kafka.streams.DependencyStoreTopologySupplier; +import zipkin2.storage.kafka.streams.TraceStoreTopologySupplier; + +import static zipkin2.storage.kafka.streams.DependencyStoreTopologySupplier.DEPENDENCIES_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.REMOTE_SERVICE_NAMES_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.SERVICE_NAMES_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.SPAN_IDS_BY_TS_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.SPAN_NAMES_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.TRACES_STORE_NAME; /** - * Span Store based on Kafka Streams. - * - * This store supports all searches (e.g. findTraces, getTrace, getServiceNames, getSpanNames, and - * getDependencies). - * - * NOTE: Currently State Stores are based on global state stores (i.e., all data is replicated on - * every Zipkin instance with spanStoreEnabled=true). + * Span store backed by Kafka Stream local stores built by {@link TraceStoreTopologySupplier} and + * {@link DependencyStoreTopologySupplier}. + *

+ * These stores are currently supporting only single instance as there is not mechanism implemented + * for scatter gather data from different instances. */ -public class KafkaSpanStore implements SpanStore { - private static final Logger LOG = LoggerFactory.getLogger(KafkaSpanStore.class); - // Store names - final String tracesStoreName; - final String servicesStoreName; - final String dependenciesStoreName; - // Kafka Streams +public class KafkaSpanStore implements SpanStore, ServiceAndSpanNames { + static final Logger LOG = LoggerFactory.getLogger(KafkaSpanStore.class); + // Kafka Streams Store provider final KafkaStreams traceStoreStream; - final KafkaStreams serviceStoreStream; final KafkaStreams dependencyStoreStream; - // Span index - final SpanIndexService spanIndexService; KafkaSpanStore(KafkaStorage storage) { - tracesStoreName = storage.traceStoreName; - servicesStoreName = storage.serviceStoreName; - dependenciesStoreName = storage.dependencyStoreName; traceStoreStream = storage.getTraceStoreStream(); - serviceStoreStream = storage.getServiceStoreStream(); dependencyStoreStream = storage.getDependencyStoreStream(); - spanIndexService = storage.getSpanIndexService(); } - @Override - public Call>> getTraces(QueryRequest request) { - try { - ReadOnlyKeyValueStore> traceStore = - traceStoreStream.store(tracesStoreName, QueryableStoreTypes.keyValueStore()); - return new GetTracesCall(traceStore, spanIndexService, request); - } catch (Exception e) { - LOG.error("Error getting traces", request, e); - return Call.emptyList(); - } + @Override public Call>> getTraces(QueryRequest request) { + ReadOnlyKeyValueStore> tracesStore = + traceStoreStream.store(TRACES_STORE_NAME, QueryableStoreTypes.keyValueStore()); + ReadOnlyKeyValueStore> traceIdsByTsStore = + traceStoreStream.store(SPAN_IDS_BY_TS_STORE_NAME, QueryableStoreTypes.keyValueStore()); + return new GetTracesCall(tracesStore, traceIdsByTsStore, request); } @Override public Call> getTrace(String traceId) { - try { - ReadOnlyKeyValueStore> traceStore = - traceStoreStream.store(tracesStoreName, QueryableStoreTypes.keyValueStore()); - return new GetTraceCall(traceStore, traceId); - } catch (Exception e) { - LOG.error("Error getting trace {}", traceId, e); - return Call.emptyList(); - } + ReadOnlyKeyValueStore> traceStore = + traceStoreStream.store(TRACES_STORE_NAME, QueryableStoreTypes.keyValueStore()); + return new GetTraceCall(traceStore, traceId); } - @Override - public Call> getServiceNames() { - try { - ReadOnlyKeyValueStore> serviceStore = - serviceStoreStream.store(servicesStoreName, QueryableStoreTypes.keyValueStore()); - return new GetServiceNamesCall(serviceStore); - } catch (Exception e) { - LOG.error("Error getting service names", e); - return Call.emptyList(); - } + @Deprecated @Override public Call> getServiceNames() { + ReadOnlyKeyValueStore serviceStore = + traceStoreStream.store(SERVICE_NAMES_STORE_NAME, QueryableStoreTypes.keyValueStore()); + return new GetServiceNamesCall(serviceStore); } - @Override - public Call> getSpanNames(String serviceName) { - try { - ReadOnlyKeyValueStore> serviceStore = - serviceStoreStream.store(servicesStoreName, QueryableStoreTypes.keyValueStore()); - return new GetSpanNamesCall(serviceStore, serviceName); - } catch (Exception e) { - LOG.error("Error getting span names from service {}", serviceName, e); - return Call.emptyList(); - } + @Deprecated @Override public Call> getSpanNames(String serviceName) { + ReadOnlyKeyValueStore> spanNamesStore = + traceStoreStream.store(SPAN_NAMES_STORE_NAME, QueryableStoreTypes.keyValueStore()); + return new GetSpanNamesCall(spanNamesStore, serviceName); } - @Override - public Call> getDependencies(long endTs, long lookback) { - try { - ReadOnlyKeyValueStore dependenciesStore = - dependencyStoreStream. - store(dependenciesStoreName, QueryableStoreTypes.keyValueStore()); - return new GetDependenciesCall(endTs, lookback, dependenciesStore); - } catch (Exception e) { - LOG.error("Error getting dependencies", e); - return Call.emptyList(); - } + @Override public Call> getRemoteServiceNames(String serviceName) { + ReadOnlyKeyValueStore> remoteServiceNamesStore = + traceStoreStream.store(REMOTE_SERVICE_NAMES_STORE_NAME, + QueryableStoreTypes.keyValueStore()); + return new GetRemoteServiceNamesCall(remoteServiceNamesStore, serviceName); + } + + @Override public Call> getDependencies(long endTs, long lookback) { + ReadOnlyWindowStore dependenciesStore = + dependencyStoreStream.store(DEPENDENCIES_STORE_NAME, + QueryableStoreTypes.windowStore()); + return new GetDependenciesCall(endTs, lookback, dependenciesStore); } static class GetServiceNamesCall extends KafkaStreamsStoreCall> { - ReadOnlyKeyValueStore> serviceStore; + ReadOnlyKeyValueStore serviceStore; - GetServiceNamesCall(ReadOnlyKeyValueStore> serviceStore) { + GetServiceNamesCall(ReadOnlyKeyValueStore serviceStore) { this.serviceStore = serviceStore; } - @Override - List query() { - try { - List keys = new ArrayList<>(); - serviceStore.all().forEachRemaining(keyValue -> keys.add(keyValue.key)); - return keys; - } catch (Exception e) { - LOG.error("Error looking up services", e); - return new ArrayList<>(); - } + @Override public List query() { + List serviceNames = new ArrayList<>(); + serviceStore.all().forEachRemaining(keyValue -> serviceNames.add(keyValue.value)); + // comply with Zipkin API as service names are required to be ordered lexicographically + Collections.sort(serviceNames); + return serviceNames; } - @Override - public Call> clone() { + @Override public Call> clone() { return new GetServiceNamesCall(serviceStore); } } static class GetSpanNamesCall extends KafkaStreamsStoreCall> { - final ReadOnlyKeyValueStore> serviceStore; + final ReadOnlyKeyValueStore> spanNamesStore; final String serviceName; - GetSpanNamesCall(ReadOnlyKeyValueStore> serviceStore, String serviceName) { - this.serviceStore = serviceStore; + GetSpanNamesCall(ReadOnlyKeyValueStore> spanNamesStore, + String serviceName) { + this.spanNamesStore = spanNamesStore; this.serviceName = serviceName; } - @Override - List query() { - try { - if (serviceName == null || serviceName.equals("all")) return new ArrayList<>(); - Set spanNames = serviceStore.get(serviceName); - if (spanNames == null) return new ArrayList<>(); - return new ArrayList<>(spanNames); - } catch (Exception e) { - LOG.error("Error looking up for span names for service {}", serviceName, e); - return new ArrayList<>(); - } + @Override public List query() { + if (serviceName == null) return new ArrayList<>(); + Set spanNamesSet = spanNamesStore.get(serviceName); + if (spanNamesSet == null) return new ArrayList<>(); + List spanNames = new ArrayList<>(spanNamesSet); + // comply with Zipkin API as service names are required to be ordered lexicographically and store returns unordered values + Collections.sort(spanNames); + return spanNames; } - @Override - public Call> clone() { - return new GetSpanNamesCall(serviceStore, serviceName); + @Override public Call> clone() { + return new GetSpanNamesCall(spanNamesStore, serviceName); + } + } + + static class GetRemoteServiceNamesCall extends KafkaStreamsStoreCall> { + final ReadOnlyKeyValueStore> remoteServiceNamesStore; + final String serviceName; + + GetRemoteServiceNamesCall(ReadOnlyKeyValueStore> remoteServiceNamesStore, + String serviceName) { + this.remoteServiceNamesStore = remoteServiceNamesStore; + this.serviceName = serviceName; + } + + @Override public List query() { + if (serviceName == null) return new ArrayList<>(); + Set remoteServiceNamesSet = remoteServiceNamesStore.get(serviceName); + if (remoteServiceNamesSet == null) return new ArrayList<>(); + List remoteServiceNames = new ArrayList<>(remoteServiceNamesSet); + // comply with Zipkin API as service names are required to be ordered lexicographically + Collections.sort(remoteServiceNames); + return remoteServiceNames; + } + + @Override public Call> clone() { + return new GetRemoteServiceNamesCall(remoteServiceNamesStore, serviceName); } } static class GetTracesCall extends KafkaStreamsStoreCall>> { - final ReadOnlyKeyValueStore> traceStore; - final SpanIndexService spanIndexService; + final ReadOnlyKeyValueStore> tracesStore; + final ReadOnlyKeyValueStore> traceIdsByTsStore; final QueryRequest queryRequest; GetTracesCall( - ReadOnlyKeyValueStore> traceStore, - SpanIndexService spanIndexService, + ReadOnlyKeyValueStore> tracesStore, + ReadOnlyKeyValueStore> traceIdsByTsStore, QueryRequest queryRequest) { - this.traceStore = traceStore; - this.spanIndexService = spanIndexService; + this.tracesStore = tracesStore; + this.traceIdsByTsStore = traceIdsByTsStore; this.queryRequest = queryRequest; } - List> query() { - List> result = new ArrayList<>(); - for (String traceId : spanIndexService.getTraceIds(queryRequest)) { - List spans = traceStore.get(traceId); - result.add(spans); - } - - LOG.info("Total results of query {}: {}", queryRequest, result.size()); - - return result; + @Override public List> query() { + List> traces = new ArrayList<>(); + List traceIds = new ArrayList<>(); + // milliseconds to microseconds + long from = (queryRequest.endTs() - queryRequest.lookback()) * 1000; + long to = queryRequest.endTs() * 1000; + // first index + KeyValueIterator> spanIds = traceIdsByTsStore.range(from, to); + spanIds.forEachRemaining(keyValue -> { + for (String traceId : keyValue.value) { + if (!traceIds.contains(traceId) && traces.size() < queryRequest.limit()) { + List spans = tracesStore.get(traceId); + if (spans != null && queryRequest.test(spans)) { // apply filters + traceIds.add(traceId); // adding to check if we have already add it later + traces.add(spans); + } + } + } + }); + LOG.debug("Traces found from query {}: {}", queryRequest, traces.size()); + return traces; } @Override public Call>> clone() { - return new GetTracesCall(traceStore, spanIndexService, queryRequest); + return new GetTracesCall(tracesStore, traceIdsByTsStore, queryRequest); } } @@ -221,85 +226,41 @@ static class GetTraceCall extends KafkaStreamsStoreCall> { this.traceId = traceId; } - @Override - List query() { - try { - final List spans = traceStore.get(traceId); - if (spans == null) return new ArrayList<>(); - return spans; - } catch (Exception e) { - LOG.error("Error getting trace with ID {}", traceId, e); - return null; - } + @Override public List query() { + final List spans = traceStore.get(traceId); + if (spans == null) return new ArrayList<>(); + return spans; } - @Override - public Call> clone() { + @Override public Call> clone() { return new GetTraceCall(traceStore, traceId); } } static class GetDependenciesCall extends KafkaStreamsStoreCall> { final long endTs, loopback; - final ReadOnlyKeyValueStore dependenciesStore; + final ReadOnlyWindowStore dependenciesStore; GetDependenciesCall(long endTs, long loopback, - ReadOnlyKeyValueStore dependenciesStore) { + ReadOnlyWindowStore dependenciesStore) { this.endTs = endTs; this.loopback = loopback; this.dependenciesStore = dependenciesStore; } - @Override - List query() { - try { - Map dependencyLinks = new HashMap<>(); - long from = endTs - loopback; - dependenciesStore.range(from, endTs) - .forEachRemaining(dependencyLink -> { - String pair = String.format("%s-%s", dependencyLink.value.parent(), - dependencyLink.value.child()); - dependencyLinks.put(pair, dependencyLink.value); - }); - - LOG.info("Dependencies found from={}-to={}: {}", from, endTs, dependencyLinks.size()); - - return new ArrayList<>(dependencyLinks.values()); - } catch (Exception e) { - LOG.error("Error looking up for dependencies", e); - return new ArrayList<>(); - } + @Override public List query() { + List links = new ArrayList<>(); + Instant from = Instant.ofEpochMilli(endTs - loopback); + Instant to = Instant.ofEpochMilli(endTs); + dependenciesStore.fetchAll(from, to) + .forEachRemaining(keyValue -> links.add(keyValue.value)); + List mergedLinks = DependencyLinker.merge(links); + LOG.debug("Dependencies found from={}-to={}: {}", from, to, mergedLinks.size()); + return mergedLinks; } - @Override - public Call> clone() { + @Override public Call> clone() { return new GetDependenciesCall(endTs, loopback, dependenciesStore); } } - - abstract static class KafkaStreamsStoreCall extends Call.Base { - - KafkaStreamsStoreCall() { - } - - @Override - protected T doExecute() throws IOException { - try { - return query(); - } catch (Exception e) { - throw new IOException(e); - } - } - - @Override - protected void doEnqueue(Callback callback) { - try { - callback.onSuccess(query()); - } catch (Exception e) { - callback.onError(e); - } - } - - abstract T query(); - } } diff --git a/storage/src/main/java/zipkin2/storage/kafka/KafkaStorage.java b/storage/src/main/java/zipkin2/storage/kafka/KafkaStorage.java index 77ec228c..b9ba7370 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/KafkaStorage.java +++ b/storage/src/main/java/zipkin2/storage/kafka/KafkaStorage.java @@ -13,25 +13,20 @@ */ package zipkin2.storage.kafka; -import java.io.IOException; import java.time.Duration; +import java.util.ArrayList; import java.util.Arrays; -import java.util.HashMap; -import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Properties; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.TimeUnit; import org.apache.kafka.clients.admin.AdminClient; import org.apache.kafka.clients.admin.AdminClientConfig; -import org.apache.kafka.clients.admin.NewTopic; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.Producer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.common.KafkaFuture; -import org.apache.kafka.common.config.TopicConfig; import org.apache.kafka.common.record.CompressionType; import org.apache.kafka.common.serialization.ByteArraySerializer; import org.apache.kafka.common.serialization.Serdes; @@ -45,204 +40,84 @@ import zipkin2.CheckResult; import zipkin2.DependencyLink; import zipkin2.Span; +import zipkin2.storage.AutocompleteTags; import zipkin2.storage.QueryRequest; +import zipkin2.storage.ServiceAndSpanNames; import zipkin2.storage.SpanConsumer; import zipkin2.storage.SpanStore; import zipkin2.storage.StorageComponent; -import zipkin2.storage.kafka.index.SpanIndexService; -import zipkin2.storage.kafka.streams.DependencyAggregationStream; -import zipkin2.storage.kafka.streams.DependencyStoreStream; -import zipkin2.storage.kafka.streams.ServiceAggregationStream; -import zipkin2.storage.kafka.streams.ServiceStoreStream; -import zipkin2.storage.kafka.streams.TraceAggregationStream; -import zipkin2.storage.kafka.streams.TraceRetentionStoreStream; -import zipkin2.storage.kafka.streams.TraceStoreStream; +import zipkin2.storage.kafka.streams.AggregationTopologySupplier; +import zipkin2.storage.kafka.streams.DependencyStoreTopologySupplier; +import zipkin2.storage.kafka.streams.TraceStoreTopologySupplier; /** - * Kafka Storage entry-point. - * - * Storage implementation based on Kafka Streams State Stores, supporting aggregation of spans, - * indexing of traces and retention management. + * Zipkin's Kafka Storage. + *

+ * Storage implementation based on Kafka Streams, supporting: + *

    + *
  • repartitioning of spans,
    • + *
  • trace aggregation,
    • + *
  • autocomplete tags, and
    • + *
  • indexing of traces and dependencies.
    • + *
*/ public class KafkaStorage extends StorageComponent { static final Logger LOG = LoggerFactory.getLogger(KafkaStorage.class); // Kafka Storage modes - final boolean spanConsumerEnabled, spanStoreEnabled, aggregationEnabled; - final boolean ensureTopics; + final boolean spanConsumerEnabled, searchEnabled; + // Autocomplete Tags + final List autocompleteKeys; // Kafka Storage configs - final String storageDirectory, traceStoreName, dependencyStoreName, serviceStoreName; + final String storageDirectory; // Kafka Topics - final Topic spansTopic, tracesTopic, spanServicesTopic, servicesTopic, spanDependenciesTopic, - dependenciesTopic; + final String spansTopicName, traceTopicName, dependencyTopicName; // Kafka Clients config final Properties adminConfig; final Properties producerConfig; // Kafka Streams topology configs - final Properties traceStoreStreamConfig, serviceStoreStreamConfig, dependencyStoreStreamConfig, - serviceAggregationStreamConfig, dependencyAggregationStreamConfig, traceRetentionStreamConfig, - traceAggregationStreamConfig; - final Topology traceStoreTopology, serviceStoreTopology, dependencyStoreTopology, - serviceAggregationTopology, dependencyAggregationTopology, traceAggregationTopology, - traceRetentionTopology; - final String spanIndexDirectory; + final Properties aggregationStreamConfig, traceStoreStreamConfig, dependencyStoreStreamConfig; + final Topology aggregationTopology, traceStoreTopology, dependencyStoreTopology; // Resources volatile AdminClient adminClient; volatile Producer producer; - volatile KafkaStreams serviceAggregationStream, dependencyAggregationStream, traceStoreStream, - serviceStoreStream, dependencyStoreStream, traceAggregationStream, traceRetentionStream; + volatile KafkaStreams traceAggregationStream, traceStoreStream, dependencyStoreStream; volatile boolean closeCalled, topicsValidated; - volatile SpanIndexService spanIndexService; - volatile Map> serviceSpanMap; KafkaStorage(Builder builder) { // Kafka Storage modes this.spanConsumerEnabled = builder.spanConsumerEnabled; - this.spanStoreEnabled = builder.spanStoreEnabled; - this.aggregationEnabled = builder.aggregationEnabled; + this.searchEnabled = builder.searchEnabled; + // Autocomplete tags + this.autocompleteKeys = builder.autocompleteKeys; // Kafka Topics config - this.ensureTopics = builder.ensureTopics; - this.spansTopic = builder.spansTopic; - this.tracesTopic = builder.tracesTopic; - this.spanServicesTopic = builder.spanServicesTopic; - this.servicesTopic = builder.servicesTopic; - this.spanDependenciesTopic = builder.spanDependenciesTopic; - this.dependenciesTopic = builder.dependenciesTopic; + this.spansTopicName = builder.spansTopicName; + this.traceTopicName = builder.traceTopicName; + this.dependencyTopicName = builder.dependencyTopicName; // State store directories - this.storageDirectory = builder.storeDirectory; - this.traceStoreName = builder.traceStoreName; - this.dependencyStoreName = builder.dependencyStoreName; - this.serviceStoreName = builder.serviceStoreName; - // Span Index service - spanIndexDirectory = builder.spanIndexDirectory(); - // Service:Span names map - serviceSpanMap = new ConcurrentHashMap<>(); - // Kafka Clients configuration - adminConfig = new Properties(); - adminConfig.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, builder.bootstrapServers); - // Kafka Producer configuration - producerConfig = new Properties(); - producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, builder.bootstrapServers); - producerConfig.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); - producerConfig.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class); - producerConfig.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); - producerConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, builder.compressionType.name); - // Trace Store Stream Topology configuration - traceStoreStreamConfig = new Properties(); - traceStoreStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, builder.bootstrapServers); - traceStoreStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - traceStoreStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - traceStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.traceStoreStreamAppId); - traceStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, - builder.traceStoreDirectory()); - traceStoreStreamConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, - builder.compressionType.name); - traceStoreStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); - traceStoreTopology = - new TraceStoreStream(spansTopic.name, traceStoreName, getSpanIndexService()).get(); - // Service Aggregation topology - serviceAggregationStreamConfig = new Properties(); - serviceAggregationStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, - builder.bootstrapServers); - serviceAggregationStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - serviceAggregationStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - serviceAggregationStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.serviceAggregationStreamAppId); - serviceAggregationStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, - builder.serviceStoreDirectory()); - serviceAggregationStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); - serviceAggregationTopology = - new ServiceAggregationStream(spanServicesTopic.name, servicesTopic.name) - .get(); - // Service Store topology - serviceStoreStreamConfig = new Properties(); - serviceStoreStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, builder.bootstrapServers); - serviceStoreStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - serviceStoreStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - serviceStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.serviceStoreStreamAppId); - serviceStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, builder.serviceStoreDirectory()); - serviceStoreStreamConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, - builder.compressionType.name); - serviceStoreStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); - serviceStoreTopology = new ServiceStoreStream(servicesTopic.name, serviceStoreName).get(); - // Dependency Aggregation topology - dependencyAggregationStreamConfig = new Properties(); - dependencyAggregationStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, - builder.bootstrapServers); - dependencyAggregationStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - dependencyAggregationStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - dependencyAggregationStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.dependencyAggregationStreamAppId); - dependencyAggregationStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, - builder.dependencyStoreDirectory()); - dependencyAggregationStreamConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, - builder.compressionType.name); - dependencyAggregationStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, - StreamsConfig.OPTIMIZE); - dependencyAggregationTopology = - new DependencyAggregationStream(tracesTopic.name, spanDependenciesTopic.name, - dependenciesTopic.name).get(); - // Dependency Store topology - dependencyStoreStreamConfig = new Properties(); - dependencyStoreStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, - builder.bootstrapServers); - dependencyStoreStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - dependencyStoreStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - dependencyStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.dependencyStoreStreamAppId); - dependencyStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, - builder.dependencyStoreDirectory()); - dependencyStoreStreamConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, - builder.compressionType.name); - dependencyStoreStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); - dependencyStoreTopology = - new DependencyStoreStream(dependenciesTopic.name, dependencyStoreName).get(); - // Trace Retention topology - traceRetentionStreamConfig = new Properties(); - traceRetentionStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, - builder.bootstrapServers); - traceRetentionStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - traceRetentionStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - traceRetentionStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.traceRetentionStreamAppId); - traceRetentionStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, builder.traceStoreDirectory()); - traceRetentionStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); - traceRetentionStreamConfig.put( - StreamsConfig.PRODUCER_PREFIX + ProducerConfig.COMPRESSION_TYPE_CONFIG, - builder.compressionType.name); - traceRetentionTopology = new TraceRetentionStoreStream(spansTopic.name, traceStoreName, - builder.retentionScanFrequency, builder.retentionMaxAge).get(); - // Trace Aggregation topology - traceAggregationStreamConfig = new Properties(); - traceAggregationStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, - builder.bootstrapServers); - traceAggregationStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, - Serdes.StringSerde.class); - traceAggregationStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, - Serdes.ByteArraySerde.class); - traceAggregationStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, - builder.traceAggregationStreamAppId); - traceAggregationStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, builder.traceStoreDirectory()); - traceAggregationStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); - traceAggregationStreamConfig.put( - StreamsConfig.PRODUCER_PREFIX + ProducerConfig.COMPRESSION_TYPE_CONFIG, - builder.compressionType.name); - traceAggregationTopology = new TraceAggregationStream(spansTopic.name, tracesTopic.name, - builder.traceInactivityGap).get(); + this.storageDirectory = builder.storeDir; + // Kafka Configs + this.adminConfig = builder.adminConfig; + this.producerConfig = builder.producerConfig; + this.aggregationStreamConfig = builder.aggregationStreamConfig; + this.traceStoreStreamConfig = builder.traceStoreStreamConfig; + this.dependencyStoreStreamConfig = builder.dependencyStoreStreamConfig; + + aggregationTopology = new AggregationTopologySupplier( + spansTopicName, + traceTopicName, + dependencyTopicName, + builder.traceTimeout).get(); + traceStoreTopology = new TraceStoreTopologySupplier( + spansTopicName, + autocompleteKeys, + builder.traceTtl, + builder.traceTtlCheckInterval, + builder.minTracesStored).get(); + dependencyStoreTopology = new DependencyStoreTopologySupplier( + dependencyTopicName, + builder.dependencyTtl, + builder.dependencyWindowSize).get(); } public static Builder newBuilder() { @@ -251,13 +126,9 @@ public static Builder newBuilder() { @Override public SpanConsumer spanConsumer() { - if (ensureTopics && !topicsValidated) ensureTopics(); - if (aggregationEnabled) { - getTraceAggregationStream(); - getServiceAggregationStream(); - getDependencyAggregationStream(); - } + checkTopics(); if (spanConsumerEnabled) { + getAggregationStream(); return new KafkaSpanConsumer(this); } else { // NoopSpanConsumer return list -> Call.create(null); @@ -265,13 +136,30 @@ public SpanConsumer spanConsumer() { } @Override - public SpanStore spanStore() { - if (ensureTopics && !topicsValidated) ensureTopics(); - if (aggregationEnabled) { - getServiceAggregationStream(); - getDependencyAggregationStream(); + public ServiceAndSpanNames serviceAndSpanNames() { + if (searchEnabled) { + return new KafkaSpanStore(this); + } else { // NoopServiceAndSpanNames + return new ServiceAndSpanNames() { + @Override public Call> getServiceNames() { + return Call.emptyList(); + } + + @Override public Call> getRemoteServiceNames(String serviceName) { + return Call.emptyList(); + } + + @Override public Call> getSpanNames(String s) { + return Call.emptyList(); + } + }; } - if (spanStoreEnabled) { + } + + @Override + public SpanStore spanStore() { + checkTopics(); + if (searchEnabled) { return new KafkaSpanStore(this); } else { // NoopSpanStore return new SpanStore() { @@ -283,11 +171,11 @@ public SpanStore spanStore() { return Call.emptyList(); } - @Override public Call> getServiceNames() { + @Override @Deprecated public Call> getServiceNames() { return Call.emptyList(); } - @Override public Call> getSpanNames(String s) { + @Override @Deprecated public Call> getSpanNames(String s) { return Call.emptyList(); } @@ -298,26 +186,34 @@ public SpanStore spanStore() { } } - void ensureTopics() { + @Override public AutocompleteTags autocompleteTags() { + checkTopics(); + if (searchEnabled) { + return new KafkaAutocompleteTags(this); + } else { + return super.autocompleteTags(); + } + } + + /** + * Ensure topics are created before Kafka Streams applications start. + *

+ * It is recommended to created these topics manually though, before application is started. + */ + void checkTopics() { if (!topicsValidated) { synchronized (this) { if (!topicsValidated) { try { Set topics = getAdminClient().listTopics().names().get(1, TimeUnit.SECONDS); - List requiredTopics = - Arrays.asList(spansTopic, spanServicesTopic, servicesTopic, spanDependenciesTopic, - dependenciesTopic, tracesTopic); - Set newTopics = new HashSet<>(); - for (Topic requiredTopic : requiredTopics) { - if (!topics.contains(requiredTopic.name)) { - NewTopic newTopic = requiredTopic.newTopic(); - newTopics.add(newTopic); - } else { - LOG.info("Topic {} already exists.", requiredTopic.name); + List requiredTopics = + Arrays.asList(spansTopicName, dependencyTopicName, traceTopicName); + for (String requiredTopic : requiredTopics) { + if (!topics.contains(requiredTopic)) { + LOG.error("Topic {} not found", requiredTopic); + throw new RuntimeException("Required topics are not created"); } } - - getAdminClient().createTopics(newTopics).all().get(); topicsValidated = true; } catch (Exception e) { LOG.error("Error ensuring topics are created", e); @@ -331,6 +227,20 @@ void ensureTopics() { try { KafkaFuture maybeClusterId = getAdminClient().describeCluster().clusterId(); maybeClusterId.get(1, TimeUnit.SECONDS); + if (spanConsumerEnabled) { + KafkaStreams.State state = getAggregationStream().state(); + if (!state.isRunning()) { + return CheckResult.failed( + new IllegalStateException("Aggregation stream not running. " + state)); + } + } + if (searchEnabled) { + KafkaStreams.State state = getTraceStoreStream().state(); + if (!state.isRunning()) { + return CheckResult.failed( + new IllegalStateException("Store stream not running. " + state)); + } + } return CheckResult.OK; } catch (Exception e) { return CheckResult.failed(e); @@ -349,32 +259,20 @@ void ensureTopics() { void doClose() { try { - if (adminClient != null) adminClient.close(1, TimeUnit.SECONDS); + if (adminClient != null) adminClient.close(Duration.ofSeconds(1)); if (producer != null) { producer.flush(); - producer.close(1, TimeUnit.SECONDS); + producer.close(Duration.ofSeconds(1)); } if (traceStoreStream != null) { traceStoreStream.close(Duration.ofSeconds(1)); } - if (traceRetentionStream != null) { - traceRetentionStream.close(Duration.ofSeconds(1)); + if (dependencyStoreStream != null) { + dependencyStoreStream.close(Duration.ofSeconds(1)); } if (traceAggregationStream != null) { traceAggregationStream.close(Duration.ofSeconds(1)); } - if (serviceAggregationStream != null) { - serviceAggregationStream.close(Duration.ofSeconds(1)); - } - if (serviceStoreStream != null) { - serviceStoreStream.close(Duration.ofSeconds(1)); - } - if (dependencyAggregationStream != null) { - dependencyAggregationStream.close(Duration.ofSeconds(1)); - } - if (dependencyStoreStream != null) { - dependencyStoreStream.close(Duration.ofSeconds(1)); - } } catch (Exception | Error e) { LOG.warn("error closing client {}", e.getMessage(), e); } @@ -408,32 +306,31 @@ KafkaStreams getTraceStoreStream() { if (traceStoreStream == null) { traceStoreStream = new KafkaStreams(traceStoreTopology, traceStoreStreamConfig); traceStoreStream.start(); - getTraceRetentionStream(); } } } return traceStoreStream; } - KafkaStreams getTraceRetentionStream() { - if (traceRetentionStream == null) { + KafkaStreams getDependencyStoreStream() { + if (dependencyStoreStream == null) { synchronized (this) { - if (traceRetentionStream == null) { - traceRetentionStream = - new KafkaStreams(traceRetentionTopology, traceRetentionStreamConfig); - traceRetentionStream.start(); + if (dependencyStoreStream == null) { + dependencyStoreStream = + new KafkaStreams(dependencyStoreTopology, dependencyStoreStreamConfig); + dependencyStoreStream.start(); } } } - return traceRetentionStream; + return dependencyStoreStream; } - KafkaStreams getTraceAggregationStream() { + KafkaStreams getAggregationStream() { if (traceAggregationStream == null) { synchronized (this) { if (traceAggregationStream == null) { traceAggregationStream = - new KafkaStreams(traceAggregationTopology, traceAggregationStreamConfig); + new KafkaStreams(aggregationTopology, aggregationStreamConfig); traceAggregationStream.start(); } } @@ -441,139 +338,100 @@ KafkaStreams getTraceAggregationStream() { return traceAggregationStream; } - KafkaStreams getServiceAggregationStream() { - if (serviceAggregationStream == null) { - synchronized (this) { - if (serviceAggregationStream == null) { - serviceAggregationStream = - new KafkaStreams(serviceAggregationTopology, serviceAggregationStreamConfig); - serviceAggregationStream.start(); - } - } - } - return serviceAggregationStream; - } + public static class Builder extends StorageComponent.Builder { + boolean spanConsumerEnabled = true; + boolean searchEnabled = true; - KafkaStreams getServiceStoreStream() { - if (serviceStoreStream == null) { - synchronized (this) { - if (serviceStoreStream == null) { - serviceStoreStream = new KafkaStreams(serviceStoreTopology, serviceStoreStreamConfig); - serviceStoreStream.start(); - } - } - } - return serviceStoreStream; - } + List autocompleteKeys = new ArrayList<>(); - KafkaStreams getDependencyAggregationStream() { - if (dependencyAggregationStream == null) { - synchronized (this) { - if (dependencyAggregationStream == null) { - dependencyAggregationStream = - new KafkaStreams(dependencyAggregationTopology, dependencyAggregationStreamConfig); - dependencyAggregationStream.start(); - } - } - } - return dependencyAggregationStream; - } + Duration traceTtl = Duration.ofDays(3); + Duration traceTtlCheckInterval = Duration.ofHours(1); + Duration traceTimeout = Duration.ofMinutes(1); + Duration dependencyTtl = Duration.ofDays(7); + Duration dependencyWindowSize = Duration.ofMinutes(1); - KafkaStreams getDependencyStoreStream() { - if (dependencyStoreStream == null) { - synchronized (this) { - if (dependencyStoreStream == null) { - dependencyStoreStream = - new KafkaStreams(dependencyStoreTopology, dependencyStoreStreamConfig); - dependencyStoreStream.start(); - } - } - } - return dependencyStoreStream; - } + long minTracesStored = 10_000; - SpanIndexService getSpanIndexService() { - if (spanIndexService == null) { - synchronized (this) { - if (spanIndexService == null) { - try { - spanIndexService = SpanIndexService.create(spanIndexDirectory); - } catch (IOException e) { - LOG.error("Error creating span index service", e); - } - } - } - } - return spanIndexService; - } + String storeDir = "/tmp/zipkin-storage-kafka"; - public static class Builder extends StorageComponent.Builder { - boolean spanConsumerEnabled = true; - boolean spanStoreEnabled = true; - boolean aggregationEnabled = true; - - Duration retentionScanFrequency = Duration.ofMinutes(1); - Duration retentionMaxAge = Duration.ofMinutes(2); - - String bootstrapServers = "localhost:29092"; - CompressionType compressionType = CompressionType.NONE; - - Duration traceInactivityGap = Duration.ofMinutes(1); - - String traceStoreStreamAppId = "zipkin-trace-store-v1"; - String traceAggregationStreamAppId = "zipkin-trace-aggregation-v1"; - String traceRetentionStreamAppId = "zipkin-trace-retention-v1"; - String serviceStoreStreamAppId = "zipkin-service-store-v1"; - String serviceAggregationStreamAppId = "zipkin-service-aggregation-v1"; - String dependencyStoreStreamAppId = "zipkin-dependency-store-v1"; - String dependencyAggregationStreamAppId = "zipkin-dependency-aggregation-v1"; - - String storeDirectory = "/tmp/zipkin"; - - String traceStoreName = "zipkin-trace-store-v1"; - String dependencyStoreName = "zipkin-dependency-v1"; - String serviceStoreName = "zipkin-service-v1"; - - Topic spansTopic = Topic.builder("zipkin-spans-v1").build(); - Topic spanServicesTopic = Topic.builder("zipkin-span-services-v1").build(); - Topic spanDependenciesTopic = Topic.builder("zipkin-span-dependencies-v1").build(); - Topic tracesTopic = Topic.builder("zipkin-traces-v1") - .config(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_COMPACT) - .build(); - Topic servicesTopic = Topic.builder("zipkin-services-v1") - .config(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_COMPACT) - .build(); - Topic dependenciesTopic = Topic.builder("zipkin-dependencies-v1") - .config(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_COMPACT) - .build(); - - boolean ensureTopics = true; + Properties adminConfig = new Properties(); + Properties producerConfig = new Properties(); + Properties aggregationStreamConfig = new Properties(); + Properties traceStoreStreamConfig = new Properties(); + Properties dependencyStoreStreamConfig = new Properties(); + + String traceStoreStreamAppId = "zipkin-trace-store"; + String dependencyStoreStreamAppId = "zipkin-dependency-store"; + String aggregationStreamAppId = "zipkin-aggregation"; + + String spansTopicName = "zipkin-spans"; + String traceTopicName = "zipkin-trace"; + String dependencyTopicName = "zipkin-dependency"; Builder() { + // Kafka Producer configuration + producerConfig.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); + producerConfig.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class); + producerConfig.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); + producerConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, CompressionType.SNAPPY.name); + producerConfig.put(ProducerConfig.BATCH_SIZE_CONFIG, 500_000); + producerConfig.put(ProducerConfig.LINGER_MS_CONFIG, 5); + // Trace Aggregation Stream Topology configuration + aggregationStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, + Serdes.StringSerde.class); + aggregationStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, + Serdes.ByteArraySerde.class); + aggregationStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, aggregationStreamAppId); + aggregationStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, traceStoreDirectory()); + aggregationStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); + aggregationStreamConfig.put( + StreamsConfig.PRODUCER_PREFIX + ProducerConfig.COMPRESSION_TYPE_CONFIG, + CompressionType.SNAPPY.name); + // Trace Store Stream Topology configuration + traceStoreStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, + Serdes.StringSerde.class); + traceStoreStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, + Serdes.ByteArraySerde.class); + traceStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, traceStoreStreamAppId); + traceStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, traceStoreDirectory()); + traceStoreStreamConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, + CompressionType.SNAPPY.name); + traceStoreStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); + // Dependency Store Stream Topology configuration + dependencyStoreStreamConfig.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, + Serdes.StringSerde.class); + dependencyStoreStreamConfig.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, + Serdes.ByteArraySerde.class); + dependencyStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, + dependencyStoreStreamAppId); + dependencyStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, dependencyStoreDirectory()); + dependencyStoreStreamConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, + CompressionType.SNAPPY.name); + dependencyStoreStreamConfig.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); } @Override - public StorageComponent.Builder strictTraceId(boolean strictTraceId) { - if (!strictTraceId) throw new IllegalArgumentException("unstrict trace ID not supported"); + public Builder strictTraceId(boolean strictTraceId) { + if (!strictTraceId) throw new IllegalArgumentException("non-strict trace ID not supported"); return this; } @Override - public StorageComponent.Builder searchEnabled(boolean searchEnabled) { - if (searchEnabled) throw new IllegalArgumentException("search not supported"); + public Builder searchEnabled(boolean searchEnabled) { + this.searchEnabled = searchEnabled; return this; } @Override public Builder autocompleteKeys(List keys) { if (keys == null) throw new NullPointerException("keys == null"); - if (!keys.isEmpty()) throw new IllegalArgumentException("autocomplete not supported"); + this.autocompleteKeys = keys; return this; } /** - * Enable consuming spans from collectors and store them in Kafka topics. - * + * Enable consuming spans from collectors, aggregation, and store them in Kafka topics. + *

* When disabled, a NoopSpanConsumer is instantiated to do nothing with incoming spans. */ public Builder spanConsumerEnabled(boolean spanConsumerEnabled) { @@ -582,18 +440,13 @@ public Builder spanConsumerEnabled(boolean spanConsumerEnabled) { } /** - * Enable storing spans to aggregate and index spans, traces, and dependencies. - * - * When disabled, a NoopSpanStore is instantiated to return empty lists for all searches. + * How long to wait for a span in order to trigger a trace as completed. */ - public Builder spanStoreEnabled(boolean spanStoreEnabled) { - this.spanConsumerEnabled = spanStoreEnabled; - return this; - } - - public Builder traceInactivityGap(Duration traceInactivityGap) { - if (traceInactivityGap == null) throw new NullPointerException("traceInactivityGap == null"); - this.traceInactivityGap = traceInactivityGap; + public Builder traceTimeout(Duration traceTimeout) { + if (traceTimeout == null) { + throw new NullPointerException("traceTimeout == null"); + } + this.traceTimeout = traceTimeout; return this; } @@ -602,57 +455,74 @@ public Builder traceInactivityGap(Duration traceInactivityGap) { */ public Builder bootstrapServers(String bootstrapServers) { if (bootstrapServers == null) throw new NullPointerException("bootstrapServers == null"); - this.bootstrapServers = bootstrapServers; + adminConfig.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + aggregationStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + traceStoreStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); + dependencyStoreStreamConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); return this; } - /** - * Kafka topic name where incoming spans are stored. - * - * A Span is received from Collectors that contains all metadata and is partitioned - * by Trace Id. - */ - public Builder spansTopic(Topic spansTopic) { - if (spansTopic == null) throw new NullPointerException("spansTopic == null"); - this.spansTopic = spansTopic; + public Builder aggregationStreamAppId(String aggregationStreamAppId) { + if (aggregationStreamAppId == null) { + throw new NullPointerException("aggregationStreamAppId == null"); + } + this.aggregationStreamAppId = aggregationStreamAppId; + aggregationStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, aggregationStreamAppId); return this; } - /** - * Kafka topic name where span services events are stored. - */ - public Builder spanServicesTopic(Topic spanServicesTopic) { - if (spanServicesTopic == null) throw new NullPointerException("spanServicesTopic == null"); - this.spanServicesTopic = spanServicesTopic; + public Builder traceStoreStreamAppId(String traceStoreStreamAppId) { + if (traceStoreStreamAppId == null) { + throw new NullPointerException("traceStoreStreamAppId == null"); + } + this.traceStoreStreamAppId = traceStoreStreamAppId; + traceStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, traceStoreStreamAppId); + return this; + } + + public Builder dependencyStoreStreamAppId(String dependencyStoreStreamAppId) { + if (dependencyStoreStreamAppId == null) { + throw new NullPointerException("dependencyStoreStreamAppId == null"); + } + this.dependencyStoreStreamAppId = dependencyStoreStreamAppId; + dependencyStoreStreamConfig.put(StreamsConfig.APPLICATION_ID_CONFIG, + dependencyStoreStreamAppId); return this; } /** - * Kafka topic name where services changelog are stored. + * Kafka topic name where incoming spans are stored. + *

+ * A Span is received from Collectors that contains all metadata and is partitioned by Trace + * Id. */ - public Builder servicesTopic(Topic servicesTopic) { - if (servicesTopic == null) throw new NullPointerException("servicesTopic == null"); - this.servicesTopic = servicesTopic; + public Builder spansTopicName(String spansTopicName) { + if (spansTopicName == null) throw new NullPointerException("spansTopicName == null"); + this.spansTopicName = spansTopicName; return this; } /** - * Kafka topic name where span dependencies events are stored. + * Kafka topic name where incoming spans are stored. + *

+ * A Span is received from Collectors that contains all metadata and is partitioned by Trace + * Id. */ - public Builder spanDependenciesTopic(Topic spanDependenciesTopic) { - if (spanDependenciesTopic == null) { - throw new NullPointerException("spanDependenciesTopic == null"); - } - this.spanDependenciesTopic = spanDependenciesTopic; + public Builder tracesTopicName(String tracesTopicName) { + if (tracesTopicName == null) throw new NullPointerException("tracesTopicName == null"); + this.traceTopicName = tracesTopicName; return this; } /** * Kafka topic name where dependencies changelog are stored. */ - public Builder dependenciesTopic(Topic dependenciesTopic) { - if (dependenciesTopic == null) throw new NullPointerException("dependenciesTopic == null"); - this.dependenciesTopic = dependenciesTopic; + public Builder dependenciesTopicName(String dependenciesTopicName) { + if (dependenciesTopicName == null) { + throw new NullPointerException("dependenciesTopicName == null"); + } + this.dependencyTopicName = dependenciesTopicName; return this; } @@ -661,124 +531,156 @@ public Builder dependenciesTopic(Topic dependenciesTopic) { */ public Builder storeDirectory(String storeDirectory) { if (storeDirectory == null) throw new NullPointerException("storageDirectory == null"); - this.storeDirectory = storeDirectory; + this.storeDir = storeDirectory; + traceStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, traceStoreDirectory()); + dependencyStoreStreamConfig.put(StreamsConfig.STATE_DIR_CONFIG, dependencyStoreDirectory()); return this; } /** * Frequency to check retention policy. */ - public Builder retentionScanFrequency(Duration retentionScanFrequency) { - this.retentionScanFrequency = retentionScanFrequency; + public Builder traceTtlCheckInterval(Duration traceTtlCheckInterval) { + if (traceTtlCheckInterval == null) { + throw new NullPointerException("traceTtlCheckInterval == null"); + } + this.traceTtlCheckInterval = traceTtlCheckInterval; return this; } /** - * Maximum age for traces and spans to be retained on State Stores. + * Traces time-to-live on local state stores. */ - public Builder retentionMaxAge(Duration retentionMaxAge) { - this.retentionMaxAge = retentionMaxAge; + public Builder traceTtl(Duration traceTtl) { + if (this.traceTtl == null) throw new NullPointerException("traceTtl == null"); + this.traceTtl = traceTtl; return this; } /** - * If enabled, will create Topics if they do not exist. + * Dependencies time-to-live on local state stores. */ - public Builder ensureTopics(boolean ensureTopics) { - this.ensureTopics = ensureTopics; - return this; - } - - public Builder compressionType(String compressionType) { - if (compressionType == null) throw new NullPointerException("compressionType == null"); - this.compressionType = CompressionType.valueOf(compressionType); + public Builder dependencyTtl(Duration dependencyTtl) { + if (dependencyTtl == null) throw new NullPointerException("dependencyTtl == null"); + this.dependencyTtl = dependencyTtl; return this; } String traceStoreDirectory() { - return storeDirectory + "/streams/traces"; - } - - String serviceStoreDirectory() { - return storeDirectory + "/streams/services"; + return storeDir + "/traces"; } String dependencyStoreDirectory() { - return storeDirectory + "/streams/dependencies"; + return storeDir + "/dependencies"; } - String spanIndexDirectory() { - return storeDirectory + "/index"; + /** + * By default, an Admin Client will be built from properties derived from builder defaults, as + * well as "client.id" -> "zipkin-storage". Any properties set here will override the admin + * client config. + * + *

For example: Set the client ID for the AdminClient. + * + * 

{@code
+     * Map overrides = new LinkedHashMap<>();
+     * overrides.put(AdminClientConfig.CLIENT_ID_CONFIG, "zipkin-storage");
+     * builder.overrides(overrides);
+     * }
+ * + * @see org.apache.kafka.clients.admin.AdminClientConfig + */ + public final Builder adminOverrides(Map overrides) { + if (overrides == null) throw new NullPointerException("overrides == null"); + adminConfig.putAll(overrides); + return this; } - @Override - public StorageComponent build() { - return new KafkaStorage(this); + /** + * By default, a produce will be built from properties derived from builder defaults, as well as + * "batch.size" -> 1000. Any properties set here will override the consumer config. + * + *

For example: Only send batch of list of spans with a maximum size of 1000 bytes + * + * 

{@code
+     * Map overrides = new LinkedHashMap<>();
+     * overrides.put(ProducerConfig.BATCH_SIZE_CONFIG, 1000);
+     * builder.overrides(overrides);
+     * }
+ * + * @see org.apache.kafka.clients.producer.ProducerConfig + */ + public final Builder producerOverrides(Map overrides) { + if (overrides == null) throw new NullPointerException("overrides == null"); + producerConfig.putAll(overrides); + return this; } - } - - public static class Topic { - final String name; - final Integer partitions; - final Short replicationFactor; - final Map configs; - Topic(Builder builder) { - this.name = builder.name; - this.partitions = builder.partitions; - this.replicationFactor = builder.replicationFactor; - this.configs = builder.configs; + /** + * By default, a Kafka Streams applications will be built from properties derived from builder + * defaults, as well as "poll.ms" -> 5000. Any properties set here will override the Kafka + * Streams application config. + * + *

For example: to change the Streams poll timeout: + * + * 

{@code
+     * Map overrides = new LinkedHashMap<>();
+     * overrides.put(StreamsConfig.POLL_MS, 5000);
+     * builder.aggregationStreamOverrides(overrides);
+     * }
+ * + * @see org.apache.kafka.streams.StreamsConfig + */ + public final Builder aggregationStreamOverrides(Map overrides) { + if (overrides == null) throw new NullPointerException("overrides == null"); + aggregationStreamConfig.putAll(overrides); + return this; } - NewTopic newTopic() { - NewTopic newTopic = new NewTopic(name, partitions, replicationFactor); - newTopic.configs(configs); - return newTopic; + /** + * By default, a Kafka Streams applications will be built from properties derived from builder + * defaults, as well as "poll.ms" -> 5000. Any properties set here will override the Kafka + * Streams application config. + * + *

For example: to change the Streams poll timeout: + * + * 

{@code
+     * Map overrides = new LinkedHashMap<>();
+     * overrides.put(StreamsConfig.POLL_MS, 5000);
+     * builder.traceStoreStreamOverrides(overrides);
+     * }
+ * + * @see org.apache.kafka.streams.StreamsConfig + */ + public final Builder traceStoreStreamOverrides(Map overrides) { + if (overrides == null) throw new NullPointerException("overrides == null"); + traceStoreStreamConfig.putAll(overrides); + return this; } - public static Builder builder(String name) { - return new Builder(name); + /** + * By default, a Kafka Streams applications will be built from properties derived from builder + * defaults, as well as "poll.ms" -> 5000. Any properties set here will override the Kafka + * Streams application config. + * + *

For example: to change the Streams poll timeout: + * + * 

{@code
+     * Map overrides = new LinkedHashMap<>();
+     * overrides.put(StreamsConfig.POLL_MS, 5000);
+     * builder.dependencyStoreStreamOverrides(overrides);
+     * }
+ * + * @see org.apache.kafka.streams.StreamsConfig + */ + public final Builder dependencyStoreStreamOverrides(Map overrides) { + if (overrides == null) throw new NullPointerException("overrides == null"); + dependencyStoreStreamConfig.putAll(overrides); + return this; } - public static class Builder { - final String name; - Integer partitions = 1; - Short replicationFactor = 1; - Map configs = new HashMap<>(); - - Builder(String name) { - if (name == null) throw new NullPointerException("topic name == null"); - this.name = name; - } - - public Builder partitions(Integer partitions) { - if (partitions == null) throw new NullPointerException("topic partitions == null"); - if (partitions < 1) throw new IllegalArgumentException("topic partitions < 1"); - this.partitions = partitions; - return this; - } - - public Builder replicationFactor(Short replicationFactor) { - if (replicationFactor == null) { - throw new NullPointerException("topic replicationFactor == null"); - } - if (replicationFactor < 1) { - throw new IllegalArgumentException("topic replicationFactor < 1"); - } - this.replicationFactor = replicationFactor; - return this; - } - - Builder config(String key, String value) { - if (key == null) throw new NullPointerException("topic config key == null"); - if (value == null) throw new NullPointerException("topic config value == null"); - this.configs.put(key, value); - return this; - } - - public Topic build() { - return new Topic(this); - } + @Override + public StorageComponent build() { + return new KafkaStorage(this); } } } diff --git a/storage/src/main/java/zipkin2/storage/kafka/index/SpanIndexService.java b/storage/src/main/java/zipkin2/storage/kafka/index/SpanIndexService.java deleted file mode 100644 index 699c029c..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/index/SpanIndexService.java +++ /dev/null @@ -1,248 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.index; - -import java.io.IOException; -import java.nio.file.Paths; -import java.util.HashSet; -import java.util.Map; -import java.util.Set; -import org.apache.lucene.analysis.standard.StandardAnalyzer; -import org.apache.lucene.document.Document; -import org.apache.lucene.document.Field; -import org.apache.lucene.document.LongPoint; -import org.apache.lucene.document.NumericDocValuesField; -import org.apache.lucene.document.SortedDocValuesField; -import org.apache.lucene.document.StringField; -import org.apache.lucene.document.TextField; -import org.apache.lucene.index.DirectoryReader; -import org.apache.lucene.index.IndexReader; -import org.apache.lucene.index.IndexWriter; -import org.apache.lucene.index.IndexWriterConfig; -import org.apache.lucene.index.Term; -import org.apache.lucene.queryparser.classic.ParseException; -import org.apache.lucene.queryparser.classic.QueryParser; -import org.apache.lucene.search.BooleanClause; -import org.apache.lucene.search.BooleanQuery; -import org.apache.lucene.search.IndexSearcher; -import org.apache.lucene.search.Query; -import org.apache.lucene.search.ScoreDoc; -import org.apache.lucene.search.Sort; -import org.apache.lucene.search.SortField; -import org.apache.lucene.search.TermQuery; -import org.apache.lucene.search.grouping.GroupDocs; -import org.apache.lucene.search.grouping.GroupingSearch; -import org.apache.lucene.search.grouping.TopGroups; -import org.apache.lucene.store.Directory; -import org.apache.lucene.store.MMapDirectory; -import org.apache.lucene.util.BytesRef; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import zipkin2.Annotation; -import zipkin2.Span; -import zipkin2.storage.QueryRequest; - -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.ANNOTATION; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.DURATION; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.ID; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.LOCAL_SERVICE_NAME; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.NAME; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.SORTED_TIMESTAMP; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.SORTED_TRACE_ID; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.TIMESTAMP; -import static zipkin2.storage.kafka.index.SpanIndexService.SpanFields.TRACE_ID; - -public class SpanIndexService { - static final Logger LOG = LoggerFactory.getLogger(SpanIndexService.class); - - final Directory directory; - - volatile IndexWriter indexWriter; - - SpanIndexService(Builder builder) throws IOException { - LOG.info("Storing index on path={}", builder.indexDirectory); - directory = new MMapDirectory(Paths.get(builder.indexDirectory)); - getIndexWriter(); - } - - public static SpanIndexService create(String indexDirectory) throws IOException { - return new Builder().indexDirectory(indexDirectory).build(); - } - - IndexWriter getIndexWriter() { - if (indexWriter == null) { - synchronized (this) { - if (indexWriter == null) { - try { - StandardAnalyzer analyzer = new StandardAnalyzer(); - IndexWriterConfig indexWriterConfigs = new IndexWriterConfig(analyzer); - indexWriter = new IndexWriter(directory, indexWriterConfigs); - indexWriter.commit(); - } catch (Exception e) { - LOG.error("Error opening index writer", e); - } - } - } - } - return indexWriter; - } - - public void deleteByTraceId(String traceId) { - try { - TermQuery query = new TermQuery(new Term(TRACE_ID, traceId)); - IndexWriter indexWriter = getIndexWriter(); - indexWriter.deleteDocuments(query); - indexWriter.commit(); - } catch (IOException e) { - e.printStackTrace(); - } - } - - public Set getTraceIds(QueryRequest queryRequest) { - // Parsing query - Query query = parseQuery(queryRequest); - GroupingSearch groupingSearch = parseGrouping(); - try (IndexReader reader = DirectoryReader.open(directory)) { - IndexSearcher indexSearcher = new IndexSearcher(reader); - TopGroups search = - groupingSearch.search(indexSearcher, query, 0, queryRequest.limit()); - // Collecting trace ids - Set traceIds = new HashSet<>(); - for (GroupDocs groupDocs : search.groups) { - for (ScoreDoc scoreDoc : groupDocs.scoreDocs) { - Document document = indexSearcher.doc(scoreDoc.doc); - String traceId = document.get(TRACE_ID); - traceIds.add(traceId); - } - } - return traceIds; - } catch (IOException e) { - LOG.error("Error in group query", e); - return new HashSet<>(); - } - } - - public void insert(Span span) { - try { - Document doc = new Document(); - doc.add( - new SortedDocValuesField(SORTED_TRACE_ID, new BytesRef(span.traceId()))); - doc.add(new NumericDocValuesField(SORTED_TIMESTAMP, span.timestampAsLong())); - - doc.add(new StringField(TRACE_ID, span.traceId(), Field.Store.YES)); - doc.add(new StringField(ID, span.id(), Field.Store.YES)); - - String localServiceName = - span.localServiceName() != null ? span.localServiceName() : ""; - doc.add( - new StringField(LOCAL_SERVICE_NAME, localServiceName, Field.Store.YES)); - - String name = span.name() != null ? span.name() : ""; - doc.add(new StringField(NAME, name, Field.Store.YES)); - - doc.add(new LongPoint(TIMESTAMP, span.timestampAsLong())); - doc.add(new LongPoint(DURATION, span.durationAsLong())); - - for (Map.Entry tag : span.tags().entrySet()) { - doc.add(new TextField(ANNOTATION, tag.getKey() + "=" + tag.getValue(), - Field.Store.YES)); - } - - for (Annotation annotation : span.annotations()) { - doc.add(new TextField(ANNOTATION, annotation.value(), Field.Store.YES)); - } - - IndexWriter indexWriter = getIndexWriter(); - indexWriter.addDocument(doc); - indexWriter.commit(); - } catch (Exception e) { - LOG.error("Error indexing span {}", span, e); - } - } - - GroupingSearch parseGrouping() { - GroupingSearch groupingSearch = new GroupingSearch(SORTED_TRACE_ID); - Sort sort = new Sort(new SortField(SORTED_TIMESTAMP, SortField.Type.LONG, true)); - groupingSearch.setGroupDocsLimit(1); - groupingSearch.setGroupSort(sort); - return groupingSearch; - } - - Query parseQuery(QueryRequest queryRequest) { - BooleanQuery.Builder builder = new BooleanQuery.Builder(); - - if (queryRequest.serviceName() != null) { - String serviceName = queryRequest.serviceName(); - TermQuery serviceNameQuery = new TermQuery(new Term(LOCAL_SERVICE_NAME, serviceName)); - builder.add(serviceNameQuery, BooleanClause.Occur.MUST); - } - - if (queryRequest.spanName() != null) { - String spanName = queryRequest.spanName(); - TermQuery spanNameQuery = new TermQuery(new Term(NAME, spanName)); - builder.add(spanNameQuery, BooleanClause.Occur.MUST); - } - - if (queryRequest.annotationQueryString() != null) { - try { - QueryParser queryParser = new QueryParser(ANNOTATION, new StandardAnalyzer()); - Query annotationQuery = queryParser.parse(queryRequest.annotationQueryString()); - builder.add(annotationQuery, BooleanClause.Occur.MUST); - } catch (ParseException e) { - e.printStackTrace(); - } - } - - if (queryRequest.maxDuration() != null) { - Query durationRangeQuery = LongPoint.newRangeQuery( - DURATION, queryRequest.minDuration(), queryRequest.maxDuration()); - builder.add(durationRangeQuery, BooleanClause.Occur.MUST); - } - - long start = queryRequest.endTs() - queryRequest.lookback(); - long end = queryRequest.endTs(); - long lowerValue = start * 1000; - long upperValue = end * 1000; - Query tsRangeQuery = LongPoint.newRangeQuery(TIMESTAMP, lowerValue, upperValue); - builder.add(tsRangeQuery, BooleanClause.Occur.MUST); - - return builder.build(); - } - - static class Builder { - String indexDirectory; - - SpanIndexService build() throws IOException { - return new SpanIndexService(this); - } - - public Builder indexDirectory(String indexDirectory) { - if (indexDirectory == null) throw new NullPointerException("indexDirectory == null"); - this.indexDirectory = indexDirectory; - return this; - } - } - - static class SpanFields { - static final String TRACE_ID = "trace_id"; - static final String SORTED_TRACE_ID = "trace_id_sorted"; - static final String ID = "id"; - static final String LOCAL_SERVICE_NAME = "local_service_name"; - static final String NAME = "name"; - static final String ANNOTATION = "annotation"; - static final String TIMESTAMP = "ts"; - static final String SORTED_TIMESTAMP = "ts_sorted"; - static final String DURATION = "duration"; - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/internal/KafkaStreamsStoreCall.java b/storage/src/main/java/zipkin2/storage/kafka/internal/KafkaStreamsStoreCall.java new file mode 100644 index 00000000..042975a5 --- /dev/null +++ b/storage/src/main/java/zipkin2/storage/kafka/internal/KafkaStreamsStoreCall.java @@ -0,0 +1,42 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.internal; + +import java.io.IOException; +import zipkin2.Call; +import zipkin2.Callback; + +public abstract class KafkaStreamsStoreCall extends Call.Base { + + protected KafkaStreamsStoreCall() { + } + + @Override protected T doExecute() throws IOException { + try { + return query(); + } catch (Exception e) { + throw new IOException(e); + } + } + + @Override protected void doEnqueue(Callback callback) { + try { // TODO check how to make queries async + callback.onSuccess(query()); + } catch (Exception e) { + callback.onError(e); + } + } + + protected abstract T query(); +} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/AggregationTopologySupplier.java b/storage/src/main/java/zipkin2/storage/kafka/streams/AggregationTopologySupplier.java new file mode 100644 index 00000000..c72f429c --- /dev/null +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/AggregationTopologySupplier.java @@ -0,0 +1,113 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.streams; + +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.function.Supplier; +import org.apache.kafka.common.serialization.Serdes; +import org.apache.kafka.streams.StreamsBuilder; +import org.apache.kafka.streams.Topology; +import org.apache.kafka.streams.kstream.Aggregator; +import org.apache.kafka.streams.kstream.Consumed; +import org.apache.kafka.streams.kstream.KStream; +import org.apache.kafka.streams.kstream.Materialized; +import org.apache.kafka.streams.kstream.Merger; +import org.apache.kafka.streams.kstream.Produced; +import org.apache.kafka.streams.kstream.SessionWindows; +import org.apache.kafka.streams.kstream.ValueMapper; +import zipkin2.DependencyLink; +import zipkin2.Span; +import zipkin2.internal.DependencyLinker; +import zipkin2.internal.Trace; +import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; +import zipkin2.storage.kafka.streams.serdes.SpansSerde; + +import static org.apache.kafka.streams.kstream.Suppressed.BufferConfig.unbounded; +import static org.apache.kafka.streams.kstream.Suppressed.untilWindowCloses; +import static zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde.linkKey; + +/** + * Processing of spans partitioned by trace Id, into traces and dependency links. + */ +public class AggregationTopologySupplier implements Supplier { + // Kafka topics + final String spansTopicName; + final String traceTopicName; + final String dependencyTopicName; + // Config + final Duration traceTimeout; + // SerDes + final SpansSerde spansSerde; + final DependencyLinkSerde dependencyLinkSerde; + + public AggregationTopologySupplier( + String spansTopicName, + String traceTopicName, + String dependencyTopicName, + Duration traceTimeout) { + this.spansTopicName = spansTopicName; + this.traceTopicName = traceTopicName; + this.dependencyTopicName = dependencyTopicName; + this.traceTimeout = traceTimeout; + spansSerde = new SpansSerde(); + dependencyLinkSerde = new DependencyLinkSerde(); + } + + @Override public Topology get() { + StreamsBuilder builder = new StreamsBuilder(); + // Aggregate Spans to Traces + KStream> tracesStream = + builder.stream(spansTopicName, Consumed.with(Serdes.String(), spansSerde)) + .groupByKey() + // how long to wait for another span + .windowedBy(SessionWindows.with(traceTimeout).grace(Duration.ZERO)) + .aggregate(ArrayList::new, aggregateSpans(), joinAggregates(), + Materialized.with(Serdes.String(), spansSerde)) + // hold until a new record tells that a window is closed and we can process it further + .suppress(untilWindowCloses(unbounded())) + .toStream() + .selectKey((windowed, spans) -> windowed.key()); + // Downstream to traces topic + tracesStream.to(traceTopicName, Produced.with(Serdes.String(), spansSerde)); + // Map to dependency links + tracesStream.flatMapValues(spansToDependencyLinks()) + .selectKey((key, value) -> linkKey(value)) + .to(dependencyTopicName, Produced.with(Serdes.String(), dependencyLinkSerde)); + return builder.build(); + } + + Merger> joinAggregates() { + return (aggKey, aggOne, aggTwo) -> { + aggOne.addAll(aggTwo); + return Trace.merge(aggOne); + }; + } + + Aggregator, List> aggregateSpans() { + return (traceId, spans, allSpans) -> { + allSpans.addAll(spans); + return Trace.merge(allSpans); + }; + } + + ValueMapper, List> spansToDependencyLinks() { + return (spans) -> { + if (spans == null) return new ArrayList<>(); + DependencyLinker linker = new DependencyLinker(); + return linker.putTrace(spans).link(); + }; + } +} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyAggregationStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyAggregationStream.java deleted file mode 100644 index b499c921..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyAggregationStream.java +++ /dev/null @@ -1,113 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.util.ArrayList; -import java.util.List; -import java.util.function.Supplier; -import java.util.stream.Collectors; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.common.utils.Bytes; -import org.apache.kafka.streams.KeyValue; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.kstream.KeyValueMapper; -import org.apache.kafka.streams.kstream.Materialized; -import org.apache.kafka.streams.kstream.Produced; -import org.apache.kafka.streams.kstream.Reducer; -import org.apache.kafka.streams.state.KeyValueStore; -import zipkin2.DependencyLink; -import zipkin2.Span; -import zipkin2.internal.DependencyLinker; -import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; -import zipkin2.storage.kafka.streams.serdes.SpanSerde; -import zipkin2.storage.kafka.streams.serdes.SpansSerde; - -/** - * Reduction of span dependency events, with call/error counter equals to 0 or 1, into ever - * increasing dependency link with updated counters. - */ -public class DependencyAggregationStream implements Supplier { - static final String KEY_PATTERN = "%s:%s"; - // Kafka topics - final String tracesTopicName; - final String spanDependenciesTopicName; - final String dependenciesTopicName; - // SerDes - final SpanSerde spanSerde; - final SpansSerde spansSerde; - final DependencyLinkSerde dependencyLinkSerde; - - public DependencyAggregationStream( - String tracesTopicName, - String spanDependenciesTopicName, - String dependenciesTopicName) { - this.tracesTopicName = tracesTopicName; - this.spanDependenciesTopicName = spanDependenciesTopicName; - this.dependenciesTopicName = dependenciesTopicName; - spanSerde = new SpanSerde(); - spansSerde = new SpansSerde(); - dependencyLinkSerde = new DependencyLinkSerde(); - } - - @Override public Topology get() { - StreamsBuilder builder = new StreamsBuilder(); - // Changelog of dependency links over time - builder.stream(tracesTopicName, Consumed.with(Serdes.String(), spansSerde)) - .flatMap(spansToDependencyLinks()) - .through(spanDependenciesTopicName, Produced.with(Serdes.String(), dependencyLinkSerde)) - .groupByKey() - .reduce(reduceDependencyLinks(), - Materialized.>with( - Serdes.String(), - dependencyLinkSerde).withLoggingDisabled().withCachingEnabled()) - .toStream() - .selectKey((key, value) -> key) - .to(dependenciesTopicName, Produced.with(Serdes.String(), dependencyLinkSerde)); - return builder.build(); - } - - KeyValueMapper, List>> spansToDependencyLinks() { - return (windowed, spans) -> { - if (spans == null) return new ArrayList<>(); - DependencyLinker linker = new DependencyLinker(); - return linker.putTrace(spans).link().stream() - .map(link -> KeyValue.pair(key(link), link)) - .collect(Collectors.toList()); - }; - } - - /** - * Reducing link events into links with updated results - */ - Reducer reduceDependencyLinks() { - return (link1, link2) -> { - if (link2 == null) { - return link1; - } else { - return DependencyLink.newBuilder() - .parent(link1.parent()) - .child(link1.child()) - .callCount(link1.callCount() + link2.callCount()) - .errorCount(link1.errorCount() + link2.errorCount()) - .build(); - } - }; - } - - String key(DependencyLink link) { - return String.format(KEY_PATTERN, link.parent(), link.child()); - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreStream.java deleted file mode 100644 index 27e5c03a..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreStream.java +++ /dev/null @@ -1,93 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.util.function.Supplier; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.processor.Processor; -import org.apache.kafka.streams.processor.ProcessorContext; -import org.apache.kafka.streams.state.KeyValueStore; -import org.apache.kafka.streams.state.StoreBuilder; -import org.apache.kafka.streams.state.Stores; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import zipkin2.DependencyLink; -import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; - -/** - * Stream topology supplier for Dependency aggregation. - * - * Source: Traces topic (aggregated Traces aggregation) - * Store: Dependencies store (global state store) - */ -public class DependencyStoreStream implements Supplier { - static final Logger LOG = LoggerFactory.getLogger(DependencyStoreStream.class); - // Kafka Topics - final String dependenciesTopic; - // Store names - final String globalDependenciesStoreName; - // SerDes - final DependencyLinkSerde dependencyLinkSerde; - - public DependencyStoreStream(String dependenciesTopic, String globalDependenciesStoreName) { - this.dependenciesTopic = dependenciesTopic; - this.globalDependenciesStoreName = globalDependenciesStoreName; - - dependencyLinkSerde = new DependencyLinkSerde(); - } - - @Override public Topology get() { - // Preparing state stores - StoreBuilder> globalDependenciesStoreBuilder = - Stores.keyValueStoreBuilder( - Stores.persistentKeyValueStore(globalDependenciesStoreName), - Serdes.Long(), - dependencyLinkSerde) - .withCachingEnabled() - .withLoggingDisabled(); - - StreamsBuilder builder = new StreamsBuilder(); - // Store Dependencies changelog by time - builder.addGlobalStore( - globalDependenciesStoreBuilder, - dependenciesTopic, - Consumed.with(Serdes.String(), dependencyLinkSerde), - () -> new Processor() { - KeyValueStore dependenciesStore; - - @Override public void init(ProcessorContext context) { - LOG.info("Initializing Dependency Store Stream"); - dependenciesStore = - (KeyValueStore) context.getStateStore( - globalDependenciesStoreName); - } - - @Override - public void process(String linkKey, DependencyLink dependencyLink) { - Long millis = System.currentTimeMillis(); - LOG.debug("Storing dependency: {} at {}", dependencyLink, millis); - dependenciesStore.put(millis, dependencyLink); - } - - @Override public void close() { // Nothing to close - } - } - ); - - return builder.build(); - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplier.java b/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplier.java new file mode 100644 index 00000000..d3b2a01b --- /dev/null +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplier.java @@ -0,0 +1,113 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.streams; + +import java.time.Duration; +import java.time.Instant; +import java.util.function.Supplier; +import org.apache.kafka.common.serialization.Serdes; +import org.apache.kafka.streams.KeyValue; +import org.apache.kafka.streams.StreamsBuilder; +import org.apache.kafka.streams.Topology; +import org.apache.kafka.streams.kstream.Consumed; +import org.apache.kafka.streams.processor.Processor; +import org.apache.kafka.streams.processor.ProcessorContext; +import org.apache.kafka.streams.state.Stores; +import org.apache.kafka.streams.state.WindowStore; +import org.apache.kafka.streams.state.WindowStoreIterator; +import zipkin2.DependencyLink; +import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; + +/** + * Windowed storage of dependency links. + */ +public class DependencyStoreTopologySupplier implements Supplier { + public static final String DEPENDENCIES_STORE_NAME = "zipkin-dependencies"; + + // Kafka topics + final String dependencyTopicName; + // Configs + final Duration dependencyTtl; + final Duration dependencyWindowSize; + // SerDes + final DependencyLinkSerde dependencyLinkSerde; + + public DependencyStoreTopologySupplier(String dependencyTopicName, + Duration dependencyTtl, Duration dependencyWindowSize) { + this.dependencyTopicName = dependencyTopicName; + this.dependencyTtl = dependencyTtl; + this.dependencyWindowSize = dependencyWindowSize; + dependencyLinkSerde = new DependencyLinkSerde(); + } + + @Override public Topology get() { + StreamsBuilder builder = new StreamsBuilder(); + + // Dependency links window store + builder.addStateStore(Stores.windowStoreBuilder( + Stores.persistentWindowStore( + DEPENDENCIES_STORE_NAME, + dependencyTtl, + dependencyWindowSize, + false), + Serdes.String(), + dependencyLinkSerde + )); + // Consume dependency links stream + builder.stream(dependencyTopicName, Consumed.with(Serdes.String(), dependencyLinkSerde)) + // Storage + .process(() -> new Processor() { + ProcessorContext context; + WindowStore dependenciesStore; + + @Override + public void init(ProcessorContext context) { + this.context = context; + dependenciesStore = + (WindowStore) context.getStateStore( + DEPENDENCIES_STORE_NAME); + } + + @Override + public void process(String linkKey, DependencyLink link) { + // Event time + Instant now = Instant.ofEpochMilli(context.timestamp()); + Instant from = now.minus(dependencyWindowSize); + WindowStoreIterator currentLinkWindow = + dependenciesStore.fetch(linkKey, from, now); + // Get latest window. Only two are possible. + KeyValue windowAndValue = null; + if (currentLinkWindow.hasNext()) windowAndValue = currentLinkWindow.next(); + if (currentLinkWindow.hasNext()) windowAndValue = currentLinkWindow.next(); + // Persist dependency link per window + if (windowAndValue != null) { + DependencyLink currentLink = windowAndValue.value; + DependencyLink aggregated = currentLink.toBuilder() + .callCount(currentLink.callCount() + link.callCount()) + .errorCount(currentLink.errorCount() + link.errorCount()) + .build(); + dependenciesStore.put(linkKey, aggregated, windowAndValue.key); + } else { + dependenciesStore.put(linkKey, link); + } + } + + @Override + public void close() { + } + }, DEPENDENCIES_STORE_NAME); + + return builder.build(); + } +} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/ServiceAggregationStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/ServiceAggregationStream.java deleted file mode 100644 index 7bab0202..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/ServiceAggregationStream.java +++ /dev/null @@ -1,73 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.util.HashSet; -import java.util.Set; -import java.util.function.Supplier; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.common.utils.Bytes; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Aggregator; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.kstream.Materialized; -import org.apache.kafka.streams.kstream.Produced; -import org.apache.kafka.streams.state.KeyValueStore; -import zipkin2.storage.kafka.streams.serdes.SpanNamesSerde; - -/** - * Aggregation of span names per service into set of span names per service. - */ -public class ServiceAggregationStream implements Supplier { - // Kafka topics - final String spanServiceTopicName; - final String servicesTopicName; - // SerDes - final SpanNamesSerde spanNamesSerde; - - public ServiceAggregationStream( - String spanServiceTopicName, - String servicesTopicName) { - this.spanServiceTopicName = spanServiceTopicName; - this.servicesTopicName = servicesTopicName; - spanNamesSerde = new SpanNamesSerde(); - } - - @Override public Topology get() { - StreamsBuilder builder = new StreamsBuilder(); - // Aggregate ServiceName:SpanName into ServiceName:Set[SpanName] - builder - .stream(spanServiceTopicName, Consumed.with(Serdes.String(), Serdes.String())) - .groupByKey() - .aggregate(HashSet::new, - aggregateSpanNames(), - Materialized - ., KeyValueStore>with(Serdes.String(), - spanNamesSerde) - .withCachingEnabled() - .withLoggingDisabled()) - .toStream() - .to(servicesTopicName, Produced.with(Serdes.String(), spanNamesSerde)); - return builder.build(); - } - - // Collecting span names into a set of names. - Aggregator> aggregateSpanNames() { - return (serviceName, spanName, spanNames) -> { - spanNames.add(spanName); - return spanNames; - }; - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/ServiceStoreStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/ServiceStoreStream.java deleted file mode 100644 index e404592b..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/ServiceStoreStream.java +++ /dev/null @@ -1,85 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.util.Set; -import java.util.function.Supplier; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.processor.Processor; -import org.apache.kafka.streams.processor.ProcessorContext; -import org.apache.kafka.streams.state.KeyValueStore; -import org.apache.kafka.streams.state.StoreBuilder; -import org.apache.kafka.streams.state.Stores; -import zipkin2.storage.kafka.streams.serdes.SpanNamesSerde; - -public class ServiceStoreStream implements Supplier { - - // Topic names - final String servicesTopicName; - - // Store names - final String globalServicesStoreName; - - // SerDes - final SpanNamesSerde spanNamesSerde; - - public ServiceStoreStream( - String servicesTopicName, - String globalServicesStoreName) { - this.servicesTopicName = servicesTopicName; - this.globalServicesStoreName = globalServicesStoreName; - - spanNamesSerde = new SpanNamesSerde(); - } - - @Override public Topology get() { - // Preparing state stores - StoreBuilder>> globalServiceStoreBuilder = - Stores.keyValueStoreBuilder( - Stores.persistentKeyValueStore(globalServicesStoreName), - Serdes.String(), - spanNamesSerde) - .withCachingEnabled() - .withLoggingDisabled(); - - StreamsBuilder builder = new StreamsBuilder(); - - // Aggregate Service:SpanNames - builder - .addGlobalStore( - globalServiceStoreBuilder, - servicesTopicName, - Consumed.with(Serdes.String(), spanNamesSerde), - () -> new Processor>() { - KeyValueStore> servicesStore; - - @Override public void init(ProcessorContext context) { - servicesStore = (KeyValueStore>) context.getStateStore( - globalServicesStoreName); - } - - @Override public void process(String serviceName, Set spanNames) { - servicesStore.put(serviceName, spanNames); - } - - @Override public void close() { // Nothing to close - } - }); - - return builder.build(); - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceAggregationStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/TraceAggregationStream.java deleted file mode 100644 index 4d088a10..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceAggregationStream.java +++ /dev/null @@ -1,95 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.time.Duration; -import java.util.ArrayList; -import java.util.List; -import java.util.Objects; -import java.util.function.Supplier; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.common.utils.Bytes; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Aggregator; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.kstream.Materialized; -import org.apache.kafka.streams.kstream.Merger; -import org.apache.kafka.streams.kstream.Produced; -import org.apache.kafka.streams.kstream.SessionWindows; -import org.apache.kafka.streams.state.SessionStore; -import zipkin2.Span; -import zipkin2.storage.kafka.streams.serdes.SpanSerde; -import zipkin2.storage.kafka.streams.serdes.SpansSerde; - -import static org.apache.kafka.streams.kstream.Suppressed.BufferConfig.unbounded; -import static org.apache.kafka.streams.kstream.Suppressed.untilWindowCloses; - -/** - * - */ -public class TraceAggregationStream implements Supplier { - // Kafka topics - final String spansTopicName; - final String tracesTopicName; - // SerDes - final SpanSerde spanSerde; - final SpansSerde spansSerde; - // Config - final Duration traceInactivityGap; - - public TraceAggregationStream( - String spansTopicName, - String tracesTopicName, - Duration traceInactivityGap) { - this.spansTopicName = spansTopicName; - this.tracesTopicName = tracesTopicName; - this.traceInactivityGap = traceInactivityGap; - spanSerde = new SpanSerde(); - spansSerde = new SpansSerde(); - } - - @Override public Topology get() { - StreamsBuilder builder = new StreamsBuilder(); - // Aggregate Spans to Traces - builder.stream(spansTopicName, Consumed.with(Serdes.String(), spanSerde)) - .filter((key, value) -> Objects.nonNull(value)) - .groupByKey() - .windowedBy(SessionWindows.with(traceInactivityGap).grace(traceInactivityGap)) - .aggregate(ArrayList::new, aggregateSpans(), joinAggregates(), - Materialized. - , SessionStore>with(Serdes.String(), spansSerde) - .withCachingDisabled() - .withLoggingDisabled()) - .suppress(untilWindowCloses(unbounded()).withName("traces-suppressed")) - .toStream() - .selectKey((windowed, spans) -> windowed.key()) - .to(tracesTopicName, Produced.with(Serdes.String(), spansSerde)); - return builder.build(); - } - - Merger> joinAggregates() { - return (aggKey, aggOne, aggTwo) -> { - aggOne.addAll(aggTwo); - return aggOne; - }; - } - - Aggregator> aggregateSpans() { - return (traceId, span, spans) -> { - spans.add(span); - return spans; - }; - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceRetentionStoreStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/TraceRetentionStoreStream.java deleted file mode 100644 index c3ebeaeb..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceRetentionStoreStream.java +++ /dev/null @@ -1,125 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.time.Duration; -import java.time.Instant; -import java.util.function.Supplier; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.streams.KeyValue; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.kstream.Transformer; -import org.apache.kafka.streams.processor.ProcessorContext; -import org.apache.kafka.streams.processor.PunctuationType; -import org.apache.kafka.streams.state.KeyValueIterator; -import org.apache.kafka.streams.state.KeyValueStore; -import org.apache.kafka.streams.state.Stores; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import zipkin2.Span; -import zipkin2.storage.kafka.streams.serdes.SpanSerde; - -/** - * Retention topology to validate every defined period of time (e.g. 1 day) old spans and mark them - * for deletion. - * - * Deletion is handled in the other streams. - */ -public class TraceRetentionStoreStream implements Supplier { - static final Logger LOG = LoggerFactory.getLogger(TraceRetentionStoreStream.class); - // Kafka topics - final String spansTopic; - // Store names - final String traceTsStoreName; - // Retention attributes - final Duration scanFrequency; - final Duration maxAge; - // SerDe - final SpanSerde spanSerde; - - public TraceRetentionStoreStream( - String spansTopic, - String traceTsStoreName, - Duration scanFrequency, - Duration maxAge) { - this.spansTopic = spansTopic; - this.traceTsStoreName = traceTsStoreName; - this.scanFrequency = scanFrequency; - this.maxAge = maxAge; - - spanSerde = new SpanSerde(); - } - - @Override public Topology get() { - StreamsBuilder builder = new StreamsBuilder(); - builder - .addStateStore(Stores.keyValueStoreBuilder( - Stores.persistentKeyValueStore(traceTsStoreName), - Serdes.String(), - Serdes.Long()) - .withCachingEnabled() - .withLoggingDisabled()) - .stream(spansTopic, Consumed.with(Serdes.String(), spanSerde)) - .transform( - () -> new Transformer>() { - KeyValueStore stateStore; - - @Override public void init(ProcessorContext context) { - stateStore = (KeyValueStore) context.getStateStore(traceTsStoreName); - // Schedule deletion of traces older than maxAge - context.schedule( - scanFrequency, - PunctuationType.WALL_CLOCK_TIME, // Run it independently of insertion - timestamp -> { - final long cutoff = timestamp - maxAge.toMillis(); - final long ttl = cutoff * 1000; - - // Scan all records indexed - try (final KeyValueIterator all = stateStore.all()) { - int deletions = 0; - while (all.hasNext()) { - final KeyValue record = all.next(); - if (record.value != null && record.value < ttl) { - deletions++; - // if a record's last update was older than our cutoff, emit a tombstone. - context.forward(record.key, null); - } - } - LOG.info("Traces deletion emitted: {}, older than {}", - deletions, Instant.ofEpochMilli(cutoff)); - } - }); - } - - @Override - public KeyValue transform(String key, Span value) { - if (value == null) { // clean state when tombstone - stateStore.delete(key); - } else { // update store when traces are available - Long timestamp = value.timestamp(); - stateStore.put(key, timestamp); - } - return null; // no need to return anything here. the punctuator will emit the tombstones when necessary - } - - @Override public void close() { - // no need to close anything; Streams already closes the state store. - } - }, traceTsStoreName) - .to(spansTopic); - return builder.build(); - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreStream.java b/storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreStream.java deleted file mode 100644 index e831a712..00000000 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreStream.java +++ /dev/null @@ -1,109 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import java.util.Collections; -import java.util.List; -import java.util.function.Supplier; -import org.apache.kafka.common.serialization.Serdes; -import org.apache.kafka.streams.StreamsBuilder; -import org.apache.kafka.streams.Topology; -import org.apache.kafka.streams.kstream.Consumed; -import org.apache.kafka.streams.processor.Processor; -import org.apache.kafka.streams.processor.ProcessorContext; -import org.apache.kafka.streams.state.KeyValueStore; -import org.apache.kafka.streams.state.StoreBuilder; -import org.apache.kafka.streams.state.Stores; -import zipkin2.Span; -import zipkin2.storage.kafka.index.SpanIndexService; -import zipkin2.storage.kafka.streams.serdes.SpanSerde; -import zipkin2.storage.kafka.streams.serdes.SpansSerde; - -/** - * Aggregation and storage of spans into traces. - */ -public class TraceStoreStream implements Supplier { - // Kafka topics - final String spansTopic; - // Store names - final String tracesStoreName; - // SerDes - final SpanSerde spanSerde; - final SpansSerde spansSerde; - // Index Service - final SpanIndexService spanIndexService; - - public TraceStoreStream( - String spansTopic, - String tracesStoreName, - SpanIndexService spanIndexService) { - this.spansTopic = spansTopic; - this.tracesStoreName = tracesStoreName; - this.spanIndexService = spanIndexService; - spanSerde = new SpanSerde(); - spansSerde = new SpansSerde(); - } - - @Override public Topology get() { - // Preparing state stores - StoreBuilder>> globalTracesStoreBuilder = - Stores.keyValueStoreBuilder( - Stores.persistentKeyValueStore(tracesStoreName), - Serdes.String(), - spansSerde) - .withCachingEnabled() - .withLoggingDisabled(); - - StreamsBuilder builder = new StreamsBuilder(); - - // Aggregate TraceId:Spans - // This store could be removed once an RPC is used to getTraceIds Traces per instance based on prior - // aggregation. - builder - .addGlobalStore( - globalTracesStoreBuilder, - spansTopic, - Consumed.with(Serdes.String(), spanSerde), - () -> new Processor() { - KeyValueStore> tracesStore; - - @Override public void init(ProcessorContext context) { - tracesStore = - (KeyValueStore>) context.getStateStore(tracesStoreName); - } - - @Override public void process(String traceId, Span span) { - if (span == null) { - tracesStore.delete(traceId); - spanIndexService.deleteByTraceId(traceId); - } else { - List currentSpans = tracesStore.get(traceId); - if (currentSpans == null) { - currentSpans = Collections.singletonList(span); - } else { - currentSpans.add(span); - } - tracesStore.put(traceId, currentSpans); - spanIndexService.insert(span); - } - } - - @Override public void close() { - } - } - ); - - return builder.build(); - } -} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplier.java b/storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplier.java new file mode 100644 index 00000000..0e5899fd --- /dev/null +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplier.java @@ -0,0 +1,240 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.streams; + +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneId; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.function.Supplier; +import org.apache.kafka.common.serialization.Serdes; +import org.apache.kafka.streams.KeyValue; +import org.apache.kafka.streams.StreamsBuilder; +import org.apache.kafka.streams.Topology; +import org.apache.kafka.streams.kstream.Consumed; +import org.apache.kafka.streams.kstream.KStream; +import org.apache.kafka.streams.processor.Processor; +import org.apache.kafka.streams.processor.ProcessorContext; +import org.apache.kafka.streams.processor.PunctuationType; +import org.apache.kafka.streams.state.KeyValueIterator; +import org.apache.kafka.streams.state.KeyValueStore; +import org.apache.kafka.streams.state.Stores; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import zipkin2.Span; +import zipkin2.storage.kafka.streams.serdes.NamesSerde; +import zipkin2.storage.kafka.streams.serdes.SpanIdsSerde; +import zipkin2.storage.kafka.streams.serdes.SpansSerde; + +/** + * Storage of Traces, Service names and Autocomplete Tags. + */ +public class TraceStoreTopologySupplier implements Supplier { + public static final String TRACES_STORE_NAME = "zipkin-traces"; + public static final String SPAN_IDS_BY_TS_STORE_NAME = "zipkin-traces-by-timestamp"; + public static final String SERVICE_NAMES_STORE_NAME = "zipkin-service-names"; + public static final String SPAN_NAMES_STORE_NAME = "zipkin-span-names"; + public static final String REMOTE_SERVICE_NAMES_STORE_NAME = "zipkin-remote-service-names"; + public static final String AUTOCOMPLETE_TAGS_STORE_NAME = "zipkin-autocomplete-tags"; + + static final Logger LOG = LoggerFactory.getLogger(TraceStoreTopologySupplier.class); + // Kafka topics + final String spansTopicName; + // Limits + final List autoCompleteKeys; + final Duration traceTtl; + final Duration traceTtlCheckInterval; + final long minTracesStored; + // SerDes + final SpansSerde spansSerde; + final SpanIdsSerde spanIdsSerde; + final NamesSerde namesSerde; + + public TraceStoreTopologySupplier(String spansTopicName, List autoCompleteKeys, + Duration traceTtl, Duration traceTtlCheckInterval, long minTracesStored) { + this.spansTopicName = spansTopicName; + this.autoCompleteKeys = autoCompleteKeys; + this.traceTtl = traceTtl; + this.traceTtlCheckInterval = traceTtlCheckInterval; + this.minTracesStored = minTracesStored; + spansSerde = new SpansSerde(); + spanIdsSerde = new SpanIdsSerde(); + namesSerde = new NamesSerde(); + } + + @Override public Topology get() { + StreamsBuilder builder = new StreamsBuilder(); + + builder + .addStateStore(Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore(TRACES_STORE_NAME), + Serdes.String(), + spansSerde)) + .addStateStore(Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore(SPAN_IDS_BY_TS_STORE_NAME), + Serdes.Long(), + spanIdsSerde)) + .addStateStore(Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore(SERVICE_NAMES_STORE_NAME), + Serdes.String(), + Serdes.String())) + .addStateStore(Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore(SPAN_NAMES_STORE_NAME), + Serdes.String(), + namesSerde)) + .addStateStore(Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore(REMOTE_SERVICE_NAMES_STORE_NAME), + Serdes.String(), + namesSerde)) + .addStateStore(Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore(AUTOCOMPLETE_TAGS_STORE_NAME), + Serdes.String(), + namesSerde)); + // Traces stream + KStream> spansStream = builder + .stream(spansTopicName, Consumed.with(Serdes.String(), spansSerde)); + // Store traces + spansStream + .process(() -> new Processor>() { + ProcessorContext context; + // Actual traces store + KeyValueStore> tracesStore; + // timestamp index for trace IDs + KeyValueStore> spanIdsByTsStore; + + @Override public void init(ProcessorContext context) { + this.context = context; + tracesStore = + (KeyValueStore>) context.getStateStore(TRACES_STORE_NAME); + spanIdsByTsStore = + (KeyValueStore>) context.getStateStore(SPAN_IDS_BY_TS_STORE_NAME); + // Retention scheduling + context.schedule( + traceTtlCheckInterval, + PunctuationType.STREAM_TIME, + timestamp -> { + if (traceTtl.toMillis() > 0 && + tracesStore.approximateNumEntries() > minTracesStored) { + // preparing range filtering + long from = 0L; + long to = timestamp - traceTtl.toMillis(); + long toMicro = to * 1000; + // query traceIds active during period + try (final KeyValueIterator> all = + spanIdsByTsStore.range(from, toMicro)) { + int deletions = 0; // logging purpose + while (all.hasNext()) { + final KeyValue> record = all.next(); + spanIdsByTsStore.delete(record.key); // clean timestamp index + for (String traceId : record.value) { + tracesStore.delete(traceId); // clean traces store + deletions++; + } + } + if (deletions > 0) { + LOG.info("Traces deletion emitted: {}, older than {}", + deletions, + Instant.ofEpochMilli(to).atZone(ZoneId.systemDefault())); + } + } + } + }); + } + + @Override public void process(String traceId, List spans) { + if (!spans.isEmpty()) { + // Persist traces + List currentSpans = tracesStore.get(traceId); + if (currentSpans == null) currentSpans = new ArrayList<>(); + currentSpans.addAll(spans); + tracesStore.put(traceId, currentSpans); + // Persist timestamp indexed span ids + long timestamp = spans.get(0).timestamp(); + Set currentSpanIds = spanIdsByTsStore.get(timestamp); + if (currentSpanIds == null) currentSpanIds = new HashSet<>(); + currentSpanIds.add(traceId); + spanIdsByTsStore.put(timestamp, currentSpanIds); + } + } + + @Override public void close() { + } + }, TRACES_STORE_NAME, SPAN_IDS_BY_TS_STORE_NAME); + // Store service, span and remote service names + spansStream.process(() -> new Processor>() { + KeyValueStore serviceNameStore; + KeyValueStore> spanNamesStore; + KeyValueStore> remoteServiceNamesStore; + KeyValueStore> autocompleteTagsStore; + + @Override + public void init(ProcessorContext context) { + serviceNameStore = + (KeyValueStore) context.getStateStore(SERVICE_NAMES_STORE_NAME); + spanNamesStore = + (KeyValueStore>) context.getStateStore(SPAN_NAMES_STORE_NAME); + remoteServiceNamesStore = + (KeyValueStore>) context.getStateStore( + REMOTE_SERVICE_NAMES_STORE_NAME); + autocompleteTagsStore = + (KeyValueStore>) context.getStateStore( + AUTOCOMPLETE_TAGS_STORE_NAME); + } + + @Override + public void process(String traceId, List spans) { + for (Span span : spans) { + if (span.localServiceName() != null) { // if service name + serviceNameStore.putIfAbsent(span.localServiceName(), + span.localServiceName()); // store it + if (span.name() != null) { // store span names + Set spanNames = spanNamesStore.get(span.localServiceName()); + if (spanNames == null) spanNames = new HashSet<>(); + spanNames.add(span.name()); + spanNamesStore.put(span.localServiceName(), spanNames); + } + if (span.remoteServiceName() != null) { // store remote service names + Set remoteServiceNames = remoteServiceNamesStore.get(span.localServiceName()); + if (remoteServiceNames == null) remoteServiceNames = new HashSet<>(); + remoteServiceNames.add(span.remoteServiceName()); + remoteServiceNamesStore.put(span.localServiceName(), remoteServiceNames); + } + } + if (!span.tags().isEmpty()) { + span.tags().forEach((key, value) -> { + if (autoCompleteKeys.contains(key)) { + Set values = autocompleteTagsStore.get(key); + if (values == null) values = new HashSet<>(); + values.add(value); + autocompleteTagsStore.put(key, values); + } + }); + } + } + } + + @Override public void close() { + } + }, + SERVICE_NAMES_STORE_NAME, + SPAN_NAMES_STORE_NAME, + REMOTE_SERVICE_NAMES_STORE_NAME, + AUTOCOMPLETE_TAGS_STORE_NAME); + + return builder.build(); + } +} diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/DependencyLinkSerde.java b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/DependencyLinkSerde.java index 23a362f6..eb3a8f44 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/DependencyLinkSerde.java +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/DependencyLinkSerde.java @@ -13,6 +13,7 @@ */ package zipkin2.storage.kafka.streams.serdes; +import java.util.Map; import org.apache.kafka.common.serialization.Deserializer; import org.apache.kafka.common.serialization.Serde; import org.apache.kafka.common.serialization.Serializer; @@ -20,9 +21,12 @@ import zipkin2.codec.DependencyLinkBytesDecoder; import zipkin2.codec.DependencyLinkBytesEncoder; -import java.util.Map; - public class DependencyLinkSerde implements Serde { + static final String KEY_PATTERN = "%s:%s"; + + public static String linkKey(DependencyLink link) { + return String.format(KEY_PATTERN, link.parent(), link.child()); + } @Override public void configure(Map configs, boolean isKey) { diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanNamesSerde.java b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/NamesSerde.java similarity index 97% rename from storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanNamesSerde.java rename to storage/src/main/java/zipkin2/storage/kafka/streams/serdes/NamesSerde.java index 3200fbb7..f5095923 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanNamesSerde.java +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/NamesSerde.java @@ -13,18 +13,17 @@ */ package zipkin2.storage.kafka.streams.serdes; -import org.apache.kafka.common.serialization.Deserializer; -import org.apache.kafka.common.serialization.Serde; -import org.apache.kafka.common.serialization.Serializer; - import java.util.Arrays; import java.util.HashSet; import java.util.Map; import java.util.Set; +import org.apache.kafka.common.serialization.Deserializer; +import org.apache.kafka.common.serialization.Serde; +import org.apache.kafka.common.serialization.Serializer; import static java.nio.charset.StandardCharsets.UTF_8; -public class SpanNamesSerde implements Serde> { +public class NamesSerde implements Serde> { @Override public void configure(Map configs, boolean isKey) { //Nothing to do. diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanSerde.java b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanIdsSerde.java similarity index 55% rename from storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanSerde.java rename to storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanIdsSerde.java index 90e8028b..b21cc0e1 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanSerde.java +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpanIdsSerde.java @@ -13,79 +13,70 @@ */ package zipkin2.storage.kafka.streams.serdes; +import java.util.Arrays; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; import org.apache.kafka.common.serialization.Deserializer; import org.apache.kafka.common.serialization.Serde; import org.apache.kafka.common.serialization.Serializer; -import zipkin2.Span; -import zipkin2.codec.SpanBytesDecoder; -import zipkin2.codec.SpanBytesEncoder; - -import java.util.Map; - -public class SpanSerde implements Serde { - private final SpanBytesDecoder spanBytesDecoder; - - private final SpanBytesEncoder spanBytesEncoder; - - public SpanSerde() { - spanBytesDecoder = SpanBytesDecoder.PROTO3; - spanBytesEncoder = SpanBytesEncoder.PROTO3; - } +import static java.nio.charset.StandardCharsets.UTF_8; +public class SpanIdsSerde implements Serde> { @Override public void configure(Map configs, boolean isKey) { - // Nothing to configure + //Nothing to do. } @Override public void close() { - // No resources to close } @Override - public Serializer serializer() { - return new SpanSerializer(); + public Serializer> serializer() { + return new SpanNamesSerializer(); } @Override - public Deserializer deserializer() { - return new SpanDeserializer(); + public Deserializer> deserializer() { + return new SpanNamesDeserializer(); } - private class SpanSerializer implements Serializer { + public static class SpanNamesSerializer implements Serializer> { @Override public void configure(Map configs, boolean isKey) { - // Nothing to configure + //Nothing to do. } @Override - public byte[] serialize(String topic, Span data) { - return spanBytesEncoder.encode(data); + public byte[] serialize(String topic, Set data) { + String values = String.join("|", data); + return values.getBytes(UTF_8); } @Override public void close() { - // No resources to close } } - private class SpanDeserializer implements Deserializer { + public static class SpanNamesDeserializer implements Deserializer> { @Override public void configure(Map configs, boolean isKey) { - // Nothing to configure + //Nothing to do. } @Override - public Span deserialize(String topic, byte[] data) { - return spanBytesDecoder.decodeOne(data); + public Set deserialize(String topic, byte[] data) { + String decoded = new String(data, UTF_8); + String[] values = decoded.split("\\|"); + return new HashSet<>(Arrays.asList(values)); } @Override public void close() { - // No resources to close } } } diff --git a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpansSerde.java b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpansSerde.java index 94cd928f..f2d247d5 100644 --- a/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpansSerde.java +++ b/storage/src/main/java/zipkin2/storage/kafka/streams/serdes/SpansSerde.java @@ -14,6 +14,8 @@ package zipkin2.storage.kafka.streams.serdes; import java.util.ArrayList; +import java.util.List; +import java.util.Map; import org.apache.kafka.common.serialization.Deserializer; import org.apache.kafka.common.serialization.Serde; import org.apache.kafka.common.serialization.Serializer; @@ -21,9 +23,6 @@ import zipkin2.codec.SpanBytesDecoder; import zipkin2.codec.SpanBytesEncoder; -import java.util.List; -import java.util.Map; - public class SpansSerde implements Serde> { private final SpanBytesDecoder spanBytesDecoder; diff --git a/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageIT.java b/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageIT.java index f4e774b0..fa4726e6 100644 --- a/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageIT.java +++ b/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageIT.java @@ -14,44 +14,56 @@ package zipkin2.storage.kafka; import java.time.Duration; -import java.time.Instant; +import java.util.ArrayList; import java.util.Arrays; -import java.util.HashMap; +import java.util.Collection; +import java.util.Collections; import java.util.List; -import java.util.Map; import java.util.Properties; import java.util.concurrent.TimeUnit; +import org.apache.kafka.clients.admin.NewTopic; import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.common.serialization.ByteArrayDeserializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.kafka.streams.errors.InvalidStateStoreException; import org.apache.kafka.streams.integration.utils.IntegrationTestUtils; -import org.junit.After; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; import org.testcontainers.containers.KafkaContainer; -import zipkin2.Call; -import zipkin2.Callback; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; import zipkin2.CheckResult; +import zipkin2.DependencyLink; import zipkin2.Endpoint; import zipkin2.Span; import zipkin2.storage.QueryRequest; +import zipkin2.storage.ServiceAndSpanNames; import zipkin2.storage.SpanConsumer; import zipkin2.storage.SpanStore; +import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; +import zipkin2.storage.kafka.streams.serdes.SpansSerde; import static org.awaitility.Awaitility.await; import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; -import static zipkin2.TestObjects.TODAY; -public class KafkaStorageIT { - @Rule - public KafkaContainer kafka = new KafkaContainer("5.1.0"); +@Testcontainers +class KafkaStorageIT { + private static final long TODAY = System.currentTimeMillis(); + @Container private KafkaContainer kafka = new KafkaContainer("5.3.0"); + + private Duration traceTimeout; private KafkaStorage storage; private Properties testConsumerConfig; + private KafkaProducer> tracesProducer; + private KafkaProducer dependencyProducer; - @Before - public void start() { + @BeforeEach void start() { testConsumerConfig = new Properties(); testConsumerConfig.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); testConsumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, "test"); @@ -62,409 +74,204 @@ public void start() { if (!kafka.isRunning()) fail(); - long epochMilli = Instant.now().toEpochMilli(); - storage = (KafkaStorage) new KafkaStorage.Builder().ensureTopics(true) + traceTimeout = Duration.ofSeconds(5); + storage = (KafkaStorage) new KafkaStorage.Builder() .bootstrapServers(kafka.getBootstrapServers()) - .storeDirectory("target/zipkin_" + epochMilli) - .spansTopic(KafkaStorage.Topic.builder("zipkin").build()) - .traceInactivityGap(Duration.ofSeconds(5)) + .storeDirectory("target/zipkin_" + System.currentTimeMillis()) + .traceTimeout(traceTimeout) .build(); + + await().atMost(10, TimeUnit.SECONDS).until(() -> { + Collection newTopics = new ArrayList<>(); + newTopics.add(new NewTopic(storage.spansTopicName, 1, (short) 1)); + newTopics.add(new NewTopic(storage.traceTopicName, 1, (short) 1)); + newTopics.add(new NewTopic(storage.dependencyTopicName, 1, (short) 1)); + storage.getAdminClient().createTopics(newTopics).all().get(); + storage.checkTopics(); + return storage.topicsValidated; + }); + + await().atMost(10, TimeUnit.SECONDS).until(() -> storage.check().ok()); + + Properties producerConfig = new Properties(); + producerConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers()); + tracesProducer = new KafkaProducer<>(producerConfig, new StringSerializer(), + new SpansSerde().serializer()); + dependencyProducer = new KafkaProducer<>(producerConfig, new StringSerializer(), + new DependencyLinkSerde().serializer()); } - @After - public void closeStorageReleaseLock() { + @AfterEach void close() { + dependencyProducer.close(Duration.ofSeconds(1)); + dependencyProducer = null; + tracesProducer.close(Duration.ofSeconds(1)); + tracesProducer = null; storage.close(); storage = null; } - @Test - public void shouldCreateSpanAndService() throws Exception { - Span root = Span.newBuilder() - .traceId("a") - .id("a") + @Test void should_aggregate() throws Exception { + // Given: a set of incoming spans + Span parent = Span.newBuilder().traceId("a").id("a").name("op_a").kind(Span.Kind.CLIENT) .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) - .name("op_a") - .kind(Span.Kind.CLIENT) - .timestamp(TODAY) - .duration(10) + .timestamp(System.currentTimeMillis() * 1000).duration(10) .build(); - Span child = Span.newBuilder() - .traceId("a") - .id("b") + Span child = Span.newBuilder().traceId("a").id("b").name("op_b").kind(Span.Kind.SERVER) .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) - .name("op_b") - .kind(Span.Kind.SERVER) - .timestamp(TODAY) - .duration(2) + .timestamp(System.currentTimeMillis() * 1000).duration(2) .build(); - final SpanConsumer spanConsumer = storage.spanConsumer(); - - List spans = Arrays.asList(root, child); - spanConsumer.accept(spans).execute(); - + // When: are consumed by storage + spanConsumer.accept(Arrays.asList(parent, child)).execute(); + storage.getProducer().flush(); + // Then: they are partitioned IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spansTopic.name, 2, 10000); + testConsumerConfig, storage.spansTopicName, 2, 10000); + // Given: some time for stream processes to kick in + Thread.sleep(traceTimeout.toMillis() * 2); + // Given: another span to move 'event time' forward + Span another = Span.newBuilder().traceId("c").id("d").name("op_a").kind(Span.Kind.SERVER) + .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) + .timestamp(System.currentTimeMillis() * 1000).duration(2) + .build(); + // When: published + spanConsumer.accept(Collections.singletonList(another)).execute(); + storage.getProducer().flush(); + // Then: a trace is published + IntegrationTestUtils.waitUntilMinRecordsReceived( + testConsumerConfig, storage.spansTopicName, 1, 1000); IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spanServicesTopic.name, 2, 10000); + testConsumerConfig, storage.traceTopicName, 1, 30000); + // Then: and a dependency link created + IntegrationTestUtils.waitUntilMinRecordsReceived( + testConsumerConfig, storage.dependencyTopicName, 1, 1000); } - // TODO: implement dependency building validation as it is unclear how to test suppress feature i.e. how long to wait for dependencies? - - @Test - public void shouldFindTraces() throws Exception { - Span root = Span.newBuilder() - .traceId("a") - .id("a") + @Test void should_return_traces_query() throws Exception { + // Given: a trace prepared to be published + Span parent = Span.newBuilder().traceId("a").id("a").name("op_a").kind(Span.Kind.CLIENT) .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) .remoteEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) - .name("op_a") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) + .timestamp(TODAY * 1000).duration(10) .build(); - Span child = Span.newBuilder() - .traceId("a") - .id("b") + Span child = Span.newBuilder().traceId("a").id("b").name("op_b").kind(Span.Kind.SERVER) .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) - .name("op_b") - .kind(Span.Kind.SERVER) - .timestamp(Long.valueOf(TODAY + "000")) - .timestamp(TODAY) - .duration(2) + .timestamp(TODAY * 1000).duration(2) .build(); - List spans = Arrays.asList(root, child); - final SpanConsumer spanConsumer = storage.spanConsumer(); - final SpanStore spanStore = storage.spanStore(); - - spanConsumer.accept(spans).execute(); - + List spans = Arrays.asList(parent, child); + // When: been published + tracesProducer.send(new ProducerRecord<>(storage.spansTopicName, parent.traceId(), spans)); + tracesProducer.flush(); + // Then: stored IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spansTopic.name, 2, 10000); + testConsumerConfig, storage.spansTopicName, 1, 10000); + // When: and stores running + SpanStore spanStore = storage.spanStore(); + ServiceAndSpanNames serviceAndSpanNames = storage.serviceAndSpanNames(); + // Then: services names are searchable await().atMost(30, TimeUnit.SECONDS) .until(() -> { - List> traces = - spanStore.getTraces(QueryRequest.newBuilder() - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 1 && traces.get(0).size() == 2; - }); - } - - @Test - public void shouldFindTracesByTags() throws Exception { - Map annotationQuery = - new HashMap() { - { - put("key_tag_a", "value_tag_a"); + List> traces = new ArrayList<>(); + try { + traces = + spanStore.getTraces(QueryRequest.newBuilder() + .endTs(TODAY + 1) + .lookback(Duration.ofMinutes(1).toMillis()) + .serviceName("svc_a") + .limit(10) + .build()) + .execute(); + } catch (InvalidStateStoreException e) { // ignoring state issues + System.err.println(e.getMessage()); + } catch (Exception e) { + e.printStackTrace(); } - }; - - Span span1 = - Span.newBuilder() - .traceId("a") - .id("a") - .putTag("key_tag_a", "value_tag_a") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) - .name("op_a") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - Span span2 = - Span.newBuilder() - .traceId("b") - .id("b") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) - .putTag("key_tag_c", "value_tag_d") - .addAnnotation(Long.valueOf(TODAY + "000"), "annotation_b") - .name("op_b") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - final SpanConsumer spanConsumer = storage.spanConsumer(); - final SpanStore spanStore = storage.spanStore(); - - List spans = Arrays.asList(span1, span2); - spanConsumer.accept(spans).execute(); - - IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spansTopic.name, 2, 10000); - - // query by annotation {"key_tag_a":"value_tag_a"} = 1 trace - await() - .atMost(10, TimeUnit.SECONDS) - .until(() -> { - List> traces = - spanStore.getTraces(QueryRequest.newBuilder() - .annotationQuery(annotationQuery) - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 1; + return traces.size() == 1 + && traces.get(0).size() == 2; // Trace is found and has two spans }); - - // query by annotation {"key_tag_non_exist_a":"value_tag_non_exist_a"} = 0 trace - await() - .pollDelay(5, TimeUnit.SECONDS) - .until(() -> { - List> traces = - spanStore.getTraces(QueryRequest.newBuilder() - .annotationQuery( - new HashMap() {{ - put("key_tag_non_exist_a", "value_tag_non_exist_a"); - }}) - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 0; - }); - } - - @Test - public void shouldFindTracesByAnnotations() throws Exception { - Span span1 = - Span.newBuilder() - .traceId("a") - .id("a") - .putTag("key_tag_a", "value_tag_a") - .addAnnotation(TODAY, "log value") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) - .name("op_a") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - Span span2 = - Span.newBuilder() - .traceId("b") - .id("b") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) - .putTag("key_tag_c", "value_tag_d") - .addAnnotation(Long.valueOf(TODAY + "000"), "annotation_b") - .name("op_b") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - final SpanConsumer spanConsumer = storage.spanConsumer(); - final SpanStore spanStore = storage.spanStore(); - - List spans = Arrays.asList(span1, span2); - spanConsumer.accept(spans).execute(); - - IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spansTopic.name, 2, 10000); - - // query by annotation {"key_tag_a":"value_tag_a"} = 1 trace - await() - .atMost(10, TimeUnit.SECONDS) - .until(() -> { - List> traces = - spanStore.getTraces(QueryRequest.newBuilder() - .parseAnnotationQuery("log*") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 1; - }); - } - - @Test - public void shouldFindTracesBySpanName() throws Exception { - Span span1 = - Span.newBuilder() - .traceId("a") - .id("a") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) - .name("op_a") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - Span span2 = - Span.newBuilder() - .traceId("b") - .id("b") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) - .name("op_b") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - final SpanConsumer spanConsumer = storage.spanConsumer(); - final SpanStore spanStore = storage.spanStore(); - - List spans = Arrays.asList(span1, span2); - spanConsumer.accept(spans).execute(); - - IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spansTopic.name, 2, 10000); - - // query by span name `op_a` = 1 trace - await() - .atMost(5, TimeUnit.SECONDS) + await().atMost(5, TimeUnit.SECONDS) .until(() -> { - List> traces = - spanStore.getTraces( - QueryRequest.newBuilder() - .spanName("op_a") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 1; - }); - - // query by span name `op_b` = 1 trace - await() - .atMost(5, TimeUnit.SECONDS) + List services = new ArrayList<>(); + try { + services = serviceAndSpanNames.getServiceNames().execute(); + } catch (InvalidStateStoreException e) { // ignoring state issues + System.err.println(e.getMessage()); + } catch (Exception e) { + e.printStackTrace(); + } + return services.size() == 2; + }); // There are two service names + await().atMost(5, TimeUnit.SECONDS) .until(() -> { - List> traces = - spanStore.getTraces( - QueryRequest.newBuilder() - .spanName("op_b") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 1; - }); - - // query by span name `non_existing_span_name` = 0 trace - await() - .pollDelay(5, TimeUnit.SECONDS) + List spanNames = new ArrayList<>(); + try { + spanNames = serviceAndSpanNames.getSpanNames("svc_a") + .execute(); + } catch (InvalidStateStoreException e) { // ignoring state issues + System.err.println(e.getMessage()); + } catch (Exception e) { + e.printStackTrace(); + } + return spanNames.size() == 1; + }); // Service names have one span name + await().atMost(5, TimeUnit.SECONDS) .until(() -> { - List> traces = - spanStore.getTraces( - QueryRequest.newBuilder() - .spanName("non_existing_span_name") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 0; - }); + List services = new ArrayList<>(); + try { + services = serviceAndSpanNames.getRemoteServiceNames("svc_a").execute(); + } catch (InvalidStateStoreException e) { // ignoring state issues + System.err.println(e.getMessage()); + } catch (Exception e) { + e.printStackTrace(); + } + return services.size() == 1; + }); // And one remote service name } - @Test - public void shouldFindTracesByServiceName() throws Exception { - Span span1 = - Span.newBuilder() - .traceId("a") - .id("a") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) - .name("op_a") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - Span span2 = - Span.newBuilder() - .traceId("b") - .id("b") - .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) - .name("op_b") - .kind(Span.Kind.CLIENT) - .timestamp(Long.valueOf(TODAY + "000")) - .duration(10) - .build(); - - final SpanConsumer spanConsumer = storage.spanConsumer(); - final SpanStore spanStore = storage.spanStore(); - - List spans = Arrays.asList(span1, span2); - spanConsumer.accept(spans).execute(); - + @Test void should_find_dependencies() throws Exception { + //Given: two related dependency links + // When: sent first one + dependencyProducer.send( + new ProducerRecord<>(storage.dependencyTopicName, "svc_a:svc_b", + DependencyLink.newBuilder() + .parent("svc_a") + .child("svc_b") + .callCount(1) + .errorCount(0) + .build())); + // When: and another one + dependencyProducer.send( + new ProducerRecord<>(storage.dependencyTopicName, "svc_a:svc_b", + DependencyLink.newBuilder() + .parent("svc_a") + .child("svc_b") + .callCount(1) + .errorCount(0) + .build())); + dependencyProducer.flush(); + // Then: stored in topic IntegrationTestUtils.waitUntilMinRecordsReceived( - testConsumerConfig, storage.spansTopic.name, 2, 10000); - - // query by service name `srv_a` = 2 trace - await() - .atMost(10, TimeUnit.SECONDS) - .until(() -> { - List> traces = - spanStore.getTraces( - QueryRequest.newBuilder() - .serviceName("svc_a") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - return traces.size() == 2; - }); - - List> traces = spanStore.getTraces( - QueryRequest.newBuilder() - .serviceName("non_existing_span_name") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()) - .execute(); - assertEquals(0, traces.size()); - } - - @Test - public void shouldEnqueueTraceQuery() { - final SpanStore spanStore = storage.spanStore(); - Call>> callTraces = - spanStore.getTraces( - QueryRequest.newBuilder() - .serviceName("non_existing_span_name") - .endTs(TODAY + 1) - .limit(10) - .lookback(Duration.ofMinutes(1).toMillis()) - .build()); - - Callback>> callback = new Callback>>() { - @Override - public void onSuccess(List> value) { - System.out.println("Here: " + value); + testConsumerConfig, storage.dependencyTopicName, 2, 10000); + // When: stores running + SpanStore spanStore = storage.spanStore(); + // Then: + await().atMost(10, TimeUnit.SECONDS).until(() -> { + List links = new ArrayList<>(); + try { + links = + spanStore.getDependencies(System.currentTimeMillis(), Duration.ofMinutes(2).toMillis()) + .execute(); + } catch (InvalidStateStoreException e) { // ignoring state issues + System.err.println(e.getMessage()); + } catch (Exception e) { + e.printStackTrace(); } - - @Override - public void onError(Throwable t) { - t.printStackTrace(); - } - }; - - try { - callTraces.enqueue(callback); - } catch (Exception e) { - fail(); - } - - try { - callTraces.enqueue(callback); - fail(); - } catch (Exception ignored) { - } + return links.size() == 1 + && links.get(0).callCount() == 2; // link stored and call count aggregated. + }); } - @Test - public void shouldFailWhenKafkaNotAvailable() { + @Test void shouldFailWhenKafkaNotAvailable() { CheckResult checked = storage.check(); assertEquals(CheckResult.OK, checked); diff --git a/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageTest.java b/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageTest.java index 3844e277..c4297f04 100644 --- a/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageTest.java +++ b/storage/src/test/java/zipkin2/storage/kafka/KafkaStorageTest.java @@ -13,71 +13,34 @@ */ package zipkin2.storage.kafka; -import java.util.Arrays; -import org.junit.Test; +import org.junit.jupiter.api.Test; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.fail; -public class KafkaStorageTest { +// Testing builder +class KafkaStorageTest { - @Test - public void notSupported() { + @Test void notSupported() { try { KafkaStorage.newBuilder().strictTraceId(false); fail(); } catch (IllegalArgumentException ignored) { } - - try { - KafkaStorage.newBuilder().searchEnabled(true); - fail(); - } catch (IllegalArgumentException ignored) { - } - - try { - KafkaStorage.newBuilder().autocompleteKeys(null); - fail(); - } catch (NullPointerException ignored) { - } - - try { - KafkaStorage.newBuilder().autocompleteKeys(Arrays.asList("key1", "key2")); - fail(); - } catch (IllegalArgumentException ignored) { - } } - @Test - public void buildDefaultBuilder() { + @Test void buildDefaultBuilder() { KafkaStorage.Builder builder = KafkaStorage.newBuilder(); - assertNotNull(builder.dependencyStoreName); - assertNotNull(builder.storeDirectory); + assertNotNull(builder.storeDir); try { - builder.spansTopic(null); + builder.spansTopicName(null); fail(); } catch (NullPointerException ignored) { } try { - builder.spanServicesTopic(null); - fail(); - } catch (NullPointerException ignored) { - } - try { - builder.servicesTopic(null); - fail(); - } catch (NullPointerException ignored) { - } - - try { - builder.spanDependenciesTopic(null); - fail(); - } catch (NullPointerException ignored) { - } - try { - builder.dependenciesTopic(null); + builder.dependenciesTopicName(null); fail(); } catch (NullPointerException ignored) { } @@ -88,40 +51,4 @@ public void buildDefaultBuilder() { } catch (NullPointerException ignored) { } } - - @Test - public void topicDefault() { - try{ - KafkaStorage.Topic.builder(null); - fail(); - } catch (NullPointerException ignored){} - - KafkaStorage.Topic.Builder topicBuilder = KafkaStorage.Topic.builder("topic-1"); - - try { - topicBuilder.partitions(0); - fail(); - } catch (IllegalArgumentException ignored){} - - try { - topicBuilder.partitions(null); - fail(); - } catch (NullPointerException ignored){} - - try { - topicBuilder.partitions(-1); - fail(); - } catch (IllegalArgumentException ignored){} - - try { - topicBuilder.replicationFactor(null); - fail(); - } catch (NullPointerException ignored){} - - try { - topicBuilder.replicationFactor( (short) 0); - fail(); - } catch (IllegalArgumentException ignored){} - - } } diff --git a/storage/src/test/java/zipkin2/storage/kafka/streams/AggregationTopologySupplierTest.java b/storage/src/test/java/zipkin2/storage/kafka/streams/AggregationTopologySupplierTest.java new file mode 100644 index 00000000..751efc90 --- /dev/null +++ b/storage/src/test/java/zipkin2/storage/kafka/streams/AggregationTopologySupplierTest.java @@ -0,0 +1,92 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.streams; + +import java.time.Duration; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Properties; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.kafka.streams.StreamsConfig; +import org.apache.kafka.streams.Topology; +import org.apache.kafka.streams.TopologyDescription; +import org.apache.kafka.streams.TopologyTestDriver; +import org.apache.kafka.streams.test.ConsumerRecordFactory; +import org.apache.kafka.streams.test.OutputVerifier; +import org.junit.jupiter.api.Test; +import zipkin2.DependencyLink; +import zipkin2.Endpoint; +import zipkin2.Span; +import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; +import zipkin2.storage.kafka.streams.serdes.SpansSerde; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; + +class AggregationTopologySupplierTest { + + @Test void should_aggregate_spans_and_map_dependencies() { + // Given: configuration + String spansTopicName = "spans"; + String tracesTopicName = "traces"; + String dependencyLinksTopicName = "dependencies"; + Duration traceTimeout = Duration.ofSeconds(1); + SpansSerde spansSerde = new SpansSerde(); + DependencyLinkSerde dependencyLinkSerde = new DependencyLinkSerde(); + // When: topology built + Topology topology = new AggregationTopologySupplier( + spansTopicName, tracesTopicName, dependencyLinksTopicName, traceTimeout).get(); + TopologyDescription description = topology.describe(); + System.out.println("Topology: \n" + description); + // Then: single threaded topology + assertEquals(1, description.subtopologies().size()); + // Given: test driver + Properties props = new Properties(); + props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); + props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234"); + TopologyTestDriver testDriver = new TopologyTestDriver(topology, props); + // When: two related spans coming on the same Session window + ConsumerRecordFactory> factory = + new ConsumerRecordFactory<>(spansTopicName, new StringSerializer(), spansSerde.serializer()); + Span a = Span.newBuilder().traceId("a").id("a").name("op_a").kind(Span.Kind.CLIENT) + .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) + .build(); + Span b = Span.newBuilder().traceId("a").id("b").name("op_b").kind(Span.Kind.SERVER) + .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) + .build(); + testDriver.pipeInput(factory.create(spansTopicName, a.traceId(), Collections.singletonList(a), 0L)); + testDriver.pipeInput(factory.create(spansTopicName, b.traceId(), Collections.singletonList(b), 0L)); + // When: and new record arrive, moving the event clock further than inactivity gap + Span c = Span.newBuilder().traceId("c").id("c").build(); + testDriver.pipeInput(factory.create(spansTopicName, c.traceId(), Collections.singletonList(c), traceTimeout.toMillis() + 1)); + // Then: a trace is aggregated.1 + ProducerRecord> trace = + testDriver.readOutput(tracesTopicName, new StringDeserializer(), spansSerde.deserializer()); + assertNotNull(trace); + OutputVerifier.compareKeyValue(trace, a.traceId(), Arrays.asList(a, b)); + // Then: a dependency link is created + ProducerRecord linkRecord = + testDriver.readOutput(dependencyLinksTopicName, new StringDeserializer(), + dependencyLinkSerde.deserializer()); + assertNotNull(linkRecord); + DependencyLink link = DependencyLink.newBuilder() + .parent("svc_a").child("svc_b").callCount(1).errorCount(0) + .build(); + OutputVerifier.compareKeyValue(linkRecord, "svc_a:svc_b", link); + } + +} \ No newline at end of file diff --git a/storage/src/test/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplierTest.java b/storage/src/test/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplierTest.java new file mode 100644 index 00000000..8f6c8d70 --- /dev/null +++ b/storage/src/test/java/zipkin2/storage/kafka/streams/DependencyStoreTopologySupplierTest.java @@ -0,0 +1,93 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.streams; + +import java.time.Duration; +import java.util.Properties; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.kafka.streams.StreamsConfig; +import org.apache.kafka.streams.Topology; +import org.apache.kafka.streams.TopologyDescription; +import org.apache.kafka.streams.TopologyTestDriver; +import org.apache.kafka.streams.kstream.Windowed; +import org.apache.kafka.streams.state.KeyValueIterator; +import org.apache.kafka.streams.state.WindowStore; +import org.apache.kafka.streams.state.WindowStoreIterator; +import org.apache.kafka.streams.test.ConsumerRecordFactory; +import org.junit.jupiter.api.Test; +import zipkin2.DependencyLink; +import zipkin2.storage.kafka.streams.serdes.DependencyLinkSerde; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static zipkin2.storage.kafka.streams.DependencyStoreTopologySupplier.DEPENDENCIES_STORE_NAME; + +class DependencyStoreTopologySupplierTest { + + @Test void should_store_dependencies() { + // Given: configs + String dependencyTopicName = "zipkin-dependency"; + DependencyLinkSerde dependencyLinkSerde = new DependencyLinkSerde(); + Duration dependenciesRetentionPeriod = Duration.ofMinutes(1); + Duration dependenciesWindowSize = Duration.ofMillis(100); + // When: topology created + Topology topology = new DependencyStoreTopologySupplier( + dependencyTopicName, + dependenciesRetentionPeriod, + dependenciesWindowSize + ).get(); + TopologyDescription description = topology.describe(); + System.out.println("Topology: \n" + description); + // Then: 2 threads prepared + assertEquals(1, description.subtopologies().size()); + // Given: streams configuration + Properties props = new Properties(); + props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); + props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234"); + props.put(StreamsConfig.STATE_DIR_CONFIG, + "target/kafka-streams-test/" + System.currentTimeMillis()); + TopologyTestDriver testDriver = new TopologyTestDriver(topology, props); + // When: a trace is passed + ConsumerRecordFactory factory = + new ConsumerRecordFactory<>(dependencyTopicName, new StringSerializer(), + dependencyLinkSerde.serializer()); + DependencyLink dependencyLink = DependencyLink.newBuilder() + .parent("svc_a").child("svc_b").callCount(1).errorCount(0) + .build(); + String dependencyLinkId = "svc_a:svc_b"; + testDriver.pipeInput( + factory.create(dependencyTopicName, dependencyLinkId, dependencyLink, 10L)); + WindowStore links = + testDriver.getWindowStore(DEPENDENCIES_STORE_NAME); + // Then: dependency link created + WindowStoreIterator fetch1 = links.fetch(dependencyLinkId, 0L, 100L); + assertTrue(fetch1.hasNext()); + assertEquals(fetch1.next().value, dependencyLink); + // When: new links appear + testDriver.pipeInput( + factory.create(dependencyTopicName, dependencyLinkId, dependencyLink, 90L)); + // Then: dependency link increases + WindowStoreIterator fetch2 = links.fetch(dependencyLinkId, 0L, 100L); + assertTrue(fetch2.hasNext()); + assertEquals(fetch2.next().value.callCount(), 2); + // When: time moves forward + testDriver.advanceWallClockTime(dependenciesRetentionPeriod.toMillis() + 91L); + testDriver.pipeInput( + factory.create(dependencyTopicName, dependencyLinkId, dependencyLink)); + // Then: dependency link is removed and restarted + KeyValueIterator, DependencyLink> fetch3 = links.all(); + assertTrue(fetch3.hasNext()); + assertEquals(fetch3.next().value.callCount(), 1); + } +} \ No newline at end of file diff --git a/storage/src/test/java/zipkin2/storage/kafka/streams/StreamGraphPrinter.java b/storage/src/test/java/zipkin2/storage/kafka/streams/StreamGraphPrinter.java deleted file mode 100644 index 3981dc69..00000000 --- a/storage/src/test/java/zipkin2/storage/kafka/streams/StreamGraphPrinter.java +++ /dev/null @@ -1,57 +0,0 @@ -/* - * Copyright 2019 jeqo - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except - * in compliance with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package zipkin2.storage.kafka.streams; - -import no.sysco.middleware.kafka.util.StreamsTopologyGraphviz; -import org.apache.kafka.streams.Topology; - -public class StreamGraphPrinter { - public static void main(String[] args) { - String spanTopicName = "zipkin-span-v1"; - String spanServicesTopicName = "zipkin-span-services-v1"; - String servicesTopicName = "zipkin-services-v1"; - String spanDependenciesTopicName = "zipkin-span-dependencies-v1"; - String dependenciesTopicName = "zipkin-dependencies-v1"; - - System.out.println("# TRACE STORE TOPOLOGY"); - Topology traceStoreTopology = new TraceStoreStream(spanTopicName, spanTopicName, - null).get(); - System.out.println(StreamsTopologyGraphviz.print(traceStoreTopology)); - System.out.println(); - - System.out.println("# SERVICE AGGREGATION TOPOLOGY"); - Topology serviceAggregationTopology = - new ServiceAggregationStream(spanServicesTopicName, servicesTopicName).get(); - System.out.println(StreamsTopologyGraphviz.print(serviceAggregationTopology)); - System.out.println(); - - System.out.println("# SERVICE STORE TOPOLOGY"); - Topology serviceStoreTopology = - new ServiceStoreStream(servicesTopicName, servicesTopicName).get(); - System.out.println(StreamsTopologyGraphviz.print(serviceStoreTopology)); - System.out.println(); - - System.out.println("# DEPENDENCY AGGREGATION TOPOLOGY"); - Topology dependencyAggregationTopology = - new DependencyAggregationStream(spanTopicName, spanDependenciesTopicName, - dependenciesTopicName).get(); - System.out.println(StreamsTopologyGraphviz.print(dependencyAggregationTopology)); - System.out.println(); - - System.out.println("# DEPENDENCY STORE TOPOLOGY"); - Topology dependencyStoreTopology = - new DependencyStoreStream(dependenciesTopicName, dependenciesTopicName).get(); - System.out.println(StreamsTopologyGraphviz.print(dependencyStoreTopology)); - } -} diff --git a/storage/src/test/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplierTest.java b/storage/src/test/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplierTest.java new file mode 100644 index 00000000..f5497571 --- /dev/null +++ b/storage/src/test/java/zipkin2/storage/kafka/streams/TraceStoreTopologySupplierTest.java @@ -0,0 +1,119 @@ +/* + * Copyright 2019 jeqo + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package zipkin2.storage.kafka.streams; + +import java.time.Duration; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Properties; +import java.util.Set; +import org.apache.kafka.common.serialization.StringSerializer; +import org.apache.kafka.streams.StreamsConfig; +import org.apache.kafka.streams.Topology; +import org.apache.kafka.streams.TopologyDescription; +import org.apache.kafka.streams.TopologyTestDriver; +import org.apache.kafka.streams.state.KeyValueIterator; +import org.apache.kafka.streams.state.KeyValueStore; +import org.apache.kafka.streams.test.ConsumerRecordFactory; +import org.junit.jupiter.api.Test; +import zipkin2.Endpoint; +import zipkin2.Span; +import zipkin2.storage.kafka.streams.serdes.SpansSerde; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.AUTOCOMPLETE_TAGS_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.SERVICE_NAMES_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.SPAN_IDS_BY_TS_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.SPAN_NAMES_STORE_NAME; +import static zipkin2.storage.kafka.streams.TraceStoreTopologySupplier.TRACES_STORE_NAME; + +class TraceStoreTopologySupplierTest { + + @Test void should_persist_stores() { + // Given: configs + String spansTopicName = "zipkin-spans"; + Duration traceTtl = Duration.ofMillis(5); + Duration traceTtlCheckInterval = Duration.ofMinutes(1); + List autocompleteKeys = Collections.singletonList("environment"); + SpansSerde spansSerde = new SpansSerde(); + // When: topology provided + Topology topology = new TraceStoreTopologySupplier( + spansTopicName, + autocompleteKeys, + traceTtl, + traceTtlCheckInterval, + 0).get(); + TopologyDescription description = topology.describe(); + System.out.println("Topology: \n" + description); + // Then: 2 threads prepared + assertEquals(1, description.subtopologies().size()); + // Given: streams config + Properties props = new Properties(); + props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); + props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234"); + props.put(StreamsConfig.STATE_DIR_CONFIG, + "target/kafka-streams-test/" + System.currentTimeMillis()); + TopologyTestDriver testDriver = new TopologyTestDriver(topology, props); + // When: a trace is passed + ConsumerRecordFactory> factory = + new ConsumerRecordFactory<>(spansTopicName, new StringSerializer(), + spansSerde.serializer()); + Span a = Span.newBuilder().traceId("a").id("a").name("op_a").kind(Span.Kind.CLIENT) + .localEndpoint(Endpoint.newBuilder().serviceName("svc_a").build()) + .timestamp(10000L).duration(11L) + .putTag("environment", "dev") + .build(); + Span b = Span.newBuilder().traceId("a").id("b").name("op_b").kind(Span.Kind.SERVER) + .localEndpoint(Endpoint.newBuilder().serviceName("svc_b").build()) + .timestamp(10000L).duration(10L) + .build(); + List spans = Arrays.asList(a, b); + testDriver.pipeInput(factory.create(spansTopicName, a.traceId(), spans, 10L)); + // Then: trace stores are filled + KeyValueStore> traces = + testDriver.getKeyValueStore(TRACES_STORE_NAME); + assertEquals(traces.get(a.traceId()), spans); + KeyValueStore> spanIdsByTs = + testDriver.getKeyValueStore(SPAN_IDS_BY_TS_STORE_NAME); + KeyValueIterator> ids = spanIdsByTs.all(); + assertTrue(ids.hasNext()); + assertEquals(ids.next().value, Collections.singleton(a.traceId())); + // Then: service name stores are filled + KeyValueStore serviceNames = + testDriver.getKeyValueStore(SERVICE_NAMES_STORE_NAME); + assertEquals("svc_a", serviceNames.get("svc_a")); + assertEquals("svc_b", serviceNames.get("svc_b")); + KeyValueStore> spanNames = + testDriver.getKeyValueStore(SPAN_NAMES_STORE_NAME); + assertEquals(Collections.singleton("op_a"), spanNames.get("svc_a")); + assertEquals(Collections.singleton("op_b"), spanNames.get("svc_b")); + KeyValueStore> autocompleteTags = + testDriver.getKeyValueStore(AUTOCOMPLETE_TAGS_STORE_NAME); + assertEquals(Collections.singleton("dev"), autocompleteTags.get("environment")); + // When: clock moves forward + Span c = Span.newBuilder() + .traceId("c").id("c") + .timestamp(traceTtlCheckInterval.toMillis() * 1000 + 20000L) + .build(); + testDriver.pipeInput( + factory.create(spansTopicName, c.traceId(), Collections.singletonList(c), + traceTtlCheckInterval.toMillis() + 1)); + + // Then: Traces store is empty + assertNull(traces.get(a.traceId())); + } +} \ No newline at end of file