Broker Side Tracing (alpha)

[eolivelli]

JMS Traces is a configurable pulsar interceptor that logs a variety of internal events.

Traces are sent to the regular SLF4J logger with the logger name “jms-tracing-<event>”. See jmsTracingEventList’s description below for a full list of possible events (lower cased in logger name).
One can configure a logger component (log4j etc) to separate these into a separate file, send to a separate destination, etc. as needed, see example below.
Alternatively this can be used in grep/splunk searches across the logs.

Traces are JSON and can be parsed as such.

Configuration options

brokerInterceptorsDirectory is a standard pulsar’s parameter that points to a directory with interceptors’ nar files. Place the nar into the directory configured there.

brokerInterceptors is a standard pulsar’s parameter that enables available (placed in the directory above) interceptors. Add jms-tracing to the list to enable.

jmsTracingEventList is a comma-separated list of event types to trace
Available options are:

• ADMINISTRATIVE - client-related events (connection opened/closed, consumer or producer connected /disconnected))
• COMMANDS - pulsar command executed (not sure if all pulsar-admin etc commands go through this interface, some internal commands do)
• MESSAGE - details of a message flow (before send, publish, dispatch, ack)
• TRANSACTION - details of transaction (opened/closed)

By default ALL events are traced.

jmsTracingLevel is a tracing level for all events except for the MESSAGE events.
Available options are:

• OFF - disables traces
• ON - enables traces

Default is OFF.

jmsTracingTraceSystemTopics is respected where the topic is known.

MESSAGE events will use per topic’s or per subscription’s jmsTraceLevel:

• Topic property “trace” (bin/pulsar-admin topics update-properties -p trace=ON public/default/mytopic) for producers and consumers, OFF if not set.
• Consumers can override topic property with a subscription property “trace” (bin/pulsar-admin topics update-subscription-properties -p trace=ON -s sub1 public/default/test).
• Producers/consumers for system topics are only traced when jmsTracingTraceSystemTopics=true; otherwise topic/subscription property won’t be checked and tracing will not happen.

cacheTraceLevelsDurationSec sets duration in seconds of how frequently trace levels set per topic or per subscription should be re-read. Default is 10 seconds.

jmsTracingMaxBinaryDataLength controls if the data preview for the entries should be included (value over 0) and the max length of data to trim it to. Data will be output as a text unless the conversion to text throws an exception, in that case as a hex string. Default is 256.

jmsTracingTraceSystemTopics controls whether events related to the system (pulsar-internal) topics should be traced. Applicable to the events where the topic is known. Default is false.

jmsTracingTraceSchema controls whether details of topic schema should be traced. Default is false.