logs-in-context-log4j2

Logs In Context With Log4j2

Introduction

This project contains a Java application configured to use Log4j2 to write JSON structured logs that propagate OpenTelemetry trace context onto log messages. It also contains a docker-compose.yaml which illustrates how to forward these logs to an OpenTelemetry Collector, and onto New Relic.

The log4j2.xml configures the application to log out to the console with a JSON Template Layout defined in Log4j2EventLayout.json. The layout follows the New Relic structured logging conventions.

The application uses the OpenTelemetry Log4j2 Integration to inject trace context to Log4j2 thread context.

The result is JSON structured logs, with one JSON object per line, which have the span.id and trace.id from OpenTelemetry included:

{
  "timestamp": "2021-05-19T15:51:16.063-05:00",
  "thread.name": "http-nio-8080-exec-1",
  "log.level": "INFO",
  "logger.name": "com.newrelic.app.Controller",
  "message": "A sample log message!",
  "trace.id": "6aae93314fe034149cd85f07eac24bc5",
  "span.id": "f1be31bc6e4471d8"
}

Run

The application runs with Docker. The docker-compose.yaml contains service definitions for the application and an OpenTelemetry Collector. The application is configured to use the Fluentd logging driver to forward logs the collector. The collector is configured to receive Fluentd logs and forward them to New Relic over OTLP.

The following image illustrates a similar example using FluentBit:

Next, build and run the application:

// Build the application with gradle
./gradlew logs-in-context-log4j2:bootJar

// Export your New Relic API key as an environment variable
export NEW_RELIC_API_KEY=<INSERT-API-KEY-HERE>

// Run the application and the collector with docker compose
docker-compose -f logs-in-context-log4j2/docker-compose.yaml up

Exercise logs in context by calling the GET /ping, which generated a log message inside the context of a trace:

curl http://localhost:8080/ping

You should be able to see a mix of trace and log data flowing through the collector. If you navigate to the distributed traces of the application in New Relic One, you should be able to find traces related to the call to GET /ping, and see the logs in context:

Trace With Logs

Trace Logs In Context