Dynamic tracing with BTrace

Sometimes logging and operation tracing give too much or too little information for tracing a specific issue.

BTrace is a tool for dynamically tracing and instrumenting java code (the java equivalent of DTrace). You write a probe in java then dynamically insert it into a running Xenon host.

$ PID=$(jps | awk '/my-service-host-1.0.0.jar/ { print $1 }')
$ bin/btrace -cp /path/to/my-service-host-1.0.0.jar $PID Xenon.java 

Some examples of things that can be done for Xenon with BTrace:

Print all the the operations being sent in the host

Print Operation details such as URI, body as well as the stacktrace leading to the operation.

package com.vmware.xenon.btrace;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.net.*;

@BTrace public class Xenon {
    @OnMethod(
        clazz="+com.vmware.xenon.common.ServiceClient",
        method="send")
    public static void onClientSend(com.vmware.xenon.common.Operation operation) {
        // print the toString of the Operation (alternatively can print only specific fields):
        println("Client Send: " + operation);

        // optionally print the stacktrace useful for understanding which service the operation originated from
        jstack(); 
    }
}

This is similar to the operation tracing feature, but you can filter for specific URIs, services or include more relevant information.

Create a histogram of accessed URIs

A histogram with the number of times each URI has been accessed, printed every 4 seconds. As an example this also filters out some operations based on path and referer (requires reflection since BTrace can't access methods/instances directly).

package com.vmware.xenon.btrace;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.net.*;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;

@BTrace public class Xenon {
    public static String ignorePath = "/core/node-groups/default";
    public static String ignoreReferer = "/core/document-index";
    private static Map<String, AtomicInteger> histo = Collections.newHashMap();

    @OnMethod(
        clazz="+com.vmware.xenon.common.ServiceClient",
        method="send")
    public static void onClientSend(com.vmware.xenon.common.Operation operation) {
        // filter our operations where the referer is the document index
        Object refererUri = Reflective.get(Reflective.field("com.vmware.xenon.common.Operation", "referer"), operation);
        String referer = str(Reflective.get(Reflective.field("java.net.URI", "path"), refererUri));
        if (compare(ignoreReferer, referer)) {
            return;
        }

        // filter out node-group operations
        Object uri = Reflective.get(Reflective.field("com.vmware.xenon.common.Operation", "uri"), operation);
        String path = str(Reflective.get(Reflective.field("java.net.URI", "path"), uri));
        if (compare(ignorePath, path)) {
            return;
        }

        AtomicInteger ai = Collections.get(histo, path);
        if (ai == null) {
            ai = Atomic.newAtomicInteger(1);
            Collections.put(histo, path, ai);
        } else {
            Atomic.incrementAndGet(ai);
        } 
    }

    @OnTimer(4000) 
    public static void print() {
        if (Collections.size(histo) != 0) {
            printNumberMap("Path Histogram", histo);
        }
    }
}

More Info

• Download btrace from here: https://github.com/jbachorik/btrace/releases/latest
• BTrace docs and samples: https://github.com/jbachorik/btrace