Optimizing the json-serialization of the Json-Logger

[dsidirop]

The json-logger uses "encoding/json".

Any insights as to whether it would make sense to refactor the logger to use a more performant json-library like "json-iterator" or maybe an adapter for "zero-log"?

Thanks in advance!

[peterbourgon]

As far as I'm aware, encoding/json does not represent a bottleneck in this codepath.

[dsidirop]

As far as I'm aware, encoding/json does not represent a bottleneck in this codepath.

My understanding (and feel free to correct me if I'm wrong) is that devs using go-kit can use the json-logger at their heart's desire to log stuff however they see fit.

So the critical path can be arbitrary for each application / project.

In this sense for applications that log in the hot path, reflection-based serialization and string formatting are prohibitively expensive — they're CPU-intensive and make many small allocations. Put differently, using encoding/json and fmt.Fprintf to log tons of interface{}s makes applications slow.

Granted one can roll out his own adapter for logging and do away with the default json-logger provided by go-kit, but I was just thinking whether it would make sense for go-kit to be a bit smarter and offer a more performant json-logging solution right out of the box.

Sorry if I'm missing something.

[peterbourgon]

In this sense for applications that log in the hot path, reflection-based serialization and string formatting are prohibitively expensive

The Go kit log system doesn't actually serialize keyvals until they get to the concrete e.g. JSONLogger implementation of the abstract Logger interface. If you emit logs in a hot path loop, and the cost of log serialization affects the performance of that loop, it implies that you serialize log events synchronously with each iteration, which is fundamentally a design error. If a bit of code has performance SLOs that would be impacted by encoding/json vs. jsoniter, then you simply can't do that work in-band.

(If you're emitting log events end-to-end with each iteration, serialization is just one part of the cost equation. Another equally important cost is the time it takes to write to your output io.Writer. In Linux, os.Stdout has a default buffer of a few kilobytes, and os.Stderr no buffer at all. If the thing that receives the output streams of a given process blocks, writes to those file descriptors will also block. Doing this work as part of a performance-sensitive hot loop is risky business!)

If you insist on serializing hot-path log events as part of each iteration, then you can — and should — provide your own concrete Logger implementation. The log package is designed to make this very easy.

Ultimately, though, decisions like this are a function of network effects. There are many, many, many consumers of Go kit's log package in the broader ecosystem. Every transitive dependency we introduce to this module is an enormous cost that must carry equivalent justification. Right now, the only dependency is on github.com/go-logfmt/logfmt, and to be honest if I could do it all over again, I would design the module so that even that one was removed.

[ChrisHines]

There is some work going on to create a better json package for the standard library that I've been helping to review. Once it's ready, the plan is to put it through the proposal process and hopefully get it accepted. If that happens, then I think Go kit would consider using it since standard library packages shouldn't be off limits for us. The experimental code is publicly mirrored at https://github.com/go-json-experiment/json. The README there explains the project in more detail.

Some of my contributions to the design discussions were related to performance concerns for logging use cases such as Go kit. Although, subject to change, some recent benchmarks I saw show the new version is significantly more performant than the existing standard library package for the kinds of use that Go kit has for logging. So there is hope for a better answer in the future, but there are no guarantees or well known timelines about when it will be available in a way Go kit would be willing to commit to using it as a default.