Instrument code with opentracing library

[KyleAMathews]

It can be complex to understand what is taking time and where inside Gatsby.

It would help immensely if there were fine-grained data we could use for understanding this. In the distributed computing world, "tracing" has become quite common for this sort of thing. One effort there, opentracing.io, has lead to a number of open source libraries for tracing.

We could take their https://github.com/opentracing/opentracing-javascript and use that with Gatsby. Track every time a source/transformer plugin is called, how long it takes, if there's any errors, etc. Put that all in the data layer so it can be queried and reports/etc be built on top of it. With GraphQL subscription support, we could also build logging from it.

Lemme know if you want to tackle this :-) if no one else gets to it, I'll probably get around to building it soonish as it'd help a lot with debugging and optimizing things.

[KyleAMathews]

SImilar to the dev 404 page, we could have a dev-only 404 page with debugging info drawn from this.

[KyleAMathews]

Ooo and this opentracing open source project looks perfect http://jaeger.readthedocs.io/en/latest/

They have an all-in-one Docker image you can download and run. Would be a really powerful understanding/debugging for Gatsby. Both in working on core, building plugins, and general site development.

So what we'd want to do is add hooks for opentracing in core and then people can add plugins for different compatible services.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[tiffon]

I'd be curious about contributing to this ticket.

I'd like to better understand the goals of the instrumentation, which is to say how granular it should be and if there are hot-spots that are priorities. And, from there, what common elements of Gatsby can be instrumented to achieve that. For instance, instrumenting the plugin APIs will get a baseline level of visibility into all plugins. Then we'll know what remaining functionality should be instrumented by hand and to what extent (or if) OpenTracing APIs should be exposed / proliferated for things like enabling plugins to add tags to spans, etc.

Sound ok? LMK if a different approach makes sense.

[Moocar]

@tiffon Yep, I think this level of granularity makes a lot of sense. We can start by instrumenting bootstrap/build phases, then add api/plugin calls, and then inject the parent span into the plugin's args or pluginOptions. Then, as you mentioned, plugin authors can add tags, new spans etc.

I just did a whole bunch of research on the state of tracing, and chatted with @KyleAMathews about it. Here's the plan:

Instrument Gatsby with opentracing-javascript. Also add the following open tracing implementation libraries to packages.json for gatsby core:

• zipkin-javascript-opentracing (do some due diligence on this library first)
• jaeger-client-node

Add command-line options to gatsby-cli so that we can turn on tracing at build time. This is also where we provide library specific configuration. Something like:

gatsby --tracer=zipkin --zipkin-port=94103 --zipkin-endpoint=https://url build

In Gatsby core, depending on the tracer specified (either zipkin or jaeger), require the appropriate library and use the other command line options to configure it. We can add other open tracing implementation libraries as required.

[Moocar]

I instrumented the codebase with open tracing and plugged in zipkin-javascript-opentracing and was able to get spans appearing in my trial honeycomb account (via the honeycomb-opentracing-proxy). Unfortunately, the zipkin library has some quirks.

• Span.setTags should allow arbitrary key/value pairs (as per open tracing docs), but this library enforces that tags must be in the set of these tags). The result is that we can't set tags such as api = onCreateNode, plugin = transformer-sharp.
• Span.addTags does not exist, yet it is defined in the open tracing spec. There is an open issue for this, though I disagree with the answer.

I'll start asking creating some issues with on the github project.

[tiffon]

One thing to note, briefly: I don't think Zipkin is interested in being compatible with OpenTracing: opentracing/opentracing.io#258

…

On Thu, Jun 28, 2018, 6:56 PM Anthony Marcar ***@***.***> wrote: I instrumented the codebase with open tracing and plugged in zipkin-javascript-opentracing <https://github.com/DanielMSchmidt/zipkin-javascript-opentracing> and was able to get spans appearing in my trial honeycomb account (via the honeycomb-opentracing-proxy <https://github.com/honeycombio/honeycomb-opentracing-proxy>). Unfortunately, the zipkin library has some quirks. - Span.setTags should allow arbitrary key/value pairs (as per open tracing docs <https://opentracing-javascript.surge.sh/classes/span.html#addtags>), but this library enforces that tags must be in the set of these tags <https://github.com/opentracing/opentracing-javascript/blob/master/src/ext/tags.ts>). The result is that we can't set tags such as api = onCreateNode, plugin = transformer-sharp. - Span.addTags does not exist, yet it is defined in the open tracing spec. There is an open issue <DanielMSchmidt/zipkin-javascript-opentracing#61> for this, though I disagree with the answer. I'll start asking creating some issues with on the github project. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1074 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACMpUdWzFR-7Pobeas-vfjJIAahyWEXYks5uBV8IgaJpZM4Nr93N> .

[Moocar]

Ahh, thanks for that info. That clarifies why I see no mention of open tracing in Zipkin. Interestingly, Honeycomb claim they support open tracing via their honeycomb-opentracing-proxy. But with Zipkin moving away from open tracing, this doesn't make any sense.

I was really hoping to be able to use the open tracing API so that Gatsby users could choose their own backend, but it's increasingly looking like we'll have to either

1. settle on one tracing implementation
2. Create our own tracing library that provides a lowest common denominator API. Then we would provide implementations inside Gatsby to as many tracing backends as required
3. Hack our own open tracing implementation and figure out how to fit it to existing backends.

I think 1) is too simplistic. Plus, I'm not sure which backend we'd choose. 2) would certainly work, but I feel bad introducing yet another tracing API. 3) to me seems like the best of both worlds. But it really depends how far Zipkin and other tracing implementations will diverge from the open tracing spec.

[KyleAMathews]

Looks like Jaeger accepts Zipkin. It says Zipkin doesn't support "advanced" features but it's unclear if that'd affect us https://github.com/jaegertracing/jaeger#backwards-compatibility-with-zipkin

[Moocar]

Nice find. It's clear there's no perfect answer here. One thing to keep in mind is that one of the benefits of the open tracing API is the ability for lots of different services to all take part in a distributed trace. Since they can work off the same datastructures over the wire. But Gatsby isn't a services based distributed system. It's just a build tool that runs periodically. There are services involved, but they're 3rd party services (Contentful etc) that wouldn't take part in a distributed trace anyway.

Bearing that in mind, if we did move to zipkin-js, we would enable Gatsby users to view their traces in Zipkin, Jaeger (via backwards compatibility), and Honeycomb (via the proxy). Those are 3 big platforms.

That will allow us to wait and see if open tracing stablizes and more companies get on board, at which point we can migrate over. It's worth noting that if we expose zipkin spans to plugin options, and plugin authors start extending tracing in their library, then the migration will be more painful. I recommend we either limit tracing to core Gatsby, or mark plugin specific tracing as experimental.

[Moocar]

I added some basic tracing using zipkin-js and confirmed that it is compatible with zipkin backend, jaeger, and honeycomb. And it supports tags and parent spans. I'm calling that a win. I'll start implementing this properly

[tiffon]

@Moocar, awesome! I was too slow man... Looking forward to giving the instrumentation a gander... I work on the front-end for Jaeger but know pretty much nil about instrumenting.

FYI, a comparison between Jaeger and Zipkin.

Also, re

I recommend we either limit tracing to core Gatsby, or mark plugin specific tracing as experimental.

Would it make sense to expose a reduced set of functionality which is the common-denominator between zipkin and OpenTracing to plugins? (Assuming it would still be a useful set of functionality.) It could still be marked experimental.

Side note: This may be relevant in the future... W3C Distributed Trace Context WG

[Moocar]

@tiffon Awesome to have someone with an actual tracing background here. If I knew that before I probably would've just let you implement the thing. It's doing my head in 😣.

I just created an issue on honeycomb to get an official answer to the open tracing vs zipkin thing from their point of view.

Meanwhile, I'm having issues instrumenting child spans that exist within promises using the zipkin-js tracer.local function. I think it's related to the way that zipkin handles context:

This package is not suitable if your code inside the context uses promises. The context is then not properly propagated. There is work underway called async_hooks, but is at the time of this writing (node v10) in Experimental state.

Which means we'll have to use ExplicitContext, which i'm learning about now. I can definitely say that the open tracing API is easier to use that the zipkin API for local (non remote call) tracing.

[Moocar]

Update: In yet another backflip, I've decided to stop using zipkin directly and instrument the API using open tracing and use zipkin-javascript-opentracing. I fixed one of their bugs which wasn't allowing setting arbitrary span tags and it's working great now.

Next step is to create the cli interface and docs.

[calcsam]

Fixed with #6347, thanks @Moocar!