Translate between x-cloud-trace-context and grpc-trace-bin for http/grpc transcoding

[qiwzhang]

For tracing: HTTP uses x-cloud-trace-context, but grpc uses grpc-trace-bin

In http/grpc transcoding, ESP should translate from x-cloud-trace-context to grpc-trace-bin in request, and from grpc-trace-bin to x-cloud-trace-context in response.

[qiwzhang]

1. The actual translation should be done here
   here is for request direction, and here is for response direction.

2. here can be optimized to set the proper context type for request direction so they don't need to be translated again.

backend_protocol can be found by calling request_->GetBackendProtocol();

[humphrej]

I created a partial workaround for this in my java/grpc downstream by doing the x-cloud-trace-context to grpc-trace-bin conversion in a custom ServerStreamTracer implementation. This allows me to link subsequent downstream spans with the trace created by ESP, but I am unable to intercept the call early enough to prevent opencensus from creating a second trace id for the sent/ received stats. Summary - it would be super if this translation happened in the ESP !

[farmdawgnation]

Instead of doing a translation that replaces the X-Cloud-Trace-Context header could this instead add the binary header in addition to the X-Cloud-Trace-Context when sending requests to the backend? This would prevent breaking existing code that is perhaps relying on the existing header to be there.

[qiwzhang]

My understanding is: if backend is grpc, it should use grpc-trace-bin. If it is http, it will use X-Cloud-Trace-Context. It doesn't make sense to send both.

[farmdawgnation]

I agree in the general. However, ESP currently has been sending X-Cloud-Trace-Context. If anyone built production software on that functionality it would be good to have a few releases where you send both when you implement sending grpc-trace-bin to give folks a chance to migrate to the new behavior.

[farmdawgnation]

@qiwzhang Any chance y'all have an ETA on this? I ran into a case where I will need to switch to using grpc-trace-bin on my end 😭

[qiwzhang]

We don't have ETA for ESPv1 which is from this repo.

Is it possible for you to use ESPv2? It is here. It allows you to do by using this flag

It is available for Serverless platforms (Cloud Run, Cloud Function, and AppEngine), it is also ready for GKE and GCE, but the public doc is not updated yet.

[farmdawgnation]

Would it work on a non-GKE Kubernetes cluster? If so, I'm game to give it a shot. We're not fully on GKE yet but making our way there.

[qiwzhang]

Yes, same as ESPv1, if not in GCP, we could not use its metadata server to get access token, you need to pass a service_account key file to ESP to generate access token to talk to Google services.