Unable to use in conjunction with IAP in GAE Flex

[maroux]

We have a few app engine services which are fronted by IAP (works with Google user auth etc). We also want to deploy a few services which are fronted by ESP (works with Auth0). Because IAP needs to be enabled for the "App Engine app" and can't be selectively enabled for App Engine services, we have to enable IAP for services which are fronted by ESP and add allUsers permission in IAP for those services. Now - the problem is that ESP honors IAP header over Authorization header, and since IAP sets it's header regardless if you use allUsers permissions or not - this leads to ESP services not working even with correct Authorization header.

Note we're using app.yaml ESP configuration method, instead of Cloud Run.

Is there any solution for this situation? Please advise.

[qiwzhang]

Do you need to use ESP for JWT authentication? If not, you can specify Endpoint config that JWT auth is not required, ESP will not check JWT token.

If you need to use JWT authentication feature and IAP, yes, we have a problem, they don't work together. ESP auth always checks IAP header first, if there is a JWT token, just use it. That could be the wrong JWT to check.

One solution is to change ESP not to check IAP header for auth. But we are not sure if such change is safe for all users. Some users maybe using ESP to verify IAP token, such change will break them.

We could add a esp start_up flag to enable such change, but it is not easy for FLEX users to set ESP start_up flags.

Another solution is to enhance Endpoint service config to allow users to specify the locations to extract JWT token.

[maroux]

Hm - yes I was hoping to use ESP for JWT authentication and avoid IAP messing it up since I don't have an option to disable IAP selectively. What's the use case for ESP respecting IAP headers?

[qiwzhang]

Some users use ESP to verify IAP signed header, so their backend don't need to verify it.

[qiwzhang]

We are working on an approach to allow users to specify the JWT extracting location in OpenAPI spec.
With this, you can specify where to extract it, you can exclude IAP signed header in your service config.

[maroux]

Okay that makes sense. Is there a timeline for when open api spec would support this?

[qiwzhang]

@maroux I am not familiar with how IAP works in GAE. I have a question, if you have IAP enabled for a GAE service, how do you pass auth0 JWT header to it (or its ESP)? JWT token passed by Authorization header will be consumed by IAP in GFE. It will reject the request if it is not intended for IAP. The auth0 JWT will never reach GAE. am I correct?

[maroux]

That was my concern as well. But as it turns out, IAP passes through the Authorization header if you don't use it for authentication (i.e. allow all users). Odd that it still tacks on IAP header though.

[maroux]

Hi @qiwzhang is there an update on this? Can we specify JWT extracting location in OpenAPI spec now?

[qiwzhang]

Yes, you can specify x-google-jwt-locations

But it is not implemented in ESPv1, only implemented in ESPv2.
For supporting services deploying in serverless platforms, such as Cloud Run, Cloud Function, and AppEngine standard, we recommend deploying ESPv2 in Cloud Run.

[qiwzhang]

BTW, this doc is still using ESPv1, we will update it to use ESPv2, the instruction should be similar to the one for Cloud Run.

[maroux]

nice!

Follow-up: So to deploy ESP v2 in app engine flex - we'd have to deploy one service for ESP and one for the actual API, correct? Does ESPv2 not support managed config rollout?

(trying to avoid Cloud Run) Does this picture look correct?

(internet) ->
IAP fronting ESP -> allows all users ->
ESP in App Engine Flex -> validates Authorization header instead of IAP header ->
IAP fronting API service -> validates ESP SA
API service in App Engine Flex -> looks at X-Endpoint-API-UserInfo header to get end user info

[maroux]

There's no way to specify ESPv2 here correct?

[qiwzhang]

AppEngine flex has a build-in ESPv1 which is configured the app.yaml, in its endpoints_api_service section (as described here. Sorry, it doesn't support ESPv2.

My suggestion is to deploy ESPv2 on Cloud Run and use x-google-backend to point to your AppEngine service. It could be either AppEngine standard and flex. For Flex, you just don't specify endpoints_api_service to disable ESPv1.

But if you insists on using Flex endpoints_api_service feature, it is ESPv1. You depends on ESPv1 implementation of this feature.