Error 403: Request had insufficient authentication scopes.

[dsiebel]

Preface

This might be more of a "request for comment/clarification" than an actual bug.
What we are currently observing is the following:

We can plan and apply changes just fine from a GKE cluster with a Service Account using Workload Identity, but when we try to run import/plan/apply locally, using our personal users it fails with the error described below.
The Service Account has role Groups Admin, while the personal accounts we were testing with have SuperAdmin role in Google Workspace.

We were already in contact with Google (Workspace) Support several times and got nowhere.
They are not officially supporting Terraform and always suggest to "reach out to terraform".

So here it goes:

Terraform Version

1.1.5

Affected Resource(s)

• googleworkspace_group

subsequently:

• googleworkspace_group_settings
• googleworkspace_group_members

Terraform Configuration Files

provider "googleworkspace" {
  customer_id = "XXXXXXX"
  // using all the scopes out of pure desperation
  oauth_scopes = [
    "https://www.googleapis.com/auth/apps.groups.settings",
    "https://www.googleapis.com/auth/chrome.management.policy",
    "https://www.googleapis.com/auth/admin.directory.user",
    "https://www.googleapis.com/auth/cloud-platform",
	"https://www.googleapis.com/auth/iam",
    "https://www.googleapis.com/auth/admin.directory.userschema",
    "https://www.googleapis.com/auth/gmail.settings.sharing",
    "https://www.googleapis.com/auth/gmail.settings.basic",
    "https://www.googleapis.com/auth/admin.directory.customer",
    "https://www.googleapis.com/auth/admin.directory.domain",
    "https://www.googleapis.com/auth/admin.directory.group",
    "https://www.googleapis.com/auth/admin.directory.orgunit",
    "https://www.googleapis.com/auth/admin.directory.rolemanagement",
  ]
}

resource "googleworkspace_group" "group" {
  description = "test group"
  email       = "[email protected]"
}

Debug Output

https://gist.github.com/dsiebel/5950f119bff40c11d9c27f737c964fb8

Panic Output

N/A

Expected Behavior

A new group to be created in Google Workspace.

Actual Behavior

terraform plan shows the group will be created. On terraform apply the API response is:

Error 403: Request had insufficient authentication scopes.

Also when running plan after a previous apply (see preface), the plan already fails when trying to read an existing group.

Steps to Reproduce

terraform apply

Important Factoids

We are running terraform via Atlantis in a GKE cluster which uses a Service Account via Workload Identity to apply the changes. This Service Account has "Groups Admin" role in Google Workspace.
This work perfectly fine, however when trying to plan/apply locally, using our local users (SuperAdmin on Google Workspace), the aforementioned error occurs.

References

N/A

[megan07]

Hi @dsiebel ! I'm sorry that you're running into this issue, I'd like to help out here as best I can. It seems to me it might be a scopes issue. So, the scopes that you grant to your service account needs to match, or be a superset, of the oauth_scopes granted to the provider. So, I see you listed all of those scopes out of desperation - but that may either be part of the issue, or hiding a different issue, so what I first want to have you do is see if you can limit the oauth_scopes to just "https://www.googleapis.com/auth/admin.directory.group" and "https://www.googleapis.com/auth/cloud-platform" and double-check in the admin console that the Oauth Scopes granted to the service account match those same scopes, or that there are more in the admin console. Does that make sense? See here.

The other thing I want to check is that I see the service account being impersonated via workload identity is working correctly, are you using credentials from a different service account locally?

[dsiebel]

Hey @megan07, thanks for taking the time and getting back to me!
The original set of oauth scopes (before desperation mode) was

oauth_scopes = [
    "https://www.googleapis.com/auth/admin.directory.user",
    "https://www.googleapis.com/auth/admin.directory.group",
    "https://www.googleapis.com/auth/apps.groups.settings",
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/iam",
  ]

and neither this nor configuring the two proposed scopes works. The error remains the same.
The link you included links to Domain Wide Delegation which we are not using anymore.
We were previously using it, including setting impersonated_user_email and credentials to the JSON key of the service account. This is not required anymore and we opted for assigning Groups Admin to the service account directly.

It works perfectly fine with the service account, but it does not work with my personalized user, relying on application default credentials.

[megan07]

@dsiebel ahh! ok! Did you use the --scopes flag to specify the scopes when you logged in?
https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login

I think you'll want to list the same scopes there. Let me know if that helps! Thanks!

[dsiebel]

When I tried Google Login was telling me that the "App is blocked", so, naturally, I assumed that this was something I wasn't supposed to do.
Turns out we needed to trust the "Google Auth Library" / gcloud SDK to access all Google Services and I was able to expand the scopes on my ADC login.

Thanks a bunch @megan07 ! 🙂

[megan07]

Thanks for the extra information @dsiebel ! I'm happy it's working!

[gmile]

Turns out we needed to trust the "Google Auth Library" / gcloud SDK to access all Google Services and I was able to expand the scopes on my ADC login.

@dsiebel would you be so kind as to share how exactly you enable "trust" to Google Auth Library / gcloud SDK applications?

I'm failing to generate application default credentials using gcloud. After running this, I am basically hitting an error when trying to consent to authorise the app in a browser:

gcloud auth application-default login --scopes openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/admin.directory.user,https://www.googleapis.com/auth/admin.directory.group,https://www.googleapis.com/auth/apps.groups.settings,https://www.googleapis.com/auth/iam

...after attempting to consent in browser I am getting:

This app is blocked. This app tried to access sensitive info in your Google Account. To keep your account safe, Google blocked this access.

Upd. Found it. One should go to https://admin.google.com/ac/owl/list?tab=configuredApps, and whitelist the app there using gcloud's client_id, the value for client_id can be seen in the https://accounts.google.com/o/oauth2/auth?... URL during OAuth2 consent.

[dsiebel]

Upd. Found it. One should go to admin.google.com/ac/owl/list?tab=configuredApps, and whitelist the app there using gcloud's client_id, the value for client_id can be seen in the https://accounts.google.com/o/oauth2/auth?... URL during OAuth2 consent.

@gmile Did you manage to solve it with that?
I ran into some other issues but didn't have the time yet to further look into it:

googleworkspace_group.group: Creating...
╷
│ Error: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the admin.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/764086051850",
│       "service": "admin.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with googleworkspace_group.group,
│   on main.tf line 43, in resource "googleworkspace_group" "group":
│   43: resource "googleworkspace_group" "group" {
│ 

The project number mentioned is none of ours though.

UPDATE I am guessing that's what #264 is about?

[dsiebel]

I found this medium article researching the project number. I have been meaning to give it a try.

[dsiebel]

I managed to get it working using service account impersonation all the way.
Using one google provider to get the access token and configure another google provider and the googleworkspace provider via access_token argument:

provider "google" {
  alias   = "sa_auth"
  project = local.service_project
}

data "google_service_account_access_token" "impersonatee" {
  provider = google.sa_auth
  target_service_account = local.service_account
  scopes = [
    "https://www.googleapis.com/auth/admin.directory.user",
    "https://www.googleapis.com/auth/admin.directory.group",
    "https://www.googleapis.com/auth/apps.groups.settings",
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/iam",
  ]
}

provider "googleworkspace" {
  customer_id     = "<redacted>"
  access_token    = data.google_service_account_access_token.impersonatee.access_token
  service_account = local.service_account
  oauth_scopes = [
    "https://www.googleapis.com/auth/admin.directory.user",
    "https://www.googleapis.com/auth/admin.directory.group",
    "https://www.googleapis.com/auth/apps.groups.settings",
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/iam",
  ]
}

[...]

Important to note, that the access scopes need to match, as also mentioned in this issue and it obviously requires you to have roles/iam.serviceAccountTokenCreator on local.service_account.

[gmile]

I am guessing that's what #264 is about?

@dsiebel precisely, yes.

I managed to get it working using service account impersonation all the way.

It appears that service account's access token is being part of Terraform state? I was looking for ways to specifically avoid storing any service account credentials as part of .tf state 🤔

For context, we are a small team, each of us runs terraform locally on our computers. Ideally, we don't want to infiltrate the .tf state with the access tokens that allow controlling users / groups at the organisation level, as I find it to be quite risky. Ideally, in our situation, all control would be done using local developer credentials, found on the computer.

[gabor-farkas]

@gmile it's also possible to specify impersonation with an environment variable, https://cloud.google.com/blog/topics/developers-practitioners/using-google-cloud-service-account-impersonation-your-terraform-code - afaik that approach avoids storing tokens in the state.

(Otherwise, the Terraform state storage can be set up to be accessible only by people who have 'generateAccessToken' privileges anyway, and the token otherwise has a short expiration.)

[dsiebel]

Sorry, for the delay. My answers below.

It appears that service account's access token is being part of Terraform state? I was looking for ways to specifically avoid storing any service account credentials as part of .tf state 🤔

No it is not. The access token is retrieved using a Terraform datasource. That is why the user interacting with the terraform state need to have roles/iam.serviceAccountTokenCreator on said service account.

EDIT: now that I wrote that I am suddently not sure whether the value of the data source will be persisted in the remote state (GCS) or not 🤔