Service Account based authentication for Provider Vault

[mustafaStakater]

What problem are you facing?

I want to configure crossplane provider vault to authenticate to vault using service account based token. Since crossplane runs on k8s clusters, it makes sense to implement it.

How could Upbound help solve your problem?

Add service account authentication for provider vault. It should look something like:

apiVersion: vault.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: provider-vault
spec:
  address: 'https://vault.com'
  credentials:
    source: ServiceAccount
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-vault
spec:
  package: 'xpkg.upbound.io/upbound/provider-vault:v0.1.0'
  controllerConfigRef:
    name: vault-controller

---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: vault-controller
spec:
  podSecurityContext: {}
  replicas: 1
  serviceAccountName: provider-vault
  securityContext: {}

Considering provider-vault service account has the appropriate permissions configured in vault.

Not Sure about how this would be implemented, it requires the kubernetes cluster configured as an OIDC/JWT provider in vault (https://developer.hashicorp.com/vault/docs/auth/jwt/oidc-providers/kubernetes). Provider uses service account as jwt token and refreshes when it expires( in 1hr ).

But maybe there might be a better way to go about implementing this.

[turkenh]

But maybe there might be a better way to go about implementing this.

I believe this should be implemented per by using the Kubernetes as the auth method instead of JWT/OIDC as the auth method and Kubernetes as a provider, i.e. https://developer.hashicorp.com/vault/docs/auth/kubernetes

[mwienk]

+1 for this feature, although I think the source should be set to InjectedIdentity as per https://github.com/crossplane/crossplane-runtime/blob/b53745680067a8c96b0258e174d7212179c1c547/apis/common/v1/resource.go#L248C47-L248C47

[ronit87]

+1 for this feature.
As we are already creating ServiceAccount with Vault Helm chart deployment and if we can use the same to configure ProviderConfig as well.
Is this feature planned in new release? I am waiting for this feature..

[ronit87]

Is there an update on this feature request??

[headyj]

@ronit87 actually it would already work if we could find a way to inject the service account token in the secret, which I didn't manage until now:

---
apiVersion: v1
kind: Secret
metadata:
  name: vault-provider-secret
  annotations:
    kubernetes.io/service-account.name: vault-provider
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-creds
type: Opaque
stringData:
  credentials: |
    {
    "auth_login_jwt": {
        "mount": "<vault mound name>",
        "role": "<vault role name>",
        "jwt": "<token value from service account secret above>"
        }
     }

I tested by manually copy-pasting the value of the service account token in jwt field and it's working. Then I tried to use something like file(\"/var/run/secrets/kubernetes.io/serviceaccount/token\")" (that's how you would do in the terraform provider from which this provider is derived) but I can say that the value is not injected, as I have this error from vault:

2023-12-20T10:02:33.671Z [DEBUG] auth.kubernetes.auth_kubernetes_1bf45bee: login unauthorized: err="error verifying token signature: square/go-jose: compact JWS format must have three parts"

[headyj]

I think I found a way to make it work using TERRAFORM_VAULT_AUTH_JWT variable:

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: vault-config
  labels:
    app: crossplane-provider-vault
spec:
  serviceAccountName: vault-provider
  env:
    - name: TERRAFORM_VAULT_AUTH_JWT
      valueFrom:
        secretKeyRef:
          name: vault-provider-secret
          key: token
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-provider-secret
  annotations:
    kubernetes.io/service-account.name: vault-provider
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-creds
type: Opaque
stringData:
  credentials: |
    {
    "auth_login_jwt": {
        "mount": "<vault mound name>",
        "role": "<vault role name>"
        }
     }

[ronit87]

Hi @headyj ,

Thanks for sharing this solution. It is helpful at some extent.

However, the challenge here is to create and use long lived ServiceAccount token that represent a higher risk to information security.

Therefore, if we find some solution to use ServiceAccount in ControllerConfig/DeploymentRuntimeConfig itself and Crossplane Vault Provider manage this token with expiration time and auto-rotate on expiration then it will be good feature to have as per Kubernetes security best practices.

[zebesh]

i find solution for this problem. Try this:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-vault
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        metadata:
          annotations:
            linkerd.io/inject: enabled
            vault.hashicorp.com/agent-init-first: 'true'
            vault.hashicorp.com/agent-inject: 'true'
            vault.hashicorp.com/agent-inject-token: 'true'
            vault.hashicorp.com/agent-limits-cpu: '0.05'
            vault.hashicorp.com/agent-pre-populate: 'false'
            vault.hashicorp.com/agent-pre-populate-only: 'false'
            vault.hashicorp.com/agent-requests-cpu: '0.01'
            vault.hashicorp.com/agent-run-as-user: '2000'
            vault.hashicorp.com/log-format: json
            vault.hashicorp.com/role: "crossplane"
            vault.hashicorp.com/agent-inject-secret-vault-token: "auth/token/lookup-self"
            vault.hashicorp.com/agent-inject-template-vault-token: |
              {{- with secret "auth/token/lookup-self" -}}
                {
                  "token_name": "crossplane-vault-creds",
                  "token": "{{ .Data.id }}"
                }
              {{- end }}         
  serviceAccountTemplate:
    metadata:
      name: crossplane-vault

[davimmt]

+1

I think it should definitely be an automatic behaviour when using AWS IAM Roles and source: InjectedIdentity.

Based on @zebesh snippet, but with a little more detail:

First, I had to install Vault Agent Injector to work with sidecar container injections:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm upgrade --install vault hashicorp/vault --set injector.enabled=true,server.enabled=false

Then, configure the DeploymentRuntimeConfig, Provider and ProviderConfig:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: vault
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        metadata:
          annotations:
            vault.hashicorp.com/agent-init-first: 'true'
            vault.hashicorp.com/agent-inject: 'true'
            vault.hashicorp.com/agent-inject-token: 'true'
            vault.hashicorp.com/agent-limits-cpu: '0.05'
            vault.hashicorp.com/agent-pre-populate: 'false'
            vault.hashicorp.com/agent-inject-status: "update"
            vault.hashicorp.com/agent-pre-populate-only: 'false'
            vault.hashicorp.com/agent-requests-cpu: '0.01'
            vault.hashicorp.com/agent-run-as-user: '2000'
            vault.hashicorp.com/service: <VAULT_ADDR>
            vault.hashicorp.com/namespace: <VAULT_NAMESPACE>
            vault.hashicorp.com/auth-type: aws
            vault.hashicorp.com/auth-path: auth/aws
            vault.hashicorp.com/auth-config-type: iam
            vault.hashicorp.com/auth-config-region: sa-east-1
            vault.hashicorp.com/role: terraform
            vault.hashicorp.com/agent-inject-secret-vault-token: "auth/token/lookup-self"
            vault.hashicorp.com/agent-inject-template-vault-token: |
              {{- with secret "auth/token/lookup-self" -}}
                {
                  "token_name": "crossplane-vault-creds",
                  "token": "{{ .Data.id }}"
                }
              {{- end }}
  serviceAccountTemplate:
    metadata:
      name: crossplane
      annotations:
        eks.amazonaws.com/role-arn: <AWS_ROLE_ARN>
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-vault
spec:
  package: xpkg.upbound.io/upbound/provider-vault:v0.4.0
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1
    kind: DeploymentRuntimeConfig
    name: vault
---
apiVersion: vault.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  address: <VAULT_ADDR>
  add_address_to_env: true
  credentials:
    source: Filesystem
    fs:
      path: /vault/secrets/vault-token

Now it should work fine.

It seems to suffice Kubernetes security best practices as pointed out by @ronit87, since the sidecar container continuously refreshes/rotates the token.

[romanlehner]

@davimmt , I don't quite get all aspects of your config. I created a kind cluster with Vault and crossplane running on it. To setup vault agent to supply a vault token I got this here:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: vault-provider
spec:
  package: xpkg.upbound.io/upbound/provider-vault:v0.4.0
  runtimeConfigRef:
    name: vault
---
apiVersion: vault.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: vault-gitops
spec:
  address: http://vault.vault:8200
  add_address_to_env: true
  credentials:
    fs:
      path: /vault/secrets/token
    source: Filesystem
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: vault
spec:
  deploymentTemplate:
    spec:
      replicas: 1
      selector: {}
      template:
        metadata:
          annotations:
            vault.hashicorp.com/agent-init-first: "true"
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-inject-token: "true" #defaults to /vault/secrets/token
            vault.hashicorp.com/agent-run-as-group: "2000"
            vault.hashicorp.com/auth-path: auth/kubernetes
            vault.hashicorp.com/auth-type: kubernetes
            vault.hashicorp.com/service: "http://vault.vault:8200"
            vault.hashicorp.com/role: crossplane-providers
  serviceAccountTemplate:
    metadata:
      name: vault-provider
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: provider-vault
  namespace: crossplane-system

As far as I saw from the vault agent docs, it will pass the vault token to /vault/secrets/token and not /vault/secrets/vault-token. So I am not sure how your config actually works.

But I run into a different issue here where my managed resource can't be reconciled because the vault token cannot be parsed for some reason:

Warning CannotConnectToProvider 1s (x2 over 2s) managed/vault.vault.upbound.io/v1alpha1, kind=policy cannot get terraform setup: cannot unmarshal vault credentials as JSON: invalid character 'h' looking for beginning of value

The vault token indeed starts with h like hvs.xxxx, so I am just guessing this must be a parsing error. Probably happening around here: https://github.com/upbound/provider-vault/blob/main/internal/clients/vault.go#L120

[davimmt]

@romanlehner

The default secret's path mount is indeed /vault/secrets/token, however I'm rewriting this path in my annotation key.

You are getting this error because crossplane expects a JSON file just as configured also in my annotations, not the token string itself.

Both of these configurations are in this annotation:

          vault.hashicorp.com/agent-inject-template-vault-token: |
              {{- with secret "auth/token/lookup-self" -}}
                {
                  "token_name": "crossplane-vault-creds",
                  "token": "{{ .Data.id }}"
                }
              {{- end }}

[romanlehner]

@davimmt , ok now I also got it to work. Thanks a ton for this hint. Although I wished the provider could just work with the original injected token.

[excelsi0r]

@davimmt I am also trying to do something similar. In my case I have vault installed via bank-vaults operator and I am trying to instantiate secrets using crossplane. What was the configuration that made it work for you? I am not being able to configure it based on the examples here...

[davimmt]

@davimmt I am also trying to do something similar. In my case I have vault installed via bank-vaults operator and I am trying to instantiate secrets using crossplane. What was the configuration that made it work for you? I am not being able to configure it based on the examples here...

You should post your configuration files and any error logs you're encountering. That way we can better assist you.