Serverless Vault with Cloud Run

This tutorial walks you through deploying Hashicorp's Vault on Cloud Run, Google Cloud's container based Serverless compute platform.

Rational

Vault is a tool for encrypting data, managing secrets, and auditing access to them. Vault should be deployed to a secure and highly available environment to ensure applications have reliable access to secrets and credentials. Vault can leverage managed services such as Cloud KMS and Google Cloud Storage to protect and store its data, and Cloud Run to serve it and capture audit logs.

While Vault can be deployed to Kubernetes, or a virtual machine, Cloud Run reduces operational complexity by delegating infrastructure management to the cloud provider. Cloud Run can scale Vault to zero when not in use, or be configured to keep a single instance available to serve concurrent request at the lowest latency possible. Cloud Run automatically captures Vault's audit logs and sends them to Cloud Logging for centralized storage and analysis.

Cloud Run can also be configured to ensure only one instance of Vault is running at a given time. See the Cloud Run maximum instances docs.

Tutorial

Create a new Google Cloud project:

gcloud projects create \
  --name vault-on-cloud-run \
  --set-as-default

Type y at the prompt and press enter:

No project id provided.

Use [vault-on-cloud-run-XXXXXX] as project id (Y/n)?  y

Before you can continue you must enable billing on the project to enable the use of Cloud KMS as an auto-unseal mechanism.

To streamline the rest of the tutorial define the key configuration settings and assign them to environment variables:

PROJECT_ID=$(gcloud config get-value project)

PROJECT_ID holds the project id generated at the start of the tutorial.

GCS_BUCKET_NAME="${PROJECT_ID}-data"

GCS_BUCKET_NAME holds the Google Cloud Storage bucket name used to persist Vault's data.

SERVICE_ACCOUNT_EMAIL="vault-server@${PROJECT_ID}.iam.gserviceaccount.com"

SERVICE_ACCOUNT_EMAIL holds the Google Cloud IAM email address representing the vault-server service account.

CURRENT_USER_EMAIL=$(gcloud config list account --format "value(core.account)")

CURRENT_USER_EMAIL holds the email address representing the current logged in Google Cloud user.

REGION=us-west1

REGION holds the region in which to deploy the vault-server.

Enable the Cloud KMS, Cloud Run, Cloud Storage, and Secret Manager APIs:

gcloud services enable --async \
  cloudkms.googleapis.com \
  run.googleapis.com \
  secretmanager.googleapis.com \
  storage.googleapis.com

Create a service account for the Vault server:

gcloud iam service-accounts create vault-server

Create a GCS storage bucket to hold Vault's encrypted data:

gsutil mb gs://${GCS_BUCKET_NAME}

Creating gs://vault-on-cloud-run-XXXXXX-data/...

Grant the necessary permissions on GCS storage bucket for the vault-server service account:

gsutil iam ch \
  serviceAccount:${SERVICE_ACCOUNT_EMAIL}:objectAdmin \
  gs://${GCS_BUCKET_NAME}

Store the vault server config file in Secret Manager:

gcloud secrets create vault-server-config \
  --replication-policy automatic \
  --data-file vault-server.hcl

Grant access to the vault-server-config secret to the vault-server service account:

gcloud secrets add-iam-policy-binding vault-server-config \
  --member "serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role "roles/secretmanager.secretAccessor"

Create a Cloud KMS key ring that will be used to hold Vault seal encryption key:

gcloud kms keyrings create "vault-server" \
  --location "global"

Create the Cloud KMS encryption key that will be used to encrypt and decrypt the Vault master key:

gcloud kms keys create "seal" \
  --location "global" \
  --keyring "vault-server" \
  --purpose "encryption"

Grant the necessary permissions on the seal Cloud KMS encryption key for the vault-server service account:

gcloud kms keys add-iam-policy-binding seal \
  --keyring "vault-server" \
  --location "global" \
  --member serviceAccount:${SERVICE_ACCOUNT_EMAIL} \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Deploy the vault-server Cloud Run service:

The initial deployment will be made private to prevent someone else from initializing the Vault server. Clients will need to provide authentication credentials that will be validated by the Cloud Run control plane. After Vault has been initialized the Cloud Run deployment will be updated to bypass Cloud Run authentication and delegate the authentication to Vault.

gcloud beta run deploy vault-server \
  --no-allow-unauthenticated \
  --concurrency 50 \
  --cpu 2 \
  --image gcr.io/hightowerlabs/vault:1.7.1 \
  --no-cpu-throttling \
  --memory '2G' \
  --min-instances 1 \
  --max-instances 1 \
  --platform managed \
  --port 8200 \
  --service-account ${SERVICE_ACCOUNT_EMAIL} \
  --set-env-vars="GOOGLE_PROJECT=${PROJECT_ID},GOOGLE_STORAGE_BUCKET=${GCS_BUCKET_NAME}" \
  --set-secrets="/etc/vault/config.hcl=vault-server-config:1" \
  --timeout 300 \
  --region ${REGION}

Grant permission to the current logged in GCP user to invoke the vault-server Cloud Run service:

gcloud run services add-iam-policy-binding vault-server \
  --member="user:${CURRENT_USER_EMAIL}" \
  --role='roles/run.invoker' \
  --platform managed \
  --region ${REGION}

Retrieve the vault-server service URL:

VAULT_SERVICE_URL=$(gcloud run services describe vault-server \
  --platform managed \
  --region ${REGION} \
  --format 'value(status.url)')

Use curl to retrieve the status of the Vault server:

The gcloud command can be used to generate an identity token which can be used to authenticate to the private vault-server Cloud Run service.

curl -s -X GET \
  ${VAULT_SERVICE_URL}/v1/sys/seal-status \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)"

{
  "type": "gcpckms",
  "initialized": false,
  "sealed": true,
  "t": 0,
  "n": 0,
  "progress": 0,
  "nonce": "",
  "version": "1.7.1",
  "migration": false,
  "recovery_seal": true,
  "storage_type": "gcs"
}

From the output above you can see that the Vault server has not been initialized and is currently sealed. Use the curl command to initialize the Vault server:

cat <<EOF > init.json
{
  "recovery_shares": 1,
  "recovery_threshold": 1,
  "secret_shares": 1,
  "secret_threshold": 1,
  "stored_share": 1
}
EOF

curl -s -X PUT \
  ${VAULT_SERVICE_URL}/v1/sys/init \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  --data @init.json

{
  "keys": [],
  "keys_base64": [],
  "recovery_keys": [
    "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  ],
  "recovery_keys_base64": [
    "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  ],
  "root_token": "X.XXXXXXXXXXXXXXXXXXXXXXXX"
}

At this point the Vault server has been initialized. If you list the GCS storage bucket you will see a new set of directories created by Vault:

gsutil ls gs://${GCS_BUCKET_NAME}

gs://vault-on-cloud-run-XXXXXX-data/core/
gs://vault-on-cloud-run-XXXXXX-data/logical/
gs://vault-on-cloud-run-XXXXXX-data/sys/

Now that Vault has been initialized make the vault-server service public:

Requests to Vault will no longer be authenticated by the Cloud Run control plane. Requests will be authenticated directly by Vault.

gcloud beta run deploy vault-server \
  --allow-unauthenticated \
  --concurrency 50 \
  --cpu 2 \
  --image gcr.io/hightowerlabs/vault:1.7.1 \
  --no-cpu-throttling \
  --memory '2G' \
  --min-instances 1 \
  --max-instances 1 \
  --platform managed \
  --port 8200 \
  --service-account ${SERVICE_ACCOUNT_EMAIL} \
  --set-env-vars="GOOGLE_PROJECT=${PROJECT_ID},GOOGLE_STORAGE_BUCKET=${GCS_BUCKET_NAME}" \
  --set-secrets="/etc/vault/config.hcl=vault-server-config:1" \
  --timeout 300 \
  --region ${REGION}

At this point Vault is up and running and can be configured using the Vault UI by visiting the vault-server service URL in browser:

gcloud run services describe vault-server \
  --platform managed \
  --region ${REGION} \
  --format 'value(status.url)'

You can also use the vault command line tool as described in the next section.

Retrieve the Vault Server Status Using the Vault Client

Download the Vault binary and add it to your path:

vault version

Vault v1.7.1 (917142287996a005cb1ed9d96d00d06a0590e44e)

Configure the vault CLI to use the vault-server Cloud Run service URL by setting the VAULT_ADDR environment variable:

export VAULT_ADDR=$(gcloud run services describe vault-server \
  --platform managed \
  --region ${REGION} \
  --format 'value(status.url)')

Retrieve the status of the remote Vault server:

vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.7.1
Storage Type             gcs
Cluster Name             vault-cluster-XXXXXXXX
Cluster ID               XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
HA Enabled               false

Clean Up

gcloud projects delete ${PROJECT_ID}