multitenant-fleet

GKE Multitenant Blueprint

This blueprint presents an opinionated architecture to handle multiple homogeneous GKE clusters. The general idea behind this blueprint is to deploy a single project hosting multiple clusters leveraging several useful GKE features.

The pattern used in this design is useful, for blueprint, in cases where multiple clusters host/support the same workloads, such as in the case of a multi-regional deployment. Furthermore, combined with Anthos Config Sync and proper RBAC, this architecture can be used to host multiple tenants (e.g. teams, applications) sharing the clusters.

This blueprint is used as part of the FAST GKE stage but it can also be used independently if desired.

The overall architecture is based on the following design decisions:

• All clusters are assumed to be private, therefore only VPC-native clusters are supported.
• Logging and monitoring configured to use Cloud Operations for system components and user workloads.
• GKE metering enabled by default and stored in a bigquery dataset created within the project.
• Optional GKE Fleet support with the possibility to enable any of the following features:
  • Fleet workload identity
  • Anthos Config Management
  • Anthos Service Mesh
  • Anthos Identity Service
  • Multi-cluster services
  • Multi-cluster ingress.
• Support for Config Sync, Hierarchy Controller, and Policy Controller when using Anthos Config Management.
• Groups for GKE can be enabled to facilitate the creation of flexible RBAC policies referencing group principals.
• Support for application layer secret encryption.
• Support to customize peering configuration of the control plane VPC (e.g. to import/export routes to the peered network)
• Some features are enabled by default in all clusters:
  • Intranode visibility
  • Dataplane v2
  • Shielded GKE nodes
  • Workload identity
  • Node local DNS cache
  • Use of the GCE persistent disk CSI driver
  • Node auto-upgrade and auto-repair for all node pools

Basic usage

The following example shows how to deploy two clusters and one node pool for each

locals {
  cluster_defaults = {
    private_cluster_config = {
      enable_private_endpoint = true
      master_global_access    = true
    }
  }
  subnet_self_links = {
    ew1 = "projects/prj-host/regions/europe-west1/subnetworks/gke-0"
    ew3 = "projects/prj-host/regions/europe-west3/subnetworks/gke-0"
  }
}

module "gke-fleet" {
  source             = "./fabric/blueprints/gke/multitenant-fleet/"
  project_id         = var.project_id
  billing_account_id = var.billing_account_id
  folder_id          = var.folder_id
  prefix             = "myprefix"
  iam_by_principals = {
    "group:[email protected]" = [
      "roles/container.admin"
    ]
  }
  iam = {
    "roles/container.clusterAdmin" = [
      "serviceAccount:[email protected]"
    ]
  }
  clusters = {
    cluster-0 = {
      location               = "europe-west1"
      private_cluster_config = local.cluster_defaults.private_cluster_config
      vpc_config = {
        subnetwork             = local.subnet_self_links.ew1
        master_ipv4_cidr_block = "172.16.10.0/28"
      }
    }
    cluster-1 = {
      location               = "europe-west3"
      private_cluster_config = local.cluster_defaults.private_cluster_config
      vpc_config = {
        subnetwork             = local.subnet_self_links.ew3
        master_ipv4_cidr_block = "172.16.20.0/28"
      }
    }
  }
  nodepools = {
    cluster-0 = {
      nodepool-0 = {
        node_config = {
          disk_type    = "pd-balanced"
          machine_type = "n2-standard-4"
          spot         = true
        }
      }
    }
    cluster-1 = {
      nodepool-0 = {
        node_config = {
          disk_type    = "pd-balanced"
          machine_type = "n2-standard-4"
        }
      }
    }
  }
  vpc_config = {
    host_project_id = "my-host-project-id"
    vpc_self_link   = "projects/prj-host/global/networks/prod-0"
  }
}
# tftest modules=8 resources=46

GKE Fleet

This example deploys two clusters and configures several GKE Fleet features:

• Enables multi-cluster ingress and sets the configuration cluster to be cluster-eu1.
• Enables Multi-cluster services and assigns the required roles to its service accounts.
• A default Config Management template is created with binary authorization, config sync enabled with a git repository, hierarchy controller, and policy controller.
• The two clusters are configured to use the default Config Management template.

locals {
  subnet_self_links = {
    ew1 = "projects/prj-host/regions/europe-west1/subnetworks/gke-0"
    ew3 = "projects/prj-host/regions/europe-west3/subnetworks/gke-0"
  }
}

module "gke" {
  source             = "./fabric/blueprints/gke/multitenant-fleet/"
  project_id         = var.project_id
  billing_account_id = var.billing_account_id
  folder_id          = var.folder_id
  prefix             = "myprefix"
  clusters = {
    cluster-0 = {
      location = "europe-west1"
      vpc_config = {
        subnetwork = local.subnet_self_links.ew1
      }
    }
    cluster-1 = {
      location = "europe-west3"
      vpc_config = {
        subnetwork = local.subnet_self_links.ew3
      }
    }
  }
  nodepools = {
    cluster-0 = {
      nodepool-0 = {
        node_config = {
          disk_type    = "pd-balanced"
          machine_type = "n2-standard-4"
          spot         = true
        }
      }
    }
    cluster-1 = {
      nodepool-0 = {
        node_config = {
          disk_type    = "pd-balanced"
          machine_type = "n2-standard-4"
        }
      }
    }
  }
  fleet_features = {
    configmanagement             = true
    identityservice              = true
    multiclusteringress          = "cluster-0"
    multiclusterservicediscovery = true
    servicemesh                  = true
  }
  fleet_workload_identity = true
  fleet_configmanagement_templates = {
    default = {
      binauthz = true
      config_sync = {
        git = {
          policy_dir    = "configsync"
          secret_type   = "none"
          source_format = "hierarchy"
          sync_branch   = "main"
          sync_repo     = "https://github.com/myorg/myrepo"
        }
        prevent_drift = true
        source_format = "hierarchy"
      }
      hierarchy_controller = {
        enable_hierarchical_resource_quota = true
        enable_pod_tree_labels             = true
      }
      policy_controller = {
        audit_interval_seconds     = 30
        exemptable_namespaces      = ["kube-system"]
        log_denies_enabled         = true
        referential_rules_enabled  = true
        template_library_installed = true
      }
      version = "1.10.2"
    }
  }
  fleet_configmanagement_clusters = {
    default = ["cluster-0", "cluster-1"]
  }
  vpc_config = {
    host_project_id = "my-host-project-id"
    vpc_self_link   = "projects/prj-host/global/networks/prod-0"
  }
}
# tftest modules=9 resources=57

Files

name	description	modules
gke-clusters.tf	GKE clusters.	gke-cluster-standard
gke-hub.tf	GKE hub configuration.	gke-hub
gke-nodepools.tf	GKE nodepools.	gke-nodepool
main.tf	Project and usage dataset.	bigquery-dataset · iam-service-account · project
outputs.tf	Output variables.
variables.tf	Module variables.

Variables

name	description	type	required	default
billing_account_id	Billing account ID.	string	✓
folder_id	Folder used for the GKE project in folders/nnnnnnnnnnn format.	string	✓
prefix	Prefix used for resource names.	string	✓
project_id	ID of the project that will contain all the clusters.	string	✓
vpc_config	Shared VPC project and VPC details.	object({…})	✓
clusters	Clusters configuration. Refer to the gke-cluster module for type details.	map(object({…}))		{}
deletion_protection	Prevent Terraform from destroying data storage resources (storage buckets, GKE clusters, CloudSQL instances) in this blueprint. When this field is set in Terraform state, a terraform destroy or terraform apply that would delete data storage resources will fail.	bool		false
fleet_configmanagement_clusters	Config management features enabled on specific sets of member clusters, in config name => [cluster name] format.	map(list(string))		{}
fleet_configmanagement_templates	Sets of config management configurations that can be applied to member clusters, in config name => {options} format.	map(any)		{}
fleet_features	Enable and configure fleet features. Set to null to disable GKE Hub if fleet workload identity is not used.	object({…})		null
fleet_workload_identity	Use Fleet Workload Identity for clusters. Enables GKE Hub if set to true.	bool		false
iam	Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format.	map(list(string))		{}
iam_by_principals	Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the iam variable.	map(list(string))		{}
labels	Project-level labels.	map(string)		{}
nodepools	Nodepools configuration. Refer to the gke-nodepool module for type details.	map(map(object({…})))		{}
project_services	Additional project services to enable.	list(string)		[]

Outputs

name	description	sensitive
cluster_ids	Cluster ids.
clusters	Cluster resources.
project_id	GKE project id.