DevOps

Documentation

Architecture

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Architecture

Landing Zone Install

Managed vs Unmanaged Landing Zones

Landing Zone Package Dependencies

These are the dependencies on the Landing Zone packages in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions. There are direct/deploy-time and indirect/runtime dependencies (required/optional). The following diagram is the direct dependences. For example the hub-env deployment needs the networking-sa service account defined in core-landing-zone. An example of a indirect optional package is the org-policies folder in any package - it is optional for development but recommended for production.

graph LR;
    style LZV2 fill:#44f,stroke:#f66,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
    %% mapped and documented
    project/hub-env-->core-landing-zone;
    client-project-->client-project-setup;
    client-setup;
    client-setup-->dns-project;
    client-setup-->kcc-management-project;
    client-landing-zone-->client-setup;
    client-project-setup-->client-landing-zone;
    client-project-setup-->client-management-project;
    gatekeeper-policies;

    kcc-management-project;
    core-landing-zone-->kcc-management-project;
    dns-project-->core-landing-zone;
    logging-project-->core-landing-zone;
    client-management-project-->client-setup;
    host-project-->client-landing-zone;

mermaid - diagrams as code

Full client package structure

Package: core-landing-zone

GKE object browser - view of log sinks - specific only to the core-landing-zone Screenshot 2024-02-16 at 11 52 35

Cluster with 4 packages clz + 4 client Screenshot 2024-02-16 at 11 56 05

Package: client-setup

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-client.md#add-client-setup-package

Package: client-landing-zone

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/onboarding-client.md#add-the-client-landing-zone-package

Note: PSC forwarding rule gcloud addition https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/823

Resource manager view for combined (core-landing-zone, client-setup and client-landing-zone)

Package: client-project-setup

Package: projects/hub-env

IaaS security - not serverless Firewall Plus

Package: gatekeeper-policies

Landing Zone Solutions

CFT https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/config-connector/solutions
KCC https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions
Simple PubSub example to get familiar with https://cloud.google.com/config-connector/docs/how-to/getting-started
KPT reference https://kpt.dev/reference/schema/plan/
Google Architecture Docs on config controller and config sync https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt

Monorepo LZ

Follow monorepo instructions from SSC around the hydration script (wraps kpt fn render)

Fully Automated Landing Zone Install

Automation procedure

see a clean run of the landing zone install from a clean GCP organization in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/611
see detailed procedures in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/446
create a HD bootstrap folder like ~/kcc
create a HD github folder like ~/kcc/github
clone the repo inside github/
create a HD kpt folder like ~/kcc/kpt
create a root gcp folder
create a bootstrap project like kcc-kls (use your domain as a unique id for example kcc.landing.systems = kls)
we get the Project Number and id from the bootstrap project (for vars.sh to write out setters.yaml)

PROJECT_NUMBER=$(gcloud projects list --filter="${CC_PROJECT_ID}" '--format=value(PROJECT_NUMBER)')

fill out setters.yaml and save it in place in your repo
gcloud config set project <bootstrap_project>
in ~/kcc/github/pubsec-declarative-toolkit/solutions run the setup.sh script after editing vars.sh

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-629)$ gcloud config set project kcc-oi
Updated property [core/project].
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ cd ../github/pubsec-declarative-toolkit/solutions/

Manual: Install the KCC cluster and minimal set of Landing Zone packages

0 overview https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/README.md#quickstart
1 see kcc cluster install script in https://github.com/ssc-spc-ccoe-cei/gcp-tools/tree/main/scripts/bootstrap via https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh and parts of https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh
2 see landing zone install docs in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/docs/landing-zone-v2/README.md#2-create-your-landing-zone

Landing Zone Developer Procedures

Developing for the KCC Landing Zone. Follow the original config controller docs...

Go to https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#landing-zone-developer-procedures

Landing Zone User Procedures

How to install the KCC landing zone - both the cluster and the LZ packages.

What the target Landing Zone looks like in GCP

core-landing-zone

Resource Manager

GKE object view

Networking

IAM

Billing

The projects-sa SA needs to have BAU (Billing Account User) enabled

[iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions](http://iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions)

Quota

Print remaining quotas by using

gcloud compute regions describe us-central1 --format=json | jq --raw-output ".quotas[]  | .metric , (.limit-.usage|tostring) "

....
NVIDIA_L4_GPUS
1
PREEMPTIBLE_NVIDIA_L4_GPUS
1
COMMITTED_NVIDIA_L4_GPUS
1


or
gcloud compute regions describe us-central1 --format=json | jq --raw-output ".quotas[] | select (.metric==\"COMMITTED_NVIDIA_L4_GPUS\") | .metric , (.limit-.usage|tostring) " 

COMMITTED_NVIDIA_L4_GPUS
1

reference: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/590

kubectl gcp view

There are 5 namespaces we target in the larger set - in the config-control namespace

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get namespaces
NAME                              STATUS   AGE
hierarchy                         Active   33m
logging                           Active   33m
networking                        Active   33m
policies                          Active   33m
projects                          Active   33m

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubens config-control
Context "gke_kcc-boot-ls-8704_northamerica-northeast1_krmapihost-kcc-oi4" modified.
Active namespace is "config-control".
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp
NAME                                                                AGE   READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           31m   True    UpToDate   31m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   31m   True    UpToDate   31m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           31m   True    UpToDate   31m
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           31m   True    UpToDate   30m

NAME                                                                                                             AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   31m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   31m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 31m   True    UpToDate   31m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    31m   True    UpToDate   30m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   31m   True    UpToDate   29m
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              31m   True    UpToDate   29m

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           31m   True    UpToDate   31m
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           31m   True    UpToDate   30m

NAME                                                                               AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-accesscontextmanager   31m   True    UpToDate   31m
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudbilling           31m   True    UpToDate   31m
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-cloudresourcemanager   31m   True    UpToDate   31m
service.serviceusage.cnrm.cloud.google.com/kcc-boot-ls-8704-serviceusage           31m   True    UpToDate   31m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get projects
No resources found in config-control namespace.

The log sink permissions issues are being worked out in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/586

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   24m   True    UpToDate   24m

NAME                                                                                                                   AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   24m   False   DependencyNotFound   24m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         24m   False   DependencyNotFound   24m
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               24m   False   DependencyNotFound   24m
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      24m   False   DependencyNotFound   24m

NAME                                                                AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-ls4       24m   True    UpToDate   21m
project.resourcemanager.cnrm.cloud.google.com/logging-project-ls4   30m   True    UpToDate   24m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    31m   True    UpToDate   28m
folder.resourcemanager.cnrm.cloud.google.com/clients                   31m   True    UpToDate   28m
folder.resourcemanager.cnrm.cloud.google.com/services                  31m   True    UpToDate   28m
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   31m   True    UpToDate   28m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n logging
NAME                                                                                   AGE   READY   STATUS     STATUS AGE
logginglogbucket.logging.cnrm.cloud.google.com/platform-and-component-log-bucket-ls4   25m   True    UpToDate   25m
logginglogbucket.logging.cnrm.cloud.google.com/security-log-bucket-ls4                 25m   True    UpToDate   25m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/mgmt-project-cluster-disable-default-bucket   25m   True    UpToDate   25m
root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n networking
No resources found in networking namespace.

root_@cloudshell:~/pdt-ls/obriensystems/pubsec-declarative-toolkit/solutions (kcc-boot-ls-8704)$ kubectl get gcp -n policies
NAME                                                                                                         AGE   READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project   32m   True    UpToDate   30m

core-landing-zone + hub-env

Scenarios

https://cloud.google.com/config-connector/docs/how-to/managing-deleting-resources
https://cloud.google.com/anthos-config-management/docs/tutorials/manage-resources-config-controller#delete_the_individual_resources
workflow 1 - shut down cluster only (create kcc cluster - create lz (render kpt lz packages) - delete kcc cluster - recreate cluster (automatically acquire gcp resources) - see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/794
workflow 2 - shut down lz and cluster (create kcc cluster - create lz (render kpt lz packages) - delete lz packages - delete kcc cluster

prereq = billing quota above 5, liens commented in the code, org polices (gatekeeper) omitted

create kcc cluster = https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L107

gcloud anthos config controller get-credentials $CLUSTER  --location $REGION

but use https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/setup-kcc.sh

gcloud anthos config controller create "$CLUSTER" --location "$REGION" --network "$NETWORK" --subnet "$SUBNET" --master-ipv4-cidr-block="172.16.0.128/28" --full-management "${args[@]}"
else

create lz = https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L165 see reconcile-timeout in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/802

# packages core-landing-zone, client-landing-zone, client-setup, project/hub-env
kpt live init core-landing-zone --namespace config-control --force
kpt fn render core landing-zone
kpt live apply core-landing-zone --reconcile-timeout=15m --output=table

delete lz = (including liens)

gcloud alpha resource-manager liens delete $NONPROD_LIEN # all 3
kpt live destroy core-landing-zone

delete kcc cluster
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/794

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh#L896

gcloud anthos config controller delete --location $REGION $CLUSTER --quiet

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ gcloud anthos config controller delete --location northamerica-northeast1 kcc  --quiet
Delete request issued for: [kcc]
Waiting for operation [projects/kcc-oi-7970/locations/northamerica-northeast1/operations/operation-1708103535888-61182d87f07de-28fac1d2-2f39769d] to complete...working... 

6 min
Deleted instance [kcc].

Onboarding to GCP

GCP Account and Organization

see example at https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/534

Full Automation

All up: bootstrap project creation, GKE cluster creation, LZ packages deployment
All down: (more of a developer workflow)

New GCP Organization

Developer Configuration

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/559

Google Cloud Shell

development environment out of the box. http://shell.cloud.google.com/ Note the 180 day file system reset (log in at least once per H1/H2) and the 50 hour/week quota -

Local Development - Containerized

We have a containerized Dockerfile dev environment referenced in gcp-tools https://github.com/ssc-spc-ccoe-cei/gcp-tools/tree/main/devcontainer

Local Development - Manual

see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/559

Use ADC credentials via SA

https://cloud.google.com/docs/authentication/client-libraries https://cloud.google.com/docs/authentication/client-libraries#java

set maven https://cloud.google.com/storage/docs/reference/libraries

add to pom.xml as usual

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>26.24.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
  </dependency>

Run a GCP GCS example

http://localhost:8080/nbi/swagger-ui.html#/application-service-controller

`

Working ADC GCP for GCS example from above https://github.com/obrienlabs/magellan/commit/554164ff0fd09e59abcc174eaf5b834b75bd3579

Setup gcloud SDK

https://cloud.google.com/sdk/docs/install

Local Development - on premise system

Install the gcloud CLI

Authenticate

Don't use CLI auth
gcloud auth login

use ADC
gcloud auth application-default login

Add Roles

gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$SUPER_ADMIN_EMAIL --role=roles/logging.admin --quiet

Configure API quota

gcloud auth application-default set-quota-project <project_id>

Set Project

gcloud config set project <project_id>

Local Development - CSP VM

Local GCP SDK docs - using a separate ubuntu VM on AWS

create a VM (with ssh key)
create EIP
associate EIP
ssh from local laptop

michaelobrien@mbp7 gcp-tools % dig bastion.obrienlabs.cloud
bastion.obrienlabs.cloud. 212	IN	A	54.167.180.91
michaelobrien@mbp7 gcp-tools % ssh [email protected]

follow the linux (not the debian/ubuntu) version - even for ubuntu (via curl/tar) https://cloud.google.com/sdk/docs/install#linux

Install the gcloud CLI

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-450.0.0-linux-x86_64.tar.gz
# will take about a min
tar -xf google-cloud-cli-450.0.0-linux-x86_64.tar.gz

ubuntu@ip-172-31-46-142:~$ ./google-cloud-sdk/install.sh
Welcome to the Google Cloud CLI!
Your current Google Cloud CLI version is: 450.0.0
The latest available version is: 450.0.0
ubuntu@ip-172-31-46-142:~$ source ~/.bashrc
ubuntu@ip-172-31-46-142:~$ gcloud version
Google Cloud SDK 450.0.0
bq 2.0.98
bundled-python3-unix 3.9.16
core 2023.10.06
gcloud-crc32c 1.0.0
gsutil 5.26

$ gcloud components update

Google API Calls

Local to GCP API calls over public IPs

Get Authorization Token

gcloud auth application-default login

Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

michaelobrien@mbp7 magellan % gcloud config set project $GOOGLE_CLOUD_PROJECT WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

To update your Application Default Credentials quota project, use the gcloud auth application-default set-quota-project command. Updated property [core/project].

after we work with the token from the previous auth request

michaelobrien@mbp7 magellan % GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json
michaelobrien@mbp7 magellan % gcloud auth application-default print-access-token                                  
ya29.c.c0AY_....2cyh62haq1k8-cmjb

michaelobrien@mbp7 magellan % ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
michaelobrien@mbp7 magellan % echo $ACCESS_TOKEN
ya29.c.......8VYaYhr_c6ooqq6

create a default big query table - in this case one of my GPS data exports

Follow for bigquery dataset creation https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494#issuecomment-1715822826 using 201611185_gps_distinct_rollerblad.csv Screenshot 2023-09-12 at 10 18 49

biometric:benchmark michaelobrien$ export GOOGLE_CLOUD_PROJECT=bigquery-ol


biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN" "https://www.googleapis.com/bigquery/v2/projects/$GOOGLE_CLOUD_PROJECT/datasets"
{
  "kind": "bigquery#datasetList",
  "etag": "irRvAdMvQcDWywbnuMPyBg==",
  "datasets": [
    {
      "kind": "bigquery#dataset",
      "id": "bigquery-ol:rollerblade",
      "datasetReference": {
        "datasetId": "rollerblade",
        "projectId": "bigquery-ol"
      },
      "location": "northamerica-northeast1"
    }
  ]
}
curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade
{
  "kind": "bigquery#dataset",
  "etag": "YGr4j2cVE8o77F0DFWySFg==",
  "id": "bigquery-ol:rollerblade",
  "selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade",
  "datasetReference": {
    "datasetId": "rollerblade",
    "projectId": "bigquery-ol"
  },
  ...
}

biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "kind": "bigquery#tableDataList",
  "etag": "wVKEep3xEdb51S3g1w7o3w==",
  "totalRows": "4331",
  "pageToken": "BHGUBPMJRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAWYTJM5YXKZLSPEWW63ARQX7PZB65AAAAAEQLOJXWY3DFOJRGYYLEMUNCINLEHE2GIYTEHEWWMMJUMUWTIM3GMQWTQZJQMYWTEMJVGFRWEOJTME3DMMQSBNZG63DMMVZGE3DBMRSRUJBYG5RTQNZUGM3S2YZZGBSC2NBWGVTC2OJYGA4C2ZJXGY4WGMBTGRSGEMLE",
  "rows": [
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.698209"

Google API calls using Service Account Impersonation

On a local system

Service Account Impersonation between orgs is detailed in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/449

Create an empty GCS bucket on your project so we can query it.

start https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev

michaelobrien@mbp7 magellan % export GOOGLE_CLOUD_PROJECT=bigquery-ol 
michaelobrien@mbp7 magellan % gcloud config set project $GOOGLE_CLOUD_PROJECT 
michaelobrien@mbp7 magellan % gcloud auth application-default login
Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login&state=2F1qPHOCSnylEphtdlusVqnCyRn3n1&access_type=offline&code_challenge=hcc2MVIAbZs0m2v99f3eArtCU1DUFx86CIsayULiA84&code_challenge_method=S256


Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "bigquery-ol" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.


michaelobrien@mbp7 magellan % cat /Users/michaelobrien/.config/gcloud/application_default_credentials.json                                                  
{
  "client_id": "76408605...pt8ejuq83di341hur.apps.googleusercontent.com",
  "client_secret": "d-F....D0Ty",
  "quota_project_id": "bigquery-ol",
  "refresh_token": "1//0...pPG-yUrK12H2dYwxz....D7RJo0q0M0",
  "type": "authorized_user"
}%

ADD service account impersonation

https://cloud.google.com/docs/authentication/use-service-account-impersonation

Service Account Impersonation between orgs is detailed in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/449

create service account https://console.cloud.google.com/iam-admin/serviceaccounts?project=bigquery-ol Screenshot 2023-10-31 at 10 15 14

impersonate at the ADC level for SDKs

michaelobrien@mbp7 magellan % SERVICE_ACCT_EMAIL=bigquery-ol-sa@bigquery-ol.iam.gserviceaccount.com                                                                                    
michaelobrien@mbp7 magellan % gcloud auth application-default login --impersonate-service-account $SERVICE_ACCT_EMAIL
Your browser has been opened to visit:
    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086...hFvqqXKWzq8u4&code_challenge_method=S256
Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

check before

michaelobrien@mbp7 magellan % gcloud storage buckets list
---
creation_time: 2023-10-31T14:20:38+0000
default_storage_class: STANDARD
location: NORTHAMERICA-NORTHEAST1
location_type: region
metageneration: 1
name: empty-bigquery-ol
public_access_prevention: enforced
storage_url: gs://empty-bigquery-ol/
uniform_bucket_level_access: true
update_time: 2023-10-31T14:20:38+0000

Impersonate at the cli level

michaelobrien@mbp7 magellan % gcloud config set auth/impersonate_service_account $SERVICE_ACCT_EMAIL
Updated property [auth/impersonate_service_account].
michaelobrien@mbp7 magellan % gcloud storage buckets list                                           
WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]].
WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]].
---
creation_time: 2023-10-31T14:20:38+0000
default_storage_class: STANDARD
location: NORTHAMERICA-NORTHEAST1
location_type: region
metageneration: 1
name: empty-bigquery-ol
public_access_prevention: enforced
storage_url: gs://empty-bigquery-ol/
uniform_bucket_level_access: true
update_time: 2023-10-31T14:20:38+0000
michaelobrien@mbp7 magellan %

Query bucket list using service account impersonation for a CURL REST call

https://cloud.google.com/storage/docs/listing-buckets#rest-list-buckets

michaelobrien@mbp7 magellan % ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
michaelobrien@mbp7 magellan % curl -X GET -H "Authorization: Bearer $ACCESS_TOKEN" "https://storage.googleapis.com/storage/v1/b?project=$GOOGLE_CLOUD_PROJECT"
{
  "kind": "storage#buckets",
  "items": [
    {
      "kind": "storage#bucket",
      "selfLink": "https://www.googleapis.com/storage/v1/b/empty-bigquery-ol",
      "id": "empty-bigquery-ol",
      "name": "empty-bigquery-ol",
      "projectNumber": "951469276805",
      "metageneration": "1",
      "location": "NORTHAMERICA-NORTHEAST1",
      "storageClass": "STANDARD",
      "etag": "CAE=",
      "timeCreated": "2023-10-31T14:20:38.638Z",
      "updated": "2023-10-31T14:20:38.638Z",
      "iamConfiguration": {
        "bucketPolicyOnly": {
          "enabled": true,
          "lockedTime": "2024-01-29T14:20:38.638Z"
        },
        "uniformBucketLevelAccess": {
          "enabled": true,
          "lockedTime": "2024-01-29T14:20:38.638Z"
        },
        "publicAccessPrevention": "enforced"
      },
      "locationType": "region"
    }
  ]
}

Local to GCP API calls over private IPs in a PSC (Private Service Connect) Configuration

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/494#issuecomment-1715822826

GCP SDK Developer Guide

Setup/Deployment

1 - Bootstrap Project

2 - KCC GKE Cluster Deployment

3 - KCC LZ V2 Package Deployments

Deletion/Teardown

Deleting an individual GCP service

Deleting KCC Packages

Deleting the KCC GKE cluster

Stopping/Restarting the KCC GKE cluster

Workarounds/Triage

Developer / DevOps Throughput

Developer Configuration - Relaxed Constraints

The landing zone comes up with near production level constraints - it helps to turn off most of these during development.

Organization Policy Overrides

Ref: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/550

Organization policies like the resource location restriction constraints/gcp.resourceLocations in https://console.cloud.google.com/iam-admin/orgpolicies/gcp-resourceLocations needs an override at the org, folder or project level if you are experimenting with resource in other regions.

The following kcc based services account has write access to organization policies

[email protected]
policies-sa
Organization Policy Administrator

However if you wish to have more than list rights on your own user identity account add the following role to your user. roles/orgpolicy.policyAdmin

Before....

GCloud

gcloud projects add-iam-policy-binding $PROJECT_ID --member=user:$USER_EMAIL --role=roles/orgpolicy.policyAdmin
or
gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$USER_EMAIL --role=roles/orgpolicy.policyAdmin

Console

Workflow Examples

Developer Workflows

FAQ

see Landing Zone procedures around create/update/delete package deployment actions - in addition to mitigation of incomplete/misconfigured/miscoded partial service deployments/deletes and their dependencies https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/750

Connecting to the GKE cluster using kubectl

Connecting to the GKE cluster using kubectl - when there is an API group list timeout

you get the following

michael@cloudshell:~ (kcc-cso-4380)$ kubectl get gcp -n config-control
E0301 13:42:02.846496    1425 memcache.go:265] couldn't get current server API group list: Get "https://34.86.120.56/api?timeout=32s": dial tcp 34.86.120.56:443: i/o timeout

refresh kubectl credentials
michael@cloudshell:~ (kcc-cso-4380)$ gcloud anthos config controller get-credentials kcc --location northamerica-northeast1
Fetching cluster endpoint and auth data.
kubeconfig entry generated for krmapihost-kcc.

Deployment: Determine kubernetes gcp service status

If during deployment of a particular solution package you see that there are errors during kpt apply - check the pod status. Checking a pod in kubernetes is done using the describe command of kubectl.

Get the service names.

# this assumes you are in a specific namespace
kubens config-control

kubectl get gcp
NAME                                                                  AGE   READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin            66d   True    UpToDate       66d
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role   66d   False   UpdateFailed   66d

Take an offending service and describe it. In the following case the org-id is not set.

kubectl describe iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role
Name:         hub-fortigatesdnreader-role
Namespace:    config-control
Labels:       <none>
Annotations:  cnrm.cloud.google.com/blueprint: kpt-pkg-fn-live
              cnrm.cloud.google.com/deletion-policy: abandon
              cnrm.cloud.google.com/management-conflict-prevention-policy: none
              cnrm.cloud.google.com/organization-id: 123456789012
              cnrm.cloud.google.com/state-into-spec: merge
              config.k8s.io/owning-inventory: abfad438df75719484ab97c58408cf033b706bf4-1692064998262569676
              config.kubernetes.io/depends-on: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dmu-admin1-hub-kls
              internal.kpt.dev/upstream-identifier: iam.cnrm.cloud.google.com|IAMCustomRole|config-control|hub-fortigatesdnreader-role
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMCustomRole
Status:
  Conditions:
    Last Transition Time:  2023-08-15T02:26:01Z
    Message:               Update call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing organizations/123456789012/roles/FortigateSdnViewer: googleapi: Error 403: You don't have permission to get the role at organizations/123456789012/roles/FortigateSdnViewer.

Deployment: Verifying kubernetes GCP resources

see https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources

For example the following check is on a system with core-landing-zone deployed and hub-env partially deployed - we are search on the flag "Ready" = False

first get our GCP namespaces
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get namespaces
NAME                              STATUS   AGE
...
hierarchy                         Active   67d
logging                           Active   67d
networking                        Active   67d
policies                          Active   67d
projects                          Active   67d


root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n projects | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n hierarchy | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n logging | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$

We checkout one of the services by using a describe

oot_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl describe computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr -n networking
Name:         hub-allow-fortigates-ha-fwr
Namespace:    networking
...
The issue is 
Status:
  Conditions:
    Last Transition Time:  2023-08-15T02:28:40Z
    Message:               reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found
    Reason:                DependencyNotFound
    Status:                False
    Type:                  Ready

Deployment: checking on folders

note deletion of child folder trees is not directly supported.
"There is a known issue where you cannot delete a Folder and its child Folder or Project resources at the same time. That is, you must either delete the parent Folder and wait for it to be gone from the Kubernetes API Server first or delete the children and wait for them to be gone from the Kubernetes API Server first."
https://cloud.google.com/config-connector/docs/reference/resource-docs/resourcemanager/folder

Deployment: check CRDs managed by config connector

see https://cloud.google.com/config-connector/docs/reference/overview

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true | grep folder
folders.resourcemanager.cnrm.cloud.google.com

Checking cnrm logs

get controller pod
michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kubectl get pods --all-namespaces | grep cnrm
cnrm-system                       cnrm-controller-manager-2ngn5mteag2v6r5itiwa-0            2/2     Running   0               77m
cnrm-system                       cnrm-controller-manager-c3w3isgmjny4adkmvixa-0            2/2     Running   0               76m
cnrm-system                       cnrm-controller-manager-it3zylhy24y5aobjjbha-0            2/2     Running   0               76m
cnrm-system                       cnrm-controller-manager-jl4awbbvx5nutfp7yq3a-0            2/2     Running   0               77m
cnrm-system                       cnrm-controller-manager-ovcmntvmtm3wq73uhuzq-0            2/2     Running   0               76m
cnrm-system                       cnrm-controller-manager-p2jcfga4lzvznyzcjuha-0            2/2     Running   0               4h45m
cnrm-system                       cnrm-controller-manager-yfi5fm3zvyuoan6qjobq-0            2/2     Running   0               76m
cnrm-system                       cnrm-controller-manager-zw3egolzoau5iyevttxa-0            2/2     Running   0               76m
cnrm-system                       cnrm-deletiondefender-0                                   1/1     Running   0               4h38m
cnrm-system                       cnrm-resource-stats-recorder-6b78d6845b-b5mdr             2/2     Running   0               4h41m
cnrm-system                       cnrm-unmanaged-detector-0                                 1/1     Running   0               4h41m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-75z5c                     1/1     Running   0               4h38m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-bqfzt                     1/1     Running   0               73m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-d952k                     1/1     Running   0               77m
cnrm-system                       cnrm-webhook-manager-6f9999f7fb-jp7sp                     1/1     Running   0               4h41m

kubectl logs cnrm-controller-manager-2ngn5mteag2v6r5itiwa-0  -n cnrm-system

check kpt tree

michael@cloudshell:~/kcc-cso/kpt (kcc-cso-4380)$ kpt pkg tree core-landing-zone
Package "core-landing-zone"
├── [Kptfile]  Kptfile core-landing-zone
├── [resourcegroup.yaml]  ResourceGroup config-control/inventory-49821483
├── [setters.yaml]  ConfigMap setters
├── audits
│   ├── [folder.yaml]  Folder hierarchy/audits
│   └── logging-project
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/platform-and-component-log-bucket-cso1
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/security-log-bucket
│       ├── [cloud-storage-buckets.yaml]  StorageBucket logging/security-incident-log-bucket
│       ├── [project-iam.yaml]  IAMAuditConfig projects/logging-project-data-access-log-config
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-infra-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/security-log-bucket-writer-permissions
│       ├── [project-sink.yaml]  LoggingLogSink logging/logging-project-cso1-data-access-sink
│       ├── [project.yaml]  Project projects/logging-project-cso1
│       ├── [services.yaml]  Service projects/logging-project-cso1-logging
│       ├── [services.yaml]  Service projects/logging-project-cso1-monitoring
│       └── monitoring
│           └── [metrics-scope.yaml]  MonitoringMonitoredProject logging/kcc-cso-4380
├── clients
│   └── [folder.yaml]  Folder hierarchy/clients
├── services
│   ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-log-sink
│   ├── [folder.yaml]  Folder hierarchy/services
│   └── services-infrastructure
│       ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-infra-log-sink
│       ├── [folder.yaml]  Folder hierarchy/services-infrastructure
│       └── dns-project
│           ├── [dns.yaml]  DNSManagedZone networking/dns-project-cso1-standard-core-public-dns
│           ├── [project.yaml]  Project projects/dns-project-cso1
│           └── [services.yaml]  Service projects/dns-project-cso1-dns
├── mgmt-project
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-accesscontextmanager
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-anthos
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-cloudbilling
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-cloudresourcemanager
│   ├── [services.yaml]  Service config-control/kcc-cso-4380-serviceusage
│   └── org-policies
│       ├── [compute-disable-serial-port-logging-except-mgt-project.yaml]  ResourceManagerPolicy policies/compute-disable-serial-port-logging-except-kcc-cso-4380
│       ├── [compute-require-shielded-vm-except-mgmt-project.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm-except-kcc-cso-4380
│       └── [compute-restrict-cloud-nat-usage-except-mgt-project.yaml]  ResourceManagerPolicy policies/compute-restrict-cloud-nat-usage-except-kcc-cso-4380
├── namespaces
│   ├── [config-management-monitoring.yaml]  IAMServiceAccount config-control/config-mgmt-mon-default-sa
│   ├── [config-management-monitoring.yaml]  IAMPolicyMember config-control/config-mgmt-mon-default-sa-metric-writer-permissions
│   ├── [config-management-monitoring.yaml]  IAMPartialPolicy config-control/config-mgmt-mon-default-sa-workload-identity-binding
│   ├── [config-management-monitoring.yaml]  ConfigConnectorContext config-management-monitoring/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [gatekeeper-system.yaml]  IAMServiceAccount config-control/gatekeeper-admin-sa
│   ├── [gatekeeper-system.yaml]  IAMPolicyMember config-control/gatekeeper-admin-sa-metric-writer-permissions
│   ├── [gatekeeper-system.yaml]  IAMPartialPolicy config-control/gatekeeper-admin-sa-workload-identity-binding
│   ├── [gatekeeper-system.yaml]  ConfigConnectorContext gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [hierarchy.yaml]  Namespace hierarchy
│   ├── [hierarchy.yaml]  IAMServiceAccount config-control/hierarchy-sa
│   ├── [hierarchy.yaml]  IAMPolicyMember config-control/hierarchy-sa-folderadmin-permissions
│   ├── [hierarchy.yaml]  IAMPartialPolicy config-control/hierarchy-sa-workload-identity-binding
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-folders-resource-reference-to-logging
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-config-control
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-policies
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-projects
│   ├── [hierarchy.yaml]  ConfigConnectorContext hierarchy/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  Namespace logging
│   ├── [logging.yaml]  IAMServiceAccount config-control/logging-sa
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-logadmin-permissions
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-monitoring-admin-kcc-cso-4380-permissions
│   ├── [logging.yaml]  IAMPartialPolicy config-control/logging-sa-workload-identity-binding
│   ├── [logging.yaml]  RoleBinding logging/allow-logging-resource-reference-from-projects
│   ├── [logging.yaml]  ConfigConnectorContext logging/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  IAMPolicyMember projects/logging-sa-monitoring-admin-logging-project-cso1-permissions
│   ├── [logging.yaml]  IAMPolicyMember projects/logging-sa-storageadmin-logging-project-cso1-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-editor-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-serviceaccountadmin-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-orgroleadmin-permissions
│   ├── [networking.yaml]  Namespace networking
│   ├── [networking.yaml]  IAMServiceAccount config-control/networking-sa
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-dns-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-networkadmin-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-security-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-service-control-org-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-servicedirectoryeditor-permissions
│   ├── [networking.yaml]  IAMPartialPolicy config-control/networking-sa-workload-identity-binding
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-xpnadmin-permissions
│   ├── [networking.yaml]  ConfigConnectorContext networking/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [policies.yaml]  Namespace policies
│   ├── [policies.yaml]  IAMServiceAccount config-control/policies-sa
│   ├── [policies.yaml]  IAMPolicyMember config-control/policies-sa-orgpolicyadmin-permissions
│   ├── [policies.yaml]  IAMPartialPolicy config-control/policies-sa-workload-identity-binding
│   ├── [policies.yaml]  ConfigConnectorContext policies/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [projects.yaml]  Namespace projects
│   ├── [projects.yaml]  IAMServiceAccount config-control/projects-sa
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-billinguser-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectcreator-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectdeleter-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectiamadmin-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectmover-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-serviceusageadmin-permissions
│   ├── [projects.yaml]  IAMPartialPolicy config-control/projects-sa-workload-identity-binding
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-logging
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-networking
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-policies
│   └── [projects.yaml]  ConfigConnectorContext projects/configconnectorcontext.core.cnrm.cloud.google.com
└── org
    ├── [org-sink.yaml]  LoggingLogSink logging/org-log-sink-data-access-logging-project-cso1
    ├── [org-sink.yaml]  LoggingLogSink logging/org-log-sink-security-logging-project-cso1
    └── custom-roles
        ├── [gke-firewall-admin.yaml]  IAMCustomRole config-control/gke-firewall-admin
        ├── [tier2-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier2-dnsrecord-admin
        ├── [tier2-vpcpeering-admin.yaml]  IAMCustomRole config-control/tier2-vpcpeering-admin
        ├── [tier3-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier3-dnsrecord-admin
        ├── [tier3-firewallrule-admin.yaml]  IAMCustomRole config-control/tier3-firewallrule-admin
        ├── [tier3-subnetwork-admin.yaml]  IAMCustomRole config-control/tier3-subnetwork-admin
        ├── [tier3-vpcsc-admin.yaml]  IAMCustomRole config-control/tier3-vpcsc-admin
        └── [tier4-secretmanager-admin.yaml]  IAMCustomRole config-control/tier4-secretmanager-admin

Continue KPT fn render after failed services fixed

expand these developer docs in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/800
see https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt#applying_a_kpt_package Sometimes some services high in the dependency block for example folder and project creation. Here is an example where a missing role on the GKE Yakima service account blocked iam access. The role is added to the automation script and/or manually added in IAM and the resources get fixed - however, to fix the dependencies - do another run of kpt.

see - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/568 see - https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh#L35

add
gcloud projects add-iam-policy-binding "kcc-oi-3552" --member "serviceAccount:[email protected]" --role "roles/serviceusage.serviceUsageConsumer" --project "kcc-oi-3552" --quiet

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl get gcp | grep False
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager   35m   False   UpdateFailed   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling           35m   False   UpdateFailed   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager   35m   False   UpdateFailed   35m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage           35m   False   UpdateFailed   35m
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-9428)$ kubectl get gcp | grep UpToDate
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   35m   True    UpToDate       87s
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             35m   True    UpToDate       88s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     35m   True    UpToDate       83s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager   35m   True    UpToDate   25s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling           35m   True    UpToDate   25s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager   35m   True    UpToDate   25s
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage           35m   True    UpToDate   25s

also fixed via

  gcloud organizations add-iam-policy-binding "${ORG_ID}" --member="serviceAccount:${SA_EMAIL}" --role=roles/iam.serviceAccountAdmin --condition=None --quiet
  
 for
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-9428)$ kubectl describe iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding
 Warning  UpdateFailed  110s (x5 over 5m55s)  iampartialpolicy-controller  Update call failed: error fetching live state for resource: error reading underlying resource: summary: Error when reading or editing Resource "service account 'projects/kcc-oi-9428/serviceAccounts/[email protected]'" with IAM Policy: Error retrieving IAM policy for service account 'projects/kcc-oi-9428/serviceAccounts/[email protected]': googleapi: Error 403: Permission 'iam.serviceAccounts.getIamPolicy' denied on resource (or it may not exist).
Details:

kpt live apply core-landing-zone

fixed
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   49m   True    UpToDate   55s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          49m   True    UpToDate   55s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            49m   True    UpToDate   55s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         49m   True    UpToDate   59s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           49m   True    UpToDate   54s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           49m   True    UpToDate   19s

Browse services in the GKE object browser

https://console.cloud.google.com/kubernetes/object/browser

Matching IAM view

Delete the landing zone

Delete via kpt - recommended

https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt#applying_a_kpt_package

Destroy the packages in reverse order that they were applied

kpt live destroy core-landing-zone


reconcile phase finished
inventory update started
inventory update finished
delete result: 90 attempted, 90 successful, 0 skipped, 0 failed
reconcile result: 90 attempted, 90 successful, 0 skipped, 0 failed, 0 timed out

Delete via kubectl - fallback

Keep the deletions split out by namespace for now
20231130 - restest with the required order - config-control (with permissions last) - re: comment from Alain in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/pull/740

  kubectl delete gcp -n networking --all
  kubectl delete gcp -n logging --all
  kubectl delete gcp -n policies --all
  kubectl delete gcp -n projects --all
  kubectl delete gcp -n hierarchy --all
  # retest and adjust/test order above and add at the end
  kubectl delete gcp -n config-control --all

Delete the KCC GKE cluster

see section that should be moved here in

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues?q=is%3Aissue+is%3Aopen+label%3Adelete

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/794

see automation in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh#L896

gcloud anthos config controller delete --location $REGION $CLUSTER --quiet

michael@cloudshell:~/kcc-oi-20231206/github/pubsec-declarative-toolkit/solutions (kcc-oi-7970)$ gcloud anthos config controller delete --location northamerica-northeast1 kcc  --quiet
Delete request issued for: [kcc]
Waiting for operation [projects/kcc-oi-7970/locations/northamerica-northeast1/operations/operation-1708103535888-61182d87f07de-28fac1d2-2f39769d] to complete...working... 

6 min
Deleted instance [kcc].

kpt delete - reconcile fails

check dependencies not met in the deployment blocking the delete - in the GKE object view
see https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/587

GCP Workarounds

GCP Organization Policy Overrides

GCP Organization Policy Overrides - constraint/resourceLocation policy

for when you need to create/move resources between regions
override at the organization or (preferred) project level
remember to limit permissions

Example

michael@cloudshell:~/cuda-old (cuda-old)$ vi constraint.yaml
michael@cloudshell:~/cuda-old (cuda-old)$ cat constraint.yaml 
name: projects/cuda-old/policies/gcp.resourceLocations
spec:
  rules:
  - values:
      allowedValues:
      - in:us-locations
      - in:northamerica-northeast1-locations
      - in:us-central1-locations
      - in:europe-west4-locations
      - in:northamerica-northeast2-locations

michael@cloudshell:~/cuda-old (cuda-old)$ gcloud org-policies set-policy --project cuda-old constraint.yaml
API [orgpolicy.googleapis.com] not enabled on project [cuda-old]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [orgpolicy.googleapis.com] on project [cuda-old]...
Operation "operations/acat.p2-196717963363-226ffd52-3de1-4e9a-a2d4-63b8acdee616" finished successfully.
Created policy [projects/cuda-old/policies/gcp.resourceLocations].
name: projects/196717963363/policies/gcp.resourceLocations
spec:
  etag: CPTivasGEMiTsqoC
  rules:
  - values:
      allowedValues:
      - in:us-locations
      - in:northamerica-northeast1-locations
      - in:us-central1-locations
      - in:europe-west4-locations
      - in:northamerica-northeast2-locations
  updateTime: '2023-12-05T18:52:36.625773Z'

Links

CC get resource status https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources
CFT https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/config-connector/solutions
KCC https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions
Simple PubSub example to get familiar with https://cloud.google.com/config-connector/docs/how-to/getting-started
KPT reference https://kpt.dev/reference/schema/plan/
Google Architecture Docs on config controller and config sync https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt
https://cloud.google.com/config-connector/docs/how-to/managing-deleting-resources
https://cloud.google.com/config-connector/docs/concepts/reconciliation#configuring_the_reconciliation_interval
Config Controller HA https://cloud.google.com/anthos-config-management/docs/how-to/config-controller-availability

Reference Architecture

GCP Google Cloud Functions V2 https endpoint

Use the following GCP Google Cloud Functions V2 https endpoint (org: oldev - proj: dev/eventstream-dev
list parameter is input into a random selector

https://northamerica-northeast1-eventstream-dev.cloudfunctions.net/random2?list=first,second,third,forth

{key: 1, value: second}