[IT-3984] Update for Schematic app

* Setup container deployment for Schematic * Update docs * OpenChallenges container uses a list of shared secrets however Schematic uses different secrets in each environment therefore we need to refactor the secrets handling to get secrets per environment.
Sage-Bionetworks-IT · Nov 6, 2024 · 2de9740 · 2de9740
1 parent a000ab9
commit 2de9740
Show file tree

Hide file tree

Showing 9 changed files with 109 additions and 197 deletions.
diff --git a/.github/workflows/deploy-dev.yaml b/.github/workflows/deploy-dev.yaml
@@ -3,7 +3,7 @@ name: deploy-dev
 on:
   workflow_run:
     workflows:
-      - check
+      - pre-deploy-check
     types:
       - completed
     branches:

diff --git a/.github/workflows/deploy-prod.yaml b/.github/workflows/deploy-prod.yaml
@@ -3,7 +3,7 @@ name: deploy-prod
 on:
   workflow_run:
     workflows:
-      - check
+      - pre-deploy-check
     types:
       - completed
     branches:

diff --git a/.github/workflows/deploy-stage.yaml b/.github/workflows/deploy-stage.yaml
@@ -3,7 +3,7 @@ name: deploy-stage
 on:
   workflow_run:
     workflows:
-      - check
+      - pre-deploy-check
     types:
       - completed
     branches:

diff --git a/.github/workflows/pre-deploy-check.yml b/.github/workflows/pre-deploy-check.yml
@@ -0,0 +1,25 @@
+name: pre-deploy-check
+
+on:
+  workflow_run:
+    workflows:
+      - check
+    types:
+      - completed
+    branches: [dev, stage, prod]
+
+jobs:
+  synth:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Install dependencies
+        run: pip install -r requirements.txt -r requirements-dev.txt
+      - name: Generate cloudformation
+        uses: youyo/aws-cdk-github-actions@v2
+        with:
+          cdk_subcommand: 'synth'
+          actions_comment: false
+          debug_log: true
+          cdk_args: '--output ./cdk.out'
diff --git a/README.md b/README.md
@@ -3,7 +3,7 @@
 
 AWS CDK app for deploying [Schematic](schematic.api.sagebionetworks.org).
 
-# Perequisites
+# Prerequisites
 
 AWS CDK projects require some bootstrapping before synthesis or deployment.
 Please review the [bootstapping documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap)
@@ -101,6 +101,7 @@ python -m pytest tests/ -s -v
 
 Deployment context is set in the [cdk.json](cdk.json) file.  An `ENV` environment variable must be
 set to tell the CDK which environment's variables to use when synthesising or deploying the stacks.
+The deployment context is the place to set infrastructure variables.
 
 Set an environment in cdk.json in `context` section of cdk.json:
 
@@ -117,7 +118,7 @@ Set an environment in cdk.json in `context` section of cdk.json:
   }
 ```
 
-For example, using the `prod` environment:
+For example, synthesis with the `prod` environment variables:
 
 ```console
 ENV=prod cdk synth
@@ -141,97 +142,40 @@ Once created take the ARN of the certificate and add it to a context in cdk.json
 
 # Secrets
 
-Secrets can be stored in one of the following locations:
-  * AWS SSM parameter store
-  * Local context in [cdk.json](cdk.json) file
+Secrets can be manually created in the [AWS SSM parameter store](./docs/aws-parameter-store.png)
+Then it can be retrieved using the `get_ssm_secret` method in [utils.py](./src/utils.py).
 
-## Loading directly from cdk.json
+Parameter store references can be passed into the ServiceProp objects:
 
-Set secrets directly in cdk.json in `context` section of cdk.json:
-
-```text
-  "context": {
-    "secrets": {
-        "MARIADB_PASSWORD": "Dummy",
-        "MARIADB_ROOT_PASSWORD": "Dummy",
-        "GIT_HOST_KEY": "Host123",
-        "GIT_PRIVATE_KEY": "-----BEGIN OPENSSH PRIVATE KEY-----\nDUMMY_GIT_PRIVATE_KEY\n-----END OPENSSH PRIVATE KEY-----",
-        "AWS_LOADER_S3_ACCESS_KEY_ID": "AccessKey123",
-        "AWS_LOADER_S3_SECRET_ACCESS_KEY": "SecretAccessKey123",
-        "SECURITY_KEY": "SecurityKey123"
-    }
-  }
-```
-
-## Loading from ssm parameter store
-
-Set secrets to the SSM parameter names in `context` section of cdk.json:
-
-```text
-  "context": {
-    "secrets": {
-        "MARIADB_PASSWORD": "/openchallenges/MARIADB_PASSWORD",
-        "MARIADB_ROOT_PASSWORD": "/openchallenges/MARIADB_ROOT_PASSWORD",
-        "GIT_HOST_KEY": "/openchallenges/GIT_HOST_KEY",
-        "GIT_PRIVATE_KEY": "/openchallenges/GIT_PRIVATE_KEY",
-        "AWS_LOADER_S3_ACCESS_KEY_ID": "/openchallenges/AWS_LOADER_S3_ACCESS_KEY_ID",
-        "AWS_LOADER_S3_SECRET_ACCESS_KEY": "/openchallenges/AWS_LOADER_S3_SECRET_ACCESS_KEY",
-        "SECURITY_KEY": "/openchallenges/SECURITY_KEY"
-    }
-  }
+```python
+app_service_props = ServiceProps(
+    "app", 443, 1024, f"ghcr.io/sage-bionetworks/app:v0.1.90-beta",
+    {
+        "MARIADB_USER": utils.get_ssm_secret("/app/dev/MARIADB_USER"),
+        "MARIADB_PASSWORD": utils.get_ssm_secret("/app/dev/MARIADB_PASSWORD")
+    },
+)
 ```
 
-where the values of these KVs (e.g. `/openchallenges/MARIADB_PASSWORD`) refer to SSM parameters that
+where the values of these KVs (e.g. `/app/dev/MARIADB_PASSWORD`) refer to SSM parameter which
 must be created manually.
 
 ![AWS secrets manager](docs/aws-parameter-store.png)
 
-## Specify secret location
-
-Set the `SECRETS` environment variable to specify the location where secrets should be loaded from.
-
-Load secrets directly from cdk.json file:
-
-```console
-SECRETS=local cdk synth
-```
-
-Load secrets from AWS SSM parameter store:
-
-```console
-AWS_PROFILE=<your-aws-profile> AWS_DEFAULT_REGION=us-east-1 SECRETS=ssm cdk synth
-```
-
 > [!NOTE]
-> Setting `SECRETS=ssm` requires access to an AWS account
-
-## Override secrets from command line
-
-The CDK CLI allows overriding context variables:
-
-To load secrets directly from passed in values:
-
-```console
-SECRETS=local cdk --context  secrets='{"MARIADB_PASSWORD": "Dummy", "MARIADB_ROOT_PASSWORD": "Dummy", ..}' synth
-```
-
-To load secrets from SSM parameter store with overridden SSM parameter names:
-
-```console
-SECRETS=ssm cdk --context  "secrets"='{"MARIADB_PASSWORD": "/test/mariadb-root-pass", "MARIADB_ROOT_PASSWORD": "/test/mariadb-root-pass", ..}' synth
-```
+> Retrieving secrets requires access to the AWS Service Systems Manager
 
 # Deployment
 
 ## Bootstrap
 
 There are a few items that need to be manually bootstrapped before deploying the application.
 
-* Add application [secrets](#Secrets) to either the cdk.json or the AWS System Manager parameter store
+* Add secrets to the AWS System Manager parameter store
 * Create an [ACM certificate for the application](#Certificates) using the AWS Certificates Manager
 * Add the Certificate ARN to the cdk.json
 * Update references to the docker images in [app.py](app.py)
-  (i.e. `ghcr.io/sage-bionetworks/schematic-xxx:<tag>`)
+  (i.e. `ghcr.io/sage-bionetworks/app-xxx:<tag>`)
 * (Optional) Update the `ServiceProps` objects in [app.py](app.py) with parameters specific to
   each container.
 
@@ -273,7 +217,7 @@ Deployment requires setting up an [AWS profile](https://docs.aws.amazon.com/cli/
 then executing the following command:
 
 ```console
-AWS_PROFILE=itsandbox-dev AWS_DEFAULT_REGION=us-east-1 ENV=dev SECRETS=ssm cdk deploy --all
+AWS_PROFILE=itsandbox-dev AWS_DEFAULT_REGION=us-east-1 ENV=dev cdk deploy --all
 ```
 
 ## Force new deployment
@@ -294,9 +238,9 @@ Example to get an interactive shell run into a container:
 
 ```console
 AWS_PROFILE=itsandbox-dev AWS_DEFAULT_REGION=us-east-1 aws ecs execute-command \
-  --cluster SchematicEcs-ClusterEB0386A7-BygXkQgSvdjY \
+  --cluster AppEcs-ClusterEB0386A7-BygXkQgSvdjY \
   --task a2916461f65747f390fd3e29f1b387d8 \
-  --container schematic-mariadb \
+  --container app-mariadb \
   --command "/bin/sh" --interactive
 ```
 
@@ -310,8 +254,8 @@ The workflow for continuous integration:
 * Create PR from the git dev branch
 * PR is reviewed and approved
 * PR is merged
-* CI deploys changes to the dev environment (dev.schematic.io) in the AWS dev account.
+* CI deploys changes to the dev environment (dev.app.io) in the AWS dev account.
 * Changes are promoted (or merged) to the git stage branch.
-* CI deploys changes to the staging environment (stage.schematic.io) in the AWS prod account.
+* CI deploys changes to the staging environment (stage.app.io) in the AWS prod account.
 * Changes are promoted (or merged) to the git prod branch.
-* CI deploys changes to the prod environment (prod.schematic.io) in the AWS prod account.
+* CI deploys changes to the prod environment (prod.app.io) in the AWS prod account.
diff --git a/app.py b/app.py
@@ -1,35 +1,35 @@
 import aws_cdk as cdk
+import src.utils as utils
 
 from src.network_stack import NetworkStack
 from src.ecs_stack import EcsStack
 from src.service_stack import LoadBalancedServiceStack
 from src.load_balancer_stack import LoadBalancerStack
 from src.service_props import ServiceProps
-import src.utils as utils
-
-cdk_app = cdk.App()
 
 # get the environment
 environment = utils.get_environment()
 stack_name_prefix = f"schematic-{environment}"
-image_version = "0.0.11"
 
-# get VARS from cdk.json
-env_vars = cdk_app.node.try_get_context(environment)
-fully_qualified_domain_name = env_vars["FQDN"]
+cdk_app = cdk.App()
+context_vars = cdk_app.node.try_get_context(environment)
+fully_qualified_domain_name = context_vars["FQDN"]
 subdomain, domain = fully_qualified_domain_name.split(".", 1)
-vpc_cidr = env_vars["VPC_CIDR"]
-certificate_arn = env_vars["CERTIFICATE_ARN"]
+vpc_cidr = context_vars["VPC_CIDR"]
+certificate_arn = context_vars["CERTIFICATE_ARN"]
+env_tags = context_vars["TAGS"]
 
-# get secrets from cdk.json or aws parameter store
-secrets = utils.get_secrets(cdk_app)
+# recursively apply tags to all stack resources
+if env_tags:
+    for key, value in env_tags.items():
+        cdk.Tags.of(cdk_app).add(key, value)
 
+# Generate stacks
 network_stack = NetworkStack(cdk_app, f"{stack_name_prefix}-network", vpc_cidr)
 
 ecs_stack = EcsStack(
     cdk_app, f"{stack_name_prefix}-ecs", network_stack.vpc, fully_qualified_domain_name
 )
-ecs_stack.add_dependency(network_stack)
 
 # From AWS docs https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts-deploy.html
 # The public discovery and reachability should be created last by AWS CloudFormation, including the frontend
@@ -39,22 +39,22 @@
     cdk_app, f"{stack_name_prefix}-load-balancer", network_stack.vpc
 )
 
-apex_service_props = ServiceProps(
-    "schematic-apex",
-    8000,
-    200,
-    f"ghcr.io/sage-bionetworks/schematic-apex:{image_version}",
+app_service_props = ServiceProps(
+    "schematic-app",
+    443,
+    4096,
+    "ghcr.io/sage-bionetworks/schematic:v24.10.2",
     {
-        "API_DOCS_HOST": "schematic-api-docs",
-        "API_DOCS_PORT": "8010",
-        "API_GATEWAY_HOST": "schematic-api-gateway",
-        "API_GATEWAY_PORT": "8082",
-        "APP_HOST": "schematic-app",
-        "APP_PORT": "4200",
-        "THUMBOR_HOST": "schematic-thumbor",
-        "THUMBOR_PORT": "8889",
-        "ZIPKIN_HOST": "schematic-zipkin",
-        "ZIPKIN_PORT": "9411",
+        "project_id": "sagebio-integration-testing",
+        "client_email": "sage-bio-integration-test-dev@sagebio-integration-testing.iam.gserviceaccount.com",
+        "client_id": utils.get_ssm_secret("/schematic/dev/client_id"),
+        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+        "token_uri": "https://oauth2.googleapis.com/token",
+        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/sage-bio-integration-test-dev%40sagebio-integration-testing.iam.gserviceaccount.com",  # noqa
+        "universe_domain": "googleapis.com",
+        "private_key_id": utils.get_ssm_secret("/schematic/dev/private_key_id"),
+        "private_key": utils.get_ssm_secret("/schematic/dev/private_key"),
     },
 )
 
@@ -63,12 +63,11 @@
     f"{stack_name_prefix}-app",
     network_stack.vpc,
     ecs_stack.cluster,
-    apex_service_props,
+    app_service_props,
     load_balancer_stack.alb,
     certificate_arn,
-    health_check_path="/health",
+    health_check_path="/v1/ui/",
     health_check_interval=5,
 )
-app_service_stack.add_dependency(load_balancer_stack)
 
 cdk_app.synth()
diff --git a/cdk.json b/cdk.json
@@ -66,26 +66,26 @@
     "dev": {
         "VPC_CIDR": "10.254.192.0/24",
         "FQDN": "dev.schematic.io",
-        "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:631692904429:certificate/0e9682f6-3ffa-46fb-9671-b6349f5164d6"
-      },
+        "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:631692904429:certificate/0e9682f6-3ffa-46fb-9671-b6349f5164d6",
+        "TAGS": {
+          "CostCenter": "NO PROGRAM / 000000"
+        }
+    },
     "stage": {
         "VPC_CIDR": "10.254.193.0/24",
         "FQDN": "stage.schematic.io",
-        "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:878654265857:certificate/d11fba3c-1957-48ba-9be0-8b1f460ee970"
-      },
+        "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:878654265857:certificate/d11fba3c-1957-48ba-9be0-8b1f460ee970",
+        "TAGS": {
+          "CostCenter": "NO PROGRAM / 000000"
+        }
+    },
     "prod": {
         "VPC_CIDR": "10.254.194.0/24",
         "FQDN": "prod.schematic.io",
-        "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:878654265857:certificate/d11fba3c-1957-48ba-9be0-8b1f460ee970"
-      },
-    "secrets": {
-        "MARIADB_PASSWORD": "/openchallenges/MARIADB_PASSWORD",
-        "MARIADB_ROOT_PASSWORD": "/openchallenges/MARIADB_ROOT_PASSWORD",
-        "GIT_HOST_KEY": "/openchallenges/GIT_HOST_KEY",
-        "GIT_PRIVATE_KEY": "/openchallenges/GIT_PRIVATE_KEY",
-        "AWS_LOADER_S3_ACCESS_KEY_ID": "/openchallenges/AWS_LOADER_S3_ACCESS_KEY_ID",
-        "AWS_LOADER_S3_SECRET_ACCESS_KEY": "/openchallenges/AWS_LOADER_S3_SECRET_ACCESS_KEY",
-        "SECURITY_KEY": "/openchallenges/SECURITY_KEY"
+        "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:878654265857:certificate/d11fba3c-1957-48ba-9be0-8b1f460ee970",
+        "TAGS": {
+          "CostCenter": "NO PROGRAM / 000000"
+        }
     }
   }
 }