Cost monitoring with kubecost and AMP

bin/single-new-eks-cost-monitoring.ts

@@ -0,0 +1,11 @@
+import { configureApp, errorHandler } from '../lib/common/construct-utils';
+import SingleNewEksCostMonitoringPattern from '../lib/single-new-eks-cost-monitoring-pattern';
+
+const app = configureApp();
+
+new SingleNewEksCostMonitoringPattern()
+    .buildAsync(app, 'single-new-eks-cost-monitoring')
+    .catch((e) => {
+        errorHandler(app, "Secure Ingress Cost monitoring Pattern could not be deployed. \
+            See Secure Ingress Cost monitoring pattern in the readme for instructions", e);
+    });

docs/patterns/single-new-eks-observability-accelerators/single-new-eks-cost-monitoring-ingress-observability.md

@@ -0,0 +1,99 @@
+# Single Cluster Observability - Kubecost Cost Monitoring with Secure Ingress using Cognito
+
+Implementing Kubecost for monitoring EKS clusters provides invaluable insights into resource utilization and cost management. Kubecost offers granular visibility into the cost breakdown of Kubernetes workloads, enabling efficient allocation of resources and optimization of infrastructure spending. By integrating with Amazon Managed Prometheus (AMP) and AWS services such as Application Load Balancer, Amazon Cognito, and Amazon Route 53, Kubecost ensures a comprehensive monitoring solution with secure access control mechanisms. With alerts and recording rules provided by Amazon Managed Service for Prometheus, teams can proactively identify and address potential issues, ensuring optimal performance and cost-effectiveness of EKS deployments. Kubecost's user-friendly dashboard and reporting capabilities empower organizations to make informed decisions, maximize resource efficiency, and maintain cost predictability in their EKS environments, ultimately enhancing operational excellence and driving business growth.
+
+## Architecture
+
+The following figure illustrates the architecture of the pattern we will be deploying for Single EKS cost monitoring (Kubecost) pattern with Application Load Balancer, Amazon Cognito, and a Transport Layer Security (TLS) Certificate on AWS Certificate Manager (ACM) with Amazon Route 53 hosted zone to authenticate users to Kubecost
+
+![Architecture](../images/costmonitoring-ingress.png)
+
+## Objective
+
+- Deploys one production grade Amazon EKS cluster.
+- AWS Kubecost with Amazon Managed Prometheus (AMP) integration
+- [Secure Ingress with AWS Cognito](https://aws.amazon.com/blogs/containers/securing-kubecost-access-with-amazon-cognito/)
+- AWS Certificate Manager with Amazon Route 53 hosted zone 
+- Alerts and recording rules with Amazon Managed Service for Prometheus
+
+## Prerequisites:
+
+An existing hosted zone in Route53 with the ability to add records.
+
+Ensure that you have installed the following tools on your machine.
+
+1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
+2. [kubectl](https://Kubernetes.io/docs/tasks/tools/)
+3. [cdk](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install)
+4. [npm](https://docs.npmjs.com/cli/v8/commands/npm-install)
+
+## Configuring domain
+
+The CDK code expects the allowed domain and subdomain names in the CDK context file (cdk.json).
+
+Create two environment variables. The PARENT_HOSTED_ZONE variable contains the name of your Route 53 public hosted zone. The DEV_SUBZONE_NAME will be the address for your Kubecost dashboard.
+
+When users register to cognito they will have to provide an email address, using the `allowed.domains.list` you can specify you enterprise's email domain to only allow your employees to sign up for the service
+
+Generate the cdk.json file:
+
+```bash
+PARENT_HOSTED_ZONE=mycompany.a2z.com
+DEV_SUBZONE_NAME=kubecost.mycompany.a2z.com
+ALLOWED_DOMAIN_LIST=amazon.com
+cat << EOF > cdk.json
+{
+    "app": "npx ts-node dist/lib/common/default-main.js",
+    "context": {
+        "parent.hostedzone.name": "${PARENT_HOSTED_ZONE}",
+        "dev.subzone.name": "${DEV_SUBZONE_NAME}",
+        "allowed.domains.list": "${ALLOWED_DOMAIN_LIST}"
+      }
+}
+EOF
+```
+
+
+## Deploying
+
+Please follow the _Deploying_ instructions of the [New EKS Cluster Open Source Observability Accelerator](./single-new-eks-opensource-observability.md) pattern till step 7.
+At step 8, execute the following
+
+```bash
+make build
+make pattern single-new-eks-cost-monitoring deploy single-new-eks-cost-monitoring-observability-accelerator
+```
+
+## Updating CNAME record for Route53
+
+Open the AWS console once the deployment is complete. 
+Navigate to EC2>AWS Load Balancer. Select the newly created load balancer and copy to the DNS A Record to your clipboard.
+Navigate to Route53 in AWS console and select the hosted zone you want to use for the deployment. 
+Add new CNAME record type. 
+Set name as the DEV_SUBZONE_NAME defined previously and value as the load balancer DNS A record copied to your clipboard.
+Wait 60 seconds for DNS to get propogated.
+
+## Verify the resources
+
+Run update-kubeconfig command. You should be able to get the command from CDK output message.
+
+```bash
+aws eks update-kubeconfig --name single-new-eks-fargate-opensource-observability-accelerator --region <your region> --role-arn arn:aws:iam::xxxxxxxxx:role/single-new-eks-fargate-op-singleneweksfargateopens-xxxxxxxx
+```
+
+
+Let’s verify the resources created by steps above.
+
+```bash
+kubectl get pods -o wide -A
+```
+
+Now, lets navigate to the URL described as our dev.subzone.name in the cdk.json file and signup with a new cognito user profile.
+
+## Teardown
+
+You can teardown the whole CDK stack with the following command:
+
+```bash
+make pattern single-new-eks-cost-monitoring destroy
+```

lib/common/resources/otel-collector-config.yml

@@ -44,6 +44,19 @@ spec:
             external_labels:
               cluster: "{{clusterName}}"
           scrape_configs:
+            {{ start kubecostJob }}
+            - job_name: kubecost
+              honor_labels: true
+              scrape_interval: 1m
+              scrape_timeout: 10s
+              metrics_path: /metrics
+              scheme: http
+              dns_sd_configs:
+              - names:
+                - "{{subdomain}}"
+                type: 'A'
+                port: 9003
+            {{ stop kubecostJob }}
             {{ start enableAdotMetricsCollectionJob }}
             - job_name: otel-collector-metrics
               scrape_interval: 10s

lib/single-new-eks-cost-monitoring-pattern/cognito-idp-stack.ts

@@ -0,0 +1,129 @@
+import * as cdk from 'aws-cdk-lib';
+import * as blueprints from '@aws-quickstart/eks-blueprints';
+import { Construct } from 'constructs';
+import * as cognito from 'aws-cdk-lib/aws-cognito';
+import * as lambda from 'aws-cdk-lib/aws-lambda';
+import * as iam from 'aws-cdk-lib/aws-iam';
+
+export default class CognitoIdpStack extends cdk.Stack {
+
+    public readonly userPoolOut: cognito.UserPool;
+    public readonly userPoolClientOut: cognito.UserPoolClient;
+    public readonly userPoolDomainOut: cognito.UserPoolDomain;
+
+    constructor(scope: Construct, id: string, subDomain: string, props?: cdk.StackProps) {
+        super(scope, id, props);
+
+        const lambdaExecutionRole = new iam.Role(this, 'Lambda Execution Role', {
+            assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
+        });
+
+        lambdaExecutionRole.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName("service-role/AWSLambdaBasicExecutionRole"));
+        lambdaExecutionRole.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonSSMReadOnlyAccess"));     
+
+        const authChallengeFn = new lambda.Function(this, 'authChallengeFn', {
+            runtime: lambda.Runtime.PYTHON_3_12,
+            code: lambda.Code.fromAsset('./lib/single-new-eks-cost-monitoring-pattern/lambda'),
+            handler: 'lambda_function.lambda_handler',
+            role: lambdaExecutionRole,
+            environment: {
+                "ALLOWED_DOMAINS_LIST": blueprints.utils.valueFromContext(scope, "allowed.domains.list", "amazon.com")
+            }
+        });
+
+
+        // Cognito User Pool
+        const userPool = new cognito.UserPool(this, 'CognitoIDPUserPool', {
+            userPoolName: 'CognitoIDPUserPool',
+            selfSignUpEnabled: true,
+            signInAliases: {
+                email: true,
+                username: true
+            },
+            standardAttributes: {
+                email: {
+                    mutable: true,
+                    required: true
+                },
+                givenName: {
+                    mutable: true,
+                    required: true
+                },
+                familyName: {
+                    mutable: true,
+                    required: true
+                }
+            },
+            lambdaTriggers: {
+                preSignUp: authChallengeFn,
+                preAuthentication: authChallengeFn,
+            },            
+        });
+
+
+        // Output the User Pool ID
+
+        this.userPoolOut = userPool;
+
+        new cdk.CfnOutput(this, 'CognitoIDPUserPoolOut', {
+            value: userPool.userPoolId,
+            exportName: 'CognitoIDPUserPoolId'
+        });
+
+        new cdk.CfnOutput(this, 'CognitoIDPUserPoolArnOut', {
+            value: userPool.userPoolArn,
+            exportName: 'CognitoIDPUserPoolArn'
+        });
+
+
+        // We will ask the IDP to redirect back to our domain's index page
+        const redirectUri = `https://${subDomain}/oauth2/idpresponse`;
+
+        // Configure the user pool client application 
+        const userPoolClient = new cognito.UserPoolClient(this, 'CognitoAppClient', {
+            userPool,
+            authFlows: {
+                userPassword: true
+            },
+            oAuth: {
+                flows: {
+                    authorizationCodeGrant: true
+                },
+                scopes: [
+                    cognito.OAuthScope.OPENID
+                ],
+                callbackUrls: [redirectUri]
+                // TODO - What about logoutUrls?
+            },
+            generateSecret: true,
+            userPoolClientName: 'Web',
+            supportedIdentityProviders: [cognito.UserPoolClientIdentityProvider.COGNITO]
+        });
+
+        // Output the User Pool App Client ID
+        this.userPoolClientOut = userPoolClient;
+
+        new cdk.CfnOutput(this, 'CognitoIDPUserPoolClientOut', {
+            value: userPoolClient.userPoolClientId,
+            exportName: 'CognitoIDPUserPoolClientId'
+        });
+
+        // Add the domain to the user pool
+        const randomText = (Math.random() + 1).toString(36).substring(7);
+        const userPoolDomain = userPool.addDomain('CognitoDomain', {
+            cognitoDomain: {
+                domainPrefix: `my-cdk-blueprint-${randomText}`,
+            },
+        });
+
+        // Output the User Pool App Client ID
+
+        this.userPoolDomainOut = userPoolDomain;
+
+        new cdk.CfnOutput(this, 'CognitoIDPUserPoolDomainOut', {
+            value: userPoolDomain.domainName,
+            exportName: 'CognitoIDPUserPoolDomain'
+        });
+
+    }
+}