Docs for Managed Service for Apache Flink

content/en/user-guide/aws/firehose/index.md

@@ -1,27 +1,27 @@
 ---
-title: "Kinesis Data Firehose"
-linkTitle: "Kinesis Data Firehose"
+title: "Data Firehose"
+linkTitle: "Data Firehose"
 description: >
-  Get started with Kinesis Data Firehose on LocalStack
+  Get started with Data Firehose on LocalStack
 aliases:
 - /user-guide/aws/kinesis-firehose/
 ---
 
 {{< callout >}}
-Amazon recently renamed Kinesis Data Firehose to Data Firehose.
+This service was formerly called as 'Kinesis Data Firehose'.
 {{< /callout >}}
 
 ## Introduction
 
-Kinesis Data Firehose is a service provided by AWS that allows you to extract, transform and load streaming data into various destinations, such as Amazon S3, Amazon Redshift, and Elasticsearch.
-With Kinesis Data Firehose, you can ingest and deliver real-time data from different sources as it automates data delivery, handles buffering and compression, and scales according to the data volume.
+Data Firehose is a service provided by AWS that allows you to extract, transform and load streaming data into various destinations, such as Amazon S3, Amazon Redshift, and Elasticsearch.
+With Data Firehose, you can ingest and deliver real-time data from different sources as it automates data delivery, handles buffering and compression, and scales according to the data volume.
 
-LocalStack allows you to use the Kinesis Data Firehose APIs in your local environment to load and transform real-time data.
-The supported APIs are available on our [API coverage page](https://docs.localstack.cloud/references/coverage/coverage_firehose/), which provides information on the extent of Kinesis Data Firehose's integration with LocalStack.
+LocalStack allows you to use the Data Firehose APIs in your local environment to load and transform real-time data.
+The supported APIs are available on our [API coverage page](https://docs.localstack.cloud/references/coverage/coverage_firehose/), which provides information on the extent of Data Firehose's integration with LocalStack.
 
 ## Getting started
 
-This guide is designed for users new to Kinesis Data Firehouse and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.
+This guide is designed for users new to Data Firehouse and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.
 
 Start your LocalStack container using your preferred method.
 We will demonstrate how to use Firehose to load Kinesis data into Elasticsearch with S3 Backup with the AWS CLI.
@@ -162,7 +162,7 @@ Additionally, take a look at the designated S3 bucket to ensure the backup proce
 
 ## Examples
 
-The following code snippets and sample applications provide practical examples of how to use Kinesis Data Firehose in LocalStack for various use cases:
+The following code snippets and sample applications provide practical examples of how to use Data Firehose in LocalStack for various use cases:
 
 - [Search application with Lambda, Kinesis, Firehose, ElasticSearch, S3](https://github.com/localstack/sample-fuzzy-movie-search-lambda-kinesis-elasticsearch)
 - [Streaming Data Pipeline with Kinesis, Tinybird, CloudWatch, Lambda](https://github.com/localstack/serverless-streaming-data-pipeline)

content/en/user-guide/aws/kinesisanalytics/index.md

@@ -1,24 +1,24 @@
 ---
-title: "Kinesis Data Analytics"
-linkTitle: "Kinesis Data Analytics"
+title: "Kinesis Data Analytics for SQL Applications"
+linkTitle: "Kinesis Data Analytics for SQL Applications"
 description: >
-  Get started with Kinesis Data Analytics on LocalStack
+  Get started with Kinesis Data Analytics for SQL Applications on LocalStack
 tags: ["Pro image"]
 ---
 
+{{< callout "warning" >}}
+This service is deprecated and marked for removal.
+{{< /callout >}}
+
 ## Introduction
 
-Kinesis Data Analytics is a service offered by Amazon Web Services (AWS) that enables you to process and analyze streaming data in real-time.
-Kinesis Data Analytics allows you to apply transformations, filtering, and enrichment to streaming data using standard SQL syntax.
-You can also run Java or Scala programs against streaming sources to perform various operations on the data using Apache Flink.
+Kinesis Data Analytics for SQL Applications is a service offered by Amazon Web Services (AWS) that enables you to process and analyze streaming data in real-time.
+It allows you to apply transformations, filtering, and enrichment to streaming data using standard SQL syntax.
 
 LocalStack allows you to use the Kinesis Data Analytics APIs in your local environment.
 The API coverage is available on:
 
-* [Kinesis Data Analytics V1](https://docs.localstack.cloud/references/coverage/coverage_kinesisanalytics/)
-* [Kinesis Data Analytics V2](https://docs.localstack.cloud/references/coverage/coverage_kinesisanalyticsv2/)
-
-This provides information on the extent of Kinesis Data Analytics integration with LocalStack.
+* [Kinesis Data Analytics](https://docs.localstack.cloud/references/coverage/coverage_kinesisanalytics/)
 
 ## Getting started
 
@@ -105,8 +105,7 @@ The following output would be retrieved:
 }
 ```
 
-## Current Limitations
+## Limitations
 
-* LocalStack supports basic emulation for the version 1 of the Kinesis Data Analytics API.
+* LocalStack supports basic emulation for Kinesis Data Analytics for SQL Applications.
   However, the queries are not fully supported and lack parity with AWS.
-* LocalStack supports CRUD mocking for the version 2 of the Kinesis Data Analytics API.

content/en/user-guide/aws/kinesisanalyticsv2/index.md

@@ -0,0 +1,175 @@
+---
+title: "Managed Service for Apache Flink"
+linkTitle: "Managed Service for Apache Flink"
+description: >
+  Get started with Managed Service for Apache Flink on LocalStack
+tags: ["Pro image"]
+---
+
+{{< callout >}}
+This service was formerly known as 'Kinesis Data Analytics for Apache Flink'.
+{{< /callout >}}
+
+## Introduction
+
+[Apache Flink](https://flink.apache.org/) is a framework for building applications that process and analyze streaming data.
+[Managed Service for Apache Flink (MSAF)](https://docs.aws.amazon.com/managed-flink/latest/java/what-is.html) is an AWS service that provides the underlying infrastructure and a hosted Apache Flink cluster that can run Apache Flink applications.
+
+LocalStack lets you to run Flink applications locally and implements several [AWS-compatible API operations](https://docs.localstack.cloud/references/coverage/coverage_kinesisanalyticsv2/).
+
+{{< callout "note" >}}
+The emulated MSAF provider was introduced and made the default in LocalStack v4.1.
+
+If you wish to use the older mock provider, you can set `PROVIDER_OVERRIDE_KINESISANALYTICSV2=legacy`.
+{{< /callout >}}
+
+## Getting Started
+
+This guide builds a demo Flink application and deploys it to LocalStack.
+The application generates synthetic records, processes them and sends the output to an S3 bucket.
+
+Start the LocalStack container using your preferred method.
+
+### Build Application Code
+
+Begin by cloning the AWS sample repository.
+We will use the [S3 Sink](https://github.com/localstack-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/S3Sink) application in this example.
+
+{{< command >}}
+$ git clone https://github.com/localstack-samples/amazon-managed-service-for-apache-flink-examples.git
+$ cd java/S3Sink
+{{< /command >}}
+
+Next, use [Maven](https://maven.apache.org/) to compile and package the Flink application into a jar.
+
+{{< command >}}
+$ mvn package
+{{< /command >}}
+
+The Flink application jar file will be placed in the `./target/flink-kds-s3.jar` directory.
+
+### Upload Application Code
+
+MSAF requires that all application code resides in S3.
+
+Create an S3 bucket and upload the compiled Flink application jar.
+
+{{< command >}}
+$ awslocal s3api create-bucket --bucket flink-bucket
+$ awslocal s3api put-object --bucket flink-bucket --key job.jar --body ./target/flink-kds-s3.jar
+{{< /command >}}
+
+### Output Sink
+
+As mentioned earlier, this Flink application writes the output to an S3 bucket.
+
+Create the S3 bucket that will serve as the sink.
+
+{{< command >}}
+$ awslocal s3api create-bucket --bucket sink-bucket
+{{< /command >}}
+
+### Permissions
+
+MSAF requires a service execution role which allows it to connect to other services.
+Without the proper permissions policy and role, this example application will not be able to connect to S3 sink bucket to output the result.
+
+Create an IAM role for the running MSAF application to assume.
+
+```json
+# role.json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Principal": {"Service": "kinesisanalytics.amazonaws.com"},
+            "Action": "sts:AssumeRole"
+        }
+    ]
+}
+```
+
+{{< command >}}
+$ awslocal iam create-role --role-name msaf-role --assume-role-policy-document file://role.json
+{{< /command >}}
+
+Next create add a permissions policy to this role that permits read and write access to S3.
+
+```json
+# policy.json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:PutObject"],
+            "Resource": "*"
+        }
+    ]
+}
+```
+
+{{< command >}}
+$ awslocal iam put-role-policy --role-name msaf-role --policy-name msaf-policy --policy-document file://policy.json
+{{< /command >}}
+
+Now, when the running MSAF application assumes this role, it will have the necessary permissions to write to the S3 sink.
+
+### Deploy Application
+
+With all prerequisite resources in place, the Flink application can now be created and started.
+
+{{< command >}}
+$ awslocal kinesisanalyticsv2 create-application \
+    --application-name msaf-app \
+    --runtime-environment FLINK-1_20 \
+    --application-mode STREAMING \
+    --service-execution-role arn:aws:iam::000000000000:role/msaf-role \
+    --application-configuration '{
+        "ApplicationCodeConfiguration": {
+            "CodeContent": {
+                "S3ContentLocation": {
+                    "BucketARN": "arn:aws:s3:::flink-bucket",
+                    "FileKey": "job.jar"
+                }
+            },
+            "CodeContentType": "ZIPFILE"
+        },
+        "EnvironmentProperties": {
+            "PropertyGroups": [{
+                "PropertyGroupId": "bucket", "PropertyMap": {"name": "sink-bucket"}
+            }]
+        }
+    }'
+
+$ awslocal kinesisanalyticsv2 start-application --application-name msaf-app
+{{< /command >}}
+
+Once the Flink cluster is up and running, the application will stream the results to the sink S3 bucket.
+You can verify this with:
+
+{{< command >}}
+$ awslocal s3api list-objects --bucket sink-bucket
+{{< /command >}}
+
+## Supported Flink Versions
+
+| Flink version | Supported by LocalStack | Supported by Apache |
+|:---:|:---:|:---:|
+| 1.20.0 | yes | yes |
+| 1.19.1 | yes | yes |
+| 1.18.1 | yes | yes |
+| 1.15.2 | yes | yes |
+| 1.13.1 | yes | no |
+
+## Limitations
+
+- Application versions are not maintained
+- Only S3 zipfile code is supported
+- Values of 20,000 ms for `execution.checkpointing.interval` and 5,000 ms for `execution.checkpointing.min-pause` are used for checkpointing.
+  They can not be overridden.
+- [Tagging](https://docs.aws.amazon.com/managed-flink/latest/java/how-tagging.html) is not supported
+- In-place [version upgrades](https://docs.aws.amazon.com/managed-flink/latest/java/how-in-place-version-upgrades.html) and [roll-backs](https://docs.aws.amazon.com/managed-flink/latest/java/how-system-rollbacks.html) are not supported
+- [Snapshot/savepoint management](https://docs.aws.amazon.com/managed-flink/latest/java/how-snapshots.html) is not implemented
+- CloudWatch and CloudTrail integration is not implemented