This repository has been archived by the owner on Jan 29, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #2361 from aiven/harshini-flink-jars-howto
Howto content Flink Custom JARs application
- Loading branch information
Showing
6 changed files
with
171 additions
and
102 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,61 +1,16 @@ | ||
| Create an Aiven for Apache Flink® application | ||
| ============================================== | ||
|
|
||
| :doc:`Aiven for Flink applications </docs/products/flink/concepts/flink-applications>` in Aiven for Apache Flink® servers as a container that includes everything connected to a Flink job, including source and sink connections and data processing logic. The `Aiven Console <https://console.aiven.io/>`_ provides a guided wizard to help you build and deploy applications, simplifying the process of selecting the source and sink tables, writing data transformation statements, and validating and ingesting data through the interactive query feature. | ||
|
|
||
| This article provides the information required to build and deploy applications on Aiven for Apache Flink service. | ||
|
|
||
| .. note:: | ||
| You must set up the :doc:`data service integration </docs/products/flink/howto/create-integration>` before building applications. | ||
|
|
||
|
|
||
| Create an application via the Aiven console | ||
| -------------------------------------------- | ||
|
|
||
| Follow these steps to build your first Aiven for Flink application: | ||
|
|
||
| 1. In the `Aiven Console <https://console.aiven.io/>`_, open the Aiven for Apache Flink service for which you want to create an application. | ||
| 2. Select **Applications** from the left sidebar and select **Create application** to create your first application. | ||
| 3. In the **Create new application** screen, enter the name of your application and configure the necessary deployment settings. select **Create application**. | ||
| 4. Select **Create first version** to create the first version of the application. | ||
| 5. Select **Add your first source table** to add a source table. | ||
|
|
||
| .. note:: | ||
| Since this is your first application, there are currently no other applications where you can import source tables. | ||
|
|
||
| 6. In the **Add new source table** screen, | ||
|
|
||
| * Select the **Integrated service** from the drop-down list. | ||
| * In the **Table SQL** section, enter the statement that will be used to create the table. | ||
| * Optionally, select **Run** to view how data is being pulled from the data source. This could take some time based on the data and the connection. | ||
| * Select **Add table**. | ||
| 7. Select **Next** to add the sink table, and then select **Add your first sink table**. | ||
|
|
||
| .. note:: | ||
| Since this is your first application, there are currently no other applications where you can import sink tables. | ||
|
|
||
| 8. In the **Add new sink table** screen, | ||
|
|
||
| * Select **Integrated service** from the drop-down list. | ||
| * In the **Table SQL** section, enter the statement that will be used to create the table. | ||
| * Select **Add table**. | ||
| 9. Select **Next** to enter the **SQL statement** that transforms the data from the source stream. Optionally, select **Run** to view how data is being pulled from the data source. | ||
| 10. Select **Save and deploy later** to save the application. You will see the application on the landing page that provides you with an overview of your application. | ||
|
|
||
| .. image:: /images/products/flink/application_landingpage_view.png | ||
| :scale: 50 % | ||
| :alt: Application landing page with an view of source table, SQL statement and sink table | ||
|
|
||
| 11. Select **Create deployment**. On the **Create new deployment** screen, | ||
|
|
||
| * Choose the version you wish to deploy. The default version for the first deployment is **Version: 1**. | ||
| * Choose the :doc:`savepoint </docs/products/flink/concepts/savepoints>` from where you want to deploy. No savepoints are available for the first application deployment. | ||
| * Use the toggle for **Restart on failure** to enable or disable the option of automatically restarting a Flink job in case it fails. | ||
| * Enter the number of `parallel instances <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/>`_ you want to have for the task. | ||
| 12. Select **Deploy without a savepoint** to deploy your application. | ||
| 13. The deployment status will show **Initializing: version 1** and then **Running: version 1**. | ||
|
|
||
| Your first application is now created and deployed, and you can view the data related to the actions the application needs to perform in your sink source. | ||
| Create and manage Aiven for Apache Flink® applications | ||
| =========================================================== | ||
|
|
||
| :doc:`Aiven for Flink applications </docs/products/flink/concepts/flink-applications>` in Aiven for Apache Flink® servers as a container that includes everything connected to a Flink job, including source and sink connections and data processing logic. | ||
|
|
||
| Using the `Aiven Console <https://console.aiven.io/>`_, you can create applications that run SQL queries or deploy custom JARs, catering to diverse data processing requirements. The console's guided wizard simplifies the application configuration process, from selecting source and sink tables for SQL applications to uploading and managing JAR files for custom job execution. | ||
|
|
||
| .. important:: | ||
|
|
||
| Custom JARs for Aiven for Apache Flink is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at [email protected]. | ||
|
|
||
| .. tableofcontents:: | ||
|
|
||
|
|
||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| Create a JAR application | ||
| ======================== | ||
| Aiven for Apache Flink® enables you to upload and deploy custom code as a JAR file, enhancing your Flink applications with advanced data processing capabilities. | ||
|
|
||
| .. important:: | ||
|
|
||
| Custom JARs for Aiven for Apache Flink is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at [email protected]. | ||
|
|
||
| Prerequisite | ||
| ------------ | ||
|
|
||
| * To enable custom JARs for a new Aiven for Apache Flink service, toggle the feature during service creation. | ||
| * For an existing service, in `Aiven Console <https://console.aiven.io/>`_ , select your project and then choose your Aiven for Apache Flink® service. | ||
|
|
||
| * Click **Service settings** on the left sidebar. | ||
| * Scroll to the **Advanced configuration** section, and click **Configure**. | ||
| * In the **Advanced configuration** screen, click **Add configuration options**, and using the search box find and set ``custom_code`` configuration to **Enabled** position. | ||
|
|
||
| Create and deploy application | ||
| --------------------------------- | ||
|
|
||
| 1. Access the `Aiven Console <https://console.aiven.io/>`_ and select the Aiven for Apache Flink service where you want to deploy a JAR application. | ||
| 2. From the left sidebar, click **Applications** and then click **Create application**. | ||
| 3. In the **Create application** dialog, enter a name for your JAR application, and select **JAR** as the application type from the drop-down. | ||
| 4. Click **Create application** to proceed. | ||
| 5. Click **Upload first version** to upload the first version of the application. | ||
| 6. In the **Upload new version** dialog: | ||
|
|
||
| * Click **Choose file** to select your custom JAR file. | ||
| * Select the **Terms of Service** checkbox to indicate your agreement. | ||
| * Click **Upload version** to upload your JAR file. | ||
|
|
||
| 7. After the upload, you are redirected to the application's overview page. | ||
| 8. To deploy the application, click **Create deployment**. In the **Create new deployment** dialog: | ||
|
|
||
| * Select the application version to deploy. | ||
| * Select a :doc:`savepoint </docs/products/flink/concepts/savepoints>` if you wish to deploy from a specific state. No savepoints are available for the first application deployment. | ||
| * Toggle **Restart on failure** to automatically restart Flink jobs upon failure. | ||
| * In the **Program args** field, provide command-line arguments consisting of variables and configurations relevant to your application's logic upon submission. Each argument is limited to 64 characters, with a total limit of 32 separate items. | ||
| * Specify the number of `parallel instances <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/>`_ you require for the task. | ||
|
|
||
| 9. Click **Deploy without a savepoint** to begin the deployment process. | ||
| 10. While deploying, the application status shows **Initializing**. Once deployed, the status changes to **Running**. | ||
|
|
||
|
|
||
| Related pages | ||
| -------------- | ||
|
|
||
| * :doc:`Manage Aiven for Apache Flink® applications </docs/products/flink/howto/manage-flink-applications>` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| Create an SQL application | ||
| ============================ | ||
|
|
||
| In Aiven for Apache Flink®, you can create an SQL application that uses Apache Flink SQL to streamline the process of building data processing pipelines. The SQL application simplifies the process of defining source and sink tables, implementing data processing logic, and managing application deployment. | ||
|
|
||
| Prerequisite | ||
| ------------ | ||
| Before creating applications, configure the :doc:`data service integration </docs/products/flink/howto/create-integration>` for seamless integration and data management within your Flink applications. | ||
|
|
||
| Create and deploy application | ||
| ------------------------------- | ||
|
|
||
| Create an SQL application in Aiven for Apache Flink® using the `Aiven Console <https://console.aiven.io/>`_: | ||
|
|
||
| 1. In the `Aiven Console <https://console.aiven.io/>`_, select the Aiven for Apache Flink service where you want to create and deploy a Flink application. | ||
| 2. From the left sidebar, click **Applications** and then click **Create application**. | ||
| 3. In the **Create application** dialog, enter the name of your application and select **SQL** as the application type. | ||
| 4. Click **Create application**. | ||
| 5. Click **Create first version** to create the first version of the application. | ||
| 6. Click **Add your first source table** to add a source table. | ||
|
|
||
| .. note:: | ||
| As this is your first application, no other applications are available to import source tables. | ||
|
|
||
| 7. On the **Add new source table** screen: | ||
|
|
||
| * Use the **Integrated service** drop-down to select the service. | ||
| * In the **Table SQL** section, enter the SQL statement to create the source table. | ||
| * Optionally, click **Run** to test how data is being retrieved from the data source. This may vary in time based on the data volume and connection speed. | ||
| * Click **Add table**. | ||
| 8. Click **Next** to proceed to adding a sink table and click **Add your first sink table**. | ||
|
|
||
| .. note:: | ||
| As this is your first application, no other applications are available to import sink tables. | ||
|
|
||
| 9. On the **Add new sink table** screen: | ||
|
|
||
| * Use the **Integrated service** drop-down to select the service. | ||
| * In the **Table SQL** section, enter the SQL statement to create the sink table. | ||
| * Click **Add table**. | ||
| 10. Click **Next** to enter the **SQL statement** that transforms the data from the source stream. Optionally, click **Run** to see how the data is extracted from the source. | ||
|
|
||
| 11. Click **Save and deploy later** to save the application. You can view and access the application you created on the application overview page. | ||
|
|
||
| .. image:: /images/products/flink/application_landingpage_view.png | ||
| :scale: 50 % | ||
| :alt: Application landing page with a view of the source table, SQL statement, and sink table | ||
|
|
||
| 12. To deploy the application, click **Create deployment**. In the **Create new deployment** dialog: | ||
|
|
||
| * Select the application version to deploy. The default version for the first deployment is **Version: 1**. | ||
| * Select a :doc:`savepoint </docs/products/flink/concepts/savepoints>` if you wish to deploy from a specific state. No savepoints are available for the first application deployment. | ||
| * Toggle **Restart on failure** to automatically restart Flink jobs upon failure. | ||
| * Specify the number of `parallel instances <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution/parallel/>` you require for the task. | ||
| 13. Click **Deploy without a savepoint** to begin the deployment process. | ||
| 14. While deploying, the application status shows **Initializing**. Once deployed, the status changes to **Running**. | ||
|
|
||
| Create SQL applications using Aiven CLI | ||
| ------------------------------------------ | ||
| For information on creating and managing Aiven for Apache Flink application using :doc:`Aiven CLI </docs/tools/cli>`, see :doc:`Manage Aiven for Apache Flink® applications </docs/tools/cli/service/flink>` document. | ||
|
|
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.