diff --git a/_toc.yml b/_toc.yml index d1574c7932..9079230bb2 100644 --- a/_toc.yml +++ b/_toc.yml @@ -654,11 +654,13 @@ entries: title: Integrate with Apache Kafka - file: docs/products/flink/howto/connect-bigquery title: Integrate with Google BigQuery - - file: docs/products/flink/howto/list-flink-applications - title: Aiven for Apache Flink applications + - file: docs/products/flink/howto/create-flink-applications + title: Apache Flink applications entries: - - file: docs/products/flink/howto/create-flink-applications - title: Create Apache Flink applications + - file: docs/products/flink/howto/create-sql-application + title: Create an SQL application + - file: docs/products/flink/howto/create-jar-application + title: Create a JAR application - file: docs/products/flink/howto/manage-flink-applications title: Manage Apache Flink applications - file: docs/products/flink/howto/list-flink-tables @@ -919,10 +921,6 @@ entries: title: Advanced parameters - - - - # -------- GRAFANA -------- - file: docs/products/grafana diff --git a/docs/products/flink/howto/create-flink-applications.rst b/docs/products/flink/howto/create-flink-applications.rst index 9520b1de65..33315aa0e2 100644 --- a/docs/products/flink/howto/create-flink-applications.rst +++ b/docs/products/flink/howto/create-flink-applications.rst @@ -1,61 +1,16 @@ -Create an Aiven for Apache Flink® application -============================================== - -:doc:`Aiven for Flink applications ` in Aiven for Apache Flink® servers as a container that includes everything connected to a Flink job, including source and sink connections and data processing logic. The `Aiven Console `_ provides a guided wizard to help you build and deploy applications, simplifying the process of selecting the source and sink tables, writing data transformation statements, and validating and ingesting data through the interactive query feature. - -This article provides the information required to build and deploy applications on Aiven for Apache Flink service. - -.. note:: - You must set up the :doc:`data service integration ` before building applications. - - -Create an application via the Aiven console --------------------------------------------- - -Follow these steps to build your first Aiven for Flink application: - -1. In the `Aiven Console `_, open the Aiven for Apache Flink service for which you want to create an application. -2. Select **Applications** from the left sidebar and select **Create application** to create your first application. -3. In the **Create new application** screen, enter the name of your application and configure the necessary deployment settings. select **Create application**. -4. Select **Create first version** to create the first version of the application. -5. Select **Add your first source table** to add a source table. - - .. note:: - Since this is your first application, there are currently no other applications where you can import source tables. - -6. In the **Add new source table** screen, - - * Select the **Integrated service** from the drop-down list. - * In the **Table SQL** section, enter the statement that will be used to create the table. - * Optionally, select **Run** to view how data is being pulled from the data source. This could take some time based on the data and the connection. - * Select **Add table**. -7. Select **Next** to add the sink table, and then select **Add your first sink table**. - - .. note:: - Since this is your first application, there are currently no other applications where you can import sink tables. - -8. In the **Add new sink table** screen, - - * Select **Integrated service** from the drop-down list. - * In the **Table SQL** section, enter the statement that will be used to create the table. - * Select **Add table**. -9. Select **Next** to enter the **SQL statement** that transforms the data from the source stream. Optionally, select **Run** to view how data is being pulled from the data source. -10. Select **Save and deploy later** to save the application. You will see the application on the landing page that provides you with an overview of your application. - - .. image:: /images/products/flink/application_landingpage_view.png - :scale: 50 % - :alt: Application landing page with an view of source table, SQL statement and sink table - -11. Select **Create deployment**. On the **Create new deployment** screen, - - * Choose the version you wish to deploy. The default version for the first deployment is **Version: 1**. - * Choose the :doc:`savepoint ` from where you want to deploy. No savepoints are available for the first application deployment. - * Use the toggle for **Restart on failure** to enable or disable the option of automatically restarting a Flink job in case it fails. - * Enter the number of `parallel instances `_ you want to have for the task. -12. Select **Deploy without a savepoint** to deploy your application. -13. The deployment status will show **Initializing: version 1** and then **Running: version 1**. - -Your first application is now created and deployed, and you can view the data related to the actions the application needs to perform in your sink source. +Create and manage Aiven for Apache Flink® applications +=========================================================== + +:doc:`Aiven for Flink applications ` in Aiven for Apache Flink® servers as a container that includes everything connected to a Flink job, including source and sink connections and data processing logic. + +Using the `Aiven Console `_, you can create applications that run SQL queries or deploy custom JARs, catering to diverse data processing requirements. The console's guided wizard simplifies the application configuration process, from selecting source and sink tables for SQL applications to uploading and managing JAR files for custom job execution. + +.. important:: + + Custom JARs for Aiven for Apache Flink is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at sales@Aiven.io. + +.. tableofcontents:: + diff --git a/docs/products/flink/howto/create-jar-application.rst b/docs/products/flink/howto/create-jar-application.rst new file mode 100644 index 0000000000..a34bf29958 --- /dev/null +++ b/docs/products/flink/howto/create-jar-application.rst @@ -0,0 +1,49 @@ +Create a JAR application +======================== +Aiven for Apache Flink® enables you to upload and deploy custom code as a JAR file, enhancing your Flink applications with advanced data processing capabilities. + +.. important:: + + Custom JARs for Aiven for Apache Flink is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at sales@Aiven.io. + +Prerequisite +------------ + +* To enable custom JARs for a new Aiven for Apache Flink service, toggle the feature during service creation. +* For an existing service, in `Aiven Console `_ , select your project and then choose your Aiven for Apache Flink® service. + + * Click **Service settings** on the left sidebar. + * Scroll to the **Advanced configuration** section, and click **Configure**. + * In the **Advanced configuration** screen, click **Add configuration options**, and using the search box find and set ``custom_code`` configuration to **Enabled** position. + +Create and deploy application +--------------------------------- + +1. Access the `Aiven Console `_ and select the Aiven for Apache Flink service where you want to deploy a JAR application. +2. From the left sidebar, click **Applications** and then click **Create application**. +3. In the **Create application** dialog, enter a name for your JAR application, and select **JAR** as the application type from the drop-down. +4. Click **Create application** to proceed. +5. Click **Upload first version** to upload the first version of the application. +6. In the **Upload new version** dialog: + + * Click **Choose file** to select your custom JAR file. + * Select the **Terms of Service** checkbox to indicate your agreement. + * Click **Upload version** to upload your JAR file. + +7. After the upload, you are redirected to the application's overview page. +8. To deploy the application, click **Create deployment**. In the **Create new deployment** dialog: + + * Select the application version to deploy. + * Select a :doc:`savepoint ` if you wish to deploy from a specific state. No savepoints are available for the first application deployment. + * Toggle **Restart on failure** to automatically restart Flink jobs upon failure. + * In the **Program args** field, provide command-line arguments consisting of variables and configurations relevant to your application's logic upon submission. Each argument is limited to 64 characters, with a total limit of 32 separate items. + * Specify the number of `parallel instances `_ you require for the task. + +9. Click **Deploy without a savepoint** to begin the deployment process. +10. While deploying, the application status shows **Initializing**. Once deployed, the status changes to **Running**. + + +Related pages +-------------- + +* :doc:`Manage Aiven for Apache Flink® applications ` diff --git a/docs/products/flink/howto/create-sql-application.rst b/docs/products/flink/howto/create-sql-application.rst new file mode 100644 index 0000000000..72f37aa9db --- /dev/null +++ b/docs/products/flink/howto/create-sql-application.rst @@ -0,0 +1,61 @@ +Create an SQL application +============================ + +In Aiven for Apache Flink®, you can create an SQL application that uses Apache Flink SQL to streamline the process of building data processing pipelines. The SQL application simplifies the process of defining source and sink tables, implementing data processing logic, and managing application deployment. + +Prerequisite +------------ +Before creating applications, configure the :doc:`data service integration ` for seamless integration and data management within your Flink applications. + +Create and deploy application +------------------------------- + +Create an SQL application in Aiven for Apache Flink® using the `Aiven Console `_: + +1. In the `Aiven Console `_, select the Aiven for Apache Flink service where you want to create and deploy a Flink application. +2. From the left sidebar, click **Applications** and then click **Create application**. +3. In the **Create application** dialog, enter the name of your application and select **SQL** as the application type. +4. Click **Create application**. +5. Click **Create first version** to create the first version of the application. +6. Click **Add your first source table** to add a source table. + + .. note:: + As this is your first application, no other applications are available to import source tables. + +7. On the **Add new source table** screen: + + * Use the **Integrated service** drop-down to select the service. + * In the **Table SQL** section, enter the SQL statement to create the source table. + * Optionally, click **Run** to test how data is being retrieved from the data source. This may vary in time based on the data volume and connection speed. + * Click **Add table**. +8. Click **Next** to proceed to adding a sink table and click **Add your first sink table**. + + .. note:: + As this is your first application, no other applications are available to import sink tables. + +9. On the **Add new sink table** screen: + + * Use the **Integrated service** drop-down to select the service. + * In the **Table SQL** section, enter the SQL statement to create the sink table. + * Click **Add table**. +10. Click **Next** to enter the **SQL statement** that transforms the data from the source stream. Optionally, click **Run** to see how the data is extracted from the source. + +11. Click **Save and deploy later** to save the application. You can view and access the application you created on the application overview page. + + .. image:: /images/products/flink/application_landingpage_view.png + :scale: 50 % + :alt: Application landing page with a view of the source table, SQL statement, and sink table + +12. To deploy the application, click **Create deployment**. In the **Create new deployment** dialog: + + * Select the application version to deploy. The default version for the first deployment is **Version: 1**. + * Select a :doc:`savepoint ` if you wish to deploy from a specific state. No savepoints are available for the first application deployment. + * Toggle **Restart on failure** to automatically restart Flink jobs upon failure. + * Specify the number of `parallel instances ` you require for the task. +13. Click **Deploy without a savepoint** to begin the deployment process. +14. While deploying, the application status shows **Initializing**. Once deployed, the status changes to **Running**. + +Create SQL applications using Aiven CLI +------------------------------------------ +For information on creating and managing Aiven for Apache Flink application using :doc:`Aiven CLI `, see :doc:`Manage Aiven for Apache Flink® applications ` document. + diff --git a/docs/products/flink/howto/list-flink-applications.rst b/docs/products/flink/howto/list-flink-applications.rst deleted file mode 100644 index d228344516..0000000000 --- a/docs/products/flink/howto/list-flink-applications.rst +++ /dev/null @@ -1,6 +0,0 @@ -Create and manage Aiven for Apache Flink® applications -====================================================== - -This section will walk you through the steps of setting up and managing your Aiven for Apache Flink® applications. You will learn how to create, configure, and manage your Apache Flink applications. - -.. tableofcontents:: diff --git a/docs/products/flink/howto/manage-flink-applications.rst b/docs/products/flink/howto/manage-flink-applications.rst index d3ab510eae..cd9afb5391 100644 --- a/docs/products/flink/howto/manage-flink-applications.rst +++ b/docs/products/flink/howto/manage-flink-applications.rst @@ -8,21 +8,34 @@ Creating a new version of an application To create a new version of the application deployed, follow these steps: 1. Log in to the `Aiven Console `_, and select your Aiven for Apache Flink® service. -2. Select **Applications** from the left sidebar, and select the application that contains the deployment you want to create a new version. -3. On the application landing screen, select **Create new version**. -4. In the create new version screen, make any desired changes to the create statement, source, or sink tables. -5. Select **Save and deploy later**. You can see the new version listed in the versions drop-down list on the left side of the screen. -6. To deploy the new version of the application, :ref:`stop ` any existing version that is running. -7. Select **Create deployment**, and on the **Create new deployment** screen: - - * Choose the version you want to deploy. - * Choose the savepoint from where you want to deploy. - * Use the toggle for **Restart on failure** to enable or disable the option of automatically restarting a Flink job in case it fails. +2. From the left sidebar, select **Applications**. +3. On the **Applications** landing page, click on the application name for which you want to create a new version. + +For SQL application +````````````````````` +1. Click **Create new version**. +2. In the **Create new version** page, modify the create statement, source, or sink tables as needed. +3. Click **Save and deploy later**. You can see the new version listed in the versions drop-down list. +4. To deploy the new version of the application, :ref:`stop ` any existing version that is running. +5. Click **Create deployment**, and in the **Create new deployment** dialog: + + * Select the version you want to deploy. + * Select the savepoint from where you want to deploy. + * Toggle **Restart on failure** to automatically restart Flink jobs upon failure. * Enter the number of `parallel instances `_ you want to have for the task. + * Click **Deploy from a savepoint** or **Deploy without savepoint** depending on your previous selection. -8. Select **Deploy from a savepoint** or **Deploy without savepoint** depending on your previous selection. +For JAR application +````````````````````` +1. Click **Upload new version**. +2. In the **Upload new version** dialog: + * Click **Choose file** to select your custom JAR file. + * Review and accept the terms of service by checking the box. + * Click **Upload version** to upload your JAR file. +3. In the **Deployment history** you can see the latest version running. + .. _stop-flink-application: Stop application deployment @@ -30,26 +43,24 @@ Stop application deployment To stop a deployment for your Flink application, follow these steps: -1. Select **Applications** from the left sidebar on your Aiven for Apache Flink service, and select the application that contains the deployment you want to stop. -2. On the application landing screen, select **Stop deployment**. -3. On the **Stop deployment** screen, enable the option to **Create a savepoint before stopping** to preserve the current state of the application. - -.. note:: - To stop a deployment without saving the current state of the application, disable the option for **Create a savepoint before stopping** and select **Stop without savepoint**. - -4. Select **Create savepoint & stop** to initiate the stopping process. +1. In your Aiven for Apache Flink service, select **Applications** from the left sidebar. +2. On the **Applications** landing page, click on the application name you want to stop. +3. In the application's overview page, click **Stop deployment**. +4. In the **Stop deployment** dialog, enable the option to **Create a savepoint before stopping** to save the current state of the application. If you want to stop a deployment without saving the current state of the application, disable the option for **Create a savepoint before stopping** and click **Stop without creating savepoint**. +5. Click **Create savepoint & stop** to initiate the stopping process. The application status will display ``Saving_and_stop_requested`` and then ``Finished`` once the stopping process is completed. -Additionally, you can view a history of all the application deployments and statuses by selecting **Deployment history**. +Additionally, the **Deployment history** provides a record of all the application deployments and statuses. Rename application ------------------- To rename an application, follow these steps: -1. Select **Applications** from the left sidebar on your Aiven for Apache Flink service, and select the application you want to rename. -2. On the application landing screen, select the **Application action menu** (ellipsis) located on the right side of the screen, and select **Update application** from the menu options. -3. In the **Update Application** screen, enter the new name for the application and select **Save changes** to confirm the new name and update the application. +1. In your Aiven for Apache Flink service, select **Applications** from the left sidebar. +2. On the **Applications** landing page, click on the application name you want to rename. +3. In the application's overview page, click the **Application action menu (...)** , and click **Update application** from the menu options. +4. In the **Update Application** dialog, enter the new name for the application and select **Save changes** to confirm the new name and update the application. .. _flink-deployment-history: @@ -66,17 +77,18 @@ The **Deployment History** screen provides the following: To view and delete the deployment history of an application, follow these steps: -1. Select **Applications** from the left sidebar on your Aiven for Apache Flink service, and select the application for which you want to view the deployment history. -2. On the application landing screen, select **Deployment History** to view the deployment history. -3. To remove a specific deployment from the history, locate it in the deployment history screen and select the **Delete** icon next to it. +1. In your Aiven for Apache Flink service, select **Applications** from the left sidebar. +2. On the **Applications** landing page, click on the application name for which you want to view the deployment history. +3. In the application landing page, click **Deployment History** to view the deployment history. +4. To remove a specific deployment from the history, locate it in the deployment history page and click the **Delete** icon next to it. -To learn about the different application statuses, see Delete application ------------------- Before deleting an application, it is necessary to remove all associated :ref:`deployment history `. -1. Select **Applications** from the left sidebar on your Aiven for Apache Flink service, and select the application you want to delete. -2. On the application landing screen, select the **Application action menu** (ellipsis) located on the right side of the screen, and select **Delete application** from the menu options. -3. In the **Delete Confirmation** screen, enter the name of the application and select **Confirm** to proceed with the deletion. +1. In your Aiven for Apache Flink service, select **Applications** from the left sidebar. +2. On the **Applications** landing page, click on the application name you want to delete. +3. In the application's overview page, click the **Application action menu (...)**, and click **Delete application** from the menu options. +4. In the **Delete Confirmation** dialog, enter the name of the application and click **Confirm** to proceed with the deletion.