From 380b730dc3d2a00092176ab1c65739fae8b37642 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Sun, 26 Nov 2023 19:45:05 +0100 Subject: [PATCH 1/7] Custom JARs for Aiven for Apache Flink overview --- docs/products/flink/concepts/custom-jars.rst | 29 ++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 docs/products/flink/concepts/custom-jars.rst diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst new file mode 100644 index 0000000000..83038afb44 --- /dev/null +++ b/docs/products/flink/concepts/custom-jars.rst @@ -0,0 +1,29 @@ +Custom JARs in Aiven for Apache Flink® +================================================= + +Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with Custom JARs. This feature aims to expand Flink's capabilities, allowing you to add custom functions that extend beyond the typical SQL-based features. With Custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. + +Why use Custom JARs? +--------------------- +Using Custom JARs in Aiven for Apache Flink offers several key benefits: + +* **Enhanced functionality:** Using Custom JARs, you can add unique features to your Aiven for Apache Flink service that are not available in the standard APIs. +* **Code reusability:** This feature allows you to seamlessly reuse and integrate your existing Java code into your Aiven for Apache Flink applications. +* **Simplified management:** The need to deploy and manage separate Flink clusters is significantly reduced, as custom functionalities can be directly utilized within the Aiven for Apache Flink application. + +Use cases +--------- + +Custom JARs can be applied in various scenarios, including but not limited to: + +* **Custom data processing and enrichment:** Custom JARs facilitate the processing and enrichment of data from sources not natively supported by Flink's core APIs. This includes implementing unique data processing logic and applying custom functions for data transformation and enrichment. +* **Connector customization:** With Custom JARs, it is possible to create and integrate unique connectors tailored to specific data integration needs. This feature enables users to develop and deploy connectors that are not available in Flink's standard offerings, providing a more flexible and customized data flow within their applications. + + +Related reading +----------------- + +How to use Custom JARs in Aiven for Apache Flink +------------------------------------------------ + +You can easily manage your Custom JARs within the Aiven for Apache Flink application through the Aiven console. The process involves uploading a JAR file, which can then be deployed and utilized within the Flink application. This integration allows for a seamless experience in enhancing and customizing the application's data processing capabilities. From e6b53e7d4c9dffe650299d5a5e16e0db9af3adf2 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Sun, 26 Nov 2023 20:00:18 +0100 Subject: [PATCH 2/7] Update TOC --- _toc.yml | 1 + docs/products/flink/concepts/custom-jars.rst | 14 ++++++++------ 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/_toc.yml b/_toc.yml index 9270a871da..99e711b2fc 100644 --- a/_toc.yml +++ b/_toc.yml @@ -619,6 +619,7 @@ entries: - file: docs/products/flink/concepts/flink-applications title: Aiven Flink applications - file: docs/products/flink/concepts/supported-syntax-sql-editor + - file: docs/products/flink/concepts/custom-jars - file: docs/products/flink/concepts/tables title: Flink tables - file: docs/products/flink/concepts/checkpoints diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst index 83038afb44..ef8069d76c 100644 --- a/docs/products/flink/concepts/custom-jars.rst +++ b/docs/products/flink/concepts/custom-jars.rst @@ -3,16 +3,21 @@ Custom JARs in Aiven for Apache Flink® Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with Custom JARs. This feature aims to expand Flink's capabilities, allowing you to add custom functions that extend beyond the typical SQL-based features. With Custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. +What are Custom JARs? +------------------------- +Java ARchive (JAR) files are package files used in Java to aggregate many class files, metadata, and resources like text and images into a single file for distribution. Custom JARs are Java Archive files containing custom code and resources developed for specific functionalities outside standard Java or Apache Flink libraries. + + Why use Custom JARs? --------------------- Using Custom JARs in Aiven for Apache Flink offers several key benefits: * **Enhanced functionality:** Using Custom JARs, you can add unique features to your Aiven for Apache Flink service that are not available in the standard APIs. * **Code reusability:** This feature allows you to seamlessly reuse and integrate your existing Java code into your Aiven for Apache Flink applications. -* **Simplified management:** The need to deploy and manage separate Flink clusters is significantly reduced, as custom functionalities can be directly utilized within the Aiven for Apache Flink application. +* **Simplified management:** The need to deploy and manage separate Flink service is significantly reduced, as custom functionalities can be directly used within the Aiven for Apache Flink application. Use cases ---------- +-------------- Custom JARs can be applied in various scenarios, including but not limited to: @@ -22,8 +27,5 @@ Custom JARs can be applied in various scenarios, including but not limited to: Related reading ----------------- +* How to use Custom JARs in Aiven for Apache Flink aaplication -How to use Custom JARs in Aiven for Apache Flink ------------------------------------------------- - -You can easily manage your Custom JARs within the Aiven for Apache Flink application through the Aiven console. The process involves uploading a JAR file, which can then be deployed and utilized within the Flink application. This integration allows for a seamless experience in enhancing and customizing the application's data processing capabilities. From 078f6ff5d97ff1b3b5370c3eaeea40772b884c6e Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Mon, 27 Nov 2023 07:35:43 +0100 Subject: [PATCH 3/7] fixed vale errors --- .github/vale/dicts/aiven.dic | 1 + docs/products/flink/concepts/custom-jars.rst | 12 ++++++------ 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/.github/vale/dicts/aiven.dic b/.github/vale/dicts/aiven.dic index adda1c65c8..5c8d0c9f5f 100644 --- a/.github/vale/dicts/aiven.dic +++ b/.github/vale/dicts/aiven.dic @@ -121,6 +121,7 @@ io iops IPsec iptables +JARs java_lang_Memory Java JMX diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst index ef8069d76c..549ddcd4ba 100644 --- a/docs/products/flink/concepts/custom-jars.rst +++ b/docs/products/flink/concepts/custom-jars.rst @@ -1,18 +1,18 @@ Custom JARs in Aiven for Apache Flink® ================================================= -Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with Custom JARs. This feature aims to expand Flink's capabilities, allowing you to add custom functions that extend beyond the typical SQL-based features. With Custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. +Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with custom JARs. This feature aims to expand the capabilities of Aiven for Apache Flink, allowing you to add custom functions that extend beyond the typical SQL-based features. With Custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. -What are Custom JARs? +What are custom JARs? ------------------------- Java ARchive (JAR) files are package files used in Java to aggregate many class files, metadata, and resources like text and images into a single file for distribution. Custom JARs are Java Archive files containing custom code and resources developed for specific functionalities outside standard Java or Apache Flink libraries. -Why use Custom JARs? +Why use custom JARs? --------------------- Using Custom JARs in Aiven for Apache Flink offers several key benefits: -* **Enhanced functionality:** Using Custom JARs, you can add unique features to your Aiven for Apache Flink service that are not available in the standard APIs. +* **Enhanced functionality:** Using custom JARs, you can extend the native capabilities of your Aiven for Apache Flink service, incorporating functionalities that go beyond the core Flink APIs. * **Code reusability:** This feature allows you to seamlessly reuse and integrate your existing Java code into your Aiven for Apache Flink applications. * **Simplified management:** The need to deploy and manage separate Flink service is significantly reduced, as custom functionalities can be directly used within the Aiven for Apache Flink application. @@ -22,10 +22,10 @@ Use cases Custom JARs can be applied in various scenarios, including but not limited to: * **Custom data processing and enrichment:** Custom JARs facilitate the processing and enrichment of data from sources not natively supported by Flink's core APIs. This includes implementing unique data processing logic and applying custom functions for data transformation and enrichment. -* **Connector customization:** With Custom JARs, it is possible to create and integrate unique connectors tailored to specific data integration needs. This feature enables users to develop and deploy connectors that are not available in Flink's standard offerings, providing a more flexible and customized data flow within their applications. +* **Connector customization:** With custom JARs, it is possible to create and integrate unique connectors tailored to specific data integration needs. This feature enables users to develop and deploy connectors that are not available in Flink's standard offerings, providing a more flexible and customized data flow within their applications. Related reading ----------------- -* How to use Custom JARs in Aiven for Apache Flink aaplication +* How to use custom JARs in Aiven for Apache Flink application From 106060ff490657667def5db5e1924b29df6af375 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Fri, 15 Dec 2023 16:41:22 +0100 Subject: [PATCH 4/7] Addressed review comments and updated Flink application topic --- _toc.yml | 1 + docs/products/flink/concepts/custom-jars.rst | 11 ++-- .../flink/concepts/flink-applications.rst | 58 ++++++++++--------- 3 files changed, 37 insertions(+), 33 deletions(-) diff --git a/_toc.yml b/_toc.yml index 99e711b2fc..ad3f40e019 100644 --- a/_toc.yml +++ b/_toc.yml @@ -620,6 +620,7 @@ entries: title: Aiven Flink applications - file: docs/products/flink/concepts/supported-syntax-sql-editor - file: docs/products/flink/concepts/custom-jars + title: Custom JARs - file: docs/products/flink/concepts/tables title: Flink tables - file: docs/products/flink/concepts/checkpoints diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst index 549ddcd4ba..d9d9c99e9f 100644 --- a/docs/products/flink/concepts/custom-jars.rst +++ b/docs/products/flink/concepts/custom-jars.rst @@ -1,11 +1,11 @@ Custom JARs in Aiven for Apache Flink® ================================================= -Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with custom JARs. This feature aims to expand the capabilities of Aiven for Apache Flink, allowing you to add custom functions that extend beyond the typical SQL-based features. With Custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. +Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with custom JARs. This feature expands the capabilities of Aiven for Apache Flink, allowing you to add custom capabilities that extend beyond the default SQL features. With custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. What are custom JARs? ------------------------- -Java ARchive (JAR) files are package files used in Java to aggregate many class files, metadata, and resources like text and images into a single file for distribution. Custom JARs are Java Archive files containing custom code and resources developed for specific functionalities outside standard Java or Apache Flink libraries. +Custom JARs are specialized Java Archive files containing code and resources for functionalities beyond standard Java or Apache Flink libraries. The capabilities of any custom JAR are defined by your organization and use cases, allowing for tailored solutions that meet specific technical requirements and objectives. Why use custom JARs? @@ -13,8 +13,9 @@ Why use custom JARs? Using Custom JARs in Aiven for Apache Flink offers several key benefits: * **Enhanced functionality:** Using custom JARs, you can extend the native capabilities of your Aiven for Apache Flink service, incorporating functionalities that go beyond the core Flink APIs. -* **Code reusability:** This feature allows you to seamlessly reuse and integrate your existing Java code into your Aiven for Apache Flink applications. -* **Simplified management:** The need to deploy and manage separate Flink service is significantly reduced, as custom functionalities can be directly used within the Aiven for Apache Flink application. +* **Code reuse:** This feature allows you to seamlessly reuse and integrate your existing Java code into your Aiven for Apache Flink applications. +* **Simplified management:** Aiven handles the complexities of hosting and operating a Flink service. This streamlined approach requires you only to upload and deploy the JAR file, enabling direct use of its functionalities within the Aiven for Apache Flink application. + Use cases -------------- @@ -22,7 +23,7 @@ Use cases Custom JARs can be applied in various scenarios, including but not limited to: * **Custom data processing and enrichment:** Custom JARs facilitate the processing and enrichment of data from sources not natively supported by Flink's core APIs. This includes implementing unique data processing logic and applying custom functions for data transformation and enrichment. -* **Connector customization:** With custom JARs, it is possible to create and integrate unique connectors tailored to specific data integration needs. This feature enables users to develop and deploy connectors that are not available in Flink's standard offerings, providing a more flexible and customized data flow within their applications. + Related reading diff --git a/docs/products/flink/concepts/flink-applications.rst b/docs/products/flink/concepts/flink-applications.rst index 8b29ceb102..775be64f2b 100644 --- a/docs/products/flink/concepts/flink-applications.rst +++ b/docs/products/flink/concepts/flink-applications.rst @@ -1,28 +1,31 @@ Aiven for Apache Flink® applications ==================================== -An Aiven for Apache Flink® Application is an abstraction layer on top of Apache Flink SQL that includes all the elements related to a Flink job to help build your data processing pipeline. An application contains the definition of source and sink tables, data processing logic, deployment parameters, and other necessary metadata. +An Aiven for Apache Flink® Application is an abstraction layer that simplifies building data processing pipelines in Apache Flink. It supports two types of applications: + +* **SQL applications:** Built on top of Apache Flink SQL, these applications include elements like source and sink table definitions, data processing logic using SQL, deployment parameters, and other necessary metadata. +* **JAR applications:** This type allows users to upload and deploy custom JAR files, enabling the integration of custom code and resources for specialized functionalities beyond the standard Flink capabilities. :doc:`Applications ` are the starting point for running an Apache Flink job within the Aiven managed service. The `Aiven Console `_ provides a user-friendly, guided wizard to help you build and deploy applications, create source and sink tables, write transformation statements, and validate and ingest data using the interactive query feature. -Each application created and deployed in the Aiven for Apache Flink service performs a specific data transformation task allowing you to map and manage different workflows separately within the Aiven for Apache Flink service. For example, you can create an application to get user IDs from one topic and publish them on another topic. You can run multiple applications within the Aiven for Apache Flink service. +Each application whether SQL or JAR-based, is designed to perform specific data transformation tasks, allowing you to map and manage different workflows within the Aiven for Apache Flink service. For example, you can create a SQL application to process user IDs from one topic and publish them on another, or use a custom JAR application for more specialized data processing needs. -Applications significantly improve the developer experience and simplify the development and deployment of Flink applications. Applications are **automatically versioned** on every edit of the underline definition (tables, or transformation SQL), allowing you to experiment with new transformations and revert to a previously stored definition if the result of the edits doesn't meet expectations. +Applications significantly improve the developer experience and simplify the development and deployment of Flink applications. Applications are **automatically versioned** on every edit of the underline definition (tables, transformation SQL, or custom JARs), allowing you to experiment with new transformations and revert to a previously stored definition if the result of the edits doesn't meet expectations. .. seealso:: For infromation on how to create applications, see :doc:`Create an Aiven for Apache Flink® application ` -Features of Aiven for Apache Flink applications +Application features ----------------------------------------------- Some of key feature of Aiven for Flink applications include: -* **Intuitive Interface**: The `Aiven Console `_ provides a user-friendly, guided wizard to help you quickly build and deploy your applications. -* **Source and sink tables** definition with **SQL autocomplete**: You can create source and sink tables, the SQL autocomplete feature allows you to start from a specific set of pre-filled parameters depending on the type of connector. -* **Interactive query**: The interactive query feature allows you to validate and preview your data in the table or SQL transformation before deploying it. +* **Intuitive interface**: The `Aiven Console `_ provides a user-friendly, guided wizard for building and deploying applications. +* **Source and sink tables** definition with **SQL autocomplete**: For SQL applications, create source and sink tables with guidance from SQL autocomplete based on connector types. +* **Interactive query**: For SQL applications, the interactive query feature allows you to validate and preview your data in the table or SQL transformation before deploying it. * **Start and stop**: You can start and stop applications stop and start anytime. -* **Versioning**: By creating a new application version every time you edit the table or data transformation definition, you can keep track of changes and have the ability to revert to a previous version if needed. +* **Versioning**: Creating a new application version after editing the table or data transformation definition allows you to track changes and revert to a previous version when needed. * **Savepoints**: You can stop your application with a savepoint, allowing you to restart it from the previous state at a later time. * **Scalability:** You can parallelize the workload into multiple tasks and executing them concurrently in a cluster @@ -32,29 +35,28 @@ Limitations * **Concurrent applications:** You can create and run up to 4 concurrent applications in the Aiven for Apache Flink service. * **Savepoints:** You can store up to 10 savepoints history per application deployment, allowing you to start from a previous state if necessary. If you exceed this limit, you must clear the savepoints history before creating a new application deployment. -Aiven for Apache Flink application status +Application status ----------------------------------------- -Flink applications can have different statuses based on their current execution status. The status of a Flink application is closely related to the state of its savepoints. The most common statuses you will see in the application on Aiven Console include: +Flink applications can have different statuses based on their execution and savepoint state. Below are the most common statuses you may encounter: -* **CANCELED:** the application has been stopped without a savepoint. -* **CREATED:** the application has been created but has yet to start running. -* **FAILED:** the application has failed to run. -* **FINISHED:** the application has been stopped with a savepoint. -* **INITIALIZING:** the application is in the process of being initialized. -* **RUNNING:** the application is currently running and can be viewed with its version. -* **RESTARTING:** the application is in the process of being restarted. -* **SAVING_AND_STOP:** the application version is being saved in a savepoint, and the application is being stopped. +* **CANCELED**: Your application has stopped without creating a savepoint. +* **CREATED**: Your application has been created but has yet to start running. +* **FAILED**: Your application attempted to run but was unsuccessful. +* **FINISHED**: Your application has completed its execution and stopped, with a savepoint created. +* **INITIALIZING**: Your application is in the process of being initialized and preparing to run. +* **RUNNING**: Your application is actively running. You can view its current version. +* **RESTARTING**: Your application is being restarted. +* **SAVING_AND_STOP**: Your application is currently saving its current state in a savepoint and then stopping. Other statuses and transient statuses include: -* **CANCELLING_REQUESTED:** the application is in the process of being stopped without a savepoint. -* **CANCELLING:** the application is in the process of being cancelled. -* **DELETE_REQUESTED:** application has been requested to be deleted. -* **DELETING:** the application is in the process of being deleted. -* **FAILING:** the application is in the process of failing. -* **RESTARTING:** the application is in the process of being restarted. -* **SAVING:** the application is in the process of creating a savepoint. -* **SAVING_AND_STOP_REQUESTED:** the application is in the process of creating a savepoint and stopping. -* **SUSPENDED:** the application has been suspended. - +* **CANCELLING_REQUESTED**: A stop request for your application has been made without creating a savepoint. +* **CANCELLING**: Your application is currently being stopped. +* **DELETE_REQUESTED**: A request to delete your application has been initiated. +* **DELETING**: Your application is in the process of being removed. +* **FAILING**: Your application is encountering issues and failing. +* **RESTARTING**: Your application is undergoing a restart process. +* **SAVING**: Your application is creating a savepoint. +* **SAVING_AND_STOP_REQUESTED**: A request has been made to save the current state of your application in a savepoint and then stop it. +* **SUSPENDED**: Your application has been suspended. From 352f321e726639e81949badecc73889e7bcc334f Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Wed, 17 Jan 2024 11:39:35 +0100 Subject: [PATCH 5/7] updated intro text --- docs/products/flink/concepts/custom-jars.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst index d9d9c99e9f..9a9f05449a 100644 --- a/docs/products/flink/concepts/custom-jars.rst +++ b/docs/products/flink/concepts/custom-jars.rst @@ -1,7 +1,7 @@ Custom JARs in Aiven for Apache Flink® ================================================= -Aiven for Apache Flink® allows you to upload, deploy, and manage your own Java code with custom JARs. This feature expands the capabilities of Aiven for Apache Flink, allowing you to add custom capabilities that extend beyond the default SQL features. With custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. +Aiven for Apache Flink enables you to upload, deploy, and manage your own Java code as custom JARs within a JAR application. This feature expands the capabilities of Aiven for Apache Flink, allowing you to add custom capabilities that extend beyond the default SQL features. With custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application. What are custom JARs? ------------------------- From 9c8a95c19484174634aad8e07bdccd93eff9eb9b Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Thu, 18 Jan 2024 15:25:03 +0100 Subject: [PATCH 6/7] added cross reference to howto page --- docs/products/flink/concepts/custom-jars.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst index 9a9f05449a..69dfc09df8 100644 --- a/docs/products/flink/concepts/custom-jars.rst +++ b/docs/products/flink/concepts/custom-jars.rst @@ -26,7 +26,7 @@ Custom JARs can be applied in various scenarios, including but not limited to: -Related reading +Related pages ----------------- -* How to use custom JARs in Aiven for Apache Flink application +* :doc:`How to use custom JARs in Aiven for Apache Flink application ` From b3d2f29024af26b2576dd097f47563d307412f9c Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Thu, 18 Jan 2024 15:34:00 +0100 Subject: [PATCH 7/7] removed link --- docs/products/flink/concepts/custom-jars.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/products/flink/concepts/custom-jars.rst b/docs/products/flink/concepts/custom-jars.rst index 69dfc09df8..4bf7f625cf 100644 --- a/docs/products/flink/concepts/custom-jars.rst +++ b/docs/products/flink/concepts/custom-jars.rst @@ -26,7 +26,4 @@ Custom JARs can be applied in various scenarios, including but not limited to: -Related pages ------------------ -* :doc:`How to use custom JARs in Aiven for Apache Flink application `