Skip to content
This repository has been archived by the owner on Jan 29, 2024. It is now read-only.

Custom JARs for Aiven for Apache Flink overview #2291

Merged
merged 8 commits into from
Jan 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/vale/dicts/aiven.dic
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ io
iops
IPsec
iptables
JARs
java_lang_Memory
Java
JMX
Expand Down
2 changes: 2 additions & 0 deletions _toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -630,6 +630,8 @@ entries:
- file: docs/products/flink/concepts/flink-applications
title: Aiven Flink applications
- file: docs/products/flink/concepts/supported-syntax-sql-editor
- file: docs/products/flink/concepts/custom-jars
title: Custom JARs
- file: docs/products/flink/concepts/tables
title: Flink tables
- file: docs/products/flink/concepts/checkpoints
Expand Down
29 changes: 29 additions & 0 deletions docs/products/flink/concepts/custom-jars.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
Custom JARs in Aiven for Apache Flink®
=================================================

Aiven for Apache Flink enables you to upload, deploy, and manage your own Java code as custom JARs within a JAR application. This feature expands the capabilities of Aiven for Apache Flink, allowing you to add custom capabilities that extend beyond the default SQL features. With custom JARs, you can swiftly develop and maximize the potential of your Aiven for Flink application.

What are custom JARs?

Check failure on line 6 in docs/products/flink/concepts/custom-jars.rst

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/products/flink/concepts/custom-jars.rst#L6

[Aiven.capitalization_headings] 'What are custom JARs?' should be in sentence case
Raw output
{"message": "[Aiven.capitalization_headings] 'What are custom JARs?' should be in sentence case", "location": {"path": "docs/products/flink/concepts/custom-jars.rst", "range": {"start": {"line": 6, "column": 1}}}, "severity": "ERROR"}
-------------------------
Custom JARs are specialized Java Archive files containing code and resources for functionalities beyond standard Java or Apache Flink libraries. The capabilities of any custom JAR are defined by your organization and use cases, allowing for tailored solutions that meet specific technical requirements and objectives.


Why use custom JARs?

Check failure on line 11 in docs/products/flink/concepts/custom-jars.rst

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/products/flink/concepts/custom-jars.rst#L11

[Aiven.capitalization_headings] 'Why use custom JARs?' should be in sentence case
Raw output
{"message": "[Aiven.capitalization_headings] 'Why use custom JARs?' should be in sentence case", "location": {"path": "docs/products/flink/concepts/custom-jars.rst", "range": {"start": {"line": 11, "column": 1}}}, "severity": "ERROR"}
---------------------
Using Custom JARs in Aiven for Apache Flink offers several key benefits:

* **Enhanced functionality:** Using custom JARs, you can extend the native capabilities of your Aiven for Apache Flink service, incorporating functionalities that go beyond the core Flink APIs.
* **Code reuse:** This feature allows you to seamlessly reuse and integrate your existing Java code into your Aiven for Apache Flink applications.
* **Simplified management:** Aiven handles the complexities of hosting and operating a Flink service. This streamlined approach requires you only to upload and deploy the JAR file, enabling direct use of its functionalities within the Aiven for Apache Flink application.


Use cases
--------------

Custom JARs can be applied in various scenarios, including but not limited to:

* **Custom data processing and enrichment:** Custom JARs facilitate the processing and enrichment of data from sources not natively supported by Flink's core APIs. This includes implementing unique data processing logic and applying custom functions for data transformation and enrichment.

Check failure on line 25 in docs/products/flink/concepts/custom-jars.rst

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/products/flink/concepts/custom-jars.rst#L25

[Aiven.aiven_spelling] 'Flink's' does not seem to be a recognised word
Raw output
{"message": "[Aiven.aiven_spelling] 'Flink's' does not seem to be a recognised word", "location": {"path": "docs/products/flink/concepts/custom-jars.rst", "range": {"start": {"line": 25, "column": 146}}}, "severity": "ERROR"}




58 changes: 30 additions & 28 deletions docs/products/flink/concepts/flink-applications.rst
Original file line number Diff line number Diff line change
@@ -1,28 +1,31 @@
Aiven for Apache Flink® applications
====================================

An Aiven for Apache Flink® Application is an abstraction layer on top of Apache Flink SQL that includes all the elements related to a Flink job to help build your data processing pipeline. An application contains the definition of source and sink tables, data processing logic, deployment parameters, and other necessary metadata.
An Aiven for Apache Flink® Application is an abstraction layer that simplifies building data processing pipelines in Apache Flink. It supports two types of applications:

* **SQL applications:** Built on top of Apache Flink SQL, these applications include elements like source and sink table definitions, data processing logic using SQL, deployment parameters, and other necessary metadata.
* **JAR applications:** This type allows users to upload and deploy custom JAR files, enabling the integration of custom code and resources for specialized functionalities beyond the standard Flink capabilities.

:doc:`Applications </docs/products/flink/howto/create-flink-applications>` are the starting point for running an Apache Flink job within the Aiven managed service. The `Aiven Console <https://console.aiven.io/>`_ provides a user-friendly, guided wizard to help you build and deploy applications, create source and sink tables, write transformation statements, and validate and ingest data using the interactive query feature.

Each application created and deployed in the Aiven for Apache Flink service performs a specific data transformation task allowing you to map and manage different workflows separately within the Aiven for Apache Flink service. For example, you can create an application to get user IDs from one topic and publish them on another topic. You can run multiple applications within the Aiven for Apache Flink service.
Each application whether SQL or JAR-based, is designed to perform specific data transformation tasks, allowing you to map and manage different workflows within the Aiven for Apache Flink service. For example, you can create a SQL application to process user IDs from one topic and publish them on another, or use a custom JAR application for more specialized data processing needs.

Applications significantly improve the developer experience and simplify the development and deployment of Flink applications. Applications are **automatically versioned** on every edit of the underline definition (tables, or transformation SQL), allowing you to experiment with new transformations and revert to a previously stored definition if the result of the edits doesn't meet expectations.
Applications significantly improve the developer experience and simplify the development and deployment of Flink applications. Applications are **automatically versioned** on every edit of the underline definition (tables, transformation SQL, or custom JARs), allowing you to experiment with new transformations and revert to a previously stored definition if the result of the edits doesn't meet expectations.

.. seealso::
For infromation on how to create applications, see :doc:`Create an Aiven for Apache Flink® application </docs/products/flink/howto/create-flink-applications>`


Features of Aiven for Apache Flink applications
Application features
-----------------------------------------------

Some of key feature of Aiven for Flink applications include:

* **Intuitive Interface**: The `Aiven Console <https://console.aiven.io/>`_ provides a user-friendly, guided wizard to help you quickly build and deploy your applications.
* **Source and sink tables** definition with **SQL autocomplete**: You can create source and sink tables, the SQL autocomplete feature allows you to start from a specific set of pre-filled parameters depending on the type of connector.
* **Interactive query**: The interactive query feature allows you to validate and preview your data in the table or SQL transformation before deploying it.
* **Intuitive interface**: The `Aiven Console <https://console.aiven.io/>`_ provides a user-friendly, guided wizard for building and deploying applications.
* **Source and sink tables** definition with **SQL autocomplete**: For SQL applications, create source and sink tables with guidance from SQL autocomplete based on connector types.
* **Interactive query**: For SQL applications, the interactive query feature allows you to validate and preview your data in the table or SQL transformation before deploying it.
* **Start and stop**: You can start and stop applications stop and start anytime.
* **Versioning**: By creating a new application version every time you edit the table or data transformation definition, you can keep track of changes and have the ability to revert to a previous version if needed.
* **Versioning**: Creating a new application version after editing the table or data transformation definition allows you to track changes and revert to a previous version when needed.
* **Savepoints**: You can stop your application with a savepoint, allowing you to restart it from the previous state at a later time.
* **Scalability:** You can parallelize the workload into multiple tasks and executing them concurrently in a cluster

Expand All @@ -32,29 +35,28 @@ Limitations
* **Concurrent applications:** You can create and run up to 4 concurrent applications in the Aiven for Apache Flink service.
* **Savepoints:** You can store up to 10 savepoints history per application deployment, allowing you to start from a previous state if necessary. If you exceed this limit, you must clear the savepoints history before creating a new application deployment.

Aiven for Apache Flink application status
Application status
-----------------------------------------

Flink applications can have different statuses based on their current execution status. The status of a Flink application is closely related to the state of its savepoints. The most common statuses you will see in the application on Aiven Console include:
Flink applications can have different statuses based on their execution and savepoint state. Below are the most common statuses you may encounter:

* **CANCELED:** the application has been stopped without a savepoint.
* **CREATED:** the application has been created but has yet to start running.
* **FAILED:** the application has failed to run.
* **FINISHED:** the application has been stopped with a savepoint.
* **INITIALIZING:** the application is in the process of being initialized.
* **RUNNING:** the application is currently running and can be viewed with its version.
* **RESTARTING:** the application is in the process of being restarted.
* **SAVING_AND_STOP:** the application version is being saved in a savepoint, and the application is being stopped.
* **CANCELED**: Your application has stopped without creating a savepoint.
* **CREATED**: Your application has been created but has yet to start running.
* **FAILED**: Your application attempted to run but was unsuccessful.
* **FINISHED**: Your application has completed its execution and stopped, with a savepoint created.
* **INITIALIZING**: Your application is in the process of being initialized and preparing to run.
* **RUNNING**: Your application is actively running. You can view its current version.
* **RESTARTING**: Your application is being restarted.
* **SAVING_AND_STOP**: Your application is currently saving its current state in a savepoint and then stopping.

Other statuses and transient statuses include:

* **CANCELLING_REQUESTED:** the application is in the process of being stopped without a savepoint.
* **CANCELLING:** the application is in the process of being cancelled.
* **DELETE_REQUESTED:** application has been requested to be deleted.
* **DELETING:** the application is in the process of being deleted.
* **FAILING:** the application is in the process of failing.
* **RESTARTING:** the application is in the process of being restarted.
* **SAVING:** the application is in the process of creating a savepoint.
* **SAVING_AND_STOP_REQUESTED:** the application is in the process of creating a savepoint and stopping.
* **SUSPENDED:** the application has been suspended.

* **CANCELLING_REQUESTED**: A stop request for your application has been made without creating a savepoint.
* **CANCELLING**: Your application is currently being stopped.
* **DELETE_REQUESTED**: A request to delete your application has been initiated.
* **DELETING**: Your application is in the process of being removed.
* **FAILING**: Your application is encountering issues and failing.
* **RESTARTING**: Your application is undergoing a restart process.
* **SAVING**: Your application is creating a savepoint.
* **SAVING_AND_STOP_REQUESTED**: A request has been made to save the current state of your application in a savepoint and then stop it.
* **SUSPENDED**: Your application has been suspended.
Loading