Skip to content

Commit

Permalink
introduction rewritten
Browse files Browse the repository at this point in the history
  • Loading branch information
Arek Burdach committed Oct 4, 2024
1 parent ed60727 commit 525b6ee
Showing 1 changed file with 8 additions and 9 deletions.
17 changes: 8 additions & 9 deletions docs/docs/flink-writes.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,15 +374,14 @@ orphan files that are old enough.

# Flink Writes (SinkV2 based implementation)

The [SinkV2 interface](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction)
was introduced in Flink 1.15.
The previous [SinkV1 interface](https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API)
had some limitations - for example it created a lot of small files when writing to it. This problem is called
the `small-file-compaction` problem in
the [FLIP-191 document](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction).
The default `FlinkSink` implementation available in `iceberg-flink` module builds its own chain of `StreamOperator`s terminated by `DiscardingSink`.
However, in the same module, there is also `IcebergSink` which is based on the SinkV2 API.
The SinkV2 based `IcebergSink` is currently an experimental feature.
At the time when the current default, `FlinkSink` implementation was created, Flink Sink's interface had some
limitations that were not acceptable for the Iceberg tables purpose. Due to these limitations, `FlinkSink` is based
on a custom chain of `StreamOperator`s terminated by `DiscardingSink`.

In the Flink 1.15 version, [SinkV2 interface](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction)
was introduced. This interface is used in the new, `IcebergSink` implementation that is also available in the `iceberg-flink` module.
The new implementation will be a base for further work on features such as [table maintenance](maintenance.md).
The SinkV2 based implementation is currently an experimental feature so please use it with caution.

## Writing with SQL

Expand Down

0 comments on commit 525b6ee

Please sign in to comment.