MINOR: Adds possibility to use custom FilenameCreator #731
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
When writing to S3 there is no possibility to decide on the actual filename that ends up in the bucket. Sometimes, we want to be able to do this to make it more clear on what the file contains and to have more control.
I have described it in this: #631 issue
Solution
I have added an interface called FilenameCreator. This interface can be implemented on a custom class and added as config. If none is provided we fall back to TopicPartitionFilenameCreator that preserves the way that file names where created before this change. So no breaking changes.
Does this solution apply anywhere else?
If yes, where?
TopicPartitionWriter now uses a fileName creator, but the change does not affect previous behaviour.
S3SinkConnectorConfig now has a new key: s3.filename.creator.class for specifying your own FilenameCreator
Test Strategy
Changed existing tests to use new FilenameCreator
Testing done:
Release Plan
Merging to master