in_kafka: add parser support and improve performance #9726
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lyhgo
requested review from
edsiper,
leonardo-albertovich,
fujimotos and
koleini
as code owners
When processing record, the in_kafka plugin lack of
parser ability, This commit added support to
parse payload. It has been tested with kafka.
When processing record, the in_kafka plugin currently
will commit every single message poll from kafka.
which is good at normal case. But with this default
behavier, the performance is strictly limited. This
commit has add auto_commit as an option when
the performance matter more. It has been tested
with kafka. There is a sample benchmark as following.
topic_size topic_offset time_cost
before 251 MB 506701 1m30s
after 251 MB 506701 10s
Signed-off-by: luoyy8 <[email protected]>
Signed-off-by: luoyy8 <[email protected]>
|
#7991 Related issue |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
in_kafka: add parser support and improve performance
When processing records, the in_kafka plugin lacks
parser ability, This commit added support to
parse payload. It has been tested with Kafka.
When processing the record, the in_kafka plugin currently
will commit every single message poll from kafka.
which is good in normal cases. But with this default
behavior, the performance is strictly limited. This
commit has added auto_commit as an option when
the performance matters more. It has been tested
with Kafka. There is a sample benchmark as follows.
Signed-off-by: luoyy8 [email protected]