Additional information from Kafka output logging

[hartfordfive]

Logstash version

6.1.1 (with X-Pack)

OS

CentOS 7.3

Description

I sometimes encounter issues with some of my pipelines where the kafka output is unable to publish messages to Kafka. For example, when the failures start occurring, messages such as the following start to flood the log file:

[2018-07-16T08:48:35,632][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>2, :failures=>2, :sleep=>0.01}

My issue right now is that there is no indication of which logstash kafka output is failing in this case. Including the pipeline ID and kafka output ID (as well as any other potentially useful details) in the error message would definitely help in the debugging process. Would it be able to add these to the standard log message of the kafka output?

[abdalians]

+1 same issue here.. logstash-6.5.4

[sasubillis]

+1 same issue here with logstash 6.4.2

[memedadexiaobai]

+1

[memedadexiaobai]

+1

[Grimmish]

Not only does it not indicate which pipeline is responsible, but in fact this error might be thrown by a single malformed message that is being rejected by Kafka. In that case, all other messages are still flowing through all pipelines perfectly, but this single message is spamming the log with errors infinitely.

Implementing finite retries is not a solution because (as the docs stress repeatedly) you are then dropping all messages in the event of a REAL Kafka outage.

I need both better detail in the Kafka failure logging and some sort of alternative routing solution for individual messages that cannot be published to Kafka.