Skip to content

Commit

Permalink
Move section about Logstash to bottom of Kafka output page (#1430) (#…
Browse files Browse the repository at this point in the history
…1435)

(cherry picked from commit b05a3ab)

Co-authored-by: David Kilfoyle <[email protected]>
  • Loading branch information
mergify[bot] and kilfoyle authored Nov 5, 2024
1 parent d93997b commit f7c223b
Showing 1 changed file with 30 additions and 27 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,7 @@

Specify these settings to send data over a secure connection to Kafka. In the {fleet} <<output-settings,Output settings>>, make sure that the Kafka output type is selected.

== Kafka output and using {ls} to index data to {es}

If you are considering using {ls} to ship the data from `kafka` to {es}, please
be aware the structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec in order
to make sure the input doesn't edit the fields and their contents.

The data streams setup by the integrations expect to receive events having the same structure and
field names as they were sent directly from an {agent}.

The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec.

Refer to the <<ls-output-settings,{ls} output for {agent}>> documentation for more details.

[source,yaml]
----
inputs {
kafka {
...
ecs_compatibility => "disabled"
codec => json { ecs_compatibility => "disabled" }
...
}
}
...
----
NOTE: If you plan to use {ls} to modify {agent} output data before it's sent to Kafka, please refer to our <<kafka-output-settings-ls-warning,guidance>> for doing so, further in on this page.

[discrete]
== General settings
Expand Down Expand Up @@ -570,4 +544,33 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[

|===

[[kafka-output-settings-ls-warning]]
== Kafka output and using {ls} to index data to {es}

If you are considering using {ls} to ship the data from `kafka` to {es}, please
be aware the structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec in order
to make sure the input doesn't edit the fields and their contents.

The data streams setup by the integrations expect to receive events having the same structure and
field names as they were sent directly from an {agent}.

The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec.

Refer to the <<ls-output-settings,{ls} output for {agent}>> documentation for more details.

[source,yaml]
----
inputs {
kafka {
...
ecs_compatibility => "disabled"
codec => json { ecs_compatibility => "disabled" }
...
}
}
...
----

:type!:

0 comments on commit f7c223b

Please sign in to comment.