Implement a strategy to handle OOM in direct memory#475

Open

andsel wants to merge 19 commits intologstash-plugins:mainfrom andsel:mitigate_thundering_herd_2

+545-16

Commits on Sep 27, 2023

Avoid to run Beats parser and Beats protocol handler in separate executors group (beatsHandlerExecutorGroup)
andsel
committed
Ported core part of original logstash-plugins#410 PR
andsel
committed
Added pull way of fetching data
andsel
committed
Extracted the pull mode part into a separate handler
andsel
committed
Update the flow control handler to avoid new reads if the channel becomes not writable, to excert backpressure to the sender system
andsel
committed
Separated the logic to drop incoming connections into specific handler
andsel
committed
Fist draft of the integration test
andsel
committed
Removed tests using EmbeddedChannel because doesn't manage the writeable status due to that offload every message to the outbound list
andsel
committed
Reshaped the asynch code to be more linear
andsel
committed
Covered the number of connection limiter with unit test
andsel
committed
Pessimistic remediation, when a direct OOM happens close the channel
andsel
committed
Removed from the log string any reference to Filebeat
andsel
committed
Raised up the log level level when dropping connections becuase of thundering and going to OOM condition
andsel
committed
Better actionable suggestion to user in case of OOM

andsel
and
jsvd
committed
Updated OOMConnectionCloser to monitor the consumption of memory also during the read opeartion and not only on exception
andsel
committed
Re-introduce the beats handlers worker group to separata the Beats protocol processing from the boss group that accepts and listed to new sockets
andsel
committed
Added feature flag named protect_direct_memory to control the usage of OOM checking or not. Enabled by default.
andsel
committed
Throw a configuration error if Netty reserved direct memory is not anough (256MB)
andsel
committed
Add missed shutdown of beat's worker loop
andsel
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a strategy to handle OOM in direct memory#475

Commits on Sep 27, 2023

Avoid to run Beats parser and Beats protocol handler in separate executors group (beatsHandlerExecutorGroup)

Ported core part of original logstash-plugins#410 PR

Added pull way of fetching data

Extracted the pull mode part into a separate handler

Update the flow control handler to avoid new reads if the channel becomes not writable, to excert backpressure to the sender system

Separated the logic to drop incoming connections into specific handler

Fist draft of the integration test

Removed tests using EmbeddedChannel because doesn't manage the writeable status due to that offload every message to the outbound list

Reshaped the asynch code to be more linear

Covered the number of connection limiter with unit test

Pessimistic remediation, when a direct OOM happens close the channel

Removed from the log string any reference to Filebeat

Raised up the log level level when dropping connections becuase of thundering and going to OOM condition

Better actionable suggestion to user in case of OOM

Updated OOMConnectionCloser to monitor the consumption of memory also during the read opeartion and not only on exception

Re-introduce the beats handlers worker group to separata the Beats protocol processing from the boss group that accepts and listed to new sockets

Added feature flag named protect_direct_memory to control the usage of OOM checking or not. Enabled by default.

Throw a configuration error if Netty reserved direct memory is not anough (256MB)

Add missed shutdown of beat's worker loop