Plugin stops working after ~30 minutes

[indietyp]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

<system>
   workers 4
   log_level debug
</system>

####
## Source descriptions:
##

<source>
  @type syslog
  port 42185
  bind 0.0.0.0
  tag syslog
</source>

<source>
  @type forward
  @id input_forward
</source>

## live debugging agent
<source>
  @type debug_agent
  @id input_debug_agent
  bind 127.0.0.1
  port 24230
</source>


####
## Output descriptions:
##

<match {docker.**,syslog.**}>
  @type copy
  #<store>
  #  @type stdout
  #</store>
  <store>
    @type opensearch_data_stream
    host <REDACTED>
    port 9200
    user <REDACTED>
    password <REDACTED>
    scheme https
    ssl_verify false

    http_backend typhoeus
    index_name fluentd-${tag}
    include_timestamp true
    data_stream_name <REDACTED>

    <buffer tag>
      @type memory
      flush_mode immediate
      flush_thread_count 8
    </buffer>
  </store>
</match>

Expected Behavior or What you need to ask

We are currently trying to deploy fluentd as our log aggregator on every host, inputting syslog and docker and outputting to OpenSearch. This works fine for ~30 minutes, but after that, it just crashes.

I have attached the complete log, which indefinitely repeats the same error, showing that this seems to be a non-recoverable error. Here is the relevant snippet with the error:

2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0c52349bd0eb51e1b23ba9aec"
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0c67862ebd116dfc2804764bc"
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0cea2050e13086797edfbcfce"
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0cea2050e13086797edfbcfce" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0ced9cb2ee7500ebbb2d93469"
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0ced9cb2ee7500ebbb2d93469" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0b4e6cd1e6dc14aca9cb6f2b3"
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0b4e6cd1e6dc14aca9cb6f2b3" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0c4469ce4bbc121088f5c75ed"
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0c4469ce4bbc121088f5c75ed" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0ce424443802f570577c2f28f"
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0ce424443802f570577c2f28f" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [debug]: #1 taking back chunk for errors. chunk="5e17e9b0cfb9299cff459ded11ea4a1f"
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0cfb9299cff459ded11ea4a1f" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0c52349bd0eb51e1b23ba9aec" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-06-15 17:59:29 +0200 [warn]: #1 failed to flush the buffer. retry_times=0 next_retry_time=2022-06-15 17:59:31 +0200 chunk="5e17e9b0c67862ebd116dfc2804764bc" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (<REDACTED>): Couldn't connect to server"
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-06-15 17:59:29 +0200 [warn]: #1 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Using Fluentd and OpenSearch plugin versions

• OS version: Ubuntu 20.04.4 LTS

• Bare Metal or within Docker or Kubernetes or others?: Bare metal

• Fluentd v1.0 or later: td-agent 4.3.1 fluentd 1.14.6 (c0f48a0080550eff6aa6fa19d269e480684e7a45)

• OpenSearch version: 7.10.2

• OpenSearch plugin version:

addressable (2.8.0)
async (1.30.1)
async-http (0.56.5)
async-io (1.33.0)
async-pool (0.3.9)
aws-eventstream (1.2.0)
aws-partitions (1.577.0)
aws-sdk-core (3.130.1)
aws-sdk-kms (1.55.0)
aws-sdk-s3 (1.113.0)
aws-sdk-sqs (1.51.0)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bindata (2.4.10)
bundler (2.3.11, default: 2.1.4)
cgi (default: 0.1.0.1)
cmetrics (0.2.5)
concurrent-ruby (1.1.10)
console (1.15.0)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
digest-murmurhash (1.1.1)
elastic-transport (8.0.0)
elasticsearch (8.1.2)
elasticsearch-api (8.1.2)
etc (default: 1.1.0)
ethon (0.15.0)
excon (0.92.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
ffi (1.15.5)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (1.6.0, default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-diagtool (1.0.1)
fluent-logger (0.9.0)
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-elasticsearch (5.2.2)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.17.5)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.0.7, 1.0.4)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.6.1)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.1.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)
fluentd (1.14.6)
forwardable (default: 1.3.1)
getoptlong (default: 0.1.0)
hirb (0.7.3)
http_parser.rb (0.8.0)
httpclient (2.8.3)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.1)
json (2.6.1, default: 2.3.0)
linux-utmpx (0.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
mini_portile2 (2.8.0)
minitest (5.13.0)
msgpack (1.5.1)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
nokogiri (1.13.4 x86_64-linux)
observer (default: 0.1.0)
oj (3.13.11)
open3 (default: 0.1.0)
opensearch-api (1.0.0)
opensearch-ruby (1.0.0)
opensearch-transport (1.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
parallel (1.22.1)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (2.1.0)
protocol-hpack (1.4.2)
protocol-http (0.22.5)
protocol-http1 (0.14.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.7)
racc (1.6.0, default: 1.4.16)
rake (13.0.6, 13.0.1)
rdkafka (0.11.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.4.0)
ruby-progressbar (1.11.0)
ruby2_keywords (0.0.5)
rubyzip (1.3.0)
sdbm (default: 1.0.0)
serverengine (2.2.5)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
systemd-journal (1.4.2)
td (0.16.9)
td-client (1.0.8)
td-logger (0.3.27)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
typhoeus (1.4.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webhdfs (0.10.2)
webrick (1.7.0, default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.2)
yaml (default: 0.1.0)
zip-zip (0.3)
zlib (default: 1.1.0)

2022-06-15 17:33:19 +0200 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-calyptia-monitoring' version '0.1.3'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.2'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-kafka' version '0.17.5'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-metrics-cmetrics' version '0.1.2'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-opensearch' version '1.0.7'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-opensearch' version '1.0.4'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-s3' version '1.6.1'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-td' version '1.1.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2022-06-15 17:33:19 +0200 [info]: gem 'fluentd' version '1.14.6'
2022-06-15 17:33:19 +0200 [debug]: adding store type="opensearch_data_stream"
2022-06-15 17:33:19 +0200 [debug]: 'host <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:20 +0200 [debug]: 'host_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:20 +0200 [debug]: 'data_stream_name_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:20 +0200 [info]: Specified data stream does not exist. Will be created: <[404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}>
2022-06-15 17:33:20 +0200 [debug]: No fluent logger for internal event
2022-06-15 17:33:20 +0200 [info]: using configuration file: <ROOT>
  <system>
    workers 4
    log_level debug
  </system>
  <source>
    @type syslog
    port 42185
    bind "0.0.0.0"
    tag "syslog"
  </source>
  <source>
    @type forward
    @id input_forward
  </source>
  <source>
    @type debug_agent
    @id input_debug_agent
    bind "127.0.0.1"
    port 24230
  </source>
  <match {docker.**,syslog.**}>
    @type copy
    <store>
      @type "opensearch_data_stream"
      host "<REDACTED>"
      port 9200
      user "<REDACTED>"
      password xxxxxx
      scheme https
      ssl_verify false
      http_backend typhoeus
      index_name "fluentd-${tag}"
      include_timestamp true
      data_stream_name "<REDACTED>"
      <buffer tag>
        @type "memory"
        flush_mode immediate
        flush_thread_count 8
      </buffer>
    </store>
  </match>
</ROOT>
2022-06-15 17:33:20 +0200 [info]: starting fluentd-1.14.6 pid=3153244 ruby="2.7.6"
2022-06-15 17:33:20 +0200 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2022-06-15 17:33:22 +0200 [info]: adding match pattern="{docker.**,syslog.**}" type="copy"
2022-06-15 17:33:22 +0200 [debug]: #2 adding store type="opensearch_data_stream"
2022-06-15 17:33:22 +0200 [debug]: #0 adding store type="opensearch_data_stream"
2022-06-15 17:33:22 +0200 [debug]: #3 adding store type="opensearch_data_stream"
2022-06-15 17:33:22 +0200 [debug]: #1 adding store type="opensearch_data_stream"
2022-06-15 17:33:23 +0200 [debug]: #2 'host <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #2 'host_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #2 'data_stream_name_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [info]: #2 Specified data stream does not exist. Will be created: <[404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}>
2022-06-15 17:33:23 +0200 [debug]: #3 'host <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #0 'host <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #2 No fluent logger for internal event
2022-06-15 17:33:23 +0200 [info]: #2 starting fluentd worker pid=3153324 ppid=3153318 worker=2
2022-06-15 17:33:23 +0200 [debug]: #2 buffer started instance=2860 stage_size=0 queue_size=0
2022-06-15 17:33:23 +0200 [info]: #2 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24232" object="Fluent::Engine" worker=2
2022-06-15 17:33:23 +0200 [info]: #2 [input_forward] listening port port=24224 bind="0.0.0.0"
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #2 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #2 listening syslog socket on 0.0.0.0:42185 with udp
2022-06-15 17:33:23 +0200 [info]: #2 fluentd worker is now running worker=2
2022-06-15 17:33:23 +0200 [debug]: #3 'host_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #1 'host <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #0 'host_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #3 'data_stream_name_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #0 'data_stream_name_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [info]: #3 Specified data stream does not exist. Will be created: <[404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}>
2022-06-15 17:33:23 +0200 [info]: #0 Specified data stream does not exist. Will be created: <[404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}>
2022-06-15 17:33:23 +0200 [info]: adding source type="syslog"
2022-06-15 17:33:23 +0200 [info]: adding source type="forward"
2022-06-15 17:33:23 +0200 [info]: adding source type="debug_agent"
2022-06-15 17:33:23 +0200 [debug]: #3 No fluent logger for internal event
2022-06-15 17:33:23 +0200 [info]: #3 starting fluentd worker pid=3153325 ppid=3153318 worker=3
2022-06-15 17:33:23 +0200 [debug]: #3 buffer started instance=2860 stage_size=0 queue_size=0
2022-06-15 17:33:23 +0200 [info]: #3 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24233" object="Fluent::Engine" worker=3
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #3 [input_forward] listening port port=24224 bind="0.0.0.0"
2022-06-15 17:33:23 +0200 [debug]: #0 No fluent logger for internal event
2022-06-15 17:33:23 +0200 [info]: #0 starting fluentd worker pid=3153321 ppid=3153318 worker=0
2022-06-15 17:33:23 +0200 [debug]: #0 buffer started instance=2860 stage_size=0 queue_size=0
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #3 listening syslog socket on 0.0.0.0:42185 with udp
2022-06-15 17:33:23 +0200 [info]: #0 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2022-06-15 17:33:23 +0200 [info]: #3 fluentd worker is now running worker=3
2022-06-15 17:33:23 +0200 [info]: #0 [input_forward] listening port port=24224 bind="0.0.0.0"
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #0 listening syslog socket on 0.0.0.0:42185 with udp
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #3 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #0 fluentd worker is now running worker=0
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #0 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #1 'host_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [debug]: #1 'data_stream_name_placeholder <REDACTED>' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: <REDACTED>' doesn't have tag placeholder
2022-06-15 17:33:23 +0200 [info]: #1 Specified data stream does not exist. Will be created: <[404] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [<REDACTED>]","index":"<REDACTED>","resource.id":"<REDACTED>","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}>
2022-06-15 17:33:23 +0200 [debug]: #1 No fluent logger for internal event
2022-06-15 17:33:23 +0200 [info]: #1 starting fluentd worker pid=3153322 ppid=3153318 worker=1
2022-06-15 17:33:23 +0200 [debug]: #1 buffer started instance=2860 stage_size=0 queue_size=0
2022-06-15 17:33:23 +0200 [info]: #1 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24231" object="Fluent::Engine" worker=1
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #1 [input_forward] listening port port=24224 bind="0.0.0.0"
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #1 listening syslog socket on 0.0.0.0:42185 with udp
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [info]: #1 fluentd worker is now running worker=1
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running
2022-06-15 17:33:23 +0200 [debug]: #1 flush_thread actually running

complete.log

[cosmo0920]

Have you tried the following parameter combination in the configuration section of data streams plugin?

  reconnect_on_error true
  reload_on_failure true
  reload_connections false

With your configuration, when occurring an error, requests are failing forever.

[indietyp]

We'll try the configuration you outlined but would like to know why the issue is occurring in the first place. From what we can tell, there's no network interruption between the two nodes, even though the error suggests otherwise.