Wrong number of arguments for i18n after upgrade to Version 8.10.3

[anikafranz]

Logstash information:

1. Logstash version: 8.10.3
2. Logstash installation source: docker
3. How is Logstash being run: docker

Plugins installed: no additional ones

JVM (e.g. java -version): the one provided in docker container from https://www.docker.elastic.co/

OS version (uname -a if on a Unix-like system): docker container

Description of the problem including expected versus actual behavior:

After we upgraded from 8.8.x to 8.10.3 the connection between logstash and elasticsearch stopped working. After investigating the only error we were able to find is located in logstash logs after startup:

warning: thread "[main]<file" terminated with exception (report_on_exception is true):
ArgumentError: wrong number of arguments (given 2, expected 0..1)
    translate at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/i18n-1.14.1/lib/i18n.rb:210
  inputworker at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:427
  start_input at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405
[2023-10-10T11:43:47,488][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: wrong number of arguments (given 2, expected 0..1)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/i18n-1.14.1/lib/i18n.rb:210:in `translate'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:427:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'"]}

The warnings in the full log provide the impression of some connection issues, but if we connect to container and run for example a curl command to elasticsearch it works fine. Only error we can find is this i18n ruby issue in logstash.
I assume that this is something related to the Ruby version upgrade? Has anyone idea how to fix this at least temporarily?

Provide logs (if relevant):

Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-10-10T11:42:46,187][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-10-10T11:42:46,191][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.10.3", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.8+7 on 17.0.8+7 +indy +jit [x86_64-linux]"}
[2023-10-10T11:42:46,193][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx4096m, -Xms4096m, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-10-10T11:42:46,771][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Elastic Agent to monitor Logstash. Documentation can be found at:
https://www.elastic.co/guide/en/logstash/current/monitoring-with-elastic-agent.html
[2023-10-10T11:42:47,025][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:[email protected]:9200/]}}
[2023-10-10T11:42:47,279][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"https://elastic:[email protected]:9200/"}
[2023-10-10T11:42:47,284][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (8.10.3) {:es_version=>8}
[2023-10-10T11:42:47,285][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-10-10T11:42:47,307][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2023-10-10T11:42:47,307][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2023-10-10T11:42:47,393][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-10-10T11:42:47,826][INFO ][org.reflections.Reflections] Reflections took 151 ms to scan 1 urls, producing 132 keys and 464 values
[2023-10-10T11:42:47,915][INFO ][logstash.javapipeline    ] Pipeline `.monitoring-logstash` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
[2023-10-10T11:42:47,927][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["https://elasticsearch.somethinginternal.com:9200"]}
[2023-10-10T11:42:47,931][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:[email protected]:9200/]}}
[2023-10-10T11:42:47,958][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://elastic:[email protected]:9200/"}
[2023-10-10T11:42:47,964][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (8.10.3) {:es_version=>8}
[2023-10-10T11:42:47,964][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-10-10T11:42:47,973][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2023-10-10T11:42:47,980][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x5317b544 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-10-10T11:42:48,710][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.73}
[2023-10-10T11:42:48,740][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2023-10-10T11:42:51,182][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_enabled' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"ssl", :plugin=><LogStash::Outputs::ElasticSearch password=><password>, hosts=>[//elasticsearch.somethinginternal.com:9200], index=>"%{[@metadata][target_index]}", action=>"create", manage_template=>false, id=>"06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd", ssl=>true, user=>"elastic", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_40937816-0f91-474f-9815-f82dd2f86323", enable_metric=>true, charset=>"UTF-8">, workers=>1, ssl_certificate_verification=>true, ssl_verification_mode=>"full", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, retry_initial_interval=>2, retry_max_interval=>64, dlq_on_failed_indexname_interpolation=>true, data_stream_type=>"logs", data_stream_dataset=>"generic", data_stream_namespace=>"default", data_stream_sync_fields=>true, data_stream_auto_routing=>true, template_overwrite=>false, template_api=>"auto", doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy">}
[2023-10-10T11:42:51,211][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_enabled' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"ssl", :plugin=><LogStash::Outputs::ElasticSearch password=><password>, hosts=>[//elasticsearch.somethinginternal.com:9200], index=>"logs-unknown-any", action=>"create", manage_template=>false, id=>"c78c8db997a68b778506465d0a77c966a81f7ee55fa45d643808dbd211272360", ssl=>true,user=>"elastic", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_3bdff138-eeb2-4f5e-8e2c-fde2bf41f331", enable_metric=>true, charset=>"UTF-8">, workers=>1, ssl_certificate_verification=>true, ssl_verification_mode=>"full", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, retry_initial_interval=>2, retry_max_interval=>64, dlq_on_failed_indexname_interpolation=>true, data_stream_type=>"logs", data_stream_dataset=>"generic", data_stream_namespace=>"default", data_stream_sync_fields=>true, data_stream_auto_routing=>true, template_overwrite=>false, template_api=>"auto", doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy">}
[2023-10-10T11:42:51,221][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
[2023-10-10T11:42:51,269][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch.somethinginternal.com:9200"]}
[2023-10-10T11:42:51,276][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:[email protected]:9200/]}}
[2023-10-10T11:42:51,310][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:[email protected]:9200/"}
[2023-10-10T11:42:51,316][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.3) {:es_version=>8}
[2023-10-10T11:42:51,317][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-10-10T11:42:51,325][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-10-10T11:42:51,326][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-10-10T11:42:51,329][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch.somethinginternal.com:9200"]}
[2023-10-10T11:42:51,341][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:[email protected]:9200/]}}
[2023-10-10T11:42:51,379][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:[email protected]:9200/"}
[2023-10-10T11:42:51,388][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.3) {:es_version=>8}
[2023-10-10T11:42:51,389][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-10-10T11:42:51,394][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-10-10T11:42:51,395][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-10-10T11:42:51,701][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x7d8605cf /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-10-10T11:42:53,510][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.81}
[2023-10-10T11:42:53,519][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-10-10T11:42:53,671][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-10-10T11:42:53,671][INFO ][logstash.inputs.http     ][main][2fb8f9d1b54e7a5291574e6d0a334470c62a5efdb19927f93096247cdb0f7712] Starting http input listener {:address=>"0.0.0.0:5055", :ssl=>"false"}
[2023-10-10T11:42:53,672][INFO ][filewatch.observingread  ][main][9d8b3e33511b0b1a0392cf079359f2e4e43dd3633630f1c12882359ea7b4b30b] START, creating Discoverer, Watch with file and sincedb collections
[2023-10-10T11:42:53,673][INFO ][org.logstash.beats.Server][main][39bba9440619053121088836e6e6ebc941194e8ec87884b59d6e6e2629ceffcb] Starting server on port: 5044
[2023-10-10T11:42:53,675][INFO ][filewatch.observingread  ][main][689aedd9954f50c200dc1bd97747613d003ed08c5949d76f68fbcb8b53316897] START, creating Discoverer, Watch with file and sincedb collections
[2023-10-10T11:42:53,678][INFO ][logstash.inputs.tcp      ][main][6ae77cb9a25c6fedc297e7b9cfdf63c678ccc8b1be36d9f9a7d24b19417c320f] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enabled=>false}
[2023-10-10T11:42:53,681][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2023-10-10T11:42:53,720][INFO ][logstash.inputs.udp      ][main][c8396066524a7a6be95fc91fd2469fb8eeff8613025d85c6f61065df6e8338fd] Starting UDP listener {:address=>"0.0.0.0:5000"}
[2023-10-10T11:42:53,730][INFO ][logstash.inputs.udp      ][main][c8396066524a7a6be95fc91fd2469fb8eeff8613025d85c6f61065df6e8338fd] UDP listener started {:address=>"0.0.0.0:5000", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
warning: thread "[main]<file" terminated with exception (report_on_exception is true):
ArgumentError: wrong number of arguments (given 2, expected 0..1)
    translate at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/i18n-1.14.1/lib/i18n.rb:210
  inputworker at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:427
  start_input at /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405
[2023-10-10T11:43:47,488][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: wrong number of arguments (given 2, expected 0..1)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/i18n-1.14.1/lib/i18n.rb:210:in `translate'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:427:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'"]}
[2023-10-10T11:43:47,528][INFO ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:43:47,529][INFO ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:43:47,530][INFO ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:43:47,538][INFO ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:43:47,538][WARN ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down {:url=>https://elastic:[email protected]:9200/, :error_message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-10-10T11:43:47,544][WARN ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down {:url=>https://elastic:[email protected]:9200/, :error_message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-10-10T11:43:47,546][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,547][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,549][WARN ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down {:url=>https://elastic:[email protected]:9200/, :error_message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-10-10T11:43:47,551][WARN ][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down {:url=>https://elastic:[email protected]:9200/, :error_message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-10-10T11:43:47,568][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,568][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/_bulk][Manticore::ClientStoppedException] Connection pool shut down", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,573][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,576][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,589][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,611][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>2}
[2023-10-10T11:43:47,668][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"", :exception=>Java::OrgJrubyExceptions::ThreadKill, :backtrace=>[], :thread=>"#<Thread:0x7d8605cf /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2023-10-10T11:43:47,768][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"", :exception=>Java::OrgJrubyExceptions::ThreadKill, :backtrace=>[], :thread=>"#<Thread:0x7d8605cf /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-10-10T11:43:47,834][WARN ][logstash.javapipeline    ][.monitoring-logstash] Waiting for input plugin to close {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x5317b544 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-10-10T11:43:48,836][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2023-10-10T11:43:49,560][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>4}
[2023-10-10T11:43:49,571][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>4}
[2023-10-10T11:43:49,571][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>4}
[2023-10-10T11:43:49,579][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>4}
[2023-10-10T11:43:49,591][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>4}
[2023-10-10T11:43:49,615][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>4}
[2023-10-10T11:43:51,525][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:43:51,526][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:43:53,565][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>8}
[2023-10-10T11:43:53,573][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>8}
[2023-10-10T11:43:53,573][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>8}
[2023-10-10T11:43:53,581][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>8}
[2023-10-10T11:43:53,594][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>8}
[2023-10-10T11:43:53,617][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>8}
[2023-10-10T11:43:56,529][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:43:56,530][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:01,533][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:44:01,534][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:01,567][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}
[2023-10-10T11:44:01,575][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}
[2023-10-10T11:44:01,575][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}
[2023-10-10T11:44:01,584][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}
[2023-10-10T11:44:01,595][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}
[2023-10-10T11:44:01,620][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}
[2023-10-10T11:44:06,537][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:44:06,538][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:11,546][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:44:11,547][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:16,565][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:44:16,567][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:17,315][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:44:17,316][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down {:url=>https://elastic:[email protected]:9200/, :error_message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-10-10T11:44:17,318][WARN ][logstash.licensechecker.licensereader] Attempt to fetch Elasticsearch cluster info failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:17,339][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2023-10-10T11:44:17,340][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2023-10-10T11:44:17,351][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
[2023-10-10T11:44:17,352][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:[email protected]:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch.somethinginternal.com:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
[2023-10-10T11:44:17,573][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>32}
[2023-10-10T11:44:17,577][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>32}
[2023-10-10T11:44:17,577][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>32}
[2023-10-10T11:44:17,586][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>32}
[2023-10-10T11:44:17,599][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>32}
[2023-10-10T11:44:17,622][ERROR][logstash.outputs.elasticsearch][main][06609e009389877afb3f8285ea3f38b3f07470ea0f240f1083b1758f15b5b6cd] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>32}

[kkornienko-aparavi]

Same in 8.10.2, 8.10.1, 8.10.0

[lost2]

Same on elk 8.10.4 (on-prem).
All other pipelines work well except the one using (http input, empty filter, elasticsearch output). Error bellow:

[2023-10-23T23:12:12,257][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: wrong number of arguments (given 2, expected 0..1)>, :backtrace=>["/usr/s
hare/logstash/vendor/bundle/jruby/3.1.0/gems/i18n-1.14.1/lib/i18n.rb:210:in translate'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:427:in inputworker'", "/usr/share
/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'"]}
[2023-10-23T23:12:12,311][ERROR][logstash.agent ] Exception handling servers {:error=>#<IOError: closed stream>, :req=>nil}

[praveenmak]

Anyone know if we downgrade to 8.9.X, will help fix this?

[andsel]

It's a bug in the usage of hash arguments in method i18n.t
The usage location of the method in java_pipeline is

logstash/logstash-core/lib/logstash/java_pipeline.rb

Line 427 in 16da966

@logger.error(I18n.t(

where it invokes

I18n.t("something", {"key" => "value"})

but it has been deprecated in Ruby 2.7 and removed in 3.0, as reported in the method comment
https://github.com/ruby-i18n/i18n/blob/395aa5e9c0f29ca20af83750fb1c7e01cb07a67b/lib/i18n.rb#L195-L209

it's a missed leftover from the original migration PR #14861