HTTPClient::KeepAliveDisconnected when the node request is timing out #1032
Comments
|
Came across this issue - 68350f9 monkey-patched the 6.8.0 gem with the following commit but I still see the same error. This only happens when I use reload_connections/reload_on_failure option while creating the connection. Whenever one of the host goes down, the request timeouts and the see the message - ` [2020-09-19T09:41:16.806831 #155817] ERROR -- : [Faraday::TimeoutError] Connection timed out - connect(2) for "10.1.14.130" port 9200 (10.1.14.130:9200) {:scheme=>"https", :user=>nil, :password=>, :host=>"x.x.x.x", :path=>"", :port=>9200, :protocol=>"https"} /mnt/apps/.gembundle/ruby/2.7.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use |
|
It seems like we're experiencing the same issue, once one of the 3 nodes is down, we get these timeout errors and every request to ElasticSearch will fail. How would one configure the client to still make requests to the other nodes, while keeping the broken one out of the pool? |
|
We are facing the same issue as well. one of the node going down causes all requests to elastic search timeout! |
|
I'm going to try the advanced options from here: https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/7.17/advanced-config.html I cannot reproduce a host being unresponsive so I have to wait until that happens in our production cluster. What I can confirm is that if the host is completely down, these advanced config options are working correctly and connect to a different host |
|
In my case the issue was that I was not passing the protocol in scheme options while setting up the connection, so it worked initially for some request and when it use to reload the connection(because of the reload_connections and reload_on_failure) option, it was taking the default scheme: 'http' instead of 'https'. Took me a while to understand the issue but finally resolved it by passing scheme during the connection creation itself. |
I am initializing my connection like
Elasticsearch::Client.new hosts: ['https://<ip-of-node1>:9200', 'https://<ip-of-node2>:9200'].with following options - {reload_on_failure: true, reload_connections: true}.
Whenever there is request timeout on any of these nodes, I am getting the following traceback -
Traceback (most recent call last): from faraday (0.9.2) lib/faraday/connection.rb:377:inrun_request'from faraday (0.9.2) lib/faraday/rack_builder.rb:139:in
build_response' from faraday (0.9.2) lib/faraday/adapter/httpclient.rb:36:incall'from httpclient (2.7.2) lib/httpclient.rb:858:in
request' from httpclient (2.7.2) lib/httpclient.rb:1016:indo_request'from httpclient (2.7.2) lib/httpclient.rb:1127:in
protect_keep_alive_disconnected' from httpclient (2.7.2) lib/httpclient.rb:1134:inrescue in protect_keep_alive_disconnected'from httpclient (2.7.2) lib/httpclient.rb:1021:in
block in do_request' from httpclient (2.7.2) lib/httpclient.rb:1241:indo_get_block'from httpclient (2.7.2) lib/httpclient.rb:1294:in
do_get_header' from httpclient (2.7.2) lib/httpclient/session.rb:547:inget_header'from httpclient (2.7.2) lib/httpclient/session.rb:771:in
read_header' from httpclient (2.7.2) lib/httpclient/session.rb:788:inparse_header'from /opt/ruby-2.7.1/lib/ruby/2.7.0/timeout.rb:105:in
timeout' from /opt/ruby-2.7.1/lib/ruby/2.7.0/timeout.rb:95:inblock in timeout'from httpclient (2.7.2) lib/httpclient/session.rb:795:in
block in parse_header' HTTPClient::KeepAliveDisconnected (HTTPClient::KeepAliveDisconnected: )I see that in elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb, we have defined
def host_unreachable_exceptions [::Faraday::Error::ConnectionFailed, ::Faraday::Error::TimeoutError] endthen why is TimeoutError is not reloading connection, instead giving the above exception.
To verify this, I did a curl on the same node and the request did timeout but from the gem, i am facing some issues
The text was updated successfully, but these errors were encountered: