Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

haproxy-1.5.18 tcp mode occasionally occur cD message item in haproxy.log, but backend return response quickly, and frontend service also not slow. below is the detail: #1

Open
timusketeers opened this issue Oct 12, 2017 · 0 comments

Comments

@timusketeers
Copy link

timusketeers commented Oct 12, 2017

Haproxy-1.5.18 tcp mode occasionally occur cD message item in haproxy.log, but backend return response quickly, and frontend service also not slow. below is the detail:

Our message flow as below:
           172.30.49.193     172.30.49.18       172.30.49.29
            UI-Console  --->   Haproxy  ---->  Tomcat
            alias:  (A)           (B)                     (C)
       note: 
            message come from Internet Browser to UI-Console application(marked as A), then UI-Console 
       redispatch it to Haproxy(marked as B), and Haproxy transmit it to backend service who using 
       tomcat as web container(marked as C). 

We find that occasionally the request will response slow, as a result the UI-Console sometimes can not get response in time, thus can not show the response result on UI (Internet Browser). we analysis our log, and find that service C return response quickly, logs as below:

172.30.49.18 - - [29/Sep/2017:15:43:27 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 462
172.30.49.18 - - [29/Sep/2017:15:43:32 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 446
172.30.49.18 - - [29/Sep/2017:15:43:36 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 387
172.30.49.18 - - [29/Sep/2017:15:43:38 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 417
172.30.49.18 - - [29/Sep/2017:15:43:39 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 422
172.30.49.18 - - [29/Sep/2017:15:43:40 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 433
172.30.49.18 - - [29/Sep/2017:15:43:42 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 429
172.30.49.18 - - [29/Sep/2017:15:43:43 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 479
172.30.49.18 - - [29/Sep/2017:15:43:44 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 434
172.30.49.18 - - [29/Sep/2017:15:43:45 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 405
    note:
        The last column is the response time of a request.

Then we analysis haproxy.log and find cD terminal code in haproxy.log as below:
    Sep 29 15:44:10 localhost haproxy[67984]: 172.30.49.193:58298[29/Sep/2017:15:43:27.244] VPC VPC/VPC-172.30.49.29 1/0/120193 40293 cD 7/1/1/0/0 0/0

It seems the problem occur below Service A and B, considering that TCP packets should be transmitted and switched directly via A to B, the murderer seems B. Is haproxy delay the response? To check it we use tcpdump tool to capture tcp packages.

From captured tcp packages, we find Service A not close the socket immediately when A send packages over, and then after 20s Haproxy send a FIN-ACK package closed the socket. one question is that why haproxy close the socket after 20s, is there any configure item to config it.

Also we discover another problem,  when haproxy send packages over to backend service C, sometimes haproxy does not close the socket immediately, then service C will close the socket after 20s(because Tomcat connector's connectTimeout is configed as 20s in server.xml on tomcat). so another question is that why sometimes haproxy does not close the socket immediately when haproxy send packages over to backend service C.  is there any configuration could cause the delay? plus, below is our haproxy configuration:
haproxy.cfg:
    global
         log 127.0.0.1   local2 debug
         maxconn 250000
         user root
         group root
         daemon
         nbproc 4
    defaults
         log global
         mode    tcp
         option  tcplog
         option  dontlognull
         retries 3
         option redispatch
         maxconn 4000
         timeout connect 30000
         timeout client  120000
         timeout server  120000
   listen  AS-Server    172.30.49.153:8443
         mode tcp
         maxconn 4000
         balance roundrobin
         server  AS-Server-172.30.49.54  172.30.49.54:7443 check inter 500 fall 2 rise 2
         server  AS-Server-172.30.49.132  172.30.49.132:7443 check inter 500 fall 2 rise 2
   listen  VPC    172.30.49.152:8443
         mode tcp
         maxconn 4000
         balance roundrobin
         server  VPC-172.30.49.29  172.30.49.29:7443 check inter 500 fall 2 rise 2
         server  VPC-172.30.49.30  172.30.49.30:7443 check inter 500 fall 2 rise 2
         server  VPC-172.30.49.233  172.30.49.233:7443 check inter 500 fall 2 rise 2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant