[backport -> release/3.5.x] perf(proxy): use higher default keepalive request value for Nginx tuning (#12223)

[chobits]

Bumped default values of nginx_http_keepalive_requests and upstream_keepalive_max_requests to 10000.
backport #12223

Summary

Checklist

• N/AThe Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• N/AThere is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix KAG-3360

[bungle]

What I have found is that on Mac the default ulimit -n seems to be 256 and these settings make the Kong not to work correctly. So perhaps a word somewhere about ulimit -n, would be good.

[chobits]

What I have found is that on Mac the default ulimit -n seems to be 256 and these settings make the Kong not to work correctly. So perhaps a word somewhere about ulimit -n, would be good.

hi @bungle What do you find in macOS? I tested this branch in my macOS Bazel environment and did not find any related error reports.
I re-reviewed the implementation of these two parameters in Nginx, thinking that they should not be related to any ulimit -n options (fd limit). These two options simply increase the allowed number of requests within a single connection.

---

If I set the limit to 256 mannually to test, the warnning message I get is not related to this PR, I revert this PR, and still get them:

(kong-dev) xc kong $ ulimit -n 256
(kong-dev) xc kong $ kong restart
2024/02/06 16:24:28 [warn] ulimit is currently set to "256". For better performance set it to at least "4096" using "ulimit -n"
Kong stopped
2024/02/06 16:24:29 [warn] ulimit is currently set to "256". For better performance set it to at least "4096" using "ulimit -n"
Kong started

[chobits]

Although this PR modifies the default value of NGINX, it doesn't have a significant impact on the client's side. Therefore, I don't believe we need to categorize this feature as a breaking_change.

Typically, users won't notice this alteration. However, under stressful testing scenarios, they might observe an increase in NGINX's performance, specifically in RPS

[bungle]

What I have found is that on Mac the default ulimit -n seems to be 256 and these settings make the Kong not to work correctly. So perhaps a word somewhere about ulimit -n, would be good.

hi @bungle What do you find in macOS?

See this: https://kongstrong.slack.com/archives/C02L7HR45DY/p1706256832168089

I think that was related. Aka kong started, and displayed those warnings. But then timers or events didn't work anymore.

In logs you will get:

2024/02/06 19:51:51 [alert] 78828#0: 1024 worker_connections are not enough
2024/02/06 19:51:51 [alert] 78828#0: lua failed to run timer with function defined at @/Users/bungle/.luaver/luarocks/3.9.1_5.1/share/lua/5.1/resty/timerng/thread/loop.lua:162: could not create fake connection
2024/02/06 19:51:51 [alert] 78828#0: 1024 worker_connections are not enough
2024/02/06 19:51:51 [alert] 78828#0: lua failed to run timer with function defined at @/Users/bungle/.luaver/luarocks/3.9.1_5.1/share/lua/5.1/resty/timerng/thread/loop.lua:162: could not create fake connection
2024/02/06 19:51:51 [alert] 78828#0: 1024 worker_connections are not enough
2024/02/06 19:51:51 [error] 78828#0: *3 [lua] worker.lua:138: communicate(): failed to connect: failed to receive response header: closed, context: ngx.timer
2024/02/06 19:51:51 [debug] 78828#0: *5 [lua] upstreams.lua:108: no upstreams were specified

Aka, kong starts and accepts admin api calls, but doesn't work. But perhaps it is releated to some other bump, e.g. did we change something else too? Before some change, it worked fine with ulimit -n 256, it does not work anymore.

[chobits]

What I have found is that on Mac the default ulimit -n seems to be 256 and these settings make the Kong not to work correctly. So perhaps a word somewhere about ulimit -n, would be good.

hi @bungle What do you find in macOS?

See this: https://kongstrong.slack.com/archives/C02L7HR45DY/p1706256832168089

I think that was related. Aka kong started, and displayed those warnings. But then timers or events didn't work anymore.

In logs you will get:

2024/02/06 19:51:51 [alert] 78828#0: 1024 worker_connections are not enough
2024/02/06 19:51:51 [alert] 78828#0: lua failed to run timer with function defined at @/Users/bungle/.luaver/luarocks/3.9.1_5.1/share/lua/5.1/resty/timerng/thread/loop.lua:162: could not create fake connection
2024/02/06 19:51:51 [alert] 78828#0: 1024 worker_connections are not enough
2024/02/06 19:51:51 [alert] 78828#0: lua failed to run timer with function defined at @/Users/bungle/.luaver/luarocks/3.9.1_5.1/share/lua/5.1/resty/timerng/thread/loop.lua:162: could not create fake connection
2024/02/06 19:51:51 [alert] 78828#0: 1024 worker_connections are not enough
2024/02/06 19:51:51 [error] 78828#0: *3 [lua] worker.lua:138: communicate(): failed to connect: failed to receive response header: closed, context: ngx.timer
2024/02/06 19:51:51 [debug] 78828#0: *5 [lua] upstreams.lua:108: no upstreams were specified

Aka, kong starts and accepts admin api calls, but doesn't work. But perhaps it is releated to some other bump, e.g. did we change something else too? Before some change, it worked fine with ulimit -n 256, it does not work anymore.

Got it, let me try, will get back to you if I have some new discoveries.

[chobits]

We have pinpointed the reason why kong worker generated so many alert logs in #12530 (comment), it is related to this PR: https://github.com/Kong/kong-ee/pull/7880. So we can advance the merging progress of this PR.

[ADD-SP]

Rebased on master as master has too many changes.

[ADD-SP]

Merge on CI passes