Changing TX-Queuelength to enhance Bandwidth

[CodeFetch]

Momentarily we use Debian's default values for e.g. the txqueuelen. These values are not optimized for routing. Increasing the buffer sizes and queue lengths will likely reduce the number of context switches. When setting them too high we might starve the userspace. Therefore we need to find reasonable values depending on the interface type (e.g. userspace VPN like fastd vs. kernel land WireGuard).

[CodeFetch]

Buffer sizes should be symmetric for servers:
ethtool -G eth0 rx 4096 tx 4096

[CodeFetch]

Increase input queue lengths:
echo 16384 > /proc/sys/net/core/netdev_max_backlog
ip link set eth0 txqueuelen 16384

[1977er]

I guess, this goes into the ffh.supernode role.

[lemoer]

Please make sure, that this does not introduce additional buffer bloat.

[CodeFetch]

@lemoer I did, but I didn't measure any improvement. So it seems it is not our bottleneck. A queue length of 16384 is enough for flattening spikes on a 2 Gbit/s link for about 5 ms. So this is nothing critical for bufferbloat, but just what we need for a 2 GBit/s link to not excessively start dropping on a high load. About the ring buffer sizes... They should be symmetric on a server that has symmetric up and down rates. Debian's defaults are for "client devices" which download more than they upload.

[1977er]

@CodeFetch If you did not measure any improvement, can this issue be closed?

[CodeFetch]

We will sooner or later run into these issues when we have higher bandwidth demands e.g. with WireGuard, because the default configuration of Debian is not for routers or better say machines that symmetrically receive/transmit packets.

[1977er]

So, what did you mean by "but I didn't measure any improvement"?

[CodeFetch]

@1977er That it's not the bottleneck at the moment. But I think it becomes relevant with more than 1 gbit/s traffic forwarding from different connections.

[1977er]

This will never happen with our current hardware and our resources.

[AiyionPrime]

@CodeFetch @1977er So this is stalled/blocked, at least until we rolled out WireGuard for the broad mass?

[1977er]

If these settings do not harm, we can introduce them for future use.

[AiyionPrime]

@CodeFetch if they don't, do you want to implement this?

[lemoer]

@1977er Settings like TXQueuelength can drastically reduce network perfomace due to bufferbloat. Therefore they can cause harm.

[1977er]

I guess then it's settled. As long as its not needed, don't change it.

@CodeFetch if you disagree, please re-open again.

[CodeFetch]

That's not how codel etc. work as these algorithms look at the latency for deciding to drop packets. Therefore we should give it a try as the current buffer sizes are not optimized for routing and this could partially explain our current problems.

[1977er]

Tested the setting of 16k again on a supernode with quite decent wg usage (already having the problem of dropping packets due to overusage). Looking at the % of neighbours with TQ < 95% chart I can see no effect after introducing the setting.

(injected at 9:38)

I suggest to close this issue until we know, that we need this.

[AiyionPrime]

Maybe we can re-test this once we have WG roled out, just to settle this once and for all then.

[CodeFetch]

@1977er @AiyionPrime We should at least set this setting symmetric as we are using the servers as routers and not desktop machines. The current/asymmetric values are definitely wrong.

[CodeFetch]

ip link set eth0 txqueuelen 16384

Tuning txqueuelen on pfifo_fast

The default txqueuelen was set to 1000 in Linux, in 2006. This is arguably wrong, even for gigE. Most network simulations at 100Mbit and below use 100, or even 50 as their default. Setting it below 50 is generally wrong, too, at any speed, except on wireless.

Source: https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/

Whatever that means...

Edit: "Time Squeeze" constantly increases so we might want to try tuning NAPI budgets
https://levelup.gitconnected.com/linux-kernel-tuning-for-high-performance-networking-receive-backlog-5b3f54fb82a7

Edit2: txqueuelen and netdev_max_backlog need to be set high enough for avoiding packet drops. At the same time NAPI weight and budget need to be set high enough for the softirq to handle all packets in time. Our issue is indeed softirq backpressure, but it seems that the reason for it is that the CPU cores are actually saturated. Maybe it makes sense to dedicate cores to softirq handling as recommended here (to reduce rescheduling and thereby increase cache warmth?), but that needs to be tested:
https://www.linkedin.com/pulse/navigating-linux-kernel-network-stack-receive-path-mark-price

Edit3: I like this guide more than the previous ones as it is more compact still covering all tips I've read about so far:
https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf