[question] Libre QoS - few questions before testing

[interduo]

1. Does it supports flat configuration configs with subscriber services parameters for example:
   { CUSTOMERIPS: 172.16.1.1, 172.16.1.2, DLRATE: 100M, UPRATE: 10M, DLCEIL: 200M, UPCEIL, 20M, DLBURST: 400M, DLBURSTTIME: 5sec,
   CUSTOMERIPS: 172.16.3.0/29, DLRATE: 100M, UPRATE: 10M, DLCEIL: 200M, UPCEIL, 20M, DLBURST: 400M, DLBURSTTIME: 5sec}

2. Can I change only one class?

3. Does Libre QoS got an API?

4. Does it supports HTB Offloading for filters (https://marc.info/?l=linux-netdev&m=160770071222418&w=2) ?

5. Does it supports counting per customer?

6. Is it possible to view class network usage in realtime?

7. Is NUMA [x] better for routers or not?

8. Is there any documentation?

[marsalans]

1. Yes, (https://github.com/rchac/LibreQoS/blob/main/v1.1/Shaper.csv)
2. LibreQoS code needs to be modified to do this
3. Yes, https://github.com/rchac/LibreQoS/blob/main/v1.1/lqAPI.py
4. Not this patch but it does same with xdp (thanks for highlighting this patch though)
5. If you are writing this in csv, you already know the count
7. If your application supports it completely
8. Yes https://github.com/rchac/LibreQoS/wiki/LibreQoS-v1.1-Installation-&-Usage-Guide-Physical-Server-and-Ubuntu-21.10

[interduo]

1. Could IP could be declared as:

• network /29
• two separate addresses like (192.168.1.55 and 192.168.1.59)
  for subscribers?

5. I mean: total bandwidth-download/upload-bandwitch for every subscriber, packet rate counters. This rate changes every second ;-)
6. Is LibreQoS for EdgeBGP routers or its recommended only as a QoS machine between EdgeBGP and subscribers?
7. How there is a NAT organized? What kind of NAT do LibreQoS supports?
8. Do LibreQoS supports HFSC?
9. Do You use mark packets by one-rule nftables with map and make QoS on it like:

table ip compressor_tc {
        chain prerouting {
                type nat hook prerouting priority -99; policy accept;
                ip saddr 10.50.0.3 mark set 0x00000007
                ip saddr 10.50.0.4 mark set 0x00000008
        }
}

?

14. Does it have integration with wanguard or fastnetmon? If some IP is blackholed - we need to change it for some time. (because many subscribers can use one NAT)
15. Could it be virtualized with passthrough NIC on Proxmox?
16. Are You planing to use VPP in future (https://www.youtube.com/watch?v=KXM4waZ4HLI)?

There are so many Question because I am comparing those three projects:

• BizonRouter,
• LibreQoS,
• VyOS,

[interduo]

It looks nice: I will probbably write an integration between with our CRM LMS and LibreQoS.

[rchac]

1. Does it supports flat configuration configs with subscriber services parameters for example:
   { CUSTOMERIPS: 172.16.1.1, 172.16.1.2, DLRATE: 100M, UPRATE: 10M, DLCEIL: 200M, UPCEIL, 20M, DLBURST: 400M, DLBURSTTIME: 5sec,
   CUSTOMERIPS: 172.16.3.0/29, DLRATE: 100M, UPRATE: 10M, DLCEIL: 200M, UPCEIL, 20M, DLBURST: 400M, DLBURSTTIME: 5sec}

You can do so by creating manual entries in Shaper.csv. However, the burst parameters aren't implemented, and each IP is treated as a distinct host (except when you define a subnet in Shaper.csv like 10.0.0.0/24 - those are all shaped under one queue)

2. Can I change only one class?

It rips and replaces all classes at each run. For ~500 hosts thats usually just a few seconds. this is done because it gets very convoluted to track class ids when changing them on the fly.

3. Does Libre QoS got an API?

Yes https://github.com/rchac/LibreQoS/blob/main/v1.1/lqAPI.py

4. Does it supports HTB Offloading for filters (https://marc.info/?l=linux-netdev&m=160770071222418&w=2) ?

The problem with HTB offloading is that only Mellanox has implemented it (please correct me if that has changed), and i wouldn't want to vendor-lock folks. It also limits to something like 3 levels of HTB depth, which kind of defeats a lot of the cool potential of nested HTBs.

6. Is it possible to view class network usage in realtime?

Yes, with v1.1 and InfluxDB

7. Is NUMA [x] better for routers or not?

If I'm understanding correctly, you mean if LibreQoS middelboxes should have pinned CPU affinity? If so, ideally you want to have a bare metal machine (xdp works better on bare metal) where you don't have to worry about other VMs. That said, in production I run it on a VM and it seems to be fine without any need to pin cores, but each VM hypervisor setup will be different.

8. Is there any documentation?

https://github.com/rchac/LibreQoS/wiki/LibreQoS-v1.1-Installation-&-Usage-Guide-Physical-Server-and-Ubuntu-21.10

1. Could IP could be declared as:

   • network /29

   • two separate addresses like (192.168.1.55 and 192.168.1.59)
     for subscribers?

Yup! just put 192.168.1.55/29 as the IPv4 in Shaper.csv for your customer.

5. I mean: total bandwidth-download/upload-bandwitch for every subscriber, packet rate counters. This rate changes every second ;-)

Using influxDB in v1.1, sort of. It tracks Mbps, rather than exact packet rate.

6. Is LibreQoS for EdgeBGP routers or its recommended only as a QoS machine between EdgeBGP and subscribers?

only as a QoS machine between EdgeBGP and subscribers - exactly that.
Illustration

7. How there is a NAT organized? What kind of NAT do LibreQoS supports?

LibreQoS should be in place downstream of wherever NAT occurs on your network. So if you use 100.64.0.0 for NAT addresses for example, LibreQoS should "see" those IPs, and shape those, rather than the external Public IP.

8. Do LibreQoS supports HFSC?

No, but it could if you adapt the code just slightly i think.

9. Do You use mark packets by one-rule nftables with map and make QoS on it like:

table ip compressor_tc {
        chain prerouting {
                type nat hook prerouting priority -99; policy accept;
                ip saddr 10.50.0.3 mark set 0x00000007
                ip saddr 10.50.0.4 mark set 0x00000008
        }
}

?
No packet marking occurs, and it doesnt look at packet marks. Instead, CAKE diffserv differentiates types of traffic (voip, bulk, etc). It's way easier and has very good results.
14. Does it have integration with wanguard or fastnetmon? If some IP is blackholed - we need to change it for some time. (because many subscribers can use one NAT)
No. But if you shape with LibreQoS downstream of NAT it might work around that.

15. Could it be virtualized with passthrough NIC on Proxmox?

Yes!

16. Are You planing to use VPP in future (https://www.youtube.com/watch?v=KXM4waZ4HLI)?

We're leaning the XDP route. XDP allows for a more neutral approach to NICs and vendors. We want ISPs to be able to use a variety of setups and NICs. Also I think XDP is kind of our only hope to work around the qdisc locking problem - and to have high throughput for HTB based hierarchical shaping.

[interduo]

You can do so by creating manual entries in Shaper.csv. However, the burst parameters aren't implemented, and each IP is treated as a distinct host (except when you define a subnet in Shaper.csv like 10.0.0.0/24 - those are all shaped under one queue)

Burts parameter is important when there are many smaller classes (eg. 1000x 10M) - total router cpu load is less then.

Yup! just put 192.168.1.55/29 as the IPv4 in Shaper.csv for your customer.

The main thing is that LibreQoS not support getting different IPs from 192.168.1.2/24 and 192.168.2.2/24 and mark it as one flowid and limit into one queue :( Is it hard to implement this?

This is the case for two main cases:

1. Subscriber have 2 publics IPs and one service,
2. Some LXC containers got two ip addresses from different VLANS and two gateways depending on conditions the routing is getting through one or second ip.

It rips and replaces all classes at each run. For ~500 hosts thats usually just a few seconds. this is done because it gets very convoluted to track class ids when changing them on the fly.

0 Packet lost when commiting changes ?

The problem with HTB offloading is that only Mellanox has implemented it (please correct me if that has changed), and i wouldn't want to vendor-lock folks. It also limits to something like 3 levels of HTB depth, which kind of defeats a lot of the cool potential of nested HTBs.

New intels (i40e) also supports this.

If I'm understanding correctly, you mean if LibreQoS middelboxes should have pinned CPU affinity? If so, ideally you want to have a bare metal machine (xdp works better on bare metal) where you don't have to worry about other VMs. That said, in production I run it on a VM and it seems to be fine without any need to pin cores, but each VM hypervisor setup will be different.

We will install LibreQoS on Proxmox and VM got settings: [x] enable NUMA - I am just curious what do You think about it in performance aspect.

Also I think XDP is kind of our only hope to work around the qdisc locking problem - and to have high throughput for HTB based hierarchical shaping.

This is the main thing I came to test LibreQoS :)

No, but it could if you adapt the code just slightly i think.

16. Why don't You use HFSC? It's better than HTB in many cases. It's getting faster to full speed in most cases.
17. Is it possible to use LibreQoS on one network card using different VLANs or just use one interface for network conectivity?
18. Do LibreQoS supports 22.04 ubuntu LTS?

[interduo]

@rchac i added some more questions,

This is really important info for project - You could add some to README.TXT or make FAQ for future generations.

[rchac]

You can do so by creating manual entries in Shaper.csv. However, the burst parameters aren't implemented, and each IP is treated as a distinct host (except when you define a subnet in Shaper.csv like 10.0.0.0/24 - those are all shaped under one queue)

Burts parameter is important when there are many smaller classes (eg. 1000x 10M) - total router cpu load is less then.

Could you explain more? @dtaht found that changing HTB burst parameters seemed to mess with CAKE.

Yup! just put 192.168.1.55/29 as the IPv4 in Shaper.csv for your customer.

The main thing is that LibreQoS not support getting different IPs from 192.168.1.2/24 and 192.168.2.2/24 and mark it as one flowid and limit into one queue :( Is it hard to implement this?

I suppose the best way to implement this would be to add a customer_ID field on Shaper.csv (in addition to the existing ID field). With that, if multiple entries in Shaper.csv share the same ID, they can be in the same queue.

This is the case for two main cases:

1. Subscriber have 2 publics IPs and one service,

2. Some LXC containers got two ip addresses from different VLANS and two gateways depending on conditions the routing is getting through one or second ip.

That makes sense. Let's work to add this functionality.

It rips and replaces all classes at each run. For ~500 hosts thats usually just a few seconds. this is done because it gets very convoluted to track class ids when changing them on the fly.

0 Packet lost when commiting changes ?

I tested right now. On a network of 350 subs, I saw just 1 single packet lost during the refresh. I tested at an interval of 100ms. So not enough packet loss to cause perceptible issues on most networks.

The problem with HTB offloading is that only Mellanox has implemented it (please correct me if that has changed), and i wouldn't want to vendor-lock folks. It also limits to something like 3 levels of HTB depth, which kind of defeats a lot of the cool potential of nested HTBs.

New intels (i40e) also supports this.

I looked this up but could not find anything to corroborate it. Could you please help me with a link, if you're able to find one?

If I'm understanding correctly, you mean if LibreQoS middelboxes should have pinned CPU affinity? If so, ideally you want to have a bare metal machine (xdp works better on bare metal) where you don't have to worry about other VMs. That said, in production I run it on a VM and it seems to be fine without any need to pin cores, but each VM hypervisor setup will be different.

We will install LibreQoS on Proxmox and VM got settings: [x] enable NUMA - I am just curious what do You think about it in performance aspect.

I would enable NUMA.

Also I think XDP is kind of our only hope to work around the qdisc locking problem - and to have high throughput for HTB based hierarchical shaping.

This is the main thing I came to test LibreQoS :)

No, but it could if you adapt the code just slightly i think.

16. Why don't You use HFSC? It's better than HTB in many cases. It's getting faster to full speed in most cases.

Part of how we achieve multi core is to have an HTB per MQ qdisc on each CPU core. I'm not sure if HFSC can reside within an MQ qdisc. If it can, I'm not sure how HFSC would interact with CAKE/fq_codel. I could anticipate those potentially interacting in unexpected ways. However, you are welcome to test it and if you find it helps we can look more into it!

You could replace mentions of "htb" with 'hfsc" in LibreQoS.py such as on the line:

shell('tc qdisc add dev ' + thisInterface + ' parent 7FFF:' + hex(queue+1) + ' handle ' + hex(queue+1) + ': htb default 2')

with

shell('tc qdisc add dev ' + thisInterface + ' parent 7FFF:' + hex(queue+1) + ' handle ' + hex(queue+1) + ': hfsc default 2')
There are some other hfsc parameters mentioned here.

17. Is it possible to use LibreQoS on one network card using different VLANs or just use one interface for network conectivity?

I don't think xdp-cpumap-tc has VLAN support yet. xdp-cpumap-tc is what we use to redirect packets to the appropriate CPU quickly. If you follow this network design, you can have any VLANs desired on the Core router at least, and as they pass, routed, over the LibreQoS box, they'll be shaped correctly.

18. Do LibreQoS supports 22.04 ubuntu LTS?

Yes definitely.

[interduo]

I will - answer questions in other issues as this would help to save information better - sorry for not doing it first time - as I didn't know that will be so many questions.

[rchac]

I will - answer questions in other issues as this would help to save information better - sorry for not doing it first time - as I didn't know that will be so many questions.

Great, thanks. Thank you for your interest in the project and for your contributions, I appreciate it.

[interduo]

New intels (i40e) also supports this.
I looked this up but could not find anything to corroborate it. Could you please help me with a link, if you're able to find one?

Situation for now: only Mellanox is supporting hierarhical rate limiting in HW+drivers. I checked again and found that I interpreted badly mails from netdev@kernel list.

[dtaht]

That was an awful lot of questions.

We explored HFSC, but in the context of a per customer shaper, htb + fq_codel seems better. There are a lot of advocates for HFSC + SFQ where its characterisics (per packet shaping) were ideal for in-customer flows, but as speeds cracked 25Mbits or so SFQ started to show it's need for an AQM component, and HFSC failed to differentiate between flows enough. I totally applaud more attempts to use HFSC + fq_codel or cake. Please see https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper_Must_Die/ for where smart queue management and eventually cake, came from.

To clarify: our experiment with the burst parameter was with in trying to shape a pre-wireless-n network, which can only tolerate a single packet. In each direction! so it proved best to spread the load at the default quantum to provide better service to more customers and more accurate rtt estimates to the end points.

I think that in wireless-n networks where aggregation is a thing, fiddling with the burst parameter may be a win. An even bigger win though would be having proper handling of aggregation in the 802.11(n,ac,ax) router hardware itself. https://www.cs.kau.se/tohojo/airtime-fairness/ . As much as I think preseem and libreqos are really good, needed, tools for ISPs, applying rfc7567 at the actual bottleneck points is also very needed.

However IMHO "saving on cpu" should not be a goal of leveraging the burst parameter in this scenario. Hardware capable of libreqos is cheap, it seems best to make few tradeoffs in terms of network quality. The xdp approach is working well, and now that we're doing that, perhaps a longer term evolution of this would be towards doing even more work in userspace.

[dtaht]

I hadn't heard of bisonrouter before. The HQOS system in the most current ddpk contains a not-quite-fully-baked implementation of the pie AQM. A couple bug fixes for that just went by, but if bisonrouter has a working pie, I'd love to hear results from that.

btw, here's one of the papers on cake. https://arxiv.org/abs/1804.07617 - it went into openwrt and most third party firmwares starting in 2015, linux mainline in 2018.

We didn't design it to be a middlebox!! At the time we started cake XDP didn't exist, and while we got rid of a lot of the locks in htb, it didn't seem like it would scale... until libreqos showed it did. I speak as a dev that has never owned more than a 6 core desktop, and a 12 core xeon server.

I am enjoying the data we are getting back from the deployment. One recent deployment ended up - "we deployed libreqos over a weekend, and all - and I mean all - our calls complaining about speed vanished" - and they called me to poke into their stats to see how well it actually working. It was really astonishing how many packet marks and drops they had at peak load, and yet how much better their network "felt".

[rchac]

@dtaht thank you for explaining, this helps a lot.

"we deployed libreqos over a weekend, and all - and I mean all - our calls complaining about speed vanished" - and they called me to poke into their stats to see how well it actually working. It was really astonishing how many packet marks and drops they had at peak load, and yet how much better their network "felt".

Best feedback possible. How awesome to hear!

[interduo]

This issue was smashed into smaller one - closing it now.