Bql support desperately needed in many drivers

Installing and Using OpenWrt
1
  • Spectrum (formerly Charter Communications) ISP [ 60dn 5up ]
  • Docsis 3.0 Standard
  • Arris (formerly Motorola) SB6183 CPE Modem
  • Netgear WNDR3800CH (same as WNDR3800)
  • LEDE Stable Branch 17.0.1.04

1st run ( sqm off, bql limit_max 3000 )

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc noqueue 0: dev br-lan root refcnt 2

2nd run ( sqm on, bq limit_max 3000, cake set @ 90% 54000,4500 keyword docsis dual-dsthost/dual-srchost nat )

root@LEDE:~# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc cake 8030: dev eth1 root refcnt 2 bandwidth 4500Kbit besteffort dual-srchost nat rtt 100.0ms noatm overhead 18 via-ethernet total_overhead 18 hard_header_len 14 mpu 64 
qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc cake 8031: dev ifb4eth1 root refcnt 2 bandwidth 54Mbit besteffort dual-dsthost nat wash rtt 100.0ms noatm overhead 18 via-ethernet total_overhead 18 hard_header_len 14 mpu 64

3rd run (same parameters as 2nd, just for sanity)

2

Your issue does not have to do with BQL. The WNDR3800 supports it. It's the modem.

3

My router does indeed support it, but the set limit is far beyond what is should be even for 100 mbits ( 1879048192 ) by default. it has to be tuned to the link speed, to actually be effective considering the device driver has the code. I'm amazed how well it works and wondering why more drivers don't support this. It's the same principle as having a vehicle with "Eco-Boost" but never activating it.

echo 3000 > /sys/devices/platform/ag71xx.1/net/eth1/queues/tx-0/byte_queue_limits/limit_max

https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#enabling-byte-queue-limits

4

This is only for TX queue.
Maybe you want to have a look at ethtool -g / -G

5

I should mention tso, ufo, gro, and gso are disabled via ethtool currently with my tests. It's not news, just wondering why it's not being utilized in more drivers?

6

If you want greater download buffer. you want to tweak rx queue not the tx queue.

7

that's the point of all these innovations, to USE them and have a happy medium. Relying on the pressure from a set bql without adjust the link speed is an awesome feature to reduce the tx queues while reducing latency. If only such a thing existed for the downstream and more devices supported bql.

8

you can change the rx queue with ethtool when your driver support this.
But normally this shouldnt be necessary.
I guess the problem lies somewhere else.

9

The WNDR3800 has cpu issues above 60Mbits using fq_codel/cake. That's why it's tuned to 54 Mbits.

10

Plus on Docsis networks when a customer reaches above 80% of the Maximum Sustained Traffic Rate, they are throttled. My MSTR is 69 Mbits dn, 6 Mbits up.

11

Im sorry.
I misread what you wrote.
You asking about the bql queue limits.
In your article you linked, they also provide a script to set the bql queue limits.

12

No worries. I'm not a much of developer...just adding to the results of edge use testing for wan links.

13

So in your tests BQL is never really exercised, if you traffic shape below line-rate, the NIC bufffers (that BQL basically tries to keep at a reasonable length) never fill. And even without SQM in the mix the fact that your uplink is only 5 Mbps means that the NIC's queues should never grow excessively.
I gess I am missing something here (and I do endorse your message bql support would be much appreciated in the DOCSIS modem and in XDSL-modems).

14

Wasn't the claim of BQL that it will tune itself? Now in your tests it should not, since it never actually gets to see many queued up packets....

15

This is an average test with no adjustments at all, just plain ole' stock LEDE

BQL ootb doesn't sway results in our favor for small links below 1 Gbit as pointed by dtaht. Now tuned even with 2 packets worth does wonders. Anything below that is just semantics.

16

Forgive me if my posts weren't concise. BQL is the main reason I decided to dust off the wndr3800ch and give bql a try to live up to it's claims. I'm glad I did.

17

In this test you should not see any effect of the BQL limits, as the bloated Buffers on the upstream leg of your link are probably inside your modem's DOCSIS section, which in all likelihood does not use bql.Now if you set your NIC to 10Mbps on a >=10 Mbps DOCSIS link, BQL will get exercised and might actually keep bufferbloat low even without a shaper like HTB or cake, but your downstream will now also be limited to 10Mbps...
So I agree that BQL is great, but it really really needs to be implemented inside the device at the near side of the real bottleneck link.

18

Sounds great, but could you explain again how you actually see bql doing the right thing in your tests? I am certain that you are correct, I just can not see it.

19

Observing the tx latency. My isp doesn't provide congestion control in any sense, so this new testing threw up some very different results. I've also test with flent on several occasions and even using stable on my r7000 with no changes and bql is not activated in that unit. Prior to BQL I could never achieve an A+ for bufferbloat unless I more than halved my download/upload bandwidth.

20

I believe you, but it really sounds odd, unless your modem uses pause frames or something to create back pressure, in that case BQL on your router's wan interface might still be triggered...