Can't go >90mbps with SQM enabled on R7800 (115mbps with software offloading)

As the title says, using r19677.

I don't know why but I discovered that if I enabled SQM I can't go over 85-90mbps, obviously I set 150mpbs in SQM (I have 200/20mbps FTTC).

If I disable the SQM I get about 180mbps.
If I enable SQM with software flow offloading I get about 115mbps (a bit better).

For what I remember some months ago SQM was working correctly with the same settings.

My SQM settings are:

config queue 'eth0'
	option debug_logging '0'
	option linklayer 'ethernet'
	option overhead '34'
	option upload '17800'
	option enabled '1'
	option verbosity '5'
	option interface 'eth0.2'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'
	option download '150000'

And

grep offload /etc/config/firewall
	option flow_offloading '1'

I also tried to enable/disable the CPU scaling governator but no change.

What I'm doing wrong? :face_with_raised_eyebrow:
Thanks!

I would try the following tests (with no advanced config options) -

  1. 195000 download/19500 upload

  2. Replace Cake/Piece of Cake with fq_codel and simplest_tbf. Same download/upload.

  3. 0 download/19500 upload.

Test on a wired connection using Waveform's bufferbloat/speedtest.

Post the links Waveform provides for test results.

1 Like

Okay thanks,

test 1 with 195000 download/19500 upload and Cake

test 2 with same settings and fq_codel/simplest_tbf

test 3 with 0 download/19500 upload and fq_codel

So from your test I think the first one should be the best setting? But 195mbps is not 5-10% of 200mbps, I should leave 195 or is better set something less?

You should test with various overhead values. The ISPs have so varying protocols, that the generic advice about overhead per connection type may not hold true. (I get best results with either 0 or 18)

I have R7800 with a connection of 200/60
with up-to-date master OpenWrt SNAPSHOT, r19689-19ef3b54f4

With 190000/57000 limits I get:

But interestingly there are errors when SQM is restarted:

root@router1:/etc/config# /etc/init.d/sqm restart
SQM: Stopping SQM on eth0.2
SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc qdisc del dev eth0.2 ingress
SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Invalid argument
SQM: Starting SQM script: simple.qos on eth0.2, in: 190000 Kbps, out: 57000 Kbps
SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc qdisc add dev eth0.2 handle ffff: ingress
SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: File exists
SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc filter add dev eth0.2 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0.2
SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Invalid argument
We have an error talking to the kernel
SQM: WARNING: sqm_start_default: simple.qos lacks an ingress() function
SQM: simple.qos was started on eth0.2 successfully

To me it looks like there may be currently something wrong with the sqm / tc / firewall co-operation, possibly related to the download side.
tc -s -d qdisc show does not show stats for the download side.

1 Like

I you value smooth browsing, look for settings that give you an A score for bufferbloat and then tweak parameters for getting highest possible throughput.

The "5-10%" is meant to be left free for protocol traffic, so that you use 90-95% of the bandwidth. 195 is 97.5% of 200, but that 5 Mbit left free is likely fine.
But if your case the download side is not that crucial. The narrow upload causes you more trouble, as you have 200/20 asymmetric connection.

Okay thanks, but in the end I discovered that my trouble was only with speedtest.net, also on DSLreports the speed is normal. On Speedtest.net I can't go over 90mbps with SQM enabled. I have no idea why!

Another point is that, as the OpenWRT tutorial reports, I should test without SQM (and I get about 170/180mbps) and then set 90/95% of it, like 160mpbs i.e., but this doesn't work good for me, I got very low speedtest and hight bufferbloat.

If I set 190/195mbps (that exceed my speed tests without SQM), I get a normal speed of 130-140mbps and low bufferbloat.
So I have no idea why. Anyway I would left irq balance active.

PS: I don't have errors and tc -s -d qdisc show shows the stats correctly here:

qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 1437805110 bytes 6342483 pkt (dropped 0, overlimits 0 requeues 6) 
 backlog 0b 0p requeues 6
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 1437805110 bytes 6342483 pkt (dropped 0, overlimits 0 requeues 6) 
 backlog 0b 0p requeues 6
  maxpacket 1514 drop_overlimit 0 new_flow_count 12 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root 
 Sent 44484923 bytes 214119 pkt (dropped 0, overlimits 0 requeues 6) 
 backlog 0b 0p requeues 6
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 44484923 bytes 214119 pkt (dropped 0, overlimits 0 requeues 6) 
 backlog 0b 0p requeues 6
  maxpacket 1506 drop_overlimit 0 new_flow_count 7 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev ifb-dns root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8019: dev eth0.2 root refcnt 2 bandwidth 19500Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 
 Sent 682951659 bytes 5089770 pkt (dropped 1392, overlimits 554789 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 1256Kb of 4Mb
 capacity estimate: 19500Kbit
 min/max network layer size:           16 /    1500
 min/max overhead-adjusted size:       50 /    1534
 average network hdr offset:           14

                  Tin 0
  thresh      19500Kbit
  target            5ms
  interval        100ms
  pk_delay       1.04ms
  av_delay         63us
  sp_delay          3us
  backlog            0b
  pkts          5091162
  bytes       685029476
  way_inds       778317
  way_miss        50952
  way_cols            0
  drops            1392
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum           595

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 16554427668 bytes 11682160 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wg0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 1301404659 bytes 6340147 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 7200 drop_overlimit 0 new_flow_count 682 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 801a: dev ifb4eth0.2 root refcnt 2 bandwidth 195Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34 
 Sent 16641453302 bytes 11631309 pkt (dropped 50851, overlimits 4254996 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 2229568b of 9750000b
 capacity estimate: 195Mbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       80 /    1534
 average network hdr offset:           14

                  Tin 0
  thresh        195Mbit
  target            5ms
  interval        100ms
  pk_delay       2.07ms
  av_delay        1.3ms
  sp_delay         39us
  backlog            0b
  pkts         11682160
  bytes     16717977908
  way_inds      1723274
  way_miss        50770
  way_cols            0
  drops           50851
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum          1514

That would very much indicate that your bufferbloat is not caused by your router (or that there are transient CPU overload conditions that randomly affect some tests but not others)...

If you want to dive deeper into this just holler, there is no guarantee for a positive outcome though....

Maybe... if the shaper rate and per-packet-overhead is set correctly there is little need to leave room for protocol traffic. For ingress leaving some room is somewhat important as SQM needs to create an artificial bottleneck, if the SQM-rate is the >= as the upstream rate then in download/ingress SQM will never built up a queue (all queueing happens in the upstream device) and will hence not be able to effectively control bufferbloat (because it never sees it). Cake's ingress keyword helps a bit in that it makes cake's control loop look primarily at the rate of incoming traffic and not at the traffic leaving cake, but that still requires that the configured download rate in cake is smaller than the true rate.

Which tc version and OpenWrt version are you running?

1 Like

Like said above, current master

1 Like

Shoot, I guess I will need to install a snapshot on my old wndr3700v2 for testing...

I would suggest using the Speedtest app.

It takes the browser out of the equation.

Yes, I was using it, but the speed is always almost the half than other services. I tried a lot of servers also…but all are under 80/90mbps

Same device used for both tests?

Yes, it’s a screenshot of my iPad with the browser and the Speedtest app side by side.

What do you get with a wired connection?

Unfortunately at the moment I can't test wired, but anyway both the services are running wireless, so it should be similar. I suspect the weird difference also wired.

You might be surprised.

I get close to wired speeds on 5 GHz, not so much on 2.4 GHz.

I get vastly different results on speedtest.net depending on the browser I use. Firefox is fastest. Chrome is a dog on speedtest.net. Best try to use iperf3 instead

iperf is for internal speed, not ISP :slight_smile:

Funny thing is that with every client (iPhone macOS iPad) on my lan, only with Speedtest.net app/website, with SQM enabled, can’t go over 90mbps, disable SQM and the speed goes back to 150/160mbps.

But with other Speedtest like dslreports or waveform, the speed is normal (130/150mbps) also with SQM enabled.

I don’t know, can’t understand what is going on with Speedtest.net and SQM. Maybe because it sends multiple connections and the the SQM are cutting its bandwitch? Could be possible?!

Strictly speaking, what stops us from using iperf to measure internet access speed is mostly the lack of fast enough and publicly reachable servers on the internet, specifically at the desired locations in regards to one's own ISP. But there are some iperf servers out there which can be used (google is your friend when trying to find them).

Not really, essentially all speedtests default to multiple connections nowadays, so it is unlikely for that to be your problem. But ATM we have too little data to even speculate...

1 Like

Good point to find out some iperf servers out in the wild :smiley: I haven't tought about it.

Indeed, for that I don't know why this limit on speedtest.net. I simply avoid it and I test with other services.