Best practice for SQM 2.5Gigabit on WAN and 1GB on LAN?

Installing and Using OpenWrt
1

Hi all,
I currently have a DOCSIS 3 cable modem that has a 2.5GB port, my OpenWRT router (x86 N5105 @ 2.00GHz) is connected to it and it has 4x2.5GB ports, one is setup as the WAN, I can see the link at 2.5GB on Openwrt.

My contracted ISP speed is 1.5GB (Rogers Canada).

All my computers, IPTV boxes and game consoles are connected through LAN via switches, switches have 1GB ports and are connected to the Openwrt Router.

My question is, How do I properly setup SQM under this circumstance? Should I run the speedtest on the modem or on the PC devices? If I do it on the PC Devices I fear that I will configure SQM with a smaller limit for the download shaper that it should. But if I measure speedtests on the modem, I get higher speed limit for the shaper, but I do not know if this is the best practice for this scenarios.

It is worthy to mention that my bufferbloat tests on waveform.com have never reached an A+, best I get is an A with around 20ms for download no matter how I configure SQM.

Upload is great it always gives me 0ms under load when I turn on SQM.

Thanks,
Roberto

1 Like
2

What is your upload speed?

3

The upload is 50mbps

4

That is insufficient if you disabled SACK "for gaming latency".
1300B packet now gets 64B backtalk and 75Mbps up is needed to sustain 1.5Gbps down.
Enable sack, that quarters backtalk.

1 Like
5

In this case you should run the speedtest from your OpenWrt router... (typically that is not recommended as many routers are fine with routing and traffic shaping at X Mbps, but if you ask them to also source and sink that traffic the performance goes down). Or connect a computer via 2G5 Ethernet to the OpenWrt router.
For SQM to work well, it needs to see all traffic, so never connect anything else to the cable modem except for the OpenWrt router (that also includes WiFi devices)...

How did you do these? Could you post a link to the results page please?

Where is this coming from, I do not see any mention of selective ACKs in the OP's post?

For TCP Reno with the default delayed ACKs and full MTU sizes forward traffic, the reverse ACK traffic is ~1/40 of forward traffic, so at 1500 Mbps that would be 1500/40 = 37.5 Mbps which still fits into 50 Mbps, modern TCP stacks often ACK less often then every two full segments received, so these 1/40 is sort of the worst case scenario.
That said, 1500/50 Mbps is genuinely a unfortunate offer, but you always can shape the download to get a better/smaller asymmetry if that bothers you.

1 Like
6

SACK acks 4 1500B (or 13xx B frames for major CDN-s) frames at once. Exactly four. Nothing modern.

7

Sorry, the big thing about SACK is that is allows to selectively tell the sender about individual missing segments (classic accumulative ACK can only signal a single missing segment per RTT), so the sender does not need to resend everything after the missing segment again. How often you send an ACK is independent on whether you use SACKs or not, at least that is my take away from reading:

But I might be missing something, if so please help me understand what I got wrong.

Sidenote: the fact that up top 4 SACK blocks can be transmitted per ACK does not mean it needs to be sent every 4 packets, as the sack options for continuously received segments are still cumulative.

1 Like
8

In your opinion - what happens when acks disappear? who notifies what?

9

If a single ACK disappears nothing much happens, as ACKs are cumulative (even SACKs) the sender will learn about the segments that reached the receiver just a bit later... if all ACKs are lost the sender will eventually stop transmitting, after having exhausted the min(receiver window, congestion window), but for all of these in-flight transmission there will be an active retransmission timer that initiates a retransmission (the sender can not disambiguate between all data segments or all ACKs were lost, and resending data segment will elicit ACKs in the receiver).
This is also why ACK filtering/ACK thinning works, a router that sees a queue of back to back ACKs for the same connection, can simply merge all of these thereby reducing the amount of ACK traffic at the cost of having less redundancy and being less resilient to additional ACK losses. This can also help on bursty links. In cake you can use the ack-filter keyword to achieve that, please note that this typically makes more sense in the upstream direction, in the download direction all ACKs will already have passed the bottleneck anyway. Also on nicely symmetrical up/down ratios this is less of an issue, than for extremely asymmetric link's like the OP's.

1 Like
10

Asymmetry shoots itself in the foot.... Delay based retransmit....

11

Thank you so much both of you, for your time on this and for your replies.

When I do speedtest from the router and before enabling SQM I get this. I repeated it a few times and I get similar results:

IP: xxx.xxx.x.xxx ( Rogers Cable ) Location: [xx, xx]
Server #1651151856:  wirelinespeedtestkanata.srvr.bell.ca.prod.hosts.ooklaserver.net:8080 by  (4.57552e-41 km from you): 0 ms (chosen by recommendation)
Ping: 0 ms.
Jitter: 2 ms.

server selected profile detected: profile selected from server configuration

Download: 1933.04 Mbit/s
Upload: 46.48 Mbit/s

When I enable SQM and run tests from a computer wired to the router I get different results, some like this:

I tried to follow the SQM config guide to do my current sqm config, it is below: ETH1 is my wan.

config queue 'eth1'
	option enabled '1'
	option interface 'eth1'
	option download '1732000'
	option upload '36318'
	option qdisc 'cake'
	option script 'piece_of_cake.qos'
	option linklayer 'ethernet'
	option debug_logging '0'
	option verbosity '5'
	option overhead '42'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option iqdisc_opts 'mpu 84 nat dual-dsthost ingress'
	option eqdisc_opts 'mpu 84 nat dual-srchost'

I believe SACK is enabled by default, I havent changed this on sysctl, verified the defaults and it is on in sysctl.d

Everything goes through the router, I have a main switch that distribute the network to the different rooms small switches in each of the room to distribute the lan there, I do have a unifi AP that broadcasts the wifi this one connected to the Openwrt router.

12

What speed does that switch support?

13

Only 1 gbps

14

Mmmh, odd, in download direction BQL and the default fq_codel should do the right thing...

Time to check whether your lan ethernet port supports BQL....

15

The bad thing is that downloader side voluntarily sends some acks, at max rate they fill upload.
Only on WAN, br-lan can have pfifo or noqueue depending on if you have vlans.
0 - fq_codel target 20ms
1 - pfifo_fast or sqm or fq_codel
2 - tbf with limit and one of above after
3 - cake but with ack filters

16

What does:
cat /sys/class/net/eth1/queues/tx-0/byte_queue_limits/limit
return?
And what do you get on the port connected to the switch?

17

The content is:

root@homerouter:/etc/config# cat /sys/class/net/eth1/queues/tx-0/byte_queue_limi
ts/limit
3028
18

I tried fq_codel on WAN as per:

config queue 'eth1'
	option enabled '1'
	option interface 'eth1'
	option download '1732000'
	option upload '36318'
	option qdisc 'fq_codel'
	option script 'simple.qos'
	option linklayer 'ethernet'
	option debug_logging '0'
	option verbosity '5'
	option overhead '42'
	option qdisc_advanced '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option itarget '20ms'

results:

19

I get this for the one that has the switch connected to it:

root@homerouter:/etc/config# cat /sys/class/net/eth0/queues/tx-0/byte_queue_limi
ts/limit
48448
20

'tc -s qdisc`
And compare to setting even higher latency targets.
What ping latency ispinging wan gateway or providers website? You cannot get much better than that.