FYI. I have noticed with my ISP Charter.com Docsis 3.0, with SQM my goodput is -8% ingress and -6% egress from any total bandwidth entered. This is whether or not 18 bytes overhead is specified.
Well, the expected goodput would be at (100 * (1500 - 20 - 20)/(1500 + 18)) = 96.18% so around 4% of the "loss" you observe is purely caused by the fact that the shaper's bandwidth is specified as gross bandwidth, while goodput is defined typically as TCP/IP payload size. So we are talking about 2 to 4% percent under performance, but how do you test that, especially how many concurrent streams do you use?
Dslreports 16/4 betterspeedtest.sh 5/5
Ah, seems reasonable,. The reason I asked is that a single TCP stream will not be able to saturate a link, but once you add a few tcp streams the loss by TCPs up- and down ramping (attempting to probe the available bandwidth) gets smaller, at 16 streams I certainly would expect less than 4% though.
One other question to think about would be TCP options or smaller MTU than 1500 (for example I use PPPoE and tcp timestamps and 26 bytes of overhead, so expect I goodput to be maximally at (100 * (1500 - 8 - 20 - 20 - 12)/(1500 + 26))) = 94.36% of (dePTMd) Syncrate. PPPoE is not going to be your problem, but TCP options and smaller MTUs on the path t the speedtest server might be... For a quick and dirty test I recommend http://www.speedguide.net/analyzer.php which should show the path MTU as well as whether tcp timestamps are in use...
here it is SG TCP/IP Analyzer
IP Address: 68.185.237.4 ()
Client OS: Mac OS
Browser: Chrome 57.0.2987.133
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
Please Read the Analyzer FAQ if the above is not your IP address.
TCP options string = 020405b4010303050101080a35d9693c0000000004020000
MTU = 1500
MTU is fully optimized for broadband.
MSS = 1460
Maximum useful data in each packet = 1448, which is less than MSS because of Timestamps, or other TCP/IP options used.
Default TCP Receive Window (RWIN) = 131744
RWIN Scaling (RFC1323) = 5 bits (scale factor: 2^5=32)
Unscaled TCP Receive Window = 4117
Your TCP Window limits you to: 5270 kbps (659 KBytes/s) @ 200ms
Your TCP Window limits you to: 2108 kbps (263 KBytes/s) @ 500ms
MTU Discovery (RFC1191) = ON
Time to live left = 54 hops
TTL value is ok.
Timestamps (RFC1323) = ON
Note: Timestamps add 12 bytes to the TCP header of each packet, reducing the space available for useful data.
Selective Acknowledgements (RFC2018) = ON
IP type of service field (RFC1349) = 00000000 (0)
Ah, so RFC1323 timestamps are enabled, so the expected maximal goodput will be at:
(100 * (1500 - 20 - 20 -12)/(1500 + 18)) = 95.39% of the shaped rate. That still leaves 1.5 to 3.5% missing. Not too bad, actually. How long do you run the speedtests? I typically recommend to go at least for 30 seconds see https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803 for how to get the most out of the dslreports speedtest...
The only other ting to try would be to repeat the speedtest at different timepoints (including in the dead of night) to see whether it might be related to overall traffic in charter's network...
Best Regards
ECN must be enabled on the client you're using for speed testing. (It's not enabled by default on about every operating systems)
I've tested many many times between 8-10am, 12am-4am, and after 6pm. The same results no matter the bandwidth entered, overhead, what-not. The throughput remains the same; as far as ecn, enabled that long ago on my Macbook Pro running Yosemite. See below>
Jasons-MacBook-Pro:~ Jason$ sysctl -a | grep ecn
net.inet.tcp.ecn_initiate_out: 1
net.inet.tcp.ecn_negotiate_in: 1
net.inet.ipsec.ecn: 1
net.inet6.ipsec6.ecn: 1
root@LEDE:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 384438196 bytes 1026218 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8013: dev eth1 root refcnt 2 bandwidth 5Mbit besteffort dual-srchost nat rtt 40.0ms noatm overhead 18 via-ethernet mpu 64
Sent 23611531 bytes 92966 pkt (dropped 7, overlimits 141652 requeues 0)
backlog 0b 0p requeues 0
memory used: 294272b of 4Mb
capacity estimate: 5Mbit
Tin 0
thresh 5Mbit
target 3.6ms
interval 41.6ms
pk_delay 7.8ms
av_delay 3.4ms
sp_delay 14us
pkts 92973
bytes 23616188
way_inds 12
way_miss 315
way_cols 0
drops 7
marks 2537
sp_flows 1
bk_flows 1
un_flows 0
max_len 1514
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 221169550 bytes 157285 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8014: dev ifb4eth1 root refcnt 2 bandwidth 66Mbit besteffort dual-dsthost nat wash rtt 40.0ms noatm overhead 18 via-ethernet mpu 64
Sent 223371540 bytes 157285 pkt (dropped 0, overlimits 202856 requeues 0)
backlog 0b 0p requeues 0
memory used: 238080b of 4Mb
capacity estimate: 66Mbit
Tin 0
thresh 66Mbit
target 2.0ms
interval 40.0ms
pk_delay 54us
av_delay 14us
sp_delay 5us
pkts 157285
bytes 223371540
way_inds 5
way_miss 315
way_cols 0
drops 0
marks 123
sp_flows 1
bk_flows 1
un_flows 0
max_len 1514
The RTT 40ms ingress & egress is experimenting for gaming to a dedicated call of duty server located in San Antonio per MSN Network. The sole reason I even cared about latency was from latency sensitive games and voip. What a journey it's been haha. I also noticed latency and/or buffer bloat doesn't oscilate much from full throughput to even 1/2 using sqm. Lower than 30% goodput cause damage it seems. Something I'm not seeing here.
OK, so disabling GRO on smaller links should be stickied! 
It didn't change my bandwidth mystery, but did improve latency a lot.
All the information I've read points to ethtool.
1 Like
OP solved! Replacing the Thomson DCM475 modem with a Hitron CDA-RES did it. Piece of cake SQM on an Archer C7 led to A+ bufferbloat at full 130Mbps. The Thomson modem simply had an oscillating bufferbloat that couldn't be controlled by SQM, whereas the Hitron has a constant 500ms bufferbloat that can be smoothed out. 
@Sunspark; Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully (specify overhead NN) e.g. overhead 28
My Config:
config queue 'eth1'
option interface 'eth1'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option itarget 'auto'
option etarget 'auto'
option linklayer 'none'
option debug_logging '0'
option verbosity '5'
option qdisc 'cake'
option qdisc_advanced '1'
option squash_dscp '1'
option squash_ingress '1'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'overhead 35 dual-dsthost nat mpu 64'
option eqdisc_opts 'overhead 28 dual-srchost nat mpu 64'
option enabled '1'
option script 'piece_of_cake.qos'
option download '60000'
option upload '4000'
For a cable link these look suspicious, I would believe that setting both up- and downstream overhead to 18 would be theoretically more correct for a cable link. But if you empirically found your values, please ignore my theoretical concerns...
Best Regards
I'm all over the map here and although things mostly work, i'm ready to chunk this wndr3800ch and buy a newer router. It's been a test router till lede gets sorted but it's acting weird. Upgrading to the newer 4.4.61 has seemingly corrected some odd behavior in sqm but other bugs with redirects/port forwading I've posted are there. It's hard to get a firm conclusion without something stable. I'm gonna try fq_codel/simple.qos for a week on this new kernel and check for differences. Cake is too inconsistent with ANY setting applied.
Well, sad to hear that. I run a wndr3700v2 on a 50/10 link and cake works pretty well for me, but I do not play on-line games, twitch, stream or even bit-torrent, so my traffic patterns are probably pretty boring and well behaved....
My main issues is the latency spikes well over 30ms. Charter has unusually fluctuations in rtt and jitter even considering the routes. Here are some mtr runs
Pearl River,LA <----->Google
My traceroute [v0.85]
LEDE (0.0.0.0) Fri Apr 21 15:19:02 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
- ???
- dtr03slidla-tge-0-1-0-1.slid.la. 0.0% 16 11.0 13.7 10.0 21.0 3.8
- 96-34-88-16.static.unas.la.chart 0.0% 16 23.9 15.9 11.3 26.4 4.1
- bbr02slidla-bue-1.slid.la.charte 0.0% 16 11.8 15.1 10.5 24.4 3.6
- bbr01sgnwmi-tge-0-2-0-10.sgnw.mi 0.0% 16 26.3 28.7 25.6 34.1 2.4
- prr01snjsca-tge-0-0-0-1.snjs.ca. 0.0% 16 29.7 30.0 26.7 51.8 5.9
- 72.14.220.17 0.0% 15 25.2 28.9 25.2 35.8 3.4
- 72.14.239.100 0.0% 15 27.4 27.5 24.7 34.4 2.7
- 72.14.233.199 0.0% 15 27.3 35.1 25.8 87.8 17.6
- 216.239.40.131 0.0% 15 40.9 45.0 39.4 75.3 9.0
- 216.239.48.7 0.0% 15 55.7 48.3 42.4 75.3 9.6
- 216.239.49.180 0.0% 15 39.3 42.3 39.3 50.0 3.2
- 108.170.246.1 0.0% 15 40.9 42.1 38.4 48.7 3.1
- 216.239.54.111 0.0% 15 39.7 41.6 39.1 55.1 3.9
- iad30s07-in-f14.1e100.net 0.0% 15 42.2 43.7 41.3 48.7 1.7
Pearl River,LA<----->Nola.com
My traceroute [v0.85]
LEDE (0.0.0.0) Fri Apr 21 15:21:48 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
- ???
- dtr03slidla-tge-0-1-0-1.slid.la. 0.0% 63 21.9 12.8 8.7 29.6 4.3
- 96-34-88-16.static.unas.la.chart 0.0% 62 13.1 15.0 10.2 43.1 5.2
- lag-102.bear1.Houston1.Level3.ne 0.0% 62 51.2 22.8 16.9 52.8 8.0
- ae-1-3500.ear1.Washington111.Lev 4.8% 62 43.0 48.7 43.0 151.0 14.1
- DYNAMIC-NET.ear1.Washington111.L 0.0% 62 44.4 46.6 42.2 81.2 6.5
- redirector2.dynect.net 0.0% 62 43.9 46.0 43.1 62.2 3.8
Peal River,LA<----->www.uno.edu
My traceroute [v0.85]
LEDE (0.0.0.0) Fri Apr 21 15:24:25 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
- ???
- dtr03slidla-tge-0-1-0-1.slid.la. 0.0% 24 17.5 11.8 8.4 20.0 2.7
- 96-34-88-16.static.unas.la.chart 0.0% 24 24.1 15.8 11.4 29.9 4.2
- bbr01olvemo-tge-0-1-0-6.olve.mo. 0.0% 24 24.7 26.8 23.8 38.9 3.1
- prr02dllstx-bue-2.dlls.tx.charte 0.0% 24 25.5 26.0 22.9 35.4 3.2
- 96-34-149-54.static.unas.mo.char 0.0% 24 22.9 37.4 22.9 83.3 16.9
- 64.57.21.66 0.0% 24 46.4 35.4 32.3 46.4 3.0
- ???
I will say both FQ_codel/Cake have alleviated close to 95% of my bufferbloat both on ethernet connections and 5ghz band. 2.4ghz is borderline saturation so depending on weather, time of day, etc it's still very nice. Gaming servers and Voip have been my main hotpoints and taking into consideration the other "end" is not well configured, too many variables come into play. I need to contruct a test lab of own to show conclusive lan results vs wan. The Xbox One is not the greatest design from Microsoft. My quest for the lowest bufferbloat and rtt figures continues...
I do have one burning question, would 2 routers, 1 acting as the edge router with sqm, and a switch/router to the lan with sqm make any difference in reducing bloat? My theory is the extra processing power may help, but also induce latency from having the extra routing maybe? check out this video https://youtu.be/T5BHDbAgdng
2 Likes
So I have 2 Netgear R7000's laying around. I installed the latest lede on one acting as a gateway, and the other running stock firmware. Wireless is definitely the culprit of my latency spikes. Any wifi activity judo chops my connection with spikes. WIthout wifi active everything is smooth as butter. Arrgggg! My little wndr3800ch is still fine so I'll keep it as a spare. Bufferbloat is still good with this setup. For the time being, I'll consider it done.