SQM any other than 'upload 0' worsens on tagged VLAN fiber WAN

willyj · September 29, 2020, 9:18pm

I'm testing it from a 802.11ac R7800 5GHz channel (148) connected to my WAN modem with a tagged VLAN (Nokia, fiber), no software/hardware offloading enabled, with a client Intel Dual Band Wireless AC 8260 adapter on Windows 10. I've been satisfied with both the router and the client and remain one of the best consumer client/router combinations I've tried. Upload is probably traffic shaped somehow, given the results I get (always worse when capping upload) as mentioned in https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details on fiber networks.

Since I've read a lot of threads here on sqm and tried to do my own research as best as I can, I also took note of what commands and tests are asked most frequently, so here is the long list of outputs:
Some starting tests:
no sqm: http://www.dslreports.com/speedtest/65765295
with sqm: http://www.dslreports.com/speedtest/65765303

# uname -a

Linux OpenWrt 4.14.195 #0 SMP Sun Sep 6 16:19:39 2020 armv7l GNU/Linux

# /etc/config/sqm (Left additional inactive option comments for what I've tried. To explain it quick and accurate).

config queue 'eth0'
        option enabled '1'
        option interface 'eth0.11'
        option qdisc_advanced '0'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
#       14-17ms peak varies on fast.com with 4 connections, bufferbloat class A on dslreports
        option download '214000'
#  Anything other than '0' gives me worse results after many tests.
        option upload '0'
# bursty and assymmetric links? upload shaping? Does not improve
#       option eqdisc_opts 'nat dual-srchost ack-filter'
# 4 byte VLAN tag and fiber, should equal to 44 or so overhead, so far no overhead setting has improved results
#        option overhead '4' # also tried '44', '36', '38'
# Zone wan MTU fixing is enabled per default in firewall on forward chain.
# Linklayer never had effect
#        option linklayer 'ethernet'
# I also tried these two following ingress/egress combinations
#         option ingress_ecn 'ECN'
# And tried this one alone again because upload seems specially managed
#         option egress_ecn 'NOECN'

# tc -s qdisc

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 1306345 bytes 3514 pkt (dropped 0, overlimits 0 requeues 9)
 backlog 0b 0p requeues 9
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 1306345 bytes 3514 pkt (dropped 0, overlimits 0 requeues 9)
 backlog 0b 0p requeues 9
  maxpacket 1494 drop_overlimit 0 new_flow_count 4 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
 Sent 55873 bytes 598 pkt (dropped 0, overlimits 0 requeues 3)
 backlog 0b 0p requeues 3
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 55873 bytes 598 pkt (dropped 0, overlimits 0 requeues 3)
 backlog 0b 0p requeues 3
  maxpacket 110 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.11 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

# tc -d qdisc

qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth1.1 root refcnt 2
qdisc noqueue 0: dev eth0.11 root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2

# ifstatus wan

{
        "up": true,
        "pending": false,
        "available": true,
        "autostart": true,
        "dynamic": false,
        "uptime": 1389,
        "l3_device": "eth0.11",
        "proto": "dhcp",
        "device": "eth0.11",
        "updated": [
                "data"
        ],
        "metric": 0,
        "dns_metric": 0,
        "delegation": false,
        "ipv4-address": [
                {
                        "address": "96.1.1.54",
                        "mask": 23
                }
        ],
        "ipv6-address": [

        ],
        "ipv6-prefix": [

        ],
        "ipv6-prefix-assignment": [

        ],
        "route": [
                {
                        "target": "0.0.0.0",
                        "mask": 0,
                        "nexthop": "96.1.1.1",
                        "source": "96.1.1.54/32"
                }
        ],
        "dns-server": [
                "89.112.1.2",
                "89.112.1.3"
        ],
        "dns-search": [
                "dns.srv.recv.com"
        ],
        "neighbors": [

        ],
        "inactive": {
                "ipv4-address": [

                ],
                "ipv6-address": [

                ],
                "route": [

                ],
                "dns-server": [

                ],
                "dns-search": [

                ],
                "neighbors": [

                ]
        },
        "data": {
                "leasetime": 3600
        }
}

# SQM_DEBUG=1 SQM_VERBOSITY_MAX=11 /etc/init.d/sqm stop

SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root fq_codel
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root fq_codel
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: QDISC fq_codel is useable.
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root codel
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root pie
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root sfq
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: QDISC cake is useable.
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
/usr/lib/sqm/update-available-qdiscs: line 57: can't create : nonexistent directory
SQM: Acquired run lock
/usr/lib/sqm/run.sh: line 57: can't create : nonexistent directory
Command failed: Not found

# SQM_DEBUG=1 SQM_VERBOSITY_MAX=11 /etc/init.d/sqm start

SQM: Acquired run lock
/usr/lib/sqm/run.sh: line 57: can't create : nonexistent directory
SQM:
SQM: Tue Sep 29 20:54:13 UTC 2020: Starting.
SQM: Starting SQM script: piece_of_cake.qos on eth0.11, in: 214000 Kbps, out: 0 Kbps
SQM: fn_exists: function candidate name: sqm_start
SQM: fn_exists: TYPE_OUTPUT: sqm_start: not found
SQM: fn_exists: return value: 1
SQM: Using generic sqm_start_default function.
SQM: fn_exists: function candidate name: sqm_prepare_script
SQM: fn_exists: TYPE_OUTPUT: sqm_prepare_script is a function
SQM: fn_exists: return value: 0
SQM: sqm_start_default: starting sqm_prepare_script
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
SQM: QDISC cake is useable.
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
SQM: QDISC cake is useable.
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: sqm_start_default: Starting piece_of_cake.qos
SQM: ifb associated with interface eth0.11:
SQM: Currently no ifb is associated with eth0.11, this is normal during starting of the sqm system.
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name ifb4eth0.11 type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name ifb4eth0.11 type ifb
SQM: sqm_start_default: egress shaping deactivated
SQM: cmd_wrapper: tc: invocation silenced by request, FAILURE either expected or acceptable.
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc del dev eth0.11 root
SQM: cmd_wrapper: tc: FAILURE (2): /usr/sbin/tc qdisc del dev eth0.11 root
SQM: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: No such file or directory
SQM: cmd_wrapper: COMMAND: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM ingress
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM ingress
SQM: QDISC ingress is useable.
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: cmd_wrapper: COMMAND: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: fn_exists: function candidate name: ingress
SQM: fn_exists: TYPE_OUTPUT: ingress is a function
SQM: fn_exists: return value: 0
SQM: ingress
SQM: cmd_wrapper: tc: invocation silenced by request, FAILURE either expected or acceptable.
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc del dev eth0.11 handle ffff: ingress
SQM: cmd_wrapper: tc: FAILURE (2): /usr/sbin/tc qdisc del dev eth0.11 handle ffff: ingress
SQM: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: No such file or directory
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc add dev eth0.11 handle ffff: ingress
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc add dev eth0.11 handle ffff: ingress
SQM: cmd_wrapper: tc: invocation silenced by request, FAILURE either expected or acceptable.
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc del dev ifb4eth0.11 root
SQM: cmd_wrapper: tc: FAILURE (2): /usr/sbin/tc qdisc del dev ifb4eth0.11 root
SQM: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: No such file or directory
SQM: LLA: default link layer adjustment method for cake is cake
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc qdisc add dev ifb4eth0.11 root cake bandwidth 214000kbit besteffort wash
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc qdisc add dev ifb4eth0.11 root cake bandwidth 214000kbit besteffort wash
SQM: cmd_wrapper: COMMAND: /sbin/ip link set dev ifb4eth0.11 up
SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev ifb4eth0.11 up
SQM: cmd_wrapper: COMMAND: /usr/sbin/tc filter add dev eth0.11 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0.11
SQM: cmd_wrapper: tc: SUCCESS: /usr/sbin/tc filter add dev eth0.11 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0.11
SQM: sqm_start_default: ingress shaping activated
SQM: piece_of_cake.qos was started on eth0.11 successfully

https://www.speedguide.net/analyzer.php

https://www.speedguide.net/analyzer.php
Got notice about: https://www.speedguide.net/faq/how-to-disable-windows-vista-tcpip-auto-tuning-247
« SpeedGuide.net TCP Analyzer Results » 
Tested on: 2020.09.29 14:34 
IP address: xx.xx.xx.xx 
Client OS/browser: Windows 10 (Firefox 78.0) 
 
TCP options string: 020405b40103030801010402 
MSS: 1460 
MTU: 1500 
TCP Window: 131328 (not multiple of MSS) 
RWIN Scaling: 8 bits (2^8=256) 
Unscaled RWIN : 513 
Recommended RWINs: 64240, 128480, 256960, 513920, 1027840 
BDP limit (200ms): 5253kbps (657KBytes/s)
BDP limit (500ms): 2101kbps (263KBytes/s) 
MTU Discovery: ON 
TTL: 108 
Timestamps: OFF 
SACKs: ON 
IP ToS: 00000000 (0)

Upload seems more jumpy than download, but also worsens when capped with anything other than caption upload '0' I don't know, but I assume the way the upload speed is managed is different from download, and hence why another layer with SQM doesn't improve things, but rather worsens it.

At some point while testing the ingress_ecn and egress_ecn options enabling and disabling each of them I noticed that bufferbloat changed from spikes and flickering numbers ranging from 0-30ms to a predictable pause that changed only from 0, 100ms without any other numbers displayed. Like it matched up 100% to what is displayed in tc -s qdisc command output above ^

qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn

Exactly: interval 100.0ms

To find more about my network I did a sudo mtr -ezb4 -i 0.5 1.1.1.1 from within WSL (thats why the starting IP is 172.29.x.x IP)

 Host                                                                                                                                                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    lenovo.mshome.net (172.29.224.1)                                                                                                                         0.0%   212    0.6   0.6   0.2   1.4   0.2
 2. AS???    OpenWrt.lan (10.0.0.1)                                                                                                                                     46.7%   212    3.7   2.3   1.3   5.6   0.7
 3. AS43557  81.2.195.1 (85.2.195.1)                                                                                                                                 0.0%   212    3.1   5.4   2.2  16.2   2.1
 4. AS43557  host01-02.net.de (85.2.195.55)                                                                                                             0.0%   212    5.0   6.1   3.1  13.8   1.7
    [MPLS: Lbl 53 TC 0 S 1 TTL 1]
 5. AS43557  host01-02-03.net.de (85.2.195.22)                                                                                                                  0.0%   212    7.6   6.9   3.5  16.8   1.7
    [MPLS: Lbl 84 TC 0 S 1 TTL 1]
 6. AS???    10.10.1.53 (10.10.1.53)                                                                                                                                     0.0%   211    7.9   7.7   4.8  25.8   2.1
    [MPLS: Lbl 20 TC 0 S 1 TTL 1]
 7. AS???    10.10.0.1 (10.10.0.1)                                                                                                                                       0.0%   211    5.8   7.8   4.9  14.4   1.5
 8. AS43557  host-01-02-03-04.net.de (201.195.201.206)                                                                                                               0.0%   211    6.8   8.2   5.5  12.8   1.3
 9. AS39642  host-01-02-03-04.link.net.de (89.101.128.135)                                                                                                             0.0%   211   12.7  12.4   9.2  24.1   1.7
    [MPLS: Lbl 24014 TC 0 S 0 TTL 1]
    [MPLS: Lbl 24003 TC 0 S 1 TTL 1]
10. AS39642  host-01-02-03-04-05.link.net.de (89.101.128.135)                                                                                                            0.0%   211   13.2  12.2   9.4  16.2   1.4
    [MPLS: Lbl 24041 TC 0 S 0 TTL 1]
    [MPLS: Lbl 24003 TC 0 S 1 TTL 2]
11. AS197288 host-01-02-03-04-05.br0.link.net.de (202.9.40.197)                                                                                                              0.0%   211   13.6  11.7   8.9  15.2   1.3
12. AS197288 210.11.12.253 (210.11.12.253)                                                                                                                               0.0%   211   13.8  13.3   9.3  42.0   4.7
13. AS13335  one.one.one.one (1.1.1.1)                                                                                                                                   0.0%   211   11.1  11.3   8.5  14.8   1.3

The only interesting thing I see here is the [MPLS] tags, latency doesn't seem too bad for the average consumer ISP.

I hope someone can help me with what the next step should be, so I don't try a lot of advanced stuff that isn't relevant to my connection.

I simply want to bring bufferbloat and latency down as much as possible when capping my speeds. I hope my config, speedtest, commands and the comments were helpful. If not, don't hesitate I will provide the information.

moeller0 · September 30, 2020, 1:09pm

Please help me understand your set-up.

Is the following sketch roughly correct:

ISP <-( fiber )-> Nokia-FiberModem <-( wired ethernet )-> OpenWrtR7800 router <-( 5GHz channel 148 )-> Win10 Client

How do the results look when you connect Win10 with an ethernet cable to OpenWrtR7800?

Where does the VLAN tagging come into play, only on ( fiber ) or also on ( wired ethernet ) ?

Which ISP, mtr and the dslreports speedtest seems to implicate AS43557 (which seems to be in Danmark, while 'whois 85.2.195.1' seems to implicate bluewin.ch/swisscom in Switzerland).

willyj · September 30, 2020, 1:40pm

The VLAN tagging is only on the WAN interface.

You are right, it is in Denmark, provider Waoo/Eniig, I only masked the addresses to not leak any sensitive information - simply

I have not checked dslreports from ethernet cable, but it is possible and will do as soon as I come home (help a friend moving). This will be via my laptop however, as I cannot move the desktop right now.

I appreciate the response.

moeller0 · September 30, 2020, 2:05pm

Thanks for clarifying.

Yes stripping the IP is not a bad idea (but I be the IP is dynamically assigned, so not a really sensitoive thing to leak either).

Great, typically it is much easier to configure SQM for wired use-cases as that avoids all the uncertainty wifi can drag in. Don't get me wrong, wifi can be low latency as well, but typically is harder to configure and also prone to RF ingress issues that can make low latency impossible.
BTW, the depth of your posting and the level of detail was/is excellent!

willyj · September 30, 2020, 10:19pm

Thank you, are you danish as well? (møller? hence your name )

reboot router openwrt r7800, no sqm, laptop (with adapter: realtek PCIe GbE Family Controller, wired), Best Performance battery level. No other clients connected.
no sqm

again the 0,100ms bufferbloat flashing during test.

with sqm

I then found some power management options on the realtek driver, I disabled Green Ethernet, Energy-Efficient Ethernet

I also found an option called "Flow Control", thought it might interfere so made 3 more tests without power management and flow control.

as I said, I'm really lucky with the pair of adapter to my desktop on 802.11ac located ~3m away. So even this laptop can't compete wired on Cat5e and the 149-164 5 GHz channels are completely free.

It looks almost like upload is 100mbps until it is boosted towards 200mbps from the provider - but maybe this is normal in fiber splitters. There's actually a great diagram here of Eniig's infrastructure:

I salute your SQM skills I have read many of your posts, its quite complicated but I notice especially secondary clients even with one host qbittorrent + 1000 connections blazing, I'm getting 10-40ms on continous ping with sqm to my AMS (vultr.com) server, and I believe that wasn't the case before. Probably the torrent host has overloaded tcp stack and/or driver and hence why sqm is very noticeable on other clients.

moeller0 · October 1, 2020, 7:33am

No sorry, I am from your southern neighbor, it is just that in our northern dialect the ü gets replaced by a ö, which phonetically is close to your ø...

so I note that all bufferbloat measurements show terrible spikes, and even without sqm the uplink throughput is quite variable (174.1/172.1/136.9)

but this looks just as bad, but I notice that the bufferbloat measurements all have issues ("54.8s Timer drops: frames=47 total ms=11150 slip=0") in that they drop probes.
At that point, I am not convinced that the dslreports speedtest is reliable enough from your position to be used as a de-bloating tool, sorry.

If you have a linux host, you could try my quick and dirty replacement script (which is really just a hack), just copy and paste the following into a file (e.g. combined_mtr_speedtest.sh), set it to be executable and run it like sudo ./combined_mtr_speedtest.sh. This requires both mtr and speedtest-cli to be installed and is only a rough first attempt at getting something cobbled together. I use speedtest-cli here not because I like the Ookla speedtest.net infrastructure, but simply because it is quite ubiquitious and it is very likely that one can find some close by nodes for generatimng a saturating load. ATM this defaults to google's 8.8.8.8 DNS server address which thanks to anycast should be relatively close by, if the RTT to 8.8.8.8 is above a few dozens ms you need to change that address.

#! /bin/bash
# if.

# show what is going to be run
#cat $0
#echo ""

SESSION_DATETIME=$( date "+%Y%m%dT%H%M%S" )
MTR_INTERVAL_SECS=0.2
MTR_HOST_IP="8.8.8.8"

echo "Starting unidirectional speedtest"
echo ""

# idle RTT
mtr -ezb4w -i ${MTR_INTERVAL_SECS} -c 100 ${MTR_HOST_IP} > mtr_idle_${SESSION_DATETIME}.out ;.
# loaded RTTs
mtr -ezb4w -i ${MTR_INTERVAL_SECS} -c 150 ${MTR_HOST_IP} > mtr_loaded_${SESSION_DATETIME}.out &.
# der speeedtest
speedtest-cli > speedtest_${SESSION_DATETIME}.out

# wait for background job to finish
wait
echo "Unidirectional speedtests finished..."
echo ""


echo "RTT to ${MTR_HOST_IP} idle:"
cat ./mtr_idle_${SESSION_DATETIME}.out
echo ""

echo "Speedtest.net results:"
cat ./speedtest_${SESSION_DATETIME}.out
echo ""

echo "RTT to ${MTR_HOST_IP} unidirectional loads:"
cat ./mtr_loaded_${SESSION_DATETIME}.out
echo ""



echo "Starting bidirectional speedtest"
echo ""


## loaded RTTs
mtr -ezb4w -i ${MTR_INTERVAL_SECS} -c 150 ${MTR_HOST_IP} > mtr_fully_loaded_${SESSION_DATETIME}.out &.

# der speeedtest
speedtest-cli --no-upload > speedtest_no-upload_${SESSION_DATETIME}.out &
speedtest-cli --no-download > speedtest_no-download_${SESSION_DATETIME}.out

# wait for background job to finish
wait
echo "Biidirectional speedtests finished..."
echo ""

echo "Speedtest.net result DOWN:"
cat ./speedtest_no-upload_${SESSION_DATETIME}.out
echo ""

echo "Speedtest.net results UP:"
cat ./speedtest_no-download_${SESSION_DATETIME}.out
echo ""

echo "RTT to ${MTR_HOST_IP} bidirectional load:"
cat ./mtr_fully_loaded_${SESSION_DATETIME}.out
echo ""



exit 0

Here is example output:

Starting unidirectional speedtest

Unidirectional speedtests finished...

RTT to 8.8.8.8 idle:
Start: 2020-10-01T09:21:32+0200
HOST: work-horse                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.42.1)                                             75.0%   100    0.4   0.4   0.3   0.5   0.1
  2. AS6805   loopback1.0002.acln.01.ham.de.net.telefonica.de (62.52.200.148)      0.0%   100   13.3  18.6  12.8  42.4   7.8
  3. AS6805   bundle-ether28.0005.dbrx.01.ham.de.net.telefonica.de (62.53.12.16)   0.0%   100   13.1  13.4  13.0  14.4   0.3
  4. AS6805   ae0-0.0001.prrx.01.ham.de.net.telefonica.de (62.53.25.57)            0.0%   100   13.0  12.9  12.5  19.1   0.8
  5. AS15169  74.125.48.102                                                        0.0%   100   13.6  13.1  12.7  13.7   0.2
  6. AS15169  216.239.63.21                                                        0.0%   100   15.3  15.1  14.5  17.7   0.5
  7. AS15169  216.239.54.187                                                       0.0%   100   13.3  13.1  12.7  13.7   0.2
  8. AS15169  dns.google (8.8.8.8)                                                 0.0%   100   12.9  13.1  12.6  13.6   0.2

Speedtest.net results:
Retrieving speedtest.net configuration...
Testing from O2 Deutschland (77.6.149.133)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by PLUTEX GmbH (Bremen) [101.12 km]: 23.481 ms
Testing download speed................................................................................
Download: 77.14 Mbit/s
Testing upload speed......................................................................................................
Upload: 22.41 Mbit/s

RTT to 8.8.8.8 unidirectional loads:
Start: 2020-10-01T09:22:04+0200
HOST: work-horse                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.42.1)                                             76.7%   150    0.3   0.3   0.2   0.5   0.1
  2. AS6805   loopback1.0002.acln.01.ham.de.net.telefonica.de (62.52.200.148)      0.0%   150   13.0  24.9  12.7  65.9  14.6
  3. AS6805   bundle-ether28.0005.dbrx.01.ham.de.net.telefonica.de (62.53.12.16)   0.0%   150   13.4  14.5  12.9  42.9   3.5
  4. AS6805   ae0-0.0001.prrx.01.ham.de.net.telefonica.de (62.53.25.57)            0.0%   150   13.0  14.1  12.5  46.0   3.8
  5. AS15169  74.125.48.102                                                        0.0%   150   14.2  14.3  12.8  44.2   3.5
  6. AS15169  216.239.63.21                                                        0.0%   150   15.4  16.2  14.6  40.6   3.1
  7. AS15169  216.239.54.187                                                       0.0%   150   12.8  14.0  12.8  32.9   2.2
  8. AS15169  dns.google (8.8.8.8)                                                 0.0%   150   13.2  13.8  12.7  32.4   1.8

Starting bidirectional speedtest

Biidirectional speedtests finished...

Speedtest.net result DOWN:
Retrieving speedtest.net configuration...
Testing from O2 Deutschland (77.6.149.133)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Spacken.net (Hagen) [302.91 km]: 35.587 ms
Testing download speed................................................................................
Download: 86.45 Mbit/s
Skipping upload test

Speedtest.net results UP:
Retrieving speedtest.net configuration...
Testing from O2 Deutschland (77.6.149.133)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Spacken.net (Hagen) [302.91 km]: 33.956 ms
Skipping download test
Testing upload speed......................................................................................................
Upload: 13.94 Mbit/s

RTT to 8.8.8.8 bidirectional load:
Start: 2020-10-01T09:22:40+0200
HOST: work-horse                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.42.1)                                             76.7%   150    0.3   0.3   0.2   0.4   0.1
  2. AS6805   loopback1.0002.acln.01.ham.de.net.telefonica.de (62.52.200.148)      0.0%   150   14.1  14.3  12.8  35.3   2.9
  3. AS6805   bundle-ether28.0005.dbrx.01.ham.de.net.telefonica.de (62.53.12.16)   0.0%   150   13.6  14.0  13.0  28.8   1.8
  4. AS6805   ae0-0.0001.prrx.01.ham.de.net.telefonica.de (62.53.25.57)            0.0%   150   20.2  14.1  12.4  41.8   3.8
  5. AS15169  74.125.48.102                                                        0.0%   150   12.9  13.8  12.7  42.6   2.6
  6. AS15169  216.239.63.21                                                        0.0%   150   15.6  15.7  14.5  30.2   1.6
  7. AS15169  216.239.54.187                                                       0.0%   150   13.2  13.7  12.8  23.9   1.3
  8. AS15169  dns.google (8.8.8.8)                                                 0.0%   150   13.5  13.6  12.7  19.0   0.8

The ting to look at here are the "Avg Best Wrst StDev" columns of the final hop "AS15169 dns.google (8.8.8.8) " und the different conditions. In this example:

Avg  Best  Wrst StDev   condition
13.1  12.6  13.6   0.2  idle
13.8  12.7  32.4   1.8  unidirectional load
13.6  12.7  19.0   0.8  bidirectional load

especially the Wrst column should be relevant. I would only look at the throuput as reported by speedtest-cli to confirm that this is roughly in the expected range for the link's capacity, the goal is to saturate the link.

In essence the same idea can be implemented with any load generator (like a download manager or a torrent client set to pull in loads of torrents simultaneously) and just combined with the appropriate mtr invocation (e.g. sudo mtr -ezb4 -i 0.5 8.8.8.8 to keep it running in monitoring mode).

moeller0 · October 1, 2020, 7:44am

The splitters are really just that passive devices without any smarts, but in GPON/XGS-PON any customer modem that wants to send anything needs to first request transmission timeslots from the upstream OLT before being allowed to send, I could well imaging, that this reqyest/grant process could well require back to back packets in the modem's queue to work at the full rate. You could test this by just setting sqm to say 90 Mbps for the upstream and see whether that helps a bit (no, that might/need not be a long term acceptable setting, depending on your use-cases, but for testing it might be interesting)).

Not really, I just am not that good at explaining things in simple terms, the principles behind the whole sqm-thing are really not that complicated, the details however, as important as they are, easily make the whole issue look complicated.

"but I notice especially secondary clients even with one host qbittorrent + 1000 connections blazing, I'm getting 10-40ms on continous ping with sqm to my AMS (vultr.com) server, and I believe that wasn't the case before."

Hmm, was the ping RTT larger before or smaller? If ping and torrent are running on different machines this is exactly the situation where the nat dual-dsthost ingres (for ingress) and nat dual-srchost (for egress) stanzas are supposed to help.

Torrents often use µTP as protocol and that will try to measure induced delay and throttle itself back if that induced delay reaches 100ms, which is better than nothing, but leaves you with up 0-100ms RTT jitter, which for interactive applications can be quite annoying.

willyj · October 5, 2020, 12:44pm

Hello again, sorry for the delay I had to catch up on some sleep since moving for my friend.

Firstly, I had to edit your script a little, I removed &. and ;. from the end of the two first mtr lines. I'm well versed in POSIX/bash/shellscript and I think its a mistake. Dot would mean current directory, but in combination with & (background process) it doesn't make sense. Did you try to background the idle test? That also wouldn't make much sense if loaded test came right after, ; means new line like C.

After this the script ran. Which are you particularly interested in? I can start by giving the MTR results idle on wired (no SQM, and reboot router before), the loss at router is peculiar (It doesn't happen on all mtrs), the 192.168 IP is WSL's internal network, its usually on 172.x.x for some reason its on 192.168 on my laptop.

mtr_idle_20201005T04 (out_wired_laptop_no_sqm_2)

Start: 2020-10-05T04:47:28+0200
HOST: LAPTOP-60I7IP68                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    LAPTOP-60I7IP68.mshome.net (192.168.224.81)         0.0%   100    0.4   0.4   0.3   0.9   0.1
  2. AS???    OpenWrt.lan (10.0.0.1)                             74.0%   100    1.0   0.9   0.9   1.1   0.0
  3. AS43557  85.203.195.1                                        0.0%   100    4.9   3.8   1.6  15.8   1.7
  4. AS43557  stovr01ds01_ae23.eniig-net.dk (85.191.209.80)       0.0%   100    4.2   4.7   2.1  17.2   1.9
       [MPLS: Lbl 53 TC 0 S u TTL 1]
  5. AS43557  aarsx01cr01_ae11.em-net.dk (85.191.209.38)          0.0%   100    3.9   5.7   3.3  16.4   1.8
       [MPLS: Lbl 84 TC 0 S u TTL 1]
  6. AS???    10.10.1.53                                          0.0%   100    6.6   6.5   4.0  35.0   3.2
       [MPLS: Lbl 20 TC 0 S u TTL 1]
  7. AS???    10.10.0.1                                           0.0%   100    5.6   7.3   4.3  21.6   2.8
  8. AS43557  aarh-br02-bexx.eniig-net.dk (217.198.210.206)       0.0%   100    8.4   6.6   4.7   8.7   1.2
  9. AS39642  be185.ahy-cor03.link.stofa.net (89.184.134.24)      0.0%   100   18.4  18.2  16.0  23.0   1.3
       [MPLS: Lbl 24028 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 1]
 10. AS39642  be182.koldb-cor03.link.stofa.net (89.184.128.170)   0.0%   100   16.4  17.9  15.8  20.0   1.2
       [MPLS: Lbl 24016 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 2]
 11. AS39642  te1-2.hos-cor-01.bbsyd.net (89.184.134.181)         0.0%   100   18.5  17.4  15.3  22.9   1.3
 12. AS197288 212.10.11.15                                        0.0%   100   16.5  17.2  14.8  37.1   2.4
 13. AS15169  108.170.251.129                                     0.0%   100   15.4  17.1  15.1  19.8   1.2
 14. AS15169  66.249.94.245                                       0.0%   100   18.8  18.1  15.9  21.6   1.2
 15. AS15169  dns.google (8.8.8.8)                                0.0%   100   16.3  16.9  15.0  19.0   1.2

mtr_idle_20201005T042458.out (out_wired_laptop_no_sqm)

Start: 2020-10-05T04:24:58+0200
HOST: LAPTOP-60I7IP68                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    LAPTOP-60I7IP68.mshome.net (192.168.224.81)         0.0%   100    0.5   0.6   0.4   0.9   0.1
  2. AS???    OpenWrt.lan (10.0.0.1)                             74.0%   100    1.1   2.6   0.9  24.4   4.9
  3. AS43557  85.203.195.1                                        0.0%   100    4.7   4.3   1.5  13.3   1.7
  4. AS43557  stovr01ds01_ae23.eniig-net.dk (85.191.209.80)       0.0%   100    2.9   6.5   2.3  32.5   4.8
       [MPLS: Lbl 53 TC 0 S u TTL 1]
  5. AS43557  aarsx01cr01_ae11.em-net.dk (85.191.209.38)          0.0%   100    5.9   7.4   3.4  41.8   5.7
       [MPLS: Lbl 84 TC 0 S u TTL 1]
  6. AS???    10.10.1.53                                          0.0%   100    6.2   6.8   3.9  17.5   2.0
       [MPLS: Lbl 20 TC 0 S u TTL 1]
  7. AS???    10.10.0.1                                           0.0%   100    9.6   7.0   4.3  12.1   1.6
  8. AS43557  aarh-br02-bexx.eniig-net.dk (217.198.210.206)       0.0%   100    6.4   8.4   4.9  34.8   4.4
  9. AS39642  be185.ahy-cor03.link.stofa.net (89.184.134.24)      0.0%   100   19.7  18.3  16.0  20.7   1.3
       [MPLS: Lbl 24028 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 1]
 10. AS39642  be182.koldb-cor03.link.stofa.net (89.184.128.170)   0.0%   100   16.9  18.7  15.9  28.8   1.6
       [MPLS: Lbl 24016 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 2]
 11. AS39642  te1-2.hos-cor-01.bbsyd.net (89.184.134.181)         0.0%   100   18.8  18.0  15.5  31.5   2.0
 12. AS197288 212.10.11.15                                        0.0%   100   18.4  17.7  14.9  27.1   1.9
 13. AS15169  108.170.251.129                                     0.0%   100   18.4  19.0  15.2  55.0   6.0
 14. AS15169  66.249.94.245                                       0.0%   100   16.5  18.4  15.6  34.5   2.2
 15. AS15169  dns.google (8.8.8.8)                                0.0%   100   18.0  17.0  14.9  19.6   1.4

these are the 2 wired no_sqm mtr_idle tests ^ from both attempts without sqm

out_wired_laptop_no_sqm
out_wired_laptop_no_sqm_2

and the rest of the test from the second attempt

mtr_fully_loaded_20201005T044725.out (out_wired_laptop_no_sqm_2)

Start: 2020-10-05T04:49:14+0200
HOST: LAPTOP-60I7IP68                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    LAPTOP-60I7IP68.mshome.net (192.168.224.81)         0.0%   150    0.6   0.5   0.3   0.8   0.1
  2. AS???    OpenWrt.lan (10.0.0.1)                             76.0%   150    0.9   0.8   0.7   0.9   0.1
  3. AS43557  85.203.195.1                                        0.0%   150    3.7   3.9   1.4  31.5   3.3
  4. AS43557  stovr01ds01_ae23.eniig-net.dk (85.191.209.80)       0.0%   150    2.7   4.7   2.0  18.0   2.1
       [MPLS: Lbl 53 TC 0 S u TTL 1]
  5. AS43557  aarsx01cr01_ae11.em-net.dk (85.191.209.38)          0.0%   150    6.2   5.3   2.9   9.2   1.3
       [MPLS: Lbl 84 TC 0 S u TTL 1]
  6. AS???    10.10.1.53                                          0.0%   150    5.8   6.8   3.9  48.9   4.4
       [MPLS: Lbl 20 TC 0 S u TTL 1]
  7. AS???    10.10.0.1                                           0.0%   150    4.5   6.8   4.2  27.0   3.0
  8. AS43557  aarh-br02-bexx.eniig-net.dk (217.198.210.206)       0.0%   150   12.0   7.0   4.6  20.8   2.0
  9. AS39642  be185.ahy-cor03.link.stofa.net (89.184.134.24)      0.0%   150   17.6  18.5  15.8  32.8   2.6
       [MPLS: Lbl 24028 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 1]
 10. AS39642  be182.koldb-cor03.link.stofa.net (89.184.128.170)   0.0%   150   15.9  18.1  15.9  30.6   1.8
       [MPLS: Lbl 24016 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 2]
 11. AS39642  te1-2.hos-cor-01.bbsyd.net (89.184.134.181)         0.0%   150   18.1  17.5  15.2  34.5   2.1
 12. AS197288 212.10.11.15                                        0.0%   150   15.9  18.3  14.8 126.2   9.3
 13. AS15169  108.170.251.129                                     0.0%   150   18.6  17.4  15.1  31.8   2.1
 14. AS15169  66.249.94.245                                       0.0%   150   17.8  18.6  15.6  40.9   3.2
 15. AS15169  dns.google (8.8.8.8)                                0.0%   150   15.5  17.4  14.9  40.7   2.6

mtr_loaded_20201005T044725.out (out_wired_laptop_no_sqm_2)

Start: 2020-10-05T04:48:08+0200
HOST: LAPTOP-60I7IP68                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    LAPTOP-60I7IP68.mshome.net (192.168.224.81)         0.0%   150    0.5   0.5   0.4   0.9   0.1
  2. AS???    OpenWrt.lan (10.0.0.1)                             76.0%   150    0.8   0.8   0.7   0.9   0.0
  3. AS43557  85.203.195.1                                        0.0%   150    4.0   3.6   1.4  13.0   1.6
  4. AS43557  stovr01ds01_ae23.eniig-net.dk (85.191.209.80)       0.0%   150    2.9   4.4   2.0   9.4   1.5
       [MPLS: Lbl 53 TC 0 S u TTL 1]
  5. AS43557  aarsx01cr01_ae11.em-net.dk (85.191.209.38)          0.0%   150    7.0   5.4   3.0  10.8   1.4
       [MPLS: Lbl 84 TC 0 S u TTL 1]
  6. AS???    10.10.1.53                                          0.0%   150    6.7   7.0   3.9  35.5   3.8
       [MPLS: Lbl 20 TC 0 S u TTL 1]
  7. AS???    10.10.0.1                                           0.0%   150    4.6   6.4   4.2  12.6   1.5
  8. AS43557  aarh-br02-bexx.eniig-net.dk (217.198.210.206)       0.0%   150    7.4   6.7   4.7   9.6   1.2
  9. AS39642  be185.ahy-cor03.link.stofa.net (89.184.134.24)      0.0%   150   17.5  18.0  15.9  21.0   1.2
       [MPLS: Lbl 24028 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 1]
 10. AS39642  be182.koldb-cor03.link.stofa.net (89.184.128.170)   0.7%   150   19.8  18.0  15.9  20.2   1.2
       [MPLS: Lbl 24016 TC 0 S u TTL 1]
       [MPLS: Lbl 24001 TC 0 S u TTL 2]
 11. AS39642  te1-2.hos-cor-01.bbsyd.net (89.184.134.181)         0.0%   150   17.9  17.4  15.3  19.5   1.2
 12. AS197288 212.10.11.15                                        0.0%   150   15.8  17.6  14.7 125.1   9.0
 13. AS15169  108.170.251.129                                     0.0%   150   18.8  17.4  15.2  32.4   1.7
 14. AS15169  66.249.94.245                                       0.0%   150   18.0  18.2  15.6  26.1   1.8
 15. AS15169  dns.google (8.8.8.8)                                0.0%   150   15.7  17.0  14.9  19.1   1.2

speedtest_20201005T044725.out
Retrieving speedtest.net configuration...
Testing from EnergiMidt Fiberbredband (xx.xx.xx.xx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nygreen IT ApS (Hirtshals) [59.31 km]: 13.685 ms
Testing download speed................................................................................
Download: 185.03 Mbit/s
Testing upload speed......................................................................................................
Upload: 204.31 Mbit/s

speedtest_no-download_20201005T044725.out (out_wired_laptop_no_sqm_2)

Retrieving speedtest.net configuration...
Testing from EnergiMidt Fiberbredband (xx.xx.xx.xx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nygreen IT ApS (Hirtshals) [59.31 km]: 9.895 ms
Skipping download test
Testing upload speed......................................................................................................
Upload: 140.30 Mbit/s

speedtest_no-upload_20201005T044725.out (out_wired_laptop_no_sqm_2)

Retrieving speedtest.net configuration...
Testing from EnergiMidt Fiberbredband (xx.xx.xx.xx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nygreen IT ApS (Hirtshals) [59.31 km]: 8.313 ms
Testing download speed................................................................................
Download: 216.64 Mbit/s
Skipping upload test

These tests are all from
combined_speedtest_laptop/out_wired_laptop_no_sqm_2

These are the available tests if you need to see some comparison:

├── out_802.11ac_laptop_no_sqm
├── out_802.11ac_laptop_sqm
├── out_wired_laptop_no_sqm
├── out_wired_laptop_no_sqm_2
├── out_wired_laptop_sqm
├── out_802.11ac_no_sqm
└── out_802.11ac_sqm

btw. fast.com shows me the best ping of the tests I've found, naturally because netflix has so many nodes. 14-17ms when it has SQM, a little higher peaks without initially 50ms+ on upload its similar .. 50ms both with/without SQM (and also not better as I said if I limit the upload). I did try to limit the upload, but no better result.

moeller0 · October 5, 2020, 2:26pm

Yes, these dots were added by the copy and paste process somehow, thez are not part of my actual script, aparently I did a sloppy job at editing.

Nope, the background test was intended to block, and I believe it does, only the loaded tests get started in the background so they run parallel to the test.

But as I tried to indicate, this is a really quick and dirty script, for ad hoc testing.

The different RTT and packet loss numbers for the final hop.

That probably is caused by a rate limit to the router's ICMP echo generation, with 0.2 seconds instead of the default 1 second intervals, quite a number of intermediate hops show similar packet loss on many paths, but that really is not a sign of any real issues.

idle: 15.0 to 19.0 ms / 14.9 to 19.6

bi-directional:
loaded: 14.9 to 40.7
uni-directional:
loaded: 14.9 to 19.1.

But the output of the script was intended to be looked at in sequence, the speedtest results give an idea/confirmation of the load, while the mtr results show how latency increased under load.

That is a rather peculiar result, still. No doubt about your data, just puzzled what your ISP is doing there....

willyj · October 5, 2020, 3:05pm

Okay! Then I think the script ran correctly

Thanks for pointing that out with ICMP/ping rate-limit, I can sleep safer at night now!

Should I do another experiment?

I can try to make a severely upload limited 90mbit output and a sane 170mbit limit from your "combined_speedtest" if you are comfortable with that output I assume what you are interested in here is the loaded and the idle speedtest.

Or I can try to use fast.com and I can see if any upload limitation would impact this, so far however, its been stale at 50ms on both SQM upload limit and without as soon as it starts the upload test. Only the download speed gives noticeable less peaks in ms

willyj · October 9, 2020, 5:42pm

I believe I have acquired some info that might help this problem @moeller0

I asked provider and they told they are using non-status reporting dynamic bandwidth allocation. This means, the upload is allocated based on use and hence why limiting the upload means more idle frames to keep the allocation low (from a service availability perspective from the ISP, splitting 1 optical fiber between multiple customers), maybe there even is a fixed starting rate @ 90-100mbps on upload, which is then boost scaled to 200mbps~ when the rate of idle frames drops (I pay for 200/200mbps).

Looking at tc-cake(8) I can see there's an autorate-ingress but not an autorate-egress. I tried to apply percentages of RATE as explained in tc(8), this failed however. Maybe you @moeller0 knows if the upload '99.5%' field fails on the % due to simple config validation and tc without the abstraction config (/etc/config/sqm) layer would be able to do what I want.

SQM: Starting SQM script: piece_of_cake.qos on eth0.11, in: 214000 Kbps, out: 99.5% Kbps
SQM: Using generic sqm_start_default function.
sh: 99.5%: bad number # <-- Is this fixable to work with `tc` ? It almost looks like a simple 'quoting' issue in commands to `tc`
SQM: piece_of_cake.qos was started on eth0.11 successfully

It looks like sh shell fails here due to lack of quoting in passing args to tc maybe?

Also, between some of the "dynamic bandwidth allocation" resources online I found that 20ms and a significant percentage (%) packet loss is presumably considered normal so it seems obvious status reporting DBA is superior in many aspects.

I hope this wasn't a waste of your time