Hello everyone!
I have a problem with SQM supposedly not using cake/cake not shaping, and I have no idea how to check if there is some problem of configuration or is related to driver.
I have an (unfortunately) unsupported device which uses a fork of OpenWRT called HomeWare. My device is a Technicolor DGA4132, and I like the hardware characteristic since it has 4 gigabit port, wireless (albeit the 5 GHz not working), SFP port, DSL input and telephones ports, but the software is utterly awful.
Since there are a lot of drivers closed, and the bootloader checks if the firmware is signed, I can't just put a freshly compiled OpenWRT image. However, there are techinques to gain root access to the device, which I think is enough to play with, even if I'd like a more open-source way to do this, but I can't throw away silicon now, can I?
I am a developer, so I don't mind fiddling with code and terminal, but the learning curve for these things is pretty steep. Another problem is that device is currently serving internet to my family, so I can't afford risky procedures that could brick the router.
So far, I understood this firmware needs some patches from the bcm63xx
target, and there are compiled patches for this firmware (dubbed brcm63xx-tch
) that can be found here
Anyway, straight to the problem: I have a DSL line, nominally 20Mbit, real throughput 12Mbit, that in the evening just drop to 3Mbit, and is causing massive problem when interactive programs mixes with social media application which are bandwidth hungry.
I managed to solve the issues by placing a OpenWRT install on a Raspberry PI between the main router and the ISP router, and running SQM witch cake on top of it.
ISP router/modem <--> raspberry <--> own wireless AP
Supposedly, this new device should replace the wireless AP and the Raspberry, but I am having issues with SQM (it should be fast enough for up to 12 MBit shaping).
I installed sqm-scripts
package, and I have fq_codel
, cake
and sfq
available through LUCI. While fq_codel
and sfq
effectively shape, it seems that cake
, while having the qdisc attached, it's simply not shaping the network when I do a speedtest. From my test, neither fq_codel
nor sfq
help to avoid high latency when bandwidth hungry application are fired, but maybe it's just a problem of configuration.
I don't know which exact details are required, so I willl share some of the information about network and SQM. This exact configuration of SQM is in place in the Raspberry, and it effectively shape the traffic almost always (sometimes I have 3 sec of packet lost, but it is acceptable)
root@stargate:~# cat /etc/config/network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option default_ps '0'
option ula_prefix 'fd00::/48'
config interface 'lan'
option type 'bridge'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '64'
option force_link '0'
option ipv6 '1'
option rpfilter '1'
list pppoerelay 'wanptm0'
option ipaddr '192.168.1.1'
option ifname 'eth0 eth1 eth2 eth3 radio_2G.network1 radio_5G.network1'
config switch 'bcmsw'
option reset '1'
option enable_vlan '0'
option qosimppauseenable '0'
option jumbo '0'
config interface 'wlnet_b_24'
option proto 'static'
option ip6assign '64'
option ipv6 '1'
option ip6hint '1'
option netmask '255.255.255.128'
option ipaddr '192.168.168.1'
option ifname 'wl0_1'
option force_link '0'
option name 'Ospiti-TIM-18178276'
config interface 'wlnet_b_5'
option proto 'static'
option ip6assign '64'
option ipv6 '1'
option ip6hint '2'
option netmask '255.255.255.128'
option ipaddr '192.168.168.129'
option ifname 'wl1_1'
option force_link '0'
option name 'Ospiti-TIM-18178276'
config device 'wlnet_b_5eth5'
option type '8021q'
option name 'wl1_1'
option enabled '1'
option ifname 'eth5'
option vid '3'
option ipv6 '1'
config interface 'public_lan'
option ifname '@lan'
option proto 'static'
option ipaddr '0.0.0.0'
option netmask '0.0.0.0'
option disabled_info 'proto=static,ip=0.0.0.0,mask=255.255.255.248'
option auto '0'
config device 'waneth4'
option type '8021q'
option name 'waneth4'
option macaddr '20:B0:01:15:60:E3'
option vid '835'
option ipv6 '0'
config device 'wanptm0'
option type '8021q'
option name 'wanptm0'
option macaddr '20:B0:01:15:60:E3'
option ifname 'ptm0'
option vid '835'
option ipv6 '0'
config interface 'wan'
option demand '0'
option macaddr '20:B0:01:15:60:E3'
option ipv6 '0'
option peerdns '1'
option reqopts '1 3 6 15 26 33 42 51 121 249'
option keepalive_adaptive '0'
option dns_metric '0'
option username 'vodafoneadsl'
option password 'vodafoneadsl'
option ifname 'eth4'
option proto 'static'
option netmask '255.255.255.0'
option gateway '192.168.3.1'
option dns '192.168.3.1'
option ipaddr '192.168.3.2'
config config 'config'
option wan_mode 'static'
config interface 'wwan'
option proto 'mobiled'
option session_id '0'
option profile '1'
option iface_464xlat '0'
option auto '0'
config interface 'wan6'
option ifname '@wan'
option proto 'dhcpv6'
option reqopts '12 21 22 23 24 25 31 56 64 67 82 83'
option noslaaconly '1'
option iface_464xlat '0'
option auto '0'
option dns_metric '20'
config interface 'sfp'
option proto 'static'
option ifname 'eth4'
option ipaddr '192.168.2.2'
option netmask '255.255.255.0'
option auto '0'
config device 'voipeth4'
option enabled '0'
option type '8021q'
option name 'voipeth4'
option macaddr '20:B0:01:15:60:E4'
option ifname 'eth4'
option vid '84'
option mtu '1400'
option ipv6 '0'
config interface 'voip'
option auto '0'
option proto 'dhcp'
option ifname '<set by script>'
option macaddr '20:B0:01:15:60:E4'
option ipv6 '0'
option peerdns '1'
option reqopts '1 3 6 15 26 33 42 51 121 249'
option vendorid 'DGA4132_CP1852RAJQT'
option ip4table 'voip'
option defaultroute '1'
option dnsset 'voip'
config interface 'ipoe'
option proto 'dhcp'
option metric '1'
option reqopts '1 3 6 43 51 58 59'
option release '1'
option neighreachabletime '1200000'
option neighgcstaletime '2400'
option ipv6 '1'
option auto '0'
root@stargate:~# cat /etc/config/sqm
config queue
option linklayer 'atm'
option overhead '44'
option upload '800'
option interface 'eth4'
option enabled '1'
option download '2100'
option qdisc 'cake'
option verbosity '10'
option debug_logging '1'
option qdisc_advanced '1'
option squash_dscp '1'
option squash_ingress '1'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'nat dual-dsthost ingress diffserv4 rtt 60ms'
option eqdisc_opts 'nat dual-srchost diffserv4 rtt 60ms'
option script 'layer_cake.qos'
root@stargate:~# service sqm restart
SQM:
SQM: Sun Mar 21 14:43:27 CET 2021: Stopping.
SQM: Stopping SQM on eth4
SQM: ifb associated with interface eth4: ifb4eth4
SQM: /usr/sbin/tc qdisc del dev eth4 ingress
SQM: /usr/sbin/tc qdisc del dev eth4 root
SQM: /usr/sbin/tc qdisc del dev ifb4eth4 root
SQM: /usr/lib/sqm/stop-sqm: ifb4eth4 shaper deleted
SQM: iptables -t mangle -D POSTROUTING -o ifb4eth4 -m mark --mark 0x00 -g QOS_MARK_eth4
SQM: ip6tables -t mangle -D POSTROUTING -o ifb4eth4 -m mark --mark 0x00 -g QOS_MARK_eth4
SQM: iptables -t mangle -D POSTROUTING -o eth4 -m mark --mark 0x00/0xff -g QOS_MARK_eth4
SQM: ip6tables -t mangle -D POSTROUTING -o eth4 -m mark --mark 0x00/0xff -g QOS_MARK_eth4
SQM: iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
SQM: ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
SQM: iptables -t mangle -D PREROUTING -i eth4 -m dscp ! --dscp 0 -j DSCP --set-dscp-class be
SQM: ip6tables -t mangle -D PREROUTING -i eth4 -m dscp ! --dscp 0 -j DSCP --set-dscp-class be
SQM: iptables -t mangle -D PREROUTING -i eth4 -m mark --mark 0x00/0xff -g QOS_MARK_eth4
SQM: ip6tables -t mangle -D PREROUTING -i eth4 -m mark --mark 0x00/0xff -g QOS_MARK_eth4
SQM: iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
SQM: ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
SQM: iptables -t mangle -F QOS_MARK_eth4
SQM: ip6tables -t mangle -F QOS_MARK_eth4
SQM: iptables -t mangle -X QOS_MARK_eth4
SQM: ip6tables -t mangle -X QOS_MARK_eth4
SQM: /sbin/ip link set dev ifb4eth4 down
SQM: /sbin/ip link delete ifb4eth4 type ifb
SQM: /usr/lib/sqm/stop-sqm: ifb4eth4 interface deleted
SQM:
SQM: Sun Mar 21 14:43:28 CET 2021: Starting.
SQM: Starting SQM script: layer_cake.qos on eth4, in: 2100 Kbps, out: 800 Kbps
SQM: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM root cake
SQM: QDISC cake is useable.
SQM: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: Starting layer_cake.qos
SQM: ifb associated with interface eth4:
SQM: Currently no ifb is associated with eth4, this is normal during starting of the sqm system.
SQM: /sbin/ip link add name ifb4eth4 type ifb
SQM: LLA: default link layer adjustment method for cake is cake
SQM: cake link layer adjustments: atm overhead 44 mpu 0
SQM: /usr/sbin/tc qdisc add dev eth4 root cake bandwidth 800kbit atm overhead 44 mpu 0 diffserv3 nat dual-srchost diffserv4 rtt 60ms
SQM: egress shaping activated
SQM: /sbin/ip link add name TMP_IFB_4_SQM type ifb
SQM: /usr/sbin/tc qdisc replace dev TMP_IFB_4_SQM ingress
SQM: QDISC ingress is useable.
SQM: /sbin/ip link set dev TMP_IFB_4_SQM down
SQM: /sbin/ip link delete TMP_IFB_4_SQM type ifb
SQM: /usr/sbin/tc qdisc add dev eth4 handle ffff: ingress
SQM: LLA: default link layer adjustment method for cake is cake
SQM: cake link layer adjustments: atm overhead 44 mpu 0
SQM: /usr/sbin/tc qdisc add dev ifb4eth4 root cake bandwidth 2100kbit atm overhead 44 mpu 0 diffserv3 besteffort wash nat dual-dsthost ingress diffserv4 rtt 60ms
SQM: /sbin/ip link set dev ifb4eth4 up
SQM: /usr/sbin/tc filter add dev eth4 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth4
SQM: ingress shaping activated
SQM: layer_cake.qos was started on eth4 successfully
root@stargate:~# tc -s qdisc
qdisc qos-tch f0ef: dev lo root refcnt 2
Sent 63192 bytes 316 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev eth0 root refcnt 2 [Unknown qdisc, optlen=104]
Sent 51211888 bytes 174088 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev eth1 root refcnt 2 [Unknown qdisc, optlen=104]
Sent 23694657 bytes 123599 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev eth2 root refcnt 2 [Unknown qdisc, optlen=104]
Sent 22009230 bytes 88552 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev eth3 root refcnt 2 [Unknown qdisc, optlen=104]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8070: dev eth4 root refcnt 2 bandwidth 800Kbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 60.0ms atm overhead 44
Sent 10493 bytes 156 pkt (dropped 0, overlimits 12 requeues 0)
backlog 0b 0p requeues 0
memory used: 1696b of 4Mb
capacity estimate: 800Kbit
min/max network layer size: 28 / 1378
min/max overhead-adjusted size: 106 / 1590
average network hdr offset: 6
Bulk Best Effort Video Voice
thresh 50Kbit 800Kbit 400Kbit 200Kbit
target 363.4ms 22.7ms 45.4ms 90.8ms
interval 726.7ms 79.7ms 102.4ms 181.7ms
pk_delay 0us 704us 0us 12us
av_delay 0us 22us 0us 0us
sp_delay 0us 7us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 152 0 4
bytes 0 10309 0 184
way_inds 0 0 0 0
way_miss 0 24 0 4
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 0 1
bk_flows 0 2 0 0
un_flows 0 0 0 0
max_len 0 1392 0 46
quantum 300 300 300 300
qdisc ingress ffff: dev eth4 parent ffff:fff1 ----------------
Sent 195651 bytes 230 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev eth5 root refcnt 2
Sent 3069766 bytes 50606 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev wl0 root refcnt 2 [Unknown qdisc, optlen=104]
Sent 27868397 bytes 123953 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc qos-tch f0ef: dev wl0_1 root refcnt 2
Sent 23540 bytes 110 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8071: dev ifb4eth4 root refcnt 2 bandwidth 2100Kbit diffserv4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 60.0ms atm overhead 44
Sent 159659 bytes 204 pkt (dropped 26, overlimits 234 requeues 0)
backlog 0b 0p requeues 0
memory used: 61312b of 4Mb
capacity estimate: 2100Kbit
min/max network layer size: 32 / 1492
min/max overhead-adjusted size: 106 / 1696
average network hdr offset: 8
Bulk Best Effort Video Voice
thresh 131248bit 2100Kbit 1050Kbit 525Kbit
target 138.4ms 8.7ms 17.3ms 34.6ms
interval 276.8ms 65.7ms 74.3ms 91.6ms
pk_delay 0us 134.3ms 0us 9us
av_delay 0us 22.0ms 0us 0us
sp_delay 0us 14us 0us 0us
backlog 0b 0b 0b 0b
pkts 0 225 0 5
bytes 0 198627 0 188
way_inds 0 0 0 0
way_miss 0 33 0 5
way_cols 0 0 0 0
drops 0 26 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 1 0 0
bk_flows 0 5 0 0
un_flows 0 0 0 0
max_len 0 1506 0 60
quantum 300 300 300 300
root@stargate:~# tc -d qdisc
qdisc qos-tch f0ef: dev lo root refcnt 2
qdisc qos-tch f0ef: dev eth0 root refcnt 2 [Unknown qdisc, optlen=104]
qdisc qos-tch f0ef: dev eth1 root refcnt 2 [Unknown qdisc, optlen=104]
qdisc qos-tch f0ef: dev eth2 root refcnt 2 [Unknown qdisc, optlen=104]
qdisc qos-tch f0ef: dev eth3 root refcnt 2 [Unknown qdisc, optlen=104]
qdisc cake 8070: dev eth4 root refcnt 2 bandwidth 800Kbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 60.0ms atm overhead 44
qdisc ingress ffff: dev eth4 parent ffff:fff1 ----------------
qdisc qos-tch f0ef: dev eth5 root refcnt 2
qdisc qos-tch f0ef: dev wl0 root refcnt 2 [Unknown qdisc, optlen=104]
qdisc qos-tch f0ef: dev wl0_1 root refcnt 2
qdisc cake 8071: dev ifb4eth4 root refcnt 2 bandwidth 2100Kbit diffserv4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 60.0ms atm overhead 44
What can I try to do to identify where the issues are? I can put the raspberry behind the WAN to sink (or generate) network traffic
Thank you for reading up to here, I tried to be as detailed as possible to give a good overview, even if experience taught me I surely miss half of the stuff that are needed