This is my current network topology. I am using the Archer A6v3 as a switch to bridge the connection from my ont to rpi4 over VLAN10.
The Archer A6 then gets a bridged lan back from my rpi4 over VLAN1.
On my rpi4 which dials a PPPoe, whenever I apply sqm I get terrible bufferbloat (without sqm 5 to 20, with sqm 50+), I tried applying the sqm on the pppoe, eth0.10 and @wan interfaces but nothing helped also I used the Cake/Layer cake choices.
Can someone please help me here.
Please post the content of"
cat /etc/config/sqm
tc -s qdisc
ifstatus wan # you might want to redact the router's IPv4 IPv6 addresses a bit.
and
cat /etc/config/network #but make sure to redact username and password in the output
Personally, I probably would try to use a USB3 ethernet dongle like the TP-Link UE300 and use a direct connection between ONT and RPi4B for the WAN side, but unless your internet access is larger than say ~500 Mbps in any direction the "router on a stick" configuration should work as well.
With SQM disabled, what speed do you measure in online speedtests, and what are your contracted rates?
1 Like
I get about 120mbps without sqm and I try to set the values to 110 with sqm.
I'll be posting all the configs in a while.
1 Like
Take your time, it would be great if you could also post links* to two waveform bufferboat tests like this:
https://www.waveform.com/tools/bufferbloat?test-id=4f16b84c-94a8-4e06-9df4-c8d13e08b6b8
(this test run over WiFi and really is just to illustrate what kind of link to post ;))
one of these tests should be with SQM disabled and one with SQM enabled.
*) The links can be found in the text box labeled "Share Your Results:" at the bottom of the test
That indicates that the multiplexing of ingress and egress over two VLANs probably is not your problem.
I got nice bufferbloat for 1 run.
2nd run
3rd run
Without Sqm
root@Aashish:~# cat /etc/config/sqm
config queue 'eth1'
option qdisc 'cake'
option verbosity '5'
option script 'layer_cake.qos'
option qdisc_advanced '1'
option squash_dscp '1'
option squash_ingress '1'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option interface 'eth0.10'
option debug_logging '0'
option linklayer 'ethernet'
option overhead '44'
option download '112640'
option upload '112640'
option enabled '1'
root@Aashish:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 10188007947 bytes 10116027 pkt (dropped 0, overlimits 0 requeues 11074)
backlog 0b 0p requeues 11074
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 1840667378 bytes 1901949 pkt (dropped 0, overlimits 0 requeues 8190)
backlog 0b 0p requeues 8190
maxpacket 2276 drop_overlimit 0 new_flow_count 6821 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 1883868967 bytes 1830833 pkt (dropped 0, overlimits 0 requeues 1152)
backlog 0b 0p requeues 1152
maxpacket 1506 drop_overlimit 0 new_flow_count 3370 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2245812952 bytes 2194056 pkt (dropped 0, overlimits 0 requeues 1300)
backlog 0b 0p requeues 1300
maxpacket 1514 drop_overlimit 0 new_flow_count 6600 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2429717240 bytes 2254851 pkt (dropped 0, overlimits 0 requeues 401)
backlog 0b 0p requeues 401
maxpacket 1506 drop_overlimit 0 new_flow_count 3696 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 1787941410 bytes 1934338 pkt (dropped 0, overlimits 0 requeues 31)
backlog 0b 0p requeues 31
maxpacket 1514 drop_overlimit 0 new_flow_count 3634 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8011: dev eth0.10 root refcnt 2 bandwidth 112640Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
Sent 344823008 bytes 383863 pkt (dropped 27, overlimits 407873 requeues 0)
backlog 0b 0p requeues 0
memory used: 1369728b of 5500Kb
capacity estimate: 112640Kbit
min/max network layer size: 16 / 1500
min/max overhead-adjusted size: 60 / 1544
average network hdr offset: 14
Bulk Best Effort Voice
thresh 7040Kbit 112640Kbit 28160Kbit
target 5ms 5ms 5ms
interval 100ms 100ms 100ms
pk_delay 0us 2.46ms 0us
av_delay 0us 867us 0us
sp_delay 0us 3us 0us
backlog 0b 0b 0b
pkts 0 383890 0
bytes 0 344862482 0
way_inds 0 23 0
way_miss 0 601 0
way_cols 0 0 0
drops 0 27 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 1 0
bk_flows 0 1 0
un_flows 0 0 0
max_len 0 1514 0
quantum 300 1514 859
qdisc ingress ffff: dev eth0.10 parent ffff:fff1 ----------------
Sent 410080468 bytes 361170 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-WAN root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 4784209503 bytes 5237230 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 63360 drop_overlimit 0 new_flow_count 974030 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8012: dev ifb4eth0.10 root refcnt 2 bandwidth 112640Kbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
Sent 406020241 bytes 355145 pkt (dropped 6025, overlimits 510843 requeues 0)
backlog 0b 0p requeues 0
memory used: 3935360b of 5500Kb
capacity estimate: 112640Kbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 90 / 1544
average network hdr offset: 14
Tin 0
thresh 112640Kbit
target 5ms
interval 100ms
pk_delay 130us
av_delay 6us
sp_delay 4us
backlog 0b
pkts 361170
bytes 415136848
way_inds 0
way_miss 551
way_cols 0
drops 6025
marks 0
ack_drop 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 1514
quantum 1514
root@Aashish:~# ifstatus WAN
{
"up": true,
"pending": false,
"available": true,
"autostart": true,
"dynamic": false,
"uptime": 2089,
"l3_device": "pppoe-WAN",
"proto": "pppoe",
"device": "eth0.10",
"updated": [
"addresses",
"routes"
],
"metric": 0,
"dns_metric": 0,
"delegation": true,
"ipv4-address": [
{
"address": "223.236.159.43",
"mask": 32,
"ptpaddress": "106.213.111.255"
}
],
"ipv6-address": [
{
"address": "fe80::c1d5:515:852d:b7d9",
"mask": 128
}
],
"ipv6-prefix": [
],
"ipv6-prefix-assignment": [
],
"route": [
{
"target": "0.0.0.0",
"mask": 0,
"nexthop": "106.213.111.255",
"source": "0.0.0.0/0"
}
],
"dns-server": [
"59.144.144.45",
"125.22.47.125"
],
"dns-search": [
],
"neighbors": [
],
"inactive": {
"ipv4-address": [
],
"ipv6-address": [
],
"route": [
],
"dns-server": [
],
"dns-search": [
],
"neighbors": [
]
},
"data": {
}
}
root@Aashish:~# cat /etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fddf:bb72:1604::/48'
option packet_steering '1'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '64'
config device
option type '8021q'
option ifname 'eth0'
option vid '10'
option name 'eth0.10'
config interface 'WAN'
option proto 'pppoe'
option device 'eth0.10'
option username 'xxxxxxx'
option password 'xxxxxxx'
option ipv6 '1'
config interface 'WAN6'
option proto 'dhcpv6'
option reqaddress 'try'
option device '@WAN'
option reqprefix 'auto'
Mmmh, what would be interesting is to see the output of
tc -s qdisc before and after waveform tests especially around one of these A+/A and one around one of the D tests, that might indicate what happens inside cake during these tests.
I tried but it is very hard to get a good result like the first one, maybe after a reboot it works nice for a minute or two?
Can it be a problem with my access point that induces latency?
Ah, a result before and after a bad wavefront test is probably enough. This should be easier to get, no?
TEST
BEFORE
root@Aashish:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 12551913976 bytes 12647078 pkt (dropped 0, overlimits 0 requeues 26961)
backlog 0b 0p requeues 26961
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2353180593 bytes 2574698 pkt (dropped 0, overlimits 0 requeues 20295)
backlog 0b 0p requeues 20295
maxpacket 59046 drop_overlimit 0 new_flow_count 9854 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2333123600 bytes 2292957 pkt (dropped 0, overlimits 0 requeues 2045)
backlog 0b 0p requeues 2045
maxpacket 1514 drop_overlimit 0 new_flow_count 3974 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2633814061 bytes 2584311 pkt (dropped 0, overlimits 0 requeues 1415)
backlog 0b 0p requeues 1415
maxpacket 16654 drop_overlimit 0 new_flow_count 7234 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 3007982509 bytes 2827915 pkt (dropped 0, overlimits 0 requeues 3163)
backlog 0b 0p requeues 3163
maxpacket 68130 drop_overlimit 0 new_flow_count 4297 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2223813213 bytes 2367197 pkt (dropped 0, overlimits 0 requeues 43)
backlog 0b 0p requeues 43
maxpacket 1514 drop_overlimit 0 new_flow_count 3791 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8015: dev eth0.10 root refcnt 2 bandwidth 112640Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
Sent 307003 bytes 1251 pkt (dropped 0, overlimits 127 requeues 0)
backlog 0b 0p requeues 0
memory used: 11008b of 5500Kb
capacity estimate: 112640Kbit
min/max network layer size: 16 / 1500
min/max overhead-adjusted size: 60 / 1544
average network hdr offset: 14
Bulk Best Effort Voice
thresh 7040Kbit 112640Kbit 28160Kbit
target 5ms 5ms 5ms
interval 100ms 100ms 100ms
pk_delay 0us 226us 0us
av_delay 0us 21us 0us
sp_delay 0us 3us 0us
backlog 0b 0b 0b
pkts 0 1251 0
bytes 0 307003 0
way_inds 0 170 0
way_miss 0 420 0
way_cols 0 0 0
drops 0 0 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 10 0
bk_flows 0 1 0
un_flows 0 0 0
max_len 0 1514 0
quantum 300 1514 859
qdisc ingress ffff: dev eth0.10 parent ffff:fff1 ----------------
Sent 430746 bytes 1102 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-WAN root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 555034991 bytes 723574 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 29840 drop_overlimit 0 new_flow_count 20383 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8016: dev ifb4eth0.10 root refcnt 2 bandwidth 112640Kbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
Sent 446174 bytes 1102 pkt (dropped 0, overlimits 350 requeues 0)
backlog 0b 0p requeues 0
memory used: 35776b of 5500Kb
capacity estimate: 112640Kbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 90 / 1544
average network hdr offset: 14
Tin 0
thresh 112640Kbit
target 5ms
interval 100ms
pk_delay 294us
av_delay 37us
sp_delay 3us
backlog 0b
pkts 1102
bytes 446174
way_inds 0
way_miss 438
way_cols 0
drops 0
marks 0
ack_drop 0
sp_flows 6
bk_flows 1
un_flows 0
max_len 1514
quantum 1514
AFTER
root@Aashish:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 13244905879 bytes 13355664 pkt (dropped 0, overlimits 0 requeues 27014)
backlog 0b 0p requeues 27014
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2468388624 bytes 2656485 pkt (dropped 0, overlimits 0 requeues 20296)
backlog 0b 0p requeues 20296
maxpacket 59046 drop_overlimit 0 new_flow_count 9856 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2397242369 bytes 2392619 pkt (dropped 0, overlimits 0 requeues 2046)
backlog 0b 0p requeues 2046
maxpacket 1514 drop_overlimit 0 new_flow_count 4046 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2693651642 bytes 2751899 pkt (dropped 0, overlimits 0 requeues 1457)
backlog 0b 0p requeues 1457
maxpacket 16654 drop_overlimit 0 new_flow_count 7567 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 3176321403 bytes 2968130 pkt (dropped 0, overlimits 0 requeues 3169)
backlog 0b 0p requeues 3169
maxpacket 68130 drop_overlimit 0 new_flow_count 4418 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 2509301841 bytes 2586531 pkt (dropped 0, overlimits 0 requeues 46)
backlog 0b 0p requeues 46
maxpacket 1514 drop_overlimit 0 new_flow_count 3958 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8015: dev eth0.10 root refcnt 2 bandwidth 112640Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44
Sent 373891363 bytes 368249 pkt (dropped 593, overlimits 518105 requeues 0)
backlog 0b 0p requeues 0
memory used: 3357888b of 5500Kb
capacity estimate: 112640Kbit
min/max network layer size: 16 / 1500
min/max overhead-adjusted size: 60 / 1544
average network hdr offset: 14
Bulk Best Effort Voice
thresh 7040Kbit 112640Kbit 28160Kbit
target 5ms 5ms 5ms
interval 100ms 100ms 100ms
pk_delay 0us 176us 0us
av_delay 0us 22us 0us
sp_delay 0us 2us 0us
backlog 0b 0b 0b
pkts 0 368842 0
bytes 0 374746417 0
way_inds 0 732 0
way_miss 0 2606 0
way_cols 0 0 0
drops 0 593 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 1 0
bk_flows 0 1 0
un_flows 0 0 0
max_len 0 1514 0
quantum 300 1514 859
qdisc ingress ffff: dev eth0.10 parent ffff:fff1 ----------------
Sent 323399133 bytes 346913 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-WAN root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 921387243 bytes 1091145 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 29840 drop_overlimit 0 new_flow_count 67377 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8016: dev ifb4eth0.10 root refcnt 2 bandwidth 112640Kbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 44
Sent 322477998 bytes 342904 pkt (dropped 4009, overlimits 420316 requeues 0)
backlog 0b 0p requeues 0
memory used: 4593088b of 5500Kb
capacity estimate: 112640Kbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 90 / 1544
average network hdr offset: 14
Tin 0
thresh 112640Kbit
target 5ms
interval 100ms
pk_delay 6.22ms
av_delay 3.54ms
sp_delay 3us
backlog 0b
pkts 346913
bytes 328255915
way_inds 167
way_miss 2745
way_cols 0
drops 4009
marks 0
ack_drop 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 1514
quantum 1514
Mmmh, looking at the drops we see a delta of 4009 for download AFTER-BEFORE, but only 593 for upload in spite of the upload achieving 102.4 Mbps, while the download only achieved 84.1 Mbps. At the very least this indicates that the ingress traffic hitting the shaper is not terribly smooth...
What do you get when you (for testing purposes) set the shaper to 50/50 Mbps and repeat the waveform test (with before and after tc -s qdisc output)?
1 Like
I assume that you removed the VLAN carrying the WAN traffic from the lan bridge?
Yes the lan is on a totally different software vlan interface.
I think I should just buy a usb to eth adapter at this point.
Can it be due to the fact that I am using firewall4?