BufferBloat between ADSL and 4G lines

Installing and Using OpenWrt Network and Wireless Configuration
1

Hello, I am using Open MTCP Router who is based on OpenWRT. I have bad BufferBloat who affects my daily work on Skype and Teams calls.

I am wondering if the issue is not coming from mixing different type of internet accesses (2 ADSL: wan1 and wan2 and 2 4G: wan3 and wan4).

What should I be configuring on the 4G interfaces to be working (as currently I have SQM enabled on ADSL but that are the most stable lines so I need to do something on th 4G lines for it to work).

My config:

7: wan1@eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 
    inet 192.168.2.2/24 brd 192.168.2.255 scope global wan1
8: wan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
    inet 192.168.3.2/24 brd 192.168.3.255 scope global wan2
9: wan3@eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1464 
    inet 192.168.8.2/24 brd 192.168.8.255 scope global wan3
10: wan4@eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1490 
    inet 192.168.9.2/24 brd 192.168.9.255 scope global wan4

cat /etc/config/sqm
config queue 'wan1'
        option qdisc_advanced '0'
        option interface 'wan1'
        option download '0'
        option upload '0'
        option debug_logging '0'
        option verbosity '5'
        option enabled '1'
        option linklayer 'atm'
        option script 'piece_of_cake.qos'
        option overhead '44'
        option qdisc 'cake'
config queue 'wan2'
        option qdisc_advanced '0'
        option interface 'wan2'
        option download '0'
        option upload '0'
        option debug_logging '0'
        option verbosity '5'
        option enabled '1'
        option script 'piece_of_cake.qos'
        option linklayer 'atm'
        option overhead '44'
        option qdisc 'cake'
config queue 'wan3'
        option interface 'wan3'
        option qdisc 'fq_codel'
        option script 'simple.qos'
        option qdisc_advanced '0'
        option linklayer 'none'
        option enabled '0'
        option debug_logging '0'
        option verbosity '5'
        option download '0'
        option upload '0'
config queue 'wan4'
        option interface 'wan4'
        option qdisc 'fq_codel'
        option script 'simple.qos'
        option qdisc_advanced '0'
        option linklayer 'none'
        option enabled '0'
        option debug_logging '0'
        option verbosity '5'
        option download '0'
        option upload '0'

tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 5895161540 bytes 9801918 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn 
 Sent 649460361 bytes 869526 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wan2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wan3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wan4 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn 
 Sent 805998 bytes 3657 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn 
qdisc noqueue 0: dev wan1 root refcnt 2 
qdisc noqueue 0: dev wan2 root refcnt 2 
qdisc noqueue 0: dev wan3 root refcnt 2 
qdisc noqueue 0: dev wan4 root refcnt 2 
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn

Basically I am trying to understand what should I be doing on the 4G interfaces around SQM.

http://www.dslreports.com/speedtest/59286022

Thanks

2

4G is difficult since the speed varies-- a lot. Generally you'd find the worst-case speed then tell SQM to run slightly lower than that.

If the worst-case 4G speed is slower than DSL, I would suggest using the two DSL lines as primary and relegate 4G to a backup role that isn't used at all except in case the DSL goes down.

1 Like
3

Mmmh, you have both download and upload bandwidth set to zero in all four configurations. Due to that you have sqm completely disabled on all interfaces. Is this just an artifact from testing, say an intermediary config for a noSQM test? If not maybe start debloating by enabling sqm for up- and downlink of the two ADSL lines?
At the current status, for variable rate links the only thing you can halfway reliably do is configure a shaper to rates that, from your experience, you are quite likely to actually get in that time of the day you desire lowest latency under load/bufferbloat.
A better solution would be to include an fq_codel/cake like AQM into the wireless stack of late modem and basestation, but I would not hold my breath for that (nevertheless with 5G mobile standards became noticeably more sensitive to latency under load so the future might bring better solutions)

4

Hello All and thanks for the quick constructive replies.

I have tested each line independently and activated SQM at the end on both ADSL. Not sure why but as soon as I activate SQM (cake, piece_of_cake) on a 4G line the results are terrible. I also noticed that my WAN4 is having very low UL/DL even withtout SQM.

SQW effects per WAN interface

After
http://www.dslreports.com/speedtest/59303832

Are there some recommended actions to be done on 4G lines with SQM (other algorithms for ex)?
I will look into either removing WAN4 or fixing it as its capability is currently not acceptable.

Merci

tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 620125629 bytes 1577281 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 200985036 bytes 302703 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 36890595 bytes 81987 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 100194374 bytes 188201 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 49801410 bytes 348114 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 83480737 bytes 298296 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 35291381 bytes 79860 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 56136356 bytes 88768 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 57345740 bytes 189352 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc htb 1: dev wan1 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 1 direct_qlen 100
Sent 62315781 bytes 456448 pkt (dropped 640, overlimits 39602 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 120: dev wan1 parent 1:12 limit 1001p flows 1024 quantum 300 target 22.0ms interval 117.0ms memory_limit 32Mb ecn
Sent 34721916 bytes 174428 pkt (dropped 635, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 3028 drop_overlimit 0 new_flow_count 66128 ecn_mark 0
new_flows_len 0 old_flows_len 3
qdisc fq_codel 130: dev wan1 parent 1:13 limit 1001p flows 1024 quantum 300 target 22.0ms interval 117.0ms memory_limit 32Mb ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev wan1 parent 1:11 limit 1001p flows 1024 quantum 300 target 22.0ms interval 117.0ms memory_limit 32Mb ecn
Sent 27592363 bytes 282019 pkt (dropped 5, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 3028 drop_overlimit 0 new_flow_count 9 ecn_mark 0
new_flows_len 0 old_flows_len 1
qdisc ingress ffff: dev wan1 parent ffff:fff1 ----------------
Sent 249012986 bytes 503976 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev wan2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 1 direct_qlen 100
Sent 27102459 bytes 158061 pkt (dropped 169, overlimits 10717 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 120: dev wan2 parent 1:12 limit 1001p flows 1024 quantum 300 target 14.7ms interval 109.7ms memory_limit 32Mb ecn
Sent 12810439 bytes 60568 pkt (dropped 165, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 3028 drop_overlimit 0 new_flow_count 32338 ecn_mark 0
new_flows_len 0 old_flows_len 9
qdisc fq_codel 130: dev wan2 parent 1:13 limit 1001p flows 1024 quantum 300 target 14.7ms interval 109.7ms memory_limit 32Mb ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev wan2 parent 1:11 limit 1001p flows 1024 quantum 300 target 14.7ms interval 109.7ms memory_limit 32Mb ecn
Sent 14290518 bytes 97492 pkt (dropped 4, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 3028 drop_overlimit 0 new_flow_count 3 ecn_mark 0
new_flows_len 0 old_flows_len 1
qdisc ingress ffff: dev wan2 parent ffff:fff1 ----------------
Sent 42402748 bytes 165460 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wan3 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4wan1 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 direct_qlen 32
Sent 255460809 bytes 502995 pkt (dropped 983, overlimits 54571 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4wan1 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 255460809 bytes 502995 pkt (dropped 983, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 3028 drop_overlimit 0 new_flow_count 90631 ecn_mark 0
new_flows_len 0 old_flows_len 1
qdisc htb 1: dev ifb4wan2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 direct_qlen 32
Sent 44740308 bytes 165460 pkt (dropped 0, overlimits 45 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4wan2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 44740308 bytes 165460 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 2236 drop_overlimit 0 new_flow_count 36768 ecn_mark 0
new_flows_len 0 old_flows_len 17
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 56917477 bytes 166854 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0

tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc htb 1: dev wan1 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 1 ver 3.17 direct_qlen 100
qdisc fq_codel 120: dev wan1 parent 1:12 limit 1001p flows 1024 quantum 300 target 22.0ms interval 117.0ms memory_limit 32Mb ecn
qdisc fq_codel 130: dev wan1 parent 1:13 limit 1001p flows 1024 quantum 300 target 22.0ms interval 117.0ms memory_limit 32Mb ecn
qdisc fq_codel 110: dev wan1 parent 1:11 limit 1001p flows 1024 quantum 300 target 22.0ms interval 117.0ms memory_limit 32Mb ecn
qdisc ingress ffff: dev wan1 parent ffff:fff1 ----------------
qdisc htb 1: dev wan2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 1 ver 3.17 direct_qlen 100
qdisc fq_codel 120: dev wan2 parent 1:12 limit 1001p flows 1024 quantum 300 target 14.7ms interval 109.7ms memory_limit 32Mb ecn
qdisc fq_codel 130: dev wan2 parent 1:13 limit 1001p flows 1024 quantum 300 target 14.7ms interval 109.7ms memory_limit 32Mb ecn
qdisc fq_codel 110: dev wan2 parent 1:11 limit 1001p flows 1024 quantum 300 target 14.7ms interval 109.7ms memory_limit 32Mb ecn
qdisc ingress ffff: dev wan2 parent ffff:fff1 ----------------
qdisc noqueue 0: dev wan3 root refcnt 2
qdisc htb 1: dev ifb4wan1 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4wan1 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc htb 1: dev ifb4wan2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4wan2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn