Bufferbloat help

Installing and Using OpenWrt
1

Hi there,

I'm pretty new to OpenWRT and networking in general. The problem I'm trying so solve right now is having an unstable ping, or occasional ping spikes to be more precise.

I bought a GL-MT6000 to enable SQM Cake on my connection and that certainly helped with bufferbloat. I can't achieve a stable A+ on Waveform so far but the Bufferbloat is greatly reduced in comparison to my old Router.

My network:

ISP DOCSIS 3.1 (250Mbps Down / 25Mbps Up) Cable -> Modem/Router (ARRIS TG344DE) in Bridge Mode -> Ethernet -> Router GL-MT6000 -> Ethernet -> My Computer

Yesterday I wanted to check whether the ping spikes keep happening, when I directly connect to the ARRIS TG344DE. I found out that they do keep happening, but I also noticed that I got only 10ms latency when no spikes happened, while I get around 15-20ms latency when connected to the GL-MT6000.

Now the simple question is: Is this to be expected?
5-10ms increase in latency by adding a router sounds a lot to me. I couldn't find an answer to this question by searching the internet.

The more complicated question is: How to fix the Ping Spikes? But from my testing results yesterday it seems that the ISP is the problem here. So I don't have much hope for solving this.

I'm including some info in advance in case that this increase in latency is not to be expected:

Installed on GL-MT6000:

  • Adguard Home with DoH from Cloudfare enabled (Running on Port 53 while the GL-MT6000 DNS runs on Port 54 for local DNS)
  • Docker (Just tinkering with it. Nothing special here)
  • SQM Cake (Configured by following the documentation

I also configured a Guest-WiFi but that's not in use as of now.

I can provide more details if needed. I just included what I think would be helpful as a newcomer to networking.

Thanks for your time!

root@OpenWrt:~# ubus call system board
{
        "kernel": "5.15.162",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 4",
        "model": "GL.iNet GL-MT6000",
        "board_name": "glinet,gl-mt6000",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.4",
                "revision": "r24012-d8dd03c46f",
                "target": "mediatek/filogic",
                "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
        }
}

root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix '***********'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'
        list ports 'lan5'
        option ipv6 '0'

config device
        option name 'lan1'
        option macaddr '***********'

config device
        option name 'lan2'
        option macaddr '***********'

config device
        option name 'lan3'
        option macaddr '***********'

config device
        option name 'lan4'
        option macaddr '***********'

config device
        option name 'lan5'
        option macaddr '***********'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        list dns '127.0.0.1'

config device
        option name 'eth1'
        option macaddr '***********'

config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'
        option vendorid 'openwrt'
        option peerdns '0'

config interface 'modem'
        option proto 'static'
        option device '@wan'
        option ipaddr '192.168.100.101'
        option netmask '255.255.255.0'

config interface 'Guest'
        option proto 'static'
        option ipaddr '192.168.3.1'
        option netmask '255.255.255.0'
        option device 'br-guest'

config device
        option type 'bridge'
        option name 'br-guest'
        option mtu '1500'
        option macaddr '***********'
        option txqueuelen '1000'
        option bridge_empty '1'
        option ipv6 '0'

config interface 'docker'
        option proto 'none'
        option auto '0'
        option device 'docker0'

config device
        option type 'bridge'
        option name 'docker0'
        option bridge_empty '1'

root@OpenWrt:~# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'platform/soc/18000000.wifi'
        option channel '7'
        option band '2g'
        option htmode 'HE20'
        option cell_density '0'
        option country 'DE'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'platform/soc/18000000.wifi+1'
        option channel '136'
        option band '5g'
        option htmode 'HE80'
        option cell_density '0'
        option country 'DE'

config wifi-iface 'wifinet0'
        option device 'radio1'
        option mode 'ap'
        option ssid '***********'
        option encryption 'psk2'
        option key '***********'
        option network 'lan'

config wifi-iface 'wifinet1'
        option device 'radio0'
        option mode 'ap'
        option ssid '***********'
        option encryption 'psk2'
        option key '***********'
        option network 'lan'

config wifi-iface 'wifinet2'
        option device 'radio0'
        option mode 'ap'
        option ssid '***********_Guest'
        option encryption 'psk2'
        option key '***********'
        option network 'Guest'

root@OpenWrt:~# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '0'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option port '54'
        list server '192.168.1.1'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        list dns 'fdc1:***********'
        list dhcp_option '6,192.168.1.1'
        list dhcp_option '3,192.168.1.1'
        list dhcp_option '42,192.168.1.1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config host
        option name '***********'
        option ip '192.168.1.111'
        list mac '***********'
        option leasetime 'infinite'

config dhcp 'Guest'
        option interface 'Guest'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list dhcp_option '6,192.168.1.1'

config host
        option ip '192.168.1.136'
        option name '***********'
        list mac '***********'
        option leasetime 'infinite'

config host
        option name '***********'
        option ip '192.168.1.159'
        list mac '***********'
        option leasetime 'infinite'


root@OpenWrt:~# cat /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'modem'
        list network 'wan'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'guest'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'Guest'

config zone
        option name 'wanGuest'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        list network 'wan'

config rule
        option name 'Allow-GuestDNS'
        option src 'guest'
        option dest_port '53'
        option target 'ACCEPT'

config rule
        option name 'Allow-GuestDHCP'
        option src 'guest'
        option dest_port '67-68'
        option target 'ACCEPT'
        list proto 'udp'

config forwarding
        option src 'guest'
        option dest 'wanGuest'

config redirect
        option dest 'docker'
        option target 'DNAT'
        option name 'test'
        option src 'lan'
        option src_dport '9999'
        option dest_port '9999'

config forwarding
        option src 'docker'
        option dest 'wan'

config zone 'docker'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option name 'docker'
        list network 'docker'

root@OpenWrt:~# cat /etc/sqm/sqm.conf
SQM_LIB_DIR=/usr/lib/sqm
SQM_STATE_DIR=/var/run/sqm
SQM_QDISC_STATE_DIR=${SQM_STATE_DIR}/available_qdiscs
SQM_CHECK_QDISCS="fq_codel codel pie sfq cake"
SQM_SYSLOG=1
IP6TABLES_BINARY=$(command -v ip6tables-nft)
IPTABLES_BINARY=$(command -v iptables-nft)

root@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
        option enabled '1'
        option interface 'eth1'
        option download '250000'
        option upload '25000'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option linklayer 'ethernet'
        option debug_logging '0'
        option verbosity '5'
        option overhead '22'
2

Why not put your isp modem in bridge mode, so that the Openwrt router is directly connected to the internet?

1 Like
3

Can you post two links to waveform tests you are referring to, plus from another with sqm disabled and openwrt rebooted?

4

That's exactly what I did. Maybe I didn't put it clearly by listing the ISP before. But the ARRIS TG344DE is the first device after the coaxial cable and is set to bridge mode.

1 Like
5

I will get back to you in 1-2 hours. Can't shut down the internet right now unfortunately.

6

I can't do it today sorry. Will get back tomorrow.

7

No rush, take your time, weekend is same for everybody.

8

Still cannot reboot the modem to do a waveform test while being connected directly to the modem. But I made some changes to the SQM Cake settings and did waveform tests again:

  • Ingress: 225000
  • Egress: 22500
  • Per Packet overhead: 48 bytes

SQM active: Bufferbloat and Internet Speed Test - Waveform

SQM disabled and rebooted router: Bufferbloat and Internet Speed Test - Waveform

I'm surprised by the SQM disabled results. The last time I tested this the result of Download Active was in the range of ~+50ms. Maybe it depends on the time of the day and how many other people are using the same cable network in my neighbourhood?

9

Set ingress to zero and try to tune egress side of SQM:
Set bandwidth limit to half of prescribed
25Mbps equals 25*1024 = 25600 kbps
Start by setting egress bandwidth limit to 12800
Upload latency should be 0...+1 , now add half of remaining BW ie 19200kbps and so on until loaded latency increases and then make one step back to have it under control.
Now repeat same with ingress. Just one zero more. If there is high jitter with half download bandwidth your provider has some unbearable traffic management, still you should be able to find sweet spot when it turns from good to average.

1 Like
10

Thanks for your help. I got the upload sweetspot at 22400 kbps.
Download on the other side seems very random. I did 3 tests each for 128000 kbps and 192000 kbps.

128000:
Bufferbloat and Internet Speed Test - Waveform
Bufferbloat and Internet Speed Test - Waveform
Bufferbloat and Internet Speed Test - Waveform

192000:
Bufferbloat and Internet Speed Test - Waveform
Bufferbloat and Internet Speed Test - Waveform
Bufferbloat and Internet Speed Test - Waveform

Is it normal to have these single high latency outliers?
The jitter is noticeably higher at 192000 but the results seem good to, at least according to waveforms grading.

11

Some buffer shared by you and 2^N co-customers in the area and when enough of yous up/download at once few packets get stuck in unmanaged queue.
150ms @ 12Gbps = 256MB buffer, sounds credible.
Likely will go away with firmware update by provider, but dont expect wonders.

1 Like
12

So I just rebooted my modem and connected directly to it. This time the latency was about the same as when I connected through the router. Must've been some stupid coincidence on my side. Sorry for the troubles but this topic can now be closed.

Thanks again @brada4 for your help.

13

Tap the checkbox under most informative/entertaining comment, and it is done.

14

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.