Can’t fix bufferbloat

vicking · August 11, 2024, 2:36pm

Hi all,

Since yesterday I am using my raspberry pi 4 for my 1gb fibre connection.

I followed the guide to set it up, including SqM.
Everything is working great except the SqM part and bufferbloat test. I cant get a A+ score..

Unloaded my latency is 4/5ms, which should be fine but down and upload still reach 11ms, even with SqM.

I ran several speedtests without SqM and was getting a 930/930mbit result. So I set it to 90% which is 837.000 kbit using cake, peace of cake, ethernet and 44 as overhead.

Other setting I use are irqbalance and packet steering. With htop I checked the cpu and the load was nicely mannaged between the cores while running a speedtest so none of them was a 100%.

Any ideas left?

brada4 · August 11, 2024, 2:42pm

Lets set some baseline to start with:
Run latency test connected to ISP, connected via rpi without sqm.
https://www.waveform.com/tools/bufferbloat
(post links, they contain no IP address)
Note that raspberry wifi is sub-standard by means of supportability, you need mnaged switch with vlans and extra dumb-ap to use it at its full ability.

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall
cat /etc/config/sqm
tc -s qdisc

brada4 · August 11, 2024, 2:50pm

The normal procedure for setting sqm is to set download control to zero, set upload to half of measured bandwidth, then chop halfs up and make a step back when upload latency starts to grow. Then repeat with download direction.

vicking · August 11, 2024, 5:52pm

Thanks for your answer. I use the RPI with a switch and AP's indeed with VLAN for IPTV. All is working nicely!

Herby the files and tests you asked for!

{
        "kernel": "5.15.162",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 4 Model B Rev 1.4",
        "board_name": "raspberrypi,4-model-b",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.4",
                "revision": "r24012-d8dd03c46f",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
        }
}

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd01:0286:9fd4::/48'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device
        option name 'eth1'
        option mtu '1508'

config device
        option type '8021q'
        option ifname 'eth1'
        option vid '6'
        option name 'eth1.6'

config interface 'wan'
        option proto 'pppoe'
        option device 'eth1.6'
        option username 'internet'
        option password 'internet'
        option ipv6 '1'
        option mtu '1500'
        option peerdns '0'
        option metric '1'
        list dns '1.1.1.1'
        option dns_metric '50'

config interface 'wan6'
        option proto 'dhcpv6'
        option device 'pppoe-wan'
        option reqaddress 'try'
        option reqprefix 'auto'
        option peerdns '0'
        option metric '0'
        list dns '2606:4700:4700::1111'
        option dns_metric '55'

config device
        option type '8021q'
        option ifname 'eth1'
        option vid '4'
        option name 'eth1.4'

config interface 'iptv'
        option proto 'dhcp'
        option device 'eth1.4'
        option force_link '1'
        option delegate '0'
        option vendorid 'IPTV_RG'
        option peerdns '0'
        option defaultroute '0'

config device
        option type 'bridge'
        option name 'br-iptv'
        option igmp_snooping '1'
        option bridge_empty '1'
        option stp '1'
        option ipv6 '0'
        list ports 'eth0.4'

config interface 'iptv_lan'
        option proto 'static'
        option device 'br-iptv'
        option ipaddr '192.168.4.1'
        option netmask '255.255.255.0'
        option delegate '0'

config device
        option type '8021q'
        option ifname 'eth0'
        option vid '4'
        option name 'eth0.4'
        list egress_qos_mapping '0:5'
        list egress_qos_mapping '1:5'
        list egress_qos_mapping '2:5'
        list egress_qos_mapping '3:5'
        list egress_qos_mapping '4:5'

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc01:1'
        option channel '36'
        option band '5g'
        option htmode 'VHT80'
        option disabled '1'

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan.local/'
        option domain 'lan.local'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases.lan'
        option localservice '1'
        option ednspacket_max '1232'
        list server '0::1#5453'
        list server '127.0.0.1#5453'
        option noresolv '1'
        option strictorder '1'
        list notinterface 'iptv_lan'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'iptv_lan'
        option interface 'iptv_lan'
        option start '100'
        option limit '150'
        option leasetime '12h'

config dnsmasq 'IPTV'
        option domainneeded '1'
        option authoritative '1'
        option local '/iptv.local/'
        option domain 'iptv.local'
        list server '195.121.1.34'
        list server '195.121.1.66'
        option rebind_protection '1'
        option rebind_localhost '1'
        option localservice '1'
        list notinterface 'loopback'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases.iptv'
        option noresolv '1'
        option strictorder '1'
        option localise_queries '1'
        option ednspacket_max '1232'
        option cachesize '1000'
        list interface 'iptv_lan'

config host
        option name 'IPCAM'
        option ip '192.168.1.60'
        option mac 'A0:9D:C1:25:6C:A9'

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'iptv'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'iptv'

config forwarding
        option src 'iptv_lan'
        option dest 'iptv'

config rule
        option name 'Allow-IPTV-To-Lan'
        option src 'iptv_lan'
        option dest 'iptv'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP-Proxy'
        option family 'ipv4'
        list proto 'udp'
        option src 'iptv'
        option dest 'iptv_lan'
        option target 'ACCEPT'
        list dest_ip '224.0.0.0/4'

config zone
        option name 'iptv_lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'iptv_lan'

config forwarding
        option src 'iptv_lan'
        option dest 'wan'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'IPCAM'
        option family 'ipv4'
        list proto 'tcp'
        option src 'wan'
        option src_dport '554'
        option dest_ip '192.168.1.60'
        option dest_port '554'

config queue 'eth1'
        option enabled '0'
        option interface 'eth1'
        option download '837000'
        option upload '837000'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option linklayer 'ethernet'
        option debug_logging '0'
        option verbosity '5'
        option overhead '44'

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 4067528260 bytes 3448815 pkt (dropped 832, overlimits 0 requeues 764901)
 backlog 0b 0p requeues 764901
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 1255569179 bytes 889297 pkt (dropped 818, overlimits 0 requeues 409086)
 backlog 0b 0p requeues 409086
  maxpacket 1514 drop_overlimit 0 new_flow_count 27837 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 1000317019 bytes 952210 pkt (dropped 0, overlimits 0 requeues 176084)
 backlog 0b 0p requeues 176084
  maxpacket 3150 drop_overlimit 0 new_flow_count 62968 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 665594607 bytes 632933 pkt (dropped 1, overlimits 0 requeues 87851)
 backlog 0b 0p requeues 87851
  maxpacket 1434 drop_overlimit 0 new_flow_count 40421 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 923705425 bytes 817489 pkt (dropped 0, overlimits 0 requeues 91116)
 backlog 0b 0p requeues 91116
  maxpacket 1434 drop_overlimit 0 new_flow_count 58312 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 222342030 bytes 156886 pkt (dropped 13, overlimits 0 requeues 764)
 backlog 0b 0p requeues 764
  maxpacket 1514 drop_overlimit 0 new_flow_count 908 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 3014616501 bytes 3086224 pkt (dropped 1376, overlimits 0 requeues 6820)
 backlog 0b 0p requeues 6820
  maxpacket 1442 drop_overlimit 1344 new_flow_count 22471 ecn_mark 0 drop_overmemory 1344
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-iptv root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.6 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 2948545251 bytes 3087110 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 53960 drop_overlimit 0 new_flow_count 277746 ecn_mark 0
  new_flows_len 0 old_flows_len 0

It's quite a big and advanced config!
Unfortunately I can't run a bufferbloat test at the moment because it keeps haning on "warming up"... will try that later!

Edit: Bufferbloat test results:
https://www.waveform.com/tools/bufferbloat?test-id=84424020-e320-4b94-9939-3125b14c0cae

speedtest:

moeller0 · August 11, 2024, 6:16pm

Mmh, what is eth1? A USB dongle perhaps?

vicking · August 11, 2024, 6:21pm

Indeed, I use the TP Link eu300 USB dongle for the WAN connection

brada4 · August 11, 2024, 6:21pm

I wanted 2 reports - PC connected to provider's internet port and PC connected behind rpi.
You measured 850down 830up for non-throttled location.

vicking · August 11, 2024, 6:23pm

Is that even possible? The provider box is not a modem but FTTH from KPN like this:

Yes, with waveform test I do. But running speedtest with the speedtest app at my computer I get a speed of 930 up and down. That is the speed I used to calculate because I think browser speed tests are somehow slowing it down..

brada4 · August 11, 2024, 6:26pm

Yes, nearby speedtest servers are throttled to maximum. You can try speed.cloudflare.com too.

Start from 850 down and 830 up.:

Set dl limit to zero ul to 415 and so forth.

vicking · August 11, 2024, 6:29pm

Oké, but do you see anything in all the configs I posted?

brada4 · August 11, 2024, 6:43pm

UL bw is above line upload bw, so ul policing does not work at all.
And the eth1 network setup like ethtool -i eth1 should be checked.

vicking · August 11, 2024, 7:08pm

driver: r8152
version: v1.12.13
firmware-version: rtl8153b-2 v2 04/27/23
expansion-rom-version:
bus-info: usb-0000:01:00.0-1
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

This is the output, but not sure what to do with it?

What do you mean?

What I don't understand is: If I run a speedtest with the Speedtest app I always get around 930/930 but now running the cloudflare speedtest I get around 850/500. Not even close to the 930/930.

What test should I use....

brada4 · August 11, 2024, 7:47pm

Move from 512 up towards 1024

moeller0 · August 11, 2024, 9:35pm

What type of galsfaser are you using, AON (active ethernet over a dedicated fiber) or PON ("passive" that is multiple users share the same fiber to/from the ISP to a splitter, from which each users gets their own fiber, but all users share the same fiber part between splitter and ISP). And if PON, which variant, GPON or XGSPON? (My goal is to figure out what the real limits likely are... and how much sharing is likely happening)

vicking · August 11, 2024, 10:07pm

I have AON/GoF fiber

moeller0 · August 12, 2024, 8:13am

Thanks! That helps a bit...

Mmmh, I note that for A+ the latency increase during both upload and download saturation needs to stay below 5 ms... but the default target for cake/fq_codel is 5ms. And under real load you need to expect the average latency increase to reach target order of magnitude...

Could you please perform the following capacity-tests:
https://speed.cloudflare.com (you need to make and post a screenshot of the full page after the test has finished, maybe redact your location and IP address if you consider that sensitive)
https://

Both with SQM disabled and with SQM enabled...

vicking · August 12, 2024, 2:57pm

Oke.. i've ran two tests, both with SQM disabled but one with only Ipv4 and the other with Ipv6 enabled on my machine.

I thought I noticed that when I run a test on my machine with ipv6 enabled the test is slower...

I didn't make any tests yet with SQM enabled because of the results of this speed I am not sure which paramaters to use for Up and Down at the moment. I think the results are very disappointed for a 1gb Up and Down connection...

Ipv4 only:

Ipv6 enabled:

moeller0 · August 13, 2024, 5:04am

Rule of thumb: use the robustly achievable throughput rates from your preferred capacity test as gross shaper rates...

But make sure to pick a test that looks like it can saturate your link...

vicking · August 13, 2024, 5:24am

Oké, I ran some tests with the Ookla speedtest app and these where the results:
919/884, 921/927, 880/857, 919/849, 928/871, 928/929.

So with these results the worst speeds are 880/849 and I will be using 90% of them for the up and down setting, oke?

These are the results with SqM:

Bufferbloat:
https://www.waveform.com/tools/bufferbloat?test-id=dd856cb5-039b-41fb-8884-51462ef16a4f

https://www.waveform.com/tools/bufferbloat?test-id=e51d4998-9824-4736-8933-07a257069081

2 times a A+, but unfortunately it is not consistent
https://www.waveform.com/tools/bufferbloat?test-id=3a1328d2-d532-4b22-b03c-c9c7fc726bb9

brada4 · August 13, 2024, 6:15am

You have less control over download delays, the method rising BW from half hope was followed. Check whether waveform test celebrates A+.