Router: Raspberry Pi 4B running OpenWrt 23.05.4
Until recently, SQM on a PPPoE connection worked seamlessly alongside NAT offloading. However, after upgrading from version 23.05.3 to the latest release, it no longer functions correctly on ingress (though it still works on egress). Changing the SQM interface from PPPoE to eth1 resolves the issue.
I am aware that the documentation under Routing/NAT Offloading mentions Not fully compatible with QoS/SQM , but SQM has been functioning without issues for the past two years on older versions. Additionally, there are no errors in the System Log.
The only current workaround is to disable NAT offloading.
Network:
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd88:faba:979d::/48'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
config interface 'wan'
option proto 'pppoe'
option device 'eth1'
option username ''
option password ''
option ipv6 'auto'
option peerdns '0'
Firewall:
config defaults
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'
option flow_offloading '1'
option flow_offloading_hw '1'
config zone
option name 'lan'
list network 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
config zone
option name 'wan'
list network 'wan'
list network 'wan6'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
config forwarding
option src 'lan'
option dest 'wan'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
SQM:
config queue 'eth1'
option enabled '1'
option interface 'pppoe-wan'
option download '21000'
option upload '21000'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option qdisc_advanced '1'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option qdisc_really_really_advanced '1'
option itarget 'auto'
option etarget 'auto'
option linklayer 'none'
option debug_logging '0'
option verbosity '5'
option squash_dscp '1'
option squash_ingress '1'
option iqdisc_opts 'nat ingress'
option eqdisc_opts 'nat ack-filter'
brada4
August 2, 2024, 8:54am
2
No hardware offload on raspberry.
Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
ubus call system board
At least software offload should work.
brada4:
ubus call system board
{
"kernel": "5.15.162",
"hostname": "OpenWrt",
"system": "ARMv8 Processor rev 3",
"model": "Raspberry Pi 4 Model B Rev 1.2",
"board_name": "raspberrypi,4-model-b",
"rootfs_type": "squashfs",
"release": {
"distribution": "OpenWrt",
"version": "23.05.4",
"revision": "r24012-d8dd03c46f",
"target": "bcm27xx/bcm2711",
"description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
}
}
brada4
August 2, 2024, 1:55pm
4
Funny thing is pppoe-wan offload is finally working in 23.05.4 and it no longer crosses slowpath that was subject to SQM. Your approach to have offloaded firewall and SQM on different interfaces is the correct one.
But it was working in older versions. After upgrade it doesn't work on ingress pppoe interface. So is this a bug?
brada4
August 2, 2024, 6:33pm
6
No, it is how it is supposed to work - offload bypasses qdisc.
2 Likes
Understood thank you. Until now I thought software NAT offloading was working along with SQM. So, I should just disable Offload from now on to make SQM work on pppoe.
brada4
August 2, 2024, 8:03pm
8
Correct, you never needed offload cpu-wise, you also figured out workaround with chaining interfaces.
1 Like
I have software offload working fine with SQM (piece_of_cake.qos). Do you mean HW offload?
https://thermalcircle.de/doku.php?id=blog:linux:flowtables_1_a_netfilter_nftables_fastpath
@brada4 @dave14305 How can I confirm if offload is working?
I'm only shaping on upload these days, and have plenty of offloaded connections. I just rebooted yesterday for the 23.05.4 upgrade.
root@router:~# tc -s qdisc show dev eth1
qdisc cake 800b: root refcnt 5 bandwidth 23Mbit besteffort dual-srchost nat nowash ack-filter split-gso rtt 100ms noatm overhead 18
Sent 1702964456 bytes 3425619 pkt (dropped 186545, overlimits 4031980 requeues 65)
backlog 0b 0p requeues 65
memory used: 2371685b of 4Mb
capacity estimate: 23Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 46 / 1518
average network hdr offset: 14
Tin 0
thresh 23Mbit
target 5ms
interval 100ms
pk_delay 2.75ms
av_delay 157us
sp_delay 3us
backlog 0b
pkts 3612164
bytes 1761550534
way_inds 95494
way_miss 124982
way_cols 0
drops 30539
marks 172
ack_drop 156006
sp_flows 2
bk_flows 1
un_flows 0
max_len 66455
quantum 701
root@router:~# grep -c OFFLOAD /proc/net/nf_conntrack
41
Can you try it on ingress and report back?
Works for me still. Added the ingress qdisc and ran a speedtest. No PPPoE here though.
root@router:~# grep -c OFFLOAD /proc/net/nf_conntrack
59
root@router:~# tc -s qdisc show root
qdisc cake 800e: dev eth1 root refcnt 5 bandwidth 23Mbit besteffort dual-srchost nat nowash ack-filter split-gso rtt 100ms noatm overhead 18
Sent 46431577 bytes 58247 pkt (dropped 16493, overlimits 159490 requeues 14)
backlog 0b 0p requeues 14
memory used: 1155Kb of 4Mb
capacity estimate: 23Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 46 / 1518
average network hdr offset: 14
Tin 0
thresh 23Mbit
target 5ms
interval 100ms
pk_delay 241us
av_delay 21us
sp_delay 6us
backlog 0b
pkts 74740
bytes 48338737
way_inds 43
way_miss 444
way_cols 0
drops 360
marks 0
ack_drop 16133
sp_flows 5
bk_flows 1
un_flows 0
max_len 9084
quantum 701
qdisc cake 800f: dev ifb4eth1 root refcnt 2 bandwidth 800Mbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 18
Sent 699470154 bytes 481913 pkt (dropped 20, overlimits 513885 requeues 0)
backlog 0b 0p requeues 0
memory used: 3052868b of 15140Kb
capacity estimate: 800Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 64 / 1518
average network hdr offset: 14
Tin 0
thresh 800Mbit
target 5ms
interval 100ms
pk_delay 67us
av_delay 31us
sp_delay 2us
backlog 0b
pkts 481933
bytes 699500434
way_inds 70
way_miss 383
way_cols 0
drops 20
marks 0
ack_drop 0
sp_flows 4
bk_flows 1
un_flows 0
max_len 68130
quantum 1514
Thanks really good! Although you're supposed to apply SQM on WAN interface and not eth1 correct? Moeller told me so back then
On my x86, eth1 is my WAN. eth0 is my LAN.
1 Like
With software offload:
root@OpenWrt:~# conntrack -L -u offload
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2620:1ec:bdf::68 sport=60857 dport=443 packets=13 bytes=4035 src=2620:1ec:bdf::68 dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60857 packets=16 bytes=9153 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=172.105.109.165 sport=61783 dport=4222 packets=118 bytes=18069 src=172.105.109.165 dst=100.64.55.195 sport=4222 dport=61783 packets=118 bytes=5152 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=151.101.194.133 sport=60854 dport=443 packets=14 bytes=3123 src=151.101.194.133 dst=100.64.55.195 sport=443 dport=60854 packets=26 bytes=24252 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=151.101.194.133 sport=60852 dport=443 packets=11 bytes=2982 src=151.101.194.133 dst=100.64.55.195 sport=443 dport=60852 packets=15 bytes=6240 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=151.101.1.194 sport=60884 dport=443 packets=229 bytes=74878 src=151.101.1.194 dst=100.64.55.195 sport=443 dport=60884 packets=232 bytes=110526 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=20.212.88.117 sport=60531 dport=443 packets=35 bytes=6998 src=20.212.88.117 dst=100.64.55.195 sport=443 dport=60531 packets=46 bytes=9042 [OFFLOAD] mark=0 use=2
udp 17 src=192.168.1.101 dst=35.190.43.134 sport=57445 dport=443 packets=206 bytes=105850 src=35.190.43.134 dst=100.64.55.195 sport=443 dport=57445 packets=170 bytes=35241 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=204.79.197.203 sport=60831 dport=443 packets=78 bytes=27432 src=204.79.197.203 dst=100.64.55.195 sport=443 dport=60831 packets=143 bytes=157796 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700::6812:b2b sport=60848 dport=443 packets=11 bytes=1562 src=2606:4700::6812:b2b dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60848 packets=12 bytes=6060 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700::6812:1f02 sport=60773 dport=443 packets=33 bytes=24068 src=2606:4700::6812:1f02 dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60773 packets=48 bytes=11622 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.105 dst=15.164.43.102 sport=49153 dport=8883 packets=802 bytes=181487 src=15.164.43.102 dst=100.64.55.195 sport=8883 dport=49153 packets=681 bytes=50094 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700:3030::6815:30c7 sport=60931 dport=443 packets=7 bytes=854 src=2606:4700:3030::6815:30c7 dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60931 packets=7 bytes=4098 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=185.199.110.153 sport=60807 dport=443 packets=13 bytes=1418 src=185.199.110.153 dst=100.64.55.195 sport=443 dport=60807 packets=15 bytes=4832 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=151.101.1.194 sport=60882 dport=443 packets=235 bytes=77149 src=151.101.1.194 dst=100.64.55.195 sport=443 dport=60882 packets=236 bytes=113680 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=151.101.2.133 sport=60856 dport=443 packets=11 bytes=3048 src=151.101.2.133 dst=100.64.55.195 sport=443 dport=60856 packets=14 bytes=4719 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=157.90.91.74 sport=60578 dport=443 packets=42 bytes=3765 src=157.90.91.74 dst=100.64.55.195 sport=443 dport=60578 packets=51 bytes=14801 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.108 dst=20.157.85.165 sport=58376 dport=1883 packets=139 bytes=11612 src=20.157.85.165 dst=100.64.55.195 sport=1883 dport=58376 packets=126 bytes=6720 [OFFLOAD] mark=0 use=3
tcp 6 src=192.168.1.101 dst=64.233.170.188 sport=39948 dport=5228 packets=497 bytes=31085 src=64.233.170.188 dst=100.64.55.195 sport=5228 dport=39948 packets=537 bytes=465795 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700::6812:a2b sport=60929 dport=443 packets=10 bytes=1538 src=2606:4700::6812:a2b dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60929 packets=12 bytes=5570 [OFFLOAD] mark=0 use=2
On older version (23.05.3)
tcp 6 src=192.168.1.100 dst=13.67.9.5 sport=58769 dport=443 packets=26 bytes=16143 src=13.67.9.5 dst=100.64.55.195 sport=443 dport=58769 packets=23 bytes=9434 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=69.173.158.64 sport=54236 dport=443 packets=12 bytes=3279 src=69.173.158.64 dst=100.64.55.195 sport=443 dport=54236 packets=13 bytes=5212 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=34.160.128.112 sport=60062 dport=443 packets=20 bytes=1990 src=34.160.128.112 dst=100.64.55.195 sport=443 dport=60062 packets=24 bytes=7976 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=34.111.94.218 sport=60111 dport=443 packets=16 bytes=3582 src=34.111.94.218 dst=100.64.55.195 sport=443 dport=60111 packets=14 bytes=4923 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=67.199.150.77 sport=59985 dport=443 packets=12 bytes=3093 src=67.199.150.77 dst=100.64.55.195 sport=443 dport=59985 packets=13 bytes=5463 [OFFLOAD] mark=0 use=2
udp 17 src=192.168.1.101 dst=35.244.195.33 sport=40942 dport=443 packets=4 bytes=2804 src=35.244.195.33 dst=100.64.55.195 sport=443 dport=40942 packets=5 bytes=3369 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=199.232.192.134 sport=60026 dport=443 packets=20 bytes=3380 src=199.232.192.134 dst=100.64.55.195 sport=443 dport=60026 packets=35 bytes=34604 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=54.230.46.225 sport=56281 dport=443 packets=32 bytes=10722 src=54.230.46.225 dst=100.64.55.195 sport=443 dport=56281 packets=41 bytes=12145 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=104.18.36.155 sport=58823 dport=443 packets=21 bytes=6944 src=104.18.36.155 dst=100.64.55.195 sport=443 dport=58823 packets=24 bytes=8028 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=146.75.116.134 sport=60059 dport=443 packets=12 bytes=3583 src=146.75.116.134 dst=100.64.55.195 sport=443 dport=60059 packets=16 bytes=6915 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=34.120.111.33 sport=54249 dport=443 packets=19 bytes=2388 src=34.120.111.33 dst=100.64.55.195 sport=443 dport=54249 packets=23 bytes=7020 [OFFLOAD] mark=0 use=3
tcp 6 src=192.168.1.100 dst=151.101.153.194 sport=60108 dport=443 packets=13 bytes=4567 src=151.101.153.194 dst=100.64.55.195 sport=443 dport=60108 packets=15 bytes=3937 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700::6812:80d8 sport=60127 dport=443 packets=14 bytes=3291 src=2606:4700::6812:80d8 dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60127 packets=13 bytes=6289 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=207.65.33.79 sport=50769 dport=443 packets=20 bytes=3768 src=207.65.33.79 dst=100.64.55.195 sport=443 dport=50769 packets=26 bytes=7348 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=35.244.154.8 sport=59982 dport=443 packets=17 bytes=1656 src=35.244.154.8 dst=100.64.55.195 sport=443 dport=59982 packets=21 bytes=7967 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=52.223.2.229 sport=50633 dport=443 packets=25 bytes=4398 src=52.223.2.229 dst=100.64.55.195 sport=443 dport=50633 packets=33 bytes=9587 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.101 dst=216.239.36.135 sport=37683 dport=443 packets=16 bytes=4099 src=216.239.36.135 dst=100.64.55.195 sport=443 dport=37683 packets=17 bytes=8597 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=34.36.216.150 sport=58741 dport=443 packets=18 bytes=1979 src=34.36.216.150 dst=100.64.55.195 sport=443 dport=58741 packets=20 bytes=5332 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700::6812:572a sport=60130 dport=443 packets=15 bytes=3662 src=2606:4700::6812:572a dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=60130 packets=15 bytes=5882 [OFFLOAD] mark=0 use=2
tcp 6 src=2407:8700:0:5070:f9ec:eaff:f748:520b dst=2606:4700:10::6816:3456 sport=59996 dport=443 packets=20 bytes=2066 src=2606:4700:10::6816:3456 dst=2407:8700:0:5070:f9ec:eaff:f748:520b sport=443 dport=59996 packets=38 bytes=34833 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=4.153.25.145 sport=63184 dport=443 packets=9 bytes=728 src=4.153.25.145 dst=100.64.55.195 sport=443 dport=63184 packets=7 bytes=5567 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=34.96.105.8 sport=60007 dport=443 packets=16 bytes=1635 src=34.96.105.8 dst=100.64.55.195 sport=443 dport=60007 packets=18 bytes=6179 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=35.71.178.8 sport=54265 dport=443 packets=16 bytes=3370 src=35.71.178.8 dst=100.64.55.195 sport=443 dport=54265 packets=22 bytes=7392 [OFFLOAD] mark=0 use=2
tcp 6 src=192.168.1.100 dst=185.184.8.90 sport=60006 dport=443 packets=16 bytes=3196 src=185.184.8.90 dst=100.64.55.195 sport=443 dport=60006 packets=16 bytes=5460 [OFFLOAD] mark=0 use=2
conntrack v1.4.8 (conntrack-tools): 66 flow entries have been shown.
Seems like offload works in both versions? So, why in the new version SQM on PPPoE isn't working with software flow offloading?
brada4
August 2, 2024, 8:43pm
18
Question is whether every packet passes qdisc or just offload makes it more of pfifo.
Yes that. How to check that? i mean lemme try tc qdisc command and report back
What does "isn't working" mean to you? Speed not being limited? Stats not updating? qdisc not setup by sqm-scripts?
