How do you assign these downloads to the bulk tin in your rules?
On new connections going to a particular destination port, I set the connection mark to 1 (lephb). The tc filter then converts this to DSCP on each packet.
What mark is shown in /proc/net/nf_conntrack for all these connections?
In both cases mark=1 on each connection. In the fastpath case I can also confirm fastpath is working because I also see [OFFLOAD] on each connection.
What hits do you observe on the egress filter rule before and after the download?
tc -s filter show dev wan
This is getting interesting. The filter statistics show that the packets are flowing through the filter in both cases. In the fastpath off case, all packets are getting the DSCP set by the ctinfo dscp action. However, in the fastpath on case, none of the packets are getting the DSCP set. The statistics in that case show that there are no errors setting the the DSCP: DSCP set 6443637 error 0.
Looking at the code in the kernel, the only way this can happen is if the ct mark is not set. In that case dscp == newdscp because they are both 0. However, I previously verified in conntrack -L that the connections show that the ct mark is set to 1. Is this a bug in the kernel where the ct mark is somehow not properly getting passed through the netfilter code when fastpath is used?
Raw statistics follow:
No fastpath:
root@router:~# tc -s filter show dev wan
filter parent 1: protocol all pref 49152 matchall chain 0
filter parent 1: protocol all pref 49152 matchall chain 0 handle 0x1
not_in_hw (rule hit 10121643)
action order 1: ctinfo zone 0 continue
index 1 ref 1 bind 1 dscp 0x0000003f 0000000000 installed 83188 sec used 0 sec firstused 83183 sec DSCP set 5535376 error 0 CPMARK set 0
Action statistics:
Sent 3730824707 bytes 10505636 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
root@router:~# tc -s filter show dev wan
filter parent 1: protocol all pref 49152 matchall chain 0
filter parent 1: protocol all pref 49152 matchall chain 0 handle 0x1
not_in_hw (rule hit 11027232)
action order 1: ctinfo zone 0 continue
index 1 ref 1 bind 1 dscp 0x0000003f 0000000000 installed 83236 sec used 0 sec firstused 83231 sec DSCP set 6439820 error 0 CPMARK set 0
Action statistics:
Sent 3793514959 bytes 11411232 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
Fastpath:
root@router:~# tc -s filter show dev wan
filter parent 1: protocol all pref 49152 matchall chain 0
filter parent 1: protocol all pref 49152 matchall chain 0 handle 0x1
not_in_hw (rule hit 11033409)
action order 1: ctinfo zone 0 continue
index 1 ref 1 bind 1 dscp 0x0000003f 0000000000 installed 83354 sec used 0 sec firstused 83349 sec DSCP set 6440026 error 0 CPMARK set 0
Action statistics:
Sent 3794717853 bytes 11417431 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
root@router:~# tc -s filter show dev wan
filter parent 1: protocol all pref 49152 matchall chain 0
filter parent 1: protocol all pref 49152 matchall chain 0 handle 0x1
not_in_hw (rule hit 11913138)
action order 1: ctinfo zone 0 continue
index 1 ref 1 bind 1 dscp 0x0000003f 0000000000 installed 83411 sec used 0 sec firstused 83406 sec DSCP set 6443637 error 0 CPMARK set 0
Action statistics:
Sent 3855562855 bytes 12297162 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
I'm conscious of:
But it looks like you indicate above that the conntrack is getting correctly set?
So the issue is rather with the 'tc action' it seems(?):
I don't believe the issue is in the script. According to conntrack, the connections are all marked properly
@ldir can you shed any light on this I wonder? Specifically see this post:
Are they all accounted for in conntrack? Any missing?
Are they all accounted for in conntrack? Any missing?
root@router:~# grep "dport=$PORT " /proc/net/nf_conntrack | grep mark=1 | wc -l
150
Looks like I’m not the first person to notice the fastpath issue: Software flow offloading implications - #30 by Luka
Sure, but then you have to account for different syntaxes for your users.
Thanks and understood. Also is that only for ipv4 or for both?
Both can be simplified.
Hopefully I got this right with this commit:
I just updated to this version and noticed in the output of nft list ruleset
chain store-dscp-in-conntrack {
meta nfproto ipv4 ct mark set @nh,8,8 & 0xfc [invalid type]
I found mention of it in this bug report[1]. I’m not sure of its ramifications. Is this an issue?
I saw that too, but the DSCPs seem to be getting set correctly:
root@OpenWrt-1:~# service cake-qos-simple download
qdisc cake 1: root refcnt 2 bandwidth 40Mbit diffserv4 triple-isolate nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 0
Sent 64282474755 bytes 55699019 pkt (dropped 46790, overlimits 75856554 requeues 0)
backlog 0b 0p requeues 0
memory used: 3219768b of 4Mb
capacity estimate: 20Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 46 / 1500
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 2500Kbit 40Mbit 20Mbit 10Mbit
target 7.27ms 5ms 5ms 5ms
interval 102ms 100ms 100ms 100ms
pk_delay 7.79ms 805us 3.06ms 532us
av_delay 568us 90us 1.41ms 67us
sp_delay 13us 10us 7us 10us
backlog 0b 0b 0b 0b
pkts 417030 53390887 1569530 368362
bytes 469252271 62043102983 1697044949 136490897
way_inds 3173 2176738 13 1618
way_miss 2446 76382 229 18078
way_cols 0 0 0 0
drops 66 40207 6516 1
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 1 5 1 1
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 61830 68700 31602 3093
quantum 300 1220 610 305
Can also be confirmed by inspecting the DSCP values on ingress (following restoration from conntrack) using: tcpdump -i ifb-wan -v.
Are they getting set correctly on your system too?
For that specific statement, it’s enough to check that the mark field in /proc/net/nf_conntrack is getting set to the DSCP values.
grep -oE "mark=[0-9]+" /proc/net/nf_conntrack | sort | uniq -c
Nice:
root@OpenWrt-1:~# grep -oE "mark=[0-9]+" /proc/net/nf_conntrack | sort | uniq -c
176 mark=0
25 mark=16
10 mark=32
4 mark=8
Everything looks good on my end too. I won’t worry about it then. Thank you!
I found an issue involving port forwarding. I have a fw4 rule on the router like:
config redirect
option target 'DNAT'
option name 'Redacted'
option src 'wan'
option dest_ip '192.168.1.5'
option dest 'lan'
list proto 'tcp'
list proto 'udp'
option dest_port '55550'
option src_dport '55550'
On my Linux server I have an nft rule like:
table inet filter {
chain output {
type filter hook output priority filter; policy accept;
meta skuid 1111 ip dscp set cs1
}
}
With the port forwarding rule disabled all download and upload traffic from the user is marked CS1 but I have to add ‘established’ or remove ‘ct state’ entirely to get it to work with the port forwarding rule enabled.
# Does not work with my setup and port forwarding
oifname wan ct state new,untracked goto classify-and-store-dscp
# Both of these commands work
oifname wan ct state new,untracked,established goto classify-and-store-dscp
oifname wan goto classify-and-store-dscp