Hello, what´s the best and easiest way to completely wash/clear the ingress traffic that my isp sends?
I can see alot of traffic with DSCP tags, Flowlabels etc. while monitoring with tcpdump.
Network: Modem in bridge mode which sets the isp vlan tag and behind that my OpenWrt 22.03.5 device to which my pc connects.
thanks in advance
You can rewrite them in the firewall, but if no application honours them why bother?
Because I´m using Classify and i want the packets from my isp to end up in my sqm layer_cake CS0 tin but they end up in the highest tin... also i think cake evaluates the ingress dscps before the firewall rewrite. Someone said veth0 would be a solution but i dont know how to set that up and i cant find any usefull information.
Is your LAN throughput lower than the uplink to the ISP and you have applied SQM ingress?
I’m also using dscpclassify for a while now and based on my testings it will still work even if you set squash dscp to “1” aka set the wash option for cake on ingress. For it to work Ignore DSCP on ingress has to be on “Allow”.
The DSCPCLASSIFY README on github clearly states that:
It is important that Squash DSCP and Ignore DSCP on ingress are not enabled in SQM setup otherwise cake will ignore the service's DSCP classes.
.. but as I said when I’m testing it DSCP get’s restored even if Squash DSCP on inbound packets is enabled.
Please try it for yourself and report back if it solves your issue.
Maybe @moeller0, @amteza and @yelreve can chime in and shed some light on my findings.
Maybe there is something wrong with my testing:
root@OpenWrtHudra:~# tcpdump -i eth1 icmp and host 8.8.8.8 -vv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:00:39.109219 IP (tos 0x20, ttl 127, id 44282, offset 0, flags [none], proto ICMP (1), length 60)
185.217.1.12 > dns.google: ICMP echo request, id 1, seq 1614, length 40
root@OpenWrtHudra:~# tcpdump -i br-lan icmp and host 8.8.8.8 -vv
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:02:33.831923 IP (tos 0x20, ttl 111, id 0, offset 0, flags [none], proto ICMP (1), length 60)
dns.google > 192.168.1.190: ICMP echo reply, id 1, seq 1727, length 40
root@OpenWrtHudra:~# tcpdump -i eth1 icmp and host 8.8.8.8 -vv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:06:02.130249 IP (tos 0x20, ttl 127, id 44572, offset 0, flags [none], proto ICMP (1), length 60)
185.217.1.12 > dns.google: ICMP echo request, id 1, seq 1904, length 40
root@OpenWrtHudra:~# tcpdump -i br-lan icmp and host 8.8.8.8 -vv
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:07:01.027866 IP (tos 0x20, ttl 111, id 0, offset 0, flags [none], proto ICMP (1), length 60)
dns.google > 192.168.1.190: ICMP echo reply, id 1, seq 1962, length 40
I removed the unnecessary stuff from my tcpdump…
Fresh tc -s qdisc with wash on ingress where you can see it's clearly hitting the bulk tin:
root@OpenWrtHudra:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 95555289116 bytes 103176397 pkt (dropped 0, overlimits 0 requeues 5330)
backlog 0b 0p requeues 5330
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 26515675408 bytes 32419442 pkt (dropped 0, overlimits 0 requeues 2550)
backlog 0b 0p requeues 2550
maxpacket 66616 drop_overlimit 0 new_flow_count 1371 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 19465295422 bytes 22066614 pkt (dropped 0, overlimits 0 requeues 1204)
backlog 0b 0p requeues 1204
maxpacket 66616 drop_overlimit 0 new_flow_count 949 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 29671817116 bytes 28368599 pkt (dropped 0, overlimits 0 requeues 936)
backlog 0b 0p requeues 936
maxpacket 1514 drop_overlimit 0 new_flow_count 741 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 19902501170 bytes 20321742 pkt (dropped 0, overlimits 0 requeues 640)
backlog 0b 0p requeues 640
maxpacket 66616 drop_overlimit 0 new_flow_count 658 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 806c: dev eth1 root refcnt 5 bandwidth 45Mbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 25ms noatm overhead 40 mpu 84
Sent 187338 bytes 620 pkt (dropped 1, overlimits 200 requeues 0)
backlog 0b 0p requeues 0
memory used: 29952b of 4Mb
capacity estimate: 45Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 84 / 1540
average network hdr offset: 13
Bulk Best Effort Video Voice
thresh 2812Kbit 45Mbit 22500Kbit 11250Kbit
target 6.46ms 1.25ms 1.25ms 1.61ms
interval 30.2ms 25ms 25ms 25.4ms
pk_delay 5us 400us 1.44ms 0us
av_delay 0us 29us 70us 0us
sp_delay 0us 1us 1us 0us
backlog 0b 0b 0b 0b
pkts 10 428 182 1
bytes 764 128514 59499 42
way_inds 0 0 0 0
way_miss 2 47 19 1
way_cols 0 0 0 0
drops 0 1 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 2 0 0
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 98 1481 13080 42
quantum 300 1373 686 343
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 343036 bytes 2369 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth2 root
Sent 7580172529 bytes 8995740 pkt (dropped 0, overlimits 0 requeues 712)
backlog 0b 0p requeues 712
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 1556126611 bytes 1703795 pkt (dropped 0, overlimits 0 requeues 181)
backlog 0b 0p requeues 181
maxpacket 6056 drop_overlimit 0 new_flow_count 182 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 1637800422 bytes 1819200 pkt (dropped 0, overlimits 0 requeues 173)
backlog 0b 0p requeues 173
maxpacket 66616 drop_overlimit 0 new_flow_count 198 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 1931951861 bytes 2023430 pkt (dropped 0, overlimits 0 requeues 179)
backlog 0b 0p requeues 179
maxpacket 66616 drop_overlimit 0 new_flow_count 154 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 2454293635 bytes 3449315 pkt (dropped 0, overlimits 0 requeues 179)
backlog 0b 0p requeues 179
maxpacket 66616 drop_overlimit 0 new_flow_count 173 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth3 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth3 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth4 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth4 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth4 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth5 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth5 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth5 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.10 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.15 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.25 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.33 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.50 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev lxcbr0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.44 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wg0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wg01 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev docker0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-eff9b1a93cc8 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tun1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 11236476 bytes 126729 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 684 drop_overlimit 0 new_flow_count 103 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun2 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 6558 bytes 24 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 474162 bytes 1018 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev vethd27dee9 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 806d: dev ifb4eth1 root refcnt 2 bandwidth 88Mbit diffserv4 dual-dsthost nat wash ingress no-ack-filter split-gso rtt 25ms noatm overhead 40 mpu 84
Sent 375762 bytes 2367 pkt (dropped 2, overlimits 733 requeues 0)
backlog 0b 0p requeues 0
memory used: 31032b of 4Mb
capacity estimate: 88Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 86 / 1540
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 5500Kbit 88Mbit 44Mbit 22Mbit
target 3.3ms 1.25ms 1.25ms 1.25ms
interval 27.1ms 25ms 25ms 25ms
pk_delay 5us 545us 1.34ms 22us
av_delay 0us 69us 52us 3us
sp_delay 0us 4us 3us 0us
backlog 0b 0b 0b 0b
pkts 9 449 173 1738
bytes 666 214487 59357 104280
way_inds 0 0 0 0
way_miss 1 40 19 1
way_cols 0 0 0 0
drops 0 1 1 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 1 0 1
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 74 10598 14810 60
quantum 300 1514 1342 671
This is because for cake's wash option only removes a DSCP after actually applying it to sort a packet into a priority tin, if you do not want that use the besteffort mode with only a single priority tin...
Yes I know that but it looks like dscpclassify restores the DSCP from conntrack afterwards. Otherwise a tcpdump on the lan interface would show no mark.
But after thinking about it again my solution probably won’t even help the op. Even when washing his marks on ingress his isp marked packets would have first landed in his ingress cake instance because as you said washing happens after sorting it into the tins.
But if it did that then cake would only ever see the ISPs DSCPs, is that what you observe?
In sqm-scripts we combine 'wash' with best effort, so ingress DSCPs are:
first, not acted upon as there is only a single priority tier
and then washed away so the internal network does not see them.
I do think with tc it should be possible to clean the full DSCP field before cake sees it, but at that point cake will only see DSCP CS0/BE and al packets will end up in a single tier, so why use anything else then besteffort? If there is some other tool in play to remark a few packets to treat them to other priority tiers (like qosify or dscpclassify) that other tool likely already has the capability to re-map the ISP's DSCPs to arbitrary values?
It doesn´t work for me, the isp traffic is still landing in the wrong tins.
No i´m using a 32/7 vdsl connection for this setup.
Yes ingress & egress
To be clear I just wanted to help the op. I don’t have any issues with DSCPCLASSIFY. I’m using it and everything works as it should. My isp doesn’t mark my incoming packets so it’s hard for me to test the op’s use case. I would have to set up two routers with the upstream router marking packets and not washing them.
No, with DSCPCLASSIFY DSCPs are also restored before from conntrack.
Well here is my question, if you do not re-mark DSCPs on ingress, why do you have different priority tins for ingress in the first place?
The important traffic I marked on egress will be automatically prioritized on the ingress side through dscpclassify and for that i have to enable diffserv on ingress and hence the isp traffic often ends up in my highest tin together with my important traffic. I want all ingress traffic to be CS0 except my important traffic that automatically gets EF marked for ingress through dscpclassify.
My knowledge with nftables is pretty limited but reading through the wiki there is the ingress hook. Please correct me if I’m wrong but this hook comes before prerouting and even before tc. So maybe we can use it to wash the DSCP marks from your isp.
You could try this:
cat << "DSCP" > /etc/nftables.d/05-rules-ingress.nft
### DSCP marking rules ###
chain ingress {
type filter hook ingress device eth1 priority -500; policy accept;
iif eth1 counter ip dscp set cs0 comment "Wash all ISP DSCP marks to CS0 (IPv4)"
iif eth1 counter ip6 dscp set cs0 comment "Wash all ISP DSCP marks to CS0 (IPv6)"
}
DSCP
This creates an ingress chain and 2 rules to set all incoming traffic on “eth1” to cs0. Change “eth1” to your wan interface.
After you pasted the commands you should reload your firewall:
fw4 reload
If it's not working as intended you can delete the file with:
rm /etc/nftables.d/05-rules-ingress.nft
then restart your firewall:
service firewall restart
Even the netdev table ingress hook happens after tc ingress.
But as far as I can see dscpclassify will also copy CS0 from the conntrack table, so for connections initiated from your own network conntrack should do the right thing, no?
Your ISP is sending quantitative traffic with CS6/CS7, odd?
Well, I would try a tc filter to clean these up then?
Thanks for the clarification and the link. This level of detail is missing in the nftables wiki.
Looks like the ip4 traffic is set to cs0 but for ip6 i get this for example:
IP6 (class 0x80, hlim 62, next-header UDP (17) payload length: 30)
IP6 (flowlabel 0x6c28f, hlim 63, next-header UDP (17) payload length: 39)
As @dave14305 stated: the ingress hook happens after tc ingress and not as I thought before. With my rules your isp DSCP will be set to cs0 but only after cake. So, if you look at:
tc -s qdisc
your isp DSCP’s will probably still land in your higher priority tins…
Is this IPv6 traffic originating from your network?
No but 0x40 and 0x80 and thousands of ip6 flowlabels.
I´m not sure what you mean. If i mark something with cs5 and tcpdump my pppoe-wan i can´t see marks in both direction only egress but if i tcpdump br-lan i can see the cs5 mark on both directions.
How do i do that correctly ?