How to clear ISP traffic with OpenWrt?

N1K · July 6, 2023, 4:11am

Hey @Lynx , i set up your script and everything felt more responsive so great work but there are still wrong marks on the pppoe-wan site the marking on br-lan works.

03:43:24.251710 IP6 (flowlabel 0xaeed1, hlim 63, next-header UDP (17) payload length: 1238) 2001:a62:1437:6a00:450c:5482:3703:2fe6.53651 > fra24s22-in-x0a.1e100.net.443: [udp sum ok] UDP, length 1230
03:43:24.260754 IP6 (class 0x80, hlim 62, next-header UDP (17) payload length: 1238) fra15s29-in-x03.1e100.net.443 > 2001:a62:1437:6a00:450c:5482:3703:2fe6.59385: [udp sum ok] UDP, length 1230
03:43:24.262903 IP6 (flowlabel 0x47470, hlim 63, next-header UDP (17) payload length: 1238) 2001:a62:1437:6a00:450c:5482:3703:2fe6.59385 > fra15s29-in-x03.1e100.net.443: [udp sum ok] UDP, length 1230
03:43:24.331769 IP6 (class 0x80, hlim 62, next-header UDP (17) payload length: 1234) fra24s22-in-x0a.1e100.net.443 > 2001:a62:1437:6a00:450c:5482:3703:2fe6.53651: [udp sum ok] UDP, length 1226
03:43:24.332379 IP6 (class 0x80, hlim 62, next-header UDP (17) payload length: 1238) fra24s22-in-x0a.1e100.net.443 > 2001:a62:1437:6a00:450c:5482:3703:2fe6.53651: [udp sum ok] UDP, length 1230
03:43:24.332379 IP6 (class 0x80, hlim 62, next-header UDP (17) payload length: 1238) fra24s22-in-x0a.1e100.net.443 > 2001:a62:1437:6a00:450c:5482:3703:2fe6.53651: [udp sum ok] UDP, length 1230
03:43:24.332809 IP6 (flowlabel 0xaeed1, hlim 63, next-header UDP (17) payload length: 43) 2001:a62:1437:6a00:450c:5482:3703:2fe6.53651 > fra24s22-in-x0a.1e100.net.443: [udp sum ok] UDP, length 35
03:43:24.333126 IP6 (class 0x80, hlim 62, next-header UDP (17) payload length: 1238) fra24s22-in-x0a.1e100.net.443 > 2001:a62:1437:6a00:450c:5482:3703:2fe6.53651: [udp sum ok] UDP, length 1230

i set everything to cs0 but there is still traffic in the wrong tins

vi /usr/share/nftables.d/ruleset-post/cake-qos-simple.nft

 # designate packet for cake tin: bulk
        chain dscp_set_bulk {
                ip dscp set cs0
        }

        # designate packet for cake tin: besteffort
        chain dscp_set_besteffort {
                ip dscp set cs0
        }

        # designate packet for cake tin: video
        chain dscp_set_video {
                ip dscp set cs0
        }

        # designate packet for cake tin: voice
        chain dscp_set_voice {
                ip dscp set cs0
        }

        chain store-dscp-in-conntrack {

                ip version 4 ct mark set (@nh,8,8 & 252) >> 2
                ip6 version 6 ct mark set (@nh,0,16 & 4032) >> 6
                ct mark set ct mark or 128
        }

After 1 session:

root@OpenWrt:~# tc -s qdisc show dev pppoe-wan
qdisc cake 1: root refcnt 2 bandwidth 6Mbit diffserv4 triple-isolate nat wash ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 41666862 bytes 227798 pkt (dropped 58, overlimits 19823 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 25856b of 4Mb
 capacity estimate: 6Mbit
 min/max network layer size:           30 /    1492
 min/max overhead-adjusted size:       68 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh        375Kbit        6Mbit        3Mbit     1500Kbit
  target         48.4ms          5ms       6.05ms       12.1ms
  interval        143ms        100ms        101ms        107ms
  pk_delay          0us       4.97ms         33us        244us
  av_delay          0us        602us          2us         25us
  sp_delay          0us         13us          2us         12us
  backlog            0b           0b           0b           0b
  pkts                0        38806           24       189026
  bytes               0      6073169         1824     35599175
  way_inds            0          262            0            0
  way_miss            0         2529           24           93
  way_cols            0            0            0            0
  drops               0            3            0            0
  marks               0            0            0            0
  ack_drop            0           55            0            0
  sp_flows            0            2            1            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0        14920           76         1328
  quantum           300          300          300          300

qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 124818347 bytes 283363 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
root@OpenWrt:~#

root@OpenWrt:~# tc -s qdisc show dev ifb-pppoe-wan
qdisc cake 1: root refcnt 2 bandwidth 22Mbit diffserv4 triple-isolate nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 68
 Sent 123615828 bytes 282635 pkt (dropped 825, overlimits 95324 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 254592b of 4Mb
 capacity estimate: 22Mbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       68 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Video        Voice
  thresh       1375Kbit       22Mbit       11Mbit     5500Kbit
  target         13.2ms          5ms          5ms          5ms
  interval        108ms        100ms        100ms        100ms
  pk_delay         15us        576us         47us        168us
  av_delay          0us        119us          4us         24us
  sp_delay          0us         17us          4us         15us
  backlog            0b           0b           0b           0b
  pkts                5        58078           50       225327
  bytes             260     64171798         2252     60661547
  way_inds            0          216            0            0
  way_miss            3         2828           48           66
  way_cols            0            0            0            0
  drops               0          825            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            0            0
  bk_flows            0            2            0            0
  un_flows            0            0            0            0
  max_len            60         1492           60         1328
  quantum           300          671          335          300

root@OpenWrt:~#

Can i experiment with the cake keywords or does that brake something? For example nonat or delete the ingress keyword or add dual-dsthost/dual-srchost, mpu

vi /etc/init.d/cake-qos-simple

# cake-qos-simple configuration options:

ul_if=pppoe-wan # upload interface
dl_if=""  # download interface override (normally left blank and IFB derived for $ul_if ingress)

cake_ul_rate_Mbps=6  # cake upload rate in Mbit/s
cake_dl_rate_Mbps=22 # cake download rate in Mbit/s

cake_ul_options="diffserv4 triple-isolate nat wash ack-filter noatm overhead 34 mpu 68"
cake_dl_options="diffserv4 triple-isolate nat nowash ingress no-ack-filter noatm overhead 34 mpu 68"

overwrite_ecn_val_ul=0 # overwrite existing ecn bits with decimal value (e.g. 0, 1, 2, 3), else "" to disable
overwrite_ecn_val_dl=0 # overwrite existing ecn bits with decimal value (e.g. 0, 1, 2, 3), else "" to disable

# end of cake-qos-simple configuration options

And what does that do? It looks like my windows marking works without it.

Setting DSCPs in Microsoft Windows Based LAN Clients
If using Microsoft Windows, DSCPs can be set at the application level by creating the registry key 'QoS' (it not present) as in:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\QoS\

And then creating the string "Do not use NLA" inside the QoS key with value "1"

Lynx · July 6, 2023, 10:06am

There are probably a lot of issues to untangle here. With cake-qos-simple setup correctly all the incoming packets should get marked with the markings applied to the associated outgoing packets relating to each connection. That should mean by default cs0 and otherwise anythign else set either in nftables or by LAN clients. To me it makes no sense to replace all those chains with ip dscp set cs0. If you don't want to have nftables mark certain traffic with certian markings then just change:

        map rules_proto_dport {
                type inet_proto . inet_service : verdict
                elements = {
			tcp . 53 : goto dscp_set_voice,  # DNS
                        udp . 53 : goto dscp_set_voice,  # DNS
                        tcp . 853 : goto dscp_set_voice, # DNS-over-TLS
                        udp . 853 : goto dscp_set_voice, # DNS-over-TLS
                        udp . 123 : goto dscp_set_voice  # NTP
                }
        }

To be clear, this means only set those markings for tcp/udp on those ports, and otherwise the markings will be left alone (this should be cs0 or anything set by LAN client).

Then, on download, the markings will get restored based on how they were set on upload packets (whether cs0 or using the rules_proto_dport above or in LAN clients).

So you shouldn't need to change much at all - it should just work by default, and by work I mean the marks set on upload in nftables as above or in your LAN clients will get restored on download. Where no marks were seton upload, cs0, those cs0 markings should then get set on download and overwrite your ISP set marks.

So it doesn't make much sense either to look at cake stats and say: oh hey, look, some marks on download are not cs0. Do you want all marks to be treated as cs0? If so, just use the besteffort mode for cake and forget about markings altogether.

Probably you need to try and take a step back and really make sure you understand what you want and what cake-qos-simple is trying to do and then work out whether and how it may be adapted for your needs.

dave14305 · July 6, 2023, 10:20am

For IPv6, each chain will also need a ip6 dscp set cs0 rule.

moeller0 · July 6, 2023, 11:18am

I thought we had established that nftables runs to late on ingress, so cake will still see and act upon your ISP's DSCPs and they will be re-marked to CS0 too late. If you stick to to using an IFB for ingress you need something else than nftables to wipe the DSCPs, this is why we toyed with tc filter invocations above (which should work, there the question is more, will they work too well, that is will they also re-set those DSCPs that got set based on your egress DSCPs).

Lynx · July 6, 2023, 11:55am

Here is why I think cake-qos-simple should work for @N1K.

These nftables rules apply on upload:

table inet cake-qos-simple {

        chain hook-output {

                type filter hook output priority filter

                # OpenWrt->wan
                oifname wan goto classify-and-store-dscp

        }

        chain hook-forward {

                type filter hook forward priority filter

                # lan->wan
                iifname $IFACE_NAMES goto classify-and-store-dscp

        }

	chain classify-and-store-dscp {

		jump classify-dscp
		jump store-dscp-in-conntrack
	}

And ultimately whether the DSCPs are altered by 'classify-dscp' or not, in respect of all upload packets, their DSCPs should get set to the corresponding conntrack:

	chain store-dscp-in-conntrack {

                ip version 4 ct mark set (@nh,8,8 & 252) >> 2
                ip6 version 6 ct mark set (@nh,0,16 & 4032) >> 6
                ct mark set ct mark or 128
        }

And in respect of download packets, the DSCPs are restored from conntrack in tc, as follows:

tc filter add dev "${ul_if}" parent ffff: protocol all matchall action ctinfo dscp 63 128 mirred egress redirect dev "${dl_if}"

So this way cake sees the modified upload packets at wan egress, and also the modified download packets at wan ingress. And the latter should overwrite anything the ISP sets anyway. So if upload packets bear cs0, then the corresponding download packets should also get set to cs0.

Now I don't understand why @N1K would want to set everything to cs0. If he does this he may as well just use the 'besteffort' keyword. Rather surely @N1K wants to apply his own DSCPs to certain packets, and have the remaining packets default to cs0 and have those restored on download, overwriting whatever the ISP does. So that way all the upload/download packets have the DSCPs that are actually desired.

@dave14305 thanks for the heads up I'll add those lines for the various chains. Is there a shorthand for this by the way:

        # designate packet for cake tin: bulk
        chain dscp_set_bulk {
                ip dscp set cs1
                ip6 dscp set cs1
        }

I mean can ip/ip6 calls be reduced to just one line?

dave14305 · July 6, 2023, 12:08pm

Not in this situation.

N1K · July 6, 2023, 1:51pm

I want every packet to be cs0 for ingress and egress except my marked packets through windows but i also want them marked on ingress thats why i use dscpclassify or your cake-qos-simple so it transfers the mark to ingress and if i would use the besteffort keyword there would´t be priority queuing on ingress or did i understand everything wrong?

Lynx · July 6, 2023, 1:55pm

Well sure that's fine, but then just use cake-qos-simple without your modifications. If setup correctly it should achieve what you want for the reasons I provided above.

N1K · July 6, 2023, 1:57pm

okay i will try ur updated config. Can i change/add the cake keywords ?

Lynx · July 6, 2023, 1:58pm

Yes of course. Those should be set as required. Feel free to switch to the cake-qos-simple thread though, if it's purely about setting up cake-qos-simple.

N1K · July 6, 2023, 2:07pm

so i used your updated config with the default settings and i still get the

14:03:45.117288 IP6 (class 0x80

and 

14:12:30.118930 IP6 (class 0x60

always port 443

if i look at tcpdump -i pppoe-wan -vv while opening youtube

did i do something wrong ?
i also restartet the firewall

Lynx · July 6, 2023, 2:17pm

One has to be careful with tcpdump because it's not clear exactly at what point it is seeing the packets - before or after the remarkings.

What are the upload packets getting set to? Maybe have a look on br-lan to see what the DSCP states of the upload/download packets look like?

N1K · July 6, 2023, 2:22pm

14:19:26.170908 IP (tos 0xa0, ttl 128, id 54314, offset 0, flags [none], proto UDP (17), length 138)
    192.168.1.227.53555 > 155.133.226.69.27034: [udp sum ok] UDP, length 110
14:19:26.183243 IP (tos 0xa0, ttl 58, id 62862, offset 0, flags [DF], proto UDP (17), length 967)
    155.133.226.69.27034 > 192.168.1.227.53555: [udp sum ok] UDP, length 939

this is my marked traffic that works.

and this is port 443 br-lan

14:21:18.766118 IP6 (hlim 60

14:21:19.194870 IP (tos 0x0

Lynx · July 6, 2023, 2:23pm

Output from: tc -s filter show dev ppoe-wan ingress?

N1K · July 6, 2023, 2:24pm

root@OpenWrt:~# tc -s filter show dev pppoe-wan ingress
filter parent ffff: protocol all pref 49152 matchall chain 0
filter parent ffff: protocol all pref 49152 matchall chain 0 handle 0x1
  not_in_hw (rule hit 24297)
        action order 1: ctinfo zone 0 pipe
         index 1 ref 1 bind 1 dscp 0x0000003f 0x00000080 installed 1319 sec used 1 sec firstused 1316 sec DSCP set 12908 error 0 CPMARK set 0
        Action statistics:
        Sent 19960621 bytes 24297 pkt (dropped 0, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0

        action order 2: mirred (Egress Redirect to device ifb-pppoe-wan) stolen
        index 1 ref 1 bind 1 installed 1319 sec used 1 sec firstused 1316 sec
        Action statistics:
        Sent 19960621 bytes 24297 pkt (dropped 0, overlimits 0 requeues 0)
        backlog 0b 0p requeues 0

root@OpenWrt:~#

Lynx · July 6, 2023, 2:25pm

That looks OK.

I'm not sure what this is meant to show.

N1K · July 6, 2023, 2:27pm

on the br-lan site there is no class 0x80 or class 0x60 for the ip6 port 443 packets
and for the ip4 port 443 packets the dscp is 0x0

Lynx · July 6, 2023, 2:29pm

Are you trying to say that IPv6 packets go out as cs0 but come back with something different?

N1K · July 6, 2023, 2:30pm

looks like that to me

Lynx · July 6, 2023, 2:31pm

I'm not sure how best to help you here. Maybe @dave14305 has some ideas? You can view a file to see the linux connracks and marks associated with those. And you can add counters / debugs ot the nftables lines to look at the marks that get set. And check what cake sees using service cake-qos-simple download/upload. I guess you need to have a play around and figure out what's going on. It's not easy for me to work out what to suggest exactly.

Also @N1K don't forget to tcpdump on ifb-ppoe-wan to see what is set there.