Checking DSCP tags

tievolu · April 1, 2020, 2:10pm

I'm using layer cake SQM with diffserv4 and I think I've figured out how to apply appropriate DSCP tags using custom firewall rules. I can see that traffic is getting classified and placed in the four available tins (by looking at the output from tc -s qdisc), so it is definitely working to some degree:

                   Bulk  Best Effort        Video        Voice
  thresh       1343Kbit    21500Kbit    10750Kbit     5375Kbit
  target         13.5ms        5.0ms        5.0ms        5.0ms
  interval      108.5ms      100.0ms      100.0ms      100.0ms
  pk_delay        1.4ms        294us        710us        109us
  av_delay        118us         18us         50us         12us
  sp_delay          2us          3us          4us          2us
  backlog            0b           0b           0b           0b
  pkts          2771621      5374593      4806887      1517536
  bytes      3624738613   2397416212    680142558    252877354
  way_inds        27112       122715       130563         8355
  way_miss        87608        36930        73735         3323
  way_cols            0            0            0            0
  drops             593          816          760            1
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            0            4            0
  bk_flows            0            0            0            0
  un_flows            0            0            0            0
  max_len         36336        40878        24378         1382
  quantum           300          656          328          300

However, I'd like to know for sure that individual connections are being classified correctly. Is there any way to see which connections are going into each tin?

I've recently migrated from Tomato, which has an excellent interface for checking QOS rules - you get a live list of connections which includes the QOS classification for each one. It also tells you which specific rule resulted in that classification. This makes it easy to spot rules that need tweaking. I'm hoping there's a way to achieve something similar in OpenWRT?

mindwolf · April 1, 2020, 2:54pm

Are the DSCP tags being forwarded to the internet or just locally over the LAN?

If they are generated only for local use, you could verify them by performing a packet capture on the switch/router using tcpdump and view the data using wireshark. The WAN part is a bit tricky as you need to make sure the receiving machine allows incoming DSCP tags and that machine would have to do a packet capture on ingress traffic as well. Your ISP and any other firewall over the internet may strip the tags regardless.

tievolu · April 1, 2020, 3:49pm

I'm applying my own DSCP tags on the router itself using firewall rules which match against ports, mac addresses etc. I suspect these are being forwarded to the WAN but I don't expect anyone else to honour them - they're only there to get the connections into the correct tin in my WAN SQM instance.

I was hoping for an simpler method than trawling through Wireshark traces, but if that's what I have to do I'll dive in. I think I've been spoiled by working with Tomato's GUI for many years

ldir · April 1, 2020, 4:20pm

The DSCP lives on the individual packet/s, not on the connection, so there's nothing connection related to track.

If the DSCP were also stored in say the connection tracking connmark then it could be used to infer the DSCP value for that connection, but that's no guarantee that each packet actually had it applied.

If you're feeling 'brave', then openwrt has some specials in the form of iptables connmark -setdscp and (from upstream linux) tc act_ctinfo. They can be used in combination to store DSCPs from packets into connmarks and restore from connmarks into packet DSCPs.

mindwolf · April 1, 2020, 5:16pm

In Wireshark, you can use the DSCP information applied as a column to follow it easier. [wireshark dscp apply as a column](http://wireshark dscp apply as a column)

tievolu · April 2, 2020, 2:11pm

Ok, I think I've come up with a solution. In my custom firewall rules I do the following:

Match packets against various criteria (dport, sport, source mac etc.) and set the connection mark to the decimal value for the appropriate DSCP class
Check all inbound/outbound packets for connection marks which match DSCP class decimal values, and if found, set the appropriate DSCP class on the packet.

This ensures that all packets, both inbound and outbound, are classified correctly for connections marked by the rules in part 1.

I also now have a script which parses /proc/net/nf_conntrack into an easily readable table containing src/dst IPs and ports, bytes in/out, and the mark on the connection. I can run this whenever I like to check whether connections are being classified correctly.

tievolu · April 2, 2020, 2:14pm

Is there any doc anywhere for connmark -setdscp? I couldn't find any...

It sounds like it might allow me to streamline my script.

ldir · April 2, 2020, 4:01pm

Have a look at https://patchwork.ozlabs.org/patch/1207651/

Also as an example of how I use connmark -setdscp in combination with tc act_ctinfo to do DSCP storing/restoration over my wan link https://github.com/ldir-EDB0/sqm-scripts/commit/8d950e9b83653b3009e3ee3476cd6cd875c36df2

tievolu · April 2, 2020, 5:03pm

Here's an example of what I'm doing right now. It seems to work but I'm not sure it's the best way:

# Identify outbound UDP traffic from the Xbox and mark the connection with the value "40" (DSCP CS5)
iptables -t mangle -A FORWARD -o eth0 -p udp -m mac --mac-source [Xbox MAC address] -j CONNMARK --set-mark 40

# Identify inbound or outbound packets on connections with a connmark of "40", and tag the packets with DSCP 40
iptables -t mangle -A FORWARD -i eth0 -m connmark --mark 40 -j DSCP --set-dscp 40
iptables -t mangle -A FORWARD -o eth0 -m connmark --mark 40 -j DSCP --set-dscp 40

Reeves0724 · April 2, 2020, 6:05pm

Wow im working on the same thing more or less lots of fun...

mindwolf · April 2, 2020, 7:01pm

@ldir I've tried 19.07.01 & 19.07.2 and can't get this to work.

ldir · April 2, 2020, 7:26pm

Ahh, I forgot It's in master only, happening some time after the branch to 19.07.

tievolu · April 7, 2020, 2:29pm

Firstly, apologies if I'm covering well-worn ground here. I've only just started digging into this within the last few days.

Looking at my SQM stats, my approach isn't working properly for ingress (interestingly some packets are being classified into different tins, but definitely not all of them). So I switched to the PREROUTING chain for tagging the packets based on the connmark:

# Identify outbound UDP traffic from the Xbox and mark the connection with the value "40" (DSCP CS5)
iptables -t mangle -A FORWARD -o eth0 -p udp -m mac --mac-source [Xbox MAC address] -j CONNMARK --set-mark 40

# Identify inbound or outbound packets on connections with a connmark of "40", and tag the packets with DSCP 40
iptables -t mangle -A PREROUTING -m connmark --mark 40 -j DSCP --set-dscp 40

The packets and connections are definitely tagged correctly, but this still doesn't work for ingress. After some reading on the forum and elsewhere it seems that this is because ingress traffic is subject to SQM before it even goes through the PREROUTING chain. Now I understand that this is the problem @ldir is tackling in his patches

So my current setup looks something like this:

While we wait for @ldir's patches to land in a release (I don't want to get into manually patching), the only solution I've found in the forums is to set up a veth pair between br-lan and eth0, and enable SQM on veth0 instead of eth0, so that iptables gets a look-in first when the packets arrive on eth0:

However, I also have a guest VLAN, so I think I'd have to create two veth pairs and two SQM instances, like this:

This is all starting to look a bit messy! I don't see how things could behave properly with two independent SQM instances, and presumably any traffic from the router itself (e.g. DNS traffic from dnsmasq) would not be subject to shaping under this model, so I wouldn't be able to prioritise it?

Would it not be better to set up a veth pair with veth1 bridged with eth0, enable SQM on veth0 and use veth0 as the WAN interface, like this:

This would allow me to shape br-lan and the guest network with one SQM instance, and it would also catch traffic from the router itself.

So is this approach even possible? Is anyone currently doing anything like this?

tievolu · April 7, 2020, 2:45pm

I will just add that my network seems to be working very well even though SQM isn't classifying packets correctly on ingress. Prioritisation of outbound traffic appears to be sufficient for my household's usage. It's just bugging me that the ingress side isn't working properly

moeller0 · April 7, 2020, 4:57pm

Wellcome to the club After years of accepting the common wisdom that QoS means having to twiddle with arcane IP/port/protocol rules, it is rather striking how well in general a competent AQM like fq_codel or cake works without lots of special cases
I do note that both fq_codel and cake (by default) use a flow-separation and a competent AQM to keep different traffic sources out of each other's way, and both employ a rather simple (at least in retrospect) heuristic of giving flows that have no packets queued a slight priority boost. That for many cases isa sufficient to not wanting to deal anymore with port-port/per-IP rules anymore (Sure, that does not work for everybody, but for quite a lot of users that is simply good enough to not bother).
I tend to recommend cake with a per-internal-IP-fairness configuration (see the Making cake sing and dance, on a tight rope without a safety net (aka advanced features) section for details) to isolate games and streamers from each other

tievolu · April 7, 2020, 5:01pm

After all that I think I've found a solution that works:

So that's three SQM instances. One shaping download traffic for the main LAN, one shaping download for the guest network, and one that shapes upload traffic on the WAN interface and ensures fairness between the main and guest networks.

Does this seem sensible?