Qosify: new package for DSCP marking + cake

Installing and Using OpenWrt Network and Wireless Configuration
1101

I took that file from a previous backup made at 8pm.


config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'
	option ipv6 '0'

config interface 'lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.50.1'
	option delegate '0'
	option device 'br-lan.50'

config interface 'wan'
	option proto 'dhcp'
	option delegate '0'
	option device 'bridge-wan.50'

config device
	option type 'bridge'
	option name 'bridge-wan'
	list ports 'wan'
	option bridge_empty '1'
	option ipv6 '0'

config bridge-vlan
	option device 'bridge-wan'
	list ports 'wan:u*'
	option vlan '50'

config bridge-vlan
	option device 'bridge-wan'
	option vlan '10'
	list ports 'wan:t'

config bridge-vlan
	option device 'bridge-wan'
	option vlan '30'
	list ports 'wan:t'

config bridge-vlan
	option device 'bridge-wan'
	option vlan '40'
	list ports 'wan:t'

config bridge-vlan
	option device 'bridge-wan'
	option vlan '100'
	list ports 'wan:t'
	option local '0'

config bridge-vlan
	option device 'bridge-wan'
	option vlan '200'
	list ports 'wan:t'
	option local '0'

config interface 'IoT'
	option proto 'static'
	option ipaddr '172.16.0.1'
	option netmask '255.255.255.0'
	option delegate '0'
	option device 'br-lan.30'

config interface 'Guest'
	option proto 'static'
	option ipaddr '10.20.30.40'
	option netmask '255.255.255.0'
	option device 'br-lan.40'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'

config bridge-vlan
	option device 'br-lan'
	option vlan '30'
	list ports 'lan3:u*'

config bridge-vlan
	option device 'br-lan'
	option vlan '40'
	list ports 'lan2:u*'

config bridge-vlan
	option device 'br-lan'
	option vlan '100'
	option local '0'

config bridge-vlan
	option device 'br-lan'
	option vlan '200'
	option local '0'

config bridge-vlan
	option device 'br-lan'
	option vlan '50'
	list ports 'lan1:u*'

config device
	option name 'bridge-wan.50'
	option type '8021q'
	option ifname 'bridge-wan'
	option vid '50'
	option ipv6 '0'

config device
	option name 'br-lan.50'
	option type '8021q'
	option ifname 'br-lan'
	option vid '50'
	option ipv6 '0'

config device
	option name 'br-lan.10'
	option type '8021q'
	option ifname 'br-lan'
	option vid '10'
	option ipv6 '0'

config device
	option name 'br-lan.200'
	option type '8021q'
	option ifname 'br-lan'
	option vid '200'
	option ipv6 '0'

config device
	option name 'br-lan.100'
	option type '8021q'
	option ifname 'br-lan'
	option vid '100'
	option ipv6 '0'

config device
	option name 'bridge-wan.10'
	option type '8021q'
	option ifname 'bridge-wan'
	option vid '10'
	option ipv6 '0'

config device
	option name 'bridge-wan.100'
	option type '8021q'
	option ifname 'bridge-wan'
	option vid '100'
	option ipv6 '0'

config device
	option name 'bridge-wan.200'
	option type '8021q'
	option ifname 'bridge-wan'
	option vid '200'
	option ipv6 '0'

config device
	option name 'bridge-wan.30'
	option type '8021q'
	option ifname 'bridge-wan'
	option vid '30'
	option ipv6 '0'

config device
	option name 'bridge-wan.40'
	option type '8021q'
	option ifname 'bridge-wan'
	option vid '40'
	option ipv6 '0'
1102

This seems to have a lot of duplicated sections, which should not be a problem, but it makes reading this harder than necessary.
Also you have quite a lot of rules (although I assume the game rules rarely trigger concurrently)... I always recommend to leave everything at the default forwarding (DF/CS0) and only selectively prioritize things up or down if you have indications that this will improve the quality of experience. Say, if you believe steam downloads from other users affect your browsing/whatever, put the steam downloads into the bulk class and conform that this actually improved the quality of experience; similar for raising the priority level for a given game, if testing shows to improve the experience keep the rue, otherwise comment it out.
At the core of cake sits an flow-fair scheduler that will both make sure no flow gets starved (and each active flow will only get an equitable share of the available capacity) and that sparse flows will get a mild boost (which typically does the right thing for smallish sparse traffic types like DNS/NTP/ACK). So with cake or fq_codel one typically can get away with a much less refined QoS priority hierarchy than in traditional QoS set-ups. IMHO simplicity is advantageous here, much simpler to understand and verify a QoS hierarchy that only changes priorities of few applications/packets.
Also, the way prioritization works is that for every packet that gets treated preferentially other packets will need to be treated worse (than they would have in a non-prioritization set-up). This in essence means you need to make sure that under congested conditions not all of your packets try to gain higher priorities...

1 Like
1103

I've been experimenting with the bufferbloat project for a long while now.
Glad to see the community get to where we are today. Much love to all.

After years of research dev and put to practice from 56K - DOCSIS - FIFO.
I've heavily modified about everything I could learn so far. (many many years)

Currently running all of this on a custom x86_64 SWITCH with 8GB Ram.
Stuff like this is not simple but with a will to understand trial and error...

I've always been interested in networking since I was a young age.
I'd like to consider my self adv when it comes to networking, but still.
That doesn't mean much imo, even as much time I've invested etc...

As well as much learning I came to acquire over the years I digress.
Prob wrong forum to ask, but maybe a GURU or a time traveler will see this.

This is a current result of several modifications I've applied over time.

Now I know some people might think this is a good test result, but I for one don't ... call me crazy.
Tho this is one of the best results on my end I've had displayed, To me it's still not good enough.
So who knows maybe the Time Travlers or GURUs here that see this is... Are more highly evolved.
I'm asking....
What kind of wizard trickery can I apply to in fact lower my deviation/jitter/latency more than it is already.

:vulcan_salute:

1104

what's your expectation? what number you'd like to see?

1 Like
1105

What is your gripe? Keep in mind that modern browsers are no high precision measuring devices so any browser based test will have some additional browser-caused delay components that are not reflective of what happens on the network.
You could try to get a packet capture while running that test and see what true delay the latency probe packets carry on the wire. Or you could switch to a different measurement system, like netperf, iperf2, iperf3 (all of which pose the challenge of finding a sufficiently beefy server on the internet).

1106

Using neutrino particles for networking would be a good next step. The big problem there is in shrinking the sizes of the emitters and receivers.

2 Likes
1107

Thank you. I'll do that in order to see what is causing me that trouble.

EDIT: after following your recommendations, now the latency was reduced a lot of.
So now, my config is almost the factory, just add some rules for YouTube and Netflix.
This weekend I'll be testing more things.

This is my current latency right now.

Thank you.

1108

It's obvious now with hindsight - @dtaht is both a time traveller and a guru.

Networking must be very complicated for time travellers. Negative round trip times? I send a packet and it is received in the past? Hmm.

1109

I highly recommend both "Time Travelers, Strictly Cash", and "-All you Zombies" if you haven't read them. The latter became a movie short, recently, I'm told. It's been really hard to work with y'all within the constraints of this timestream continuum with all the tricks to coping too well documented.

...

My neutrino joke has a history. Back in 1999 when things were frothy, I put together a 4 page funding proposal for some VCs I knew that I distributed on march 31st. It described how amazing we could make the internet if we could get past the pesky line of sight problems conventional wireless technology had - and described in vague but exciting! terms alternate waveforms and particles we could use to get there... and even through the center of the earth (8k km) rather than around! and on the third page we got to the major research problem ahead which was shrinking the size of the emitter (the sun), and the reciever (our top secret research lab under a mountain in california) down to cellphone size.

-- "very funny, Taht", came back one postit.

Anyway, I lost that darn document. I swear it was convincing as hell til it got to that third page...

...

Also, are you aware of the dangers of DiHydrogen Monoxide? I used to date someone who was there when that public education campaign got kicked off. She still has the mugs.

1110

can you show your current configuration?

1111

Of course, I can.

qosify

config defaults
	list defaults /etc/qosify/*.conf
	#option dscp_prio video
	option dscp_icmp +besteffort
	#option dscp_default_udp besteffort
	option prio_max_avg_pkt_len 600
	option dscp_default_tcp paquete_sin_nada
	option dscp_default_udp paquete_sin_nada

config	class paquete_sin_nada
	option ingress CS1
	option egress CS1
	option prio_max_avg_pkt_len 1270
	option dscp_prio CS4
	option bulk_trigger_pps 600
	option bulk_trigger_timeout 10
	option dscp_bulk CS1


config class navegando
	option ingress CS0
	option egress CS0
	option prio_max_avg_pkt_len 575
	option dscp_prio AF41
	option bulk_trigger_pps 1000
	option bulk_trigger_timeout 10
	option dscp_bulk CS1


config class besteffort
	option ingress CS0
	option egress CS0

config class bulk
	option ingress LE
	option egress LE

config class video
	option ingress AF41
	option egress AF41


config class voice
	option ingress CS6
	option egress CS6
	option bulk_trigger_pps 100
	option bulk_trigger_timeout 5
	option dscp_bulk CS0

config class brugales
	option ingress CS4
	option egress CS3

config interface wan
	option name wan
	option disabled 0
	option bandwidth_up 20mbit
	option bandwidth_down 200mbit
	option overhead_type docsis
	# defaults:
	option ingress 1
	option egress 1
	option mode diffserv4
	option nat 1
	option host_isolate 1
	option autorate_ingress 0
	option ingress_options ""
	option egress_options "wash"
	option options "ether-vlan"

config device wandev
	option disabled 1
	option name wan
	option bandwidth 200mbit

00-defaults

# DNS
tcp:53		voice
tcp:5353	voice
udp:53		voice
udp:5353	voice
tcp:853		voice
udp:853		voice

# NTP
udp:123		voice

# SSH
tcp:44225		+video

# HTTP/QUIC
#tcp:80		+besteffort
#tcp:443		+besteffort
#udp:80		+besteffort
#udp:443		+besteffort

# HTTP/QUIC
tcp:80			navegando
tcp:443			navegando
udp:80			navegando
udp:443			navegando

# GAMING
# Retroarch
tcp:55435-55438		brugales
udp:55435-55438		brugales

# YouTube
dns:*googlevideo*	video

# Netflix
dns:*nflxvideo*		video
1 Like
1112

Hello I have a question if I have it setup right. My connection is 1000/50Mbit Docsis 3.1 Cable. I want to get the best latency for my games.

config defaults
	list defaults /etc/qosify/*.conf
	option dscp_prio tin6
	option dscp_icmp tin7
	option dscp_bulk tin0
	option dscp_default_tcp tin1
	option dscp_default_udp	tin2
	option bulk_trigger_timeout 5
	option bulk_trigger_pps	100
	option prio_max_avg_pkt_len 500

config alias tin0
	option ingress LE
	option egress LE

config alias tin1
	option ingress CS1
	option egress CS1

config alias tin2
	option ingress CS0
	option egress CS0

config alias tin3
	option ingress CS3
	option egress CS3

config alias tin4
	option ingress AF21
	option egress AF21

config alias tin5
	option ingress CS2
	option egress CS2

config alias tin6
	option ingress CS5
	option egress CS5

config alias tin7
	option ingress CS6
	option egress CS6

config interface wan
        option name wan
        option disabled 0
        option bandwidth_up 42mbit
        option bandwidth_down 600mbit
        option overhead_type docsis
        # defaults:
        option ingress 1
        option egress 1
        option mode diffserv8
        option nat 1
        option host_isolate 1
        option autorate_ingress 0
        option ingress_options ""
        option egress_options "ack-filter"
        option options ""

config device wandev
	option disabled 1
	option name wan
	option bandwidth 100mbit


# GAMES
udp:30095		tin7
udp:30103		tin7

# HTTP
tcp:80		+tin3
tcp:443		+tin3
udp:80		+tin3
udp:443		+tin3

Also is this working right? Example is shatterline

wireshark
wireshark1306×41 3.13 KB

1113

I get worried about people using the cs based priority feature too much. It was obsoleted by many RFCs long, long ago, and I'd prefer people try to fit things into the diffserv3 and diffserv4 category modes so more otherwise unrecognized but correctly marked traffic from elsewhere landed there.

2 Likes
1114

If I could let myself dream a little, cake diffserv4's treatments would become a globally accepted default standard.

2 Likes
1115

Well if one uses the diffserv8 mode than the 8 CSs are convenient short hands to assign packets to the eight queues (ignoring that CS0 is in tin2). Personally I think that for normal links 8 tiers are probably too much, but not excessively so (64 tiers would clearly be excessive), but then I rarely use prioritization myself, so I could do with a diffserv2 mode... Kidding, I think diffserv3 and diffserv4 make some sense (diffserv4 in that it offers are relatively high throughput higher priority class, which for bursty traffic can be helpful*).

However cake tries to map existing "named" DSCPs into roughly appropriate tins, which holds for EF, VA, but also for the CSNs and the AFNNs.

As I said cake tries to fold in a few IETF described PHBs**, so is actually not far away from accepted behaviour. That said, this only holds if you actually trust traffic to carry "correct" DSCPs in the first place, which for ingress traffic I do not.
My personal preference would to re-dedicate lowish 3 bits of IPv6 interface identifiers (IID) to denote my intended priority tin (selected by the application running in my network) and then simply use that for priority tin sorting, DSCPs are essentially spoiled beyond recovery.
I feel that my proposal probably violates IPv6 orthodoxy, but I do not care much, as I consider the current IID rules as an effective way to preserve the capability to do more interesting things with some of these bits in the future (unlike using fixed values for most of the bits current approaches essentially randomize the bits, so even misguided firewalls will not dare to reject IPv6 packets do to unexpected bit patterns in the lower 64 bit of the addresses).

*) I saw a report here in the forum of as 12 player Valorant (a modern FPS with a 128 Hz tick-rate) session caused somewhere between 12-18 Mbps of averaged ingress traffic, even though this is not capacity seeking this needs a priority tier with sufficient capacity....
**) As well as you can do whith cake's soft-limited prioritization approach.

1116

yes it is really dangerous, everyone should avoid :wink:

1117

Hi @nbd,

I've noticed an unusual phenomenon with the mappings under /sys/fs/bpf/qosify_data/udp_ports. It appears that every 256 ports (64, 320, 576, etc.) are being incorrectly mapped to a different class value. In my case, most udp ports are mapped with 145 (default class 17 for UDP), but these other ports are being mapped to decimal 194 (class 2 which is +video in my config).

I don't understand why, however, since my udp port mappings are minimal.

/etc/qosify/*.conf entries 
# grep -h ^udp /etc/qosify/*.conf
udp:3478-3497 +video
udp:16384-16387 +video
udp:16393-16402 +video
udp:8801-8810 +video
/etc/config/qosify 
 uci export qosify
package qosify

config defaults
        list defaults '/etc/qosify/*.conf'
        option dscp_prio 'video'
        option dscp_icmp '+besteffort'
        option prio_max_avg_pkt_len '500'

config class 'besteffort'
        option ingress 'CS0'

config class 'bulk'
        option ingress 'LE'
        option egress 'LE'

config class 'video'
        option ingress 'AF41'
        option egress 'AF41'

config class 'voice'
        option ingress 'CS6'
        option egress 'CS6'
        option bulk_trigger_pps '100'
        option bulk_trigger_timeout '5'
        option dscp_bulk 'AF41'
udp_ports with 194 
root@router:/sys/fs/bpf/qosify_data# grep -v 145$ udp_ports 
# WARNING!! The output is for debug purpose only
# WARNING!! The output format will change
64: 194
320: 194
576: 194
832: 194
2368: 194
2624: 194
2880: 194
3136: 194
3392: 194
3648: 194
3904: 194
4160: 194
4416: 194
4672: 194
24866: 194
25122: 194
25378: 194
25634: 194
25890: 194
26146: 194
26402: 194
26658: 194
26914: 194
27170: 194
38413: 194
38669: 194
38925: 194
39181: 194
39437: 194
39693: 194
39949: 194
40205: 194
40461: 194
40717: 194
40973: 194
41229: 194
41485: 194
41741: 194
41997: 194
42253: 194
42509: 194
42765: 194
43021: 194
43277: 194
class_map 
root@router:/sys/fs/bpf/qosify_data# cat class_map 
# WARNING!! The output is for debug purpose only
# WARNING!! The output format will change
0: {{255,255,0,0,0,},{0,0,},1,}
1: {{255,255,0,0,0,},{1,1,},1,}
2: {{255,255,0,0,0,},{34,34,},1,}
3: {{255,34,5,100,0,},{48,48,},1,}
4: {{0,0,0,0,0,},{0,0,},0,}
5: {{0,0,0,0,0,},{0,0,},0,}
6: {{0,0,0,0,0,},{0,0,},0,}
7: {{0,0,0,0,0,},{0,0,},0,}
8: {{0,0,0,0,0,},{0,0,},0,}
9: {{0,0,0,0,0,},{0,0,},0,}
10: {{0,0,0,0,0,},{0,0,},0,}
11: {{0,0,0,0,0,},{0,0,},0,}
12: {{0,0,0,0,0,},{0,0,},0,}
13: {{0,0,0,0,0,},{0,0,},0,}
14: {{0,0,0,0,0,},{0,0,},0,}
15: {{0,0,0,0,0,},{0,0,},0,}
16: {{0,0,0,0,0,},{0,0,},0,}
17: {{0,0,0,0,0,},{0,0,},0,}

I think there's some magic answer in the fact that the interval between these "wrongly" classified ports is exactly 256, but I haven't figured out the trick.

EDIT: today Dave learned about htons() and network byte order. No problem exists except in my place on the learning curve!

2 Likes
1122

I have been playing around with this today and wanted to share what I have learned...

But first a BIG thank you to @nbd for creating this tool, @elan for sharing his awesome config, @moeller0 for all his advice and to everyone else for sharing their experience.

This thread is awesome.

A few things I have learned...

How to set this up...

  • Install the qosify package
  • Edit /etc/config/qosify
    • Edit the config interface wan section
    • You can delete the config device wandev section
    • You can adjust the DSCP class aliases if you want
  • Edit /etc/qosify/00-defaults.conf
  • Disable other SQM packages
    • For example, the commonly used luci-app-sqm
    • Make sure to also disable any other SQM or QoS scripts you have tried.
    • Make sure to also clear any firewall DSCP prioritization rules the scripts or your may have added. Otherwise you might see unexpected results.
  • Start/Restart qosify with service qosify restart && qosify-status
  • Test if it's working
    • Run qosify-status
    • Generate some traffic
    • Run qosify-status again
    • Look at the bytes column to see in which tin the traffic ends up
    • For more in-depth debugging, sniff the traffic as described below

Hopefully this will help newcomers who were as confused by this as me.

How to verify DSCP markings by sniffing them

There isn't a single interface that has both the download and upload traffic with DSCP marks, but you can work around it by verifying both separately.

Sniffing download / ingress traffic

Simply use wireshark on your local computer's wifi or ethernet interface.

Important: Make sure cake isn't configured to wash ingress traffic, otherwise the DSCP marks will be removed.

Sniffing upload / egress

This is a little more complicated. You could for example run tcpdump on the router directly, but I found the easiest way is to use Wireshark's SSH remote capture: sshdump feature.

Wireshark will connect over SSH to your openwrt router and capture packages from there.

Just set it to use the wan interface on your router.

Important: Again, it's important to make sure you are not washing the egress DSCP markings, otherwise you won't see them.

	option ingress_options "wash" # <-- Make sure `wash` this is NOT set
	option egress_options "wash" # <-- Make sure `wash` this is NOT set

My experience with the dns: option

How it (probably) works

I found this option a little finicky in my testing. Sometimes it worked and sometimes it didn't.

It probably works by subscribing to DNS requests and then applying the DSCP markings to the resolved IP addresses.

So if your computer already resolved and cached the IP address of a hostname before QoSify was started, then QoSify may not "see" the DNS request and won't know that an IP belongs to it.

So, in my testing, destination ports have been more reliable for me. But perhaps it's not a problem if you run the service for a while and are not looking for results instantly.

Forcing it to recognize a domain

You can force it to work by flushing your computer's DNS cache and then querying the DNS server for the host.

On macOS you can do it with:

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
host thedomain.com

Gotcha: The domain in your config has to match the one the application uses

This may seem obvious, but it really tripped me up.

My plex server has two domains. I use my own domain to access other services on the same server, but the plex service was configured to use the domain provided by my hosting provider.

The marking didn't work when I was using my own domain, since plex was resolving the one from my hosting provider.

Another gotcha: CNAME records

Check out [this response by dave14305](Qosify: new package for DSCP marking + cake - #1127 by dave14305).

Regarding your DNS findings, you also need to be aware of the dns_c: syntax used for CNAMEs. Mostly, QoSify is watching A or AAAA responses for matching, but if the domain is actually a CNAME, you might want to try adding a dns_c: entry for the domain.

How to match both the domain and subdomains

QoSify uses fnmatch() for the matching, which allows you to use ? to match ANY ONE character and * to match a sequence of ANY characters.

But if you want to match a domain and all it's subdomains, you need to create two entries like this:

# Boosteroid (Game streaming service like GeForce Now)
dns:cloud.boosteroid.com video_gaming
dns:*.cloud.boosteroid.com video_gaming

Elan's Configs

I am quite impressed by @elan's config.

When it works, it's really amazing... For example...

  • Started a steam download, which immediately used entire bandwidth
  • Started a 60gb 4k bluray remux stream on plex
  • The steam download immediately backed off from around 11mb/s to 2-3mb/s
  • Once plex had buffered enough, the steam download increased again to 4-5mb/s
  • Then I started boosteroid, a game streaming service similar to GoForce Now and Stadia
  • The streaming service worked perfectly and my ping didn't increase

But unfortunately there were some instances where it didn't work and the config actually made things worse.

For example, I did the same experiment, but with GeForce Now. Somehow the port rule didn't work.

But the unmarked_traffic -> bulk did work and so the GeForce stream was put into the same and lowest tin as the steam download... leading to a very bad experience.

I'll play around with it and see how it goes.

7 Likes
1123

This illustrates that 'unmarked -> bulk' is a policy that requires that your rules catch all important traffic. Personally I would probably use a more conservative ''unmarked -> besteffort' policy and only up/down prioritize a few select traffic types if absolutely required instead of having to essentially create a comprehensive rule set that more or less needs to classify all/most traffic to avoid the 'bulk trap'. But that is subjective preferrence not better or worse than the alternative policies.

1 Like
1124

Hi kimsha I'm glad you like this config from gargoyle,

but there is an error in your config in the egress option you have to put wash otherwise your packets won't have clearly assigned dscp :wink:

only option egress_options "wash"

https://www.gargoyle-router.com/phpbb/viewtopic.php?t=11953