Qosify: new package for DSCP marking + cake

francisuk1989 · 2022-02-23T02:24:15.162Z

dave14305:

Router to LAN traffic doesn’t get modified by Qosify since the BPF filters are on the WAN interface only.

Learn something new, Thanks

For me, This seems better for when things are "Downloading from the router" as is saying from MYPC port 57666 > GoogleIP port 993


tcpdump -i br-lan -v -n 'src net 192.168.1.0/24 and ( ip and ip[1] & 0xfc > 0 ) and ! port 22'

@dave14305
possible Multicast that's going into tos 0xc0 = CS6 ? Even though is not into the 0-defaults.conf

02:44:29.235719 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
HuaweiY6.lan > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 224.0.0.251 to_in, 0 source(s)]
02:44:29.298432 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
HuaweiY6.lan > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 224.0.0.251 to_in, 0 source(s)]

dave14305 · 2022-02-23T03:26:09.910Z

francisuk1989:

possible Multicast that's going into tos 0xc0 = CS6 ? Even though is not into the 0-defaults.conf

02:44:29.235719 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
HuaweiY6.lan > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 224.0.0.251 to_in, 0 source(s)]
02:44:29.298432 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
HuaweiY6.lan > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 224.0.0.251 to_in, 0 source(s)]

This is captured before it is processed by qosify. Since it’s igmp, I think it gets bypassed since it is not icmp, tcp or udp; but I’m not sure how to interpret the code where key=0 and what value gets returned.

https://lxr.openwrt.org/source/qosify/qosify-bpf.c#L273

ilshatms · 2022-02-23T07:19:31.505Z

francisuk1989:

What device do you run for qosify? @ilshatms

belkin rt3200

francisuk1989 · 2022-02-23T08:23:41.437Z

Found two way of getting around this (for now) BWARE if your ISP uses IPTV, This will affect you!

1 iptables
iptables -I FORWARD -o br-lan -s 224.0.0.0/4 -j DROP

2 Luci
Login into GUI 192.168.1.1 > Network > Devices > br-lan > configure > Advanced device options > Enable multicast support UNTICKED and save, save and apply

3 SSH /etc/config/network
Add below the config file.

# WAN
config device
	option name 'eth0.2'
	option ipv6 '0'
	option multicast '0'

# 2.4 GHz Wireless
config device
	option name 'wlan0'
	option multicast '0'

# 5 GHz Wireless
config device
	option name 'wlan1'
	option multicast '0'

and then dont forgot to Reboot the Router.

elan · 2022-02-23T19:08:03.897Z

I wanted to share these Qosify commands or script that I use to run on first boot when I build my custom OpenWrt firmware or to copy and paste at the same time in the SSH client after doing a factory reset because I upgraded the firmware, so that I already have Qosify configured as I recommend in my above post.

You only have to write your bandwidth and the type of connection you have and nothing else:

# Qosify
BANDWIDTH_UP="50mbit"
BANDWIDTH_DOWN="320mbit"
OVERHEAD_TYPE="docsis"
INGRESS_OPTIONS=""
EGRESS_OPTIONS="wash"
INGRESS_EGRESS_OPTIONS="ether-vlan"

######################################
#   JESUS CHRIST IS THE LORD! AMEN   #
######################################

# Qosify
# Clear default config
uci del qosify.@defaults[0].dscp_prio
uci del qosify.@defaults[0].prio_max_avg_pkt_len
uci del qosify.besteffort
uci del qosify.bulk
uci del qosify.video
uci del qosify.voice
# ICMP (ping) and default class for "unmarked traffic"
uci set qosify.@defaults[0].dscp_icmp="+besteffort"
uci set qosify.@defaults[0].dscp_default_tcp="unmarked_traffic"
uci set qosify.@defaults[0].dscp_default_udp="unmarked_traffic"
# unmarked_traffic class
uci set qosify.unmarked_traffic=class
uci set qosify.unmarked_traffic.ingress="CS1"
uci set qosify.unmarked_traffic.egress="CS1"
uci set qosify.unmarked_traffic.prio_max_avg_pkt_len="1256"
uci set qosify.unmarked_traffic.dscp_prio="CS4"
uci set qosify.unmarked_traffic.bulk_trigger_pps="250"
uci set qosify.unmarked_traffic.bulk_trigger_timeout="10"
uci set qosify.unmarked_traffic.dscp_bulk="CS1"
# browsing class
uci set qosify.browsing=class
uci set qosify.browsing.ingress="CS0"
uci set qosify.browsing.egress="CS0"
uci set qosify.browsing.prio_max_avg_pkt_len="575"
uci set qosify.browsing.dscp_prio="AF41"
uci set qosify.browsing.bulk_trigger_pps="1000"
uci set qosify.browsing.bulk_trigger_timeout="10"
uci set qosify.browsing.dscp_bulk="CS1"
# bulk class
uci set qosify.bulk=class
uci set qosify.bulk.ingress="CS1"
uci set qosify.bulk.egress="CS1"
# besteffort class
uci set qosify.besteffort=class
uci set qosify.besteffort.ingress="CS0"
uci set qosify.besteffort.egress="CS0"
# network_services class
uci set qosify.network_services=class
uci set qosify.network_services.ingress="CS2"
uci set qosify.network_services.egress="CS2"
# broadcast_video class
uci set qosify.broadcast_video=class
uci set qosify.broadcast_video.ingress="CS3"
uci set qosify.broadcast_video.egress="CS3"
# gaming class
uci set qosify.gaming=class
uci set qosify.gaming.ingress="CS4"
uci set qosify.gaming.egress="CS4"
# multimedia_conferencing class
uci set qosify.multimedia_conferencing=class
uci set qosify.multimedia_conferencing.ingress="AF42"
uci set qosify.multimedia_conferencing.egress="AF42"
uci set qosify.multimedia_conferencing.prio_max_avg_pkt_len="575"
uci set qosify.multimedia_conferencing.dscp_prio="AF41"
# telephony class
uci set qosify.telephony=class
uci set qosify.telephony.ingress="EF"
uci set qosify.telephony.egress="EF"
# Reorder classes
uci reorder qosify.unmarked_traffic=1
uci reorder qosify.browsing=2
uci reorder qosify.bulk=3
uci reorder qosify.besteffort=4
uci reorder qosify.network_services=5
uci reorder qosify.broadcast_video=6
uci reorder qosify.gaming=7
uci reorder qosify.multimedia_conferencing=8
uci reorder qosify.telephony=9
# CAKE
uci set qosify.wan.name="wan"
uci set qosify.wan.disabled="0"
uci set qosify.wan.bandwidth_up="$BANDWIDTH_UP"
uci set qosify.wan.bandwidth_down="$BANDWIDTH_DOWN"
uci set qosify.wan.overhead_type="$OVERHEAD_TYPE"
uci set qosify.wan.autorate_ingress="0"
uci set qosify.wan.ingress_options="$INGRESS_OPTIONS"
uci set qosify.wan.egress_options="$EGRESS_OPTIONS"
uci set qosify.wan.options="$INGRESS_EGRESS_OPTIONS"
# Rules
cat << "EOF" > /etc/qosify/00-defaults.conf
# SSH
tcp:22    network_services

# NTP
udp:123   network_services

# DNS
tcp:53    network_services
tcp:5353  network_services
udp:53    network_services
udp:5353  network_services

# DNS over TLS (DoT)
tcp:853   multimedia_conferencing
udp:853   multimedia_conferencing

# HTTP/HTTPS/QUIC
tcp:80    browsing
tcp:443   browsing
udp:80    browsing
udp:443   browsing

# Microsoft (Download)
dns:*1drv*                 bulk
dns:*backblaze*            bulk
dns:*backblazeb2*          bulk
dns:*ms-acdc.office*       bulk
dns:*onedrive*             bulk
dns:*sharepoint*           bulk
dns:*update.microsoft*     bulk
dns:*windowsupdate*        bulk

# MEGA (Download)
dns:*mega*                 bulk

# Dropbox (Download)
dns:*dropboxusercontent*   bulk

# Google (Download)
dns:*drive.google*         bulk
dns:*googleusercontent*    bulk

# Steam (Download)
dns:*steamcontent*         bulk

# Epic Games (Download)
dns:*download.epicgames*   bulk
dns:*download2.epicgames*  bulk
dns:*download3.epicgames*  bulk
dns:*download4.epicgames*  bulk
dns:*epicgames-download1*  bulk

# YouTube
dns:*googlevideo*   besteffort

# Facebook
dns:*fbcdn*         besteffort

# Twitch
dns:*ttvnw*         besteffort

# TikTok
dns:*tiktok*        besteffort

# Netflix
dns:*nflxvideo*     besteffort

# Amazon Prime Video
dns:*aiv-cdn*       besteffort
dns:*aiv-delivery*  besteffort
dns:*pv-cdn*        besteffort

# Disney Plus
dns:*disney*        besteffort
dns:*dssott*        besteffort

# HBO
dns:*hbo*           besteffort
dns:*hbomaxcdn*     besteffort

# BitTorrent
tcp:6881-7000    bulk
tcp:51413        bulk
udp:6771         bulk
udp:6881-7000    bulk
udp:51413        bulk

# Usenet
tcp:119          bulk
tcp:563          bulk

# Live Streaming to YouTube Live, Twitch, Vimeo and LinkedIn Live
tcp:1935-1936    broadcast_video
tcp:2396         broadcast_video
tcp:2935         broadcast_video

# Xbox
tcp:3074         gaming
udp:88           gaming
#udp:500         gaming # UDP port already used in "VoWiFi" rules
udp:3074         gaming
udp:3544         gaming
#udp:4500        gaming # UDP port already used in "VoWiFi" rules

# PlayStation
tcp:3478-3480    gaming
#udp:3478-3479   gaming # UDP ports already used in "Zoom" rules

# Call of Duty
#tcp:3074        gaming # TCP port already used in "Xbox" rules
tcp:3075-3076    gaming
#udp:3074        gaming # UDP port already used in "Xbox" rules
udp:3075-3079    gaming
udp:3658         gaming

# FIFA
tcp:3659         gaming
udp:3659         gaming

# Minecraft
tcp:25565        gaming
udp:19132-19133  gaming
udp:25565        gaming

# Supercell Games
tcp:9339         gaming
udp:9339         gaming

# Zoom, Microsoft Teams, Skype and FaceTime (they use these same ports)
udp:3478-3497    multimedia_conferencing

# Zoom
dns:*zoom*       multimedia_conferencing
tcp:8801-8802    multimedia_conferencing
udp:8801-8810    multimedia_conferencing

# Skype
dns:*skype*      multimedia_conferencing

# FaceTime
udp:16384-16387  multimedia_conferencing
udp:16393-16402  multimedia_conferencing

# GoToMeeting
udp:1853         multimedia_conferencing
udp:8200         multimedia_conferencing

# Webex Meeting
tcp:5004         multimedia_conferencing
udp:9000         multimedia_conferencing

# Jitsi Meet
tcp:5349         multimedia_conferencing
udp:10000        multimedia_conferencing

# Google Meet
udp:19302-19309  multimedia_conferencing

# TeamViewer
tcp:5938         multimedia_conferencing
udp:5938         multimedia_conferencing

# Voice over Internet Protocol (VoIP)
tcp:5060-5061    telephony
udp:5060-5061    telephony

# Voice over WiFi or WiFi Calling (VoWiFi)
udp:500          telephony
udp:4500         telephony
EOF

# Saving modified values
uci commit qosify
/etc/init.d/qosify restart

Information about my Qosify configuration:

Qosify: new package for DSCP marking + cake Network and Wireless Configuration

I found this information about packet length size to prioritize non-bulk unmarked traffic like games, VoIP, etc. and added it to my config and wanted to share it with you. My QoS rules for gaming, browsing/streaming, torrents and bufferbloat - Gargoyle Forum Clarification If you have a bad or unstable connection even using CAKE (because your ISP is bad), you will still have a bad connection even if you use DSCP marking. The DSCP marking doesn't help you FIX the bufferbloat, it's CAKE that …

elan · 2022-02-23T19:23:24.033Z

These keywords in CAKE have been provided to represent a number of common link technologies:

Write the keyword below
***********************
"conservative"     Failsafe     (overhead 48 atm)
"ethernet"         Ethernet     (overhead 38 mpu 84 noatm)
"docsis"           Cable Modem  (overhead 18 mpu 64 noatm)
"pppoe-ptm"        VDSL2        (overhead 30 ptm)
"bridged-ptm"      VDSL2        (overhead 22 ptm)
"pppoa-vcmux"      ADSL         (overhead 10 atm)
"pppoa-llc"        ADSL         (overhead 14 atm)
"pppoe-vcmux"      ADSL         (overhead 32 atm)
"pppoe-llcsnap"    ADSL         (overhead 40 atm)
"bridged-vcmux"    ADSL         (overhead 24 atm)
"bridged-llcsnap"  ADSL         (overhead 32 atm)
"ipoa-vcmux"       ADSL         (overhead 8  atm)
"ipoa-llcsnap"     ADSL         (overhead 16 atm)

# If you are unsure, then write "conservative" as a general safe value.
# "raw" keyword turns off all overhead compensation in CAKE.

"ether-vlan" keyword information:

"ether-vlan" adds '4 bytes' to the overhead.

- This keyword is not for standalone use, but act as a modifier to some previous presets or keywords.
- In addition to those previous presets or keywords it is common to have VLAN tags (4 extra bytes) or PPPoE encapsulation (8 extra bytes).
# This keyword "ether-vlan" may be repeated as necessary.

For true ATM links (ADSL), one often can measure the real per-packet overhead empirically, see for further information how to do that:

https://github.com/moeller0/ATM_overhead_detector

grrr2 · 2022-02-23T20:46:08.344Z

dave14305:

Router to LAN traffic doesn’t get modified by Qosify since the BPF filters are on the WAN interface only.

qosify allows multiple interface configuration, do you mean it is not working for other than wan interface as it is hardwired somehow?

dave14305 · 2022-02-23T21:19:26.702Z

I suppose I should add “by default”.

grrr2 · 2022-02-23T22:26:26.678Z

no worries, i thought you know something which under the hood silently ignores configuration.

now, i have a question: if multiple interface configuration is supported it looks there is no way tp specify config class and <port, ip, dns> rules per interface. is this correct? e.g. /etc/qosify/*.conf files rules are applied to all traffic on all configured interfaces?

francisuk1989 · 2022-02-25T01:18:18.040Z

The only way this would work, if you have multiples of internet connections (WANs) like an Cable or DSL line and then you add the "docisiscable" info from the interface you added by adding an VLAN and Interface name like so . Just remember. i wont recommend multiples of interfaces on 3G/4G5G type of connections, as they can differ in speeds the more users are using the phone masts.

Note; Cable/DOCSIS users that are going to use any type of QoS!!

most of the issues that you may experience with DOCSIS/Cable are causes by the shared access infrastructure. If youre in an overloaded CMTS cluster there's not much you can do. Of course you can have a small cable cluster that let's you always hit your full bandwidth no problem. Then you're lucky, but that won't happen often. And That's the nature of it, DOCSIS/Cable was never was designed to transmit individual users internet data.

Neither was phone wire, but with VDSL you only have shared core network, which rarely is a problem. With your dedicated twisted pair you are better off when it comes to stability and reliability of the internet access

# 63/19 plan
config interface wan
	option name wan
```	option bandwidth_down 60mbit
	option bandwidth_up 17mbit
	option overhead_type pppoe-ptm

# 50/15 plan
config interface docisiscable
	option name docsiscable
`	option bandwidth_down 48mbit
	option bandwidth_up 13mbit
	option overhead_type docsis
``

elan · 2022-02-25T02:46:49.502Z

Use Lag Report to know which ISP offers the best connection and the lowest ping where you live:

Lag Report

WHAT IS THE LAG REPORT?

moeller0 · 2022-02-25T09:01:39.931Z

francisuk1989:

DOCSIS/Cable was never was designed to transmit individual users internet data.

Funny, here I thought that DOCSIS (Data Over Cable Service Interface Specification) was specifically designed for that purpose... You can even configure a segment such that all users have a guaranteed access rate... But it turns out that this rate is rather low, and most users' access links are idling most of the time anyhow. That means that provisioning each user more than her/his guaranteed share will result in users being better of most of the time. It now becomes the challenge for the ISP to make sure that this oversubscription of the link does not result in noticeable effects for the users, if the ISP manages to do that, over subscription becomes a valid and efficient way to share a scarce resource.

francisuk1989:

Neither was phone wire, but with VDSL you only have shared core network, which rarely is a problem. With your dedicated twisted pair you are better off when it comes to stability and reliability of the internet access

In most cases the uplink of the DSLAM/MSAN is a similar bottleneck as the aggregate segment-rate is in DOCSIS. Many DSLAM with (low) hundreds of users with 100 Mbps plans are still uplinked with single or double 1Gbps connections, giving similar oversubscription as in DOCSIS networks. However it should be considerably simpler/cheaper to exchange a 1Gbps fiber uplink with e.g. a 10 Gbps simply by exchanging the "optics" at both sides of the link, compared to DOCSIS where either segment bandwidth increases or segment splits are required both of which are more involved than replacing two optic modules. (Sure replacing optics only works if the devices the optics attach to already support higher rates, but for halfway modern DSL gear that appears to be the case).

grrr2 · 2022-02-25T12:45:42.728Z

let me explain what i meant:

(note: it will be long and total misunderstanding from my end, if it is stupid please be a noble soul and just ignore or point out why it is not working like this, and not why am stupid. thanks.)

according to link qosify a) is not only supporting wan and b) not only supporting a single interface.
basically qosify does two things:

sitting on top of tc, configuring tc, but tc provides the queuing mechanism (not qosify directly)
there is a rule set system which translates rules into actual traffic marking (with hooks to dns etc.) and a specific logic how for example port rules are working.

let's use this notation, according to #1 qosify can be configured on any ( ! ) interface, it is not limited to wan, so let's use this generalized topology:

[ host0 ] - [ network0 ] --- if0 -[ router w/ qosify on if0 ]- if1 --- [ network1 ] - [ host1 ]
where ifX is interface of router; if0 is the interface qosify applied on
where networkX is a network
where hostX is member of corresponding networkX
where host0 & host1 talking to each other

according to link the specific logic how qosify marks traffic is the following:

* cosify is applied to if0
* rule: tcp:5555 	video

case1: in case of outgoing traffic from host1 via if0 (where qosify is applied) to host0:
host1:<port> -----> host0:5555 such traffic is marked as video
host1:<port> -----> host0:6666 such traffic is not marked as video as there is no rule
host1:5555 -----> host0:6666 such traffic is not marked as video as traffic is matched against destination port

case2: in case of incoming traffic from host0 via if0 (where qosify is applied) to host1:
host1:<port> <----- host0:5555 such traffic is marked as video as in case of ingress traffic source port match is used
host1:<port> <----- host0:6666 such traffic is not marked as video due to no source port match
host1:5555 <----- host0:6666 such traffic is not marked as video due to no source port match

so basically qosify always matching against the remote end.

the problem I see that there are traffic patterns where I only know/control the near end, and know nothing about the remote end prior connection so no rule can be defined. Typically VoIP, torrent is such kind of traffic. For example contrary to the VoIP configs flying around here, VoIP does not work the way some people thinks: defining the standard SIP signalling ports 5060/5061 will not help to have good call quality as the media is traveling in RTP using different ports and logic. And even if they know this and add rule for RTP, it will fail. Because usually RTP using dynamically picked port from a port range and depending on your vendor/setup it can vary by each end point, so you don't know what the other end will use. A typical voip call is a duplex steam meaning there are two streams: one from A to B, and another one from B to A. During call setup you advertise your end, i.e the port you expect the other end to connect to, which will be the destination port in case2 above and not the source port hence qosify will not mark your traffic with voice! Symmetric RTP port usage is very optional, and even so, if you really want direct media with anybody you'll have to capture any port 0:65535 ... which obviously makes no sense/and breaks qosify fine grained rule sets as will overlap with something else for sure and it is not supported. Similar problem if you use random listening port for torrent, you can only define your end and not the peer ports, so the configs flying around are neither correct because they assume all the other peers you meet will only use some handful of predefined ports.
obviously i can mark my traffic outside of qosify and just let qosify configure tc. or ... i might able to use qosify! after all it is not hardwired to only usable with single interface (by default the wan), so let's add a second interface configuration, apply qosify both on if0 and if1 as well ( ! ) on if1 just set for egress direction though. in this way I'd have this topology:

[ host0 ] - [ network0 ] --- if0 -[ router w/ qosify on if0 ]- if1 --- [ network1 ] - [ host1 ]
                                  [ qosify on if1.egress    ]

meaning two sets of tc diffserv queues, each assigned to if0 and if1 respectively 
host1:<port> -----> host0:5555 such traffic is marked as video by if0 queue
host1:<port> -----> host0:6666 such traffic is not marked

host1:<port> <----- host0:5555 such traffic is marked as video as in case of ingress traffic source port match is used
host1:<port> <----- host0:6666 such traffic is not marked as video due to no source port match
host1:5555 <----- host0:<port> such traffic is marked as video by if1 queuing !

this way i could mimic near end based matching!

the problem though (apart it may not work at all, double queuing, cpu impact etc) that due to common rule set even if i mean a large port range only for if1 ingress, it will capture traffic on if0 too. hence it would be nice to extend qosify's syntax to override direction and or specify ip:port not just port imho.

EnfermeraSexy · 2022-02-25T13:16:08.696Z

I'm testing openwrt inside an LXC container in proxmox. Everything seems to work fine but Qosify and it seems that the problem is that it can't write into the /sys/fs/bpf folder. I know it's not qosify fault but does anybody know a way to make it work? Using privileged containers with all permissions seems to not work.

libbpf: failed to mkdir /sys/fs/bpf/qosify_data/config: Permission denied
libbpf: map 'config': failed to auto-pin at '/sys/fs/bpf/qosify_data/config': -13
libbpf: map 'config': failed to create: Permission denied(-13)
libbpf: failed to load object '/lib/bpf/qosify-bpf.o'
bpf_object__load: Permission denied

dMopp · 2022-02-25T16:04:12.336Z

All i can tell you is: dont use lxc for openwrt. i tried the same and switched over to kvm

EnfermeraSexy · 2022-02-25T16:19:27.414Z

Thanks but the performance isn't even comparable. I tried SQM and cake and it didn't even use 3% of CPU, while in a KVM it uses near 30%.
For now i will use it in a VM but if i could make it to work in LXC i will use it, or at least try. As i said the only thing that does not work is qosify.

moeller0 · 2022-02-25T16:41:25.499Z

EnfermeraSexy:

Thanks but the performance isn't even comparable. I tried SQM and cake and it didn't even use 3% of CPU, while in a KVM it uses near 30%.

Are you sure that in one condition SQM really took 27%-age points more CPU cycles? Or could it be that in the container case the CPU cycles might simply not have been accounted to the container? Mind you I know little about containers so could be wrong, but the cycles SQM needs to do its work need to show up somewhere (possible that 3% is a realistic number).

EnfermeraSexy · 2022-02-25T16:53:47.829Z

Didn't make so many test but the global CPU usage from the host machine didn't increase that much. I don't know if the CPU usage was made by SQM or not. That CPU usage could be because the VM emulation it self, the drivers from the network (i used virtio)... If i'm not wrong LXC has direct acces to the network you atach so there's less overhead.

dMopp · 2022-02-25T16:55:35.341Z

Did you try to change the CPU Type to „host“? Cake (SQM) is always very CPU hungry. I have 15% on my Intel Celeron J4115. Same as native.

EnfermeraSexy · 2022-02-25T16:58:04.584Z

It's already set as host. I don't think the CPU used by the VM is wrong. I've 600 Mb and i think 30% of CPU usage (4 cores assigned) for shapping that bandwitch isn't much, so im confortable with that. But after testing an LXC container the performance is really impresive.