Qosify: new package for DSCP marking + cake

elan · 2022-02-23T19:23:24.033Z

These keywords in CAKE have been provided to represent a number of common link technologies:

Write the keyword below
***********************
"conservative"     Failsafe     (overhead 48 atm)
"ethernet"         Ethernet     (overhead 38 mpu 84 noatm)
"docsis"           Cable Modem  (overhead 18 mpu 64 noatm)
"pppoe-ptm"        VDSL2        (overhead 30 ptm)
"bridged-ptm"      VDSL2        (overhead 22 ptm)
"pppoa-vcmux"      ADSL         (overhead 10 atm)
"pppoa-llc"        ADSL         (overhead 14 atm)
"pppoe-vcmux"      ADSL         (overhead 32 atm)
"pppoe-llcsnap"    ADSL         (overhead 40 atm)
"bridged-vcmux"    ADSL         (overhead 24 atm)
"bridged-llcsnap"  ADSL         (overhead 32 atm)
"ipoa-vcmux"       ADSL         (overhead 8  atm)
"ipoa-llcsnap"     ADSL         (overhead 16 atm)

# If you are unsure, then write "conservative" as a general safe value.
# "raw" keyword turns off all overhead compensation in CAKE.

"ether-vlan" keyword information:

"ether-vlan" adds '4 bytes' to the overhead.

- This keyword is not for standalone use, but act as a modifier to some previous presets or keywords.
- In addition to those previous presets or keywords it is common to have VLAN tags (4 extra bytes) or PPPoE encapsulation (8 extra bytes).
# This keyword "ether-vlan" may be repeated as necessary.

For true ATM links (ADSL), one often can measure the real per-packet overhead empirically, see for further information how to do that:

https://github.com/moeller0/ATM_overhead_detector

grrr2 · 2022-02-23T20:46:08.344Z

dave14305:

Router to LAN traffic doesn’t get modified by Qosify since the BPF filters are on the WAN interface only.

qosify allows multiple interface configuration, do you mean it is not working for other than wan interface as it is hardwired somehow?

dave14305 · 2022-02-23T21:19:26.702Z

I suppose I should add “by default”.

grrr2 · 2022-02-23T22:26:26.678Z

no worries, i thought you know something which under the hood silently ignores configuration.

now, i have a question: if multiple interface configuration is supported it looks there is no way tp specify config class and <port, ip, dns> rules per interface. is this correct? e.g. /etc/qosify/*.conf files rules are applied to all traffic on all configured interfaces?

francisuk1989 · 2022-02-25T01:18:18.040Z

The only way this would work, if you have multiples of internet connections (WANs) like an Cable or DSL line and then you add the "docisiscable" info from the interface you added by adding an VLAN and Interface name like so . Just remember. i wont recommend multiples of interfaces on 3G/4G5G type of connections, as they can differ in speeds the more users are using the phone masts.

Note; Cable/DOCSIS users that are going to use any type of QoS!!

most of the issues that you may experience with DOCSIS/Cable are causes by the shared access infrastructure. If youre in an overloaded CMTS cluster there's not much you can do. Of course you can have a small cable cluster that let's you always hit your full bandwidth no problem. Then you're lucky, but that won't happen often. And That's the nature of it, DOCSIS/Cable was never was designed to transmit individual users internet data.

Neither was phone wire, but with VDSL you only have shared core network, which rarely is a problem. With your dedicated twisted pair you are better off when it comes to stability and reliability of the internet access

# 63/19 plan
config interface wan
	option name wan
```	option bandwidth_down 60mbit
	option bandwidth_up 17mbit
	option overhead_type pppoe-ptm

# 50/15 plan
config interface docisiscable
	option name docsiscable
`	option bandwidth_down 48mbit
	option bandwidth_up 13mbit
	option overhead_type docsis
``

elan · 2022-02-25T02:46:49.502Z

Use Lag Report to know which ISP offers the best connection and the lowest ping where you live:

Lag Report

WHAT IS THE LAG REPORT?

moeller0 · 2022-02-25T09:01:39.931Z

francisuk1989:

DOCSIS/Cable was never was designed to transmit individual users internet data.

Funny, here I thought that DOCSIS (Data Over Cable Service Interface Specification) was specifically designed for that purpose... You can even configure a segment such that all users have a guaranteed access rate... But it turns out that this rate is rather low, and most users' access links are idling most of the time anyhow. That means that provisioning each user more than her/his guaranteed share will result in users being better of most of the time. It now becomes the challenge for the ISP to make sure that this oversubscription of the link does not result in noticeable effects for the users, if the ISP manages to do that, over subscription becomes a valid and efficient way to share a scarce resource.

francisuk1989:

Neither was phone wire, but with VDSL you only have shared core network, which rarely is a problem. With your dedicated twisted pair you are better off when it comes to stability and reliability of the internet access

In most cases the uplink of the DSLAM/MSAN is a similar bottleneck as the aggregate segment-rate is in DOCSIS. Many DSLAM with (low) hundreds of users with 100 Mbps plans are still uplinked with single or double 1Gbps connections, giving similar oversubscription as in DOCSIS networks. However it should be considerably simpler/cheaper to exchange a 1Gbps fiber uplink with e.g. a 10 Gbps simply by exchanging the "optics" at both sides of the link, compared to DOCSIS where either segment bandwidth increases or segment splits are required both of which are more involved than replacing two optic modules. (Sure replacing optics only works if the devices the optics attach to already support higher rates, but for halfway modern DSL gear that appears to be the case).

grrr2 · 2022-02-25T12:45:42.728Z

let me explain what i meant:

(note: it will be long and total misunderstanding from my end, if it is stupid please be a noble soul and just ignore or point out why it is not working like this, and not why am stupid. thanks.)

according to link qosify a) is not only supporting wan and b) not only supporting a single interface.
basically qosify does two things:

sitting on top of tc, configuring tc, but tc provides the queuing mechanism (not qosify directly)
there is a rule set system which translates rules into actual traffic marking (with hooks to dns etc.) and a specific logic how for example port rules are working.

let's use this notation, according to #1 qosify can be configured on any ( ! ) interface, it is not limited to wan, so let's use this generalized topology:

[ host0 ] - [ network0 ] --- if0 -[ router w/ qosify on if0 ]- if1 --- [ network1 ] - [ host1 ]
where ifX is interface of router; if0 is the interface qosify applied on
where networkX is a network
where hostX is member of corresponding networkX
where host0 & host1 talking to each other

according to link the specific logic how qosify marks traffic is the following:

* cosify is applied to if0
* rule: tcp:5555 	video

case1: in case of outgoing traffic from host1 via if0 (where qosify is applied) to host0:
host1:<port> -----> host0:5555 such traffic is marked as video
host1:<port> -----> host0:6666 such traffic is not marked as video as there is no rule
host1:5555 -----> host0:6666 such traffic is not marked as video as traffic is matched against destination port

case2: in case of incoming traffic from host0 via if0 (where qosify is applied) to host1:
host1:<port> <----- host0:5555 such traffic is marked as video as in case of ingress traffic source port match is used
host1:<port> <----- host0:6666 such traffic is not marked as video due to no source port match
host1:5555 <----- host0:6666 such traffic is not marked as video due to no source port match

so basically qosify always matching against the remote end.

the problem I see that there are traffic patterns where I only know/control the near end, and know nothing about the remote end prior connection so no rule can be defined. Typically VoIP, torrent is such kind of traffic. For example contrary to the VoIP configs flying around here, VoIP does not work the way some people thinks: defining the standard SIP signalling ports 5060/5061 will not help to have good call quality as the media is traveling in RTP using different ports and logic. And even if they know this and add rule for RTP, it will fail. Because usually RTP using dynamically picked port from a port range and depending on your vendor/setup it can vary by each end point, so you don't know what the other end will use. A typical voip call is a duplex steam meaning there are two streams: one from A to B, and another one from B to A. During call setup you advertise your end, i.e the port you expect the other end to connect to, which will be the destination port in case2 above and not the source port hence qosify will not mark your traffic with voice! Symmetric RTP port usage is very optional, and even so, if you really want direct media with anybody you'll have to capture any port 0:65535 ... which obviously makes no sense/and breaks qosify fine grained rule sets as will overlap with something else for sure and it is not supported. Similar problem if you use random listening port for torrent, you can only define your end and not the peer ports, so the configs flying around are neither correct because they assume all the other peers you meet will only use some handful of predefined ports.
obviously i can mark my traffic outside of qosify and just let qosify configure tc. or ... i might able to use qosify! after all it is not hardwired to only usable with single interface (by default the wan), so let's add a second interface configuration, apply qosify both on if0 and if1 as well ( ! ) on if1 just set for egress direction though. in this way I'd have this topology:

[ host0 ] - [ network0 ] --- if0 -[ router w/ qosify on if0 ]- if1 --- [ network1 ] - [ host1 ]
                                  [ qosify on if1.egress    ]

meaning two sets of tc diffserv queues, each assigned to if0 and if1 respectively 
host1:<port> -----> host0:5555 such traffic is marked as video by if0 queue
host1:<port> -----> host0:6666 such traffic is not marked

host1:<port> <----- host0:5555 such traffic is marked as video as in case of ingress traffic source port match is used
host1:<port> <----- host0:6666 such traffic is not marked as video due to no source port match
host1:5555 <----- host0:<port> such traffic is marked as video by if1 queuing !

this way i could mimic near end based matching!

the problem though (apart it may not work at all, double queuing, cpu impact etc) that due to common rule set even if i mean a large port range only for if1 ingress, it will capture traffic on if0 too. hence it would be nice to extend qosify's syntax to override direction and or specify ip:port not just port imho.

EnfermeraSexy · 2022-02-25T13:16:08.696Z

I'm testing openwrt inside an LXC container in proxmox. Everything seems to work fine but Qosify and it seems that the problem is that it can't write into the /sys/fs/bpf folder. I know it's not qosify fault but does anybody know a way to make it work? Using privileged containers with all permissions seems to not work.

libbpf: failed to mkdir /sys/fs/bpf/qosify_data/config: Permission denied
libbpf: map 'config': failed to auto-pin at '/sys/fs/bpf/qosify_data/config': -13
libbpf: map 'config': failed to create: Permission denied(-13)
libbpf: failed to load object '/lib/bpf/qosify-bpf.o'
bpf_object__load: Permission denied

dMopp · 2022-02-25T16:04:12.336Z

All i can tell you is: dont use lxc for openwrt. i tried the same and switched over to kvm

EnfermeraSexy · 2022-02-25T16:19:27.414Z

Thanks but the performance isn't even comparable. I tried SQM and cake and it didn't even use 3% of CPU, while in a KVM it uses near 30%.
For now i will use it in a VM but if i could make it to work in LXC i will use it, or at least try. As i said the only thing that does not work is qosify.

moeller0 · 2022-02-25T16:41:25.499Z

EnfermeraSexy:

Thanks but the performance isn't even comparable. I tried SQM and cake and it didn't even use 3% of CPU, while in a KVM it uses near 30%.

Are you sure that in one condition SQM really took 27%-age points more CPU cycles? Or could it be that in the container case the CPU cycles might simply not have been accounted to the container? Mind you I know little about containers so could be wrong, but the cycles SQM needs to do its work need to show up somewhere (possible that 3% is a realistic number).

EnfermeraSexy · 2022-02-25T16:53:47.829Z

Didn't make so many test but the global CPU usage from the host machine didn't increase that much. I don't know if the CPU usage was made by SQM or not. That CPU usage could be because the VM emulation it self, the drivers from the network (i used virtio)... If i'm not wrong LXC has direct acces to the network you atach so there's less overhead.

dMopp · 2022-02-25T16:55:35.341Z

Did you try to change the CPU Type to „host“? Cake (SQM) is always very CPU hungry. I have 15% on my Intel Celeron J4115. Same as native.

EnfermeraSexy · 2022-02-25T16:58:04.584Z

It's already set as host. I don't think the CPU used by the VM is wrong. I've 600 Mb and i think 30% of CPU usage (4 cores assigned) for shapping that bandwitch isn't much, so im confortable with that. But after testing an LXC container the performance is really impresive.

dMopp · 2022-02-25T21:43:56.265Z

you could disable SQM and test KVM again. As i noted, there is no (or maybe 1 or 2 percent) difference between Hardware and VM. I tried that out

PS: You could also pass-trough the network adapters (except one, because somehow you should be able to reach the Hypervisor :D) But from what i can see, it makes no difference. Maybe with a 1G/1G line, but since i ahve Dual 2,5GBit Adapter even then i wouldnt care

EnfermeraSexy · 2022-02-26T15:58:13.307Z

I made it work by recompiling qosify and making it point to another directory where i mounted a BFP folder. I made some test and i found that in the VM the 23% of the CPU usage was just the usage of IRQs. So 7% was the real usage for SQM/cake.

In LXC i don't know why but i have 0% usage with qosify activated and working. The total CPU usage of the host isn't even near 1%.

EDIT: Now is the host the one that manages de IRQ, so that usage isn't counted in openwrt, but i see its 1% usage compared with the 23% from VM.

EDIT2: Finally i made /sys/fs/dpf writtable. I added the line:

lxc.apparmor.profile: unconfined

to the LXC container and now i'm able to write into the /sys folder. However i set it as read only as the openwrt wiki says, making it more secure and it just works as the bpf folder is mounted over sys.

msilletti · 2022-03-02T20:18:28.918Z

I would like to try this but I cannot find a setup guide for first time users. Is it in the 700+ replies somewhere?

anomeome · 2022-03-02T20:20:47.124Z

From the link of the OP

elan · 2022-03-02T22:38:17.230Z

In these two ways you can use my Qosify configuration:

Copy and paste my Qosify configuration in the files using WinSCP.
Copy and paste these commands at the same time in the SHH client or in the script to run on first boot.

To confirm if the traffic is marked correctly you have to use this command:

qosify-status

⠀
Information:

CAKE categories and DSCP marks information.
Keywords in CAKE for all common link technologies (overhead compensation).
Know which ISP offers the best connection and the lowest ping where you live.