Ultimate SQM settings: Layer_cake + DSCP marks


#242

Ture, it's just a wish!
i'm still reading to find a solution !
i have 2 wifi's, one is router wifi and second is usb wifi.
i will make a new br and i will put that wlan1 inside new br.then i will see how to solve this case.


#243

@hisham2630 :

root@OpenWrt:/etc/config# ip route show table main
default via 192.168.231.254 dev eth0.2 proto static src 192.168.230.25
10.1.1.0/24 dev gre4-gre3 proto kernel scope link src 10.1.1.3
192.168.1.0/24 dev br-vlan1 proto kernel scope link src 192.168.1.1
192.168.3.0/24 dev br-vlan3 proto kernel scope link src 192.168.3.1
192.168.4.0/24 dev br-vlan4 proto kernel scope link src 192.168.4.1
192.168.5.0/24 dev br-vlan5 proto kernel scope link src 192.168.5.1
192.168.230.0/23 dev eth0.2 proto kernel scope link src 192.168.230.25
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config# ip route show table 100
default dev veth0 scope link
192.168.5.0/24 dev br-vlan5 proto kernel scope link src 192.168.5.1
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config# ip route list
default via 192.168.231.254 dev eth0.2 proto static src 192.168.230.25
10.1.1.0/24 dev gre4-gre3 proto kernel scope link src 10.1.1.3
192.168.1.0/24 dev br-vlan1 proto kernel scope link src 192.168.1.1
192.168.3.0/24 dev br-vlan3 proto kernel scope link src 192.168.3.1
192.168.4.0/24 dev br-vlan4 proto kernel scope link src 192.168.4.1
192.168.5.0/24 dev br-vlan5 proto kernel scope link src 192.168.5.1
192.168.230.0/23 dev eth0.2 proto kernel scope link src 192.168.230.25
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config#
root@OpenWrt:/etc/config# route -e
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         192.168.231.254 0.0.0.0         UG        0 0          0 eth0.2
10.1.1.0        *               255.255.255.0   U         0 0          0 gre4-gre3
192.168.1.0     *               255.255.255.0   U         0 0          0 br-vlan1
192.168.3.0     *               255.255.255.0   U         0 0          0 br-vlan3
192.168.4.0     *               255.255.255.0   U         0 0          0 br-vlan4
192.168.5.0     *               255.255.255.0   U         0 0          0 br-vlan5
192.168.230.0   *               255.255.254.0   U         0 0          0 eth0.2

@dlakelan : I added br-vlan5 specific rule now but still I do not see dscp on ingress packets. Also, I tried to capture packets on veth0 and veth1 but could not see any icmp packets on these two interfaces for my ping.


#244

I think we need to take a step back. If you tag ALL traffic going to vlan4 as CS6 then there is really no differentiation on vlan4 and so it's basically the same as if you left all traffic tagged CS0. Note that because traffic is only queued once it's destined to vlan4 the fact that it's tagged CS6 doesn't affect (delay) traffic to other vlans. I think most likely you will not get what you really want from this design.

Most likely what you want from this design (in downstream direction) is: traffic comes in on WAN, all traffic goes through a single queue where traffic destined for VLAN4 is up-prioritized over traffic destined to other VLANs, then traffic goes to its individual VLAN and is distributed.

It is possible to do something like that using veth devices, but the way to do it is a little tricky: put your eth0.2 and veth0 into a separate namespace (maybe call it "prewan"), in this namespace establish the queue on veth0 and the default routing from eth0.2 to veth0, and the firewall rules for tagging (you don't need any other firewall rules in this namespace). Then use veth1 as your WAN. You should not put ip addresses on eth0.2 or veth0, they should just be layer 2 devices, so that there is no security issues in this namespace.

I think you could probably do this by setting everything up in UCI and then in some startup script like /etc/rc.local just move the eth0.2 and veth0 into their own namespace and set up the routing and tagging rules. It may be worth trying if you are adventurous, but I don't have lots of time to help debug it in detail.


#245

@dlakelan
instead of second name space is it possible to create a bridge between eth0.2 and veth0 without set any ip(keep them layer 2), then set veth1 as wan interface, then is it possible to tag packets ?

@dsinghra12
it's possible to create 5 veth pair's, then run sqm on each of them. but this not nice solution!


#246

Yes, That's exactly what I want.

Thanks for your valuable suggestions. I will definitely try that approach if not now.


#247

i think this what you need: https://superuser.com/questions/764986/howto-setup-a-veth-virtual-network/765078#765078


#248

It's an interesting idea, and it might work, it's certainly easier to try than the namespace idea. I have an old tp-link router around I could maybe test it on but it will be many days before I can get to that. If someone else can try it, please report back here.


#249

@dlakelan
i will try when network is not in use!
i think there's one more solution, which is creating a tuntap , then route traffic between wan and lan.
but not sure how!


#250

That is undoubtedly going to be much much higher CPU requirement, as it will send each packet one at a time from the kernel to userspace and back to the kernel.


#251

I set a test up on an old TP-link WDR-3600 I have lying around.
with ip-full and kmod-veth and all that installed, I used this script to create veth pair:

ip link add type veth
ip link set veth0 up
ip link set veth1 up

I then went to luci and created a new interface "prewan", I made it a bridge between eth0.2 and veth0, then I moved WAN to be physically connected to veth1.

I then put SQM instance on egress only on veth0 and veth1. The veth0 rate was "download" rate of actual wan, and the veth1 rate was "upload" rate of actual wan.

and IT WORKED just fine.

EDIT: obviously you want to fiddle around with the iptables rules for tagging with DSCP, and perhaps it requires you to do use bridge netfilter to ensure the iptables are called when the prewan bridge receives packets. But it's a very cool idea. @moeller0 might be interested.


#252

So I did wind up playing with this on my main router, and found that I had to shove the bridge into a different namespace otherwise it wouldn't forward the packets through the veth device on receive, it would just receive them on the bridge itself. But after shoving the bridge into the other namespace where none of the ip addresses existed it would just forward from the wan ethernet to the veth where it queues and then pops out on veth1 in the main namespace. Definitely the way to go to get fully customizable dscp based qos/sqm


#253

Just a thought: for most users the level of indirection/raffinesse that @hisham2630 needs seems a bit over the top, since most ISP, I know of, will not offer different rates for different traffic anyways, but just use a single shaper/policer per line. And in that case I begin to wonder whether it would not be simpler to recommend people desiring ultimate control to use a router+AP setup, then the ingress shaper can easily be instantiated on the LAN-facing interface and iptables marking will simply work without the need for either an IFB or the nifty veth-pair approach so skillfully demonstrated here.
Sidequestion @dlakelan can you quantify the CPU-cost of the veth approach as compared to the IFB approach? I ask, as maybe adding a shaper script that uses the veth-method to sqm-scripts would offer a partial solution for people wanting to play with setting the dscps for ingress without forcing them to use tc directly (which supposedly is computationallt cheaper than iptables and might still be a reasonable idea if the rules do not require access the the NATed internal addresses)?

I still have putting the newish nft-qos package through the paces to see whether that offers fine-grained shaping without introducing severe bufferbloat by itself.
Why do I bring this up here? Well, the intention for sqm-scripts was always to find a solution that works well enough for most users with needing to much individual twaeking and tuning. And I do consider the gaming case to be within reach (again @hisham2630's situation really requires the sophistication demonstrated here, but I bet a number of other's might be fine without the extreme heuristics involved in per-port rules).

IMHO this router & AP approach alao will scale better to higher speeds as it relieves the router from having to spend a variable amount of CPU cycles on WIFI. In addition this also removes the requirement for the router to offer WIFI itself, making a number of x86/arm boards a more natural fit.


#254

Hishams case is definitely unusual but it's not unusual to have a lan and guest and even maybe a third vlan. In those cases it's problematic to do wan and lan shapers since there are two or three lans and you can't prioritize them as @dsinghra12 wants to.

I will do some more fiddling as my setup isn't yet stable, I need to put my dscp rules into the separate namespace etc.

I can say that strangely my network seems to be faster after the veth stuff, though I suspect it's due to the variability of my wan speed. Gigabit fiber from ATT isn't really gigabit, as the whole neighborhood probably shares a single 10gig uplink or something, result is speeds vary from 400mbit to 900


#255

@dlakelan I will try above mentioned approaches by tomorrow as I have only one router with me now and need to give some Demo on that today. Will update my results by tomorrow.


#256

Hi, in Iraq(which is my country),Egypt and India, they offer different traffic rates, like 10 mbps for normal download and 50 mbps for google drive, even youtube will have something like 20mbps.

i see that cpu cost using veth method will be + 10~15% more than normal using 2 pair's of veth.
it's much easier to use iptables cause it's more readable and understandable than tc commands.also tc is
well known to be not fully documented!, i know that tc is cheaper but hard to configure!

i was planing to add a new tab in sqm, lets call it advanced, in this tab user can easily select which ip's/port's
he want to tag and prioritize. i'm still fine without using the second veth pair, i'm still get a stable ping with
saturated link in both or one direction(not a 100% saturated lets say 98% or less).
i'm still want to buy a x86/x64 router, but still don't know which brands and models is available in my country.


#257

i'm still want to understand/get this case to work, also preferably without using second namespace!(try to add an ip for wan bridge and see if this help!), you should be happy if you can get 400mbps, here you can't even get a 8mbps!. in the north of the country
you can get 1gbps fo 100$ but not sure if it's unlimited and stable!
EDIT:
It would be much better if there's a way to put one veth(or use another type of interface) in main table, and remove wan from main table, and second table with another veth with wan interface!


#258

IMHO something like this should live in its own independent luci-app, call it luci-app-diffserve or so. This kind of fine-grained configuration is pretty much what sqm-scripts was created to avoid (for those user's that can live without extreme configurability).
As I already indicated adding a script to sqm-scripts that uses the veth-pair method instead of the ifb route should be easier to get upstreamed :wink:


#259

this is another good idea,but who will write this new package!
@dlakelan there's an interesting things here
https://wiki.debian.org/BridgeNetworkConnections
and this
https://wiki.libvirt.org/page/Networking#Altering_the_interface_config
it might help,
under Libvirt and bridging.


#260

I think what happens is that the Linux bridge sees a packet come in say eth0.2 into the bridge. It checks this packet to see how to deliver it. it sees that it's destined for the router and then it processes the packet rather than resending it out the veth0.

Now, for many people this may not be particularly a big problem, since they are mostly routing but I also run squid proxy on my router to fine-grain control access to the web (it's for security, and time-limits for kids, and also some prioritization/bandwidth limits as well). So this means all the traffic going to squid, which is a lot, is un-queued.

Moving the bridge to the second namespace solves that problem. But it may not be a problem for most people.

Try it like this from above and see if it works: Ultimate SQM settings: Layer_cake + DSCP marks


#261

Hey @dlakelan
i think using ebtables commands will fix this problem!