So you mean that i should trust cake/fq_codel magic!
i need some ideas!:
i'm still getting a ping spikes on download only, the main problem is if i set download speed to something like
1000 kbit/s , i would loss the youtube speed and other services like caching, cause isp's here is providing 1mbps
for everything, except youtube and few site that will gave me 8mbps!.
so should i route the fast traffic to another pair of veth then using sqm on it!
No, not really magic, just that fact that both give non-queing flows a slight boost. That might be good enough already; if not you can still do research in invariants in the packet headers you can just for strict rules...
One thing that might help a bit is to add the ingress keyword to the cake instances that handle download traffic from the internet. But that will still not solve the issues posed by the complex behavior of your ISP.
That is what I would try if there are actually two independent shapers active at your ISP, so if the sum of all traffic is 8Mbps for youtube&Co. plus 1 Mbps for the rest, totaling 9Mbps then splitting the traffic into its two tiers and run independent shapers on both seems feasible. (Unless switching to an ISP with a somewhat less ornery and more rational offer is an option 
)
1 Like
Ok, i will try then lets see how it will going!
Yeah, at the moment i will route traffic to into two veth, the strange thing is upload is very stable even if put 30000kbit or more as bandwidth. bufferbloat is still low, only download crazy.
This 100% impossible in my country, all isp's do the same thing!, but at least this isp is offer 1mbps for other stuff, but other isp's is offering like 0.5 and 0.25 mbps !, i'm paying 35$ per month for this very low speed connection, cause government is sell 1 mbps for 100$ 
, but in the north of my country they sell it for less than 5$
EDIT:
should i insert the ingress keyword to "option eqdisc_opts" on veth0 ?
I just tested my setup with gaming and max download speed torrent on background, hit registration in game with max priority UDP packet <- and -> was superb, in fact unnoticable that i was downloading in background at all. I still saw in-game ping a bit higher than usual but thats because i dont tag TCP on ingress for the particular game, so that latency info the game gathers just goes into the flow with the downloading torrent data. but its all about the UDP traffic anyway for actual gameplay 
I think its all working great! Whoohoo!
@hisham2630 Are you testing ping with just pinging? I get some occasional ping spike to 150+ms but thats probably because of my 4G variable download speed. Perhaps you should prioritize ICMP packets on both direction? 
I think its super hard to QoS anything that slow of a connection, thats sad to hear :C
thats nice, but ping doesn't use tcp or udp, it's use a different protocol called ICMP, use this line to tag ping packets!
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS6 ##dscp tag for ping packets
i'm test pinging on both games and normal ping via cmd. it's not really always about variable download speed, try to tag ping packets then see how it will going, also it would be better to prioritize other traffic.
at least i'm happy with this sqm/qos results, even i tried a download on my phone + playing pubg, game
run smooth from login to gaming, but i see that qos is performing better on +4mbps speed, i'm still need to isolate fast traffic from slow one!
Hello @Emtee @hisham2630 @dlakelan
Thanks for this detailed explanation and configurations you have provided here. These are really helpful. I used these configuration in my setup which can be defined as below:
I have divided my wlan into different networks using dynamic VLAN (with the help of radius server) so that each type of device like IoT, Gaming consoles will be in different VLANs and in different subnets. Now, I want to apply different QOS on these VLANs.
Let's say Gaming devices are in subnet 192.168.4.1/24 and I want all packets coming and going from this subnet should use low latency (CS6 or EF). I am able to set DSCP value for egress packets and could see them on eth0.2(WAN interface) with correct DSCP value. But for ingress packets, It still shows CS0.
I used below rules to mark packets:
iptables -t mangle -A PREROUTING -i eth0.2 -j DSCP --set-dscp-class CS0
iptables -t mangle -A PREROUTING -s 192.168.4.1/24 -j DSCP --set-dscp-class CS6
iptables -t mangle -A PREROUTING -d 192.168.4.1/24 -j DSCP --set-dscp-class CS6
ifconfig:
root@OpenWrt:~# ifconfig
br-vlan1 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fdf0:7136:5f11::1/60 Scope:Global
inet6 addr: fe80::f6f2:6dff:fe95:267b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23113 errors:0 dropped:0 overruns:0 frame:0
TX packets:11842 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3059033 (2.9 MiB) TX bytes:2445920 (2.3 MiB)
br-vlan3 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
inet addr:192.168.3.1 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::f6f2:6dff:fe95:267b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:4272 (4.1 KiB)
br-vlan4 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0
inet6 addr: fe80::f6f2:6dff:fe95:267b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13398 errors:0 dropped:0 overruns:0 frame:0
TX packets:511 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:887471 (866.6 KiB) TX bytes:94433 (92.2 KiB)
br-vlan5 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
inet addr:192.168.5.1 Bcast:192.168.5.255 Mask:255.255.255.0
inet6 addr: fe80::f6f2:6dff:fe95:267b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:4548 (4.4 KiB)
eth0 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7C
inet6 addr: fe80::f6f2:6dff:fe95:267c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:318692 errors:0 dropped:0 overruns:0 frame:0
TX packets:35897 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:85313572 (81.3 MiB) TX bytes:5251389 (5.0 MiB)
Interrupt:4
eth0.2 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7C
inet addr:192.168.230.25 Bcast:192.168.231.255 Mask:255.255.254.0
inet6 addr: 2600:2300:400::f6f2:6dff:fe95:267c/64 Scope:Global
inet6 addr: fe80::f6f2:6dff:fe95:267c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:312261 errors:0 dropped:129 overruns:0 frame:0
TX packets:35236 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:78004609 (74.3 MiB) TX bytes:5032336 (4.7 MiB)
eth1 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
inet6 addr: fe80::f6f2:6dff:fe95:267b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:25521 errors:0 dropped:0 overruns:0 frame:0
TX packets:22588 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3827248 (3.6 MiB) TX bytes:3546426 (3.3 MiB)
Interrupt:5
eth1.1 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23112 errors:0 dropped:0 overruns:0 frame:0
TX packets:11843 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3059027 (2.9 MiB) TX bytes:2445940 (2.3 MiB)
eth1.3 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:4272 (4.1 KiB)
eth1.4 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:9275 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:423336 (413.4 KiB)
eth1.5 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7B
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:4548 (4.4 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1399 errors:0 dropped:0 overruns:0 frame:0
TX packets:1399 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:158567 (154.8 KiB) TX bytes:158567 (154.8 KiB)
veth0 Link encap:Ethernet HWaddr E2:2B:19:67:9D:70
inet6 addr: fe80::e02b:19ff:fe67:9d70/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:407 errors:0 dropped:0 overruns:0 frame:0
TX packets:12237 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:50396 (49.2 KiB) TX bytes:1685828 (1.6 MiB)
veth1 Link encap:Ethernet HWaddr 9E:AF:C2:A5:02:07
inet6 addr: fe80::9caf:c2ff:fea5:207/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:12237 errors:0 dropped:0 overruns:0 frame:0
TX packets:407 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1685828 (1.6 MiB) TX bytes:50396 (49.2 KiB)
wlan1 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7A
inet6 addr: fe80::f6f2:6dff:fe95:267a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1014 (1014.0 B) TX bytes:2096 (2.0 KiB)
wlan1.4 Link encap:Ethernet HWaddr F4:F2:6D:95:26:7A
inet6 addr: fe80::f6f2:6dff:fe95:267a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2576 errors:0 dropped:0 overruns:0 frame:0
TX packets:5258 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:359324 (350.9 KiB) TX bytes:1168786 (1.1 MiB)
brctl show:
root@OpenWrt:~# brctl show
bridge name bridge id STP enabled interfaces
br-vlan1 7fff.f4f26d95267b no eth1.1
br-vlan3 7fff.f4f26d95267b no eth1.3
br-vlan4 7fff.f4f26d95267b no eth1.4
veth1
wlan1.4
br-vlan5 7fff.f4f26d95267b no eth1.5
sqm:
config queue 'wan'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option enabled '1'
option debug_logging '0'
option verbosity '5'
option qdisc 'cake'
option qdisc_advanced '1'
option qdisc_really_really_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option download '0'
option script 'layer_cake.qos'
option linklayer 'ethernet'
option interface 'eth0.2'
option eqdisc_opts 'diffserv4 nat dual-srchost rtt 170ms'
option iqdisc_opts 'diffserv4 nat dual-dsthost rtt 170ms autorate-ingress'
option upload '5000'
option overhead '18'
config queue
option debug_logging '0'
option verbosity '5'
option ingress_ecn 'ECN'
option enabled '1'
option download '0'
option qdisc 'cake'
option qdisc_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option egress_ecn 'ECN'
option qdisc_really_really_advanced '1'
option linklayer 'ethernet'
option script 'layer_cake.qos'
option interface 'veth0'
option eqdisc_opts 'diffserv4 nat dual-dsthost rtt 170ms'
option iqdisc_opts 'diffserv4 nat dual-srchost rtt 170ms autorate-ingress'
option upload '20000'
option overhead '18'
config queue
option debug_logging '0'
option verbosity '5'
option qdisc 'cake'
option qdisc_advanced '1'
option ingress_ecn 'ECN'
option qdisc_really_really_advanced '1'
option enabled '1'
option squash_dscp '0'
option squash_ingress '0'
option egress_ecn 'ECN'
option upload '30000'
option script 'piece_of_cake.qos'
option iqdisc_opts 'dual-srchost'
option eqdisc_opts 'dual-dsthost'
option linklayer 'ethernet'
option overhead '8'
option download '0'
option interface 'wlan1.4'
I am not sure if my iptables rule is correct where I am giving destination ip.
Thanks in advance for your help.
A subnet should have zeros after the mask. 192.168.4.0/24 for example, this may solve your problem.
Thanks for your reply. But it did not make any difference. i am still seeing CS0 on ingress packets. Also, I captured packets on both eth0.2 and br-vlan4 interface.
You made the changes here?, because those source and dest rules shouldn't match anything, once the /24 mask is applied the result can not have a .1 at the end, so it will never match.
Also, at some point did you set up routing to send things to veth0 rather than br-vlan4 directly?
Yes I did change here only.
Also, I have define routing rule for veth0 as mentioned in above mentioned comments:
ip route add default dev veth0 table 100
ip rule add iif eth0.2 table 100 priority 100
Do we need to define any other routing rule?
Well those routing rules will send everything coming in eth0.2 out veth0 which probably isn't correct because you have multiple VLANs. So I do think you'll probably want more routing rules in table 100.
If your iptables rules work in terms of tagging, then you will see packets heading towards your LAN have CS6 tags when you capture on veth0 or veth1 or br-vlan4, on the other hand, packets heading for the internet will only have the tags when you capture on eth0.2
there will not be any interface where both directions have tags, because the tagging happens between receiving on one and sending on the other.
Yes, I am capturing egress packets on eth0.2 and ingress on br-vlan4.
I will check this.
@dsinghra12
Hello, you are welcome here.
let me know what is the output of those commands:
ip route show table main
ip route show table 100
ip route list
route -e
i think that you have to move br-vlan's to table 100, but not sure if it will work.
another possible way to try Forwarding, but not sure how!
EDIT:
@dlakelan
Is it possible to remove veth1 from br-vlan4 then put veth1 in main table, also move those br-vlan's from main
table to table 100 along with veth0.
I think all that's needed is to put entries in table 100 for each vlan destination, rather than just a single default rule.
what about veth1 ?
One end of the veth pair has to be in a bridge. I've tried just having a floating pair but ARP doesn't work right. Technically you could also create a second namespace but this is a bit more advanced
what about putting veth1 in main table then other vlans in table 100 with veth0
Veth1 is just the other end of veth0 it exists just to receive packets from veth0 and nothing else. Packets shouldn't be routed to it
it would be nice if we can put those br-vlan's into one br interface, this will make it easier.
No that will break the segmentation that is there for a reason I assume (like guest vs regular LAN or whatever).