@moeller0
Am I right to say that SQM creates an ifb and controls only the egress part?
+-------------+ +----------------+ +------+
|ingress | |egress | +---------+ |egress|
|qdisc +--->qdisc +--->netfilter+--->qdisc |
|eth0.2 SQM | |ifb4eth0.2 SQM | +---------+ |eth1 |
+-------------+ +----------------+ +------+If the above is true is possible to create ingress ifb4eth0.2 ifb device?
I believe it used to be able to select ifb on LuCI-SQM
yes
please rephrase that
+-------------+ +----------------+ +------+
|ingress | |**ingress** | +---------+ |egress|
|qdisc +-->qdisc +--->netfilter+--> qdisc |
|eth0.2 SQM | |ifb4eth0.2 SQM | +---------+ |eth1 |
+-------------+ +----------------+ +------+
Is the above a possible setup?
Reason why is because I think the reason ifb4eth0.2 egress is bypassed in fast-path is because fast-path bypasses egress but not ingress
The hook occurs at nat postrouting maybe that is why egress gets skipped
@moeller0
Am I right to say that SQM creates an ifb and controls only the egress
part?
Yes, in a sense. The Linux kernel, as far as I know, does not support instantiating traffic shapers on an interface's ingress side. The workaround is to redirect all ingress traffic to an ifb and instantiate the shaper on that interface's egress, which sits between the real ingress and the further kernel network stack.
In case you only want to shape a specific direction you can set the sqm bandwidth to zero for the other direction; if you set the ingress bandwidth to zero there should be no ifb interface generated...
Best Regards
P.S.: I am travelling with on and off internet...
Hi @moeller0
After more intense testing on fast path
Result is as follows:
SQM applied on eth0.1 interface -> eth0.1 + ifb4eth1.0
Works on both Upload and Download
root@LEDE:~# tc qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc cake 8020: dev eth0.1 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
qdisc ingress ffff: dev eth0.1 parent ffff:fff1 ----------------
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 8021: dev ifb4eth0.1 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
SQM applied on eth0 interface -> eth0 + ifb4eth0
Works on both Upload and Download
root@LEDE:~# tc qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc cake 801a: dev eth0 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 801b: dev ifb4eth0 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
SQM applied on br-lan interface -> br-lan + ifb4br-lan
Works on only Download Upload does not work
root@LEDE:~# tc qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc cake 8026: dev br-lan root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
qdisc ingress ffff: dev br-lan parent ffff:fff1 ----------------
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 8027: dev ifb4br-lan root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
SQM applied on eth0.2 interface -> eth0.2 + ifb4eth0.2
Works on only Upload Download does not work
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc cake 802c: dev eth0.2 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 802d: dev ifb4eth0.2 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
So question is does the interface sequence matter? Because it clearly works on eth0.1(LAN) but not eth0.2(WAN)
I am puzzled, the interface sequence should not matter, BUT a bridge interface is a rotten place to instantiate a shaper (it will also affect all internal WLAN to LAN traffic as well as WLAN/LAN to WAN), so let's ignore that. But the eth0.2 result makes me scratch my head.
Could you run a dslreports speedtest for all combinations an post the results here, these tests often give enough data to get a better hypothesis what happens... (I collected a few configuration tips for the dslreports speedtest under https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803, maybe they can be of help...)
Each time I run tc qdisc to check the rules are correct
interface name eth0.2 will create ifb4eth0.2 etc etc, full tc qdisc rules are copied as shown in the previous post
No SQM
SQM eth0.2 Ingress/Egress 50/50Mbits
SQM eth0.1 Ingress/Egress 50/50Mbits
SQM eth0 Ingress/Egress 50/50Mbits
SQM br-lan Ingress/Egress 50/50Mbits
Okay, I see, now could you repeat a speedtest with sqm instantiated with piece_of_cake/cake on eth0.2 but capture the output of "tc -s qdisc" before and after that speedtest, please. In addition it would be great if you could also add the output of "ifconfig" from before and after the speedtest. And finally it would be great if you could post the link to the detailed results for that speedtest (or the link form the sharing section of the detailed results, which also contains the test's id number, but given how clear the effects are that yuo see this will only help to satisfy my curiosity, it should not matter for the issue at hand). Finally I assume that on your test machine none of the fast-path/shortcut modules are active during the test, correct?
Best Regards
In all my test SFE is running, the aim is to enable SFE to work with rate-limiting on SQM.
The thing is with SFE on SQM-Scripts works on eth0.1 up and down but on eth0.2 only up is working which is puzzling
Before running dslreports speedtest interface eth0.2 ingress 50Mbits/s egress 50Mbits/s
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 2477109287 bytes 3082824 pkt (dropped 0, overlimits 0 requeues 791)
backlog 0b 0p requeues 791
maxpacket 1514 drop_overlimit 0 new_flow_count 611 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 802c: dev eth0.2 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
Sent 49670 bytes 236 pkt (dropped 0, overlimits 1 requeues 0)
backlog 0b 0p requeues 0
memory used: 2464b of 4Mb
capacity estimate: 50Mbit
Bulk Best Effort Voice
thresh 3125Kbit 50Mbit 12500Kbit
target 5.8ms 5.0ms 5.0ms
interval 100.8ms 100.0ms 10.0ms
pk_delay 0us 28us 4us
av_delay 0us 4us 0us
sp_delay 0us 3us 0us
pkts 0 231 5
bytes 0 49274 396
way_inds 0 0 0
way_miss 0 43 3
way_cols 0 0 0
drops 0 0 0
marks 0 0 0
sp_flows 0 0 0
bk_flows 0 1 0
un_flows 0 0 0
max_len 0 1412 135
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 71355 bytes 219 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 802d: dev ifb4eth0.2 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
Sent 74421 bytes 219 pkt (dropped 0, overlimits 51 requeues 0)
backlog 0b 0p requeues 0
memory used: 21824b of 4Mb
capacity estimate: 50Mbit
Tin 0
thresh 50Mbit
target 5.0ms
interval 100.0ms
pk_delay 1.8ms
av_delay 78us
sp_delay 64us
pkts 219
bytes 74421
way_inds 0
way_miss 45
way_cols 0
drops 0
marks 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1514
After running dslreports speedtest interface eth0.2 ingress 50Mbits/s egress 50Mbits/s
qdisc noqueue 0: dev lo root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 2983159161 bytes 3650556 pkt (dropped 0, overlimits 0 requeues 822) backlog 0b 0p requeues 822 maxpacket 1514 drop_overlimit 0 new_flow_count 799 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc noqueue 0: dev br-lan root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev eth0.1 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc cake 802c: dev eth0.2 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw Sent 144535872 bytes 243135 pkt (dropped 5582, overlimits 195546 requeues 0) backlog 0b 0p requeues 0 memory used: 121024b of 4Mb capacity estimate: 50Mbit Bulk Best Effort Voice thresh 3125Kbit 50Mbit 12500Kbit target 5.8ms 5.0ms 5.0ms interval 100.8ms 100.0ms 10.0ms pk_delay 0us 1.8ms 48us av_delay 0us 1.3ms 1us sp_delay 0us 3us 1us pkts 0 248670 47 bytes 0 152804566 4302 way_inds 0 0 0 way_miss 0 573 7 way_cols 0 0 0 drops 0 5582 0 marks 0 0 0 sp_flows 0 6 0 bk_flows 0 1 0 un_flows 0 0 0 max_len 0 1514 238 qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- Sent 8010809 bytes 9102 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev wlan1 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc noqueue 0: dev wlan0 root refcnt 2 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc cake 802d: dev ifb4eth0.2 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw Sent 8096192 bytes 9072 pkt (dropped 30, overlimits 8959 requeues 0) backlog 0b 0p requeues 0 memory used: 142848b of 4Mb capacity estimate: 50Mbit Tin 0 thresh 50Mbit target 5.0ms interval 100.0ms pk_delay 517us av_delay 48us sp_delay 4us pkts 9102 bytes 8138237 way_inds 49 way_miss 690 way_cols 0 drops 30 marks 0 sp_flows 0 bk_flows 1 un_flows 0 max_len 1514
Before running dslreports speedtest interface eth0.1 ingress 50Mbits/s egress 50Mbits/s
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 2983839889 bytes 3652952 pkt (dropped 0, overlimits 0 requeues 822)
backlog 0b 0p requeues 822
maxpacket 1514 drop_overlimit 0 new_flow_count 799 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8032: dev eth0.1 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
Sent 10756 bytes 84 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 2464b of 4Mb
capacity estimate: 50Mbit
Bulk Best Effort Voice
thresh 3125Kbit 50Mbit 12500Kbit
target 5.8ms 5.0ms 5.0ms
interval 100.8ms 100.0ms 10.0ms
pk_delay 0us 10us 9us
av_delay 0us 1us 1us
sp_delay 0us 1us 1us
pkts 0 43 41
bytes 0 5008 5748
way_inds 0 0 0
way_miss 0 19 3
way_cols 0 0 0
drops 0 0 0
marks 0 0 0
sp_flows 0 0 1
bk_flows 0 0 0
un_flows 0 0 0
max_len 0 514 690
qdisc ingress ffff: dev eth0.1 parent ffff:fff1 ----------------
Sent 13256 bytes 108 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8033: dev ifb4eth0.1 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
Sent 14768 bytes 108 pkt (dropped 0, overlimits 6 requeues 0)
backlog 0b 0p requeues 0
memory used: 1984b of 4Mb
capacity estimate: 50Mbit
Tin 0
thresh 50Mbit
target 5.0ms
interval 100.0ms
pk_delay 58us
av_delay 3us
sp_delay 2us
pkts 108
bytes 14768
way_inds 0
way_miss 22
way_cols 0
drops 0
marks 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1514
After running dslreports speedtest interface eth0.1 ingress 50Mbits/s egress 50Mbits/s
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 3181447366 bytes 3880570 pkt (dropped 0, overlimits 0 requeues 830)
backlog 0b 0p requeues 830
maxpacket 1514 drop_overlimit 0 new_flow_count 841 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8032: dev eth0.1 root refcnt 2 bandwidth 50Mbit diffserv3 triple-isolate rtt 100.0ms raw
Sent 73898189 bytes 100811 pkt (dropped 7881, overlimits 102667 requeues 0)
backlog 0b 0p requeues 0
memory used: 154752b of 4Mb
capacity estimate: 50Mbit
Bulk Best Effort Voice
thresh 3125Kbit 50Mbit 12500Kbit
target 5.8ms 5.0ms 5.0ms
interval 100.8ms 100.0ms 10.0ms
pk_delay 0us 31us 9us
av_delay 0us 4us 3us
sp_delay 0us 3us 3us
pkts 0 108502 190
bytes 0 85622179 21364
way_inds 0 0 0
way_miss 0 354 5
way_cols 0 0 0
drops 0 7881 0
marks 0 0 0
sp_flows 0 0 0
bk_flows 0 0 1
un_flows 0 0 0
max_len 0 1514 690
qdisc ingress ffff: dev eth0.1 parent ffff:fff1 ----------------
Sent 122740048 bytes 124995 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8033: dev ifb4eth0.1 root refcnt 2 bandwidth 50Mbit besteffort triple-isolate wash rtt 100.0ms raw
Sent 123146272 bytes 124092 pkt (dropped 903, overlimits 162114 requeues 0)
backlog 0b 0p requeues 0
memory used: 124992b of 4Mb
capacity estimate: 50Mbit
Tin 0
thresh 50Mbit
target 5.0ms
interval 100.0ms
pk_delay 2.4ms
av_delay 2.1ms
sp_delay 3us
pkts 124995
bytes 124489978
way_inds 0
way_miss 352
way_cols 0
drops 903
marks 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1514So if I compare the ingress number of packets (bytes) I get:
eth0.1: 124995 - 108 = 124887 (124489978 - 14768 = 124475210 / 1000^2 = 124.47521 MB)
eth0.2: 9102 - 219 = 8883 (8138237 - 74421 = 8063816 / 1000^2 = 8.063816 MB)
Now, assuming the test is otherwise of the same duration and the throughput on eth0.2 ingress is actually 200 instead of 50, I would say the SFE module steals packets from qdiscs instantiated on an ifb (sort of making the tc action redirect only partially working). I had expected no packets actually hitting ingress cake at all, but it seems the SFE module will not trigger for all packets... Now, it would have been nice to also see how the packet and byte counters exposed by "ifconfig" would have changed for the same test...
In short it looks like the shortcut module also (partially) shortcuts the ifb.
One work-around might be to not use sqm on ingress at all; instead of shaping the WAN interface's egress and ingress one can also shape on the egress of the interface that connects the router SoC with the LAN switch (in that case one needs to configure "upload" only for both independent sqm instances). This will obviously not work well with SoCs where the WLAN interfaces are directly connected (so most of them), as WLAN packets will side-step the internet download shaper. But for testing that should work out...
One more thing to test would be whether cake's deNATing which is required for per-IP-fairness still works when NAT is dome in hardware (I assume SFE does hardware NAT, is that correct?)
Best Regards
Yes I also thought of separating it into eth1.0 and eth0.2 but will in affect lan speeds?
Assuming on LAN I have 1Gbits between localhost but I only have 250Mbits to and from WAN, by applying WAN limits on LAN, LAN speeds will be affected?
I am trying to see if there is a clean way of applying the old eth0.2 and make sure it still obeys the limit when set by SQM.
SFE is actually not Hardware NAT it simply creates and connects interface like eth0.2 and br-lan and shortcuts NAT between them bypassing a lot of layers.
I am assuming that when applying SQM to eth0.2 only downloads get shortcut, uploads are still hitting SQM but in eth0.1 case both uploads and downloads hit SQM, so can we replicate this for eth0.2?
Depends on your router's architecture. If the SoC's LAN port connects to a switch that supplies the other LAN ports than the switch obviously will not care about the sahper on the switch's port to the CPU. But as I said, often the WLAN radios are directly connected to the SoC so their traffic will not be handled by the shaper, which in essence makes the shaper ineffective in controlling bufferbloat.
Well your tests show that both the WAN interface which does NAT and the bridge interface have ingress shaper/ifb issues; as far as I can tell these are also the places where the shortcut module gets active. So my hypothesis is that the SFE module will, if active, side-step the IFB entry point and hence only a few packets (presumably packets not handled by the shortcut module, BTW there might be counters to check whether the number of packets hitting ingress cake are similar to the number of packets that are not going through the short cut module?).
I guess that would require figuring out why most packets with SFE are "routed" around sqm's ingress IFB...
Best Regards
This works both directions
tc qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc cake 80b7: dev eth0.1 root refcnt 2 bandwidth 250Mbit diffserv3 triple-isolate rtt 100.0ms raw
qdisc cake 80b5: dev eth0.2 root refcnt 2 bandwidth 250Mbit diffserv3 triple-isolate rtt 100.0ms raw
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
Basically the ifb devices gets skipped do you have any idea where in the network path does the packet pass to ifb?
mmmh, I somehow have the feeling you have read https://unix.stackexchange.com/questions/288959/how-is-the-ifb-device-positioned-in-the-packet-flow-of-the-linux-kernel already (or maybe you wrote it yourself). But I have no more insight about where the packet stealing happens, but the data we have indicates, that SFE basically steals the packets earlier, so that there is nothing left for the ifb to steal. The fact that you still saw some residual packets on the ifb is probably due to the fact that SFE (AFAICT) will only start stealing after 128 packets have been send (per flow?)...
Best Regards