I'm not right person for that either, but I can help with "proof of concept" and perhaps that will generate some interest.
1 Like
That's why I try so hard to build teams!
1 Like
so there is a first time for everything and for me it's implementing an Out Of Tree kernel module. I took a shot at adapting kmod-sched-cake-oot for fq_codel_fast. The openwrt package makefile (available in a git repo here) seems to work. It downloads, attempts a build, etc.
However, I'm getting the following error during building (which I did not see on linux):
...
make[3]: Entering directory '/home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/linux-5.4.39'
CC [M] /home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b/sch_fq_codel_fast.o
/home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b/sch_fq_codel_fast.c:683:25: error: static declaration of 'fq_codel_qdisc_ops' follows non-static declaration
static struct Qdisc_ops fq_codel_qdisc_ops __read_mostly = {
^~~~~~~~~~~~~~~~~~
In file included from ./include/net/pkt_cls.h:7,
from /home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b/sch_fq_codel_fast.c:26:
./include/net/sch_generic.h:572:25: note: previous declaration of 'fq_codel_qdisc_ops' was here
extern struct Qdisc_ops fq_codel_qdisc_ops;
^~~~~~~~~~~~~~~~~~
scripts/Makefile.build:265: recipe for target '/home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b/sch_fq_codel_fast.o' failed
make[4]: *** [/home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b/sch_fq_codel_fast.o] Error 1
Makefile:1692: recipe for target '/home/n/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b' failed
...
I just started digging into it but I'd thought I'd share in the event others are interested.
1 Like
more than likely, I just missed something trying to translate the linux makefile to the openwrt build make file...
but instruction is always welcome
1 Like
As it currently is, it can be built for openwrt but only with some hackish modifications to openwrt that I do not recommend.
It appears openwrt's fq_codel implementation has evolved relative to sch_fq_codel_fast, see EDIT1_.
openwrt and ubuntu 20.04 are not similar in this respect.
ubuntu:
[31] $ cat /usr/src/linux-headers-5.4.0-29/include/net/sch_generic.h | grep fq_codel
[32] $
In other words, there is no definition for fq_codel_qdisc_ops in sch_generic.h in the kernel headers.
There is however one in openwrt's sch_generic.h. As a temporary workaround, I can build and load fq_codel_fast after commenting out a few lines in openwrt's sch_generic.h (but again I do not recommend trying to use fq_codel_fast like this).
I have to go back and understand why openwrt's fq_codel implementation has evolved to where it is.
Start EDIT1:
git log -S fq_codel_qdisc_ops --oneline
is a start to understanding how fq_codel has evolved in openwrt.
I did
colordiff build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/linux-5.4.39/net/sched/sch_fq_codel.c build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x_generic/sch-fq-codel-fast-oot-2019-08-20-51bd334b/sch_fq_codel_fast.c
and that output makes me suspect it will be easier to adapt fq_codel_fast to openwrt rather than adapting openwrt to the current fq_codel_fast.
End EDIT1
EDIT2 I edited a lot of this post as I don't like leaving detailed instructions for what not to do.
EDIT3: it looks like the following 3 patches are the ones to look at:
target/linux/generic/pending-5.4/620-net_sched-codel-do-not-defer-queue-length-update.patch
target/linux/generic/hack-5.4/660-fq_codel_defaults.patch
target/linux/generic/hack-5.4/661-use_fq_codel_by_default.patch
if it's just these 3, then maybe updating fq_codel_fast would not be difficult.
1 Like
thx for isolating the existing patches to the patcjhes on the patchset. 
I'm really busy this week (see the tsvwg mailing list for why)
https://github.com/openwrt/openwrt/blob/master/target/linux/generic/hack-4.19/660-fq_codel_defaults.patch is a hack!!! better would be to just leverage the size of free memory - but I have no idea how to get it.
does openwrt run on 10gigE systems much?
1 Like
I am glad they rip out pfifo_fast entirely but there's a can bus patch that was needed too for that. agggghhhh.... was the only major bug in fq_codel! vs the can bus... don't know if it ever went into mainline....
1 Like
I don't know but I suspect few if any as openwrt seems to me as mostly focused on SOHO routers. Perhaps there are some people running it on x86 systems that might do that - pfsense might be a better choice at that point tho.
Having made this statement, I suspect the more experienced forum members will chime in and correct me.
FWIW: as it turned out, it's only a few lines to "manually" adapted the 3 patches mentioned above to fq_codel_fast. It builds and loads for me (IMO much more cleanly than what I did above) but I have not tried to use it yet. I'll push that to my git repo in a bit as a patch set to the oot kmod package for anyone following along.
I don't recommend anyone get too attached to it tho. I just pulled in linux-5.4.y to my local openwrt repo so I can see how fq_codel has evolved in the kernel sources relative to the current fq_codel_fast. There may be some changes necessary from that as well. I post what I find.
no worries and thank you for all the links/reference material. Between reading and digesting those while mucking about with the code, I have plenty to keep me going.
pfifo_fast/CAN bugfix is interesting
I wonder if its absence in openwrt might explain some complaints I've seen on the forums about IOT devices not working...
1 Like
So I did a non-exhaustive comparison between fq_codel_fast, fq_codel in openwrt, and fq_codel in linux-5.4.y and it does not look like there have been updates to fq_codel from linux-5.4.y since fq_codel_fast was branched off.
The oot kmod for fq_codel_fast builds, installs and loads on my r7500v2 without error.
I went ahead and did a "casual" test between my r7500v2 (set up for GW/AP mode, wireless off) and a laptop through a n66u (revived as a substitute AP on my network and in use both on the wire and wireless at the time of testing) using SQM with either fq_codel or fq_codel_fast. The r7500v2 is connected to the n66u AP on the r7500v2 WAN port, its firewall is up but WAN ports are open for SSH (22) and iperf (5001). The laptop (running "iperf -s" is connected to n66u AP via wire and wireless.
This test is just to see if anything breaks.
The short answer is nothing broke or crashed.
If someone else is interested enough to try it, it looks good enough to get started.
Below are iperf test results between the router and laptop. At this time, I don't think these results mean anything other than fq_codel_fast used this way does not crash the r7500v2. I'll need to do some reading about sqm configuration, testing (yes, I'm aware of flent), etc.
### after installing fq_codel_fast and a reboot... ###
BusyBox v1.31.1 () built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt SNAPSHOT, r13285-597df570f3
-----------------------------------------------------
root@r7500v2:~# tc -d -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 13522 bytes 100 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 13522 bytes 100 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
root@r7500v2:~# lsmod | grep codel
sch_fq_codel_fast 20480 0
root@r7500v2:~# ll /var/run/sqm/available_qdiscs/
drwxr-xr-x 2 root root 80 May 19 13:52 ./
drwx------ 3 root root 60 May 19 13:59 ../
-rw-r--r-- 1 root root 0 May 19 13:52 cake
-rw-r--r-- 1 root root 0 May 19 13:52 fq_codel
root@r7500v2:~# touch /var/run/sqm/available_qdiscs/fq_codel_fast
### test with fq_codel and simple.qos before trying fq_codel_fast ###
root@r7500v2:~# tc -d -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 323204 bytes 746 pkt (dropped 0, overlimits 0 requeues 2)
backlog 0b 0p requeues 2
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 323204 bytes 746 pkt (dropped 0, overlimits 0 requeues 2)
backlog 0b 0p requeues 2
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 ver 3.17 direct_qlen 1000
linklayer ethernet overhead 22
Sent 36632 bytes 95 pkt (dropped 0, overlimits 5 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 36632 bytes 95 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 4432 drop_overlimit 0 new_flow_count 5 ecn_mark 0
new_flows_len 0 old_flows_len 1
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 6950 bytes 59 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
linklayer ethernet overhead 22
Sent 9074 bytes 59 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 9074 bytes 59 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 824 drop_overlimit 0 new_flow_count 6 ecn_mark 0
new_flows_len 0 old_flows_len 1
root@r7500v2:~# iperf -c XXX.XXX.45.137 -d -t60 -i10
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: -1.00 Byte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to XXX.XXX.45.137, TCP port 5001
TCP window size: 140 KByte (default)
------------------------------------------------------------
[ 4] local XXX.XXX.45.101 port 59406 connected with XXX.XXX.45.137 port 5001
[ 5] local XXX.XXX.45.101 port 5001 connected with XXX.XXX.45.137 port 49510
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 562 MBytes 472 Mbits/sec
[ 5] 0.0-10.0 sec 845 MBytes 709 Mbits/sec
[ 4] 10.0-20.0 sec 602 MBytes 505 Mbits/sec
[ 5] 10.0-20.0 sec 811 MBytes 681 Mbits/sec
[ 5] 20.0-30.0 sec 778 MBytes 652 Mbits/sec
[ 4] 20.0-30.0 sec 626 MBytes 526 Mbits/sec
[ 5] 30.0-40.0 sec 778 MBytes 653 Mbits/sec
[ 4] 30.0-40.0 sec 636 MBytes 533 Mbits/sec
[ 4] 40.0-50.0 sec 643 MBytes 539 Mbits/sec
[ 5] 40.0-50.0 sec 767 MBytes 644 Mbits/sec
[ 4] 50.0-60.0 sec 634 MBytes 532 Mbits/sec
[ 4] 0.0-60.0 sec 3.62 GBytes 518 Mbits/sec
[ 5] 50.0-60.0 sec 792 MBytes 664 Mbits/sec
[ 5] 0.0-60.0 sec 4.66 GBytes 667 Mbits/sec
[SUM] 0.0-60.0 sec 5.48 GBytes 785 Mbits/sec
root@r7500v2:~# tc -d -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 4066243443 bytes 2763972 pkt (dropped 0, overlimits 0 requeues 287)
backlog 0b 0p requeues 287
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 4066243443 bytes 2763972 pkt (dropped 0, overlimits 0 requeues 287)
backlog 0b 0p requeues 287
maxpacket 1514 drop_overlimit 0 new_flow_count 543 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 ver 3.17 direct_qlen 1000
linklayer ethernet overhead 22
Sent 3895985795 bytes 2763321 pkt (dropped 0, overlimits 84819 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 3895925051 bytes 2763100 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 65248 drop_overlimit 0 new_flow_count 28157 ecn_mark 0
new_flows_len 1 old_flows_len 3
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 60744 bytes 221 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 4432 drop_overlimit 0 new_flow_count 25 ecn_mark 0
new_flows_len 0 old_flows_len 1
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 5019587570 bytes 3621247 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
linklayer ethernet overhead 22
Sent 5030519194 bytes 3621241 pkt (dropped 1, overlimits 127660 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 5030519194 bytes 3621241 pkt (dropped 1, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 65384 drop_overlimit 0 new_flow_count 48637 ecn_mark 0
new_flows_len 1 old_flows_len 6
### change to fq_codel_fast via WWW SQM UI ###
root@r7500v2:~# tc -d -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 4066316399 bytes 2764148 pkt (dropped 0, overlimits 0 requeues 287)
backlog 0b 0p requeues 287
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 4066316399 bytes 2764148 pkt (dropped 0, overlimits 0 requeues 287)
backlog 0b 0p requeues 287
maxpacket 1514 drop_overlimit 0 new_flow_count 543 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 6690 bytes 40 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 ver 3.17 direct_qlen 1000
linklayer ethernet overhead 22
Sent 34580 bytes 78 pkt (dropped 0, overlimits 5 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 120: dev eth0.2 parent 1:12 [Unknown qdisc, optlen=72]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 130: dev eth0.2 parent 1:13 [Unknown qdisc, optlen=72]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 110: dev eth0.2 parent 1:11 [Unknown qdisc, optlen=72]
Sent 34580 bytes 78 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 7284 bytes 65 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
linklayer ethernet overhead 22
Sent 9624 bytes 65 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 110: dev ifb4eth0.2 parent 1:10 [Unknown qdisc, optlen=72]
Sent 9624 bytes 65 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
### test result with fq_codel_fast ###
root@r7500v2:~# iperf -c XXX.XXX.45.137 -d -t60 -i10
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to XXX.XXX.45.137, TCP port 5001
TCP window size: 70.0 KByte (default)
------------------------------------------------------------
[ 4] local XXX.XXX.45.101 port 59432 connected with XXX.XXX.45.137 port 5001
[ 5] local XXX.XXX.45.101 port 5001 connected with XXX.XXX.45.137 port 49526
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 827 MBytes 693 Mbits/sec
[ 4] 0.0-10.0 sec 578 MBytes 485 Mbits/sec
[ 5] 10.0-20.0 sec 814 MBytes 683 Mbits/sec
[ 4] 10.0-20.0 sec 593 MBytes 497 Mbits/sec
[ 4] 20.0-30.0 sec 592 MBytes 496 Mbits/sec
[ 5] 20.0-30.0 sec 823 MBytes 690 Mbits/sec
[ 4] 30.0-40.0 sec 595 MBytes 499 Mbits/sec
[ 5] 30.0-40.0 sec 815 MBytes 684 Mbits/sec
[ 4] 40.0-50.0 sec 610 MBytes 512 Mbits/sec
[ 5] 40.0-50.0 sec 803 MBytes 674 Mbits/sec
[ 4] 50.0-60.0 sec 640 MBytes 537 Mbits/sec
[ 4] 0.0-60.0 sec 3.52 GBytes 504 Mbits/sec
[ 5] 50.0-60.0 sec 766 MBytes 642 Mbits/sec
[ 5] 0.0-60.0 sec 4.73 GBytes 678 Mbits/sec
[SUM] 0.0-60.0 sec 5.54 GBytes 793 Mbits/sec
root@r7500v2:~# lsmod | grep codel
sch_fq_codel_fast 20480 4
### some time passes...
root@r7500v2:~# tc -d -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 8027329338 bytes 5461195 pkt (dropped 0, overlimits 0 requeues 479)
backlog 0b 0p requeues 479
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 8027329338 bytes 5461195 pkt (dropped 0, overlimits 0 requeues 479)
backlog 0b 0p requeues 479
maxpacket 1514 drop_overlimit 0 new_flow_count 896 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth1 root
Sent 8778 bytes 52 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 8778 bytes 52 pkt (dropped 0, overlimits 0 requeues 1)
backlog 0b 0p requeues 1
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 ver 3.17 direct_qlen 1000
linklayer ethernet overhead 22
Sent 3795604065 bytes 2697125 pkt (dropped 0, overlimits 78116 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 120: dev eth0.2 parent 1:12 [Unknown qdisc, optlen=72]
Sent 3795490377 bytes 2696747 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 130: dev eth0.2 parent 1:13 [Unknown qdisc, optlen=72]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 110: dev eth0.2 parent 1:11 [Unknown qdisc, optlen=72]
Sent 113688 bytes 378 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 5101162997 bytes 3669944 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
linklayer ethernet overhead 22
Sent 5111825653 bytes 3669923 pkt (dropped 1, overlimits 130815 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel_fast 110: dev ifb4eth0.2 parent 1:10 [Unknown qdisc, optlen=72]
Sent 5111825653 bytes 3669923 pkt (dropped 1, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
I have done additional "playing" than that reported above (e.g. placing fq_codel_fast on eth0 and run some iperf tests both with/without SQM (with SQM using fq_codel or fq_codel_fast - simple.qos) - again no apparent issues here).
Is fq_codel_fast faster than fq_codel or does it use less cpu? Testing like this likely will not show it.
2 Likes
We should find a way to test if this version is actually better than fq_codel or cake
If we have some practical data I think openwrt won't have a problem including this
1 Like
someone with a >100 mbps isp connection might see a difference.
To artifically simulate a slow connection, I'm thinking about using a tbf qdisc for limiting the rate between the switch ports connecting the r7500v2 and the laptop (on a vlan) but failing miserably to get the vlan to talk with the rest of my network.
1 Like
i have a 140 mbps connection. But i currently have a mvebu router... Can i still see the difference?
1 Like
the purpose of the test is to find out...
but given @dtaht question about 10gigE nics, I think he's hoping to see more of benefit at higher throughput.
1 Like
well he had 5% with x86 hardware... could be higher with arm device
Anyway can you give me a guide on how to install this and test this?
1 Like
for install, you can cherry pick the commits "OOT experimental fast fq_codel shaper" and "adapt openwrt patches 620, 660, and 661" here (or you can just clone the repo - it's fairly up to date with master ATM). It's pretty minimal. You should just be able to build the kmod and install it as a package.
as far as testing goes, I'm still working through that. The other sqm experts on the forum will likely jump in with suggestions once there is something worthwhile to test.
After installing (verify the module is loaded with lsmod), I used openwrt SQM scripts:
touch /var/run/sqm/available_qdiscs/fq_codel_fast
and then selected fq_codel_fast (i used simple.qos) from the luci UI for SQM.
EDIT: I also tried:
tc qdisc add dev eth0 root handle 1: mq
tc qdisc add dev eth0 parent 1:1 fq_codel_fast
and then tried SQM both with and without fq_codel_fast. I had no purpose behind this other than to see if anything broke or crashed - nothing did.
1 Like
Well, the elephant in the room is typically the actual shaper, so either HTB, TBF or cake's inbuilt shaper. Once these are operating the small changes in efficiency of the fq leaf qdiscs easily disappear into the noise. I am not saying they can not be measured, but just that you should not expect miracles in typical shaping situations. That said, I am quite curious how it compares against the references.
2 Likes
The more I try to test on nmrh's cheep and cheerful home network (while the family is using the network), the more I think I am not going to demonstrate anything useful.
Here is my latest attempt, mostly to orient myself, and shared in the event others find parts of it helpful.
I don't recommend drawing any conclusions from the results below nor do I recommend testing this way.
### on a laptop running ubuntu: setting "350 up/down rate limit" & iperf server
modprobe ifb numifbs=1
sudo ip link set dev ifb0 up
sudo tc qdisc add dev eno1 handle ffff: ingress
sudo tc filter add dev eno1 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
sudo tc qdisc add dev ifb0 root tbf rate 350mbit burst 1mbit latency 400ms
sudo tc qdisc add dev eno1 root tbf rate 350mbit burst 1mbit latency 400ms
iperf -s
### on r7500v2, mpstat and iperf -c commands for test results below
mpstat 2 30 > cpu-$(date +"%Y%m%d-%H%M").log&
iperf -c XXX.XXX.40.10 -d -t60 -i10
### laptop: 350 up/down shaping; r7500v2: sqm, fq_codel_fast, cpu-20200520-2144.log;
### Note I got the sqm up/down limits from a prior test but see below for a test without sqm
root@r7500v2:~# cat /etc/config/sqm
config queue 'eth1'
option qdisc_advanced '0'
option interface 'eth0.2'
option debug_logging '0'
option verbosity '5'
option linklayer 'ethernet'
option overhead '22'
option script 'simple.qos'
option download '180000'
option upload '300000'
option qdisc 'fq_codel_fast'
option enabled '1'
[ 4] local XXX.XXX.40.10 port 5001 connected with XXX.XXX.40.5 port 56008
------------------------------------------------------------
Client connecting to XXX.XXX.40.5, TCP port 5001
TCP window size: 204 KByte (default)
------------------------------------------------------------
[ 6] local XXX.XXX.40.10 port 52728 connected with XXX.XXX.40.5 port 5001
[ 6] 0.0-60.0 sec 1.21 GBytes 173 Mbits/sec
[ 4] 0.0-60.0 sec 2.06 GBytes 294 Mbits/sec
Linux 5.4.41 (r7500v2) 05/20/20 _armv7l_ (2 CPU)
21:44:24 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 2.59 0.02 20.25 0.00 0.00 14.90 0.00 0.00 0.00 62.25
### laptop: 350 up/down shaping; r7500v2: sqm, fq_codel, cpu-20200520-2200.log
root@r7500v2:~# cat /etc/config/sqm
config queue 'eth1'
option qdisc_advanced '0'
option interface 'eth0.2'
option debug_logging '0'
option verbosity '5'
option linklayer 'ethernet'
option overhead '22'
option script 'simple.qos'
option download '180000'
option upload '300000'
option qdisc 'fq_codel'
option enabled '1'
[ 4] local XXX.XXX.40.10 port 5001 connected with XXX.XXX.40.5 port 56010
------------------------------------------------------------
Client connecting to XXX.XXX.40.5, TCP port 5001
TCP window size: 204 KByte (default)
------------------------------------------------------------
[ 6] local XXX.XXX.40.10 port 52912 connected with XXX.XXX.40.5 port 5001
[ 6] 0.0-60.0 sec 1.21 GBytes 173 Mbits/sec
[ 4] 0.0-60.1 sec 2.05 GBytes 294 Mbits/sec
Linux 5.4.41 (r7500v2) 05/20/20 _armv7l_ (2 CPU)
22:00:54 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 2.35 0.03 19.86 0.00 0.00 15.39 0.00 0.00 0.00 62.37
### laptop: 350 up/down shaping; r7500v2: no sqm, fq_codel, cpu-20200520-2219.log
[ 4] local XXX.XXX.40.10 port 5001 connected with XXX.XXX.40.5 port 56012
------------------------------------------------------------
Client connecting to XXX.XXX.40.5, TCP port 5001
TCP window size: 246 KByte (default)
------------------------------------------------------------
[ 6] local XXX.XXX.40.10 port 52982 connected with XXX.XXX.40.5 port 5001
[ 6] 0.0-60.0 sec 1.38 GBytes 198 Mbits/sec
[ 4] 0.0-60.1 sec 2.34 GBytes 334 Mbits/sec
Linux 5.4.41 (r7500v2) 05/20/20 _armv7l_ (2 CPU)
22:19:12 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 2.03 0.00 29.46 0.00 0.00 27.13 0.00 0.00 0.00 41.37
### laptop: no shaping; r7500v2: fq_codel_fast, no sqm, cpu-20200520-2242.log
[ 4] local XXX.XXX.40.10 port 5001 connected with XXX.XXX.40.5 port 56016
------------------------------------------------------------
Client connecting to XXX.XXX.40.5, TCP port 5001
TCP window size: 187 KByte (default)
------------------------------------------------------------
[ 6] local XXX.XXX.40.10 port 53094 connected with XXX.XXX.40.5 port 5001
[ 4] 0.0-60.0 sec 4.64 GBytes 664 Mbits/sec
[ 6] 0.0-60.0 sec 4.44 GBytes 636 Mbits/sec
Linux 5.4.41 (r7500v2) 05/20/20 _armv7l_ (2 CPU)
22:42:18 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 4.26 0.01 41.45 0.00 0.00 46.50 0.00 0.00 0.00 7.78
### laptop: no shaping; r7500v2: fq_codel, no sqm, cpu-20200520-2232.log
[ 4] local XXX.XXX.40.10 port 5001 connected with XXX.XXX.40.5 port 56014
------------------------------------------------------------
Client connecting to XXX.XXX.40.5, TCP port 5001
TCP window size: 382 KByte (default)
------------------------------------------------------------
[ 6] local XXX.XXX.40.10 port 52990 connected with XXX.XXX.40.5 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-60.0 sec 5.32 GBytes 762 Mbits/sec
[ 4] 0.0-60.0 sec 4.16 GBytes 596 Mbits/sec
Linux 5.4.41 (r7500v2) 05/20/20 _armv7l_ (2 CPU)
22:32:44 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 4.90 0.02 43.11 0.00 0.00 48.10 0.00 0.00 0.00 3.87
Asking myself why I got the results above leads me to believe testing between a laptop and the GW/AP with sqm does not simulate "the internet" well at all. Perhaps not enough buffers between the two devices, poor choice of configuration to rate limit the connection, or all of the above.
Comparing fq_codel with fq_codel_fast without sqm on a home network like this might be informative but under more controlled conditions (no family using the AP serving as the switch between the laptop and r7500v2) and with some statistics (lots of replicate experiments).
1 Like
My current level of (mis)understanding:
I am curious about
Since it looks like the return value in fq_codel_drop was set to avoid calling this part of the code in fq_codel_enqueue
if (ret == idx) {
qdisc_tree_reduce_backlog(sch, prev_qlen - 1,
prev_backlog - pkt_len);
return NET_XMIT_CN;
}
I did the following to see what would happen if that part of the code was called (I think as intended in fq_codel for dropping from the current flow):
--- a/sch_fq_codel_fast.c
+++ b/sch_fq_codel_fast.c
@@ -148,16 +148,16 @@ static unsigned int fq_codel_drop(struct
sch->qstats.drops += i;
sch->qstats.backlog -= len;
sch->q.qlen -= i;
- idx = 1055; // just ignore for now
+ // idx = 1055; // just ignore for now
// idx = (q->flows - q->fat_flow) >> FIXME SOME_CORRECT_DEFINE
- return idx;
+ return 0; // success
}
static int fq_codel_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
struct fq_codel_sched_data *q = qdisc_priv(sch);
- unsigned int idx, prev_backlog, prev_qlen;
+ unsigned int idx, prev_backlog, prev_qlen, drop_current_flow = 0;
struct fq_codel_flow *flow;
int uninitialized_var(ret);
unsigned int pkt_len;
@@ -215,6 +215,7 @@ static int fq_codel_enqueue(struct sk_bu
if(flow->backlog > q->fat_backlog) {
q->fat_flow = flow;
q->fat_backlog = flow->backlog;
+ drop_current_flow = 1;
}
if (list_empty(&flow->flowchain)) {
@@ -248,9 +249,10 @@ static int fq_codel_enqueue(struct sk_bu
* If we dropped a packet for this flow, return NET_XMIT_CN,
* but in this case, our parents wont increase their backlogs.
*/
- if (ret == idx) {
+ if (drop_current_flow) {
qdisc_tree_reduce_backlog(sch, prev_qlen - 1,
prev_backlog - pkt_len);
+ printk("dropping current flow idx: %u", idx);
return NET_XMIT_CN;
}
qdisc_tree_reduce_backlog(sch, prev_qlen, prev_backlog);
With the above patch, fq_codel_fast builds, loads, and runs on my r7500v2...
I can get a few drops (say 376 according to tc -d -s qdisc show dev ifb4eth0.2 after some runs with iperf) using fq_codel_fast but apparently not enough to trigger dropping from the current flow (i.e. no messages in dmesg). I guess with 1024 flows this should be expected.
Since I appear to be spamming the OP's thread (and I'm not the person he's looking for), I'm going to take a break from this for a while and hopefully some one else will take an interest.
HTH
1 Like