@uniqe13 @anon98444528 @amteza @ka2107 @nbd @vochong how stable and non-crashy are we now?
Oh, I think I misread what you did. It's the non cake'd ethernet bql instances that have the high bql limit. (?)
Personally I think this is worse for the pk_delay or peak delay which should show the maximum, but due to the EWMA the true maximum during a load test will to survive until after the load test. I would actually prefer a maximum counter that destructively resets at every tc -s stat dump, but veridically collects the true maximum, same for the (maximum delay of) sparse (flows). The average delay could stay as is IMHO.
1 Like
Not for the first time, I'm missing bugzilla and redmine. A simple pre-catagory of "crash bugs" to sort through (closed and open) would be helpful.
MT76: https://github.com/openwrt/openwrt/issues?q=is%3Aissue+mt76+
1 Like
Ah damn, I posted all, the results for eth2 are
774
63315
120652
75363
I also have a few vlans (but not on eth2), that's why there were more than twelve results.
few enough people use the cake stats in the first place and I agree the actual peak and a destructive read for peak would be better! @tohojo ?
1 Like
are you just running that single instance of cake unlimited on eth2? If so, the only bql in still in use is the first one. I think.
(to be clear despite my original instructions, when I originally looked at that slice of data (with more than 4 results) I assumed it was a sampled stat, then I assumed cake was on eth0, not eth2, now I am assuming that I have brain damage from trying to absorb too many openwrt bugs and find common causes....
2 Likes
My connection stability is much better after applying the patch from link. I not I have a errors ... received unexpected tx_fetch_ind event: in push mode error and ... failed to lookup txq for peer_id x tid x... this way there is no blocked clients and no degradation of wifi (or I don't see it yet... 
).
Some black magic bottom...
-----------------------------------------------------
OpenWrt SNAPSHOT, r19873-a703f9ed0b
-----------------------------------------------------
root@OpenWrt:~# qosify-status
===== interface WAN_4: active =====
egress status:
qdisc cake 8009: root refcnt 2 bandwidth 25Mbit diffserv4 dual-srchost nat wash no-ack-filter split-gso rtt 100ms noatm overhead 22
Sent 38785242 bytes 247611 pkt (dropped 4, overlimits 17699 requeues 87)
backlog 0b 0p requeues 87
memory used: 54976b of 4Mb
capacity estimate: 25Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 50 / 1522
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 1562Kbit 25Mbit 12500Kbit 6250Kbit
target 11.6ms 5ms 5ms 5ms
interval 107ms 100ms 100ms 100ms
pk_delay 510us 2.03ms 381us 141us
av_delay 60us 262us 73us 45us
sp_delay 37us 41us 22us 29us
backlog 0b 0b 0b 0b
pkts 4288 87090 152029 4208
bytes 1283265 15727473 21352923 426677
way_inds 1 92 1139 0
way_miss 257 2911 9072 58
way_cols 0 0 0 0
drops 0 2 2 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 1 0
bk_flows 0 0 1 0
un_flows 0 0 0 0
max_len 21658 14014 21658 6370
quantum 300 762 381 300
ingress status:
qdisc cake 800a: root refcnt 2 bandwidth 50Mbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 22
Sent 974951628 bytes 833995 pkt (dropped 238, overlimits 1006344 requeues 0)
backlog 0b 0p requeues 0
memory used: 519680b of 4Mb
capacity estimate: 50Mbit
min/max network layer size: 40 / 1500
min/max overhead-adjusted size: 62 / 1522
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 3125Kbit 50Mbit 25Mbit 12500Kbit
target 5.81ms 5ms 5ms 5ms
interval 101ms 100ms 100ms 100ms
pk_delay 1.64ms 2.18ms 213us 182us
av_delay 1.09ms 459us 54us 60us
sp_delay 37us 49us 20us 29us
backlog 0b 0b 0b 0b
pkts 24080 745057 62926 2170
bytes 28222530 933788563 12874075 369743
way_inds 29 406 256 0
way_miss 150 2809 4153 66
way_cols 0 0 0 0
drops 4 233 1 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 1 1 0
bk_flows 0 0 1 0
un_flows 0 0 0 0
max_len 1292 1514 1514 1274
quantum 300 1514 762 381
root@OpenWrt:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
26: 35451638 2481881 6790946 2031982 GIC-0 20 Level arch_timer
28: 0 0 0 0 GIC-0 23 Level arm-pmu
30: 10882 630 710 0 GIC-0 270 Level bam_dma
31: 87376 2330 6536 6349 GIC-0 127 Level 78b5000.spi
32: 0 0 0 0 GIC-0 239 Level bam_dma
33: 7 0 0 0 GIC-0 139 Level msm_serial0
50: 7427 114688 253923 3268218 GIC-0 200 Level ath10k_ahb
67: 12 0 0 0 GIC-0 201 Level ath10k_ahb
68: 16169 80630 88914 207935 GIC-0 97 Edge edma_eth_tx0
69: 124 298 405 13 GIC-0 98 Edge edma_eth_tx1
70: 0 0 0 0 GIC-0 99 Edge edma_eth_tx2
71: 0 0 0 0 GIC-0 100 Edge edma_eth_tx3
72: 19942 95563 169294 147461 GIC-0 101 Edge edma_eth_tx4
73: 2 73 6 0 GIC-0 102 Edge edma_eth_tx5
74: 0 0 0 0 GIC-0 103 Edge edma_eth_tx6
75: 0 0 0 0 GIC-0 104 Edge edma_eth_tx7
76: 58330 210649 95556 85852 GIC-0 105 Edge edma_eth_tx8
77: 10565 61151 84607 16261 GIC-0 106 Edge edma_eth_tx9
78: 0 0 0 0 GIC-0 107 Edge edma_eth_tx10
79: 0 0 0 0 GIC-0 108 Edge edma_eth_tx11
80: 12465 144638 68081 108108 GIC-0 109 Edge edma_eth_tx12
81: 147 346 58 29 GIC-0 110 Edge edma_eth_tx13
82: 0 0 0 0 GIC-0 111 Edge edma_eth_tx14
83: 0 0 0 0 GIC-0 112 Edge edma_eth_tx15
84: 7140 26087 119424 68167 GIC-0 272 Edge edma_eth_rx0
86: 17564 58185 42271 49291 GIC-0 274 Edge edma_eth_rx2
88: 7506 66550 57481 62441 GIC-0 276 Edge edma_eth_rx4
90: 16545 40751 58443 63577 GIC-0 278 Edge edma_eth_rx6
100: 0 0 0 0 GIC-0 163 Level qcom_dwc3 HS
101: 0 0 0 0 GIC-0 518 Level qcom_dwc3 SS
102: 0 0 0 0 GIC-0 520 Level qcom_dwc3 DM_HS
103: 0 0 0 0 GIC-0 521 Level qcom_dwc3 DP_HS
104: 0 0 0 0 GIC-0 167 Level qcom_dwc3 HS
105: 0 0 0 0 GIC-0 522 Level qcom_dwc3 SS
106: 0 0 0 0 GIC-0 524 Level qcom_dwc3 DM_HS
107: 0 0 0 0 GIC-0 525 Level qcom_dwc3 DP_HS
108: 0 0 0 0 msmgpio 63 Edge keys
109: 0 0 0 0 msmgpio 0 Edge keys
110: 330559 1189832 582509 134859 GIC-0 164 Level xhci-hcd:usb1
111: 0 0 0 0 GIC-0 168 Level xhci-hcd:usb3
IPI0: 0 0 0 0 CPU wakeup interrupts
IPI1: 0 0 0 0 Timer broadcast interrupts
IPI2: 4329 6082 5443 5807 Rescheduling interrupts
IPI3: 1271857 645576 759170 528753 Function call interrupts
IPI4: 0 0 0 0 CPU stop interrupts
IPI5: 751878 343889 417044 147325 IRQ work interrupts
IPI6: 0 0 0 0 completion interrupts
Err: 0
root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy0/netdev:wlan0/aqm
ac backlog-bytes backlog-packets new-flows drops marks overlimit collisions tx-bytes tx-packets
2 0 0 12218 0 0 0 4583 12478010 48194
root@OpenWrt:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.1 port 5001 connected with 192.168.1.192 port 52612
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.11 sec 85.9 MBytes 71.3 Mbits/sec
^Croot@OpenWrt:~#
speedtest.net is not good a today ;), 15-20Mbps on RJ45 and about 15-18 wireless of 2.4GHz...
but was a days in this week when was it ethernet=wireless of 2.4GHz in near (to 45-46Mbps) and 18-24Mbps in far distance... Pings are better than without AQL patch but sometimes they still jump a little (but unfortunately my connection to Internet is LTE still...)
There are still bugs...
[99546.844452] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[155569.299402] ess_edma c080000.edma: IPv6 not supported
[166998.297360] ess_edma c080000.edma: IPv6 not supported
[167669.746779] hrtimer: interrupt took 28354 ns
but what I see are not dangerous for me
Sorry for the confusion, here is the full output of tc -s qdisc show but not during the test.
tc -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 54848637 bytes 402962 pkt (dropped 0, overlimits 0 requeues 42)
backlog 0b 0p requeues 42
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 11563210 bytes 92523 pkt (dropped 0, overlimits 0 requeues 7)
backlog 0b 0p requeues 7
maxpacket 74 drop_overlimit 0 new_flow_count 3 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 11800636 bytes 97842 pkt (dropped 0, overlimits 0 requeues 6)
backlog 0b 0p requeues 6
maxpacket 94 drop_overlimit 0 new_flow_count 5 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 17824927 bytes 100687 pkt (dropped 0, overlimits 0 requeues 10)
backlog 0b 0p requeues 10
maxpacket 94 drop_overlimit 0 new_flow_count 4 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 13659864 bytes 111910 pkt (dropped 0, overlimits 0 requeues 19)
backlog 0b 0p requeues 19
maxpacket 195 drop_overlimit 0 new_flow_count 9 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8003: dev eth1 root refcnt 9 bandwidth unlimited diffserv4 triple-isolate nat wash no-ack-filter split-gso rtt 100ms noatm overhead 38 mpu 84
Sent 1568006804 bytes 1156756 pkt (dropped 0, overlimits 0 requeues 28)
backlog 0b 0p requeues 28
memory used: 13640b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 84 / 1538
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 9us 26us 15us
av_delay 0us 4us 14us 12us
sp_delay 0us 3us 7us 7us
backlog 0b 0b 0b 0b
pkts 0 1153176 552 3028
bytes 0 1567773588 85468 147748
way_inds 0 10534 0 0
way_miss 0 11003 5 25
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 0 1 0
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 7570 750 1462
quantum 1514 1514 1514 1514
qdisc cake 8004: dev eth2 root refcnt 9 bandwidth unlimited diffserv4 triple-isolate nat wash no-ack-filter split-gso rtt 100ms noatm overhead 38 mpu 84
Sent 1686662 bytes 12848 pkt (dropped 0, overlimits 0 requeues 5)
backlog 0b 0p requeues 5
memory used: 4608b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 84 / 1538
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 0bit 0bit 0bit 0bit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 0us 23us 27us 27us
av_delay 0us 11us 0us 12us
sp_delay 0us 5us 0us 6us
backlog 0b 0b 0b 0b
pkts 0 9986 8 2854
bytes 0 1560970 2626 123066
way_inds 0 1 0 0
way_miss 0 958 1 6
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 2 0 1
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 0 1514 336 345
quantum 1514 1514 1514 1514
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.20 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wg root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.7 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8013: dev pppoe-wan root refcnt 2 bandwidth 10Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34
Sent 27636518 bytes 220627 pkt (dropped 5, overlimits 160897 requeues 0)
backlog 0b 0p requeues 0
memory used: 126720b of 4Mb
capacity estimate: 10Mbit
min/max network layer size: 30 / 1492
min/max overhead-adjusted size: 64 / 1526
average network hdr offset: 0
Bulk Best Effort Voice
thresh 625Kbit 10Mbit 2500Kbit
target 29.1ms 5ms 7.27ms
interval 124ms 100ms 102ms
pk_delay 0us 4.61ms 471us
av_delay 0us 148us 102us
sp_delay 0us 16us 6us
backlog 0b 0b 0b
pkts 0 220206 426
bytes 0 27574516 67818
way_inds 0 617 0
way_miss 0 5972 46
way_cols 0 0 0
drops 0 5 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 1 1
bk_flows 0 1 0
un_flows 0 0 0
max_len 0 4476 576
quantum 300 305 300
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
Sent 898676224 bytes 636865 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8014: dev ifb4pppoe-wan root refcnt 2 bandwidth 54Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 34
Sent 897786029 bytes 636265 pkt (dropped 600, overlimits 1151875 requeues 0)
backlog 0b 0p requeues 0
memory used: 705024b of 4Mb
capacity estimate: 54Mbit
min/max network layer size: 28 / 1492
min/max overhead-adjusted size: 62 / 1526
average network hdr offset: 0
Tin 0
thresh 54Mbit
target 5ms
interval 100ms
pk_delay 1.55ms
av_delay 865us
sp_delay 9us
backlog 0b
pkts 636865
bytes 898676224
way_inds 8989
way_miss 7375
way_cols 0
drops 600
marks 0
ack_drop 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 1492
quantum 1514
eth0 is left unchanged, eth1+2 both have a single cake instance. The Performance test always ran on eth2.
Hi,
@dtaht, I just compiled version v22.03.0-rc5 (Unifi NanoHD / mt76) removing two patches as per @nbd request:
334-mac80211-add-a-per-PHY-AQL-limit-to-improve-fairness.patch
335-mac80211-add-debugfs-file-to-display-per-phy-AQL-pen.patch
Same scenario, my devices are idle except this macOS (Silicon 2x2 MIMO), awdl0 down (to avoid periodic scanning):
@reaper$ ➜ ~ ifconfig awdl0
awdl0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 76:60:97:60:8b:ee
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: inactive
Good WiFi connectivity:
Good ping:
PING openwrt.lan (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=3.276 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=6.343 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=6.214 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=3.049 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=6.681 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=6.037 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=7.082 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=6.676 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=6.314 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=4.219 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=6.119 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=6.158 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=6.185 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=6.222 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=6.440 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=3.133 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=6.351 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=6.311 ms
64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=6.450 ms
64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=6.191 ms
64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=6.340 ms
64 bytes from 192.168.1.1: icmp_seq=21 ttl=64 time=6.361 ms
64 bytes from 192.168.1.1: icmp_seq=22 ttl=64 time=3.325 ms
64 bytes from 192.168.1.1: icmp_seq=23 ttl=64 time=6.513 ms
64 bytes from 192.168.1.1: icmp_seq=24 ttl=64 time=6.335 ms
64 bytes from 192.168.1.1: icmp_seq=25 ttl=64 time=6.211 ms
64 bytes from 192.168.1.1: icmp_seq=26 ttl=64 time=7.090 ms
64 bytes from 192.168.1.1: icmp_seq=27 ttl=64 time=6.216 ms
64 bytes from 192.168.1.1: icmp_seq=28 ttl=64 time=3.363 ms
64 bytes from 192.168.1.1: icmp_seq=29 ttl=64 time=3.574 ms
64 bytes from 192.168.1.1: icmp_seq=30 ttl=64 time=6.296 ms
64 bytes from 192.168.1.1: icmp_seq=31 ttl=64 time=6.343 ms
64 bytes from 192.168.1.1: icmp_seq=32 ttl=64 time=6.414 ms
64 bytes from 192.168.1.1: icmp_seq=33 ttl=64 time=6.055 ms
64 bytes from 192.168.1.1: icmp_seq=34 ttl=64 time=6.343 ms
64 bytes from 192.168.1.1: icmp_seq=35 ttl=64 time=6.156 ms
64 bytes from 192.168.1.1: icmp_seq=36 ttl=64 time=6.307 ms
64 bytes from 192.168.1.1: icmp_seq=37 ttl=64 time=6.174 ms
64 bytes from 192.168.1.1: icmp_seq=38 ttl=64 time=6.321 ms
64 bytes from 192.168.1.1: icmp_seq=39 ttl=64 time=6.189 ms
64 bytes from 192.168.1.1: icmp_seq=40 ttl=64 time=5.814 ms
64 bytes from 192.168.1.1: icmp_seq=41 ttl=64 time=6.259 ms
64 bytes from 192.168.1.1: icmp_seq=42 ttl=64 time=6.300 ms
64 bytes from 192.168.1.1: icmp_seq=43 ttl=64 time=6.357 ms
64 bytes from 192.168.1.1: icmp_seq=44 ttl=64 time=6.473 ms
64 bytes from 192.168.1.1: icmp_seq=45 ttl=64 time=6.273 ms
64 bytes from 192.168.1.1: icmp_seq=46 ttl=64 time=6.300 ms
64 bytes from 192.168.1.1: icmp_seq=47 ttl=64 time=6.413 ms
64 bytes from 192.168.1.1: icmp_seq=48 ttl=64 time=6.139 ms
64 bytes from 192.168.1.1: icmp_seq=49 ttl=64 time=6.452 ms
64 bytes from 192.168.1.1: icmp_seq=50 ttl=64 time=6.311 ms
64 bytes from 192.168.1.1: icmp_seq=51 ttl=64 time=3.324 ms
64 bytes from 192.168.1.1: icmp_seq=52 ttl=64 time=6.395 ms
64 bytes from 192.168.1.1: icmp_seq=53 ttl=64 time=6.475 ms
64 bytes from 192.168.1.1: icmp_seq=54 ttl=64 time=6.507 ms
64 bytes from 192.168.1.1: icmp_seq=55 ttl=64 time=6.594 ms
64 bytes from 192.168.1.1: icmp_seq=56 ttl=64 time=3.588 ms
64 bytes from 192.168.1.1: icmp_seq=57 ttl=64 time=4.248 ms
64 bytes from 192.168.1.1: icmp_seq=58 ttl=64 time=6.372 ms
64 bytes from 192.168.1.1: icmp_seq=59 ttl=64 time=6.972 ms
64 bytes from 192.168.1.1: icmp_seq=60 ttl=64 time=6.401 ms
64 bytes from 192.168.1.1: icmp_seq=61 ttl=64 time=6.366 ms
64 bytes from 192.168.1.1: icmp_seq=62 ttl=64 time=5.819 ms
^C
--- openwrt.lan ping statistics ---
63 packets transmitted, 63 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.049/5.897/7.090/1.071 ms
Waveform results (1000/50 cable DOCSIS 3.1 connection with SQM configured at 312/47:
** Note: huge difference between upload and download, and bigger latency than with 22.02.1, but is it a tad more stable?
flent rrul_be -H openwrt.lan --step-size=.05 (click me to download)
And a 300 s run:
flent rrul_be -H openwrt.lan --step-size=.05 -l 300 (click me to download)
And just for a little more completeness, a run of Apple's tool:
@reaper$ ➜ ~ networkQuality -v
==== SUMMARY ====
Upload capacity: 31.799 Mbps
Download capacity: 229.290 Mbps
Upload flows: 20
Download flows: 12
Responsiveness: Medium (893 RPM)
Base RTT: 14
Start: 11/7/2022, 7:08:30 am
End: 11/7/2022, 7:08:45 am
OS Version: Version 12.4 (Build 21F79)
** Note: in general with version 22.02.1 I get over 3,000 RPM easy. Latency, to my understanding, stills worse after removing the patches suggested by @nbd and running v22.03.0-rc5.
And this is it. @nbd, let me know if you need me to compile something else. @dtaht, any other suggestions?
Kind regards, and apologies for taking so long Real Life TM is very demanding lately. 
1 Like
LGTM +1
Looking forward to hearing back from @quarky (and others) that were experiencing latency issues either after about a 4+ day wait or intermittent latency spikes when certain devices disconnect from their AP. These make take time to report back.
thx all for jumping all over this today! I'm encouraged, and as I noted I'll settle for "stable" and "doesn't crash"! for now. Ship it, get it out the door, silence all the other bugs. Selfishly I'd like to take august off.
BUT:
It does really seem like we have something going not-so-well with download speeds, and that RPM result was poor. Not blaming this patch set and looking at other possible causes in patches elsewhere, the kernel upgrade, etc, seems sane.
@amteza A) would it be possible for you to repeat that rrul_be -l 300 result while also taking a packet capture? Ideally with ECN enabled:
sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=1
An aircap is not needed, so that would be:
tcpdump -i your_interface -s 128 -w the_file.cap
B) a packet capture during a -l 60 --te=upload_streams=1 tcp_nup test
C) -l 60 --te=download_streams=1 tcp_ndown test
I don't trust osx very far and the long period latency swings you are seeing on panels 1 and 3, um... no. You can get at apple's internal fq_codel stats also, before and after the test.
netstat -I yourwifidevice -qq > somewhere.txt
2 Likes
The latest patch for mac80211 is looking good so far for my ath10k (21.02) and mt7622 (master) routers running it. No visible degradation of performance or latency from what I can remember with the pre-VTBA scheduler.
2 Likes
@dtaht, sure can.
See my /etc/sysctl.conf:
@reaper$ ➜ ~ cat /etc/sysctl.conf
net.inet.tcp.ecn_initiate_out=1
net.inet.tcp.ecn_negotiate_in=1
net.inet.tcp.disable_tcp_heuristics=1
And, confirmation of it working:
@reaper$ ➜ ~ sysctl net.inet.tcp.ecn_initiate_out
net.inet.tcp.ecn_initiate_out: 1
@reaper$ ➜ ~ sysctl net.inet.tcp.ecn_negotiate_in
net.inet.tcp.ecn_negotiate_in: 1
@reaper$ ➜ ~ sysctl net.inet.tcp.disable_tcp_heuristics
net.inet.tcp.disable_tcp_heuristics: 1
Here you go, results from A:
Results from B:
Summary of tcp_nup test run from 2022-07-11 01:12:41.369812
Title: 'flent_tcp_nup_60s_openwrt_v22.03.0-rc5_mt76'
avg median # data pts
Ping (ms) ICMP : 16.70 17.10 ms 1400
TCP upload avg : 573.40 N/A Mbits/s 1400
TCP upload sum : 573.40 N/A Mbits/s 1400
TCP upload::1 : 573.40 565.19 Mbits/s 1400
Results from C:
Summary of tcp_ndown test run from 2022-07-11 01:17:06.113491
Title: 'flent_tcp_ndown_60s_openwrt_v22.03.0-rc5_mt76'
avg median # data pts
Ping (ms) ICMP : 19.70 21.10 ms 1400
TCP download avg : 503.43 N/A Mbits/s 1400
TCP download sum : 503.43 N/A Mbits/s 1400
TCP download::1 : 503.43 514.51 Mbits/s 1400
And, of course, all the files to download in a nice package containing all your requests (1.41 GiB in total):
Note: I still run this morning rc5 with @nbd patches removed (334, 335).
@amteza After long debugging, I finally got fping and flent working on the Macbook. I searched through Flent github issues and came across https://github.com/tohojo/flent/issues/240#issuecomment-931256026 . There's no fping6 command in macOS so I instead tried:
% fping -D -c 1 localhost
[1657503443.62438] localhost : [0], timed out (NaN avg, 100% loss)
localhost : xmt/rcv/%loss = 1/0/100%
Turns out macOS firewall has a Stealth Mode (System Preferences -> Security & Privacy -> Firewall -> Firewall Options... -> Enable Stealth Mode). That option was enabled. After disabling "Enable Stealth Mode" I confirmed fping to localhost is working:
% fping -D -c 1 localhost
[1657503635.17864] localhost : [0], 64 bytes, 0.116 ms (0.116 avg, 0% loss)
localhost : xmt/rcv/%loss = 1/1/0%, min/avg/max = 0.116/0.116/0.116
After that change, flent works.
@amteza - could you find it in your heart to repeat these tests and packet captures using 22.02.1 so we have a baseline?
1 Like
@dtaht mumbai.starlink.taht.net is not working.
Flent Log https://gist.github.com/KA2107/2f668164476e6b83ab45c17533e46379
OS: macOS 12.4 Monterey
Grump. Try just ipv4, by adding -4 to your flent command line.
Still not working.
My apologies!! That was the old cluster that I had decommissioned. I have updated both the ipv4 and ipv6 addresses for mumbai to the new cluster, but it will take a while for the dns update to propagate.
You can direct your test at mumbai.v.taht.net for now.