That is correct. Actually the Ethernet driver has a lot of opportunities for optimization. Nobody’s working on it though.
What was your iperf3 command and where did you run it, and what was the topology?
Also, hardware revision?
Archer C7 v2. Just dual NAT on my network, iPerf3 to a server on the WAN side. I did a regular test and reverse test, both were about the same speeds. Upload maybe slightly slower at 870mbit
Impossible. Show me.
Perhaps that was with overclocking?
No overclocking. Router is currently being used in a temporary setup, I'll grab some screenshots next time I get a chance.
23.05.2: ath79: TP-Link Archer C7 v2: Fix wifi shutdown and "irq 23: nobody cared" error
Does anybody care? Just kidding, but is there any more detail on what it is beyond this?
I haven't seen it happen, though I'm not sure why.
it's a hardware problem. https://github.com/openwrt/openwrt/commit/8bc8db9ad26da98ebe2d4f861f03f09ece165694 has more details.
1 Like
Hi! Sorry for ghosting this discussion, I had some personal matters that took priority. Plus, I guess I either got used to the state of my Wi-Fi as it is now, or I wasn't using it very much, or... I dunno, but it just feels quite okay most of the time.
Anyways, below is my current config, test results and the output of tc as you requested.
tc output
root@Archer:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 945235163 bytes 1874680 pkt (dropped 0, overlimits 0 requeues 659)
backlog 0b 0p requeues 659
maxpacket 1514 drop_overlimit 0 new_flow_count 651 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8005: dev eth1 root refcnt 2 bandwidth 90Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 86170465 bytes 167026 pkt (dropped 1, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 55936b of 4500000b
capacity estimate: 90Mbit
min/max network layer size: 20 / 1180
min/max overhead-adjusted size: 20 / 1180
average network hdr offset: 14
Tin 0
thresh 90Mbit
target 5ms
interval 100ms
pk_delay 18us
av_delay 14us
sp_delay 12us
backlog 0b
pkts 167027
bytes 86170994
way_inds 133
way_miss 759
way_cols 0
drops 1
marks 0
ack_drop 0
sp_flows 3
bk_flows 1
un_flows 0
max_len 1180
quantum 1514
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4eth1 root refcnt 2 bandwidth 90Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 258 bytes 3 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
memory used: 704b of 4500000b
capacity estimate: 90Mbit
min/max network layer size: 86 / 86
min/max overhead-adjusted size: 86 / 86
average network hdr offset: 0
Tin 0
thresh 90Mbit
target 5ms
interval 100ms
pk_delay 9us
av_delay 0us
sp_delay 0us
backlog 0b
pkts 3
bytes 258
way_inds 0
way_miss 1
way_cols 0
drops 0
marks 0
ack_drop 0
sp_flows 1
bk_flows 0
un_flows 0
max_len 86
quantum 1514
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
I assume SQM is not working, because of the Sent 0 bytes 0 pkt bit. Should I have selected a different interface in the first tab of settings perhaps?
Other thing: I had installed the Attended Sysupgrade package and thought it would help me upgrade to the new release easily, but by default it doesn't do that (which makes me doubt what it even does when configured with its defaults). Anyways, does 23.05.2 work fine on it? Looking at two posts above, it seems there are some new issues? Should I postpone my upgrade for now?
Replying to my own question: I suppose I should have changed interface to eth0.2 (the "Switch VLAN" interface which holds the external IP) a long time ago. After I switched it, changing other settings (e.g. reducing upload and download speeds) finally had an effect.
Once I set both speeds to 90000, I think the sicq value topped at 98% instead of the previous 100%, and my internet speed was reduced. However, I don't see how that would be helpful in any way, since other than the unloaded latency, all other readings of the Bufferbloat test actually turned out slightly worse than the last time with SQM not working at all. Plus, the speed was reduced in half.
I guess I'll keep traffic shaping off until I actually start feeling the need for it. With the way we're currently using our internet connection, it just seems unnecessary.
Oh, and here is the `tc` output after running the test twice with different max speed configurations (180000kbps and 90000kbps
root@Archer:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 1183631966 bytes 2002070 pkt (dropped 0, overlimits 0 requeues 629)
backlog 0b 0p requeues 629
maxpacket 1514 drop_overlimit 0 new_flow_count 604 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 1269869 bytes 2396 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8011: dev eth0.2 root refcnt 2 bandwidth 90Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 361726286 bytes 339570 pkt (dropped 212, overlimits 72297 requeues 0)
backlog 0b 0p requeues 0
memory used: 560032b of 4500000b
capacity estimate: 90Mbit
min/max network layer size: 42 / 1514
min/max overhead-adjusted size: 42 / 1514
average network hdr offset: 14
Tin 0
thresh 90Mbit
target 5ms
interval 100ms
pk_delay 162us
av_delay 28us
sp_delay 12us
backlog 0b
pkts 339782
bytes 362034534
way_inds 282
way_miss 1299
way_cols 0
drops 212
marks 0
ack_drop 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 30931
quantum 1514
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 269095529 bytes 307911 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8012: dev ifb4eth0.2 root refcnt 2 bandwidth 90Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 273227824 bytes 307776 pkt (dropped 135, overlimits 167608 requeues 0)
backlog 0b 0p requeues 0
memory used: 558Kb of 4500000b
capacity estimate: 90Mbit
min/max network layer size: 60 / 1514
min/max overhead-adjusted size: 60 / 1514
average network hdr offset: 14
Tin 0
thresh 90Mbit
target 5ms
interval 100ms
pk_delay 8.86ms
av_delay 2.53ms
sp_delay 9us
backlog 0b
pkts 307911
bytes 273406283
way_inds 261
way_miss 1217
way_cols 0
drops 135
marks 0
ack_drop 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1514
quantum 1514
P.S. I also just upgraded to OpenWRT 23.05.2.
Very unstable in 23.05.2 with my Archer C7, rollbacked to 23.05.0
[ 139.273885] ath10k_pci 0000:00:00.0: firmware crashed! (guid 5403da4f-0a34-4403-8b73-67b0329c5190)
[ 139.283394] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000
[ 139.293832] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[ 139.306658] ath10k_pci 0000:00:00.0: firmware ver 10.1-ct-8x-__fH-022-ecad3248 api 2 features wmi-10.x,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT crc32 1b2a161c
[ 139.336374] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[ 139.344320] ath10k_pci 0000:00:00.0: htt-ver 2.2 wmi-op 2 htt-op 2 cal nvmem max-sta 128 raw 0 hwcrypto 1
[ 139.355388] ath10k_pci 0000:00:00.0: firmware register dump:
[ 139.361350] ath10k_pci 0000:00:00.0: [00]: 0x4100016C 0x00000000 0x0099483D 0xDEADC0EE
[ 139.369619] ath10k_pci 0000:00:00.0: [04]: 0x0099483D 0x00060130 0x00000000 0x00000002
[ 139.377773] ath10k_pci 0000:00:00.0: [08]: 0x00000000 0x009BB18C 0x00426360 0x0042D6F4
[ 139.386026] ath10k_pci 0000:00:00.0: [12]: 0x00000009 0x00000000 0x0095808C 0x009580A2
[ 139.394226] ath10k_pci 0000:00:00.0: [16]: 0x00958080 0x0094085D 0x00000000 0x00000000
[ 139.402454] ath10k_pci 0000:00:00.0: [20]: 0x4099483D 0x0040AC34 0x00000000 0x004DC0DE
[ 139.410707] ath10k_pci 0000:00:00.0: [24]: 0x809950AF 0x0040AC94 0x00000011 0xC099483D
[ 139.418950] ath10k_pci 0000:00:00.0: [28]: 0x809960A0 0x0040AD14 0x0041E524 0x00426360
[ 139.427104] ath10k_pci 0000:00:00.0: [32]: 0x80997EC9 0x0040ADB4 0x0041E524 0x00426360
[ 139.435343] ath10k_pci 0000:00:00.0: [36]: 0x809AF249 0x0040AEE4 0x00425918 0x00000002
[ 139.443646] ath10k_pci 0000:00:00.0: [40]: 0x80940F18 0x0040AF14 0x00000004 0x004039D0
[ 139.451851] ath10k_pci 0000:00:00.0: [44]: 0x80940EEA 0x0040AF44 0x00400000 0x00000000
[ 139.460142] ath10k_pci 0000:00:00.0: [48]: 0x80940F31 0x0040AF64 0x00401C10 0x00400600
[ 139.468299] ath10k_pci 0000:00:00.0: [52]: 0x40940024 0x0040AF84 0x004068E8 0x004068E8
[ 139.476424] ath10k_pci 0000:00:00.0: [56]: 0x00000000 0x0040AFA4 0x009BB001 0x00040020
[ 139.484669] ath10k_pci 0000:00:00.0: Copy Engine register dump:
[ 139.491191] ath10k_pci 0000:00:00.0: [00]: 0x00057400 7 7 3 3
[ 139.497748] ath10k_pci 0000:00:00.0: [01]: 0x00057800 17 17 19 20
[ 139.504329] ath10k_pci 0000:00:00.0: [02]: 0x00057c00 12 12 75 76
[ 139.510888] ath10k_pci 0000:00:00.0: [03]: 0x00058000 7 7 11 7
[ 139.517433] ath10k_pci 0000:00:00.0: [04]: 0x00058400 2521 2521 19 238
[ 139.524167] ath10k_pci 0000:00:00.0: [05]: 0x00058800 20 20 339 340
[ 139.530722] ath10k_pci 0000:00:00.0: [06]: 0x00058c00 26 26 26 26
[ 139.537268] ath10k_pci 0000:00:00.0: [07]: 0x00059000 0 0 0 0
[ 139.545539] ath10k_pci 0000:00:00.0: debug log header, dbuf: 0x411a98 dropped: 0
[ 139.553486] ath10k_pci 0000:00:00.0: [0] next: 0x411ab0 buf: 0x40f8fc sz: 1500 len: 28 count: 1 free: 0
[ 139.563481] ath10k_pci 0000:00:00.0: ath10k_pci ATH10K_DBG_BUFFER:
[ 139.569984] ath10k: [0000]: DE480200 0100FC17 00000000 0E1C0000 24AB4000 6C010041 01000000
With the identical configuration from the last v22, I'm seeing a MemAvailable:around 40000 kB in c7v2 23.02 (I didn't try 23.00) when historically it was in the upper 50's.
I also have a very similar router (WNDR3800) which sees no discernable difference on 23.02 from its historical average, so I'm a bit confused if this is going to be laid at the feet of "things are bigger."
What kind of instability did that make for? I looked at my kernel log on the same router with the new version, but no crashes or dumps. Is it possible that it was a fluke bad flash? Is there such a thing?
I don't know
I use the archer c7 as dumb ap, so the load is very low. I don't routing with this
There is two ssid is a wpa2/3 announced in 2.4 and 5ghz and I have fast roaming activated and around 15 clients (7 and 8 at this moment) one one ssid and 1 client on the second
Symptoms before firmware crash are packet lossing
After the firmware crash with a loop until the full reboot
So I come back this morning to 23.05.2 and I flashed non ct version, and it seems to be stable, and today I revert to full-htt and seems to be good
That's not an openwrt firmware bug.
@greearb is this something you could look at?
Let @djgreg13 know if you need any more info.
My logs still flood with the following on 23.05.2:
[567139.386960] ath10k_pci 0000:00:00.0: mac flush vdev 0 drop 0 queues 0x1 ar->paused: 0x0 arvif->paused: 0x0
[567552.684705] ath10k_pci 0000:00:00.0: mac flush vdev 0 drop 0 queues 0x1 ar->paused: 0x0 arvif->paused: 0x0
[568270.573062] ath10k_pci 0000:00:00.0: mac flush vdev 0 drop 0 queues 0x1 ar->paused: 0x0 arvif->paused: 0x0
| Hostname | asdf |
|---|---|
| Model | TP-Link Archer C7 v2 |
| Architecture | Qualcomm Atheros QCA9558 ver 1 rev 0 |
| Target Platform | ath79/generic |
| Firmware Version | OpenWrt 23.05.2 r23630-842932a63d / LuCI openwrt-23.05 branch git-23.306.39416-c86c256 |
| Kernel Version | 5.15.137 |
I check in my syslog
Already exists in 22.03.5
I don't have enough retention to see previous versions
One thing, that firmware image is quite a bit older than what we are using internally. I have not been trying to update openwrt or even upload new specific version patches to my site, but you can find more recent firmware linked from this page. If someone wants to grab my latest beta images and make them available to owrt somehow, please do. I'm not likely to spend the time to do that.
I'm also not planning to make any new changes to ath10k firmware except perhaps for relatively easy bug fixes in case someone reports something that I can fix quickly.
https://www.candelatech.com/ath10k-10.1.php
If that crashes, let me know and at least I can check to see if it is one of the known issues or not.
What ath10k-ct driver version are you using? I want to know how far back I need to go in my trees to comment out that verbose logging.
ath10k-firmware-qca988x-ct 2020-11-08-1
That's what openwrt shows. Dmesg shows
[ 45.555238] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000
[ 45.564663] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[ 45.576590] ath10k_pci 0000:00:00.0: firmware ver 10.1-ct-8x-__fW-022-ecad3248 api 2 features wmi-10.x,has-wmi-mgmt-tx,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,txrate-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT crc32 3e4cf97f
[ 45.689840] ath10k_pci 0000:00:00.0: Loading BDF type 0
[ 46.449674] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[ 47.425358] ath10k_pci 0000:00:00.0: 10.1 wmi init: vdevs: 16 peers: 127 tid: 256
[ 47.442031] ath10k_pci 0000:00:00.0: wmi print 'P 128 V 8 T 410'
[ 47.448355] ath10k_pci 0000:00:00.0: wmi print 'msdu-desc: 1424 sw-crypt: 0 ct-sta: 0'
[ 47.456513] ath10k_pci 0000:00:00.0: wmi print 'alloc rem: 24984 iram: 38672'
[ 47.506362] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 2 htt-op 2 cal nvmem max-sta 128 raw 0 hwcrypto 1
[ 47.519312] ath10k_pci 0000:00:00.0: NOTE: Firmware DBGLOG output disabled in debug_mask: 0x10000000
Hm, my log is full of these lines too. How bad are they?