TP-LINK Archer C60 v2 halting on OpenWrt 19.07.0

I have OpenVPN network with server at TP-Link Archer C-7 v5
and 6 clients TP-Link Archer C-60 v2
More than a year everythings worked perfectly at OpenWRT 18.06.x

Recently I decided upgrade to 19.07.0.
It was my big mistake, do not touch things that just works.

TP-Link C7(OpenVPN server) works further without problems.
But with at least two TP-Link Archer C60 v2 (OpenVPN clients) have issues:
It freezes(halts) a few times per day.
This two routers located in different cities and at different ISP.

Unfortunately, this two routers are remote for me and I can not have access to it, when problems happens.

What I know, when problems happens:
WiFi dies
Luci dies
Chrome browser from LAN host says: DNS_PROBE_FINISHED_BAD_CONFIG
SSH dies, I opened remote access to static IP.
But ping from LAN pass
And when I try ping router WAN external IP remotly ping pass

After reboot works normaly for a few hours.
But now it's freezes(halts) while I write this post.

ubus call system board
I installed only a few additional packets:
opkg list-installed
df -h

Sometimes I get error:

Out of memory

after opkg update

P.S. My emergency rescue plan fallback to 18.06.x but decided to write this topic before do it.

P.P.S. I think it obvious that I am not a native english speaker, how to right correctly freezes or halts?

I don't see anything wrong in the logs, so I would suggest to downgrade one of them to 18.06.6 and verify that it works without issues, then downgrade the other one too.
Maybe the memory after some time indeed gets consumed and you need a device with 128MB to work 19.07 version with luci enabled.

You do not see any errors in log because this logs after router reboot.

Last night this two routers downgraded to 18.06.06 and they works perfectly.
But now have the same issues with another two routers C-60 v2 on 19.07.0
Tonight will downgrade them too.

But as conclusion it is common problem with Archer C-60 v2.

May be should write warning about this in wiki page?

I guess we could write a warning for other users to be aware.
@tmomas any objections?

No objections, but it would be good to know the root cause of this issue.
Maybe kmod-ath10k-ct?

Does the same happen with the non-ct ath10k?
Does the same happen with ath10k-ct-smallbuffers?

@oslyak maybe you could try these versions of the module on the remaining 19.07 routers?

@trendy and @thomas yes, ok!

Replaced it at the first device

# opkg list-installed|grep ath10
ath10k-firmware-qca9888-ct - 2019-10-03-d622d160-1
kmod-ath10k-ct-smallbuffers - 4.14.162+2019-09-09-5e8cd86f-1

Now will wait for customer feedback or just watch uptime to see if somebody reboot it

Second router is in halting state, there is nobody to reboot it, for now.

By the way, with any of this modules:

  • kmod-ath10k-ct - 4.14.162+2019-09-09-5e8cd86f-1
  • kmod-ath10k-ct-smallbuffers - 4.14.162+2019-09-09-5e8cd86f-1

I get:

Is there anything in the logs that might provide an explanation?

I posted logs above or you need something else?

These logs were for the initial problem. Now you have tried the alternative modules, so new logs will be produced. dmesg and/or logread, whichever has more information.

Now i connected to router with 19.0.7 and kmod-ath10k-ct-smallbuffers
For simplicity hostname: rivne1

It is not rebooted for long time:

05:26:23 up 3 days, 22:39,  load average: 0.00, 0.00, 0.00

what is good ofcourse, but But 5Ghz is still broken.

root@rivne1:~# opkg list-installed| grep  ath10k
ath10k-firmware-qca9888-ct - 2019-10-03-d622d160-1
kmod-ath10k-ct-smallbuffers - 4.14.162+2019-09-09-5e8cd86f-1

ubus call system board

Now got connection to second router with 19.07.0 and original kmod-ath10k-ct
For simplicity hostname: rivne2

BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 OpenWrt 19.07.0, r10860-a3ffeb413b
root@rivne2:~# uptime 
 05:06:41 up 12:13,  load average: 0.01, 0.02, 0.00

root@rivne2:~# vim /etc/rc.local 
-ash: can't fork: Out of memory
root@rivne2:~# vim /etc/rc.local 
-ash: can't fork: Out of memory
root@rivne2:~# dmesg
-ash: can't fork: Out of memory
root@rivne2:~# reboot
-ash: can't fork: Out of memory

It has rebooted automatically, without any appropriate setting or manual actions.

After reboot.
LUCI shows me that's 5Ghz is works, but conduct real test with device connection I have not ability

#opkg list-installed|grep ath10k
ath10k-firmware-qca9888-ct - 2019-10-03-d622d160-1
kmod-ath10k-ct - 4.14.162+2019-09-09-5e8cd86f-1

after 20 minutes

# uptime
 05:31:47 up 20 min,  load average: 0.14, 0.05, 0.03

root@rivne2:~# opkg list|grep luci|grep base|grep en
Collected errors:
 * pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_routing: Out of memory.

But I have ability to gather info
ubus call system board
opkg list-installed

Check if /tmp is growing too big for your memory.
Other than a few firmware failures and country regulatory domains I don't see anything else wrong.
If I recall correctly, the ath10k can consume a lot of memory. Verify that and maybe a roll-back to 18.06.6 will solve your problems for now.

1 Like
[    8.014147] mount_root: switching to jffs2 overlay
[    8.038496] overlayfs: upper fs does not support tmpfile.

^ interesting ( not good ) this pops up so early...

These are likely not causal... aka... concur@memory
device page

[    0.901427] 0x000000030000-0x0000001b0000 : "kernel"
[    0.907725] 0x0000001b0000-0x0000007d0000 : "rootfs"
[    0.926556] 0x000000470000-0x0000007d0000 : "rootfs_data"

your log

[    0.894491] 0x000000030000-0x0000001b33f7 : "kernel"
[    0.900249] 0x0000001b33f7-0x0000007d0000 : "rootfs"
[    0.918265] 0x0000004a0000-0x0000007d0000 : "rootfs_data"
1 Like

How to verify memory, if free says thats memory is enafe, but commands like opkg shows: Out of memory?

P.S. I already rolled back to 18.06.6

Sorry, but it is nothing says to me.
I see different offset of rootfs_data and so?

With a df -h you can see how much space is left. In this case /tmp is actually part of the memory.

In my first post I showed

df -h

There you can see only one percent of /tmp memory being used.

You can monitor that in case it starts growing too much and the device becomes unresponsive.