@BeauSlim can you share also the loglines before and after the oops? There should be a hint towards where the NULL pointer dereference happened...
Yesterday evening I changed the SSID and password on the 2.4 GHz radio to fix a tasmota device configured for a previous setup. At 22:02 I changed it back to the new SSID:
Jul 29 19:58:23 ap-lsys-cc hostapd: wlan0: STA 5c:cf:7f:xx:xx:xx IEEE 802.11: authenticated
Jul 29 19:58:23 ap-lsys-cc hostapd: wlan0: STA 5c:cf:7f:xx:xx:xx IEEE 802.11: associated (aid 1)
Jul 29 19:58:23 ap-lsys-cc hostapd: wlan0: AP-STA-CONNECTED 5c:cf:7f:xx:xx:xx
Jul 29 19:58:23 ap-lsys-cc hostapd: wlan0: STA 5c:cf:7f:xx:xx:xx RADIUS: starting accounting session DB4D77548F26F79D
Jul 29 19:58:23 ap-lsys-cc hostapd: wlan0: STA 5c:cf:7f:xx:xx:xx WPA: pairwise key handshake completed (RSN)
Jul 29 19:58:23 ap-lsys-cc hostapd: wlan0: EAPOL-4WAY-HS-COMPLETED 5c:cf:7f:xx:xx:xx
Jul 29 20:02:09 ap-lsys-cc hostapd: wlan0: STA f8:cf:c5:xx:xx:xx IEEE 802.11: authenticated
Jul 29 20:02:09 ap-lsys-cc hostapd: wlan0: STA f8:cf:c5:xx:xx:xx IEEE 802.11: associated (aid 2)
Jul 29 20:02:09 ap-lsys-cc hostapd: wlan0: AP-STA-CONNECTED f8:cf:c5:xx:xx:xx
Jul 29 20:02:09 ap-lsys-cc hostapd: wlan0: STA f8:cf:c5:xx:xx:xx RADIUS: starting accounting session E83E62E81DE81FD6
Jul 29 20:02:09 ap-lsys-cc hostapd: wlan0: STA f8:cf:c5:xx:xx:xx WPA: pairwise key handshake completed (RSN)
Jul 29 20:02:09 ap-lsys-cc hostapd: wlan0: EAPOL-4WAY-HS-COMPLETED f8:cf:c5:xx:xx:xx
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: AP-STA-DISCONNECTED 24:a0:74:xx:xx:xx
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx IEEE 802.11: authenticated
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx IEEE 802.11: associated (aid 1)
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: AP-STA-CONNECTED 24:a0:74:xx:xx:xx
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx RADIUS: starting accounting session 28945744EBB4563A
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx WPA: pairwise key handshake completed (RSN)
Jul 29 20:08:51 ap-lsys-cc hostapd: wlan1: EAPOL-4WAY-HS-COMPLETED 24:a0:74:xx:xx:xx
Jul 29 20:34:32 ap-lsys-cc hostapd: wlan1: AP-STA-DISCONNECTED 24:a0:74:xx:xx:xx
Jul 29 20:34:32 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx IEEE 802.11: disassociated due to inactivity
Jul 29 20:34:33 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
Jul 29 20:40:10 ap-lsys-cc hostapd: wlan0: AP-STA-DISCONNECTED 5c:cf:7f:xx:xx:xx
Jul 29 20:40:10 ap-lsys-cc hostapd: wlan0: STA 5c:cf:7f:xx:xx:xx IEEE 802.11: disassociated due to inactivity
Jul 29 20:40:11 ap-lsys-cc hostapd: wlan0: STA 5c:cf:7f:xx:xx:xx IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
Jul 29 21:03:07 ap-lsys-cc hostapd: nl80211: kernel reports: key addition failed
Jul 29 21:03:07 ap-lsys-cc hostapd: wlan1: STA 24:a0:74:xx:xx:xx IEEE 802.11: associated (aid 1)
Jul 29 21:03:07 ap-lsys-cc hostapd: wlan1: AP-STA-CONNECTED 24:a0:74:xx:xx:xx
Jul 29 21:08:01 ap-lsys-cc hostapd: wlan1: AP-STA-DISCONNECTED 18:3e:ef:xx:xx:xx
Jul 29 21:08:01 ap-lsys-cc hostapd: wlan1: STA 18:3e:ef:xx:xx:xx IEEE 802.11: disassociated due to inactivity
Jul 29 21:08:02 ap-lsys-cc hostapd: wlan1: STA 18:3e:ef:xx:xx:xx IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
Jul 29 21:31:53 ap-lsys-cc hostapd: wlan1: STA 18:3e:ef:xx:xx:xx IEEE 802.11: authenticated
Jul 29 21:31:53 ap-lsys-cc hostapd: wlan1: STA 18:3e:ef:xx:xx:xx IEEE 802.11: associated (aid 2)
Jul 29 21:31:53 ap-lsys-cc hostapd: wlan1: AP-STA-CONNECTED 18:3e:ef:xx:xx:xx
Jul 29 21:31:53 ap-lsys-cc hostapd: wlan1: STA 18:3e:ef:xx:xx:xx RADIUS: starting accounting session 4139BD7DB31902DE
Jul 29 21:31:53 ap-lsys-cc hostapd: wlan1: STA 18:3e:ef:xx:xx:xx WPA: pairwise key handshake completed (RSN)
Jul 29 21:31:53 ap-lsys-cc hostapd: wlan1: EAPOL-4WAY-HS-COMPLETED 18:3e:ef:xx:xx:xx
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.147336] device wlan0 left promiscuous mode
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.151862] br-lan: port 3(wlan0) entered disabled state
Jul 29 22:02:06 ap-lsys-cc hostapd: Remove interface 'wlan0'
Jul 29 22:02:06 ap-lsys-cc hostapd: wlan0: interface state ENABLED->DISABLED
Jul 29 22:02:06 ap-lsys-cc hostapd: wlan0: AP-STA-DISCONNECTED f8:cf:c5:xx:xx:xx
Jul 29 22:02:06 ap-lsys-cc hostapd: wlan0: AP-DISABLED
Jul 29 22:02:06 ap-lsys-cc hostapd: wlan0: CTRL-EVENT-TERMINATING
Jul 29 22:02:06 ap-lsys-cc hostapd: rmdir[ctrl_interface=/var/run/hostapd]: Permission denied
Jul 29 22:02:06 ap-lsys-cc hostapd: nl80211: deinit ifname=wlan0 disabled_11b_rates=0
Jul 29 22:02:06 ap-lsys-cc hostapd: nl80211: Failed to remove interface wlan0 from bridge br-lan: Invalid argument
Jul 29 22:02:06 ap-lsys-cc netifd: Network device 'wlan0' link is down
Jul 29 22:02:06 ap-lsys-cc hostapd: Configuration file: /var/run/hostapd-phy0.conf (phy wlan0) --> new PHY
Jul 29 22:02:06 ap-lsys-cc netifd: Network device 'wlan0' link is up
Jul 29 22:02:06 ap-lsys-cc netifd: Network device 'wlan0' link is down
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.648284] br-lan: port 3(wlan0) entered blocking state
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.653613] br-lan: port 3(wlan0) entered disabled state
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.659227] device wlan0 entered promiscuous mode
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.664065] br-lan: port 3(wlan0) entered blocking state
Jul 29 22:02:06 ap-lsys-cc kernel: [75805.669482] br-lan: port 3(wlan0) entered listening state
Jul 29 22:02:06 ap-lsys-cc hostapd: wlan0: interface state UNINITIALIZED->COUNTRY_UPDATE
Jul 29 22:02:07 ap-lsys-cc netifd: Network device 'wlan0' link is up
Jul 29 22:02:07 ap-lsys-cc hostapd: wlan0: interface state COUNTRY_UPDATE->ENABLED
Jul 29 22:02:07 ap-lsys-cc hostapd: wlan0: AP-ENABLED
Jul 29 22:02:07 ap-lsys-cc kernel: [75805.727138] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Jul 29 22:02:09 ap-lsys-cc kernel: [75807.725775] br-lan: port 3(wlan0) entered learning state
Jul 29 22:02:11 ap-lsys-cc kernel: [75809.805778] br-lan: port 3(wlan0) entered forwarding state
Jul 29 22:02:11 ap-lsys-cc kernel: [75809.811270] br-lan: topology change detected, sending tcn bpdu
Jul 29 22:13:48 ap-lsys-cc kernel: [76507.624097] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000053
There is nothing in the logs after the already posted Oops line until I manually restarted the device at 11 AM this morning.
I guess the oops cause the device to reboot into recovery mode (like it should in such a case). Can you extract (hopefully) more complete logs from /sys/fs/pstore?
edit: you find the device on IP 192.168.1.1/24 in recovery mode and should be able to login using SSH root user without password.
Im dont think its a good idea to automatically boot to recovery after kernel crash as it causes confusion.
How to go from recovery to normal image again? Is a rebooty by luci or power toggle enough?
You just have to clear PSTORE and reboot, then everything will be back to normal. This can either be done using rm /sys/fs/pstore/* or by disconnecting the device from power for a short moment, so DRAM content will be cleared.
I think this is extremely useful, especially in snapshot images as kernel oops will not go unnoticed in that way and users for the first time are able to report meaningful things back to us even if the device' serial console is not connected during the crash. If you don't like this, it's also simple to tell U-Boot not to care about PSTORE at all, simply change the bootcmd variable from inside OpenWrt using fw_printenv/fw_setenv.
For a production release things will have to be a bit different, of course. One option would be to let the recovery image handle and clear PSTORE automatically, ie. upload logs to a URL configured in U-Boot environment or store them on an additional UBI storage volume.
And, of course, we need clear indication in LuCI that we are currently running recovery, it'd also be great to have a notification about logs being present in PSTORE and be able to view and clear them in LuCI. I can provide a JSON-RPC interface for PSTORE ops if anyone is willing to implement the front-end part (I'm not into web/front-end stuff at all, ie. graphical stuff makes me feel lost and angry, I hate using web-browsers and prefer everything in a simple text-mode console myself, ncurses is far as it gets with UI in my case, at least I can still use that without having to use the mouse).
1 Like
kernel oops will not go unnoticed in that way
As you see it results only in "wlan did not work anymore until i power cycled" 
I would wonder too if the device "boots itself" into recovery
What about to check if /sys/fs/pstore/* exists, and then show an additional line "crash logs found" or so in Luci-Overview, maybe red? Just by if (file-exist-condition) fields.push('output' in feeds/luci/modules/luci-mod-status/htdocs/luci-static/resources/view/status/include/*.js
But only if its possible to keep /sys/fs/pstore/* when not booting to recovery automatically.
Btw, if some of my access-points boot automatically to recovery and get the default ip (=same as the router) 192.168.1.1 it will disable whole internet access and it would some tome to notice its not a problem of the router
Additional:
For my device on the Overview page in luci it show "Linksys E8450" or "Linksys E8450 (UBI)", depending. It would be great if the Revovery shows as "Linksys E8450 (UBI-Recovery)", so i could get it directly 
I have already receive many useful PSTORE dumps in the past. Ok, sometimes it took an initial confusion for users to understand what just happened, but I still believe it's worth it even if only a fraction of users manage to extract and submit logs from PSTORE.
PSTORE generally doesn't get lost unless you manually clear it or unplug power for a few seconds. The problem with not booting into recovery if there is something in PSTORE is the potential of triggering (costly, in terms of device lifetime) infinite reboot loops in case something crashes early during boot.
Also, as a useful side-effect, users can decide to manually boot into recovery using echo c > /proc/sysrq-trigger.
Regarding LuCI suggestions: there is https://github.com/openwrt/luci/pull/5041 in order to provide a generic infrastructure to display notifications of all kinds to the user in LuCI.
Regarding default IP in recovery mode: I was thinking to implement a way to store settings relevant to recovery in U-Boot env, or simply use the existing ipaddr from U-Boot env also for OpenWrt when booting into recovery.
In the meantime, maybe move your router away from 192.168.1.0/24 subnet...
When the device does not boot automatically into recovery, but shows a red notification user could still send the logs. The PR is interesting, but the discussion luci/global will still take some time. Just add a line to overview is done fast (and deleted later also, if needed)
To prevent a reboot-loop a counter could be added. Something like just a number in /sys/fs/pstore/crashcount and if some limit is exceeded fall back to recovery.
I dont want to change my routers ip, its this since my 1st lan! 
So i change the default of my images ip with files/etc/config file. Only recovery does not use this
Is $ fw_setenv bootcmd 'run boot_ubi' okay to change? Does holding the reset button durin power up then still work to get into recovery?
A reboot counter would not be enough, it'd need to be with timestamps (also tricky without RTC) to be able to recognize "early" reboots and what is "early" anyway (in seconds or ms)? I've only seen broken implementations of that approach for now...
Counting the number of records in pstore already works, but I don't see how it would be more transparent/easy for users to understand if their device hangs in recovery after 5 crashes instead of after the first time it happens. It will just delay the problem and keep users unaware of problems (unless they manually check /sys/fs/pstore or have an eye on uptime).
And yes, to not have U-Boot check PSTORE the change to U-Boot environment you stated works as expected (and you will still be able to manually trigger recovery or tftpboot by holding down RESET button during boot).
2 Likes
Thats simple: run in some startup script rm -f /sys/fs/pstore/crashcount on (every) sucessfull reboot (recovery + installed)
Not go to recovery after 1st crash helps if somthing after a long runtime goes wrong.
The boot-loop case is if eg in the kernel is something wrong and the device is not able to boot at all
In theory that's a good solution. However, it'd require an additional pstore record (crashcount) to be handled by the kernel -- for now, this is all just vanilla Linux features without patching anything related to pstore, but just using it as-is. If you think this is easy to do, please submit patches to upstream Linux and OpenWrt lists.
Imho it'd be easier to have an init-script which handles pstore in recovery, clears it and reboots (according to settings it finds in U-Boot env). I've just been to lazy to implement that (but it's on the list).
1 Like
Sorry, I had moved the device (unplugged it) before seeing your reply and I see nothing under /sys/fs/pstore. But I now know better for next time.
hello @daniel
my colleague has the same router as me but he lost the openwrt wifi on his router, he uses sqm for the video game and suddenly found himself in moderate nat,
instead of nat open, but with the difference that this time the router did not go back to the original blue interface as in the past, it disconnected 10SEC and reconnected, and the wifi came back, this problem could it be recurrent thank you?
ps I don't have the version of its software
The best would be if you manage to extract the logs from /sys/fs/pstore next time it happens, so we will be able to reproduce the cause (I didn't manage to crash it even once, but that can well be related to the behavior of wifi clients; on MT7620 there were problems which could only be triggered by WiFi action frames emitted by an Xbox.... you get the idea...).
If you that is not an option for you and you just want crashes to silently reboot the router and keep things functional, you can also disable the PSTORE feature in the U-Boot environment (see above).
Ah well seen Daniel he is on PS5, not Xbox I told him to unplug in case, like you for the moment mine has no problem 
Where is the best place for uploading pstore files? I have a couple. The device boots into recovery frequently. Finally read this thread about pstore files and how to clear them. Good stuff.
I have two Belkin RT3200 can't get mesh working. I just played around using the tutorial that I used in the past without success. Could someone post a working config to help me, please. The openwrt tutorial has worked in the past with other devices but does not work with the Belkin RT3200. I used the network/wireless setup and the command line examples. Using latest snapshot with wpad-mesh-wolfssl.
iw dev mesh0 station dump
iw dev mesh0 mpath dump
With the above two command there is no output. Log file looks like the mesh0 interface is created. Not sure if it is a belkin problem or something that I am doing wrong. Probably the later.
The tutorial that I have followed is https://openwrt.org/docs/guide-user/network/wifi/mesh/80211s
I have looked at other tutorials also but no positive results.
good evening everyone does upnp work on rt3200 it does not seem to want to make it work because i have xbox and ps5 several machines, thanks in advance
1 Like
This guide got mine going:
3 Likes
anyone can say if you are done upnp on linksys e8450 rt3200 doesnt work at my home for the moment
thanks
1 Like
Thanks for the links @elan and @smileys29 and I actually followed all the tutorials you listed before my previous post, the last article posted is excellent. I guess I should have listed all the links I reviewed. I think there is a bigger problem. I been using trying belkin rt3200 to another belkin rt3200 and mesh setup won't work even a simple setup. Decided to try Belkin rt3200 to Netgear r7800 and mesh network came up on the first try. Exact same setup. Will try Batman mesh with the Belkin to Netgear r7800 to see if that works. I have to try to figure this out. I am stumped after several days. Will try to turn on some debugging output to get more information.
Noted kernel crash when bringing the network/wifi up and down.
Pstore shows this.
Unable to handle kernel NULL pointer dereference at virtual address 000000000000000c
<1>[ 1827.151352] Mem abort info:
<1>[ 1827.154139] ESR = 0x96000005
<1>[ 1827.157186] EC = 0x25: DABT (current EL), IL = 32 bits
<1>[ 1827.162488] SET = 0, FnV = 0
<1>[ 1827.165532] EA = 0, S1PTW = 0
<1>[ 1827.168663] Data abort info:
<1>[ 1827.171533] ISV = 0, ISS = 0x00000005
<1>[ 1827.175359] CM = 0, WnR = 0
<1>[ 1827.178318] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000041858000
<1>[ 1827.184747] [000000000000000c] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
<0>[ 1827.193443] Internal error: Oops: 96000005 [#1] SMP
<7>[ 1827.198310] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat batman_adv xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc sch_cake nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables hwmon crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
<7>[ 1827.198480] ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs nfs_ssc ifb rpcsec_gss_krb5 auth_rpcgss oid_registry lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax sha1_generic seqiv md5 des_generic libdes cts usb_storage leds_gpio xhci_plat_hcd fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd gpio_button_hotplug cbc encrypted_keys trusted tpm rng_core
<7>[ 1827.338782] CPU: 1 PID: 930 Comm: ubusd Tainted: G S 5.10.54 #0
<7>[ 1827.345994] Hardware name: Linksys E8450 (UBI) (DT)
<7>[ 1827.350864] pstate: 60000085 (nZCv daIf -PAN -UAO -TCO BTYPE=--)
<7>[ 1827.356866] pc : account_kernel_stack+0xa0/0x114
<7>[ 1827.361474] lr : put_task_stack+0x5c/0xbc
<7>[ 1827.365473] sp : ffffffc01171bbf0
<7>[ 1827.368778] x29: ffffffc01171bbf0 x28: 0000000000000000
<7>[ 1827.374082] x27: 0000000000000000 x26: 0000000000000000
<7>[ 1827.379386] x25: ffffff8000d25650 x24: ffffff8000b85228
<7>[ 1827.384689] x23: 0000000000000000 x22: 0000000000000080
<7>[ 1827.389994] x21: ffffff80002fe300 x20: ffffff80016c4e00
<7>[ 1827.395298] x19: 0000000000000000 x18: 0000000000000000
<7>[ 1827.400601] x17: 0000000000000000 x16: 0000000000000000
<7>[ 1827.405905] x15: 0000000000000000 x14: 0000000000000000
<7>[ 1827.411209] x13: 00000000000000db x12: 0000000000000000
<7>[ 1827.416512] x11: ffffffc010a38540 x10: 0000000000000820
<7>[ 1827.421816] x9 : ffffff801fed2510 x8 : fffffffeffead008
<7>[ 1827.427120] x7 : ffffffc00f49a000 x6 : 0000000000000000
<7>[ 1827.432423] x5 : 0000000000000000 x4 : 0000000000000000
<7>[ 1827.437726] x3 : ffffff80002feb18 x2 : 00000000fffffff0
<7>[ 1827.443030] x1 : 0000000000000060 x0 : 0000000000000004
<7>[ 1827.448334] Call trace:
<7>[ 1827.450772] account_kernel_stack+0xa0/0x114
<7>[ 1827.455033] put_task_stack+0x5c/0xbc
<7>[ 1827.458689] finish_task_switch+0xe4/0x134
<7>[ 1827.462778] __schedule+0x1f8/0x454
<7>[ 1827.466257] schedule+0x44/0xcc
<7>[ 1827.469391] schedule_hrtimeout_range_clock+0x110/0x120
<7>[ 1827.474607] schedule_hrtimeout_range+0x10/0x20
<7>[ 1827.479132] do_epoll_wait+0x4d0/0x564
<7>[ 1827.482873] __arm64_sys_epoll_pwait+0x54/0xc4
<7>[ 1827.487309] el0_svc_common.constprop.0+0x60/0x100
<7>[ 1827.492090] do_el0_svc+0x18/0x20
<7>[ 1827.495396] el0_svc+0x10/0x20
<7>[ 1827.498441] el0_sync_handler+0xa8/0xac
<7>[ 1827.502268] el0_sync+0x148/0x180
<0>[ 1827.505578] Code: 17fffff3 d503201f d2800c01 d50342df (f9400401)
<4>[ 1827.511663] ---[ end trace e8144481bd796d18 ]---
Thanks for replying.
Update. Finally got a ping going with mesh and this is what happened on both belkin rt3200 after pinging the netgear. All three routers crashed.
[ 568.970407] WARNING: CPU: 0 PID: 1104 at __skb_flow_dissect+0x1a4/0x1130
[ 568.977098] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat batman_adv xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc sch_cake nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables hwmon crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
[ 568.977271] ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs nfs_ssc ifb rpcsec_gss_krb5 auth_rpcgss oid_registry lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax sha1_generic seqiv md5 des_generic libdes cts usb_storage leds_gpio xhci_plat_hcd fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd gpio_button_hotplug cbc encrypted_keys trusted tpm rng_core
[ 569.117578] CPU: 0 PID: 1104 Comm: napi/phy0-5 Tainted: G S 5.10.54 #0
[ 569.125397] Hardware name: Linksys E8450 (UBI) (DT)
[ 569.130266] pstate: 40000005 (nZcv daif -PAN -UAO -TCO BTYPE=--)
[ 569.136266] pc : __skb_flow_dissect+0x1a4/0x1130
[ 569.140874] lr : __skb_get_hash+0x70/0x130
[ 569.144960] sp : ffffffc010d6b980
[ 569.148265] x29: ffffffc010d6b980 x28: ffffff8002399280
[ 569.153570] x27: 0000000000000008 x26: 0000000000000000
[ 569.158873] x25: ffffff8001839a18 x24: 000000000000009c
[ 569.164177] x23: ffffffc010d6bb00 x22: 000000000000ffbe
[ 569.169481] x21: ffffffc010a49580 x20: 0000000000000000
[ 569.174785] x19: ffffff80018aa800 x18: ffffffc010de0410
[ 569.180089] x17: 0000000000000080 x16: 000000000000096c
[ 569.185394] x15: 0000000000000000 x14: 0000000000000000
[ 569.190697] x13: 02ff000000040300 x12: 0000c2d8008f0001
[ 569.196001] x11: 00000205003a1600 x10: ffffff80018aa300
[ 569.201305] x9 : ffffff80034a08c0 x8 : ffffff8000b9d800
[ 569.206608] x7 : 0000000000000000 x6 : 0000000000000000
[ 569.211912] x5 : 0000000000000000 x4 : 0000000000000000
[ 569.217216] x3 : ffffffc010d6bb00 x2 : 000000000000ffd6
[ 569.222519] x1 : 0000000000000000 x0 : 0000000000000000
[ 569.227823] Call trace:
[ 569.230263] __skb_flow_dissect+0x1a4/0x1130
[ 569.234524] __skb_get_hash+0x70/0x130
[ 569.238289] ieee80211_schedule_txq+0x794/0xb8c [mac80211]
[ 569.243776] ieee80211_schedule_txq+0xb24/0xb8c [mac80211]
[ 569.249262] ieee80211_tx_pending+0xdc/0x240 [mac80211]
[ 569.254480] tasklet_action_common.constprop.0+0x15c/0x170
[ 569.259957] tasklet_action+0x24/0x30
[ 569.263609] _stext+0x124/0x294
[ 569.266742] do_softirq+0x74/0x80
[ 569.270047] __local_bh_enable_ip+0x88/0x90
[ 569.274224] napi_threaded_poll+0x94/0xf0
[ 569.278227] kthread+0x120/0x124
[ 569.281446] ret_from_fork+0x10/0x18
[ 569.285011] ---[ end trace 0c23f40535c1c7fa ]---
Similar on second belking rt3200
[ 561.719982] ------------[ cut here ]------------
[ 561.724617] WARNING: CPU: 0 PID: 1104 at __skb_flow_dissect+0x1a4/0x1130
[ 561.731307] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat batman_adv xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc sch_cake nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables hwmon crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
[ 561.731478] ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs nfs_ssc ifb rpcsec_gss_krb5 auth_rpcgss oid_registry lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax sha1_generic seqiv md5 des_generic libdes cts usb_storage leds_gpio xhci_plat_hcd fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd gpio_button_hotplug cbc encrypted_keys trusted tpm rng_core
[ 561.871781] CPU: 0 PID: 1104 Comm: napi/phy0-5 Tainted: G S 5.10.54 #0
[ 561.879599] Hardware name: Linksys E8450 (UBI) (DT)
[ 561.884469] pstate: 40000005 (nZcv daif -PAN -UAO -TCO BTYPE=--)
[ 561.890468] pc : __skb_flow_dissect+0x1a4/0x1130
[ 561.895076] lr : __skb_get_hash+0x70/0x130
[ 561.899162] sp : ffffffc010d5b980
[ 561.902468] x29: ffffffc010d5b980 x28: ffffff8002311280
[ 561.907772] x27: 0000000000000008 x26: 0000000000000000
[ 561.913076] x25: ffffff8002bd4a18 x24: 0000000000000044
[ 561.918380] x23: ffffffc010d5bb00 x22: 000000000000ffbe
[ 561.923684] x21: ffffffc010a49580 x20: 0000000000000000
[ 561.928988] x19: ffffff8000a74800 x18: ffffffc010ddb4d0
[ 561.934292] x17: 0000000000000080 x16: 000000000000096c
[ 561.939595] x15: 0000000000000000 x14: 0000000000000080
[ 561.944899] x13: 0000000000000070 x12: ffffffc010830610
[ 561.950202] x11: 0000000000000001 x10: 0000000000000001
[ 561.955506] x9 : ffffff8002b928c0 x8 : ffffff8000294200
[ 561.960810] x7 : 0000000000000000 x6 : 0000000000000000
[ 561.966113] x5 : 0000000000000000 x4 : 0000000000000000
[ 561.971417] x3 : ffffffc010d5bb00 x2 : 000000000000ffd6
[ 561.976721] x1 : 0000000000000000 x0 : 0000000000000000
[ 561.982026] Call trace:
[ 561.984465] __skb_flow_dissect+0x1a4/0x1130
[ 561.988726] __skb_get_hash+0x70/0x130
[ 561.992493] ieee80211_schedule_txq+0x794/0xb8c [mac80211]
[ 561.997979] ieee80211_schedule_txq+0xb24/0xb8c [mac80211]
[ 562.003465] ieee80211_tx_pending+0xdc/0x240 [mac80211]
[ 562.008684] tasklet_action_common.constprop.0+0x15c/0x170
[ 562.014160] tasklet_action+0x24/0x30
[ 562.017813] _stext+0x124/0x294
[ 562.020945] do_softirq+0x74/0x80
[ 562.024251] __local_bh_enable_ip+0x88/0x90
[ 562.028428] napi_threaded_poll+0x94/0xf0
[ 562.032431] kthread+0x120/0x124
[ 562.035650] ret_from_fork+0x10/0x18
[ 562.039215] ---[ end trace 771e410f6f1a456b ]---
This is the error on the netgear r7800. All of the routers on the mesh crashed. Don't know if this helps but wanted to post it also.
[ 101.961765] ------------[ cut here ]------------
[ 101.961822] WARNING: CPU: 0 PID: 0 at net/core/flow_dissector.c:960 0xc0782264
[ 101.965484] Modules linked in: pppoe ppp_async iptable_nat batman_adv ath10k_pci ath10k_core ath xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG slhc nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 crc32c_generic leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug
[ 102.024836] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.124 #0
[ 102.047043] Hardware name: Generic DT based system
[ 102.053037] Function entered at [<c030f854>] from [<c030b870>]
[ 102.057634] Function entered at [<c030b870>] from [<c08ffe18>]
[ 102.063451] Function entered at [<c08ffe18>] from [<c031e75c>]
[ 102.069266] Function entered at [<c031e75c>] from [<c031e7c8>]
[ 102.075081] Function entered at [<c031e7c8>] from [<c0782264>]
[ 102.080899] Function entered at [<c0782264>] from [<c0783600>]
[ 102.086712] Function entered at [<c0783600>] from [<bf1e6bec>]
[ 102.092551] Function entered at [<bf1e6bec>] from [<bf1ec360>]
[ 102.098347] Function entered at [<bf1ec360>] from [<bf1edd3c>]
[ 102.104163] Function entered at [<bf1edd3c>] from [<c03226e0>]
[ 102.109977] Function entered at [<c03226e0>] from [<c0302298>]
[ 102.115793] Function entered at [<c0302298>] from [<c0322b44>]
[ 102.121608] Function entered at [<c0322b44>] from [<c036f830>]
[ 102.127426] Function entered at [<c036f830>] from [<c05dcbb4>]
[ 102.133242] Function entered at [<c05dcbb4>] from [<c0301a8c>]
[ 102.139057] Exception stack(0xc0c01ee0 to 0xc0c01f28)
[ 102.144883] 1ee0: 00000000 00000017 1ce50000 dd991a80 dcc58800 00000000 dd990e30 00000017
[ 102.150008] 1f00: 00000017 00000000 bd62b2c0 bd57dee0 00000015 c0c01f30 c0730e78 c0730e7c
[ 102.158160] 1f20: 80000013 ffffffff
[ 102.166312] Function entered at [<c0301a8c>] from [<c0730e7c>]
[ 102.169614] Function entered at [<c0730e7c>] from [<c07312c4>]
[ 102.175515] Function entered at [<c07312c4>] from [<c034ac4c>]
[ 102.181333] Function entered at [<c034ac4c>] from [<c034af5c>]
[ 102.187148] Function entered at [<c034af5c>] from [<c0b00db4>]
[ 102.193044] ---[ end trace 5e8e6f1b7fce010d ]---
I think I found the reason for going into recovery. Will try something different in the morning.
Hope this helps.
2 Likes