Belkin RT3200/Linksys E8450 WiFi AX discussion

bobbythomas · August 26, 2021, 7:25am

Hi Daniel,

Please find the output below.

root@OpenWrt:~# cat /sys/fs/pstore/*
Oops#1 Part1
<5>[    7.338224] UBIFS (ubi0:5): UBIFS: mounted UBI device 0, volume 5, name "rootfs_data"
<5>[    7.346056] UBIFS (ubi0:5): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
<5>[    7.355979] UBIFS (ubi0:5): FS size: 91295744 bytes (87 MiB, 719 LEBs), journal size 4571136 bytes (4 MiB, 36 LEBs)
<5>[    7.366410] UBIFS (ubi0:5): reserved for root: 4312123 bytes (4211 KiB)
<5>[    7.373017] UBIFS (ubi0:5): media format: w5/r0 (latest is w5/r0), UUID 3015F212-EFD6-4299-9110-AB7B1C57E37D, small LPT model
<14>[    7.513643] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[    7.523378] block: extroot: not configured
<14>[    7.528842] mount_root: switching to ubifs overlay
<6>[    7.536458] overlayfs: "xino" feature enabled using 2 upper inode bits.
<15>[    7.545939] urandom-seed: Seeding with /etc/urandom.seed
<6>[    7.589196] mt7530 mdio-bus:00 lan1: Link is Down
<14>[    7.598434] procd: - early -
<14>[    7.601352] procd: - watchdog -
<14>[    8.144788] procd: - watchdog -
<14>[    8.149874] procd: - ubus -
<14>[    8.203933] procd: - init -
<14>[    8.497159] urngd: v1.0.2 started.
<14>[    8.568882] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[    8.600475] Loading modules backported from Linux version v5.10.42-0-g65859eca4dff
<6>[    8.608094] Backport generated by backports.git v5.10.42-1-0-gbee5c545
<6>[    8.627198] xt_time: kernel timezone is -0000
<7>[    8.683467] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
<7>[    8.694264] mt7915e 0000:01:00.0: assign IRQ: got 144
<7>[    8.699463] pci 0000:00:00.0: enabling bus mastering
<6>[    8.704450] mt7915e 0000:01:00.0: enabling device (0000 -> 0002)
<7>[    8.710564] mt7915e 0000:01:00.0: enabling bus mastering
<6>[    8.746674] mt7622-wmac 18000000.wmac: HW/SW Version: 0x8a108a10, Build Time: 20190801210006a
<6>[    8.746674]
<6>[    8.778842] mt7915e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 20201105222230a
<6>[    8.778842]
<6>[    8.847393] mt7622-wmac 18000000.wmac: N9 Firmware Version: 2.0, Build Time: 20200131180931
<6>[    8.912506] mt7915e 0000:01:00.0: WM Firmware Version: ____000000, Build Time: 20201105222304
<6>[    8.937183] mt7915e 0000:01:00.0: WA Firmware Version: DEV_000000, Build Time: 20201105222323
<6>[    9.061285] PPP generic driver version 2.4.2
<6>[    9.066766] NET: Registered protocol family 24
<14>[    9.078466] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[   11.727781] mtk_soc_eth 1b100000.ethernet eth0: Link is Down
<6>[   11.736732] mtk_soc_eth 1b100000.ethernet eth0: configuring for fixed/2500base-x link mode
<6>[   11.745239] mtk_soc_eth 1b100000.ethernet eth0: Link is Up - 2.5Gbps/Full - flow control rx/tx
<6>[   11.753947] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[   11.768929] mt7530 mdio-bus:00 lan1: configuring for phy/gmii link mode
<6>[   11.777996] 8021q: adding VLAN 0 to HW filter on device lan1
<6>[   11.786055] br-lan: port 1(lan1) entered blocking state
<6>[   11.791400] br-lan: port 1(lan1) entered disabled state
<6>[   11.798499] device lan1 entered promiscuous mode
<6>[   11.803122] device eth0 entered promiscuous mode
<6>[   11.821101] mt7530 mdio-bus:00 lan2: configuring for phy/gmii link mode
<6>[   11.835735] 8021q: adding VLAN 0 to HW filter on device lan2
<6>[   11.849040] br-lan: port 2(lan2) entered blocking state
<6>[   11.854269] br-lan: port 2(lan2) entered disabled state
<6>[   11.896612] device lan2 entered promiscuous mode
<6>[   11.913070] mt7530 mdio-bus:00 lan3: configuring for phy/gmii link mode
<6>[   11.924644] 8021q: adding VLAN 0 to HW filter on device lan3
<6>[   11.945679] br-lan: port 3(lan3) entered blocking state
<6>[   11.951030] br-lan: port 3(lan3) entered disabled state
<6>[   11.973057] device lan3 entered promiscuous mode
<6>[   11.989451] mt7530 mdio-bus:00 lan4: configuring for phy/gmii link mode
<6>[   12.002761] 8021q: adding VLAN 0 to HW filter on device lan4
<6>[   12.028403] br-lan: port 4(lan4) entered blocking state
<6>[   12.033644] br-lan: port 4(lan4) entered disabled state
<6>[   12.041889] device lan4 entered promiscuous mode
<6>[   12.054669] mt7530 mdio-bus:00 wan: configuring for phy/gmii link mode
<6>[   12.062419] 8021q: adding VLAN 0 to HW filter on device wan
<6>[   14.787495] br-lan: port 5(wlan1) entered blocking state
<6>[   14.792824] br-lan: port 5(wlan1) entered disabled state
<6>[   14.798399] device wlan1 entered promiscuous mode
<6>[   14.803234] br-lan: port 5(wlan1) entered blocking state
<6>[   14.808612] br-lan: port 5(wlan1) entered forwarding state
<6>[   14.814995] br-lan: port 5(wlan1) entered disabled state
<6>[   14.930986] mt7530 mdio-bus:00 lan1: Link is Up - 1Gbps/Full - flow control off
<6>[   14.938413] br-lan: port 1(lan1) entered blocking state
<6>[   14.943664] br-lan: port 1(lan1) entered forwarding state
<6>[   14.952336] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
<6>[   15.015310] br-lan: port 6(wlan0) entered blocking state
<6>[   15.020718] br-lan: port 6(wlan0) entered disabled state
<6>[   15.026305] device wlan0 entered promiscuous mode
<6>[   15.031145] br-lan: port 6(wlan0) entered blocking state
<6>[   15.036514] br-lan: port 6(wlan0) entered forwarding state
<6>[   15.886372] br-lan: port 6(wlan0) entered disabled state
<6>[   22.870463] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[   22.876977] br-lan: port 6(wlan0) entered blocking state
<6>[   22.882291] br-lan: port 6(wlan0) entered forwarding state
<6>[   35.207122] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
<6>[   35.213608] br-lan: port 5(wlan1) entered blocking state
<6>[   35.218942] br-lan: port 5(wlan1) entered forwarding state
<1>[   71.169440] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000053
<1>[   71.178258] Mem abort info:
<1>[   71.181043]   ESR = 0x96000005
<1>[   71.184089]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[   71.189405]   SET = 0, FnV = 0
<1>[   71.192449]   EA = 0, S1PTW = 0
<1>[   71.195579] Data abort info:
<1>[   71.198458]   ISV = 0, ISS = 0x00000005
<1>[   71.202283]   CM = 0, WnR = 0
<1>[   71.205243] user pgtable: 4k pages, 39-bit VAs, pgdp=000000004140d000
<1>[   71.211681] [0000000000000053] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
<0>[   71.220455] Internal error: Oops: 96000005 [#1] SMP
<7>[   71.225356] Modules linked in: pppoe ppp_async iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG slhc nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 ipvlan iptable_mangle iptable_filter ip_tables hwmon crc_ccitt compat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 vfat fat autofs4 nls_utf8 nls_iso8859_1 nls_cp437 seqiv uas usb_storage leds_gpio xhci_plat_hcd gpio_button_hotplug
<7>[   71.286861] CPU: 0 PID: 1157 Comm: napi/phy1-9 Tainted: G S                5.10.58 #0
<7>[   71.294680] Hardware name: Linksys E8450 (UBI) (DT)
<7>[   71.299549] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
<7>[   71.305557] pc : mt76_tx_status_skb_add+0x1b0/0x230 [mt76]
<7>[   71.311035] lr : mt76_tx_status_skb_done+0x14/0x20 [mt76]
<7>[   71.316422] sp : ffffffc010e23c10
<7>[   71.319726] x29: ffffffc010e23c10 x28: ffffff800221a800
<7>[   71.325031] x27: 00000000000036c0 x26: ffffff8002a6a300
<7>[   71.330335] x25: 0000000040000000 x24: 0000000000000000
<7>[   71.335640] x23: ffffff80038f7848 x22: ffffff800187bbb8
<7>[   71.340945] x21: 0000000000000000 x20: ffffff80038f7828
<7>[   71.346249] x19: ffffff800221a800 x18: 00000000419b6780
<7>[   71.351553] x17: 0000000000000000 x16: 0000000000000001
<7>[   71.356858] x15: 000000000000c5c0 x14: ffffff8002280000
<7>[   71.362162] x13: 00000000000062e0 x12: 0000000000000000
<7>[   71.367466] x11: 0000000000000000 x10: ffffffc010a89570
<7>[   71.372771] x9 : ffffff801fec0530 x8 : 0000000003000000
<7>[   71.378076] x7 : ffffff80022c0e80 x6 : 00000000ffffa688
<7>[   71.383380] x5 : 00000000ffffa698 x4 : 0000000000000000
<7>[   71.388684] x3 : ffffffc010e23d18 x2 : 0000000000000002
<7>[   71.393989] x1 : 0000000000000000 x0 : ffffff800221a800
<7>[   71.399294] Call trace:
<7>[   71.401735]  mt76_tx_status_skb_add+0x1b0/0x230 [mt76]
<7>[   71.406868]  mt7915_queue_rx_skb+0xbcc/0x1598 [mt7915e]
<7>[   71.412087]  mt76_dma_rx_poll+0x2bc/0xc98 [mt76]
<7>[   71.416700]  __napi_poll+0x38/0x170
<7>[   71.420180]  napi_threaded_poll+0x80/0xe8
<7>[   71.424185]  kthread+0x120/0x128
<7>[   71.427407]  ret_from_fork+0x10/0x18
<0>[   71.430978] Code: a94153f3 a8c27bfd d65f03c0 d503201f (39414c24)
<4>[   71.437062] ---[ end trace 92fbbbfe12521edd ]---
Panic#2 Part1
<5>[    7.355979] UBIFS (ubi0:5): FS size: 91295744 bytes (87 MiB, 719 LEBs), journal size 4571136 bytes (4 MiB, 36 LEBs)
<5>[    7.366410] UBIFS (ubi0:5): reserved for root: 4312123 bytes (4211 KiB)
<5>[    7.373017] UBIFS (ubi0:5): media format: w5/r0 (latest is w5/r0), UUID 3015F212-EFD6-4299-9110-AB7B1C57E37D, small LPT model
<14>[    7.513643] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
<14>[    7.523378] block: extroot: not configured
<14>[    7.528842] mount_root: switching to ubifs overlay
<6>[    7.536458] overlayfs: "xino" feature enabled using 2 upper inode bits.
<15>[    7.545939] urandom-seed: Seeding with /etc/urandom.seed
<6>[    7.589196] mt7530 mdio-bus:00 lan1: Link is Down
<14>[    7.598434] procd: - early -
<14>[    7.601352] procd: - watchdog -
<14>[    8.144788] procd: - watchdog -
<14>[    8.149874] procd: - ubus -
<14>[    8.203933] procd: - init -
<14>[    8.497159] urngd: v1.0.2 started.
<14>[    8.568882] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[    8.600475] Loading modules backported from Linux version v5.10.42-0-g65859eca4dff
<6>[    8.608094] Backport generated by backports.git v5.10.42-1-0-gbee5c545
<6>[    8.627198] xt_time: kernel timezone is -0000
<7>[    8.683467] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
<7>[    8.694264] mt7915e 0000:01:00.0: assign IRQ: got 144
<7>[    8.699463] pci 0000:00:00.0: enabling bus mastering
<6>[    8.704450] mt7915e 0000:01:00.0: enabling device (0000 -> 0002)
<7>[    8.710564] mt7915e 0000:01:00.0: enabling bus mastering
<6>[    8.746674] mt7622-wmac 18000000.wmac: HW/SW Version: 0x8a108a10, Build Time: 20190801210006a
<6>[    8.746674]
<6>[    8.778842] mt7915e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 20201105222230a
<6>[    8.778842]
<6>[    8.847393] mt7622-wmac 18000000.wmac: N9 Firmware Version: 2.0, Build Time: 20200131180931
<6>[    8.912506] mt7915e 0000:01:00.0: WM Firmware Version: ____000000, Build Time: 20201105222304
<6>[    8.937183] mt7915e 0000:01:00.0: WA Firmware Version: DEV_000000, Build Time: 20201105222323
<6>[    9.061285] PPP generic driver version 2.4.2
<6>[    9.066766] NET: Registered protocol family 24
<14>[    9.078466] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[   11.727781] mtk_soc_eth 1b100000.ethernet eth0: Link is Down
<6>[   11.736732] mtk_soc_eth 1b100000.ethernet eth0: configuring for fixed/2500base-x link mode
<6>[   11.745239] mtk_soc_eth 1b100000.ethernet eth0: Link is Up - 2.5Gbps/Full - flow control rx/tx
<6>[   11.753947] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[   11.768929] mt7530 mdio-bus:00 lan1: configuring for phy/gmii link mode
<6>[   11.777996] 8021q: adding VLAN 0 to HW filter on device lan1
<6>[   11.786055] br-lan: port 1(lan1) entered blocking state
<6>[   11.791400] br-lan: port 1(lan1) entered disabled state
<6>[   11.798499] device lan1 entered promiscuous mode
<6>[   11.803122] device eth0 entered promiscuous mode
<6>[   11.821101] mt7530 mdio-bus:00 lan2: configuring for phy/gmii link mode
<6>[   11.835735] 8021q: adding VLAN 0 to HW filter on device lan2
<6>[   11.849040] br-lan: port 2(lan2) entered blocking state
<6>[   11.854269] br-lan: port 2(lan2) entered disabled state
<6>[   11.896612] device lan2 entered promiscuous mode
<6>[   11.913070] mt7530 mdio-bus:00 lan3: configuring for phy/gmii link mode
<6>[   11.924644] 8021q: adding VLAN 0 to HW filter on device lan3
<6>[   11.945679] br-lan: port 3(lan3) entered blocking state
<6>[   11.951030] br-lan: port 3(lan3) entered disabled state
<6>[   11.973057] device lan3 entered promiscuous mode
<6>[   11.989451] mt7530 mdio-bus:00 lan4: configuring for phy/gmii link mode
<6>[   12.002761] 8021q: adding VLAN 0 to HW filter on device lan4
<6>[   12.028403] br-lan: port 4(lan4) entered blocking state
<6>[   12.033644] br-lan: port 4(lan4) entered disabled state
<6>[   12.041889] device lan4 entered promiscuous mode
<6>[   12.054669] mt7530 mdio-bus:00 wan: configuring for phy/gmii link mode
<6>[   12.062419] 8021q: adding VLAN 0 to HW filter on device wan
<6>[   14.787495] br-lan: port 5(wlan1) entered blocking state
<6>[   14.792824] br-lan: port 5(wlan1) entered disabled state
<6>[   14.798399] device wlan1 entered promiscuous mode
<6>[   14.803234] br-lan: port 5(wlan1) entered blocking state
<6>[   14.808612] br-lan: port 5(wlan1) entered forwarding state
<6>[   14.814995] br-lan: port 5(wlan1) entered disabled state
<6>[   14.930986] mt7530 mdio-bus:00 lan1: Link is Up - 1Gbps/Full - flow control off
<6>[   14.938413] br-lan: port 1(lan1) entered blocking state
<6>[   14.943664] br-lan: port 1(lan1) entered forwarding state
<6>[   14.952336] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
<6>[   15.015310] br-lan: port 6(wlan0) entered blocking state
<6>[   15.020718] br-lan: port 6(wlan0) entered disabled state
<6>[   15.026305] device wlan0 entered promiscuous mode
<6>[   15.031145] br-lan: port 6(wlan0) entered blocking state
<6>[   15.036514] br-lan: port 6(wlan0) entered forwarding state
<6>[   15.886372] br-lan: port 6(wlan0) entered disabled state
<6>[   22.870463] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[   22.876977] br-lan: port 6(wlan0) entered blocking state
<6>[   22.882291] br-lan: port 6(wlan0) entered forwarding state
<6>[   35.207122] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
<6>[   35.213608] br-lan: port 5(wlan1) entered blocking state
<6>[   35.218942] br-lan: port 5(wlan1) entered forwarding state
<1>[   71.169440] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000053
<1>[   71.178258] Mem abort info:
<1>[   71.181043]   ESR = 0x96000005
<1>[   71.184089]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[   71.189405]   SET = 0, FnV = 0
<1>[   71.192449]   EA = 0, S1PTW = 0
<1>[   71.195579] Data abort info:
<1>[   71.198458]   ISV = 0, ISS = 0x00000005
<1>[   71.202283]   CM = 0, WnR = 0
<1>[   71.205243] user pgtable: 4k pages, 39-bit VAs, pgdp=000000004140d000
<1>[   71.211681] [0000000000000053] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
<0>[   71.220455] Internal error: Oops: 96000005 [#1] SMP
<7>[   71.225356] Modules linked in: pppoe ppp_async iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG slhc nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 ipvlan iptable_mangle iptable_filter ip_tables hwmon crc_ccitt compat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 vfat fat autofs4 nls_utf8 nls_iso8859_1 nls_cp437 seqiv uas usb_storage leds_gpio xhci_plat_hcd gpio_button_hotplug
<7>[   71.286861] CPU: 0 PID: 1157 Comm: napi/phy1-9 Tainted: G S                5.10.58 #0
<7>[   71.294680] Hardware name: Linksys E8450 (UBI) (DT)
<7>[   71.299549] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
<7>[   71.305557] pc : mt76_tx_status_skb_add+0x1b0/0x230 [mt76]
<7>[   71.311035] lr : mt76_tx_status_skb_done+0x14/0x20 [mt76]
<7>[   71.316422] sp : ffffffc010e23c10
<7>[   71.319726] x29: ffffffc010e23c10 x28: ffffff800221a800
<7>[   71.325031] x27: 00000000000036c0 x26: ffffff8002a6a300
<7>[   71.330335] x25: 0000000040000000 x24: 0000000000000000
<7>[   71.335640] x23: ffffff80038f7848 x22: ffffff800187bbb8
<7>[   71.340945] x21: 0000000000000000 x20: ffffff80038f7828
<7>[   71.346249] x19: ffffff800221a800 x18: 00000000419b6780
<7>[   71.351553] x17: 0000000000000000 x16: 0000000000000001
<7>[   71.356858] x15: 000000000000c5c0 x14: ffffff8002280000
<7>[   71.362162] x13: 00000000000062e0 x12: 0000000000000000
<7>[   71.367466] x11: 0000000000000000 x10: ffffffc010a89570
<7>[   71.372771] x9 : ffffff801fec0530 x8 : 0000000003000000
<7>[   71.378076] x7 : ffffff80022c0e80 x6 : 00000000ffffa688
<7>[   71.383380] x5 : 00000000ffffa698 x4 : 0000000000000000
<7>[   71.388684] x3 : ffffffc010e23d18 x2 : 0000000000000002
<7>[   71.393989] x1 : 0000000000000000 x0 : ffffff800221a800
<7>[   71.399294] Call trace:
<7>[   71.401735]  mt76_tx_status_skb_add+0x1b0/0x230 [mt76]
<7>[   71.406868]  mt7915_queue_rx_skb+0xbcc/0x1598 [mt7915e]
<7>[   71.412087]  mt76_dma_rx_poll+0x2bc/0xc98 [mt76]
<7>[   71.416700]  __napi_poll+0x38/0x170
<7>[   71.420180]  napi_threaded_poll+0x80/0xe8
<7>[   71.424185]  kthread+0x120/0x128
<7>[   71.427407]  ret_from_fork+0x10/0x18
<0>[   71.430978] Code: a94153f3 a8c27bfd d65f03c0 d503201f (39414c24)
<4>[   71.437062] ---[ end trace 92fbbbfe12521edd ]---
<0>[   71.447848] Kernel panic - not syncing: Oops: Fatal exception in interrupt
<2>[   71.454719] SMP: stopping secondary CPUs
<0>[   71.458637] Kernel Offset: disabled
<0>[   71.462118] CPU features: 0x0000002,04002004
<0>[   71.466377] Memory Limit: none
root@OpenWrt:~#

Regards,
Bob

geminis3 · August 26, 2021, 7:28am

I'm not familiar with OpenWrt's codebase but it looks like the problem lies in the mt76 WiFi driver.

hnyman · August 26, 2021, 7:34am

Sounds familiar. Similar errors in message 244 (and the next few days, a month ago) in this thread.

bobbythomas · August 26, 2021, 9:19am

This is what I see when I check the ACU status from cli.

Am I doing something wrong here?

bobbythomas · August 26, 2021, 9:25am

Is there a way to return to snapshot firmware from recovery without performing a power cycle. It's causing issues with my network. I know that these are nightly snapshots but just trying to identify what's wrong here and what work around we can apply to keep this running.

daniel · August 26, 2021, 11:37am

maybe also change /etc/config/attendedsysupgrade to use asu.aparcar.org in case chef.libremesh.org gives you trouble.

daniel · August 26, 2021, 11:39am

Yes, you can just
rm /sys/fs/pstore/* ; reboot
inside recovery, that will reboot into production firmware.

bobbythomas · August 26, 2021, 11:56am

Thanks, I will give this a try.

This is very helpful. I will keep this in mind.

Thank you.

bobbythomas · August 26, 2021, 12:17pm

Can someone confirm if you can download the snapshots from asu.aparcar.org manually using browser? I am having hard time downloading the snapshot image from asu.aparcar.org, I tried flashing the sysupgrade image from asu.aparcar.org but it didn't have luci builtin and when I installed luci it went back to old gui and I am also having some issues with network connectivity and latency.

fonix232 · August 26, 2021, 1:21pm

I've been having trouble with the latest builds (r17415+) from ASU servers. It seemingly installs fine, but after reboot, it's unresponsive (power LED solid, internet LED is off, and it neither receives an IP from upstream, nor serves IPs downstream), even to the reset long-hold. The only way to fix it was to power off, hold reset while booting, wait for recovery to kick in, and flash a clean firmware. After many, many experiments, I managed to get a build working, but it was very barebones (basically I only added luci and luci-theme-openwrt-2020 to it, and replaced the wolfssl implementations with openssl - including swapping wpad-basic-wolfssl with wpad-openssl).

@bobbythomas the "old GUI" is just a theme. If you install the package luci-theme-openwrt-2020 you'll get the new style. I don't know why the new theme isn't being used as default over the old bootstrap one.

Also, don't expect the ASU servers to be fast. It checks every 5 seconds, but a build can take up to 3-4 minutes to complete.

bobbythomas · August 26, 2021, 3:05pm

Ohh Thanks for sharing this info. I will give it a try again. Mine going into recovery mode quite often. It's only being used just as an accesspoint right now as all other services (NAT, DHCP, routing, WAN termination, etc) are handled by my firewall. Planning to add vlans to it once it becomes stable.

daniel · August 26, 2021, 8:53pm

@aparcar has changed the API of the update server. New client versions are compatible again, and he is also going to provide compatibility endpoints for existing older version clients.

themeanfarmer · August 27, 2021, 6:40am

I been having problems too using as access point. What firewall do you use?

aparcar · August 27, 2021, 8:20am

I've update the upgrade server to work with the legacy API as well. Please give it a try.

bobbythomas · August 27, 2021, 3:15pm

I am using Opnsense firewall.

bobbythomas · August 27, 2021, 3:16pm

I will check again and will let you know. Thank you.

hnyman · August 27, 2021, 4:59pm

@daniel

It seems that a new NAND flash driver for Mediatek with bad block management table (BMT) support was merged today.

Does it matter in sysupgrade (or later actions) if the UBI recovery partition is still the old one, with the old driver and without bad block management?
Should that recovery be reflashed with the new driver with BMT support?

daniel · August 27, 2021, 5:47pm

The new and the old driver both support BMT. The new driver initially did not, then it was backported. However, when using UBI, anyway we are not using BMT, hence it makes no difference from point of view of sysupgrade or existing data. Should just work[tm].

hnyman · August 27, 2021, 6:20pm

Well, apparently not.

Flashing r17435-97e32e9702 causes router to be nonfunctional.
The following pstore items are then visible in recovery.

root@OpenWrt:~# cat /sys/fs/pstore/dmesg-ramoops-0
Oops#1 Part1
<5>[    0.554467] 0x0000001c0000-0x0000002c0000 : "factory"
<5>[    0.560941] 0x000000300000-0x000008000000 : "ubi"
<5>[    0.594565] random: fast init done
<3>[    0.696772] blk_update_request: I/O error, dev mtdblock2, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
<3>[    0.707136] Buffer I/O error on dev mtdblock2, logical block 0, async page read
<3>[    0.714592] blk_update_request: I/O error, dev mtdblock2, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
<3>[    0.724939] Buffer I/O error on dev mtdblock2, logical block 0, async page read
<1>[    0.732252] Unable to handle kernel paging request at virtual address ffffffc007fff000
<1>[    0.740160] Mem abort info:
<1>[    0.742944]   ESR = 0x96000006
<1>[    0.746001]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[    0.751303]   SET = 0, FnV = 0
<1>[    0.754347]   EA = 0, S1PTW = 0
<1>[    0.757480] Data abort info:
<1>[    0.760351]   ISV = 0, ISS = 0x00000006
<1>[    0.764176]   CM = 0, WnR = 0
<1>[    0.767139] swapper pgtable: 4k pages, 39-bit VAs, pgdp=00000000449cd000
<1>[    0.773828] [ffffffc007fff000] pgd=000000005ffff003, p4d=000000005ffff003, pud=000000005ffff003, pmd=0000000000000000
<0>[    0.784438] Internal error: Oops: 96000006 [#1] SMP
<7>[    0.789305] Modules linked in:
<7>[    0.792355] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G S                5.10.60 #0
<7>[    0.799740] Hardware name: Linksys E8450 (UBI) (DT)
<7>[    0.804609] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
<7>[    0.810612] pc : fdt_check_header+0x4/0x120
<7>[    0.814788] lr : parse_fit_partitions+0x9c/0x764
<7>[    0.819395] sp : ffffffc010b8b7d0
<7>[    0.822699] x29: ffffffc010b8b7d0 x28: 0000000000000000
<7>[    0.828003] x27: fffffffffffffffb x26: 0000000000000000
<7>[    0.833307] x25: ffffffc007fff000 x24: 0000000000000000
<7>[    0.838611] x23: 0000000000000000 x22: ffffffc010b8b8ec
<7>[    0.843915] x21: 0000000000000000 x20: ffffff8000981200
<7>[    0.849219] x19: 0000000000000000 x18: 000000000000007c
<7>[    0.854523] x17: 0000000000000007 x16: 0000000000000007
<7>[    0.859827] x15: ffffffc010a569e0 x14: 0000000000000174
<7>[    0.865131] x13: 000000000000007c x12: 0000000000000040
<7>[    0.870434] x11: 0000000000000000 x10: ffffff80003f2438
<7>[    0.875739] x9 : 00000000285d2550 x8 : ffffff80009f6140
<7>[    0.881042] x7 : ffffff8000380820 x6 : 0000000000000001
<7>[    0.886347] x5 : 00000000000000c0 x4 : 0000000000000002
<7>[    0.891650] x3 : fffffffeffe20634 x2 : 0000000000000000
<7>[    0.896954] x1 : ffffffc007fff000 x0 : ffffffc007fff000
<7>[    0.902258] Call trace:
<7>[    0.904696]  fdt_check_header+0x4/0x120
<7>[    0.908523]  fit_partition+0x20/0x2c
<7>[    0.912089]  blk_add_partitions+0x120/0x510
<7>[    0.916265]  bdev_disk_changed+0xa0/0x12c
<7>[    0.920265]  __blkdev_get+0x4c8/0x650
<7>[    0.923918]  blkdev_get+0x4c/0xd0
<7>[    0.927225]  blkdev_get_by_dev+0x34/0x54
<7>[    0.931138]  __device_add_disk+0x1b0/0x430
<7>[    0.935225]  device_add_disk+0x10/0x1c
<7>[    0.938968]  register_mtd_blktrans_devs+0x6c/0xf0
<7>[    0.943666]  mtd_device_parse_register+0xb8/0x1b0
<7>[    0.948362]  mtk_snand_probe+0x414/0x5a0
<7>[    0.952279]  platform_drv_probe+0x50/0xa0
<7>[    0.956281]  really_probe+0xe4/0x480
<7>[    0.959847]  driver_probe_device+0x54/0xb0
<7>[    0.963934]  device_driver_attach+0xb4/0xc0
<7>[    0.968108]  __driver_attach+0x80/0x114
<7>[    0.971935]  bus_for_each_dev+0x5c/0x90
<7>[    0.975762]  driver_attach+0x20/0x2c
<7>[    0.979329]  bus_add_driver+0xfc/0x1e0
<7>[    0.983069]  driver_register+0x74/0x120
<7>[    0.986897]  __platform_driver_register+0x44/0x50
<7>[    0.991595]  mtk_snand_driver_init+0x18/0x20
<7>[    0.995857]  do_one_initcall+0x4c/0x1b0
<7>[    0.999687]  kernel_init_freeable+0x240/0x2ac
<7>[    1.004034]  kernel_init+0x10/0x10c
<7>[    1.007514]  ret_from_fork+0x10/0x18
<0>[    1.011084] Code: d2800501 9a813000 d65f03c0 aa0003e1 (b9400000)
<4>[    1.017171] ---[ end trace d8b54d52f64a2cb3 ]---
root@OpenWrt:~# cat /sys/fs/pstore/dmesg-ramoops-
cat: can't open '/sys/fs/pstore/dmesg-ramoops-': No such file or directory
root@OpenWrt:~# cat /sys/fs/pstore/dmesg-ramoops-1
Panic#2 Part1
<3>[    0.707136] Buffer I/O error on dev mtdblock2, logical block 0, async page read
<3>[    0.714592] blk_update_request: I/O error, dev mtdblock2, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
<3>[    0.724939] Buffer I/O error on dev mtdblock2, logical block 0, async page read
<1>[    0.732252] Unable to handle kernel paging request at virtual address ffffffc007fff000
<1>[    0.740160] Mem abort info:
<1>[    0.742944]   ESR = 0x96000006
<1>[    0.746001]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[    0.751303]   SET = 0, FnV = 0
<1>[    0.754347]   EA = 0, S1PTW = 0
<1>[    0.757480] Data abort info:
<1>[    0.760351]   ISV = 0, ISS = 0x00000006
<1>[    0.764176]   CM = 0, WnR = 0
<1>[    0.767139] swapper pgtable: 4k pages, 39-bit VAs, pgdp=00000000449cd000
<1>[    0.773828] [ffffffc007fff000] pgd=000000005ffff003, p4d=000000005ffff003, pud=000000005ffff003, pmd=0000000000000000
<0>[    0.784438] Internal error: Oops: 96000006 [#1] SMP
<7>[    0.789305] Modules linked in:
<7>[    0.792355] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G S                5.10.60 #0
<7>[    0.799740] Hardware name: Linksys E8450 (UBI) (DT)
<7>[    0.804609] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
<7>[    0.810612] pc : fdt_check_header+0x4/0x120
<7>[    0.814788] lr : parse_fit_partitions+0x9c/0x764
<7>[    0.819395] sp : ffffffc010b8b7d0
<7>[    0.822699] x29: ffffffc010b8b7d0 x28: 0000000000000000
<7>[    0.828003] x27: fffffffffffffffb x26: 0000000000000000
<7>[    0.833307] x25: ffffffc007fff000 x24: 0000000000000000
<7>[    0.838611] x23: 0000000000000000 x22: ffffffc010b8b8ec
<7>[    0.843915] x21: 0000000000000000 x20: ffffff8000981200
<7>[    0.849219] x19: 0000000000000000 x18: 000000000000007c
<7>[    0.854523] x17: 0000000000000007 x16: 0000000000000007
<7>[    0.859827] x15: ffffffc010a569e0 x14: 0000000000000174
<7>[    0.865131] x13: 000000000000007c x12: 0000000000000040
<7>[    0.870434] x11: 0000000000000000 x10: ffffff80003f2438
<7>[    0.875739] x9 : 00000000285d2550 x8 : ffffff80009f6140
<7>[    0.881042] x7 : ffffff8000380820 x6 : 0000000000000001
<7>[    0.886347] x5 : 00000000000000c0 x4 : 0000000000000002
<7>[    0.891650] x3 : fffffffeffe20634 x2 : 0000000000000000
<7>[    0.896954] x1 : ffffffc007fff000 x0 : ffffffc007fff000
<7>[    0.902258] Call trace:
<7>[    0.904696]  fdt_check_header+0x4/0x120
<7>[    0.908523]  fit_partition+0x20/0x2c
<7>[    0.912089]  blk_add_partitions+0x120/0x510
<7>[    0.916265]  bdev_disk_changed+0xa0/0x12c
<7>[    0.920265]  __blkdev_get+0x4c8/0x650
<7>[    0.923918]  blkdev_get+0x4c/0xd0
<7>[    0.927225]  blkdev_get_by_dev+0x34/0x54
<7>[    0.931138]  __device_add_disk+0x1b0/0x430
<7>[    0.935225]  device_add_disk+0x10/0x1c
<7>[    0.938968]  register_mtd_blktrans_devs+0x6c/0xf0
<7>[    0.943666]  mtd_device_parse_register+0xb8/0x1b0
<7>[    0.948362]  mtk_snand_probe+0x414/0x5a0
<7>[    0.952279]  platform_drv_probe+0x50/0xa0
<7>[    0.956281]  really_probe+0xe4/0x480
<7>[    0.959847]  driver_probe_device+0x54/0xb0
<7>[    0.963934]  device_driver_attach+0xb4/0xc0
<7>[    0.968108]  __driver_attach+0x80/0x114
<7>[    0.971935]  bus_for_each_dev+0x5c/0x90
<7>[    0.975762]  driver_attach+0x20/0x2c
<7>[    0.979329]  bus_add_driver+0xfc/0x1e0
<7>[    0.983069]  driver_register+0x74/0x120
<7>[    0.986897]  __platform_driver_register+0x44/0x50
<7>[    0.991595]  mtk_snand_driver_init+0x18/0x20
<7>[    0.995857]  do_one_initcall+0x4c/0x1b0
<7>[    0.999687]  kernel_init_freeable+0x240/0x2ac
<7>[    1.004034]  kernel_init+0x10/0x10c
<7>[    1.007514]  ret_from_fork+0x10/0x18
<0>[    1.011084] Code: d2800501 9a813000 d65f03c0 aa0003e1 (b9400000)
<4>[    1.017171] ---[ end trace d8b54d52f64a2cb3 ]---
<0>[    1.022846] Kernel panic - not syncing: Oops: Fatal exception
<2>[    1.028584] SMP: stopping secondary CPUs
<0>[    1.032499] Kernel Offset: disabled
<0>[    1.035977] CPU features: 0x0000002,04002004
<0>[    1.040236] Memory Limit: none

I tested twice, and r17421-d98738b5c1 from yesterday works, but r17435-97e32e9702 from today does not. Rather identical errors in pstore on both times.

Dopam-IT_1987 · August 27, 2021, 6:23pm

hello everyone so you advise not to update?