Kernel 4.19 upgrade issue on mvebu target(s)

For Developers
1

Wondering if anyone has successfully booted a kernel 4.19.x image using PR1646 on mvebu target mamba. I have tried a number of times as the PR has progressed, but it has always failed to run successfully. The reason for the failure has varied, but always related to an issue with nand. The kernel has grown, so it could be that my device simply has bad blocks in a critical kernel area. But the way the issue manifests itself in varying ways might indicate it is an issue with the nand support for that particular device in the PR.

So, just wondering if my particular device is EOL(or perhaps SOL) in terms of future upgrade support, or if others are seeing the same and there is possibly a resolution to be had by way of the PR.

2

Have you tried make kernel_menuconfig, removing some of kernel features to reduce kernel size? IIRC WRT series' kernel partitions are not covered by UBI so they can be more likely affected by bad blocks.

3

The 4.19 pull requests aren't prioritized and the kernel config should probably be cleaned up as there are probably old setttings in there. https://github.com/openwrt/openwrt/pull/1799
From what I recall this makes the kernel actually a bit smaller than the current 4.14 version. :slight_smile:

4

The 4.14 kernel by gcc7x is:

2623059 Mar 10 13:01 kernel

and 4.19 by gcc8x is:

2777198 Mar 10 14:10 kernel

and the config.414 and config.419 diff, which interestingly enough indicates CONFIG_MTD_NAND_PXA3xx replaced with the new CONFIG_MTD_NAND_MARVELL from here. I'll see if I can't back up to the previous driver with the 4.19 PR.

5

I had tried PR1646 on WRT3200ACM(rango) and WRT32X (venom) and both failed to boot on 4.19, they just go back to 4.14 partition.
Boot logs on WRT32X

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.19.25 (JC@Build) (gcc version 7.4.0 (OpenWrt                                                                             GCC 7.4.0 r9734-b422e1631e)) #0 SMP Tue Mar 26 22:13:30 2019
[    0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction ca                                                                            che
[    0.000000] OF: fdt: Machine model: Linksys WRT32X
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] random: get_random_bytes called from start_kernel+0x78/0x4d0 with crn                                                                            g_init=0
[    0.000000] percpu: Embedded 12 pages/cpu @(ptrval) s19212 r8192 d21748 u49152
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 130048
[    0.000000] Kernel command line: console=ttyS0,115200 root=/dev/mtdblock8
[    0.000000] Bootloader command line (ignored): console= root=/dev/mtdblock8
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 509924K/524288K available (6144K kernel code, 206K rwdata, 14                                                                            80K rodata, 1024K init, 229K bss, 14364K reserved, 0K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xe0800000 - 0xff800000   ( 496 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0x(ptrval) - 0x(ptrval)   (7136 kB)
[    0.000000]       .init : 0x(ptrval) - 0x(ptrval)   (1024 kB)
[    0.000000]       .data : 0x(ptrval) - 0x(ptrval)   ( 207 kB)
[    0.000000]        .bss : 0x(ptrval) - 0x(ptrval)   ( 230 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 D prefetch enabled, offset 1 lines
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 Coherent cache controller enabled, 16 ways, 1024 kB
[    0.000000] L2C-310 Coherent: CACHE_ID 0x410054c9, AUX_CTRL 0x56070001
[    0.000006] sched_clock: 64 bits at 933MHz, resolution 1ns, wraps every 439804651                                                                            1103ns
[    0.000015] clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0                                                                            x1ae5b571769, max_idle_ns: 881590513431 ns
[    0.000026] Switching to timer-based delay loop, resolution 1ns
[    0.000125] Ignoring duplicate/late registration of read_current_timer delay
[    0.000131] clocksource: armada_370_xp_clocksource: mask: 0xffffffff max_cycles:                                                                             0xffffffff, max_idle_ns: 76450417870 ns
[    0.000242] Calibrating delay loop (skipped), value calculated using timer freque                                                                            ncy.. 1866.00 BogoMIPS (lpj=9330000)
[    0.000249] pid_max: default: 32768 minimum: 301
[    0.000299] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000304] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000498] CPU: Testing write buffer coherency: ok
[    0.000509] CPU0: Spectre v2: using BPIALL workaround
[    0.000601] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.000733] Setting up static identity map for 0x100000 - 0x100060
[    0.000792] mvebu-soc-id: MVEBU SoC ID=0x6820, Rev=0x4
[    0.000853] mvebu-pmsu: Initializing Power Management Service Unit
[    0.000892] rcu: Hierarchical SRCU implementation.
[    0.001046] smp: Bringing up secondary CPUs ...
[    0.001147] Booting CPU 1
[    0.001286] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.001288] CPU1: Spectre v2: using BPIALL workaround
[    0.001326] smp: Brought up 1 node, 2 CPUs
[    0.001330] SMP: Total of 2 processors activated (3732.00 BogoMIPS).
[    0.001332] CPU: All CPU(s) started in SVC mode.
[    0.002591] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev                                                                             4
[    0.002638] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_id                                                                            le_ns: 19112604462750000 ns
[    0.002645] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.002689] pinctrl core: initialized pinctrl subsystem
[    0.002923] NET: Registered protocol family 16
[    0.003344] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.003931] cpuidle: using governor ladder
[    0.004051] mvebu-pmsu: CPU hotplug support is currently broken on Armada 38x: di                                                                            sabling
[    0.004057] mvebu-pmsu: CPU idle is currently broken on Armada 38x: disabling
[    0.009139] cryptd: max_cpu_qlen set to 1000
[    0.009484] SCSI subsystem initialized
[    0.009668] usbcore: registered new interface driver usbfs
[    0.009687] usbcore: registered new interface driver hub
[    0.009711] usbcore: registered new device driver usb
[    0.010249] clocksource: Switched to clocksource arm_global_timer
[    0.010595] NET: Registered protocol family 2
[    0.010786] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 byte                                                                            s)
[    0.010797] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    0.010815] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
[    0.010842] TCP: Hash tables configured (established 4096 bind 4096)
[    0.010875] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.010889] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.010954] NET: Registered protocol family 1
[    0.010965] NET: Registered protocol family 44
[    0.011703] Crashlog allocated RAM at address 0x3f00000
[    0.011786] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    0.013355] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.013363] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (                                                                            c) 2001-2006 Red Hat, Inc.
[    0.014433] io scheduler noop registered
[    0.014438] io scheduler deadline registered (default)
[    0.015002] armada-38x-pinctrl f1018000.pinctrl: registered pinctrl driver
[    0.015906] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
[    0.015915] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.015920] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff]
[    0.015924] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
[    0.016745] PCI: bus0: Fast back to back transfers disabled
[    0.016751] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfi                                                                            guring
[    0.016756] pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfi                                                                            guring
[    0.017588] PCI: bus1: Fast back to back transfers disabled
[    0.017781] pci 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5                                                                             GT/s x1 link at 0000:00:02.0 (capable of 4.000 Gb/s with 5 GT/s x1 link)
[    0.018463] PCI: bus2: Fast back to back transfers disabled
[    0.018490] pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe02fffff]
[    0.018495] pci 0000:00:02.0: BAR 8: assigned [mem 0xe0400000-0xe06fffff]
[    0.018502] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe01fffff 64bit pr                                                                            ef]
[    0.018510] pci 0000:01:00.0: BAR 2: assigned [mem 0xe0200000-0xe02fffff 64bit pr                                                                            ef]
[    0.018518] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.018523] pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xe02fffff]
[    0.018529] pci 0000:02:00.0: BAR 0: assigned [mem 0xe0400000-0xe05fffff 64bit pr                                                                            ef]
[    0.018539] pci 0000:02:00.0: BAR 2: assigned [mem 0xe0600000-0xe06fffff 64bit pr                                                                            ef]
[    0.018547] pci 0000:00:02.0: PCI bridge to [bus 02]
[    0.018552] pci 0000:00:02.0:   bridge window [mem 0xe0400000-0xe06fffff]
[    0.018754] mv_xor f1060800.xor: Marvell shared XOR driver
[    0.060591] mv_xor f1060800.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    0.060698] mv_xor f1060900.xor: Marvell shared XOR driver
[    0.120541] mv_xor f1060900.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    0.120692] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[    0.120946] console [ttyS0] disabled
[    0.141015] f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 22, base_baud = 1250                                                                            0000) is a 16550A
[    0.888304] console [ttyS0] enabled
[    0.893166] loop: module loaded
[    0.896574] ahci-mvebu f10a8000.sata: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3                                                                             impl platform mode
[    0.905675] ahci-mvebu f10a8000.sata: flags: 64bit ncq sntf led only pmp fbs pio                                                                             slum part sxs
[    0.914806] scsi host0: ahci-mvebu
[    0.918372] scsi host1: ahci-mvebu
[    0.921858] ata1: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x100 i                                                                            rq 41
[    0.929810] ata2: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x180 i                                                                            rq 41
[    0.938306] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xda
[    0.944700] nand: AMD/Spansion S34ML02G2
[    0.948639] nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 1                                                                            28
[    0.956529] Bad block table found at page 131008, version 0x01
[    0.962723] Bad block table found at page 130944, version 0x01
[    0.968839] pxa3xx_nand-0: error parsing ofpart partition /soc/internal-regs/nand                                                                            -controller@d0000/nand@0/partition@0 (/soc/internal-regs/nand-controller@d0000/nand@                                                                            0)
[    0.984320] libphy: Fixed MDIO Bus: probed
[    0.988679] libphy: orion_mdio_bus: probed
[    0.992940] mvneta_bm f10c8000.bm: Buffer Manager for network controller enabled
[    1.001613] mvneta f1070000.ethernet eth0: Using hardware mac address 60:38:e0:c0                                                                            :b1:a8
[    1.010837] mvneta f1034000.ethernet eth1: Using hardware mac address 60:38:e0:c0                                                                            :b1:a8
[    1.019010] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.025575] ehci-platform: EHCI generic platform driver
[    1.030887] ehci-orion: EHCI orion driver
[    1.034984] orion-ehci f1058000.usb: EHCI Host Controller
[    1.040421] orion-ehci f1058000.usb: new USB bus registered, assigned bus number                                                                             1
[    1.048061] orion-ehci f1058000.usb: irq 38, io mem 0xf1058000
[    1.080245] orion-ehci f1058000.usb: USB 2.0 started, EHCI 1.00
[    1.086378] hub 1-0:1.0: USB hub found
[    1.090156] hub 1-0:1.0: 1 port detected
[    1.094478] usbcore: registered new interface driver usb-storage
[    1.100659] i2c /dev entries driver
[    1.106619] orion_wdt: Initial timeout 171 sec
[    1.111324] sdhci: Secure Digital Host Controller Interface driver
[    1.117531] sdhci: Copyright(c) Pierre Ossman
[    1.153576] mmc0: SDHCI controller on f10d8000.sdhci [f10d8000.sdhci] using ADMA
[    1.161097] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.170253] marvell-cesa f1090000.crypto: CESA device successfully registered
[    1.177830] NET: Registered protocol family 10
[    1.182732] Segment Routing with IPv6
[    1.186449] NET: Registered protocol family 17
[    1.191026] 8021q: 802.1Q VLAN Support v1.8
[    1.195263] Registering SWP/SWPB emulation handler
[    1.201596] usb_phy_generic usb3_1-phy: Linked as a consumer to regulator.1
[    1.208729] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    1.214083] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 2
[    1.220714] random: fast init done
[    1.221659] xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirk                                                                            s 0x0000000000010010
[    1.226075] mmc0: new high speed SDIO card at address 0001
[    1.234306] xhci-hcd f10f8000.usb3: irq 44, io mem 0xf10f8000
[    1.245825] hub 2-0:1.0: USB hub found
[    1.249605] hub 2-0:1.0: 1 port detected
[    1.253653] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    1.258992] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 3
[    1.265186] ata1: SATA link down (SStatus 0 SControl 300)
[    1.266520] xhci-hcd f10f8000.usb3: Host supports USB 3.0  SuperSpeed
[    1.271956] ata2: SATA link down (SStatus 0 SControl 300)
[    1.278433] usb usb3: We don't know the algorithms for LPM for this host, disabli                                                                            ng LPM.
[    1.292149] hub 3-0:1.0: USB hub found
[    1.295942] hub 3-0:1.0: 1 port detected
[    1.300260] mvsw61xx 10.mvsw61xx: Found MV88E6352 at f1072004.mdio-mii:10
[    1.307079] mvsw61xx 10.mvsw61xx: Using direct addressing
[    1.319821] hctosys: unable to open rtc device (rtc0)
[    1.325214] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    1.332736] Please append a correct "root=" boot option; here are the available p                                                                            artitions:
[    1.341132] 1f00          262144 mtdblock0
[    1.341133]  (driver?)
[    1.347700] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-b                                                                            lock(0,0)
[    1.356000] CPU0: stopping
[    1.358720] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.19.25 #0
[    1.364751] Hardware name: Marvell Armada 380/385 (Device Tree)
[    1.370707] [<c010eebc>] (unwind_backtrace) from [<c010b020>] (show_stack+0x10/0x                                                                            14)
[    1.378488] [<c010b020>] (show_stack) from [<c0655e30>] (dump_stack+0x88/0x9c)
[    1.385742] [<c0655e30>] (dump_stack) from [<c010dc98>] (handle_IPI+0xf0/0x198)
[    1.393086] [<c010dc98>] (handle_IPI) from [<c0398a6c>] (gic_handle_irq+0x8c/0x90                                                                            )
[    1.400691] [<c0398a6c>] (gic_handle_irq) from [<c0101a0c>] (__irq_svc+0x6c/0x90)
[    1.408205] Exception stack(0xc0a01f38 to 0xc0a01f80)
[    1.413277] 1f20:                                                       00000000                                                                             00000e10
[    1.421491] 1f40: dfbd42b0 c0114700 ffffe000 c0a03c6c 00000001 c0a03ca8 c09484e8                                                                             00000000
[    1.429705] 1f60: 00000000 00000000 1f28e000 c0a01f88 c01086cc c01086d0 60000013                                                                             ffffffff
[    1.437922] [<c0101a0c>] (__irq_svc) from [<c01086d0>] (arch_cpu_idle+0x34/0x38)
[    1.445355] [<c01086d0>] (arch_cpu_idle) from [<c014b774>] (do_idle+0xe4/0x1c8)
[    1.452697] [<c014b774>] (do_idle) from [<c014bab4>] (cpu_startup_entry+0x18/0x1c                                                                            )
[    1.460305] [<c014bab4>] (cpu_startup_entry) from [<c0900db4>] (start_kernel+0x4c                                                                            0/0x4d0)
[    1.468519] [<c0900db4>] (start_kernel) from [<00000000>] (  (null))
[    1.475190] Rebooting in 1 seconds..
6

I assume this is the problem:

I think the NAND flash driver changed with kernel 4.19.

7

The NAND driver has changed but I thought it had been reported in the PR that the caiman target was working and so assumed (of the wrtpac series) it was just the mamba that had an issue.

Things would appear to work though, as apparently all the wrtpac devices are reported functioning on McDebian; but with a note regarding the increase in kernel size as regards the mamba.

8

@anomeome does this works on wrt3200 acm ?

9

If query is regarding OpenWrt, the above post would indicate that it does not, I have only tried on a mamba so far.

If regarding McDebian, their thread regarding firefighting the 4.19 release indicates everything resolved and released. I reconciled the dts for the mamba device from their github against what is generated by the OpenWrt buildroot and found no substantive differences in the nand bits between the two.

10

mtd hacks are evils

11

anyway mamba is pretty different than acm should i try ?

12

ya, quite different, and as I said above, I thought that early on it had been reported to work on a caiman (shm0 ???). Certainly worth a shot, but there is a small amount of bit rot with the PR currently.

13

yes i can't undestand why nobody is working on kernel 4.19... i remember that the move from 4.9 to 4.14 was pretty quick... Also ipq806x slow progress....

Anyway will waste some resource to build an image and test

14

You can check chunkeey's staging tree for an initial kernel 4.19 port for ipq806x.

15

notice that most of the works is for ipq40xx and not for 80xx

16

https://git.openwrt.org/?p=openwrt/staging/chunkeey.git;a=commitdiff;h=fd025d6217c4661bfde4b77ec35bbe347f388520 is the commit for ipq806x, and it's working on my nbg6817 (ipq8065).

17


this is also needed or kernel compilation fail...

18

PR1913 was pushed to address issue, I was testing with a 8.x GCC image.

I still have an older 4.19 rango image about if that assists, I do not have a serial connection to a rango, so have not bothered testing

19

don't know why it doesn't work for me o.o

20

Requires a dirclean (toolchain), but there is also a change to the defaults for the target in the .config (i.e. distclean, but I just reconciled the diff). see the actual push, there is also further explanation in a thread here by chunkeey.