WRT32X Sysupgrade Broken

[lantis1008]

I'm reasonably certain sysupgrade is broken on the Linksys WRT32X. I'd love to hear from another user to confirm/refute this.

Sysupgrade (do not preserve) via GUI or CLI

Symptoms:

• Device reboots (way too quickly, i don't think much of the upgrade happens at all)
  -- boot_part/bootcmd remains as it was before sysupgrade, so it comes back to same partition
  -- /etc/banner is now blank (part of the overlay is being erased??). Can't find any other files which are broken but i haven't looked extensively
  -- Config remains intact as far as i can tell

IF i interrupt boot with serial and bring the system up (run bootcmd), then run the sysupgrade, interrupt again and bring the system up (run bootcmd), everything works as you would expect.
Take serial out of the equation and it goes pear shaped.

Anyone able to point me in the right direction for debugging this? I've tried but i'm currently out of ideas. Because the problem manifests with serial disconnected, it makes life hard.

Also with serial, am i supposed to be able to invoke this at ANY time and access the console? I do not seem to get any output/input unless i interrupt the boot process first.

[lantis1008]

So the problem appears to be that the console variable is getting corrupted during boot.
When i bring it up via serial:
Kernel command line: console=ttyS0,115200 root=/dev/mtdblock6

When i let the system come up on its own:

Malformed early option 'console'
Kernel command line: console= root=/dev/mtdblock6

Discussion on IRC confirms that a lack of console would definitely cause a sysupgrade failure, as the commands would not be able to be run by the minimal user session.

Thoughts and ideas on fixing this? The u-boot environment appears to be ok, the command is just being mangled. Don't think hardcoding the cmdline in the DTS would be a great method as that would negate the benefits of dual partition firmware.

altnandboot=setenv bootargs console=ttyS0,115200 root=/dev/mtdblock8;nand read $defaultLoadAddr $altKernAddr $altKernSize; bootz $defaultLoadAddr
nandboot=setenv bootargs console=ttyS0,115200 root=/dev/mtdblock6;nand read $defaultLoadAddr $priKernAddr $priKernSize; bootz $defaultLoadAddr

[ghoffman]

hello lantis -
what build are you refering to?
sysupgraded the davidc502 5/18 build a few tumes on my wrt32x without issue.

this morning i used mtd write to place the 5/23 lede 18.06 snapshot factory.img on kernel1 and am running that now. tonight i can test whether sysupgrade works or not.

i honestly cant remember if there was banner on an ssh connection.
fyi; let me know how i can help

[lantis1008]

Any/all of them as far as I am aware.
When it sysupgraded, did you do this through the GUI? If so, are you sure it came up to a new system on the opposite partition?

If you let the system boot normally (no serial), and run cat /proc/cmdline you should see a blank console variable which is the root of the issue.

[ghoffman]

i just sysupgraded to snapshot 5/24. this is what i get from an ssh console:

Using username "root".
root@192.168.1.254's password:


BusyBox v1.28.3 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 18.06-SNAPSHOT, r6924-c97189e
 -----------------------------------------------------
root@hoffman1:~# cat /proc/cmdline
console=ttyS0,115200 root=/dev/mtdblock8
root@hoffman1:~# cat /etc/banner
  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 18.06-SNAPSHOT, r6924-c97189e
 -----------------------------------------------------
root@hoffman1:~# fw_printenv
CASset=max
MALLOC_len=5
MPmode=SMP
SMT-2D=NK90I0500B75X01
altFwSize=0x7B00000
altKernAddr=0x8400000
altKernSize=0x0600000
altnandboot=setenv bootargs console=ttyS0,115200 root=/dev/mtdblock8;nand read $defaultLoadAddr $altKernAddr $altKernSize; bootz $defaultLoadAddr
autoload=no
baudrate=115200
boot_order=hd_scr usb_scr mmc_scr hd_img usb_img mmc_img pxe net_img net_scr
boot_part_ready=3
bootargs_dflt=$console $nandEcc $mtdparts_lgcy $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end $mvNetConfig video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel
bootargs_end=:10.4.50.254:255.255.255.0:Armada38x:eth0:none
bootargs_root=root=/dev/nfs rw
bootcmd_auto=stage_boot $boot_order
bootcmd_fdt=tftpboot 0x2000000 $image_name;tftpboot $fdtaddr $fdtfile;setenv bootargs $console $nandEcc $mtdparts $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end $mvNetConfig video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel; bootz 0x2000000 - $fdtaddr;
bootcmd_fdt_boot=tftpboot 0x2000000 $image_name; setenv bootargs $console $nandEcc $mtdparts $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end $mvNetConfig video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel; bootz 0x2000000 - $fdtaddr;
bootcmd_fdt_edit=tftpboot $fdtaddr $fdtfile; fdt addr $fdtaddr; setenv bootcmd $bootcmd_fdt_boot
bootcmd_lgcy=tftpboot 0x2000000 $image_name;setenv bootargs $console $nandEcc $mtdparts_lgcy $bootargs_root nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip$bootargs_end  video=dovefb:lcd0:$lcd0_params clcd.lcd0_enable=$lcd0_enable clcd.lcd_panel=$lcd_panel;  bootm $loadaddr;
bootdelay=3
cacheShare=no
console=console=ttyS0,115200
defaultLoadAddr=0x2000000
device_partition=0:1
disaMvPnp=no
eeeEnable=no
enaClockGating=no
enaCpuStream=no
enaFPU=yes
enaMonExt=no
enaWrAllo=no
eth1addr=60:38:E0:D0:10:58
eth1mtu=1500
eth2addr=60:38:E0:D0:10:58
eth2mtu=1500
eth3addr=60:38:E0:D0:10:58
eth3mtu=1500
ethact=egiga0
ethaddr=60:38:E0:D0:10:58
ethmtu=1500
ethprime=egiga0
fdt_addr=2040000
fdt_skip_update=no
fdtaddr=0x1000000
fdtfile=armada-38x-modular.dtb
fileaddr=2000000
filesize=C20000
firmwareName=venom.img
flash_alt_image=tftpboot $defaultLoadAddr $firmwareName && nand erase $altKernAddr $altFwSize && nand write $defaultLoadAddr $altKernAddr $filesize
flash_pri_image=tftpboot $defaultLoadAddr $firmwareName && nand erase $priKernAddr $priFwSize && nand write $defaultLoadAddr $priKernAddr $filesize
ide_path=/
image_name=uImage
initrd_name=uInitrd
ipaddr=192.168.1.1
kernel_addr_r=2080000
lcd0_enable=0
lcd0_params=640x480-16@60
lcd_panel=0
limit_dram_size=yes
loadaddr=0x02000000
loads_echo=0
mtddevname=uboot
mtddevnum=0
mtdids=nand0=armada-nand
mtdparts=mtdparts=armada-nand:2048K(uboot)ro,128K(u_env),256K(s_env),256K@8064K(devinfo),123m@9m(firmware1),123m@132m(firmware2)
mvNetConfig=mv_net_config=4,(00:50:43:11:11:11,0:1:2:3),mtu=1500
mv_pon_addr=00:50:43:db:f5:00
nandEcc=nfcConfig=4bitecc
nandboot=setenv bootargs console=ttyS0,115200 root=/dev/mtdblock6;nand read $defaultLoadAddr $priKernAddr $priKernSize; bootz $defaultLoadAddr
netbsd_en=no
netmask=255.255.255.0
netretry=no
partition=nand0,0
pcieTune=no
pexMode=RC
priFwSize=0x7B00000
priKernAddr=0x0900000
priKernSize=0x0600000
pxe_files_load=:default.arm-armadaxp-db:default.arm-armadaxp:default.arm
pxefile_addr_r=3100000
ramdisk_addr_r=2880000
rootpath=/srv/nfs/
run_script=no
sata_delay_reset=0
sata_dma_mode=yes
script_addr_r=3000000
script_name=boot.scr
sd_detection_dat3=no
serverip=192.168.1.254
standalone=fsload 0x2000000 $image_name;setenv bootargs $console $nandEcc $mtdparts_lgcy root=/dev/mtdblock0 rw ip=$ipaddr:$serverip$bootargs_end; bootm 0x2000000;
stderr=serial
stdin=serial
stdout=serial
update_both_images=tftpboot $defaultLoadAddr $firmwareName && nand erase $priKernAddr $priFwSize && nand erase $altKernAddr $altFwSize && nand write $defaultLoadAddr $priKernAddr $filesize && nand write $defaultLoadAddr $altKernAddr $filesize
usb0Mode=host
usbActive=0
usbType=2
vxworks_en=no
yuk_ethaddr=00:00:00:EE:51:81
boot_part=2
bootcmd=run altnandboot
auto_recovery=yes
root@hoffman1:

[lantis1008]

Fascinating.
Can you provide a link to this snapshot please. I’ll tske a look.

Perhaps it only my local environment that is borked.

Just to confirm, you flashed this by using the GUI, and did not issue any boot commands from the U-Boot? i.e. the device was just powered on and booted without intervention.

[ghoffman]

i flashed the davidc 5/18 verion from the factory gui.
that got the davidc build on kernel2
i then ssh'd into davidc build on boot_part2 and mtd-wrote the factory snapshot (from the directory below) to kernel1, and rebooted into kernel1 (boot_part1)

this afternoon, as you had asked, I sysupgraded from the gui on kernel1. this got me back onto kernel2 (boot_part2)

the above terminal session is from ssh after the last step.

i'm not sure if htis is how it's supposed to work, but it does seem to switch partitions with sysupgrade.
(i actually would have thought not)

http://downloads.openwrt.org/releases/18.06-SNAPSHOT/targets/mvebu/cortexa9/

[spuriousoffspring]

sysupgrade from LuCI dud not work for me either.

I originally flashed the May 18 factory image via serial using run update_both_images.
Before upgrading I checked with fw_printenv boot_part. Current partition is 1.
I used Advanced Reboot to switch to partition 2. LuCI comes up as a blank slate.
I switched back to partition 1 with fw_setenv boot_part 1.
Using LuCI I upgrade firmware with May 24 sysupgrade image.
WRT32X reboots normally, but in partition 1 and on May 18 version.

[lantis1008]

Yes this is what I expect to happen currently because of the bug.

[farmwald]

I have tried sysupgrade three times this week on a wrt32x - all failed. I'm stuck on r6911.

[lantis1008]

You’re not stuck, the OEM partition is still available.
If you overwrote the OEM partition using “upgrade_both_images” then that implies you have serial access so you can get back as well.

I’m trying to come up with a solution, but I need time. I’ve come up with an idea I’ll be testing this weekend.
This thread was to ask for assistance from the big guns, but so far no dice. So you’re stuck with me.

[slh]

@lantis1008

Thoughts and ideas on fixing this? The u-boot environment appears to be ok, the command is just being mangled. Don’t think hardcoding the cmdline in the DTS would be a great method as that would negate the benefits of dual partition firmware.

You can override the bootloader's cmdline while still inheriting the rootfs from it, at least on ARM architectures, see

ip806x: nbg6817: don't hardcode the rootfs location by using append-rootblock instead

which depends on

https://github.com/openwrt/openwrt/blob/master/target/linux/ipq806x/patches-4.14/0067-generic-Mangle-bootloader-s-kernel-arguments.patch

(it might make sense to move the patch to generic, as it's needed by oxnas and ipq806x already)

[lantis1008]

Yes this is what I had thought. Will be making a similar patch to “append-console” instead.
If that works in principle, then cleaning it up for submission as a patch will go ahead.

Where were you the last 2 days while I have been pondering this lol? I had this epiphany while on lunch today.

[slh]

You don't need an append-console, you can just override the cmdline completely via dts and use the patch as-us, all what matters at that point is the "rootfs=/dev/mtdblock[68]" setting (the rest is static) so the correct rootfs for the (already loaded/ running) kernel gets mounted (I saw your questions on IRC about 6h later, but you never returned).

[lantis1008]

Good point. Appreciate your steering me in the right direction.
Yes I’m a bit flakey with IRC.

[slh]

cp target/linux/ipq806x/patches-4.14/0067-generic-Mangle-bootloader-s-kernel-arguments.patch target/linux/mvebu/patches-4.14/

and

--- a/target/linux/mvebu/files-4.14/arch/arm/boot/dts/armada-385-linksys-venom.dts
+++ b/target/linux/mvebu/files-4.14/arch/arm/boot/dts/armada-385-linksys-venom.dts
@@ -48,6 +48,12 @@
                     "marvell,armada380";
        };
 
+       chosen {
+               bootargs = "console=ttyS0,115200";
+               stdout-path = "serial0:115200n8";
+               append-rootblock = "root=/dev/mtdblock";
+       };
+
        &expander0 {
                        wan_amber@0 {
                                label = "venom:amber:wan";

should be literally everything needed (disclaimer, I'm not a DTS expert and don't know anything about mvebu, but it should work).

[lantis1008]

Compiling image with similar patch now.
I'm not sure how CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_MANGLE will interact with CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_FROM_BOOTLOADER. I don't think it will be favourable.

[lantis1008]

This is what we like to see:

[    0.000000] Kernel command line: console=ttyS0,115200 root=/dev/mtdblock8
[    0.000000] Bootloader command line (ignored): console= root=/dev/mtdblock8

Will be submitting a patch shortly.

EDIT: http://lists.infradead.org/pipermail/openwrt-devel/2018-May/012523.html

[spuriousoffspring]

Thank you!

[farmwald]

Thanks. When will the changes hit the snapshot? I assume I just need to wait for the right upgrade file and it just work.