Failed sysupgrade with invalid tar on gl-xe300

I am working on support for the Gl.iNet GL-XE300 on my github fork that I will submit as a PR. I have the device working with only the NOR flash.

I've noticed that I can't sysupgrade, even after switching to my own firmware image. When I perform the sysupgrade process the following output is seen on the console:

 -----------------------------------------------------
 OpenWrt SNAPSHOT, r16845+1-507257778c
 -----------------------------------------------------
root@OpenWrt:/#
root@OpenWrt:/#
root@OpenWrt:/# sysupgrade -F -n -v /tmp/gl-xe300.bin
Wed Jun  2 10:39:21 UTC 2021 upgrade: Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
killall: telnetd: no process killed
Wed Jun  2 10:39:22 UTC 2021 upgrade: Sending TERM to remaining processes ... logd hostapd wpa_supplicant netifd odhcpd opennhrp sh logread sh watchfrrwatchfrr[2288]: Terminating on signal
 dnsmasq zebra staticd ntpd ubusd urngd
Wed Jun  2 10:39:25 UTC 2021 upgrade: Sending KILL to remaining processes ... logread
[  218.173718] sh (3097): drop_caches: 3
Wed Jun  2 10:39:27 UTC 2021 upgrade: Switching to ramdisk...
Wed Jun  2 10:39:30 UTC 2021 upgrade: Performing system upgrade...
tar: invalid tar magic
sysupgrade successful
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource [  222.058323] reboot: Restarting system

Does anyone have any idea why I'm getting the invalid tar magic error? I'd like to be able to sysupgrade before I submit the PR, it is possible to upgrade using the uboot web interface but it's not optimal.

I've made some modest progress on this issue, it looks like the problem is on how the sysupgrade process is identifying the upgrade file.

When performing the upgrade the the image is handled by the nand_do_upgrade function which in turn identifies it as a tar formatted upgrade file; this leads to it being handled by nand_upgrade_tar . Here's from output from stage2 with set -x applied:

+ echo 'Fri Jun  4 18:54:03 UTC 2021 upgrade: Performing system upgrade...'
Fri Jun  4 18:54:03 UTC 2021 upgrade: Per+ board_name
+ + local 'board=glinet,gl-xe300-nor'
+ nand_do_upgrade /tmp/gl-xe300.bin
+ iden+ nand_get_magic+ hexdump -v -n 4 -e '1/1 "%02x"+ dd 'if=/tmp/gl-xe300.bin' 'skip=0' 'bs=4' 'cou+ identify_magic 27051956
+ local 'magic=27051956'
+ echo 'unk+ local 'file_ty+ find_mtd_index+ awk -F: '{print $1}'
+ grep '"ubi"' /proc/mtd
+ local 'PART='
+ local 'INDEX='
+ echo
+ '[' '!'  ]
+ CI_UBIPART=rootfs
+ nand_upgrade_tar /tmp/gl-xe300+ find_mtd_index+ awk -F: '{print $1}'
+ grep '"kernel"' /proc/mtd
+ local 'PART=mtd4'
+ local 'INDEX=4'
+ echo 4
+ local 'kernel_mtd=4'
+ grep -m 1 '^sysupgrade-.*/$'
+ tar tf /tmp/gl-xe300.bin

My compiled image is actually in a u-boot uImage format:

gl-xe300.bin: u-boot legacy uImage, MIPS OpenWrt Linux-5.4.123, Linux/MIPS, OS Kernel Image (lzma), 2294956 bytes, Tue Jun 1 13:27:59 2021, Load Address: 0x80060000, Entry Point: 0x80060000, Header CRC: 0x2FF623EF, Data CRC: 0xD7238DA0

How can I ensure that the bin file is properly handled?

I've been able to figure out part of the problem, it seems like ath79/nand uses a tar format for sysupgrade.

I was able to address this by modifying the definition to include both a factory.img as well as a sysupgrade in tar format.

define Device/glinet_gl-xe300-nor
  SOC := qca9531
  DEVICE_TITLE := GL.iNet GL-XE300 (NOR)
  DEVICE_PACKAGES := kmod-usb2 block-mount  kmod-usb-serial-ch341
  IMAGE_SIZE := 16000k
  KERNEL_SIZE := 2304k
  BLOCKSIZE := 128k
  PAGESIZE := 2048
  SUPPORTED_DEVICES += gl-xe300 glinet,gl-xe300
  IMAGES := factory.img sysupgrade.tar
  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi | append-metadata
  IMAGE/sysupgrade.tar := sysupgrade-tar | append-metadata
endef
TARGET_DEVICES += glinet_gl-xe300-nor

This works very well for the sysupgrade but I am no longer able to flash my bin file using the uboot web recovery interface. Compared with a build with a device definition that excludes the image definition I see the following differences on first boot. The functioning build shows detection of a squashfs-split partition as follows:

[    0.424157] Creating 2 MTD partitions on "firmware":
[    0.429348] 0x000000000000-0x000000240000 : "kernel"
[    0.435528] 0x000000240000-0x000000fa0000 : "rootfs"
[    0.441545] mtd: device 5 (rootfs) set to be root filesystem
[    0.449380] 1 squashfs-split partitions found on MTD device rootfs

The non-functional image shows doesn't have this message and shows, a few lines later:

[    2.005065] UBI: auto-attach mtd5
[    2.008514] ubi0: attaching mtd5
[    2.012064] ubi0 error: validate_ec_hdr: bad VID header offset 2048, expected 64
[    2.019750] ubi0 error: validate_ec_hdr: bad EC header
[...]
[    2.162368] ubi0 error: ubi_io_read_ec_hdr: validation failed for PEB 0
[    2.169215] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd5, error -22
[    2.179268] UBI error: cannot attach mtd5
[    2.184144] /dev/root: Can't open blockdev
[    2.188453] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    2.196198] Please append a correct "root=" boot option; here are the available partitions:

[...]
[    2.245391] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I'm eager to resolve this last issue so that I can submit my PR for support for these two devices.

$> fgrep -r factory ../../cache/openwrt/target/linux/*/image/* | grep metadata | wc -l
2

I don't seem to have a cache/openwrt anywhere inside my build directory from what I can see, running from my openwrt git checkout against target/linux I found the following entries, two of which are related to my efforts:

johnf@johnf-desktop:~/src/openwrt-gl-hw$ fgrep -r factory target/linux/*/image/* | grep metadata
target/linux/ath79/image/common-senao.mk:  IMAGE/sysupgrade.bin := $$(IMAGE/factory.bin) | append-metadata
target/linux/ath79/image/nand.mk:  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi | append-metadata
target/linux/ath79/image/nand.mk:  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi | append-metadata
target/linux/ramips/image/mt76x8.mk:  IMAGE/factory.bin := trx -M 0x746f435c | pad-rootfs | append-metadata | \

I'm not really sure what you're looking for or suggesting but I'm eager to understand more. I'm on IRC as johnf on OFTC if you want to talk there.

can you explain what you mean by related? (edit: nevermind... i see you mean't that these are the ones you created)

can you explain why you are appending metadata to a factory image?

By two of them are related I mean they are part of my changes vs upstream, I am trying to support both the gl-xe300 and the gl-x300b. Here are my current efforts.

I am appending the metadata to the factory image because I'm basing myself on the support in the GL.iNet fork which appends metadata, though their own version, via append-gl-metadata.

GL.iNet is a bit unique in that they ship devices with OpenWrt support, however this product unlike one's I've worked with previously doesn't have support in the main openwrt tree.

I tried removing the append-metadata from the factory image line but it didn't help.

I've noticed when I flash the two different images that the flash erase performed by u-boot before the first boot is different, on the working image I erase more flash than on the non-working one.

Working:

Erasing flash... 
First 0x6 last 0x5c sector size 0x10000

Non-working:

Erasing flash... 
First 0x6 last 0x97 sector size 0x10000

This configuration in nand.mk does produce a working image for uboot:

define Device/glinet_gl-xe300-nor
  SOC := qca9531
  DEVICE_TITLE := GL.iNet GL-XE300 (NOR)
  DEVICE_PACKAGES := kmod-usb2 block-mount  kmod-usb-serial-ch341
  IMAGE_SIZE := 16000k
  SUPPORTED_DEVICES += gl-xe300 glinet,gl-xe300
endef
TARGET_DEVICES += glinet_gl-xe300-nor

can you explain why you've added this?

Truthfully because I don't really know what I'm doing. I don't understand this configuration file as well as I would like to, my goal is to be able to install and sysupgrade stock openwrt on these hardware two platforms. If there's documentation that can help me I'm eager to read it, what I've got so far I've pieced together by looking at the gl-inet tree as well as commits from the addition of other similar devices to OpenWrt.

I took the IMAGE_SIZE parameter from the GL.iNet configuration, if I don't include it then the openwrt-ath79-nand-glinet_gl-xe300-nor-squashfs-sysupgrade.bin file doesn't get created in my bin directory.

So far I have been able to produce a working bin image using this configuration:

define Device/glinet_gl-xe300-nor
  SOC := qca9531
  DEVICE_TITLE := GL.iNet GL-XE300 (NOR)
  DEVICE_PACKAGES := kmod-usb2 block-mount  kmod-usb-serial-ch341
  IMAGE_SIZE := 16000k
  SUPPORTED_DEVICES += gl-xe300 glinet,gl-xe300
endef
TARGET_DEVICES += glinet_gl-xe300-nor

However, as described above when I try to use the bin file with sysupgrade it gets processed as a tar file and fails to upgrade successfully.

I have also built a sysupgrade.tar file that worked, using this configuration, however the bin file created using this method can't be installed with uboot:

define Device/glinet_gl-xe300-nor
  SOC := qca9531
  DEVICE_TITLE := GL.iNet GL-XE300 (NOR)
  DEVICE_PACKAGES := kmod-usb2 block-mount  kmod-usb-serial-ch341
  IMAGE_SIZE := 16000k
  KERNEL_SIZE := 2304k
  BLOCKSIZE := 128k
  PAGESIZE := 2048
  SUPPORTED_DEVICES += gl-xe300 glinet,gl-xe300
  IMAGES := factory.img sysupgrade.tar
  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi | append-metadata
  IMAGE/sysupgrade.tar := sysupgrade-tar | append-metadata
endef
TARGET_DEVICES += glinet_gl-xe300-nor

ok... lets step back a bit...

  1. openwrt sample
define Device/glinet_gl-e750
  SOC := qca9531
  DEVICE_VENDOR := GL.iNet
  DEVICE_MODEL := GL-E750
  DEVICE_PACKAGES := kmod-ath10k-ct ath10k-firmware-qca9887-ct kmod-usb2
  SUPPORTED_DEVICES += gl-e750
  KERNEL_SIZE := 4096k
  IMAGE_SIZE := 131072k
  PAGESIZE := 2048
  VID_HDR_OFFSET := 2048
  BLOCKSIZE := 128k
  IMAGES += factory.img
  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi
  IMAGE/sysupgrade.bin := sysupgrade-tar | append-metadata
endef
TARGET_DEVICES += glinet_gl-e750
  1. gl-inet equivalent
define Device/glinet_gl-e750-nor
  ATH_SOC := qca9531
  DEVICE_TITLE := GL.iNet GL-E750 (NOR)
  DEVICE_PACKAGES := kmod-usb2 kmod-ath10k ath10k-firmware-qca9887 block-mount
  IMAGE_SIZE := 16000k
  SUPPORTED_DEVICES += gl-e750 glinet,gl-e750
endef
TARGET_DEVICES += glinet_gl-e750-nor

define Device/glinet_gl-e750-nor-nand
  ATH_SOC := qca9531
  DEVICE_TITLE := GL.iNet GL-E750 (NOR/NAND)
  DEVICE_PACKAGES := kmod-usb2 kmod-ath10k ath10k-firmware-qca9887 block-mount PCI_SUPPORT
  KERNEL_SIZE := 2048k
  BLOCKSIZE := 128k
  PAGESIZE := 2048
  VID_HDR_OFFSET := 2048
  IMAGES := factory.img sysupgrade.tar
  IMAGE/sysupgrade.tar := sysupgrade-tar-compat-1806 | append-gl-metadata
  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi | append-gl-metadata
  SUPPORTED_DEVICES += gl-e750 glinet,gl-e750
endef
TARGET_DEVICES += glinet_gl-e750-nor-nand

can you spot the differences?

(HINT: Follow openwrt image recipes where possible... refer only to oem recipe for specific lines where different... emphasis on refer as your most likely 'extrapolating' not emulating... )

example:

In the example above the value is not verbatim... to work with openwrt it has been altered...

DISCLAIMER... I know nothing about that device.... this is pure speculation based on similar manufacturer ports... It's entirely possible my advice is faulty or that device's uboot does all sorts of unknown shennanigans )

Ok, so I've tried to follow your advice and clean it up to closely match the OpenWrt reference.

I now have:

define Device/glinet_gl-xe300-nor
  SOC := qca9531
  DEVICE_VENDOR := GL.iNet
  DEVICE_TITLE := GL-XE300 (NOR)
  DEVICE_PACKAGES := kmod-usb2 block-mount  kmod-usb-serial-ch341
  IMAGE_SIZE := 16000k
  KERNEL_SIZE := 4096k
  PAGESIZE := 2048
  SUPPORTED_DEVICES += gl-xe300 glinet,gl-xe300
  IMAGES += factory.img
  IMAGE/factory.img := append-kernel | pad-to $$$$(KERNEL_SIZE) | append-ubi
  IMAGE/sysupgrade.bin := sysupgrade-tar | append-metadata
endef
TARGET_DEVICES += glinet_gl-xe300-nor

Is there something else that I'm missing here?

The resulting image didn't work, the factory.img flashes properly (or appears to) but the root partition isn't found when the system boots.

I believe that maybe I shouldn't be calling append-ubi but rather append-rootfs based on an IRC discussion about my challenges, I'm going to try that now.

read further comment...

the peeps on IRC are propably alot more knowlegable than myself... no point duplicating effort...

hope you get it running!

Because this version doesn't support nand I moved out of nand.mk to generic.mk, the resulting images work in the uboot recovery console as well as using sysupgrade inside OpenWrt (though not from the GL.iNet firmware). I believe that nand support could be created in a fashion similar to this commit but it exceeds the time I can commit to this presently.

I've submitted the cleaned up patch in a PR. Thanks very much for the help here and on IRC.

1 Like

Great work @johnfzc! May I know if you've made any progress regarding adding support for the GL-X300B?

Most GL.iNet products ship with a slightly modified version of OpenWRT, however, some devices like the GL-X300B and the GL-XE300 isn't listed on the "Table of Hardware," may I know why this is? And if we could still use a firmware image of a similar device (i.e. the GL-X750 Spitz) which is listed on the OpenWRT TOH page?