Bugreport: Regression in ZYNQ-CPU platform not bootiing any more (ZedBoard, Zybo)

I would like to report here a bug about the ZYNQ CPU platform. Same issue on ZedBoard and Zybo. I have done some testing and know since when the regression have appeared.
OpenWrt Versions tested:
[23.05.2] - not booting (stuck in u-boot)
[22.03.6] - not booting (stuck in u-boot)
[21.02.7] - not booting (stuck in u-boot)
[19.07.9] - booting

I cant register to github to report it also there. I hope its enough to report it here.

This is the output of the openwrt snapshot build dated 1.1.2024 (not that long, posting it here):

U-Boot SPL 2019.07-OpenWrt-r24707-4693514ca8 (Jan 01 2024 - 01:34:48 +0000)
mmc boot
Trying to boot from MMC1
spl_load_image_fat_os: error reading image system.dtb, err - -2


U-Boot 2019.07-OpenWrt-r24707-4693514ca8 (Jan 01 2024 - 01:34:48 +0000)

CPU:   Zynq 7z020
Silicon: v3.1
Model: Avnet ZedBoard board
DRAM:  ECC disabled 512 MiB
MMC:   mmc@e0100000: 0
Loading Environment from SPI Flash... SF: Detected s25fl256s1 with page size 256 Bytes, erase size 64 KiB, total 32 MiB
*** Warning - bad CRC, using default environment

In:    serial@e0001000
Out:   serial@e0001000
Err:   serial@e0001000
Net:   ZYNQ GEM: e000b000, phyaddr 0, interface rgmii-id

Warning: ethernet@e000b000 (eth0) using random MAC address - da:c8:23:01:62:56
eth0: ethernet@e000b000
74 bytes read in 8 ms (8.8 KiB/s)
Importing environment from mmc ...
Checking if uenvcmd is set ...
Hit any key to stop autoboot:  0 
## Error: "bootcmd_mmc" not defined
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
SF: Detected s25fl256s1 with page size 256 Bytes, erase size 64 KiB, total 32 MiB
device 0 offset 0xfc0000, size 0x40000
SF: 262144 bytes @ 0xfc0000 Read: OK
## Executing script at 03000000
Wrong image format for "source" command
SCRIPT FAILED: continuing...
starting USB...
Bus usb@e0002000: USB EHCI 1.00
scanning bus usb@e0002000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

Device 0: unknown device
ethernet@e000b000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00000000
ethernet@e000b000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0000000
ethernet@e000b000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/000000
ethernet@e000b000 Waiting for PHY auto negotiation to complete........
FATAL: read zero bytes from port
term_exitfunc: reset failed for dev UNKNOWN: Input/output error

And this is the log from the working 19.07.9 OpenWrt release on same board:

U-Boot SPL 2018.07 (Feb 16 2022 - 20:47:59 +0000)
mmc boot
Trying to boot from MMC1
spl_load_image_fat_os: error reading image system.dtb, err - -2


U-Boot 2018.07 (Feb 16 2022 - 20:47:59 +0000)

CPU:   Zynq 7z020
Silicon: v3.1
Model: Zynq Zed Development Board
DRAM:  ECC disabled 512 MiB
MMC:   sdhci@e0100000: 0
Loading Environment from SPI Flash... SF: Detected s25fl256s_64k with page size 256 Bytes, erase size 64 KiB, total 32 MiB
*** Warning - bad CRC, using default environment

Failed (-5)
In:    serial@e0001000
Out:   serial@e0001000
Err:   serial@e0001000
Net:   ZYNQ GEM: e000b000, phyaddr 0, interface rgmii-id

Warning: ethernet@e000b000 (eth0) using random MAC address - ba:cb:64:48:a9:a5
eth0: ethernet@e000b000
74 bytes read in 6 ms (11.7 KiB/s)
Importing environment from mmc ...
Checking if uenvcmd is set ...
Hit any key to stop autoboot:  0 
Copying FIT from SD to RAM...
3714728 bytes read in 213 ms (16.6 MiB/s)
## Loading kernel from FIT Image at 02000000 ...
   Using 'config@1' configuration
   Verifying Hash Integrity ... OK
   Trying 'kernel@1' kernel subimage
     Description:  ARM OpenWrt Linux-4.14.267
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x020000e4
     Data Size:    3704838 Bytes = 3.5 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x00008000
     Entry Point:  0x00008000
     Hash algo:    crc32
     Hash value:   45a0ee26
     Hash algo:    sha1
     Hash value:   3e3bb3d6a176cb013b87cc15cd6f235fa3ffc8cf
   Verifying Hash Integrity ... crc32+ sha1+ OK
## Loading fdt from FIT Image at 02000000 ...
   Using 'config@1' configuration
   Trying 'fdt@1' fdt subimage
     Description:  ARM OpenWrt avnet_zynq-zed device tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x02388a24
     Data Size:    8020 Bytes = 7.8 KiB
     Architecture: ARM
     Hash algo:    crc32
     Hash value:   b6b92738
     Hash algo:    sha1
     Hash value:   8a3e1d922842a4373c2c41f8d7961b1cf336d4e6
   Verifying Hash Integrity ... crc32+ sha1+ OK
   Booting using the fdt blob at 0x2388a24
   Uncompressing Kernel Image ... OK
   Loading Device Tree to 1eb1b000, end 1eb1ff53 ... OK

Starting kernel ...
[regular kernel booting output removed]

I have took a look into the openwrt development history for this cpu architecture.
It was the same person bumping the u-boot version.
This one works: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=09ac4aa86b889446969df10e397798c7fe961515
And this one is broken:
https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=5ca243153b110ceddffecb70ba8a8cd0e33c8f0b

But maybe its something different then the u-boot version as a reason why its not booting.
Its easy for me to test images. I just write them with dd to a sd card and plug the card into the cardslot and apply power. I am willing to help with further debugging and test other images.

@luaraneda would you like to take a look at the issue?

Thanks for reporting this.
I'll try and take a look at it over the week.

I'm not actively working on the Zynq platform, but I should have my Zybo Z7 board around.
I recall run-testing all my changes on it, so I'm wondering on what happened.

What exact board are you using (FYI, Zybo Z7 has 2 flavors)?

I'll test on mine, but in case the issue is not observed we might need to do debugging on yours.

1 Like

Thanks for the reply. Yes, i would help debugging where i can. Maybe bumping the u-boot version again this time to v2024.1 (release on 08.01.2024 https://docs.u-boot.org/en/latest/develop/release_cycle.html ) would be the perfect way to go. The code is already freezed right now. There was some changes in u-boot in the last years for ZYNQ.

Yes, i have seen this in the commits. That is why i pinged you on the forum here.

OpenWrt currently have support for 4 boards with ZYNQ CPU like listed here:
https://downloads.openwrt.org/releases/23.05.2/targets/zynq/generic/

Yours is: https://downloads.openwrt.org/releases/23.05.2/targets/zynq/generic/openwrt-23.05.2-zynq-generic-digilent_zynq-zybo-z7-squashfs-sdcard.img.gz

And the two i have are:
https://downloads.openwrt.org/releases/23.05.2/targets/zynq/generic/openwrt-23.05.2-zynq-generic-digilent_zynq-zybo-squashfs-sdcard.img.gz

And
https://downloads.openwrt.org/releases/23.05.2/targets/zynq/generic/openwrt-23.05.2-zynq-generic-avnet_zynq-zed-squashfs-sdcard.img.gz

This is a good amount of boards for run testing :slight_smile:

Is there something i could test to report any possible changes in the boot outcome?

Apologies for the delay, I got a lot of things at work.

I did find my Zybo Z7 board and a memory card.
Now I need to find the card reader.

I finally found everything required and ran some tests.
I think this is related to a variable missing on U-Boot's environment.

Please run a test by adding "bootcmd=run sdboot" to uEnv.txt file (text file located on the same SD card partition as U-Boot binary) so it looks like the following:

bootargs=console=ttyPS0,115200n8 root=/dev/mmcblk0p2 rootwait earlyprintk
bootcmd=run sdboot

If that doesn't work, please paste the output of printenv command on U-Boot prompt.

For context, when I started testing, my Zybo z7 worked fine with official 23.05.2 and latest snapshot.
I then attempted to update to 2024.01, but due to some changes it was not taking uEnv.txt file (I'll look into this at some point) and booting wasn't happening properly.

These results lead me to investigate more about U-Boot environment and I realized my board was loading an environment stored on SPI memory (soldered to the board).

It turns out stored environment had bootcmd set to:
bootcmd=run $modeboot || run distro_bootcmd (modeboot=sdboot)
And default environment to:
bootcmd=run distro_bootcmd.
After reseting the environment (u-boot> env default -a) and saving it, I was able to reproduce a similar boot issue.

In addition to fixing this issue, I think we need to look into storing U-Boot's environment into SD card so it's more standalone.

1 Like

Working on Zynq Zybo:

U-Boot SPL 2019.07-OpenWrt-r25579-40fce9dcb8 (Mar 17 2024 - 22:41:03 +0000)
mmc boot
Trying to boot from MMC1
spl_load_image_fat_os: error reading image system.dtb, err - -2


U-Boot 2019.07-OpenWrt-r25579-40fce9dcb8 (Mar 17 2024 - 22:41:03 +0000)

CPU:   Zynq 7z010
Silicon: v3.1
Model: Digilent Zybo board
DRAM:  ECC disabled 512 MiB
MMC:   mmc@e0100000: 0
Loading Environment from SPI Flash... SF: Detected s25fl128s with page size 256 Bytes, erase size 64 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

In:    serial@e0001000
Out:   serial@e0001000
Err:   serial@e0001000
Net:   ZYNQ GEM: e000b000, phyaddr 0, interface rgmii-id

Warning: ethernet@e000b000 (eth0) using random MAC address - 5e:cc:27:0d:56:31
eth0: ethernet@e000b000
93 bytes read in 9 ms (9.8 KiB/s)
Importing environment from mmc ...
Checking if uenvcmd is set ...
Hit any key to stop autoboot:  0 
Copying FIT from SD to RAM...
5185116 bytes read in 292 ms (16.9 MiB/s)
## Loading kernel from FIT Image at 02000000 ...
   Using 'config-1' configuration
   Verifying Hash Integrity ... OK
   Trying 'kernel-1' kernel subimage
     Description:  ARM OpenWrt Linux-5.15.150
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x020000e4
     Data Size:    5172742 Bytes = 4.9 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x00008000
     Entry Point:  0x00008000
     Hash algo:    crc32
     Hash value:   cd4128df
     Hash algo:    sha1
     Hash value:   1f4de41780560778a497796d6b7423100ca62eb1
   Verifying Hash Integrity ... crc32+ sha1+ OK
## Loading fdt from FIT Image at 02000000 ...
   Using 'config-1' configuration
   Verifying Hash Integrity ... OK
   Trying 'fdt-1' fdt subimage
     Description:  ARM OpenWrt digilent_zynq-zybo device tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x024ef028
     Data Size:    10479 Bytes = 10.2 KiB
     Architecture: ARM
     Hash algo:    crc32
     Hash value:   79f4bf66
     Hash algo:    sha1
     Hash value:   98db115b5c2608ed0652f77b888c32a483aa50dd
   Verifying Hash Integrity ... crc32+ sha1+ OK
   Booting using the fdt blob at 0x24ef028
   Uncompressing Kernel Image ... OK
   Loading Device Tree to 1eb33000, end 1eb388ee ... OK

Starting kernel ...

Also tried on Zedboard and also fixed the issue. Could you merge the fix? Is this addidional configuration required with current u-boot 2024 version? Because this was somewhat a regression after the version change from u-boot 2018 to u-boot 2019. Maybe its fixed in 2024 and the additional configuration is not required.