Combined image file not created after compilation (bananapi-r64)

I've been trying to build the firmware for BPI-R64 board and the build completes without any errors. However, I'm not getting a combined IMG file that I'm used to seeing which can just flash onto the card. Instead, I'm getting separate .bin files in two different directories, the question is what steps I need to take to combine them for a bootable system? Any direction would be greatly appreciated as always.

Screenshot from 2021-08-18 09-11-15

Screenshot from 2021-08-18 09-15-56

@Probook:~/bpi64/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-mediatek_mt7622$ ls

backports-5.10.42-1                         mt7623n-preloader
base-files                                  mtd
bpi_bananapi-r64-kernel.bin                 opkg-2021-06-13-1bf042dd
bpi_bananapi-r64-kernel.bin.its             packages
gpio-button-hotplug                         ppp-default
Image                                       root.ext4
Image-initramfs                             root.squashfs
image-mt7622-bananapi-bpi-r64.dtb           symvers
image-mt7622-bananapi-bpi-r64-rootdisk.dtb  tmp
image-mt7622-elecom-wrc-2533gent.dtb        vmlinux
image-mt7622-rfb1.dtb                       vmlinux.debug
image-mt7622-rfb1-ubi.dtb                   vmlinux.elf
image-mt7622-ubnt-unifi-6-lr.dtb            vmlinux-initramfs
iptables-1.8.7                              vmlinux-initramfs.debug
linux-5.4.137                               vmlinux-initramfs.elf
linux-atm-2.5.2                             wireguard-tools-1.0.20210223


@Probook:~/bpi64/openwrt/bin/targets/mediatek/mt7622$ ls

config.buildinfo
feeds.buildinfo
openwrt-mediatek-mt7622-bpi_bananapi-r64-initramfs-kernel.bin
openwrt-mediatek-mt7622-bpi_bananapi-r64.manifest
openwrt-mediatek-mt7622-bpi_bananapi-r64-squashfs-sysupgrade.bin
packages
sha256sums
u-boot-mtk.bin
version.buildinfo

That is the right dir for the end artefacts. The images should be there.

The build_dir is a temporary build-time dir for interim results.

Based on wiki, your router does not follow the usual convetions, as it is SD card based. You need help from the device specialists.
https://openwrt.org/toh/sinovoip/bananapi_bpi-r64_v1.1

Easiest thing might be to download the official snapshot image from
https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/

and then use that to sysyupgrade to your own image, which you have.

2 Likes

from the wiki, it should create a bananapi_bpi-r64-sdcard.img.gz you then flash to the SD card, like every other device based on SD cards

and also here in the build system https://github.com/openwrt/openwrt/blob/master/target/linux/mediatek/image/mt7622.mk#L73

So I think something is blowing up when assembling the images. Maybe you have put a too big root filesystem partition size? It's not default size.

I think you need to run again the build with make -j1 and see what errors it prints on console in the last stage where it is assembling the image.

1 Like

Thank you.

I did the sysupgrade and looks like it completed successfully, however the board still keeps booting up the snapshot version, instead of the firmware I built.

root@OpenWrt:/tmp/upgrade# sysupgrade -v /tmp/upgrade/*.bin
Cannot save config while running from ramdisk.
Wed Aug 18 14:32:32 UTC 2021 upgrade: Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
Watchdog does not have CARDRESET support
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending TERM to remaining processes ...
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to ubusd (1013)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to urngd (1048)
sh: S: out of range
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to logd (1450)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to hostapd (1761)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to wpa_supplicant (1762)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to odhcpd (1868)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to ntpd (2160)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to netifd (2911)
Wed Aug 18 14:32:33 UTC 2021 upgrade: Sending signal TERM to dnsmasq (3211)
Wed Aug 18 14:32:37 UTC 2021 upgrade: Sending KILL to remaining processes ...
sh: S: out of range
[  245.030588] sh (3628): drop_caches: 3
Wed Aug 18 14:32:43 UTC 2021 upgrade: Switching to ramdisk...
Wed Aug 18 14:32:43 UTC 2021 upgrade: Performing system upgrade...
sh: 7: unknown operand
sh: 7: unknown operand
sh: 7: unknown operand
sh: 7: unknown operand
sh: 7: unknown operand
sh: 7: unknown operand
sh: 7: unknown operand
sh: 7: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 259: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 179: unknown operand
sh: 31: unknown operand
sh: 31: unknown operand
sh: 31: unknown operand
[  248.554705] ubi0: attaching mtd2
[  248.708326] ubi0: scanning is finished
[  248.712075] ubi0: empty MTD device detected
[  248.730056] ubi0: attached mtd2 (name "ubi", size 125 MiB)
[  248.735584] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[  248.742452] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[  248.749243] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[  248.756201] ubi0: good PEBs: 1004, bad PEBs: 0, corrupted PEBs: 0
[  248.762285] ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128
[  248.769504] ubi0: max/mean erase counter: 0/0, WL threshold: 4096, image sequence number: 3479548358
[  248.778632] ubi0: available PEBs: 980, total reserved PEBs: 24, PEBs reserved for bad PEB handling: 20
[  248.787940] ubi0: background thread "ubi_bgt0d" started, PID 4272
UBI device number 0, total 1004 LEBs (127483904 bytes, 121.5 MiB), available 980 LEBs (124436480 bytes, 118.6 MiB), LEB size 126976 bytes (124.0 KiB)
Volume ID 0, size 688 LEBs (87359488 bytes, 83.3 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "fit", alignment 1
Set volume size to 37076992
Volume ID 1, size 292 LEBs (37076992 bytes, 35.3 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs_data", alignment 1
sysupgrade successful
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource busy
[  269.040226] reboot: Restarting system

Thank you Alberto, running the build now with make-j1

The build finished with zero errors/warnings, clean build.

That part with all the "out of range" and "unknown operand" errors doesn't look that successful to me...

(I have no knowledge about the BananaPi specifics, but that many errors looks really strange to me)

1 Like

known bug... fix rejected by @aparcar

reasoning was not so clear...

1 Like

I've run a compile of that device with default options from master source, and all seems OK, I see the sdcard image generated. I think you are also building from master sources or at least not 19.07, please correct me if I'm wrong.

When I try to replicate your config from the second screenshot I notice something is not right.
If deselect "ext4" option the Gzip Images and Root filesystem size disappear


Screenshot from 2021-08-19 00-24-24

Can you try moving/renaming the current config files .config and .config.old you find in your ~/bpi64/openwrt folder (it is a hidden file by default, you can see hidden files with ls -la from the command line) and creating a new config with make menuconfig and see if you still have the same problem? I think something messed up the current config and this may be confusing the build system.

EDIT: I've also run a make clear and then compiled another image with ext4 enabled. I still see the sdcard image. I don't understand what is that selection even doing anyway. In other targets I know if you select ext4 you get a dedicated image with ext4 and if you select squashfs you get a dedicated image with squashfs.

But it seems that the image generation code for the sdcard.img.gz just uses the squashfs image regardless.

It's kind of hacky to just leave options that serve no purpose like that, it should probably be fixed.

1 Like

Alberto, wow thank you for taking the time to actually run through it yourself.

I saw the same behavior so I added the squashfs option to "depends on" in openwrt/config/config-images.in so the options to adjust Root filesize and Gzip would appear with it. Line 257 and 273

	config TARGET_IMAGES_GZIP
		bool "GZip images"
		depends on TARGET_ROOTFS_EXT4FS || TARGET_x86 || TARGET_ROOTFS_SQUASHFS || TARGET_armvirt || TARGET_malta
		default y
	config TARGET_ROOTFS_PARTSIZE
		int "Root filesystem partition size (in MB)"
		depends on USES_ROOTFS_PART || TARGET_ROOTFS_EXT4FS || TARGET_ROOTFS_SQUASHFS || TARGET_omap || TARGET_sunxi || TARGET_uml
		default 1500
		help
		  Select the root filesystem partition size.

I always keep them at 1500 as all the boards I work on have atleast an 8G eMMC and I've plenty of packages to install and always run out of space with the default 104Mb.

Yes, I'm building from the master, v21.02.0-rc4. I even replaced the default .config file with the one from the following link: https://downloads.openwrt.org/releases/21.02.0-rc4/targets/mediatek/mt7622/config.buildinfo, however that changed nothing.

I'm going to delete my current build folder and start over again, this time strictly using the default options first before I make any changes.

I agree that this is clunky, how and who can fix it?

Again, I really appreciate you spending time on this and verifying the build process. I will report back with the results.

afaik (some?) bpi boards are 'special' in that traditional 'combined' magic doesn't apply...

pretty sure they use funky layout as chown in your snippets above... you may get better info from the sinoviop or whatever forums...

1 Like

The BPI-R2 gets built normally with a perfectly usable image, however the ongoing chip shortage has caused them some issues with their MT7632N supply so the R64 is what they're focusing on. Both boards use like 6-7 different partitions for a proper boot.
I've reached out to the manufacturer to have them look into this, let's see how much they will be willing to support their own product.

1 Like

well... when I was looking at an R2 issue for someone...

I was able to use some image creation logic and bootstrap partitions from then sinviop repo with a newer openwrt buildroot...

but you have to create the images as expected by their bootloader...

check commits for the R2 target... pretty sure that stuff I did was (similarly handled) merged or has been updated / fixed...

1 Like

The "right way" show that menu with the part size, also if you want to open a PR to contribute this change in OpenWrt is to add "rootfs-part" in the feature list of the target, as in this commit https://github.com/openwrt/openwrt/commit/3bb44f42990a75e66972016cde75bed6a3f09ef9
for your device target it is openwrt/target/linux/mediatek/Makefile
Don't change the default 104 size to 1500 in the PR you do to OpenWrt as that will upset a lot of people and they will refuse to merge your change (for no real reason imho, who has a SDcard smaller than 2Gb nowadays).

Btw, since you build from source anyway you should probably integrate the packages in the image (press y and the package line shows a <*>) instead than installing them later. It's much less painful when you need to upgrade the firmware as packages integrated like this will also be in the firmware upgrade image already and you don't need to install them again afterwards.

Yes, I'm building from the master, v21.02.0-rc4.

the "master" is the master branch, the place where all new code goes. If you build from v21.02.0-rc4 it is a different branch that is stabilizing to become the next release.

That file I linked above looks different in branch v21.02 https://github.com/openwrt/openwrt/blob/openwrt-21.02/target/linux/mediatek/image/mt7622.mk

anyone that has the device to test, manages to understand how the image generation scripts in the folder openwrt/target/linux/mediatek/image/ work and can open and follow a PR on github for a while.
I told you above what I think is the "right way" for showing the partsize in the make menuconfig interface.
But I don't really know about the rest.

I've seen a core developer that has been very active lately on this target, he has made the rework I noticed above with this commit https://github.com/openwrt/openwrt/commit/6f5cd3bdcfce693c7f5f630043b4b5be19e0c592#diff-fa45d2c80de0d6f335af3d8f40f8e15a12d9d36586c28e08b9ccfdc020a31b9e
and he did a similar thing also on bananapi-r2 https://github.com/openwrt/openwrt/commit/86a61e716efe2e0ef2f4ce9b2fdd7a532661ef56

@daniel do you know why there is a ext4 filesystem option as shown here Combined image file not created after compilation (bananapi-r64) - #8 by bobafetthotmail , it does not seem to be used to generate images so enabling it does nothing. Also can you add "rootfs-part" in the feature list of the target in openwrt/target/linux/mediatek/Makefile so people can set the rootfs size of a squashfs image from make menuconfig? Currently that line is shown only if someone selects the ext4 filesystem option.

2 Likes

In post-21.02 (ie. only in master branch for now) there are no more adjustable partitions what-so-ever as uImage.FIT containing kernel+dtb+squashfs and rootfs_data are sharing the same GPT partition (using the newly introduced uImage.FIT partition parser in Linux). It is set to be unreasonably large, but we could also made it adjustable if there is a true need for even larger squashfs or rootfs_overlay (I prefer having the unpartitioned space available for use with uvol and autopart, to use additional persistent volumes e.g. with uxc or podman for containers)

2 Likes

Adding @anon50098793 @hnyman
So, I ran two different builds, one Snapshot and the other with v21.02.0-rc4. The combined SD Card image only gets built with the snapshot version, and not with v.21 or in anything below it, that explains why there's no SD Image file in the download section for any release.

What would need to be done to have the non snapshot versions build proper image files for MT7622?

@daniel How unreasonably large of a partition is it? If it's less than 1G, then it should be made adjustable as there're so many boards with SD cards with no space constraints. Is that why none of my builds acknowledged the partition size values I configured (1500 for example) as after booting, it gave me "out of space" errors when I tried to install a few large packages (python3-dev, luci etc).

Screenshot from 2021-08-19 11-03-11

ln: /etc/rc.d/S98samba4: No space left on device
ln: /etc/rc.d/K05samba4: No space left on device
ln: /etc/samba/smb.conf: No space left on device
Configuring python3-pip.
Configuring uhttpd.
ln: /etc/rc.d/S50uhttpd: No space left on device
verify_pkg_installable: Only have 0kb available on filesystem /overlay, pkg fdisk needs 55

root@OpenWrt:/# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.8M      4.8M         0 100% /rom
tmpfs                   496.5M    948.0K    495.5M   0% /tmp
/dev/mmcblk1p66         199.4M    199.4M         0 100% /overlay
tmpfs                   496.5M     52.0K    496.4M   0% /tmp/root
overlayfs:/tmp/root     496.5M     52.0K    496.4M   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

Of course don't just waste that space. Go ahead and install autopart to automatically assign it into an LVM2 physical volume, then you can use uvol to easily create additional storage volume for larger things. Assigning all that extra space to rootfs-overlay doesn't make too much sense imho as there aren't as many things you can install and life data wouldn't survive sysupgrade unless you back it up (or have sysupgrade back it up for you, but that's silly for gigabytes of data on every sysupgrade).
I use external storage volumes e.g. for Grafana (in an Alpine container) or additional web-content served by the device.

During the build, what exactly dictates the partition size that's changeable if I want an image with custom partition size without any post install steps? In this case for example if I want a larger overlayfs partition, what tells it to make the partition of this explicit size (199.4Mb)

The strange thing I found however, is that the overlay partition is showing to be 2.1G instead of 199M.

root@OpenWrt:/# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0     31:0    0   512K  1 disk
mtdblock1     31:2    0     2M  1 disk
mtdblock2     31:4    0 125.5M  0 disk
mmcblk1      179:0    0  29.1G  0 disk
├─mmcblk1p1  179:1    0   495K  0 part
├─mmcblk1p2  179:2    0   512K  0 part
├─mmcblk1p3  179:3    0     2M  0 part
├─mmcblk1p4  179:4    0     1M  0 part
├─mmcblk1p5  179:5    0    32M  0 part
├─mmcblk1p6  179:6    0     7M  0 part
├─mmcblk1p7  179:7    0   2.1G  0 part
├─mmcblk1p8  259:0    0    27G  0 part
├─mmcblk1p65 259:1    0   4.4M  0 part /rom
└─mmcblk1p66 259:2    0   2.1G  0 part /overlay
mmcblk0      179:8    0   7.3G  0 disk
mmcblk0boot0 179:16   0     4M  1 disk
mmcblk0boot1 179:24   0     4M  1 disk
ubiblock0_0  259:3    0  83.3M  0 disk

Update:

I installed the autopart, but don't know how to use uvol. What is the next step after installing autopart?

Did you already change the size of the partition in the target/linux/mediatek/image/mt7622.mk? https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=target/linux/mediatek/image/mt7622.mk;h=6077e623ffd61269b0178c0912788e8b43855fe3;hb=HEAD#l51

Because the default is to use the first 256MB of an SD card or the first 1GB of the eMMC for OpenWrt. Hence I'm also surprised to see that you got 2.1GB there in mmcblk1p7 (which is then split further into mmcblk1p65 and mmcblk1p66).

Anyway, autopart worked and created a 27GB LVM2 physical volume which you can now use using uvol:

# show free space in bytes
uvol free
# create new read/write volume with 16GB of space
uvol create n007 17179869184 rw
uvol up n007
# now you got a lot of space in /var/run/uvol/n007
1 Like

Still working on the build, will update the thread :slight_smile: