I'm trying to setup ImageBuilder to build updates for my x86_64 setup of OpenWRT. Is there a way to skip building images for squashfs to save some storage and would that cut down the time make image takes (my previous runs took about 1 hour and 28 min)?
- how many packages...?
- pc speed / hard drive type?
- internet speed?
- rootfs size?
typical run from scratch is around 3-5mins
1 Like
Did not expect that, that would be amazing.
Here's the output of one of the runs hoping to answer rest of the questions, it was done using docker container of imagebuilder. The docker host is a VM on virtualbox with 2 cores of Intel i7-6498DU and 4GB of ram with storage of SSD SATA. I've set rootfs size to 3GB. I would notice that when it starts, the download of the packages would be rather quick (within 10 minutes). Though after Building images... appears is where the bulk of the time is being spent.
ouch! vm io bottlenecks mostly... smaller rootfs size (or faster disks) will help you most...
try this on for size...
sed -i 's/CONFIG_TARGET_ROOTFS_SQUASHFS=y//g' ./.config
mkdir -p packages
cp -a ./dl/*.ipk ./packages/
no-sudo
note: you shouldn't be using sudo here...
build@d07b9eb1cf0d:~/openwrt$ sudo make clean
rm -rf /home/build/openwrt/tmp /home/build/openwrt/dl /home/build/openwrt/build_dir/target-x86_64_musl/root-x86 /home/build/openwrt/bin/targets/x86/64
build@d07b9eb1cf0d:~/openwrt$ time sudo make image ...
next time you unpack a fresh image builder don't use it... ( but your current ib will likely be broken if you don't )
i think your disk is failing or you hypervisor is panicking because this aint right...
3221225472 bytes (3.2 GB, 3.0 GiB) copied, 162.673 s, 19.8 MB/s
your typical 200MB/s+ is massively constrained here...
this is the end stage from my community build with 300+ packages added no-squashfs and pre-downloaded ipks copied to packages, 1GB rootfs;
Disabling nft-qos
Enabling nlbwmon
Enabling ntop
Disabling nut-server
Enabling odhcpd
Disabling open-iscsi
Enabling openvpn
Enabling persistentdata
Enabling persistentlucistatistics
imagebuilder
Enabling persistentnlbwmon
Disabling pservice
Disabling quagga
Enabling rngd
Enabling rpcd
Disabling rssileds
Disabling simple-adblock
Disabling snmpd
Disabling socat
Disabling sqm
Enabling sysctl
Enabling sysfixtime
Enabling sysntpd
Enabling system
Disabling travelmate
Enabling ucitrack
Enabling uhttpd
Enabling umount
Enabling urandom_seed
Enabling usbmode
Disabling vpn-policy-routing
Disabling watchcat
Enabling wpad
Disabling xinetd
Disabling zerotier
Building images...
Creating filesystem with parameters:
Size: 1006632960
Block size: 4096
Blocks per group: 32768
Inodes per group: 7680
Inode size: 256
Journal blocks: 0
Label: rootfs
Blocks: 245760
Block groups: 8
Reserved blocks: 0
Reserved block group size: 63
Created filesystem with 11906/61440 inodes and 58180/245760 blocks
mkfs.fat: Warning: lowercase labels might not work properly on some systems
mkfs.fat 4.2 (2021-01-31)
+ [ 5 -eq 5 ]
+ OUTPUT=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz
+ BOOTFS=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz.boot
+ ROOTFS=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/root.ext4
+ BOOTFSSIZE=384
+ ROOTFSSIZE=960
+ head=4
+ sect=63
+ ptgen -o /fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz -h 4 -s 63 -l 4096 -t c -p 384M -t 83 -p 960M -S 0x10022301
part 0 393216
part 0 983040
+ set 4194304 402653184 411041792 1006632960
+ BOOTOFFSET=8192
+ BOOTSIZE=786432
+ ROOTFSOFFSET=802816
+ ROOTFSSIZE=1966080
+ dd bs=512 if=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz.boot of=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz seek=8192 conv=notrunc
786432+0 records in
786432+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 1.39888 s, 288 MB/s
+ dd bs=512 if=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/root.ext4 of=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz seek=802816 conv=notrunc
1966080+0 records in
1966080+0 records out
1006632960 bytes (1.0 GB, 960 MiB) copied, 3.74176 s, 269 MB/s
mkfs.fat: Warning: lowercase labels might not work properly on some systems
mkfs.fat 4.2 (2021-01-31)
+ [ 5 -eq 5 ]
+ OUTPUT=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz
+ BOOTFS=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz.boot
+ ROOTFS=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/root.ext4
+ BOOTFSSIZE=384
+ ROOTFSSIZE=960
+ head=4
+ sect=63
+ ptgen -o /fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz -h 4 -s 63 -l 4096 -t c -p 384M -t 83 -p 960M -S 0x10022301
part 0 393216
part 0 983040
+ set 4194304 402653184 411041792 1006632960
+ BOOTOFFSET=8192
+ BOOTSIZE=786432
+ ROOTFSOFFSET=802816
+ ROOTFSSIZE=1966080
+ dd bs=512 if=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz.boot of=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz seek=8192 conv=notrunc
786432+0 records in
786432+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 1.39779 s, 288 MB/s
+ dd bs=512 if=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/root.ext4 of=/fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-sysupgrade.img.gz seek=802816 conv=notrunc
1966080+0 records in
1966080+0 records out
1006632960 bytes (1.0 GB, 960 MiB) copied, 3.85762 s, 261 MB/s
cp /fs/sdd1/openwrt/RTNGext/ibminiSINGLE/rpi-4/binout/rpi-4_snapshot_20211001-1204-5/bootfiles.tar.gz /fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-special.tar.gz-fw
cp /fs/sdd1/openwrt/RTNGext/ibminiSINGLE/rpi-4/binout/rpi-4_snapshot_20211001-1204-5/*-rpi-4-rootfs.tar.gz /fs/sdd1/openwrt/RTNGext/ibminiSINGLE/r17637_somesmallstufflast5.10.64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/tmp/openwrt-bcm27xx-bcm2711-rpi-4-ext4-special.tar.gz-fw
Calculating checksums...
############### build success ############## 173
173 seconds = 3mins... notice the io around 200MB/s +
3 Likes
Awesome, thanks a lot for the comprehensive feedback! I will work on the IO bottleneck issue in my next run (probably in the next update release of openwrt) and will share some updates.
1 Like
The sudo is because some of the custom files being added are owned by root without sufficient rw privileges (docker container is running with user build). I fear that chmod or chown the custom files before building would be a bad idea.
1 Like
if you really want some speedup... attach a second VDI which resides on a second physical disk and set that as your BIN_DIR...
for images over GB this will vastly reduce build times...
(as will pre-allocating the disk aka not dynamically grow when you create the VDI)
1 Like
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.