How to add a data partition at build time

For Developers
1

I am building a product based on openwrt, and I have a need to construct a separate data partition at build time.

Presently I am using an R2S with an sd card, but will also be supporting X86 platforms and likely other ARM platforms also. I am using the ext4 filesystem pretty much throughout, though at a later point I might convert the rootfs to squashfs.

I have been fiddling with scripts/gen_image_generic.sh because this is where partition tables are constructed. I have modified is like this:

#!/bin/sh
# Copyright (C) 2006-2012 OpenWrt.org
set -e -x
if [ $# -ne 5 ] && [ $# -ne 6 ]; then
    echo "SYNTAX: $0 <file> <kernel size> <kernel directory> <rootfs size> <rootfs image> [<align>]"
    exit 1
fi

OUTPUT="$1"
KERNELSIZE="$2"
KERNELDIR="$3"
#ROOTFSSIZE="$4"
ROOTFSIMAGE="$5"
ALIGN="$6"

# DATAFSSIZE is addition to get a new partition into the image
DATAFSSIZE="10"

# MOD:  Set ROOTFSSIZE in .config as the real root filesystem size plus the data file system size
# Then in this script, we shorten the root file system size by the amount of data file system size to make
# everyone happy
ROOTFSSIZE="$(($4 - DATAFSSIZE))"

rm -f "$OUTPUT"

head=16
sect=63

# create partition table
set $(ptgen -o "$OUTPUT" -h $head -s $sect ${GUID:+-g} -p "${KERNELSIZE}m" -p "${DATAFSSIZE}m" -p "${ROOTFSSIZE}m" ${ALIGN:+-l $ALIGN} ${SIGNATURE:+-S 0x$SIGNATURE} ${GUID:+-G $GUID})

KERNELOFFSET="$(($1 / 512))"
KERNELSIZE="$2"
ROOTFSOFFSET="$(($5 / 512))"
ROOTFSSIZE="$(($6 / 512))"
DATAOFFSET="$(($3 / 512))"
DATAFSSIZE="$(($4 / 512))"

[ -n "$PADDING" ] && dd if=/dev/zero of="$OUTPUT" bs=512 seek="$ROOTFSOFFSET" conv=notrunc count="$ROOTFSSIZE" && dd if=/dev/zero of="$OUTPUT" bs=512 seek="$DATAFSOFFSET" conv=notrunc count="$DATAFSSIZE"
dd if="$ROOTFSIMAGE" of="$OUTPUT" bs=512 seek="$ROOTFSOFFSET" conv=notrunc

if [ -n "$GUID" ]; then
    [ -n "$PADDING" ] && dd if=/dev/zero of="$OUTPUT" bs=512 seek="$((ROOTFSOFFSET + ROOTFSSIZE))" conv=notrunc count="$sect"
    mkfs.fat -n kernel -C "$OUTPUT.kernel" -S 512 "$((KERNELSIZE / 1024))"
    mcopy -s -i "$OUTPUT.kernel" "$KERNELDIR"/* ::/
else
    make_ext4fs -J -L kernel -l "$KERNELSIZE" "$OUTPUT.kernel" "$KERNELDIR"
    make_ext4fs -J -L data -l "$DATAFSSIZE" "Data"
fi
dd if="$OUTPUT.kernel" of="$OUTPUT" bs=512 seek="$KERNELOFFSET" conv=notrunc
rm -f "$OUTPUT.kernel"

I also have tried building my data partition after the rootfs.

This mechanism is not working. It would appear that $OUTPUT is being constructed somewhere else, and my offsets into my last partition, regardless of whether that partition is my data partition or the rootfs partition, are showing up as beyond the end of the drive.

What I am getting back when I build looks like this:

# Creates the final SD/eMMC images, 
# combining boot partition, root partition as well as the u-boot bootloader
# Generate a new partition table in /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz with 32 MiB of 
# alignment padding for the idbloader and u-boot to fit:
# http://opensource.rock-chips.com/wiki_Boot_option#Boot_flow
#
# U-Boot SPL expects the U-Boot ITB to be located at sector 0x4000 (8 MiB) on the MMC storage
/mnt/sdd5/home/jiml/buildenv/openwrt/scripts/gen_image_generic.sh /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz 16 /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz.boot 114 /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/root.ext4 32768
+ '[' 6 -ne 5 ']'
+ '[' 6 -ne 6 ']'
+ OUTPUT=/mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz
+ KERNELSIZE=16
+ KERNELDIR=/mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz.boot
+ ROOTFSIMAGE=/mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/root.ext4
+ ALIGN=32768
+ DATAFSSIZE=10
+ ROOTFSSIZE=104
+ rm -f /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz
+ head=16
+ sect=63
++ ptgen -o /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz -h 16 -s 63 -p 16m -p 10m -p 104m -l 32768
part 0 16384
part 0 10240
part 0 106496
+ set 33554432 16777216 67108864 10485760 100663296 109051904
+ KERNELOFFSET=65536
+ KERNELSIZE=16777216
+ ROOTFSOFFSET=196608
+ ROOTFSSIZE=212992
+ DATAOFFSET=131072
+ DATAFSSIZE=20480
+ '[' -n '' ']'
+ dd if=/mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/root.ext4 of=/mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz bs=512 seek=196608 conv=notrunc
233472+0 records in
233472+0 records out
119537664 bytes (120 MB, 114 MiB) copied, 0.200843 s, 595 MB/s
+ '[' -n '' ']'
+ make_ext4fs -J -L kernel -l 16777216 /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz.kernel /mnt/sdd5/home/jiml/buildenv/openwrt/build_dir/target-aarch64_generic_musl/linux-rockchip_armv8/tmp/openwrt-rockchip-armv8-friendlyarm_nanopi-r2s-ext4-sysupgrade.img.gz.boot
Creating filesystem with parameters:
    Size: 16777216
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 1024
    Inode size: 256
    Journal blocks: 0
    Label: kernel
    Blocks: 4096
    Block groups: 1
    Reserved blocks: 0
    Reserved block group size: 7
Created filesystem with 14/1024 inodes and 3120/4096 blocks
+ make_ext4fs -J -L data -l 20480 Data
Creating filesystem with parameters:
    Size: 20480
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 16
    Inode size: 256
    Journal blocks: 0
    Label: data
    Blocks: 0
    Block groups: 0
    Reserved blocks: 0
    Reserved block group size: 7
error: make_ext4fs_internal: failed to reserve first 10 inodes

I think what what I want to accomplish is fairly obvious, and I am quite willing to modify the environment if I must do so.

So can anyone tell me what I need to do to get this to work?

2

Given that there is no response after two days, I presume that there is literally no support in the OpenWRT build environment for what I want to do.

However, I did successfully modify scripts/gen_image_generic.sh sufficiently to do my job. For posterity, for anyone else with this problem, and against the possibility that in the future openwrt would decide to support the capability (this mod isn't adequate for a general-purpose capability, but it works for me), here is my modified gen_image_generic.sh file:

#!/bin/sh
# Copyright (C) 2006-2012 OpenWrt.org
set -e -x
if [ $# -ne 5 ] && [ $# -ne 6 ]; then
    echo "SYNTAX: $0 <file> <kernel size> <kernel directory> <rootfs size> <rootfs image> [<align>]"
    exit 1
fi
OUTPUT="$1"
KERNELSIZE="$2"
KERNELDIR="$3"
ROOTFSSIZE="$4"
ROOTFSIMAGE="$5"
ALIGN="$6"

# DATAFSSIZEMEG is addition to get a new data partition into the image
DATAFSSIZEMEG="64"

rm -f "$OUTPUT"

head=16
sect=63

# create partition table
set $(ptgen -o "$OUTPUT" -h $head -s $sect ${GUID:+-g} -p "${KERNELSIZE}m" -p "${ROOTFSSIZE}m" -p "${DATAFSSIZEMEG}m" ${ALIGN:+-l $ALIGN} ${SIGNATURE:+-S 0x$SIGNATURE} ${GUID:+-G $GUID})

KERNELOFFSET="$(($1 / 512))"
KERNELSIZE="$2"
ROOTFSOFFSET="$(($3 / 512))"
ROOTFSSIZE="$(($4 / 512))"
# more modified stuff
DATAOFFSETBYTES="$5"
DATAOFFSET="$(($5 / 512))"
DATAFSSIZEBYTES="$6"
DATAFSSIZE="$(($6 / 512))"

[ -n "$PADDING" ] && dd if=/dev/zero of="$OUTPUT" bs=512 seek="$ROOTFSOFFSET" conv=notrunc count="$((KERNELSIZE + DATAFSSIZE + 1))"
dd if="$ROOTFSIMAGE" of="$OUTPUT" bs=512 seek="$ROOTFSOFFSET" conv=notrunc

if [ -n "$GUID" ]; then
    [ -n "$PADDING" ] && dd if=/dev/zero of="$OUTPUT" bs=512 seek="$((ROOTFSOFFSET + ROOTFSSIZE))" conv=notrunc count="$sect"
    mkfs.fat -n kernel -C "$OUTPUT.kernel" -S 512 "$((KERNELSIZE / 1024))"
    mcopy -s -i "$OUTPUT.kernel" "$KERNELDIR"/* ::/
else
    make_ext4fs -J -L kernel -l "$KERNELSIZE" "$OUTPUT.kernel" "$KERNELDIR"
#    make_ext4fs -J -b 1024 -g 8192 -I 128 -i 35856 -L Data -l "$DATAFSSIZE" "Data"
fi
dd if="$OUTPUT.kernel" of="$OUTPUT" bs=512 seek="$KERNELOFFSET" conv=notrunc
rm -f "$OUTPUT.kernel"

# Mod to get data partition into the build.  Note that when file/cf is pushed into the package, the partition is VERY
# small.  It works now but we better not put any more stuff in cf/conf for install at build time.  Only after the package is
# enlarged is there room
make_ext4fs -J -b 1024 -g 8192 -I 128 -L Data -l "$DATAFSSIZE" "$OUTPUT.Data" "$TOPDIR/files/cf"
dd if="$OUTPUT.Data" of="$OUTPUT" bs=512 seek="$DATAOFFSET" conv=notrunc
rm -f "$OUTPUT.Data"
dd if=/dev/zero of="$TMP_DIR/newpkg.img" bs=512 count="$((DATAOFFSET + DATAFSSIZE + 1))" conv=notrunc
dd if="$OUTPUT" of="$TMP_DIR/newpkg.img" bs=512 conv=notrunc
/sbin/losetup -o "$DATAOFFSETBYTES" /dev/loop5 "$TMP_DIR/newpkg.img"
/sbin/resize2fs /dev/loop5 "${DATAFSSIZEMEG}M"
/sbin/losetup -d /dev/loop5
mv "$TMP_DIR/newpkg.img" "$OUTPUT"
1 Like
3

Maybe take a look at the autopart package in packages feed which auto-allocates all remaining space on the block device which is used for booting to LVM2.
You can then use uvol to create a data volume on first boot (if not existing), for instance using an init script, uci-defaults script or /etc/rc.local.
uvol works on eMMC/SD card/block device targets (using LVM2) as well as UBI/NAND flash-based boards (using UBI directly).