Strategy of setting up OpenWrt on 'large' system for easy upgrade, security & expansion for long haul

Installing and Using OpenWrt
1

I just bought a new router board (Banana Pi R4). It comes with built-in NAND and eMMC.

  • 128 MB NAND (onboard, pretty slow 2MB/s -ish)
  • 8 GB eMMC (onboard, fast in 100+ MB/s but haven't actually measured yet)

Banana Pi R4 has DIP switches to select where to boot from. So I plan to use the 128 MB NAND for a minimal OpenWrt installation. This installation will act as a 'recovery/emergency' contingency. It'll get updated from time to time but much less frequent than the main installation.

I plan to use the 8 GB eMMC as the main installation of OpenWrt. It'll be minimal and be the same image (and config etc) to the image installed on the 128 MiB NAND. This main installation will get updated as soon as OpenWrt stable is refreshed.

The OpenWrt installation will be minimal for my router to perform great as a router. The planned 'rootfs + rootfs_data' will be ~500MB which is the current OpenWrt default for my target. I want to deviate as little as possible from OpenWrt defaults so that on-going customisation & maintenance effort will be minimum for long haul, say, 10+ years.

Besides R4 as a router, I plan to run a couple of LXC containers

  • one for Internet facing services
  • one for LAN facing services (non router functions)
  • one or more for other stuff

All these containers will be orchestrated by OpenWrt's LXC package. Inside the containers all will run one and same copy of ArchLinux for ARM installation.

So the 8 GB eMMC will be 'partitioned' into one 500 MB for OpenWrt and 7.5 GB for ArchLinux for ARM, the containers, application data and limited user data. And my original thought on making use of the 8 GB eMMC through extroot:

But after stumbling on this discussion of extroot:

I believe extroot not a good idea, at least for my intended usage. In place of extroot, my alternative plan is to create a partition of 7.5 GB and mount it inside OpenWrt. This will retain everything stored in the 7.5 GB partition and survive OpenWrt upgrades without extra work.

Any critics? Suggestions of alternative & better practice? Soliciting feedback from anyone who had attempted or is doing something similar.

This is my ongoing exploration for the coming weeks. Sorry being a long post and thanks for spending the time to finish it here.

2

The RAX3000M eMMC version, comes with 64GB eMMC flash, only us$25/set in china.
https://openwrt.org/toh/hwdata/cmcc/cmcc_rax3000m

3

CMCC is China Mobile, the giant network operator in China. For unknown reason, perhaps surplus? RAX3000M floods the grey market. I believe $25 is heavily 'subsidized' price. People should grab one when it lasts and it suits your need and purposes.

OpenWrt One will have the same SoC and radio as in RAX3000M. Comes with 128MB SPI NAND. Hope future owners won't see your post :smile:

And we digressed...

4

Agree, RAX3000M factory firmware, cannot use on non-China Mobile network.
Install openwrt......OK.

I believe the qty. of the model, more than 10k in the market.

5

Used a 8GB micro-SD as a test medium and took the baby steps for 'proof of concept'. Successfully created the "7.5 GB partition" and get it auto mounted when OpenWrt starts up.

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0    31:0    0     2M  1 disk
mtdblock1    31:1    0   126M  0 disk
mmcblk0     179:0    0   7.3G  0 disk
├─mmcblk0p1 179:1    0     4M  0 part
├─mmcblk0p2 179:2    0   512K  0 part
├─mmcblk0p3 179:3    0     2M  0 part
├─mmcblk0p4 179:4    0     4M  0 part
├─mmcblk0p5 179:5    0    32M  0 part
├─mmcblk0p6 179:6    0    20M  0 part
├─mmcblk0p7 179:7    0   448M  0 part
└─mmcblk0p8 259:0    0   6.8G  0 part /opt
ubiblock0_1 254:0    0  54.9M  0 disk
fit0        259:1    0   7.8M  1 disk /rom
fitrw       259:2    0 434.6M  0 disk /overlay

mmcblk0 is the 8GB micro-SD test medium. Partitions 1 to 7 are created by flashing OpenWrt image. Then I manually created 'mmcblk0p8' to take up rest of the disk, which is about 6.8GiB in reality.

Partition p1 holds BL2 boot loader.
Partition p4 holds BL31 + BL33 as part of U-Boot package.
Partition p7 holds kernel + fdt + rootfs + roofs_data

All these are automatically done by the magic of OpenWrt image. I spell it out so that I won't forget and may be of interest to some in the audience.

Tested sysupgrade and it can handle nicely everything on partition p7. When necessary, I could manually update p1 and p4

Partition p8 is configured to automatically mounted on /opt. Its content will be manually managed by me. By that in the case of ArchLinux rootfs, it'll be automagically managed by me running pacman.

As for the choice of filesystem for p8, I went with f2fs because for unknown reason I can't find UBIFS on my ArchLinux PC (anyone knows why?) but f2fs is available. In the rare event, I could investigate any damaged p8 on my ArchLinux PC.

f2fs is built-in and supported by default in OpenWrt. Hence, also one less thing to customise. Played, abused and toasted a few rounds on p8 and the micro-SD card. So far pretty solid. Knock on wood.

Sad news is that I just realised multiple LXC containers can't share one rootfs. Need to start some reading & digging.

1 Like
6

Things progressed a bit faster than I expected. LXC is up and running off Partition p8. For testing, two containers run default ArchLinux for ARM from the LXC team.

One is created as privileged container. The other is unprivileged container. For production, I plan to only run unprivileged containers for security reason.

# ll /opt/srv/lxc
drwxr-xr-x    4 root     root          3452 Jun 19 04:58 ./
drwxr-xr-x    3 root     root          3452 Jun 19 02:55 ../
drwxrwx---    3 100000   100000        3452 Jun 19 05:00 ups1/
drwxrwx---    3 root     root          3452 Jun 19 04:42 vps1/

# find /opt/srv/lxc -maxdepth 2
/opt/srv/lxc
/opt/srv/lxc/vps1
/opt/srv/lxc/vps1/rootfs
/opt/srv/lxc/vps1/config
/opt/srv/lxc/ups1
/opt/srv/lxc/ups1/rootfs
/opt/srv/lxc/ups1/config

# du -ksh /opt/srv/lxc/*
811.3M	/opt/srv/lxc/ups1
809.4M	/opt/srv/lxc/vps1

The two containers have its own copy of ROOTFS. Although they run the same ArchLinux, they don't share the same base ROOTFS at the moment. So not efficient use of storage space on Partition p8.

OpenWrt's Guide is pretty self-sufficient. Except that you will not want to install 'cgroupfs-mount' package. It prevented me from starting containers. Some details in this post:

Also worth noting, on Banana Pi R4, the LAN bridge is named 'br-lan'. Replace 'lxcbr0' inside /etc/lxc/default.conf to hook up my containers to the LAN bridge. Or you may create lxcbr0 but I don't see the benefit. May look into it on a future time.

Now I recall two more types of error in running unprivileged containers. Here they're and also the fixes. So that I won't forget and may help someone in the future.

lxc: Operation not permitted - Failed to mount "proc"
lxc: Operation not permitted - Failed to mount "sys"

The workaround to the above errors is to add the following two lines to /etc/rc.local:

mount -o remount,rw,nosuid,nodev,noexec,relatime proc /proc
mount -o remount,rw,nodev,noexec,relatime sysfs /sys

Source / Credit:

mount: /sys/kernel/debug: permission denied.
mount: /sys/kernel/config: permission denied.

The above two errors won't stop guest containers from running. But would be nice to get rid of them anyway. To fix, inside the guest container and run:

systemctl mask sys-kernel-debug.mount
systemctl mask sys-kernel-config.mount

Source / Credit:

Again played, abused and toasted a few rounds. Sysupgrade'ed multiple times. So far pretty solid. Meet my original requirements. A bit more efficient use of storage space will be a bonus.

--

ArchLinux ROOTFS is about 800MiB. The LXC team already trims down a bit from >1GiB official images by the ArchLinux for ARM team. If I spend some effort, perhaps can further trimmed down to 500MiB. Stuff like manpages, header files and docs aren't needed.

Alpine ROOTFS is much smaller, starting from ~20MiB. I might decide to use Alpine. Or perhaps run one big ArchLinux for internet-facing services, and re-use the OpenWrt host for intranet-facing services.

The idea to share a base ROOTFS and stack an overlayfs on top saves space regardless of the ROOTFS size of a Linux favour. I might find time to give it a try. But for now consider this thread done.

OpenWrt (the software & the community) is great! It took much less time than I originally anticipated.

As usual, critics, suggestions, better practices, especially critics are welcome.

update

  • added a hyperlink to OpenWrt's Guide to LXC
  • added two more types of errors (and their solutions) that I met when running unprivileged containers
7

This turned out simpler than I thought. I started with, at OS level, creating the layered overlayfs of the ROOTFS. Pass that rootfs to LXC without LXC realizing it's running on top of overlayfs. Surprisingly it worked. But then I figured LXC already has overlayfs support built-in. So let me skip over and present this simpler approach. I can't believe it's that straight forward.

Take Alpine Linux as genius pig. First let's create a 'base' container. 'Base' in a sense like a 'class' in OO programming:

lxc-create -n albase -t download -- --dist alpine --release 3.20 --arch arm64

Then, let's initiate two 'instances' of this base container. The two instances will share the same ROOTFS of the base container:

lxc-create -n alups1 -t none
lxc-create -n alups2 -t none

Now here is the trick which I haven't found a better way to deal with but at least the following manual way works for me:

mkdir -p /opt/srv/lxc/alups1/overlay/upper
mkdir -p /opt/srv/lxc/alups2/overlay/upper

Note 100000 below is the re-mapped UID and GID for on my OpenWrt host for running unprivileged containers. Plenty of tutorials will show you how to create them when creating unprivileged containers. Let's assume they're created already here. Then:

chown 100000:100000 -R /opt/srv/lxc/alups1/overlay
chown 100000:100000 -R /opt/srv/lxc/alups2/overlay

For /opt/srv/lxc/alups1/config, copy from /opt/srv/lxc/albase/config and overwrite the content except these three lines:

lxc.rootfs.path = overlayfs:/opt/srv/lxc/albase/rootfs:/opt/srv/lxc/alups1/overlay/upper
lxc.uts.name = <keep the original value>
lxc.net.0.hwaddr = <keep the original value>

For /opt/srv/lxc/alups2/config, again copy from /opt/srv/lxc/albase/config and overwrite the content except these three lines:

lxc.rootfs.path = overlayfs:/opt/srv/lxc/albase/rootfs:/opt/srv/lxc/alups2/overlay/upper
lxc.uts.name = <keep the original value>
lxc.net.0.hwaddr = <keep the original value>

As we might have guessed, the very important line is 'lxc.rootfs.path'. It's easy to guess its meaning. If not, look up LXC manpage.

Bravo. We're ready to launch the two unprivileged and 'instantiated' containers. Check disk usage to make sure that 'alups1' and 'alups2' are indeed sharing the ROOTFS of 'albase':

# du -ksh /opt/srv/lxc/*
14.2M	/opt/srv/lxc/albase
59.0K	/opt/srv/lxc/alups1
59.0K	/opt/srv/lxc/alups2

EE
EE1910×1368 521 KB

Information seems scarce online. This post is perhaps the first one to lay bare how to use LXC built-in overlayfs. I can understand why LXC isn't promoting the feature. Careless users will shoot themselves in the foot. IMO, it's a matter of IT policy.

For example, system update should only happen in 'albase' the base container. Never in the instantiated containers, 'alups1' and 'alups2'. Will play around for a week or two and see if any other issues.

I believe I accomplished all my goals in the OP in a surprisingly short time and as a first-time OpenWrt user. :smiley:

Critics, suggestions, better practices are welcome as usual.

1 Like
8

Hi, have you run archlinux in opener? Or just Linux container in openwrt.

If I get an RAX3000M, then I have 64gb emmc, I wonder if I could run a full version of Ubuntu server in it.

9

For the purpose of this discussion, we shall divide a Linux distribution e.g. Ubuntu server into:

  • boot loader(s)
  • kernel
  • user space i.e. everything else. Most users actually see this as full distro

For RAX3000M, you definitely can run full Ubuntu server with the caveats that you still need

  • boot loaders from OpenWrt
  • kernel from OpenWrt

and you need to do some surgery to stitch them together.

Unless for science purpose, I would suggest to stick with LXC containers. The performance of Ubuntu server will be same running bare metal vs inside LXC on RAX3000M.

1 Like
10

So this is already done above. But I was thinking if we could improve upon it and make the whole system a bit more streamlined and coherent. The 'new' idea in my mind was to run it like a x86-64 PC box. The root partition will be persistent storage writable, and going forward just update like a Linux PC.

I saw the option to compile a 'ext4 rootfs' and I saw the boot loaders, dts binary blobs already there in the OpenWrt build directory. So why not try to stitch them together for a spin?

I dumped the 'ext4 rootfs' to '/dev/mmcblk0p7' on an existing squashfs-style image disk. mkdir '/boot' and copied kernel and dts blobs into there. Booted into U-Boot. Poked around various macro's and tried to load the dts blobs and the kernel. All loaded by using 'ext4load' command. But failed to execute by issuing 'booti' command for the obvious reason that I had little idea about U-Boot.

By looking at the existing U-Boot macro's, I got the impression that it's very well written to be fail safe. And then I stumbled on this post (which I did a screen capture but couldn't quickly find the URL in my browser cache):

EEE
EEE1392×524 256 KB

That's the moment I stopped digging further because:

  • it deviates from my original goal that stays to OpenWrt's default as much as possible to minimize customisation & maintenance on the long run
  • I can't quickly hack up a U-boot script to be as fail-safe as OpenWrt's default. Not one that's even close to functional.
  • I'll lose the ability to use F2FS for ROOTFS without further effort
  • I'll have to add custom build workflow to package for 'ext4 rootfs'

With that said, the trend for ARM systems growing bigger & bigger is a sure thing. For a device like Banana Pi R4, it is 'big' enough to not function like a traditional consumer device. So seems making lot of sense 'ext4 rootfs' will be made a standard feature/option in a future OpenWrt release and made as fail-safe as squashfs counterparts.

11

Well, in order to achieve that it will have to be read-only. Using ext4 instead of squashfs could still be seen as an advantage because it's much faster and less resource-hungry than reading from squashfs. Android does it that way as well...

Having a single read/write rootfs like traditional desktop or server Linux distributions is not really feasible on headless devices as the user doesn't have the option to "boot the old kernel" in GRUB menu or choose single-user boot, simply due to the typical lack of a local console.

A button and some LEDs is usually as good as it gets for OpenWrt devices (without having to open the case), so that has to be enough. Being able to more or less safely update remote devices, or even devices mounted in hard-to-access places like roofs or towers, is another key feature of our distribution which drives us towards trying to eliminate any possible single point of failure (such as a /boot filesystem, or a single read/write rootfs).

1 Like
12

Hi, I've installed a docker / docker-compose on my RAX3000M eMMC, openwret 23.05.3.
Will try to install a database into the docker.

I donot need to run a full-version linux in the docker, my target is an IoT database. Thanks.

1 Like
13

How did you get this working? I was able to create the extra partition and use it, but after a sysupgrade the partition was removed. I guess the GPT header was overwritten but the partition and the data is still intact - the GPT header just needs to be edited to be aware of the partition.

I don't know how to do this, but am curious how you performed a sysupgrade so that this problem didn't occur?