Strategy of setting up OpenWrt on 'large' system for easy upgrade, security & expansion for long haul

Installing and Using OpenWrt
6

Things progressed a bit faster than I expected. LXC is up and running off Partition p8. For testing, two containers run default ArchLinux for ARM from the LXC team.

One is created as privileged container. The other is unprivileged container. For production, I plan to only run unprivileged containers for security reason.

# ll /opt/srv/lxc
drwxr-xr-x    4 root     root          3452 Jun 19 04:58 ./
drwxr-xr-x    3 root     root          3452 Jun 19 02:55 ../
drwxrwx---    3 100000   100000        3452 Jun 19 05:00 ups1/
drwxrwx---    3 root     root          3452 Jun 19 04:42 vps1/

# find /opt/srv/lxc -maxdepth 2
/opt/srv/lxc
/opt/srv/lxc/vps1
/opt/srv/lxc/vps1/rootfs
/opt/srv/lxc/vps1/config
/opt/srv/lxc/ups1
/opt/srv/lxc/ups1/rootfs
/opt/srv/lxc/ups1/config

# du -ksh /opt/srv/lxc/*
811.3M	/opt/srv/lxc/ups1
809.4M	/opt/srv/lxc/vps1

The two containers have its own copy of ROOTFS. Although they run the same ArchLinux, they don't share the same base ROOTFS at the moment. So not efficient use of storage space on Partition p8.

OpenWrt's Guide is pretty self-sufficient. Except that you will not want to install 'cgroupfs-mount' package. It prevented me from starting containers. Some details in this post:

Also worth noting, on Banana Pi R4, the LAN bridge is named 'br-lan'. Replace 'lxcbr0' inside /etc/lxc/default.conf to hook up my containers to the LAN bridge. Or you may create lxcbr0 but I don't see the benefit. May look into it on a future time.

Now I recall two more types of error in running unprivileged containers. Here they're and also the fixes. So that I won't forget and may help someone in the future.

lxc: Operation not permitted - Failed to mount "proc"
lxc: Operation not permitted - Failed to mount "sys"

The workaround to the above errors is to add the following two lines to /etc/rc.local:

mount -o remount,rw,nosuid,nodev,noexec,relatime proc /proc
mount -o remount,rw,nodev,noexec,relatime sysfs /sys

Source / Credit:

mount: /sys/kernel/debug: permission denied.
mount: /sys/kernel/config: permission denied.

The above two errors won't stop guest containers from running. But would be nice to get rid of them anyway. To fix, inside the guest container and run:

systemctl mask sys-kernel-debug.mount
systemctl mask sys-kernel-config.mount

Source / Credit:

Again played, abused and toasted a few rounds. Sysupgrade'ed multiple times. So far pretty solid. Meet my original requirements. A bit more efficient use of storage space will be a bonus.

--

ArchLinux ROOTFS is about 800MiB. The LXC team already trims down a bit from >1GiB official images by the ArchLinux for ARM team. If I spend some effort, perhaps can further trimmed down to 500MiB. Stuff like manpages, header files and docs aren't needed.

Alpine ROOTFS is much smaller, starting from ~20MiB. I might decide to use Alpine. Or perhaps run one big ArchLinux for internet-facing services, and re-use the OpenWrt host for intranet-facing services.

The idea to share a base ROOTFS and stack an overlayfs on top saves space regardless of the ROOTFS size of a Linux favour. I might find time to give it a try. But for now consider this thread done.

OpenWrt (the software & the community) is great! It took much less time than I originally anticipated.

As usual, critics, suggestions, better practices, especially critics are welcome.

update

  • added a hyperlink to OpenWrt's Guide to LXC
  • added two more types of errors (and their solutions) that I met when running unprivileged containers