Using extroot is insecured for data because it is easy to unplug and steal the removable storage device. I spent a whole day trying to do a full disk encryption, but failed at the end.
I tried to encrypt the whole /dev/sda, and split it into multiple LVM logical volumes, with one of them being used for extroot. A block hotplug event script was used to auto-decrypt the sda at boot. The problem is the boot process: the hotplug script is run very late, after the kernel is tired of waiting for the extroot.
[ 1.748122] VFS: Mounted root (squashfs filesystem) readonly on device 31:5.
[ 1.760110] Freeing unused kernel memory: 1268K
[ 1.764649] This architecture does not have kernel memory protection.
[ 1.771247] Run /sbin/init as init process
[ 1.775346] with arguments:
[ 1.775353] /sbin/init
[ 1.775358] with environment:
[ 1.775363] HOME=/
[ 1.775368] TERM=linux
[ 2.299058] init: Console is alive
[ 2.302880] init: - watchdog -
[ 3.169223] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[ 8.956644] SCSI subsystem initialized
[ 8.994480] usb-storage 2-1:1.0: USB Mass Storage device detected
[ 9.001698] scsi host0: usb-storage 2-1:1.0
[ 9.006859] usbcore: registered new interface driver usb-storage
[ 9.018196] usbcore: registered new interface driver uas
[ 9.069182] encrypted_key: failed to alloc_cipher (-2)
[ 9.275475] kmodloader: 2 modules could not be probed
[ 9.280636] kmodloader: dependency not loaded encrypted-keys
[ 9.286421] kmodloader: - dm-crypt - 1
[ 9.290223] kmodloader: - encrypted-keys - 0
[ 9.532091] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[ 9.542028] block: extroot: device not present, retrying in 15 seconds
[ 10.056418] scsi 0:0:0:0: Direct-Access USB SanDisk 3.2Gen1 1.00 PQ: 0 ANSI: 6
[ 10.066771] sd 0:0:0:0: [sda] 60088320 512-byte logical blocks: (30.8 GB/28.7 GiB)
[ 10.074834] sd 0:0:0:0: [sda] Write Protect is off
[ 10.079682] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 10.080139] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 10.101121] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 24.666716] block: extroot: cannot find device with UUID 83b1246c-14fa-4ccb-890d-806093f3bfe7
[ 24.676711] mount_root: switching to jffs2 overlay
[ 24.697774] overlayfs: upper fs does not support tmpfile.
[ 24.723591] urandom-seed: Seeding with /etc/urandom.seed
[ 24.870859] mt7530 mdio-bus:1f lan1: Link is Down
[ 24.889060] procd: - early -
[ 24.892186] procd: - watchdog -
[ 25.545314] procd: - watchdog -
[ 25.549215] procd: - ubus -
[ 25.642229] procd: - init -
[ 26.471820] kmodloader: loading kernel modules from /etc/modules.d/*
[ 47.377201] mt7530 mdio-bus:1f lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[ 47.384804] br-lan: port 1(lan1) entered blocking state
[ 47.390240] br-lan: port 1(lan1) entered forwarding state
[ 47.653110] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[ 47.814243] mt7530 mdio-bus:1f wan: Link is Up - 1Gbps/Full - flow control rx/tx
[ 47.822014] IPv6: ADDRCONF(NETDEV_CHANGE): wan: link becomes ready
[ 51.990100] Device '/dev/sda' was decrypted successfully and mapped to 'decrypted_storage'.
As you can see, it tried to load extroot as soon as [ 9.542028] but failed. I set delay_root to 15 secs, yet it still waited for nothing and gave up at [ 24.676711]. Then after everything started up, it finally ran my decryption script at [ 51.990100].
The extroot configuration tutorial is incomplete, as it instructs the users to blindly follow shell commands without explaining the boot process, and the boot process pages are no use because they don't explain how the order the partitions are mounted nor how the fstab file is processed. Documentation is really a big obstacle for me, and diving into C code is not an option.
Is there any way to achieve extroot encryption with OpenWrt?
It would be by far easier to hot-glue the USB stick or chain it to the device, if stealing the USB stick (and not the whole device) is your only concern. For any kind of encryption of non(-pure)-data partitions which need to be available early on, you are facing a chicken and egg problem, which effectively isn't solvable for headless infrastructure devices.
It is clear that I want to protect the data from being stolen, not the physical devices. I always have backup copies of data, and spared devices available.
intruder steals extroot drive
They won't have a clue which software I installed on the router, how it is configured, which scripts are used, nor do they have access to the data on it.
intruder steals router
They end up with a cheap device which boots into its internal overlay (JFFS2) and works just like a Wireless Access Point.
intruder exchanges your router with an identical one, except there is some malware / spyware or similar added
The rogue device doesn't have the mechanism to decrypt my extroot, effectively brings the OpenWrt device offline. Once my software stops working, I immediately notice the problem.
intruder activates failsafe mode and gains access to your router and does $things
This one is tricky to solve because they can mount the internal overlay and see what I has done. After all, they need to know that the router is not running its original firmware (which they do not) and that they have to know how to use Linux and how to boot into OpenWrt fail-safe mode (they don't either). All I expect them to do is to unplugged the storage device and try to read it from a Windows or Linux PC, which gives them an idea which software I installed and how my shell scripts work. And I really need some protection in such scenarios.
TLDR; Here's a wiki section I created that may be a little clearer and explicit.
Have you tried adding export PREINIT=1; mount_root to /etc/rc.local as suggested in the troubleshooting section? I haven't confirmed, but I believe this could be the missing piece.
As mentioned without creating a special build, it looks like you can get extroot to work on an encypted device in preinit. However, once you've got everything up an running you should be able to setup the crypt device and switch root. I've done some testing to convince myself this works.
It should go without saying that you'll need to have all binaries required to bring up the crypt device installed into your rootfs_data. Here's some steps that I roughly did:
Make sure all needed packages are installed (eg. cryptsetup, which should pull in all the required kmods)
Create LUKS container with one of the supported extroot filesystems (eg. ext4)
Configure /etc/config/fstab as in the wiki using the UUID
The extroot will fail to mount in the preinit phase of the boot, that's expected
Edit /etc/rc.local to unlock the LUKS container and then do PREINIT=1 mount_root
If you require user input for the password (eg. not using a key file on the device), cryptsetup will fail as stdin is not setup. I'm sure there's a way to set it up, but I haven't loked into it yet.
You should not have the upper dir of the / overlay be in the LUKS container.
[OPTIONAL require user input for password] run ash /etc/rc.local from shell.
Since stdin is not setup for cryptsetup when rc.local is normally run, you'll need to manually run rc.local from a shell (which has stdin setup correctly).