Extroot over sshfs - How To?

Installing and Using OpenWrt
1

Any success yet in configuring extroot over sshfs?

Right now I'm stuck at mapping uid/gid.

I expected a no-brainer, but am already struggling the whole day.
Took some time to realize that in a mininmal installation, ssh client is provided by dropbear. And this one obviously not compatible with the ssh-options forwarded by sshfs.
But the openssh-client alone would costs the fortune of 1.6 Mbyte :woozy_face: of rare boot flash.

Anyway, finally, at least, I managed to generate a key pair, extract a ssh readable private key, connect and mount a path without asking for a password.

Many thanks to this link:

But I assume for a proper overlay fs, file ownership must be be conserved, or at least symmetrically translated, right?

2

You cannot have fuse extroot (or root)

3

sounds bad :cry:
what's the limitation?
kernel inside?

Because now, that at least I could manage

  • to sshfs mount as root
  • stay in a safe chroot jail on server
  • fail connection (instead of open access to server) if not properly configured
  • modify and keep file ownership for files copied in and out
Match Address 192.168.17.34
        AuthorizedKeysFile /home/sshfs_jails/gadget5/.ssh/authorized_keys
        ChrootDirectory /home/sshfs_jails/gadget5
        # PermitRootLogin yes
        # PermitRootLogin forced-commands-only
        PermitRootLogin prohibit-password
        ForceCommand internal-sftp
        AllowTcpForwarding no
4

Why?

For starters, the network is not initialized until after extroot is loaded.

The extroot process happens very early in the boot process, so many modules and services have not been loaded nor has the network hardware been brought up. So extroot over ssh just isn’t possible.

1 Like
5

Ok, I see.

And what about not so bloody starters?
They have to figure out themselves, eh :thinking: :wink: :innocent:

There are reports around with extroot on nfs or smbfs, right?
As far as I have heared, they might - sometimes .... - require network connectivity, too, don't they?

OK, admittedly, on openwrt I may be newbie.
But on a "real" (sorry, "full fledged") linux, there is a readonly root on startup which get's remounted rw when all the gear required for operation is up.
So my expectation was that with extroot, it might be similiar?

Of course, anything - including network and mounting the sshfs bust be performed from flash only.

Just installed block-mount to have a look at it.
Basically, I see two lines in /etc/init.d/fstab

  • ... mount
  • ... unmount

OK, I'll have to rtfm overlay....

6

Can you show those reports?
nfs root requires initramfs with nfs and network drivers on a desktop linux, though nowadays only used to netboot system installer over pxe

7

Can you show those reports?

The one point I could recall from my notes and my memories is this link to a cifs (aka smaba, right?) post.
https://openwrt.org/docs/guide-user/additional-software/extroot_configuration#remote_file_system
But yes, I agree with you, that's far from an 'officially' supported working solution.
And I also agree that obscure 3rd party halfway ideas don't count, either.

So may be that mentally I still stick deeply in the concepts and possibility of debian and bretheren? So deep that I could not even think of the possibility of major problems on our tiny little gadgets :thinking:?

nfs root requires initramfs with nfs and network drivers on a desktop linux,

Well, yes, that's the way if everything runs on nfs, aka completely diskless.
I have configured a similiar concept with aufs a decade ago for a cluster: common installation, different run data. So, may be that's where my expectations were rooted.

though nowadays only used to netboot system installer over pxe

yeah, booting an 4MB access point diskless on nfs over PXE would be the real challenge :innocent:

But back to my approach:
After gleaning over the nuts&bolts of the openWRT boot sequence, I think I'd not step not that early into the extroot meachnism.
I find the network initiated as S20 in /etc/rc.d
I'd prefer not to touch that.
Not even the normal boot sequence, may be until S95 -> done where rc.local is called.

This resembles the current stage of my DuT (a TP-Link CPE 210):

  • no luci
  • no wpad
  • no CA-bundle and other wget tls stuff
  • 2 MB Flash left for playing and growing base system in the future
  • ssh acces and sshfs installed

In that stage, I'd expect a read-only root or a remount of root a low risk.
So my idea was to install a 3 tier overlay

  • sshfs on remote server
  • existing /overlay/upper on jffss
  • unpacked root from squashfs

In the kernel documentation
https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html
we may learn that this might be implemented either by nested overlays (i. e. using the current overlay as lower for the next level) or as multi-lower-layer overlay.

However, we may also learn that there are some extra challenges for the upper fs - in particular requiring special acl features I admittedly haven't yet grasped in detail.

But as my first trials indicate, this might precisely be the point that bites me:

[84524.723235] overlayfs: upper fs does not support tmpfile.
[84524.738596] overlayfs: upper fs does not support RENAME_WHITEOUT.
[84524.744953] overlayfs: failed to set xattr on upper
[84524.750079] overlayfs: upper fs missing required features.

root@mygadget:~# /usr/bin/mount -t overlay  overlay -olowerdir=/root/test,upperdir=/root/mp_extroot/up
per,workdir=/root/mp_extroot/work /root/merged

mount ....
root@homeserver:/extroot on /root/mp_extroot type fuse.sshfs

Do we really need an overlay?

Might it even be possible to boot into the current stage and then either replace / or maybe only /usr with an unlayered netmount?
I know, in contrast to a 3-tier-overlay, that may leave us in a incoherent sytem if the net file is down, but chances are that at least still we get a dhcp handshake and ssh dialin ... at least I hope so....

8

nfs sounds a bit of chicken and egg on a network switch.

9

I think for something similiar PXE (isn't that a phrase from Intel aka x86 world anyway?) the choice at hand would be uboot and tftp.

But whatever machine we employ for early booting, at the end we need some network file protocol for unrestricted dwelling.

nfs sounds a bit of chicken and egg on a network switch.

So even if the egg is TFTP, the chicken might live on nfs.
Or Samba / CIFS. Or sshfs.

10

Some qick'n dirty late night testing:

opkg install mount-utils
costs ~ 300 k of flash (including dependencies), but for "heavy jobs" a more reliable choice than the busybox default.

... like mounting / read only:

mount -o remount,ro overlay /
touch asdf
touch: asdf: Read-only file system

I don't see any complaints from a running system.
Presumably that's a virtue of flash-friendly design of openWRT, where any living data is kept in RAM.

Copying from a read-only file system is a no-brainer, but for the records:

cp -arv   /bin /etc /mnt /root /usr /www  /lib /sbin mp_extroot/
cd mp_extroot/
md proc sys dev overlay rom tmp var

What I haven't yet managed is to remount a different source to root.
I can mount my to-be-new-root to /, either directly or by bind mount.
However, I don't get rid of the existing
overlayfs:/overlay on / type overlay dwelling there.

Even the "big" mount does not seem to recognize the overlay syntax:

root@mygadget:~# umount overlayfs:/overlay 
umount: can't unmount overlayfs:/overlay: No such file or directory

"umount /" I can call repeatedly without any effect.

What I might try as a workaround is a bind mount of all the second level directories:
/bin /etc /mnt /root /usr /www /lib /sbin

... but not his night, any more ...

11

When you're talking about the 'big distros' you have to consider that they are targeted at devices that have significantly more resources (hundreds to many gigs of RAM) than a typical OpenWrt target (as low as 64MB for 23.05). They may include the PXE boot (and other boot mechanisms) in the BIOS, so it can kick start that process and load up enough stuff to bring up the network and still have plenty of space to load a new initramfs bootable image (and then hand over to the loaded image). This is not practical (and maybe actually impossible) in most OpenWrt devices.

Another thing to consider is that OpenWrt is a firmware that is primarily focused on networking. Although an AP isn't as critical for a network as the main router and/or DHCP server, it's still an infrastructure device that should be capable of booting properly in the absence of the network or any other infrastructure. That is to say, if one were to use PXE or sshfs or whatever as an extroot file system, imagine what happens if the server is unavailable? That would mean that it would not be possible to bring up your OpenWrt devices. Maybe manageable if it's 'just' an AP, but pretty catastrophic if it's a router+DHCP server and everything else depends on it.

All this to say, it's a tall order and risky approach to rely on the network to boot a network infrastructure device.

That not withstanding... what is your goal here? What are you trying to achieve in practical terms?

2 Likes
12

Are you serous? You found no way to mount user mode fs early, you j&st keel chan5ing it works. You know, in the real world it does mot.

13

What are you trying to achieve in practical terms?

Reorganize my farm network into different vlans.
Large area, increasing number of gadgets.
Ideas from enterprise networking, but not the budget :person_facepalming:
Some spare time to have fun :wink:

So:

  • couple of old enterprise grade HP 1810 / 1820 switches as "backbone"
  • openWRT on x86 (Futro 920) as central router
  • observium & SNMP to keep track of the beast
  • "core area" WLAN access points on DAP X1860 (128 MB Flash, 256 MB, openwrt)

There are two TP-Link CP210 directional outdoor AP to cover open area with no building / LAN (8MB Flash, 64 MB RAM). Do their WLAN thing with 23.05.5, but no space left for growing openWRT in the future nor for perl to run my scrip for Q-BRIDGE snmp.

And if net-extrooting works, maybe I can try ro revive some old 4/32 MB routers as AP to fill gaps, but that's low priority. Including two TP-Link RE450, one of them bricked.

what happens if the server is unavailable?

What's the use of an Access Point for a network out of order?

Of course, it's not as simple.

My first Idea was to keep the net drives for the AP on the X86 router.
But on openWRT - at least for me - it's much harder to manage things like fine grained sshd configuration. And even with chroot jails, I don't feel comfortable with rooot access from gadgets out in the open wild to the core security controller - albeit my security level my be somewhat moderate.

Currently I use a debian home management server as ssh server for my tests.
In terms of recovery from disaster, my falling priorities (and recovery sequence) would be

  • internal LAN traffic, i.e. the switches
  • internet and cross-vlan traffic, i.e. the x86-router
  • 80 % of LAN bound IoT stuff, including the homeserver
  • cctv, pbx
  • office server and LAN user clients
  • WLAN in the core area (the DAP - standalone)
  • WLAN in the fields (dependent on net root)
  • nice-to-have IoT

So what might look like an chicken-egg cycle with Fritz-Box-like-one-gadget-for-anything in mind, gets straightened out. And inoperating AP don't stand in the way to recover core network.

Or, may be, in the end, I'd even pick another Futro 900 from the shelf (or for 15 € from ebay) for a dedicated extroot-file-server.

14

Usb extroot can work, nfs will not.
ok?

1 Like
15

Regarding the PXE-like ideas: That's not my primary focus.
But while searching the forum, I feel some spirit of reviving old devices for whatever moral or sportive or budget reasons.

From an economical point of view, it might not be viable for a couple of devices.
But if there were a community endavour to revive all those 8/32 bricks from the shelves towards a second live at some niche far from the core infrastructure, I'd not mind to contribute.

16

fuse support at early boot is non existant, ie not a valid mean for your intent.

17

fuse support at early boot is non existant,

Yes, I got that.
So I don't target at early boot any more.
What I have in mind is skd of the runlevel concept from good old sysV.

  • Widget boots into skd of maintenance state with ssh acces.
  • there is a second "run level" where all those "heavy things" are up
  • sshfs enables those heavy things
  • during setup the switching is done manually
  • on deployment, the switching is done automagically
  • there should be some kind of auto-fall back if stuff goes wrong to always ensure ssh accessibility

Fall back conditions:

  1. config mistakes during setup, testing, playing
  2. ssh server not (yet) available after boot
    (think e.g. of grid power recovery)
  3. ssh server gone away after switching to full operation

Looks like # 1) is most probable one - and the next one to expect, for sure.

How might a watchdog look like to cover those situations?
Is there anything readily at hand, or do we have to reinvent the wheel?
...just think of the Luci-restore function after faulty network changes.

What action to choose?

  • retry to mount sshfs - may be repeated forever, as long as we have ssh access
  • reboot and retry - but stay in a state of ssh access
    we might need to catch the case where remount ends in screwed access
    so we might have skd of "previous state"
  • reboot and stay calm in safe state after some "giving up" condition

What conditions to monitor?

  • after changes manually triggered from console, we might require some confirmation, and perform rollback if this is not received after some timeout (this is what luci does, right?)
  • during (post) boot sequence, we may check for milestones
    • net fs (ssh) server available
    • net fs mounted
    • net fs sanity checks
    • root fs (or parts) remounted
    • demons reloaded
    • sshd up (still/again)
    • (other) network connectivity working
    • demons working

Where to dwell?
The watchdog must not rely on ressources prone to fail during/after level switch, i.e. root file system and networking.
Can we rely on shell skripts executed by busybox if only ram fs is available?
Can we rely on cron?

18

Hi @wolfgangr

you could do something like this ...

heavy packages are extracted to NFS/CIFS/whatever

and custom init.d ... something like this

#!/bin/sh /etc/rc.common

START=19
STOP=50
USE_PROCD=1
PROG=telnetd
PREFIX=
CMD=
TAG=TELNETD
PRIO=daemon.info

create_cfg() {
    logger -s -t "$TAG" -p "$PRIO" "Config write"

    for PREFIX in / /tmp/root/; do
        logger -s -t "$TAG" -p "$PRIO" "testing $PREFIX"
        [ -x ${PREFIX}usr/sbin/${PROG} ] && CMD="${PREFIX}usr/sbin/${PROG}"
    done

    PARAM="-F -p 2323"

    logger -s -t "$TAG" -p "$PRIO" "Config end"
}

start_service() {
    logger -s -t "$TAG" -p "$PRIO" "Check for init done"
    [ -f /tmp/done1 ] || exit 0;

    create_cfg
    logger -s -t "$TAG" -p "$PRIO" "Exec: $CMD"
    logger -s -t "$TAG" -p "$PRIO" "Param: $PARAM"

    [ -z "$CMD" ] || [ "$gSETUPDONE" != 0 ] && {
        logger -t "$TAG" -p "$PRIO" "Not starting ${PROG}"
        exit
    }

    procd_open_instance
    procd_set_param stdout 1
    procd_set_param stderr 1
    procd_set_param user root
    procd_set_param env LD_LIBRARY_PATH=${PREFIX}usr/lib
    procd_set_param command $CMD $PARAM
    procd_close_instance

    logger -s -t "$TAG" -p "$PRIO" "Start"
}

stop_service() {
    logger -s -t "$TAG" -p "$PRIO" "Stop"
}

shutdown() {
    killall -9 ${PROG}
}

key is
procd_set_param env LD_LIBRARY_PATH=${PREFIX}usr/lib
so unpacked libraries could be loaded from ext storage

19

sounds interesting .... so let my try to understand it.

/etc/rc.common is skd of a wrapper around all init.d skrips, called in by the shebang, correct? And /etc/rc.common shebangs #!/bin/sh, so we get skd of ash-plus. (Sorry, I'm not the shell acrobat and prefer perl as soon as indentation exceeds level of one...)

The variables following are fodder for either the wrapping /etc/rc.common, the code following in the init-whatver or in code called from there - e.g. the procd.

TELNET is just an example, right?

Let me just recall some of the large and/or second level chunks

  • full wpad
  • luci (including uhttpd, I suppose)
  • CA-bundle and tls for opkg
  • snmpd, lldpd, collectd
  • perl, my skript and its data for Q-BRIDGE-MIB

Simple la /etc/init.d on a configured DAP-x1860 (with ample of flash) reveals e.g.

-rwxr-xr-x    1 root     root         15553 Nov 22 14:08 collectd
-rwxr-xr-x    1 root     root         13339 Sep 17 10:36 lldpd
-rwxr-xr-x    1 root     root          9726 Nov 22 14:08 snmpd
-rwxr-xr-x    1 root     root          6056 Sep 23 12:34 uhttpd
-rwxr-xr-x    1 root     root          1262 Sep 23 12:34 wpad

The sheer size of that scripts alone makes me hesitate to dive into them for customizing. And is anything else (just think of all the luci pages...) covered by LD_LIBRARY_PATH? May I rise some doubt, at least?
Feels like an administrative nightmare ahead??

OK, looking into what I had added on top of customized DA-X1860 image supports your approach to focus on /usr/lib

root@ap-x1860:/etc/init.d$ du -sh /overlay/upper/usr/*
2.6M    /overlay/upper/usr/bin
6.9M    /overlay/upper/usr/lib
268.0K  /overlay/upper/usr/libexec
88.0K   /overlay/upper/usr/local
1.6M    /overlay/upper/usr/sbin
132.0K  /overlay/upper/usr/share

I might even think of a bind mount for /usr or /usr/lib alone.
However, this all seems to risk inconsistent system, to me.

Wouldn't a root remount provide a clear cut, in contrast?
The state of the overlay is a conserved consistent snapshot of a minimum to gain ssh access and sshfs mount.

Then, if the remount succeeds, we have another consistently maintained version (easy to be snapshoted on the server, since it resides on btrfs there, by the way), without any maintenance chains between those versions beyond the process of handing over.

And for starting, I think my minimal watchdog may boil down to sth simple as

reboot -d 120 -f &  do_the_dangerous_remount_stuff
killall reboot
20

If file permissions and the like are a problem nbd would work ?
Probably beats mounting a filesystem over the top of sshfs
not encrypted though

seems like there might be a plugin that does it though
sorry if i'm way off base