OpenWRT /dev/loop0 overlay question

Hi All,

I have a question about the /dev/loop0 usage. I have a ubiquiti usg 3p router with an 16Gb usb stick. Yesterday i've upgraded the system from 23.5 to 24.1 by changing the USB stick. But while restoring the config data i am missing some files. Is it possible to retrieve these files from the USB stick using my linux laptop? Somehow the files ar'nt visible on my laptop. probably because the files are on the stick not on a filesystem but somehow on a raw partition? I can always replace the stick and boot with 23.5 once and get the files from the device but thats not a tech way. :wink:

So i think i have to create a loop device but how and where to start ...
Who has done this before?

If you change sd card obviously all info is in old card.
No idea what you mean by tech way, blowtorch or sledge.

You have to check what filesystem is there first via lsblk -f, then sudo file -s /dev/loop0 or sudo blkid /dev/loop0.
If it's an ext4 then just mount it

ls /mnt

For squashfs

sudo mount -t squashfs /dev/loop0 /mnt

For JFFS something like that:

sudo apt install mtd-utils
sudo modprobe mtdblock
sudo modprobe jffs2
sudo mount -t jffs2 /dev/loop0 /mnt
ls /mnt

If the format is unrecognised then you can try

sudo apt install binwalk
binwalk -e /dev/loop0

By the way, almost any AI could give you very nice suggestions about these type of things.

2 Likes

Thanks so far.
But it is not getting clearer for me.
My USB stick when inserted in Ubuntu 24 has 2 partitions: /dev/sda1 is of the type FAT32 and contains the kernel. /dev/sda2 is of the type squashfs and holds the root filesystem.

/dev/sda2 on /media/user/disk type squashfs (ro,nosuid,nodev,relatime,errors=continue,threads=single,uhelper=udisks2)
/dev/sda1 on /media/user/DC4A-CE4C type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)

When i look in /media/user/disk i see the root:

# ls -l
total 0
drwxr-xr-x  2 root root  778 Jul 16  2024 bin
drwxr-xr-x  2 root root   30 Jul 16  2024 dev
drwxr-xr-x 23 root root 1011 Jul 16  2024 etc
drwxr-xr-x 10 root root  506 Jul 16  2024 lib
lrwxrwxrwx  1 root root    3 Jul 16  2024 lib64 -> lib
drwxr-xr-x  2 root root    3 Jul 16  2024 mnt
drwxr-xr-x  2 root root    3 Jul 16  2024 overlay
drwxr-xr-x  2 root root    3 Jul 16  2024 proc
drwxr-xr-x  2 root root   27 Jul 16  2024 rom
drwxr-xr-x  2 root root    3 Jul 16  2024 root
drwxr-xr-x  2 root root  869 Jul 16  2024 sbin
drwxr-xr-x  2 root root    3 Jul 16  2024 sys
drwxrwxrwt  2 root root    3 Jul 16  2024 tmp
drwxr-xr-x  7 root root  114 Jul 16  2024 usr
lrwxrwxrwx  1 root root    3 Jul 16  2024 var -> tmp
drwxr-xr-x  4 root root   67 Jul 16  2024 www

But there should also be a tftpboot directory.

When i look on OpenWRT my root looks like this:

ls -l
drwxr-xr-x    2 root     root           778 Feb  4 00:09 bin
drwxr-xr-x    4 root     root          1240 Mar  9 14:15 dev
drwxr-xr-x    1 root     root          3488 Mar  7 13:27 etc
drwxr-xr-x    1 root     root          3488 Mar  7 14:38 lib
lrwxrwxrwx    1 root     root             3 Feb  4 00:09 lib64 -> lib
drwxr-xr-x    2 root     root             3 Feb  4 00:09 mnt
drwxr-xr-x    4 root     root          4096 Feb  4 00:09 overlay
dr-xr-xr-x  123 root     root             0 Jan  1  1970 proc
drwxr-xr-x   16 root     root           236 Feb  4 00:09 rom
drwxr-x---    1 root     root          3488 Feb  4 00:14 root
lrwxrwxrwx    1 root     root             8 Feb  4 00:09 run -> /var/run
drwxr-xr-x    2 root     root           838 Feb  4 00:09 sbin
dr-xr-xr-x   11 root     root             0 Jan  1  1970 sys
drwxr-xr-x    2 root     root          3488 Feb  4 00:14 tftpboot
drwxrwxrwt   19 root     root           540 Mar 10 06:00 tmp
drwxr-xr-x    1 root     root          3488 Mar  7 14:38 usr
lrwxrwxrwx    1 root     root             3 Feb  4 00:09 var -> tmp
drwxr-xr-x    1 root     root          3488 Mar  7 14:38 www

The mount command gives on OpenWRT:

# mount
/dev/root on /rom type squashfs (ro,relatime,errors=continue)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/loop0 on /overlay type f2fs (rw,lazytime,noatime,background_gc=on,nogc_merge,nodiscard,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,barrier,extent_cache,mode=adaptive,active_logs=6,alloc_mode=reuse,checkpoint_merge,fsync_mode=posix,memory=normal,errors=continue)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work,xino=off)
tmpfs on /dev type tmpfs (rw,nosuid,noexec,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,noatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,noatime)
bpffs on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)

Now my missing files were in tftpboot.

Question is, the missing data should be on the USB disk, but not in /dev/sda2.
So /dev/loop0 points at something else but where or what. There is no other storage on the router. So probably /dev/loop0 points to some space on the drive not recognizable as filesystem by ubuntu.

I'm writing up some details that hopefully will help. I hope to reply in a few minutes.

See if this makes any sense to you...

This is a bit lengthy but I included lots of detail for anyone that might need it. I also did this for the first time today.

The short version:

I didn't find a /dev/loop0 automatically showing up on my linux laptop.
I ran binwalk against /dev/sda and got the info to run:
sudo losetup -o <offset> -f /dev/sda
and then mount /dev/loop0 or whatever loop dev number it is.
"<offset>" is to be replaced with the number in the output of binwalk.

The long version:
I'm not familiar with the usg 3p device.
Looking at the device page: https://openwrt.org/toh/ubiquiti/unifi_security_gateway_3p
I searched the OpenWrt Bootlog for "overlay" and "loop0" and found:

[   11.670904] random: jshn: uninitialized urandom read (4 bytes read)
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[   14.327924] F2FS-fs (loop0): Mounted with checkpoint version = 8f5dc93
[   14.336933] mount_root: switching to f2fs overlay
[   14.344884] overlayfs: "xino" feature enabled using 32 upper inode bits.
[   14.398807] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   14.420869] urandom-seed: Seeding with /etc/urandom.seed
[   14.567751] procd: - early -
[   14.570974] procd: - watchdog -

Which looks to be from OpenWrt SNAPSHOT, r20176-a63aeaecf1.
I don't know what might be different now.
I don't know if "overlayfs: "xino" feature enabled using 32 upper inode bits." will complicate mounting on your linux laptop.

Reinforced by the filesystem types available in https://downloads.openwrt.org/releases/24.10.0/targets/octeon/generic/ , it looks like it might use the pretty standard setup with the kernel on one partition and the root filesystem of type squashfs on a second partition and the writable overlay filesystem mounted as a loop device using the second partition but an offset to skip the immutable squashfs for "rootfs_data".

It helps to know specifics from when it is in use on the usg. Hopefully showing the info from the running system will be good enough.

I'll use the info from my "FriendlyElec NanoPi R4S" and you can adapt as needed for your usg.

We'll look at the info for "kernel", "root", and overlay on the usg so we know what to expect on the linux laptop.

On running OpenWrt system:

block info
/dev/loop0: UUID="990fdb02-1784-41f2-9e93-6309aa4043d1" LABEL="rootfs_data" VERSION="1.0" MOUNT="/overlay" TYPE="ext4"
/dev/mmcblk1p1: UUID="84173db5-fa99-e35a-95c6-28613cc79ea9" LABEL="kernel" VERSION="1.0" TYPE="ext4"
/dev/mmcblk1p2: UUID="e3c74bfd-081c4550-52536fdd-7a3f08e3" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"

Note that /overlay is on /dev/loop0 and there is only one loop device.
Also note the TYPE="ext4". If your system uses jffs2, f2fs or something else a little more exotic than ext4, your linux laptop will need the drivers for that filesystem in order to mount it and give you access.
EDIT: I see in your follow-up post that your usg used f2fs for the overlay filesystem. You may need to install f2fs-tools on your linux pc and maybe a driver or two to mount and work with that if not yet installed.

If you had captured the offset when the usg was running from the usb stick that you now want to read from, it would eliminate having to have software with a lot of dependencies installed on your linux laptop.

On running OpenWrt system:

losetup -l
NAME       SIZELIMIT   OFFSET AUTOCLEAR RO BACK-FILE  DIO LOG-SEC
/dev/loop0         0 12058624         1  0 /mmcblk1p2   0     512

Without that info, you might be able to get the offset and filesystem info on your linux laptop with the following:

Plug the usb stick in to the linux laptop and determine what device it is.

On linux laptop:

lsblk or use something like Gnome Disks.

In my example it is /dev/sda.

Install if needed (lots of dependencies) and run:

sudo binwalk /dev/sda2 

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Squashfs filesystem, little endian, version 4.0, compression:xz, size: 10884307 bytes, 2079 inodes, blocksize: 262144 bytes, created: 2024-09-23 12:34:46
10944512      0xA70000        Linux EXT filesystem, blocks count: 95808, image size: 98107392, rev 1.0, ext4 filesystem data, UUID=15c16104-614c-11e2-9e4e-859fd7b2d7b2, volume name "rootfs_data"

We see that the squashfs filesystem for /rom is at offset 0.
The filesystem for "rootfs_data" is at offset 10944512.

I think that offset will differ with each build.

Attach the data blocks to a loop device with a command like:
sudo losetup -o 10944512 -f /dev/sda2
(use the offset number for your setup on that specific usb stick.)

Check it with
sudo losetup -l .

There are other options you may want like --size-limit, --sector-size and --read-only .

Once the loop device is in place, mount the filesystem with Gnome Disks or the mount command etc.
Set as read-only if you want to be safe to not mess up files. Enable write access if you want to write files to it.

lsblk -f
NAME                FSTYPE      FSVER    LABEL       UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0               ext4        1.0      rootfs_data 15c16104-614c-11e2-9e4e-859fd7b29602     70.5M     6% /media/spence/rootfs_data
sda                                                                                                        
├─sda1              ext4        1.0      kernel      84173db5-fa99-e35a-95c6-28613cc79ea9     11.4M    25% /media/spence/kernel

Read your files! :-)

I mounted the kernel and root filesystems as a test and to learn. I spent a lot of time looking for a place where the offset might be stored where OpenWrt might get the info at boot time and to avoid needing binwalk to mount it on a linux pc.

1 Like

Should that be an sd device like /dev/sda or is there something available to auto-detect the hidden filesystem and have losetup mount it on a typical linux pc?

Thanks for the info by the way.

Hmm found 2 filesystems sofar on /dev/sda2:

0             0x0             Squashfs filesystem, little endian, version 4.0, compression:xz, size: 3070182 bytes, 1210 inodes, blocksize: 262144 bytes, created: 2024-07-15 22:14:18
455293792     0x1B233B60      Squashfs filesystem, little endian, version 4.0, compression:xz, size: 848078 bytes, 4 inodes, blocksize: 131072 bytes, created: 2022-12-09 16:11:44

But when i do on my ubunu latptop:

# losetup -o 455293792 -f /dev/sda2
losetup -l
# /dev/loop28         0 3276800         0  0 /dev/sda2                                                 0     512
# mount -t f2fs /dev/loop28 /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop28, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

I've installed f2fs-tools but i must do something wrong.

It looks like your binwalk found a squashfs filesystem at offset 455293792.

Is the "root" filesystem on /dev/sda2 ?

also mounting as -t squashfs does not work, same message.
How can i find out what te filesystem type should be

The root filesystem is on /dev/sda2 offset 0
If i do:

losetup -o 0 -f /dev/sda2

i can mount /dev/loop28 without problems and it gives me the root filesystem.

Funny part if i do

# losetup -o 455293791 -f /dev/sda2
or 
# losetup -o 455293793 -f /dev/sda2

No loopback device is created so there is something at that spot 455293792.

Hmmm.
I was thinking it would be easy based on the device page info.

My thought is to list the partition table with sizes in bytes.

Is the offset and calculated end bytes from binwalk within partition 2?

The device page refers to partition 3. Do you have a 3rd partition?

No i have no third partition. I changed the 4Gb USB stick with routerOS for a 16Gb model.
Created a sda1 and a sda2:

Disk /dev/sda: 14.45 GiB, 15518924800 bytes, 30310400 sectors
Disk model: Flash Disk      
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4c406247

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1         2048   292863   290816  142M  c W95 FAT32 (LBA)
/dev/sda2       292864 30310399 30017536 14.3G 83 Linux

So somewhere within sda2 should be my data.

That squashfs at offset 455293792 looks rather small.

echo $(( (30310399 - 292864) * 512 ))
15368977920
echo $(( 455293792 + 848078 ))
456141870
          0   - start of root
    3070182   - end of root
  455293792   - start of hidden squashfs
  456141870   - end of hidden squashfs
15368977920   - end of partition

I'd look at command line options for binwalk to look for more specific info for the expected filesystem like signatures or magic numbers etc. Next I'd look around on the live system with tools like hexdump and strings to try to see what is in place for a valid loop device and mount and look for that type of info on the off-line system.

# losetup -o 455293792 -f /dev/sda2
# file -sL /dev/loop28
/dev/loop28: Squashfs filesystem, little endian, version 4.0, xz compressed, 848078 bytes, 4 inodes, blocksize: 131072 bytes, created: Fri Dec  9 16:11:44 2022
1 Like

I wonder if that is an artifact of the build process or something.

When i do this:

# losetup -o 455293792 -f /dev/sda2
root@dell:/home/klaas# binwalk /dev/loop28

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Squashfs filesystem, little endian, version 4.0, compression:xz, size: 848078 bytes, 4 inodes, blocksize: 131072 bytes, created: 2022-12-09 16:11:44
891408        0xD9A10         Unix path: /sys/bus/pci/devices
892392        0xD9DE8         Unix path: /sys/devices/system/cpu/isolated
892616        0xD9EC8         Unix path: /sys/devices/system/cpu
894776        0xDA738         Unix path: /sys/devices/system/node
1061608       0x1032E8        Ubiquiti firmware header, third party, ~CRC32: 0x74797065, version: "SSL_CONF"
1090360       0x10A338        OpenSSL encryption, salted, salt: 0x00
.........
37516325      0x23C7425       Cisco IOS experimental microcode, for ""
46549664      0x2C64AA0       POSIX tar archive (GNU), owner user name: "/netboot/16.04/amd64/linux"
152480928     0x916ACA0       POSIX tar archive (GNU), owner user name: "/netboot/18.04/amd64/ubuntu-inst", owner group name: "aller/amd64/linux"
496326924     0x1D95590C      gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
....

There is my data, /netboot/...
Now to get it off, mount does not work. dmesg tells me: unable to read id index table

1 Like

If those files were really important to me, I'd probably take the service interruption and swap that usb stick back in to the router and get the files that way rather than risk damaging the filesystem since it is not straight forward to mount on your linux laptop.

In the process, I'd include that directory in a backup by way of sysupgrade config and also look at losetup, block info, mount etc and document it. Compare it to the new 24.10 version.

You seem experienced but I'll advise to be careful unmounting and ejecting the usb stick so you don't corrupt the filesystem.

You may get some useful info searching linux forums, reddit, stack exchange etc, as this is a linux issue as much as OpenWrt.

I'll follow this topic more but may not have the experience to help more. I hope I provided more help than distraction.

Good Luck!

I haven't done this in a long time, so going from memory here:

You get offset by multiplying start sector with blocksize
on openwrt:

losetup -o $((292864*512)) -f /path/to/disk/image
losetup -a
mount /dev/loopX /mount/point

on linux pc:

mount -o loop,offset=$((292864*512)) /path/to/disk/image /mount/point

That should do it.

edit:
If partitions are available in /proc/partitions you should be able to directly use /dev/sda2 and not need offset. Those offsets are for the disk /dev/sda and not for partitions /dev/sdaX.