Trouble installing USB drive

Update: This topic has evolved to include a discussion of installing a USB drive. Subject line has changed mid-stream...

I was having trouble with my router, and SSH'd in. I was surprised to see the Please try to remove files from /overlay/upper/... and reboot! error message. Here's what I did to get out of trouble...

TL;DR - use rm /overlay/upper/...files... to regain some space and reboot...

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.0, r10860-a3ffeb413b
 -----------------------------------------------------
Your JFFS2-partition seems full and overlayfs is mounted read-only.
Please try to remove files from /overlay/upper/... and reboot!

Background

I had installed a USB drive for external storage and mounted it to the /opt directory. The program that used the space had run amok. It seems to have filled the USB drive and left the storage in a bad state.

I was not able to rm /opt/...files... because that part of the filesystem was read-only. But /opt is really part of /overlay/upper/opt Here's what it looks like:

root@ArcherC7v2-Main:~# ls -al /overlay/upper/
drwxr-xr-x    9 root     root             0 Jan 26 19:59 .
drwxr-xr-x    5 root     root             0 Dec 31  1969 ..
drwxr-xr-x   17 root     root             0 Feb 11 07:41 etc
drwxr-xr-x    7 root     root             0 Feb 15 14:29 lib
drwxr-xr-x    3 root     root             0 Feb  8 11:44 mnt
drwxr-xr-x    3 root     root             0 Feb  8 11:53 opt
drwxr-xr-x    2 root     root             0 Jan 26 19:53 sbin
drwxr-xr-x    6 root     root             0 Feb 11 07:41 usr
drwxr-xr-x    3 root     root             0 Jan 26 20:43 www

root@ArcherC7v2-Main:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.8M      2.8M         0 100% /rom
tmpfs                    60.8M    228.0K     60.6M   0% /tmp
/dev/mtdblock3           11.6M     11.4M    176.0K  99% /overlay
overlayfs:/overlay       11.6M     11.4M    176.0K  99% /
tmpfs                   512.0K         0    512.0K   0% /dev

Solution

So I cd'd to various directories, and used the du -hd 1 command to review the sizes of files in that directory. I found some big files that I could afford to lose, and used the following command to create a little space.

rm /overlay/upper/opt/...file(s)... # not rm /opt/...file(s)... which won't work

I then rebooted the router and the read-only problem disappeared.

[Note: the initial error message was exactly correct, but without the context above, I didn't understand how to apply it...]

1 Like

I'm curious why you mounted to /opt rather than performing the extroot process?

I wanted to install YAMon, which is built around using /opt for its data. (I'm running on a 16 MByte Flash/128MByte RAM router (Archer C7 v2), so I have plenty of space for packages, etc.)

This will happen if your big data application starts up and stores data without the external disk mounted. Since the mount point is in the internal flash tree, the data will go into internal flash and overfill it. Approaches to prevent that are either to store the application and/or its configuration files on the external disk so it can't possibly run without the disk mounted, or place the mount point under /tmp so that such a malfunction only fills up RAM instead of flash.

3 Likes

This sounds like really important information, but I don't quite understand how to apply it to my situation. A few questions:

  1. I never removed the USB drive - can you think of another way that this could have happened?
  2. I used the USB Drive Quick Start to format and mount the USB drive. Does it provide a good procedure for this type of application? [Note: I wrote that guide, but I don't completely understand all the ramifications. I would gladly accept new information to make it better.]
  3. It appears that directories that appear in /overlay/upper/xxx are identical to /xxx. Is this true? (I see there are directories in / that do not appear in /overlay/upper)
  4. @psherman asked why I didn't use extroot. How might that apply/help in this situation?

Many thanks.

The overlay system works like it always has:

  • /rom contains the original files that are in the flashed firmware. They never change
  • /overlay contains the later changes (additions, changes, delete marks)
  • The overlay filesystem combines those as the total file system that you see as /
1 Like

@hnyman - Thanks for this clarification. I will keep this in mind as I read more about this.

Update: Per my comment #1 above, I never intentionally removed the USB drive. However, when I ssh'd in, I found that the USB drive was not visible, although it had been previously (during the USB config process described in the Quick Start...)

  • df -h did not mention the /opt
  • ls -al /dev/sd* gives ls: /dev/sd*: No such file or directory

What might cause the USB drive to disappear? Thanks again.

The strange part of your story is tha the change to the files on the USB drive got written to overlayfs. I suspect that something is wrong in your mounting and/or yamon config, so that the files never got written to USB.

Sounds suspicious.
A mounted USB drive should be visible like here:

root@router1:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 7.3M      7.3M         0 100% /rom
tmpfs                   232.4M    980.0K    231.4M   0% /tmp
/dev/ubi0_1              75.3M    328.0K     71.1M   0% /overlay
overlayfs:/overlay       75.3M    328.0K     71.1M   0% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1                 3.8G    336.3M      3.4G   9% /mnt/sda1

Happy to provide a puzzle :-/ To confirm, after installing the USB drive, both df and ls -al /dev/sd* showed the expected output. It was only after YAMon collected data for a few weeks that I got into this state. (I had not been checking the data regularly, so I can't say when things went bad.)

Would you check the procedure in the USB Quick Start? Those instructions get the drive mounted at /mnt/sda1. I then used the LuCI GUI to move it to /opt Is there an error in my thinking? Thanks.

You might check the timestamps on the USB drive and try to figure out when the last writes have been done.

But I have no idea why the mount would get terminated.

One theory is that your fstab logic is not reboot-proof and the router has rebooted at some point. Does (did) the uptime match your expectations?

I no longer can read the USB drive (it needed to be formatted).

This is a big clue. My power goes out all the time (at least once a month, and certainly after installing the USB drive.)

The quick start says to do this: /sbin/block mount && service fstab enable Is that sufficient to make it reboot-proof? Thanks.

Well, does it work for you if you plug out the power and replug?

Let's first discuss the real life and then fix the documentation.

Personally I have just configured the fstab to allow anonmount and automount, so that all plugged drives (usually default FAT sticks) would get mounted under /mnt. I rarely use usb storage, so the ease of use is the key point for me. But your approach would allow more finegrained control, assuming that the user correctly configures the device-specific section of fstab. E.g. the /opt mountpoint.

1 Like

Yes! My intent for writing the Quick Start is a simple, can't fail procedure that will work for everyone who says, "I want to mount a USB drive on my router..." No special settings required.

What changes would you suggest to the Quick Start? Thank you.

Well, below is my fstab that I include already in the firmware.
I care nothing about having Linux ext4 file system on the stick, so no mkfs commands etc. needed. Usually the sticks have the Windows compatible FAT or FAT32 on them.

I just define that all sticks should be mounted automatically (auto_mount), also the sticks that have no defined UUID in fstab (anon_mount).

So, this is my /etc/config/fstab:

config 'global'
	option anon_swap  '0'
	option anon_mount '1'
	option auto_swap  '1'
	option auto_mount '1'
	option delay_root '0'
	option check_fs   '0'

## use 'block detect' to find out uuid, if you want device specific mount.
## anon_mount enables the automatic mounting of /dev/sdXy to /mnt/sdXy

The only changes to the defaults are the global anon_mount 1 and auto_mount 1, which enable the magic.

But that does not allow settings device-specific mount-point etc., so this is suitable mostly for occasional USB stick usage for backups etc. If you need the storage to definitely be in /opt for stick '8C6C-0792', you need to set the the device specific sections (that block detect gives info about).

Btw,
you have still not shown your actual fstab config to us...
How have you configured the global options and the device-specific section?

EDIT:
hmm. the global defaults may have changed a bit during the years. Apparently auto_mount 1 is already default, and only the anon_mount 1 is a change.
There is also currently delay_root 5 in the global defaults, but I have not needed it.
https://git.openwrt.org/?p=project/fstools.git;a=blob;f=block.c;h=50d877fbeb08e1d2b445549224f0565eecd56791;hb=HEAD#l1716

Here's my /etc/config/fstab, as it was (auto)created by the Quick Start:

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '5'
	option check_fs '1'

config mount
	option uuid '825475e4-7a1f-48ee-b145-d4102fa4eba3'
	option enabled '1'
	option target '/opt'

Step 7 imports the block detect output and this causes the mount point to be /mnt/sdaX
Step 8 sets a bunch of the options

block detect | uci import fstab
uci set fstab.@mount[0].enabled='1' && uci set fstab.@global[0].check_fs='1' && uci commit

It seems that changing Step 8 to say the following would get to the same state as your config. (It would auto-mount all volumes, no check_fs option) Correct? [NB: I updated the Quick Start to use this new command.]

uci set fstab.@global[0].anon_mount='1' && uci commit

A couple other considerations:

  • I put Step 6 into the process (to format the drive for ext4) as belt-and-braces, just so the procedure can assure that the drive has been formatted in a useful way. Is this useful?
  • I believe the last step (Step 9) - /sbin/block mount && service fstab enable - will actually mount the drive and enable fstab during the boot process. Correct?
  • I tweaked the steps of the Quick Start to incorporate this info. Please let me know if I got it wrong.

Thank you for all your thought.

Update: I re-ran the Quick Start procedure from a clean router and got the following. I believe that the procedure is working as expected.

Note: Step 9 showed an error (block: hotplug-call call failed ) Re-running the commands seems to have worked without error. Why might that fail the first time?

Click the triangle to see the full log, and compare to your results...

Detailed Log of the 'Add a USB Drive' commands
  1. Format your USB Drive on your laptop/desktop computer. Use default formatting options.

  2. SSH into the router

    root@ArcherC7v2-Main:/#
    
  3. Get the required packages:

     root@ArcherC7v2-Main:/# opkg update && opkg install block-mount e2fsprogs kmod-fs-ext4 kmod-usb-storage kmod-us
     b2 kmod-usb3
     Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ar71xx/generic/packages/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_core
     Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ar71xx/generic/packages/Packages.sig
     Signature check passed.
     Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ar71xx/generic/kmods/4.14.162-1-342af9e4f67b3447c53216ab8e3b12a1/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_kmods
     Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ar71xx/generic/kmods/4.14.162-1-342af9e4f67b3447c53216ab8e3b12a1/Packages.sig
     Signature check passed.
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/base/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_base
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/base/Packages.sig
     Signature check passed.
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/luci/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_luci
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/luci/Packages.sig
     Signature check passed.
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/packages/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_packages
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/packages/Packages.sig
     Signature check passed.
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/routing/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_routing
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/routing/Packages.sig
     Signature check passed.
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/telephony/Packages.gz
     Updated list of available packages in /var/opkg-lists/openwrt_telephony
     Downloading http://downloads.openwrt.org/releases/19.07.0/packages/mips_24kc/telephony/Packages.sig
     Signature check passed.
     Package block-mount (2020-01-05-823faa0f-1) installed in root is up to date.
     Package e2fsprogs (1.44.5-2) installed in root is up to date.
     Package kmod-fs-ext4 (4.14.162-1) installed in root is up to date.
     Package kmod-usb-storage (4.14.162-1) installed in root is up to date.
     Package kmod-usb2 (4.14.162-1) installed in root is up to date.
     Installing kmod-usb3 (4.14.162-1) to root...
     Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ar71xx/generic/kmods/4.14.162-1-342af9e4f67b3447c53216ab8e3b12a1/kmod-usb3_4.14.162-1_mips_24kc.ipk
     Configuring kmod-usb3.
    
  4. Look for USB device with ls -al /dev/sd* It hasn't been inserted, and therefore it is not listed.

    root@ArcherC7v2-Main:/# ls -al /dev/sd*
    ls: /dev/sd*: No such file or directory
    
  5. Insert the USB Device, then look to see if it's present. The ls command shows /dev/sda1; the df -h command does not show it yet.

    root@ArcherC7v2-Main:/# ls -al /dev/sd*
    brw-------    1 root     root        8,   0 Mar 15 13:03 /dev/sda
    brw-------    1 root     root        8,   1 Mar 15 13:03 /dev/sda1 <-- Yes /dev/sda1 is present
    
    root@ArcherC7v2-Main:/# df -h
    Filesystem                Size      Used Available Use% Mounted on
    /dev/root                 2.8M      2.8M         0 100% /rom
    tmpfs                    60.8M    128.0K     60.7M   0% /tmp
    /dev/mtdblock3           11.6M      3.8M      7.8M  33% /overlay
    overlayfs:/overlay       11.6M      3.8M      7.8M  33% /
    tmpfs                   512.0K         0    512.0K   0% /dev
    
  6. Format the USB drive as ext4

    root@ArcherC7v2-Main:/# mkfs.ext4 /dev/sda1
    mke2fs 1.44.5 (15-Dec-2018)
    /dev/sda1 contains a vfat file system labelled '1GBYTE'
    Proceed anyway? (y,N) y
    Creating filesystem with 258560 4k blocks and 64640 inodes
    Filesystem UUID: 7375fbf6-4b14-4e13-bae8-1aa4e5e08502
    Superblock backups stored on blocks:
    	32768, 98304, 163840, 229376
    
    Allocating group tables: done
    Writing inode tables: done
    Creating journal (4096 blocks): done
    Writing superblocks and filesystem accounting information: done
    
  7. Create the fstab configuration file

    root@ArcherC7v2-Main:/# block detect | uci import fstab
    
  8. Update the global settings to mount all volumes - named or anonymous

    root@ArcherC7v2-Main:/# uci set fstab.@global[0].anon_mount='1' && uci commit
    

    Here's what the fstab config file looks like now

    root@ArcherC7v2-Main:/# cat /etc/config/fstab
    
    config global
    	option anon_swap '0'
    	option auto_swap '1'
    	option auto_mount '1'
    	option delay_root '5'
    	option check_fs '0'
    	option anon_mount '1'
    
    config mount
    	option target '/mnt/sda1'
    	option uuid '7375fbf6-4b14-4e13-bae8-1aa4e5e08502'
    	option enabled '0'
    
  9. Mount the device and enable fstab at boot. Note that the first time, the block command failed

    root@ArcherC7v2-Main:/etc/init.d# /sbin/block mount && service fstab enable
    block: hotplug-call call failed
    
    root@ArcherC7v2-Main:/etc/init.d# /sbin/block mount
    root@ArcherC7v2-Main:/etc/init.d# service fstab enable
    
  10. That's it! The USB drive is mounted at /mnt/sda1;

    root@ArcherC7v2-Main:/etc/init.d# ls -al /mnt
    drwxr-xr-x    1 root     root             0 Feb  8 11:44 .
    drwxr-xr-x    1 root     root             0 Mar 15 12:33 ..
    drwxr-xr-x    3 root     root          4096 Mar 15 13:05 sda1
    

    df -h shows the USB drive has 909 MBytes free

    root@ArcherC7v2-Main:/etc/init.d# df -h
    Filesystem                Size      Used Available Use% Mounted on
    /dev/root                 2.8M      2.8M         0 100% /rom
    tmpfs                    60.8M      1.2M     59.6M   2% /tmp
    /dev/mtdblock3           11.6M      3.9M      7.7M  33% /overlay
    overlayfs:/overlay       11.6M      3.9M      7.7M  33% /
    tmpfs                   512.0K         0    512.0K   0% /dev
    /dev/sda1               978.1M      2.5M    909.1M   0% /mnt/sda1
    

    The USB drive is visible in /dev/sd* as well

    root@ArcherC7v2-Main:/etc/init.d# ls -al /dev/sd*
    brw-------    1 root     root        8,   0 Mar 15 13:03 /dev/sda
    brw-------    1 root     root        8,   1 Mar 15 13:05 /dev/sda1
    

    Here's the contents of /etc/config/fstab

    root@ArcherC7v2-Main:/etc/init.d# cat /etc/config/fstab
    
    config global
    	option anon_swap '0'
    	option auto_swap '1'
    	option auto_mount '1'
    	option delay_root '5'
    	option check_fs '0'
    	option anon_mount '1'
    
    config mount
    	option target '/mnt/sda1'
    	option uuid '7375fbf6-4b14-4e13-bae8-1aa4e5e08502'
    	option enabled '0'
    
  11. Now reboot to see if the USB drive is mounted again... Yes, it looks good.

    root@ArcherC7v2-Main:/etc/init.d# reboot
    root@ArcherC7v2-Main:/etc/init.d# Connection to 192.168.253.1 closed by remote host.
    Connection to 192.168.253.1 closed.
    bash-3.2$
    bash-3.2$ ssh root@192.168.253.1
    root@192.168.253.1's password:
    
    
    BusyBox v1.30.1 () built-in shell (ash)
    
      _______                     ________        __
     |       |.-----.-----.-----.|  |  |  |.----.|  |_
     |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
     |_______||   __|_____|__|__||________||__|  |____|
    		  |__| W I R E L E S S   F R E E D O M
     -----------------------------------------------------
     OpenWrt 19.07.0, r10860-a3ffeb413b
     -----------------------------------------------------
     
    root@ArcherC7v2-Main:~# ls -al /dev/sd*
    brw-------    1 root     root        8,   0 Dec 31  1969 /dev/sda
    brw-------    1 root     root        8,   1 Dec 31  1969 /dev/sda1
    
    root@ArcherC7v2-Main:~# df -h
    Filesystem                Size      Used Available Use% Mounted on
    /dev/root                 2.8M      2.8M         0 100% /rom
    tmpfs                    60.8M    124.0K     60.7M   0% /tmp
    /dev/mtdblock3           11.6M      3.9M      7.7M  33% /overlay
    overlayfs:/overlay       11.6M      3.9M      7.7M  33% /
    tmpfs                   512.0K         0    512.0K   0% /dev
    /dev/sda1               978.1M      2.5M    909.1M   0% /mnt/sda1
    
    root@ArcherC7v2-Main:~# ls -al /overlay/upper/mnt/
    drwxr-xr-x    3 root     root             0 Feb  8 11:44 .
    drwxr-xr-x    8 root     root             0 Mar 15 12:33 ..
    drwxr-xr-x    2 root     root             0 Feb  8 11:44 sda1
    

In /etc/config/fstab, the enabled option for the filesystem needs to be set to 1 to mount on boot.. (edit) but then there is this:

## anon_mount enables the automatic mounting of /dev/sdXy to /mnt/sdXy

You probably do not want that, if you have identified the filesystem by UUID and made a specific mount for it. @hnyman shows that is more for the use case of plugging in a drive temporarily to move some files.

/mnt/sda1 will always exist, since mounting of a filesystem must attach to an existing (usually empty) directory providing a mount point. If the drive is not mounted, anything you store there will go to internal flash, which was the problem in your original post.

Check mount or df to confirm that the drive is really mounted. Then the files in /mnt/sda1 and below are stored on the external disk.

Also the first level USB driver, the one that directly interacts with the port hardware-- usually but not always kmod-usb2 or kmod-usb3-- should already be installed in a distribution build, no need to install them separately. e2fstools is only needed to make a filesystem in place. Usually I would partition and format the drive on a desktop Linux computer using gparted.

@hnyman, @mk24, @psherman Thanks for these thoughts. I don't have a deep understanding of all these intricacies, so I am grateful for any advice from this forum. More questions:

  • (I see that df -h output is "/dev/sda1 978.1M 2.5M 909.1M 0% /mnt/sda1" (see earlier message in this topic) which I assume means that df is showing that block device /dev/sda1 is mounted at /mnt/sda1...)
  • But how could I use mount to test whether the drive is mounted?
  • Are you suggesting I should remove the anon_mount='1' command? (Since the Quick Start procedure explicitly enables specific device(s) by their UUID, it's unlikely to have anything that needs anon_mount, right?)
  • re: installing kmod-usb2 and kmod-usb3 - This couldn't hurt, could it? (Again, I'm striving for a simple, copy-paste "can't fail" procedure. Given that the setup is limited to devices with >= 8MBytes Flash, there shouldn't be a problem, right?)
  • re: installing e2fsprogs - same argument. Formatting the drive on a PC, then using mkfs.ext4 will guarantee there's a sane file system on the USB device, avoiding peculiar problems afterwards.

And finally the big question: What might cause data written to /mnt/sda1 (or /opt in my original case with YAMon) to go to the internal flash and not the external drive?

This last question is really what kicked off this topic: At least two times (with two separate installs of a USB stick) I have gone away from the router for a couple weeks only to find that /overlay/upper/opt/... is full and read-only. How could this happen? What's wrong with the procedure in the USB Quick Start?

Thanks again for all your good thoughts.

PS I edited the subject line of this topic to more closely reflect where it has moved...

I don't know if it is the best solution, but I think that if you pivot to an extroot prior to installing any of your additional applications/services, all of the programs that could generate any data that would need writable space would then reside on that external storage. Therefore, it would obviously be necessary for the extroot to be mounted prior to the execution of said programs/services, and thus you would know for sure that you wouldn't run into a situation where you were accidentally writing to the embedded flash storage.