Topic: No space left on device.

Greetings,

I've looked around for quite awhile but I just wanted to be sure that there's nothing I can do before I potentially brick this router trying to reset it.

I have a 4 MB WRT54g.    Well, it's actually a Motorola 850g but it shows up as "Linksys WRT54G/GS/GL."    I used to run DD-WRT o it and switched to OpenWRT several months ago.

Anyways, I have several OpenVPN tunnels running on the device.   I needed to add a new one.   I couldn't create the configuration files because it said the device was full.

Filesystem                Size      Used Available Use% Mounted on
none                      7.0M     48.0k      6.9M   1% /tmp
/dev/mtdblock/4           2.0M      1.9M     52.0k  97% /jffs
mini_fo:/jffs             1.2M      1.2M         0 100% /

I deleted the OpenVPN log file and I deleted the configuration files for one of the tunnels.  I saw some k free up on the /dev/mtdblock/4 device, but still, still get "no space left on device."

I've read around the forum, and this appears to be a normal problem?   I don't understand.   I don't have a console port or serial port on this thing so I don't know how I'm supposed to boot into a safe mode and run the wipe command - and I really would rather not wipe the configuration!

What's the options here?   Is this a known issue?   Is it an issue with all flash-based routers like this or is it specific to the WRT?

I'm out of ideas.

Re: No space left on device.

OpenVPN is a hefty packages, it and the dependencies take up a lot of space. The log files should not be stored in flash, but in the ramdisk (/tmp and /var) or not enabled.

Run `du -sh /jffs/*' then run that command on the largest folder within /jffs/, and keep doing that until you know which files/packages are taking up the space.  You can't use ipkg to remove packages (it can't record which packages were removed) until you free up space, so you'll have to delete files first.  You can save space by building packages into the image when you create it. SquashFS is a more efficient filesystem than JFFS2, but of course is read only.

Re: No space left on device.

Okay, but that doesn't explain why when I deleted some items, I still cannot make any changes or any new files.   I cleared up about 35KB by removing the configuration file for two tunnels, and I still can't create a single 1 byte file.

Re: No space left on device.

can you delete more files ?
if no then your only help is to reflash...
if jffs once was overloaded => defect (its still running but nor more changes are possible on this fs

ciao gerd

Re: No space left on device.

So that's a bug in the software?   Once it's filled up, you can no longer clear up space?    Shouldn't there be a safeguard against that - like leave 5k available for cleanup purposes?

Can I look forward to this happening over and over again?   Anything I can do to prevent this next time?

Re: No space left on device.

ps.  Because of this problem, I can't even back up the configuration.  I can't do anything.

I'm sorry I even tried OpenWRT.   Apparently I now I have a bricked router.

Thanks OpenWRT for the non-warning!

Re: No space left on device.

its not a bug of the software its behaviuor of the filesystem...
why you cannot backup files ?
copy/tar them to /tmp  and use scp to move them to a linux box.....
you can prevent  sytem with good config... :-) => write logfiles to /tmp

ciao gerd

Re: No space left on device.

I second the request for a warning of some kind... I have just encountered this for the second time (currently trying to cool down a bit to not insult anyone here)...
And i do not want to reflash everytime I try some packages which accidentally use up to much space.

(there should be a size shown in ipkg also.... then one could avoid this issue... e.g. like in APT, the selection and its prerequisites will take up X bytes.)

I understand this isn't the fault of OpenWRT itself, still it is very annoying...

Re: No space left on device.

check disk space BEFORE installing :-) or mod it with a mmc card.
btw: you can see as long as i'm registered here i never heard someone else complaining like this about the behaviour of jffs it's well known that jffs reacts like this when its too full.
the advantage is that the wrt is still functional. (with my sat receiver which also uses jffs for firmware i had it many times)
But another hint => if it this behaviuor bugs you => go back to linksys firmware and be happy that you get rid of this problem.

ciao gerd

Re: No space left on device.

Already reflashed Kamikaze...
Going back to Linksys fw is simply not an option, I'm using an Asus wl500gx ;-)

I will try to follow the overlay fs on USB HowTo in this board (I somehow missed that the last time; adding a 128MB CF I have laying around this time should be enough for the packages I want).


btw: Is the size of a package displayed somewhere else than on the repo http?

Re: No space left on device.

OKOK lets say "stock firmware" :-)
But if you use a CF card for additional packages why you do not set it up as ext2 ? I always use on my alix boards ext2 => so even if fs is corrupted or damaged i have every option on my linux server to repair or get at least cfg files...
But also on my wrt' s  (the run with  a 2 year old openwrt version) backup if fs is damaged ? they use nvram and backup of config files will be done with a terminal log so quite 20 minutes to refleash and bring em back to same state....

ciao gerd

12 (edited by LordOfThePings 2008-08-11 23:09:14)

Re: No space left on device.

I am setting it up as ext... currently I'm trying to get the pivotroot to work completely (struggling with wlan not getting an ip, could be totally unrelated).

The last time I had a smaller (32MB) CF as /usb (not pivotroot), which caused chaos with symlinks und ipks installed to the wrong fs, filling up the / jffs in main flash and finally crashing some hours ago.
I suppose this will be easier with pivotroot, as I won't have to keep track what is installed where, and how much space it takes (to some degree, 128mb could be filled quickly, but at least it won't crash the whole fs)...


edit: yes, was totally unrelated... rebooting fixed it, seems like the wlan just did not connect correctly...

13

Re: No space left on device.

Hi,

you can use the command "mtd erase mtd4" to entirely erase the jffs part of your flash memory.

After rebooting the device you should get an empty jffs filesystem.
The read-only part of your firmware images will be left intact.

Re: No space left on device.

I don´t believe it!!! Openwrt is widely used with small devices with a very low amount of memory. Going into this is very common, SOMETHING SHOULD BE MADE TOFIX THIS!! Its completely crazy blaming the users for this. It happened to me again, once in a year it happens, and there is nothing I can do, over six routers I had configured for friends and family. In a wrt54g installing openvpn and almost anything else fills the entire filesystem and there is no other solution than reflashing.

Right now I filled a router 200 miles away. So I have no choice of reflashing, not from here. Reflashing over the Internet, yeah! sure! NO WAY! And everyone points to the user: you should check always first! C'MON!!! ITS A BUG!!! It's RIDICULOUS to have this kind of behavior on an OS used so WIDELY ON SMALL DEVICES!!!! PLEASE!!!

I'm going to bed now, but GOD HOW I HATE THIS MOMENTS!! HOW I HATE THIS!!!!! Everything else is so great on openwrt, but this. Every time I install something I have to make calculations and start to wonder if I'm doing things right. F*CK I MISSED A COUPLE OF BYTES!! PLEASE!!!!!!!! THIS IS A BUG!!!!

THIS IS REALLY A BUG!!!!! And something SHOULD BE DONE!

Re: No space left on device.

It's not a tumor.

16 (edited by lexa2 2010-05-19 03:57:51)

Re: No space left on device.

tristangrimaux wrote:

In a wrt54g installing openvpn and almost anything else fills the entire filesystem and there is no other solution than reflashing.

You may always compile your own frimware from the OpenWRT sources and include in it the packages you _really_ need pre-installed on the squashfs.
You may always use the image builder to build up the firmware image with all the packages you _really_ need pre-installed on the squashfs.
You may always try to use extroot package on top of SD mod or USB stick (in case your router have got USB ports) to avoid the jffs2 usage.

tristangrimaux wrote:

Right now I filled a router 200 miles away. So I have no choice of reflashing, not from here. Reflashing over the Internet, yeah! sure! NO WAY!

Just the same applies to any system administration task that may bring the device/server down. Reconfiguring network interfaces, changing firewall rules, restarting the device, installing new packages - any operations of such kind should only be done when you've got immediate physical access to the box in case something went wrong. Ignoring this rule will put you at risk of sitting 200 miles away from device moaning loudly "Ups, I bricked it again!".

tristangrimaux wrote:

And everyone points to the user: you should check always first! C'MON!!! ITS A BUG!!! It's RIDICULOUS to have this kind of behavior on an OS used so WIDELY ON SMALL DEVICES!!!! PLEASE!!!

OpenWRT is not an "end-user" friendly OS. It require a lot of knowledge and *nix system administration skills in order to be used properly. Feel free to use more user-friendly firmwares like DD-WRT or even original vendor-supplied firmwares and you will be free from obeying the rule "triple check anything you type as root". Don't blame OS for it's extreme flexibility, blame yourself for not being able to use the system wisely.

tristangrimaux wrote:

Every time I install something I have to make calculations and start to wonder if I'm doing things right.

By the way, opkg should refuse to install any package if it would not fit within free space on the root filesystem. At least it used to do so on the firmwares I recently compiled for the D-Link DIR-320 using the latest sources from SVN trunk. If it didn't warn you and then stalled/failed during the untaring of the package due to the lack of free space feel free to fire the bugreport to the openwrt bugtracker.

Another thing to note is that I've been able to recover from the fully filled jffs2 filesystem on one of the OpenWRT boxes by the simple deletion of files using "rm", but the minimal required amount of "free" space for filesystem to become writeable again was about 256Kb. Of course, YMMW.

Last but not least. If you feel yourself adventurous and lucky and you have got enough knowledge not to make any errors you may always proceed with backing up most essential configuration files, pivot_rooting back to the read only squashfs, creating tmpfs, populating it with minimal subset of files from "stucked" jffs2 partition, pivoting_root to mini_fo with the prepared tmpfs overlaid over the read only squashfs, using mtd to erase the contents of the jffs2 mtd partition (usually called rootfs_data), formating it back to jffs2, re-mounting and re-populating with the essential configuration files you had backed up earlier and finishing by pivot_rooting back to the mini_fo with newly formated jffs2 partition overlaid over the read only squashfs. It is very hard to do it successfully on a production system especially when you're far away from the box and use VPN link over internet to get into shell. But it is possible for sure: I had just given this procedure a try on one of the DIR-320 boxes I use for experiments with the openwrt.

Re: No space left on device.

Also, I have updated firmware on a router I manage that is 300 miles away without any problems, over an OpenVPN tunnel over the internet. I just create and test the custom trx image on a local router prior to uploading it to the remote router.

Re: No space left on device.

Void Main wrote:

I just create and test the custom trx image on a local router prior to uploading it to the remote router.

That,s right. It is one of the widely accepted good practices in the administration of the openwrt-based routers.