Free up storage/disk space after unsuccesfull opkg install

Did intentionally install packages that can't fit so can check how the OpenWrt behave and what I'm capable to do with it (this is spare device so playing with it not mis-configuring my main router).


As to why is possible install packages that are above the quota available:

because fs used for adding data is JFFS2 and "is compressed, so a program (opkg in particular) cannot know in advance how much space a package will occupy"

Not sure if all devices have this as info for NAND pointing to UBIFS

Mine have: mount|grep 'jffs' -> /dev/mtdblock6 on /overlay type jffs2 (rw,noatime)

Now have some questions.


Starting on device with this space:

df -h /
Filesystem                Size      Used Available Use% Mounted on
overlayfs:/overlay        9.9M    448.0K      9.4M   4% /

Did install the set of packages:

opkg update
opkg install wireguard-tools ip-full kmod-veth iperf3 git-http

(Using this as example instead of just randomly running install whatever until fail because of no space and also the last one being really big fill the space, the remove process nicely verifiable as a lot of packages are just small and not create enough mess.)

It can't be done; this is the starting point of this adventure:

Collected errors:
 * pkg_write_filelist: Failed to open //usr/lib/opkg/info/git-http.list: No space left on device.
 * opkg_install_pkg: Failed to extract data files for git-http. Package debris may remain!
 * opkg_install_cmd: Cannot install package git-http.
 * opkg_conf_write_status_files: Can't open status file //usr/lib/opkg/status: No space left on device.

Now what to do - from 4% to 98%.

root@OpenWrt:~# df -h /
Filesystem                Size      Used Available Use% Mounted on
overlayfs:/overlay        9.9M      9.7M    188.0K  98% /

According to documentation (this whole page of wiki) it should be possible to uninstall packages:
(All info here is just to make it shorter, did that for each package by name.)

opkg remove ip-full
No packages removed.

and

opkg --autoremove remove ip-full
No packages removed.

This is same for each package attempted for install.

Wondering why - run:

opkg list-installed | grep '*all attempted to install*'

And it return none.

That make me wonder.

In general, does this mean: that the if last package, (in this example because there is no space), fails - it take down entirety from the actual opkg install run(?).

Despite seeing opkg downloading quite a lot of .ipk and some configurations:

Configuring libatomic1.
Configuring libopenssl3.
Configuring libiperf3.
Configuring kmod-veth.
Configuring librt.
Configuring git.
Configuring iperf3.

because space taken, did assume it's all there except the 'git-http' as that fails.

Well unable to uninstall anything:

(seeing some attempts ending like this)

Collected errors:
 * opkg_conf_write_status_files: Can't open status file //usr/lib/opkg/status: No space left on device.

decided continue with 'Out of space' option.

But that is not possible, unless one already can free up some space, because:

vi wq > -ash: can't create opkg-rm-pkg-deps.sh: No space left on device

For more clarity, adding it from wiki:

opkg-rm-pkg-deps.sh

Save the script

cat << "EOF" > opkg-rm-pkg-deps.sh
#!/bin/sh
opkg update
URL="$(opkg --force-space --noaction install "${@}"
| sed -n -e "/^Downloading\s*/s///p")"
rm -f /usr/lib/opkg/lock
for URL in ${URL}
do FILE="$(wget -q -O - "${URL}"
| tar -O -x -z ./data.tar.gz
| tar -t -z
| sort -r
| sed -e "s|^.|/overlay/upper|")"
for FILE in ${FILE}
do if [ -f "${FILE}" ]
then rm -f "${FILE}"
elif [ -d "${FILE}" ]
then rmdir "${FILE}"
fi
done
done
EOF
chmod +x opkg-rm-pkg-deps.sh

So had to perform 'reset to defaults', prepare the .sh before filling up the space again (or did that also by creating a bigger file in before filling up space and deleting it later avoiding the necessity of reset/reflash) and only in this scenario can proceed for:

./opkg-rm-pkg-deps.sh git-http

And run it for all of them.

After all - always the best outcome is:

root@OpenWrt:~# df -h /
Filesystem                Size      Used Available Use% Mounted on
overlayfs:/overlay        9.9M      5.6M      4.3M  56% /

Can see output from ./opkg-rm-pkg-deps.sh (shortened, many more lines, for each package few):

rmdir: '/overlay/upper/usr/lib/git-core/': Directory not empty
rmdir: '/overlay/upper/usr/lib/': Directory not empty

Would expect that it will remove them as well being instructed to do so for 'git-http'.

Do I make a mistake in the process? How one can remove files(?) so the difference 52% (measuring against clean start) is not existing anymore.

The packages can't be found with opkg list-installed - so where this remaining files, whatever it is, reside..

Well, from assuming from wiki about layout of flash its in /overlay/upper/usr.

But why it's not cleaned at all? (In my case cca 1/2.)

In /overlay/upper/usr/lib/opkg/info is quite a lot of files matching the attempted install of wireguard-tools ip-full kmod-veth iperf3 git-http

Did search forum, yes, mainly to achieve info pointing back to the wiki/reflash - but that seems to me only if some command can't be run.
(Reading wiki my initial expectations were, well I can solve this with ssh in.)

Try a lot (even had to reflash device 2 times) but no success.

Is this normal(?) or expected.

Unless user manually eradicate files (and this can be tedious work - fortunately some dirs printed out by ./opkg-rm-pkg-deps.sh) if not having enough memory/mounted USB / it's almost automatically non-recoverable state (if the package is big so the residue files are taking much space in many dirs).


This all is not critique of a the project at all as mainly an honest observation/questioning the experience.


Listing of dirs after all ./opkg-rm-pkg-deps.sh runs:

ls -d -- */ | sed -e 's/[^\/]*$//' | sort | uniq | while read dir; do du -sh "$dir"; done
| sort -rn
895.0K	www/
594.0K	bin/
390.5K	etc/
72.0K	var/
72.0K	tmp/
19.5K	root/
14.4M	usr/
14.0M	rom/
8.0M	overlay/ - in here are by name some files resembling
 what were attempted to be processed by opkg
4.3M	lib/
1.5M	sbin/
0	sys/
0	proc/
0	mnt/
0	dev/
root@OpenWrt:/# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 3840      3840         0 100% /rom
tmpfs                    28636        72     28564   0% /tmp
/dev/mtdblock6           10112      5688      4424  56% /overlay
overlayfs:/overlay       10112      5688      4424  56% /
tmpfs                      512         0       512   0% /dev

Reset the device config via luci.

Usually when you fill the overlay it goes to readonly mode and you must use failsafe to free up space. One enough space is free you can boot into the normal system and remove additional files.

2 Likes

Yeah - as reading more about FS:

1) /overlay was previously named /jffs2

2) The opkg util really has no idea how much space is available on the JFFS2 partition since it's compressed, and so it will blindly keep going until the opkg system crashes -- at that point you have so little space you probably can't even use opkg to remove anything.

--

It does fitting together more now for me - but still overall inability to clean after this event is not exactly obvious to me, and the chain reaction of opkg as it probably takes down all being installed at the run.

Not really, but jffs2 has been the most typical filesystem used for overlay, so some advice refers to it.

Like you have observed, opkg loses it's internal status info when disk gets full and status info can't be written any more to it. Using opkg reliably after that is not really possible.

The only way is to manually rm files from /overlay/upper.

Overlay contains all the changes that you have made after flashing, so it's is natural that you find the files there. And it is not only .../usr but also other subdirectories there as packages place files in various places.

Your df is misleading, as in addition to the normal (combined by overlayfs) directories it contains also the toplevel /rom (files in image) and /overlay (later changes) that actually contain all the files.