Calculate file space usage

I want to calculate how much space does a file take up in a compressed filesystem.

The problem is a bit confusing but I will try to explain it with an example:

I have a device with around 3MB of physical flash space.
Checking the filesystem space usage with df outputs that the overlayfs:/overlay filesystem has around 3MB of available space.

root@XXX:/usr/bin# df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlayfs:/overlay        3264      2136      1128  65% /

I can have a file of size 10MB inside this filesystem because it is compressed, using du <FILENAME> outputs that it takes 10MB of space.

root@XXX:/# du -sh 10MB_FILE 
10.0M	10MB_FILE

Question: is there a way to see how much space does this 10MB file take up in relation to 3MB compressed filesystem? I can get an approximation of this number by checking how much space is left on this compressed filesystem before and after adding this file but this method is not feasible for me.

That's a wrong assumption. The overlay is uncompressed, hence the standard advice to include any extra packages you need into the image you flash. Only the read-only Squashfs root is compressed.

Nevermind me, overlay is still compressed, just not as much as the read-only SquashFS. Thanks @jow.

In that case, how is it possible to have files of bigger size than shown with df? I understand that it is best to include extra packages into the image but my question is related to installing additional packages with opkg. Lastly, I'm just trying to get a better understanding of the filesystems on OpenWRT.

I will try to show it with another example. Here is a sample filesystem of my device:

root@device:/# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                10688     10688         0 100% /rom
tmpfs                    62416       172     62244   0% /tmp
/dev/mtdblock6            3264      2184      1080  67% /overlay
overlayfs:/overlay        3264      2184      1080  67% /
tmpfs                      512         0       512   0% /dev
/dev/mtdblock7             576       248       328  43% /log

Now, I transfer a new file of size 10MB from my host computer to this device on path /usr/bin:

X@host_PC:~/fabricated_files$ du -sh 10MB_FILE 
10M	10MB_FILE
X@host_PC:~/fabricated_files$ scp 10MB_FILE root@192.168.1.1:/usr/bin
10MB_FILE                          100%   10MB 203.3KB/s   00:50

Now I check again the device's filesystem and size of the file:

root@device:/usr/bin# df 10MB_FILE 
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mtdblock6            3264      2456       808  75% /overlay
root@device:/usr/bin# du -sh 10MB_FILE 
10.0M	10MB_FILE
root@device:/usr/bin# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                10688     10688         0 100% /rom
tmpfs                    62416       172     62244   0% /tmp
/dev/mtdblock6            3264      2456       808  75% /overlay
overlayfs:/overlay        3264      2456       808  75% /
tmpfs                      512         0       512   0% /dev
/dev/mtdblock7             576       252       324  44% /log

Now du shows that it still is of size 10MB yet the filesystems /dev/mtdblock6 and overlayfs:/overlay used space increased around 28% or around 272KB (2456-2184=272). I'm assuming that the new file takes up around 272KB of the compressed filesystem's space. This confuses me and that is the reason I have asked this question. Maybe I am missing some point here?

I have tried reading about filesystems on OpenWRT wiki but I still don't quite understand how this works.

There is basically the same question asked elsewhere https://stackoverflow.com/a/31622594/14984008. I wanted to ask if there is any way to calculate an approximate size a file takes up in relation to the compressed filesystem or is it pretty much impossible to do that?

How much space the file takes depends on the file's contents. There is no one, single rule for that. If you don't know the contents of the file, then you can't determine the space it will use, either.

Afair the overlayfs is compressed, but not as thoroughly as the readonly squashfs one.

A very very rough approximation would be something like cat file | xz | wc -c on your desktop. The actual storage footprint depends on further factors (data in adjacent sectors, squashfs vs. jffs, specific compression parameters and dictionary size chosen by the FS etc.)

2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.