Package Makefile: Compress Binary with UPX

go is only needed to compile the package. That's why I added it as golang/host. As far as I understood it, this would only make go available as a staging tool and not included it into the package.

I am however unsure about the librarires, because I found some PRs about discussion dynamic linking of go packages [1].

And I found that dynamic linking only seems to be working for amd64 right now [2].

Disclaimer: I'm not really familiar with go (at all), but indeed have some faint memories of static linking being considered fancy.

The easy option would be trying to install your telegraaf package on vanilla OpenWrt (after opkg update) - and to just see what and how much it pulls in during the installation (for a first look, the target arch doesn't matter, x86_64 in a VM, armhf for the RPi; also record df -h before and after the installation).

For a closer look, you can check your package's dependencies manually (*.ipk files are just a tarball, and inside the control.tar.gz contains a file called control, listing your direct dependencies under Depends:).

The control file only show Depends: libc as requirement.

The memory test on ext4 you can find in this thread Package Makefile: Compress Binary with UPX - #7 by jonnytischbein and Package Makefile: Compress Binary with UPX - #9 by jonnytischbein.

For squashfs / jiffs2 I did in my PR

Both just consider the binary file, but the rest of the package is an /etc/init.d/ script and a config file. (both in total 320K)

To be clear, yes, this:

To utilize Squashfs, you need to preinstall the package upon building the firmware image.
And then compare the size of 2 firmware images with and without the package preinstalled.

I see.

With that I would measure the impact of the sqaushfs compression right ?
If I install the package in the end via the opkg manager I will "only" have the jiffs2 compression (which are already 50% in my test).

Still, I am a bit unsure what to do with result of comparing the two images. Telegraf is in the end a metric tool and I can't image that one will use an image with a preinstalled metric tool, if it will occupy 50% of the total memory (assuming my 6M result as the maximum compression result and 12M flash memory).

In the end there are always device, which could run all packages, as I encounter on some older tp-link models with luci packages. Or would my package not be accepted because it is too large and only a few devices may ever use it ?

Btw. I did another test on a mips with squashfs image, which confuses my again.

  • Like in my PR I used the telegraf version without any plugins (21M: $ du -s telegraf 20776 telegraf)
  • I ran upx --lzma -9 on the binary. Resulting in $ du -s telegraf 7928 telegraf
  • I did df before and after scp-ing: 8584āˆ’516 = 8068
  • It is more than the compressed, but still less than using "un-upxed" and jiffs2 resulting in 10424 (see here)

As I understood, I did test here not the squashfs (because it is not preinstalled) but the jiffs2 compression. My result with combining upx and jiffs2 is in my view still rewarding, because I still reduce 2M of memory usage.
Am I wrong ?

Something tells me the JFFS2 compression capability is overestimated.
You can easily test the Squashfs compression using the image builder:
https://openwrt.org/docs/guide-user/additional-software/imagebuilder

1 Like

You mean I should test the squashfs images to see that I didn't need upx right ?

Normally, yes - but if the source is as highly compressible as here, it might still be reasonable.

1 Like

squashfs compression matters only if someone preinstalls the package so yeah you can check it for curiosity.

You have already seen that jiffs2 isn't really as good as upx at compressing

So many of us use a VM or a robust device with plenty of storage. Can there be two packages? A full fledged and a slimed down? Some of the options would be nice to have, like the logfile or docker socket in telegraf.

My package contains two versions.
A small one telgeraf with only limited inputs, output, aggregators and proccessors and the other one telegraf-full with all plugins.

You can see which plugins remains here in my branch. But I am open to suggestions! :slight_smile:

1 Like

You're my hero Jonny. How do I install? I'm used to pulling from the repo.

Right now my package isn't merged to the repository.
You could compile it yourself with my branch.
Alternatively you could download the (full) binary from the telegraf github repo (there is nearly every architecture provided) and use my configuration files.

Or if you tell me which target you use I could try to compile you package.

Hey,

I did compile the squashfs image with and without telegraf (small version).
But I do not see any difference ?

~/openwrt/bin/targets/bcm27xx/bcm2710$ du  with*/*
79024	without-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-factory.img
79024	without-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-sysupgrade.img
79024	with-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-factory.img
79024	with-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-sysupgrade.img

I did verify take the package was actually (not) compiled

~/openwrt$ make -j4
 make[1] world
 make[2] package/cleanup
 make[2] toolchain/compile
 make[3] -C toolchain/gdb compile
 make[3] -C toolchain/binutils compile
 make[3] -C toolchain/fortify-headers compile
 make[3] -C toolchain/gcc/initial compile
 make[3] -C toolchain/musl compile
 make[3] -C toolchain/kernel-headers compile
 make[3] -C toolchain/gcc/final compile
 make[2] target/compile
 make[3] -C target/linux compile
 make[2] buildinfo
 make[2] package/compile
[...]
 make[3] -C feeds/packages/utils/telegraf compile
[...]
 make[2] package/install
 make[2] target/install
 make[3] -C target/linux install
 make[2] package/index
 make[2] json_overview_image_info
 make[2] checksum

Edit: I flashed the images to the Pi and the package telegraf isn't preinstalled. I selected telegraf as package with M in `make menuconfigĀ“. Was that wrong ?

You selected it to be built as a module but NOT included in the image. You just get the .ipk file.

You need to toggle the selection to "*" = include in image.

1 Like

Ohhhhh right..! Thanks :smiley: hupis.

That did the trick.

So the resulting image sizes

$ du with*/*
79024	without-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-factory.img
79024	without-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-sysupgrade.img
86364	with-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-factory.img
86360	with-telegraf/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-sysupgrade.img
85304	with-telegraf-upx/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-factory.img
85304	with-telegraf-upx/openwrt-bcm27xx-bcm2710-rpi-3-squashfs-sysupgrade.img

And to compare the binary size

$ du ~/openwrt/build_dir/target-aarch64_cortex-a53_musl/telegraf-1.19.1/telegraf_small 
33920	/home/jonny/openwrt/build_dir/target-aarch64_cortex-a53_musl/telegraf-1.19.1/telegraf_small

And the binary size compressed with upx

$ du ~/openwrt/build_dir/target-aarch64_cortex-a53_musl/telegraf-1.19.1/telegraf_small 
6208	/home/jonny/openwrt/build_dir/target-aarch64_cortex-a53_musl/telegraf-1.19.1/telegraf_small

In total the squash compresses the 34M binary into 86360āˆ’79024=7336 so around 7M.
The with upx binary 6M consumes 85304-79024=6280 so exactly the size of the binary.

So @vgaetera you are right, that the squashfs did nearly the same thing as my upx compression. However I you agree with @slh .
Including the binary/package as preinsalled is the best way if we consider the used memory space in total.
But when using telegraf as package, as it would be the more likely way, than we would benefit from a upx compression during package, resulting in used 8M as shown here

And I do not see why an upx compressed binary conflicts with the squashfs. In my case it is just not anymore compressed - as I expected.

Would you agree that I create an PR for including upx as a host tool and add upx compressin to my Makefile for telegraf ?

1 Like

Note that upx has been a build tool, but it was removed in 2017

1 Like
Remove upx from repo, it hasn't been used for a very long time and serves little to no purpose.

Well, good news is, it was dropped for non-use and probably not being maintained, rather than other issues.

Still, are new build steps are additional complications, so personally I am still doubtful of the usefulness of it, as the first use-case package will still be really large despite using upx.

1 Like

But the difference is that with upx it is useable (for my TP archer c7 with 12M flash) and without not, because the difference between 34M and 6 or 7M is huge here.
(preinstalled it usable even without upx, you are right there)