Is there any utility to minify the shell script?

Hi all,
I am looking for a utility to minify the shell scripts( this can decrease the rootfs size of the firmware ), is there any solution that can remove all the comments and useless spaces and blank line ?? Thanks!

Your best bet is sed, most likely.

Though I doubt it'll decrease the size all that much.

I have not tested, but this could work.

2 Likes

Below is all the scripts of rootfs, if we minify them, the size of firmware will decrease ~50KB.

xxxx@xxxx-build-server:~/work/openwrt/build/ath79-master/staging_dir/target-mips_24kc_musl/root-ath79$ find . -name *.sh | xargs ls -lh
-rwxr-xr-x 1 xxxx xxxx 1.6K 3月   3 17:49 ./bin/ipcalc.sh
-rw-r--r-- 1 xxxx xxxx  872 3月   3 17:49 ./etc/diag.sh
-rw-r--r-- 1 xxxx xxxx 3.7K 3月   3 17:49 ./lib/config/uci.sh
-rwxr-xr-x 1 xxxx xxxx  991 3月   3 17:49 ./lib/functions/k2t.sh
-rw-r--r-- 1 xxxx xxxx 1.5K 3月   3 17:49 ./lib/functions/leds.sh
-rw-rw-r-- 1 xxxx xxxx  734 4月   9 09:10 ./lib/functions/migrations.sh
-rw-r--r-- 1 xxxx xxxx 7.7K 3月   3 17:49 ./lib/functions/network.sh
-rw-r--r-- 1 xxxx xxxx 1.8K 3月   3 17:49 ./lib/functions/preinit.sh
-rw-r--r-- 1 xxxx xxxx  11K 4月   9 09:22 ./lib/functions/procd.sh
-rw-r--r-- 1 xxxx xxxx 2.7K 3月   3 17:49 ./lib/functions/service.sh
-rwxr-xr-x 1 xxxx xxxx 7.9K 3月   3 17:49 ./lib/functions.sh
-rw-r--r-- 1 xxxx xxxx 3.0K 3月   3 17:49 ./lib/functions/system.sh
-rwxr-xr-x 1 xxxx xxxx  12K 3月   3 17:49 ./lib/functions/uci-defaults.sh
-rw-rw-r-- 1 xxxx xxxx   84 4月   3 17:12 ./lib/mwan3/common.sh
-rw-rw-r-- 1 xxxx xxxx  34K 4月   3 17:12 ./lib/mwan3/mwan3.sh
-rw-r--r-- 1 xxxx xxxx  27K 4月   9 09:31 ./lib/netifd/hostapd.sh
-rw-r--r-- 1 xxxx xxxx 8.4K 4月   2 04:56 ./lib/netifd/netifd-proto.sh
-rw-r--r-- 1 xxxx xxxx 6.3K 4月   2 04:56 ./lib/netifd/netifd-wireless.sh
-rwxrwxr-x 1 xxxx xxxx 2.8K 4月   3 17:09 ./lib/netifd/proto/dhcp.sh
-rwxr-xr-x 1 xxxx xxxx 4.8K 4月   9 09:32 ./lib/netifd/proto/dhcpv6.sh
-rwxr-xr-x 1 xxxx xxxx 7.4K 4月   9 09:33 ./lib/netifd/proto/ppp.sh
-rw-r--r-- 1 xxxx xxxx  761 4月   2 04:56 ./lib/netifd/utils.sh
-rwxr-xr-x 1 xxxx xxxx  21K 4月   9 09:32 ./lib/netifd/wireless/mac80211.sh
-rwxr-xr-x 1 xxxx xxxx 1.5K 3月   3 17:49 ./lib/network/config.sh
-rw-r--r-- 1 xxxx xxxx  303 4月   9 09:32 ./lib/network/switch.sh
-rw-r--r-- 1 xxxx xxxx 2.2K 4月   9 09:26 ./lib/nft-qos/core.sh
-rw-r--r-- 1 xxxx xxxx 2.2K 4月   9 09:26 ./lib/nft-qos/dynamic.sh
-rw-r--r-- 1 xxxx xxxx 1.4K 4月   9 09:26 ./lib/nft-qos/monitor.sh
-rw-r--r-- 1 xxxx xxxx 2.1K 4月   9 09:26 ./lib/nft-qos/priority.sh
-rw-r--r-- 1 xxxx xxxx 1.8K 4月   9 09:26 ./lib/nft-qos/static.sh
-rw-r--r-- 1 xxxx xxxx  771 4月   9 09:22 ./lib/uboot-envtools.sh
-rw-r--r-- 1 xxxx xxxx 5.2K 3月   3 17:49 ./lib/upgrade/common.sh
-rw-r--r-- 1 xxxx xxxx 1.5K 3月   3 17:49 ./lib/upgrade/fwtool.sh
-rw-rw-r-- 1 xxxx xxxx  360 4月   3 17:13 ./lib/upgrade/luci-add-conffiles.sh
-rw-rw-r-- 1 xxxx xxxx 1.3K 4月   3 17:09 ./lib/upgrade/platform.sh
-rw-r--r-- 1 xxxx xxxx 3.0K 4月   9 09:32 ./lib/wifi/mac80211.sh
-rwxr-xr-x 1 xxxx xxxx  496 3月   3 17:49 ./sbin/led.sh
-rwxr-xr-x 1 xxxx xxxx  839 4月   9 09:32 ./usr/lib/dnsmasq/dhcp-script.sh
-rwxr-xr-x 1 xxxx xxxx  104 3月   3 17:49 ./usr/libexec/login.sh
-rw-r--r-- 1 xxxx xxxx 5.4K 4月   9 09:20 ./usr/share/libubox/jshn.sh

If they’re in ROM, they’re already compressed by squashfs generation.

A quick sanity check of "decrease ~50KB"

tar cvf /tmp/scripts.tar $( find /rom -name '*.sh' )

  File: scripts.tar
  Size: 169472  

 xz -9 -k scripts.tar

  File: scripts.tar.xz
  Size: 34300  

It's that roughly 34 kB compressed you're going against, so I find it hard to believe that you can save 50 kB.

Past that, since many interact, you’d need something that takes them as a whole when changing variable names.

Manually removing case statements might gain a tiny bit as set-up scripts generally contain values for each variant of the target.

Minification also renders scripts nearly unreadable by the end user. Or is that the point?

3 Likes

Minification is necessary for devices whose ROM is equal with 4MBytes. Any you can see at least 3 relevant in LuCI modules option.

@rosysong There are currently >40 TP-Link WR841N with 4 MiB flash lying on my table where I replace the chips with 16 MiB. And for each of them I need to backup the ART first. Please under no circumstance save on the chips when you produce your router :wink:.

In the long run 4 MiB won't be enough. Everything is already extremely optimized. Lua scripts are already minified on 4 MiB devices, but for bash scripts it is not that easy and likely wouldn't make a big difference. If someone comes up with a solution, great. But you are supposed to produce cool routers for a little buck. I'd love to see you coming up with a CE-certified IP40xx router for less than $50. We'd immediately buy 10,000 of them.

1 Like

As jeff already mentioned, shell scripts compress very well as part of the global squashfs-{lzma,xz} rootfs compression, if that is your concern, you have bigger fish to catch elsewhere (and shouldn't have skimped on the hardware in the first place).

2 Likes

Yes, we do have plan on this in 2019, you can contact my email (rosysong@rosinson.com) for more details.

Maybe it can't save 50 KB, but this really meaningful for size decreasing.
Below is the output after xz -9 -k xxx.sh, thanks! :slight_smile:

 root@PC# ls -l
...
-rw-rw-r-- 1 zoolan zoolan 276371 4月   9 09:38 a.lua
-rw-rw-r-- 1 zoolan zoolan   1044 4月   9 09:38 a.lua.xz
-rw-rw-r-- 1 zoolan zoolan 220770 4月   9 09:40 az.lua
-rw-rw-r-- 1 zoolan zoolan    184 4月   9 09:40 az.lua.xz
1 Like

What and how exactly have you been testing there? How did you arrive arrive at 220 KB of source that compress down to 184 bytes? Something is off there.

2 Likes

Those are Lua files. They are already being minifed using https://github.com/stravant/LuaMinify. OpenWrt images are compressed using SquashFS's LZMA in blocks. It's not comparable with the xz utility.

(Even with xz's more sophisticated compression I'm having a hard time to believe that 220 KB of Lua code compresses down to 184 bytes. That's what I was going for. If that is feasible then please forgive my ignorance.)

2 Likes

Are you working for ZOOLAN, perhaps? You seem to have nice products over there, and the user interface on this manual looks familiar; but your download page seems a bit vacant (if you know what I mean).

1 Like

@eduperez That's Rosy's OpenWrt theme https://github.com/rosywrt/luci-theme-rosy

BTW Zoolan looks like a reseller and not a manufacturer.

See include/image.mk

LZMA_XZ_OPTIONS := -Xpreset 9 -Xe -Xlc 0 -Xlp 2 -Xpb 2
target/linux/generic/config-3.18:CONFIG_SQUASHFS_XZ=y
target/linux/generic/config-4.9:CONFIG_SQUASHFS_XZ=y
target/linux/generic/config-4.19:CONFIG_SQUASHFS_XZ=y
target/linux/generic/config-4.14:CONFIG_SQUASHFS_XZ=y

Note also that my use of xz was to get a rough handle on how much the files had already been compressed in the ROM, not how much they could further be compressed.



I've got to agree that 4 MB NOR is not a good choice at this time (nor is 8 MB much better).

Digikey, a major, first-quality, commercial, component distributor in the US lists Winobond SPI NOR (W25QnnnJVSTIQ) at:

  • 4 MB -- $0.59
  • 8 MB -- $0.79
  • 16 MB -- $1.14

in single-reel quantities

No doubt they are even cheaper through Chinese channels, and at higher volumes.

Does that make any difference?

1 Like

@jeff Wow, that's expensive. I buy 16 MiB/W25Q128FV or even W25Q128JV (industrial grade) for $0.50 at eBay or Aliexpress. Shipping takes less than two weeks from China. What is a single-reel? To me a reel is a reel with a lot of chips.

@eduperez Resellers are totally overprized and you can't make a real deal with them. Furthermore they don't have a clue about the software they are using in most cases. It's not hard to find out who the manufacturers are, because there are only a few in Shenzhen and most of them do direct sales now. You can do e.g. software engineering and help them with certification or something else to get a competitive pricing. It's one hand washes the other. I'd never buy something from resellers in a bigger quantity. Actually I wouldn't buy from Apple or some other exploitative company that has lost touch with reality with their excessive pricing, too. You feed their cruelty when you do.

Edit: Xiaomi, TP-Link, Yunlink, Zhibotong, GL-iNet, Comfast, Gainstrong, Tenda, Mercury, Oolite, MTC, Dragino, Afoundry, Century Xinyang, MTN electronics, Netcore and Rosinson of course. Those are the companies that I know of which create own board designs. If there is one you know that is not listed, please tell me.

Sorry this thread is getting off-topic.

1 Like

How about 2MB and 1MB ?

I hope you're joking.

I've pointed out that the incremental cost for first-quality parts from a reputable US-based supplier is $0.20 to go from 4 MB (MBytes here, and in my previous post) to 8 MB of flash, and not much more to go to 16 MB. There should be zero design cost, as I believe they all have the same physical and electrical interface up through 16 MB. As CodeFetch points out, 16 MB flash on the secondary market are pennies a piece.

2 Likes

I am looking for a way to run openwrt in some low cost machine(high integrated SoC with 8-16MBytes ddr and switch embedded), otherwise porting utilities from openwrt to freertos is a disaster.