Ultimate Guide for 4MB flash devices – Expert only

Hi all,

As a lot of you may have old devices left in cupboards/drawers, I decided to share some tips to help you build a recent version of OpenWRT on hardware with limited specifications. I know there is a wiki page on 4MB devices, but what I will describe on this page is more dedicated to expert users. I am not responsible for any issue with you devices.

The goal is to explain you how to build a full-featured (for daily use at least) and recent image (including LuCI web interface). With the right optimizations, it is possible, to include almost a dozen of MB of software into a 4MB (squashfs is highly compressed).

First thing you need to know, is that the device-specific images provided by OpenWRT are actually generic images with specific header/packages. They all includes a lot of unneeded code. The goal is not to produce a small image with very limited functionalities, but really to get rid of the β€œdead” code included in the firmware and not used by the hardware.

I have little time so if anyone wants to continue this work, create a wiki page and improve the explanations I would be very happy. Some optimizations could also be scripted (especially driver trimming for broadcom platforms).

[EASY]
1 - Files edition
1.0 - Package selection
That is always a good starting point, I will not go into detail for this one. You can also take some ideas from https://openwrt.org/faq/build_image_for_devices_with_only_4mb_flash.

1.1 – Remove comments from config files
By placing a files folder at the root of your OpenWRT sources, you can replace files in the image. Easiest gains are default config files, as they usually contain a lot of comments. There is no impact on daily use as you will configure firewall after the first boot anyway. Note that these files are also copied on the config partition, so it is a double gain (especially is you are following instructions from the section 4). Easiest gains are to simplify firewall and uhttpd default configuration files.

etc/config/firewall

config defaults
	option syn_flood	1
	option input		ACCEPT
	option output		ACCEPT
	option forward	REJECT

config zone
	option name		lan
	list   network		'lan'
	option input		ACCEPT
	option output		ACCEPT
	option forward	ACCEPT

config include
	option path /etc/firewall.user

etc/config/uhttpd

config uhttpd main
	list listen_http	0.0.0.0:80
	list listen_http	[::]:80
	list listen_https	0.0.0.0:443
	list listen_https	[::]:443
	option redirect_https	1
	option home		/www
	option rfc1918_filter 1
	option max_requests 3
	option max_connections 100
	option cert		/etc/uhttpd.crt
	option key		/etc/uhttpd.key
	option cgi_prefix	/cgi-bin
	list lua_prefix		"/cgi-bin/luci=/usr/lib/lua/luci/sgi/uhttpd.lua"
	option script_timeout	60
	option network_timeout	30
	option http_keepalive	20
	option tcp_keepalive	1
	option days		730
	option key_type		rsa
	option bits		2048
	option ec_curve		P-256
	option country		ZZ
	option state		Somewhere
	option location		Unknown
	option commonname	'OpenWrt'

Other files can be modified to remove unneeded function (IPv6 support, switch/VLAN support) depending on your needs.

1.2 – Removal of upgrade through firmware
99% of the routers can be flash through recovery tools from manufacturer (e.g. nmrpflash for Netgear) or TFTP, therefore sysupgrade process is most of the time a nice-to-have. Considering you need to compile your firmware manually; it is unlikely you will update it very often. Therefore, you can remove the firmware upgrade functions of sysupgrade. You need to partially keep sysupgrade to ensure you can still export and import configuration backup.
The minimum functions for that are:

-	list_conffiles
-	list_changed_conffiles
-	add_conffiles
-	do_save_conffiles

You also need to keep CONF_RESTORE and CONF_BACKUP handlers, along with -r and -b options.
Your /lib/upgrade/common.sh file can also be trimmed, by keeping only the following functions:

-	libs
-	run_hooks
-	ask_bool
-	v

Once your sysupgrade is trimmed, you can add the following lines under the prepare_rootfs section of your rootfs.mk.

        rm -f $(1)/lib/upgrade/do_stage2
        rm -f $(1)/lib/upgrade/fwtool.sh
        rm -f $(1)/lib/upgrade/luci-add-conffiles.sh
        rm -f $(1)/lib/upgrade/platform.sh
        rm -f $(1)/lib/upgrade/stage2
        rm -f $(1)/etc/sysupgrade.conf
        rm -f $(1)/usr/libexec/validate_firmware_image
        rm -f $(1)/sbin/pkg_check
        rm -f $(1)/sbin/upgraded
        rm -f $(1)/usr/bin/fwtool

At the end, you will have no impact on the daily use of your device, and you can still perform firmware updates using manufacturer tools.

[MEDIUM]
2 - Selection of drivers
In most devices, you will have the following elements a chipset, a flash memory, one or more network PHY, and some buttons/leds.

Default OpenWRT images are not fully hardware-specific, even if each device uses a specific flash file. Kernel configurations are shared among a wide range of devices for each platform. Consequently, the default kernel configuration contains a lot of unrequired drivers.

My advice is to first install a build without UI and with limited package. From there look at dmesg output and identify the drivers being loaded. You can also explore /sys/ to gain some knowledge on your system.

Once drivers are properly identified, it is time to configure the Device Drivers section, using make kernel_menuconfig command. Why is that important? Because the kernel and the root filesystem are compressed independently, and small gains in the kernel are usually more significant than small gains in the root filesystem. Kernel is also decompressed in RAM, so every byte gained here is RAM savings.

Below, an example of an (almost) minimal kernel driver list for WN3000RP (BCM47xx platform).

Storage
Memory Technology Device (MTD) support

  • BCM47xx partitioning support
  • Partition parsers > Parser for TRX format partition
  • Caching block device access to MTD devices
  • RAM/ROM/Flash chip drivers > Detect flash chips by CFI probe
  • RAM/ROM/Flash chip drivers > Support for CFI command 002
  • Self-contained MTD device drivers > Support for serial flash on BCMA bus
    ChipCommon-attached serial flash support

Ethernet
Network device support

  • Network core driver support
  • Ethernet driver support > Broadcom iProc GBIT BCMA support
  • MDIO bus device-drivers
  • PHY Device support and infrastructure > Switch configuration API
  • PHY Device support and infrastructure > Broadcom bcm53xx managed switch support (MDIO only)
  • MDIO Bus/PHY emulation with fixed speed/link PHYs

GPIO (necessary for buttons, led)
GPIO Support > /sys/class/gpio
BCMA GPIO driver
LED Support > LED Class Support (+ some LED triggers if you want)

Watchdog (else device will autoreboot every X seconds)
Watchdog Timer Support

  • WatchDog Timer Driver Core
  • Update boot-enabled watchdog until userspace takes overs
  • Broadcom BCM47xx Watchdog Timer

Chipset
Broadcom specific AMBA (SoC support + MIPS core driver)

[EXPERT]
3 - Driver trimming
Linux kernel is in fact developed to cover thousands of devices. One driver can cover dozens of devices. Some drivers are split in configurable parts through kernel config, but most are delivered as a single package.

This means that, when you are loading b53 driver (for example), you are actually loading a driver able to handle more than 10 different devices / hardware revisions. But… your device only contains one specific hardware revision! It means that a significant amount of space is lost to store useless functions. No optimization is done at compile time, as the conditionals checks from most drivers depends on the devices detected on the bus at runtime.

Here, we want to develop a truly device-specific OpenWrt image, so the idea is to get rid of all the codes which is not related to our specific embedded hardware. Optimization is a never-ending work; you should only focus on easy optimizations on large files. In the table below, an example for WN3000RP device.

Board
arch/mips/bcm47xx
board.c, buttons.c, leds.c, setup.c, time.c
arch/include/asm/mach-bcm47xx/
bcm47xx_board.h

Chipset
drivers/bcma
core.c, driver_chipcommon.c, driver_chipcommon_pmu.c, driver_chipcommon_sflash.c, driver_gpio.c, driver_mips.c, main.c, scan.c, sprom.c
drivers/firmware/broadcom
bcm47xx_nvram.c, bcm47xx_sprom.c

Ethernet
drivers/net/ethernet/broadcom
bgmac-bcma.c

Ethernet PHY
drivers/net/phy/b53
b53_common.c, b53_mdio.c, b53_priv.h

You may wonder what needs to be changed. My advice is to focus on the case statements. Typically, your device only has one chip ID. In the example below, if your device is BCM53573, there is no reason to keep code BCM57357 / BCM5372.
Capture d’écran 2020-12-11 aΜ€ 19.53.14
4 - Flash optimization
4.1 – Reclaiming unused space
Some vendor uses a partition between the rootfs and the nvram partition (detected as POT partition). In some case, you can reclaim some blocks from this partition for rootfs-data but you need to be very careful.

Three things should you NEVER erase:

  • The CFE
  • The NVRAM (used by the CFE)
  • Magic number, if any

Taking the example of WN3000RP, the POT partition contains one empty block, and the next block starts with a magic number. It is possible to trick the POT partition detection to reclaim the free block before the magic number. Once you have done that you should always use firmware compiled with the same trick, as installing another firmware will erase the magic number, hence making the firmware non-flashable through CFE. If you do that by mistake and you do not have mtd installed in your firmware your only solution to flash is unsoldering the NAND/NOR chip and using programmer.

However, if you are careful enough, this is an easy way to reclaim a free block from your flash memory.

Example for BCM47xx platform:
In drivers/mtd/bcm47xxpart.c, replace the POT header by the start of the magic number from the next block.

1

4.2 – Optimizing configuration partition
Note that this section is mainly useful if your flash memory uses 64KB blocks (which is the most common). If you have 4KB blocks, there is no reason to move away from JFFS2 (you are not losing space).
Also note that NAND have limited erase cycles. The smaller the configuration partition is, the more wear to the flash memory. You won’t do config update every day so you are unlikely to encounter issues (most default firmware use NVRAM config on a single block anyway), but keep that in mind. For NOR no issue is expected under normal use.

4.2.1 Easy – JFFS2 reserved block reduction - From 256KB to 192KB
As best practice from point 1, you should embed all the packages you need at compile time and avoid the use of opkg. Without a package manager, there is no need for a large storage partition. However, you will notice that, by default OpenWRT forces you to use 256KB of the flash memory for the configuration partition. It is more than 6% of the space that you cannot use fully. The reason behind this is that JFFS2 uses garbage collection and needs free blocks for that. This is not a limitation specific to OpenWRT, JFFS2 needs 4 blocks by default.

The good news is that JFFS2 can work with 3 blocks, so you can easily reduce the partition size to 192KB. You will simply need to ensure you use less than 64KB of data (which is usually more than enough for configuration, even with VPN certs).
How to do that:
In drivers/fs/jffs2/fs.c, update the minimum number of sectors from <5 to <4.
2

In addition to that, edit include/image.mk and remove the 256k mark.

We could add a 192k mark, but there is no added value, and padjffs2 only handle 2^n parameters, so it would be additional rework. JFFS2 partition will expand to 192KB on the first boot anyway. The only caveat is that the flash file will be 64KB smaller than the actual size needed, so take that into account when building your image. If you forget, you will easily notice it, it will fail to create the JFFS2 partition on boot and it will not be possible to save config.

4.2.2 Advanced – Moving away from JFFS2 - From 256KB to 64KB
The previous fix is simple to implement but you are still losing 128KB for the filesystem. The most efficient fix is to go for a non-filesystem-based configuration management.

One option is to slightly rework the snapshot-tool package, to ensure every configuration update is written over the same block. Basically, you are not using snapshots, but you benefit from the fact it is already developed to be compatible with overlay.

[TIME-CONSUMING]
5. LuCI trimming
LuCi contains code to covers all the potential situation. It means that the base module contains code to handle Wireless, Switch, VLAN, etc.
If your router does not have switch configuration or VLAN, you may want to remove the related code in LuCI. This is time-consuming, but it is really worth the size gain.

The main drawback is that, unlike drivers, LuCI undergoes a lot of development so you have to redo the work for each release, which takes time.

7 Likes

Hi,
Can you give an example of what you achieved after applying the above "optimizations"?
I only focused on https://openwrt.org/docs/guide-user/additional-software/saving_space

Both are really complementary, the saving_space guide shall be the starting point for anyone.

Some optimizations however impact the performance of the device. For example, increasing squashfs block size can be an option BUT it will increase RAM usage a lot:

  • 256KB BS, 3 blocks cached in RAM = 768KB
  • 1MB BS, 3 blocks cached in RAM = 3MB used
    One of the most valuable trick to use in the saving_space guide is "Making all kernel modules built-in". Note that you need to first compile the kernel once with modules non built-in, run the command and recompile (else, it will not compile). You also need to reissue that command every time you run kernel_menuconfig.

This thread offers optimizations that do not impact daily use functionalities (we only removed what is unneeded), and not affecting performance (even some slight gains of performance, by reducing and trimming drivers). The tradeoff is that it may not be worth the time investment. Driver trimming could really be automated/scripted for the most common drivers like b53 for example.

For WN3000RP I was able to trim down the ROM from more than 5MB to 3.6MB, and to allow default image size increase from 3.4MB to 3.6MB using the tricks from section 4.

It allowed to create an image with:

  • Speed optimized kernel
  • LuCI GUI
  • OpenVPN with OpenSSL optimized for speed rather than size
  • Wireguard
  • SQM
  • DDNS

Image without OpenVPN is smaller than 3MB, so it is plenty of room to put additional packages !

Something else I did not mention, by default, kernel is compiled with instructions available for all the CPUs belonging to the target (e.g. MIPS 74K) and optimized for a specific subset of the architecture (mtune flag). You can add change the march flag to ensure your kernel is built specifically for your device CPU. Refer to [SOLVED] Target options - march & mtune for additional information.

1 Like

Is this guide still valid for latest OpenWRT 22.03.5?

Theoretically, yes. But since dec '20 the default kernel + squashfs has grown by 50%. So the firmware you could squeeze in 4MB back in 2020 now needs an extra 2MB to squeeze away. You can try, but I think it's mission impossible.

1 Like

It's not impossible. I successfully built 22.03.5 for 4MB devices.

1 Like

+1. But needs careful (and "expert") tuning of included modules/packages. To be built from source, of course.

@kent_c and @reinerotto Can you share your diffconfig. I will be great if you can also write a few worlds about extra options which you disable/enable

1 Like

if you really want to go crazy, it might be possible to just get rid of JFFS2 altogether. AFAIK the only drawback would be that change of configuration would require flashing again.

For whatever reason the kernel keeps growing, so that might be necessary one day to continue running the latest on 4 MB.

This is really not something that will save any space, there is no difference in instruction sets between 24k MIPS and 74k MIPS, it's simply different in the hardware only (what they call pipelines i guess). there are only some extensions to the MIPS instruction set, no way to cut down on it, and then there are chips that are less-than MIPS which are not supported by linux at all, like the RTL819x (lexra) which would be treated like a separate arch if ever supported. for example there are extra instructions available to use on 24kec MIPS which is for complex maths related to digital signal processing (DSP). remember that more instructions available will almost always decrease the code size. however standard networking will not be using that DSP extension anyway. On the other hand, mips16 instructions are supported by all modern MIPS and are (or should be) enabled for packages that don't suffer large performance loss from using that hardware feature. so you can reclaim some space by enabling mips16 on some packages (like busybox or dropbear I think, i don't remember) but you would expect a noticable difference in behavior, which is why they turned it off for that package. At the end of the day, they made that decision based on performance only as far as I can tell, the space saved by enabling mips16 for a package is in the magnitude of bytes per kb, maybe not even 1 percent in some cases, but I don't really know

@kent_c and @reinerotto Any chance you can share your diffconfig, pls?

hello

i just see that topic
very big TY for the author btw
it's among the very very rare topics whom doesn't typically/usually too much tell us "change device" "throw your 4MB within the bin" and instead, of helping or accompany to provide 4MB images.

the only thing is, where i see several topics in the "community builds" : why not share images by upload them on addded links on the topic, or even add the link on the wiki page?
it would permit, seeing that they are about ~dozen or ~hundred (maybe more!!) community-made images for lot of specific devices, that they would be added to the wiki's specific page for each device?

where i see the few devices i have on the wiki, i only see the download/archive.openwrt.org, never customized or community-based images... it would be highly better convenient !

ty vm :slight_smile: