Hi all,
As a lot of you may have old devices left in cupboards/drawers, I decided to share some tips to help you build a recent version of OpenWRT on hardware with limited specifications. I know there is a wiki page on 4MB devices, but what I will describe on this page is more dedicated to expert users. I am not responsible for any issue with you devices.
The goal is to explain you how to build a full-featured (for daily use at least) and recent image (including LuCI web interface). With the right optimizations, it is possible, to include almost a dozen of MB of software into a 4MB (squashfs is highly compressed).
First thing you need to know, is that the device-specific images provided by OpenWRT are actually generic images with specific header/packages. They all includes a lot of unneeded code. The goal is not to produce a small image with very limited functionalities, but really to get rid of the βdeadβ code included in the firmware and not used by the hardware.
I have little time so if anyone wants to continue this work, create a wiki page and improve the explanations I would be very happy. Some optimizations could also be scripted (especially driver trimming for broadcom platforms).
[EASY]
1 - Files edition
1.0 - Package selection
That is always a good starting point, I will not go into detail for this one. You can also take some ideas from https://openwrt.org/faq/build_image_for_devices_with_only_4mb_flash.
1.1 β Remove comments from config files
By placing a files
folder at the root of your OpenWRT sources, you can replace files in the image. Easiest gains are default config files, as they usually contain a lot of comments. There is no impact on daily use as you will configure firewall after the first boot anyway. Note that these files are also copied on the config partition, so it is a double gain (especially is you are following instructions from the section 4). Easiest gains are to simplify firewall and uhttpd default configuration files.
etc/config/firewall
config defaults
option syn_flood 1
option input ACCEPT
option output ACCEPT
option forward REJECT
config zone
option name lan
list network 'lan'
option input ACCEPT
option output ACCEPT
option forward ACCEPT
config include
option path /etc/firewall.user
etc/config/uhttpd
config uhttpd main
list listen_http 0.0.0.0:80
list listen_http [::]:80
list listen_https 0.0.0.0:443
list listen_https [::]:443
option redirect_https 1
option home /www
option rfc1918_filter 1
option max_requests 3
option max_connections 100
option cert /etc/uhttpd.crt
option key /etc/uhttpd.key
option cgi_prefix /cgi-bin
list lua_prefix "/cgi-bin/luci=/usr/lib/lua/luci/sgi/uhttpd.lua"
option script_timeout 60
option network_timeout 30
option http_keepalive 20
option tcp_keepalive 1
option days 730
option key_type rsa
option bits 2048
option ec_curve P-256
option country ZZ
option state Somewhere
option location Unknown
option commonname 'OpenWrt'
Other files can be modified to remove unneeded function (IPv6 support, switch/VLAN support) depending on your needs.
1.2 β Removal of upgrade through firmware
99% of the routers can be flash through recovery tools from manufacturer (e.g. nmrpflash for Netgear) or TFTP, therefore sysupgrade process is most of the time a nice-to-have. Considering you need to compile your firmware manually; it is unlikely you will update it very often. Therefore, you can remove the firmware upgrade functions of sysupgrade. You need to partially keep sysupgrade to ensure you can still export and import configuration backup.
The minimum functions for that are:
- list_conffiles
- list_changed_conffiles
- add_conffiles
- do_save_conffiles
You also need to keep CONF_RESTORE and CONF_BACKUP handlers, along with -r and -b options.
Your /lib/upgrade/common.sh
file can also be trimmed, by keeping only the following functions:
- libs
- run_hooks
- ask_bool
- v
Once your sysupgrade is trimmed, you can add the following lines under the prepare_rootfs section of your rootfs.mk
.
rm -f $(1)/lib/upgrade/do_stage2
rm -f $(1)/lib/upgrade/fwtool.sh
rm -f $(1)/lib/upgrade/luci-add-conffiles.sh
rm -f $(1)/lib/upgrade/platform.sh
rm -f $(1)/lib/upgrade/stage2
rm -f $(1)/etc/sysupgrade.conf
rm -f $(1)/usr/libexec/validate_firmware_image
rm -f $(1)/sbin/pkg_check
rm -f $(1)/sbin/upgraded
rm -f $(1)/usr/bin/fwtool
At the end, you will have no impact on the daily use of your device, and you can still perform firmware updates using manufacturer tools.
[MEDIUM]
2 - Selection of drivers
In most devices, you will have the following elements a chipset, a flash memory, one or more network PHY, and some buttons/leds.
Default OpenWRT images are not fully hardware-specific, even if each device uses a specific flash file. Kernel configurations are shared among a wide range of devices for each platform. Consequently, the default kernel configuration contains a lot of unrequired drivers.
My advice is to first install a build without UI and with limited package. From there look at dmesg output and identify the drivers being loaded. You can also explore /sys/ to gain some knowledge on your system.
Once drivers are properly identified, it is time to configure the Device Drivers section, using make kernel_menuconfig command. Why is that important? Because the kernel and the root filesystem are compressed independently, and small gains in the kernel are usually more significant than small gains in the root filesystem. Kernel is also decompressed in RAM, so every byte gained here is RAM savings.
Below, an example of an (almost) minimal kernel driver list for WN3000RP (BCM47xx platform).
Storage
Memory Technology Device (MTD) support
- BCM47xx partitioning support
- Partition parsers > Parser for TRX format partition
- Caching block device access to MTD devices
- RAM/ROM/Flash chip drivers > Detect flash chips by CFI probe
- RAM/ROM/Flash chip drivers > Support for CFI command 002
- Self-contained MTD device drivers > Support for serial flash on BCMA bus
ChipCommon-attached serial flash support
Ethernet
Network device support
- Network core driver support
- Ethernet driver support > Broadcom iProc GBIT BCMA support
- MDIO bus device-drivers
- PHY Device support and infrastructure > Switch configuration API
- PHY Device support and infrastructure > Broadcom bcm53xx managed switch support (MDIO only)
- MDIO Bus/PHY emulation with fixed speed/link PHYs
GPIO (necessary for buttons, led)
GPIO Support > /sys/class/gpio
BCMA GPIO driver
LED Support > LED Class Support (+ some LED triggers if you want)
Watchdog (else device will autoreboot every X seconds)
Watchdog Timer Support
- WatchDog Timer Driver Core
- Update boot-enabled watchdog until userspace takes overs
- Broadcom BCM47xx Watchdog Timer
Chipset
Broadcom specific AMBA (SoC support + MIPS core driver)
[EXPERT]
3 - Driver trimming
Linux kernel is in fact developed to cover thousands of devices. One driver can cover dozens of devices. Some drivers are split in configurable parts through kernel config, but most are delivered as a single package.
This means that, when you are loading b53 driver (for example), you are actually loading a driver able to handle more than 10 different devices / hardware revisions. But⦠your device only contains one specific hardware revision! It means that a significant amount of space is lost to store useless functions. No optimization is done at compile time, as the conditionals checks from most drivers depends on the devices detected on the bus at runtime.
Here, we want to develop a truly device-specific OpenWrt image, so the idea is to get rid of all the codes which is not related to our specific embedded hardware. Optimization is a never-ending work; you should only focus on easy optimizations on large files. In the table below, an example for WN3000RP device.
Board
arch/mips/bcm47xx
board.c, buttons.c, leds.c, setup.c, time.c
arch/include/asm/mach-bcm47xx/
bcm47xx_board.h
Chipset
drivers/bcma
core.c, driver_chipcommon.c, driver_chipcommon_pmu.c, driver_chipcommon_sflash.c, driver_gpio.c, driver_mips.c, main.c, scan.c, sprom.c
drivers/firmware/broadcom
bcm47xx_nvram.c, bcm47xx_sprom.c
Ethernet
drivers/net/ethernet/broadcom
bgmac-bcma.c
Ethernet PHY
drivers/net/phy/b53
b53_common.c, b53_mdio.c, b53_priv.h
You may wonder what needs to be changed. My advice is to focus on the case statements. Typically, your device only has one chip ID. In the example below, if your device is BCM53573, there is no reason to keep code BCM57357 / BCM5372.
4 - Flash optimization
4.1 β Reclaiming unused space
Some vendor uses a partition between the rootfs and the nvram partition (detected as POT partition). In some case, you can reclaim some blocks from this partition for rootfs-data but you need to be very careful.
Three things should you NEVER erase:
- The CFE
- The NVRAM (used by the CFE)
- Magic number, if any
Taking the example of WN3000RP, the POT partition contains one empty block, and the next block starts with a magic number. It is possible to trick the POT partition detection to reclaim the free block before the magic number. Once you have done that you should always use firmware compiled with the same trick, as installing another firmware will erase the magic number, hence making the firmware non-flashable through CFE. If you do that by mistake and you do not have mtd installed in your firmware your only solution to flash is unsoldering the NAND/NOR chip and using programmer.
However, if you are careful enough, this is an easy way to reclaim a free block from your flash memory.
Example for BCM47xx platform:
In drivers/mtd/bcm47xxpart.c
, replace the POT header by the start of the magic number from the next block.
4.2 β Optimizing configuration partition
Note that this section is mainly useful if your flash memory uses 64KB blocks (which is the most common). If you have 4KB blocks, there is no reason to move away from JFFS2 (you are not losing space).
Also note that NAND have limited erase cycles. The smaller the configuration partition is, the more wear to the flash memory. You wonβt do config update every day so you are unlikely to encounter issues (most default firmware use NVRAM config on a single block anyway), but keep that in mind. For NOR no issue is expected under normal use.
4.2.1 Easy β JFFS2 reserved block reduction - From 256KB to 192KB
As best practice from point 1, you should embed all the packages you need at compile time and avoid the use of opkg. Without a package manager, there is no need for a large storage partition. However, you will notice that, by default OpenWRT forces you to use 256KB of the flash memory for the configuration partition. It is more than 6% of the space that you cannot use fully. The reason behind this is that JFFS2 uses garbage collection and needs free blocks for that. This is not a limitation specific to OpenWRT, JFFS2 needs 4 blocks by default.
The good news is that JFFS2 can work with 3 blocks, so you can easily reduce the partition size to 192KB. You will simply need to ensure you use less than 64KB of data (which is usually more than enough for configuration, even with VPN certs).
How to do that:
In drivers/fs/jffs2/fs.c
, update the minimum number of sectors from <5 to <4.
In addition to that, edit include/image.mk and remove the 256k mark.
We could add a 192k mark, but there is no added value, and padjffs2 only handle 2^n parameters, so it would be additional rework. JFFS2 partition will expand to 192KB on the first boot anyway. The only caveat is that the flash file will be 64KB smaller than the actual size needed, so take that into account when building your image. If you forget, you will easily notice it, it will fail to create the JFFS2 partition on boot and it will not be possible to save config.
4.2.2 Advanced β Moving away from JFFS2 - From 256KB to 64KB
The previous fix is simple to implement but you are still losing 128KB for the filesystem. The most efficient fix is to go for a non-filesystem-based configuration management.
One option is to slightly rework the snapshot-tool package, to ensure every configuration update is written over the same block. Basically, you are not using snapshots, but you benefit from the fact it is already developed to be compatible with overlay.
[TIME-CONSUMING]
5. LuCI trimming
LuCi contains code to covers all the potential situation. It means that the base module contains code to handle Wireless, Switch, VLAN, etc.
If your router does not have switch configuration or VLAN, you may want to remove the related code in LuCI. This is time-consuming, but it is really worth the size gain.
The main drawback is that, unlike drivers, LuCI undergoes a lot of development so you have to redo the work for each release, which takes time.