Boot custom openwrt firmware from extroot

Installing and Using OpenWrt
1

Hello everyone. I have a TP-Link TL-MR3420 v2 with a 4MB flash and 32MB DDR1 RAM running OpenWRT 17.01.5 with extroot support enabled. I have been using openwrt for the past couple of months and I'm really grateful that such a community exists on providing free and open source software on proprietary hardware.

I'm looking into building a custom openwrt firmware from the openwrt sdk and flash it onto the router. As the router only has 4MB of space on its flash and the custom firmware wouldn't fit in the flash, is it possible to boot the router from an external card containing the custom firmware?

I have an 8gb sd card formatted with ext4 connected to the usb port of the router. I want to add support for glibc to the firmware as musl is kind of limited in terms of running programs such as php and nodejs.

2

if your bootloader supports it... or you hardcode the cmdline in the dts, and a new kernel with usb/mmc/scsi/fs drivers will fit within the allowed space on the device...

chances are this will either not be feasable or way too much messing around to be worth it....

but, you could use imagebuilder/files and include block-mount / config-fstab so that the external disk gets mounted at boot... with the caveat that when you flash the device, you also have to flash(copy rootfs to) the external disk...

1 Like
3

The last option sounds more interesting though. Would like to know that how can the router boot to the external card with the firmware residing on the card itself

4

this is decided by the bootloader, does your bootloader support that?

5

I'm not sure though. Would the dmesg log indicate that it supports the option?

Screenshot_20200926-115512
Screenshot_20200926-1155121280Ă—720 110 KB

[    0.000000] Linux version 4.4.140 (buildbot@builds-02.infra.lede-project.org) (gcc version 5.4.0 (LEDE GCC 5.4.0 r3101-bce140e) ) #0 Fri Jul 13 19:25:14 2018
[    0.000000] MyLoader: sysp=e72c40a6, boardp=3050a00a, parts=15d661ab
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001974c (MIPS 74Kc)
[    0.000000] SoC: Atheros AR9341 rev 3
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 02000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] No valid device tree found, continuing without
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000001ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000001ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000001ffffff]
[    0.000000] On node 0 totalpages: 8192
[    0.000000] free_area_init_node: node 0, pgdat 803de4b0, node_mem_map 81000000
[    0.000000]   Normal zone: 64 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 8192 pages, LIFO batch:0
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8128
[    0.000000] Kernel command line:  board=TL-MR3420-v2  console=ttyS0,115200 rootfstype=squashfs,jffs2 noinitrd
[    0.000000] PID hash table entries: 128 (order: -3, 512 bytes)
[    0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Memory: 27800K/32768K available (3122K kernel code, 166K rwdata, 416K rodata, 320K init, 205K bss, 4968K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:51
[    0.000000] Clocks: CPU:535.000MHz, DDR:400.000MHz, AHB:200.000MHz, Ref:25.000MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7144898866 ns
[    0.000011] sched_clock: 32 bits at 267MHz, resolution 3ns, wraps every 8027976190ns
[    0.008236] Calibrating delay loop... 266.64 BogoMIPS (lpj=1333248)
[    0.091054] pid_max: default: 32768 minimum: 301
[    0.096067] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.103058] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.113269] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.123671] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.131394] NET: Registered protocol family 16
[    0.137625] MIPS: machine is TP-LINK TL-MR3420 v2
[    0.582594] Can't analyze schedule() prologue at 800670fc
[    0.600100] clocksource: Switched to clocksource MIPS
[    0.606686] NET: Registered protocol family 2
[    0.612395] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.619762] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.626535] TCP: Hash tables configured (established 1024 bind 1024)
[    0.633351] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.639531] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.646478] NET: Registered protocol family 1
[    0.651153] PCI: CLS 0 bytes, default 32
[    0.656257] Crashlog allocated RAM at address 0x1f00000
[    0.679185] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.685401] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.698613] io scheduler noop registered
[    0.702811] io scheduler deadline registered (default)
[    0.708463] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.718326] console [ttyS0] disabled
[    0.742234] serial8250.0: ttyS0 at MMIO 0x18020000 (irq = 11, base_baud = 1562500) is a 16550A
[    0.751345] console [ttyS0] enabled
[    0.758892] bootconsole [early0] disabled
[    0.773371] m25p80 spi0.0: found s25sl032p, expected m25p80
[    0.779152] m25p80 spi0.0: s25sl032p (4096 Kbytes)
[    0.785466] 5 tp-link partitions found on MTD device spi0.0
[    0.791284] Creating 5 MTD partitions on "spi0.0":
[    0.796240] 0x000000000000-0x000000020000 : "u-boot"
[    0.803031] 0x000000020000-0x000000156d5c : "kernel"
[    0.810005] 0x000000156d5c-0x0000003f0000 : "rootfs"
[    0.817068] mtd: device 2 (rootfs) set to be root filesystem
[    0.823032] 1 squashfs-split partitions found on MTD device rootfs
[    0.829429] 0x0000003a0000-0x0000003f0000 : "rootfs_data"
[    0.836905] 0x0000003f0000-0x000000400000 : "art"
[    0.843749] 0x000000020000-0x0000003f0000 : "firmware"
[    0.868912] libphy: ag71xx_mdio: probed
[    1.462196] ag71xx ag71xx.0: connected to PHY at ag71xx-mdio.1:00 [uid=004dd042, driver=Generic PHY]
[    1.472452] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode:MII
[    2.061981] ag71xx-mdio.1: Found an AR934X built-in switch
[    2.104338] eth1: Atheros AG71xx at 0xba000000, irq 5, mode:GMII
[    2.113563] NET: Registered protocol family 10
[    2.123702] NET: Registered protocol family 17
[    2.128377] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    2.141589] 8021q: 802.1Q VLAN Support v1.8
[    2.147723] hctosys: unable to open rtc device (rtc0)
[    2.157841] VFS: Mounted root (squashfs filesystem) readonly on device 31:2.
[    2.167280] Freeing unused kernel memory: 320K
[    3.286063] init: Console is alive
[    3.289820] init: - watchdog -
[    4.620815] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    4.722935] usbcore: registered new interface driver usbfs
[    4.728732] usbcore: registered new interface driver hub
[    4.734391] usbcore: registered new device driver usb
[    4.784766] SCSI subsystem initialized
[    4.796093] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.804668] ehci-platform: EHCI generic platform driver
[    4.810284] ehci-platform ehci-platform: EHCI Host Controller
[    4.816274] ehci-platform ehci-platform: new USB bus registered, assigned bus number 1
[    4.826586] ehci-platform ehci-platform: TX-TX IDP fix enabled
[    4.832651] ehci-platform ehci-platform: irq 3, io mem 0x1b000000
[    4.850148] ehci-platform ehci-platform: USB 2.0 started, EHCI 1.00
[    4.857654] hub 1-0:1.0: USB hub found
[    4.862048] hub 1-0:1.0: 1 port detected
[    4.869393] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.877518] ohci-platform: OHCI generic platform driver
[    4.885755] uhci_hcd: USB Universal Host Controller Interface driver
[    4.897414] usbcore: registered new interface driver usb-storage
[    4.904404] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    4.914355] init: - preinit -
[    5.470254] usb 1-1: new high-speed USB device number 2 using ehci-platform
[    5.625767] random: procd: uninitialized urandom read (4 bytes read, 9 bits of entropy available)
[    5.636473] usb-storage 1-1:1.0: USB Mass Storage device detected
[    5.645229] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[    5.653388] scsi host0: usb-storage 1-1:1.0
[    6.652151] scsi 0:0:0:0: Direct-Access     Generic  STORAGE DEVICE   1404 PQ: 0 ANSI: 6
[    6.855806] sd 0:0:0:0: [sda] 15564800 512-byte logical blocks: (7.97 GB/7.42 GiB)
[    6.865171] sd 0:0:0:0: [sda] Write Protect is off
[    6.870204] sd 0:0:0:0: [sda] Mode Sense: 21 00 00 00
[    6.871677] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    6.889111]  sda: sda1 sda2
[    6.897918] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    7.808136] mount_root: loading kmods from internal overlay
[    7.836335] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[    7.845202] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
[    8.129454] jffs2: notice: (441) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[    8.145976] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[    9.483488] EXT4-fs (sda1): recovery complete
[    9.512823] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
[    9.525162] mount_root: switched to extroot
[    9.533836] urandom-seed: Seeding with /etc/urandom.seed
[    9.686162] procd: - early -
[    9.689314] procd: - watchdog -
[   10.285455] procd: - watchdog -
[   10.289042] procd: - ubus -
[   10.334109] random: ubusd: uninitialized urandom read (4 bytes read, 76 bits of entropy available)
[   10.355095] random: ubusd: uninitialized urandom read (4 bytes read, 77 bits of entropy available)
[   10.365424] random: ubusd: uninitialized urandom read (4 bytes read, 77 bits of entropy available)
[   10.374939] random: ubusd: uninitialized urandom read (4 bytes read, 77 bits of entropy available)
[   10.385368] random: ubusd: uninitialized urandom read (4 bytes read, 77 bits of entropy available)
[   10.394905] random: ubusd: uninitialized urandom read (4 bytes read, 77 bits of entropy available)
[   10.405017] random: ubusd: uninitialized urandom read (4 bytes read, 77 bits of entropy available)
[   10.417973] random: ubusd: uninitialized urandom read (4 bytes read, 78 bits of entropy available)
[   10.428043] Adding 524284k swap on /dev/sda2.  Priority:-1 extents:1 across:524284k 
[   10.436597] procd: - init -
[   11.836129] kmodloader: loading kernel modules from /etc/modules.d/*
[   11.892488] exFAT: Version 1.2.9
[   11.920250] ntfs: driver 2.1.32 [Flags: R/O MODULE].
[   11.975822] fuse init (API version 7.23)
[   11.995217] Loading modules backported from Linux version wt-2017-01-31-0-ge882dff19e7f
[   12.003580] Backport generated by backports.git backports-20160324-13-g24da7d3c
[   12.019126] nf_conntrack version 0.5.0 (439 buckets, 1756 max)
[   12.051082] wireguard: WireGuard 0.0.20180519 loaded. See www.wireguard.com for information.
[   12.059809] wireguard: Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[   12.092373] xt_time: kernel timezone is -0000
[   12.126705] ip_tables: (C) 2000-2006 Netfilter Core Team
[   12.243983] ath: EEPROM regdomain: 0x0
[   12.244008] ath: EEPROM indicates default country code should be used
[   12.244018] ath: doing EEPROM country->regdmn map search
[   12.244042] ath: country maps to regdmn code: 0x3a
[   12.244054] ath: Country alpha2 being used: US
[   12.244064] ath: Regpair used: 0x3a
[   12.257914] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   12.264898] ieee80211 phy0: Atheros AR9340 Rev:3 mem=0xb8100000, irq=47
[   12.320567] kmodloader: done loading kernel modules from /etc/modules.d/*
[   12.342813] random: nonblocking pool is initialized
[   46.686773] device eth1 entered promiscuous mode
[   46.725758] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   46.833953] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   52.832367] ath: EEPROM regdomain: 0x81ca
[   52.836560] ath: EEPROM indicates we should expect a country code
[   52.842903] ath: doing EEPROM country->regdmn map search
[   52.848392] ath: country maps to regdmn code: 0x5d
[   52.853354] ath: Country alpha2 being used: MY
[   52.857940] ath: Regpair used: 0x5d
[   52.861559] ath: regdomain 0x81ca dynamically updated by user
[   61.310398] IPv6: ADDRCONF(NETDEV_UP): TP-LINK_659534: link is not ready
[   61.396470] device TP-LINK_659534 entered promiscuous mode
[   67.239255] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   70.394615] wlan0: authenticate with d8:5b:2a:95:ed:9b
[   70.417680] wlan0: send auth to d8:5b:2a:95:ed:9b (try 1/3)
[   70.433811] wlan0: authenticated
[   70.510234] wlan0: associate with d8:5b:2a:95:ed:9b (try 1/3)
[   70.529159] wlan0: RX AssocResp from d8:5b:2a:95:ed:9b (capab=0x411 status=0 aid=2)
[   70.537437] wlan0: associated
[   70.540718] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   70.625434] wlan0: Limiting TX power to 20 (20 - 0) dBm as advertised by d8:5b:2a:95:ed:9b
[   71.924100] IPv6: ADDRCONF(NETDEV_CHANGE): TP-LINK_659534: link becomes ready
[   71.931745] br-lan: port 2(TP-LINK_659534) entered forwarding state
[   71.938273] br-lan: port 2(TP-LINK_659534) entered forwarding state
[   71.995703] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   73.930110] br-lan: port 2(TP-LINK_659534) entered forwarding state
6

You are looking for trouble.
The hard limiting factor is the 32MB RAM. It will severely impact more complicated programs.
See

4 Likes
7

Okay, so I went away for a while and now I had set up uboot for this router. These are the commands available from the uboot console. Any ideas on how to set up uboot to boot off the custom firmware?

U-Boot 1.1.4  (Nov 19 2014, 09:48:39)

uboot> ?
?
?          - alias for 'help'
boot       - boot default, i.e., run 'bootcmd'
bootd      - boot default, i.e., run 'bootcmd'
bootm      - boot application image from memory
cp         - memory copy
dhcp       - invoke DHCP client to obtain IP/boot params
echo       - echo args to console
erase      - erase FLASH memory
exit       - exit script
go         - start application at address 'addr'
help       - print embedded help
httpd      - start www server for firmware recovery
itest      - return true/false on integer compare
md         - memory display
mm         - memory modify (auto-incrementing)
mtest      - RAM test
mw         - memory write (fill)
nm         - memory modify (constant address)
ping       - send ICMP ECHO_REQUEST to network host
printenv   - print environment variables
printmac   - print MAC address stored in FLASH
printmodel - print router model stored in FLASH
printpin   - print WPS pin stored in FLASH
reset      - perform RESET of the CPU
run        - run commands in an environment variable
setenv     - set environment variables
setmac     - save new MAC address in FLASH
sntp       - send NTP request to NTP server
startnc    - start net console
startsc    - start serial console
test       - minimal test like /bin/sh
tftpboot   - boot image via network using TFTP protocol
version    - print U-Boot version

uboot>

Edit: here are the modifiable environment variables

uboot> printenv
printenv
bootargs=console=ttyS0,115200 root=31:02 rootfstype=squashfs init=/sbin/init mtdparts=ath-nor0:128k(u-boot),1024k(kernel),2816k(rootfs),64k(config),64k(art)
bootcmd=bootm 0x9F020000
bootdelay=1
baudrate=115200
ipaddr=192.168.1.1
serverip=192.168.1.2
bootfile="firmware.bin"
loadaddr=0x80800000
ncport=6666
uboot_addr=0x9F000000
uboot_name=uboot.bin
uboot_size=0x10000
uboot_upg=if ping $serverip; then tftp $loadaddr $uboot_name && if itest.l $filesize == $uboot_size; then erase $uboot_addr +$filesize && cp.b $loadaddr $uboot_addr $filesize && echo OK!; else echo ERROR! Wrong file size!; fi; else ERROR! Server not reachable!; fi
firmware_addr=0x9F020000
firmware_name=firmware.bin
firmware_upg=if ping $serverip; then tftp $loadaddr $firmware_name && erase $firmware_addr +$filesize && cp.b $loadaddr $firmware_addr $filesize && echo OK!; else ERROR! Server not reachable!; fi
stdin=nc
stdout=nc
stderr=nc
ethact=eth1

Environment size: 922 bytes

uboot>
8

that u-boot lacks the commands to boot from usb (not surprising, they always remove most unused functionality)

Your device is supported by a custom uboot from a known OpenWrt contributor here https://github.com/pepe2k/u-boot_mod
It's an ancient u-boot source so I don't know exactly where/how it defines what commands are available, but it seems they are here https://github.com/pepe2k/u-boot_mod/tree/master/u-boot/common
so you would probably need to edit that source to add the command to boot from USB, and then flash the new u-boot, hope it does not brick the device, and then you can edit the bootcommand line to load the kernel from USB.

I personally think this is WAY too much effort to save a device that is not going to be able to run latest OpenWrt anyway or any custom application (especially with Glibc) because it has 32MB of RAM and it's already near that limit with the 17.XX release.

If your end goal is running stuff in PHP and Node I'd suggest to just retire that device and get a minirouter from GL.Inet. https://www.gl-inet.com/

It's like 20 euro/bucks on Amazon (for the yellow ones, the black ones are better but cost a bit more, the whites are using older hardware and have worse specs), it is the same size of your current minirouter, it has way more space on flash and more RAM (even on the yellow one you got 128MB RAM, which means you have more than 64MB free for your custom applications), they break out unused GPIOs on their board into holes you can solder a header on, and it is supported by OpenWrt, it's even its stock firmware.
See for example the spec sheet of the yellow "Mango" ones https://docs.gl-inet.com/en/3/hardware/mt300n-v2/

3 Likes
9

Alright, thanks for the detailed explanation. I really appreciate it. I understand that it is not really powerful enough to run php and such but i guess it isn't worth it to go through all of the hassle of editing the source code of uboot and hope that it doesn't brick the device which ironically uboot is created to prevent a router from ever being bricked.

10

That's because when you flash a firmware you are not touching the u-boot, but only the firmware partitions. So no matter what you do, you always have u-boot, and you can flash a new firmware.

The device CPU loads uboot, then uboot loads the firmware.

If you write a broken uboot that cannot load a firmware (and respond on serial console), no firmware will be loaded, and the CPU is very dumb and has no interface to send a new uboot.

To recover from this, you need to connect to the flash chip with a SPI flashing tool and write the uboot manually in the flash chip while the device is off. Or more complex tools if it is using NAND flash chips, like JTAG.

Some devices with Marvell CPUs are smart and can be recovered even if u-boot is deleted, since the CPU checks serial console for the commands of a recovery tool running on your PC, and will load the u-boot from serial (very slow) from your PC and let you recover even if you erased all the device flash.
Your device does not have this feature.

3 Likes
11

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.