OpenWrt Forum Archive

Topic: Asus WL500gP V2 working with kernel and rootfs loaded from USB disk!

The content of this topic has been archived on 6 Apr 2015. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

For quite a long time I was trying to get the USB flash disk based environment
for development of drivers and applications for Asus WL500gP v2.
My main goal was to avoid reflashing after each recompilation of the kernel.
I also wanted to have the root filesystem on a media big enough to
put e.g. full Python and other development tools on it.

Just yesterday I've managed to obtain the working environment, and today I have
polished it sufficiently to give you quite short description.

In my approach the system works with the kernel loaded from the USB drive, so when
you change the kernel, you only have to change the contents of the USB drive.
The flashed in system is just the "kexec based bootloader" (I think,
that it could be nice to add possibility to block booting from the USB drive,
e.g. when one of buttons is pressed. Then you could modify the contents
of the USB drive from the "bootloader", as it is also fully functional OpenWRT system.
I'll try to add this functionality later)
.

Of course this result was possible only due to hard work of all OpenWRT developers,
so many thanks to them all for bringing the trunk version to the state,
where the above was possible.

1. Preparation of the flashed in, kexec enabled system

I start from the fresh OpenWRT downloaded with the below commands:

svn co svn://svn.openwrt.org/openwrt/trunk/
cd trunk
./scripts/feeds update
make V=99 defconfig
make V=99 package/symlinks

Then I execute the "make menuconfig" and select the following options:
a) In "Kernel modules/Filesystems" I select: nfs, e2fs, e3fs, vfat
   and code-pages: cp437, iso8859-1, utf-8
b) In "Kernel modules/Device drivers/USB" i select: usb-core, usb-storage, usb-ohci, usb2
In the "Utilities" I select "kexec-tools" package.

All the above selections have to be done as "*", not as "M".
As the type of the image I select "squashfs".

Then I run the "make kernel_menuconfig" and in "Kernel type" I select "kexec system call".

Finally I run "make V=99 world".

After compilation I have entered the
trunk/build_dir/linux-brcm47xx/linux-2.6.32.9/arch/mips/kernel
directory and modified the file machine_kexec.c

the 55th line I have changed from:     kexec_start_address = image->start;
into: kexec_start_address = (unsigned long) phys_to_virt(image->start);

Then I have ran "make V=99" in the trunk directory and get the ready to flash image
in trunk/bin...

After flashing and rebooting i waited until the jffs2 system is populated
after first reboot. (In the serial console the following messages appeared:
jffs2_scan_eraseblock(): End of filesystem marker found at 0x10000             
jffs2_build_filesystem(): unlocking the mtd device... done.                     
jffs2_build_filesystem(): erasing all blocks after the end marker... done.     
switching to jffs2                                                             
mini_fo: using base directory: /                                               
mini_fo: using storage directory: /jffs                                         
- config restore -     )

Then I had to modify the system, so it can automatically boot the files from the USB disk.

I have added that functionality to the /etc/rc.local script.
After modification it looks as follows

# Put your custom commands here that should be executed once                    
# the system init finished. By default this file does nothing.                  
mkdir /new                                                                      
until [ -e /dev/sda1 ]; do                                                      
  sleep 2                                                                       
done                                                                            
mount /dev/sda1 /new                                                            
if [ -f /new/linux.elf ]; then                                                  
  kexec -l /new/linux.elf                                                       
  kexec -e                                                                      
fi                                                                              
exit 0

2. Preparation of the USB bootable system
I start from the fresh OpenWRT downloaded with the below commands:

svn co svn://svn.openwrt.org/openwrt/trunk/
cd trunk
./scripts/feeds update
make V=99 defconfig
make V=99 package/symlinks

Then I execute the "make menuconfig" and select the following options:
a) As the image type I select "ramfs"
b) In "Kernel modules/Filesystems" I select: e2fs, e3fs
c) In "Kernel modules/Device drivers/USB" i select: usb-core, usb-storage, usb-ohci, usb2
Then I run the "make V=99 world" command.
After compilation is finished I modify the trunk/package/base-files/files/etc/preinit, adding
the line:
'rootfs=/dev/sda1'
right after line '. /etc/diag.sh'

You can also modify the file trunk/target/linux/generic-2.6/base-files/init
line 47: from:  'mount $rootfs /mnt' to: 'mount $rootfs /mnt -o noatime',
to make sure, that the access time of the read files is not stored on the USB drive
(however the standard "relatime" option also reasonably limits wear of the USB flash disk).

After that I run "make V=99"

Finally I get file trunk/bin/brcm47xx/openwrt-brcm47xx-vmlinux.elf , which I copy
as linux.elf to the root directory of the USB disk
and directory trunk/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx, which I copy
(with "cp -a") to the USB disk.

3. Results
The environment prepared in that way works quite reliably, however sometimes
I get strange problems. One of these is presented below.
When the kernel is booted from USB, sometimes right after unused kernel memory is freed,
the error occurs:

Freeing unused kernel memory: 2596k freed                                       
Instruction bus error, epc == 80004d2c, ra == 80000018                          
Oops[#1]:                                                                       
Cpu 0                                                                           
$ 0   : 00000000 10008800 80000000 00000000                                     
$ 4   : 7f8549ec 80cc5e00 0000002d 80cb7e50                                     
$ 8   : 40646461 646f6d2f 2f656c75 67616964                                     
$12   : 54434100 0000004d 0000004d 3d4e4f49                                     
$16   : 0000004d 0000004d 00000a00 0000004d                                     
$20   : 00000000 80cb7d60 80cc4cc0 80cb7e50                                     
$24   : 0000000d 80cc5e00                                                       
$28   : 80cb6000 80cb7cb0 7f8549ec 80000018                                     
Hi    : 00000000                                                                
Lo    : 9abbd100                                                                
epc   : 80004d2c 0x80004d2c                                                     
    Not tainted                                                                 
ra    : 80000018 0x80000018                                                     
Status: 10008803    KERNEL EXL IE                                               
Cause : 00804018                                                                
PrId  : 00029029 (Broadcom BCM3302)                                             
Modules linked in: diag(+)                                                      
Process hotplug2 (pid: 157, threadinfo=80cb6000, task=81c5a9d8, tls=00000000)   
Stack : 00000a00 8019d094 00000000 80cb7d60 81804560 00418280 80cc4cc0 80c9d400 
        00000a00 80cb7e58 00000000 80cb7d60 81804560 0000004d 7f8549ec 801c2dd8 
        00000000 80001444 007fffff 00000000 ffffffff 80063b34 000200da 000200da 
        00000000 fffffffe 001fe154 80cb87f8 80c9f000 80cb7da0 80cb7e74 00000a00 
        7f8549ec 00000000 7f8548b8 00410000 00418280 801903d0 fffffffc 00410000 
        ...                                                                     
Call Trace:[<8019d094>] 0x8019d094                                              
[<801c2dd8>] 0x801c2dd8                                                         
[<80001444>] 0x80001444                                                         
[<80063b34>] 0x80063b34                                                         
[<801903d0>] 0x801903d0                                                         
[<8007906c>] 0x8007906c                                                         
[<8001aca8>] 0x8001aca8                                                         
[<80079104>] 0x80079104                                                         
[<8003d814>] 0x8003d814                                                         
[<8007b45c>] 0x8007b45c                                                         
[<80190674>] 0x80190674                                                         
[<800325a0>] 0x800325a0                                                         
[<80032668>] 0x80032668                                                         
[<8000aefc>] 0x8000aefc                                                         
[<801906ec>] 0x801906ec                                                         
[<80003230>] 0x80003230                                                         
                                                                                
                                                                                
Code: 24c6ffe0  8cac0010  8caf0014 <ac880000> ac890004  8ca80018  8ca9001c  24a 
Disabling lock debugging due to kernel taint                                    
diag: Detected 'ASUS WL-500g Premium V2'                                        
Instruction bus error, epc == 80005050, ra == 80000018                          
Oops[#2]:                                                                       
Cpu 0                                                                           
$ 0   : 00000000 10008800 fffffff2 00000000                                     
$ 4   : 0047e5ef 00000000 00000a11 0047e5ef                                     
$ 8   : ffffffff 800050fc 00000a11 00000001                                     
$12   : 80cacc60 80279e24 80524bc8 00010000                                     
$16   : 0047e5ef 80cabf00 00000000 0047fb50                                     
$20   : 80c91980 80cffe48 0047e5ef 0047fb50                                     
$24   : 80516880 00000001                                                       
$28   : 80cfe000 80cffc28 00000008 80000018                                     
Hi    : 00000000                                                                
Lo    : a4bf4b00                                                                
epc   : 80005050 0x80005050                                                     
    Tainted: G      D                                                           
ra    : 80000018 0x80000018                                                     
Status: 10008803    KERNEL EXL IE                                               
Cause : 00804018                                                                
PrId  : 00029029 (Broadcom BCM3302)                                             
Modules linked in: diag                                                         
Process hotplug.failsaf (pid: 166, threadinfo=80cfe000, task=80cc69f8, tls=0000)
Stack : 80cabf00 800d4cf4 00000000 00000000 80c91980 00000080 00006012 00000000 
        00000000 00000001 ffffffff 81d938c0 00000000 00000000 00000000 00000000 
        80cc69f8 0047e5ef 0047e000 0046daa0 00400000 80cc69f8 80c91880 80ca4280 
        00000001 00000000 00400000 81c30da0 00000000 00002021 8027aac0 fffffff8 
        80c91980 8027a750 00000002 00010000 80516880 00000000 fffffff8 8009898c 
        ...                                                                     
Call Trace:[<800d4cf4>] 0x800d4cf4                                              
[<8009898c>] 0x8009898c                                                         
[<800d3fa4>] 0x800d3fa4                                                         
[<8007b720>] 0x8007b720                                                         
[<8007b720>] 0x8007b720                                                         
[<80098484>] 0x80098484                                                         
[<80098678>] 0x80098678                                                         
[<8009898c>] 0x8009898c                                                         
[<80099e2c>] 0x80099e2c                                                         
[<8009d3fc>] 0x8009d3fc                                                         
[<80012c30>] 0x80012c30                                                         
[<80003230>] 0x80003230                                                         
[<80044a6c>] 0x80044a6c                                                         
[<80037f44>] 0x80037f44                                                         
[<80012858>] 0x80012858                                                         
[<8000f7b4>] 0x8000f7b4                                                         
[<80037e28>] 0x80037e28                                                         
[<8000f7a4>] 0x8000f7a4                                                         
                                                                                
                                                                                
Code: 30880003  11000004  2508fffc <b8850000> 00882023  00c83021  34c9003f  392 
Instruction bus error, epc == 2ab5e080, ra == 80000018                          
Instruction bus error, epc == 80027a8c, ra == 80000018                          
Oops[#3]:                                                                       
Cpu 0                                                                           
$ 0   : 00000000 10008801 00000000 80000000                                     
$ 4   : 7fb752a0 00000000 00000000 0000000a                                     
$ 8   : 00000000 00000000 00000000 00000000                                     
$12   : 81c18d60 80281b14 81c13968 00000001                                     
$16   : 000000a5 80cc6570 00000000 00000000                                     
$20   : 81c13938 81c15ee8 20000000 81c15f00                                     
$24   : 00000008 800189a4                                                       
$28   : 81c14000 81c15e60 fffffff6 80000018                                     
Hi    : 00000000                                                                
Lo    : ae85bc00                                                                
epc   : 80027a8c 0x80027a8c                                                     
    Tainted: G      D                                                           
ra    : 80000018 0x80000018                                                     
Status: 10008803    KERNEL EXL IE                                               
Cause : 00804018                                                                
PrId  : 00029029 (Broadcom BCM3302)                                             
Modules linked in: diag                                                         
Process init (pid: 1, threadinfo=81c14000, task=81c13938, tls=00000000)         
Stack : 80cacc60 8000aefc 80cacde0 80017bbc 00000000 00000000 80cc6570 00000000 
        81c13a48 81c15ee8 81c13938 ffffffff 20000000 81c15f00 fffffff6 80028410 
        00000012 00000000 00000000 80cc6570 00000000 00000000 7fb752a0 00000000 
        00000002 00000000 0000002d 00480008 ffffffff 8002860c 00000000 000081ed 
        00000001 00000000 00000003 00000004 00000000 00000000 7fb752a0 00000000 
        ...                                                                     
Call Trace:[<8000aefc>] 0x8000aefc                                              
[<80017bbc>] 0x80017bbc                                                         
[<80028410>] 0x80028410                                                         
[<8002860c>] 0x8002860c                                                         
[<80026424>] 0x80026424                                                         
[<80003230>] 0x80003230                                                         
                                                                                
                                                                                
Code: 00431024  1440021c  00000000 <ac870000> 14a0003f  8ea4000c  1080003e  248 
Kernel panic - not syncing: Attempted to kill init!

Anyway, the presented environment works sufficiently reliable to be useful.
I'll appreciate any corrections and improvements

(Last edited by wzab on 14 Mar 2010, 18:55)

You should use the block-extroot package and 'make package/symlinks' is obsolete.

(Last edited by Dogge on 14 Mar 2010, 19:12)

Does block-extroot allow me to use the kernel loaded from external disk?
The main goal was to be able to replace BOTH kernel and filesystem without reflashing.

Thank you for the guide. It inspired me to try the solution on mips based TL-WR1043ND (ar71xx architecture)

First of all I was not sure about "ramfs" option from point 2 of the guide. Do you mean ramdisk image? Yet you still take the openwrt-brcm47xx-vmlinux.elf kernel image only not the initramfs image. Initramfs elf image doesn't boot via kexec anyway. My conclusion would be that if the vmlinux.elf image gets built with default target images you can stick to it, am I correct?

So I was happy to see kexec running vmlinux.elf image the kernel however it halts with following error (I've changed kernel command line using make kernel_menuconfig set root to /dev/sdb2):

VFS: Cannot open root device "sdb1" or unknown-block(0,0)
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

It seems to me that it mustn't work since the usb/ext3 modules even when marked [*] are compiled as modules and we don't have root filesystem with access modules yet.
Still you have it working. I wonder I you could give your impressions what might went wrong.

mikiel, I think you're probably right that the "second" kernel needs USB support compiled in, not in module form. You should be able to do this from the same kernel_menuconfig stage that you mentioned in your post.

It's good to see this guide. I tried using kexec on a WL-700gE about a year ago with no luck, so it's encouraging to hear that it can be done.

- Angus

I'm using the "ramfs" option, which AFAIR generates the ELF image with initramfs included.
This image boots with kexec (modified as described in my first post).
Support for ext2fs, ext3fs and USB is provided by the modules located in the initial ramdisk.
However to have these modules included (not only compiled for later installation),
you have to select them as "*", and not as "M" after "make menuconfig".
To Mikiel
AFAIK the root filesystem on the USB may be mounted only after the USB system
is initialized, which is performed by files located in the initial ramdisk.
I've never managed to get the USB disk running without either initramfs or initrd.
Kexec on MIPS platform is not able to boot the kernel with initrd (as you are not able
to load the ramdisk file). The only option is to use the initramfs, which is compiled
into the kernel.
That's why I change the rootfs to /dev/sda1 in the /etc/preinit.

(Last edited by wzab on 29 Mar 2010, 18:15)

Thanks wzab for the reply. You might want to edit the initial post to change trunk/bin/brcm47xx/openwrt-brcm47xx-vmlinux.elf to trunk/bin/brcm47xx/openwrt-brcm47xx-vmlinux-initramfs.elf to make the procedure super clear.

initramfs finally booted! It must have been to large the previous time or something. Unfortunatelly the usb-storage obviously has a problem finding/listing the card reader with card. Instead I get hw watchdog induced (I suppose) reboot after several seconds.

In fact after compilation I get two files in bin:
openwrt-brcm47xx-initramfs.trx  and openwrt-brcm47xx-vmlinux.elf
There is no "initramfs" in the name of the ELF image.

The size of the initial image must be really small. I select most packages as "M", not as "*" to keep
the initramfs equipped ELF image small.
I install all needed packages after rebooting, when USB disk is mounted as the new root filesystem.

Rejoyce TL-WR1043nd users!
This procedure works for TP-LINK device as well. The modifications you must take into account are:
1. Add

board=TL-WR1043ND

to kernel command line. Use make kernel_menuconfig / Kernel hacking to do that.
2. Initramfs filename is openwrt-ar71xx-vmlinux-initramfs.elf.
3. Make sure you disabled watchdog before you kexec -e new kernel! Otherwise you'll get a reboot if the boot process takes too much time which is almost always the case. The easiest way is to kill the watchdog process.

Additionally I do clean umount of the usb device before running kexec -e so it's cleanly mounted when used as a /

wzab:
It looks the naming is not consistent across the platforms - sorry for nagging.

edit:

root@OpenWrt:/proc# lsmod |grep ath9k
ath9k                  75696  0
ath9k_common            5168  1 ath9k
ath9k_hw              202896  2 ath9k,ath9k_common
ath                     6992  3 ath9k,ath9k_common,ath9k_hw
mac80211              192224  2 ath9k,ath9k_common
cfg80211              116768  4 ath9k,ath9k_common,ath,mac80211
root@OpenWrt:/proc# rmmod ath9k
ath9k: Driver unloaded
root@OpenWrt:/proc# insmod ath9k
ath9k ath9k: failed to initialize device
ath9k: probe of ath9k failed with error -22
root@OpenWrt:/proc#

Anyone has an idea how to reset/initialize device properly in kexec'ed environment?

(Last edited by mikiel on 1 Apr 2010, 16:20)

I'm glad to hear that thing gets shaped up so nicely in the trunk.
Prospect of moving the whole system to external storage is very appealing especially that tp-wr1043nd flashing sometimes fails without a reason.
Maybe someone has an idea how to cope with the eeprom/mac issue the other platform has.
Please note the macs below, also eeprom area seems to be zeroed.

root@OpenWrt:/# dmesg |grep ath
ath: Bad EEPROM checksum 0x0 or revision 0x0000
ath: Unable to initialize hardware; initialization status: -22
ath9k ath9k: failed to initialize device
ath9k: probe of ath9k failed with error -22
root@OpenWrt:/# ifconfig
br-lan    Link encap:Ethernet  HWaddr 00:00:00:01:00:00
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:402 (402.0 B)

eth0      Link encap:Ethernet  HWaddr 00:00:00:01:00:00
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1134 (1.1 KiB)  TX bytes:2678 (2.6 KiB)
          Interrupt:4

eth0.1    Link encap:Ethernet  HWaddr 00:00:00:01:00:00
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:406 (406.0 B)

eth0.2    Link encap:Ethernet  HWaddr 00:00:00:01:00:00
          inet addr:192.168.1.229  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:814 (814.0 B)  TX bytes:1229 (1.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@OpenWrt:/#

Kernel expects the eeprom and macs to be in memory (mach-tl-wr1043nd.c):

static void __init tl_wr1043nd_setup(void)
{
        u8 *mac = (u8 *) KSEG1ADDR(0x1f01fc00);
        u8 *eeprom = (u8 *) KSEG1ADDR(0x1fff1000);

I've also downloaded GPL tarbal from TP-LINK site and there's a lot of board initialization stuff in the u-boot sources they use however I don't know how to incorporate it into kexec procedure or if it's even possible.

Maybe we have some atheros internals/ath9k specialist that could have a look and say if the thing is doable in the first place?

pqa wrote:

Many thanks for this. I have it working on a WL500GPv1; it's just what I have been trying to do for ages.

I have produced some patches to incorporate all the configuration into the build system - see https://lists.openwrt.org/pipermail/openwrt-devel/2010-April/006671.html for details.

whats next after applying the patch? i tryed flashing the nomal squashfs image with the added packeges enabled and the proconfigured configuration:

/etc/config/kexec:

config file   reboot
      option  enabled         1
      option  filename        openwrt-brcm47xx-vmlinux.elf
      option  params          "vga=386 p2=2"
      option  add_params      "vga=683 p1=1"

/etc/config/fstab

config mount
        option target   /overlay
        option device   /dev/sda1
        option fstype   ext2
        option options  rw,sync, noatime
        option enabled  1
        option enabled_fsck 0

then i copied everything from build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/ to the usb drive together with openwrt-brcm47xx-vmlinux.elf

after that i got

root@OpenWrt:/# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
none on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
root on /tmp/root type tmpfs (rw,relatime)
mini_fo:/tmp/root on / type mini_fo (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /overlay type ext2 (rw,sync,relatime,errors=continue)
/dev/sda2 on /mnt/sda2 type ext2 (rw,relatime,errors=continue)

whats wrong with it?

regards

mikiel wrote:

3. Make sure you disabled watchdog before you kexec -e new kernel! Otherwise you'll get a reboot if the boot process takes too much time which is almost always the case. The easiest way is to kill the watchdog process.

Or build the 'bootloader' without watchdog - it's in the busybox config when you do 'make menuconfig'.

I've got this working on a WL500gP V2 with backfire trunk following the instructions at the top of the thread. I did worry that as I've got two USB drives - a 250GB hdd and a 4GB flash stick - the root file system might not always be on /dev/sda1 but it seems to be working OK.

When I want to disable booting from the USB drive temporarily, I just rename /linux.elf before rebooting. I can then make changes to the 'bootloader', mount the USB drive and rename the image back to linux.elf, then run /etc/rc.local to boot to the USB drive.

Thanks.

I just needed to refresh the "bootloader" kernel in my Asus WL500gP V2, and I recompiled it according to the recipe formulated in my first post in this thread.
So now it compiled with the 3.2.9 kernel.
I've switched on the router keeping pressed the small black button "RESTORE", so it started waiting for TFTP server.
Then I started the TFTP server on my host: 

atftp --trace --option "timeout 1" --option "mode octet" --put --local-file openwrt-brcm47xx-squashfs.trx 192.168.1.1

The firmware got uploaded successfully and flashed correctly.
After I switched off and on the power, The system started correctly, so I mounted the USB disk and tried to kexec to new kernel:

root@OpenWrt:/# mkdir /tmp/new
root@OpenWrt:/# mount /dev/sda1 /tmp/new
root@OpenWrt:/# kexec -l /tmp/new/vmlinux.elf 
root@OpenWrt:/# kexec -e
[  137.660000] b44 ssb0:0: eth0: powering down PHY
[  137.668000] device eth0 left promiscuous mode
[  137.672000] br-lan: port 1(eth0.0) entering forwarding state
[  137.684000] Starting new kernel
[  137.688000] kexec_relocate_size = 5416
[  137.692000] reboot_code_buffer = 80f84000
[  137.696000] kexec_start_address(entry point of new kernel) = 802371a0
[  137.704000] kexec_indirection_page = 81a93000
[  137.708000] Copy kexec_relocate section from 80243808 to reboot_code_buffer: 80f84000
[  137.716000] kexec_args[0] (argc): 2151909152
[  137.720000] kexec_args[1] (argv):   (null)
[  137.724000] kexec_args[2] (env ): 804029fc
[  137.728000] kexec_args[3] (desc): 43464531
[  137.732000] kexec_argv[0] =   (null), (null)
[  137.740000] kexec_argv[1] =   (null), (null)
[  137.744000] kexec_argv[2] =   (null), (null)
[  137.748000] kexec_argv[3] =   (null), (null)
[  137.752000] kexec_argv[4] =   (null), (null)
[  137.756000] kexec_argv[5] =   (null), (null)
[  137.760000] kexec_argv[6] =   (null), (null)
[  137.764000] kexec_argv[7] =   (null), (null)
[...]
[  162.628000] kexec_argv[3523] =   (null), (null)
[  162.632000] kexec_argv[3524] =   (null), (null)
[  162.640000] CPU 0 Unable to handle kernel paging request at virtual address 74696e69, epc == 801273fc, ra == 8012886c
[  162.640000] Oops[#1]:
[  162.640000] Cpu 0
[  162.640000] $ 0   : 00000000 1000c800 74696e69 80129910
[  162.640000] $ 4   : 74696e69 ffffffff 74696e69 00000000
[  162.640000] $ 8   : 0000002d 0000002b 00000000 203d205d
[  162.640000] $12   : 00000000 00000000 00000005 00000000
[  162.640000] $16   : 74696e69 802d8bbc ffffffff 802d8f9c
[  162.640000] $20   : 00000000 00000002 80258a7a 00000400
[  162.640000] $24   : 00000002 8015c6e0                  
[  162.640000] $28   : 81afe000 81affc80 8024df4c 8012886c
[  162.640000] Hi    : 00000000
[  162.640000] Lo    : 00000000
[  162.640000] epc   : 801273fc 0x801273fc
[  162.640000]     Tainted: G           O
[  162.640000] ra    : 8012886c 0x8012886c
[  162.640000] Status: 1000c802    KERNEL EXL 
[  162.640000] Cause : 00800008
[  162.640000] BadVA : 74696e69
[  162.640000] PrId  : 00029029 (Broadcom BMIPS3300)
[  162.640000] Modules linked in: usb_storage ohci_hcd nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp ipt_MASQUERADE iptable_nat nf_nat xt_conntrack xt_CT xt_NOTRACK iptable_raw xt_state nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ehci_hcd sd_mod pppoe pppox ipt_REJECT xt_TCPMSS ipt_LOG xt_comment xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables nfs ppp_async ppp_generic slhc vfat fat lockd sunrpc ext4 jbd2 mbcache b43legacy(O) b43(O) nls_utf8 nls_iso8859_1 nls_cp437 mac80211(O) usbcore usb_common scsi_mod nls_base crc16 crc_ccitt cfg80211(O) compat(O) ssb_hcd bcma_hcd arc4 aes_generic crypto_algapi switch_robo(O) switch_core(O) diag(O)
[  162.640000] Process kexec (pid: 1264, threadinfo=81afe000, task=81b259d8, tls=77d92440)
[  162.640000] Stack : 00000008 ffffffff 802d8f9c 81affdc0 81affdc4 802d8bbc 802d8f9c 802d8b9c
[  162.640000]         80258a78 80129934 80258a79 00000000 00000005 00000000 ffffffff ffffffff
[  162.640000]         ffffffff ffffffff ff0a0004 ffffffff 00000400 80258a60 802d8b9c 81affdbc
[  162.640000]         80240000 802e0000 80240000 00000002 80290000 80129b80 1000c803 00000000
[  162.640000]         00000077 80290000 00000000 8001cf54 ffffffda 80290000 000000a2 0009a4c0
[  162.640000]         ...
[  162.640000] Call Trace:[<80129934>] 0x80129934
[  162.640000] [<80240000>] 0x80240000
[  162.640000] [<80240000>] 0x80240000
[  162.640000] [<80129b80>] 0x80129b80
[  162.640000] [<8001cf54>] 0x8001cf54
[  162.640000] [<8016302c>] 0x8016302c
[  162.640000] [<80240000>] 0x80240000
[  162.640000] [<80240000>] 0x80240000
[  162.640000] [<8023cb88>] 0x8023cb88
[  162.640000] [<8000f6f8>] 0x8000f6f8
[  162.640000] [<80051f68>] 0x80051f68
[  162.640000] [<8002e2e4>] 0x8002e2e4
[  162.640000] [<80208680>] 0x80208680
[  162.640000] [<8020a464>] 0x8020a464
[  162.640000] [<80200a0c>] 0x80200a0c
[  162.640000] [<800a98c0>] 0x800a98c0
[  162.640000] [<800a9c24>] 0x800a9c24
[  162.640000] [<800a9ac0>] 0x800a9ac0
[  162.640000] [<80209e0c>] 0x80209e0c
[  162.640000] [<800a6f4c>] 0x800a6f4c
[  162.640000] [<8019ace8>] 0x8019ace8
[  162.640000] [<800a707c>] 0x800a707c
[  162.640000] [<8019b1e8>] 0x8019b1e8
[  162.640000] [<800948d4>] 0x800948d4
[  162.640000] [<8001576c>] 0x8001576c
[  162.640000] [<80090960>] 0x80090960
[  162.640000] [<800a3ba8>] 0x800a3ba8
[  162.640000] [<80090a20>] 0x80090a20
[  162.640000] [<8000e150>] 0x8000e150
[  162.640000] 
[  162.640000] 
[  162.640000] Code: 24420001  10a00004  00000000 
[  162.640000]  1460fffb  24a5ffff  03e00008  00441023  08049d10 
[  162.972000] ---[ end trace e7ee7d69e02fc92a ]---

As you can see, there was a problem with kernel command line arguments.
I've tried work around the problem by setting the command line for the kernel when calling kexec.
I've restarted the router, and tried once again:

root@OpenWrt:/tmp# mkdir /tmp/new
root@OpenWrt:/tmp# mount /dev/sda1 /tmp/new
root@OpenWrt:/tmp# cd /tmp/new
root@OpenWrt:/tmp/new# kexec -l vmlroot@OpenWrt:/tmp/new# kexec -l vmlinux.elf --command-line="console=ttyS0,115200"
root@OpenWrt:/tmp/new# kexec -e
[  157.928000] b44 ssb0:0: eth0: powering down PHY
[  157.936000] device eth0 left promiscuous mode
[  157.940000] br-lan: port 1(eth0.0) entering forwarding state
[  157.952000] Starting new kernel
[  157.956000] kexec_relocate_size = 5416
[  157.960000] reboot_code_buffer = 81a84000
[  157.964000] kexec_start_address(entry point of new kernel) = 802371a0
[  157.972000] kexec_indirection_page = 81b02000
[  157.976000] Copy kexec_relocate section from 80243808 to reboot_code_buffer: 81a84000
[  157.984000] kexec_args[0] (argc): 1
[  157.988000] kexec_args[1] (argv): 81a840e4
[  157.992000] kexec_args[2] (env ):   (null)
[  157.996000] kexec_args[3] (desc):   (null)
[  158.000000] kexec_argv[0] = 81a84529, console=ttyS0,115200
[  158.008000] Will call new kernel at 802371a0
[  158.008000] Bye ...
[    0.000000] Linux version 3.2.9 (wzab@wzab) (gcc version 4.6.3 20120201 (prerelease) (Linaro GCC 4.6-2012.02) ) #2 Fri Mar 2 23:50:59 CET 2012
[    0.000000] CFE's entrypoint seal doesn't match.
[    0.000000] CPU revision is: 00029029 (Broadcom BMIPS3300)
[    0.000000] bcm47xx: using ssb bus
[    0.000000] ssb: Found chip with id 0x5354, rev 0x02 and package 0x00
[    0.000000] ssb: chipcommon status is 0x0
[    0.000000] ssb: Initializing MIPS core...
[    0.000000] ssb: set_irq: core 0x0806, irq 4 => 4
[    0.000000] ssb: set_irq: core 0x0812, irq 5 => 5
[    0.000000] ssb: after irq reconfiguration
[    0.000000] ssb: core 0x0800, irq : 2(S)  3* 4  5  6  D  I 
[    0.000000] ssb: core 0x0806, irq : 2(S)  3  4* 5  6  D  I 
[    0.000000] ssb: core 0x0816, irq : 2(S)* 3  4  5  6  D  I 
[    0.000000] ssb: core 0x0819, irq : 2(S)  3  4  5  6* D  I 
[    0.000000] ssb: core 0x080f, irq : 2(S)  3  4  5  6  D  I*
[    0.000000] ssb: core 0x0812, irq : 2(S)  3  4  5* 6  D  I 
[    0.000000] ssb: core 0x081c, irq : 2(S)  3  4  5  6  D  I*
[    0.000000] found parallel flash.
[    0.000000] ssb: Sonics Silicon Backplane found at address 0x18000000
[    0.000000] Determined physical RAM map:
[...]

Now the new kernel started correctly.
So it seems, that when using kexec with the new 3.2.9 kernel, it is necessary to provide the kernel command line.
What's interesting, I had the command line compiled into my new kernel (for first experiments I've used the same
image, which I've flashed into the router):
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="root=/dev/mtdblock2 rootfstype=squashfs,jffs2 noinitrd console=ttyS0,115200"

I have also tested the old USB disk, which I've created, when starting this thread. Results were the same, so the problem
is associated not with the way the new kernels start, but with the way the new kernels perform kexec...

So to have a flexible way to start the new system from a flashdisk I propose to change the script rc.local:

# Put your custom commands here that should be executed once                    
# the system init finished. By default this file does nothing.                  
mkdir /tmp/new                                                                      
until [ -e /dev/sda1 ]; do                                                      
  sleep 2                                                                       
done                                                                            
mount /dev/sda1 /tmp/new                                                            
if [ -f /tmp/new/linux.elf ]; then
  if [ -f /tmp/new/linux.cmd ]; then                                               
     kexec -l /tmp/new/linux.elf --command-line="`cat /tmp/new/linux.cmd`"
     #Set the watchdog, so that the booting should complete before the watchdog restarts the system
     killall watchdog
     watchdog -T 300 -t 30 /dev/watchdog                                                      
     kexec -e                                                    
  fi                  
fi                                                                              
exit 0

(Last edited by wzab on 3 Mar 2012, 19:47)

it would be nice, if you post this in the wiki smile

I've done yet another correction. Sometimes my development system located on the USB disk is put into the state in which it is not able to boot correctly. Until now, the only method to fix it was to remove the USB disk, and to put it to my host machine for fixing.
In fact it happened frequently enough, that I got tired of it, and I've decided to add functionality allowing to block booting from the USB disk.

I've added the kmod-diag package (CONFIG_PACKAGE_kmod-diag=y in the .config)
After flashing, I've created the /etc/hotplug.d/button directory containing the following /etc/hotplug.d/button/script  script (based on https://forum.openwrt.org/viewtopic.php?id=8151 ):

#!/bin/sh
# RED button in WL500 gP V2 - used to stop booting from USB
if [ "$BUTTON" = "ses" ]; then
 if [ "$ACTION" = "pressed" ]; then
  echo "don't kexec" > /tmp/kexec_stop
  echo "1" > /proc/diag/led/power
  echo "0" > /proc/diag/led/wlan
  sleep 1
  echo "0" > /proc/diag/led/power
  echo "1" > /proc/diag/led/wlan
  sleep 1
  echo "1" > /proc/diag/led/power
  echo "0" > /proc/diag/led/wlan
  sleep 1
  echo "0" > /proc/diag/led/power
  echo "0" > /proc/diag/led/wlan
 fi
fi

Of course you must set the "executable" attribute of this script, by "chmod +x /etc/hotplug.d/button/script"

I have also modified my /etc/rc.local script to the following form

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
# First signal, that we are about to check USB disk
echo "1" > /proc/diag/led/power                                               
echo "1" > /proc/diag/led/wlan                                                
sleep 1                                                                       
echo "0" > /proc/diag/led/power                                               
echo "0" > /proc/diag/led/wlan                                                
sleep 1
echo "1" > /proc/diag/led/power                                               
echo "1" > /proc/diag/led/wlan                                                
sleep 1                                                                       
echo "0" > /proc/diag/led/power                                               
echo "0" > /proc/diag/led/wlan     
sleep 1
echo "1" > /proc/diag/led/power
echo "1" > /proc/diag/led/wlan
if [ -e /tmp/kexec_stop ]; then
  # Booting from USB disk has been stopped by pressing of SES button
  echo "0" > /proc/diag/led/wlan
  exit 0
else  
  echo "0" > /proc/diag/led/wlan
  echo "0" > /proc/diag/led/power
  mkdir /tmp/new                                                               
  until [ -e /dev/sda1 ]; do                                                   
    sleep 2                                                                    
  done                                                                         
  mount /dev/sda1 /tmp/new                                                     
  if [ -f /tmp/new/linux.elf ]; then
    if [ -f /tmp/new/linux.cmd ]; then                                         
       kexec -l /tmp/new/linux.elf --command-line="`cat /tmp/new/linux.cmd`"
       #Set the watchdog, so that the booting should complete before the watchdog restarts the system
       killall watchdog
       watchdog -T 300 -t 30 /dev/watchdog                                                      
       kexec -e                                                    
    fi                  
  fi                                                                           
fi
exit 0

Certainly, you should also do "chmod +x /etc/rc.local" to set the executable attribute of the above script.

It works in the following way:
When the system boots from flash and starts to execute the rc.local script, both POWER and WLAN diodes start to blink simultaneously. If you press then the RED button (marked as "EZSETUP"), booting from USB is stopped (this is confirmed by POWER and WLAN diodes blinking alternately). After that you can log into the system loaded from flash, manually mount the USB disk and fix it.

You can even install to flash most important utilities needed e.g. to repartition and reformat the USB disk. To achieve that I have set the following options in the .config file:
CONFIG_BUSYBOX_CONFIG_FSCK=y
CONFIG_BUSYBOX_CONFIG_FDISK=y
CONFIG_BUSYBOX_CONFIG_FEATURE_FDISK_WRITABLE=y
CONFIG_BUSYBOX_CONFIG_MKFS_EXT2=y
CONFIG_BUSYBOX_CONFIG_MKFS_VFAT=y

The above lets me to (re)create vfat and ext2 partitions on the USB disk, and together with the "tar" functionality offered by the busybox,  I can populate the newly created partitions.

(Last edited by wzab on 4 Mar 2012, 14:30)

wzab wrote:

I'm using the "ramfs" option, ...

There is no "ramfs" option in "Target Images", only these:
[ ] ramdisk  --->                                                                           
--- Root filesystem archives                                                         
[ ] cpio.gz                                                                                 
[ ] tar.gz                                                                                 
--- Root filesystem images                                                       
[ ] ext2                                                                                   
[ ] jffs2                                                                                   
[ ] squashfs                                                                           
--- Image Options
Also e2fs, e3fs not present in In "Kernel modules/Filesystems", only EXT4.

Yes, you are right.
I tried to update my system, and I've stated that the old recipe doesn't work with the newest trunk.
In fact I have also discovered that the old "bootloader" does not load correctly OpenWRT compiled from the newest trunk - https://forum.openwrt.org/viewtopic.php?id=46536

I'll try to update it.
Maybe the following thread (even though describing another router) will be helpful:
https://forum.openwrt.org/viewtopic.php?id=36462

(Last edited by wzab on 7 Oct 2013, 00:38)

Hi,
Thanks to the information found in https://forum.openwrt.org/viewtopic.php?id=36462 and to support I have received in threads: https://forum.openwrt.org/viewtopic.php?id=46594 and https://forum.openwrt.org/viewtopic.php?id=46536 I can now present the recipe which works with the current trunk.

To create the "USB loader" which is flashed into the device, I have done

git clone git://git.openwrt.org/openwrt.git
cd openwrt
make menuconfig

In the menuconfig I have set the following options:

Target System (Broadcom BCM947xx/953xx)
Target Profile (Broadcom SoC, b44 Ethernet, BCM43xx WiFi (b43, default))
Kernel modules -> Filesystems -> kmod-fs-ext4 (*)
Kernel modules -> USB Support -> kmod-usb-ohci (*)
Kernel modules -> USB Support -> kmod-usb-storage (*)
Kernel modules -> USB Support -> kmod-usb2 (*)
Utilities -> Filesystem -> e2fsprogs (*)
Utilities -> kexec-tools (*)
Global build settings -> [*] Enable kexec support

Afterwards I have created subdirectory "files" with the following files
files/etc/hotplug.d/button/script:

#!/bin/sh
# RED button in WL500 gP V2 - used to stop booting from USB
if [ "$BUTTON" = "ses" ]; then
 if [ "$ACTION" = "pressed" ]; then
  echo "don't kexec" > /tmp/kexec_stop
  echo "1" > /proc/diag/led/power
  echo "0" > /proc/diag/led/wlan
  sleep 1
  echo "0" > /proc/diag/led/power
  echo "1" > /proc/diag/led/wlan
  sleep 1
  echo "1" > /proc/diag/led/power
  echo "0" > /proc/diag/led/wlan
  sleep 1
  echo "0" > /proc/diag/led/power
  echo "0" > /proc/diag/led/wlan
 fi
fi

files/etc/rc.local:

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
# First signal, that we are about to check USB disk
echo "1" > /proc/diag/led/power                                               
echo "1" > /proc/diag/led/wlan                                                
sleep 1                                                                       
echo "0" > /proc/diag/led/power                                               
echo "0" > /proc/diag/led/wlan                                                
sleep 1
echo "1" > /proc/diag/led/power                                               
echo "1" > /proc/diag/led/wlan                                                
sleep 1                                                                       
echo "0" > /proc/diag/led/power                                               
echo "0" > /proc/diag/led/wlan     
sleep 1
echo "1" > /proc/diag/led/power
echo "1" > /proc/diag/led/wlan
if [ -e /tmp/kexec_stop ]; then
  # Booting from USB disk has been stopped by pressing of SES button
  echo "0" > /proc/diag/led/wlan
  exit 0
else  
  echo "0" > /proc/diag/led/wlan
  echo "0" > /proc/diag/led/power
  mkdir /tmp/new                                                               
  until [ -e /dev/sda1 ]; do                                                   
    sleep 2                                                                    
  done                                                                         
  mount /dev/sda1 /tmp/new -o rw,sync,relatime
  if [ -f /tmp/new/linux.elf ]; then
    if [ -f /tmp/new/linux.cmd ]; then                                         
       kexec -l /tmp/new/linux.elf --command-line="`cat /tmp/new/linux.cmd`"
       #Set the watchdog, so that the booting should complete before the watchdog restarts the system
       killall watchdog
       watchdog -T 300 -t 30 /dev/watchdog                                                      
       kexec -e                                                    
    fi                  
  fi                                                                           
fi
exit 0

After that I have compiled my system image with "make V=99 world" (in fact to speed up compilation on my machine I have used "make -j 8 V=99 world") , and then sent it to the AP (started with the RESTORE button pressed down) with:
"atftp --trace --option "timeout 1" --option "mode octet" --put --local-file openwrt-brcm47xx-squashfs.trx 192.168.1.1"

To create the "user" system which is loaded from the external USB disk, I have done:

git clone git://git.openwrt.org/openwrt.git
cd openwrt
make menuconfig

In the menuconfig I have set the following options:

Target System (Broadcom BCM947xx/953xx)
Target Profile (Broadcom SoC, b44 Ethernet, BCM43xx WiFi (b43, default))
Kernel modules -> Filesystems -> kmod-fs-ext4 (*)
Kernel modules -> USB Support -> kmod-usb-ohci (*)
Kernel modules -> USB Support -> kmod-usb-storage (*)
Kernel modules -> USB Support -> kmod-usb2 (*)
Utilities -> Filesystem -> e2fsprogs (*)
Base system -> block-mount (*)

Due to some problems with passing the kernel commandline via kexec, I hade to compile-in the appropriate command line:

make kernel_menuconfig
#Kernel hacking -> Built-in kernel command line [*]
#Kernel hacking -> Default kernel command string (rdinit=/myinit noinitrd console=ttyS0,115200)
#Kernel hacking -> Built-in command line overrides firmware arguments [*]

Then I have created the subdirectory "files" with the following file
files/myinit:

#!/bin/sh
mount proc /proc -t proc
modprobe diag
modprobe ehci-platform
modprobe ssb-hcd
modprobe sd-mod
modprobe usb-storage
/bin/sleep 10
mknod /dev/sda1 b 8 1
mkdir /tmp/new
modprobe ext4
mount /dev/sda1 /tmp/new -o rw,sync,relatime
exec /sbin/switch_root /tmp/new /init

Afterwards I have compiled my system with "make V=99 world" (in fact to speed up compilation on my machine I have used "make -j 8 V=99 world").
Then I have copied the kernel image bin/brcm47xx/openwrt-brcm47xx-squashfs.trx  as linux.elf to the main directory of my USB flashdisk.
Next I have copied the contents of the build_dir/target-mipsel_mips32_uClibc-0.9.33.2/root-brcm47xx directory to the main directory of my USB flashdisk, and finally I have created there the linux.cmd file containing the "rdinit=/myinit noinitrd console=ttyS0,115200"  (in fact the contents of this file is not important, as due to some problems with passing the command line via kexec, I had to force the kernel to use compiled-in command line).

It seems that so prepared system works correctly, allowing to load the kernel from external USB disk and to mount the rootfs from the same disk.

(Last edited by wzab on 8 Oct 2013, 09:30)

The discussion might have continued from here.