Cisco Meraki MX64/MX65 image support

For Developers
191

Hi all, I'm happy to have found this thread! I have both an MX64 and MX65 that I want to re-use. I've read through the years of thread here (whew) and I'm still not quite clear on the process. Is it essentially this?

  • Connect to serial header
  • Interrupt boot sequence by serial
  • Something about the image being on a MBR+FAT flash disk?

Thanks for any pointers to the latest procedure. :slight_smile:

Ooh I found this thread, maybe I don't need serial access any more? That would be handy.
https://github.com/openwrt/openwrt/pull/3996

192

Would be nice if someone could document the installation process on the devicepages:
https://openwrt.org/inbox/toh/meraki/meraki_mx64
https://openwrt.org/inbox/toh/meraki/meraki_mx65

1 Like
193

I have updated u-boot and the install procedure so that larger kernels (>3MiB) can be used, which should help get the PR accepted. However this means that existing users will need to update u-boot. I've detailed the update procedure in the commit message (not the PR message). While I have flashed this successfully on several devices it would be good if someone (preferably with a spare device) could test this procedure and report back any issues found. You will also need to compile your own images - initramfs and sysupgrade.

3 Likes
194

I tried the reset-button method, but all I get is a solid orange light that never changes. So I've connected to the serial port, and I can talk to U-Boot, but it seems to be failing. Looks like I have a "small" MX65, which I have along with the kernel on a FAT flash drive. Once I'm in u-boot, how can I get it to go the rest of the way? Here's my boot output:
https://pastebin.com/cutAscMr

195

Hi! I tried to install this to a MX64 0A, I managed to get clayface U-boot installed, booted initramfs from usb, did the sysupgrade -F using the images provided by clayface ( openwrt-bcm53xx-generic-meraki_mx64-squashfs.sysupgrade.bin ) but after reboot I get the u-boot showing Bad kernel image, do anyone have any idee where I get it wrong ?
Everything works as expected but after the sysupgrade the unit doesn't boot.
Thanks!

196

Can you tell me which image that is?

197

Sorry for the confusion, I used:
https://app.box.com/s/9kkyjm0t2n54kqte8a659tqtibweqy69
and U-boot/initramfs from:

If the images are correct I can post full logs tomorrow.

Thanks!

EDIT:

I tried with all images found in this thread, the result is identical.

Back with the logs for sysupgrade:

Also booting from USB doesn't work, I have to do the:
fatload usb 0:1 0x60008000 openwrt-bcm5862x-generic-meraki_mx64a0-initramfs-kernel.bin
and then bootbk from that address.

Logs:

198

You need to use a bootm compatible image. I provided a bootbk image so that users with the older u-boot could change to the new one.

These images should work (same URLs but images are updated):
initramfs
sysupgrade

199

Thanks!
I can flash a newer uboot but dont know where i can find it. Thanks!

200

You already have the newer one

201

Thanks @clayface !
The images provided works as expected, really appreciate your effort!

202

I'm working on a full walk-through doc right now as I follow the various steps mentioned in clayface's PR #3996 and will post it once I get this MX65 functional with OpenWRT.

A few days prior I flashed a Z1 and didn't run into any issues, it's all good (and I have a very detailed install doc for that as well if the community is interested; can post to git and link if so.)

The issue I'm running into with the MX65 is during this stage:

U-boot installation: MX65 Only

Prepare a USB drive formatted to FAT. Download uboot_mx65 to the USB drive from the following location:
https://github.com/clayface/U-boot-MX64-20190430_MX65

Once you have telnet access to the MX65, plug in the USB stick and run the following commands. This is the most dangerous step so please ensure the dd command is entered correctly:
cd /tmp/media/sda1
dd if=uboot_mx65 of=/dev/mtdblock0

Once this has successfully completed, power off the device.

When I ran that command dd did not complete as it usually does and threw an error. So I tried the small u-boot image uboot_mx65_small, also with the same dd error output. FWIW I validated the files checksums before and after checking out the repo versions and copying them to a valid/working 32GB USB flash drive (same one I used for the Z1) formatted as FAT32. So the files were fine but they refused to write to the MX65... no idea why.

Then tried a few other debugging steps over both TTL serial and via telnet. Simply didn't work. I looked back through the boot logs and everything looked ok. Now when I reboot the device it seems bricked; holding down reset to get into busybox no longer works, there's no TTL output with or without reset being pressed, the orange LED never changes to white or other colors like it had previously done.

Hopefully it's not bricked; I'd rather not buy a used one as this was a brand new model in original packaging that was gifted for free (licensed but never used; company didn't want it anymore).

Any ideas on how to get this booted again? Here's the dd error output:

# cd /tmp/media/sda1
# dd if=uboot_mx65 of=/dev/mtdblock0
dd: uboot_mx65: Input/output error
# hotplug detected product:  90c/1000/1100
hotplug detected product:  90c/1000/1100

It then seems to unmount the USB drive, so I unplug it and replug it for auto-mount to do its thing, and try the small image:

cd /tmp/media/sda1/
# dd if=uboot_mx65_small of=/dev/mtdblock0
# dd: uboot_mx65_small: Input/output error
# hotplug detected product:  90c/1000/1100
hotplug detected product:  90c/1000/1100

Before running dd I did run a backup of mtdblock0 to the USB drive, so I have that but not sure how to get it back onto the MX65 from the USB drive if there's no TTL post

# dd if=/dev/mtdblock0 of=/tmp/media/sda1/original-blockdevs/mtdblock0 bs=1M
1+0 records in
1+0 records out
# ls -alh /tmp/media/sda1/original-blockdevs/
drwxr-xr-x    2 0        0           16.0k Dec 31 16:49 .
drwxr-xr-x    3 0        0           16.0k Dec 31 16:46 ..
-rwxr-xr-x    1 0        0            1.0M Dec 31 16:49 mtdblock0

Files from the repo look fine as well:

# cd /tmp/media/sda1
# ls -al
drwxr-xr-x    3 0        0           16384 Dec 31 16:46 .
drwxr-xr-x    3 0        0               0 Dec 31 16:28 ..
-rwxr-xr-x    1 0        0         5242912 Jun 28  2022 openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin
drwxr-xr-x    2 0        0           16384 Dec 31 16:49 original-blockdevs
-rwxr-xr-x    1 0        0          587072 Jun 28  2022 uboot_mx65
-rwxr-xr-x    1 0        0             141 Jun 28  2022 uboot_mx65.sha512
-rwxr-xr-x    1 0        0          524288 Jun 28  2022 uboot_mx65_small
-rwxr-xr-x    1 0        0             141 Jun 28  2022 uboot_mx65_small.sha512

Eventually I decide to reboot and try again, but that seems to have been a mistake.

# exit
Hit enter to continue...

BusyBox v1.7.2 (2016-01-11 12:37:47 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# 
# reboot now
Terminated
Sending SIGTERM to all processes
# IGD shutdown!
SendiRestarting system.

Bricked? Hope not.

203

Terribly sorry but looks like it has bricked. Seems like there was some kind of issue reading from the USB. Since the NAND is TSOP48 there is a chance to recover using a clip - for example see here.

204

well that's a shame. I'm willing to give the OFNI method a try... though I don't have that particular board or connector. can you recommend a seller for those items?

205

To recover an MR33 I used a Teensy++ 2.0 with a 360-Clip (further purchase information in bellows link). Depending where you are I might also be able to help you out getting it all done.

206

That's a great solution. I don't have the Teensy++ 2.0 but I can order one, which would arrive by July 8th (costs $40), or I can order one of the other Teensy models (with pins):

  • 2.0 (standard, not ++ version), $18, one day shipping
  • LC, $21, one day shipping
  • 4.0, $29, arrive by July 9th

Then I'd need the other parts of course, so the quick shipping isn't really an issue except for the 360 Clip, which I can only find being delivered from China on AliExpress for $36 and it would arrive July 29th... which is a frustratingly long time to wait. I do not mind investing in new debugging/embedded devices, and cost priorities in this project are subjective as long as it's enjoyable.

The thread mentions another method than the 360 clip (ZIF socket and hot air station) so perhaps that is worth exploring? I have a decent digital solder kit (Weller WE1010NA) and don't mind going that direction if necessary. Any ideas on which parts would be required for this ZIF method?

207

I once used those before. You find a picture of some other chap having converted his board into a dev board using a similar socket.

https://www.aliexpress.com/item/1005001355412303.html

Then, of course, there are also the traditional ZIF ones. While I purchased one of those on eBay once, I never actually used it so far. You might have it if you find any use for it. Just PM me.

1 Like
208

Thanks for the link. That one is rather more affordable than the one I was looking at today, though with the benefit of including a FlashCat USB device. I'd take you up on the offer but I suspect shipping from the EU to USA would be about the same cost as the Flashcat version.

What do you think of this one: https://www.ebay.com/itm/304520602445

209

Good news on the hardware for this side project... I found two additional new-in-box "license claimed" MX65 units for sale on the cheap and so now have a couple more units to try not to brick like last time. :wink:

So the first one, I did the usual first steps and got to the early stage of resetting into busybox and then backing up /dev/mtdblock0 onto a 8GB thumb drive with dd. Decided to enable term capture for logging the output, did so, then issued a reboot. I left the usb drive in the board, didn't manually unmount before reboot. When it came back up while watching the capture log via TTL connection, I see it's loading uboot as such:

U-Boot 2012.10 (May 04 2022 - 23:46:39)Meraki MX64 Boot Kernel Loader

Then a flood of unexpected output from Single STEP Calibration, which ended with Log: DDR Tuning Complete. Finally, it kicks over to a u-boot> prompt after notifying that the following initramfs cannot be found:

1046   │ Volume kernel not found!
1047   │ (Re)start USB...
1048   │ USB:   Before reset, USB clk enable register is: 000002dd
1049   │ Before reset, USB clk enable register is: 000002dd
1050   │ usb2_reset_state is: 00000000
1051   │ Register 1212 NbrPorts 2
1052   │ USB EHCI 1.00
1053   │ scanning bus for devices... 2 USB Device(s) found
1054   │        scanning bus for storage devices... 1 Storage Device(s) found
1055   │ reading openwrt-bcm53xx-generic-meraki_mx65-initramfs.bin
1056   │ ..
1057   │ ** Unable to read "openwrt-bcm53xx-generic-meraki_mx65-initramfs.bin" from usb 0:1 **
1058   │ Wrong Image Format for bootm command
1059   │ ERROR: can't get kernel image!
1060   │ u-boot>

So, according to the last thing I read about the initramfs, I was expecting the boot sequence to access openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin instead, which is on the mounted USB drive in the base directory. Grep'ing for the other initramfs file mentioned in the boot log, I see it over here in the USB drive's directory for the build process:

» grep -R "openwrt-bcm53xx" *
U-boot.MX64/include/configs/kingpin.h:        "initramfs_filename=openwrt-bcm53xx-generic-meraki_mx65-initramfs.bin\0" \
Binary file uboot_mx65 matches

Some Process Reference Details

For posterity sake and reference, here's the directory/structure for the files that get moved over to the USB drive after formatting it to FAT32 w/ DOS partition table, along with the process I follow for formatting the drive and blah blah blah.

Given the bricking from the first device by using a problematic 32GB drive, I made sure to not use GPT and wipe the drive entirely before any use with the MX65. Process is scripted as such (sda is always a usb drive on the workstation as the other drives are all nvme).

» cat sda-refresh.sh
   1   │ wipefs -a /dev/sda
   2   │ sgdisk --zap-all /dev/sda
   3   │ sgdisk --gpttombr /dev/sda

» sudo ./sda-refresh.sh
/dev/sda: 5 bytes were erased at offset 0x00008001 (iso9660): 43 44 30 30 31
/dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sda: calling ioctl to re-read partition table: Success
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or other utilities
  • Command overview during fdisk usage:
  1. DOS label is auto-added by fdisk because it has no part table yet
  2. create primary partition to full disk size, label it FAT32, write the changes, exit fdisk
  3. use ` sudo mkfs.fat -F 32
  • Then Formatting FAT32...
» sudo fdisk /dev/sda
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x7307a6dd.

Command (m for help): p
Disk /dev/sda: 29.9 GiB, 32080200192 bytes, 62656641 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7307a6dd

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):
Partition number (1-4, default 1):
First sector (2048-62656640, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-62656640, default 62656640):

Created a new partition 1 of type 'Linux' and of size 29.9 GiB.
Command (m for help): p

Disk /dev/sda: 29.9 GiB, 32080200192 bytes, 62656641 sectors
Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1        2048 62656640 62654593 29.9G 83 Linux

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): b
Changed type of partition 'Linux' to 'W95 FAT32'.

Disklabel type: dos
Disk identifier: 0x7307a6dd
Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1        2048 62656640 62654593 29.9G  b W95 FAT32

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Now the big question...

So... does this look correct and/or expected w/re to the initramfs or should I modify the contents of the USB drive to include the mentioned file openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin instead of the openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin one?

Here are the SHA256 hashed of the newer bin file. I cannot locate the other file to generate a hash for it.

» openssl sha256 openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin 
SHA256(openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin)= 9b3f5ac07ac6f0487fc7775c570df97b375808e466e80abc446765e07839440d

And the tree structure of the data that ends up on the USB drive:

[~/Downloads/Firmware/Cisco-Meraki-Firmware/meraki_mx65]
» tree -L 2
.
├── mx64
│   ├── mtd
│   ├── mtd-rw.ko
│   ├── openwrt-bcm5862x-generic-meraki_mx64-initramfs-kernel.bin
│   ├── openwrt-bcm5862x-generic-meraki_mx64.manifest
│   ├── openwrt-bcm5862x-generic-meraki_mx64-squashfs-sysupgrade.bin
│   └── uboot_mx64
├── OpenWRT_MX64-MX65.rar
└── U-boot-MX64-20190430_MX65
    ├── openwrt-bcm5862x-generic-meraki_mx65-initramfs-kernel.bin
    ├── U-boot.MX64
    ├── uboot_mx65
    ├── uboot_mx65.sha512
    ├── uboot_mx65_small
    └── uboot_mx65_small.sha512

Before continuing I want to validate/verify with @sumo and/or @clayface that this initramfs file is the expected version. I made sure to pull the most recent commit from the MX65 repo ( https://github.com/clayface/U-boot-MX64-20190430_MX65.git) but I don't see the openwrt-bcm53xx-generic-meraki_mx65-initramfs.bin in the repo anywhere... just that uboot is complaining as such:

» tail -f mx65-dev-2.cap 
Register 1212 NbrPorts 2
USB EHCI 1.00
scanning bus for devices... 2 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
reading openwrt-bcm53xx-generic-meraki_mx65-initramfs.bin
..
** Unable to read "openwrt-bcm53xx-generic-meraki_mx65-initramfs.bin" from usb 0:1 **
Wrong Image Format for bootm command
ERROR: can't get kernel image!
u-boot>
210

To provide insight, previously I provided a u-boot image with fewer modifications, that allowed booting from the existing bootkernel partition. This had a big limitation which was the 3MiB limit for the bootkernel partition size, which is likely to be exceeded in the near future. To solve this, the bootkernel needed to be moved to within the UBI, which consumes almost all the 1GiB NAND, more than sufficient for the kernel.

However this presented another issue, whereby the existing u-boot partition was limited to 512KiB, with the shmoo config located immediately after. Adding support for UBI and bootm to u-boot would take us beyond 512KiB. To solve this, I have made use (diff) of the NVRAM partition that is located between the two bootkernels and consumes an excessive 1MiB, when 32KiB would be enough to store the required data (it's not really required apart from the W variants). Shmoo has been moved within there along with a new u-boot-env partition. When you booted after installing u-boot, the output you saw was the shmoo calibration data being created.

The bcm5862x initramfs is provided to help users of that old u-boot change to the new version. The expectation is that once upstreamed, OpenWrt will provide the initramfs file. For the time being you can use this initramfs image:
https://app.box.com/s/ndvtb8yipis0ov6uvkkd0mhigjibdyi9
and the sysupgrade:
https://app.box.com/s/aa19cbirkcgnuqzf3g1dhrjdmh7nc3p2

1 Like