There actually is a magic value like for LZOR, after you get the dump there is LZ77 is ASCII
Hi John,
This is my factory hard_config for the Chateau 12(LZ77):
I never used a decompiler, so it might be more efficient if you can take a look at it 
MOD: I upgraded my device to ROS7.7/routerboot7.7 and John's method still works: I can boot initramfs then sysupgrade to Openwrt.
1 Like
I wonder if anyone has other RoS7 only devices, it would be interesting to see how widespread this LZ77 issue is. @robimarko do you happen to have any such devices? Even if it is ipq60xx, it might worth to take a look at them. I have a feeling that this is going to be widespread....
@johnth I tried to look at this and I can see that the current LZOR method uses the in-kenrel functions. As much as I understand LZ77 is LZ2, but I am not sure if LZ4 is compatible or if it can be used with the LZ4.h methods. If not, then there is the problem of external dependencies like zlib in order to do the decompress.
you probably know this but just in case.
was experimenting a bit with a board from a Mtik SXTsq 5 ac that I had bricked and opened. Accessing NOR with a SOP8 clip and programmer.
To restore a working ROS unit two things that are unique to the device are needed: the mtd1 partition and the license file. Apart from the bootloaders and its three config data internal mtds, mtd1 contains a header before the first bootloader with a.o. serial number and mac address, and a string near 0x20000 from dtb1 start that probably relates to the license. That string remains unaltered when bootloader version changes. mtd0 also needs to be there but is not unique to the device and common for the ipq40xx devices I could test (hAPac2, SXTsq 5ac, LHG 5 ac).
I was wondering whether it would be possible to install OpenWrt by just abandoning the entire Mtik boot process after backup of the first 1 MB of flash (mtd0 & 1), and install e.g. uboot in mtd0, followed by kernel and rootfs in mtd5. Essentially mtd1,2,3,4 would remain untouched, with hard_config and soft_config used by OpenWrt.
Could mtd0, pretty useless for OpenWrt anyway, with its 524288 bytes fit a bootloader that would allow tftp and bootp to still function? Returning to ROS would imply putting back mtd0, and using MTik's netinstall.
With an open source bootloader we would not have to spend time on anything MTik engineers are changing in the boot process.
Only worry would be changes in the hard_config and soft_config format
I dont have any ROSv7 only devices currently, not really keen on getting ipq60xx ones are they are overpriced.
But you are right that they are gonna move to LZ77 only, and I dont think that kernel provides a way to decompress LZ77 currently
1 Like
Two things:
-
As much as I understand, we have no issues with booting and installing Openwrt on factory bootloader since John's work seems to work. So replacing the bootloader has no good reason and in general we try to avoid that (unless it is not possible to get around it).
-
No matter if you replace the bootloader, the hard_config extract method will be needed to get the BDF/Cal specific to the device.
yes, hard_config will be needed, but that would be all. No need to figure out other changes to RB. But it would only be a 'simple' fix when 512kB is enough for an alternative bootloader that offers basic functionality via ethernet. If not, the partitioning would have to be changed and compatibility with the original system would be lost.
I checked the RB_config part on my Chateau 12LTE, and it seems besides the compression change, the rest is intact. So at the moment the only issue is the LZ77 compression of the BDF. Of course with ipq60xx that can change, that is one of the reason I asked if someone can provide hard_configs of other ROS7-only devices 
I wonder if we can "patch" the current linux/lz4.h to provide support for LZ2/LZ77, as much as it seems LZ4 is a more complex version of LZ2, so maybe the diff is not that big. I would assume that adding external dependency (zlib fox example) to this process is the least favorable.
@johnth I wonder if you had a bit of time to take a look at the hard_config dump by any chance?
@johnth or anyone interested, this is a summary of what I did so far:
The factory hard_config partition of the device: LINK (as the old one is gone).
The ROS7 bootlog say:
[ 12.208043] radio data lz77 decompressed from 2756 to 11924
...
[ 22.632744] probe_er_radio_data chunkid 8001 caldataid -1
[ 22.632760] found extended radio data with id 32769
[ 22.633119] radio data rle decompressed from 5464 to 24128
...
[ 24.891557] probe_er_radio_data chunkid 8201 caldataid -1
[ 24.891576] found extended radio data with id 33281
[ 24.891936] radio data rle decompressed from 6442 to 24128
The magic number is: 77ZL and the size is exactly 2756 bytes as the first dmesg entry of ROS7 indicates.
According to hc_wlan_data_unpack_lzor() the payload after the magic number must be appended with hc_lzor_prefix[], question is if this is the same for LZ77 as well? I looked at the flash.ko in Ghidra and the only "dictionary" like element is the one we already have, but again I am not sure if it is used for LZ77 as well, or only for LZOR.
I tried to unpack the "payload" with various accessible LZ77 implementations, but failed so far (I did not tried to append the payload to hc_lzor_prefix[] yet.).
This is where I am. Hope @johnth can take a look at this.
Thank you for the RouterOS bootlog. Very helpful to see what we are expecting in messages, uncompressed length, and the order of lz77, chunks, rle. It looks the same as lzor in that data is RLE encoded, then packed into chunk tags, then compressed. Would expected this compressed chunk to decompress to starting with DRE\x00, and if this was only LZ77 (in a block format without packed bytes), would also expect to see this in the compressed data. FastLZ https://ariya.github.io/FastLZ/ has a nice explanation of their block format.
Still have not had enough time to be able to dig into this.
From what I can see, there are functions for both compress and decompress of this lz77 in flash.ko (starting with decompress in ROS ARM around v6.46). The decompress function gets passed an output buffer ptr, output buffer length (0x40000), the input buffer ptr, and input length. For input, the 77ZL magic is skipped, and the hardcfg wlan-data length tag is reduced by 4 for this skipped magic (there is no constant prefix (like done for lzor1x)). The decompress function returns uncompressed length, or a negative number for error.
The Ghidra decompiled decompress function dumps very early on me, so I might have to try to do something else to determine the block layout. Look at the compress function, or debug, or walk through the assembly.
You already know that? Thats great! It was a bit confusing as LZ77 does not have a dictionary, and I was afraid this was another trick added by MikroTik to mislead us, like they call it LZ77 but added some extra flavors to it
Non the less, I tried many decompression parameters and I was never able to get even close to the 11924 bytes length the ROS7 bootlog suggests...
I think I worked out the Mikrotik LZ77 block format, but I would like some additional hard_config with LZ77 data examples to test (or hardware…, hardware is always nice). Details and demo python decompressor here (worked for 1/1 example so far). Demo decompressor only decompresses LZ77, not the Mikrotik RLE you get after it (already done by rb_hardconfig).
Hi @wwortel and @cezary, you have both previously mentioned Mikrotik LZ77 caldata. I would appreciate it if you could send me a link to a NOR dump (at minimum I need hard_config tag 0x16, but that probably does include your MAC address)
2 Likes
This is great news.
What I can do for you is test, and next week we likely get a second Chateau 12, so I can send you the hard_config of that device too, so we can compare CAL data of the same series on two devices which are apart from each other by 4-5 months in terms of manufacturing.
I can get you full flash dumps of six RB911-5Hn at ROS 7.8, would that help? I just need to zip them.
They originally came with much older release, though - but I experienced the booting problem with ROS7 bootloader on them.
These likely will not have want I want to test, so don't bother, thanks.
The hard_config is only written during production, and from what I have seen LZ77 encoded wlan-data is only used on a few newer arm or aarch64 devices.
2 Likes
Johnth needs dumps from ROS7 only devices, those are the ones that likely has LZ77 encoded. It is not enough that the device runs ROS7
How do I get the dumps from my Chateau 5G and both LTE12?
I don't have access to any MT Chateau right now. If I have something, I'll post it.
Both devices are ipq4019. I can send you an initramfs image, which you need to netboot on the device, then you can log in via SSH and dump the hard_config partition and copy it via SCP to your PC.
If any of the above seems awfully complicated, I'd rather not write down the complete process, but if you up for it, and you have at least basic networking knowledge, then I can help you.