Depending on or defaulting to extroot? (emmc, change rootfs_data)

Is this acceptable?

Background: Trying to add support for a MT7988D device (Zyxel EE4600-00, but this could be an operator specific designation) with 8GB emmc. The OEM firmware uses a dual firmware scheme with a chainloaded vendor bootloader as the final step (zloader, as a FIT image this time)

I would prefer to keep the vendor bootloader. At least as an option. This makes OpenWrt installation simpler and less risky, and it allows keeping vendor firmware in the other slot. It's even possible to switch back and forth with full functionality in place, except for vendor firmware upgrade (would overwrite OpenWrt). All this is feasible and I have it working.

The problem is partitions. The vendor table is crazy:

Device      Start   End Sectors Type-UUID              UUID                 Name     Attrs  
/dev/mmcblk0p1   8192  9215  1024 3DE21764-95BD-54BD-A5C3-4ABE786F38A8 5C35B638-6901-11F0-B5A4-379A290AE74C u-boot-env     
/dev/mmcblk0p2   9216  17407  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B639-6901-11F0-B5A4-379A290AE74C factory      
/dev/mmcblk0p3  17408  21503  4096 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B63A-6901-11F0-B5A4-379A290AE74C fip        
/dev/mmcblk0p4  21504  22527  1024 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B63B-6901-11F0-B5A4-379A290AE74C zloader      
/dev/mmcblk0p5  22528  88063  65536 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B63C-6901-11F0-B5A4-379A290AE74C kernel       
/dev/mmcblk0p6  88064 219135 131072 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B63D-6901-11F0-B5A4-379A290AE74C rootfs       
/dev/mmcblk0p7  219136 221183  2048 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B63E-6901-11F0-B5A4-379A290AE74C zyfwinfo      
/dev/mmcblk0p8  221184 223231  2048 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B63F-6901-11F0-B5A4-379A290AE74C zydefault     
/dev/mmcblk0p9  223232 288767  65536 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B640-6901-11F0-B5A4-379A290AE74C kernel2      
/dev/mmcblk0p10 288768 419839 131072 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B641-6901-11F0-B5A4-379A290AE74C rootfs2      
/dev/mmcblk0p11 419840 421887  2048 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B642-6901-11F0-B5A4-379A290AE74C zyfwinfo2     
/dev/mmcblk0p12 421888 423935  2048 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B643-6901-11F0-B5A4-379A290AE74C zydefault2     
/dev/mmcblk0p13 423936 440319  16384 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B644-6901-11F0-B5A4-379A290AE74C rootfs_data    
/dev/mmcblk0p14 440320 448511  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B645-6901-11F0-B5A4-379A290AE74C rom-d       
/dev/mmcblk0p15 448512 456703  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B646-6901-11F0-B5A4-379A290AE74C romfile      
/dev/mmcblk0p16 456704 464895  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B647-6901-11F0-B5A4-379A290AE74C teconfig      
/dev/mmcblk0p17 464896 473087  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B648-6901-11F0-B5A4-379A290AE74C zyMFG       
/dev/mmcblk0p18 473088 538623  65536 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B649-6901-11F0-B5A4-379A290AE74C data        
/dev/mmcblk0p19 538624 800767 262144 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B64A-6901-11F0-B5A4-379A290AE74C syslog       
/dev/mmcblk0p20 800768 808959  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B64B-6901-11F0-B5A4-379A290AE74C wwan        
/dev/mmcblk0p21 808960 2621439 1812480 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B64C-6901-11F0-B5A4-379A290AE74C misc        
/dev/mmcblk0p22 2621440 3252223 630784 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B64D-6901-11F0-B5A4-379A290AE74C misc1       
/dev/mmcblk0p23 3252224 3452927 200704 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B64E-6901-11F0-B5A4-379A290AE74C firmware      
/dev/mmcblk0p24 3452928 3653631 200704 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B64F-6901-11F0-B5A4-379A290AE74C firmware2     
/dev/mmcblk0p25 3653632 3670015  16384 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B650-6901-11F0-B5A4-379A290AE74C rootfs_data2    
/dev/mmcblk0p26 3670016 3671039  1024 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B651-6901-11F0-B5A4-379A290AE74C reserved512K    
/dev/mmcblk0p27 3671040 3672063  1024 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B652-6901-11F0-B5A4-379A290AE74C reserved512K1   
/dev/mmcblk0p28 3672064 3674111  2048 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B653-6901-11F0-B5A4-379A290AE74C reserved1M     
/dev/mmcblk0p29 3674112 3678207  4096 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B654-6901-11F0-B5A4-379A290AE74C reserved2M     
/dev/mmcblk0p30 3678208 3686399  8192 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B655-6901-11F0-B5A4-379A290AE74C reserved4M     
/dev/mmcblk0p31 3686400 3702783  16384 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B656-6901-11F0-B5A4-379A290AE74C reserved8M     
/dev/mmcblk0p32 3702784 3735551  32768 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B657-6901-11F0-B5A4-379A290AE74C reserved16M    
/dev/mmcblk0p33 3735552 3801087  65536 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B658-6901-11F0-B5A4-379A290AE74C reserved32M    
/dev/mmcblk0p34 3801088 3932159 131072 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B659-6901-11F0-B5A4-379A290AE74C reserved64M    
/dev/mmcblk0p35 3932160 4194303 262144 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B65A-6901-11F0-B5A4-379A290AE74C reserved128M    
/dev/mmcblk0p36 4194304 4718591 524288 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B65B-6901-11F0-B5A4-379A290AE74C reserved256M    
/dev/mmcblk0p37 4718592 5767167 1048576 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B65C-6901-11F0-B5A4-379A290AE74C reserved512M    
/dev/mmcblk0p38 5767168 7864319 2097152 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5C35B65D-6901-11F0-B5A4-379A290AE74C reserved1024M  

But I consider changing it without changing the bootloader out of the question. Too risky.

The existence of a tiny rootfs_data partition will result in a bad user experience by default. We need something larger. But how?

I've considered pointing root to one of the larger unused partitions, like misc1 or reserved1024M. But this locks us there. And becomes a problem if the vendor changes their table. Maybe unlikely, but I have a bad feeling about this. Labels, uuids and numbers could all just disappear with a vendor firmware update.

So I started thinking about extroot as the configurable alternative. Would it be acceptable to have a tiny rootfs by default and recommending extroot? Or maybe even provide a default extroot configuration?

We had similar issues with the Unifi 6 Plus. It ended up with most of the emmc unused by default. But I think that has worked out fine. Not as extreme though.

Thoughts?

Is 100% of flash allocated, I don't think so ?

That sounds the most promising option to me.
But I foresee possible difficulties in getting it flashed to proper places from OEM firmware.

The easiness might depend on how OEM uses partitions. I see the kernel 64 MB, rootfs 128 MB, firmware 200 MB, rootfs_data 16 MB. How does OEM use them? If you could use kernel and rootfs as they are, and it is only about rootfs_data (the overlay), it might be relatively easy.

I don't understand why you think that there should be an option to still do vendor firmware updates after you have already installed OpenWrt. That kind of functionality is rarity and would be luxury. (In many modern routers you may have to change boot parameters for OpenWrt, so vendor firmware is out in any case.)

The plan is a two step install. First install an initramfs and then use that to sysupgrade. Ram booting the initramfs is of course also an option, but considered for advanced users only since console is necessary then. Installing an OpenWrt initramfs from OEM works. It ends up in one of the kernel partitions. We can't control which one directly. But that doesn't matter.

OEM boots from one of the kernel partitions. We can't change that. But we can force it to one of them after installation, using an ubootenv variable. So the dual firmware scheme is not a problem wrt final state.

I did experiment with a direct install from OEM. It's not impossible, but the dual firmware complicates it. Decided it wasn't worth the effort. Much easier to document the two step procedure for everyone. And then we have free choice of rootfs partition.

Yes. We can do that. What is the proper way to point OpenWrt to another rootfs_data partition?

I didn't mean that it should be supported. Just that it could happen if the user decides to boot OEM. This is an ISP device and it will be remotely managed by default. So there's always the possibility of unexpected upgrades if the user isn't careful (when booting OEM).

But this isn't a real problem. Just a caveat to document.

The end of the last partition is at 7864319, so I believe most of it is assigned to partitions. But most of them are not used for anything AFAICS.

OEM uses partitions up to and including "misc". I haven't seen it use anything after that. But the bootloader does refer to "firmware" and "firmware2" so they could be in use for something I haven't figured out. Likewise with "rootfs_data2". It would make sense if that was related to "rootfs2", and there could be an option to make it so

EDIT. Doh! That's only 4GB. Yes, so you're right. Or there is only 4GB. I have to check that

That's what I was thinking, you could perhaps allocate the rest, and leave the other partitions untouched.
Unless it is 4GB, of course :slight_smile:

I have never done that myself, but there are several routers that define rootfs_data in their dts(i) files. I think that if it is defined and is found, it is then used, and no "split rootfs to /rom and rootfs_data" action will happen.

You might look these through and check if there is anything special in their sysupgrade things:

I think that as OpenWrt uses the names defined in .dts, it could simply work like this:

Modify your DTS definition to match your target partition location. E.g. the large reserved1024M location and size.

That should work for normal flash based routers, as there is no real partition table (outside the DTS), but not quite sure about emmc...

Well, that's the problem. I don't think it's possible to modify any partition attributes like labels or uuids without changing the table on the emmc. Which would work, but I'm trying to avoid

Sysupgrade is easy. It supports direct specification of the data partition on emmc. But mount_root seems to hardcode rootfs_data unless extroot is in use. Which is why I asked about that

Dunno if doable, but root=/dev/mmcblk0pXX passed by boot loader as param ?

Yes, that works and might be the best option. Worrying about the high number if using the last and most attractive partition though. It could go away. But I guess I am overthinking this? If it happens then it won't break existing installations. And we can adapt.

1 Like

I was wrong. There is a blkdevparts cmdline parameter. It has been used for a few OpenWrt devices like for example

This allows redefining the table for OpenWrt without modifying the ondisk version. But it's rather ugly, having to map every partition we care about on the command line. And the feature is not yet in the filogic kernels. I don't think I want to be the first one to start this mess without very good reasons :slight_smile:

I'll just go for the simple root=/dev/mmcblk0p38

Thanks to both of you for sharing thoughts and ideas!

And yes, @frollic was right - half of the emmc is not allocated to any partition at all. It is 8GB as I thought it was:
Disk /dev/mmcblk0: 7.28 GiB, 7818182656 bytes, 15269888 sectors

Thanks for spotting that. The remaining space is empty. So there is definitely room for improvements for advanced users willing to play with this.

1 Like

Well, that is the way we did it before DTS changed things to be more structured...
Old way is still visible in old 18.06 sources.
E.g. wndr3700