Adding support for Ubiquiti UniFi 6 Plus (U6+)

Hmm, failing to find a good example here....

The wifi eeprom data seems to be in /dev/mmcblk0p3 (named "factory" by U-Boot, which makes sense). But how do I point the wifi driver to an emmc partition?

EDIT: Found it, I think.
target/linux/mediatek/filogic/base-files/etc/hotplug.d/firmware/11-mt76-caldata

got wifi with this:

diff --git a/target/linux/mediatek/filogic/base-files/etc/hotplug.d/firmware/11-mt76-caldata b/target/linux/mediatek/filogic/base-files/etc/hotplug.d/firmware/11-mt76-caldata
index c3d7c0997405..a858ace04a76 100644
--- a/target/linux/mediatek/filogic/base-files/etc/hotplug.d/firmware/11-mt76-caldata
+++ b/target/linux/mediatek/filogic/base-files/etc/hotplug.d/firmware/11-mt76-caldata
@@ -14,6 +14,13 @@ case "$FIRMWARE" in
                ;;
        esac
        ;;
+"mediatek/mt7981_eeprom_mt7976_dbdc.bin")
+       case "$board" in
+       ubnt,unifi-6-lite)
+               caldata_extract_mmc "factory" 0x0 0x1000
+               ;;
+       esac
+       ;;
 "mediatek/mt7986_eeprom_mt7976.bin")
        case "$board" in
        acer,predator-w6)

You're right. The cmdline is the real problem. There are no device tree nodes added.

Assuming @daniel is right too, and I believe he always is :slight_smile: , I guess we could just drop the bootloader command line. This works for me without changing mt7981.dtsi:

--- a/target/linux/mediatek/dts/mt7981a-ubnt-unifi-6-plus.dts
+++ b/target/linux/mediatek/dts/mt7981a-ubnt-unifi-6-plus.dts
@@ -16,6 +16,11 @@
 
        chosen {
                stdout-path = "serial0:115200n8";
+               bootargs-override = "console=ttyS0,115200n8";
+       };
+
+       memory {
+               reg = <0 0x40000000 0 0x10000000>;
        };
 
        gpio-keys {

But we lose the ubntbootid=0 which might be useful for dual firmware support? Or is this something else?

I can confirm this is for dual firmware support. I've been probing around the stock firmware, and whenever I do a fwupdate.real -m it toggles ubntbootid between 0 and 1.

Downgrading to 6.5.54:

U6-Plus-BZ.6.5.67# fwupdate.real -m /tmp/BZ.MT7981_6.5.54\+14746.230522.2228.bin 
Writing 'gpt            ' to /dev/mmcblk0  (gpt            ) ...  [%100]
Writing 'bl2            ' to /dev/mmcblk0p1(bl2            ) ...  [%100]
Writing 'u-boot         ' to /dev/mmcblk0p4(u-boot         ) ...  [%100]
Writing 'kernel0        ' to /dev/mmcblk0p7(kernel1        ) ...  [%100]
WARNING: Invalid bootselection magic = 0
Done


Copying /dev/mmcblk0p1 to mmcblk0boot0...
4096+0 records in
4096+0 records out
Done

Upgrading to 6.5.64:

U6-Plus-BZ.6.5.54# fwupdate.real -m /tmp/BZ.MT7981_6.5.64\+14808.230711.0243.bin 
Writing 'gpt            ' to /dev/mmcblk0  (gpt            ) ...  [%100]
Writing 'bl2            ' to /dev/mmcblk0p1(bl2            ) ...  [%100]
Writing 'u-boot         ' to /dev/mmcblk0p4(u-boot         ) ...  [%100]
Writing 'kernel0        ' to /dev/mmcblk0p6(kernel0        ) ...  [%100]
Done


Copying /dev/mmcblk0p1 to mmcblk0boot0...
4096+0 records in
4096+0 records out
Done

Upgrading to 6.5.67:

U6-Plus-BZ.6.5.64# fwupdate.real -m /tmp/BZ.MT7981_6.5.67\+14811.230802.0406.bin 
Writing 'gpt            ' to /dev/mmcblk0  (gpt            ) ...  [%100]
Writing 'bl2            ' to /dev/mmcblk0p1(bl2            ) ...  [%100]
Writing 'u-boot         ' to /dev/mmcblk0p4(u-boot         ) ...  [%100]
Writing 'kernel0        ' to /dev/mmcblk0p7(kernel1        ) ...  [%100]
Done


Copying /dev/mmcblk0p1 to mmcblk0boot0...
4096+0 records in
4096+0 records out
Done

It appears that the normal firmware upgrade process doesn't touch the SPI NOR flash at all. Probably something to keep in mind once we actually try flashing OpenWrt on U6+.


One of the reasons why I'm spelunking in the stock firmware is because apparently /sys/class/leds only partially works. On stock firmware:

U6-Plus-BZ.6.5.67# cd /sys/class/leds
U6-Plus-BZ.6.5.67# echo heartbeat > 'ubnt:white:personality/trigger'
U6-Plus-BZ.6.5.67# echo heartbeat > 'ubnt:blue:personality/trigger'

White LED blinks as expected, blue LED doesn't do anything (wtf?). Yes, the blue LED does physically work. When using UniFi Recovery Mode, the blue LED blinks just fine during TFTP recovery. Considering the blue LED is how UniFi indicates normal operating mode, I'm convinced stock firmware is directly manipulating the GPIOs to operate the LEDs and not through /sys/class/leds.

You guessed right, was trying to figure out mmc_select_hs200 as well. EMMC04G-MK27 does support HS200 mode, in my (limited) embedded Linux experience this error usually means voltages or clocks to eMMC are suspect and needs to be checked. I don't see this error on stock, so either stock is not using HS200 mode or I need to study the stock DTS some more.

I noticed the same. My assumption is that we (and stock) have the wrong GPIO for the blue LED.

EDIT: This works for me (names taken from the U6 Lite for consistency):

	leds {
		compatible = "gpio-leds";

		led_blue: dome-blue {
			label = "blue:dome";
			gpios = <&pio 9 GPIO_ACTIVE_HIGH>;
		};

		led_white: dome-white {
			label = "white:dome";
			gpios = <&pio 34 GPIO_ACTIVE_HIGH>;
		};
	};

Stock firmware apparently does not use HS200 mode at all. I see no references to HS200 or HS400 anywhere in stock DTS. I removed these lines:

mmc-hs200-1_8v;
mmc-hs400-1_8v;
hs400-ds-delay = <0x14014>;

from the &mmc0 node and the mmc_select_hs200 error goes away, while still booting normally.

I say we go for the bootargs-override approach since it's the simplest way to get OpenWrt to not crash at boot and it leaves mt7981.dtsi alone. That's what I'm using now in my work-in-progress OpenWrt build. Also, since I am using bootargs-override, I removed stdout-path from the chosen node and it seems to still work.

Confirmed, blue LED now works.

After figuring out how nvmem-cells work, I think I got the correct MAC address to show up in ip a. I made these changes to mt7981a-ubnt-unifi-6-plus.dts:

// Add eeprom label
eeprom: partition@00000 {
	label = "EEPROM";
	reg = <0x00000 0x10000>;
	read-only;
};
// Add new node
&eeprom {
	compatible = "nvmem-cells";
	#address-cells = <1>;
	#size-cells = <1>;
	macaddr_eeprom_0000: macaddr@0000 {
		reg = <0x0000 0x6>;
	};
};
// Add nvmem-cells properties
gmac1: mac@1 {
	compatible = "mediatek,eth-mac";
	reg = <1>;
	phy-mode = "gmii";
	phy-handle = <&int_gbe_phy>;
	nvmem-cells = <&macaddr_eeprom_0000>;
	nvmem-cell-names = "mac-address";
};

Since we're well on our way with OpenWrt support for this device, I'm going to clean up what I have a bit and start writing something for the OpenWrt wiki. Testing the U6+ "for real" will commence soon after a few more config files that needs to be written, I believe.

Great! This is certainly starting to look almost complete.

uboot-envtools support:

diff --git a/package/boot/uboot-envtools/files/mediatek_filogic b/package/boot/uboot-envtools/files/mediatek_filogic
index b0810184e5af..6d6128ad5802 100644
--- a/package/boot/uboot-envtools/files/mediatek_filogic
+++ b/package/boot/uboot-envtools/files/mediatek_filogic
@@ -47,6 +47,9 @@ mercusys,mr90x-v1)
 netgear,wax220)
        ubootenv_add_uci_config "/dev/mtd1" "0x0" "0x20000" "0x20000"
        ;;
+ubnt,unifi-6-lite)
+       ubootenv_add_uci_config "/dev/mtd1" "0x0" "0x80000" "0x10000"
+       ;;
 xiaomi,mi-router-wr30u-112m-nmbm|\
 xiaomi,mi-router-wr30u-stock|\
 xiaomi,redmi-router-ax6000-stock)

Still missing mac-address support for wifi. I can't figure out how to set separate addresses for the two phys. For reference, this is the stock mac address config on my unit (still running the original 6.5.28 which doesn't seem to be available for DL):

U6-Plus-BZ.6.5.28# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP300> mtu 1500 qdisc pfifo master br0 state DOWN qlen 1000
    link/ether e4:38:83:e5:85:d4 brd ff:ff:ff:ff:ff:ff
3: ra0: <BROADCAST,MULTICAST300> mtu 1500 qdisc pfifo master br0 state DOWN qlen 1000
    link/ether e4:38:83:e5:85:d5 brd ff:ff:ff:ff:ff:ff
4: ra1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:15:85:d5 brd ff:ff:ff:ff:ff:ff
5: ra2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:25:85:d5 brd ff:ff:ff:ff:ff:ff
6: ra3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:35:85:d5 brd ff:ff:ff:ff:ff:ff
7: ra4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:45:85:d5 brd ff:ff:ff:ff:ff:ff
8: ra5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:55:85:d5 brd ff:ff:ff:ff:ff:ff
9: ra6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:65:85:d5 brd ff:ff:ff:ff:ff:ff
10: ra7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:75:85:d5 brd ff:ff:ff:ff:ff:ff
11: rai0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo state UNKNOWN qlen 1000
    link/ether e6:38:83:85:85:d5 brd ff:ff:ff:ff:ff:ff
12: rai1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:95:85:d5 brd ff:ff:ff:ff:ff:ff
13: rai2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:a5:85:d5 brd ff:ff:ff:ff:ff:ff
14: rai3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:b5:85:d5 brd ff:ff:ff:ff:ff:ff
15: rai4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:c5:85:d5 brd ff:ff:ff:ff:ff:ff
16: rai5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:d5:85:d5 brd ff:ff:ff:ff:ff:ff
17: rai6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:e5:85:d5 brd ff:ff:ff:ff:ff:ff
18: rai7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e6:38:83:f5:85:d5 brd ff:ff:ff:ff:ff:ff
19: apcli0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e2:38:83:e5:85:d5 brd ff:ff:ff:ff:ff:ff
20: apclii0: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP300> mtu 1500 qdisc pfifo master br0 state DORMANT qlen 1000
    link/ether ee:38:83:e5:85:d5 brd ff:ff:ff:ff:ff:ff
21: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP200> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether e4:38:83:e5:85:d4 brd ff:ff:ff:ff:ff:ff

The ethernet mac is found directly in the eeprom nor partition as you noted. There's also a locally modified version of it at pos 6. This is the beginning of my "eeprom":

00000000  e4 38 83 e5 85 d4 e6 38  83 e5 85 d4 a6 42 07 77  |.8.....8.....B.w|
00000010  00 04 9d 10 ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000020

The second version can be used to construct one of the wifi addresses along with

mac-address-increment = <1>;

but if I do that then both phys end up with this same address. Is that correct?

The stock device names confuse me. I don't understand what is what there. But I note that all the ra* devices use this eeprom_6+1 address. The apcli* devices use further locallly modified versions. These could be userspace generated?

EDIT: This wasn't entirely correct. The ra0 device uses the eeprom_0+1 address as shown. So I conclude that the two wifi phys should use eeprom_0+1 and eeprom_6+1 .

Any idea what the rest of the mtd-nor EEPROM partition is by the way? There's a blob starting with "UBNT" magic at 8000., and some larger code blobs at 9000 and d0000. Comparing to the U6 Lite I see pretty much the same. Keystores, maybe?

Guess it doesn't really matter. It seems completely unused by OpenWrt for the U6 Lite, except for the mac addresses at the start.

Wondering if this is actually a mt76 bug? Ref:

This would have made a lot more sense if either the call to mt76_eeprom_override() was moved into the if block, or the copy there took the already overridden version.

The problem with the current code, if I read it correctly, is that mt76_eeprom_override() is the final mac fixup for both phys, using of_get_mac_address() on the exact same device tree node. So you end up with duplicate addresses if you attempt to use device tree override on a DBDC device.

Or am I missing something?

EDIT: I created https://github.com/openwrt/mt76/pull/817 after a simple "works for me" test. Should at least provoke an answer to the "how?" question in case I'm wrong

On my device, the stock WiFi MAC addresses are assigned:

eeprom_0 eeprom_6
f4:e2:c6:43:38:a4 f6:e2:c6:43:38:a4
ra{0..7} rai{0..7}
f4:e2:c6:43:38:a5 f6:e2:c6:83:38:a5
f6:e2:c6:13:38:a5 f6:e2:c6:93:38:a5
f6:e2:c6:23:38:a5 f6:e2:c6:a3:38:a5
f6:e2:c6:33:38:a5 f6:e2:c6:b3:38:a5
f6:e2:c6:43:38:a5 f6:e2:c6:c3:38:a5
f6:e2:c6:53:38:a5 f6:e2:c6:d3:38:a5
f6:e2:c6:63:38:a5 f6:e2:c6:e3:38:a5
f6:e2:c6:73:38:a5 f6:e2:c6:f3:38:a5

Based on a sample size of two devices, I posit that the stock MAC address assignment algorithm is:
(EDIT: use bitwise ORs)

eth0 = eeprom_0 # eeprom_6 is redundant
ra0 = eth0 + 00:00:00:00:00:01
temp = ra0 & ff:ff:ff:0f:ff:ff
ra1 = temp | 02:00:00:10:00:00
ra2 = temp | 02:00:00:20:00:00
ra3 = temp | 02:00:00:30:00:00
// and so on...
rai0 = temp | 02:00:00:80:00:00
rai1 = temp | 02:00:00:90:00:00
rai2 = temp | 02:00:00:a0:00:00
// and so on...

The two access point radios are ra[n] and rai[n]. Stock firmware appears to allow up to 8 dual-band WiFi networks. The apcli* interfaces are WiFi client interfaces.

No idea. My wild guess is per-device data for use by the Ubiquiti's UniFi software stack.

I have added this node in my DTS:

&wifi {
	status = "okay";
};

And I have applied your changes to 11-mt76-caldata. Upon booting, I get this dmesg:

[    5.078211] mt798x-wmac 18000000.wifi: eeprom load fail, use default bin
[    5.084991] mt798x-wmac 18000000.wifi: Direct firmware load for mediatek/mt7981_eeprom_mt7976_dbdc.bin failed with error -2
[    5.096115] mt798x-wmac 18000000.wifi: Falling back to sysfs fallback for: mediatek/mt7981_eeprom_mt7976_dbdc.bin

I assume the driver is using the "sysfs fallback", since the calibration data is stored on the eMMC and not on the SPI NOR flash. If this dmesg is expected, I should make a note of it in the wiki entry for this device in case a user peruses the kernel log.

Is there a command or log we can use to verify if the driver is actually using the file provided by 11-mt76-caldata?

I believe the warning is expected.

I guess you can compare /sys/kernel/debug/ieee80211/phy0/mt76/eeprom with the emmc partition to verify that it was actually loaded.

But if it works, then the driver got its eeprom data from somewhere.

Depends on what you believe is touching. I just tested an upgrade myself, running

fwupdate.real -d -m ...

and noticed

Writing U-Boot environment to /dev/mtd1

right before the update of mmcblk0boot0. Which might be because it hardcodes the fdtaddr and fdtcontroladdr addresses? Pretty weird.

The ubntbootid value is also written to the "bs" partition (/dev/mmcblk0p8) similar to how it's done on other Unifi's, using the exact same magic. So after upgrade on my U6+:

U6-Plus-BZ.6.5.64# hexdump -C /dev/mmcblk0p8
00000000  01 00 00 00 2b e8 4d a3  00 00 00 00 00 00 00 00  |....+.M.........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800

Compared to U6 Lite:

root@u6-2:~# hexdump -C /dev/mtd4
00000000  00 00 00 00 2b e8 4d a3  ff ff ff ff ff ff ff ff  |....+.M.........|
00000010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00010000

Or UAP AC Pro:(big endian):

root@unifiac2:~# hexdump -C /dev/mtd6
00000000  00 00 00 00 a3 4d e8 2b  00 00 00 00 00 00 00 00  |.....M.+........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00020000

FWIW, my U6+ came with /dev/mmcblk0p8 being all zeros. I.e no magic or anything. Which obviously still maps to "kernel0". I gues yours did too? I noticed this in your first upgrade:

WARNING: Invalid bootselection magic = 0

and it was gone from the remaining upgrades/downgrades.

The normal OpenWrt Unifi instructions say that you should write 4 zero bytes to "bs" to force booting from "kernel0". But it doesn't actually matter, on the U6+ at least. I just tested

U6-Plus-BZ.6.5.64# dd if=/dev/zero of=/dev/mmcblk0p8
dd: writing '/dev/mmcblk0p8': No space left on device
5+0 records in
4+0 records out

and it booted nicely into the 6.5.28 still left in "kernel0" as expected. Anyway, this confirmd that the bootargs is rewritten on flash on reboot, since the ubntbootid stored there was changed from 1 to 0 by that experiment. The fdtaddr was also modified to match the running firmware, while the fdtcontroladdr didnt change (as expected since U-Boot is still the upgraded version from 6.5.64).

I think this should allow us to install and boot OpenWrt pretty controlled. We can force "kernel0" booting for now.

I've just realized I made a copy-and-paste fail in my DTS, the compatible property needs ubnt,unifi-6-plus, not ubnt,unifi-6-lite :person_facepalming:


After fixing that fail, and the associated case statement in 11-mt76-caldata, I've taken a closer look at the latter. I added a line writing to the kernel log right before the call to caldata_extract_mmc:

"mediatek/mt7981_eeprom_mt7976_dbdc.bin")
	case "$board" in
	ubnt,unifi-6-plus)
		echo "11-mt76-caldata: extracting $FIRMWARE on $board" > /dev/kmsg
		caldata_extract_mmc "factory" 0x0 0x1000
		;;
	esac
	;;

The kernel log shows that caldata_extract_mmc runs after driver initialization:

[    4.947508] mt798x-wmac 18000000.wifi: WM Firmware Version: ____000000, Build Time: 20221208201806
[    4.988177] mt798x-wmac 18000000.wifi: WA Firmware Version: DEV_000000, Build Time: 20221208202048
[    5.079434] mt798x-wmac 18000000.wifi: eeprom load fail, use default bin
[    5.086193] mt798x-wmac 18000000.wifi: Direct firmware load for mediatek/mt7981_eeprom_mt7976_dbdc.bin failed with error -2
[    5.097316] mt798x-wmac 18000000.wifi: Falling back to sysfs fallback for: mediatek/mt7981_eeprom_mt7976_dbdc.bin
[    5.128258] 11-mt76-caldata: extracting mediatek/mt7981_eeprom_mt7976_dbdc.bin on ubnt,unifi-6-plus

Clearly, the order of operations are incorrect here. To further confirm this, I've manually copied the factory data from /dev/mmcblk0p3 and put mt7981_eeprom_mt7976_dbdc.bin into /base-files/lib/firmware/mediatek/ before building the image. The dmesg now looks like:

[    4.957596] mt798x-wmac 18000000.wifi: WM Firmware Version: ____000000, Build Time: 20221208201806
[    4.997486] mt798x-wmac 18000000.wifi: WA Firmware Version: DEV_000000, Build Time: 20221208202048
[    5.088866] mt798x-wmac 18000000.wifi: eeprom load fail, use default bin

which is much more in line with what I expected. How do we get hotplug running earlier in the process of bringing up the radios?

It seems to me then, that the flashing process for U6+ should be very similar to U6 lite, is that right? That means to get sysupgrade working on U6+ we can just copy what U6 lite does, with only minor tweaks needed.

I did notice that too, but was busy with getting the other stuff on U6+ working.

No, I don't think so. The firmware loader looks for the file first and then it calls the script as a fallback. The script runs after that and provides the requested file to the driver. This works fine, but you will get those warnings in the log. There are no messages on success.

I see, I stand corrected. Definitely confusing, but it does mean less things that actually need fixing. :+1:

Not sure about that. The emmc makes this very different. I don't understand how that's supposed to work and integrate with what we inherit from stock firmware.

Unless we add partitions, it seems we have to cope with a single one for kernel and rootfs. And rootfs_data if we do the squashfs+overlay thing. This is possible on mtd due to dynamic partition splitting. But that's not available on emmc, is it? So how do we lay this out?

AFAICS, the partitions we have are kernel0, kernel1, and log. And if we want to be compatible with stock, then we can only use kernel0. And log for persistent "scratch" data.

Do we have any other OpenWrt devices that use eMMC we can look at? Ignoring single-board computer modules like Raspberry Pi CM4, I see this device which has eMMC and uses emmc_do_upgrade in filogic/base-files/lib/upgrade/platform.sh. Perhaps we should be using that?