Adding support for MikroTik hAP ac3 LTE6 kit (D53GR_5HacD2HnD)

Hi there,

Development effort to add support for MikrotTik hAP AC3 LTE6 kit devices.

Hardware: MikroTik RouterBOARD hAP ac 3 LTE6

Github: https://github.com/dchard/openwrt/tree/hap_ac3_lte6_kit

Original ROS 7.2.3 bootlog: https://pastebin.com/KpjhKfcR

What works: UART, USB, Ethernet, Wifi, LTE, sysupgrade (from initramfs, the device needs to be on ROS v6, the Routerboard image as well!).

Any and all help is appreciated!

Upon request, I can provide full dump of the original flash (with ROS) for analysis.

How to install:

  1. Export your MikroTik license key! (via winbox or terminal).
  2. You need to build the image yourself.
  3. Download this modified Tftpd64 image link, that will provide the necessary BOOTP and TFTP server that will be needed for netbooting the device into Openwrt initramfs. Then copy the "openwrt-ipq40xx-mikrotik-mikrotik_hap-ac3-lte6-kit-initramfs-kernel.bin" file to the TFTP server's root, and rename it to "vmlinux". For netbooting, only the first ethernet port can be used. Turn off power, hold the reset button, connect power, and wait for the center led: first it will start blinking green, then it turns solid green, then to solid blue. When it turned to solid blue the reset button can be release, and in Tftpd64 you will see that the unit is downloading the initramfs image called "vmlinux". After this, Openwrt boots to initramfs. The unit default IP is 192.168.1.1, ethernet ports 2-5 can be used. SCP the "openwrt-ipq40xx-mikrotik-mikrotik_hap-ac3-lte6-kit-squashfs-sysupgrade.bin" image to the /tmp directory and issue sysupgrade /tmp/openwrt-ipq40xx-mikrotik-mikrotik_hap-ac3-lte6-kit-squashfs-sysupgrade.bin The unit will flash Openwrt to the device and will boot into Openwrt after this. This takes 2-3 minutes, during the process the red LED will blink.
  4. How to recover to factory firmware: use the MikroTik Netinstall utility.

Update: the device support is added to Openwrt master as of 2022-10-31.

Hi dchard,
Thank you for working on this board!

For modem, you will need to enable the SoC hardware it uses (PCIE, USB?) in DTS.
You can copy the (stock) DTB from NOR, and use dtc to convert it to a DTS to see if there is hardware missed. It will be a little different for the drivers they use. Also worth checking the RouterOS v7 GPL dump.

It helps to style your add-device commit similar to existing commits: example https://github.com/openwrt/openwrt/commit/7f54bf6fe2ac331d018cd273bb1abe04493b5457
Then git rebase master into your working branch, rather than merge.

RouterBOOT v7 does not load the OpenWrt mikrotik minor NOR layout.
Easy path is to downgrade the primary RouterBOOT to v6.
Better path is to work out what RouterBOOT v7 is looking for, and adjust our boot process to suit.
Look at the filesystem, offsets, and filenames on stock firmware, or decompress (UCL NRV2B) RouterBOOT and decompile it to see what it wants to load.

Hi,

As much as I know, the mini PCIe socket is USB only, so theoretically we dont need PICe in the DTS. Although in the original DTS does have PCI part in it. What we might need for the modem part is the enabble/disable GPIO legs:

Registered gpio 44 as usb-power-off
Registered gpio 51 as mpcie-power-off
Registered gpio 45 as lte-ant-sw1
Registered gpio 46 as lte-ant-sw2
Registered gpio 49 as lte-reset

Donwgrading to ROS6 is going to be the quick solution for now, and when the HW itself works fully, we can further dig in to ROS7 flash support.

To be honest, I am looking at the full flash dump, and I can't even find where the hell the kernel should be on the ROS7 dump...

Dont trust the DTB that is shipped on the devices, its just a hacked up reference DTS that is good enough to just get it booting and is wrong more times than its right

@robimarko

I donwgraded to ROS 6.49.2 (routerboot as well), and the first 0x10000 part of the "firmware" partition is still completely empty. So the offset is still there, now I am going to try and sysupgrade.

OK, the good news is: we have a bootable image that can be flashed.

The modem is still not showing up, so probably some more hacking needs to be done with the GPIO PINs.

On the modem side:

The mikrotik R11e-LTE6 modem incorporates a baseband vendor called ASR. The chip is ASR18xx, likely the ASR1826, I can't confirm for sure, as the letters and numbers are very faint... Non the less, I have no idea if there is an official linux driver for this at all.

MOD: I installed a Quectel EC25 modem into the mPCIe slot, relevant drivers are present yet the unit is not detecting the device.

I removed the modem, and started measuring on the mPCIe interface: no power rails are present at all. So likely something is still missing.

MOD2: after toggling the "mpcie_power" GPIO leg, 3.3 Volt is present:

echo 463 > /sys/class/gpio/export
echo out > /sys/class/gpio/gpio463/direction
echo 1 > /sys/class/gpio/gpio463/value

MOD3: even after "mpcie_power" is set, power is present on the mPCIe slot, the modem is no recognized. So likely something on the secondary USB interface is not right, although both host controllers are detected:

Bus 002 Device 001: ID 1d6b:0003 Linux 5.15.68 xhci-hcd xHCI Host Controller
Bus 001 Device 001: ID 1d6b:0002 Linux 5.15.68 xhci-hcd xHCI Host Controller

The issue is likely with missing USB definitions in the DTS.

OWRT dmesg:

xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
xhci-hcd xhci-hcd.0.auto: hcc params 0x0220f665 hci version 0x100 quirks 0x0000002002010010
xhci-hcd xhci-hcd.0.auto: irq 104, io mem 0x06000000
xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
hub 2-0:1.0: USB hub found
hub 2-0:1.0: config failed, hub doesn't have any ports! (err -19)

ROS dmesg:

xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
xhci-hcd xhci-hcd.0.auto: hcc params 0x0228f665 hci version 0x100 quirks 0x0000000002010010
xhci-hcd xhci-hcd.0.auto: irq 112, io mem 0x08a00000
xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
xhci-hcd xhci-hcd.1.auto: hcc params 0x0220f665 hci version 0x100 quirks 0x0000000002010010
xhci-hcd xhci-hcd.1.auto: irq 113, io mem 0x06000000
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 1 port detected
xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
hub 4-0:1.0: USB hub found
hub 4-0:1.0: config failed, hub doesn't have any ports! (err -19)
usb 1-1: new high-speed USB device number 2 using xhci-hcd
usb 3-1: new high-speed USB device number 2 using xhci-hcd

Again, you need to enable the second USB controller as there is a user-facing USB port as well

Yeah I realized that, what I am not sure about is how to do that :slight_smile:

Just add:

&usb3_ss_phy {
	status = "okay";
};

&usb3_hs_phy {
	status = "okay";
};

&usb3 {
	status = "okay";
};

&usb2_hs_phy {
	status = "okay";
};

&usb2 {
	status = "okay";
};

Remove existing USB nodes and just add it to the bottom

1 Like

Did that, I also added:

        enable-mpcie-power {
                gpio-hog;
                gpios = <51 GPIO_ACTIVE_HIGH>;
                output-high;
                line-name = "enable mPCI-E power";
        };

so I dont need to turn it on manually each time.

The extra USB interface shows up (it matches the ROS dmesg output), but still no sign of the modem. Again I tried with a Quectel EC25 as well, and nothing shows up.

This is how it looks now:

root@OpenWrt:/# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/0p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M

Some more digging.

Under ROS I reset the modem and it is for sure USB:

[  359.251936] usb 1-1: USB disconnect, device number 3
[  359.252240] rndis_host 1-1:1.0 wwan0: unregister 'rndis_host' usb-xhci-hcd.0.auto-1, RNDIS device
[  359.905215] usb 1-1: new high-speed USB device number 4 using xhci-hcd
[  383.112927] usb 1-1: USB disconnect, device number 4
[  383.805206] usb 1-1: new high-speed USB device number 5 using xhci-hcd
[  410.328212] usb 1-1: USB disconnect, device number 5
[  410.725193] usb 1-1: new high-speed USB device number 6 using xhci-hcd
[  410.932246] rndis_host 1-1:1.0: hard mtu 1559 (1580 from dev), rx buflen 1728, align 1
[  410.933186] cdc_acm 1-1:1.2: ttyACM0: USB ACM device
[  410.945068] cdc_acm 1-1:1.4: ttyACM1: USB ACM device
[  410.946089] rndis_host 1-1:1.0 wwan0: register 'rndis_host' at usb-xhci-hcd.0.auto-1, RNDIS device, ac:50:43:1a:ee:fd

PCIe modems have two hardware control inputs, usually one resets the modem / holds it in reset and the other puts it in hardware RFKILL "airplane mode". These would likely be connected to GPIO pins.

From the ROS bootlog we know that there is a PIN called Registered gpio 49 as lte-reset

I already tried to toggle this, no luck so far.

mPCIE gpio relay might be ACTIVE_LOW instead of high?
Also might be worth not hogging these, so that you can switch them. See base-files/etc/board.d/03_gpio_switches

If you can give me an example, happy to try. Although when it was not hogged, there were no 3.3Volts on the mPCIe, now that it is hogged, now there is 3.3Volts. But this active high/low theory might be interesting for the lte_reset_pin?

More data from ROS:

/sys/firmware/devicetree/base/soc # ls -la
total 0
-r--r--r--    1 0        0                4 Sep 21 20:38 #address-cells
-r--r--r--    1 0        0                4 Sep 21 20:38 #size-cells
drwxr-xr-x   36 0        0                0 Sep 21 20:38 .
drwxr-xr-x   14 0        0                0 Sep 21 20:16 ..
drwxr-xr-x    2 0        0                0 Sep 21 20:38 clock-controller@1800000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 clock-controller@b088000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 clock-controller@b098000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 clock-controller@b0a8000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 clock-controller@b0b8000
-r--r--r--    1 0        0               11 Sep 21 20:38 compatible
drwxr-xr-x    2 0        0                0 Sep 21 20:38 dma@7884000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 edma@c080000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 ess-switch@c000000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 interrupt-controller@b000000
-r--r--r--    1 0        0                4 Sep 21 20:38 name
drwxr-xr-x    2 0        0                0 Sep 21 20:38 pci@40000000
drwxr-xr-x    5 0        0                0 Sep 21 20:38 pinctrl@1000000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 qcom,nand@7980000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 qcom,sps
drwxr-xr-x    2 0        0                0 Sep 21 20:38 qcrypto
-r--r--r--    1 0        0                0 Sep 21 20:38 ranges
drwxr-xr-x    2 0        0                0 Sep 21 20:38 regulator@b089000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 regulator@b099000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 regulator@b0a9000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 regulator@b0b9000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 restart@4ab000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 rng@22000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 sdhci@7824000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 serial@78af000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 serial@78b0000
drwxr-xr-x    3 0        0                0 Sep 21 20:38 spi@78b5000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 tcsr@1949000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 tcsr@1957000
drwxr-xr-x    3 0        0                0 Sep 21 20:38 usb@60f8800
drwxr-xr-x    3 0        0                0 Sep 21 20:38 usb@8af8800
drwxr-xr-x    2 0        0                0 Sep 21 20:38 usbphy@9a000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 usbphy@a6000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 usbphy@a8000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 watchdog@b017000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 wifi@a000000
drwxr-xr-x    2 0        0                0 Sep 21 20:38 wifi@a800000

@johnth this is from ROS, and it seems you are right (LTE modem works during this readout):

/dev # cat /sys/class/gpio/lte-reset/value
0
/dev # cat /sys/class/gpio/lte-reset/active_low
0
/dev # cat /sys/class/gpio/lte-ant-sw1/value
0
/dev # cat /sys/class/gpio/lte-ant-sw1/active_low
0
/dev # cat /sys/class/gpio/lte-ant-sw2/active_low
0
/dev # cat /sys/class/gpio/lte-ant-sw2/value
0
/dev # cat /sys/class/gpio/lte-reset/value
0
/dev # cat /sys/class/gpio/lte-reset/active_low
0
/dev # cat /sys/class/gpio/usb-power-off/value
0
/dev # cat /sys/class/gpio/usb-power-off/active_low
1
/dev #
/dev # cat /sys/class/gpio/mpcie-power-off/value
0
/dev # cat /sys/class/gpio/mpcie-power-off/active_low
1

Not really.
It looks like modem works with: lte-reset GPIO low, & mpcie-power-off GPIO high
Mikrotik just define mpcie-power-off as GPIO_ACTIVE_LOW, so that they can set (in software) power-off, and it takes the gpio output / voltage from high to low.