Support for Mikrotik RB3011UiAS-RM?

For Developers
222

An important moment. I worked on RB3011 and improved my program for analyzing the structure of a mikrotik's NOR: mikrotik_x_config_reader
Now it can find an MIBIB - Multi-Image Boot Info Block and decode it in DTS like format.

for RB450Gx4 
	SBL1@0 {
		label = "SBL1";
		reg = <0x00000000 0x210000>;
		read-only;
	};
	MIBIB@210000 {
		label = "MIBIB";
		reg = <0x00210000 0x20000>;
		read-only;
	};
	QSEE@230000 {
		label = "QSEE";
		reg = <0x00230000 0x540000>;
		read-only;
	};
	CDT@770000 {
		label = "CDT";
		reg = <0x00770000 0x10000>;
		read-only;
	};
	APPSBLENV@780000 {
		label = "APPSBLENV";
		reg = <0x00780000 0x80000>;
		read-only;
	};
	APPSBL@800000 {
		label = "APPSBL";
		reg = <0x00800000 0xffff0000>;
		read-only;
	};
for LHGG60-ad 
		label = "SBL1";
		reg = <0x00000000 0x210000>;
		read-only;
	};
	MIBIB@210000 {
		label = "MIBIB";
		reg = <0x00210000 0x20000>;
		read-only;
	};
	QSEE@230000 {
		label = "QSEE";
		reg = <0x00230000 0x540000>;
		read-only;
	};
	CDT@770000 {
		label = "CDT";
		reg = <0x00770000 0x90000>;
		read-only;
	};
	APPSBL@800000 {
		label = "APPSBL";
		reg = <0x00800000 0xffff0000>;
		read-only;
	};
for RB3011 
	SBL1@0 {
		label = "SBL1";
		reg = <0x00000000 0x20000>;
		read-only;
	};
	MIBIB@20000 {
		label = "MIBIB";
		reg = <0x00020000 0x20000>;
		read-only;
	};
	SBL2@40000 {
		label = "SBL2";
		reg = <0x00040000 0x40000>;
		read-only;
	};
	SBL3@80000 {
		label = "SBL3";
		reg = <0x00080000 0x80000>;
		read-only;
	};
	DDRCONFIG@100000 {
		label = "DDRCONFIG";
		reg = <0x00100000 0x10000>;
		read-only;
	};
	SSD@110000 {
		label = "SSD";
		reg = <0x00110000 0x10000>;
		read-only;
	};
	TZ@120000 {
		label = "TZ";
		reg = <0x00120000 0x80000>;
		read-only;
	};
	RPM@1a0000 {
		label = "RPM";
		reg = <0x001a0000 0x20000>;
		read-only;
	};
	APPSBL@1c0000 {
		label = "APPSBL";
		reg = <0x001c0000 0x40000>;
		read-only;
	};
	APPSBLENV@200000 {
		label = "APPSBLENV";
		reg = <0x00200000 0x40000>;
		read-only;
	};
	ART@240000 {
		label = "ART";
		reg = <0x00240000 0x40000>;
		read-only;
	};
	NSS0@280000 {
		label = "NSS0";
		reg = <0x00280000 0x100000>;
		read-only;
	};
	NSS1@380000 {
		label = "NSS1";
		reg = <0x00380000 0x100000>;
		read-only;
	};
	HLOS@480000 {
		label = "HLOS";
		reg = <0x00480000 0x200000>;
		read-only;
	};
	rootfs@680000 {
		label = "rootfs";
		reg = <0x00680000 0xffff0000>;
		read-only;
	};
223

Awesome job.
It looks like LHGG60-ad has some other weird MDIO offset as 0x18 does not work.
I also tried couple of other values but no MDIO is detected

224

At a Mikrotik each device with the trick. Unfortunately, I do not have an LHGG60-ad so I cannot help in this matter. To solve this riddle, we need to jailbreak the device and implement the software mdio sniffer which I did for the RB450Gx4.

225

Yes, but the issue is that I dont know how to jailbreak this one as it does not have SD card slot or USB and it runs 6.42.1 at minimum and only exploit I know works on 6.42 version at max.
There is also some weird stuff going on with PCI-E too.
It simply refuses to go UP, only thing I can dig is that there is some trick called gpio-booster on GPIO66.
I tried exporting the GPIO as both ACTIVE_LOW and ACTIVE_HIGH and no difference.
It looks like they patched it in the PCI driver, but I am still not sure what it actually does

+	ret = of_get_named_gpio_flags(np, "gpio-booster", 0, &flags);
+	if (ret >= 0) {
+		unsigned gpio_num = ret;
+		unsigned val_on = !(flags & OF_GPIO_ACTIVE_LOW);
+		ret = gpio_request(gpio_num, pdev->name);
+		if (ret >= 0) {
+			msm_pcie_dev.gpio_booster = gpio_num;
+//			msm_pcie_dev.gpio_booster_on = val_on;
+			gpio_direction_output(gpio_num, !val_on);
+			printk("pcie booster on gpio %d\n", gpio_num);
+		}
+	}
226

Oh, so in the GPL sources of the Mikrotik, everything is there!

+static int msm_pcie_enable(struct msm_pcie_dev_t *dev, u32 options)
.........................
+	retries = 0;
+
+	while ((!(val & XMLH_LINK_UP) ||
+		!msm_pcie_confirm_linkup(dev, false, false))
+		&& (retries < MAX_LINK_RETRIES)) {
+		if (val_last != val) {
+			val_last = val;
+			PCIE_ERR(dev, "RC%d:No. %ld:LTSSM_STATE:0x%x\n",
+				 dev->rc_idx,
+				 retries + 1, (val >> 0xC) & 0x1f);
+		}
+		msm_pcie_dev_reset(dev, 1);
+//		gpio_set_value(dev->gpio_rst, 1);
+		usleep_range(PERST_PROPAGATION_DELAY_US_MIN,
+					 PERST_PROPAGATION_DELAY_US_MAX);
+		msm_pcie_dev_reset(dev, 0);
+//		gpio_set_value(dev->gpio_rst, 0);
+		usleep_range(LINK_RETRY_TIMEOUT_US_MIN,
+				LINK_RETRY_TIMEOUT_US_MAX);
+		retries++;
+		if (retries == MAX_LINK_RETRIES && dev->gpio_booster) {
+			if (jiffies_end == 0) {
+				jiffies_end = jiffies + 150 * HZ;
+				gpio_set_value(dev->gpio_booster, 1);
+				printk("enable pcie booster\n");
+			}
+			else if (time_after(jiffies, jiffies_end)) {
+				break;
+			}
+			--retries;
+		}
+		val =  readl_relaxed(dev->elbi + PCIE20_ELBI_SYS_STTS);
227

It looks like some kind of hardware bug and an attempt to programmatically fix it ...
In the while retry cycle, it is checked if the link is up in the pcie bus and if the retry cycle has passed MAX_LINK_RETRIES, then the booster gpio is set to 1. At the end of the function, the booster turns off again. It seems that he either increases the supply voltage or something similar. But it seems like it's impossible to turn it on for a long time(>150 sec)! You can burn something!?

228

Hm, well that would explain why the heatsink was getting really hot after couple of minutes with GPIO exported.
Luckily it looks like nothing has died as RouterOS still boots fine and 60GHz card is detected and active.

So basically, I gotta try to patch that retry function into upstream driver somehow.
But there appears to be something else since according to suppout extracted there is newer "enable pcie booster" printed

18.63@0: libata version 3.00 loaded.
		 18.63@0: pcie booster on gpio 66
		 18.63@0: msm_pcie_enable: PCIe: trigger the reset of endpoint of RC0.
		 18.63@0: msm_pcie_enable: PCIe RC0 PHY is ready!
		 18.63@0: msm_pcie_enable: PCIe: Release the reset of endpoint of RC0.
		 18.63@0: msm_pcie_enable: PCIe RC0 link initialized
		 18.63@0: PCI host bridge to bus 0000:00
		 18.63@0: pci_bus 0000:00: root bus resource [io  0x40200000-0x402fffff]
		 18.63@0: pci_bus 0000:00: root bus resource [mem 0x40300000-0x40ffffff]
		 18.63@0: pci 0000:00:00.0: [17cb:1001] type 1 class 0x000604
		 18.63@0: pci 0000:00:00.0: reg 10: [mem 0x00000000-0x00000fff 64bit]
		 18.63@0: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
		 18.63@0: PCI: bus0: Fast back to back transfers disabled
		 18.63@0: pci 0000:01:00.0: [1ae9:0310] type 0 class 0x000280
		 18.63@0: pci 0000:01:00.0: reg 10: [mem 0x00000000-0x001fffff 64bit]
		 18.63@0: PCI: bus1: Fast back to back transfers disabled
		 18.63@0: pci 0000:00:00.0: BAR 8: assigned [mem 0x40400000-0x405fffff]
		 18.63@0: pci 0000:00:00.0: BAR 0: assigned [mem 0x40300000-0x40300fff 64bit]
		 18.63@0: pci 0000:01:00.0: BAR 0: assigned [mem 0x40400000-0x405fffff 64bit]
		 18.63@0: pci 0000:00:00.0: PCI bridge to [bus 01-01]
		 18.63@0: pci 0000:00:00.0:   bridge window [mem 0x40400000-0x405fffff]
		 18.63@0: pci 0000:00:00.0: PCI bridge to [bus 01-01]
		 18.63@0: pci 0000:00:00.0:   bridge window [mem 0x40400000-0x405fffff]
		 18.63@0: PCI: enabling device 0000:00:00.0 (0144 -> 0147)
		 18.63@0: msm_pcie_probe: RC0 is enabled in bootup
229

Apparently the trick is that the booster needs to be turned on, wait for the link on the bus to be up and then immediately turn it off.

230

Well it looks like it.
But weird thing is that with the booster active PCI-E still does not come up in OpenWrt.
Those wil6210 cards most likely can draw a lot of power and they had to hack this in.

I am gonna try something similar, gotta add a retry mechanism as currently it looks like driver attempts to probe only once.

231

Theres gotta be something else too.
I hacked in booster GPIO and its turned on by probe function.
Even gave the link function 5 retries and nothing still.
Maybe probe is executed later than link?

From ebb4d06a95be89797591e29a37b831f0c13c7447 Mon Sep 17 00:00:00 2001
From: Robert Marko <robimarko@gmail.com>
Date: Wed, 6 Mar 2019 20:56:22 +0100
Subject: [PATCH] ipq4019: Add PCI-E probe retry

Signed-off-by: Robert Marko <robimarko@gmail.com>
---
 pcie-qcom.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
index 4352c1c..84dabc6 100644
--- a/drivers/pci/controller/dwc/pcie-qcom.c
+++ b/drivers/pci/controller/dwc/pcie-qcom.c
@@ -165,6 +165,7 @@ struct qcom_pcie {
 	union qcom_pcie_resources res;
 	struct phy *phy;
 	struct gpio_desc *reset;
+	struct gpio_desc *booster;
 	const struct qcom_pcie_ops *ops;
 };
 
@@ -1088,6 +1089,7 @@ static int qcom_pcie_host_init(struct pcie_port *pp)
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 	struct qcom_pcie *pcie = to_qcom_pcie(pci);
 	int ret;
+	int retries = 0;
 
 	pm_runtime_get_sync(pci->dev);
 	qcom_ep_reset_assert(pcie);
@@ -1113,7 +1115,10 @@ static int qcom_pcie_host_init(struct pcie_port *pp)
 
 	qcom_ep_reset_deassert(pcie);
 
-	ret = qcom_pcie_establish_link(pcie);
+	do{
+		ret = qcom_pcie_establish_link(pcie);
+		retries++;
+	}while(retries <=5);
 	if (ret)
 		goto err;
 
@@ -1228,6 +1233,12 @@ static int qcom_pcie_probe(struct platform_device *pdev)
 	if (IS_ERR(pcie->reset))
 		return PTR_ERR(pcie->reset);
 
+	pcie->booster = devm_gpiod_get_optional(dev, "booster", GPIOD_OUT_LOW);
+	if (IS_ERR(pcie->booster))
+		return PTR_ERR(pcie->booster);
+
+	gpiod_direction_output(pcie->booster, 1);
+
 	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "parf");
 	pcie->parf = devm_ioremap_resource(dev, res);
 	if (IS_ERR(pcie->parf))
-- 
2.20.1
1 Like
232

Sorry for the spam. No, I haven't had the "pleasure" to deal with that. But it looks like you all are better equipped at figuring this out too. I don't think I would be much of a help here.
At least it

233

No need to be sorry.
I tried couple of variants, but I have hit a wall and have no idea how to get bloody PCI working.
Mikrotik did a lot of stuff that is hard to replicate in the upstream driver.
GPIO is pulled HIGH by probe but even with 20 retries in link function it will not link UP.
I am sure that I am missing something, but have no idea what

234

Well, I can say I have no idea either. If the WIL6210 is as heavy duty as it's said to be, it might have an external power supply as well that needs to be enabled. I've seen stuff like this on the Meraki MR32 where both BCM94352 11ac chips have to be enabled via a gpio. This has to be done before the pcie link comes up, otherwise it's the same no-show as you are describing. What helped me track this down was that I could toggle the gpios from the stock firmware. So when I disabled the power by setting the gpio to 0, the wifis just "disappeared" (along with the driver crashing). But this is all different hardware, I don't want to sent you off to a wild goose chase.

235

I think that your code does not exactly what we need.
Here I sketched an approximate code (not tested!):

--- a/pcie-qcom.c	2019-02-15 11:09:54.000000000 +0300
+++ b/pcie-qcom.c	2019-03-07 03:08:44.437941017 +0300
@@ -1088,6 +1088,12 @@
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 	struct qcom_pcie *pcie = to_qcom_pcie(pci);
 	int ret;
+	struct gpio_desc *booster;
+	int retries = 0;
+
+	booster = devm_gpiod_get_optional(dev, "booster", GPIOD_OUT_LOW);
+	if (IS_ERR(pcie->booster))
+		return PTR_ERR(pcie->booster);
 
 	pm_runtime_get_sync(pci->dev);
 	qcom_ep_reset_assert(pcie);
@@ -1113,7 +1119,22 @@
 
 	qcom_ep_reset_deassert(pcie);
 
-	ret = qcom_pcie_establish_link(pcie);
+	do{
+		ret = qcom_pcie_establish_link(pcie);
+		if(!ret){
+			break;
+		}else{
+			qcom_ep_reset_assert(pcie);
+			usleep_range(1000, 1005);
+			qcom_ep_reset_deassert(pcie);
+			usleep_range(20000, 25000);
+			retries++;
+			if(retries == 5){
+				gpiod_direction_output(booster, 1);
+			}
+		}
+	}while(retries <= 10);
+	gpiod_direction_output(booster, 0);
 	if (ret)
 		goto err;
236

Try using this metod to jailbreak lhg.

ps: you two are doing some nice work here.

237

I already tried something like that, but you fetch anything from OF there.
You get this nice error

drivers/pci/controller/dwc/pcie-qcom.c: In function 'qcom_pcie_host_init':
drivers/pci/controller/dwc/pcie-qcom.c:1094:36: error: 'dev' undeclared (first use in this function); did you mean 'cdev'?
  booster = devm_gpiod_get_optional(dev, "booster", GPIOD_OUT_LOW);
                                    ^~~
                                    cdev
drivers/pci/controller/dwc/pcie-qcom.c:1094:36: note: each undeclared identifier is reported only once for each function it appears in
drivers/pci/controller/dwc/pcie-qcom.c:1095:17: error: 'struct qcom_pcie' has no member named 'booster'
  if (IS_ERR(pcie->booster))
                 ^~
drivers/pci/controller/dwc/pcie-qcom.c:1096:22: error: 'struct qcom_pcie' has no member named 'booster'
   return PTR_ERR(pcie->booster);
238

Hello. Try this one:

diff -rNu a/pcie-qcom.c b/pcie-qcom.c
--- a/pcie-qcom.c	2019-02-15 11:09:54.000000000 +0300
+++ b/pcie-qcom.c	2019-03-07 13:25:28.527003218 +0300
@@ -1088,6 +1088,12 @@
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 	struct qcom_pcie *pcie = to_qcom_pcie(pci);
 	int ret;
+	struct gpio_desc *booster;
+	int retries = 0;
+
+	booster = devm_gpiod_get_optional(pci->dev, "booster", GPIOD_OUT_LOW);
+	if (IS_ERR(booster))
+		return PTR_ERR(booster);
 
 	pm_runtime_get_sync(pci->dev);
 	qcom_ep_reset_assert(pcie);
@@ -1113,7 +1119,22 @@
 
 	qcom_ep_reset_deassert(pcie);
 
-	ret = qcom_pcie_establish_link(pcie);
+	do{
+		ret = qcom_pcie_establish_link(pcie);
+		if(!ret){
+			break;
+		}else{
+			qcom_ep_reset_assert(pcie);
+			usleep_range(1000, 1005);
+			qcom_ep_reset_deassert(pcie);
+			usleep_range(20000, 25000);
+			retries++;
+			if(retries == 5){
+				gpiod_direction_output(booster, 1);
+			}
+		}
+	}while(retries <= 10);
+	gpiod_direction_output(booster, 0);
 	if (ret)
 		goto err;
239

Actually,a slightly modified your first example somewhat worked.

From ebb4d06a95be89797591e29a37b831f0c13c7447 Mon Sep 17 00:00:00 2001
From: Robert Marko <robimarko@gmail.com>
Date: Wed, 6 Mar 2019 20:56:22 +0100
Subject: [PATCH] ipq4019: Add PCI-E probe retry

Signed-off-by: Robert Marko <robimarko@gmail.com>
---
 pcie-qcom.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
index 4352c1c..84dabc6 100644
--- a/drivers/pci/controller/dwc/pcie-qcom.c
+++ b/drivers/pci/controller/dwc/pcie-qcom.c
@@ -165,6 +165,7 @@ struct qcom_pcie {
 	union qcom_pcie_resources res;
 	struct phy *phy;
 	struct gpio_desc *reset;
+	struct gpio_desc *booster;
 	const struct qcom_pcie_ops *ops;
 };
 
@@ -1088,6 +1089,7 @@ static int qcom_pcie_host_init(struct pcie_port *pp)
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 	struct qcom_pcie *pcie = to_qcom_pcie(pci);
 	int ret;
+	int retries = 0;
 
 	pm_runtime_get_sync(pci->dev);
 	qcom_ep_reset_assert(pcie);
@@ -1113,9 +1115,22 @@ static int qcom_pcie_host_init(struct pcie_port *pp)
 
 	qcom_ep_reset_deassert(pcie);
 
-	ret = qcom_pcie_establish_link(pcie);
-	if (ret)
-		goto err;
+	do{
+		ret = qcom_pcie_establish_link(pcie);
+		if(!ret){
+			break;
+		}else{
+			qcom_ep_reset_assert(pcie);
+			usleep_range(1000, 1005);
+			qcom_ep_reset_deassert(pcie);
+			usleep_range(20000, 25000);
+			retries++;
+			if(retries == 5){
+				gpiod_direction_output(pcie->booster, 1);
+			}
+		}
+	}while(retries <= 10);
+	gpiod_direction_output(pcie->booster, 0);
 
 	return 0;
 err:
@@ -1228,6 +1243,10 @@ static int qcom_pcie_probe(struct platform_device *pdev)
 	if (IS_ERR(pcie->reset))
 		return PTR_ERR(pcie->reset);
 
+	pcie->booster = devm_gpiod_get_optional(dev, "booster", GPIOD_OUT_LOW);
+	if (IS_ERR(pcie->booster))
+		return PTR_ERR(pcie->booster);
+
 	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "parf");
 	pcie->parf = devm_ioremap_resource(dev, res);
 	if (IS_ERR(pcie->parf))
-- 
2.20.1

Linkup happens and root bridge is detected, but wil6210 is not detected by lspci.
Maybe booster is needed longer.

[    0.178168] qcom-pcie 40000000.pci: host bridge /soc/pci@40000000 ranges:
[    0.178209] qcom-pcie 40000000.pci:    IO 0x40200000..0x402fffff -> 0x40200000
[    0.178233] qcom-pcie 40000000.pci:   MEM 0x40300000..0x40ffffff -> 0x40300000
[    1.297669] qcom-pcie 40000000.pci: Phy link never came up
[    2.357659] qcom-pcie 40000000.pci: Phy link never came up
[    3.417658] qcom-pcie 40000000.pci: Phy link never came up
[    4.477658] qcom-pcie 40000000.pci: Phy link never came up
[    5.537658] qcom-pcie 40000000.pci: Phy link never came up
[    6.597658] qcom-pcie 40000000.pci: Phy link never came up
[    7.657658] qcom-pcie 40000000.pci: Phy link never came up
[    8.717658] qcom-pcie 40000000.pci: Phy link never came up
[    9.777658] qcom-pcie 40000000.pci: Phy link never came up
[   10.837658] qcom-pcie 40000000.pci: Phy link never came up
[   11.897658] qcom-pcie 40000000.pci: Phy link never came up
[   11.957814] qcom-pcie 40000000.pci: PCI host bridge to bus 0000:00
[   11.957835] pci_bus 0000:00: root bus resource [bus 00-ff]
[   11.957852] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x40200000-0x402fffff])
[   11.957865] pci_bus 0000:00: root bus resource [mem 0x40300000-0x40ffffff]
[   11.959558] PCI: bus0: Fast back to back transfers disabled
[   11.960935] PCI: bus1: Fast back to back transfers enabled
[   11.960987] pci 0000:00:00.0: BAR 0: assigned [mem 0x40300000-0x40300fff 64bit]
[   11.961012] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[   11.963386] pcieport 0000:00:00.0: AER enabled with IRQ 28

root@OpenWrt:/# lspci -v
00:00.0 PCI bridge: Qualcomm Device 1001 (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 28
        Memory at 40300000 (64-bit, non-prefetchable) [size=4K]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: None
        Prefetchable memory behind bridge: None
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] L1 PM Substates
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12
240

Try to play with delays: usleep_range(20000, 25000)
Try to remove all usleep_range(because qcom_ep_reset_X has it own delay) or vice versa add more long delays.

241

I played with the usleep and went up to usleep_range(20000, 60000); but still nothing.
Next thing would be to try extending the reset functions delay altough current numbers are in line with Mikrotik GPL driver, its like something else is missing