MikroTik RB5009UPr+S+IN (RB5009UG with PoE)

For Developers
1

Latest update (separate UG and UPr models):
OpenWrt branch rb5009upr+s+in
Packages branch mtpoe-ctrl-rb5009

This is a split from Add support for MikroTik RB5009UG - #1131 by v01t to focus on the PoE model.

The recent proposed OpenWrt tree changes provide a common build for base RB5009 and PoE-featured version RB5009Upr+S+IN (same bootloader, compatible sysupgrade, devices selectable separately in the menuconfig whilst producing the same image):

Those give access to the on-board PoE hardware but do not include the mtpoe-ctrl tool, which belongs to the packages/feeds. The tool can be included so it can be part of the image build result:

Note that those gather other people work with necessary changes to run on RB5009 and aligning all this together.

Build
https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem
Steps below refer to the direct git branches but obviously changes can be taken and applied as patches.

git clone --depth 1 https://github.com/prudy/openwrt.git -b rb5009-with-poe
cd openwrt

# this gets therepo with already added tool, but instead only the mtpoe-ctrl dir can be put into local packages
sed '/git packages /s/https.*$/https:\/\/github.com\/prudy\/openwrt-packages.git;mtpoe-ctrl-rb5009/' feeds.conf.default > feeds.conf

./scripts/feeds update -a
./scripts/feeds install -a
cp rb5009-with-poe.diffconfig .config
make defconfig
make menuconfig
# verify
make -j

Flashing
Since all matches base rb5009, simply follow the procedure from https://openwrt.org/toh/mikrotik/rb5009ug_s_in.
I don’t use usb so I load u-boot and initramfs with loader.sh script, run in bin/targets/mvebu/cortexa72/.

loader.sh script

#!/bin/bash
IFNAME=eth0
/sbin/ip addr replace 192.168.88.2/24 dev $IFNAME
/sbin/ip link set dev $IFNAME up
/usr/sbin/dnsmasq \
--no-daemon \
--listen-address 192.168.88.2 \
--bind-interfaces \
-p0 \
--dhcp-authoritative \
--dhcp-range=192.168.88.100,192.168.88.200 \
--bootp-dynamic \
--dhcp-boot=u-boot-rb5009/u-boot.elf \
--log-dhcp \
--enable-tftp \
--tftp-root=$(pwd)

example rb5009-with-poe.diffconfig

CONFIG_TARGET_mvebu=y
CONFIG_TARGET_mvebu_cortexa72=y
CONFIG_TARGET_mvebu_cortexa72_DEVICE_mikrotik_rb5009=y
CONFIG_PACKAGE_cgi-io=y
CONFIG_PACKAGE_kmod-spi-dev=y
CONFIG_PACKAGE_libiwinfo=y
CONFIG_PACKAGE_libiwinfo-data=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-ucode=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-app-opkg=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-light=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-network=y
CONFIG_PACKAGE_luci-mod-status=y
CONFIG_PACKAGE_luci-mod-system=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-ssl=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_mtpoe-ctrl=y
CONFIG_PACKAGE_px5g-mbedtls=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-luci=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_rpcd-mod-ucode=y
CONFIG_PACKAGE_ucode-mod-html=y
CONFIG_PACKAGE_ucode-mod-math=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
1 Like
4

I'm sure I created the branch against master, that was on kernel 6.x that time. By any chace, didn't you miss the branch check-out step after cloning and before building?

5

I clean up my previous posts to not confuse others, my main issue seems to be:

root@OpenWrt:~# cat /sys/firmware/mikrotik/hard_config/booter_version
7.8

while according to docs I should downgrade to RouterOS v6 first, @prudy what you have as an booter_version in your RB5009UPr+S+IN? As after fighting with this device for a third day, now I stuck in opposite situation, in attempt of downgrade to RouterOS v6.x I restore all mtd's using backups and dd and this restore RouterOS at the end, but my problem is that RouterOS doesn't allow me to downgrade below 7.8, I waste several hours while searching for combination of netinstall and routeros version that will do actual downgrade but now I stuck in completely shocking situation - I cant --dhcp-boot anything anymore :exploding_head:
I tried several RouterOS versions, all possible booting combinations in RouterOS settings but previously working approach, just give me tons of

dnsmasq-tftp: sent /src/u-boot-rb5009/u-boot.elf to 192.168.88.186
dnsmasq-tftp: sent /src/u-boot-rb5009/u-boot.elf to 192.168.88.186
dnsmasq-tftp: sent /src/u-boot-rb5009/u-boot.elf to 192.168.88.186

logs instead.

So now I'm completely blocked and desperate to give up on this device as I can't even boot previously bullet-proof working option from here

Would really appreciate any help

6

I never had to downgrade from 7. You've got the console so you booted fine too.
I propose you go with a ready-made UG image first to see all starts, only then switch to own build.

7

Any suggestion on those? as now problem are not with sysupdate but before with u-boot/patched initramfs that should be loaded into ram to get into OpenWRT over ssh

As after restoring mtd's with original content, I somehow can't escape "routeros jail" anymore, router simply doesn't pick files that was sent, I see correct vendor class: ARM64__boot as before but as result instead of vendor class like odhcp from OpenWrt, I just see bunch (like 50) lines in dnsmasq log like:

dnsmasq-tftp: sent /src/u-boot-rb5009/u-boot.elf to 192.168.88.186

or (even one that was always working before)

dnsmasq-dhcp: 2433593973 available DHCP range: 192.168.88.100 -- 192.168.88.200
dnsmasq-dhcp: 2433593973 vendor class: ARM64__boot
dnsmasq-dhcp: 2433593973 tags: bootp, eno2
dnsmasq-dhcp: 2433593973 BOOTP(eno2) 192.168.88.186 78:9a:18:94:34:a8
dnsmasq-dhcp: 2433593973 bootfile name: openwrt-mvebu-cortexa72-mikrotik_rb5009-initramfs-fit-uImage.elf
dnsmasq-dhcp: 2433593973 next server: 192.168.88.2
dnsmasq-dhcp: 2433593973 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp: 2433593973 sent size:  4 option: 28 broadcast  192.168.88.255
dnsmasq-dhcp: 2433593973 sent size:  4 option:  3 router  192.168.88.2

dnsmasq-tftp: sent /src/openwrt-mvebu-cortexa72-mikrotik_rb5009-initramfs-fit-uImage.elf to 192.168.88.186
...
dnsmasq-tftp: sent /src/openwrt-mvebu-cortexa72-mikrotik_rb5009-initramfs-fit-uImage.elf to 192.168.88.186

and after some time router give-up and reboot back to normal boot :frowning:

here is what I have currently:

[admin@gw0] > /system/routerboard/print
       routerboard: yes
             model: RB5009UPr+S+
     serial-number: xxx
     firmware-type: 70x0
  factory-firmware: 7.8
  current-firmware: 7.15.3 #were trying with 7.8 too
  upgrade-firmware: 7.8

[admin@gw0] > /system/routerboard/settings/print
              auto-upgrade: no
               boot-device: nand-if-fail-then-ethernet
             cpu-frequency: auto
             boot-protocol: bootp #also tried with dhcp
       force-backup-booter: no
               silent-boot: no
      protected-routerboot: disabled
      reformat-hold-button: 20s
  reformat-hold-button-max: 10m

As in last 3 days I did exactly the same blind netbooting for like 50 times already so I'm sure that something got changed on router side in this context, yes I was having issues with sysupdate's or so but almost never with initial ones that specified in --dhcp-boot option that get me into ram loaded OpenWrt, I see that after restoring mtd's, led sequence (blinking) got a bit changed, in the past I were just spin-up dnsmasq, connect to eth0 and power-up device, while holding reset button till some log line appear in dnsmasq log (~20s), led sequence blinking were like - sfp to eth0, few blinks, all led off, eth0 only > log line in dnsmasq, now I see that last sequence with eth0 blinks actually happen twice and only then line appears in dnsmasq log, so I start to think that maybe some boot order got changed or I ending up in some "wrong" mode, without console its quite hard to understand what is going on as I basically detecting progress based on dnsmasq log lines like vendor class: ARM64__boot

Any hints would help, maybe some ROS settings are wrong, but from what I see everything looks or maybe I forgot about some original jailbreak trick to get first bootp happen or some other piece of puzzle that I miss

8

Can you monitor the wire for packages tfansfer? I don't remember the tftp ack scheme, but those repeating 'sent', don't they sugest the packets lost? Is everythink ok if you load ROS image over the same net connection?

BTW. do not give up on this, I almost did it when my PoE was not working fully on a new router. Thanks to switching to OpenWrt I opened the router, noticed some residual soldering, that when cleaned up made all working.

9

New day, new attempt, this time somehow successful.

Yes, you were right, those repeating 'sent' were somehow related to lost packets, maybe because while serving dnsmasq over eth to router, laptop itself were connected over wireless connection.

So as result of today's test:

Using self made u-boot from yours rb5009upr branch, lead to the following, I see that u-boot are sent, I see that device restarts and get initramfs it looks like this:

dnsmasq-tftp: sent /src/u-boot-rb5009/u-boot.elf to 192.168.88.158
dnsmasq-tftp: file /src/openwrt-mvebu-cortexa72-mikrotik_rb5009-initramfs-uImage.itb not found for 192.168.88.158
dnsmasq-tftp: sent /src//openwrt-mvebu-cortexa72-mikrotik_rb5009upr-initramfs-uImage.itb to 192.168.88.158

but after restart it boot back to ROS instead of OpenWrt :confused:

As I were not having much time (as this device should bring internet to whole house over SFP GPON modem and should power wireless router over PoE to share internet to all other devices), I tried previously successful approach, that use adron-s initramfs for dhcp-boot option and future steps looks like:

wget https://github.com/adron-s/aux-loader2/raw/main/releases/2.xx-latest/rbt-with-aux-for-mtd5.bin \
  -O- | mtd write - RouterBOOT
echo cfg > /sys/firmware/mikrotik/soft_config/boot_device
echo 1 > /sys/firmware/mikrotik/soft_config/commit

wget http://marvin.ariekanarie.nl/openwrt/rb5009/openwrt-mvebu-cortexa72-mikrotik_rb5009-squashfs-sysupgrade.bin \
  -O- > /tmp/fw.bin && sysupgrade /tmp/fw.bin

(sysupdate are not from adron-s but marvin, that I found in rb5009 thread before) with that and assuming that this is done on top of ROS, I'm having OpenWrt with working SFP and PoE-out (most likely because it was auto enabled in ROS) so while I'm having no control of PoE, auto-on option on all ports satisfy my needs, problem is only that I'm stuck with 5.15.102 kernel in that case.

From previous attempts I already know that using adron-s initramfs and mtd5 (RouterBOOT) with any other sysupdates (with kernel 6 or so) will not work, I'm not sure why but I assume that sysupdate itself somehow dependent on initramfs and/or RouterBOOT partition (or because sysupdate with 6.x can't be done on top of initramfs with 5.x), also I assume that adron-s patch for mtd5 force booting OpenWrt (that's most likely why my u-boot, doesn't boot into OpenWrt after sysupdate)

With all that I still plan to try with OpenWrt that I will make locally and that will contain some extra packages I need, but for that I would like to double check branch that I suppose to use?

  • rb5009-with-poe
  • rb5009upr+s+in
  • rb5009poe
  • rb5009upr

as I finally uncover confusion with PoE in RB5009 case, that basically misleading as there are PoE-in and PoE-out functionalities in different versions, and in my case (RB5009UPr+S+IN) I'm mainly interested in PoE-out to power connected devices (not router itself)

I'm grateful for your support @prudy that help a lot :beers:
if you can share details regarding correct branch in my case (did you use PoE-out?) and share ideas on how I could overtake problem when after u-boot and sysupdate router boot back to ROS instead of OpenWrt that would help a lot.

Thanks for your support again

10

As for the branch, please stick to the one in the first post. The others are abandoned or different approaches. I went against those steps myself before posting.
As for the bootloader, the very first ever was a modified origin I believe, then the u-boot was created that you can find in the linked commit. Use the latter. If you look at the boot script it first tries to boot flashed image. Then it searches for the usb/net, then gives up. I did not look deeper but if none is found and the ROS is started it might revert to original bootloader, and you must load all again.

11

Thanks for explanation, regarding

then the u-boot was created that you can find in the linked commit

that would explain a lot as all that time I were trying u-boot that was created in bin/targets/mvebu/cortexa72/u-boot-rb5009/u-boot.elf as result of make. Is that correct? or I should use some patched one from somewhere instead? if so can you please provide reference as I can't find it.
Thanks

12

Another attempt today, unfortunately with same outcome,
I write back original ROS mtd's, reboot and get into Flashboot/ARM64__boot I see that u-boot booted (vendor class: U-Boot.armv8), I see that initramfs was sent, I see that device reboots, but then device stuck with rebooting (I was waiting for like 15min) :frowning:

I start to think that maybe its somehow related to mtds that I have:

root@OpenWrt:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 40000000 00020000 "ubi"
mtd1: 00095c04 00010000 "ATF"
mtd2: 000193fc 00010000 "backup_RouterBOOT"
mtd3: 00001000 00010000 "hard_config"
mtd4: 00010000 00010000 "RouterBOOT"
mtd5: 00010000 00010000 "soft_config"
mtd6: 00010000 00010000 "DTS"
mtd7: 00020000 00010000 "free_space"
mtd8: 00f00000 00010000 "kernel"
mtd9: 00100000 00010000 "first_1M"

is it same for you @prudy ?

because I'm lack in understanding why u-boot method doesn't work in my case at all, I'm expecting to get OpenWrt booted so I could sysupdate but it just never come up, so or my mtds are wrong or I'm using wrong u-boot (I'm using the one that are builded as result of following make instructions from your branch)

Regards

13

I cannot get hands onto my router yet for ~2 weeks to list the mtds.
But what I can do is to upload that latest binaries I used, so you could try with them:

One thing to clarify - after the initramfs delivery, it does start but then the flashed file does not or even the initramfs does not come up?

14

I am using the last snapshot

root@MikroTik:~# ubus call system board
{
        "kernel": "6.6.49",
        "hostname": "MikroTik",
        "system": "ARMv8 Processor rev 1",
        "model": "MikroTik RB5009",
        "board_name": "mikrotik,rb5009",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "SNAPSHOT",
                "revision": "r27317-8e6543fae5",
                "target": "mvebu/cortexa72",
                "description": "OpenWrt SNAPSHOT r27317-8e6543fae5"
        }
}

Is it possible to turn off the LEDs on the switch (p2-p8)?

15

That is what my mtd looks like if using the old .elf file provided by adron.

This is what the mtd information looks like using the new u-boot.elf from the openwrt repository:

root@OpenWrt:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00800000 00020000 "YAFFS"
mtd1: 3f800000 00020000 "ubi"
mtd2: 00fe0000 00010000 "MikroTik"
mtd3: 00001000 00010000 "hard_config"
mtd4: 00001000 00010000 "soft_config"
mtd5: 00005b56 00010000 "dtb_config"
mtd6: 00010000 00010000 "RouterBOOT-primary"
mtd7: 00020000 00010000 "u-boot-env"

I'm not sure if flashing isn't working properly or booting after the flash isn't working properly. The loaded initramfs is working, but upon flashing I also encounter this boot-loop. I may try building it myself to see if something works better.

The method I use for net booting is tinypxe. I simply supply the u-boot.elf file with the .itb file in the same root directory.

For what it's worth a previous snapshot build with the modified routerboot has been working and stable for months on another device.

16

Hi guys thanks for valuable information,

@bassblu did you modified those mtd's or those are stock ones, I mean count and sizes of those because as you can see above mine looks completely different

@prudy using uboot in dnsmasq logs I see that initramfs file was sent I see that device restarts, but nothing come up and device goes in infinitive loop

do I need to have mtd's like bassblu above? to make uboot work, is there are some mandatory uboot preparation steps that I miss because mine are stock ones and I see different number of those together with size that most likely causing problems with uboot

I can boot from adrons fit initramfs and modify those mtd's but I need to get them from somewhere

Regards.

17

What I posted is the mtd information after using the new u-boot from the supplemental files and loading the initramfs from a snapshot on Sept. 14. I can only get the router to boot using netboot and encounter a boot loop after trying to flash it.

The mtd information you posted looks the same as Adron's modified image from a couple years ago and most likely the stock mtd information. I do not have a factory device to check this against.

Here are a couple links from the other thread.

Keep in mind that this information is somewhat outdated. I also seem to remember there being a problem with YAFFS, but I will have to do some more searching. I suspect something broke in the bootloader, but my skills are limited.

18

The RG5009 should support Ethernet cable test and also full raw cable TDR data retrieval for the 1G PHYs.

Has anyone tried it? It'll need the ethtool-full package installed, and then you should be able to run:

ethtool --cable-test-tdr devname

This can be used to detect damaged or broken cabling.

Would someone mind giving this a spin? I don't have a RB5009, but I'm debugging a cable test driver bug with another OpenWrt platform (Huawei AP5030DN) with a similar PHY... The platform I'm on hits a:

Timeout while waiting for cable test to finish

message in the kernel logs (see logread or dmesg) during the TDR test. I also have patch which fixes that on the Huawei AP5030DN.

19

Is there any documentation available on how to use this?
When I plug in the device (over eternet clable), the device does not power up.

20

With both OpenWrt 23.05 (non-upstream device support) and OpenWrt 24.10 I am running into a similar problem:

~# ethtool --cable-test-tdr p6
Cable test TDR started for device p6.
Cable test TDR completed for device p6.
TDR Pulse 1000mV
Step configuration: 0.80-149.73 meters in 0.80m steps
Pair A Amplitude  164
Pair B Amplitude  585
Pair C Amplitude  562
Pair D Amplitude  703
~# dmesg
[..]
[  111.102749] Marvell 88E6393 Family !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:03: Timeout while waiting for cable test to finish
1 Like
21

Thanks! Would you mind giving this patch a go?

Index: linux-6.6.61/drivers/net/phy/marvell.c
===================================================================
--- linux-6.6.61.orig/drivers/net/phy/marvell.c
+++ linux-6.6.61/drivers/net/phy/marvell.c
@@ -38,6 +38,9 @@
 #include <asm/irq.h>
 #include <linux/uaccess.h>
 
+/* For timing debugging only TODO remove */
+#include <linux/time.h>
+
 #define MII_MARVELL_PHY_PAGE           22
 #define MII_MARVELL_COPPER_PAGE                0x00
 #define MII_MARVELL_FIBER_PAGE         0x01
@@ -2042,14 +2045,24 @@ static int marvell_vct5_wait_complete(st
 {
        int i;
        int val;
+       u64 entrystamp, exitstamp;
+       entrystamp = ktime_get_ns();
+
+       usleep_range(9500, 10000);
+       /* msleep(10); */
 
-       for (i = 0; i < 32; i++) {
+       for (i = 0; i < 100; i++) {
                val = __phy_read(phydev, MII_VCT5_CTRL);
                if (val < 0)
                        return val;
 
-               if (val & MII_VCT5_CTRL_COMPLETE)
+               if (val & MII_VCT5_CTRL_COMPLETE) {
+                       exitstamp = ktime_get_ns();
+                       phydev_warn(phydev, "Got VCT5 ctrl data after polling %d times\n", i);
+                       phydev_warn(phydev, "Got VCT5 after %llu ns\n", exitstamp - entrystamp);
                        return 0;
+               }
+               usleep_range(800, 1200);
        }
 
        phydev_err(phydev, "Timeout while waiting for cable test to finish\n");

Here's the same patch bug without the debug prints - but it would be useful to see some debug timings, because the values I've used for usleep_range were just those found on the hardware I have here...

Index: linux-6.6.61/drivers/net/phy/marvell.c
===================================================================
--- linux-6.6.61.orig/drivers/net/phy/marvell.c
+++ linux-6.6.61/drivers/net/phy/marvell.c
@@ -2043,13 +2043,17 @@ static int marvell_vct5_wait_complete(st
        int i;
        int val;
 
-       for (i = 0; i < 32; i++) {
+       usleep_range(9500, 10000);
+
+       for (i = 0; i < 100; i++) {
                val = __phy_read(phydev, MII_VCT5_CTRL);
                if (val < 0)
                        return val;
 
                if (val & MII_VCT5_CTRL_COMPLETE)
                        return 0;
+
+               usleep_range(800, 1200);
        }
 
        phydev_err(phydev, "Timeout while waiting for cable test to finish\n");
22

In the sources, there is a readme file. You can manually set each port on/off/auto and check the status. The satartup setup is taken from the network config section. Note that by default all ports are set to auto - if it does not come up maybe its PoE is on passive standard?