[Unifi 6LR] OpenWrt ethernet frames swallowed by PoE injector

I have a Unifi 6 LR AP that I have successfully flashed OpenWRT onto. After flashing, I found that it was unreachable over Ethernet. I cracked it open and attached a serial debugger, and through a series of debugging steps, I have found the following to be true:

  1. OpenWRT is able to transmit ethernet frames out to the rest of the world when it is connected to a PoE-providing switch, but not when it is connected through a PoE injector. I have tried two PoE injectors, the official Ubiquiti one, and a cheapo one off of Amazon. Both exhibit the same issue.

  2. The Stock Ubiquiti firmware works with everything, so it's not a hardware issue.

  3. The kernel sees the ethernet device just fine, and I can even tcpdump -i eth0 and see packets like DHCP requests going out, but I never see anything coming in (confirmed by connecting AP <-> PoE injector <-> laptop and running wireshark on the laptop; no ethernet frames are seen that do not originate from the laptop). If I look at packet statistics on the AP, I see there are some errors on the RX side:

# cd /sys/class/net/eth0/statistics
# for f in *; do echo -n  "${f}: "; cat ${f}; done
collisions: 0
multicast: 0
rx_bytes: 366
rx_compressed: 0
rx_crc_errors: 11
rx_dropped: 1
rx_errors: 0
rx_fifo_errors: 0
rx_frame_errors: 0
rx_length_errors: 55
rx_missed_errors: 0
rx_nohandler: 0
rx_over_errors: 0
rx_packets: 3
tx_aborted_errors: 0
tx_bytes: 29306
tx_carrier_errors: 0
tx_compressed: 0
tx_dropped: 0
tx_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_packets: 105
tx_window_errors: 0

I'm willing to go pretty far to get this resolved; can anyone give me a pointer on how to discern why the packets are not making it through the PoE injector(s)? I thought the issue might be link negotiation, but experiments with ethtool, locking things to a particular speed so far have not been productive. I am open to any and all suggestions, including building custom firmware, as I already have done that to get tools like tcpdump and whatnot bundled in for debugging.

2 Likes

Poe injectors are passive devices and do not control any link state themselves and only pass pins 1,2,3,6 to the device behind it. It could be a pc or a switch and it doesn't matter.

Please post your network config file.

2 Likes

Poe injectors are passive devices and do not control any link state themselves and only pass pins 1,2,3,6 to the device behind it.

This is what I expected, and why I am so confused. Please note I have updated my original post to include an important piece of information I forgot to mention; I ran wireshark on a machine just on the other side of the PoE injector(s) and verified that there are no packets originating from any MAC address other than the laptop itself.

Please post your network config file.

My network config file is intended to be as barebones as possible; if there are further things that I can omit, please do let me know:

# cat /etc/config/network
config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd62:e857:130a::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'
config interface 'lan'
        option device 'br-lan'
        option ip6assign '60'
        option proto 'dhcp'

This config works fine when I plug directly into the Ubiquiti PoE switch, but it doesn't work when plugged into the PoE injector, even when I've just interposed the PoE injector in between the AP and the powered switch. (That is, AP <-> Switch works, but AP <-> PoE injector <-> Switch does not). And yes, I unfortunately need this to work with a PoE injector, and not with the nice big switch.

See if this will work for you as this works on my U6 Lite. You can't assign eth0 to the bridge directly as eth0 technically isn't eth0 and it's part of the switch chip.

/etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'auto'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan'

config interface 'lan'
	option device 'br-lan'
	option proto 'dhcp'

Maybe it is a simple cable issue. Did you use the very same ethernet cable for connecting the injectors or the switch?

1 Like

You can't assign eth0 to the bridge directly as eth0 technically isn't eth0 and it's part of the switch chip.

I applied your config and it does not work with the PoE injector, same issue. Plugging the AP straight into the switch, it now doesn't automatically try to get a DHCP address on eth0, I have to manually udhcpc -i eth0 in order to get an address on the interface. Perhaps I have some kind of misconfiguration in my DHCP config:

# cat /etc/config/dhcp 

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        option ra_slaac '1'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

Maybe it is a simple cable issue. Did you use the very same ethernet cable for connecting the injectors or the switch?

I've swapped out cables multiple times, no difference in behavior. Also, as I said in the original post, when I used the stock firmware, everything works with the PoE injector, so I don't think it's a hardware issue (or at least, if it is, the stock firmware can work around something that OpenWrt can't).

You're trying to use this as an AP and not a router correct?

config dnsmasq 'dnsmasq1'
	option authoritative '1'
	option boguspriv '1'
	option domain 'lan'
	option domainneeded '1'
	option ednspacket_max '1232'
	option expandhosts '1'
	option filterwin2k '0'
	option leasefile '/tmp/dhcp.leases'
	option local '/lan/'
	option localise_queries '1'
	option localservice '1'
	option nonegcache '0'
	option nonwildcard '1'
	option readethers '1'
	option rebind_localhost '1'
	option rebind_protection '1'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

You're trying to use this as an AP and not a router correct?

That's correct. I have removed the config dhcp 'lan' section and re-ordered my stanzas to make it identical to yours now. Despite this, I still do not get a DHCP address when connected directly to the switch, and if I tcpdump -i eth0, I don't see any DHCP request packets going out. I do see many packets coming in from broadcast traffic on the switch.

If I connect it through the PoE injector again, we get the same situation as last time (no incoming packets at all) and this time I don't see my own DHCP requests going out.

What version are you building? I will kick off a build and attempt to reproduce.

I also don't have any udhcpc processes running with your /etc/config/network file:

# ps | grep [d]hcp
 1723 root      1424 S    /usr/sbin/odhcpd

Whereas when /etc/config/network has list ports 'eth0', I get the following:

# ps | grep [d]hcp
 1723 root      1552 S    /usr/sbin/odhcpd
 3952 root      1248 S    udhcpc -p /var/run/udhcpc-br-lan.pid -s /lib/netifd/

I'm curious about the list ports 'lan'; where does the lan port come from? I'm not sure how to get a list of valid identifiers to put into that configuration spot.

What version are you building? I will kick off a build and attempt to reproduce.

This issue is reproducible for me with the build linked to from the wiki page. I have been messing around with my own custom build based on the imagebuilder docker image, but to make this easier for you as well, I'm happy to flash the wiki-linked build onto the device.

Should I do anything other than sysupgrade to revert back to vanilla OpenWrt?

Are you on V21.02 or V22.03?

I am on version v22.03.

Alright, to make things easier to debug, I have flashed the build of v22.03 linked from the wiki page, and then run firstboot -y; reboot which has cleared out all configuration to the bare default.

Without making any configuration changes at all, if I inspect the packet statistics in /sys/class/net/eth0/statistics, when I am connected directly to the switch, I get readings like the following:

collisions: 0
multicast: 0
rx_bytes: 51006
rx_compressed: 0
rx_crc_errors: 0
rx_dropped: 20
rx_errors: 0
rx_fifo_errors: 0
rx_frame_errors: 0
rx_length_errors: 0
rx_missed_errors: 0
rx_nohandler: 0
rx_over_errors: 0
rx_packets: 364
tx_aborted_errors: 0
tx_bytes: 2684
tx_carrier_errors: 0
tx_compressed: 0
tx_dropped: 0
tx_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_packets: 22
tx_window_errors: 0

So you see there are packets being properly received, but dropped. However, if I connect to the PoE injector, I get readings like the following:

collisions: 0
multicast: 0
rx_bytes: 0
rx_compressed: 0
rx_crc_errors: 0
rx_dropped: 0
rx_errors: 0
rx_fifo_errors: 0
rx_frame_errors: 0
rx_length_errors: 105
rx_missed_errors: 0
rx_nohandler: 0
rx_over_errors: 0
rx_packets: 0
tx_aborted_errors: 0
tx_bytes: 2446
tx_carrier_errors: 0
tx_compressed: 0
tx_dropped: 0
tx_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_packets: 19
tx_window_errors: 0

Sent you a DM with a link to the config files from mine that I just flashed and configured the same way you were looking for. I didn't have an injector and only a POE switch to test but I will get one from storage tomorrow.

Thank you for your effort here, I really appreciate it.

I have downloaded/unpacked your configs into my /etc/config directory. After unpacking those and trying out both the switch and the injector, it appears that we still have the same behavior as before; e.g. I never receive any packets when plugged into the injector, and everything works just fine when plugged into the switch.

I switched back to the stock firmware to doubly make sure that the PoE injector, the cables, etc... all work properly, which they do. Interestingly, the stock firmware selects 1000Mb mode through the PoE injector, rather than 100Mb which is what OpenWrt auto-selects. When OpenWrt selects an 100Mb link, my switch on the other end correctly detects that there is a 100Mb link, however the switch stats show the mirror image of the packet statistics from within OpenWrt: many packets transmitted, but not packets received. It's like there's some kind of serious signal degradation that gets applied when transmitting through the PoE injector, but only with OpenWrt, not with the stock firmware. I also tried using ethtool to force OpenWrt to use a gigabit link, but when I do that, the switch and OpenWrt both think the cable is unplugged, which is a bit frustrating.

I thought this might be a kernel driver bug; the stock firmware is running on v4.4 while v22.03 is on v5.10. I saw that there's been some work on the mtk_eth_soc driver within the kernel over the years, so I flashed a snapshot build of OpenWrt to see if the updated kernel (v5.15) would fix anything, but no dice.

I also compared the stock firmware's kernel config (how nice of them to expose it via /proc/config.gz) to the one from OpenWrt v22.03 (I got this from the builder image, nice of you guys to provide it there as well). I didn't see any major differences, sadly, although even if I had, I'm not sure how I would properly build my own kernel to test new configurations, the documentation around that seems to mostly be "don't, it's hard". :wink:

1 Like

Just checking to see if you have had a chance to look into this? If you've been too busy, that's totally, fine, but I just want to make sure this doesn't fall through the cracks, as I'm very eager to debug this. If there's somewhere else you recommend I dig for answers, I'm happy to reach out in other venues as well.

Sorry i injured my back last week and I wasn't able to go dig through the tool box in the truck.

But today i was feeling better so i went and grabbed a 48v ubiquiti black gigabit power supply. I put it place in between my miktoik 24 port poe switch. It received a dhcp lease just like normal. Link state is 1000mbits.

I'm not sure what is wrong with your unit.

My unit is a V1 access point.

Oh no, I'm sorry to hear you injured your back! I'm glad you're feeling better.

Hmm, my device is a V2, so perhaps there is a slight difference there.