Mt76 wireless driver debugging

I believe I've just pinpointed a rare crash that I've been searching for, that seems to occur when using 802.11X authentication.

If I'm correct, this occurs in the mac80211 library in the rx.c file, ieee80211_rx_8023( ) -> ieee80211_deliver_skb_to_local_stack( )

The crash happens on the last line here, I believe where ehdr is dereferenced, as the other variables have already been dereferenced above this point without crashing. But the code comment is interesting in that it states it is overwriting the mac address with the bridge address when sending EAPOL frames.

My first theory is the ehdr pointer can be null if the STA disconnects at precisely the right moment immediately after connecting, while still in the state of 802.11X authentication.

I found that ehdr is almost always null unless skb->protocol == sdata->control_port_protocol, then it's not supposed to be null. Except when it is, perhaps due to a race condition?
I've added a sanity check to see where that leads.

struct ethhdr *ehdr = (void *)skb_mac_header(skb);

memset(skb->cb, 0, sizeof(skb->cb));

/*
 * 802.1X over 802.11 requires that the authenticator address
 * be used for EAPOL frames. However, 802.1X allows the use of
 * the PAE group address instead. If the interface is part of
 * a bridge and we pass the frame with the PAE group address,
 * then the bridge will forward it to the network (even if the
 * client was not associated yet), which isn't supposed to
 * happen.
 * To avoid that, rewrite the destination address to our own
 * address, so that the authenticator (e.g. hostapd) will see
 * the frame, but bridge won't forward it anywhere else. Note
 * that due to earlier filtering, the only other address can
 * be the PAE group address, unless the hardware allowed them
 * through in 802.3 offloaded mode.
 */
if (unlikely(skb->protocol == sdata->control_port_protocol &&
             ehdr &&                              <-- I added this extra check
	     !ether_addr_equal(ehdr->h_dest, sdata->vif.addr)))
	ether_addr_copy(ehdr->h_dest, sdata->vif.addr);

Here's the original stacktrace, which is misleading because it looks like the code crashes in eth_type_trans, but if you use gdb to dereference that address ends up in ieee80211_deliver_skb_to_local_stack

<1>[43203.019950] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
<7>[43203.159413] Hardware name: Linksys E8450 (UBI) (DT)
<7>[43203.171241] pc : eth_type_trans+0x44/0x190
<7>[43203.175342] lr : ieee80211_sta_ps_transition+0x12a0/0x3090 [mac80211]
<7>[43203.181833] sp : ffffffc008ce3bf0
<7>[43203.185139] x29: ffffffc008ce3bf0 x28: 0000000000000000 x27: 0000000000000000
<7>[43203.192273] x26: ffffff8002863820 x25: ffffffc008ce3d88 x24: ffffff80029a284a
<7>[43203.199406] x23: ffffffc008ce3d30 x22: ffffff80027b4aa8 x21: ffffff8006111980
<7>[43203.206539] x20: ffffffc008ce3c68 x19: ffffff8005a08000 x18: 0000000000000000
<7>[43203.213672] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
<7>[43203.220804] x14: 0000000000000001 x13: cbcccbcc0f000100 x12: 0000004040200e9a
<7>[43203.227937] x11: 143c08bc0004eb08 x10: ffffff80027b4735 x9 : ffffff8005a08050
<7>[43203.235070] x8 : ffffff80029a284a x7 : 000000000008beae x6 : ffffff8005a08028
<7>[43203.242204] x5 : 6c416a8400000000 x4 : 0000000000000000 x3 : b07cc422846a416c
<7>[43203.249336] x2 : ffffff80029a284a x1 : ffffff8006b88000 x0 : ffffff8005a08000
<7>[43203.256469] Call trace:
<7>[43203.258908]  eth_type_trans+0x44/0x190
<7>[43203.262653]  ieee80211_rx_list+0x260/0x28c [mac80211]
<7>[43203.267738]  mt76_rx_complete+0xdc/0x158 [mt76]
<7>[43203.272274]  mt76_rx_poll_complete+0x5c/0x134 [mt76]
<7>[43203.277242]  mt76_dma_rx_poll+0x84/0xe4 [mt76]
<7>[43203.281690]  __napi_poll.constprop.0+0x6c/0x154
<7>[43203.286217]  napi_threaded_poll+0x7c/0xac
<7>[43203.290221]  kthread+0xcc/0xdc
<7>[43203.293271]  ret_from_fork+0x10/0x20
<0>[43203.296847] Code: 91003843 f9006403 f9419024 f9400043 (f9400084)

I am having unkown no data flow mode sporadically, using wpa2 enterprise. The status showed connected just no data flow at all for a while.

My device is D-Link DIR-860L B1, running OpenWRT snapshot. And I am not sure if your finding is also the cause of my issue, I guess we'll find out once your fix of this bug got committed.

So if I were looking to create some patches to apply to my 22.03.x builds I would need to pick commits:

Anything else?

To get my fixes you could edit openwrt/package/kernel/mt76/Makefile

Change these these lines:
PKG_SOURCE_URL:=https://github.com/Brain2000/mt76
PKG_SOURCE_VERSION:=3176270ef706f9797113136f6bd07a64a4534cb1
#PKG_MIRROR_HASH:=f8cd90499fcbfb5155c46bfcdee097559ed65294e51b2c5b479a2309c8d46040

If I push a new version, you just need to update the PKG_SOURCE_VERSION to the latest commit id.

[If you keep the PKG_MIRROR_HASH rem'd out it will download from the repo every time, but other than that, the line is really just an optimization so every package isn't redownloaded.]

Quick update, I found there were two ways to look at the stack depending on the order of the .o files pulled in, which led me to a 2nd place to perform sanity check.

From what I can gather, a device is transmitting a multicast packet (most likely iPhone mDNS) when another STA disconnects, and the MAC address pointer location is deallocated before it can recieve the packet. The crash seems point to to a location that performs checks on MAC addresses for multicast to see if it is broadcast or multicast that end up being null pointers.

I believe this is not in the MT76 driver, but the mac80211, or even down to the linux ethernet libraries.
I am adding more and more safety checks until I find where the crash happens. Hopefully we'll have something soon. It takes me about 3 days per test, as it takes that long to make it crash.

As for the WED, I've found several suspect places where that might be happening. So I've updated my repo to include some extra WARN logging when certain events take place that cause the DMA to reset, or there's a function that "kicks" the MCU when the receive packets get stuck. That might point to a potential issue in the Mediatek firmware itself.... this might all take some time, but I feel we're getting closer to some issues here.

The latest changes are available by changing the following lines in openwrt/packages/kernel/mt76/Makefile:

PKG_SOURCE_URL:=https://github.com/Brain2000/mt76
PKG_SOURCE_VERSION:=f7fea5b2f0ef08c8154316c607689699881251ad
PKG_MIRROR_HASH:=e6394795bc11dc343d9399a7c51f3d0ef928592dce22ad4e22775ccf627a9e36

Have you have a solution so that the STA/AP mode can work with all French internet boxes without having to use the relay mode because the WDS does not work ?

It works with very old OpenWrt version 17.01
*

Alright, after a brief hiatus, I'm back to being able to jump in and do more testing. I'm back up and running off your latest commit (3d8b848) now.

Do you want me to enable WED again and look for any particular behavior or output?

Yes. After enabling WED, when it suddenly stops, check this file>
cat /sys/kernel/brain2000/brain2000_mt76->token_count
Which is the file that I'm using to monitor values live.
I'll probably need to add a bunch more as we search for what goes wrong when WED stops.

BTW, I pinpointed the most recent crash location. It's appears that the device MAC address gets null'd out. I thought it was in something called an skbuf (socket buffer for transmitting data), but it's not. I have a line in to log and skip the crash added yesterday, and as soon as I see that it works, I'm going to submit a patch. This one is in the mac80211, not the MT76 driver, so unless MT76 does something to cause this through some weird side effect, this should be a crash fix across all devices.

If you're interested, in net/mac80211/rx.c line 4763:

	/* deliver to local stack */
	skb->protocol = eth_type_trans(skb, fast_rx->dev);
	ieee80211_deliver_skb_to_local_stack(skb, rx);

fast_rx->dev is a null value in very rare cases, causing the 00000000 dereference panic in eth_type_trans..

I'm watching that new debug file you added as well--nice!

I've re-enabled WED while watching your new token_count debug file at a refresh rate of 10x per second. While running iPerf3 with my well-connected router as the sender, I see no offloaded flows indicated in /sys/kernel/debug/ppe0/bind. However, the token_count sits north of 180--generally between 190-200 during the 30 second test.

Are you seeing offloaded flows in ppe0 during the same sort of test?

I would be happy to test this patch as a backport on 22.03 to see whether it helps other devices. The mix of mDNS and disconnecting client’s definitely fits with my day-to-day network usage.

I discovered it's worse than I thought. I can cause the router to go sideways by sending a stream of multicast until it saturates the receive buffer, and then turning on airplane mode while sending. It basically DoS's the unit where the packets get lost leaving the token_counter stuck. The only recovery is a manual reboot....

mt76 repo got updated. It now includes a few more of your proposals @Brain2000. wifi: mt76: add missing locking to protect against concurrent rx/status calls

Kudos @Brain2000 !!!

I'm not sure whether this is helpful or not, but I had an unexpected slow down (almost to zero) with my WiFi which resolved after about 60 seconds, and I managed to capture a stack trace from the kernel log.

It's from 22.03.4 and visible in the bug here: https://github.com/openwrt/openwrt/issues/12431 - it looks to be the mt7603e doing something odd.

I updated my repo with the latest and rebased two missing fixes. Along with the variable watcher I have set up (which I need to update to add more WED variables in so we can find that issue).

This new mt76 update is not yet included in the openwrt master branch. In order to get it to compile I had to remove

openwrt/packages/kernel/mt76/patches/100-api_update.patch
openwrt/packages/kernel/mt76/patches/110-wifi-mt76-ignore-key-disable-commands.patch

as these two seem to have been moved directly into the mt76 source.
Though there still may be other patches that need removed as well. I noticed that this new build, while a regular access point works, 802.11s mesh is not working. AMSDU support was added for mesh, so maybe that is causing an issue?

You know, I just realized that even though WED is enabled on my router, nothing is ever offloaded.
The bind file is always empty and the entries file always shows "packets=0 bytes=0" for every entry.

cat /sys/module/mt7915e/parameters/wed_enable contains Y, so not sure what is going on here.

Note that if you are using the device as a bridged access point you will need to install the bridger package as well. Otherwise there aren't any flows to offload...
If you are using the device as NAT router you do not need bridger.
In both cases you also need to switch on hardware flow offloading in /etc/config/firewall, ie.:

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'
	option flow_offloading_hw '1'
	option flow_offloading '1'

Aha, I do not have software or hardware offloading enabled.

One of the WED caveats reads It can be used alone or in addition to other hardware offloading, and I read that as "I don't need software/hardware offloading to use WED"...

Thank you Daniel

@daniel Wow! If enabling SW & HW offloading is all that was needed, I feel like a big dummy. :upside_down_face:

But I do see the same point that @Brain2000 raised in the E8450(/RT3200) wiki:

Specifically this within the wiki:
"A device configured as a router will likely benefit from other hardware offloading, but this is not a requirement to use WED."

FWIW, I do have Bridger installed and have from the start of trying WED since my APs are traditional WAPs and I leave the firewall/routing to another upstream device.