How to troubleshoot sirq storm?

For Developers
1

Hi -

I would just like a quick starter to start troubleshooting a mwl8k client mode sirq usage issue.

A ea4500 that does > 100mbps in AP mode has maxed out SIRQ in client mode < 36mbps.

How would one be able to see which part of mwl8k driver is causing the maxed out SIRQ?

Thanks!

2

I think this page has some useful info here:

http://www.linuxgrill.com/anonymous/fire/netfilter/kernel-hacking-HOWTO-5.html

I think I can do a standard perf record and then see what functions are calling spin_lock / other sirq sources.

3

Here's a flame graph in client mode doing throughput test (throughput less than 30mbps at 100% SIRQ)

image

A flame graph in AP mode doing throughput test. Throughput fluctuates but can reach ~100mbps with a 1x1 802.11ac client.

image

The culprit looks quite clear here - looks like in client mode mwl8k is busy doing en/decryption. In AP mode there's no such calls - I think it's probably hardware accelerated.

4

in mwl8k driver 80211 setkey call:

static int mwl8k_set_key(struct ieee80211_hw *hw,
             enum set_key_cmd cmd_param,
             struct ieee80211_vif *vif,
             struct ieee80211_sta *sta,
             struct ieee80211_key_conf *key)
{


...

    if (vif->type == NL80211_IFTYPE_STATION && !priv->ap_fw)
        return -EOPNOTSUPP;

from
http://ww2.cs.fsu.edu/~rosentha/linux/2.6.26.5/docs/DocBook/mac80211/ch06.html

The set_key call for the SET_KEY command should return 0 if the key is now in use, -EOPNOTSUPP or -ENOSPC if it couldn't be added; if you return 0 then hw_key_idx must be assigned to the hardware key index, you are free to use the full u8 range.

Oh well, I guess they probably found that in client mode with hardware crpto enabled there were some issues so they just disabled it instead of fixing it - no reason why the same hardware that is used for crpto in AP mode cannot be used in client mode.

Still I would have to try enabling hw crpto in client mode and see what happens. I will most likely just refer to the mwlwifi code since they look really similar in this setkey call.

5

I am really not sure what it means:

https://github.com/torvalds/linux/commit/751930cb1d6da7b7891d8ffe84877cbf21915e0d

what does it mean to have a station interface running on AP firmware? As in like - repeater mode where the same radio is both client and AP at the same time. In such case why bother having a separate firmware for client mode... jesus.

In this case I should probably just try enabling AP mode on the same radio and see what happens now...