Huge latency after setting the "smp irq affinity" on MT7621 based board

For Developers
1

Hello!
I have a router device with Mediatek MT7620A SoC & MT7603e(2.4g)& MT7603e (5g).
The device is running OpenWRT version CHAOS CALMER with kernel v3.10.14.(as @slh said, it's differnet with real "chaos calmer", ok ,but I still have to use it for now)

It works fine but its wireless throughput is not good enough. We want to improve its network performance.

I read many documents about "smp affinity" ,such as :
https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt

I think its what we need, so I check the "/proc/interrupts" , it shows all of the net related irqs are on the CPU0 ,like this:

           CPU0       CPU1       CPU2       CPU3       

  7:   16657551   16378292   16217011   16296965      MIPS  timer

11:    1873048          0          0          0  MIPS GIC  eth0

12:   23878093          0          0          0  MIPS GIC  ra0(2.4g wireless)

30:          0          0          0          0  MIPS GIC  xhci-hcd:usb1

31:         13          0          0          0  MIPS GIC  ralink_esw

32:    9094675          0          0          0  MIPS GIC  rai0(5g wireless)

34:         32          0          0          0  MIPS GIC  serial

64:     560447          0          0          0  MIPS GIC  ipi resched

65:          0     335000          0          0  MIPS GIC  ipi resched

66:          0          0     278863          0  MIPS GIC  ipi resched

67:          0          0          0     403813  MIPS GIC  ipi resched

68:    2062637          0          0          0  MIPS GIC  ipi call

69:          0    3077783          0          0  MIPS GIC  ipi call

70:          0          0    3797212          0  MIPS GIC  ipi call

71:          0          0          0    3585205  MIPS GIC  ipi call

I checked the setting of my device:

cat /proc/irq/11/smp_affinity (eth0)

cat /proc/irq/12/smp_affinity (2.4g wireless)

cat /proc/irq/32/smp_affinity (5g wireless)

the values are all "f".

so I tried cmds like this:

echo 2 > /proc/irq/11/smp_affinity

echo 4 > /proc/irq/12/smp_affinity

echo 8 > /proc/irq/32/smp_affinity

It works, the interrupts of eth0/rai0/ra0 are separated.

But after my setting, the problem comes:

1. All the network interface has huge latency, for example , the ping 's delay grows to 1~10ms/ 10~100ms/100~1000ms (it is not sure, sometimes huge, sometims is fine , but never less than 1ms as before )

2. The related interrupts in /proc/interrupts increased much slower than before!**

Anyone knows what's the problem? I've tired to solve this for many days...Thanks very much!

PS:

Here is the our kernel config about SMP & irq:

CONFIG_IRQ_CPU=y

CONFIG_IRQ_GIC=y

CONFIG_CPU_MIPSR2_IRQ_VI=y

CONFIG_CPU_MIPSR2_IRQ_EI=y

CONFIG_IRQ_WORK=y

CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_IRQ_DOMAIN=y

CONFIG_INLINE_WRITE_UNLOCK_IRQ=y

CONFIG_INLINE_READ_UNLOCK_IRQ=y

CONFIG_IRQ_FORCED_THREADING=y

CONFIG_OF_IRQ=y

CONFIG_IRQCHIP=y

CONFIG_TRACE_IRQFLAGS_SUPPORT=y

CONFIG_MIPS_MT_SMP=y

CONFIG_SMP=y

CONFIG_SMP_UP=y

CONFIG_SYS_SUPPORTS_SMP=y

CONFIG_USE_GENERIC_SMP_HELPERS=y

CONFIG_GENERIC_SMP_IDLE_THREAD=y
2

It quite obviously is not, "chaos calmer" was released with kernel 3.18.20, respectively 3.18.23, (and "barrier breaker with kernel 3.10.49). So while the firmware might identify itself as such, it's most likely running some kind of OEM firmware or vendor SDK, using a completely different kernel and (most probably) proprietary drivers, please contact your OEM for support about their firmware (or install a (current/ supported-) version of real OpenWrt).

2 Likes
3

thank you slh.
I made a mistake, our kernel version is 3.10.34. Thank you for your remind that the device is running a unsupported and unreal OpenWRT version.You are right and I think We will update the kernle later.
But still,We need to improve it in a short time, I hope to get some advise about my problem, maybe someone encountered it before, maybe we just need to remove some configs?

4

i dont think this will work the way you want it to.

spreading irq's on different processors is usefull if one cant cope, but will reduce overall system effiency drastically at low loads (i.e. the coordination between the involved processors and duplication of data is not "free")