This post says the interrupts are not evenly distributed on xrx200. It seems the 4.14.93 openwrt kernel doesn't support the interrupt controller for the second VPE of xrx200, this seemed weird to me, so I dug on internet and I've found a kernel here which supports it. There was no devicetree with base address of the second icu, but the GPL sources from TP-W9980 contains a kernel 2.6 header (located at GPL_TD-W9980/lantiq/target/linux/ltqcpe/files/include/asm-mips/ifx/vr9/vr9.h) with hardcoded base address:
#define IFX_ICU_VPE1 (KSEG1 | 0x1F880300)
The base address was confirmed by an address space scan (devmem).
These informations combines in attached files. Basically interrupt functions require a check for which CPU they run on and use a correct base address. There were some bugfixes in the 3.10 kernel, which are missing in openwrt (but it seems to work without them). There are spinlocks too but the system seems to work without them (maybe the new kernels have implicit locking? anyway race conditions would appeared only during big interrupt load, which I didn't test).
Other thing required is the enable of the second VPE (core) interrupts. The function vsmp_init_secondary() in arch/mips/kernel/smp-mt.c seems to activate only IP0, IP1, IP6 and IP7, which seems it is not sufficient to get the interrupts to MIPS VPE core, so I decided to enable them all by:
change_c0_status(ST0_IM, STATUSF_IP0 | STATUSF_IP1 | STATUSF_IP2 | STATUSF_IP3 | STATUSF_IP4 | STATUSF_IP5 | STATUSF_IP6 | STATUSF_IP7);
It seems v3.10 kernel did use mips_gic_present() part of the if-else condition, but it seems it is not compatible with 4.14 kernel code and the part doesn't enable IP0 and IP1 which are (IMO) IPI calls on xrx200 (first VPE gets them enabled in the interrupt driver). The second change is and irrelevant fix (seems to be already in current vanilla kernel) for a correct VPE numbering in /proc/cpuinfo.
Last part of the change is the devicetree arch/mips/boot/dts/vr9.dtsi:
icu0: icu0@80200 {
#interrupt-cells = <1>;
interrupt-controller;
compatible = "lantiq,icu";
reg = <0x80200 0x28
0x80228 0x28
0x80250 0x28
0x80278 0x28
0x802a0 0x28>;
};
//second ICU for irq affinity/balancing
icu1: icu1@80300 {
#interrupt-cells = <1>;
compatible = "lantiq,icu1";
reg = <0x80300 0x28
0x80328 0x28
0x80350 0x28
0x80378 0x28
0x803a0 0x28>;
};
Can somebody test if the balancing it works? I did it successfully on my TP-W9980B. Ideas for a better solution are always welcomed too ;-).
# cat /proc/interrupts
CPU0 CPU1
0: 14066 9147 MIPS 0 IPI_resched
1: 841 1046 MIPS 1 IPI_call
7: 333849 332983 MIPS 7 timer
8: 0 0 MIPS 0 IPI call
9: 0 0 MIPS 1 IPI resched
22: 25251 1 icu 22 spi_rx
23: 3145 1 icu 23 spi_tx
24: 0 0 icu 24 spi_err
62: 0 7478 icu 62 1e101000.usb, dwc2_hsotg:usb1
63: 1 10848 icu 63 mei_cpe
72: 0 12886 icu 72 vrx200_rx
73: 15506 0 icu 73 vrx200_tx
75: 0 0 icu 75 vrx200_tx_2
91: 0 0 icu 91 1e106000.usb, dwc2_hsotg:usb2
112: 3325 0 icu 112 asc_tx
113: 0 123 icu 113 asc_rx
114: 0 0 icu 114 asc_err
126: 0 0 icu 126 gptu
127: 0 0 icu 127 gptu
128: 0 0 icu 128 gptu
129: 0 0 icu 129 gptu
130: 0 0 icu 130 gptu
131: 0 0 icu 131 gptu
144: 0 0 icu 144 ath9k
161: 0 0 icu 161 ifx_pcie_rc0
P.S. There was SMTC mode, where kernel managed any TC as a single core. But it was removed somewhere in v3.15. With SMTC you could have 4 cores in your xrx200 device (but there would be a higher latency ofc...). In my TD-W9980B only two TC are used (mapped to 2 VPEs), so the second 2 TC are disabled and unused. Maybe on other boards the second VPE is using them for some proprietary DSP operations (as I read in old topics here). The compatibility of the balancing should be tested on such boards too.