Xrx200 IRQ balancing between VPEs

ahmar16 · 2019-01-30T12:35:32.873Z

I can't test the patches on my system because support for cifs is broken in snapshots for now. I couldn't make it work on my router. So I had to go back to stable release. If you have tested it on your end it should be okay to submit the pull request for now.

Plonk34 · 2019-01-30T19:12:02.036Z

Sorry, Happy too early my created patch end up in:

 make[2] package/install
 make[2] target/install
 make[3] -C target/linux install
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `ltq_disable_irq':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:121: multiple definition of `ltq_disable_irq'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:121: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `ltq_mask_and_ack_irq':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:137: multiple definition of `ltq_mask_and_ack_irq'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:137: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `ltq_enable_irq':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:172: multiple definition of `ltq_enable_irq'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:172: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `get_c0_perfcount_int':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:626: multiple definition of `get_c0_perfcount_int'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:626: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `ltq_eiu_get_irq':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:113: multiple definition of `ltq_eiu_get_irq'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:113: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `arch_init_ipiirq':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:407: multiple definition of `arch_init_ipiirq'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:407: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `icu_of_init':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:487: multiple definition of `icu_of_init'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:487: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `plat_irq_dispatch':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:447: multiple definition of `plat_irq_dispatch'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:447: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `get_c0_compare_int':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:633: multiple definition of `get_c0_compare_int'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:633: first defined here
mips-openwrt-linux-musl-ld: arch/mips/kernel/smp-mt.o: in function `arch_init_irq':
/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/kernel/smp-mt.c:642: multiple definition of `arch_init_irq'; arch/mips/lantiq/irq.o:/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/arch/mips/lantiq/irq.c:642: first defined here
make[5]: *** [vmlinux] Error 1
make[4]: *** [/opt/build/owrt/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/.image] Error 2
make[3]: *** [install] Error 2
time: target/linux/install#70.82#22.53#94.29
make[2]: *** [target/linux/install] Error 2
make[1]: *** [/opt/build/owrt/staging_dir/target-mips_24kc_musl/stamp/.target_install] Error 2
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
/opt/build/owrt/include/toplevel.mk:216: die Regel für Ziel „world“ scheiterte
make: *** [world] Fehler 1

But i am 100% shure that the files are replaced by new like copy.
Can someone confirm the error ?

Sparks · 2019-01-30T19:53:09.474Z

Im not entire sure either , but seems like in your workspace there is already a patch applied that does some of the changes youve added from @pc2005

Maybe try do a recursive grep on the multiple definition var errors. Eg grep -rn ltq_disable_irq /opt/build/owrt/build_dir/target-mips_24kc_musl/

Plonk34 · 2019-01-30T20:46:49.615Z

Sorry ashes on my head
there was an fail in my script it copy irq.c to smp-mp.c
i have edit it. but anyway i try the patches from pc2005

Plonk34 · 2019-01-30T22:35:12.438Z

@pc2005 the patches end up in:

Applying /opt/build/openwrt-sb1-master/target/linux/lantiq/patches-4.14/0901-add-icu-smp-support.patch using plaintext: 
(Stripping trailing CRs from patch; use --binary to disable.)
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- ./a/arch/mips/lantiq/irq.c 2019-01-30 02:20:35.739994259 +0100
|+++ ./b/arch/mips/lantiq/irq.c 2019-01-30 04:30:31.152538191 +0100
--------------------------
No file to patch.  Skipping patch.
patch unexpectedly ends in middle of line
10 out of 10 hunks ignored
Patch failed!  Please fix /opt/build/openwrt-sb1-master/target/linux/lantiq/patches-4.14/0901-add-icu-smp-support.patch!
Makefile:23: recipe for target '/opt/build/openwrt-sb1-master/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/.prepared_f12eb2f9aed002097e49068404c55dea' failed
make[3]: *** [/opt/build/openwrt-sb1-master/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.14.96/.prepared_f12eb2f9aed002097e49068404c55dea] Error 1
make[3]: Leaving directory '/opt/build/openwrt-sb1-master/target/linux/lantiq'
Makefile:13: recipe for target 'prepare' failed
make[2]: *** [prepare] Error 2
make[2]: Leaving directory '/opt/build/openwrt-sb1-master/target/linux'
time: target/linux/prepare#41.75#16.02#160.44
target/Makefile:23: recipe for target 'target/linux/prepare' failed
make[1]: *** [target/linux/prepare] Error 2
make[1]: Leaving directory '/opt/build/openwrt-sb1-master'
/opt/build/openwrt-sb1-master/include/toplevel.mk:216: die Regel für Ziel „target/linux/prepare“ scheiterte
make: *** [target/linux/prepare] Fehler 2
irq.c
smp-mt.c

I guest it must be named a/arch/mips/lantiq/irq.c instead ./a/arch/mips/lantiq/irq.c
I fear there is a "./" to much.

But in the moment it does not make sense to make a build because the package postgres will not build and in consequence of that a lot of other needed packages are missed

pc2005 · 2019-01-30T22:44:52.358Z

Plonk34:

I guest it must be named a/arch/mips/lantiq/irq.c instead ./a/arch/mips/lantiq/irq.c
I fear there is a "./" to much.

My mistake without both ./ in patch header the openwrt will patch and compile fine.

Plonk34:

But in the moment it does not make sense to make a build because the package postgres will not build ...

I'm using rootfs on NFS (and kernel is downloaded only into RAM), so its fine for me to make many devel builds.

Fixed patches:
0901-add-icu-smp-support.patch
0902-enable-external-irqs-for-second-vpe.patch
0903-add-icu1-node-for-smp.patch

DermotMcDonnell · 2019-01-31T14:23:11.233Z

Hi Petr,

Thank you for your efforts. I have a BT HomeHub 5a (xrx200 rev 1.2) and was keen to try your patches. I downloaded trunk this morning (kernel 4.14.96) and placed your patches (all 5 of them) in target/linux/lantiq/patches-4.14:

mac@mac-desktop:~/source/target/linux/lantiq/patches-4.14$ ls -al 09*
-rw-r--r-- 1 mac mac  8657 Jan 31 12:33 0901-add-icu-smp-support.patch
-rw-r--r-- 1 mac mac   432 Jan 31 12:33 0902-enable-external-irqs-for-second-vpe.patch
-rw-r--r-- 1 mac mac   921 Jan 31 12:34 0903-add-icu1-node-for-smp.patch
-rw-r--r-- 1 mac mac 26619 Jan 31 12:30 0904-backport-vanilla-eth-driver.patch
-rw-r--r-- 1 mac mac   545 Jan 31 12:31 0905-increase-dma-burst-size.patch

Sadly they failed to build:

Applying /home/mac/source/target/linux/lantiq/patches-4.14/0701-NET-lantiq-etop-of-mido.patch using plaintext: 
patching file drivers/net/ethernet/lantiq_etop.c

Applying /home/mac/source/target/linux/lantiq/patches-4.14/0901-add-icu-smp-support.patch using plaintext: 
patching file arch/mips/lantiq/irq.c
Hunk #1 FAILED at 49.
Hunk #2 succeeded at 62 with fuzz 2.
Hunk #3 FAILED at 120.
Hunk #4 FAILED at 131.
Hunk #5 FAILED at 143.
Hunk #6 FAILED at 197.
Hunk #7 succeeded at 242 with fuzz 2.
Hunk #8 FAILED at 280.
Hunk #9 FAILED at 300.
Hunk #10 FAILED at 344.
8 out of 10 hunks FAILED -- saving rejects to file arch/mips/lantiq/irq.c.rej
Patch failed!  Please fix /home/mac/source/target/linux/lantiq/patches-4.14/0901-add-icu-smp-support.patch!

Not sure why.

pc2005 · 2019-01-31T23:44:56.155Z

DermotMcDonnell:

I downloaded trunk this morning (kernel 4.14.96) and placed your patches (all 5 of them) in target/linux/lantiq/patches-4.14

I downloaded current git https://github.com/openwrt/openwrt.git with last commit efa6b8b6b6463a406bbbff30839ef9c05d0daaac and it works OK. But by looking at your patch sizes, they are much bigger, did you click on "raw" or "download" button on linked webpage?

Plonk34 · 2019-02-01T20:47:09.315Z

Hi, there I can install the new patch, but it looks like not working on Easybox 904 xDSL:

root@OpenWrt:/# cat /proc/interrupts
           CPU0       CPU1       
  7:     387744     388066      MIPS   7  timer
  8:       4751       4485      MIPS   0  IPI call
  9:       9468      10813      MIPS   1  IPI resched
 62:          0          0       icu  62  1e101000.usb, dwc2_hsotg:usb1
 63:      41170          0       icu  63  mei_cpe
 72:          0      14729       icu  72  vrx200_rx
 73:          0      20599       icu  73  vrx200_tx
 75:          0          0       icu  75  vrx200_tx_2
 91:          0        178       icu  91  1e106000.usb, dwc2_hsotg:usb2
 96:          0      27498       icu  96  ptm_mailbox_isr
112:          0       8808       icu 112  asc_tx
113:          0       3011       icu 113  asc_rx
114:          0          0       icu 114  asc_err
126:          0          0       icu 126  gptu
127:          0          0       icu 127  gptu
128:          0          0       icu 128  gptu
129:          0          0       icu 129  gptu
130:          0          0       icu 130  gptu
131:          0          0       icu 131  gptu
ERR:          0

root@OpenWrt:/# cat /proc/cpuinfo
system type             : xRX200 rev 1.2
machine                 : EasyBox 904 xDSL
processor               : 0
cpu model               : MIPS 34Kc V5.6
BogoMIPS                : 332.54
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 16
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa                     : mips1 mips2 mips32r1 mips32r2
ASEs implemented        : mips16 dsp mt
Options implemented     : tlb 4kex 4k_cache prefetch mcheck ejtag llsc dc_aliases userlocal vint perf_cntr_intr_bit nan_legacy nan_2008 perf
shadow register sets    : 1
kscratch registers      : 0
package                 : 0
core                    : 0
VPE                     : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : MIPS 34Kc V5.6
BogoMIPS                : 333.82
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 16
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa                     : mips1 mips2 mips32r1 mips32r2
ASEs implemented        : mips16 dsp mt
Options implemented     : tlb 4kex 4k_cache prefetch mcheck ejtag llsc dc_aliases userlocal vint perf_cntr_intr_bit nan_legacy nan_2008 perf
shadow register sets    : 1
kscratch registers      : 0
package                 : 0
core                    : 0
VPE                     : 1
VCED exceptions         : not available
VCEI exceptions         : not available

Download speed should be 96Mb/s but it is 85Mb/s

Does it sense to test the files from beginning of this thread ?

QAuge · 2019-02-02T06:52:28.076Z

Did you changed the lantiq switch driver as well? Or only the IRQ Patches?
I tried to use all patches from the mailing list, but I can't get activated the wifi for example.
EDIT: Nevermind. I mixed up IRQ and How can we make the lantiq xrx200 devices faster. The latter one causes some problems ( My testing branch is located here ).

Plonk34 · 2019-02-02T22:57:59.763Z

Great !
I use the files from Post #1 + the script from Post 7#
And it looks like working:

[    0.076718] smp: Bringing up secondary CPUs ...

[    0.082140] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.

[    0.082154] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes

[    0.082299] !!!!!!!!init secondary smp-mt.c

[    0.082319] CPU1 revision is: 00019556 (MIPS 34Kc)

[    0.113360] Synchronize counters for CPU 1: 

[    0.136976] smp finish!

[    0.136980] done.

[    0.145775] === MIPS MT State Dump ===

[    0.149588] -- Global State --

[    0.152715]    MVPControl Passed: deadbeef

[    0.156881]    MVPControl Read: 00000003

[    0.160874]    MVPConf0 : b8008403

[    0.164344] -- per-VPE State --

[    0.167558]   VPE 0

[    0.169727]    VPEControl : 00000001

[    0.173373]    VPEConf0 : 800f0003

[    0.176847]    VPE0.Status : 1100ff00

[    0.180605]    VPE0.EPC : 8000d470 r4k_wait_irqoff+0x1c/0x24

[    0.186308]    VPE0.Cause : 50808000

[    0.189954]    VPE0.Config7 : 80080400

[    0.193771]   VPE 1

[    0.195941]    VPEControl : 00000001

[    0.199588]    VPEConf0 : 802f0003

[    0.203061]    VPE1.Status : 1100ff00

[    0.206810]    VPE1.EPC : 80018ff0 vsmp_smp_finish+0x3c/0x5c

[    0.212522]    VPE1.Cause : 50808000

[    0.216168]    VPE1.Config7 : 80080400

[    0.219985] -- per-TC State --

[    0.223111]   TC 0

[    0.225194]    TCStatus : 18102000

[    0.228666]    TCBind : 00000000

[    0.231973]    TCRestart : 8000d468 r4k_wait_irqoff+0x14/0x24

[    0.237780]    TCHalt : 00000000

[    0.241079]    TCContext : 4a04c89e

[    0.244638]   TC 1 (current TC with VPE EPC above)

[    0.249499]    TCStatus : 00000001

[    0.252971]    TCBind : 00200001

[    0.256278]    TCRestart : 8007daa0 printk+0x10/0x30

[    0.261304]    TCHalt : 00000000

[    0.264603]    TCContext : 284c1256

[    0.268161]   TC 2

[    0.270245]    TCStatus : 00000400

[    0.273717]    TCBind : 00400001

[    0.277020]    TCRestart : fdaeaa4e 0xfdaeaa4e

[    0.281529]    TCHalt : 00000001

[    0.284828]    TCContext : 2a6cc6e3

[    0.288386]   TC 3

[    0.290470]    TCStatus : 00000400

[    0.293942]    TCBind : 00600001

[    0.297244]    TCRestart : 90840b2e 0x90840b2e

[    0.301754]    TCHalt : 00000001

[    0.305053]    TCContext : acfc2301

[    0.308611] ===========================

[    0.312760] smp: Brought up 1 node, 2 CPUs

[    0.321970] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns

root@OpenWrt:/# cat /proc/interrupts
           CPU0       CPU1       
  0:       1597       1534      MIPS   0  IPI_resched
  1:        741       7335      MIPS   1  IPI_call
  7:      40889      41039      MIPS   7  timer
  8:          0          0      MIPS   0  IPI call
  9:          0          0      MIPS   1  IPI resched
 62:          0          0       icu  62  1e101000.usb, dwc2_hsotg:usb1
 63:        314          0       icu  63  mei_cpe
 72:       6227          0       icu  72  vrx200_rx
 73:          0       6278       icu  73  vrx200_tx
 75:          0          0       icu  75  vrx200_tx_2
 91:        178          0       icu  91  1e106000.usb, dwc2_hsotg:usb2
112:        891          0       icu 112  asc_tx
113:          0        154       icu 113  asc_rx
114:          0          0       icu 114  asc_err
126:          0          0       icu 126  gptu
127:          0          0       icu 127  gptu
128:          0          0       icu 128  gptu
129:          0          0       icu 129  gptu
130:          0          0       icu 130  gptu
131:          0          0       icu 131  gptu
ERR:          0

I will do more tests when the build are complett ready

@QAuge

Did you changed the lantiq switch driver as well?

As if I'm deserting what I'm doing
But i guess no, but i must do a look into the vr9.dtsi
Unfortunately i have not build the wifi driver yet, because buildproblems.

pc2005 · 2019-02-03T08:51:33.157Z

I've sent a benchmark mail to the openwrt devel mailing list. If somebody wants a tarball upload somewhere, please tell me where.

Plonk34 · 2019-02-03T20:33:52.327Z

I have do some little test but my VDSL2 speed does not increase, it is beetween 80Mb/s and 88Mb/s and should 95Mb/s (Data Rate)
But i thing the VDSL2 Modem does not 100% working.
It shows me Annex A but my Provider have Annexe B and i have choice an Annex B firmware
"option ds_snr_offset" Have no effect
And normally i should have 100Mb/s and last but not least i have no idea how much my ISP reduce my line at the moment.

I remember i have reached the same speed without the irq-balancing patch

With only one core + telefon support i get only 70Mb/s

pc2005 · 2019-03-14T04:46:07.292Z

New combined patches in thread How can we make the lantiq xrx200 devices faster.

pc2005 · 2019-05-17T03:46:59.070Z

Finally sent patch to openwrt-devel, you can watch the discussion in this mailing list thread.

chw22 · 2019-05-29T07:55:42.636Z

Hi,

as was already pointed out on the mailing list, your mail client appears to have broken the patch.
Any chance for another resend and or upload somewhere?

I'm also interested in the eth driver patches, as I only get about 80 MBit/s throughput on LAN with both of my FRITZ!Box 3370s.
Are these still up to date?

Thanks.

pc2005 · 2019-06-02T03:28:35.047Z

Hi,

All new patches are here. The last ethernet patch is in the "[OpenWrt-Devel,RFC,v4] lantiq: IRQ balancing, ethernet driver, wave300" thread. I don't have a working snapshot now as I'm trying to find the best way to implement 5+ port variants of the platform.

The broken patch is fine, you just need to correct the lines ;-). I will post better series when I manage to create the ethernet one.

ahmar16 · 2019-06-02T06:36:48.808Z

Well I'll just stick to the old ones for now. I don't wanna rebuild my firmware from scratch. I'll wait for it to get merged so I can use it in the next major update maybe. Hopefully it will get merged by then.

chw22 · 2019-06-04T12:34:12.549Z

Hi again.

So, I've fixed up the smp patch and applied the latest ethernet driver patch, as suggested.
I've also changed the target optimization flags to:

-O2 -pipe -mno-branch-likely -march=34kc -mmt -mdsp -mtune=34kc

With all this the iperf3 LAN throughput did go up about 10 Mbit/s, so it's roughly around 90 Mbit/s now.
I have no idea if it happened before the patches, but during the iperf3 tests [ksoftirqd/1] uses up to 50% of the CPU according to top.
So, the box just chokes on interrupts.

Also, enabling/disabling software flow offloading does not do anything at all.

Anything else I can try?

pc2005 · 2019-06-05T02:26:53.419Z

If your modem and your host machine really have 1Gbit ports both, then 90Mbps is still sort of low. But if your modem has MII1 (WAN port) it could be incompatibility of my betaversion of the ethernet driver which doesn't support it (currently working on it so my driver can fully replace the old one).

Note my tests were done with a minimal openwrt configuration (no WIFI enabled, no DSL, only the cable to the host, no NAT, no web configuration). Any of these can impact the speed negatively. Also does your modem use both VPEs for the openwrt or is one VPE reserved for the voice firmware? There still may be bugs for a single VPE configurations. A single VPE system will impact the speed too.

Please try to run the iperf on a vanilla kernel so we can exclude vanilla oriented speed impacts (there were some vanilla slowdowns around 4.14.100, so if it goes up 10Mbps it was either fixed or it was improved by my patches ). Also run the iperf on the different LAN ports of your modem (to see if it really works and if your modem uses WAN port we will know).

BTW if you run iperf on localhost of your modem (both iperf server and client on the modem), you should observe speeds around 300-400Mbps. It would be nice to know how fast does your openwrt configuration works, localhost is just memcopy oriented test, so my patches should not impact it and it basicaly benchmarks the raw kernel network stack. Only thing from my patches which could impact the performace that way would be probably a high IRQ load. But if there is no other LAN/VDSL/WIFI traffic, it should not matter that much. For my setup the TX speed from xrx200 never crossed the maximal localhost speed.

The IRQ driver needs to be tested (thats why it is not in the vanilla patch queue yet). There is a define:

#define AUTO_AFFINITY_ROTATION

Which basically route every odd IRQ to the first VPE and every even IRQ to the second VPE. This will assure that IRQ are autobalanced. For low frequency interrupts it doesn't impact the speed, probably even RX DMA fifo for ethernet is fine. But TX DMA fifo is impacted by it (altought only about 33% in my setup, the TX speed is much higher than 90Mbps). You can disable this function by commenting out the macro and recompiling the kernel (but all interrupts will be on VPE0, which will again impact the TX speed) ... or you can disable the rotation in the running kernel by limiting the VPEs on which it can rotate the affinity:

echo 1 > /proc/irq/73/smp_affinity #TX FIFO for VPE0
echo 2 > /proc/irq/75/smp_affinity #TX FIFO for VPE1

You can set this thing even with AUTO_AFFINITY_ROTATION commented out. And you can set it for every interrupt from ICU. Possible interrupts can be seen in:

cat /proc/interrupts

Also, enabling/disabling software flow offloading does not do anything at all.
What command did you use? My ethernet driver has only added the support of fragmented packets (parts of the packet may be in a different location in the vanilla driver it must be first copied to a linear buffer, my driver just skip this step).