Orange Pi R2S: Packet Steering, Flow Offloading and Gigabit Throughput Issues

Hello,

I have an Orange Pi R2S device (Ky X1 8-core RISC-V AI CPU).
The ONT is connected directly to the WAN port and PPPoE is configured.
IPv6 is disabled.

I tested the following configurations:

  • Packet steering OFF, flow offload OFF
    → ~400 / 400 Mbps down/up
  • Packet steering ENABLED, flow offload OFF
    → ~780 / 900 Mbps down/up
  • Packet steering ENABLED (all CPUs), flow offload OFF
    → ~910 / 450 Mbps down/up
  • Packet steering ENABLE (normal, not all CPUs) or DISABLE, hardware flow offload ON
    → ~930 / 930 Mbps down/up

Software flow offloading does not seem to have any noticeable effect.

Enabling hardware flow offloading solves the throughput issue, but then SQM cannot be used.
Without hw-offloading, I cannot reach full gigabit speeds.

My questions are:

  1. Is it possible to achieve ~930 / 930 Mbps purely in software (CPU path) without enabling offload on this device?
  2. Why does upload speed drop significantly when packet steering is set to “enable all CPUs”?
  3. What is the best packet steering / offload combination for this device and for achieving symmetric gigabit speeds?

Thank you in advance for your help.

cat /proc/interrupts(with packet steering enable all cpu);

root@OpenWrt:~# cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
 11:      40698      32605      29054      26321      25115      53410      24441      26150  RISC-V INTC   5 Edge      riscv-timer
 12:          0          0          0          0          0          0          0          0  SiFive PLIC  60 Edge      pinctrl
 13:          0          0          0          0          0          0          0          0  SiFive PLIC  58 Edge      x1-gpio
 14:          6          0          0          0          0          0          0          0  SiFive PLIC  72 Edge      pdma
 15:          0          0          0          0          0          0          0          0  SiFive PLIC  36 Edge      d4010800.i2c
 16:          0          0          0          0          0          0          0          0  SiFive PLIC  37 Edge      d4011000.i2c
 17:          0          0          0          0          0          0          0          0  SiFive PLIC  38 Edge      d4012000.i2c
 18:       2198          0          0          0          0          0          0          0  SiFive PLIC  19 Edge      d401d800.i2c
 19:          0          0          0          0          0          0          0          0  SiFive PLIC  64 Edge      spm8821
 30:          0          0          0          0          0          0          0          0   spm8821  10 Edge      rtc@pmic
 57:          0          0          0          0          0          0          0          0   spm8821  37 Edge      ky_pwrkey_rise
 58:          0          0          0          0          0          0          0          0   spm8821  38 Edge      ky_pwrkey_fall
 59:          0          0          0          0          0          0          0          0   spm8821  39 Edge      ky_pwrkey_skey
 60:          0          0          0          0          0          0          0          0   spm8821  40 Edge      ky_pwrkey_lkey
 63:          0          0          0          0          0          0          0          0  SiFive PLIC 146 Edge      x1-pcie
 66:          0          0          0          0          0          0          0          0   PCI-MSI 134217728 Edge      PCIe PME, aerdrv
 67:          0          0          0          0          0          0          0          0  SiFive PLIC 147 Edge      x1-pcie
 70:          0          0          0          0          0          0          0          0   PCI-MSI 268435456 Edge      PCIe PME, aerdrv
 71:          0          0          0          0          0          0          0          0  SiFive PLIC  87 Edge      JPU
 72:          0          0          0          0          0          0          0          0  SiFive PLIC  86 Edge      v2d-irq
 73:          0          0          0          0          0          0          0          0  SiFive PLIC  42 Edge      UART1
 74:          0          0          0          0          0          0          0          0  SiFive PLIC  44 Edge      UART3
 76:    1700587          0          0          0          0          0          0          0  SiFive PLIC 131 Edge      eth0
 77:     724339          0          0          0          0          0          0          0  SiFive PLIC 133 Edge      eth1
 78:          0          0          0          0          0          0          0          0  SiFive PLIC 149 Edge      dwc3-usb-wakeup
 79:          0          0          0          0          0          0          0          0  SiFive PLIC 125 Edge      xhci-hcd:usb2
 80:          0          0          0          0          0          0          0          0  SiFive PLIC 105 Edge      mv_udc, ehci_hcd:usb4
 81:          0          0          0          0          0          0          0          0  SiFive PLIC 106 Edge      usb-wakeup
 82:          0          0          0          0          0          0          0          0  SiFive PLIC 118 Edge      ehci_hcd:usb1
 83:          0          0          0          0          0          0          0          0  SiFive PLIC 148 Edge      usb-wakeup
 84:          0          0          0          0          0          0          0          0  SiFive PLIC  74 Edge      c0500000.linlon-v5
 85:          0          0          0          0          0          0          0          0  SiFive PLIC  81 Edge      x1ccic
 86:          0          0          0          0          0          0          0          0  SiFive PLIC  82 Edge      x1ccic
 87:          0          0          0          0          0          0          0          0  SiFive PLIC  83 Edge      x1ccic
 88:          0          0          0          0          0          0          0          0  SiFive PLIC  84 Edge      mars-cpp
 89:          0          0          0          0          0          0          0          0  SiFive PLIC  79 Edge      feisp
 90:          1          0          0          0          0          0          0          0  SiFive PLIC  61 Edge      thermal_zone0, thermal_zone1
 91:       9048          0          0          0          0          0          0          0  SiFive PLIC 101 Edge      mmc2
 92:          0          0          0          0          0          0          0          0  SiFive PLIC 113 Edge      ky-crypto-engine-0
 93:         12          0          0          0          0          0          0          0  SiFive PLIC  52 Edge      d4013400.mailbox
 94:          0          0          0          0          0          0          0          0   PCI-MSI 134742016 Edge      eth2-0
 95:          0          0          0          0          0          0          0          0   PCI-MSI 134742017 Edge      eth2-1
 96:          0          0          0          0          0          0          0          0   PCI-MSI 134742018 Edge      eth2-2
 97:          0          0          0          0          0          0          0          0   PCI-MSI 134742019 Edge      eth2-3
 98:          0          0          0          0          0          0          0          0   PCI-MSI 134742020 Edge      eth2-4
 99:          0          0          0          0          0          0          0          0   PCI-MSI 134742021 Edge      eth2-5
100:          0          0          0          0          0          0          0          0   PCI-MSI 134742022 Edge      eth2-6
101:          0          0          0          0          0          0          0          0   PCI-MSI 134742023 Edge      eth2-7
102:          0          0          0          0          0          0          0          0   PCI-MSI 134742024 Edge      eth2-8
103:          0          0          0          0          0          0          0          0   PCI-MSI 134742025 Edge      eth2-9
104:          0          0          0          0          0          0          0          0   PCI-MSI 134742026 Edge      eth2-10
105:          0          0          0          0          0          0          0          0   PCI-MSI 134742027 Edge      eth2-11
106:          0          0          0          0          0          0          0          0   PCI-MSI 134742028 Edge      eth2-12
107:          0          0          0          0          0          0          0          0   PCI-MSI 134742029 Edge      eth2-13
108:          0          0          0          0          0          0          0          0   PCI-MSI 134742030 Edge      eth2-14
109:          0          0          0          0          0          0          0          0   PCI-MSI 134742031 Edge      eth2-15
110:          0          0          0          0          0          0          0          0   PCI-MSI 134742032 Edge      eth2-16
111:          0          0          0          0          0          0          0          0   PCI-MSI 134742033 Edge      eth2-17
112:          0          0          0          0          0          0          0          0   PCI-MSI 134742034 Edge      eth2-18
113:          0          0          0          0          0          0          0          0   PCI-MSI 134742035 Edge      eth2-19
114:          0          0          0          0          0          0          0          0   PCI-MSI 134742036 Edge      eth2-20
115:          0          0          0          0          0          0          0          0   PCI-MSI 134742037 Edge      eth2-21
116:          0          0          0          0          0          0          0          0   PCI-MSI 134742038 Edge      eth2-22
117:          0          0          0          0          0          0          0          0   PCI-MSI 134742039 Edge      eth2-23
118:          0          0          0          0          0          0          0          0   PCI-MSI 134742040 Edge      eth2-24
119:          0          0          0          0          0          0          0          0   PCI-MSI 134742041 Edge      eth2-25
120:          0          0          0          0          0          0          0          0   PCI-MSI 134742042 Edge      eth2-26
121:          0          0          0          0          0          0          0          0   PCI-MSI 134742043 Edge      eth2-27
122:          0          0          0          0          0          0          0          0   PCI-MSI 134742044 Edge      eth2-28
123:          0          0          0          0          0          0          0          0   PCI-MSI 134742045 Edge      eth2-29
124:          0          0          0          0          0          0          0          0   PCI-MSI 134742046 Edge      eth2-30
125:          0          0          0          0          0          0          0          0   PCI-MSI 134742047 Edge      eth2-31
126:          0          0          0          0          0          0          0          0   PCI-MSI 268959744 Edge      eth3-0
127:          0          0          0          0          0          0          0          0   PCI-MSI 268959745 Edge      eth3-1
128:          0          0          0          0          0          0          0          0   PCI-MSI 268959746 Edge      eth3-2
129:          0          0          0          0          0          0          0          0   PCI-MSI 268959747 Edge      eth3-3
130:          0          0          0          0          0          0          0          0   PCI-MSI 268959748 Edge      eth3-4
131:          0          0          0          0          0          0          0          0   PCI-MSI 268959749 Edge      eth3-5
132:          0          0          0          0          0          0          0          0   PCI-MSI 268959750 Edge      eth3-6
133:          0          0          0          0          0          0          0          0   PCI-MSI 268959751 Edge      eth3-7
134:          0          0          0          0          0          0          0          0   PCI-MSI 268959752 Edge      eth3-8
135:          0          0          0          0          0          0          0          0   PCI-MSI 268959753 Edge      eth3-9
136:          0          0          0          0          0          0          0          0   PCI-MSI 268959754 Edge      eth3-10
137:          0          0          0          0          0          0          0          0   PCI-MSI 268959755 Edge      eth3-11
138:          0          0          0          0          0          0          0          0   PCI-MSI 268959756 Edge      eth3-12
139:          0          0          0          0          0          0          0          0   PCI-MSI 268959757 Edge      eth3-13
140:          0          0          0          0          0          0          0          0   PCI-MSI 268959758 Edge      eth3-14
141:          0          0          0          0          0          0          0          0   PCI-MSI 268959759 Edge      eth3-15
142:          0          0          0          0          0          0          0          0   PCI-MSI 268959760 Edge      eth3-16
143:          0          0          0          0          0          0          0          0   PCI-MSI 268959761 Edge      eth3-17
144:          0          0          0          0          0          0          0          0   PCI-MSI 268959762 Edge      eth3-18
145:          0          0          0          0          0          0          0          0   PCI-MSI 268959763 Edge      eth3-19
146:          0          0          0          0          0          0          0          0   PCI-MSI 268959764 Edge      eth3-20
147:          0          0          0          0          0          0          0          0   PCI-MSI 268959765 Edge      eth3-21
148:          0          0          0          0          0          0          0          0   PCI-MSI 268959766 Edge      eth3-22
149:          0          0          0          0          0          0          0          0   PCI-MSI 268959767 Edge      eth3-23
150:          0          0          0          0          0          0          0          0   PCI-MSI 268959768 Edge      eth3-24
151:          0          0          0          0          0          0          0          0   PCI-MSI 268959769 Edge      eth3-25
152:          0          0          0          0          0          0          0          0   PCI-MSI 268959770 Edge      eth3-26
153:          0          0          0          0          0          0          0          0   PCI-MSI 268959771 Edge      eth3-27
154:          0          0          0          0          0          0          0          0   PCI-MSI 268959772 Edge      eth3-28
155:          0          0          0          0          0          0          0          0   PCI-MSI 268959773 Edge      eth3-29
156:          0          0          0          0          0          0          0          0   PCI-MSI 268959774 Edge      eth3-30
157:          0          0          0          0          0          0          0          0   PCI-MSI 268959775 Edge      eth3-31
IPI0:      2159       2487       2582       2441       2417       2182       2696       2408  Rescheduling interrupts
IPI1:      9558      31765     385773      97839     206227     230585      93692     102282  Function call interrupts
IPI2:         0          0          0          0          0          0          0          0  CPU stop interrupts
IPI3:         0          0          0          0          0          0          0          0  CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0          0          0          0          0  IRQ work interrupts
IPI5:         0          0          0          0          0          0          0          0  Timer broadcast interrupts

Even with packet steering set to "all CPUs", network interrupts still appear to be processed mainly by CPU0.

root@OpenWrt:/proc/irq/76# cat /proc/irq/76/smp_affinity
ff
root@OpenWrt:/proc/irq/76# cat /proc/irq/77/smp_affinity
ff
root@OpenWrt:/proc/irq/76# 

Hi,

Try playing around with the IRQs as described here

The rps/xps cpu’s can bound to multiple cores, the actual nic irq’s only bind to a single core.

Try irqbalance in place of steering.

Thanks for the help.
I tried different irqbalance and rps/xps config but they didn't really help.

In the end, I moved the LAN interface from eth1 to eth2.(eth0,eth1 -> 1Giga // eth2,eth3 -> 2.5G different chipset)

with irqbalance on and rps set to ff -> I get around 930/890 down/up, which is almost perfect.

Howewer, under load the CPUs get too hot and after some time the device becomes unstable. Also, SQM is not available on RISC-V. I can't install.
For this device, hardware offloading seems to be the best option.

You do not need both irqbalance AND packet steering.