ACEMAGICIAN T8PLUS (Intel N95)

darksky · June 30, 2023, 4:34pm

In your diagram, is your "switch" is replaced by your T8 using the same network architecture and same physical cables?
What is your "switch?"
What kernel is your T8 running?

My T8's:

/etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1d:692b:58dc::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'
	option ipv6 '0'

config bridge-vlan
	option device 'br-lan'
	option vlan '3'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '4'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '5'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'eth0:t'

config device
  option type 'bridge'
  option name 'lxcbr0'
  option ipv6 '0'
  option bridge_empty '1'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'

config interface 'guest'
	option device 'br-lan.3'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'

config interface 'homeoffice'
	option device 'br-lan.4'
	option proto 'static'
	option ipaddr '10.9.6.1'
	option netmask '255.255.255.0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'iot'
	option device 'br-lan.5'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'

config interface 'lan'
	option device 'br-lan.10'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'

config interface 'lxc'
	option device 'lxcbr0'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'

config interface 'wg0'
...

fodiator · June 30, 2023, 5:49pm

@darksky -
thank you for your prompt feedback and config.

Regarding your questions:

Not an in-place swap, but reconnected cables to the T8 (instead of wall outlet). Same cables were used, just total length was shorter. Cables were swapped to rule out cable issue
Cisco SG350-20
"kernel": "5.10.176"

Essentially I had the same set up as you (but without VLANs). DHCP client on eth1 (WAN), static on eth0 (LAN).
I then moved them into the same bridge and zone for the re-test, without effect on the iperf3 results.

I’ll have a closer look at htop as suggested by @konus and @frollic above.

I’ll report back with more detail

darksky · June 30, 2023, 6:17pm

I am using a snapshot I build which is running the 5.15 series of kernel so if your network hardware is identical, recommend that you try a snapshot.

EDIT:

We are not running the same hardware at all:

# ubus call system board
{
	"kernel": "5.15.119",
	"hostname": "t8plus",
	"system": "Intel(R) N95",
	"model": "Default string Default string",
	"board_name": "default-string-default-string",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "SNAPSHOT",
		"revision": "r0+23454-01885bc6a3",
		"target": "x86/64",
		"description": "OpenWrt SNAPSHOT r0+23454-01885bc6a3"
	}
}

https://www.cpu-monkey.com/en/compare_cpu-intel_celeron_n5095-vs-intel_processor_n95

fodiator · June 30, 2023, 8:42pm

Yes, you are right - mine is the T8 Pro (not T8 plus).

Glad you’re not seeing the issue. Maybe it’s just the T8 Pro. Will post findings here.

-- EDIT --
Tried again, different cables & different machines (xeon && i7) - issue has gone away (kind off)
Iperf3 now reports ~700 Mbps in either direction, both forward and reverse testing.

Hooking up the raspberry pi shows the same behaviour - so probably bad cable (connection). Still puzzled why that would be unidirectional...

In comparison to my switch, there is still a ~30% performance hit (~700 vs ~940 Mbps) when using the T8pro. Maybe caused by the slower CPU as compared to the T8plus.

fodiator · July 1, 2023, 7:46pm

Network interface is indeed the same:

root@OpenWrtX86:~# lspci
...
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
03:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8821CE 802.11ac PCIe Wireless Network Adapter

darksky · July 6, 2023, 4:28pm

Yes, apparently it's a bind mount at work, see: /boot is mounted twice - #4 by qwrty-ftw

devast · August 8, 2023, 1:10pm

I've been having roughly the same issue since i got my T8plus. It's still in the testing phase, and i could narrow my issue to this:
Somehow, the nics in this mini pc do not like switches.
If i do a direct cable between a pc and the device ( doesnt matter if its the lan or wan interface) iperf3 results are stable ~950mbit. If i put any switch ( i tried unifi switch and a cisco) between a pc and the device, one direction in iperf3 will be ~80-90mbit, while the reverse will be ~940mbit. And it gets more weird... If i do iperf3 with 4-5 parts, sometimes it does ~940mbit combined, sometimes ~70-80mbit.
Also, for a quick test i just plugged the WAN port into my existing network, disabled the input fw on it and did some speed tests with the ookloa speedtest script. My gigabit fiber had like... 4 mbit down and ~900mbit up according to it. If i leave out the switch connecting directly to my old routers lan port, its 940/940. Something is wrong here but i can't really find it why.
Tried replacing cables, changing ports on the switch, changing ports on the t8plus, nothing really helps. Also its the same with 23.05-rc2 and 22.03.5.
Does anyone have any advice about this? I can't put this into production in this state...

Thanks!

frollic · August 8, 2023, 1:14pm

I'd try two things

disable auto neg, and force 1gbit full duplex using ethtool (if it works for RTL NICs)
boot some recent Linux dist from a flash drive, see if the same issue occurs there

devast · August 8, 2023, 2:24pm

Tried the first, nothing happened. Even when auto is on, it is negotiated at 1gbit. Also tried disabling energy efficient stuff on the switch still the same.
So quickly booted an ubuntu 23.04 with 6.2 kernel, and... everything is perfect there.
Same r8169 driver... so i guess it's probably the kernel. 5.15 is quite old. Also it feels like the intel pstate driver behaves differently on 6.2, though nothing scientific i can provide to prove that.

frollic · August 8, 2023, 3:16pm

Instead of waiting for the stable coming after the one currently in RC2, try going backwards..

devast · August 8, 2023, 3:25pm

I mentioned, that i tried 22.03.5 briefly, and it behaved exactly like 23.05-rc2.
Next step to try would be to virtualize the whole thing, it seems like i won't be able to avoid it.

frollic · August 8, 2023, 3:25pm

Well, there's 22.03, and so on, too ....

fodiator · August 8, 2023, 8:13pm

great testing! Will try to replicate your findings and report here.

devast · August 8, 2023, 8:29pm

Fisihed debugging and found the solution.
It seemed fine on ubuntu 23.04, but after installing proxmox ve 8.0 it behaved exactly the same. Both have 6.2 kernel, so it's not in the kernel. Did some more digging, ended there: https://bugzilla.redhat.com/show_bug.cgi?id=1671958
Issue is exactly like mine. It seems there are some problems with realtek nics and aspm.
Long story short, on proxmox i can fix the nics with:

echo 0 > /sys/class/net/eth0/device/link/l1_aspm
echo 0 > /sys/class/net/eth1/device/link/l1_aspm

I'll reinstall openwrt and check it with that as well. I hate realtek nics.

A clean way should be to add this udev rule:

DRIVER=="r8169", ATTR{link/clkpm}="0", ATTR{link/l1_1_aspm}="0", ATTR{link/l1_1_pcipm}="0", ATTR{link/l1_2_aspm}="0", ATTR{link/l1_2_pcipm}="0", ATTR{link/l1_aspm}="0"

Update:
Reinstalled openwrt, but it does not have any link under /sys/class/net/eth0/device/ .
So the root cause is known, but i have no clue how to apply this onto openwrt.
Maybe someone has some idea.

frollic · August 8, 2023, 8:37pm

Check out https://bugzilla.redhat.com/show_bug.cgi?id=1671958#c73 too.

devast · August 9, 2023, 7:59am

Checked it, not working.
Unfortunately the x86 target kernel needs CONFIG_PCIEASPM=y in the kernel config to expose this functionality.

frollic · August 9, 2023, 8:09am

bummer.
I assume the param isn't set on the 6.2 kernel you'd tested, so it's fixed in higher kernel versions.

I'd still try the 22.03, it could be a new bug, even though the RH bugzilla report if for kernel v4.

devast · August 9, 2023, 8:26am

The param is set on both 6.2 kernels i tested with. This just exposes the sysfs knobs. Ubuntu has an udev rule that disables aspm on r8169 pcie links, while the proxmox kernel does not. Openwrt x86 kernels are not compiled with that config option. Interestingly, some other platforms do have it.
Also, ASPM controls are not exposed in the device bios either so can't fix it there.
I might create a ticket on github to include this config flag in x86 kernel, though i guess that won't go anywhere anytime soon, if ever.

frollic · August 9, 2023, 8:33am

Wow, I wouldn't have expected that, so it seems to be a device quirk ...

It's probably easier to get it enabled for x86, than it'd have been for other platforms, since there aren't any real space restrictions, compared to "regular" routers.
It's no big deal, if the kernel grows by a couple kb ...

Excellent detective work, btw.

frollic · August 11, 2023, 5:20pm

I see you did, thnx.

Adding the link for future reference

github.com/openwrt/openwrt

r8169 ASPM issues on x86

opened 09:00AM - 09 Aug 23 UTC

closed 06:12PM - 14 Aug 23 UTC

devastgh

bug

### Describe the bug r8169 has long standing issues with pcie link aspm. Spe…ed is very slow for rx. Currently the known workarounds are - enabling flow control on the other end of the cable OR - disabling aspm in the bios of the device for that pcie link, if the configuration is exposed OR - disabling aspm in sysfs for the pcie device In the current x86 target pcie aspm controls are not exposed in sysfs. Please add ``` CONFIG_PCIEASPM=y CONFIG_PCIEASPM_DEFAULT=n ``` options to the x86 target kernel. ### OpenWrt version r23228-cd17d8df2a ### OpenWrt target/subtarget x86/64 ### Device Default string Default string ### Image kind Official downloaded image ### Steps to reproduce _No response_ ### Actual behaviour _No response_ ### Expected behaviour _No response_ ### Additional info _No response_ ### Diffconfig _No response_ ### Terms - [X] I am reporting an issue for OpenWrt, not an unsupported fork.