I'd try two things
disable auto neg, and force 1gbit full duplex using ethtool (if it works for RTL NICs)
boot some recent Linux dist from a flash drive, see if the same issue occurs there
Tried the first, nothing happened. Even when auto is on, it is negotiated at 1gbit. Also tried disabling energy efficient stuff on the switch still the same.
So quickly booted an ubuntu 23.04 with 6.2 kernel, and... everything is perfect there.
Same r8169 driver... so i guess it's probably the kernel. 5.15 is quite old. Also it feels like the intel pstate driver behaves differently on 6.2, though nothing scientific i can provide to prove that.
Instead of waiting for the stable coming after the one currently in RC2, try going backwards..
I mentioned, that i tried 22.03.5 briefly, and it behaved exactly like 23.05-rc2.
Next step to try would be to virtualize the whole thing, it seems like i won't be able to avoid it.
Well, there's 22.03, and so on, too ....
great testing! Will try to replicate your findings and report here.
Fisihed debugging and found the solution.
It seemed fine on ubuntu 23.04, but after installing proxmox ve 8.0 it behaved exactly the same. Both have 6.2 kernel, so it's not in the kernel. Did some more digging, ended there: https://bugzilla.redhat.com/show_bug.cgi?id=1671958
Issue is exactly like mine. It seems there are some problems with realtek nics and aspm.
Long story short, on proxmox i can fix the nics with:
echo 0 > /sys/class/net/eth0/device/link/l1_aspm
echo 0 > /sys/class/net/eth1/device/link/l1_aspm
I'll reinstall openwrt and check it with that as well. I hate realtek nics.
A clean way should be to add this udev rule:
DRIVER=="r8169", ATTR{link/clkpm}="0", ATTR{link/l1_1_aspm}="0", ATTR{link/l1_1_pcipm}="0", ATTR{link/l1_2_aspm}="0", ATTR{link/l1_2_pcipm}="0", ATTR{link/l1_aspm}="0"
Update:
Reinstalled openwrt, but it does not have any link under /sys/class/net/eth0/device/ .
So the root cause is known, but i have no clue how to apply this onto openwrt.
Maybe someone has some idea.
2 Likes
Checked it, not working.
Unfortunately the x86 target kernel needs CONFIG_PCIEASPM=y in the kernel config to expose this functionality.
bummer.
I assume the param isn't set on the 6.2 kernel you'd tested, so it's fixed in higher kernel versions.
I'd still try the 22.03, it could be a new bug, even though the RH bugzilla report if for kernel v4.
The param is set on both 6.2 kernels i tested with. This just exposes the sysfs knobs. Ubuntu has an udev rule that disables aspm on r8169 pcie links, while the proxmox kernel does not. Openwrt x86 kernels are not compiled with that config option. Interestingly, some other platforms do have it.
Also, ASPM controls are not exposed in the device bios either so can't fix it there.
I might create a ticket on github to include this config flag in x86 kernel, though i guess that won't go anywhere anytime soon, if ever.
2 Likes
Wow, I wouldn't have expected that, so it seems to be a device quirk ...
It's probably easier to get it enabled for x86, than it'd have been for other platforms, since there aren't any real space restrictions, compared to "regular" routers.
It's no big deal, if the kernel grows by a couple kb ...
Excellent detective work, btw.
I see you did, thnx.
Adding the link for future reference
1 Like
Seems the ASPM config's been added ?
Yep. Now i just have to figure out, how to set specific sysfs stuff to "0" without udev on boot, that does not look totally dirty 
why not simply use the grub kernel command line workaround ?
I have no clue how would that affect pcie nvme power saving states. I really do not want to nuke that.
Hi, have you solved this issue, t8plus is one of devices I'm evaluating for purchase. Thanks
Yes it works ok (as a router) with the two echo lines in a startup script using 23.05-rc4? or 23.05-snapshot. Nothing else to add 
1 Like