Fisihed debugging and found the solution.
It seemed fine on ubuntu 23.04, but after installing proxmox ve 8.0 it behaved exactly the same. Both have 6.2 kernel, so it's not in the kernel. Did some more digging, ended there:
Issue is exactly like mine. It seems there are some problems with realtek nics and aspm.
Long story short, on proxmox i can fix the nics with:

echo 0 > /sys/class/net/eth0/device/link/l1_aspm
echo 0 > /sys/class/net/eth1/device/link/l1_aspm

I'll reinstall openwrt and check it with that as well. I hate realtek nics.

A clean way should be to add this udev rule:

DRIVER=="r8169", ATTR{link/clkpm}="0", ATTR{link/l1_1_aspm}="0", ATTR{link/l1_1_pcipm}="0", ATTR{link/l1_2_aspm}="0", ATTR{link/l1_2_pcipm}="0", ATTR{link/l1_aspm}="0"

Reinstalled openwrt, but it does not have any link under /sys/class/net/eth0/device/ .
So the root cause is known, but i have no clue how to apply this onto openwrt.
Maybe someone has some idea.