OrangePi Zero H2+ having issues with 100baseT

Hello there!
I've been a long OpenWrt time user in all sorts of hardware, and a Linux-only user as well.
I got two of these boards, with different ram sizes. They both exhibit the same behvaiour:

Whenever they are set to work at 100baseT (100mbps ethernet) the transmission does not work well. So things like downloading big files work well and fast, since the board only needs to reply with TCP Acks every now and then, that seems fine. However if I try to upload a file, or fetch a file via scp on the device, or use it for VPN, it just does not work. The speed goes to a few KBs per sec, like 10KB/s or so.

My workaround at the moment (since I use it as VPN and I do not need that much bandwidth) is:
ethtool -s eth0 speed 10 duplex full

This makes it work at 10mbps but works well overall. However the other mode is broken. Since this is happening to two boards I suspect the HW is actually OK. I tried connecting the board to other ports in the router (it sits next to a fiber router) but does not improve.

Any ideas on how can I debug this? I checked a few wireshark dumps taken at the device and I can only see a ton of waiting time between packets. It seems the other node does not get the packets (and obviously does not send ACKs), causing the device to eventually retransmit the packet which at some point makes it through. So unless the packets are somehow not 100% kosher and the router decides to drop them, or something fishy is going on (perhaps the NIC drops them without acutally putting them in the wire?).

Thanks a ton!