I've read reports that the choice of USB Ethernet for the second NIC matters when using the Raspberry Pi 4 as a router. In particular, RTL8153-based NICs work better than basically any other USB NIC. My hypothesis is that the AX88179 can't handle the UDP traffic. I can't say for sure as I'm using a Raspberry Pi CM4 with RTL8111, which is a PCIe NIC.
Some packet loss is normal and expected when transiting the global Internet. Protocols like TCP and QUIC do a really good job of hiding this from applications. The fact that it works much better when using your desktop's NIC supports the hypothesis that the USB NIC is the problem.
One way to test this without buying new hardware is to swap the LAN/WAN roles of the two NICs. That is, use the built-in NIC on the Raspberry Pi for the WAN connection and do the UDP test from the Raspbery Pi itself.