I don't know if it makes any difference, but I have already tried passing through my NIC to the VM (and I got the same error). Booting on bare metal is a bit of an inconvenience, but I'll try with a spare Raspberry Pi 4 I have.
So it's definitely something weird with how my ISP handles things. A Tp-Link router from 2014 connects just fine (with minimal configuration) while both a Proxmox VM and a Raspberry Pi 4 get the same error
Okkk, so I found out that:
1- I'm an idiot
2- I'm definitely an idiot
3- I'm probably the stupidest person on this planet
4- The MAC address printed on the back of my old router is being used for LAN, while it uses the same MAC address incremented by 1 for WAN. So, using the WAN address fixed everything