Normally I'd be suspicious of the GND connection, but that would be less likely to experience on two different routers.
Another potential cause could be the voltage level of your USB2serial adaptor, and the pl2303 family has a few members that are 5V based. Confirming (measuring) its voltage level (Vcc <--> GND) would be a first step, getting a cheap cp2102 adapter might be another option.