See Why you need at least 3Mbps upload to get good game performance with ~1500byte packets: Doing the math for why latencies are destined to be annoying until you have at least 3Mbps each way.
Try redoing your experiment with at least 3Mbps each way and see how it goes.