I have also TL-WDR4300 and I tested you scenario. I tested it by run iperf test through ssh connection to my remote server.
After 5 minutes I did not notice any degradation in speed. CPU usage was the same, at low level. Please check your connection in /sys/fast_classifier/debug_info. My looks as follows
root@OpenWrt:~# cat /sys/fast_classifier/debug_info | grep ":22"
o=1, p=6 [dc:a9:04:88:84:52]:192.168.0.64:49569 my_remote_server_IP here:22:[b8:af:67:70:86:7f] m=00000000 h=128
o=1 means that connection is offloaded. You can check this state before and after performance is degrading