Hey Dan... which TP-LINK AP's are you using, or actually, what is chipset?
Don't know if it might be helpful, but on my ath10k C7's I experimented with lowering the TX queue limit... with improvements in the latency.
Now, I had more latency on the download and very little on the upload before the change, and maybe you all are running a different chipset, but it would be an easy experiment to try.
Here's a link to (hopefully) the part of the thread where it was mentioned and my trying it out: https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002/925
Tried it on the main house AP, and basically left it there the past 3 weeks, with no issues.
Edit: I should say my testing was with a Win10 client. Quick testing with a few other things handy, my Android phone had +35-45ms on download, +0ms on upload. Another Win10 laptop, had +4ms, 0ms on download/upload, but much slower download speed. Interesting that you are having loads on the upload side.