not sure if it will help you, i fixed the latency problem on mine by disabling odhcpd's NDP proxying - i couldn't work out why, but something to do with the way it sends pings to trigger the kernel to do a neighbour solicitation/advertisement and then responds. instead i had to update the ndppd package to the master branch so it would update the routing tables (enable autowire on the proxy entries).
this branch uses master of my fork (which is currently the same as the source master, i just didn't want it changing underneath me): https://github.com/facboy/routing/tree/ndppd