Also check that in network/interfaces/global packet steering checkbox is enabled.
The load should be +/-10% between cores and certainly not hitting 100% on one core.
Current source of latency is few packets dropped and retransmitted due to unavailable CPU time.