Also, for adding to the "solution": most OEM will set the clock to performance in order to reduce latency even when the network is quiet. OpenWrt uses the ondemand governor (which I personally doubt it's a wise choice). What are you seeing is a small and variable delay when Linux checks your firewall rules.
May look at this for a real life example (from Linksys):