Unfortunately, it doesn't compile for me:
/lede1701/build_dir/target-arm_cortex-a15+neon-vfpv4_musl-1.1.16_eabi/linux-ipq806x/shortcut-fe/sfe_ipv4.c:1366:5: error: 'struct sk_buff' has no member named 'fast_forwarded'
Does that look familiar?
Edit: I see that this error is because the patches in hack-4.4 hasn't been applied. I didn't manually copy the patches to the patches-4.4 directory, which I should have. I'll leave this post here anyway, even if it shows my utter lack of understanding... 
Edit2: It doesn't seem to affect my performance issue. If anything, it actually got worse (but that could be just a coincidence, though, as I didn't test much). Ah well, thanks for suggesting it anyway.