I'm only speaking from my hands-on experience with my x86 hardware. Just for giggles (and a sanity check), I switched off TurboBoost in real-time on my router just now:
echo 1 | tee /sys/devices/system/cpu/intel_pstate/no_turbo
Running speed tests from an ethernet connected host and I'm getting ~940mbps both directions (I have 1Gb symmetric fiber and running SQM). Even with TurboBoost switched off, there is so much CPU headroom with my i7, it's not even funny. My router load average goes to ~0.35 at the 1Gb line-rate. Also, I see zero measurable effect on jitter even with TurboBoost off. I'm in the 0.10ms - 0.18ms range with TurboBoost on or off.
I don't disagree with you at all. I think my point is that these two things are not necessarily give or take. In other words, you can select a reasonable i5/i7 CPU with a low TDP (now called Processor Base Power) and still disable TurboBoost if it truly isn't needed. This gives you the low idle usage you called out, but also keeps the CPU from hitting higher temps during TurboBoost if it really has little measurable effect on performance (as I demonstrated in my particular use-case above).
But, should the OP need TurboBoost in their particular scenario--so let's say it does provide @kaivorth a measurable effect--then leave it enabled. No biggy.