Not only is it possible but its downright easy!, fool proof, and if you go too high or don't give her enough of the voltage and she won't boot you can simply pop out the sd card edit the config.txt on a computer then put it back in and you are good to go!
I found a sweet spot for my Pi4B and I am at 2ghz cpu, voltage set to 3 and gpu at 750. I am using a cheap-o $5usd clear case I drilled about 25 holes in all 4 of the sides and the top of the case with a 10mm drill mounted in my drill press because I do not want to use a fan but desire great cooling at dirt cheap cost. Besides that, little fans can be very noisy they have to spin at high speed to make up for small size. Any ways I have a heat sink off of a dead router shoehorned in to the case and I epoxied it into place with special thermal epoxy! I stumbled onto the fact that the double sided thermal tape is nowhere near as good as a quality thermal paste or epoxy. My idle temps are right around 50C, full load (using stress-ng on all 4 cores) brings her up to about 65C and I am happy with that. I also have epoxied a heat sink on the ram chip, the USB controller chip and the ethernet chip. (I had extra heat sinks so I figure why not). The only part I have left to do is figure out a way to put a heat sink on the voltage regulators. I have read many sources that state they are a weak link in overclocking the Pi4B and I must say they do get very hot to the finger.
At first glance you would think the gpu freq would not matter in OpenWRT because we do not use graphics but the gpu speed is tied to other parts of the device like the cache I think it was? My memory is a bit foggy on that but it's something like the cache. I tried to duckduckgo search that but strangely I am coming up empty. So in other words, overclocking the gpu increases the performance of the cpu!
With SQM off it should be spreading packet handling among multiple CPUs through RPS. Which speed test are you using? Perhaps you are connecting to just one IP? RPS keeps packets on the same stream together on the same CPU