Is it possible that SQM / QOS interface could be made to allow greater flexibility more like the firewall application where you can set rules for control of the traffic?
The ability to group by mac, interface, device, zone, time based traffic shaping, traffic match based shaping, shape specific protocol,source destination shaping etc...
It seems QOS and SQM are relativly simple and you have to choose one or the other and any thing advanced requires custom scripting , cronjobs etc...
That's incredibly open ended and you can imagine facilitating a generic solution to cover such variety isn't easy. The existing SQM utility will get you started and covers basic use cases but for more interesting use cases like VPN use things become rather specific and a DIY solution is needed that handles your specific requirements.
What I can report is that with scripting you can achieve pretty much anything you want.
Here is what I use to set up CAKE instances and work with VPN and nftables:
This sets up two IFB interfaces (one for download and one for upload) and applies CAKE on them and this way you can control using nftables and fwmarks what gets seen and shaped by CAKE. It also allows control over DSCPs and restoring those from conntrack on download etc.
I think this is offers a fairly generic template.
You could surely adapt this solution for your requirements.
SQM is intended to work for "the rest of us" to set a decent baseline and allow non-experts to easily get access to competent traffic shaping, AQM, and scheduling for their (home-)network. So the default scripts and luci app are intended to be simplistic. However the sqm-scripts machinery allows you to easily incorporate arbitrary self-written scripts and go wild. Does that answer your request for better luci integration of advanced features? Unfortunately no, and I do not want to propose to write your own GUI.
However one thing that SQM aims for is "good enough" performance out of the box, with minimal configuration (interface, shaper rates and per-packet-overhead need to be configured explicitly). Have you tried how well that works for your use-case (I would certainly recommend cake's per-internal-IP-fairness), this will not be optimal for any network but it might already offer a better baseline over what you have now and will allow you to develop your own solution without the pressure of having to suffer abysmal responsiveness.
occasionally I feel an urge to point people to review https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper_Must_Die/
qosify might be a building block. In my case I feel an increasing urge to block facebook and twitter during business hours..
When @moeller0 says "good enough", he really means, "really, quite good..." SQM gives a simple one-stop, set-and-forget setting that allows you to minimize latency for all people on your network and share bandwidth equally between all devices.
My advice is always to try it as-is, without any further tweaks. If it works, and you don't notice any problems, then you're done. You can go fishing, watch videos, or indulge your favorite pastime.
If it's not "good enough", you have a specific set of test cases/applications that you can describe and ask for advice to address. Good luck on whichever path you take!
SQM with cake works quite well so far but not optimal for the structure of my network and my network is nothing fancy.
Basically I have two dnsmasq instances kids and adults I have setup sqm on wan side and kids/ adults Lan instances with adblocking, httpdns proxy.and dns hyjacking setup on the kids lan.
but I don't believe I can setup sqm across two dnsmasq instances to evenly share across everything I just have to half the available wan bandwith with upload and download limits on each instance but this limits devices on one instance when the other instance is unloaded.
One thing is a major pain is no time based ruling, Id like to slow an instance down to promote homework over video at speciific times, dns solutions mostly don't work because of ealilly acessable free vpn's and proxy servers, and devices defaulting to dnssec, but limiting bandwith surely does if it pauses every three seconds the kids quickly give up.
If I could limit bandwith to group of devices without totally blocking and alearting the kids to my blocking activities (stealth bandwith limiting) it would be cool.
In reality I know little of how to acomplish anything too fancy or beyond what tutiorials are posted on the wiki as I'm still new to openwrt.
Why? If you have a single wan interface, just instantiate SQM on that interface SQM does not care about your internal network structure or how or which IP/ranges you distribute to different internal hosts.
As long as you want to have some traffic by-pass these restrictions things will be fickle, if you are prepared to throttle all you internet traffic a simplish cron script should do the trick.
You really can't your users will notice that throughput is diminished, the only question is how they explain that rate reduction...
Well, there was a package called nft-qos if I recall correctly that allowed per IP throttling, which seems to be the level of control that you are after. I have never used that myself so can not add any first-hand experience based recommendations.