Ultimate SQM settings: Layer_cake + DSCP marks


NDPI can't decrypt proper encryption, it can only identify which certificates are in use:

From their website:

Handling Encrypted Content

The trend of Internet traffic is going towards encrypted content often using SSL. In order to let nDPI support encrypted connections, we have added a decoder for SSL (both client and server) certificates, thus we can figure out the protocol using the encryption certificate. This allows us to identify protocols such as Citrix Online and Apple iCloud that otherwise would be undetected.

Although Squid can't decrypt either (unless you set your clients up to trust squid's own certificates) It can do better than this by looking at the domain name you're connecting to and prioritizing based on that.

Is this perhaps an issue with your browser and cpu usage on your machine? Some web pages require a fair amount of work to render. Also another thing that can cause this is DNS latency, perhaps increase the size of the DNSmasq cache?


says in /etc/config/dhcp

option cachesize 1000

will make dnsmasq keep the last 1000 dns lookups in cache


That's good, and you're absolutely right that these kinds of connections are difficult to classify without something like NDPI. I wonder though about how long before encryption makes this kind of thing impossible.


It's not a CPU nor browser problem, my laptop CPU is 8th gen cpu (intel core i7 8750H). also the same problem
on my phone which SD845 CPU !
this only happen when network is loaded with a download!
i don't think it's a DNS latency problem,this happen with page refresh too, but i think it's a something with TCP like delayed SYN or ACK!
i will try increase DNS cache size but will this affect memory usage on router?


Lets use it until it's get useless like L7.
I don't know how isp's get managed everything, i think it's UTM firewall magic!
my isp is using a cache server called Thunder cache, how do they cache youtube?!, i saw my isp ip's when loading a youtube video (googlevideo.com)


For the most part they don't :wink:

How do they cache YouTube? My guess is they have squid acting as a transparent proxy and they redirect the query to a Google owned massive cache box directly installed on site. Then google hands the data to squid, squid hands the data to you.

Yes, but I can't imagine 1000 DNS entries takes very much memory. Should be something like say a few hundred bytes for the domain name and a few bytes for the ip addresses. Let's call it a kB per entry, so 1000 of them should be around a MB. So set your cache to somewhere between 200 and 500 if you need to shave down a bit of memory usage.

Try prioritizing DNS requests, this sounds like latency caused by DNS being stalled. it's one of the most irritating things about bufferbloat. It works like this, suppose you have 500ms of latency. Then you type in somesite.com to your browser. the browser looks up somesite.com (takes 500ms) then connects (takes 500ms) then downloads the content of the page... which has additional scripts an things, so now we need to look up say 20 or 30 other DNS entries, (takes somewhere between 500ms and 20*500ms) and then you make further connections (takes several 100ms) and then you finally get all the content... so 10s later you finally can display the web-page. Actual time spent transferring data could be maybe 1s but time spent waiting for DNS requests could be 5, 10, even 20s.

in this kind of case caching DNS is really important. Each website can easily require you to lookup tens of DNS entries, so it's very plausible that even if you looked something up recently you don't have it in cache unless you have multiple hundreds of DNS cache entries.

But the other thing that you should do is tag DNS queries as something like CS4 or AF41 and up-prioritize them on your WAN outbound.


For cache they are using thunder cache, but don't know if they are changed to squid. it's a transparent proxy.
But HTTPS pages and downloads is not cached, i think it's duo to certification and SSL.

1000 is ok for now!

Thanks for clarifications, I already tagged DNS(port 53) traffic,but i forgot that i used port 5353 opendns
dns provider, so i head now and tagged port 5353, let check and see if this will help fix the delay!

EDIT: can snort be useful with QOS ?


Googling around it seems "Thunder Cache" is just some brand name custom configuration for squid :wink:

Squid can't cache things that are SSL/TLS because it can't see the content. But what it can do is redirect requests to the "close" servers. So if you try to load some URL from say googlevideo.com (the source of actual youtube video files) instead of using whatever DNS tells it to use for googlevideo.com it could use a known device that google has placed in the ISP server cabinet on the ISP local network.

Yeah, that could be a DNS starvation issue, so hopefully that helps! One of the things about QoS is that once you start prioritizing things specially, if you accidentally down-prioritize other stuff that should be prioritized, it can be starved.

I haven't played with it. I've used ntop to identify what's going on on my network in the past in order to get an idea of if there are issues that need to be addressed like huge numbers of connections open or long running flows that need down-prioritizing etc. But you wouldn't want to run it constantly. If you detect there's a problem, firing up ntop during that problem can be super helpful. I do think Snort could inspect traffic and dynamically alter the prioritization based on what it discovered, kind of like the NDPI stuff, but again, lots of CPU required.

What is the main goal? If it's to specifically down-prioritize long-running flows, then the conntrack bytes mechanism is probably a good one and doesn't require anything special. If it's to up-prioritize voice and gaming type stuff that happens on unpredictable "random" ports, then this is probably the hardest thing to do. Sometimes the easiest thing is to create a whole separate VLAN for high priority stuff. So you could have an SSID called "voipwifi" and also a router port labeled "gaming" and plug your latency sensitive devices into these ports/SSIDs.

Another option would be to use the QoS tagging in Windows to tag outbound packets from your games with DSCP, and then connmark these flows and use the connmark to mark inbound packets as well...

The biggest difficulty I think is with stuff like WhatsApp/Signal/Whatever encrypted voice calls to random IPs on random ports. But, then you could connect your phone to a special "voipwifi" SSID for calling and tag everything going through that with DSCP=CS6 for example.

Another idea I just had is to try identifying real-time information by the fact that it typically goes over UDP and has a steady flow (ie. a packet every 10 or 20 or 50 ms) so that it's less than say 100 packets per second and less than say 500kbps, the idea is first tag all udp as CS6, then once it exceeds our limits, tag it back down to CS1 assuming it's not a real-time flow (this code not tested)

udp dport > 1024 ct state established,related meter udprtflow {ip saddr . udp sport . ip daddr . udp dport timeout 3s limit rate over 40 kbytes/second burst 20 kbytes } ip dscp set cs1
ip protocol udp ip dscp != cs1 ip dscp set cs6

Steady flows of game control information and voip voice packets etc that stay below the 40 kbytes/second should wind up tagged high priority here, and everything else should start out high priority and rapidly be down-prioritized if it's a high bandwidth flow, and then stay down prioritized. The only concern is if your game briefly exceeds the bandwidth limit it will get down prioritized permanently so it's a good idea to set the limit say twice as high as you expect the game flow to be :wink:

I'm pretty sure you can do this with hashlimit in iptables doing a little googling as well.


this is the official thunder-cache website: https://bmsoftware.org/
thanks for clarifying this about youtube!

this help a bit, but i think this is duo to my slow connection; prioritizing small ACK's seems help to reduce
the loading time!

I was doing some research's, that's why i took longer time to respond to your reply, then i found a better approach, to avoid complications.
i think it's possible to use connbyte and maybe length to tag UDP traffic, usually voip and gaming traffic have
a small bytes size and small length than downloads.
i have a capture for voip call:
if you see that packet length is from 92 and can grow to 273 and maybe more!
so i don't know which value should i use for length match or connbytes!?

i use windows DSCP tags, but i don't know how to use connmark in this case.

As i said before we can avoid complexity, by using either connbytes or length match.

Interesting idea, sensitive traffic always use UDP, so it's somewhat easy to control.
I saw that you used NFT code here, it's seems nice, but i think it's still hard to watch the packets flow and
rate in kbps.
Yeah i read about hashlimit, but i think there's still a better way!
i saw many are using this way to identify interactive traffic:

# small packet is probably interactive or flow control
    iptables -t mangle -A OUTPUT -p tcp --sport $ip_port -m length --length 0:500 -j RETURN

    # small packet connections: multi purpose (don't harm since not maxed out)
    iptables -t mangle -A OUTPUT -p tcp --sport $ip_port -m connbytes --connbytes 0:250 --connbytes-dir both --connbytes-mode avgpkt -j RETURN

i will edit those lines according to my setup.
Cheers :wink::innocent:


In your router, match packets outbound from your windows machine that have a given DSCP and then set the connmark. On inbound from the internet, restore the connmark and then DSCP tag connections that have the connmark so you get the same DSCP tag on inbound packets.

Yeah, the connbytes avgpkt mode seems appropriate but I would do it for UDP not tcp. It's rare that you're going to get realtime interactive TCP streams (ssh is one exception).


hmm, connmark will work only for windows, but i don't have a problem with games on pc cause they are using
a known ports!
the problem is with the phone, it's possible to use dscp tag and iptables uid to tag games and apps packets
but this is for rooted phones only, non rooted phone's will not permit to execute commands!

Yeah, i know it's useful for UDP only in this case. i don't use ssh by the way. for now i don't have aproblem with tcp!
how should i calculate connbytes avgpkt?!, and length for length match!


If you have a packet capture of several tens of seconds of the kind of traffic you're interested in, I'd just take the average packet size you saw, multiply by say 1.2 and set that as your avgpkt size.


Edit: tbh i should probably not clog up this thread at all as this is pretty much unrelated to OpenWRT.


Hi, you are welcome.

hmm, maybe game menu is showing ping using ICMP. and use tcp or udp to show in game ping.
i think your isp is some how add more latency or they treat your dscp tags strangely.

no problem, but it's related to bufferbloat, you can start a new thread if you like.
also sorry i was just busy, that's why my response is late


No worries! I'm not mad xD

Yea I dont know but only thing i know:

WTFast, pingkiller etc that can change routes, didnt work.

  • Speedify fixes it, just a VPN really. Enabling/disabling encryption had no effect. Nor changing TCP/UDP
  • OpenWRT is unrelated to issue
  • Disable Windows QoS tagging had no effect.

I dont know about deep packet inspection from ISP? Mhh... What about ECN? I enabled that in my network adapter so, i believe this is also a packet tag.


This would be thwarted by encryption, but you say disabled encryption still works, so I don't think so.

I'm having trouble following the issue because it is split around several threads, I suggest to start your own thread specific to why you have different bufferbloat conditions under different circumstances where the new thread will be easier to follow your specific issue and you can explain what tests you did etc there


This looks still like routing/peering issue to me...


Sure i will try to help you!, i'm waiting for your new thread!
@dlakelan @moeller0
i'm still can't/don't know how to calculate connbytes avgpkt size, because for voip packet size will start
from 60 the after some minutes it will rise slowly until it reach 1208 and more, for games it will start from 45
to 555 and above.


That's weird, it seems like probably there's something wrong with the calculation. You might do well to just prioritize small udp packets and set the upper threshold based on the biggest packet you see from your voip packet capture of 10 seconds or so, rather that trying to get it to calculate average packet size using conntrack.


i used this rule to tag small udp packets that's have the specified length:
$IPT -t mangle -A PREROUTING -p udp -m multiport ! --ports 53,80,443,60887 -m length --length 28:1500 -j DSCP --set-dscp-class CS6
it's working and tagging the games and voip packets in the specified length, but the only problem it tag
only the ingress traffic but not egress traffic.
another idea comes to my mind is rules like this:
$IPT -t mangle -A PREROUTING -p udp -m conntrack --ctorigsrc -m multiport ! --ports 53,80,443,60887 -m length --length 28:1500 -j DSCP --set-dscp-class CS6


this will match (almost) any packet, might as well leave it out.