I have a question regarding the performance impact on DNS queries when running Adblock. While my router with 128MB RAM can easily handle relatively large adblock lists, I am noticing a lineair performance loss depending on list size, or a O(n) complexity. I am using dnsmasq as the DNS forwarder.
Is there a way to make dnsmasq use a hashtable for looking up entries in the list, and hence a O(1) complexity, or put differently, DNS performance independent of list size? If not, is it worth switching to the unbound backend? Ie does unbound support hashtables for the lookup?
Do you use trunk builds?
Just of curiosity does adding:
option min_cache_ttl '1800'
to /etc/config/dhcp under config dnsmasq make things a little bit better?
For example, some sites like google use very small TTL values of like 5min.
Adding:
option nonegcache '0'
option cachesize '10000'
Maybe is also worth a try.
Changing caching behavior might mask the problem in some cases, but it will never improve uncached results, which will still suffer a pretty severe performance penalty due to the slow O(n) lookups. I really want my uncached results to be O(1) through proper use of hashtables.
Does it feel faster, or do you also happen to have objective performance figures?
No, I have not benchmarked it
The first lookup with unbound can be slow too.
But after the cache is build up everything loads instantly.
You can make a script to save and load the cache. (using unbound-control)
When you have to reboot the device you can load cache.
I also have prefetching enabled and set a min cache TTL of 1h.
But unbound also uses a little bit more of ram.
I put it to the test. I fetched the list adblock created from the router. I then edited into the host file format, and uploaded it back unto the router. I then manually imported the host file in dnsmasq to make sure addn-hosts is used. Lo and behold, adblocking was working, and DNS performance was not impacted at all!
@dibdot would it be possible to change the workings of this wonderful adblocking package to create and import a list via addn-hosts instead of the currently used method? The speed-up is insane. It now performs the same as pi-hole with very large lists!
@shm0 Where did you find above documentation? I am unable to create a file by specifying no IP (which should result in NXDOMAIN). This forces me to add each domain twice to the list, once for IPv4 and once for IPv6. While it works, the list is needlessly larger, which is a bummer with the constrained space routers have.
For my tests I had to use a IPv4 only list, or the list wouldn't fit on the router. To test whether it was blocked properly I used ping -4 to force IPv4 pings to a few domains on the list. They all resolved to 127.0.0.1 instead of NXDOMAIN, which isn't ideal. To test the performance, I reran the benchmark, which showed unimpaired performance compared to not running the host file.
In principle it's no problem to add another dnsmasq output variant. Said that, did the described approach deliver "NXDOMAIN" in return? What about IPv4/IPv6? If I remember correctly, the addn-host-method blocks only on single host level, e.g. 'doubleclick.net' blocks only this host and not 'www.doubleclick.net', is this assumption correct? If so, then this method probably leads into much bigger lists with serious memory constraints ...
No, unfortunately via the manually crafted host file as posted above, localhost is returned. Having said that, according to the following documentation posted by @shm0 it should be possible. I just haven't managed to find the correct format/syntax of the host file:
Unfortunately, both IPv4 and IPv6 require a separate entry (=big host file) according to the following documentation:
Having said that, seeing as specifying no IP should result in NXDOMAIN, I am hoping this would work for both at the same time (it would only make sense). I am still struggling with getting the format/syntax correct though.
That is correct unfortunately.
Is there even a list available that contains all subdomains?
How much larger would you expect the list to be?
I have a few questions for people following this discussion:
Does anyone happen to know the correct syntax for addn-hosts to return NXDOMAIN?
Does anyone happen to know how the filtering is done for Pi-hole? They used to use dnsmasq, and are now using a dnsmasq fork. The Raspberry Pi 1st generation is really slow, yet more than adequate for large lists. Does anyone know how they handle the blocking? At first sight, addn-hosts seems to have quite a few large drawbacks, but then again, so does having multi-second DNS resolution times as can be seen from my benchmarks above.
...
To include multiple IP addresses for a single query, use --addn-hosts= instead.
...
As for --server , one or more domains with no address returns a no-such-domain answer, so --address=/example.com/ is equivalent to --server=/example.com/ and returns NXDOMAIN for example.com and all its subdomains. An address specified as '#' translates to the NULL address of 0.0.0.0 and its IPv6 equivalent of :: so --address=/example.com/# will return NULL addresses for example.com and its subdomains.
Did you try to create an addn-hosts file in the following format?
address=/example.com/
address=/example2.com/
address=/example3.com/
.
.
.
I am fairly sure I did, because that's the format adblock uses by default (or am I mixing things up?). I tried to load that one directly via addn-hosts, and it would fail.