Adblock-lean: set up adblock using dnsmasq blocklist

@anon52770028 Right, I don't know how widely supported it's currently, which is why I asked this to the other person suggesting DNS over HTTPS. My comment about lack of support dates back to 2017/2018, on my old router. I didn't have any major reasons to re-consider encrypted name resolutions until now.

@Lynx It's more of a "time" thing. Time to setup and maintain yet another service. I'd hoped it would also provide DNS over TLS on its own, so I would need only one service for both features (adblocking and DNS over TLS).
I am already late with the release of my Wireless ARP Monitor application, so free time is really in short demand for me.

There isn't going to be any significant performance difference between DNS over HTTPS and DNS over TLS (DNS over TLS has less layers, and in theory should be faster).
DNSCrypt through dnscrypt-proxy2 might be slightly faster aswell as provide solutions to some of the shortcomings of the other two protocols. Less DNS servers support it though.

1 Like

I can pretty much guarantee you that adblock-lean will never include a DoT inbuilt option. It's simply not in the scope of this project. And there is little need for it, given that stubby exists.

Everyone is short on time really, but the beauty of both adblock-lean and stubby is that once setup, neither require ongoing maintenance.

1 Like

Always a question of how we prioritise things.

I setup stubby ages ago and it has just worked since. And adblock-lean surely only takes about 5-10 minutes to try (download, launch, tweak config potentially, launch again, uninstall).

I guess I drew conclusions too quickly. I had noticed my final blocklist had shrunk significantly recently, probably because complete blocklists are being ignored because of new rogue elements. Here are some examples:

Warning: Rogue element: 'server=/25z5g623wpqpdwis.onion.to/' identified originating in blocklist file part from: https://blocklistproject.github.io/Lists/dnsmasq-version/ransomware-dnsmasq.txt.

Warning: Rogue element: 'server=/10tradeoption.com/' identified originating in blocklist file part from: https://blocklistproject.github.io/Lists/dnsmasq-version/scam-dnsmasq.txt.

Warning: Rogue element: 'local=/pokolorujfb*.glt.pl/' identified originating in blocklist file part from: https://raw.githubusercontent.com/FiltersHeroes/KADhosts/master/KADdnsmasq.txt.

My reasons for thinking about skipping only the element is that, as in those cases, they are either improperly formatted or wrongly detected by adblock-lean, but they are clearly not malicious. I also see clearly the security issue with only skipping elements. I'm not sure what the best approach is while keeping things secure!

The first 2 are using outdated dnsmasq format, where instead of the keyword 'local' they are using the keyword 'server'. Currently our filter doesn't support this outdated format and hence these entries are flagged as rogue. In the upcoming release, we will transition to using raw domain lists rather than dnsmasq lists, so this sort of thing won't be an issue anymore, as long as you find an equivalent substitute for that blocklist in the raw domains format (these are sometimes also called "wildcard domains", of which there are 2 types, and the one which we will recommend to use with adblock-lean is the one which strangely has no or very few wildcards. it's a bit of a mess, I know - but the processing is faster for equivalent protection). If a substitute doesn't exist, you will be able to continue using the dnsmasq-formatted lists. We will consider allowing the above outdated format with them.

The 3rd one is flagged as rogue because it has the wildcard - * not in the very beginning of the URL. Generally wildcard processing is quite heavy on the CPU, plus they introduce some security issues, so we decided to only allow them only once in a domain name, and only in the beginning of it. We may reopen the discussion about that if this is important to you.

That said, did you consider using the Hagezi and/or the OISD lists?

Actually not necessarily outdated, they are still valid entries and can be used by dnsmasq. Earlier versions of adblock-lean would convert both "server" and "domain" (can't remember exactly, I think it was domain), to "local". It would probably be best to re-add this conversion to sed dnsmasq format lists.

This is technically a rogue entry even if we were allowing it before. From the dnsmasq instructions, dnsmasq only allows wildcards such as:
*.google.com
*.amazon.com.
etc

1 Like

"Server" and "address" should be converted to "local"
Not domain as I incorrectly mentioned earlier

You guys should be proud to know that among adblock, adblock-fast, and adblock-lean, you are the only ones who fulfill all the allowlist scenarios I harped on before:

  1. Allow a subdomain of a blocked domain (add server=/my.ads.example.com/# when ads.example.com is blocked)
  2. Allow a perfectly matched domain from the blocklist (removed by awk and explicitly allowed with server=/ads.example.com/#)
  3. Allow a higher level domain when subdomains are blocked (allow example.com when ads.example.com and tracking.example.com are in the blocklist)

Adblock only permits #1 and #2. Adblock-fast only permits #2 and #3. Adblock-lean permits #1, #2 and #3.

3 Likes

That's nice feedback cheers.
Last time I checked adblock-lean was the only one that could handle large allowlists also (eg 1,500 entries) extremely quickly.
Actually surprised the other two haven't "borrowed" this code...

1 Like

Adblock does no line removal from the blocklist at all (only local=/.../#) and Adblock-fast compiles a sed script to remove allowed domains from the blocklist, but does no local=/.../#.

I think both are constrained somewhat by the desire to be able to serve multiple DNS backends.

2 Likes

I've had a look at this code before, it is very slow on large lists (1,500 allowlist entries, 900k blocklist). I terminated the script after 30 mins as it was still running. Approx 10 seconds on ablock-lean. We put of heeeeap of effort into getting allowlist's right...

They are different scope packages in that regard, and full respect for that type of flexibility.

1 Like

Thanks for the clarifications. It is not especially important to me to allow more than one wildcard or to allow wildcards at the end, although I do enjoy using those blocklists. I do not think compromising on security for general usage is a good idea, but if the option was there to allow it in the config, I would probably use it.

As for the "server=" entries, I think it would be a good idea to re-add the conversion, unless it causes problems.

1 Like

This shouldn't be a problem to get re implemented, let you know when.

I'm going to look into this a bit more. eg why this blocklist contains these entries etc. More curious than anything.

2 Likes

Just to update you on this. It's a little bit off-topic, but considering the issue I ran into and that you guys are also running dns proxies.

I ended up installing dnsproxy; it's significantly larger than stubby, but it supports, all-in-one, DNS over TLS, DNS over HTTPS, DNS over HTTPS/3, DNS over QUIC and DNSCrypt.
Currently, I've a mix of DNS over TLS and DNS over HTTPS/3 upstream servers, but I will switch to DNS over QUIC (with DNS over TLS as fallback) once more servers start supporting it.

With that being said, I ran into a minor issue with the network.rrdns ubus call. This ubus call is part of LuCI's rpcd-mod-rrdns and is used by LuCI to perform reverse DNS lookups for some of its display information.

While network.rrdns takes an optional DNS server address and port, LuCI doesn't actually pass any to it, which causes rrdns to fallback to looking at /etc/resolv.conf, taking the first valid server, see https://github.com/openwrt/luci/blob/openwrt-23.05/libs/rpcd-mod-rrdns/src/rrdns.c#L251-L284

Now, once the dns proxy was set, dnsmasq option noresolv set to 1 and all network.[interface].dns entries removed, /etc/resolv.conf started pointing to an empty file.

root@router1:~# cat /etc/resolv.conf
# Interface wan
# Interface wan6

This causes network.rrdns to fail, if no server is explicitly passed to it as it can no longer obtain a server from /etc/resolv.conf.

A quick look at dnsmasq's service init script showed me that there is a "hidden" option that controls writing to /tmp/resolv.conf (pointed by /etc/resolv.conf).
This option is always forced to 1, if noresolv is not 1, resolvfile is /tmp/resolv.conf.d/resolv.conf.auto (or not set, defaulting to this path) and the option itself is not explicitly set in dhcp.dnsmasq. See: https://github.com/openwrt/openwrt/blob/openwrt-23.05/package/network/services/dnsmasq/files/dnsmasq.init#L1050-L1057 and https://github.com/openwrt/openwrt/blob/openwrt-23.05/package/network/services/dnsmasq/files/dnsmasq.init#L1182-L1192

I am leaving the fix here, in case any of you also run into this issue. Simply setting localuse to 1 forces the script to write a proper local resolv.conf.

uci set dhcp.@dnsmasq[0].localuse='1'
uci commit
service dnsmasq restart

Result:

root@router1:~# cat /etc/resolv.conf
search intranet
nameserver 127.0.0.1
nameserver ::1
root@router1:~# ubus call network.rrdns lookup '{"addrs":["139.178.84.217", "2604:1380:4641:c500::1"], "limit":2}'
{
        "139.178.84.217": "dfw.source.kernel.org",
        "2604:1380:4641:c500::1": "dfw.source.kernel.org"
}
2 Likes

I know this is supposed to be "lean" but is it possible to get a basic luci app to just modify the config file or is there an easy way for me to do this within luci webui?

Thank you. It's running great on a GL.iNet GL-MT6000 Flint 2 router

A luci app is in the works - see https://github.com/rickparrish/luci-app-adblock-lean.
I would however wait with installing this until the developer calls out for a (public) test

4 Likes

The next update to adblock-lean is up. It features a variety of changes, some are pretty important, so users - please read carefully. Main changes:

  • Support raw domains lists (smaller download size, improves processing speed). dnsmasq-formatted lists continue to be supported.
  • Pack entries with gawk or sed (produces smaller final blocklist file, improves processing speed)
  • Implement automatic URLs conversion from dnsmasq format to raw domains format (supports Hagezi and oisd URLs). Automatic URLs conversion only runs once.
  • Move the config file from /root/adblock-lean/config to /etc/adblock-lean/config
  • Change default local allowlist path to /etc/adblock-lean/allowlist
  • Same for local blocklist
    (these paths will change in config only when resetting the config with service adblock-lean gen_config, otherwise previously configured paths will continue to be used)
  • Stop the dnsmasq service before running dnsmasq --test on the new blocklist (frees up a lot of memory, so updating the lists becomes much less memory-intensive)
  • Implement config values validation
  • Fix restart command sometimes not working correctly
  • Support server=, address= for dnsmasq-formatted lists
  • After some performance testing, we found that GNU sed is a fair bit faster than Busybox sed. So while we still support out-of-the-box OpenWrt tools, we recommend to install GNU sed, as well as GNU awk and GNU sort. You can do this all at once:
    opkg update
    opkg install gawk sed coreutils-sort
    

To update to the latest version, run service adblock-lean update.

We did our best to implement easy automated migration from the previous versions to this one, and to test the code. But as software development goes, there might be bugs which we didn't catch. If you find a bug, please report it and it will get fixed.

The readme has been updated as well.

Enjoy!

4 Likes

TLDR :wink:
Lots of under the hood improvements for even faster processing speed, and even less memory usage during processing. This was a biiiig update.

Config file should auto-update it's own variables. This is working really well, but you can always run service adblock-lean gen_config and just start with a fresh config if all else fails.

Lots of time and effort from @antonk - awesome work! Plus all the input and testing from everyone.

6 Likes

I'm recommending to everyone to use adblock-lean, I don't even use pihole/adguard, is so simple yet elegant! thanks @Wizballs and @Lynx

i've a question, im probably using an old version from the github since is working fine for me but with these new changes and adjustments so I should update or replace with a new version?

2 Likes

Yes it’s very powerful and set and forget. I’ve used it for ages now and it just works.

Regarding upgrading, I’d keep a backup of your config and try updating - the new code is much faster, uses less memory and has many other nifty improvements thanks to a complete overhaul by @antonk with input from myself and @Wizballs. We’ve had fun trying to optimise the code as much as possible, facilitating use of ever larger blocklists for a given router.

Hopefully the update process will take care of pulling from your old config. If not you may just have to generate new config and manually copy over the equivalent settings.

5 Likes