Adblock-lean: set up adblock using dnsmasq blocklist

Wizballs · May 21, 2023, 2:23am

Feels like we can relax for a little whilte after these pretty significant updates :). Well, until the next good ideas come along anyway.
Food for thought, do you think leave the github script defaulting to current two big lists. Or go back to just one eg OISD big? Next thing I'm looking for is a cookie consent blocker list. Which won't get them all anyway due to DNS blocking method, but hey some is better than none.

Lynx · May 21, 2023, 7:16am

Ah yes good point. Perhaps we should both experiment with different lists and try to work out what works best for a good default. Stopping those stupid cookie consent things would be terrific. Those have spoilt browsing experience for years.

@Wizballs I agree we are almost in a position where we can call it a day for a while, but two more things come to mind.

Firstly, since we added support for an allowlist (default path /root/adblock-lean/allowlist) shouldn't we add support for a manual blocklist (default path /root/adblock-lean/blocklist)? That would just be a case of appending the manual blocklist to the downloaded blocklist part(s), and then business as usual, right?

Secondly, what about pixelserv? pixelserv is used a lot by the AsusWrt Merlin users and serves up a transparent pixel in place of adverts:

What's the status of pixelserv in OpenWrt I wonder? @stangri?

I just came across this thread by @kvic:

Amusingly, the second link in that old opening post now leads to spam.

There is also this:

but that is pretty old.

Wizballs · May 21, 2023, 11:40am

Definitely not finished with this project, there just had some good progress last two weeks!
Pixelserv looks interesting but I'll read more to understand it properly. Does seem to be both pros and cons from a quick read.

As for downloadable Allowlist(s), could potentially just work like Blocklist download, append each together etc. And depending on the list syntax, may just need a sed line to format lines in a similar manner.

Lynx · May 21, 2023, 1:39pm

I meant manual blocklist specified in file /root/adblock-lean/blocklist in form:

site1.com
site2.com

Woud you be able to come up with an awk/sed line to convert those lines into the dnsmasq format that I can append to /tmp/blocklist?

@colo on IRC suggests:

sed 's@.*@local=/&/@'

Lynx · May 21, 2023, 7:55pm

@Wizballs I have added two new commits to 'testing':

and

Now an optional local blocklist can be used in the form:

site1.com
site2.com

The so-called 'preprocessed blocklist' is now formed from:

the local blocklist; and/or
downloaded blocklist file part(s),

and must have at least one line.

This preprocessed blocklist is then processed as before.

Also, I altered the dnsmasq check because I just noticed today that the blocked URLs on my system now seem to show up as NXDOMAIN:

root@OpenWrt-1:/tmp/dnsmasq.d# nslookup 0-29.com
Server:         127.0.0.1
Address:        127.0.0.1:53

** server can't find 0-29.com: NXDOMAIN

** server can't find 0-29.com: NXDOMAIN

So I've changed the dnsmasq check to:

        for domain in google.com amazon.com microsoft.com
        do
                if ! nslookup "${domain}" &> /dev/null
                then
                        log_msg "Lookup of '${domain}' failed with new blocklist."
                        return 1
                elif nslookup "${domain}" | grep -A1 ^Name | grep -q '^Address: *0\.0\.0\.0$'
                then
                        log_msg "Lookup of '${domain}' resulted in 0.0.0.0 with new blocklist."
                        return 1
                fi
        done

in order to check for either NXDOMAIN or 0.0.0.0.

New example output:

root@OpenWrt-1:/etc/init.d# service adblock-lean start
Started adblock-lean.
No local blocklist identified.
Downloading new blocklist file part(s).
Downloading file part from: https://big.oisd.nl/dnsmasq2.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Downloading file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Successfully generated preprocessed blocklist file with 668143 line(s).
Processing and checking new blocklist file.
Cleaning whitespace and formatting blocklist file.
Whitepsace removed and formatting completed.
Removing duplicates from blocklist file.
Duplicates removed.
No local allowlist identified.
Checking for any rogue elements.
New blocklist file check passed.
Restarting dnsmasq.
Checking dnsmasq instance.
The dnsmasq check passed with new blocklist file.
New blocklist installed with good line count: 403557.

Please could you (and anyone else interested) stress test the new code in 'testing' in order to test the local blocklist and the local allowlist with or without downloaded lists?

Wizballs · May 22, 2023, 11:34am

Ah ok manual blocklist, I misread somehow

I called this out in the old thread, but might have been missed. OISD at some point changed to ...../ (NXDOMAIN) instead of ...../# (0.0.0.0) so we just followed suit.

Can also use ~ as the suffix instead of if you don't want it looking like email address
sed 's~.*~local=/&/~'

I'll give a test run in next day or two.

Lynx · May 23, 2023, 9:24am

Do we force everything to / (NXDOMAIN) instead of /# (0.0.0.0) or do we allow both? If the latter, should we force one or the other? It might be weird if we allow both and have some domains returning NXDOMAIN and others returning 0.0.0.0.

Shall we force NXDOMAIN, and then this code:

for domain in google.com amazon.com microsoft.com
        do
                if ! nslookup "${domain}" &> /dev/null
                then
                        log_msg "Lookup of '${domain}' failed with new blocklist."
                        return 1
                elif nslookup "${domain}" | grep -A1 ^Name | grep -q '^Address: *0\.0\.0\.0$'
                then
                        log_msg "Lookup of '${domain}' resulted in 0.0.0.0 with new blocklist."
                        return 1
                fi
        done

Can be simplified just to:

for domain in google.com amazon.com microsoft.com
        do
                if ! nslookup "${domain}" &> /dev/null
                then
                        log_msg "Lookup of '${domain}' failed with new blocklist."
                        return 1
                fi
        done

Wizballs · May 23, 2023, 10:37am

Hey Lynx, everything is being set as / (NXDOMAIN) via this line. It's cleaning whitespace, and converting anything with /# to just /

# Clean whitespace and format all entries as local=/.../
sed -i -e '\~^\s*$~d;s/^[ \t]*//;s/[ \t]*$//;s/^address/local/;s/^server/local/;s/#$//' /tmp/blocklist

It could easily be set to go either way, but that is how it is set now, largely due to OISD switching to this. I don't think there is a definite winner between NXDOMAIN and 0.0.0.0 anyway, both have pros and cons vs the other.

Another reason for not allowing a mix of both is that it makes duplicate checking harder, but also slower as it will no longer be exact line matches. Substrings will need to be extracted first etc etc. So yeah, stick to one format.

Lynx · May 23, 2023, 12:26pm

OK thanks for the clarification. Defaulting to NXDOMAIN seems ideal. But perhaps we should add a toggle to switch between NXDOMAIN and null IP?

Is it just these lines that would need to change:

# your suggested replacement for sed 's@.*@local=/&/@'
sed 's~.*~local=/&/~' 

# Clean whitespace and format all entries as local=/.../
sed -i -e '\~^\s*$~d;s/^[ \t]*//;s/[ \t]*$//;s/^address/local/;s/^server/local/;s/#$//' /tmp/blocklist

What would the strings need to be to make it all /#?

I can make, and weave in, a simple toggle variable into the relevant calls.

No rest for the wicked ;) - albeit we've probably added about all there is to add then!

dave14305 · May 23, 2023, 12:45pm

Null IP alone is insufficient for iOS devices where HTTPS (Type65) queries are prevalent. That’s why I use both address and local in my own script.

Lynx · May 23, 2023, 1:01pm

Thanks for your input here @dave14305 .

How do you mean?

For our purposes would you recommend just a toggle, like I'm proposing? That is, all blocks go to either NXDOMAIN or null IP in dependence on user toggle?

dave14305 · May 23, 2023, 1:16pm

I prefer null IP myself for A and AAAA queries using address=/example.com/# and block other query types using local=/example.com/. Just using address wasn’t enough to block Type65, which is probably why Oisd switched format.

I wouldn’t bother offering a toggle, unless it’s doing null IP plus NXDOMAIN when toggling null IP on.

Lynx · May 23, 2023, 1:20pm

I am confused because I thought with the dnsmasq style blocking one can choose either for any one entry like so:

# NXDOMAIN
local=/zzzregsizzz.com.ru/

OR

# null IP
local=/zzzregsizzz.com.ru/#

But reading what you write above it seems the situation is more complicated and/or I am not understanding something.

How can one have both say for zzzregsizzz.com.ru?

dave14305 · May 23, 2023, 1:36pm

# return 0.0.0.0 or :: for A or AAAA queries
address=/zzzregsizzz.com.ru/#
# return NXDOMAIN for all other query types
local=/zzzregsizzz.com.ru/

Lynx · May 23, 2023, 1:40pm

Ah, gotcha. Might a sensible default be to have everything NXDOMAIN:

local=/zzzregsizzz.com.ru/

But an optional 'generate_null_IP_entries' toggle to additionally include for every entry:

address=/zzzregsizzz.com.ru/#

dave14305 · May 23, 2023, 1:46pm

This is what I do:

# address=/example.com/#  # Returns 0.0.0.0 or :: for blocked domains
# local=/example.com/     # Prevents HTTPS (type65) queries from being forwarded
grep -E "^local=/[A-Za-z0-9._-]+/$" /tmp/oisd/dnsmasq2 | \
grep -E -v -i -f /etc/oisd/whitelist | \
awk -F/ ' { printf "address=/%s/#\nlocal=/%s/\n", $2, $2 } ' > /tmp/oisd/oisd.new

Lynx · May 23, 2023, 1:49pm

OK thanks for your input here.

@Wizballs - phew - any thoughts on the above? Seems I opened a bit of a can of worms. Perhaps just leaving things as NXDOMAIN is best.

Wizballs · May 23, 2023, 7:35pm

Of course having both would double memory usage, which isn't a problem for many people / routers anyway. A toggle wouldn't be that hard however...

To help me understand this a bit better, can I ask why you prefer null for A and AAAA queries? There must be some benefit if you going to the effort.

dave14305 · May 23, 2023, 7:44pm

Old habits from previous router platforms and their adblock solutions. Most of the OpenWrt solutions rely on NXDOMAIN, so I doubt I’m smarter than the collective wisdom. But I prefer to respond to a blocked query instead of saying it doesn’t exist.

Wizballs · May 23, 2023, 7:49pm

Got it thanks. Ah yes ok this old debate, I remember both options had pro and cons.

@Lynx maybe ask everyone/anyone if they want this toggle/functionality? So everyone/anyone out there want this?

NXDOMAIN is working well for me as is, with no noticeable side effects