Adblock-oisd : 22.03 allows you to use huge blocklists with dnsmasq

Lynx · February 27, 2023, 7:48pm

But browsing patterns vary from day to day.

dave14305 · February 27, 2023, 10:19pm

I’ve contemplated something like this before.

Run for a week or so with a full Adblock list and dnsmasq logging enabled.
Gather all blocked domains from the dnsmasq log.
Make a smaller block list based on those actual results.
Every n days, compare the latest downloaded blocklist with the queried domains in the dnsmasq log to see if any would-be-blocked domains detected. If so, add them to the custom blocklist.
Every week or so, check if any domains in the block list are not being hit anymore in the dnsmasq log. After a suitable waiting period, remove them.

I just never did anything about it.

Lynx · February 27, 2023, 10:25pm

Seems an intriguing concept but isn't it somewhat academic since the OISD list seems to work perfectly well without using much memory anyway?

dave14305 · February 27, 2023, 10:28pm

Sure. This was on that other router platform before dnsmasq 2.86 made larger lists more efficient to use.

castillofrancodamian · February 28, 2023, 4:50pm

Already with the first point, I see that it can handle a complete list, therefore the following points are unnecessary. The idea was that the "heavy" work would be done by some service like Github and the router would only have to update the blocklist every so often.

castillofrancodamian · February 28, 2023, 4:50pm

That is why it would have to be updated from time to time.

castillofrancodamian · February 28, 2023, 4:51pm

But on what kind of hardware? It is still limited by the amount of RAM.

Lynx · February 28, 2023, 7:06pm

Well it's completely academic now isn't it since dnsmasq 2.86 has rendered even mediocre devices able to use enormous lists like oisd that deal with everything anyway, right?

castillofrancodamian · February 28, 2023, 7:10pm

No matter how many versions of dnsmasq there are, I doubt it can do "magic". Even the "small" version makes the 64 MB RAM device very slow. And I don't know what you mean by "academic". And that the people of OISD do a great job, do not mean that it is specific and ultra optimized for my use.

This is how the router is seen today:

This is how Oisd Big's list is seen:

And this having the 5GHz band disabled.

Wizballs · February 28, 2023, 8:21pm

Have you tried using the oisd small list?

castillofrancodamian · February 28, 2023, 9:05pm

No block list:

With "Small" lock list:

Wizballs · February 28, 2023, 9:40pm

Looks like a good option for low mem devices. Unless you want to use an external device eg pihole, or external DNS adblockers eg nextdns, adguard.

castillofrancodamian · February 28, 2023, 9:45pm

Apparently, it is a good result. I don't understand why it was like that and with adblock and simple-adblock had "unpleasant" results.

Lynx · February 28, 2023, 9:47pm

Interesting. You can use the service script in this thread with minimal change I imagine?

@Wizballs perhaps we should generalized this out to a variable defined at the start of the service file?

github.com

lynxthecat/adblock-oisd/blob/9462b71b9ad14eeed1a64d1e66d7ce57d98b335c/adblock-oisd#L106


      
          	if [[ "${RANDOM_DELAY}" == "1" ]]; then
          		random_delay_mins=$(($(hexdump -n 1 -e '"%u"' </dev/urandom)%60))
          		log_msg "Delaying download by ${random_delay_mins} minutes (thundering herd prevention)."
          		sleep "${random_delay_mins}m"
          	fi
          
          
	log_msg "Downloading new oisd.txt."
          	
                  for retries in 1 2 3
                  do
                          wget https://dnsmasq2.oisd.nl -O /tmp/oisd.txt &> /dev/null
                          if [[ ${?} -eq 0 ]]; then
          			log_msg "Download of new oisd.txt file suceeded."
          			break
          		fi
          	
                          sleep 5
          	done
          	
          	if [[ ! -f /tmp/oisd.txt ]]; then
          		log_msg "Download of new oisd.txt file failed."

castillofrancodamian · March 1, 2023, 12:41am

Don't you think simple-adblock should do this job?

Wizballs · March 1, 2023, 2:09am

Only one way to find out really

dave14305 · March 1, 2023, 2:21am

I wanted a blocking list that would return NULL addresses AND prevent the HTTPS (type65) queries from bypassing the adblocking. This script has some potentially useful ideas:

Use curl option -z to only download the updated file from oisd.nl if it is newer than the local file.
Produce both local= and address= entries per domain.
Simple whitelisting through a grep -vf command.
Diffing the new and existing lists to make sure there is a real difference before restarting dnsmasq and destroying the existing cache.
Running dnsmasq --test --conf-file=/tmp/oisd/oisd.new to make sure no syntax errors before restarting dnsmasq.

Memory usage for dnsmasq is 12328 kB (VmSize).

I don't know that I'll keep using this or not, but since this seemed like a good do-it-yourself adblock thread, I thought I'd share it.

#!/bin/sh

logmsg() { logger -t "$(basename $0)" $*; }

logmsg "Starting adblock update"

[ -d /tmp/oisd ] || mkdir -p /tmp/oisd
[ -f /etc/oisd/whitelist ] || touch /etc/oisd/whitelist

# Only download the file if it is newer on the server, or missing locally
if ! curl -z /tmp/oisd/dnsmasq2 -s -R https://small.oisd.nl/dnsmasq2 -o /tmp/oisd/dnsmasq2; then
        logmsg "oisd.nl download failed!"
        exit 1
fi

# Exit if the downloaded file is still older than the current local blocklist
if [ -f /tmp/dnsmasq.d/oisd.list ] && [ /tmp/oisd/dnsmasq2 -ot /tmp/dnsmasq.d/oisd.list ]; then
        logmsg "No change in remote oisd list"
        exit
fi

# Reformat the list with address= commands and remove whitelist entries
if [ -f /tmp/oisd/dnsmasq2 ]; then
        # address=/example.com/#  # Returns 0.0.0.0 or :: for blocked domains
        # local=/example.com/     # Prevents HTTPS (type65) queries from being forwarded
        awk -F/ '$1 == "local=" { printf "address=/%s/#\nlocal=/%s/\n", $2, $2 } ' /tmp/oisd/dnsmasq2 | \
        grep -vf /etc/oisd/whitelist > /tmp/oisd/oisd.new
else
        logmsg "No blocklist found after download!"
        exit 1
fi

if [ -f /tmp/dnsmasq.d/oisd.list ] && diff -q /tmp/oisd/oisd.new /tmp/dnsmasq.d/oisd.list 2>/dev/null; then
        logmsg "No changes to blocking list today"
        rm /tmp/oisd/oisd.new
        exit
fi

# Test if dnsmasq has any problems with the new blocklist before promoting it to the live blocklist
if dnsmasq --test --conf-file=/tmp/oisd/oisd.new >/dev/null 2>&1; then
        logmsg "Restarting dnsmasq with new blocking list"
        mv /tmp/oisd/oisd.new /tmp/dnsmasq.d/oisd.list
        service dnsmasq restart
else
        logmsg "dnsmasq did not like the new file"
        exit 1
fi

Lynx · March 1, 2023, 7:13am

Nice script and excellent ideas.

I intend to review and amalgamate these ideas where applicable into adblock-oisd.

@castillofrancodamian the two major existing adblock solutions are overly complicated for my taste and lack some important features we've implemented in adblock-oisd. The oisd adblock basically just works and adblock-oisd is just a service script wrapper to use it and perform a whole bunch of sanity and safety checks.

account4538 · March 1, 2023, 2:21pm

Hi,

logmsg gave -ash: logmsg: not found for me but logger worked to add to syslog.

dave14305 · March 1, 2023, 2:52pm

That’s strange since the logmsg() function is declared at the top of the script.