Adblock-lean: set up adblock using dnsmasq blocklist

BTW, this is an alternate method of whitelisting. local is a synonym for server and the trailing # says, “use the standard servers for this domain.”

1 Like

But isn't whitelisting the absence of a block line? Surely I'm being dense here, but with our allowlist we just ensure we don't have any blocks. So I'm trying to understand significance of line that is not a block but an allow.


@Wizballs likewise NXDOMAIN seems OK so far for me too. So maybe let's just stick with NXDOMAIN and park toggling for now, and see if demand for null IP emerges in the future.

In any case, it seems apt to me now to leave our check_dnsmasq with both the NXDOMAIN and null IP checks in place just to keep options open.

For my main DNS over TLS provider I actually just switched from CleanBrowsing to Cloudflare Family (1.1.1.3), and webpage load times seem a little faster. I only just discovered that Cloudflare has an option for families.

1 Like

Yes, I was just offering a commentary on the example you posted. Plus I see that is how @dibdot processes whitelists in Adblock.

TL;DR Don’t mind me…

1 Like

Not in the slightest. In my book you're one of Jedi Masters on this platform.

And crikey Charlie that's some serious complexity. @Wizballs take a look at the code @dave14305 just linked above.

1 Like

Not even close, but doing my best to catch up. You must be thinking of the “old dave14305”. :sauropod:

guys I'm testing your new version, on a linksys E5600 is more limited with ram but it is working fine I'm going to add a wireguard server soon. My report is working flawless at the moment.

I notice that OISD Big list for dnsmasq ver 2.86 uses this syntax is that okay?

local=/0--foodwarez.da.ru/

HaGeZi uses

address=/0123movies.cam/

I just wonder if I can try to mix and match other lists to see the performance but so far is doing well (dnsmasq only).

This is my setup for this tiny router Linksys e5600

  • DNS HTTPS Proxy Settings DNSMASQ (this super enforces all the dns request and I get DoT/DoH to cloudflare)

  • DDNS to cloudflare

  • Adblock-Lean (blocking ads with dnsmasq) OISD + HaGeZi Pro

  • WireGuard Server (soon)

thank you again!

Thanks a lot for your report.

Yes we accommodate this difference and swap any instances of the latter to the former.

Yes we allow for this. Please try different combinations and let us know what works well.

Is this an alternative to DNS over TLS via stubby? I use the latter. Not sure what the difference is. Anyone?

Ace! Glad to hear this.

All running good here. Sent a small github pull request to save one step, and a few CPU cycles. See what you think.

Having some issue upon and it dumps the collated work when:

Wed May 31 16:21:31 2023 user.notice adblock-lean: Lookup of 'google.com' failed with new blocklist.

Full~ish log

Wed May 31 16:20:27 2023 user.notice adblock-lean: Successfully generated preprocessed blocklist file with 638416 line(s).
Wed May 31 16:20:27 2023 user.notice adblock-lean: Processing and checking new blocklist file.
Wed May 31 16:20:27 2023 user.notice adblock-lean: Removing duplicates from blocklist file.
Wed May 31 16:20:45 2023 user.notice adblock-lean: Duplicates removed.
Wed May 31 16:20:45 2023 user.notice adblock-lean: No local allowlist identified.
Wed May 31 16:20:45 2023 user.notice adblock-lean: Checking for any rogue elements.
Wed May 31 16:21:09 2023 user.notice adblock-lean: New blocklist file check passed.
Wed May 31 16:21:09 2023 user.notice adblock-lean: Restarting dnsmasq.
Wed May 31 16:21:10 2023 daemon.info dnsmasq[1]: exiting on receipt of SIGTERM
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: started, version 2.86 cachesize 150
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: DNS service limited to local subnets
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: UBus support enabled: connected to system bus
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: DNSSEC validation enabled
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: configured with trust anchor for <root> keytag 20326
Wed May 31 16:21:16 2023 daemon.info dnsmasq-dhcp[1]: DHCP, IP range 192.168.44.100 -- 192.168.44.249, lease time 12h
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for zzztt49.com
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using only locally-known addresses for dlrect-smtb.jp.ap1.ib.zzzpl.com
Wed May 31 16:21:16 2023 daemon.info dnsmasq[1]: using 405385 more local addresses
Wed May 31 16:21:20 2023 user.notice adblock-lean: Checking dnsmasq instance.
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: reading /tmp/resolv.conf.d/resolv.conf.auto
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using nameserver 9.9.9.11#53
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using nameserver 149.112.112.11#53
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using nameserver 208.67.222.222#53
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using nameserver 208.67.220.220#53
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for zzztt49.com
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using only locally-known addresses for dlrect-smtb.jp.ap1.ib.zzzpl.com
Wed May 31 16:21:26 2023 daemon.info dnsmasq[1]: using 405385 more local addresses
Wed May 31 16:21:31 2023 user.notice adblock-lean: Lookup of 'google.com' failed with new blocklist.
Wed May 31 16:21:31 2023 user.notice adblock-lean: The dnsmasq check failed with new blocklist file.
Wed May 31 16:21:31 2023 user.notice adblock-lean: No previous blocklist file found. Stopping adblock-lean.
Wed May 31 16:21:31 2023 user.notice adblock-lean: Stopping adblock-lean.
Wed May 31 16:21:31 2023 user.notice adblock-lean: Removing /tmp/dnsmasq.d/blocklist and restarting dnsmasq.
Wed May 31 16:21:32 2023 user.notice adblock-lean: Stopped adblock-lean.
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: started, version 2.86 cachesize 150
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: DNS service limited to local subnets
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: UBus support enabled: connected to system bus
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: DNSSEC validation enabled
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: configured with trust anchor for <root> keytag 20326
Wed May 31 16:21:36 2023 daemon.info dnsmasq-dhcp[1]: DHCP, IP range 192.168.44.100 -- 192.168.44.249, lease time 12h
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for lan
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: reading /tmp/resolv.conf.d/resolv.conf.auto
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using nameserver 9.9.9.11#53
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using nameserver 149.112.112.11#53
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using nameserver 208.67.222.222#53
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using nameserver 208.67.220.220#53
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: using only locally-known addresses for lan
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: read /etc/hosts - 4 addresses
Wed May 31 16:21:36 2023 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Wed May 31 16:21:36 2023 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses

Service restart/stop/start header warning also in Luci Gui.

Failed to execute "/etc/init.d/adblock-lean restart" action: Error: XHR request timed out

What happens if you run nslookup google.com on the router over SSH?

Hey dave. it works fine after the Adblock-lean init.d script finished without the desired blocklist.. See log above for the script flow.

Luci Diag:

Summary

Server: 127.0.0.1
Address: 127.0.0.1:53

Non-authoritative answer:
Name: google.com
Address: 64.233.177.139
Name: google.com
Address: 64.233.177.102
Name: google.com
Address: 64.233.177.113
Name: google.com
Address: 64.233.177.138
Name: google.com
Address: 64.233.177.101
Name: google.com
Address: 64.233.177.100

Non-authoritative answer:
Name: google.com
Address: 2607:f8b0:4002:c08::65
Name: google.com
Address: 2607:f8b0:4002:c08::66
Name: google.com
Address: 2607:f8b0:4002:c08::8a
Name: google.com
Address: 2607:f8b0:4002:c08::8b

Oh, I missed the log entry showing it start the dnsmasq check before dnsmasq had read the upstream servers. Try increasing the sleep 10 line to sleep 20

Larger lists might take more than 10 seconds to load.

Failed with 20 sec delay.

Whilst the putty command window was processing the command:

/etc/init.d/adblock-lean enabled && export RANDOM_DELAY="0" && /etc/init.d/adblock-lean start

Luci Gui diag tools showed this.

;; connection timed out; no servers could be reached

nslookup: write to '::1': Connection refused

What about the syslog messages?

Same as init post.

root@routernamehere:~# /etc/init.d/adblock-lean enabled && export RANDOM_DELAY="0" && /etc/init.d/adblock-lean start
Started adblock-lean.
No local blocklist identified.
Downloading new blocklist file part(s).
Downloading new blocklist file part from: https://big.oisd.nl/dnsmasq2.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Cleaning whitespace and formatting blocklist file part as local=/.../.
Downloading new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Cleaning whitespace and formatting blocklist file part as local=/.../.
Successfully generated preprocessed blocklist file with 638416 line(s).
Processing and checking new blocklist file.
Removing duplicates from blocklist file.
Duplicates removed.
No local allowlist identified.
Checking for any rogue elements.
New blocklist file check passed.
Restarting dnsmasq.
Checking dnsmasq instance.
Lookup of 'google.com' failed with new blocklist.
The dnsmasq check failed with new blocklist file.
No previous blocklist file found. Stopping adblock-lean.
Stopping adblock-lean.
Removing /tmp/dnsmasq.d/blocklist and restarting dnsmasq.
Stopped adblock-lean.

As a test, reverted to Adblock-lean, 7 revisions behind current. Apr 17 "c5be465" "lynxthecat Render printf conditional on terminal use"

Moving up in commits to see if I can catch, or unveil my issue.

Settle here for now.

Putty log output-added echo and commit number to Adblock-lean script.
Executing Adblock-lean commit a2c52af
Executing Adblock-lean commit a2c52af
Started adblock-lean.
No local blocklist identified.
Downloading new blocklist file part(s).
Downloading new blocklist file part from: https://big.oisd.nl/dnsmasq2.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Downloading new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Successfully generated preprocessed blocklist file with 638417 line(s).
Processing and checking new blocklist file.
Cleaning whitespace and formatting blocklist file.
Whitepsace removed and formatting completed.
Removing duplicates from blocklist file.
Duplicates removed.
No local allowlist identified.
Checking for any rogue elements.
New blocklist file check passed.
Restarting dnsmasq.
Checking dnsmasq instance.
Lookup of 'google.com' failed with new blocklist.
The dnsmasq check failed with new blocklist file.
No previous blocklist file found. Stopping adblock-lean.
Stopping adblock-lean.
Removing /tmp/dnsmasq.d/blocklist and restarting dnsmasq.
Stopped adblock-lean.
root@Akita:~# /etc/init.d/adblock-lean enabled && export RANDOM_DELAY="0" && /etc/init.d/adblock-lean start
Running commit 356f7ac
Running commit 356f7ac
Started adblock-lean.
Downloading new blocklist file.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Removing duplicates from downloaded blocklist file part(s).
Checking new blocklist file.
New blocklist file check passed.
Restarting dnsmasq.
Checking dnsmasq instance.
The dnsmasq check passed with new blocklist file.
New blocklist installed with good line count: 638400.
root@Akita:~# /etc/init.d/adblock-lean enabled && export RANDOM_DELAY="0" && /etc/init.d/adblock-lean start
Running commit 356f7ac
Running commit 356f7ac
Started adblock-lean.
Downloading new blocklist file.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Removing duplicates from downloaded blocklist file part(s).
Checking new blocklist file.
New blocklist file check passed.
Restarting dnsmasq.
Checking dnsmasq instance.
The dnsmasq check passed with new blocklist file.
New blocklist installed with good line count: 638400.
root@Akita:~#
1 Like

By reverting to a commit prior to support for multiple blocklists, you could try going back to the current release and just removing the second blocklist from the top.

Just keep the first one and be happy!

I tried that, yet it did not work and had the same failure. The script has been working fine till today ~ 2PM. This is when I have the cron job (thundering heard) scheduled to run.

All past scripts that had this below worked. I edited the current "Main" script to include this section...

}

check_dnsmasq()
{
	log_msg "Checking dnsmasq instance."

	if ! pgrep -x dnsmasq &> /dev/null
	then
		log_msg "No instance of dnsmasq detected with new blocklist."
		return 1
	fi

	for domain in google.com amazon.com microsoft.com
	do
		nslookup "${domain}" | grep -A1 ^Name | grep -q '^Address: *0\.0\.0\.0$'
		if [[ "${?}" -eq 0 ]]
		then
			log_msg "Lookup of '${domain}' resulted in 0.0.0.0 with new blocklist"
			return 1
		fi
	done
	
	return 0
}

Tested the timing as doing anything past the April 17 update took 5 minutes to complete the sorting/removing/compacting.

This is a truncated "system log" with the versions run and times till complete.

Both Block lists

Wed May 31 20:52:14 2023 kern.notice kernel: [    0.000000] Linux version 5.10.176 (builder@buildhost) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r20134-5f15225c1e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu Apr 27 20:28:15 2023
###     TRUNCATED    ######   Facilitate local blocklist and restructure 74b398c Latest commit May 21, 2023
Wed May 31 20:57:02 2023 user.notice adblock-lean: The dnsmasq check passed with new blocklist file.
Wed May 31 20:57:02 2023 user.notice adblock-lean: New blocklist installed with good line count: 406110.
Wed May 31 20:57:02 2023 daemon.info procd: - init complete -

Original Blocklist

Wed May 31 21:02:02 2023 kern.notice kernel: [    0.000000] Linux version 5.10.176 (builder@buildhost) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r20134-5f15225c1e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu Apr 27 20:28:15 2023
###	   TRUNCATED    ######   Replace cat w/ sed clean of individual parts (#8)…Latest commit 0c7ebc8 May 28, 2023
Wed May 31 21:07:51 2023 user.notice adblock-lean: The dnsmasq check passed with new blocklist file.
Wed May 31 21:07:51 2023 user.notice adblock-lean: New blocklist installed with good line count: 253096.
Wed May 31 21:07:51 2023 daemon.info procd: - init complete -


Original Blocklist

Wed May 31 21:15:32 2023 kern.notice kernel: [    0.000000] Linux version 5.10.176 (builder@buildhost) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r20134-5f15225c1e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu Apr 27 20:28:15 2023
###     TRUNCATED   ####    Render printf conditional on terminal use Latest commit c5be465 on Apr 17
Wed May 31 21:18:05 2023 user.notice adblock-lean: The dnsmasq check passed with new blocklist file.
Wed May 31 21:18:05 2023 user.notice adblock-lean: New blocklist installed with good line count: 253095.
Wed May 31 21:18:05 2023 daemon.info procd: - init complete -

My need for blocklist depth is not as great as my need for turnup speed, and the old osid list is just fine for my need.

I have and will keep encouraging the developers to keep pace and stay focused on their baby..lean!
Thanks all for looking.

Hi Bill, what router do you have, and how much ram does it have? See if we can figure out what is going wrong.

Also does this mean that the current script was running fine for you, then suddenly stopped working. Or am I reading this wrong...?

Even using the default script with two large blocklists, your router only took 18 seconds to remove duplicates, 24 seconds to check for rogue entries. What is taking up the other 4.x minutes?
PS no sorting going on :wink:

Running on:

As far as I know, I just noticed something wrong EOM and looked at the System log to see it failing on version #8

I see there have been 3 new submission since my last post on the original problem of the script failing:
Wed May 31 16:21:31 2023 user.notice adblock-lean: Lookup of 'google.com' failed with new blocklist.

Did the work involved in #9, #10 and #11 Resolve some issue with lookup and leaving an empty blocklist?


Commits on Apr 17, 2023 Render printf conditional on terminal use

I'm currently running the above within a cron job just once a week to really reduce the "Heard" and park with a known good turn around time and blocklist I know works since, Adblock-oisd : 22.03 allows you to use huge blocklists with dnsmasq - #6 by jackiechun