Adblock-lean: set up adblock using dnsmasq blocklist

Hi Bill,
512mb ram so that definitely isn't an issue.
Lynx just pushed a really neat update to wait until dnsmasq has properly restarted, before final nslookup checks. It's in the master branch if you want to give the latest script a run. It should solve the issue you were having with google.com unreachable and therefore restarting without a blocklist. We think it was due to your router taking longer than the (previously) hardcoded 10 seconds to restart dnsmasq.

Remember to edit this line if you just want to run oisd blocklist alone:
blocklist_urls="https://big.oisd.nl/dnsmasq2"

Very Good.

Syslog
Tue Jun  6 16:47:19 2023 authpriv.info dropbear[19216]: Child connection from 192.168.44.100:58216
Tue Jun  6 16:47:31 2023 authpriv.notice dropbear[19216]: Password auth succeeded for 'root' from 192.168.44.100:58216
Tue Jun  6 16:47:56 2023 user.notice adblock-lean: Stopping adblock-lean.
Tue Jun  6 16:47:56 2023 user.notice adblock-lean: Removing /tmp/dnsmasq.d/blocklist and restarting dnsmasq.
Tue Jun  6 16:48:00 2023 user.notice adblock-lean: Stopped adblock-lean.
Tue Jun  6 16:48:00 2023 user.notice adblock-lean: Started adblock-lean.
Tue Jun  6 16:48:00 2023 user.notice adblock-lean: No local blocklist identified.
Tue Jun  6 16:48:00 2023 user.notice adblock-lean: Downloading new blocklist file part(s).
Tue Jun  6 16:48:00 2023 user.notice adblock-lean: Downloading new blocklist file part from: https://big.oisd.nl/dnsmasq2.
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: started, version 2.86 cachesize 150
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: DNS service limited to local subnets
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: UBus support enabled: connected to system bus
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: DNSSEC validation enabled
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: configured with trust anchor for <root> keytag 20326
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq-dhcp[1]: DHCP, IP range 192.168.44.100 -- 192.168.44.249, lease time 12h
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for lan
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: reading /tmp/resolv.conf.d/resolv.conf.auto
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using nameserver 9.9.9.11#53
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using nameserver 149.112.112.11#53
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using nameserver 208.67.222.222#53
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using nameserver 208.67.220.220#53
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: using only locally-known addresses for lan
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: read /etc/hosts - 4 addresses
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq[1]: read /tmp/hosts/odhcpd - 0 addresses
Tue Jun  6 16:48:00 2023 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Tue Jun  6 16:48:04 2023 user.notice adblock-lean: Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Tue Jun  6 16:48:04 2023 user.notice adblock-lean: Cleaning whitespace and formatting blocklist file part as local=/.../.
Tue Jun  6 16:48:20 2023 user.notice adblock-lean: Downloading new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
Tue Jun  6 16:48:25 2023 user.notice adblock-lean: Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Tue Jun  6 16:48:25 2023 user.notice adblock-lean: Cleaning whitespace and formatting blocklist file part as local=/.../.
Tue Jun  6 16:49:01 2023 user.notice adblock-lean: Successfully generated preprocessed blocklist file with 678622 line(s).
Tue Jun  6 16:49:01 2023 user.notice adblock-lean: Processing and checking new blocklist file.
Tue Jun  6 16:49:01 2023 user.notice adblock-lean: Removing duplicates from blocklist file.
Tue Jun  6 16:49:20 2023 user.notice adblock-lean: Duplicates removed.
Tue Jun  6 16:49:20 2023 user.notice adblock-lean: No local allowlist identified.
Tue Jun  6 16:49:20 2023 user.notice adblock-lean: Checking for any rogue elements.
Tue Jun  6 16:49:41 2023 user.notice adblock-lean: Performing dnsmasq --test on the processed blocklist.
Tue Jun  6 16:49:46 2023 user.notice adblock-lean: dnsmasq --test output: dnsmasq: syntax check OK.
Tue Jun  6 16:49:46 2023 user.notice adblock-lean: The dnsmasq --test on the processed blocklist passed.
Tue Jun  6 16:49:46 2023 user.notice adblock-lean: New blocklist file check passed.
Tue Jun  6 16:49:46 2023 user.notice adblock-lean: Restarting dnsmasq.
Tue Jun  6 16:49:46 2023 daemon.info dnsmasq[1]: exiting on receipt of SIGTERM
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: started, version 2.86 cachesize 150
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: DNS service limited to local subnets
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: UBus support enabled: connected to system bus
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: DNSSEC validation enabled
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: configured with trust anchor for <root> keytag 20326
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq-dhcp[1]: DHCP, IP range 192.168.44.100 -- 192.168.44.249, lease time 12h
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for zzztt49.com
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using only locally-known addresses for zzznews.ru
Tue Jun  6 16:49:56 2023 daemon.info dnsmasq[1]: using 411813 more local addresses
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: reading /tmp/resolv.conf.d/resolv.conf.auto
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using nameserver 9.9.9.11#53
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using nameserver 149.112.112.11#53
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using nameserver 208.67.222.222#53
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using nameserver 208.67.220.220#53
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for zzztt49.com
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using only locally-known addresses for zzznews.ru
Tue Jun  6 16:50:06 2023 daemon.info dnsmasq[1]: using 411813 more local addresses
Tue Jun  6 16:50:16 2023 daemon.info dnsmasq[1]: read /etc/hosts - 4 addresses
Tue Jun  6 16:50:16 2023 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Tue Jun  6 16:50:16 2023 daemon.info dnsmasq[1]: read /tmp/hosts/odhcpd - 0 addresses
Tue Jun  6 16:50:16 2023 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Tue Jun  6 16:50:16 2023 user.notice adblock-lean: Restart of dnsmasq completed.
Tue Jun  6 16:50:16 2023 user.notice adblock-lean: Checking dnsmasq instance.
Tue Jun  6 16:50:17 2023 user.notice adblock-lean: The dnsmasq check passed with new blocklist file.
Tue Jun  6 16:50:17 2023 user.notice adblock-lean: New blocklist installed with good line count: 411814.

Great. And your total runtime is 2m17s. That's with the 880mhz CPU and also two (very large) lists. That's what we are aiming for. It will complete even faster with more powerful hardware of course, or just one blocklist etc.

@Lynx aka Lynx, @Wizballs

Any chance to get a Version numbering system applied to the adblock-lean script echo'ed in the System Log?

1 Like

Sorry being so late to reply. I've done pixelserv in the early versions of simple-adblock, the cons to it are:

  1. You'll have to start an uhttpd instance to serve it
  2. With https requests you've have certificate errors

That was years ago, now with more and more of the web moving to https, it will be even more problematic.

2 Likes

Pixelserv-tls has run its course on the Asuswrt-Merlin firmware as an Entware package used by the Diversion ad-blocking script. It won't be included in the next release of the Diversion script, effectively killing pixelserv on that platform. kvic seems to have abandoned the project, and it became more problematic to use as browsers and apps started pinning certificates.

The Entware guys seem to have adapted it for OpenSSL 3.0 recently, but better to leave it alone.

2 Likes

Thanks. So it seems that we shouldn't bother with it then. I think in principal it is a good idea to replace those advertisement image frames with nothing, but there doesn't seem to be an easy effective fix.

Our very own adblock-lean has been ticking away just nicely for ages now for me.

@Wizballs added a bunch of efficiency improvements recently, so I still think the 'lean' in adblock-lean is still justified notwithstanding all the features we've added like local allowlist and blocklist and merging blocklists from multiple URLs.

@Bill to your very valid point about versioning, we could do I suppose, but I am pretty lazy on that front - e.g. cake-autorate has been due a version 2.00 release for a very long time now!

Any thoughts on versioning @Wizballs? We could arbitrarily release version 1.0.0 based on the latest commit, or we could be lazy and not bother with any versioning for now.

Looks like pixel serving worked easily before https. No matter now really anway, I don't actually notice that many blank areas of a page anyway (yes there are some though).

Regarding a version number for now, how about just put the last date and time updated as a comment at the top? Pretty easy to spot for everyone I think.

Why once openwrt is rebooted, adblock-lean service is not started automatically although it is enabled at boot. I got it setup on two router and both have the same issue..
Can you please advise if there is something i need to check to make sure adblock-lean service starts at boot?

Hmm. Is it loaded too early?

How to check that?

Hello,
On the github page there are some instructions, did you also do the chmod to enable file permissions?

Installation on OpenWrt

wget https://raw.githubusercontent.com/lynxthecat/adblock-lean/main/adblock-lean -O /etc/init.d/adblock-lean
chmod +x /etc/init.d/adblock-lean
service adblock-lean enable

ls -al /etc/init.d/adblock-lean
rwxr-xr-x 1 root root 9535 Jul 8 01:31 /etc/init.d
/adblock-lean

Hm ok, what happens if you try and restart adblock-lean via eg system>startup>restart ?
Maybe check logs also....

i have just restarted the service through luci, but got no error, when i checked the service status through the command line i got the following

service adblock-lean status
No /tmp/dnsmasq.d/blocklist identified.
adblock-lean is not active.

What about log entries?

logread -e adblock-lean

Anything about “thundering herd”? No, that’s only via cron. Log entries would still be useful.

1 Like

i have noticed while now trying to start the adblock-lean service, the memory is being consumed almost fully during service start.

This is when I get

sed: write error

but the service starts successfully.

service adblock-lean restart
Stopping adblock-lean.
Removing /tmp/dnsmasq.d/blocklist and restarting dnsmasq.
Stopped adblock-lean.
Started adblock-lean.
No local blocklist identified.
Downloading new blocklist file part(s).
Downloading new blocklist file part from: https://big.oisd.nl/dnsmasq2.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 failed.
Downloading new blocklist file part from: https://big.oisd.nl/dnsmasq2.
Download of new blocklist file part from: https://big.oisd.nl/dnsmasq2 suceeded.
Cleaning whitespace and formatting blocklist file part as local=/.../.
Downloading new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded.
Cleaning whitespace and formatting blocklist file part as local=/.../.
sed: write error
Successfully generated preprocessed blocklist file with 620657 line(s).
Processing and checking new blocklist file.
Removing duplicates from blocklist file.
Killed
Duplicates removed.
No local allowlist identified.
Checking for any rogue elements.
Performing dnsmasq --test on the processed blocklist.
dnsmasq --test output: dnsmasq: syntax check OK.
The dnsmasq --test on the processed blocklist passed.
New blocklist file check passed.
Restarting dnsmasq.
Restart of dnsmasq completed.
Checking dnsmasq instance.
The dnsmasq check passed with new blocklist file.
New blocklist installed with good line count: 134976.

64MB ram? Using smaller blocklist likely a good idea!

@wizballs and @dave14305 maybe we should check for sufficient free memory before downloading and installing blocklist?

Then again we already have variables relating to memory consumption:

1 Like

Simple-adblock actually has some free memory code we could look over and possibly adapt :wink:

@wshamroukh Try using oisd small only, remove any/all other lists...

1 Like

what do u mean? Can you elaborate?