UPDATE: an outcome of this thread has been the development of adblock-oisd - a super lean script written as an OpenWrt service file that implements adblocking via the oisd blocklist and incorporates several safety checks and useful features. adblock-oisd will download the latest oisd list and perform various safety checks associated with the file before and after restarting dnsmasq and act accordingly. It is designed to be run as a daily scheduled task.
adblock-oisd is available here:
ORIGINAL POST:
Not my work, just reposting here for more people to see. I'm using this method - it's truly insanely fast, and the full OISD blocklist is only consuming ~30mb ram on my r7800 / 22.03.
Thanks to luigi_xp whoever/wherever you are. Link to original reddit thread:
22.03-rcx comes with dnsmasq 2.86, which brings a full rewrite of the dns handling code making it much more efficient in both ram and processing for various things, including common blocklist syntax.
I'm running the full 1M+ line oisd blocklist with no noticeable performance penalty on my humble MT7621/128MB Archer C6.
and setup it to run every day and on startup, and it's working silk smooth. Dnsmasq is using a "large" amount of ram (17%), but considering what it's doing and that there's still a lot of free ram left, i'm happy with the result.
Unfortunately plugins such as adblock still choke because they process the lists before, but if you use dnsmasq directly and create a little script to download and update the lists it works as good if not better than the standalone plugins.
Great - I run dnsmasq, now also at version 2.86, on my Linux desktop, with several blocklists that in total, in their human readable text format, consume over 27 Mbytes. I've been using dnsmasq since at least 2016. It's outstanding.
Kudos for sharing this, afaik there are lot of people here on the forum who don't frequent reddit and vice a versa.
Obviously to each their own, and yes, that blocklist would probably choke simple-adblock on a 128Mb router, but there are upsides of "processing", especially if you're using multiple sources -- adblock pioneered and simple-adblock shamelessly copied the functionality to remove duplicates and unnecessary 3rd level domains from the final block list for supported dnsmasq/unbound configs.
Another upside of using adblock or simple-adblock is a built-in enforcement of OpenWrt's resolver across your network, so even if your clients are set up to use hardcoded DNS (but not DoT/DoH) servers, the ads would still be blocked.
PS. Have you run any actual tests to measure the performance of dnsmasq with and without that list?
Hey no problems re-posting it here. Exactly why I did it, I think a lot of people would never visit the Reddit/Openwrt community. In fact I only found it by complete accident doing a google search. Hit the jackpot in this case tho.
I'll send details of performance etc when I get home (which is very, very good BTW!)
I totally get the usage cases dnsmasq vs adblock & simple. Ie wanting to manage client DNS. However I'm pretty liberal with that, if guests want to use their own DNS, go for it.
The below graph is anecdotal. But context - I'm now running the full OISD blocklist, Layer Cake SQM, ~15 devices, 3x IP motion cameras, IOT & Guest WLAN, 100/20 (Australia speeds) on R7800. For comparison the full OISD blocklist on Adblock with 21.02 would hover around 20% on one CPU and could take literal seconds to load a webpage. The full OISD blocklist on dnsmasq is also noticeably faster than Adblock on 21.02 with OISD basic (webpage load speeds). No other graphs sorry, just what I described here.
I got home around 17:00, web browsing / searching etc. Video chat started around 17:35. All in all this router/config is just idling along. This is easily the fastest setup I've run....pages just load quickly.
One thing to note and IDKW, but after running the dnsmasq command on bootup, WLAN1-1 and WLAN1-2 stop providing DHCP. I just run "/etc/init.d/network restart" straight after in local startup, and all is good.
wget https://dnsmasq.oisd.nl -O /tmp/oisd.txt downloads the list to /tmp/oisd.txt (a temporary file)
mv /tmp/oisd.txt /tmp/dnsmasq.d/oisd.txt moves the temporary file to the actual location. This is so that if the wget download fails it doesn't load an empty file. If the download failed, this command will fail as there won't be any /tmp/oisd.txt file to move.
/etc/init.d/dnsmasq restart restarts the dnsmasq service with the updated list
And for those who aren't familiar, anything in the /tmp directory is stored on RAM (not non-volatile storage), which is good for the longevity of the device.
Ah, you meant zero impact in CPU load, I thought you meant zero impact on speed of resolution requests (which was an issue with the older dnsmasq versions for large servers lists).
@jackiechun does some minimum amount of control need to be put into place to ensure the source of the information isn't malicious and poisoning DNS entries?
Maybe I'm a nervous Nelly, because I never heard of oisd.nl before?
Example1: Maybe grep -v out any entries that map FQDN to an IP??
Thanks for the update! This experience made me realize that my modest Archer C7 is too old to handle these possible fluctuations so I've decided to generate my own static dnsmasq blocklist in place of the OISD list, and changed to an adblocking public dns.
I wrote earlier that with the startup script I lost DHCP for WLAN1-1 and WLAN1-2 (my guest and IOT WLAN's) and was using "/etc/init.d/network restart" at startup as a workaround. Now just sharing that I'm using sleep 60 before the script (last in local startup) and everything is working 100% without a network restart at startup. IMO sleep 60 is the nicer solution for me...
@jackiechun Thanks for that script with sense checking - good idea. I guess you could add a file size check and ignore if it's oversized - just an idea tho. I have used both NextDNS and Adguard before in round robin to accommodate the 300k dns lookup per month limit both of them have.
I've decided to give it another try: the allure of using Cloudflare/Quad9 DNS servers was too great.
My solution: simple bash script that will pre-process the blocklist and push to Github if < 400,000 entries (arbitrarily chosen), and I've modified my command to download that blocklist from Github. Currently the list is ~340,000, when it ballooned 2 weeks ago it was ~ 950,000.
A bonus to this is I can append domains I want blocked (such as autoupdates for my Amazon Firesticks). Will run and see how it goes, I'll only report back if there are any issues.
EDIT: recent update went to 437,051 entries so I bumped my threshold up to 500,000 and my Archer C7 is still able to handle it.
Last night there was an issue with OISD and this morning I found out with DNSMasq not working.
The blocklist had a one line error message so since the download succeeded the command continued with restarting DNSMasq.
Log:
Thu Oct 20 05:10:00 2022 cron.err crond[1447]: USER root pid 21924 cmd curl --max-time 60 --retry 4 --retry-delay 6 --url https://dnsmasq.oisd.nl/ --output /tmp/oisd.txt && mv /tmp/oisd.txt /tmp/dnsmasq.d/oisd.txt && /etc/init.d/dnsmasq restart
Thu Oct 20 05:10:00 2022 daemon.info dnsmasq[1]: exiting on receipt of SIGTERM
Thu Oct 20 05:10:04 2022 daemon.crit dnsmasq[1]: bad option at line 1 of /tmp/dnsmasq.d/oisd.txt
Thu Oct 20 05:10:04 2022 daemon.crit dnsmasq[1]: FAILED to start up
Thu Oct 20 05:10:09 2022 daemon.crit dnsmasq[1]: bad option at line 1 of /tmp/dnsmasq.d/oisd.txt
Thu Oct 20 05:10:09 2022 daemon.crit dnsmasq[1]: FAILED to start up
Thu Oct 20 05:10:14 2022 daemon.crit dnsmasq[1]: bad option at line 1 of /tmp/dnsmasq.d/oisd.txt
Thu Oct 20 05:10:14 2022 daemon.crit dnsmasq[1]: FAILED to start up
Thu Oct 20 05:10:19 2022 daemon.crit dnsmasq[1]: bad option at line 1 of /tmp/dnsmasq.d/oisd.txt
Thu Oct 20 05:10:19 2022 daemon.crit dnsmasq[1]: FAILED to start up
Thu Oct 20 05:10:24 2022 daemon.crit dnsmasq[1]: bad option at line 1 of /tmp/dnsmasq.d/oisd.txt
Thu Oct 20 05:10:24 2022 daemon.crit dnsmasq[1]: FAILED to start up
Thu Oct 20 05:10:29 2022 daemon.crit dnsmasq[1]: bad option at line 1 of /tmp/dnsmasq.d/oisd.txt
Thu Oct 20 05:10:29 2022 daemon.crit dnsmasq[1]: FAILED to start up
Thu Oct 20 05:10:29 2022 daemon.info procd: Instance dnsmasq::cfg01411c s in a crash loop 6 crashes, 0 seconds since last crash
I was thinking just adding || rm /tmp/dnsmasq.d/oisd.txt && /etc/init.d/dnsmasq restart to the end of the command so if DNSMasq doesn't start the blocklist file is deleted and DNSMasq restarted again.
However, when doing a /etc/init.d/dnsmasq restart from the command line it always appears to start and doesn't throw an error exit code, the errors appear in the log later:
EDIT: This should do the trick: pgrep -x dnsmasq >/dev/null || rm /tmp/dnsmasq.d/oisd.txt && /etc/init.d/dnsmasq restart
Check for the DNSMasq process and if not there then delete the blocklist file and restart
While those are indeed 'errors', they're expected and the intended outcome. Background for this is dnsmasq first checking for another (rogue) DHCPd on your network (by trying to get an IP via DHCP by the means of udhcpc itself), this ought to fail (or there's something seriously wrong on your network), before dnsmasq itself gets started.
