Keep in mind DNSMasq does not start if there is a syntax error in the list. This was the reason for the process running check added. This would make some of the other checks redundant.
However, checking for an IP address in place of the NULL address (#) would check against the rare chance there is a rouge blocklist. In that case I think it would be best to either remove that list or revert to the previous instead of trying to filter or fix it with a script.
If there really was a rouge blocklist human eyes would need to find out what is going on.
Isn't it worth checking dnsmasq is running and pings to google.com and cloudlfare.com work? Would you change or add to these checks?
I can see the logic in this.
Maybe we should test for rogue entries - sed filter @Wizballs(?) and return without further processing on any identified rogue entries. So we leave things as they were - old oisd.txt or not. Same for failed download of new oisd.txt file.
And if the above test reveals no rogue entries, then filter out non-rogue, but erroneous entries with:
sed -i '\|^address=/[[:alnum:]]|!d;\|/#$|!d' /tmp/oisd.txt
Because there could be non-rogue entries resulting from typographical errors. Or just comments, etc.
By this do you mean check against a list of "essential" sites that must work? (ie should never be in the oisd list)
eg google.com, cloudflare.com & a few other favourite sites.
Or am I missing the logic here?
Yes I think keep this in after rouge check, might just filter out that one bad character which would make the whole list unusable (dnsmasq no start due to syntax error). Plus it processes so fast, only takes a few seconds. No downside really.
Ok I think I get this. If say google.com was erroneously in the oisd list, ping test would not return an error
eg ping 0001.2waky.com (in oisd blocklist)
PING 0001.2waky.com (0.0.0.0): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.139 ms
Ping test returns ok
Still, is there any harm in doing a ping test to a couple of sites after applying oisd.txt? Assuming no rouge entries, it's just checking the internet is up and running as an additional test to dnsmasq running. Could there be a case where dnsmasq is running, but no internet connectivity due to oisd.txt? That's the assumed logic here.
Any thoughts on nxdomain vs 0.0.0.0? Just curious which people here think is better for adblocking purposes. Pi-hole docs recommend 0.0.0.0 and I read an article advocating it to stop device consulting another DNS server.
Looks like nslookup returns exit code 2 for nxdomain and exit code 0 for 0.0.0.0.
Looks like dnsmasq conf /# (what we are using already) is returning 0.0.0.0 as is.
address=/oisdblanktest.com/
*** OpenWrt.lan can't find oisdblanktest.com: Non-existent domain
address=/oisdhashtest.com/#
Name: oisdhashtest.com
Addresses: ::
0.0.0.0
address=/oisd0000test.com/0.0.0.0
Name: oisd0000test.com
Address: 0.0.0.0
This is simple for now, filter out anything that starts with address=, and anything that doesn't end with /#.
I'll try and add a check for no more "/" in between. eg max 2x "/" per line. But this should work as a placeholder until me or someone can figure it out. '\|^address=|!d;\|/#|d' > shallnotpass.txt
So comments, blank lines, or even just random words won't be sent to shallnotpass.txt. These would all be cleaned out afterwards with '\|^address=/[[:alnum:]]|!d;\|/#$|!d'. PS this existing command cleans out for example address=/.com/#, which does actually block all .com addresses.
Even though none are necessarily dangerous, you could argue that if the oisd.txt list included these, something may have gone wrong with that particular list compile. I've searched through several list versions, and happily have not found anything at all out of order.
What I do want to include is something to rip this type of entry out (note the 3x "/"). I'll work on it.
address=/oisdmytest.com/1.1.1.1/#
Edit: this returns 0.0.0.0 anyway, so not really an issue. But wouldn't hurt to test regardless.
Name: oisdmytest.com
Addresses: ::
0.0.0.0
Short answer? 0.0.0.0 is better. its an answer rather than a don't know which can lead to more queries to find out or trying to use an alternate DNS server.
NXDOMAIN may cause your device to fall back to some other DNS resolver, whereas 0.0.0.0 is an answer so your device will stop asking.
Also android tends to go looking again if you use NXDOMAIN. See this GitHub thread about it.
Regarding trust of blocking/filter lists. All you can do is try the lists and see if they work for your purposes. Some lists are very well run with dedicated reviewers and a process to get wrongly blocked addresses removed/reviewed.
I've included a few I use below from my AGH thread. They can be used with dnsmasq, AdGuard Home, Pihole or simple Adblock.
Filter Sites:
https://firebog.net/ https://oisd.nl/ For AdGuardHome you must use abp.oisd.nl
This has a major collection of blocklists allowing you to pick and choose and even gives them ratings for safe usage.
This set of lists are for keeping your smart tv/devices from spying and reporting back.
This is a very important list. It is a whitelist to always allow these sites. Its more useful for programs like AdGuard Home and PiHole as these allow you to put in exclusions like this but its still worth knowing about.
Last thing? Using AdGuard Home or PiHole while initially bit more complex to setup, does give you greater handling of blocklists and exclusions. It makes customising your needs simpler and also gives you a dashboard to check on.
Dnsmasq with a list its defiantly useful for smaller routers with low ram/disk space as an intermediary step but I still recommend AGH or PiHole, especially if you are using for parental purposes.
The good news is that oisd full already combines these sites you listed, or a variant of them, plus many more.
And then it applies a whitelist over the top also:
"Where most other blocklists go for the "block ads/trackers and when something breaks it's up to the user to figure out what to whitelist"-approach .. This list prioritizes functionality over blocking."
Indeed. but I prefer to be sure and treat these lists like Venn diagrams. As long as there is overlap its good (and any dupes will be stripped out when they are built into a master list by AGH.) Its actually why I don't use OISD and just use source lists instead.
My reasoning for this is AGH will tell you which blocklist has blocked something when you check the logs. Thus its easier to find the offending list and track it down vs one big master list. Just a personal choice. That being said its been a while since I've had issues. I believe last set of exceptions I had to include was for streaming IPTV (just needed a domain enabling). Also its because originally I used a 128mb router and using huge blocklists would cause out of memory errors. I could get about 140k of lists before it would die.
I've just listed all the ones I use and reference sites as a new post on my AGH thread if you are curious.
Yeah cool I do like to understand what options are out there. And there is definitely some upsides to running dedicated software ie logging as you mentioned.
Also benefits to running just a list, which is no additional software needed.
Its one reason I try to lay out my reasonings and references. Help people understand the why and how etc.
As I said earlier I originally started out with 128mb router so you have to be so careful to make sure you can do blocking without killing performance or memory.
Approaching from an engineering view I've always tried to keep it small but functional. There are some on pihole sites who have 4million plus sites in blocklists. Now that is ok if your router is powerful and has memory but it can be rather overkill. For instance if you aren't in china why include their blocklists?
I originally started with simple blocklist and curated lists, I later moved to a pihole and finally to AGH. But I now have a router with 4gb of ram so don't have to be as careful. Others however will need other options and understanding how to fit a solution for smaller routers is important.
Also dnsmasq and lists for a small light travel router is ideal and improves privacy and protection while traveling. Hostile wifi points are a always there threat.
Some final, I hope!, edits to my one liner for anyone that wishes to use this. Lots of other options in this thread to choose from also of course. Now a rolling log kept at ~999 lines, stored in /tmp (ram). Log is just a crude screen dump but contains enough to troubleshoot if ever needed. Record the number of oisd lines before and after sed filter. Variables to shorten everything up a bit.
I'm personally still running ping connectivity check after applying oisd and restarting dnsmasq. At least until someone can definitively say it's impossible to have dnsmasq running, yet no internet connectivity due to oisd list.
1st pass check anything starting with 'address=' and not ending with '/#' '\|^address=|!d;\|/#$|d'
I'm so sorry somehow didn't originally paste the $ for end of line
Second pass filter will behave exactly as I want now -E '\~^address=/[[:alnum:]][[:alnum:].-]+/#$~!d'
-E = extended options
and delete lines such as:
address=/.com/#
address=test.com/#
address=/test/com/#
address=/test.com/1.1.1.1/#
address=/test.com/1.1.1.1
address=/test.com/ (this line isn't actually a problem, but oisd dnsmasq uses /# anyway)
If a rogue element is identified in the first pass then processing is discontinued and first rogue element is printed out. Otherwise the file is processed according to the second pass.
