Adblock-oisd : 22.03 allows you to use huge blocklists with dnsmasq

Installing and Using OpenWrt
181

@Wizballs should we add a simple array variable to add in manual blocks like for facebook.com or the like?

182

What I do and is easier to manage is just a separate list with your static manual blocklist.

1 Like
183

So for custom blocks, just create another txt file in the /tmp/dnsmasq.d/ directory?

184

Exactly.
Since it's in ram though I just copy it from disk on start-up otherwise you'll lose it on reboot.

1 Like
185

Hmmm oisd dnsmasq entries have changed from:
address=/0--foodwarez.da.ru/#
to:
server=/0--foodwarez.da.ru/

I'll try and find out what/why, and if a permanent change or temporary oversight etc. Good news is sed filter worked as intended and deleted all the server= entries. Which basically left an empty file with just the comments, but no blocked addresses. I've posted on reddit:

@Lynx I think it's worth revisiting the one-pass check option. In this case adblock-oisd would have reverted to last good list, until either oisd is corrected or adblock-oisd is updated.

1 Like
186

According the the DNSMasq Man page, address=/example.com/# is the same as server=/example.com/. I'm not sure why the OISD maintainer decided to change. I can reach out if you haven't already. He's responsive on Reddit & email.

If scripting a check, the real danger is an IP address redirect. My opinion is simply anything other than nothing or # doesn't pass:

/example.com/
/example.com/#

187

IMHO server is better as it saves 2 bytes per entry. :wink: I'm glad they've changed it, I'm updating simple-adblock to support this change in OISD dnsmasq file.

1 Like
188

Good point.

189

I posted the question on reddit (link above), so guess will wait and see response...Really just wanting to know if it's permanent change so current code can be updated to accomodate.

Similar, but '/#' returns NULL, '/' returns NXDOMAIN

chickenwings=/example.com/ will cause dnsmasq to stop running altogether

190

Sorry, typo. Meant address=/example.com/# = server=/example.com/#

and

address=/example.com/ = server=/example.com/

"address" and "server" do the same according to the man page.

1 Like
191

Did you get a response?

BTW in this instance with our present logic in adblock-oisd, the new filtered oisd file with no entries gets injected because no rogue elements were identified and we don't check for at least one match.

So I think we should check for at least one match after checking for rogue elements here:

And if no match then return false and the update will just abandon ship and leave things as they were.

Could you propose a sed line to check for at least one match? I suppose it should be very similar to the check for rogue elements except in that it will check for at least one good element.

And we'll need to update the existing sed commands in dependence upon what oisd decides for the new file format.

1 Like
192

No reply yet... but new sed command might as well be flexible to accept any dnsmasq compliant syntax/formatting, regardless of what option the oisd list decides to use. Are you okay with just the one check instead of check then clean? I think one check with pass or fail is more than sufficient, it would have handled all recent scenarios. I'm yet to see any oisd list build that has needed the cleaning step however. Even if so, this type of entry would probably be rectified by oisd soon after (via user reports etc), AND would be picked up by the pass/fail method anyway. So pass/fail method covers all scenarios.

Can now start with either address=/ or server=/ . Can end with either / or /# . Addresses must still be properly formed/compliant.

-E '\~(^address=/|^server=/)[[:alnum:]][[:alnum:].-]+(/$|/#$)|^#|^\s*$~d'

I'll work on the count lines part next

Edit: Since the first check presumably passed, count the number of lines that aren't blank or # comments:

sed '\|^#|d;\|^\s*$|d' /tmp/oisd.txt | wc -l

How about check there is over eg 100,000 lines? current list has ~245k lines...

1 Like
193

One more thought. This will probably never come into play, but just in the interest of bullet proofing everything. Run a sed before anything else, to clean all whitespace (blanks/spaces/tabs) at the start and end of lines, and delete whitespace lines:

sed -i '\~^\s*$~d;s/^[ \t]*//;s/[ \t]*$//' /tmp/oisd.txt

So now the flow could go:

  1. clean whitespace
  2. check for rogue/malformed. Stop if detected, otherwise continue.
  3. check number of lines for minimum count

But if you really really want the original plan ie check for rogue, then clean typos etc, let me know and I'll update those commands also

1 Like
194

OK @Wizballs how about this now:

I checked and it seemed to work for both:

  • getting and updating new oisd.txt; and

  • failing with oisd.txt when I disabled downloading and added a rogue element in oisd.txt.

Any thoughts on the overall flow now? See:

Thanks to user: ente` on #openwrt - is that @ente here I wonder? - for some sed help.

1 Like
195

Nice work, applied and working on first adblock-oisd service restart. I'll find some time at some point, to do a proper debug run through if you like. The print line number and rogue match if found, is a good addition.

The nslookup check may need to be updated to exclude both 0.0.0.0 and NXDOMAIN, since oisd is now returning NXD instead of 0.0.0.0. But best to include both options? Nothing urgent as it passes as is, since nothing is currently returning 0.0.0.0.

1 Like
196

Yes I like this a lot too. And in this connection, we might be able to simplify this line:

rogue_element=`sed -nE '\~(^address=/|^server=/)[[:alnum:]][[:alnum:].-]+(/$|/#$)|^#|^\s*$~d;{p;=;q}' /tmp/oisd.txt | { read match; read line; [[ ! -z $match ]] && echo "$line: $match"; }`

Would you know how to modify sed to output either:

  • nothing on no rogue element; or
  • in the form "$line: $match" with a rogue element,

so that we don't have to add this funky shell dual read stuff?

Also, I presume the line count sed has to be run separately as we are doing?

Overall I am pretty chuffed with our new adblock-oisd utility. It's slim and elegant with no dependencies and incorporates some useful safeguarding features.

Last night I tested the following scenarios:

  • disabling oisd.txt download and manually downloading and adding a bogus line to oisd.txt
    - this was handled; adblock-oisd gave up
  • disabling the oisd file check and retaining bogus line in oisd.txt
    - this was handled; adblock-oisd detected no running instance of dnsmasq following restart and adblock-oisd reverted to the previous good oisd.txt

So that seems pretty positive.

197

Very happy with how 'bullet proof' this is becoming. Something I could run on my family's router without worry.

I think, this is not in the scope of sed to handle, and does need to be managed 'outside' of sed. As an option, could run sed by itself or with | wc -l, and if no rogues, then do nothing (or echo "passed" etc). And if rogues found, then produce line numbers and matches. And assuming rogues pretty much never occur, this step will pretty much never run anyway. It is a few more lines of code, but might be easier to read.

Yes I think so. sed is has a specific scope, pretty powerful tool tho.

1 Like
198

Worked for me just now! Thanks!

1 Like
199

OISD has currently been down for hours and I had to d/l from the mirror. I'm wondering if we should put in another failsafe that if the d/l fails to use the GitHub mirror which is the official one from the maintainer himself: https://github.com/sjhgvr/oisd

1 Like
200

@Wizballs what do you think? My initial gut reaction is that if a scheduled download fails then it's no biggie because the next one or one after that will surely work? And in any case each time the download fails there will be an output to the system log.

But it would be easy to put in a fallback url in the script.

1 Like