Adblock-oisd : 22.03 allows you to use huge blocklists with dnsmasq

[Lynx]

Thanks. So in other words:

'\|^address=/[[:alnum:]]|!d;\|/#$|!d'

can be broken up in two as follows:

'\|^address=/[[:alnum:]]|!d;

\|/#$|!d'

So does \| and | count as beginning and end of statement, and the ^ means at the beginning of line and $ means at the end of line? And !d means delete anything not matching? And ; means start of second statement?

Also @Wizballs someone on IRC explained that we can get whether service is enabled using 'service adblock-oisd enabled' and then checking $?. So a line could be placed inside Scheduled Tasks to call 'service adblock-oisd start' if $? is 0, i.e. adblock-oisd service is enabled.

[Wizballs]

@Lynx

So does \| and | count as beginning and end of statement, and the ^ means at the beginning of line and $ means at the end of line? And !d means delete anything not matching? And ; means start of second statement?

Spot on - all correct!

@Lynx

Also @Wizballs someone on IRC explained that we can get whether service is enabled using 'service adblock-oisd enabled' and then checking $?. So this coudl be placed inside Scheduled Tasks to call 'service adblock-oisd start' if $? is 0, i.e. adblock-oisd service was enabled.

This I'm not familiar with sorry. I'm just a hobby linux/openwrt'er

[Lynx]

Any other improvements to the script you can think of? Does it work for you? If update fails it should revert back to previous list.

[Wizballs]

Just at work for the day (happy Friday) but keen to have a look over the weekend. I'm no expert but can read code just well enough to get through...

[Lynx]

@Wizballs

I'm just a hobby linux/openwrt'er

Me too. But you know sed and I know bash so between us we have what we need!

Despite the existence of 'simple-adblock' and 'adblock', I still think there's a place for this simple 'adblock-oisd' script or your one liner.

[Wizballs]

Hey yeah that's what geat about this community everyone helps each other. Definitely there is always room for more options, everyone has different wants/needs. Plus if everyone's project can improve/benefit from others works, then even better!

simp-adb will always have it place with a nice Luci interface etc. There's a few thing I would (selfishly) like to see in there, but that is totally up to @stangri and his direction with that already awesome work.

Personally I'm focused on oisd for it's hands-free ease of use. Don't care if a few ads get through, and websites work and don't break (especially for the family)

@Lynx

Me too. But you know sed and I know bash so between us we have what we need!

Despite the existence of 'simple-adblock' and 'adblock', I still think there's a place for this simple 'adblock-oisd' script or your one liner.

[account4538]

@Wizballs

deletes /tmp/dnsmasq/oisd.txt

I'm thinking it would be best to not delete the blocklist on DNSMasq restart because if for whatever reason it restarted (by https-dns-proxy for example) then you now running with no blocklist at all until the next refresh.

The current OISD full list is only 8.3MB. If a user is that short on RAM they aren't doing this anyway.

[Wizballs]

@account4538

I'm thinking it would be best to not delete the blocklist on DNSMasq restart because if for whatever reason it restarted (by https-dns-proxy for example) then you now running with no blocklist at all until the next refresh.

I've never had an unexpected restart of stubby, it just does it's thing. You maybe correct re https - I wouldn't know

@account4538

The current OISD full list is only 8.3MB. If a user is that short on RAM they aren't doing this anyway.

The 128mb router users may disagree

However, what you want is something like:

{ date && curl --max-filesize 20971520 --max-time 60 --retry 3 --url https://dnsmasq.oisd.nl/ --output /tmp/dnsmasq.d/oisd.txt && head -n 20 /tmp/dnsmasq.d/oisd.txt && sed -i '\|^address=/[[:alnum:]]|!d;\|/[\d35]$|!d' /tmp/dnsmasq.d/oisd.txt && /etc/init.d/dnsmasq restart; sleep 60; { pgrep -x dnsmasq && { ping -c 1 1.1.1.1 || ping -c 1 9.9.9.9 ;} ;} || { rm /tmp/dnsmasq.d/oisd.txt; /etc/init.d/dnsmasq restart ;} ;} 2>&1 | tee /tmp/oisd.log

[stangri]

@Wizballs

I've never had an unexpected restart of stubby, it just does it's thing. You maybe correct re https - I wouldn't know

The https-dns-proxy needs to be restarted on WAN updates, so it will restart dnsmasq in turn.

[Wizballs]

@stangri

The https-dns-proxy needs to be restarted on WAN updates, so it will restart dnsmasq in turn.

Good point account4538 and Stangri .... keep /tmp/dnsmasq.d/oisd.txt if using https-dns-proxy.

I trialled restarting wan & wan6 (without tmp/dnsmasq.d/oisd.txt), ad-blocking is still working using stubby FYI. Is restarting wan&6 enough to test?

[Lynx]

An alternative might be to modify the dnsmasq service file to gunzip and then store back to gzip the log file upon restart like what I've implemented in my script when replacing old oisd.txt with new one in case the checks fail.

[bigsmile]

If curl or wget succeed you are online, why would you also ping some external dns server ?

[Lynx]

I think the idea is to verify connectivity after dnsmasq has been restarted with the new oisd.txt list.

As part of verification that dnsmasq with the present 'oisd.txt' list is in good order at present my script performs the following checks:

• dnsmasq is running
• ping to 'google.com' suceeds
• ping to 'cloudflare.com' suceeds

check_dnsmasq()
{
	check_return=0;

	pgrep -x dnsmasq &> /dev/null

    (($? != 0)) && check_return=1;

    ping -c 1 google.com &> /dev/null

    (($? != 0)) && check_return=1

	ping -c 1 cloudflare.com &> /dev/null

	(($? != 0)) && check_return=1

	return $check_return
}

See here.

Not sure if this is the best thing to be doing, but that's part of my first draft. Suggestions welcome!

If this fails, then if a previous oisd.txt was available (stored as .gz file to avoid eating up too much RAM) it will fallback to that, recheck, and if checks succeed then stick with the previous oisd.txt. If even those checks based on the previous oisd.txt fail, it will then fallback to not using any oisd.txt at all.

My script retains the oisd.txt in /tmp/dnsmasq.d/ to avoid any issues with external dnsmasq restarts and only uses the .gz store when attempting to swap out to a new oisd.txt in case the swap fails so that it can revert back to the old one.

[bigsmile]

Bash is not installed by default, its better to use sh/ash.
Then enable xtrace in your shell so you can see what's going on.

set -x

Something like this can be used to check if a service is running.
You can substitute pidof with pgrep if you want.

root@OpenWRT:/tmp# is_running(){ [ ! -z "$(pidof $1)" ];}
root@OpenWRT:/tmp# is_running dnsmasq && echo yes || echo no
+ is_running dnsmasq
+ pidof dnsmasq
+ '[' '!' -z '2203 3772' ]
+ echo yes
yes
root@OpenWRT:/tmp# 

And you need something to check if "oisd" is working. So for example take the last address line from the blocklist and check if that is blocked. That would be a proper indicator I think.

root@OpenWRT:/tmp# lastaddress=$(tail -n1 /tmp/oisd.txt|awk -F/ '{print $2}')
+ tail -n1 /tmp/oisd.txt
+ awk -F/ '{print $2}'
+ lastaddress=zzzzzzzzzzz.no-ip.biz
root@OpenWRT:/tmp# is_blocking(){ nslookup $1 | { ! grep -q $1;};}
root@OpenWRT:/tmp# is_blocking $lastaddress && echo yes || echo no
+ is_blocking zzzzzzzzzzz.no-ip.biz
+ nslookup zzzzzzzzzzz.no-ip.biz
+ grep -q zzzzzzzzzzz.no-ip.biz
+ echo yes
yes
root@OpenWRT:/tmp# is_blocking cloudflare.com && echo yes || echo no
+ is_blocking cloudflare.com
+ nslookup cloudflare.com
+ grep -q cloudflare.com
+ echo no
no
root@OpenWRT:/tmp# 

This should give you some idea's

[moeller0]

@bigsmile

Bash is not installed by default, its better to use sh/ash.

But only from the perspective of compatibility. IMHO sh/ash/dash are way to restricted, as if they aim to make creating useful programs in shell hard to steer people to other languages

[bigsmile]

It sure feels that way doesn't it? As you progress (assuming that you are not a veteran shell programmer) that bar will drop significantly. But something like this should not suffer because of it

[Lynx]

Thanks for the suggestions. I'm a bash fan - it's not got a large footprint at all and adds so many features. The code looks way nicer.

[Wizballs]

@bigsmile

If curl or wget succeed you are online, why would you also ping some external dns server ?

@Lynx

I think the idea is to verify connectivity after dnsmasq has been restarted with the new oisd.txt list.

Yes this - test after apply oisd.txt. I'm leaning towards internet connectivity as an absolute priority, especially with cctv and home automation etc. Side effect is for example, if the internet dropped out momentarily at exactly the wrong time, this would of course restart dnsmasq without oisd.txt.

But yeah any feedback on if this is actually needed or a better way? Eg would there ever be a case where dnsmasq is running but internet is blocked due to a bad oisd.txt??

[Lynx]

@Wizballs

Eg would there ever be a case where dnsmasq is running but internet is blocked due to a bad oisd.txt??

I imagine so and that's why I thought we should test ping -c 1 google.com and cloudflare.com to verify all is well with new oisd.txt?

It seems for Scheduled Tasks can render start conditional on service being enabled as follows:

service adblock-oisd enabled; [[ $? -eq 1 ]] && service adblock-oisd start

[Lynx]

OK your comment about sh preyed on my mind. After all, the premise of my script is that it is supposed to be super simple - and the bash dependency undermined that. So I switched from bash over to sh:

Switch from bash to sh to prevent bash dependency