Adblock-lean: set up adblock using dnsmasq blocklist

Lynx · 2024-06-29T06:19:25.176Z

jarg:

No-no. I meant that I am using Gl.iNet custom OpenWRT will you lr app work on it?

You ought really to ask GI.iNet that question because this forum is directed to OpenWrt and adblock-lean has been written for use on OpenWrt. adblock-lean is written as a service script and requires no dependencies. So it might work.

jarg:

But… 128 MB of /tmp is ok? If so, no problem.

So it doesn’t affect flash at all now (installation not counting)?

Sure, users have reported things working just fine on 128 MB.

jarg · 2024-06-29T11:40:05.892Z

Lynx:

ask GI.iNet that question

They told me that GL gui doesn’t affect LuCi and works simultaneously. So, in this case everything will work fine?

Wizballs · 2024-06-29T11:56:32.147Z

I've used gl.inet mini portable router before. I did become cautious that gl.inet wasn't a pure Owrt build and eventually flashed stock Owrt on to it.
My unit electronically burnt itself out eventually somehow. But anyway, It's basically a skin over the top of OpenWRT. You can still access the default Luci interface, Google how to do that. My guess is that lean will work just fine. But use at your own risk.

Documibozu · 2024-06-29T16:34:33.311Z

AdBlock lean needs a dnsmasq version that you won't find in every gl.inet routers.
Many are based on eol versions of openwrt (18.xx or 19.xx trunks).

Lynx · 2024-06-30T08:30:13.559Z

@Wizballs any chance you could you answer hagezi’s question here:

github.com/lynxthecat/adblock-lean

Allowlist - Which syntax/format is supported?

opened 06:40AM - 30 Jun 24 UTC

hagezi

Hi @lynxthecat, I had a request to provide my referral allowlist for adblockl…ean. As it is a wildcard list, it is currently only available in AdBlock format, see: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/whitelist-referral.txt Which format/syntax is supported by adblocklean? The lists contains rules like `track*.example.com` and the rule `example.com` unblocks `example.com` and all subdomains (`*.example.com`). How can this be mapped for adblocklean? Would that work? https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/whitelist-referral.txt Thanks, Gerd

spike0en · 2024-06-30T12:03:13.460Z

Is it possible to add "allowlist_urls" similar to "blocklist_urls" in the config file besides the local ones?

github.com/hagezi/dns-blocklists

add DNSmasq allowlist referral for testing - @lynxthecat

committed 06:52AM - 30 Jun 24 UTC

hagezi

+423 -0

Lynx · 2024-06-30T12:46:44.124Z

If the question is whether there is facilitated a local allowlist then the answer is yes. I’m hazy on the syntax though and hopefully @Wizballs can remind us soon.

dave14305 · 2024-06-30T12:56:34.886Z

I think the question is whether you support a “hosted/remote” allowlist, just as the blocklists are “hosted/remote”.

Lynx · 2024-06-30T13:25:44.575Z

Ah. Not at present. I wonder if this is something we should add?

Wizballs · 2024-06-30T23:51:30.350Z

Copied the github comment here for convenience:
Hi Hazezi, lynx,

adblock-lean has allowlist implemented in the following way:

example.com and all subdomains such as *.example.com are deleted from the blocklist
and a positive allow entry is added server=/example.com/# (this is to allow cases for example where example.com is blocked along with all subdomains, but the user wishes to unblock only one.example.com and leave all other example.com blocked)

I'll have to do some trials on dnsmasq wildcard handling over say the next 24 hours. but dnsmasq seems to accept them at least:
root@OpenWrt:~# dnsmasq --test --conf-file=/tmp/dnsmasq.d/blocklist
dnsmasq: syntax check OK.

Lynx · 2024-07-01T10:28:03.447Z

spike0en:

Is it possible to add "allowlist_urls" similar to "blocklist_urls" in the config file besides the local ones?

OK, I'm adding this now - hagezi himself also just asked that we add it.

Wizballs:

the github comment

Thanks @Wizballs for expanding on this.

Lynx · 2024-07-01T11:22:26.611Z

I've almost completing adding configurable allowlist_urls - please see here:

GitHub

Comparing master...add-allowlist_urls · lynxthecat/adblock-lean

Lean and powerful adblocking solution for OpenWrt. Contribute to lynxthecat/adblock-lean development by creating an account on GitHub.

but I have encountered the following issue in the resulting /tmp/blocklist:

829160 server=/# title: hagezi's allowlist referral/#
829161 server=/# description: this list unblocks affiliate & tracking referral links that appear in mails, search results etc./#
829162 server=/# homepage: https://github.com/hagezi/dns-blocklists/#
829163 server=/# license: https://github.com/hagezi/dns-blocklists/blob/main/license/#
829164 server=/# issues: https://github.com/hagezi/dns-blocklists/issues/#
829165 server=/# expires: updated regularly/#
829166 server=/# last modified: 30 jun 2024 21:58 utc/#
829167 server=/# version: 2024.0630.2158.57/#
829168 server=/# syntax: domains (without subdomains)/#
829169 server=/# number of entries: 1548/#

Right now I have:

	if [[ -f "${local_allowlist_path}" ]] && local_allowlist_line_count=$(grep -vEc '^\s*$|^#' ${local_allowlist_path}) && [[ $local_allowlist_line_count -gt 0 ]]
	then
		log_msg "Found local allowlist with ${local_allowlist_line_count} lines."
		log_msg "Sanitizing allowlist file part."
		# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace;
		sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d' ${local_allowlist_path} > /tmp/allowlist
	else
		log_msg "No local allowlist identified."
	fi

	allowlist_id=0
	for allowlist_url in ${allowlist_urls}
	do
		for retries in $(seq 1 3)
		do
			log_msg "Downloading new allowlist file part from: ${allowlist_url}."
			uclient-fetch "${allowlist_url}" -O- --timeout=2 2> /tmp/uclient-fetch_err > /tmp/allowlist.${allowlist_id}
			if grep -q "Download completed" /tmp/uclient-fetch_err
			then
				allowlist_file_part_size_KB=$(du -bk /tmp/allowlist.${allowlist_id} | awk '{print $1}')
				allowlist_file_part_line_count=$(grep -vEc '^\s*$|^#' /tmp/allowlist.${allowlist_id})
				log_msg "Download of new allowlist file part from: ${allowlist_url} suceeded (downloaded file size: ${allowlist_file_part_size_KB} KB; line count: ${allowlist_file_part_line_count})."
				log_msg "Sanitizing allowlist file part."
				# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace
				sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d' /tmp/allowlist.${allowlist_id} >> /tmp/allowlist
				rm -f  /tmp/allowlist.${allowlist_id}
				allowlist_id=$((allowlist_id+1))
				continue 2
			else
				log_msg "Download of new allowlist file part from: ${allowlist_url} failed."
				log_msg "Sleeping for 5 seconds after failed download attempt."
				sleep 5
				continue
			fi
		done
		log_msg "Download failed after three failed download attempts. Continuing further operation."
	done

	rm -f /tmp/allowlist.*

	if [[ -f "/tmp/allowlist" ]] && allowlist_line_count=$(grep -vEc '^\s*$|^#' /tmp/allowlist) && [[ $allowlist_line_count -gt 0 ]]
	then
		log_msg "Successfully generated allowlist with ${local_allowlist_line_count} lines."
		log_msg "Will remove any (sub)domain matches from the blocklist and append corresponding server entries to the blocklist."
		use_allowlist=1
	else
		log_msg "Not using any allowlist for blocklist processing."
		use_allowlist=0
	fi

How should the sed calls be amended to remove the comments in the allowlist file part? See example allowlist file part here:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt

Here are the lines with the sed calls:

log_msg "Sanitizing allowlist file part."
# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace;
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d' ${local_allowlist_path} > /tmp/allowlist

and also:

log_msg "Sanitizing allowlist file part."
# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d' /tmp/allowlist.${allowlist_id} >> /tmp/allowlist

And as a separate issue, hagezi has also asked if we can add support for bogus-nxdomain lines:

github.com/lynxthecat/adblock-lean

TIF - Block IPs with DNSmasq - `bogus-nxdomain=1.2.3.4`

opened 05:57AM - 30 Jun 24 UTC

hagezi

Hi @lynxthecat, could you add support and test my TIF IP list in DNSmasq form…at in adblocklean? `bogus-nxdomain=1.2.3.4` All domains resolving to the "blocked" IP should be answered with NXDOMAIN. The list: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif-ips.txt Thank you, Gerd

See example file for the format here:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif-ips.txt

This allows preventing lookups of domains that would result in blocked IPs. Pretty nifty addition.

Wizballs · 2024-07-01T22:34:39.173Z

Ok for the allowlist lines:

# 1 Convert to lowercase; 2 Remove comment lines and trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Add newline

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; $a\'

I'll put something together for the bogus-nxdomain entries. I think this should be a bit more universal than just "bogus-nxdomain specific". I have an idea, will get back to you later today.

Wizballs · 2024-07-02T12:41:10.071Z

For the bogus-nxdomain part, this might be a start. It attempts to pull the address out of any given blocklist in any format, and encapsulate it in local=/...../
I think this can be improved, but will work as is:

The question this raises is:
Do you want to allow any blocklist input file and rework the dnsmasq compliance checks?
Or have separate blocklist inputs. Ie one for dnsmasq specific input files, and one for all other files?

I think both options are worth considering....

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/$[^#]$#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; s/.*=//; s/\///g; s/^/local=\//; s/$/\//; $a\'

Lynx · 2024-07-02T14:20:06.568Z

Not sure I’m following entirely. So with the new bogus-nxdomain lines, aren’t those just to be retained as is without any modification (except perhaps removing trailing comments)?

Lynx · 2024-07-02T15:21:28.845Z

Wizballs:

Ok for the allowlist lines:

# 1 Convert to lowercase; 2 Remove comment lines and trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Add newline

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; $a\'

Thanks a lot for this.

How does this look to you:

github.com/lynxthecat/adblock-lean

Introduce configurable allowlist_urls

committed 03:19PM - 02 Jul 24 UTC

lynxthecat

+47 -6

For the new allowlist file part(s), I skipped some of the checks that we employ for the blocklist file part(s) on the basis that checking for file part size and the like doesn't seem so important for these. Hope that's the right call?

It works for me when testing with:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt

So @spike0en I think your request to add this feature has now pretty much been satisfied. Just need thumbs up from @Wixballs and I will pull this into the master branch.

Wizballs · 2024-07-02T19:27:41.483Z

Lynx:

Not sure I’m following entirely. So with the new bogus-nxdomain lines, aren’t those just to be retained as is without any modification

I treated the bogus-nxdomain file as a blocklist (non-specific blocklist input format), and therefore outputted as dnsmasq blocklist format ie local=/..../

Are you saying that bogus-nxdomain list should be treated as something else rather than a blocklist?

dave14305 · 2024-07-02T19:45:33.652Z

The idea with bogus-nxdomain is to reply with NXDOMAIN if any query replies with one of the IPs indicated. It’s the case where you know the bad IP, but not necessarily aware of the bad domain.

Wizballs · 2024-07-02T20:01:04.018Z

Right got it now - thankyou! So bogus-nxdomain is already a dnsmasq allowed format.

In that case we can just recycle this line to cleanup the list

# 1 Convert to lowercase; 2 Remove comment lines and trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Add newline

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; $a\'

Wizballs · 2024-07-02T20:45:18.239Z

Lynx:

It works for me when testing with:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt

Yes working well for me also!
I do have a suggestion to reduce peak memory usage. Currently the remove (sub)domain awk command is operating on the total combined blocklist. For people with large/multiple blocklists, this is a large array for awk to load into memory. By removing (sub)domain matches from the individually downloaded blocklist parts, awk will have smaller chunks to work with.

Current order is:

Download individual blocklists, sanitise, and combine
download allowlists and combine
remove (sub)domain allowlist matches from combined blocklist
add positive allowlist server= entries

Suggested order:

Download allowlists and combine
Download individual blocklists, and sanitise
remove (sub)domain allowlist matches from individual blocklist parts
combine individual blocklist parts
add positive allowlist server= entries