Adblock-lean: set up adblock using dnsmasq blocklist

Wizballs · 2024-07-02T12:41:10.071Z

For the bogus-nxdomain part, this might be a start. It attempts to pull the address out of any given blocklist in any format, and encapsulate it in local=/...../
I think this can be improved, but will work as is:

The question this raises is:
Do you want to allow any blocklist input file and rework the dnsmasq compliance checks?
Or have separate blocklist inputs. Ie one for dnsmasq specific input files, and one for all other files?

I think both options are worth considering....

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/$[^#]$#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; s/.*=//; s/\///g; s/^/local=\//; s/$/\//; $a\'

Lynx · 2024-07-02T14:20:06.568Z

Not sure I’m following entirely. So with the new bogus-nxdomain lines, aren’t those just to be retained as is without any modification (except perhaps removing trailing comments)?

Lynx · 2024-07-02T15:21:28.845Z

Wizballs:

Ok for the allowlist lines:

# 1 Convert to lowercase; 2 Remove comment lines and trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Add newline

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; $a\'

Thanks a lot for this.

How does this look to you:

github.com/lynxthecat/adblock-lean

Introduce configurable allowlist_urls

committed 03:19PM - 02 Jul 24 UTC

lynxthecat

+47 -6

For the new allowlist file part(s), I skipped some of the checks that we employ for the blocklist file part(s) on the basis that checking for file part size and the like doesn't seem so important for these. Hope that's the right call?

It works for me when testing with:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt

So @spike0en I think your request to add this feature has now pretty much been satisfied. Just need thumbs up from @Wixballs and I will pull this into the master branch.

Wizballs · 2024-07-02T19:27:41.483Z

Lynx:

Not sure I’m following entirely. So with the new bogus-nxdomain lines, aren’t those just to be retained as is without any modification

I treated the bogus-nxdomain file as a blocklist (non-specific blocklist input format), and therefore outputted as dnsmasq blocklist format ie local=/..../

Are you saying that bogus-nxdomain list should be treated as something else rather than a blocklist?

dave14305 · 2024-07-02T19:45:33.652Z

The idea with bogus-nxdomain is to reply with NXDOMAIN if any query replies with one of the IPs indicated. It’s the case where you know the bad IP, but not necessarily aware of the bad domain.

Wizballs · 2024-07-02T20:01:04.018Z

Right got it now - thankyou! So bogus-nxdomain is already a dnsmasq allowed format.

In that case we can just recycle this line to cleanup the list

# 1 Convert to lowercase; 2 Remove comment lines and trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Add newline

'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; $a\'

Wizballs · 2024-07-02T20:45:18.239Z

Lynx:

It works for me when testing with:

https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt

Yes working well for me also!
I do have a suggestion to reduce peak memory usage. Currently the remove (sub)domain awk command is operating on the total combined blocklist. For people with large/multiple blocklists, this is a large array for awk to load into memory. By removing (sub)domain matches from the individually downloaded blocklist parts, awk will have smaller chunks to work with.

Current order is:

Download individual blocklists, sanitise, and combine
download allowlists and combine
remove (sub)domain allowlist matches from combined blocklist
add positive allowlist server= entries

Suggested order:

Download allowlists and combine
Download individual blocklists, and sanitise
remove (sub)domain allowlist matches from individual blocklist parts
combine individual blocklist parts
add positive allowlist server= entries

Lynx · 2024-07-02T20:48:53.415Z

Right now we have:

github.com

lynxthecat/adblock-lean/blob/86eb29aa73e7c99980c463fba586a0c872dd47a7/adblock-lean#L220


      
          			then
          				blocklist_file_part_size_KB=$(du -bk /tmp/blocklist.${blocklist_id} | awk '{print $1}')
          				blocklist_file_part_line_count=$(grep -vEc '^\s*$|^#' /tmp/blocklist.${blocklist_id})
          
          				if [[ "${blocklist_file_part_line_count}" -ge "${min_blocklist_file_part_line_count}" ]]
          				then
          					log_msg "Download of new blocklist file part from: ${blocklist_url} suceeded (downloaded file size: ${blocklist_file_part_size_KB} KB; line count: ${blocklist_file_part_line_count})."
          					
          					log_msg "Sanitizing blocklist file part."
          					# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Convert to local=; 5 Add newline
          					sed -i 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; s/\(^address=\|^server=\)/local=/; $a\' "/tmp/blocklist.${blocklist_id}"
          
          					if [[ "${rogue_element_action}" != "IGNORE" ]] 
          					then
          						log_msg "Checking for any rogue elements."
          
          						# Get line number and match of any rogue elements
          						rogue_element=$(sed -nE '\~(^(local|server|address)=/)[[:alnum:]*][[:alnum:]*_.-]+(/$)|^#|^\s*$~d;{p;=;q}' /tmp/blocklist.${blocklist_id} | { read match; read line; [[ ! -z "${match}" ]] && echo "${line}: ${match}"; })
          
          						if [[ ! -z "${rogue_element}" ]]
          						then

We need to either leave the bogus domain entries as they are (perhaps remove trailing comments) but still convert remaining entries to local.

And we can amalgamate the more sophisticated comment removal now too, right?

Wizballs · 2024-07-02T20:55:25.386Z

Lynx:

We need to either leave the bogus domain entries as they are (perhaps remove trailing comments) but still convert remaining entries to local.

bogus-nxdomains also have a different syntax and will fail the rogue_element check. So options would be to run a seperate sanitise and seperate rogue check for bogus-nxdomains. Or I can try and combine them, but I think this could be more difficult. What do you think?

Lynx · 2024-07-02T21:15:16.147Z

Taking a look right now. Hmm.

Am I right that the sanitizing aspect can remain the same:

github.com

lynxthecat/adblock-lean/blob/86eb29aa73e7c99980c463fba586a0c872dd47a7/adblock-lean#L220


      
          			then
          				blocklist_file_part_size_KB=$(du -bk /tmp/blocklist.${blocklist_id} | awk '{print $1}')
          				blocklist_file_part_line_count=$(grep -vEc '^\s*$|^#' /tmp/blocklist.${blocklist_id})
          
          				if [[ "${blocklist_file_part_line_count}" -ge "${min_blocklist_file_part_line_count}" ]]
          				then
          					log_msg "Download of new blocklist file part from: ${blocklist_url} suceeded (downloaded file size: ${blocklist_file_part_size_KB} KB; line count: ${blocklist_file_part_line_count})."
          					
          					log_msg "Sanitizing blocklist file part."
          					# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Convert to local=; 5 Add newline
          					sed -i 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; s/\(^address=\|^server=\)/local=/; $a\' "/tmp/blocklist.${blocklist_id}"
          
          					if [[ "${rogue_element_action}" != "IGNORE" ]] 
          					then
          						log_msg "Checking for any rogue elements."
          
          						# Get line number and match of any rogue elements
          						rogue_element=$(sed -nE '\~(^(local|server|address)=/)[[:alnum:]*][[:alnum:]*_.-]+(/$)|^#|^\s*$~d;{p;=;q}' /tmp/blocklist.${blocklist_id} | { read match; read line; [[ ! -z "${match}" ]] && echo "${line}: ${match}"; })
          
          						if [[ ! -z "${rogue_element}" ]]
          						then

But the rogue element check would need to be changed:

github.com

lynxthecat/adblock-lean/blob/86eb29aa73e7c99980c463fba586a0c872dd47a7/adblock-lean#L227


      
          					
          					log_msg "Sanitizing blocklist file part."
          					# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Convert to local=; 5 Add newline
          					sed -i 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; s/\(^address=\|^server=\)/local=/; $a\' "/tmp/blocklist.${blocklist_id}"
          
          					if [[ "${rogue_element_action}" != "IGNORE" ]] 
          					then
          						log_msg "Checking for any rogue elements."
          
          						# Get line number and match of any rogue elements
          						rogue_element=$(sed -nE '\~(^(local|server|address)=/)[[:alnum:]*][[:alnum:]*_.-]+(/$)|^#|^\s*$~d;{p;=;q}' /tmp/blocklist.${blocklist_id} | { read match; read line; [[ ! -z "${match}" ]] && echo "${line}: ${match}"; })
          
          						if [[ ! -z "${rogue_element}" ]]
          						then
          							log_msg "Rogue element: '${rogue_element}' identified originating in blocklist file part from: ${blocklist_url}."
          							if [[ "${rogue_element_action}" == "STOP" ]]
          							then
          								return 1
          							else
          								log_msg "Skipping file part and continuing."
          								rm -f "/tmp/blocklist.${blocklist_id}"

Wizballs · 2024-07-02T21:23:23.495Z

Lynx:

Am I right that the sanitizing aspect can remain the same:

The sanitizing part needs to be changed also. As it doesn't currently account for the bogus-nxdomain syntax.

Can use the sanitise from this post a couple hours ago, especially as a temporary placeholder. But yes I need to do this part better regardless

Adblock-lean: set up adblock using dnsmasq blocklist Community Builds, Projects & Packages

Right got it now - thankyou! So bogus-nxdomain is already a dnsmasq allowed format. In that case we can just recycle this line to cleanup the list # 1 Convert to lowercase; 2 Remove comment lines and trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Add newline 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/$[^#]$#.*$/\1/g; /^#/d; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; $a\'

Lynx · 2024-07-02T21:29:26.649Z

Wizballs:

The sanitizing part needs to be changed also. As it doesn't currently account for the bogus-nxdomain syntax.

Why? Won't the existing line just skip over the bogus-nxdomain lines?

github.com

lynxthecat/adblock-lean/blob/86eb29aa73e7c99980c463fba586a0c872dd47a7/adblock-lean#L220


      
          			then
          				blocklist_file_part_size_KB=$(du -bk /tmp/blocklist.${blocklist_id} | awk '{print $1}')
          				blocklist_file_part_line_count=$(grep -vEc '^\s*$|^#' /tmp/blocklist.${blocklist_id})
          
          				if [[ "${blocklist_file_part_line_count}" -ge "${min_blocklist_file_part_line_count}" ]]
          				then
          					log_msg "Download of new blocklist file part from: ${blocklist_url} suceeded (downloaded file size: ${blocklist_file_part_size_KB} KB; line count: ${blocklist_file_part_line_count})."
          					
          					log_msg "Sanitizing blocklist file part."
          					# 1 Convert to lowercase; 2 Remove trailing comments; 3 Remove trailing address hash, and all whitespace; 4 Convert to local=; 5 Add newline
          					sed -i 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/; s/\([^#]\)#.*$/\1/g; s/#$//; s/^[ \t]*//; s/[ \t]*$//; /^$/d; s/\(^address=\|^server=\)/local=/; $a\' "/tmp/blocklist.${blocklist_id}"
          
          					if [[ "${rogue_element_action}" != "IGNORE" ]] 
          					then
          						log_msg "Checking for any rogue elements."
          
          						# Get line number and match of any rogue elements
          						rogue_element=$(sed -nE '\~(^(local|server|address)=/)[[:alnum:]*][[:alnum:]*_.-]+(/$)|^#|^\s*$~d;{p;=;q}' /tmp/blocklist.${blocklist_id} | { read match; read line; [[ ! -z "${match}" ]] && echo "${line}: ${match}"; })
          
          						if [[ ! -z "${rogue_element}" ]]
          						then

Wizballs · 2024-07-02T21:39:03.076Z

Ah yep ignore me, it does work. So yes just the rogue check needs updating. I'll have to try this after work today, it's a bit more complicated and will take longer.

currently allows only local|server|address=/..../(#)

I'll see if an 'or' statement can allow bogus-nxdomains with the format:
bogus-nxdomain=xx.xx.xx

Lynx · 2024-07-02T21:41:00.731Z

Wizballs:

I do have a suggestion to reduce peak memory usage. Currently the remove (sub)domain awk command is operating on the total combined blocklist. For people with large/multiple blocklists, this is a large array for awk to load into memory. By removing (sub)domain matches from the individually downloaded blocklist parts, awk will have smaller chunks to work with.

Current order is:

Download individual blocklists, sanitise, and combine

download allowlists and combine

remove (sub)domain allowlist matches from combined blocklist

add positive allowlist server= entries

Suggested order:

Download allowlists and combine

Download individual blocklists, and sanitise

remove (sub)domain allowlist matches from individual blocklist parts

combine individual blocklist parts

add positive allowlist server= entries

Makes sense. But one complexity. Does this:

github.com

lynxthecat/adblock-lean/blob/0f98dc5335f78972b58a7cd3b38e56b7f02358ba/adblock-lean#L342


      
          		use_allowlist=1
          	else
          		log_msg "Not using any allowlist for blocklist processing."
          		use_allowlist=0
          	fi
          
          	for blocklist_file_part_gz in /tmp/blocklist.*.gz
          	do
          		if [[ "${use_allowlist}" == 1 ]]
          		then
          			gunzip -c "${blocklist_file_part_gz}" | awk -F'/' 'NR==FNR { allow[$0]; next } { n=split($2,arr,"."); addr = arr[n]; for ( i=n-1; i>=1; i-- ) { addr = arr[i] "." addr; if ( addr in allow ) next } } 1' "/tmp/allowlist" -
          		else  
          			gunzip -c "${blocklist_file_part_gz}"
          		fi
          		rm -f "${blocklist_file_part_gz}"
          	done | sort -u > /tmp/blocklist
          
          	[[ "${use_allowlist}" == 1 ]] && sed 's~.*~server=/&/#~; $a\' /tmp/allowlist >> /tmp/blocklist
          
          	rm -f /tmp/allowlist

result in the the remove (sub)domain awk command operating on the total combined blocklist? I think this runs an awk command per blocklist file part as it is extracted and all those bits are piped to the sort.

Does this change your recommendation?

Wizballs · 2024-07-02T21:49:24.401Z

Lynx:

think this runs an awk command per blocklist file part as it is extracted and all those bits are piped to the sort.

Ok but why am I seeing this order in the log? (look at second last line). Or is this actually operating on individual blocklist entries? If so, ignore me again, I'll go back to bed

Wed Jul 3 06:23:48 2024 user.notice adblock-lean: Downloading new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt.
Wed Jul 3 06:23:49 2024 user.notice adblock-lean: Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt suceeded (downloaded file size: 4145 KB; line count: 154097).
Wed Jul 3 06:23:49 2024 user.notice adblock-lean: Sanitizing blocklist file part.
Wed Jul 3 06:23:54 2024 user.notice adblock-lean: Checking for any rogue elements.
Wed Jul 3 06:23:57 2024 user.notice adblock-lean: Downloading new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.txt.
Wed Jul 3 06:24:05 2024 user.notice adblock-lean: Download of new blocklist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.txt suceeded (downloaded file size: 19959 KB; line count: 735408).
Wed Jul 3 06:24:05 2024 user.notice adblock-lean: Sanitizing blocklist file part.
Wed Jul 3 06:24:26 2024 user.notice adblock-lean: Checking for any rogue elements.
Wed Jul 3 06:24:41 2024 user.notice adblock-lean: Downloading new blocklist file part from: https://raw.githubusercontent.com/sjhgvr/oisd/main/dnsmasq2_big.txt.
Wed Jul 3 06:24:44 2024 user.notice adblock-lean: Download of new blocklist file part from: https://raw.githubusercontent.com/sjhgvr/oisd/main/dnsmasq2_big.txt suceeded (downloaded file size: 6723 KB; line count: 249820).
Wed Jul 3 06:24:44 2024 user.notice adblock-lean: Sanitizing blocklist file part.
Wed Jul 3 06:24:51 2024 user.notice adblock-lean: Checking for any rogue elements.
Wed Jul 3 06:24:57 2024 user.notice adblock-lean: Successfully generated preprocessed blocklist file with 1139325 line(s).
Wed Jul 3 06:24:57 2024 user.notice adblock-lean: No local allowlist identified.
Wed Jul 3 06:24:57 2024 user.notice adblock-lean: Downloading new allowlist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt.
Wed Jul 3 06:24:57 2024 user.notice adblock-lean: Download of new allowlist file part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/whitelist-referral-onlydomains.txt suceeded (downloaded file size: 30 KB; line count: 1552).
Wed Jul 3 06:24:57 2024 user.notice adblock-lean: Sanitizing allowlist file part.
Wed Jul 3 06:24:57 2024 user.notice adblock-lean: Successfully generated allowlist with lines.
**Wed Jul 3 06:24:57 2024 user.notice adblock-lean: Will remove any (sub)domain matches from the blocklist and append corresponding server** entries to the blocklist.
Wed Jul 3 06:26:33 2024 user.notice adblock-lean: Processed blocklist file size: 24284 KB.

Lynx · 2024-07-02T21:55:15.659Z

The comment might be misleading. The relevant code snippet is here:

	if [[ -f "/tmp/allowlist" ]] && allowlist_line_count=$(grep -vEc '^\s*$|^#' /tmp/allowlist) && [[ $allowlist_line_count -gt 0 ]]
	then
		log_msg "Successfully generated allowlist with ${local_allowlist_line_count} lines."
		log_msg "Will remove any (sub)domain matches from the blocklist and append corresponding server entries to the blocklist."
		use_allowlist=1
	else
		log_msg "Not using any allowlist for blocklist processing."
		use_allowlist=0
	fi

	for blocklist_file_part_gz in /tmp/blocklist.*.gz
	do
		if [[ "${use_allowlist}" == 1 ]]
		then
			gunzip -c "${blocklist_file_part_gz}" | awk -F'/' 'NR==FNR { allow[$0]; next } { n=split($2,arr,"."); addr = arr[n]; for ( i=n-1; i>=1; i-- ) { addr = arr[i] "." addr; if ( addr in allow ) next } } 1' "/tmp/allowlist" -
		else  
			gunzip -c "${blocklist_file_part_gz}"
		fi
		rm -f "${blocklist_file_part_gz}"
	done | sort -u > /tmp/blocklist

	[[ "${use_allowlist}" == 1 ]] && sed 's~.*~server=/&/#~; $a\' /tmp/allowlist >> /tmp/blocklist

So: "Will remove any (sub)domain matches from the blocklist and append corresponding server entries to the blocklist." really means: remove the (sub)domain matches from the blocklist file parts when generating the sorted blocklist and then add the corresponding server entries.

Wizballs · 2024-07-02T22:01:30.774Z

Lynx:

So: "Will remove any (sub)domain matches from the blocklist and append corresponding server entries to the blocklist." really means: remove the (sub)domain matches from the blocklist file parts when generating the sorted blocklist and then add the corresponding server entries.

I can see that order in the code, and I even looked at that order ages ago when you first implemented it. I'll look at it again later to get my head around it. Cheers

Lynx · 2024-07-02T22:01:39.232Z

Wizballs:

I'll have to try this after work today, it's a bit more complicated and will take longer.

currently allows only local|server|address=/..../(#)

I'll see if an 'or' statement can allow bogus-nxdomains with the format:
bogus-nxdomain=xx.xx.xx

Looking forward! For now I've added in the comment removal into the blocklist file part sanitization:

github.com/lynxthecat/adblock-lean

Improve blocklist file part sanitization

committed 09:58PM - 02 Jul 24 UTC

lynxthecat

+2 -2

Lynx · 2024-07-02T22:05:26.121Z

Wizballs:

If so, ignore me again, I'll go back to bed

Well to be fair we've been running multiple parallel conversations!

Wizballs:

I can see that order in the code, and I even looked at that order ages ago when you first implemented it. I'll look at it again later to get my head around it. Cheers

Yes it's a bit complicated now. Since we have this system of piping the individual awk lines that are run to a global sort. You could well still be right. I certainly notice that the allowlist lines really slow things down a lot.

Wizballs · 2024-07-02T22:10:24.370Z

Lynx:

Well to be fair we've been running multiple parallel conversations!

Haha yes, and I'm just having a shocker today lol.

Lynx:

I certainly notice that the allowlist lines really slow things down a lot.

Waaay back when the (sub)domain removal part was first implement, it was by far the fastest option I could find. And by a large speed margin at that. You can remove exact full line matches very quickly, but it's the removal of the subdomains that slows things down. All open to faster code suggestions, but I'll almost put money there won't be any significant gains for this operation.

Ultimately I think the removal of subdomains is the right thing to do for the end user.