I just pushed a few more commits, one of them re-adds debug messages, these time they are printed with time (in seconds, since measurement start).
@Lynx could you test with it (service adblock-lean update -v commit=39e4476) and post the output with the 3 URLs as you had above and DL_THREADS=2, PROCESS_THREADS=2?
Seems it’s working? Although second job seemed to take a long time to schedule. Love the colours.
root@OpenWrt:~# service adblock-lean start
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating compressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Starting DL job (PID: 24056)
[ 0 ] Scheduling DL job (running jobs: 2)
[ 0 ] Waiting for DL vacancy (running jobs: 2)
[ 0 ] Starting DL job (PID: 24057)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 0)
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 24076)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 24078)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.5 MiB, lines: 186,880).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 20 ] DL job 24056 completed.
[ 20 ] DL job 24057 completed.
[ 20 ] Starting DL job (PID: 24179)
Downloading raw blocklist part from https://big.oisd.nl/domainswild
[ 20 ] Scheduling PROCESS job (running jobs: 2)
[ 20 ] PROCESS job 24076 completed.
[ 20 ] Starting PROCESS job (PID: 24190)
Processing raw blocklist part from https://big.oisd.nl/domainswild
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 17.26 MiB, lines: 930,826).
Successfully downloaded list part from https://big.oisd.nl/domainswild
[ 24 ] DL job 24179 completed.
[ 24 ] PROCESS job 24078 completed.
Successfully processed list part from https://big.oisd.nl/domainswild (size: 3.41 MiB, lines: 168,045).
[ 24 ] PROCESS job 24190 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,285,751 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 23.45 MiB.
Checking dnsmasq instances.
Successfully imported new compressed blocklist file for use by dnsmasq with size: 8.15 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:58s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,232,892.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
The processing of the file parts seems super fast.
And the big sort at the end seems to be the bottleneck now.
I wonder if you gain any time by setting DL_THREADS to 3?
Also could you check with htop that both cores are fully utilized when processing?
I'm not sure what you are referring to. Looking at the output, PROCESS jobs are scheduled 1 second after DL jobs start, which is probably a consequence of the sleep 1 here:
The PROCESS scheduler has to sleep until it finds files to process, otherwise it would stress the CPU by non-stop polling.
Otherwise I'm not seeing any delays anywhere. Could you elaborate?
P.s. I think the logged times you posted above were simply imprecise because logger was delaying them for one reason or another.
Test results from my R4S...
Summary:
It looks like with the two hagezi lists, concurrency doesn't make much difference with a second CPU hitting 100% for a fraction of a second with commit=39e4476. Adding the oisd list made 5 of the 6 CPUs hit 100% for a few seconds.
blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt"
Runs through Wed Mar 12 05:31:33 2025 were the "latest" build from before the weekend.
Run Wed Mar 12 20:12:22 2025 was installing current latest (released a couple days ago).
Run Wed Mar 12 20:43:02 2025 was installing commit=39e4476 and downloading https://nsfw.oisd.nl/domainswild2.
THREADS="1"
Run Wed Mar 12 20:56:00 2025 was downloading https://nsfw.oisd.nl/domainswild2.
THREADS="6"
Run Wed Mar 12 21:10:12 2025 was only downloading the two hagezi lists and THREADS="1"
root@R4S-wrt:~# logread | grep "adblock-lean: Processing time for blocklist generation and import"
Sat Mar 8 13:16:49 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Sun Mar 9 05:51:33 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Mon Mar 10 05:35:32 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:33s.
Tue Mar 11 05:21:32 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 05:31:33 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:33s.
Wed Mar 12 20:12:22 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 20:43:02 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:41s.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:38s.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 20:11:03 2025
service adblock-lean update -v latest
Wed Mar 12 20:11:03 2025 user.info adblock-lean: Stopping adblock-lean.
Wed Mar 12 20:11:03 2025 user.info adblock-lean: Removing any adblock-lean blocklist files in dnsmasq config directories.
Wed Mar 12 20:11:03 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:11:03 2025 user.info adblock-lean: Restarting dnsmasq.
Wed Mar 12 20:11:21 2025 user.info adblock-lean: Waiting for dnsmasq initialization.
Wed Mar 12 20:11:22 2025 user.info adblock-lean: Restart of dnsmasq completed.
Wed Mar 12 20:11:22 2025 user.info adblock-lean: Stopped adblock-lean.
Wed Mar 12 20:11:22 2025 user.info adblock-lean: Getting GitHub ref for latest version of adblock-lean from GitHub.
Wed Mar 12 20:11:22 2025 user.info adblock-lean: Downloading adblock-lean, version 'v0.6.1'.
Wed Mar 12 20:11:23 2025 user.info adblock-lean: Copying file 'adblock-lean'.
Wed Mar 12 20:11:23 2025 user.info adblock-lean: Copying file 'abl-lib.sh'.
Wed Mar 12 20:11:23 2025 user.info adblock-lean: Copying file 'abl-process.sh'.
Wed Mar 12 20:11:23 2025 user.info adblock-lean: The adblock-lean service is already enabled.
Wed Mar 12 20:11:23 2025 user.info adblock-lean: adblock-lean has been updated to version 'latest'.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Starting adblock-lean, version v0.6.1.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: gawk detected so using gawk for fast (sub)domain match removal and entries packing.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: GNU sed detected so list processing will be fast.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: coreutils-sort detected so sort will be fast.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: No existing compressed or uncompressed blocklist identified.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Testing connectivity.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: No local allowlist identified.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Found local blocklist. Sanitizing.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Successfully processed blocklist (source file size: 432 B, sanitized line count: 16).
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Starting raw blocklist part(s) download.
Wed Mar 12 20:11:50 2025 user.info adblock-lean: Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt.
Wed Mar 12 20:11:51 2025 user.info adblock-lean: Successfully processed blocklist (source file size: 3.5 MiB, sanitized line count: 186,880).
Wed Mar 12 20:11:51 2025 user.info adblock-lean: Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt.
Wed Mar 12 20:11:55 2025 user.info adblock-lean: Successfully processed blocklist (source file size: 17.26 MiB, sanitized line count: 930,826).
Wed Mar 12 20:11:55 2025 user.info adblock-lean: Successfully generated preprocessed blocklist file with 1,117,722 entries.
Wed Mar 12 20:11:55 2025 user.info adblock-lean: Sorting and merging the blocklist parts into a single blocklist file.
Wed Mar 12 20:11:59 2025 user.info adblock-lean: Stopping dnsmasq.
Wed Mar 12 20:11:59 2025 user.info adblock-lean: Checking the resulting blocklist with 'dnsmasq --test'.
Wed Mar 12 20:12:00 2025 user.info adblock-lean: New blocklist file check passed.
Wed Mar 12 20:12:00 2025 user.info adblock-lean: Final list uncompressed file size: 20.01 MiB.
Wed Mar 12 20:12:00 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:12:00 2025 user.info adblock-lean: Successfully imported new blocklist file for use by dnsmasq with size: 20.01 MiB.
Wed Mar 12 20:12:01 2025 user.info adblock-lean: Restarting dnsmasq.
Wed Mar 12 20:12:18 2025 user.info adblock-lean: Waiting for dnsmasq initialization.
Wed Mar 12 20:12:22 2025 user.info adblock-lean: Restart of dnsmasq completed.
Wed Mar 12 20:12:22 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 20:12:22 2025 user.info adblock-lean: Checking active blocklist.
Wed Mar 12 20:12:22 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Wed Mar 12 20:12:23 2025 user.info adblock-lean: Testing adblocking.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: Testing DNS resolution.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: Active blocklist check passed with the new blocklist file.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: New blocklist installed with entries count: 1,064,859.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: Checking for adblock-lean updates.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: Getting GitHub ref for latest version of adblock-lean.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: The locally installed adblock-lean is the latest version.
################################################################
#date ; service adblock-lean update -v commit=cd1d2d2 ; date
date ; service adblock-lean update -v commit=39e4476 ; date
Wed Mar 12 20:41:24 EDT 2025
Stopping adblock-lean.
Removing any adblock-lean blocklist files in dnsmasq config directories.
Checking dnsmasq instances.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Stopped adblock-lean.
Downloading adblock-lean, version '39e4476'.
NOTE: config format has changed from v6 to v7.
Error: Missing keys in config: 'DL_THREADS PROCESS_THREADS'.
Corresponding default config entries:
DL_THREADS="1"
PROCESS_THREADS="1"
Perform following automatic changes?
1. Re-add missing config entries with default values
y|n: y
Old config file was saved as /tmp/adblock-lean_config.old.
This will overwrite existing config. Proceed?
y|n: y
Saving new config file to '/etc/adblock-lean/config'.
Copying file 'adblock-lean'.
Copying file 'abl-lib.sh'.
Copying file 'abl-process.sh'.
Copying file 'abl-arrays.sh'.
The adblock-lean service is already enabled.
adblock-lean has been updated to version 'commit=39e4476'.
Start adblock-lean now? (y|n)
y|n: y
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
No existing compressed or uncompressed blocklist identified.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 23503)
Processing raw blocklist part from /etc/adblock-lean/blocklist
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Waiting for DL vacancy (running jobs: 1)
[ 0 ] Starting DL job (PID: 23515)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 23503 completed.
[ 1 ] Starting PROCESS job (PID: 23552)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 2 ] DL job 23515 completed.
[ 2 ] Scheduling DL job (running jobs: 1)
[ 2 ] Waiting for DL vacancy (running jobs: 1)
[ 2 ] Starting DL job (PID: 23582)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.5 MiB, lines: 186,880).
[ 2 ] Scheduling PROCESS job (running jobs: 1)
[ 2 ] PROCESS job 23552 completed.
[ 2 ] Starting PROCESS job (PID: 23602)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 6 ] DL job 23582 completed.
[ 6 ] Starting DL job (PID: 23635)
Downloading raw blocklist part from https://nsfw.oisd.nl/domainswild2
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 17.26 MiB, lines: 930,826).
[ 6 ] Scheduling PROCESS job (running jobs: 1)
[ 6 ] PROCESS job 23602 completed.
[ 6 ] Starting PROCESS job (PID: 23655)
Processing raw blocklist part from https://nsfw.oisd.nl/domainswild2
Successfully downloaded list part from https://nsfw.oisd.nl/domainswild2
[ 8 ] DL job 23635 completed.
Successfully processed list part from https://nsfw.oisd.nl/domainswild2 (size: 7.86 MiB, lines: 437,649).
[ 9 ] PROCESS job 23655 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,555,371 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 27.86 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 27.86 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:41s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,498,247.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Wed Mar 12 20:43:02 EDT 2025
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="1"
PROCESS_THREADS="1"
root@R4S-wrt:~#
################################################################
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="6"
PROCESS_THREADS="6"
root@R4S-wrt:~#
date ; /etc/init.d/adblock-lean start ; date
root@R4S-wrt:~# date ; /etc/init.d/adblock-lean start ; date
Wed Mar 12 20:55:21 EDT 2025
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating uncompressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 24377)
Processing raw blocklist part from /etc/adblock-lean/blocklist
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Starting DL job (PID: 24391)
[ 0 ] Scheduling DL job (running jobs: 2)
[ 0 ] Starting DL job (PID: 24393)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 0 ] Starting DL job (PID: 24395)
Downloading raw blocklist part from https://nsfw.oisd.nl/domainswild2
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 24377 completed.
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 24446)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 2)
[ 1 ] Starting PROCESS job (PID: 24447)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 24449)
Processing raw blocklist part from https://nsfw.oisd.nl/domainswild2
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 2 ] DL job 24391 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.5 MiB, lines: 186,880).
Successfully downloaded list part from https://nsfw.oisd.nl/domainswild2
Successfully processed list part from https://nsfw.oisd.nl/domainswild2 (size: 7.86 MiB, lines: 437,649).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 6 ] DL job 24393 completed.
[ 6 ] DL job 24395 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 17.26 MiB, lines: 930,826).
[ 6 ] PROCESS job 24446 completed.
[ 6 ] PROCESS job 24447 completed.
[ 6 ] PROCESS job 24449 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,555,371 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 27.86 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 27.86 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:38s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,498,246.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Wed Mar 12 20:56:00 EDT 2025
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~# logread | grep "^Wed Mar 12 20:5.*adblock-lean"
Wed Mar 12 20:55:21 2025 user.info adblock-lean: Starting adblock-lean, version 39e4476.
Wed Mar 12 20:55:21 2025 user.info adblock-lean: gawk detected so using gawk for fast (sub)domain match removal and entries packing.
Wed Mar 12 20:55:21 2025 user.info adblock-lean: GNU sed detected so list processing will be fast.
Wed Mar 12 20:55:21 2025 user.info adblock-lean: coreutils-sort detected so sort will be fast.
Wed Mar 12 20:55:21 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:55:21 2025 user.info adblock-lean: Creating uncompressed backup of existing blocklist.
Wed Mar 12 20:55:21 2025 user.info adblock-lean: Testing connectivity.
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Scheduling processing for the local blocklist.
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Starting raw blocklist part(s) download.
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Processing raw blocklist part from /etc/adblock-lean/blocklist
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Downloading raw blocklist part from https://nsfw.oisd.nl/domainswild2
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Wed Mar 12 20:55:22 2025 user.info adblock-lean: Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
Wed Mar 12 20:55:23 2025 user.info adblock-lean: Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Wed Mar 12 20:55:23 2025 user.info adblock-lean: Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Wed Mar 12 20:55:23 2025 user.info adblock-lean: Processing raw blocklist part from https://nsfw.oisd.nl/domainswild2
Wed Mar 12 20:55:24 2025 user.info adblock-lean: Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Wed Mar 12 20:55:24 2025 user.info adblock-lean: Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.5 MiB, lines: 186,880).
Wed Mar 12 20:55:25 2025 user.info adblock-lean: Successfully downloaded list part from https://nsfw.oisd.nl/domainswild2
Wed Mar 12 20:55:25 2025 user.info adblock-lean: Successfully processed list part from https://nsfw.oisd.nl/domainswild2 (size: 7.86 MiB, lines: 437,649).
Wed Mar 12 20:55:28 2025 user.info adblock-lean: Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Wed Mar 12 20:55:28 2025 user.info adblock-lean: Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 17.26 MiB, lines: 930,826).
Wed Mar 12 20:55:28 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 20:55:28 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 20:55:28 2025 user.info adblock-lean: Successfully generated preprocessed blocklist file with 1,555,371 entries.
Wed Mar 12 20:55:28 2025 user.info adblock-lean: Sorting and merging the blocklist parts into a single blocklist file.
Wed Mar 12 20:55:33 2025 user.info adblock-lean: Stopping dnsmasq.
Wed Mar 12 20:55:33 2025 user.info adblock-lean: Checking the resulting blocklist with 'dnsmasq --test'.
Wed Mar 12 20:55:34 2025 user.info adblock-lean: New blocklist file check passed.
Wed Mar 12 20:55:34 2025 user.info adblock-lean: Final list uncompressed file size: 27.86 MiB.
Wed Mar 12 20:55:34 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:55:34 2025 user.info adblock-lean: Successfully imported new blocklist file for use by dnsmasq with size: 27.86 MiB.
Wed Mar 12 20:55:34 2025 user.info adblock-lean: Restarting dnsmasq.
Wed Mar 12 20:55:52 2025 user.info adblock-lean: Waiting for dnsmasq initialization.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Restart of dnsmasq completed.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:38s.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Checking active blocklist.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Testing adblocking.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Testing DNS resolution.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Active blocklist check passed with the new blocklist file.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: New blocklist installed with entries count: 1,498,246.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
root@R4S-wrt:~#
htop showed 5 cpus at 100% for 2 or 3 seconds and the 6th was at about 70%.
After, there was one at 100% for a brief time with several other at 30% 0 40% for a few seconds.
################################################################
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="6"
PROCESS_THREADS="6"
root@R4S-wrt:~#
date ; /etc/init.d/adblock-lean start ; date
root@R4S-wrt:~# date ; /etc/init.d/adblock-lean start ; date
Wed Mar 12 21:09:39 EDT 2025
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating uncompressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 25284)
Processing raw blocklist part from /etc/adblock-lean/blocklist
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Starting DL job (PID: 25299)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 0 ] Starting DL job (PID: 25309)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 25284 completed.
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 25343)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 25344)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 2 ] DL job 25299 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.5 MiB, lines: 186,880).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 5 ] DL job 25309 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 17.26 MiB, lines: 930,826).
[ 5 ] PROCESS job 25343 completed.
[ 5 ] PROCESS job 25344 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,117,722 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 20.01 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 20.01 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:32s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,064,859.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Wed Mar 12 21:10:12 EDT 2025
root@R4S-wrt:~#
root@R4S-wrt:~# logread | grep "^Wed Mar 12 21.*adblock-lean"
Wed Mar 12 21:09:39 2025 user.info adblock-lean: Starting adblock-lean, version 39e4476.
Wed Mar 12 21:09:39 2025 user.info adblock-lean: gawk detected so using gawk for fast (sub)domain match removal and entries packing.
Wed Mar 12 21:09:39 2025 user.info adblock-lean: GNU sed detected so list processing will be fast.
Wed Mar 12 21:09:39 2025 user.info adblock-lean: coreutils-sort detected so sort will be fast.
Wed Mar 12 21:09:39 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 21:09:39 2025 user.info adblock-lean: Creating uncompressed backup of existing blocklist.
Wed Mar 12 21:09:39 2025 user.info adblock-lean: Testing connectivity.
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Scheduling processing for the local blocklist.
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Starting raw blocklist part(s) download.
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Processing raw blocklist part from /etc/adblock-lean/blocklist
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Wed Mar 12 21:09:40 2025 user.info adblock-lean: Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
Wed Mar 12 21:09:41 2025 user.info adblock-lean: Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Wed Mar 12 21:09:41 2025 user.info adblock-lean: Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Wed Mar 12 21:09:41 2025 user.info adblock-lean: Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Wed Mar 12 21:09:41 2025 user.info adblock-lean: Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.5 MiB, lines: 186,880).
Wed Mar 12 21:09:45 2025 user.info adblock-lean: Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Wed Mar 12 21:09:45 2025 user.info adblock-lean: Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 17.26 MiB, lines: 930,826).
Wed Mar 12 21:09:45 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 21:09:45 2025 user.info adblock-lean: Not using any allowlist for blocklist processing.
Wed Mar 12 21:09:45 2025 user.info adblock-lean: Successfully generated preprocessed blocklist file with 1,117,722 entries.
Wed Mar 12 21:09:45 2025 user.info adblock-lean: Sorting and merging the blocklist parts into a single blocklist file.
Wed Mar 12 21:09:48 2025 user.info adblock-lean: Stopping dnsmasq.
Wed Mar 12 21:09:48 2025 user.info adblock-lean: Checking the resulting blocklist with 'dnsmasq --test'.
Wed Mar 12 21:09:49 2025 user.info adblock-lean: New blocklist file check passed.
Wed Mar 12 21:09:49 2025 user.info adblock-lean: Final list uncompressed file size: 20.01 MiB.
Wed Mar 12 21:09:49 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 21:09:50 2025 user.info adblock-lean: Successfully imported new blocklist file for use by dnsmasq with size: 20.01 MiB.
Wed Mar 12 21:09:50 2025 user.info adblock-lean: Restarting dnsmasq.
Wed Mar 12 21:10:07 2025 user.info adblock-lean: Waiting for dnsmasq initialization.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Restart of dnsmasq completed.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Checking active blocklist.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Checking dnsmasq instances.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Testing adblocking.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Testing DNS resolution.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Active blocklist check passed with the new blocklist file.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: New blocklist installed with entries count: 1,064,859.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
root@R4S-wrt:~#
htop showed 2 cpus at 100% briefly, less than 2 seconds with 1 cpu at 100$ for a few seconds and several others at abour 30%.
Much lower peak cpu usage with the oisd list removed.
Thank you for the effort.
My conclusions:
DL_THREADS=1 and PROCESS_THREADS=1. Which is good.dnsmasq --test on the final blocklist, and loading it into dnsmasq. Because of that, the improvement seems very mild (about 8% for @spence, 3%-10% for @Lynx, ~11% on my VM).So in general, this is the sort of performance improvement we can probably expect. Not something to write home about, but also not completely useless.
In the meantime, I looked into the aforementioned idea of splitting the input stream into chunks with awk, but turns out that fifo writes in awk are blocking, and I didn't manage to come up with a way to work around this. Considering that even if such way is found, we are talking about shaving off a couple of extra seconds at best, probably this makes no sense to pursue further.
I also looked again into the later processing chain to see if there is anything to parallelize there. The only thing I found was zcat's which probably can be parallelized. I'll look into this a bit further.
All that said, the very good news are that the implementation seems to be working reliably.
I was curious so I did 3 more tests with THREADS= 6, 2, 1, using the following eleven blocklist_urls= (linefeeds added for post readability)
blocklist_urls="
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
"
resulting in 1,749,148 entries and size 34.38MiB.
New blocklist installed with entries count: 1,697,802.
UPDATE: did a run with THREADS="12"
Processing time for blocklist generation and import: 0m:40s.
/UPDATE
THREADS="6",
Processing time for blocklist generation and import: 0m:46s.
htop: All 6 cores at 100% for a few seconds at DL/processing time.
THREADS="2"
Processing time for blocklist generation and import: 0m:50s.
htop: All 6 cores at 100% for 2 seconds or a bit less at DL/processing time.
THREADS="1"
Processing time for blocklist generation and import: 0m:55s.
htop: Max of 2 cores at 100% concurrently for a brief time at DL/processing time.
root@R4S-wrt:~#
root@R4S-wrt:~# grep ^blocklist_urls /etc/adblock-lean/config
blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt"
root@R4S-wrt:~#
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="6"
PROCESS_THREADS="6"
root@R4S-wrt:~#
root@R4S-wrt:~# date ; /etc/init.d/adblock-lean start ; date
Thu Mar 13 06:53:25 EDT 2025
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating uncompressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 29187)
Processing raw blocklist part from /etc/adblock-lean/blocklist
Warning: Following Hagezi URLs are in dnsmasq format and should be either changed to raw list URLs
or moved to one of the 'dnsmasq_' config entries:
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Warning: Following Hagezi URLs are missing the '-onlydomains' suffix in the filename:
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Scheduling DL job (running jobs: 2)
[ 0 ] Starting DL job (PID: 29240)
[ 0 ] Scheduling DL job (running jobs: 3)
[ 0 ] Starting DL job (PID: 29241)
[ 0 ] Starting DL job (PID: 29242)
[ 0 ] Scheduling DL job (running jobs: 4)
[ 0 ] Scheduling DL job (running jobs: 5)
[ 0 ] Starting DL job (PID: 29244)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 0 ] Starting DL job (PID: 29247)
[ 0 ] Scheduling DL job (running jobs: 6)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
[ 0 ] Waiting for DL vacancy (running jobs: 6)
[ 0 ] Starting DL job (PID: 29249)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
Downloading raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 29187 completed.
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 29301)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 2)
[ 1 ] Starting PROCESS job (PID: 29302)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 3)
[ 1 ] Starting PROCESS job (PID: 29304)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
[ 1 ] Scheduling PROCESS job (running jobs: 4)
[ 1 ] Starting PROCESS job (PID: 29307)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
[ 1 ] Scheduling PROCESS job (running jobs: 5)
[ 1 ] Starting PROCESS job (PID: 29312)
Processing raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
[ 1 ] Starting PROCESS job (PID: 29337)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt (size: 3.62 KiB, lines: 93).
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt (size: 12.23 KiB, lines: 334).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt (size: 223.66 KiB, lines: 9,700).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.48 MiB, lines: 185,498).
Successfully downloaded list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
Successfully processed list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt (size: 13.95 MiB, lines: 618,886).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 6 ] DL job 29240 completed.
[ 6 ] DL job 29241 completed.
[ 6 ] DL job 29242 completed.
[ 6 ] DL job 29244 completed.
[ 6 ] DL job 29247 completed.
[ 6 ] DL job 29249 completed.
[ 6 ] Scheduling DL job (running jobs: 1)
[ 6 ] Starting DL job (PID: 29529)
[ 6 ] Scheduling DL job (running jobs: 2)
[ 6 ] Starting DL job (PID: 29530)
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 16.91 MiB, lines: 916,493).
[ 6 ] Scheduling DL job (running jobs: 3)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 6 ] Starting DL job (PID: 29533)
[ 6 ] Scheduling DL job (running jobs: 4)
Downloading raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
[ 6 ] Starting DL job (PID: 29536)
[ 6 ] Starting DL job (PID: 29539)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 7 ] Scheduling PROCESS job (running jobs: 6)
[ 7 ] PROCESS job 29301 completed.
[ 7 ] PROCESS job 29302 completed.
[ 7 ] PROCESS job 29304 completed.
[ 7 ] PROCESS job 29307 completed.
[ 7 ] PROCESS job 29312 completed.
[ 7 ] PROCESS job 29337 completed.
[ 7 ] Scheduling PROCESS job (running jobs: 1)
[ 7 ] Starting PROCESS job (PID: 29591)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
[ 7 ] Scheduling PROCESS job (running jobs: 2)
[ 7 ] Starting PROCESS job (PID: 29592)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 7 ] Scheduling PROCESS job (running jobs: 3)
[ 7 ] Starting PROCESS job (PID: 29594)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 7 ] Scheduling PROCESS job (running jobs: 4)
[ 7 ] Starting PROCESS job (PID: 29598)
Processing raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
[ 7 ] Starting PROCESS job (PID: 29603)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt (size: 4.85 KiB, lines: 184).
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt (size: 44.92 KiB, lines: 2,211).
[ 7 ] DL job 29529 completed.
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt (size: 90.95 KiB, lines: 3,830).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt (size: 262.36 KiB, lines: 10,501).
Successfully downloaded list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Successfully processed list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt (size: 29.14 KiB, lines: 1,402).
[ 7 ] DL job 29530 completed.
[ 7 ] DL job 29533 completed.
[ 7 ] DL job 29536 completed.
[ 7 ] DL job 29539 completed.
[ 8 ] PROCESS job 29591 completed.
[ 8 ] PROCESS job 29592 completed.
[ 8 ] PROCESS job 29594 completed.
[ 8 ] PROCESS job 29598 completed.
[ 8 ] PROCESS job 29603 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,749,148 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 34.38 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 34.38 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:46s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,697,802.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Thu Mar 13 06:54:12 EDT 2025
root@R4S-wrt:~#
#########################################################
root@R4S-wrt:~#
root@R4S-wrt:~# vi /etc/adblock-lean/config
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="2"
PROCESS_THREADS="2"
root@R4S-wrt:~#
root@R4S-wrt:~# date ; /etc/init.d/adblock-lean start ; date
Thu Mar 13 06:55:49 EDT 2025
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating uncompressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 30473)
Warning: Following Hagezi URLs are in dnsmasq format and should be either changed to raw list URLs
Processing raw blocklist part from /etc/adblock-lean/blocklist
or moved to one of the 'dnsmasq_' config entries:
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Warning: Following Hagezi URLs are missing the '-onlydomains' suffix in the filename:
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Starting DL job (PID: 30522)
[ 0 ] Scheduling DL job (running jobs: 2)
[ 0 ] Waiting for DL vacancy (running jobs: 2)
[ 0 ] Starting DL job (PID: 30525)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 30473 completed.
[ 1 ] Starting PROCESS job (PID: 30547)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 30549)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.48 MiB, lines: 185,498).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 5 ] DL job 30522 completed.
[ 5 ] DL job 30525 completed.
[ 5 ] Scheduling DL job (running jobs: 1)
[ 5 ] Starting DL job (PID: 30618)
[ 5 ] Scheduling DL job (running jobs: 2)
[ 5 ] Waiting for DL vacancy (running jobs: 2)
[ 5 ] Starting DL job (PID: 30619)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 16.91 MiB, lines: 916,493).
[ 5 ] Scheduling PROCESS job (running jobs: 2)
[ 5 ] PROCESS job 30547 completed.
[ 5 ] PROCESS job 30549 completed.
[ 5 ] Scheduling PROCESS job (running jobs: 1)
[ 5 ] Starting PROCESS job (PID: 30649)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
[ 5 ] Starting PROCESS job (PID: 30650)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt (size: 12.23 KiB, lines: 334).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt (size: 3.62 KiB, lines: 93).
[ 5 ] DL job 30618 completed.
[ 5 ] DL job 30619 completed.
[ 5 ] Scheduling DL job (running jobs: 1)
[ 5 ] Starting DL job (PID: 30726)
[ 5 ] Scheduling DL job (running jobs: 2)
[ 5 ] Waiting for DL vacancy (running jobs: 2)
[ 5 ] Starting DL job (PID: 30727)
Downloading raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
[ 6 ] Scheduling PROCESS job (running jobs: 2)
[ 6 ] PROCESS job 30649 completed.
[ 6 ] PROCESS job 30650 completed.
[ 6 ] Scheduling PROCESS job (running jobs: 1)
[ 6 ] Starting PROCESS job (PID: 30748)
Processing raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
[ 6 ] Starting PROCESS job (PID: 30749)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt (size: 223.66 KiB, lines: 9,700).
Successfully downloaded list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
[ 9 ] DL job 30726 completed.
[ 9 ] DL job 30727 completed.
[ 9 ] Scheduling DL job (running jobs: 1)
[ 9 ] Scheduling DL job (running jobs: 2)
[ 9 ] Starting DL job (PID: 30818)
[ 9 ] Waiting for DL vacancy (running jobs: 2)
[ 9 ] Starting DL job (PID: 30819)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Downloading raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Successfully processed list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt (size: 13.95 MiB, lines: 618,886).
[ 9 ] Scheduling PROCESS job (running jobs: 2)
[ 9 ] PROCESS job 30748 completed.
[ 9 ] PROCESS job 30749 completed.
[ 9 ] Scheduling PROCESS job (running jobs: 1)
[ 9 ] Starting PROCESS job (PID: 30849)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 9 ] Starting PROCESS job (PID: 30850)
Processing raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt (size: 44.92 KiB, lines: 2,211).
Successfully downloaded list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
[ 10 ] DL job 30818 completed.
[ 10 ] DL job 30819 completed.
[ 10 ] Scheduling DL job (running jobs: 1)
[ 10 ] Starting DL job (PID: 30920)
[ 10 ] Scheduling DL job (running jobs: 2)
Successfully processed list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt (size: 29.14 KiB, lines: 1,402).
[ 10 ] Waiting for DL vacancy (running jobs: 2)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
[ 10 ] Starting DL job (PID: 30921)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
[ 10 ] Scheduling PROCESS job (running jobs: 2)
[ 10 ] PROCESS job 30849 completed.
[ 10 ] PROCESS job 30850 completed.
[ 10 ] Scheduling PROCESS job (running jobs: 1)
[ 10 ] Starting PROCESS job (PID: 30948)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
[ 10 ] Starting PROCESS job (PID: 30949)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt (size: 4.85 KiB, lines: 184).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
[ 11 ] DL job 30920 completed.
[ 11 ] DL job 30921 completed.
[ 11 ] Starting DL job (PID: 31019)
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt (size: 90.95 KiB, lines: 3,830).
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 11 ] Scheduling PROCESS job (running jobs: 2)
[ 11 ] PROCESS job 30948 completed.
[ 11 ] PROCESS job 30949 completed.
[ 11 ] Starting PROCESS job (PID: 31037)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 12 ] DL job 31019 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt (size: 262.36 KiB, lines: 10,501).
[ 12 ] PROCESS job 31037 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,749,148 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 34.38 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 34.38 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:50s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,697,802.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Thu Mar 13 06:56:40 EDT 2025
root@R4S-wrt:~#
#########################################################
Run log for run 3 in reply post.
Let me know if you want logread output etc.
root@R4S-wrt:~#
root@R4S-wrt:~# vi /etc/adblock-lean/config
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="1"
PROCESS_THREADS="1"
root@R4S-wrt:~#
root@R4S-wrt:~# date ; /etc/init.d/adblock-lean start ; date
Thu Mar 13 06:59:05 EDT 2025
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating uncompressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 31768)
Warning: Following Hagezi URLs are in dnsmasq format and should be either changed to raw list URLs
Processing raw blocklist part from /etc/adblock-lean/blocklist
or moved to one of the 'dnsmasq_' config entries:
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Warning: Following Hagezi URLs are missing the '-onlydomains' suffix in the filename:
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Waiting for DL vacancy (running jobs: 1)
[ 0 ] Starting DL job (PID: 31820)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 31768 completed.
[ 1 ] Starting PROCESS job (PID: 31832)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 2 ] DL job 31820 completed.
[ 2 ] Scheduling DL job (running jobs: 1)
[ 2 ] Waiting for DL vacancy (running jobs: 1)
[ 2 ] Starting DL job (PID: 31862)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.48 MiB, lines: 185,498).
[ 2 ] Scheduling PROCESS job (running jobs: 1)
[ 2 ] PROCESS job 31832 completed.
[ 2 ] Starting PROCESS job (PID: 31882)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 6 ] DL job 31862 completed.
[ 6 ] Scheduling DL job (running jobs: 1)
[ 6 ] Waiting for DL vacancy (running jobs: 1)
[ 6 ] Starting DL job (PID: 31915)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 16.91 MiB, lines: 916,493).
[ 6 ] Scheduling PROCESS job (running jobs: 1)
[ 6 ] PROCESS job 31882 completed.
[ 6 ] Starting PROCESS job (PID: 31935)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt (size: 12.23 KiB, lines: 334).
[ 6 ] DL job 31915 completed.
[ 6 ] Scheduling DL job (running jobs: 1)
[ 6 ] Waiting for DL vacancy (running jobs: 1)
[ 6 ] Starting DL job (PID: 31974)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
[ 7 ] Scheduling PROCESS job (running jobs: 1)
[ 7 ] PROCESS job 31935 completed.
[ 7 ] Starting PROCESS job (PID: 31985)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt (size: 3.62 KiB, lines: 93).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt
[ 7 ] DL job 31974 completed.
[ 7 ] Scheduling DL job (running jobs: 1)
[ 7 ] Waiting for DL vacancy (running jobs: 1)
[ 7 ] Starting DL job (PID: 32024)
Downloading raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
[ 8 ] Scheduling PROCESS job (running jobs: 1)
[ 8 ] PROCESS job 31985 completed.
[ 8 ] Starting PROCESS job (PID: 32035)
Processing raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
Successfully downloaded list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt
[ 11 ] DL job 32024 completed.
[ 11 ] Scheduling DL job (running jobs: 1)
[ 11 ] Waiting for DL vacancy (running jobs: 1)
[ 11 ] Starting DL job (PID: 32067)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
Successfully processed list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt (size: 13.95 MiB, lines: 618,886).
[ 11 ] Scheduling PROCESS job (running jobs: 1)
[ 11 ] PROCESS job 32035 completed.
[ 11 ] Starting PROCESS job (PID: 32087)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt
[ 11 ] DL job 32067 completed.
[ 11 ] Scheduling DL job (running jobs: 1)
[ 11 ] Waiting for DL vacancy (running jobs: 1)
[ 11 ] Starting DL job (PID: 32117)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt (size: 223.66 KiB, lines: 9,700).
[ 12 ] Scheduling PROCESS job (running jobs: 1)
[ 12 ] PROCESS job 32087 completed.
[ 12 ] Starting PROCESS job (PID: 32137)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 12 ] DL job 32117 completed.
[ 12 ] Scheduling DL job (running jobs: 1)
[ 12 ] Waiting for DL vacancy (running jobs: 1)
[ 12 ] Starting DL job (PID: 32167)
Downloading raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt (size: 44.92 KiB, lines: 2,211).
[ 13 ] Scheduling PROCESS job (running jobs: 1)
[ 13 ] PROCESS job 32137 completed.
[ 13 ] Starting PROCESS job (PID: 32187)
Processing raw blocklist part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Successfully downloaded list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt
Successfully processed list part from https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt (size: 29.14 KiB, lines: 1,402).
[ 13 ] DL job 32167 completed.
[ 13 ] Scheduling DL job (running jobs: 1)
[ 13 ] Waiting for DL vacancy (running jobs: 1)
[ 13 ] Starting DL job (PID: 32225)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
[ 14 ] Scheduling PROCESS job (running jobs: 1)
[ 14 ] PROCESS job 32187 completed.
[ 14 ] Starting PROCESS job (PID: 32237)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt (size: 4.85 KiB, lines: 184).
[ 15 ] DL job 32225 completed.
[ 15 ] Scheduling DL job (running jobs: 1)
[ 15 ] Waiting for DL vacancy (running jobs: 1)
[ 15 ] Starting DL job (PID: 32276)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
[ 15 ] Scheduling PROCESS job (running jobs: 1)
[ 15 ] PROCESS job 32237 completed.
[ 15 ] Starting PROCESS job (PID: 32287)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt
[ 16 ] DL job 32276 completed.
[ 16 ] Starting DL job (PID: 32317)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt (size: 90.95 KiB, lines: 3,830).
[ 16 ] Scheduling PROCESS job (running jobs: 1)
[ 16 ] PROCESS job 32287 completed.
[ 16 ] Starting PROCESS job (PID: 32337)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt
[ 17 ] DL job 32317 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt (size: 262.36 KiB, lines: 10,501).
[ 18 ] PROCESS job 32337 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,749,148 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 34.38 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 34.38 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:55s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,697,802.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Thu Mar 13 07:00:01 EDT 2025
root@R4S-wrt:~#
UPDATE:
I changed the blocklists to wildcard onlydomains except for hosters and ran with THREADS="12". It ran faster and the installed entries is a bit smaller!
Successfully generated preprocessed blocklist file with 1,749,148 entries.
Successfully imported new blocklist file for use by dnsmasq with size: 29.13 MiB.
Processing time for blocklist generation and import: 0m:40s.
New blocklist installed with entries count: 1,679,110.
@antonk would it be an idea for the final sort -u to be split up based on the processing threads? Such that lines are split according to the first character into one of X files/fifos, which are then passed through individual instances of sort -u?
How would this fit in against (or not fit in against) the existing data flow path? I wonder if you could summarize that now. How the data is processed and then merged and at what points we are using temporary files?
By the way if the user right now hits ctrl-c mid-operation will all the forked processes get pulled down too?
I think this could be improved by placing a blocking read and having other process nudge it on by writing to FIFO. We should be using FIFOs for this stuff if we are using polling and files right now because FIFOs are designed to work with concurrency.
I see a dramatic improvement using that same blocklist URL set:
blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/native.amazon.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/native.apple.txt https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/gambling.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/anti.piracy.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt https://codeberg.org/hagezi/mirror2/raw/branch/main/dns-blocklists/dnsmasq/dyndns.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/nosafesearch.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/doh-vpn-proxy-bypass.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/fake.txt"
With:
# Number of parallel download and processing threads
DL_THREADS="1"
PROCESS_THREADS="1"
I see:
Thu Mar 13 12:31:01 2025 user.info adblock-lean: Processing time for blocklist generation and import: 1m:36s.
And with:
# Number of parallel download and processing threads
DL_THREADS="2"
PROCESS_THREADS="2"
I see:
Thu Mar 13 12:32:31 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:55s.
So here the multi-threaded approach is almost twice as fast.
I can also test the effect when lowering the connection speed by throttling using CAKE.
After all, the benefits of this new approach is affected not just by processing power, but also bandwidth.
My router has 6 cores so I did a run with 12 threads and it ran considerably faster.
Try running with THREADS="4" and THREADS="6".
Since each process may be throughput limited, they get context switched in and out of each processor so it makes sense that more threads than cores can be faster when each is I/O limited.
Here is my fixed blocklist:
blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt"
Only the last one is not "wildcard-onlydomains"
Thanks @spence and @Lynx! I'll analyze the results a bit later. And yes @spence, logs or, better, console output would be useful, if it's not too much work.
I suppose this could be implemented with a chain of tee>(grep "^${first_letter}" > "${first_letter}_fifo) commands. Not sure how much benefit this would provide if any, but you are welcome to experiment.
As to how this would fit: the final processing steps in the Concurrency branch are essentially the same as in latest release. I don't see a principal issue with implementing this sort of split and then processing the resulting fifo's, rather than having it arranged in one pipeline.
BTW probably a similar approach could be used to split the download streams into chunks, except using a combination of head and tail commands, rather than grep. My concern is that this adds multiple extra processing steps, which may benefit the use case of large number of cores but hurt the use case of small number of cores. Worth testing nevertheless, I guess.
I think a brief look at the code in abl-process.sh will provide a better answer, as I am a bit short of time rn, but I'll try to summarize this in a nutshell.
In latest release we are sequentially looping over the download URLs, piping the download stream directly into processing, which then gets written into a temp file. Those temp files are then futher processed and merged into the final blocklist.
In the Concurrency branch, the first download and processing step is split into 2 separate functions: dl_list_part(), process_list_part(). The scheduler stuff is just there to coordinate calls to these functions and handle errors. Basically dl_list_part() writes to a fifo file and process_list_part() reads from that file and writes to a temp file. The resulting temp files are then further processed and merged the same way as in the older code.
Actually, I think the forked processes may still stick around in this case. I was planning on implementing a trap mechanism which would kill those but didn't get around to doing this, yet. The other fault scenarios should be handled correctly (e.g. failed download, failed processing or stalled download thread).
Here is the run log with THREADS="12" using 11 blocklist_urls on my R4S.
I did also change away from using codeberg as a source with switching the new list to mostly wildcard onlydomains but did not look into the differences in download times.
root@R4S-wrt:~#
root@R4S-wrt:~# vi /etc/adblock-lean/config
root@R4S-wrt:~#
root@R4S-wrt:~# grep THREADS /etc/adblock-lean/config
DL_THREADS="12"
PROCESS_THREADS="12"
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~# date ; /etc/init.d/adblock-lean start ; date
Thu Mar 13 08:24:59 EDT 2025
Starting adblock-lean, version 39e4476.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating uncompressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 2187)
Processing raw blocklist part from /etc/adblock-lean/blocklist
Warning: Following Hagezi URLs are in dnsmasq format and should be either changed to raw list URLs
or moved to one of the 'dnsmasq_' config entries:
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Warning: Following Hagezi URLs are missing the '-onlydomains' suffix in the filename:
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 0 ] Scheduling DL job (running jobs: 0)
[ 1 ] Scheduling DL job (running jobs: 1)
[ 1 ] Starting DL job (PID: 2241)
[ 1 ] Scheduling DL job (running jobs: 2)
[ 1 ] Starting DL job (PID: 2242)
[ 1 ] Scheduling DL job (running jobs: 3)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Starting DL job (PID: 2244)
[ 1 ] Scheduling DL job (running jobs: 4)
[ 1 ] Starting DL job (PID: 2246)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Scheduling DL job (running jobs: 5)
[ 1 ] Starting DL job (PID: 2249)
[ 1 ] Scheduling DL job (running jobs: 6)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
[ 1 ] Scheduling DL job (running jobs: 7)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 1 ] Starting DL job (PID: 2253)
[ 1 ] Starting DL job (PID: 2257)
[ 1 ] Scheduling DL job (running jobs: 8)
[ 1 ] Starting DL job (PID: 2262)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 1 ] Scheduling DL job (running jobs: 9)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 1 ] Starting DL job (PID: 2271)
[ 1 ] Scheduling DL job (running jobs: 10)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
[ 1 ] Starting DL job (PID: 2278)
[ 1 ] Starting DL job (PID: 2289)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 2187 completed.
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 2352)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 2)
[ 1 ] Starting PROCESS job (PID: 2354)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 3)
[ 1 ] Starting PROCESS job (PID: 2357)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
[ 1 ] Scheduling PROCESS job (running jobs: 4)
[ 1 ] Starting PROCESS job (PID: 2360)
[ 1 ] Scheduling PROCESS job (running jobs: 5)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 2 ] Starting PROCESS job (PID: 2379)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
[ 2 ] Starting PROCESS job (PID: 2386)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
[ 2 ] Scheduling PROCESS job (running jobs: 6)
[ 2 ] Scheduling PROCESS job (running jobs: 7)
[ 2 ] Starting PROCESS job (PID: 2413)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 2 ] Scheduling PROCESS job (running jobs: 8)
[ 2 ] Starting PROCESS job (PID: 2423)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 2 ] Starting PROCESS job (PID: 2446)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 2 ] Scheduling PROCESS job (running jobs: 9)
[ 2 ] Scheduling PROCESS job (running jobs: 10)
[ 2 ] Starting PROCESS job (PID: 2466)
[ 2 ] Starting PROCESS job (PID: 2462)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt (size: 2.9 KiB, lines: 93).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt (size: 44.92 KiB, lines: 2,211).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt (size: 9.63 KiB, lines: 334).
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt (size: 3.42 KiB, lines: 184).
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt (size: 18.19 KiB, lines: 1,402).
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt (size: 61.04 KiB, lines: 3,830).
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt (size: 180.33 KiB, lines: 10,501).
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt (size: 147.88 KiB, lines: 9,700).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 3 ] DL job 2241 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.48 MiB, lines: 185,498).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt (size: 9.23 MiB, lines: 618,886).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 7 ] DL job 2242 completed.
[ 7 ] DL job 2244 completed.
[ 7 ] DL job 2246 completed.
[ 7 ] DL job 2249 completed.
[ 7 ] DL job 2253 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 16.91 MiB, lines: 916,493).
[ 7 ] DL job 2257 completed.
[ 7 ] DL job 2262 completed.
[ 7 ] DL job 2271 completed.
[ 7 ] DL job 2278 completed.
[ 7 ] DL job 2289 completed.
[ 8 ] PROCESS job 2352 completed.
[ 8 ] PROCESS job 2354 completed.
[ 8 ] PROCESS job 2357 completed.
[ 8 ] PROCESS job 2360 completed.
[ 8 ] PROCESS job 2379 completed.
[ 8 ] PROCESS job 2386 completed.
[ 8 ] PROCESS job 2413 completed.
[ 8 ] PROCESS job 2423 completed.
[ 8 ] PROCESS job 2446 completed.
[ 8 ] PROCESS job 2462 completed.
[ 8 ] PROCESS job 2466 completed.
Not using any allowlist for blocklist processing.
Not using any allowlist for blocklist processing.
Successfully generated preprocessed blocklist file with 1,749,148 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 29.13 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 29.13 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:40s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,679,110.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
Thu Mar 13 08:25:39 EDT 2025
root@R4S-wrt:~#
Sounds good. Should be simple enough to keep track of PIDs in main process and add list to kill on trap.
In case this gives any helpful ideas:
I just realised there is no timeout feature in the ash read:
https://linux.die.net/man/1/ash
Nor is there the special -t 0 feature available with the bash read to peak as to whether there is data to be read.
I don't know if that'd be problematic in terms of avoiding polling with my idea of blocking read on fifo. If process blocks on that read it can still be killed by main process. And we still avoid polling.
Sorry about the delayed response.
First of all, this:
means that there is a bug in blocklist processing code because this dnsmasq-formatted blocklist which you included in raw blocklist URLs should have been detected as invalid. Other dnsmasq-formatted blocklists as well. I can reproduce this behavior on my system. So this is something we'll need to look into.
Otherwise the results seem consistent with previous observations. The more URLs and cores, the higher the performance benefit.
I suppose this can be explained by the fact that each core in your machine can process data faster than it's being fetched. So it would be idling periodically when the data buffer is exhausted. When you configure a higher number of threads, the kernel (which ultimately manages what the cores are doing) occasionally assigns other threads which are waiting for processing to idle cores and this produces faster processing.
This is a good thing to know. Perhaps we should consider defaulting to a higher number of threads than cores. This might be architecture-dependent, though, and in some cases counterproductive. So I'm not sure that we can generalize based on this rather small sample size.
Super, super impressed that you've managed to implement this multi-threaded capability. It's a really exciting development - unlocking new possibilities.
Many thanks for your super helpful testing and insights @spence.
I just pushed another update to the Concurrency branch (current commit: 9266318).
It fixes rogue elements detection (apparently I broke it some time ago when adding support for top-level domains in blocklists), and implements killing off scheduler PID's and (recursively) all processes they spawned when cleanup_and_exit() is called. So now Ctrl+C no longer leaves hanging processes behind. Also fixes some cosmetic issues in the console messages.
BTW @Lynx take a look at kill_pids_recursive() - it can be reused almost as-is elsewhere, so maybe it'll be good for cake-autorate as well. It really goes all the way and kills absolutely everything spawned by given PID's, recursively.
Shouldn’t we first try regular kill for graceful exit then later try kill -9, which is akin to kill by fire?
I guess it's a good idea to use kill -9 in the final attempt. Added the code for that (not yet pushed to GH).
@antonk here is info on a test run for commit: 9266318
I also updated my config to replace
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/dnsmasq/hoster.txt
https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
for blocklist_urls= so no blocklist format check test although I think the "dnsmasq" format is accepted anyway.
I looked a couple times for an "wildcard/-onlydomains" url when I originally added that but there were only a few sections listed at that time but "wildcard" was listed today.
I got :
Error: Processing job (PID 11106) for list 'https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt' returned code 2.
Skipping file and continuing.
on the run with the update...
root@R4S-wrt:~# service adblock-lean update -v commit=9266318
Stopping adblock-lean.
Removing any adblock-lean blocklist files in dnsmasq config directories.
Checking dnsmasq instances.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Stopped adblock-lean.
Downloading adblock-lean, version '9266318'.
Copying file 'adblock-lean'.
Copying file 'abl-lib.sh'.
Copying file 'abl-process.sh'.
Copying file 'abl-arrays.sh'.
The adblock-lean service is already enabled.
adblock-lean has been updated to version 'commit=9266318'.
Start adblock-lean now? (y|n)
y|n: y
Starting adblock-lean, version 9266318.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
No existing compressed or uncompressed blocklist identified.
Testing connectivity.
Not using any allowlist for blocklist processing.
Scheduling processing for the local blocklist.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling PROCESS job (running jobs: 0)
[ 0 ] Starting PROCESS job (PID: 10946)
Processing raw blocklist part from /etc/adblock-lean/blocklist
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Scheduling DL job (running jobs: 2)
[ 0 ] Scheduling DL job (running jobs: 3)
[ 0 ] Starting DL job (PID: 10963)
[ 0 ] Starting DL job (PID: 10961)
[ 0 ] Starting DL job (PID: 10973)
[ 0 ] Scheduling DL job (running jobs: 4)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 0 ] Scheduling DL job (running jobs: 5)
[ 0 ] Starting DL job (PID: 10979)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 0 ] Scheduling DL job (running jobs: 6)
[ 0 ] Starting DL job (PID: 10982)
[ 0 ] Scheduling DL job (running jobs: 7)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
[ 0 ] Starting DL job (PID: 10984)
[ 0 ] Starting DL job (PID: 10987)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
[ 0 ] Scheduling DL job (running jobs: 8)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 0 ] Scheduling DL job (running jobs: 9)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 0 ] Starting DL job (PID: 10990)
[ 0 ] Scheduling DL job (running jobs: 10)
[ 0 ] Starting DL job (PID: 11004)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 0 ] Starting DL job (PID: 10998)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
[ 0 ] Starting DL job (PID: 11009)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
Successfully processed list from /etc/adblock-lean/blocklist (size: 432 B, lines: 16).
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] PROCESS job 10946 completed.
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 11096)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 2)
[ 1 ] Scheduling PROCESS job (running jobs: 3)
[ 1 ] Starting PROCESS job (PID: 11097)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 11099)
[ 1 ] Scheduling PROCESS job (running jobs: 4)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 11102)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 5)
[ 1 ] Starting PROCESS job (PID: 11106)
[ 1 ] Starting PROCESS job (PID: 11113)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 6)
[ 1 ] Scheduling PROCESS job (running jobs: 7)
[ 1 ] Starting PROCESS job (PID: 11154)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 8)
[ 1 ] Scheduling PROCESS job (running jobs: 9)
[ 1 ] Starting PROCESS job (PID: 11167)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 11194)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 10)
[ 1 ] Starting PROCESS job (PID: 11208)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 11222)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt (size: 27.2 KiB, lines: 2,184).
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt (size: 2.9 KiB, lines: 93).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt (size: 18.16 KiB, lines: 1,399).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt (size: 3.42 KiB, lines: 184).
Error: Line count in downloaded blocklist part from 'https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt' is 0, which is less than configured minimum: 1.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt (size: 61.16 KiB, lines: 3,836).
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt (size: 147.95 KiB, lines: 9,703).
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt (size: 180.72 KiB, lines: 10,524).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 2 ] DL job 10961 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.53 MiB, lines: 187,885).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt (size: 9.69 MiB, lines: 644,589).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 7 ] DL job 10963 completed.
[ 7 ] DL job 10973 completed.
[ 7 ] DL job 10979 completed.
[ 7 ] DL job 10982 completed.
[ 7 ] DL job 10984 completed.
[ 7 ] DL job 10987 completed.
[ 7 ] DL job 10990 completed.
[ 7 ] DL job 10998 completed.
[ 7 ] DL job 11004 completed.
[ 7 ] DL job 11009 completed.
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 18.04 MiB, lines: 963,462).
[ 8 ] PROCESS job 11096 completed.
[ 8 ] PROCESS job 11097 completed.
[ 8 ] PROCESS job 11099 completed.
[ 8 ] PROCESS job 11102 completed.
[ 8 ] PROCESS job 11106 completed.
Error: Processing job (PID 11106) for list 'https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt' returned code 2.
Skipping file and continuing.
[ 8 ] PROCESS job 11113 completed.
[ 8 ] PROCESS job 11154 completed.
[ 8 ] PROCESS job 11167 completed.
[ 8 ] PROCESS job 11194 completed.
[ 8 ] PROCESS job 11208 completed.
[ 8 ] PROCESS job 11222 completed.
Successfully generated preprocessed blocklist file with 1,823,875 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 30.65 MiB.
Checking dnsmasq instances.
Successfully imported new blocklist file for use by dnsmasq with size: 30.65 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 0m:41s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,747,942.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
root@R4S-wrt:~#
root@R4S-wrt:~#
root@R4S-wrt:~# wget https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt -O /tmp/native.amazon-onlydomains.txt
Downloading 'https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt'
Connecting to 185.199.110.133:443
Writing to '/tmp/native.amazon-onlydomains.txt'
/tmp/native.amazon-o 100% |*******************************| 9867 0:00:00 ETA
Download completed (9867 bytes)
root@R4S-wrt:~# wc /tmp/native.amazon-onlydomains.txt
345 387 9867 /tmp/native.amazon-onlydomains.txt
root@R4S-wrt:~#
But a manual wget worked fine. A few minutes later I ran service adblock-lean start to test for an issue like a parsing error and it was error free. Looks like it was probably a download issue external to adblock-lean.