Tested this and see largely similar times.
Is the way the execution time is getting monitored working correctly?
I'm not finding this commit in my git log but I can update adblock-lean to it, so apparently this is some commit which github is aware of but my git is not?
Theoretically there could be a bug in the simpler_concurrency branch, but I think it was working correctly in the Concurrency branch and I didn't change anything related to this for a while in that branch.
But overall, considering that previously you posted a result of 55s with the Concurrency branch, and now you are getting ~1m20s with same branch and (presumably) same URLs, doesn't it look like today your connection is slower and this is the reason why thread number doesn't affect the processing time?
I think the reason for that discrepancy is because the blocklist URLs changed as @spence updated them.
Ah, now I get it. I was posting the literally latest commit hashes on Github which correspond to 'merge' commits, but that merge is not reflected in my local git, which is why I wasn't finding the commit. Should be posting the actual code change commit hashes instead, going forward.
1 Like
The timings all seem stubbornly similar. But what’s weird is with some runs the logging showing progress looks much faster/smoother. That’s why I’m wondering about the accuracy of the timer.
How many threads does this refer to? Also could you post the exact reported time?
And could you test the Concurrency branch with 1 thread?
Sorry about all the testing, I'm just trying to get to the bottom of the issue.
Could you be more specific? What do you mean by faster/smoother?
That was with 8 threads.
If you see my console output between runs with different versions and settings the rate of seeing the logging of progress seemed to vary a lot.
How do you mean?
Well, you wrote 'similar times' but I'd like to see the exact figure, just so it's easier to understand what's going on.
Also if you could test with the 'Concurrency' branch (commit ae3b22b) and DL_THREADS=1, PROCESS_THREADS=1, this should provide a more complete picture.
With changing list sizes and short of full console output,
logread | grep -e "Processing time for blocklist generation" -e "adblock-lean: New blocklist installed"Might help with indicating processing volume...
Sat Mar 8 13:16:49 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Sat Mar 8 13:16:49 2025 user.info adblock-lean: New blocklist installed with entries count: 1,044,177.
Sun Mar 9 05:51:33 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Sun Mar 9 05:51:33 2025 user.info adblock-lean: New blocklist installed with entries count: 1,043,946.
Mon Mar 10 05:35:32 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:33s.
Mon Mar 10 05:35:33 2025 user.info adblock-lean: New blocklist installed with entries count: 1,053,081.
Tue Mar 11 05:21:32 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Tue Mar 11 05:21:32 2025 user.info adblock-lean: New blocklist installed with entries count: 1,033,318.
Wed Mar 12 05:31:33 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:33s.
Wed Mar 12 05:31:33 2025 user.info adblock-lean: New blocklist installed with entries count: 1,052,550.
Wed Mar 12 20:12:22 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 20:12:23 2025 user.info adblock-lean: New blocklist installed with entries count: 1,064,859.
Wed Mar 12 20:43:02 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:41s.
Wed Mar 12 20:43:02 2025 user.info adblock-lean: New blocklist installed with entries count: 1,498,247.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:38s.
Wed Mar 12 20:56:00 2025 user.info adblock-lean: New blocklist installed with entries count: 1,498,246.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 21:10:12 2025 user.info adblock-lean: New blocklist installed with entries count: 1,064,859.
Wed Mar 12 21:17:39 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Wed Mar 12 21:17:39 2025 user.info adblock-lean: New blocklist installed with entries count: 1,064,859.
Thu Mar 13 05:40:32 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:32s.
Thu Mar 13 05:40:33 2025 user.info adblock-lean: New blocklist installed with entries count: 1,050,727.
Thu Mar 13 06:54:12 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:46s.
Thu Mar 13 06:54:12 2025 user.info adblock-lean: New blocklist installed with entries count: 1,697,802.
Thu Mar 13 06:56:40 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:50s.
Thu Mar 13 06:56:40 2025 user.info adblock-lean: New blocklist installed with entries count: 1,697,802.
Thu Mar 13 07:00:01 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:55s.
Thu Mar 13 07:00:01 2025 user.info adblock-lean: New blocklist installed with entries count: 1,697,802.
Thu Mar 13 07:35:42 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:54s.
Thu Mar 13 07:35:42 2025 user.info adblock-lean: New blocklist installed with entries count: 1,697,802.
Thu Mar 13 08:25:39 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:40s.
Thu Mar 13 08:25:39 2025 user.info adblock-lean: New blocklist installed with entries count: 1,679,110.
Sat Mar 15 05:52:41 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:40s.
Sat Mar 15 05:52:41 2025 user.info adblock-lean: New blocklist installed with entries count: 1,746,840.
Sat Mar 15 13:21:23 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:41s.
Sat Mar 15 13:21:23 2025 user.info adblock-lean: New blocklist installed with entries count: 1,747,942.
Sat Mar 15 13:32:48 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:42s.
Sat Mar 15 13:32:48 2025 user.info adblock-lean: New blocklist installed with entries count: 1,747,990.
Sun Mar 16 05:15:43 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:42s.
Sun Mar 16 05:15:43 2025 user.info adblock-lean: New blocklist installed with entries count: 1,775,185.
Mon Mar 17 05:15:43 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:42s.
Mon Mar 17 05:15:43 2025 user.info adblock-lean: New blocklist installed with entries count: 1,783,342.
Tue Mar 18 05:33:43 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:43s.
Tue Mar 18 05:33:44 2025 user.info adblock-lean: New blocklist installed with entries count: 1,795,449.
Wed Mar 19 05:52:43 2025 user.info adblock-lean: Processing time for blocklist generation and import: 0m:42s.
Wed Mar 19 05:52:44 2025 user.info adblock-lean: New blocklist installed with entries count: 1,802,820.
EDIT: edits were just to play with formatting...
[code]logread | grep -e "Processing time for blocklist generation" -e "adblock-lean: New blocklist installed"[/code]
vs
`logread | grep -e "Processing time for blocklist generation" -e "adblock-lean: New blocklist installed"`
2 Likes
Processing time for blocklist generation and import: 1m:30s.
root@OpenWrt:~# service adblock-lean start
Starting adblock-lean, version ae3b22b.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
No existing compressed or uncompressed blocklist identified.
Testing connectivity.
Not using any allowlist for blocklist processing.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Waiting for DL vacancy (running jobs: 1)
[ 0 ] Starting DL job (PID: 31239)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 0)
[ 1 ] Starting PROCESS job (PID: 31250)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.56 MiB, lines: 190,498).
[ 5 ] DL job 31239 completed.
[ 5 ] Scheduling DL job (running jobs: 1)
[ 5 ] Waiting for DL vacancy (running jobs: 1)
[ 5 ] Starting DL job (PID: 31296)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 6 ] Scheduling PROCESS job (running jobs: 1)
[ 6 ] PROCESS job 31250 completed.
[ 6 ] Starting PROCESS job (PID: 31313)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 18.45 MiB, lines: 998,494).
[ 25 ] DL job 31296 completed.
[ 25 ] Starting DL job (PID: 31396)
[ 25 ] Scheduling DL job (running jobs: 1)
[ 25 ] Waiting for DL vacancy (running jobs: 1)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
[ 26 ] Scheduling PROCESS job (running jobs: 1)
[ 26 ] PROCESS job 31313 completed.
[ 26 ] Starting PROCESS job (PID: 31410)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt (size: 9.63 KiB, lines: 334).
[ 26 ] DL job 31396 completed.
[ 26 ] Scheduling DL job (running jobs: 1)
[ 26 ] Waiting for DL vacancy (running jobs: 1)
[ 26 ] Starting DL job (PID: 31449)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
[ 27 ] Scheduling PROCESS job (running jobs: 1)
[ 27 ] PROCESS job 31410 completed.
[ 27 ] Starting PROCESS job (PID: 31463)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt (size: 2.9 KiB, lines: 93).
[ 27 ] DL job 31449 completed.
[ 27 ] Scheduling DL job (running jobs: 1)
[ 27 ] Waiting for DL vacancy (running jobs: 1)
[ 27 ] Starting DL job (PID: 31501)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 28 ] Scheduling PROCESS job (running jobs: 1)
[ 28 ] PROCESS job 31463 completed.
[ 28 ] Starting PROCESS job (PID: 31514)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt (size: 9.95 MiB, lines: 662,500).
[ 38 ] DL job 31501 completed.
[ 38 ] Scheduling DL job (running jobs: 1)
[ 38 ] Waiting for DL vacancy (running jobs: 1)
[ 38 ] Starting DL job (PID: 31574)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 39 ] Scheduling PROCESS job (running jobs: 1)
[ 39 ] PROCESS job 31514 completed.
[ 39 ] Starting PROCESS job (PID: 31586)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt (size: 150.46 KiB, lines: 9,859).
[ 39 ] DL job 31574 completed.
[ 39 ] Scheduling DL job (running jobs: 1)
[ 39 ] Starting DL job (PID: 31626)
[ 39 ] Waiting for DL vacancy (running jobs: 1)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 40 ] Scheduling PROCESS job (running jobs: 1)
[ 40 ] PROCESS job 31586 completed.
[ 40 ] Starting PROCESS job (PID: 31638)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt (size: 18.17 KiB, lines: 1,400).
[ 40 ] DL job 31626 completed.
[ 40 ] Scheduling DL job (running jobs: 1)
[ 40 ] Starting DL job (PID: 31676)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
[ 40 ] Waiting for DL vacancy (running jobs: 1)
[ 41 ] Scheduling PROCESS job (running jobs: 1)
[ 41 ] PROCESS job 31638 completed.
[ 41 ] Starting PROCESS job (PID: 31688)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt (size: 3.42 KiB, lines: 184).
[ 41 ] DL job 31676 completed.
[ 41 ] Scheduling DL job (running jobs: 1)
[ 41 ] Waiting for DL vacancy (running jobs: 1)
[ 41 ] Starting DL job (PID: 31725)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
[ 42 ] Scheduling PROCESS job (running jobs: 1)
[ 42 ] PROCESS job 31688 completed.
[ 42 ] Starting PROCESS job (PID: 31738)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt (size: 61.68 KiB, lines: 3,870).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
[ 42 ] DL job 31725 completed.
[ 42 ] Scheduling DL job (running jobs: 1)
[ 42 ] Waiting for DL vacancy (running jobs: 1)
[ 42 ] Starting DL job (PID: 31776)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
[ 43 ] Scheduling PROCESS job (running jobs: 1)
[ 43 ] PROCESS job 31738 completed.
[ 43 ] Starting PROCESS job (PID: 31788)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt (size: 184.26 KiB, lines: 10,741).
[ 44 ] DL job 31776 completed.
[ 44 ] Starting DL job (PID: 31825)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
[ 44 ] Scheduling PROCESS job (running jobs: 1)
[ 44 ] PROCESS job 31788 completed.
[ 44 ] Starting PROCESS job (PID: 31838)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
[ 44 ] DL job 31825 completed.
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt (size: 27.26 KiB, lines: 2,189).
[ 45 ] PROCESS job 31838 completed.
Successfully generated preprocessed blocklist file with 1,880,162 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 31.37 MiB.
Checking dnsmasq instances.
Successfully imported new compressed blocklist file for use by dnsmasq with size: 10.24 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 1m:30s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,802,845.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
And now with DL_THREADS=4 , PROCESS_THREADS=4:
Processing time for blocklist generation and import: 1m:22s.
root@OpenWrt:~# service adblock-lean start
Starting adblock-lean, version ae3b22b.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
Creating compressed backup of existing blocklist.
Testing connectivity.
Not using any allowlist for blocklist processing.
Starting raw blocklist part(s) download.
[ 0 ] Scheduling DL job (running jobs: 0)
[ 0 ] Scheduling DL job (running jobs: 1)
[ 0 ] Starting DL job (PID: 32503)
[ 0 ] Scheduling DL job (running jobs: 2)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 0 ] Starting DL job (PID: 32504)
[ 0 ] Starting DL job (PID: 32506)
[ 0 ] Scheduling DL job (running jobs: 3)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 0 ] Starting DL job (PID: 32510)
[ 0 ] Scheduling DL job (running jobs: 4)
[ 0 ] Waiting for DL vacancy (running jobs: 4)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 0)
[ 1 ] Scheduling PROCESS job (running jobs: 1)
[ 1 ] Starting PROCESS job (PID: 32547)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
[ 1 ] Scheduling PROCESS job (running jobs: 2)
[ 1 ] Starting PROCESS job (PID: 32548)
[ 1 ] Scheduling PROCESS job (running jobs: 3)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 32552)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
[ 1 ] Starting PROCESS job (PID: 32555)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt (size: 2.9 KiB, lines: 93).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt (size: 9.63 KiB, lines: 334).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt (size: 3.56 MiB, lines: 190,498).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt (size: 18.45 MiB, lines: 998,494).
[ 24 ] DL job 32503 completed.
[ 24 ] DL job 32504 completed.
[ 24 ] DL job 32506 completed.
[ 24 ] DL job 32510 completed.
[ 24 ] Scheduling DL job (running jobs: 1)
[ 24 ] Starting DL job (PID: 32734)
[ 24 ] Scheduling DL job (running jobs: 2)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 24 ] Starting DL job (PID: 32735)
[ 24 ] Starting DL job (PID: 32737)
[ 24 ] Scheduling DL job (running jobs: 3)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 24 ] Starting DL job (PID: 32743)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 24 ] Scheduling DL job (running jobs: 4)
[ 24 ] Waiting for DL vacancy (running jobs: 4)
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
[ 25 ] Scheduling PROCESS job (running jobs: 4)
[ 25 ] PROCESS job 32547 completed.
[ 25 ] PROCESS job 32548 completed.
[ 25 ] PROCESS job 32552 completed.
[ 25 ] PROCESS job 32555 completed.
[ 25 ] Scheduling PROCESS job (running jobs: 1)
[ 25 ] Starting PROCESS job (PID: 315)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 25 ] Starting PROCESS job (PID: 316)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
[ 25 ] Scheduling PROCESS job (running jobs: 2)
[ 25 ] Scheduling PROCESS job (running jobs: 3)
[ 25 ] Starting PROCESS job (PID: 321)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
[ 25 ] Starting PROCESS job (PID: 322)
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt (size: 18.17 KiB, lines: 1,400).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt (size: 3.42 KiB, lines: 184).
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt (size: 150.46 KiB, lines: 9,859).
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt (size: 9.95 MiB, lines: 662,500).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt
[ 35 ] DL job 32734 completed.
[ 35 ] DL job 32735 completed.
[ 35 ] DL job 32737 completed.
[ 35 ] DL job 32743 completed.
[ 35 ] Scheduling DL job (running jobs: 1)
[ 35 ] Starting DL job (PID: 481)
[ 35 ] Scheduling DL job (running jobs: 2)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
[ 35 ] Starting DL job (PID: 484)
[ 35 ] Starting DL job (PID: 482)
Downloading raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Downloading raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
[ 36 ] Scheduling PROCESS job (running jobs: 4)
[ 36 ] PROCESS job 315 completed.
[ 36 ] PROCESS job 316 completed.
[ 36 ] PROCESS job 321 completed.
[ 36 ] PROCESS job 322 completed.
[ 36 ] Scheduling PROCESS job (running jobs: 1)
[ 36 ] Scheduling PROCESS job (running jobs: 2)
[ 36 ] Starting PROCESS job (PID: 518)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
[ 36 ] Starting PROCESS job (PID: 519)
[ 36 ] Starting PROCESS job (PID: 520)
Processing raw blocklist part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Processing raw blocklist part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt (size: 61.68 KiB, lines: 3,870).
[ 36 ] DL job 481 completed.
Successfully downloaded list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt
Successfully processed list part from https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt (size: 27.26 KiB, lines: 2,189).
Successfully downloaded list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt
Successfully processed list part from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt (size: 184.26 KiB, lines: 10,741).
[ 36 ] DL job 482 completed.
[ 36 ] DL job 484 completed.
[ 37 ] PROCESS job 518 completed.
[ 37 ] PROCESS job 519 completed.
[ 37 ] PROCESS job 520 completed.
Successfully generated preprocessed blocklist file with 1,880,162 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 31.37 MiB.
Checking dnsmasq instances.
Successfully imported new compressed blocklist file for use by dnsmasq with size: 10.24 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 1m:22s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,802,845.
adblock-lean was installed from a specific Git commit. Automatic updates check is disabled.
1 Like
I'll summarize your results as I understand them.
- With the
simpler_concurrencybranch, total time is 1:22 - 1:24, regardless of the number of threads (including with 1 thread). - With the
Concurrencybranch, total time varies between 1:19 (4 threads, older commit), 1:22 (4 threads, latest commit) and 1:30 (1 thread, latest commit). - Looks like at ~1:20 is a hard-limit of your download/processing speed right now.
- The
simpler_concurrencybranch provides almost the same performance with 1 thread as theConcurrencybranch with 4 threads.
My interpretation: the simpler_concurrency branch eliminates some scheduler overheads which makes it faster than the Concurrency branch. The ~3s difference between the 4-threads result with the Concurrency branch and the simpler_concurrency branch might be a result of your connection speed/latency fluctuating a bit. The fact that increasing the number of threads makes almost no difference with the simpler_concurrency branch, and the fact that there is a hard-limit around 1:20 leaves me no other good explanation except download bandwidth limitation.
Maybe it's your provider today? Maybe something is loading up your Internet connection? Maybe it's some cake/qos settings?
For now, I don't have any better ideas. Adding threads at minimum means that more downloads and processing are run in parallel, the simpler_concurrency branch doesn't change that. I triple-checked the code responsible for calculating the total time - it's simple, solid and correct.
So how can one explain these results otherwise, except bandwidth limitation? Maybe the processing with 1 thread with the simpler_concurrency branch loads the CPU enough for it to hit some sort of thermal or power limit, which is hit with any higher number of threads as well, which makes it throttle down with a higher number of threads? And this wasn't happening with the Concurrency branch with 1 thread because there were some moments when the CPU was idling? Maybe the 1 thread in the simpler_concurrency branch is actually loading both of your CPU cores?
BTW if not too hard, could you test the latest release version with the same URLs, just to have another point of data on single-threaded performance.
1 Like
Uh oh:
root@OpenWrt:~# service adblock-lean start
Starting adblock-lean, version v0.6.1.
gawk detected so using gawk for fast (sub)domain match removal and entries packing.
GNU sed detected so list processing will be fast.
coreutils-sort detected so sort will be fast.
Checking dnsmasq instances.
No existing compressed or uncompressed blocklist identified.
Testing connectivity.
No local allowlist identified.
Not using any allowlist for blocklist processing.
No local blocklist identified.
Starting raw blocklist part(s) download.
Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt.
Successfully processed blocklist (source file size: 3.56 MiB, sanitized line count: 190,498).
Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt.
Successfully processed blocklist (source file size: 18.45 MiB, sanitized line count: 998,494).
Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/native.amazon-onlydomains.txt.
Successfully processed blocklist (source file size: 9.63 KiB, sanitized line count: 334).
Downloading, checking and sanitizing raw blocklist part from: https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/native.apple-onlydomains.txt.
Successfully processed blocklist (source file size: 2.9 KiB, sanitized line count: 93).
Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt.
Successfully processed blocklist (source file size: 9.95 MiB, sanitized line count: 662,500).
Downloading, checking and sanitizing raw blocklist part from: https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/anti.piracy-onlydomains.txt.
Successfully processed blocklist (source file size: 150.46 KiB, sanitized line count: 9,859).
Downloading, checking and sanitizing raw blocklist part from: https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/dyndns-onlydomains.txt.
Successfully processed blocklist (source file size: 18.17 KiB, sanitized line count: 1,400).
Downloading, checking and sanitizing raw blocklist part from: https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/nosafesearch-onlydomains.txt.
Successfully processed blocklist (source file size: 3.42 KiB, sanitized line count: 184).
Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-vpn-proxy-bypass-onlydomains.txt.
Successfully processed blocklist (source file size: 61.68 KiB, sanitized line count: 3,870).
Downloading, checking and sanitizing raw blocklist part from: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt.
Successfully processed blocklist (source file size: 184.26 KiB, sanitized line count: 10,741).
Downloading, checking and sanitizing raw blocklist part from: https://gitlab.com/hagezi/mirror/-/raw/main/dns-blocklists/wildcard/hoster-onlydomains.txt.
Successfully processed blocklist (source file size: 27.26 KiB, sanitized line count: 2,189).
Successfully generated preprocessed blocklist file with 1,880,162 entries.
Sorting and merging the blocklist parts into a single blocklist file.
Stopping dnsmasq.
Checking the resulting blocklist with 'dnsmasq --test'.
New blocklist file check passed.
Final list uncompressed file size: 31.37 MiB.
Checking dnsmasq instances.
Successfully imported new compressed blocklist file for use by dnsmasq with size: 10.24 MiB.
Restarting dnsmasq.
Waiting for dnsmasq initialization.
Restart of dnsmasq completed.
Processing time for blocklist generation and import: 1m:21s.
Checking active blocklist.
Checking dnsmasq instances.
Using following nameservers for DNS resolution verification: 127.0.0.1, ::1
Testing adblocking.
Testing DNS resolution.
Active blocklist check passed with the new blocklist file.
New blocklist installed with entries count: 1,802,845.
Checking for adblock-lean updates.
Getting GitHub ref for latest version of adblock-lean.
The locally installed adblock-lean is the latest version.
How is this possible?
Running htop I am seeing:
So does this mean that piping processes in ash allows distribution of processes across cores already?
Look at those two sed processes - they appear to be running simultaneously and presumably on different cores?
1 Like
Well, technically we are running a lot of processes in our processing chain, for each "thread". I guess it is possible that Linux distributes the load between multiple cores automatically, in some magical way, to some degree, and with only 2 cores, both of them are loaded with our current processing chain. I don't have an insight into this, unfortunately. But clearly this is the reason why parallelization is making no difference right now.
Because one of the processes loading your CPU in that screenshot is gzip, I suppose you could disable compression and re-test with single thread and multiple threads - perhaps this will lift enough load from the CPU to leave room for parallelization?
Disabling compression gives:
Processing time for blocklist generation and import: 1m:12s.
And now with 4 threads using -v commit=ae3b22b:
Processing time for blocklist generation and import: 1m:14s.
1 Like
And 1, 2, 4 threads with -v commit=88e1514? I hope I'm not asking too much.
4 threads:
Processing time for blocklist generation and import: 1m:10s.
2 threads:
Processing time for blocklist generation and import: 1m:13s.
1 thread:
Processing time for blocklist generation and import: 1m:16s.
1 Like