Currently I'm working on adblock 3.6.x - major change is a new reporting engine (based on pcap files generated by tcpdump). Currently the report includes the following figures - do we need more?
Thanks for your feedback!
1 Like
@dibdot This upcoming development for 3.6.x is absolutely fantastic and I am looking forward to it. It is thorough, yet well organized and concise. Perfection!
I honestly cannot think of anything to add since it seems to cover the most relevant information already. Keep up the great work.
Hi dibdot!
I need your advice: adblock is counting overall twice.
My log:
Sun Dec 9 11:28:15 2018 user.info adblock-3.5.5-2[31809]: adblock instance started ::: action: reload, priority: 0, pid: 31809
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_jsnup::: status: running, mode: normal mode, cnt: 35742, notify: 0, notify_cnt: 0, notify_pid: -
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_uci ::: config: dhcp, change:
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_uci ::: config: firewall, change:
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: dns: dnsmasq, fetch_util: /bin/uclient-fetch (libustream-ssl), backup: 0, backup_mode: 0, dns_jail: 0, force_srt: 0, force_dns: 1, mem_total: 60, mem_free: 23, max_queue: 16
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: cameleon_pihole_def, enabled: 1
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: malware_pihole_def, enabled: 1
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: disconnect_malware, enabled: 1
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: disconnect_ad_pihole_def, enabled: 1
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: zeus_pihole_def, enabled: 1
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: blacklist, enabled: 1
Sun Dec 9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: blacklist, mode: download, cnt: 10, in_rc: 0, out_rc: 0
Sun Dec 9 11:28:16 2018 user.debug adblock-3.5.5-2[31809]: f_tld ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.blacklist, cnt: 10, cnt_srt: 10, cnt_tld: 10
Sun Dec 9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_tld ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.zeus_pihole_def, cnt: 381, cnt_srt: 381, cnt_tld: 380
Sun Dec 9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: disconnect_malware, mode: download, cnt: 2831, in_rc: 0, out_rc: 0
Sun Dec 9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: disconnect_ad_pihole_def, mode: download, cnt: 2703, in_rc: 0, out_rc: 0
Sun Dec 9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_tld ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: 2668
Sun Dec 9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_tld ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.disconnect_malware, cnt: 2831, cnt_srt: 2831, cnt_tld: 2790
Sun Dec 9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: cameleon_pihole_def, mode: download, cnt: 20567, in_rc: 0, out_rc: 0
Sun Dec 9 11:28:34 2018 user.debug adblock-3.5.5-2[31809]: f_tld ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.cameleon_pihole_def, cnt: 20567, cnt_srt: 20567, cnt_tld: 16409
Sun Dec 9 11:28:54 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: malware_pihole_def, mode: download, cnt: 26850, in_rc: 0, out_rc: 0
Sun Dec 9 11:29:06 2018 user.debug adblock-3.5.5-2[31809]: f_tld ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.malware_pihole_def, cnt: 26850, cnt_srt: 26850, cnt_tld: 26362
Sun Dec 9 11:29:07 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: overall, mode: merge, cnt: 97238, in_rc: 0, out_rc: 0
Sun Dec 9 11:29:07 2018 user.debug adblock-3.5.5-2[31809]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 68da62929ec9c976afb1add6172756da231c369e6a85446dd48dcd848c1163f5, out_rc: 1
Sun Dec 9 11:29:12 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: overall, mode: final, cnt: 97238, in_rc: 0, out_rc: 0
Summary of all the "cnt_tld" is exactly the half of the "overall cnt"
How can that be?
Thank you!
Problem (?) nr. 2:
if i only load 1 list, it counts more at the end as it is in the list!
For example malware list :
entries: 27.043
adblock count: 52.724
See log below:
Mon Dec 10 22:02:12 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: malware_pihole_def, mode: download, cnt: 26850, in_rc: 0, out_rc: 0
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_tld ::: source: /tmp/tmp.PEafGi/tmp.fJEfbN.malware_pihole_def, cnt: 26850, cnt_srt: 26850, cnt_tld: 26362
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: overall, mode: merge, cnt: 52724, in_rc: 0, out_rc: 0
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, out_rc: 1
Mon Dec 10 22:02:27 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: overall, mode: final, cnt: 52724, in_rc: 0, out_rc: 0
Thank you!
Clearly a bug in the f_tld() function, seen only on low memory devices - will be fixed in 3.6.0. In the meantime you could change it in /usr/bin/adblock.sh on your own:
old: line 577 ff.
[...]
if [ ${?} -eq 0 ]
then
cnt_tld="$(wc -l 2>/dev/null < "${source}")"
else
mv -f "${temp}" > "${source}"
fi
fi
[...]
new, added missing "rm" command :
[...]
if [ ${?} -eq 0 ]
then
cnt_tld="$(wc -l 2>/dev/null < "${source}")"
rm -f "${temp}" # that's the bugfix!
else
mv -f "${temp}" > "${source}"
fi
fi
[...]
Sorry for the mess!
Thanx dibdot!
It works , but it seems tld is not working anymore?
with bugfix: "cnt_tld: -"
without bugfix: "cnt_tld: 2668"
See logs:
with bugfix
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_tld ::: source: /tmp/tmp.CADBOK/tmp.LAAJnc.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: -
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_tld ::: source: /tmp/tmp.CADBOK/tmp.LAAJnc.disconnect_malware, cnt: 2828, cnt_srt: 2828, cnt_tld: -
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_list ::: name: overall, mode: merge, cnt: 5456, in_rc: 0, out_rc: 0
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b, out_rc: 1
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_list ::: name: overall, mode: final, cnt: 5456, in_rc: 0, out_rc: 0
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: e26d71fcb03cd82d6058a9ef1479665f685da05e0b358b27ec18ba780f61fdb0, out_rc: 1
Tue Dec 11 14:55:11 2018 user.debug adblock-3.5.5-2[6440]: f_dnsup::: cache_util: -, cache_rc: -, cache_flush: 1, cache_cnt: 0, rc: 0
Tue Dec 11 14:55:11 2018 user.debug adblock-3.5.5-2[6440]: f_jsnup::: status: enabled, mode: normal mode, cnt: 5456, notify: 0, notify_cnt: 0, notify_pid: -
Tue Dec 11 14:55:11 2018 user.info adblock-3.5.5-2[6440]: blocklist with overall 5456 domains loaded successfully (TP-Link TL-WR1043N/ND v3, OpenWrt 18.06.1 r7258-5eb055306f)
without bugfix
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_tld ::: source: /tmp/tmp.FOEMBh/tmp.kpLFCK.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: 2668
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_tld ::: source: /tmp/tmp.FOEMBh/tmp.kpLFCK.disconnect_malware, cnt: 2828, cnt_srt: 2828, cnt_tld: 2788
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_list ::: name: overall, mode: merge, cnt: 10912, in_rc: 0, out_rc: 0
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b, out_rc: 1
Tue Dec 11 15:02:01 2018 user.debug adblock-3.5.5-2[7230]: f_list ::: name: overall, mode: final, cnt: 10912, in_rc: 0, out_rc: 0
Tue Dec 11 15:02:01 2018 user.debug adblock-3.5.5-2[7230]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 059c2450db2bec061477fc79e3755923663bfcac6bf98b322774b9f189d1edc8, out_rc: 1
Tue Dec 11 15:02:04 2018 user.debug adblock-3.5.5-2[7230]: f_dnsup::: cache_util: -, cache_rc: -, cache_flush: 1, cache_cnt: 0, rc: 0
Tue Dec 11 15:02:05 2018 user.debug adblock-3.5.5-2[7230]: f_jsnup::: status: enabled, mode: normal mode, cnt: 10912, notify: 0, notify_cnt: 0, notify_pid: -
Tue Dec 11 15:02:05 2018 user.info adblock-3.5.5-2[7230]: blocklist with overall 10912 domains loaded successfully (TP-Link TL-WR1043N/ND v3, OpenWrt 18.06.1 r7258-5eb055306f)
Thank you!
No, the change is unrelated to that counter. Two options that comes in my mind:
-
you've messed up the tld function with a wrong placed insert
-
you've run in an (unrelated) OOM condition on your router (for whatever reason)
Please contact me directly via dev@brenken.org and I'll send you a fixed adblock version in return.
Just in time before Christmas I've pushed adblock release 3.6.0 to master branch. Major change is the new "DNS Query Report", one of the most pihole like feature requests ...
In LuCI it looks like this:
You'll get a new "Top10" reporting and the last 50 requested domains. Whithin the domain list you could directly blacklist or whitelist these domains.
This report is also available via console (only the Top10 reporting part!).
To make use of this new feature, please install package "tcpdump-mini" and start with a fresh adblock config (use the "--force-maintainer" option during opkg install). By default tcpdump will be started as a background task and write the pcap files to "/tmp" (in a loop max. 5 files in chunks of 1 MByte) ... of course, you could configure that to your needs.
BTW, please also reset your luci caches (rm -rf /tmp/luci-*)!
Merry Christmas! 
9 Likes
Hello
@dibdo Congratulations for everything you do!
We have the possibility to exclude certain sites if we know we are interested in accessing them, but they are blocked?
When clicking the whitelist button, that site must arrive in the white list, right?
From what i see does not work.
Does the domain already exist in your whitelist? Only new/not existing domains will be added - just noticed that there was a bug in this function - fix is on the way with this PR (https://github.com/openwrt/luci/pull/2389).
Yes he appears to be in the whitelist, only that it continues to be blocked.
Please refresh/restart adblock processing after adding new domains to white- or blacklist.
Hi @dibdot
I have multi vlan with difference subnet segment on same router. Can I set with multi report interface ?
Interesting statistics, my chromecast seems to be very chatty, in just 1 day with no real usage (I haven't watched anything using Chromecast)
Count 1736 - ChromeCast
yeah, it's a google product ...
How many dns instances did you use? Currently tcpdump only listens on port 53 on "br-lan" interface - the latter one is configurable.
Two dns instances. Dns service listen on eth0.1 & eth0.2, work on port 53. Tks.
Is this the proper way to add the snapshots source? I placed this in the Distribution feeds section.
src/gz openwrt_snapshot https://downloads.openwrt.org/snapshots/packages/x86_64/packages/Packages
Everything works well, thanks for the answer!
Hi please the youtube skip ad is still not removed or not blocked by adblock
Is there any way to block it?