Adblock support thread


#610

Currently I'm working on adblock 3.6.x - major change is a new reporting engine (based on pcap files generated by tcpdump). Currently the report includes the following figures - do we need more?
adblock

Thanks for your feedback!


#611

@dibdot This upcoming development for 3.6.x is absolutely fantastic and I am looking forward to it. It is thorough, yet well organized and concise. Perfection!

I honestly cannot think of anything to add since it seems to cover the most relevant information already. Keep up the great work.


#612

Hi dibdot!
I need your advice: adblock is counting overall twice.

My log:

Sun Dec  9 11:28:15 2018 user.info adblock-3.5.5-2[31809]: adblock instance started ::: action: reload, priority: 0, pid: 31809
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_jsnup::: status: running, mode: normal mode, cnt: 35742, notify: 0, notify_cnt: 0, notify_pid: -
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_uci  ::: config: dhcp, change:
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_uci  ::: config: firewall, change:
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: dns: dnsmasq, fetch_util: /bin/uclient-fetch (libustream-ssl), backup: 0, backup_mode: 0, dns_jail: 0, force_srt: 0, force_dns: 1, mem_total: 60, mem_free: 23, max_queue: 16
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: cameleon_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: malware_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: disconnect_malware, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: disconnect_ad_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: zeus_pihole_def, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_main ::: name: blacklist, enabled: 1
Sun Dec  9 11:28:15 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: blacklist, mode: download, cnt: 10, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:16 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.blacklist, cnt: 10, cnt_srt: 10, cnt_tld: 10
Sun Dec  9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.zeus_pihole_def, cnt: 381, cnt_srt: 381, cnt_tld: 380
Sun Dec  9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: disconnect_malware, mode: download, cnt: 2831, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:18 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: disconnect_ad_pihole_def, mode: download, cnt: 2703, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: 2668
Sun Dec  9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.disconnect_malware, cnt: 2831, cnt_srt: 2831, cnt_tld: 2790
Sun Dec  9 11:28:24 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: cameleon_pihole_def, mode: download, cnt: 20567, in_rc: 0, out_rc: 0
Sun Dec  9 11:28:34 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.cameleon_pihole_def, cnt: 20567, cnt_srt: 20567, cnt_tld: 16409
Sun Dec  9 11:28:54 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: malware_pihole_def, mode: download, cnt: 26850, in_rc: 0, out_rc: 0
Sun Dec  9 11:29:06 2018 user.debug adblock-3.5.5-2[31809]: f_tld  ::: source: /tmp/tmp.kjDhnf/tmp.BdkPcG.malware_pihole_def, cnt: 26850, cnt_srt: 26850, cnt_tld: 26362
Sun Dec  9 11:29:07 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: overall, mode: merge, cnt: 97238, in_rc: 0, out_rc: 0
Sun Dec  9 11:29:07 2018 user.debug adblock-3.5.5-2[31809]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 68da62929ec9c976afb1add6172756da231c369e6a85446dd48dcd848c1163f5, out_rc: 1
Sun Dec  9 11:29:12 2018 user.debug adblock-3.5.5-2[31809]: f_list ::: name: overall, mode: final, cnt: 97238, in_rc: 0, out_rc: 0

Summary of all the "cnt_tld" is exactly the half of the "overall cnt"
How can that be?

Thank you!


#613

Problem (?) nr. 2:

if i only load 1 list, it counts more at the end as it is in the list!
For example malware list :
entries: 27.043
adblock count: 52.724
See log below:

Mon Dec 10 22:02:12 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: malware_pihole_def, mode: download, cnt: 26850, in_rc: 0, out_rc: 0
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_tld  ::: source: /tmp/tmp.PEafGi/tmp.fJEfbN.malware_pihole_def, cnt: 26850, cnt_srt: 26850, cnt_tld: 26362
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: overall, mode: merge, cnt: 52724, in_rc: 0, out_rc: 0
Mon Dec 10 22:02:24 2018 user.debug adblock-3.5.5-2[6478]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, out_rc: 1
Mon Dec 10 22:02:27 2018 user.debug adblock-3.5.5-2[6478]: f_list ::: name: overall, mode: final, cnt: 52724, in_rc: 0, out_rc: 0

Thank you!


#614

Clearly a bug in the f_tld() function, seen only on low memory devices - will be fixed in 3.6.0. In the meantime you could change it in /usr/bin/adblock.sh on your own:

old: line 577 ff.

[...]
			if [ ${?} -eq 0 ]
				then
					cnt_tld="$(wc -l 2>/dev/null < "${source}")"
				else
					mv -f "${temp}" > "${source}"
				fi
fi
[...]

new, added missing "rm" command :

[...]
			if [ ${?} -eq 0 ]
				then
					cnt_tld="$(wc -l 2>/dev/null < "${source}")"
                                        rm -f "${temp}"   # that's the bugfix!
				else
					mv -f "${temp}" > "${source}"
				fi
fi
[...]

Sorry for the mess!


#615

Thanx dibdot!
It works , but it seems tld is not working anymore?
with bugfix: "cnt_tld: -"
without bugfix: "cnt_tld: 2668"

See logs:

with bugfix

Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_tld  ::: source: /tmp/tmp.CADBOK/tmp.LAAJnc.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: -
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_tld  ::: source: /tmp/tmp.CADBOK/tmp.LAAJnc.disconnect_malware, cnt: 2828, cnt_srt: 2828, cnt_tld: -
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_list ::: name: overall, mode: merge, cnt: 5456, in_rc: 0, out_rc: 0
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b, out_rc: 1
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_list ::: name: overall, mode: final, cnt: 5456, in_rc: 0, out_rc: 0
Tue Dec 11 14:55:07 2018 user.debug adblock-3.5.5-2[6440]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: e26d71fcb03cd82d6058a9ef1479665f685da05e0b358b27ec18ba780f61fdb0, out_rc: 1
Tue Dec 11 14:55:11 2018 user.debug adblock-3.5.5-2[6440]: f_dnsup::: cache_util: -, cache_rc: -, cache_flush: 1, cache_cnt: 0, rc: 0
Tue Dec 11 14:55:11 2018 user.debug adblock-3.5.5-2[6440]: f_jsnup::: status: enabled, mode: normal mode, cnt: 5456, notify: 0, notify_cnt: 0, notify_pid: -
Tue Dec 11 14:55:11 2018 user.info adblock-3.5.5-2[6440]: blocklist with overall 5456 domains loaded successfully (TP-Link TL-WR1043N/ND v3, OpenWrt 18.06.1 r7258-5eb055306f)

without bugfix

Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_tld  ::: source: /tmp/tmp.FOEMBh/tmp.kpLFCK.disconnect_ad_pihole_def, cnt: 2703, cnt_srt: 2703, cnt_tld: 2668
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_tld  ::: source: /tmp/tmp.FOEMBh/tmp.kpLFCK.disconnect_malware, cnt: 2828, cnt_srt: 2828, cnt_tld: 2788
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_list ::: name: overall, mode: merge, cnt: 10912, in_rc: 0, out_rc: 0
Tue Dec 11 15:02:00 2018 user.debug adblock-3.5.5-2[7230]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b, out_rc: 1
Tue Dec 11 15:02:01 2018 user.debug adblock-3.5.5-2[7230]: f_list ::: name: overall, mode: final, cnt: 10912, in_rc: 0, out_rc: 0
Tue Dec 11 15:02:01 2018 user.debug adblock-3.5.5-2[7230]: f_hash ::: hash_util: /usr/bin/sha256sum, hash: 059c2450db2bec061477fc79e3755923663bfcac6bf98b322774b9f189d1edc8, out_rc: 1
Tue Dec 11 15:02:04 2018 user.debug adblock-3.5.5-2[7230]: f_dnsup::: cache_util: -, cache_rc: -, cache_flush: 1, cache_cnt: 0, rc: 0
Tue Dec 11 15:02:05 2018 user.debug adblock-3.5.5-2[7230]: f_jsnup::: status: enabled, mode: normal mode, cnt: 10912, notify: 0, notify_cnt: 0, notify_pid: -
Tue Dec 11 15:02:05 2018 user.info adblock-3.5.5-2[7230]: blocklist with overall 10912 domains loaded successfully (TP-Link TL-WR1043N/ND v3, OpenWrt 18.06.1 r7258-5eb055306f)

Thank you!


#616

No, the change is unrelated to that counter. Two options that comes in my mind:

  • you've messed up the tld function with a wrong placed insert :wink:

  • you've run in an (unrelated) OOM condition on your router (for whatever reason)

Please contact me directly via dev@brenken.org and I'll send you a fixed adblock version in return.


#617

Just in time before Christmas I've pushed adblock release 3.6.0 to master branch. Major change is the new "DNS Query Report", one of the most pihole like feature requests ...:wink:

In LuCI it looks like this:
dns_report_1
dns_report2

You'll get a new "Top10" reporting and the last 50 requested domains. Whithin the domain list you could directly blacklist or whitelist these domains.

This report is also available via console (only the Top10 reporting part!).

To make use of this new feature, please install package "tcpdump-mini" and start with a fresh adblock config (use the "--force-maintainer" option during opkg install). By default tcpdump will be started as a background task and write the pcap files to "/tmp" (in a loop max. 5 files in chunks of 1 MByte) ... of course, you could configure that to your needs.

BTW, please also reset your luci caches (rm -rf /tmp/luci-*)!

Merry Christmas! :slight_smile:


Davidc502- wrt1200ac wrt1900acx wrt3200acm wrt32x builds
#618

Hello
@dibdo Congratulations for everything you do!
We have the possibility to exclude certain sites if we know we are interested in accessing them, but they are blocked?
When clicking the whitelist button, that site must arrive in the white list, right?
From what i see does not work.
3


#619

Does the domain already exist in your whitelist? Only new/not existing domains will be added - just noticed that there was a bug in this function - fix is on the way with this PR (https://github.com/openwrt/luci/pull/2389).


#620

Yes he appears to be in the whitelist, only that it continues to be blocked.3


#621

Please refresh/restart adblock processing after adding new domains to white- or blacklist.


#622

Hi @dibdot
I have multi vlan with difference subnet segment on same router. Can I set with multi report interface ?


#623

Interesting statistics, my chromecast seems to be very chatty, in just 1 day with no real usage (I haven't watched anything using Chromecast)
Count 1736 - ChromeCast


#624

yeah, it's a google product ... :wink:


#625

How many dns instances did you use? Currently tcpdump only listens on port 53 on "br-lan" interface - the latter one is configurable.


#626

Two dns instances. Dns service listen on eth0.1 & eth0.2, work on port 53. Tks.


#627

Is this the proper way to add the snapshots source? I placed this in the Distribution feeds section.

src/gz openwrt_snapshot https://downloads.openwrt.org/snapshots/packages/x86_64/packages/Packages


#628

Everything works well, thanks for the answer!:beers:


#629

Hi please the youtube skip ad is still not removed or not blocked by adblock

Is there any way to block it?