OpenWrt Forum Archive

Topic: adblock package, release 2.x

The content of this topic has been archived between 22 Mar 2018 and 4 May 2018. Unfortunately there are posts – most likely complete pages – missing.

dibdot wrote:

Wait for 1.2.7 ... smile

You're awesome, i'm still on 1.2.6 on the builds i do for people in Romania.

hi

thank you for the great addon!

when i open a blocked domain domain through https chrome tells me the connection got closed.
How can i fix this?

(Last edited by shm0 on 23 Jun 2016, 03:01)

shm0 wrote:

hi

thank you for the great addon!

when i open a blocked domain domain through https chrome tells me the connection got closed.
How can i fix this?

this is the normal behaviour for ad related ssl connections, they can't be redirected to a pixel server cause a https connections requires a valid certificate (i.e. from doubleclick.net). In times of ssl pinning I'm not willing to fake certificates on the fly ... it's evil stuff in my opinion.

So the uhttp instance on port 65535/443 is unnecessary?
Why does http://a.admob.com:443/ give an empty respsonse?

shm0 wrote:

So the uhttp instance on port 65535/443 is unnecessary?
Why does http://a.admob.com:443/ give an empty respsonse?

No, this "ssl" instance has a network timeout of "0" ... so connection will be instantly closed. This is the fastest way to bring down a ssl connection on client side.

(Last edited by dibdot on 23 Jun 2016, 15:10)

Thank you.

Another question :<

Can this work with multiple lan interfaces? (Two networks 192.168.0.X 192.168.1.X)
Especially the "Redirect all DNS queries to the local resolver" option.
One of the networks is only allowed to connect to the internet and to dhcp/dns on the router (but to any router address).

Is this adblock package in anything different then the adblock package included in openwrt?

(Last edited by shm0 on 23 Jun 2016, 17:19)

shm0 wrote:

Thank you.

Another question :<

Can this work with multiple lan interfaces? (Two networks 192.168.0.X 192.168.1.X)
Especially the "Redirect all DNS queries to the local resolver" option.
One of the networks is only allowed to connect to the internet and to dhcp/dns on the router (but to any router address).

Is this adblock package in anything different then the adblock package included in openwrt?

I don't know. Both networks have to use the same dns resolver and the router(uhttpd ip must be reachable from both networks. The force_dns rule is limited to one lan device/interface - if you need more, then disable this in the adblock config and put your own rules in firewall.user.

(Last edited by dibdot on 24 Jun 2016, 15:21)

Is this compatible with Attitude Adjustment? Thanks

@dibdot, 1.3.0 seems to consistently fail for me when downloading source lists.  I have tried restarting the service and also rebooting the device.

Wed Jun 29 18:05:22 2016 user.notice adblock[1368] info : adblock service started due to 'ifup' of 'wan' interface
Wed Jun 29 18:05:22 2016 user.notice adblock[1457] info : domain adblock processing started (1.3.0, r706, 29.06.2016 18:05:22)
Wed Jun 29 18:05:22 2016 user.notice adblock[1457] info : backup/restore will be disabled
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : created volatile firewall rulesets
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : created volatile uhttpd instances
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : => processing source 'adaway'
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info :    no online timestamp
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info :    source download failed, skipped
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : => processing source 'disconnect'
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info :    no online timestamp
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info :    source download failed, skipped
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : => processing source 'yoyo'
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info :    no online timestamp
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info :    source download failed, skipped
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : block lists with overall 0 domains are still valid, no update required
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : firewall statistics (IPv4/IPv6): 0%/0% of all packets in prerouting chain are ad related & blocked
Wed Jun 29 18:05:23 2016 user.notice adblock[1457] info : domain adblock processing finished successfully (1.3.0, r706, 29.06.2016 18:05:23)

Are there any other details that I can provide to help diagnose this issue?

The upgrade from 1.2.8 to 1.3.0 mentioned removing wget, but otherwise no errors with configuration or anything like that.  I always use the option with each new release to start with a fresh configuration file based on each new release.

WildByDesign wrote:

@dibdot, 1.3.0 seems to consistently fail for me when downloading source lists.

Hi, works for me without problems on a tp-link wdr4900 with adblock 1.3.0 and LEDE r817 ..

WildByDesign wrote:

@dibdot, 1.3.0 seems to consistently fail for me when downloading source lists.  I have tried restarting the service and also rebooting the device.

Did you remove wget in your build? There was a gap in the uclient-fetch detection, that was fixed in 1.3.1. Please retest with that version ... at least you should get an error message with missing wget/uclient-fetch.

br
dirk

P.S. If you still receive download errors with 1.3.1, please set "adb_debug" in /etc/init.d/adblock to "1" and restart adblock processing and send me the output to my email - thanks.

(Last edited by dibdot on 30 Jun 2016, 13:49)

dibdot wrote:
WildByDesign wrote:

@dibdot, 1.3.0 seems to consistently fail for me when downloading source lists.  I have tried restarting the service and also rebooting the device.

Did you remove wget in your build? There was a gap in the uclient-fetch detection, that was fixed in 1.3.1. Please retest with that version ... at least you should get an error message with missing wget/uclient-fetch.

br
dirk

P.S. If you still receive download errors with 1.3.1, please set "adb_debug" in /etc/init.d/adblock to "1" and restart adblock processing and send me the output to my email - thanks.

Thank you for the suggestions.  I am running 1.3.1 now for testing.  The wget package was removed by installing 1.3.0, the install had mentioned the removal of wget.

So I assume the failure initially was that there was no uclient-fetch or wget to download the source lists.  So I installed uclient-fetch now after installing 1.3.1 because I assume that 1.3.0 removed wget for a reason.

Unfortunately, uclient-fetch on my system is not working correctly.  I keep getting "runtime error, please check the log!".  I tried a few reboots and still failures.

So I decided to install wget, and successfully downloaded the source lists and adblock was working. One time anyway, it was successful.

Then I realized that since it was not working with uclient-fetch but it was working with wget, I'd remove uclient-fetch since I no longer needed it.

I simply removed uclient-fetch from System - Software section and for whatever reason, removing uclient-fetch has borked things quite a bit.  opkg is no longer working.  adblock is no longer working.  Within LuCI, the System - Software page is completely missing even after reboot.

So at the moment I just have to either re-install LEDE (either re-flash or probably try restore defaults first) and then I will go back to 1.3.1 and I will just stick with wget for now. I'll let you know how it's going once I get this problem sorted out.

WildByDesign wrote:

So at the moment I just have to either re-install LEDE (either re-flash or probably try restore defaults first) and then I will go back to 1.3.1 and I will just stick with wget for now. I'll let you know how it's going once I get this problem sorted out.

Sorry for that. The default uclient-fetch installation (with libustream-polarssl) is not supported, cause polarssl didn't support certain https sites. You need to switch the underlying libustream-library for this to work (check latest documentation on how to do that). For normal use cases 'wget' is still the best selection.

Again, sorry for the troubles!
dirk

P.S. The mentioned debug output is only visible via logread -e "adblock", not in LuCI.

(Last edited by dibdot on 30 Jun 2016, 15:45)

dibdot wrote:

Sorry for that. The default uclient-fetch installation (with libustream-polarssl) is not supported, cause polarssl didn't support certain https sites. You need to switch the underlying libustream-library for this to work (check latest documentation on how to do that). For normal use cases 'wget' is still the best selection.

Again, sorry for the troubles!
dirk

P.S. The mentioned debug output is only visible via logread -e "adblock", not in LuCI.

No worries, you never need to apologize, Dirk.  Thank you for the extra details.  Also, I wanted to mention that your documentation is always top-notch, very much appreciated.

Sometimes the best way for me to learn is to make mistakes and learn the hard way, so I always try to turn everything into a learning moment.  I have now flashed with the latest trunk of LEDE from today along with adblock 1.3.1 and the latest relevant packages.  I will continue to use wget for now because it requires a bit less understanding for me and as long as you continue to support wget as well.

Anyway, everything is working flawlessly as per usual.  Keep up the great work!

dibdot wrote:

The default uclient-fetch installation (with libustream-polarssl) is not supported, cause polarssl didn't support certain https sites. You need to switch the underlying libustream-library for this to work (check latest documentation on how to do that). For normal use cases 'wget' is still the best selection.

For whom was successfully in switching "the underlying libustream-library" - how is that better in space saving compared to the regular wget (+libpcre) installation? Thanks!

libopenssl (which is required by wget to handle https links) is approx 740Kb, libmbedtls and its libustream (one of the options for ufetch-client) are only 166Kb combined.

Replying to the post below -- it's still 'cheaper' to use libpolarssl for VPN in addition to libmbedtls.

(Last edited by stangri on 7 Jul 2016, 01:31)

Likewise libopenssl may come 'for free', if you already need it for other purposes (such as VPN).

If i already need libopenssl on a 8 MB flash firmware then how do i switch from libpolarssl to only libopenssl to save space? Thanks!

dape wrote:

If i already need libopenssl on a 8 MB flash firmware then how do i switch from libpolarssl to only libopenssl to save space? Thanks!

You need to elaborate on your current setup/requirements and may be better off starting a separate thread on that.

Thanks dibdot for this ad blocking solution. I have been using this for the past couple of weeks and its the best router ad blocking solution. Thanks so much

Hi dibdot

First of all, thank you for writing this package. The recent releases solved many issues (eg. mwan3 compatibility and the need to use wget with heavy openssl libraries) and now it seems like an almost perfect solution.
I have a couple of ideas for better procd/hotplug integration. I can contribute by writing some patches, but before doing that, I'd like to discuss it.

The most urgent issue is running adblock-update from init.d script. It's not as great as it may look at first glance. Since network connections are managed asynchronously in background, fetching files during boot is unreliable. In my case, network comes up when update script is already executing, so I end up with partially successful fetch. Also, pid file lock prevents running update via hotplug, when WAN interface actually comes up. Moreover, update script delays system boot sequence.
What I suggest is to separate setup logic (creating uhttpd instance and adding iptables rules, which belong to init.d)  from actual update script (which should be triggered by hotplug events). Ideally, init.d script would clean up dynamic uhttpd instance and iptables rules on stop (but it's not that important). There might be a point in restoring backup on boot sequence but it's a bit iffy because boot scripts should be fast and lightweight.

There's a minor issue in hotplug triggered update. It should be possible to disable it via config option. Moreover, I'd add an option to override auto-detected wan interface by providing a config-defined list of interfaces (remember mwan3 setup).

kr

marcin1j wrote:

Hi dibdot

First of all, thank you for writing this package. The recent releases solved many issues (eg. mwan3 compatibility and the need to use wget with heavy openssl libraries) and now it seems like an almost perfect solution.
I have a couple of ideas for better procd/hotplug integration.

Hi, thanks for your feedback.

PROCD support is already on my todo list, but at this stage it's poorly documented and I see no benefit for scripts like adblock which didn't daemonize.

The adblock update process is only triggered by hotplug event (whenever a WAN interface comes up), not by init.d (empty boot function in adblock.init). It's definitely needless to put an additional "/etc/init.d/adblock start" in rc.local or any other startup script.

The pid file check/lock only avoids multiple/concurrent starts of the same script - nothing more.

A split of 'setup' and 'update' logic makes no sense, cause all system changes made by adblock are volatile and need to be checked during runtime.

marcin1j wrote:

Ideally, init.d script would clean up dynamic uhttpd instance and iptables rules on stop (but it's not that important). There might be a point in restoring backup on boot sequence but it's a bit iffy because boot scripts should be fast and lightweight.

That's already implemented.

marcin1j wrote:

Moreover, I'd add an option to override auto-detected wan interface by providing a config-defined list of interfaces

Interesting point, I'll put it on my todo list.
I've released 1.4.0 yesterday and it should quite stable now ... more to come after my holidays - have a good time!

(Last edited by dibdot on 15 Jul 2016, 07:40)

Dilbot

I'm afraid you misunderstood me.

I didn't mention there's a need to run init.d script from rc.local. Obviously init script is started automatically.

The problem is that init script runs adblock-update during boot sequence (which involves running init.d scripts with start parameter). Update script tries to fetch blacklists from network (not only restore from backup, as you suggested) regardless of wan interface status and that's unreliable because network interfaces are started asynchronously.

Due to pid file lock, update script is not started by a hotplug event when wan interface actually comes up because there's another instance already running (started from init.d). This doesn't mean that local file is a bad thing. The point is that running update from init.d leads to a race condition.

So to recap:

  1. Fetching blacklists from init.d scripts is unreliable, leads to a race condition and should be avoided,

  2. Update script should be triggered by a hotplug event only. There should be an option to override automatic wan interface detection and to disable this trigger altogether,

  3. You are right that all changes made by adblock are volatile. However, uhttpd and iptables setup is executed only once. Therefore it doesn't belong to update/helper script and it makes sense to separate it. You benefit from a cleaner code and moving ubus stuff away. And there's no harm in starting volatile uhttpd instance before dnsmasq rules are generated,

  4. Likewise, the wan interface detection logic makes sense only in hotplug script,

  5. You're wrong thinking that procd has no benefit for scripts like adblock which didn't daemonize. Look at the wiki page https://wiki.openwrt.org/inbox/procd-in … ce_changes. That's another reason to move ubus-related code to init.d script.

I didn't mean to criticize anything because this is a really awesome solution and can be made even better (hence my first post). Still, there's at least one issue which can't be ignored and needs addressing.
I agree that procd is poorly documented and that's one of the reasons I offered my help.

kr

marcin1j wrote:

Due to pid file lock, update script is not started by a hotplug event when wan interface actually comes up because there's another instance already running (started from init.d). This doesn't mean that local file is a bad thing. The point is that running update from init.d leads to a race condition.

Again, please check the real package sources here ...
The init script (adblock.init) makes nothing on boot:

[...]
boot()
{
    return 0
}
[...]

Maybe you've installed another script or a mixture of different scripts ...the adblock package described here is only triggered by hotplug!

Hi
My custom blacklist seems broken:

Here my list:

choice.microsoft.com
choice.microsoft.com.nsatc.net
df.telemetry.microsoft.com
diagnostics.support.microsoft.com
feedback.microsoft-hohm.com
feedback.search.microsoft.com
feedback.windows.com
oca.telemetry.microsoft.com
oca.telemetry.microsoft.com.nsatc.net
onesettings-bn2.metron.live.com.nsatc.net
onesettings-cy2.metron.live.com.nsatc.net
onesettings-db5.metron.live.com.nsatc.net
onesettings-hk2.metron.live.com.nsatc.net
reports.wes.df.telemetry.microsoft.com
services.wes.df.telemetry.microsoft.com
settings-sandbox.data.microsoft.com
settings-win.data.microsoft.com
settings.data.glbdns2.microsoft.com
sqm.df.telemetry.microsoft.com
sqm.telemetry.microsoft.com
sqm.telemetry.microsoft.com.nsatc.net
statsfe1.ws.microsoft.com
statsfe2.update.microsoft.com.akadns.net
statsfe2.ws.microsoft.com
survey.watson.microsoft.com
telecommand.telemetry.microsoft.com
telecommand.telemetry.microsoft.com.nsatc.net
telemetry.appex.bing.net
telemetry.microsoft.com
telemetry.urs.microsoft.com
v10.vortex-win.data.metron.live.com.nsatc.net
v10.vortex-win.data.microsoft.com
vortex-bn2.metron.live.com.nsatc.net
vortex-cy2.metron.live.com.nsatc.net
vortex-db5.metron.live.com.nsatc.net
vortex-hk2.metron.live.com.nsatc.net
vortex-sandbox.data.microsoft.com
vortex-win.data.metron.live.com.nsatc.net
vortex-win.data.microsoft.com
vortex.data.glbdns2.microsoft.com
vortex.data.metron.live.com.nsatc.net
vortex.data.microsoft.com
watson.live.com
watson.microsoft.com
watson.ppe.telemetry.microsoft.com
watson.telemetry.microsoft.com
watson.telemetry.microsoft.com.nsatc.net
wes.df.telemetry.microsoft.comnet
telemetry.battle.net

This is the generated output:

address=/Binary file /tmp/tmp.pKocpG matches/198.18.0.1
address=/choice.microsoft.com.nsatc.net/198.18.0.1
address=/choice.microsoft.com/198.18.0.1
address=/df.telemetry.microsoft.com/198.18.0.1
address=/diagnostics.support.microsoft.com/198.18.0.1
address=/feedback.microsoft-hohm.com/198.18.0.1
address=/feedback.search.microsoft.com/198.18.0.1
address=/feedback.windows.com/198.18.0.1
address=/oca.telemetry.microsoft.com.nsatc.net/198.18.0.1
address=/oca.telemetry.microsoft.com/198.18.0.1
address=/onesettings-bn2.metron.live.com.nsatc.net/198.18.0.1
address=/onesettings-cy2.metron.live.com.nsatc.net/198.18.0.1
address=/onesettings-db5.metron.live.com.nsatc.net/198.18.0.1
address=/onesettings-hk2.metron.live.com.nsatc.net/198.18.0.1
address=/reports.wes.df.telemetry.microsoft.com/198.18.0.1
address=/services.wes.df.telemetry.microsoft.com/198.18.0.1
address=/settings-sandbox.data.microsoft.com/198.18.0.1
address=/settings-win.data.microsoft.com/198.18.0.1
address=/settings.data.glbdns2.microsoft.com/198.18.0.1
address=/sqm.df.telemetry.microsoft.com/198.18.0.1
address=/sqm.telemetry.microsoft.com.nsatc.net/198.18.0.1
address=/sqm.telemetry.microsoft.com/198.18.0.1
address=/statsfe1.ws.microsoft.com/198.18.0.1
address=/statsfe2.update.microsoft.com.akadns.net/198.18.0.1
address=/statsfe2.ws.microsoft.com/198.18.0.1
address=/survey.watson.microsoft.com/198.18.0.1
address=/telecommand.telemetry.microsoft.com/198.18.0.1

thank you.

//Edit
Sorry. There was some weird encoding in my source list.

(Last edited by shm0 on 4 Aug 2016, 00:08)