Appreciate the feedback.
The sed runs in the same process as the wget which downloads file. I believe the error is actually that wget cannot complete writing to file despite returning 0.
I'll try to flash 17.01.5 which I've experienced this problem again to troubleshoot.
I also ran into the issue with dnsmasq barfing on special encodings over the weekend.
root@OpenWrt:~# cat /etc/banner
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 18.06.0-rc2, r7141-e4d0ee5af5
-----------------------------------------------------
root@OpenWrt:~# cat /proc/version
Linux version 4.14.54 (buildbot@builds-03.infra.lede-project.org) (gcc version 7.3.0 (OpenWrt GCC 7.3.0 r7101-a63e38b)) #0 SMP Sat Jul 14 11:10:08 2018
root@OpenWrt:~# opkg info dnsmasq
Package: dnsmasq
Version: 2.80test3-1
Depends: libc, libubus
Status: install user installed
Section: net
Architecture: arm_cortex-a9_vfpv3
Size: 110609
Filename: dnsmasq_2.80test3-1_arm_cortex-a9_vfpv3.ipk
Conffiles:
/etc/config/dhcp e90f7b32e9c4e8f526a435607f792aa0439dd69fabfe1038eed5b27efc616be0
/etc/dnsmasq.conf 1e6ab19c1ae5e70d609ac7b6246541d52042e4dee1892f825266507ef52d7dfd
Description: It is intended to provide coupled DNS and DHCP service to a LAN.
Installed-Time: 1532825827
Do you know which line/domain name or at least what was the source of the blocklist?
We're clearly not talking about the main process, but the downloading and processing of upstream lists using wget / sed, which your code backgrounds:
process_url "$hf" 'domains' 'allowed' &
Guess that wasn't clear: the stock dnsmasq (or dnsmasq-full) on the latest stable 17.01.5 or 18.06.0. So pretty much "everywhere".
That suggests perhaps your own environment doesn't represent most users'. The last line crashes dnsmasq on multiple routers (ar71xx and MIPS malta).
If your simple-adblock domain file still includes the line local=/ɢoogle.com/ then it is not filtering out extended characters safely.
Understand that we do want upstream block lists to protect against phishing and homographs attacks, but they need to be encoded properly. I believe finding upstream entries like
local=/www.turkishạirlines.com/
local=/ɢoogle.com/
was simply a mistake, but one that highlights a security hole. Indeed, in the same blocklist I did find a correctly encoded version of one line:
local=/www.xn--turkishirlines-1p8g.com/
which shows there's upstream awareness of the issue.
Cheers!
@stangri: it's a limitation of standard dnsmasq compile which comes without IDN ("no-IDN") support, e.g.:
Tue Jul 31 22:40:49 2018 daemon.info dnsmasq[2847]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-ID loop-detect inotify dumpfile
For 18.06 I wouldn''t care cause dnsmasq just prints an error and ignore these IDN entries in blocklists. Of cause, it's more serious for older dnsmasq releases in 17.01 ...
@guidosarducci: note that regular adblock 3.4.3 (in 17.01 tree) ignore these IDN domains, later releases support IDN and add these domains to the blocklist, too.
Weird, I've had the same list on 17.01.4 and dnsmsasq-full which is also compiled with no-IDN didn't have any problems with it.
No, it is not filtering out non-single-byte characters, but there are filters for other potential problems. Thanks for bringing it up, I don't know why it hadn't been a problem for my installs, I'll have IDN-characters in domain names filtered in the next version.
I see what you mean now, like I said there was a run_in_background setting in some of the previous versions which didn't affect download processes at all and at first I thought you meant that setting.
Fixed in simple-adblock 1.6.3-8 available from my repo. I'll submit a PR once I figure out what's going on with failed wget downloads.
@guidosarducci, @philthetank, @jaquer, @bigwave -- both wget/sed and IDN issues fixed in simple-adblock 1.6.3-9. I would appreciate testing on different devices.
No issues with 1.6.3-9 after two days of use.
It's true that 18.06 dnsmasq prints an error message instead of crashing, but it fails to start up and procd eventually gives up, so you still need to be careful.
How will you supporting the IDN domains? I'm guessing using the idn package for domains hitting your filter?
Pardon the delay -- I did have a chance to try out the changes but got sidetracked later, so let me address your two changes separately:
Filtering Improperly Encoded IDN Domains:
Yay, it now works! I see you rightly added character class filtering (e.g. [:alnum:]) to fix this. Be aware their behavior depends on locale, which you need to control for the solution to be robust. You should consider adding e.g. LC_ALL="C" to your script, or the equivalent.
Download Failures - process_url(), wget
It seemed to work better at first, but after several restart or reload attempts I could still trigger download errors. As both @dibdot and I mentioned before, it does seem like there's an underlying race or timing issue, which your change (below) maybe only tickled rather than fixed.
@@ -201,3 +203,3 @@
done
- if ! touch "$R_TMP" || ! $dl_command "$1" -O "$R_TMP" 2>/dev/null; then
+ if ! $dl_command "$1" -O "$R_TMP" 2>/dev/null || [ ! -s "$R_TMP" ]; then
output 2 "[DL] $type $label $__FAIL__\n"
I also tried disabling backgrounding the calls to process_url(), and that has worked reliably well, so perhaps consider adding an option to do this?
I also noticed the DNS firewall rule isn't being removed properly, which is a big problem for anyone trying to manage their own VPN/DNS/redirect scheme. You can see for yourself: disable DNS redirect in the GUI, or even stop the service, and the rule persists in both ubus and iptables (and if you investigate further you'll find other underlying serious issues). Aside from that, can I suggest a UCI default of '0' for force_dns, which is less likely to cause problems "out of the box"?
Cheers!
can't confirm any startup failure: current dnsmasq only prints an error/warning but start up finished successfully, e.g.:
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: read /etc/hosts - 4 addresses
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq-dhcp[2207]: read /etc/ethers - 0 addresses
Thu Aug 9 12:25:27 2018 daemon.err dnsmasq[2207]: error at line 26615 of /tmp/adb_list.overall
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zztxdown.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zzkrxder.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zzhgpbovlhinj.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zzhengre.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zzgqqdmnrhhals.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zzdnkvjaikjth.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zz2017.ru
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using local addresses only for domain zyxwvutsrqponmlkjihgfedcbazyxwvutsrqponmlkjihgfedcbazyxwvutsrqp.de
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using standard nameservers for domain here.com
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using nameserver 172.18.0.1#53
Thu Aug 9 12:25:27 2018 daemon.info dnsmasq[2207]: using 26612 more local addresses
Valid domain names are not limited to ascii encoding, therefore I simply accept Unicode characters in the blocklist sources as well. Other dns backends like unbound or bind fully support IDN, too - IMHO it's mainly a dnsmasq limitation.
Thank you! Added.
That's not the change which got rid of the failed downloads.
Both fixed in 1.6.4-1. The new setting is parallel_downloads and it's 1 by default.
I strongly disagree on this one, considering the number of mobile/media devices with hardcoded DNS servers and misconfigured LAN clients I believe force_dns should be 1 by default.
Hmm, what do think the actual problem was then?
Great! That will also help with CPU usage. BTW, I can see the new setting in simple-adblock, but not yet in luci-app-simple-adblock.
On service stop, the firewall rule is now removed. Thanks for that! Out of curiosity, why do you leave the orphaned ubus instance after stopping? It's still there even after uninstalling the package, which was part of the reason the FW rule persisted.
The issue remains of not being able to change the DNS (or any other) setting from the GUI. The problem lies in your init script reload definition, which effectively breaks LUCI/UCI config management. I didn't have time to better explain before, so let me now...
The normal function of the init script reload is to re-read changed service config settings:
# /etc/init.d/cron
Syntax: /etc/init.d/cron [command]
Available commands:
reload Reload configuration files (or restart if service does not implement reload)
This is core functionality needed e.g. by LUCI, UCI, ucitrack, procd, etc, but simple-adblock redefines the reload to be "download blacklists" while ignoring config changes. A cleaner approach would be to define an explicit init script download function.
Please try this for yourself a few times as requested: toggle DNS redirect in the GUI, press "Save & Apply", and then check UCI, ubus and iptables. You'll see the problem right away.
While I'm still relying on reload to redownload lists and restart to restart the service without redownloading lists, with the simple-adblock 1.6.4-2 and luci-app-simple-adblock 18 proper settings take effect after save and apply.
If you only change something like force_dns in luci app, it will just quickly restart the service to insert/remove firewall rule. If you only change verbosity it won't even restart the service. If you change anything which has to do with lists or downloads, it will redownload everything and will also absorb any other setting changed.
I did an opkg update from 18.06 and got 1.6.3-5. I'm currently getting critical DNS errors which disable any internet access until I restart the device and disable the service.
Sun Aug 12 09:26:54 2018 user.notice simple-adblock [7949]: Reloading simple-adblock 1.6.3-5...
Sun Aug 12 09:26:54 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: ssl.bblck.me [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: ransomwaretracker.abuse.ch [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: raw.githubusercontent.com [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: zeustracker.abuse.ch [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: adaway.org [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: s3.amazonaws.com [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: www.malwaredomainlist.com [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: s3.amazonaws.com [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: s3.amazonaws.com [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: pgl.yoyo.org [✓]
Sun Aug 12 09:26:55 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: dshield.org [✓]
Sun Aug 12 09:26:56 2018 user.notice simple-adblock [7949]: [DL] Blocked Domains: mirror1.malwaredomains.com [✓]
Sun Aug 12 09:26:57 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: www.mvps.org [✓]
Sun Aug 12 09:26:57 2018 user.notice simple-adblock [7949]: [DL] Blocked Hosts: someonewhocares.org [✓]
Sun Aug 12 09:26:58 2018 user.notice simple-adblock [7949]: Sorting combined list [✓]
Sun Aug 12 09:27:03 2018 user.notice simple-adblock [7949]: Optimizing combined list [✓]
Sun Aug 12 09:27:03 2018 user.notice simple-adblock [7949]: Whitelisting domains [✓]
Sun Aug 12 09:27:03 2018 user.notice simple-adblock [7949]: Formatting merged file [✓]
Sun Aug 12 09:27:03 2018 user.notice simple-adblock [7949]: Creating dnsmasq config [✓]
Sun Aug 12 09:27:03 2018 user.notice simple-adblock [7949]: Removing temporary files [✓]
Sun Aug 12 09:27:03 2018 daemon.info dnsmasq[5952]: exiting on receipt of SIGTERM
Sun Aug 12 09:27:04 2018 user.notice dnsmasq: DNS rebinding protection is active, will discard upstream RFC1918 responses!
Sun Aug 12 09:27:04 2018 user.notice dnsmasq: Allowing 127.0.0.0/8 responses
Sun Aug 12 09:27:07 2018 user.notice simple-adblock [7949]: Restarting dnsmasq [✓]
Sun Aug 12 09:27:07 2018 user.notice simple-adblock [7949]: service is blocking 46952 domains [✓]
Sun Aug 12 09:27:07 2018 daemon.crit dnsmasq[8594]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:27:07 2018 daemon.crit dnsmasq[8594]: FAILED to start up
Sun Aug 12 09:27:12 2018 daemon.crit dnsmasq[8784]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:27:12 2018 daemon.crit dnsmasq[8784]: FAILED to start up
Sun Aug 12 09:27:17 2018 daemon.crit dnsmasq[8797]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:27:17 2018 daemon.crit dnsmasq[8797]: FAILED to start up
Sun Aug 12 09:27:22 2018 daemon.crit dnsmasq[8805]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:27:22 2018 daemon.crit dnsmasq[8805]: FAILED to start up
Sun Aug 12 09:27:27 2018 daemon.crit dnsmasq[8811]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:27:27 2018 daemon.crit dnsmasq[8811]: FAILED to start up
Sun Aug 12 09:27:33 2018 daemon.crit dnsmasq[8819]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:27:33 2018 daemon.crit dnsmasq[8819]: FAILED to start up
Sun Aug 12 09:27:33 2018 daemon.info procd: Instance dnsmasq::cfg01411c s in a crash loop 6 crashes, 0 seconds since last crash
Sun Aug 12 09:28:55 2018 user.notice dnsmasq: DNS rebinding protection is active, will discard upstream RFC1918 responses!
Sun Aug 12 09:28:55 2018 user.notice dnsmasq: Allowing 127.0.0.0/8 responses
Sun Aug 12 09:28:59 2018 daemon.crit dnsmasq[9056]: error at line 45519 of /tmp/dnsmasq.d/simple-adblock
Sun Aug 12 09:28:59 2018 daemon.crit dnsmasq[9056]: FAILED to start up
Thanks for the report. Can you post which lists are you using and if you don't mind starting the service again the output of: sed -n '45518,45520p' < /tmp/dnsmasq.d/simple-adblock?
Well, that's weird. It stopped happening. Everything works fine now. Now I just get the 36198 domains blocked with error: 'Download error' error
In any case I'm using the default lists
http://mirror1.malwaredomains.com/files/justdomains
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt
https://ssl.bblck.me/blacklists/domain-list.txt
http://dshield.org/feeds/suspiciousdomains_High.txt
I also tried your sed command but nothing happened.
Try to run /etc/init.d/simple-adblock restart until all the lists download successfully. If you set verbosity to 2, you will see which specific list is having issues.
That's because the current combined block-list only has 36k lines, the problematic line in your previous list was 45519, which isn't present in the current list.
Gotcha, will do. How do I set verbosity to 2?
uci get simple-adblock.config.verbosity
uci set simple-adblock.config.verbosity=2; uci commit