OpenWrt Forum Archive

Topic: monitor outgoing Internet connections (OpenWRT WDR3600) - in progress

The content of this topic has been archived on 20 Apr 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

Hi folks!

Wanted to share my experience and get some feedback and advice. My objective is to log and analyze/visualize all Internet connections of my local network to get visibility of internet usage by individual devices and users.

Setup: cable modem (100MB down, 6MB up) – with Fritzbox 7390 for WLAN connectivity. Amazingly many networked devices (iPad, Chromebook, smartphone, Internet radio, Skype appliance, home pc, work laptop, blue ray player, little NAS, …).

Got me a TP-Link WDR3600 for 44€ at my local Mediamarkt and installed OpenWRT – worked flawless. Nice.
Now, how do I capture/log all Internet traffic?


Option 1 – transparent proxy with TinyProxy

Installation worked fine (opkg install tinyproxy luci-app-tinyproxy). Before changing routing for transparent proxy I manually changed client to use proxy. Web performance was noticable slower. And it proxies HTTP only, for HTTPS I have to install my own certificates on all devices? Is there a simpler way? And honestly, routing on OpenWRT is a piece of work. By default, the logs show the URL without the hostname.

Internet reference (http://www.farville.com/home-networks-a … itor-kids/) requires another server for log processing (yes, I already got a Raspberry lying around, but no…).


Option 2 – TCPDUMP

If setting up a transparent proxy and logging is so complex, why not listening to the traffic directly? Wireshark/Tshark is not available on OpenWRT, but TCPdump is.

Installing TCPdump is very easy: “opkg install tcpdump”.
Then use “tcpdump –D” to show all network interfaces, or “tcpdump –q –tttt” to show all connections with timestamp.

tcpdump -q -i br-lan –tttt
2014-11-14 16:51:16.282484 IP iPadwwwi2daycom.lan.53516 > ea-in-f94.1e100.net.https: tcp 0
2014-11-14 16:51:16.282818 IP iPadwwwi2daycom.lan.53516 > ea-in-f94.1e100.net.https: tcp 0
But holy moly, it logs every fart in the air. Every Ping, every ARP, everything. Piping the simplified output (without the actual packet data) to file generates a few MB for a few minutes. And without some magic it only shows the host, not the visited URL.
tcpdump –q –i br-lan –tttt > tcpdump.log
2014-11-14 19:11:04.356741 IP iPadwwwi2daycom.lan.54532 > 195.10.18.43.https: tcp 0
2014-11-14 19:11:04.357311 IP iPadwwwi2daycom.lan.54533 > 195.10.18.43.https: tcp 0
2014-11-14 19:11:04.400291 IP 195.10.18.43.https > iPadwwwi2daycom.lan.54532: tcp 0

I guess I have to invest some time in setting up proper filtering.

But here is the showstopper: tcpdump does not run in background. After you kill the SSH session, TCPdump will stop too (even if you run with “&” parameter). Did not find any successful recipe on the Internet (tried “screen” too).


Option 3  - DNS logging

Another direction is to monitor all DNS requests coming from the local network. OpenDNS offers some services/functionality here. Of course, this will only display the Internet host and not the full URL either.
Apparently OpenWRT ships with DNSMASQ for DHCP and DNS. The manpage at http://www.thekelleys.org.uk/dnsmasq/do … q-man.html shows us how to manipulate the DNS answers for a short Time-To-Live – so that every website visit triggers a new DNS query (even if the client should remember the correct IP from a minute ago).

First install text editor nano (alternative to VI) via “opkg install nano”.  Now configure DNSMASQ with “nano /etc/dnsmasq.conf”.
Add the following lines:
# Set the TTL value returned in answers from the authoritative server.
max-ttl=0
auth-ttl=0

Finally restart dnsmasq with “reboot” or “killall dnsmasq” and “/etc/init.d/dnsmasq start”.
Now you can read every DNS query on syslog with “logread | grep "query\[A"” (try logread to see every message). And with “logread –f | grep “query\[A” >> dnsmasq.log &” we write all new entries to a logfile in the background (can disconnect from SSH session). The logfile is only a few dozen kB after an hour and looks like this:

Sun Nov 16 18:14:56 2014 daemon.info dnsmasq[11659]: query[A] [url=http://www.facebook.com]www.facebook.com[/url] from 192.168.1.244
Sun Nov 16 18:14:56 2014 daemon.info dnsmasq[11659]: query[AAAA] [url=http://www.facebook.com]www.facebook.com[/url] from 192.168.1.244
Sun Nov 16 18:14:57 2014 daemon.info dnsmasq[11659]: query[A] farm.plista.com from 192.168.1.244
Sun Nov 16 18:14:57 2014 daemon.info dnsmasq[11659]: query[AAAA] farm.plista.com from 192.168.1.244
Sun Nov 16 18:14:57 2014 daemon.info dnsmasq[11659]: query[A] csi.gstatic.com from 192.168.1.244
Sun Nov 16 18:14:57 2014 daemon.info dnsmasq[11659]: query[AAAA] csi.gstatic.com from 192.168.1.244
Sun Nov 16 18:14:58 2014 daemon.info dnsmasq[11659]: query[A] pubads.g.doubleclick.net from 192.168.1.244
Sun Nov 16 18:14:58 2014 daemon.info dnsmasq[11659]: query[AAAA] pubads.g.doubleclick.net from 192.168.1.244
Sun Nov 16 18:14:58 2014 daemon.info dnsmasq[11659]: query[A] [url=http://www.google-analytics.com]www.google-analytics.com[/url] from 192.168.1.244
Sun Nov 16 18:14:58 2014 daemon.info dnsmasq[11659]: query[AAAA] [url=http://www.google-analytics.com]www.google-analytics.com[/url] from 192.168.1.244
Sun Nov 16 18:14:58 2014 daemon.info dnsmasq[11659]: query[AAAA] partnerad.l.doubleclick.net from 192.168.1.244

Nice, excellent! Small logfile, every webvisit on there, timestamp and IP address of local host. Now I just have to filter out all the advertising BS associated with every website (create your own adblocker by manipulating the DNS records for all these trackers).

Now I just have to download the logfile, and display connections over time (still have to figure this one out).



Summary:
I am surprised that there are not that many posts on the Internet for this specific use case (why has nobody posted a simple how-to?)
I am surprised how much traffic a simple website generates (yes, I knew this before, but seeing it is different).
I am curious if this forum can give me some new pointers/ideas that actually work!

(Last edited by MatthiasWDR3600 on 1 Jan 2015, 14:06)

Matthias, I guess if you want to analyze your traffic, you will need a lot of cpu power to do so. What i did is I check iptables every five minutes, save averages to RRD tool and generate graphs. From that I can see bandwidth used by each device on my LAN and that is enough for my use case. I found instructions for that on the forum, but I had to modify a lot inthe script. Interesting experience.

I'm interested in this too.  #1 is not good since it kills performance.  The problem I see with #3 is that it can be cheated by someone using numeric addresses and bypassing DNS.  I think #2 will work if you pipe the tcpdump through a filter based on awk or something to drop repeated access to the same site within a short time and only store the first one to a file (on the USB port of course).  Perhaps have a dictionary of "already logged" sites and wipe that dictionary every 5 or 10 minutes so if someone is still going to the site over a prolonged time, it will be logged again.

Use the 'local startup' (/etc/rc.local) to start something at boot and have it run in the background without needing to ssh in and start it.  It is a good idea to use a sleep command as the first line of your local startup to give time for the basic services to all come up, disks mounted, etc. before your program tries to run.

(Last edited by mk24 on 17 Nov 2014, 00:46)

Hi folks,

short update with option 3 (DNSMASQ logging) after 24 hrs with one device connected:
- browsing experience on iPad is NOT slower than normal - great!
- router is working fine (my log/grep process taking 1% CPU/Mem)
- logfile has grown to 800kB in a day with 7500 log entries (plenty of memory left...)
- download log to PC and quick analysis in Excel (pivot table & little string formatting)
- quick graphs to show #of DNS queries by minute, top-requested domain names

And voila, here you are: http://imgur.com/ofj1lHx


Summary: Option 3 (DNS Query logging)
- no noticable performance degredation
- logfile size is manageable
- detail level okay (domain name only, not the full URL)

Improvement ideas:
- switch more devices to my modded DNS server (performance? log size?...)
- check that really every connection shows up in DNS log (at least per minute)
- restart log process every day, and offload previous day via Email/FTP
- minimize logfile by writing only relevant text (grep/awk/sed magic)
- create log analysis files in HTML with charting via google chart api?
- start filtering all these advertising trackers

Comments to my commenters:

@marosq - no, CPU power on router is fine (1%), analyis is done in Excel. I'm not so interested in bandwidth tracking, more in domains/URLs with traffic

@mk24 - yes, DNS-logging can be circumvented by changing DNS setting on client. And, my log does not show traffic to IP addresses (without hostnames). I want to log/observe - not control access.
Also, TCPDUMP does NOT run in background even when starting from scripts - google "tcpdump in background" for plenty of people having tried this route (and I did not find a success story).


Stay posted for updates.....

(Last edited by MatthiasWDR3600 on 17 Nov 2014, 20:20)

2015 New Years Update...with mixed results

I was able to spend some time again on my DNS logger setup.

Connected OpenWRT router inside local LAN
This took some time and effort, I finally bridged the LAN/WAN interfaces and simply connected the OpenWRT router on the WAN port with the LAN port of my home router. I certainly need to clean up this configuration....

Changed DNS configuration to ensure all LAN clients resolve via OpenWRT
Make sure to change the DNS setup properly, otherwise you may end up with a loop like this:
- OpenWRT resolves from its gateway (=home router)
- home router in turn resolves from OpenWRT on the LAN

First I forward DNS requests from OpenWRT to my cable modem or the internet:

nano /etc/dnsmasq.conf
#forward DNS requests to public DNS - e.g. Google
server=8.8.8.8
server=8.8.4.4

Since we are already in the DNSmasq config file, lets make some changes to the logging:

# For debugging purposes, log each DNS query as it passes through
# dnsmasq.
log-queries
log-facility=/root/dnslog.txt
log-async=10

Lets restart the DNSmasq service to apply the new config:

 /etc/init.d/dnsmasq restart

Second, I changed the DNS settings on my home router (Fritzbox/Internet/Zugangsdaten) to the IP of my OpenWRT router.
Nice, I can see the log file growing and don't need monitor the syslog anymore! Apply the changes.


Now I can see all DNS queries in the dnslog.txt file:

root@OpenWrt:~# head dnslog.txt -n 20
Jan  1 13:48:49 dnsmasq[1339]: started, version 2.71 cachesize 150
Jan  1 13:48:49 dnsmasq[1339]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC
Jan  1 13:48:49 dnsmasq[1339]: using local addresses only for domain lan
Jan  1 13:48:49 dnsmasq[1339]: using nameserver 8.8.4.4#53
Jan  1 13:48:49 dnsmasq[1339]: using nameserver 8.8.8.8#53
Jan  1 13:48:49 dnsmasq[1339]: reading /tmp/resolv.conf.auto
Jan  1 13:48:49 dnsmasq[1339]: using local addresses only for domain lan
Jan  1 13:48:49 dnsmasq[1339]: using nameserver 8.8.4.4#53
Jan  1 13:48:49 dnsmasq[1339]: using nameserver 8.8.8.8#53
Jan  1 13:48:49 dnsmasq[1339]: using nameserver 192.168.0.1#53
Jan  1 13:48:49 dnsmasq[1339]: using nameserver 192.168.178.1#53
Jan  1 13:48:49 dnsmasq[1339]: read /etc/hosts - 1 addresses
Jan  1 13:48:49 dnsmasq[1339]: read /tmp/hosts/dhcp - 0 addresses
Jan  1 13:48:51 dnsmasq[1339]: query[A] ic.1f12ded8.017dfd.1.amazonmmd.loris.llnwd.net from 192.168.178.1

Awesome! Success!
All my LAN clients are now resolving their DNS requests on my OpenWRT router.

root@OpenWrt:~# grep "query\[A" dnslog.txt
Jan  1 13:48:51 dnsmasq[1339]: query[A] ic.1f12ded8.017dfd.1.amazonmmd.loris.llnwd.net from 192.168.178.1
Jan  1 13:48:52 dnsmasq[1339]: query[A] amazonmmd-mmd-cust.lldns.net from 192.168.178.1
Jan  1 13:49:56 dnsmasq[1399]: query[A] www.amazon.de from 192.168.178.1
Jan  1 13:50:56 dnsmasq[1399]: query[A] spectrum.s3.amazonaws.com from 192.168.178.1
Jan  1 13:51:09 dnsmasq[1399]: query[A] www.amazon.de from 192.168.178.1
Jan  1 13:51:51 dnsmasq[1399]: query[A] clients4.google.com from 192.168.178.1
Jan  1 13:52:27 dnsmasq[1399]: query[A] mail.google.com from 192.168.178.1
Jan  1 13:53:00 dnsmasq[1399]: query[A] plus.google.com from 192.168.178.1
Jan  1 13:53:05 dnsmasq[1399]: query[A] apple-mobile.query.yahooapis.com from 192.168.178.1
Jan  1 13:53:05 dnsmasq[1399]: query[A] csi.gstatic.com from 192.168.178.1
Jan  1 13:53:06 dnsmasq[1399]: query[A] p32-keyvalueservice.icloud.com.akadns.net from 192.168.178.1

But....hey!!! Why is my Fritzbox simply forwarding DNS and not handing out the DNS server?
I can see that really all DNS requests are properly received and answered on OpenWRT. But as you can see above - the entries are all coming from the same IP (x.x.178.1) - which is the IP of the Fritzbox.
Checking IPconfig on my home PC I realize that the Fritzbox is NOT handing out the new DNS server directly - instead it forwards them. After I manually changed DNS on my PC I can also see which client is logged.

Jan  1 14:05:24 dnsmasq[1399]: query[A] notepad-plus-plus.org from 192.168.178.38

How do I change DNS settings on all my clients in the LAN?
So...how do I convince my home router to hand out the DNS server instead of simply forwarding? I don't want to manually change the DNS settings on all my devices. Of course, I could replace my Home router with the modded OpenWRT unit alltogether - but then I can't play around with OpenWRT anymore (or, I get another WDR3600 unit?).

Next exercise: Log analysis and scripting

Hi MatthiasWDR3600,

multiple options in my eyes, depending on your setup of your home router, Fritzbox and Openwrt router:

- It seems the Fritzbox serves as a DHCP and DNS server. IN that case, try to change the IP address of the resolving DNS server for your DHCP clients in the Fritzbox setup to the IP of the Openwrt router
- next option depend on your network setup: if all three devices share the same segment (IP subnet) of e.g. 192.168.1.0/24, then deactivate the DHCP option in your home router or/and in the fritzbox and instead activate it solely in Openwrt. Effect: all DHCP clients will get the information of the DNS server from the Openwrt router itself.

Both options just working for DHCP clients as the distribution of resolving DNS server is part of DHCP process.

Hope it will help.

Hi MatthiasWDR3600, your post helped me to monitor internet connections, but the issue is I am having WR841nv11 router, which has only 4 MB flash and I created a new interface for coova-chilli and I used your this below method:-

log-queries
log-facility=/root/dnslog.txt
log-async=10

But, it only shows log for lan and wan connected users, not for chilli users. chilli users get IP in 192.168.180.xx. I want to record logs for chilli users.

tcpdump -q -i tun0 –tttt

This command records log for chilli users, but less memory to install tcpdump. Any other method to record log for chilli users, it will be great appreciation. Thanks in advance

The discussion might have continued from here.