Hi,
I need to know what websites with either partial or complete URL are been visited from my home router.
This can be either dumped to a local file and log-rotated or URL info be sent to an external syslog server.
I've gone through OpenWrt documentation but don't see anything useful.
Can someone help me know how this can be achieved.
TIA.
When clients use TLS, SSL, HTTPS, whatever you want to call it, you can't. Best you can do in that case is record the DNS queries made, the IP addresses of the hosts that they connect to, and the names on the certificates they return before the connection is encrypted. The HTTP request is made after the connection is encrypted.
For unencrypted HTTP, you can use a proxy (assuming your clients are cooperative) or snoop the HTTP request as it is made "in the clear".
ngrep is one useful tool for this kind of snooping. There are many others.
Thanks. I can use ngrep and sniff out DNS. Tried that out on my linux machine and it works with below option.
ngrep -l -q -d eth0 -i "" udp and port 53
Next Question:
Where and How can I dump this data (somewhere) ?
I can't dump it on router as there is very little space (few MBs) . Can I point this data to somewhere in cloud and later on pull that file from cloud for more analysis ?
Before you go too much further, what you are proposing to do is likely illegal in the EU, or involving anyone in the EU, under the GDPR, and likely runs afoul of privacy regulations in other jurisdictions as well.
Yes, you'll likely need both more storage and more processing power to generate any meaningful results.
Storage in the cloud would likely compound your exposure as not only would you be collecting personal data, but also failing to properly protect it.
Which means I cannot sniff my own data !
You can generally sniff your own data, but as soon as you start "monitoring of their behaviour as far as their behaviour takes place within the Union" then you should be aware of the implications of the GDPR, Regulation (EU) 2016/679. There is an exception that
- This Regulation does not apply to the processing of personal data by a natural person in the course of a purely personal or household activity and thus with no connection to a professional or commercial activity. Personal or household activities could include correspondence and the holding of addresses, or social networking and online activity undertaken within the context of such activities. However, this Regulation applies to controllers or processors which provide the means for processing personal data for such personal or household activities.
which may apply to your situation.
Well, its documented on wiki.openwrt.org/doc/uci/dhcp
"/etc/config/dhcp"
config dnsmasq
option logqueries '1'
The log will be written on syslog & log-rotated with default buffer size. You can adjust the file location at local or at your desired server(:port) along with its buffer size on
"/etc/config/system"
config system
option log_size '64'
option log_ip '192.168.6.4'
option log_port '6400'
Then wait for what the EU going to do with you!
If the native logging features are not of interest, you can consider pihole.
I would also like to inquire about this -- especially logging IP addresses. My DNS server already logs all DNS requests and all traffic on port 53 is redirected to the intranet.
Also, I'm not in the EU or EEA, so GDPR doesn't apply to me, and I'm processing data of myself only, and my devices (which do not consent to the data processing but aren't considered people, so I don't need their consent 
)
I'm using 19.07rc2 on ramips/mt76x8 with an external syslog server.
So, in summary: I want to be able to log complete HTTP traffic metadata (URL, from LAN IP, to remote IP, via WAN IP, device location (wlan0, wlan1, eth0.1, eth0.2, eth0.3, eth0.4) to syslog and sync it remotely to an external server (which the build-in syslog server does, and I have successfully set it up).
Right now I have achieved this by adding iptables rules to log all traffic, but the results are not detailed at all. I would like to log HTTP URL and hostname, HTTPS hostname, TCP data in general (from ip & port, to ip & port).
Not possible, by design of HTTP-S
(Nor is anything else in the encrypted channel, such as URLs, other metadata, or content sent/received)
Of course it's possible, I explicitly said hostname, not URL or any other metadata (well, other than IPs and ports of dest and src, time info).
Unless the browser implements Encrypted SNI, it will send the hostname as cleartext in the initial TLS handshake. And since the "browser" is not actually a browser but instead an embedded device with a really bad TLS implementation, it will never have Encrypted SNI enabled.
So why don't you edit the iptables rules - in order to get more of the details you desire?
...or are you seeking someone to assist you with those detailed rules?
iptables is not the answer to logging HTTP and HTTPs metadata, it's suited for logging TCP, mostly. It's too low level.
Also, can you stop with the agressively-attacking attitude. I have been polite with you, it's only fair that you be polite to me, given that I have not posted anything that could be interpreted as being abusive, spam-like in nature, unwanted, etc.
Asking what you're seeking is aggressive and attacking!?!?
LMFAO!
Don't worry, I won't try to help you any further.
In the context of usual forum ruleset of "ask for guidance only, don't request the whole solution", "are you seeking someone to assist you with those detailed rules?" comes off more as agressive than a genuine offer of help. Especially in the context of "So why don't you" and "or are you".
In any case, I apologize if I have misinterpreted it, English is not my mother tongue.
Well since it's clear now that you don't want to use iptables, I'm not sure what you're seeking.
Rewriting your iptables rules actually seems easy to me.
If I were trying to accomplish the goal, I'd run suricata or a similar packet-inspection tool on a suitable host. It would surprise me if a consumer all-in-one has enough compute power to do packet inspection at more than a trickle. MIPS-based SoCs have trouble with something as computationally simple as SQM at around 100-200 Mbps, for example.
Even diverting all the packets to an external host for analysis may well be beyond what an all-in-one can accomplish. Many/most achieve high throughput by offloading the flows from the CPU to the ASIC switch framework.
Seems like you'd need a topology that separates your network into VLANs (or physical LANs) so that you can accomplish your objective of identification of where the associated host is connected.
I care about this issue, because I want to control ads and what is dangerous if any.