I wanted to ask if anybody have any advice for monitoring similar to what you find in PFSense ntopng? I want something like the Active Flows where you can see what your IoT/internal devices phone home to, maybe even with a map.
I have tried various tools like Darkstat, but most of the ones I land on are more geared toward bandwidth usage monitoring, and I am quite happy with using VnStat for that. One blog I landed on had used Syslog-ng and Elasticsearch Security to build something similar, but slightly more complicated.
I would prefer not having to pay for it if I had to build it or use a package, but if there is a low cost service I don’t mind paying. I also understand that PFSense tools are x86 and BSD and may not need to offload the data to be analyzed since the hardware can handle it. For reference I have a GL-MV1000, and would prefer to keep the services on the device if possible.
I would also like to know about options for this as well... as I add, intentionally or unintentionally, more IoT and similar gizmos to the house.
Being able to easily know if they are doing just what they are supposed to do, vs sending lots of data out, or generating inordinate amounts of traffic on the local network.. would make me feel better. I've already seen suspicious (at least it looks that way) behavior out of wifi power plugs.
You need a reasonably beefy server. ELK is a bit of a resource hog. And using something like a elasticsearch's curator to trim logs older than a certain time is a must. Each elastiflow index uses about 100MB for one day of data.
It's an i7-4790s with 24GB of RAM and 8TB of SSD storage running Ubuntu 20.04
It gets used for quite a lot of other things though, including a recursive DNS server, my Ubiquiti Unifi controller for all my APs and my openwrt build system. My ELK stack also ingests all syslog output from all devices into Graylog (which is a bit like logstash with a graphical front-end), which also chews resources.
I'd say that a moderately used ELK stack will probably need about 8GB of RAM to run ok. It really depends on how long you want to maintain the indices for. More data retention = more resources required.
Incidentally, the piece where I use graylog is very similar to what you mentioned in your first post in this thread. Syslog-ng runs on my openwrt router and forwards logs including firewall logs and snort logs to the graylog server.
It's very useful in conjunction with netflow data, as you can then go and look at log file / firewall log data if you notice something odd in the netflow data, or vice-versa.
Graylog also lets you setup dashboards, so I have a firewall dashboard that at a glance lets me get a very good picture of what's going on as it displays things like histograms of the message count. This way, a concerted probe of your firewall is immediately obvious, as the message count goes up. I've also set it up to display aggregate data, such as which hosts are probing the most and which ports they're probing the most, as well as various combinations of similar data.