thankyou for clarifying and your commitment to openness... I guess i'd made a false assumption based on the forum general userbase and also taken stuff a bit too literally...
totally clear now
any news ?
Nothing yet... sorry.
I have to say I've found the whole 'netify' thing to offer lots of promise AND lots of frustration. Maybe I'm being stupid or simply not clever enough but some of the 'simple' stuff should just work. eg. I took out a month subscription (PAID MONEY), it worked for 15 days and then it seems my organisation (ie my home with single netify-agent) just simply disappeared.
I wanted to use netify-agent/netify-fwa to do some smart flow classification for bandwidth/prioritising via ipsets - I've got some of that working and had to fix a bug in the agent related to 'wide characters' vs 'json string lengths'
I wanted to have some sort of 'live dynamic' view into what netify-agent was actually detecting - netify-console appears to be just the thing, but I have yet to actually get a running instance on any flavour of linux, there usually being a missing (php ncurses) library. I'm not a php person, I'm barely a python person but I do know that python3 ncurses exists & works even in openwrt.
To be blunt, stop what you're doing and get a basic cross platform monitoring tool working.
3 Likes
Bummer. Sorry for the frustrations and quirkiness. OpenWrt is a great platform for building custom hardware/software products, but it's a difficult platform for providing solutions to the wider community. The openness of OpenWrt is great, but when there are dozens of QoS implementation and a handful of captive portal apps, it makes it difficult to provide a sane one-size-fits-all solution to end users. For comparison, products like pfSense and VyOS have specific implementations for QoS and captive portal, and that makes it possible to get further along.
We have been part of three custom products on OpenWrt, but these hardware/software products have all come with tight integration and specific firewall/QoS/apps, not to mention known hardware, memory, and CPU. That makes it possible to rely on the underlying operating system.
netify-console appears to be just the thing, but I have yet to actually get a running instance on any flavour of Linux
The PHP-based netify-console for ClearOS is definitely not intended for other Linux distributions! ClearOS has its own sandboxed version of PHP, so netify-console definitely won't install very cleanly on other Linux distros. There is some development going on in our free time (C/C++, not PHP), but there aren't enough hours in the day. The console is an open source / free-as-in-beer tool, so if there are any C/C++ developers out there willing to help, please let me know.
Ugh. That's a human error... sorry. Y'all shouldn't be paying for a subscription anyway! Anyone who contributes to the open source project is welcome to a free subscription. Sign-up again to the eval (if you dare 
) and send me a personal message -- I'll extend the trial a couple of years.
3 Likes
Hi Peter,
Thanks for getting back to me and apologies for the somewhat abrasive post. I did manage to get netify-console running in a centos 7 VM on my macbook so I've been able to play. You'll find a couple of merge requests on gitlab for the agent (man page tweak) & console (http user agent check) Hopefully you'll find those useful and the console one a hint on some validation to Darryl's c++ rewrite 
Really looking forward to the c++ re-write, should be able to make that run on openwrt natively. I can't really code for toffee, well certainly c++ would be new so not sure I can contribute new code... I can sometimes mangle existing code.
I also have a much improved & simplified netify-fwa openwrt package definition 'Makefile'. I don't know if you want that, if you'd like me to try to get it added into openwrt's package repo or if things aren't quite ready, let me know.
What's the best way of contacting you/Darryl with fwa suggestions? Using ndpi/netifyd/netify-fwa for flow classification for priority purposes is what I find most interesting, not so much the AI stuff but I might take you up on the eval again, it'll be a few days 'cos I'm busy doing the day job which is nothing to do with openwrt.
Kevin
1 Like
Hi Kevin,
Many thanks for the merge requests... one was merged today. I'll answer your questions offline, but will post any interesting open source tidbits here.
I'm going to figure out the best way to merge your prioipset concepts. I'm also hoping to get the python-based Netify FWA into the ClearOS project -- this might run into some roadblocks though.
Hi Peter,
Thank you for your updates. I do have one question, how to enable netifyd agent to send JSON to a local script or POST? I didn't find that in the documentation, my bad if it's there.
In OpenWrt, the data stream is sent to a local Unix socket by default: /var/run/netifyd/netifyd.sock This can be changed to a TCP/IP socket for external access. In /etc/netifyd.conf, change:
[socket]
listen_path[0] = /var/run/netifyd/netifyd.sock
To
[socket]
listen_address[0] = 1.2.3.4
Where 1.2.3.4 is your local LAN interface IP. The default port is 7150, though that can be changed as well.
It's in the man page (man netifyd.conf), but there are no man pages on OpenWrt! You can find information on the data streams here.
1 Like
Thank you Peter. I will check it out and come back in case of any questions.
always getting this error
Error connecting to: unix:///var/run/netifyd/netifyd.sock: No such file or directory
here is full log
root@OpenWrt:~# netify-fwa -d netify-fwa[14214]: Netify FWA v1.2.5 started. netify-fwa[14214]: IPTables Firewall driver initialized. netify-fwa[14214]: OpenWrt driver initialized. netify-fwa[14214]: Firewall engine: OpenWrt iptables v1.8.7 netify-fwa[14214]: nfa_fw_openwrt::get_external_interfaces: uci: Entry not found netify-fwa[14214]: nfa_fw_openwrt::get_external_interfaces: uci: Entry not found netify-fwa[14214]: nfa_fw_openwrt::get_external_interfaces: uci: Entry not found netify-fwa[14214]: nfa_fw_openwrt::get_external_interfaces: uci: Entry not found netify-fwa[14214]: JSON file not found: /etc/netify-fwa/matches.json netify-fwa[14214]: Loaded dynamic configuration. netify-fwa[14214]: Connecting to: unix:///var/run/netifyd/netifyd.sock netify-fwa[14214]: Error connecting to: unix:///var/run/netifyd/netifyd.sock: No such file or directory netify-fwa[14214]: Connecting to: unix:///var/run/netifyd/netifyd.sock netify-fwa[14214]: Error connecting to: unix:///var/run/netifyd/netifyd.sock: No such file or directory
why need that much hard disk space?
1 Like
There's really no disk requirements beyond the install size. Documentation updated!
Sorry for the slow reply... I don't always get the forum notifications in my inbox.
I have a feeling the following might be required in /etc/netifyd.conf:
listen_path[0] = /var/run/netifyd/netifyd.sock
On another note, we'll need to review our OpenWrt documentation - I'm sure some things have changed with the 21.x releases. What version of OpenWrt are you using?
OpenWrt 21.02.1, also tried 19.07
netify-fwa[5961]: Invalid netify-agent scheme (only tcp:// and unix:// schemes are supported).
I fixed issue by replacing socket-uri entry in netify-fwa.ini like this, functional now
[netify-agent]
#socket-uri = unix:///var/run/netifyd/netifyd.sock
socket-uri = tcp://127.0.0.1:7150
@pbaldwin Youtube on desktop is blocked but unable to stop it on mobile app, also whatsapp audio messages and photo transfer is blocked but text messages pass through, what might be possible solution?
Thank you for the help
I would recommend enabling the 14-day trial when kicking the tires (instructions here). No obligation at all, but it provides:
- The latest application signatures (/etc/netify.d/netify-sink.conf) - these are updated daily. I know that WhatsApp has had quite a few updates over the last few weeks.
- A web-based interface to visualize the data (see screenshot below). It can be easier than staring at JSON all day
For example, I just pulled up WhatsApp on my mobile and tracked all the flows. If there was a detection that was missed, it would show up as "Unclassified" and we could then update the application signatures to improve detection. Or, dive into packet captures to improve protocol detection.
WhatsApp is particularly confusing because it has 3 (yes, 3) protocols defined (WhatsAppVideo, WhatsApp, and WhatsAppVoice) and 1 application defined. Just use the "WhatsApp" application definition for best results.
2 Likes
Is it possible to create new applications in netify-sink.conf ?
for example
ip:192.168.18.143/32@11000.pi.server
or
host:"^xyz.co.uk$",host:".xyz.co.uk$"@11001.shops.shoes
or if there is any guide to do so if possible?
With Netify Informatics enabled, this signature file is automatically updated and it will get overwritten. We have tools for vendors to manage custom signatures across all of their netifyd-enabled sites from a central location. However, for hacking around, you can specify an alternate signature file using the netifyd --sink-conf command-line option. Copy and paste the existing /etc/netify.d/netify-sink.conf to something like /etc/netify.d/my-sink.conf and hack away.
Yes, that works! The first parameter (11000 in your example) is a unique ID of your choosing. The second parameter is typically a vendor code - it's just to avoid name collisions with other vendors. The third parameter is a unique tag of your choosing. A bit weird, but it keeps it compatible with the nDPI format.
2 Likes
Very much a WIP, I was playing with Python for the first time, but I created a script that uses netcat to listen for data, parse the json payload, map local_ip to hostname (requires prometheus DHCP module on the router), and takes the other_ip and adds geoip information (you need to add your maxmind key to the script). Feel free to try it out, but please remember, I've only been messing around with python for three days 
use at your own risk. Once the data is in SQL, you can do whatever you wish (Admin dashboard, Grafana, etc). Best to test this on a Ubuntu/Debian OS (RaspberryPi, VM, LXC)