I just found the answer to #1 -- you do have application categories such as "Streaming Media".
Is there a complete list of the application categories somewhere? Thanks!
Hey bizzbyster.
The Netify Firewall Agent (NFA) should be considered a skeleton for integrating applications/protocols (Layer 7 stuff) into a platform. Basically, there's a stream of metadata coming out Netify, and NFA takes that information and munges it into various firewall/QoS engines (VyOS, pfSense, etc). We haven't built support for OpenWRT yet, but it would take less than a day to get a v1.0 completed.
As for routing policies for multiwan, you are getting into the "first packet" problem that some of our SD-WAN vendors had to solve. Tricky! Deep packet inspection can take up to 10-ish packets to get enough data to do the detection. For example, the HTTPS SNI hostname is seen in packet #4. However, a routing decision for multiwan needs to be done on the first packet. There are a few ways around it.
-
Quick and dirty. Let the first initial conversation (e.g. the first Netflix payload) pass through without a WAN application policy, but cache the results. Subsequent conversations would then follow the WAN policy. Lots of leakages.
-
DNS hinting. Most Internet requests start with an initial DNS request. Netify can pre-populate a lookup table before the first application (e.g. HTTPS/QUIC) packet arrives. There's still some leakage, but it's better.
-
Other voodoo that engineers come up with.
2 Likes
And to answer your other question, the list of applications and categories are here:
The "Adult" apps aren't listed, but you can also grab the full dataset via the public API:
https://informatics.netify.ai/api/v1/lookup/applications?settings_limit=1000
The categories are in the "data_options" part of the API payload (though some might be deprecated).
1 Like
We haven't built support for OpenWRT yet, but it would take less than a day to get a v1.0 completed.
Great! Any way I can help make this happen? I am a developer and so might be able to do the work but would love to do it in a reusable way. Perhaps send a pointer to the code with some guidance?
Quick and dirty . Let the first initial conversation (e.g. the first Netflix payload) pass through without a WAN application policy, but cache the results. Subsequent conversations would then follow the WAN policy. Lots of leakages.
This may work for my use case. A bit of leakage is likely fine, especially if the application automatically reconnects when connections are forcibly closed. Any efforts in the works I may be able to tag along with?
Thanks Peter!
1 Like
Thank you for posting this. Can you confirm if netifyd is ready to go?
Can you point me to a resource how to retrieve the inspection data generated in a way that can be used locally or for POST?
Wow I'm, suprised that people like the idea of sending connection details to 3rd party.
That's a complete security/privacy nightmare.
In addition to that, you would have to get permission from every user before he uses the network, that his private data (which connection details are) will be send to your servers. You are in deep trouble with DSGVO for example.
3 Likes
Hi KONG,
Ideally, we would be able to provide an on-premise "solution-in-a-box" for small businesses (and techy households), but it's just too complicated at the moment. For the AI/machine learning bits, we need to have a 48-core 256 GB server kicking around. For customers with 250 or more sites, the Netify Network Intelligence solution can be deployed on-premise or in a private cloud.
And just to clarify, the open-source netifyd engine doesn't send metadata to the cloud at all unless you explicitly enable it.
Wow I'm, surprised that people like the idea of sending connection details to 3rd party.
In a way, it's already happening.
We realize that many network administrators are uneasy about the cloud, and that's perfectly understandable. But just keep in mind that ISPs use Sandvine-like tools to collect similar network metadata! ISPs know the makes and models of the mobiles, tablets, TVs, game consoles, and other devices are running in a business/household. And they can extract demographic and usage data too (e.g. what video games are played at what time of day). Spooky.
We like to turn the tables and put that power into the hands of the network administrator. Administrators need to know what those IoT devices, SmartTVs, and weird mobile apps are doing on their networks. [shakes fist at the ads on my Samsung TV menu].
However, we have to earn an administrator's trust. Here's what we have done so far:
- The data collector is open-source - developers can dig into the code and see what's being collected and sent.
- It's metadata in the first 10 network packets, not the whole payload (a private document attached to an email, for example).
- PII (personally identifiable information) is encrypted on the client side (browser / mobile app) with a passphrase. For example, if a network admin tags a device as "Cindy Laptop", it's just an encrypted blob in the data store.
- The metadata can be anonymized on-premise before being sent to the cloud: IP addresses, hostnames, MACs, etc. Details on our privacy page.
- The data stores are siloed from the billing information.
- On-premise options are available at scale.
And yes, there is definitely an element of trust that's very similar to the privacy-oriented VPN services. With VPN, you might be bypassing the ISP data collection, but that trust now lies with the VPN provider. [tinfoil hat on] Are they using Sandvine and selling your data? Do they really not keep logs? Are they providing warrantless access to government agencies because the VPN server is outside of the country? [tinfoil hat off]
GDPR
It's a bit counter-intuitive, but Netify is often used to comply with GDPR requirements. Weird, eh?
First, we here at Netify can't connect the network metadata to an individual. We don't know the person behind MAC address 11:22:33:44:55:66 -- no home address, no email, no mobile number, no Facebook profile, etc. GDPR is mostly about protecting personal consumer data that is in the hands of retailers, marketers, healthcare providers, etc.
Companies have started implementing tools like Netify and Netflow to comply with GDPR's various requirements and articles:
- Information access audit trails
- Asset/device inventory
- Risk assessments
- Forensics
In other words, GDPR is telling companies that they better make some effort into knowing what's happening on the network. Companies that store personal data can't just shrug off cybersecurity threats by doing nothing. And when that personal data breach occurs, GDPR is expecting a proper incident response with forensic data.
Looks like the netifyd and the netify-fwa are now available on OpenWrt !
great news...
Is there anybody testing the netify-fwa ?
I am getting errors in debug mode :
root@LPM:~# netify-fwa -d
netify-fwa[9148]: Netify FWA v1.2.5 started.
netify-fwa[9148]: IPTables Firewall driver initialized.
netify-fwa[9148]: OpenWrt driver initialized.
netify-fwa[9148]: Firewall engine: OpenWrt iptables v1.8.3
netify-fwa[9148]: nfa_fw_iptables::add_rule: Bad argument `lan1'
netify-fwa[9148]: nfa_fw_iptables::add_rule: Try `iptables -h' or 'iptables --help' for more information.
netify-fwa[9148]: nfa_fw_iptables::add_rule: Bad argument `lan1'
netify-fwa[9148]: nfa_fw_iptables::add_rule: Try `ip6tables -h' or 'ip6tables --help' for more information.
netify-fwa[9148]: Loaded dynamic configuration.
netify-fwa[9148]: Connecting to: unix:///var/run/netifyd/netifyd.sock
netify-fwa[9148]: Connected to: unix:///var/run/netifyd/netifyd.sock
netify-fwa[9148]: unix:///var/run/netifyd/netifyd.sock: Netify Agent/2.88 (aarch64; conntrack; netlink; dns-cache; plugins; regex) nDPI/2.9.0 JSON/1.90
Hi Gandalf,
That's an old version. Try upgrading to netifyd 3.05 - instructions are here: https://www.netify.ai/get-netify/openwrt-upgrade
The "ipset" rules are fairly well tested, but the "mark" and "block" rules are definitely a moving target. I'm right in the middle of upating the CI/test workflow for OpenWrt, so we should get to beta-level support for OpenWrt in the next few days. Yay!
2 Likes
Thanks,
Same problem after upgrading...
I will wait next (soon) release !
Ready to test...
At first glance, that looks like the very basic automatic network interface detection is not going to cut it. Could you send your "uci show" output to devel@netify.ai? Thanks!
1 Like
sent... thanks to look at
To be honest this still reads very much like a sales pitch. Is there a way to leverage netifyd with some other open source, locally (as in within the LAN, not necessarily on the same edge device) hosted GUI solutions that would make this function similarly as ntopng?
1 Like
Absolutely there's a sales pitch 
, but there's also some open source goodness.
On OpenWrt, netifyd has been used with other open source tools to:
- Block layer-7 protocols (e.g. BitTorrent)
- Prioritize specific applications (e.g. Zoom traffic)
- Route specific applications and protocols out different multiWAN connections
... but no ntopng replacement. I'm not aware of anyone using Netify DPI to build such a solution, but it's certainly possible with a lot of GUI development work.
1 Like
i'm a little lost... on the pricing page it has two sides... 'integrators' and 'organisations'...
i'm a home user... so if I understand it right... I pay 25$ a month for 'protocol detection'?(and related analytics?)... that's more than half my internet bill...
the OP states...
what exactly again does it offer in regards to this management featureset?
Hello wulfy23,
Sorry, Netify Informatics (the subscription available on our website) is not intended for home use. Our typical deployment sits on a network segment with 50 to 150 users. We do have some home user solutions on the roadmap, but those are a work-in-progress.
A better choice of words would have been "monitoring" instead of "managing", i.e.:
the solution is a great tool for m̶a̶n̶a̶g̶i̶n̶g̶ monitoring dozens, hundreds or thousands of edge gateways
Sorry about that.
You can get a single view across a distributed network. For example, the main dashboard for an enterprise might have an events listing like:
- 1 hour ago -- Boston Office -- Flow anomaly detected over HTTP protocol from device Canon TS6100
- 10 hours ago -- Portland Office -- Detected SSH Secure Shell connection on Ubiquiti WAP
- 1 day ago -- Toronto Office -- New device iPhone XR discovered on the network
1 Like
might want to have a partner discussion as we developed a complete turnkey openwrt management and upgrade engine, complete with web interface for monitoring and management.
root@LPM:~# netify-fwa -d
netify-fwa[5440]: Netify FWA v1.2.6 started.
netify-fwa[5440]: IPTables Firewall driver initialized.
netify-fwa[5440]: OpenWrt driver initialized.
netify-fwa[5440]: Firewall engine: OpenWrt iptables v1.8.3
netify-fwa[5440]: Prepping internal interface hooks: lan0
netify-fwa[5440]: Prepping internal interface hooks: lan1
netify-fwa[5440]: Prepping external interface hooks: wan
netify-fwa[5440]: Loaded dynamic configuration.
netify-fwa[5440]: Connecting to: unix:///var/run/netifyd/netifyd.sock
netify-fwa[5440]: Connected to: unix:///var/run/netifyd/netifyd.sock
netify-fwa[5440]: unix:///var/run/netifyd/netifyd.sock: Netify Agent/3.05 (openwrt; aarch64; conntrack; netlink; dns-cache; plugins; regex) nDPI/2.9.0 JSON/1.90
version 1.2.6 ok
thankyou for clarifying and your commitment to openness... I guess i'd made a false assumption based on the forum general userbase and also taken stuff a bit too literally...
totally clear now