Open Source DPI and Network Intelligence Engine (Beta)
Hi OpenWRT Community,
We're a Canadian company that has developed two bits of technology that can help with managing networks at the edge of the Internet:
An open source DPI engine - the Netify Agent (netifyd) - that can detect protocols, applications, and other fun network tidbits. The protocol detection part of the engine is currently based on nDPI, but we’re always adding fun features and hooks to get better network visibility.
A cloud service (gasp!) - Netify - that provides complete network traffic visibility and intelligence (screenshots) using the aforementioned DPI engine.
For network administrators, the solution is a great tool for managing dozens, hundreds or thousands of edge gateways. For integrators, the technologies have been integrated into SD-WAN devices, IoT gateways, and firewalls. Below is a pfSense screenshot showing WhatsApp and Facebook getting blocked by the Netify Agent DPI engine.
Since OpenWrt is a great solution for the edge of the network, we’re starting to provide support for the platform. With the release of OpenWrt 19.07, you can easily install the Netify DPI agent the usual OpenWrt way. More on this below.
Please keep in mind, DPI in tight spaces is tricky, but we’re going to take a crack at it!
OpenWrt users tend to lean on the technical side, so let's get down to the details. The open-source Netify Agent (netifyd) uses deep packet inspection (DPI) to extract useful metadata from a network conversation:
- Protocol (yes, encrypted protocols too)
- SSL ciphers, SNI, certificate names, etc.
- Agent strings
- Torrent hashes
- DHCP fingerprints
- SSL fingerprints
- and more
The Netify Agent only analyzes the first few packets in a network conversation and then sends a JSON-encoded stream of metadata to a socket interface (TCP/IP and Unix sockets). We’re not interested in the payload, just the metadata in a network connection. Third party tools can connect to this data stream and do all sorts of different things - firewalling, QoS, reporting, etc.
For example, the screenshot below is part of a stream of information coming from on an Avast client on a Windows 7 desktop making an HTTPS/TLS 1.2 (version: 0x0303) connection to an Avast server (wildcard SSL certificate: *.avast.com”).
As you can imagine, it’s possible to build hooks into OpenWrt’s firewall and QoS engines using this data stream. Do you want to throw BitTorrent traffic into a low-priority QoS queue? That’s the hope for 2020, but we’re first going to concentrate on tuning DPI performance.
Integrators: Kicking the Tires
Do you want to kick the tires in OpenWrt? First, you should know that deep packet inspection (DPI) requires some horsepower, so please don’t try this on low-end hardware! I have it running on an old TP-Link WDR3600. Memory is not a problem, but the CPU load is approaching max when I’m pushing the wireless network as fast as I can.
If you are interested in seeing live data on your network, you can use nc (netcat) and jq (JSON processor) to see data flows like the above screenshot. With the release of OpenWrt 19.07, you can install the netifyd engine the usual OpenWrt way.
opkg install netifyd
And jq for viewing pretty JSON output:
opkg install jq
The nc tool in OpenWrt is only able to listen to TCP sockets, so you have to add the listen_address line below to enable TCP sockets in the netifyd engine. In /etc/netifyd.conf:
listen_path = /var/run/netifyd/netifyd.sock
listen_address = 127.0.0.1
Then restart netifyd:
service netifyd restart
Then you can watch the JSON payloads go by on your terminal using netcat and jq:
nc 127.0.0.1 7150 | jq
If you are a developer, you can do all sorts of fun stuff with the data. You can find more information on the DPI engine integrator page.
Netify Network Intelligence
Well, those are some of the details about the on-device DPI engine. Now I’m going to put on my product hat for the cloud-based Netify product.
product pitch warning
For those of you who want a deep analysis of what’s happening on your network, read on.
This same JSON network data stream is used for Netify - a cloud-based subscription service that provides network intelligence and visibility. Netify makes it possible to manage network resources, identify & inventory devices, enforce company policies, provide forensics, detect weaknesses, and stay on top of cyberthreats. In essence, Netify provides insights to help manage network and devices. Here are some screenshots.
- Device Discovery
- Application Detection
- Protocol Detection
- Risk and Reputation Analysis
- Bandwidth Monitoring
- Hostname Visibility
- Geolocation Information
- Connection Tracking
/product pitch warning
What's Free, What's Not
Just to summarize what's free and what's not:
The underlying netifyd deep packet inspection agent is free, open source, and licensed under the GPLv3. We’re big fans of open source ... we used to manage and maintain a Linux distribution (ClearOS).
The Netify cloud-based service is a paid subscription service, starting at $25 per month. Subscription levels can be found on Netify's pricing page. Please feel free to take a test drive with our no-obligation 7-day free trial.
We know, we know... "cloud" is not everyone's cup of tea, but offloading the horsepower to do analysis is necessary. It’s also important for our clients who manage hundreds or thousands of endpoints on the network. Regardless, there’s an option for private hosting for enterprise deployments.
Installation and Configuration
Though the underlying netifyd engine is mature, the OpenWrt environment is new to us, so it comes with the usual first release kind of software quality. If you want to try the Netify cloud-based service on your OpenWrt 19.07 system today, you can find installation and configuration instructions here:
From time to time, we’ll post updates on the DPI engine in this thread. Feedback and comments are welcome.