Thank you it works. But I would have one more improvement you don't need to rewrite the rules from alert to block (drop) it's enough to add in the IPS section action_override = 'block'.
1 Like
I found another problem the queue entry is killed by some script when you add it for example via rc.local during the startup process. I could observe it during the startup process first it was there in the firewall rules after Openwrt was finished it was gone again. I have solved the problem by making a small script which contains a sleep command of 300 seconds before the rule is added. I will observe this if it also happens during runtime, maybe it would be better to put everything in a separate table like banip does for example.
Correct, @cuongdao mentions that in a note above. Whenever the firewall is reloaded, anything like that is transient. To get it working with firewall reload, put it in /etc/firewall.user and add it to the firewall includes in /etc/config/firewall like this:
config include
option enabled '1'
option type 'script'
option path '/etc/firewall.user'
option fw4_compatible '1'
Test it with fw4 reload and then nft list chain inet fw4 IPS to see if the commands were executed.
Thank you for the instructions. But I'm not sure if this is the ideal configuration because the performance is much worse than with afpacket which is a bit strange because I don't see any Cpu bottleneck. I think there is a reason why Banip uses its own table maybe for performance reasons?
Are you using htop for finding bottlenecks? Assuming so, do you have the "Hide kernel threads" turned off? Default is to hide it, so it might not be showing.
I'm just guessing, but creating a separate table would probably only give a tiny improvement if anything. The packet pipeline through nftables is pretty streamlined already.
Snort is going to be a hog no matter how you deal with it. If I'm recalling right, it's still single-threaded internally (or was that the reason for the snort 2 -> 3 rewrite, I'm probably mixing things up). I think we need to get Suricata up and running on OpenWrt and see if it, with it's much more modern take on things, does as good a job at IDS/IPS while hopefully reducing resource utilization (I noted on first startup that snort gobbled 255 MB of RAM, gah!).
I see you've already been to @darksky's thread on performance issues, but in case anyone else wants to jump in...
No performance problems on the part of the Cpu are not because I had afpacket Daq running before and there I got full speed with Vpn it seems like the queue is slowing down because creating a table and adding the rules to the new table has already brought a visible improvement. Maybe we should make one queue for Tcp and one for Udp or for inbound and outbound Snort is able to handle multiple streams at the same time that should alleviate the bottleneck. Problem is I have no idea about nftables. In any case the configuration is more performant and there is no problem with firewall reload.
nft add table inet snort
nft 'add chain inet snort IPS { type filter hook forward priority filter ; }'
nft insert rule inet snort IPS counter queue num 4 bypass
1 Like
You can detect the family with a meta l4proto match on the rule. The beauty of nft becomes apparent, as this grabs both IPv4 and IPv6 packets.
$ nft add rule inet snort IPS meta l4proto tcp counter queue num 4 bypass
$ nft add rule inet snort IPS meta l4proto udp counter queue num 5 bypass
$ nft list table inet snort
table inet snort {
chain IPS {
type filter hook forward priority filter; policy accept;
meta l4proto tcp counter packets 0 bytes 0 queue flags bypass to 4
meta l4proto udp counter packets 0 bytes 0 queue flags bypass to 5
}
}
Then, of course, add queue '5' to the inputs in your snort.lua...
Thank you that should work better. Apparently the reverse path filter is also a problem when I disabled it the performance was also better. But I think it is turned off by default.
Oops, just those two rules ignore the non-tcp/udp packets like ICMPv6, so I think we also need another one to grab all those
nft add rule inet snort IPS 'meta l4proto != {tcp, udp} counter queue num 7 bypass '
Thanks I will add it to my script but I will test the performance first.I have now created 2 streams with the z parameter in the start line but I still have to see if there is no error because I added the parameter directly in the service file is easier.
I found the problem nfnetlink_queue: nf_queue: full at 1024 entries, dropping packets(s) it seems the queue is to small.
So I found the problem that the nf_queue is full is strangely not the reason for the lack of performance that can be turned off with daq-var queue_maxlen=8192 quite well or delay.The problem seems to be a kernel limit / bug that limits the speed of the queue the tests with larger queue and only a third of the Snort rules brought no improvement. What brought success however is to divide the traffic on several Queues on and then Snort individually to hand over I solved it in such a way:
nft add table inet snort
nft 'add chain inet snort IPS { type filter hook forward priority filter ; }'
nft insert rule inet snort IPS counter queue num 4-6 bypass
Then pass queues 4, 5 and 6 individually to Snort (e.g. -i '5' -i '6'... -i 'x' or adding it to snort.lua) and start Snort with the -z 3 parameters in the command line. You can use more than 3 queues to increase the performance but not more than one per core. For this the queue_maxlen should be set high (I use 8192 but this could be too much) and the snaplen should be set to ~64000. But there is still a disadvantage servers that use only one connection are limited to the throughput of one queue, for me the limit was max 75 Mbit. Oh and tcp-segmentation-offload should also be disabled for the network cards with the ethtool (ethtool -K eth(x) tso off).
1 Like
I'm still trying to get data to pass through the queue on my N5105 box. Everything seems to work fine in a VM, but once I put it on the box with Intel I-226 NICs, it either locks up or nothing passes through Snort...
In any case, here are some pieces for you to make life easier. Put this in /etc/snort/snort-table.sh:
#!/bin/sh
verbose=false
nft list tables | grep -q 'snort' && nft flush table inet snort
nft -f - <<TABLE
table inet snort {
chain IPS {
type filter hook forward priority filter; policy accept;
counter queue flags bypass to 4-6
# meta l4proto tcp counter queue flags bypass to 4
# meta l4proto udp counter queue flags bypass to 5
# meta l4proto != { tcp, udp } counter queue flags bypass to 6
}
}
TABLE
$verbose && nft list table inet snort
exit 0
Point to it in /etc/config/firewall:
config include
option enabled '1'
option type 'script'
option path '/etc/snort/snort-table.sh'
option fw4_compatible '1'
Now fw4 reload and reboots will re-initialize your snort table (or create it from scratch). Whenever you change the script, just do another reload.
2 Likes
Thanks for your Script I will insert it on occasion times with first I'm glad that it runs. I had the problem with letting through once but I don't know what it was I think it was an enabled network card option I think it was the software flow option that automatically enables large-receive-offload. Generally generic-receive-offload, large-receive-offload and tcp-segmentation-offload must be disabled all three had visible influence on performance and function in my tests.
The snaplen is also important according to https://github.com/snort3/libdaq/blob/master/modules/nfq/README.nfq.md the Packets will come up from the kernel defragmented.
@efahl has you enabled Ipv6? If so have you added the Ipv6 traffic to the queue? I ask because I saw an example with Iptables and there the Ipv6 traffic had to be added with a second command to the same queue number.
You don't have to modify the rules, simply use action_override in your ips section of /etc/snort/local.lua
Example:
ips = {
-- mode = tap,
mode = inline,
variables = default_variables,
action_override = 'drop',
include = RULE_PATH .. '/snort.rules',
}
EDIT: AH! I see xxxx already suggested this.
Yes, we're running both through our queues. The nftables inet table type is the way they merged both v4 and v6 into the same rule tables. This was one of the big improvements over iptables, you only had to write a rule once if it was not IP family-specific.
table inet snort -> dual stack IPv4/IPv6 table
table ip snort -> IPv4-only table
table ip6 snort -> IPv6-only table
So, we use inet and any generic rules will pass both.
You have to be careful when writing specific rules, though, as some match clauses are v4 or v6 specific. Sometimes it's obvious:
ip saddr 202.3.4.5/32 udp port 22 ...
sometimes not.
Now I'm wondering if Snort cares about the distinction? It must, since it uses header information in some of the pattern matching. How up-to-date is it with respect to IPv6?
Actually Snort should have no problem with ipv6 because it could already Snort 2 but it may be that it needs adjustments in the snort.lua. The problem is the documentation is a disaster and is also partially no longer correct alone between the version used in the stable branch and the version used in the development branch of openwrt, there have been significant changes. In general I would try to pass the most important options in the command line because they override the options defined in the snort.lua here is my current start line I use: snort -q -c "/etc/snort/snort.lua" -i "4" -i "5" -i "6" --daq-dir /usr/lib/daq --daq nfq -Q -z 3 -s 64000 --daq-var queue_maxlen=8192
EDIT: I believe I have it running but snort isn't doing anything as far as I can tell
- I see very tiny CPU usage with show kernel threads.
- My rule to match ICMP ping isn't even getting tripped.
# cat /etc/snort/rules/test.rules
alert icmp any any <> any any (msg:"TEST ALERT"; icode:0; itype:8; sid:10000010; rev:001;)
System is RPi4. Internal NIC eth0 is LAN facing and USB NIC eth1 is WAN facing.
Running snort like this:
# snort -c /etc/snort/snort.lua --tweaks local
--------------------------------------------------
o")~ Snort++ 3.1.62.0
--------------------------------------------------
Loading /etc/snort/snort.lua:
Loading homenet.lua:
Finished homenet.lua:
Loading snort_defaults.lua:
Finished snort_defaults.lua:
Loading local.lua:
Finished local.lua:
snort
ssh
host_cache
pop
so_proxy
stream_tcp
mms
smtp
gtp_inspect
packets
dce_http_proxy
alert_fast
cip
ips
stream_icmp
hosts
normalizer
binder
wizard
appid
js_norm
file_id
http2_inspect
http_inspect
stream_udp
ftp_data
ftp_server
search_engine
port_scan
dce_http_server
dce_tcp
dce_smb
iec104
telnet
ssl
sip
rpc_decode
netflow
modbus
host_tracker
stream_user
stream_ip
process
back_orifice
classifications
dnp3
active
trace
ftp_client
decode
alerts
stream
references
daq
arp_spoof
output
network
dns
dce_udp
imap
file_policy
s7commplus
stream_file
Finished /etc/snort/snort.lua:
Loading file_id.rules_file:
Loading file_magic.rules:
Finished file_magic.rules:
Finished file_id.rules_file:
Loading rules/snort.rules:
Finished rules/snort.rules:
--------------------------------------------------
ips policies rule stats
id loaded shared enabled file
0 40127 0 40127 /etc/snort/snort.lua
--------------------------------------------------
rule counts
total rules loaded: 40127
text rules: 40127
option chains: 40127
chain headers: 1694
flowbits: 694
flowbits not checked: 83
--------------------------------------------------
port rule counts
tcp udp icmp ip
any 1786 380 457 288
src 1208 156 0 0
dst 5060 920 0 0
both 109 48 0 0
total 8163 1504 457 288
--------------------------------------------------
service rule counts to-srv to-cli
bgp: 5 1
dcerpc: 573 496
dhcp: 19 5
dnp3: 0 6
dns: 268 104
drda: 5 0
file: 275 284
ftp: 193 21
ftp-data: 561 8639
gopher: 0 1
http: 14058 11590
http2: 14058 11590
http3: 14058 11590
ident: 1 0
imap: 612 8889
irc: 40 14
ircd: 9 3
java_rmi: 51 3
kerberos: 34 6
ldap: 42 6
ldp: 1 0
modbus: 34 10
mysql: 67 7
netbios-dgm: 2 2
netbios-ns: 8 4
netbios-ssn: 809 541
netware: 2 0
nntp: 2 2
ntp: 36 7
openvpn: 16 16
pop3: 571 8893
postgresql: 8 0
printer: 3 0
radius: 3 2
rdp: 3 8
rtmp: 1 4
rtp: 1 1
rtsp: 17 2
sip: 338 44
smtp: 7875 513
snmp: 46 9
ssdp: 13 0
ssh: 10 4
ssl: 173 202
sunrpc: 118 9
syslog: 4 0
teamview: 1 2
telnet: 55 15
tftp: 11 6
vnc: 1 1
vnc-server: 12 10
total: 55103 63562
--------------------------------------------------
fast pattern groups
src: 486
dst: 1590
any: 8
to_server: 127
to_client: 92
--------------------------------------------------
search engine (ac_bnfa)
instances: 1261
patterns: 133885
pattern chars: 3092111
num states: 2337128
num match states: 346503
memory scale: MB
total memory: 76.6849
pattern memory: 8.05497
match list memory: 40.5348
transition memory: 27.9412
fast pattern only: 89388
appid: MaxRss diff: 0
appid: patterns loaded: 300
--------------------------------------------------
nfq DAQ configured to inline.
Commencing packet processing
++ [0] 4
/etc/snort/homenet.lua
HOME_NET = [[ 10.9.1.0/24 10.9.2.0/24 10.9.3.0/24 ]]
EXTERNAL_NET = "!$HOME_NET"
/etc/snort/local.lua
snort = {}
snort["-Q"] = true
ips = {
mode = inline,
variables = default_variables,
action_override = 'reject',
--action_override = 'drop',
include = RULE_PATH .. '/snort.rules',
}
daq = {
module_dirs = {
'/usr/lib/daq',
},
inputs = { '4' },
modules = {
{
name = 'nfq',
mode = 'inline',
variables = { 'device=eth1' } -- eth1 is wan interface
}
}
}
-- To log to a file, uncomment the below and manually create the dir defined in output.logdir
output.logdir = '/mnt/mmcblk0p3'
alert_fast = {
file = true,
packet = false,
}
--search_engine = { search_method = "hyperscan" }
--detection = { hyperscan_literals = true, pcre_to_regex = true }
normalizer = {
tcp = {
ips = true,
}
}
file_policy = {
enable_type = true,
enable_signature = true,
rules = {
use = {
verdict = 'log', enable_file_type = true, enable_file_signature = true
}
}
}
For me the rule works but I also use the Snort version from the stable branch was there perhaps any changes to Snort that were not taken into account?
Oh I see a problem you have not configured snaplen the packages come defragmented from the kernel there is a snaplen of ~64000 recommended. Add -s 64000 as parmeter to the snort executable.
1 Like
Thanks, I replied in the other thread... let's keep our conversation there to simplicity's sake.