IPS mode of snort3 is not dropping traffic

Yes that is clear. The Postrouting hook snort queue works with activated Generic Receive offload and tcp-segmentation-offload much faster I reach now with a single connection now almost my full bandwidth of 100 Mbit and the problems why the options should be disabled should not exist with Nfq because that works differently than afpacket what the article referred to then.
But better test also with activated reputation blacklist if the drop works correctly with Http/Https.

I have tested again something the fastest configuration is the postrouting hook with a priority of 300 there snort behaves as if it were bound with pcap to the Wan device, but is able to block on the idea I came after I have read here https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks#Priority_within_hook that the connection tracking helper work there. The WanDevice should be specified in the Snort.conf ( variables = { 'device=eth0' }, ) otherwise I could observe in my tests the strange behavior that one device was blocked on another but not or only partially blocked. Here is the modified queue script:

#!/bin/sh

verbose=false

nft list tables | grep -q 'snort' && nft flush table inet snort

nft -f - <<TABLE
    table inet snort {
        chain IPS {
            type filter hook postrouting priority 300; policy accept;

            counter  queue flags bypass to 4-7

        }
    }
TABLE

$verbose && nft list table inet snort

exit 0

#!/bin/sh

verbose=false

nft list tables | grep -q 'snort' && nft flush table inet snort

nft -f - <<TABLE
    table inet snort {
        chain IPS {
            type filter hook postrouting priority 300; policy accept;

	    ct state invalid drop;

# Add here accept or drop rules to bypass snort or drop traffic that snort not should see
# Note that if nat is enabled, snort will only see the address of the outgoing device for outgoing traffic, 
# for example for wan port the wan ip address or if you are using vpn the device address of the virtual adapter

		 counter  queue flags bypass to 4-6


        }
    }
TABLE

$verbose && nft list table inet snort

exit 0

I have slightly revised the script again the snaplen should be better 65531

Are you guys still testing setups? I am happy to test out some stuff when ready.

@xxxx - I tested your script one post above using the following. I got 165 Mbps download where running without snort is 950-1000 Mbps. Do I need to run any ethtool tweaks or modify anything?

My WAN NIC is eth1 by the way.

# ethtool -k eth1 | grep receive-offload
generic-receive-offload: on
large-receive-offload: off [fixed]

My /etc/snort/ok.lua

ips = {
  mode = inline,
  variables = default_variables,
  action_override = 'block',
filesystem
  include = RULE_PATH .. '/snort.rules',
}

output.logdir
output.logdir = '/mnt/mmcblk0p3'
alert_fast = {
	file = true,
	packet = false,
}

normalizer = {
  tcp = {
    ips = true,
  }
}

file_policy = {
  enable_type = true,
  enable_signature = true,
  rules = {
    use = {
      verdict = 'log', enable_file_type = true, enable_file_signature = true
    }
  }
}

And:

# snort -c "/etc/snort/snort.lua" -i "4" -i "5" -i "6" -i "7" --daq-dir /usr/lib/daq --daq nfq -Q -z 4 -s 64000 --daq-var queue_maxlen=8192 --daq-var device=eth1 --tweaks ok

Activate for both network ports (wan, lan) also tso: ethtool -K eth0 (1) gro on tso on and change the s parameter (snaplen) from 64000 to 65531. But I'm afraid your Cpu is too weak for much more throughput I could increase the throughput on one queue also only about 25 percent this way.

No I don't test anymore it doesn't make sense to put the queue somewhere else because the performance is worse I have done all reasonable configurations what makes sense for example to exclude the vpn traffic by accept rule before the queue or the traffic if you have different internal networks (snort also controls the traffic between the internal networks). But these are adjustments that everyone must perform for themselves.

//edit I'm not sure if your file_policy option makes sense since Snort only says yes or no to the passing traffic and can't manipulate anything itself.

@xxxx - I got x86 hardware based on Intel N95 CPU and want to revisit this.

eth0 = LAN
eth1 = WAN

I compiled in kmod-nfnetlink-queue and kmod-nft-queue

What is the setting I need to tweak with ethtool?

# ethtool -k eth1 | grep receive-offload
generic-receive-offload: on
large-receive-offload: off [fixed]

I ran your script from post#36.

Then I ran:

snort -c "/etc/snort/snort.lua" -i "4" -i "5" -i "6" -i "7" --daq-dir /usr/lib/daq --daq nfq -Q -z 4 -s 65531 --daq-var queue_maxlen=8192 --daq-var device=eth1

I am getting my full download bandwidth of around 650-700 Mbps. The CPU is a quad core and one of them hits >99% during the download test (other 3 are 30-40%).

So this is a win!

Next question is how to adjust the snort3 package config files to mimic this without the command line.

My notes indicate that when using the nft queues (nfq DAQ), you don't need to change the NIC offload settings at all; it's only needed when you use the IP filter packet path (i.e., afpacket DAQ).

But just in case, here's what I've got:

wan=$(uci get network.wan.device)
ethtool -K $wan   gro off   lro off   tso off  2> /dev/null

Where
tso = tcp-segmentation-offload
gro = generic-receive-offload
lro = large-receive-offload

I've been picking away at this, see post #15 for (still) current state of things... I'll be sure to bother you when I get something worth testing. :grin:

1 Like

Sounds good please let me know

Generic-receive-offload (GSO) and TCP Segmentation Offload (TSO) should be on, large-receive-offload (LRO) must be off.
Entering the parameters in the config is not a good idea because command line parameters are given priority and overwrite the parameters in the config, not to mention any errors caused by forgotten or incorrect characters. It is better to modify the service file, maybe you could extend it so that you only have to enter the options under /etc/config/snort like as Efahl's suggestion.

Exactly, I think that's the plan all around. Put all the settings possible in the config, then have the init.d script generate configs and/or modify its internal command line, so all of the stuff we've been doing on these two big threads boils down to 1) change config (cli or LuCI if we get really energetic), 2) run /etc/init.d/snort start, and then general users won't have to go through all the discovery steps we've been discussing here.

There's also the part about downloading rules, which is how I got started on all of this. I took @darksky's original script from the wiki and have "tuned it up" so that you can set uci snort.snort.oinkcode="blah" and if your code is valid, the script downloads and "installs" (symlinks right now) your subscription rules, in a location set in local.lua, and if no oinkcode exists, then it does the same with the community rules. (The update-rules script also has a testing mode to generate a hard-coded set of rules that you can easily test with just ping.)

Those sound like nice additions. I am thinking that passing a more complex command line is a nice option as well as coding it up in a config file. We could just have something like a SnortArgs= in a file that would get expanded by the init script.

Are you sure about TSO on? For my WAN facing eth1 it is off and performance is amazing.

generic-receive-offload: on
tx-tcp-segmentation: off
large-receive-offload: off [fixed]

EDIT: when I try to set tos to on it does not seem to be allowed:

# ethtool -K eth1 tso on gro on lro on
Cannot change large-receive-offload
Could not change any device features

# ethtool -k eth1 | grep tcp          
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [requested on]
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: off [requested on]
	tx-tcp6-segmentation: off [requested on]

I can't find my notes about the tso setting. I'm sure it's somewhere... As I recall, it was another "bug fix" that caused fragmented packets to be passed to snort, which then tried to match on disassembled pieces and snort would just pass the data without proper inspection. And, as far as I can tell, none of this is needed with nfq daq, as it delivers fully assembled packets by virtue of the queue mechanism.

In any case, if the setting is [fixed] then it's apparently hard coded into the driver or hardware, so you won't be able to change it.

Here are some explicit references to ethtool use in afpacket docs: https://github.com/snort3/libdaq/blob/master/modules/afpacket/README.afpacket.md#interface-preparation

But no equivalent mention in the nfq docs: https://github.com/snort3/libdaq/blob/master/modules/nfq/README.nfq.md

And some old discussion I had stashed that does mention segmentation in various forms: https://s3.amazonaws.com/snort-org-site/production/document_files/files/000/000/067/original/packet-offloading-issues.pdf

Oh, and that means that you also need to jack up the snaplen value as high as possible, i.e., 64k, as mentioned in that above link to the nfq README. From the snort3 documentation:

• int daq.snaplen = 1518: set snap length (same as -s) { 0:65535 }

So, CLI would be -s 65535 and in config files the equivalent would be in the daq section of local.lua:

daq = {
  snaplen = 65535,
  ...

EDIT: I did this on an x86 VM that was configured with only(!) 512 MB of RAM, and it locked the VM on snort startup. I had to back it way down to get snort to start, but then set the VM to have 1 GB RAM and it was ok with the full 64k snap buffer.

Yes that is good. The problem will be to add exception rules to the queue script via /etc/config/snort because depending on whether masquerading is enabled or disabled the source ip address sent to the queue will change therefore the exception rules to be created will also change. If masquerading is enabled, Snort sees only the source Ip address of the outgoing network card. But when masquerading is disabled Snort sees the original Ip address. This is a problem in that you can enable masquerading between internal networks as well, so exception rules between internal networks would no longer work in one direction and would have to be changed. Which brings the next problem with itself since the outgoing address is always one of the device addresses of Openwrt for example 192.168.1.1 would exclude such a rule also everything which comes from 192.168.1.1.
Better use a snaplen = 65531 if I remember correctly it will skip too small packets without content.

If I remember the tests correctly it was a few weeks ago TSO had an influence on the upload and the simultaneous handling of several simultaneous connections became better.

@xxxx @efahl - do either of you guys run snort3 on x86-64? If so, pull https://github.com/openwrt/packages/pull/21471 and modify the Makefile for snort3 as I show in the PR. It will build with hyperscan. Interested for you guys to test with it. If you don't want build your own packages, I can share the ones I built with you.

Interesting it is, problem is I use the stable Openwrt line and it is not compatible with the builds for the Development Snapshot builds. And to build a package for the stable Openwrt line now is nonsensical because a new stable version is already on the horizon and the Snort version in the current stable version is totally outdated anyway. If you are sure that the build is also on the release candidate you could share the package and I update to v23.05.0-rc2 and test it there. I hope the RC2 doesn't have the problem with my wlan modules which I had with a development snapshot build some time ago.

According to this, the snort version on that branch corresponding to v23.05.0-rc2 should be current, just a few sub versions behind snapshot.

I just checkout out openwrt-23.05 and built hyperscan for it rather than for snapshot. This way you can just install the two packages (snort3 and hyperscan-runtime) and have all the other packages matching that branch of packages.

Target system (x86)
Subtarget (x86_64)
Target Profile (Generic x86/64)

Here is a link to the two packages. Let me know how it goes!

@efahl - let me know if you want packages for snapshot and I can push them.