Strange iptables behaviour when used with ipset - x64 17.01.1

Hi, I am relatively new to the LEDE world, but I have been spending a lot of time over the last few weeks configuring a nice little networking device as my main router with some added features for helping me bypass the GFW here in China. For the most part I am experiencing some strange behaviours with iptables when used in conjuction with ipset.

I use dnsmasq-full and it's ipset functionality to maintain two ipset lists, one for ip's to proxy and one for ip's to route locally. This functionality works fine. However as the lists grow (to say about 20 or so entries) iptables routing from WAN to LAN just stops and the rules to drop packets, and do NAT forwarding simply stop. If I flush the ipset list then it starts working again! Strangely routing from LAN to WAN always works as expected and the required connections are directed to the proxy.

Although this is not a major issue, it's still not right. But I am at a loss where to start looking or even to report the issue! Is it iptables, ipset, or the kernel module kmod-ipt-ipset?

Any suggestions on things to try would be appreciated? I have ordered some more SSD drives so I can try to see if the problem exists in the latest snapshot and even in a completely different OS.


I can confirm that I'm seeing the same behaviour.

Was trying to use openvpn-policy-routing, that'd fail, so I cut it down to the smallest reproducible testcase which exhibits the issue.

I attempted to try to diagnose this by adding packet tracing, but it internally uses nftables' nftrace flag, which doesn't seem to be functional. Compiled xt_TRACE.ko /, but no result.
Updated nftables to 0.7 to get access to the trace monitor, but no activity whatsoever.

With appropriate ipset entry in dnsmasq-full, I too see zero throughput after a bunch of activity.
Will be attempting to sort out tracing before going any further with this.

Tracing got enabled, ended up working through the problem. Not entirely sure what the problem was, it started working while I was mucking about with the routing.

Had some decent luck with the following script - will let it run over the next few days to see if it's stable - if so I'll look at making a package out of it.

Run "./ on" to turn it on "./ off" to turn it off.
You'll need to fix up the WAN_IF and VPN_IF, or adapt the whole thing if you've got something more complex.


export MARK_MASK=0xF00

export WAN_TABLE=156
export WAN_MARK=0x400
export WAN_PRIO=900
export WAN_IF=eth1

export VPN_TABLE=157
export VPN_MARK=0x800
export VPN_PRIO=901
export VPN_IF=tun777

for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > $f; done

# clear the whole firewall and give us a clean slate
nft flush ruleset
/etc/init.d/firewall restart

ip rule del table $WAN_TABLE
ip route flush table $WAN_TABLE

ip rule del table $VPN_TABLE
ip route flush table $VPN_TABLE

# Try to destroy any lists that are already present
for i in `seq 1 3` ; do
for l in $(ipset list | awk '/^Name/ {print $2}') ; do ipset flush $l >/dev/null 2>&1; ipset destroy $l >/dev/null 2>&1; done

ip route flush cache

function setup_iface() {
    local interface=$1
    local ipset=$2
    local mark=$3
    local table=$4
    local prio=$5

    ip rule add fwmark $mark table $table prio $prio

    if [ -z "$(ip route show | grep $interface)" ] ; then
        # if the interface isn't present, then mark this table as unreachable
        ip route add unreachable default table $table
        # copy across all the routes from the main table which matches this interface
        ip route show table main | grep $interface | while read route ; do
            ip route add $route table $table

    ipset create ${ipset}_remotenet hash:net
    iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_remotenet dst -j MARK --set-mark $mark/$MARK_MASK

    ipset create ${ipset}_remoteport bitmap:port range 0-65535
    iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_remoteport dst -j MARK --set-mark $mark/$MARK_MASK

    ipset create ${ipset}_localnet hash:net
    iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_localnet src -j MARK --set-mark $mark/$MARK_MASK

    ipset create ${ipset}_localport bitmap:port range 0-65535
    iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_localport src -j MARK --set-mark $mark/$MARK_MASK

if [ "$ARG" = "on" ] ; then

    /etc/init.d/dnsmasq restart

    # Set up the ipset entries here!
    ipset add ${WAN_IPSETPREFIX}_remoteport 443 >/dev/null 2>&1 # Force SSL traffic through the WAN instead of the VPN

echo; echo; iptables -nvL PREROUTING -t mangle --line; echo; iptables -nvL FORWARD -t mangle --line; echo; ipset list; echo

# To test on device being routed:
# echo "HTTP: $(wget -q -O-, HTTPS: $(wget -q -O-"