Hi, I am relatively new to the LEDE world, but I have been spending a lot of time over the last few weeks configuring a nice little networking device as my main router with some added features for helping me bypass the GFW here in China. For the most part I am experiencing some strange behaviours with iptables when used in conjuction with ipset.
I use dnsmasq-full and it's ipset functionality to maintain two ipset lists, one for ip's to proxy and one for ip's to route locally. This functionality works fine. However as the lists grow (to say about 20 or so entries) iptables routing from WAN to LAN just stops and the rules to drop packets, and do NAT forwarding simply stop. If I flush the ipset list then it starts working again! Strangely routing from LAN to WAN always works as expected and the required connections are directed to the proxy.
Although this is not a major issue, it's still not right. But I am at a loss where to start looking or even to report the issue! Is it iptables, ipset, or the kernel module kmod-ipt-ipset?
Any suggestions on things to try would be appreciated? I have ordered some more SSD drives so I can try to see if the problem exists in the latest snapshot and even in a completely different OS.
Was trying to use openvpn-policy-routing, that'd fail, so I cut it down to the smallest reproducible testcase which exhibits the issue.
I attempted to try to diagnose this by adding packet tracing, but it internally uses nftables' nftrace flag, which doesn't seem to be functional. Compiled xt_TRACE.ko / libxt_TRACE.so, but no result.
Updated nftables to 0.7 to get access to the trace monitor, but no activity whatsoever.
With appropriate ipset entry in dnsmasq-full, I too see zero throughput after a bunch of activity.
Will be attempting to sort out tracing before going any further with this.
Tracing got enabled, ended up working through the problem. Not entirely sure what the problem was, it started working while I was mucking about with the routing.
Had some decent luck with the following script - will let it run over the next few days to see if it's stable - if so I'll look at making a package out of it.
Run "./test-script.sh on" to turn it on "./test-script.sh off" to turn it off.
You'll need to fix up the WAN_IF and VPN_IF, or adapt the whole thing if you've got something more complex.
ARG=$1
export MARK_MASK=0xF00
export WAN_TABLE=156
export WAN_MARK=0x400
export WAN_PRIO=900
export WAN_IF=eth1
export WAN_IPSETPREFIX=wan
export VPN_TABLE=157
export VPN_MARK=0x800
export VPN_PRIO=901
export VPN_IF=tun777
export VPN_IPSETPREFIX=vpn
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > $f; done
# clear the whole firewall and give us a clean slate
nft flush ruleset
/etc/init.d/firewall restart
ip rule del table $WAN_TABLE
ip route flush table $WAN_TABLE
ip rule del table $VPN_TABLE
ip route flush table $VPN_TABLE
# Try to destroy any lists that are already present
for i in `seq 1 3` ; do
for l in $(ipset list | awk '/^Name/ {print $2}') ; do ipset flush $l >/dev/null 2>&1; ipset destroy $l >/dev/null 2>&1; done
done
ip route flush cache
function setup_iface() {
local interface=$1
local ipset=$2
local mark=$3
local table=$4
local prio=$5
ip rule add fwmark $mark table $table prio $prio
if [ -z "$(ip route show | grep $interface)" ] ; then
# if the interface isn't present, then mark this table as unreachable
ip route add unreachable default table $table
else
# copy across all the routes from the main table which matches this interface
ip route show table main | grep $interface | while read route ; do
ip route add $route table $table
done
fi
ipset create ${ipset}_remotenet hash:net
iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_remotenet dst -j MARK --set-mark $mark/$MARK_MASK
ipset create ${ipset}_remoteport bitmap:port range 0-65535
iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_remoteport dst -j MARK --set-mark $mark/$MARK_MASK
ipset create ${ipset}_localnet hash:net
iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_localnet src -j MARK --set-mark $mark/$MARK_MASK
ipset create ${ipset}_localport bitmap:port range 0-65535
iptables -t mangle -A PREROUTING -i br-lan -m set --match-set ${ipset}_localport src -j MARK --set-mark $mark/$MARK_MASK
}
if [ "$ARG" = "on" ] ; then
setup_iface $VPN_IF $VPN_IPSETPREFIX $VPN_MARK $VPN_TABLE $VPN_PRIO
setup_iface $WAN_IF $WAN_IPSETPREFIX $WAN_MARK $WAN_TABLE $WAN_PRIO
/etc/init.d/dnsmasq restart
######################
# Set up the ipset entries here!
ipset add ${WAN_IPSETPREFIX}_remoteport 443 >/dev/null 2>&1 # Force SSL traffic through the WAN instead of the VPN
fi
echo; echo; iptables -nvL PREROUTING -t mangle --line; echo; iptables -nvL FORWARD -t mangle --line; echo; ipset list; echo
# To test on device being routed:
# echo "HTTP: $(wget -q -O- icanhazip.com), HTTPS: $(wget -q -O- https://icanhazip.com)"