Router no Internet when using iptables for policy routing

Installing and Using OpenWrt
1

My router has multiple WAN interfaces and I'd like to use iptables to load balance the traffic among them.

My goal is to load balance clients traffic among pppoe-wan_a to pppoe-wan_f, and have router use eth4.160

I put the following scripts in the /etc/firewall.user:

# Load balancing

# create routing tables
ip rule del table 100
ip rule del table 110
ip rule del table 120
ip rule del table 130
ip rule del table 140
ip rule del table 150
ip rule del table 160
ip route add default dev pppoe-wan_a table 100
ip route add default dev pppoe-wan_b table 110
ip route add default dev pppoe-wan_c table 120
ip route add default dev pppoe-wan_d table 130
ip route add default dev pppoe-wan_e table 140
ip route add default dev pppoe-wan_f table 150
ip route add default dev eth4.160 table 160

# link mark with routing table and set priority
ip rule add fwmark 100 table 100 prio 33000
ip rule add fwmark 110 table 110 prio 33000
ip rule add fwmark 120 table 120 prio 33000
ip rule add fwmark 130 table 130 prio 33000
ip rule add fwmark 140 table 140 prio 33000
ip rule add fwmark 150 table 150 prio 33000
ip rule add fwmark 160 table 160 prio 33000
ip route flush cache

# load balancing over 6 WAN
iptables -t mangle -N wan_load_balancing
iptables -t mangle -A wan_load_balancing -d 10.0.0.0/8 -j ACCEPT
iptables -t mangle -A wan_load_balancing -d 172.16.0.0/12 -j ACCEPT
iptables -t mangle -A wan_load_balancing -d 192.168.0.0/16 -j ACCEPT
iptables -t mangle -A wan_load_balancing -j CONNMARK --restore-mark
iptables -t mangle -A wan_load_balancing -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A wan_load_balancing -m conntrack --ctstate NEW -m statistic --mode nth --every 6 --packet 0 -j MARK --set-mark 100
iptables -t mangle -A wan_load_balancing -m conntrack --ctstate NEW -m statistic --mode nth --every 6 --packet 1 -j MARK --set-mark 110
iptables -t mangle -A wan_load_balancing -m conntrack --ctstate NEW -m statistic --mode nth --every 6 --packet 2 -j MARK --set-mark 120
iptables -t mangle -A wan_load_balancing -m conntrack --ctstate NEW -m statistic --mode nth --every 6 --packet 3 -j MARK --set-mark 130
iptables -t mangle -A wan_load_balancing -m conntrack --ctstate NEW -m statistic --mode nth --every 6 --packet 4 -j MARK --set-mark 140
iptables -t mangle -A wan_load_balancing -m conntrack --ctstate NEW -m statistic --mode nth --every 6 --packet 5 -j MARK --set-mark 150
iptables -t mangle -A wan_load_balancing -j CONNMARK --save-mark
iptables -t mangle -A PREROUTING -j wan_load_balancing

In this setup the custom tables are put behind the main table and no default route is set in the main table:

0:      from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default 
33000:  from all fwmark 0x64 lookup 100 
33000:  from all fwmark 0x6e lookup 110 
33000:  from all fwmark 0x78 lookup 120 
33000:  from all fwmark 0x82 lookup 130 
33000:  from all fwmark 0x8c lookup 140 
33000:  from all fwmark 0x96 lookup 150 
33000:  from all fwmark 0xa0 lookup 160

Problem: the load balancing works fine for the clients, but the router does not have Internet access (Network unreachable)

If I put a default route in the main table, then only that route is used, so load balancing is broken.

If I put a default route in the main table, and move the custom table before the main:

0:      from all lookup local 
3000:  from all fwmark 0x64 lookup 100 
3000:  from all fwmark 0x6e lookup 110 
3000:  from all fwmark 0x78 lookup 120 
3000:  from all fwmark 0x82 lookup 130 
3000:  from all fwmark 0x8c lookup 140 
3000:  from all fwmark 0x96 lookup 150 
3000:  from all fwmark 0xa0 lookup 160
32766:  from all lookup main 
32767:  from all lookup default

Problem: the router works fine but the clients no longer has Internet.

I tried to add iptables -t mangle -A OUPUT -j MARK --set-mark 160 but the router still said network unreachable

2

Using options ip4table and ip6table can significantly simplify your config:
https://openwrt.org/docs/guide-user/network/routing/pbr#pbr_with_netifd

1 Like
3

I just figured out that rules in mangle table output chain would not work if there's no default route in the main table.

4

I made it work finally. Please correct me if I made a mistake.

First, to make the Internet work for the router, main table must have a default route, it can be a dummy, but it has to be presented, otherwise the output chain could not work.

Second, all custom rules must be placed before the main rule (32766), so the custom rules catch before the default route in the main table does.

The problem here is the client no longer has Internet access, because the incoming packets are marked and catched by the custom rule and routed by the custom table. Since the table only contains one default route point to a wan interface, the router cannot route the packet back to its client.

The solution is to add rules tell the router to use the main table for routing packet back to the clients, in my scripts eg.:

# link mark with routing table and set priority

ip rule add from all to 10.0.0.0/8 table main prio 1000
ip rule add from all to 192.168.0.0/16 table main prio 1001
ip rule add from all to 172.16.0.0/12 table main prio 1002

# The above three lines are critical

ip rule add fwmark 100 table 100 prio 1003
ip rule add fwmark 110 table 110 prio 1004
ip rule add fwmark 120 table 120 prio 1005
ip rule add fwmark 130 table 130 prio 1006
ip rule add fwmark 140 table 140 prio 1007
ip rule add fwmark 150 table 150 prio 1008
ip rule add fwmark 160 table 160 prio 1009
ip route flush cache

The final ip rule prints:

0:      from all lookup local 
1000:   from all to 10.0.0.0/8 lookup main 
1001:   from all to 192.168.0.0/16 lookup main 
1002:   from all to 172.16.0.0/12 lookup main 
1003:   from all fwmark 0x64 lookup 100 
1004:   from all fwmark 0x6e lookup 110 
1005:   from all fwmark 0x78 lookup 120 
1006:   from all fwmark 0x82 lookup 130 
1007:   from all fwmark 0x8c lookup 140 
1008:   from all fwmark 0x96 lookup 150 
1009:   from all fwmark 0xa0 lookup 160 
32766:  from all lookup main 
32767:  from all lookup default
5

The local system can reach the internet with a totally empty main table.
It can use a routing rule to lookup one of the tables with a default route.

Also, you don't need to add any routes, or create rules for local subnets.
These manual actions are redundant as netifd can do it automatically.