Routing problem with 2102rc

Installing and Using OpenWrt Network and Wireless Configuration
1

I have custom vpn routing configuration which works very nicely in 19.07. Applying the same config to 2102rc2 results in misrouting, and I am not sure why.

The config is simple. It basically points unmarked packets through wg0, and everything else is routed through the normal mechanisms. This works fine in 19.07

  # UCI config
network.throughTunnel=route
network.throughTunnel.interface='wg0'
network.throughTunnel.target='0.0.0.0'
network.throughTunnel.netmask='0.0.0.0'
network.throughTunnel.table='200'

network.unmarkedThroughTable200=rule
network.unmarkedThroughTable200.mark='0x888'
network.unmarkedThroughTable200.invert='1'
network.unmarkedThroughTable200.lookup='200'

  # UCI config looks like this in shell
root@G:~# ip rule list
0:      from all lookup local
1:      not from all fwmark 0x888 lookup 200
32766:  from all lookup main
32767:  from all lookup default
root@G:~# ip route show table 200
default dev wg0 proto static scope link

On 21.02-rc2 I can see the ping from the LAN going out to the internet through wg0, and the ping response coming back. The MARKed response should be routed back to the pi (on br-lan), but is instead incorrectly being routed back into wg0.

  # OK  Original ping from pi (on LAN) to internet host via OUT=wg0
Tue Jun 15 10:04:02 2021 kern.warn kernel: [51732.719422] FwdBasicInfo IN=br-lan OUT=wg0 MAC=xx:xx:xx SRC=172.29.0.80 DST=172.217.165.14 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=2189 DF PROTO=ICMP TYPE=8 CODE=0 ID=9455 SEQ=1

   # MISROUTED Response to ping - Note MARK on packet. Should be OUT=br-lan
Tue Jun 15 10:04:02 2021 kern.warn kernel: [51732.911602] FwdBasicInfo IN=wg0 OUT=wg0 MAC= SRC=172.217.165.14 DST=172.29.0.80 LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=41118 PROTO=ICMP TYPE=0 CODE=0 ID=9455 SEQ=1 MARK=0x888

Any thoughts, or things I should check?

Note that I have the ip-tiny package installed

2

try ip-full?

3

How do you set the fwmark 0x888 ?

4

MARK is set for all forwarded output from wg0 (the tunneled packets coming out of wireguard). The rule is configured through uci and the iptables output is shown below in FORWARD. The ping response is one of these

fwmark is also set in several places. It is set by wireguard (for their stream of encrypted udp packets). It is also set for all OUTPUT of the router (so that connections originating at the router are not tunneled).

root@G:~# iptables -t mangle -vL
Chain PREROUTING (policy ACCEPT 1714 packets, 256K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1305 packets, 135K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  wg0    any     anywhere             anywhere             /* !fw3: markOutputFromWG */ MARK set 0x888
    0     0 TCPMSS     tcp  --  any    eth0.2  anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  eth0.2 any     anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 1132 packets, 187K bytes)
 pkts bytes target     prot opt in     out     source               destination
 1132  187K MARK       all  --  any    any     anywhere             anywhere             /* !fw3: markOutputFromRouter */ MARK set 0x888

Chain POSTROUTING (policy ACCEPT 1132 packets, 187K bytes)
 pkts bytes target     prot opt in     out     source               destination
5

I haven't tried ip-full. I assume I would need to first uninstall ip-tiny.

Given that "the ip rules encoded in /etc/config/network are processed by netifd C
code directly, they're not translated into busybox ip calls". Would changing this package have an impact on routing done in the kernel?

6

fair point...

iptables-save -c | grep 888
1 Like
7

These are the only firewall lines that set a mark

root@mangoG:~# iptables-save -c | grep -i mark
[0:0] -A FORWARD -i wg0 -m comment --comment "!fw3: markOutputFromWG" -j MARK --set-xmark 0x888/0xffffffff
[1435:243228] -A OUTPUT -m comment --comment "!fw3: markOutputFromRouter" -j MARK --set-xmark 0x888/0xffffffff
8

This manual lookup seems good to me. The second "ip route get" means the packet should go to br-lan and out to the pi which sent the ping.

But somehow it heads back to wg0...

root@mangoG:~# ip route get 172.217.165.14
172.217.165.14 dev wg0 table 200 src 192.168.219.246 uid 0
    cache
root@mangoG:~# ip route get 172.29.0.80 mark 0x888
172.29.0.80 dev br-lan src 172.29.0.1 mark 0x888 uid 0
    cache
10

I'm not clear what this is trying to do, given that network.throughTunnel.table='200'

The network.lan.ip4table="1" bit did create a seperate routing table for the LAN

root@G:~# ip rule list
0:      from all lookup local
10000:  from 172.29.0.1 lookup 1
20000:  from all to 172.29.0.1/24 lookup 1
30000:  not from all fwmark 0x888 lookup 2
32766:  from all lookup main
32767:  from all lookup default
90004:  from all iif lo lookup 1

root@G:~# ip route list table 1
172.29.0.0/24 dev br-lan proto static scope link

root@G:~# ip route list table 2
Error: ipv4: FIB table does not exist.
Dump terminated

root@G:~# ip route list table 200
default dev wg0 proto static scope link
1 Like
11 
uci set network.@wireguard_wg0[0].route_allowed_ips="1"
uci -q delete network.@wireguard_wg0[0].allowed_ips
uci add_list network.@wireguard_wg0[0].allowed_ips="0.0.0.0/0"
uci set network.lan.ip4table="1"
uci set network.wg0.ip4table="2"
uci -q delete network.throughTunnel
uci set network.unmarkedThroughTable200.lookup="2"
uci set network.unmarkedThroughTable200.priority="30000"
uci commit network
/etc/init.d/network restart
12

I have a configuration which has worked very well for 19.07. It is simple, and works reliably over a variety of use cases.

I would first like to understand why that no longer works in 21.02-rc2. If I have done something wrong, I'd like to first understand what that is and then I will replace it.

13

Can't of anything specific that would change core routing logic. A couple of things you could try:

  • Did you enable any kind of flow offloading? If yes, try disabling it
  • Do you observe any effect when flushing the conntrack table? (echo f > /proc/net/nf_conntrack)
14

No, I've not tried to do any flow offloading. Also, nothing has changed in my configuration to use it in 21.02 All the routing config is above, which is pretty simple.

Interesting trick with conntrack. Thanks!

I have not been able to make any progress with using marked packets. Something seems to be wrong on 21.02, but I am not able to narrow it down. In any case I can change to do the routing based on the LAN source address of the packets, and that seems to work quite reliably. In fact it simplifies a few things