Hi,
I am testing this on Netgear R7800 with 19.07.4.
There is a problem with some flows that are using default gateways from different routing tables.
This is a very similar topic to: Software flow offloading & custom routing table
The main table does not have a default route. Tables wan-isp1, wan-isp2, default have default routes.
G.G.G.G - my gateway for wan-isp2
E.E.E.E - my external IP from ISP2
L.L.L.L - my internal lan IP (on br-lan).
root@OpenWrt-S1:~# ip rule
0: from all lookup local
1000: from all lookup main
2000: from all fwmark 0xe lookup wan-isp1
3000: from all fwmark 0xf lookup wan-isp2
4000: from all fwmark 0x10 lookup wan-3g
32766: from all lookup main
32767: from all lookup default
root@OpenWrt-S1:~# iptables -t mangle -L -n -v | grep mark
...
812 54670 MARK all -- * * L.L.L.L 0.0.0.0/0 mark match 0x0 MARK set 0xf
+ marking for incoming traffic on wan-isp2 interface:
1337K 143M qos-marking-in-wan-isp2 all -- eth0.30 * 0.0.0.0/0 0.0.0.0/0
258K 30M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0 MARK set 0xf
root@OpenWrt-S1:~# iptables -t nat -L -n -v | grep 12345
0 0 SNAT tcp -- * * L.L.L.0/24 L.L.L.L tcp dpt:12345 /* !fw3:ISP2-port1 (reflection) */ to:L.L.L.1
0 0 DNAT tcp -- * * L.L.L.0/24 E.E.E.E tcp dpt:12345 /* !fw3: ISP2-port1 (reflection) */ to:L.L.L.L:12345
1147 68820 DNAT tcp -- * * 0.0.0.0/0 E.E.E.E tcp dpt:12345 /* !fw3: ISP2-port1 */ to:L.L.L.L:12345
There are no problems with software offloaded traffic that is using wan-isp1 and default routing tables - same gateway. Without SFO the whole policy routing working as expected. However when I activate SFO I have a routing issue:
00:21:26.178988 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [S], seq 1587392213, win 65535, options [mss 1460,sackOK,TS val 2304371643 ecr 0,nop,wscale 10], length 0
00:21:26.179190 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [S.], seq 835228213, ack 1587392214, win 65160, options [mss 1460,sackOK,TS val 1136620338 ecr 2304371643,nop,wscale 6], length 0
00:21:26.220322 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [.], ack 1, win 64, options [nop,nop,TS val 2304371685 ecr 1136620338], length 0
00:21:26.262656 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [P.], seq 1:248, ack 1, win 64, options [nop,nop,TS val 2304371726 ecr 1136620338], length 247
00:21:26.262822 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], ack 248, win 1015, options [nop,nop,TS val 1136620422 ecr 2304371726], length 0
00:21:26.264460 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [P.], seq 1:1504, ack 248, win 1015, options [nop,nop,TS val 1136620423 ecr 2304371726], length 1503
00:21:26.356839 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [P.], seq 1449:1504, ack 248, win 1015, options [nop,nop,TS val 1136620516 ecr 2304371726], length 55
00:21:26.511786 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [P.], seq 1:248, ack 1, win 64, options [nop,nop,TS val 2304371978 ecr 1136620338], length 247
00:21:26.511999 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], ack 248, win 1015, options [nop,nop,TS val 1136620671 ecr 2304371978,nop,nop,sack 1 {1:248}], length 0
00:21:26.604884 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], seq 1:1449, ack 248, win 1015, options [nop,nop,TS val 1136620764 ecr 2304371978], length 1448
00:21:26.763955 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [P.], seq 1:248, ack 1, win 64, options [nop,nop,TS val 2304372230 ecr 1136620338], length 247
00:21:26.764155 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], ack 248, win 1015, options [nop,nop,TS val 1136620923 ecr 2304372230,nop,nop,sack 1 {1:248}], length 0
00:21:27.096833 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], seq 1:1449, ack 248, win 1015, options [nop,nop,TS val 1136621256 ecr 2304372230], length 1448
00:21:27.280023 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [P.], seq 1:248, ack 1, win 64, options [nop,nop,TS val 2304372746 ecr 1136620338], length 247
00:21:27.280239 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], ack 248, win 1015, options [nop,nop,TS val 1136621439 ecr 2304372746,nop,nop,sack 1 {1:248}], length 0
00:21:28.088912 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], seq 1:1449, ack 248, win 1015, options [nop,nop,TS val 1136622248 ecr 2304372746], length 1448
00:21:28.279559 IP E.E.E.E.40090 > L.L.L.L.12345: Flags [P.], seq 1:248, ack 1, win 64, options [nop,nop,TS val 2304373738 ecr 1136620338], length 247
00:21:28.279731 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], ack 248, win 1015, options [nop,nop,TS val 1136622439 ecr 2304373738,nop,nop,sack 1 {1:248}], length 0
00:21:29.396478 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 60
00:21:29.396557 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 556
00:21:29.396672 IP L.L.L.L.12345 > E.E.E.E.40090: Flags [.], seq 1:1449, ack 248, win 1015, options [nop,nop,TS val 1136623555 ecr 2304373738], length 1448
00:21:29.396700 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 115
00:21:29.397700 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 72
00:21:29.397775 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 556
00:21:29.397790 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 72
00:21:29.397803 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 556
00:21:29.397816 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 72
00:21:29.398101 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 556
00:21:29.398127 IP M.M.M.M > L.L.L.L: ICMP host E.E.E.E unreachable, length 72
If I add a static route then everything is fine (however this in not the point of policy base routing):
root@OpenWrt-S1:~# ip r a E.E.E.E via G.G.G.G
00:34:16.544145 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [S], seq 1799817715, win 65535, options [mss 1460,sackOK,TS val 1318324052 ecr 0,nop,wscale 10], length 0
00:34:16.544392 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [S.], seq 593187034, ack 1799817716, win 65160, options [mss 1460,sackOK,TS val 3573750144 ecr 1318324052,nop,wscale 6], length 0
00:34:16.584548 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [.], ack 1, win 64, options [nop,nop,TS val 1318324092 ecr 3573750144], length 0
00:34:16.585239 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [P.], seq 1:248, ack 1, win 64, options [nop,nop,TS val 1318324093 ecr 3573750144], length 247
00:34:16.585399 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [.], ack 248, win 1015, options [nop,nop,TS val 3573750185 ecr 1318324093], length 0
00:34:16.587494 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [P.], seq 1:1503, ack 248, win 1015, options [nop,nop,TS val 3573750187 ecr 1318324093], length 1502
00:34:16.632688 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [.], ack 1503, win 67, options [nop,nop,TS val 1318324140 ecr 3573750187], length 0
00:34:16.632773 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [P.], seq 248:1517, ack 1503, win 67, options [nop,nop,TS val 1318324141 ecr 3573750187], length 1269
00:34:16.632789 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [P.], seq 1517:1574, ack 1503, win 67, options [nop,nop,TS val 1318324142 ecr 3573750187], length 57
00:34:16.632804 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [P.], seq 1574:1884, ack 1503, win 67, options [nop,nop,TS val 1318324142 ecr 3573750187], length 310
00:34:16.632819 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [P.], seq 1884:1999, ack 1503, win 67, options [nop,nop,TS val 1318324142 ecr 3573750187], length 115
00:34:16.633793 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [.], ack 1999, win 988, options [nop,nop,TS val 3573750234 ecr 1318324141], length 0
00:34:16.730691 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [P.], seq 1503:1965, ack 1999, win 1002, options [nop,nop,TS val 3573750331 ecr 1318324141], length 462
00:34:16.734132 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [P.], seq 1965:2766, ack 1999, win 1002, options [nop,nop,TS val 3573750334 ecr 1318324141], length 801
00:34:16.774235 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [.], ack 2766, win 73, options [nop,nop,TS val 1318324283 ecr 3573750331], length 0
00:34:16.775226 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [P.], seq 1999:2025, ack 2766, win 73, options [nop,nop,TS val 1318324285 ecr 3573750331], length 26
00:34:16.776340 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [F.], seq 2025, ack 2766, win 73, options [nop,nop,TS val 1318324285 ecr 3573750331], length 0
00:34:16.777220 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [P.], seq 2766:2790, ack 2026, win 1002, options [nop,nop,TS val 3573750377 ecr 1318324285], length 24
00:34:16.777734 IP L.L.L.L.12345 > E.E.E.E.59598: Flags [F.], seq 2790, ack 2026, win 1002, options [nop,nop,TS val 3573750378 ecr 1318324285], length 0
00:34:16.815554 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [R], seq 1799819741, win 0, length 0
00:34:16.815640 IP E.E.E.E.59598 > L.L.L.L.12345: Flags [R], seq 1799819741, win 0, length 0
Is this a limitation of SFO or a bug that potentially could be fixed?
Regards,
Peter