Routing two interfaces bypassing local loopback (should be simple, but…)

bolemo · January 18, 2023, 1:37pm

Hello,

I am trying to do something very simple, but it appears it is not so simple to implement after all…

I have a router x86_64 with the latest OpenWrt 22.03.
I have 2 WAN accesses: one through classical physical ethernet (DHCP, etc.), the other through a GRE tunnel.

To simplify, lets call IfA the interface carrying the connection through physical ethernet, with public ip IPA, and IfB the interface carrying the connection through the tunnel, with public ip IPB.

All works fine; my default route is through the tunnel IfB, and I can specifically choose from which WAN I want to go using routing rules if needed.

Now, I want to be able to ping IPA from IfB, and have the packets go through IfB, like this:
Echo Request to IPA -> IfB -> WAN -> IfA | Echo Request to IPB -> IfA -> WAN -> IfB
And the opposite:
Echo Request to IPB -> IfA -> WAN -> IfB | Echo Request to IPA -> IfB -> WAN -> IfA

I made the following routing changes:

root@OpenWrt:~# ip ru
0:	from IPA to IPB lookup 1000
0:	from IPB to IPA lookup 2000
10:	from all lookup local

root@OpenWrt:~# ip r show table 1000
default IfA-GW dev IfA proto static src IPA

root@OpenWrt:~# ip r show table 2000
default dev IfB proto static scope link src IPB mtu 1476

This allows to bypass the loopback and force the packets to go through WAN.

Then in firewall config, I added this:

config rule
	option name 'Test1'
	list proto 'all'
	list src_ip 'IPA'
	list dest_ip 'IPB'
	option target 'ACCEPT'
	option dest '*'

config rule
	option name 'Test2'
	list proto 'all'
	list src_ip 'IPA'
	list dest_ip 'IPB'
	option target 'ACCEPT'
	option src '*'

config rule
	option name 'Test3'
	list proto 'all'
	list src_ip 'IPB'
	list dest_ip 'IPA'
	option target 'ACCEPT'
	option dest '*'

config rule
	option name 'Test4'
	list proto 'all'
	list src_ip 'IPB'
	list dest_ip 'IPA'
	option target 'ACCEPT'
	option src '*'

config rule
	option name 'Test5'
	list proto 'all'
	option src '*'
	list src_ip 'IPA'
	option dest '*'
	list dest_ip 'IPB'
	option target 'ACCEPT'

config rule
	option name 'Test6'
	list proto 'all'
	option src '*'
	list src_ip 'IPB'
	option dest '*'
	list dest_ip 'IPA'
	option target 'ACCEPT'

config rule
	option name 'Test7'
	list proto 'all'
	list src_ip 'IPA'
	list dest_ip 'IPB'
	option target 'ACCEPT'

config rule
	option name 'Test8'
	list proto 'all'
	list dest_ip 'IPA'
	list src_ip 'IPB'
	option target 'ACCEPT'

This should cover all the scenarios to allow any traffic between IPA and IPB whatever the direction.

Now, here is what happens when I ping IPA from IfB:

root@OpenWrt:~# ping -I IfB IPA
PING IPA (IPA) from IPB IfB: 56(84) bytes of data.
^C
--- IPA ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2054ms

BUT a simultaneous tcpdump show this:

root@OpenWrt:~# tcpdump -i IfB -enn net IPA/32
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on IfB, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:49:34.414150 Out ethertype IPv4 (0x0800), length 100: IPB > IPA: ICMP echo request, id 63335, seq 1, length 64
13:49:34.429451  In ethertype IPv4 (0x0800), length 100: IPA > IPB: ICMP echo reply, id 63335, seq 1, length 64
13:49:35.438756 Out ethertype IPv4 (0x0800), length 100: IPB > IPA: ICMP echo request, id 63335, seq 2, length 64
13:49:35.454228  In ethertype IPv4 (0x0800), length 100: IPA > IPB: ICMP echo reply, id 63335, seq 2, length 64
13:49:36.468736 Out ethertype IPv4 (0x0800), length 100: IPB > IPA: ICMP echo request, id 63335, seq 3, length 64
13:49:36.484763  In ethertype IPv4 (0x0800), length 100: IPA > IPB: ICMP echo reply, id 63335, seq 3, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

And a tcpdump -i IfA -enn net IPB/32 shows also the requests and replies from IfA POV.

Problem 1: it appears that the packets are taking the correct route, bypassing the local loopback, are arriving, the reply is issued and sent back then received, but… the ping process does not see the replies
__

Now in the other direction, here is what happens when I ping IPB from IfA:

root@OpenWrt:~# ping -I IfA IPB
PING IPB (IPB) from IPA IfA: 56(84) bytes of data.
^C
--- IPB ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2080ms

And the tcpdump:

root@OpenWrt:~# tcpdump -i IfB -enn net IPA/32
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gre4-mw_tunnel, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:57:56.156728  In ethertype IPv4 (0x0800), length 100: IPA > IPB: ICMP echo request, id 2588, seq 1, length 64
13:57:57.156366  In ethertype IPv4 (0x0800), length 100: IPA > IPB: ICMP echo request, id 2588, seq 2, length 64
13:57:58.236551  In ethertype IPv4 (0x0800), length 100: IPA > IPB: ICMP echo request, id 2588, seq 3, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

And same if I do a tcpdump -i IfA -enn net IPB/32 I only see echo requests.

Problem 2: the echo requests packets are taking the correct route, bypassing the local loopback, are arriving, but no reply is even issued this time…
__

Any idea of what is going on?

Any suggestion for a better way to implement what I am trying to do?
I tried with ip netns a veth bridged to IfA, but since it needs to be forwarded on the main namespace, I have the exact same problem with loopback that without namespaces… I could put the IfB tunnel in its own netspace, but OpenWrt and mostly LuCi is not designed for that and that would make this simple thing very complicated at the end.

Thank you for reading and for your help.

bolemo · January 18, 2023, 1:50pm

Oh, I forgot to precise that I tried with net.ipv4.conf.all.rp_filter set to 0, 1 and 2, without any luck.

bolemo · January 19, 2023, 11:31am

First, thank you @tmomas for moving the subject in the appropriate sub-section

I found the problem and the solution.

Bypassing the local loopback worked so well that the packets never reached the machine (that was then never considered for input, all was forwarded again).
I discovered this by playing with the firewall rules and finding the ping fro IPA to IPB (or the reverse) had suddenly a TTL exhaustion error, and tcpdump would show the packets multiple times, the device would receive them then send them back as forward, making a huge loop.

The solution is simple at the end:
It does not require any firewall rule. It needs just the 2 specific routing tables and 4 routing rules*.

root@OpenWrt:~# ip ru
0:	from IPA to IPB iif IfB lookup local
0:	from IPB to IPA iif IfA lookup local
1:	from IPA to IPB lookup 1000
1:	from IPB to IPA lookup 2000
10:	from all lookup local
(… then usual rules with main, default, etc.)

root@OpenWrt:~# ip r show table 1000
default IfA-GW dev IfA proto static src IPA

root@OpenWrt:~# ip r show table 2000
default dev IfB proto static scope link src IPB mtu 1476

That's all
The two first rules are to send to local (input) the packets arriving from the WAN.
The two next rules are to bypass the local loopback for the packets being emitted.
Then the fifth rule is the usual rule for loopback, usually the first rule with a priority of 0, bet here it has to be after the bypassing rules (so I gave priority 10).

__
*There might be a way to reduce the 4 first rules into only 2 rules.

bolemo · January 19, 2023, 5:31pm

Variation on the same idea:
Alternative using named pre local table and I also found how to reduce 4 first rules to only 2.

root@OpenWrt:~# ip r list table prelocal
IPA dev IfB proto static scope link src IPB mtu 1476 
IPB via IPA-GW dev IfA proto static src IPA

root@OpenWrt:~# ip ru
0:	from IPA to IPB iif lo lookup prelocal
0:	from IPB to IPA iif lo lookup prelocal
10:	from all lookup local
(…) other rules

The rules could probably even be simplified to this, not sure if it would mess anything else:

root@OpenWrt:~# ip ru
0:	from all iif lo lookup prelocal
10:	from all lookup local

system · January 29, 2023, 5:31pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.