Asymmetric routing and NAT

Installing and Using OpenWrt
1

Hi, don’t know if the title is right, but I will try to explain my situation as best as I can.

I have one main internet service provider, I also have a public /32 address announced through BGP. BGP address is set on “lo” interface, as you can see in the config below.

My BGP upstream is reached through an IPIP tunnel and is working OK.

What is working - traffic from outside is reaching my router and is being processed as it should. Traffic originating from the router(OpenWrt) to external addresses is working as expected( for ex. ping -I MYPUB or wget).

What is not working - NAT. If I do SRCNAT from my internal /24 - it is working, but only traffic originating from OpenWrt itself(192.168.0.1), every other IP in this /24 segment is not being rewritten by the SRCNAT rule.

I want to achieve classic masquerading as it is done on my LAN< - >WAN interfaces.

Below are config snippets with the parts which are related to this topic. If you need something more, feel free to ask.

Edit: Using OpenWrt 24.10.3 r28872-daca7c049b if it makes sense.

/etc/config/network

config interface 'bgpip'
        option proto 'static'
        option device 'lo'
        option delegate '0'
        list ipaddr '<MYPUB>/32'
        option ip4table 'bgp'
        option force_link '0'
        option defaultroute '0'

config interface 'bgp_edge'
        option proto 'ipip'
        option peeraddr '.......'
        option mtu '<MYMTU>'
        option ttl '64'
        option delegate '0'
        option defaultroute '0'
        option peerdns '0'

config interface 'bgp_internal'
        option proto 'static'
        option device 'bond0.787'
        option ipaddr '192.168.0.1'
        option netmask '255.255.255.0'
        option delegate '0'
        option ip4table 'bgp'

config rule
        option src '192.168.0.0/24'
        option lookup 'bgp'

config rule
        option src '<MYIP>/32'
        option lookup 'bgp'
        option priority '100'
        option in 'bgpip'

config rule
        option src '<MYPUB>/32'
        option lookup 'bgp'
        option priority '100'

config route
        option interface 'bgp_edge'
        option target '0.0.0.0/0'
        option table 'bgp'

/etc/config/firewall - this is not working as expected

config zone
        option name 'BGP'
        option output 'ACCEPT'
        option input 'ACCEPT'
        option forward 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        option family 'ipv4'
        list subnet '<MYPUB>/32'
        option masq6 '1'
        list network 'bgpip'

config zone
        option name 'bgp_int'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option family 'ipv4'
        list network 'bgp_internal'
        list subnet '192.168.0.0/24'

config forwarding
        option src 'bgp_int'
        option dest 'BGP'

Finally, this is the rule that makes it work from OpenWrt itself

config nat
        option name 'bgp'
        option src '*'
        option target 'SNAT'
        option snat_ip '<MYPUB>'
        option src_ip '192.168.0.0/24'
        list proto 'all'
        option enabled '0'

ip route show table bgp

default dev ipip-bgp_edge proto static scope link 
<MYPUB> dev lo proto static scope link 
192.168.0.0/24 dev bond0.787 proto static scope link src <MYPUB>

ip rule show table bgp

2:      from 192.168.0.0/24 lookup bgp
100:    from <MYPUB> lookup bgp
100:    from <MYPUB> iif lo lookup bgp
10000:  from 192.168.0.1 lookup bgp
10000:  from <MYPUB> lookup bgp
20000:  from all to 192.168.0.1/24 lookup bgp
20000:  from all to <MYPUB> lookup bgp
90001:  from all iif lo lookup bgp
90124:  from all iif lo lookup bgp
2

Disclaimer, I have no final answer right of my head right now but at least some hopefully useful pointers.

First. Don't use the loopback IP you also use as a router I'd and other stuff on your router for the nat src addr.
Second. Free your router from the first address and reserve that for an high available virtual IP. Give your router each a dedicated one and also reserve an extra address as source address for nat.
Third. Please try to do the research on your own, but there was something with in net filter core and how masq is implemented that you can not set a src address. For that you need snat as you have discovered already.
With snat you could also sync connection states between two routers to load balance and or doing fail over using conntrackd.

3

Thank you for your answer. I’ve tried with veth and also dummy interface. None of them is working.

Aside from that, nice point, because I’ve read elsewhere that it’s not good to put IPs on “lo” as kernel and conntrack treats them differently.
I can not fully understand the second one. Maybe leave .1 for the router and reserve .253(for example) to do SNAT from it?

4

I would propose

.1 as the virtual gateway address for your network, this is handles by vrrp
.7 for the first router
.8 for the second router
.16 as a virtual address used for snat connections aka everything that has not for good reasons an rfc1918 on your network

When you say, something isn't working please try to give me a context and at best sample configs

I use virtual and other addresses on loopback and dummy just fine. And I've also setup snat in data center with bgp for instance VPN clients.... So for sure there should be a solution with OpenWrt just fine. I've jused Debian but the tools are the same.

5

I got your idea.

I never mentioned VRRP, VRF or something special. My goal is to use it as another “LAN” segment for my home lab.

Just this public IP with just this internal network(LAN) and NAT between them.

Still can’t understand why classic masquerade is not working.

6

If it's all just local traffic and rfc1918 anyway then do you want to use masq in the first place?!

7

Because it’s more clear to me, but can live without it. The fact that masq is not working means that something is off with the whole case (or is my config problem) and will later impose some difficulties when doing input port redirection and input firewall traffic handling.

For example, right now no Input REJECT rules are working in the BGP firewall zone and I have no control of what can or cannot reach my public IP.

8

Again. You can not select a source address when using masq! If you want to use a dedicated address you simply need to use snat.

you have no global bgp at all? Only within your local network?

9

I do have global BGP session/peering, it’s up and accessible from outside world.

10

I tried to mark packets with fwmark. It is working, but again only for traffic originating from local IPs(192.168.0.1 - OpenWrt).

If I execute nft list ruleset | grep -i 0x00000002 i get the rule, but counter is 0 when trying from 192.168.0.2

         chain srcnat {
                type nat hook postrouting priority srcnat; policy accept;
                meta nfproto ipv4 meta mark 0x00000002 counter packets 0 bytes 0 snat ip to <MYPUB> comment "!fw4: bgp" }