Proxy_arp: WAN is sometimes unavailable

rozhur · January 23, 2025, 9:57am

Continuing the topic Routing ISP public IP addresses on L3.

I am currently using a Ubiquiti EdgeRouter X-SFP router: https://openwrt.org/toh/ubiquiti/edgerouter_x_er-x_ka
I have router with OpenWRT, server and desktop with Debian 12, which I give public IP addresses from my ISP network 1.0.0.0/27:

ISP gateway -> 1.0.0.1 (eth5)
router -> 1.0.0.10
server -> 1.0.0.12 (eth0)
desktop -> 1.0.0.15 (eth1)

To make routing work for server and desktop, I first created a wan bridge including ports eth0 and eth1, and connected server and desktop to these ports. Everything worked fine.

However, I wanted to isolate these addresses from the ISP network and also make the firewall on the router work for these IP addresses, so I configured the network a little differently:

Created a bridge isowan (isolated WAN) and added ports eth0 and eth1 to it;
1.1. To allow isowan members to get their addresses from statis-leases of the dhcp server, I assigned the IP address 1.0.0.10/27 to the interface isowan;
1.2. Added the route 1.0.0.0/27 and the gateway 1.0.0.1 to wan;
1.3. Then added two routes 10.0.0.12/32 and 10.0.0.15/32 for isowan.
To make this setting work, I enabled proxy_arp: echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp.
The wan interface is no longer a bridge, but just an eth5 device.

After these steps, everything worked as expected.

But I found that very occasionally the WAN access from the router's IP address 10.0.0.10 would temporarily disappear, while server and desktop had no such problem.

I investigated the problem using tcpdump and found that when my ISP sends an ARP request to my router (to the MAC address of the wan interface), my router does not respond to them. I assume that after unsuccessful attempts to get a response, the ISP removes my router's address from its ARP cache, after which my router becomes unreachable through the ISP's network. I found that my router also sends an ARP request to the ISP gateway about once a minute, gets a response, and my router becomes reachable on the ISP network from its IP address until the next ARP request from the ISP to my router.

Let's say the MAC address of the wan interface is 11:00:00:00:00:37, and my ISP's is 22:00:00:00:00:01

The output of tcpdump arp is something like this:

22:00:00:00:00:01 -> 11:00:00:00:00:37 Request who-has 10.0.0.10 tell 10.0.0.1
22:00:00:00:00:01 -> 11:00:00:00:00:37 Request who-has 10.0.0.10 tell 10.0.0.1

(at this point the Internet becomes unavailable, I can't even ping 10.0.0.1. As I assume my ISP is trying to find my router for all members of his network since my router didn't respond to him earlier)

22:00:00:00:00:01 -> ff:ff:ff:ff:ff:ff Request who-has 10.0.0.10 tell 10.0.0.1
22:00:00:00:00:01 -> ff:ff:ff:ff:ff:ff Request who-has 10.0.0.10 tell 10.0.0.1
22:00:00:00:00:01 -> ff:ff:ff:ff:ff:ff Request who-has 10.0.0.10 tell 10.0.0.1
22:00:00:00:00:01 -> ff:ff:ff:ff:ff:ff Request who-has 10.0.0.10 tell 10.0.0.1
22:00:00:00:00:01 -> ff:ff:ff:ff:ff:ff Request who-has 10.0.0.10 tell 10.0.0.1

(the entry below can be triggered automatically by the router about once a minute or manually by the command arping 10.0.0.1)

22:00:00:00:00:01 -> 11:00:00:00:00:37 Reply 10.0.0.1 is-at 22:00:00:00:00:01

(after this the internet works until the next Request who-has 10.0.0.1 tell 10.0.0.1)

If I return the network settings with one wan without isowan, then this problem will not occur. I assume that since there are no addresses on wan, my router does not respond to ARP requests from my ISP. Please correct me if I'm wrong.

Question: what are the OpenWRT configuration options to resolve this problem so that with this proxy_arp setup my router responds to ARP requests?

Configuration `/etc/config/network`

config interface 'loopback'
    option device 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1/8'

config globals 'globals'
    option packet_steering '1'

config interface 'wan'
    option device 'eth5'
    option proto 'none'

config route 'route_wan_1'
    option interface 'wan'
    option target '1.0.0.0/27'

config route 'route_wan_2'
    option interface 'wan'
    option target '0.0.0.0/0'
    option gateway '1.0.0.1'
    option source '1.0.0.10'

config device
    option name 'isowan'
    option type 'bridge'
    list ports 'eth0'
    list ports 'eth1'

config interface 'isowan'
    option device 'isowan'
    option proto 'static'
    option ipaddr '1.0.0.10/27'
    option ip6assign '64'
    list dns '8.8.8.8'
    list dns '1.1.1.1'

config route 'route_isowan_1'
    option interface 'isowan'
    option target '1.0.0.12'

config route 'route_isowan_2'
    option interface 'isowan'
    option target '1.0.0.15'

config device
    option name 'lan'
    option type 'bridge'
    list ports 'eth2'
    list ports 'eth3'
    list ports 'eth4'

config interface 'lan'
    option device 'lan'
    option proto 'static'
    option ipaddr '192.168.74.87/24'
    option ip6assign '64'

Output of `ip addr`

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: dsa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc mq state UP group default qlen 1000
    link/ether 11:00:00:00:00:32 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e638:83ff:fedd:a832/64 scope link
       valid_lft forever preferred_lft forever
3: eth0@dsa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master isowan state UP group default qlen 1000
    link/ether 11:00:00:00:00:32 brd ff:ff:ff:ff:ff:ff
4: eth1@dsa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master isowan state LOWERLAYERDOWN group defaul
    link/ether 11:00:00:00:00:33 brd ff:ff:ff:ff:ff:ff
5: eth2@dsa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lan state UP group default qlen 1000
    link/ether 11:00:00:00:00:34 brd ff:ff:ff:ff:ff:ff
6: eth3@dsa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master lan state LOWERLAYERDOWN group default q
    link/ether 11:00:00:00:00:35 brd ff:ff:ff:ff:ff:ff
7: eth4@dsa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master lan state LOWERLAYERDOWN group default q
    link/ether 11:00:00:00:00:36 brd ff:ff:ff:ff:ff:ff
8: eth5@dsa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 11:00:00:00:00:37 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e638:83ff:fedd:a837/64 scope link
       valid_lft forever preferred_lft forever
73: isowan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 11:00:00:00:00:32 brd ff:ff:ff:ff:ff:ff
    inet 1.0.0.10/27 brd 1.0.0.31 scope global isowan
       valid_lft forever preferred_lft forever
    inet6 fe80::e638:83ff:fedd:a832/64 scope link
       valid_lft forever preferred_lft forever
74: lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 11:00:00:00:00:34 brd ff:ff:ff:ff:ff:ff
    inet 192.168.74.87/24 brd 192.168.74.255 scope global lan
       valid_lft forever preferred_lft forever
    inet6 fe80::e638:83ff:fedd:a834/64 scope link
       valid_lft forever preferred_lft forever

Output of `ip route`

default via 1.0.0.1 dev eth5 proto static src 1.0.0.10
1.0.0.0/27 dev eth5 proto static scope link
1.0.0.12 dev isowan proto static scope link
1.0.0.15 dev isowan proto static scope link
192.168.74.0/24 dev lan proto kernel scope link src 192.168.74.87

lleachii · January 23, 2025, 11:06am

Are you saying that want to isolate the ISP's addresses - from the ISP itself?

rozhur · January 23, 2025, 11:59am

No, by isolate I don't mean literally restrict access from the ISP. I want the addresses in 1.0.0.0/27 and the IP addresses that the ISP gave me from that network to be connected only via L3, not L2.

Sorry, English is not my native language, so I may have chosen inappropriate terms.

lleachii · January 23, 2025, 12:03pm

You know these are public addresses, correct?

OK...

Then separate these networks.

rozhur · January 23, 2025, 1:12pm

Yes. The prefix 1.0.0 is for example not to reveal the real IP of my ISP. For the same reason I edited the real MAC addresses in the post to fake ones.

Please can you explain what you mean by "separating these networks" in the context of OpenWRT configuration?

I think I've already done this by adding routes for my IP addresses to one interface, and routes for the rest of the IP addresses on that network to another interface, so that the connection between them is now at L3. It remains to solve the problem described in the first post.