WAN ping echo-reply going over the wrong interface

Installing and Using OpenWrt Network and Wireless Configuration
1

I'm using a ping monitor from thinkbroadband.com. I have the default WAN ping firewall rule enabled covering the WAN zone, which my WAN interfaces are all attached to. I've noticed that after a while my router seems to stop replying to the ping requests on a specific WAN interface in question (pppoe-wanb). This is evident by the graph data

image

Using tcpdump and filtering ICMP, I can see the echo request packets on the interface. However, no replies are being sent.

pingbox1.thinkbroadband.com is 80.249.99.164.

# tcpdump -i pppoe-wanb src 80.249.99.164
11:17:25.049399 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35938, length 8
11:17:26.049574 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35939, length 8
11:17:27.049500 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35940, length 8
11:17:28.050097 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35941, length 8
11:17:29.049572 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35942, length 8
11:17:30.049467 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35943, length 8
11:17:31.049621 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35944, length 8
11:17:32.049539 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35945, length 8
11:17:33.049502 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35946, length 8
11:17:34.049656 IP pingbox1.thinkbroadband.com > static-87-xx-xx-xx.vodafonexdsl.co.uk: ICMP echo request, id 32865, seq 35947, length 8

As I have multiple WAN interfaces, I have the same monitor on a different WAN eth1.2 (although on the same router) which is working fine. However I suddenly spotted that the echo reply packets for me pppoe-wanb interface are going over eth1.2, looking at tcpdump:

cpc1-xxxxxxx-x-x-xxxxxxx.xx-x.cable.virginm.net - Reverse DNS for my WAN IP

# tcpdump -i eth1.2 icmp
11:22:00.050067 IP static-87-xx-xx-xx.vodafonexdsl.co.uk > pingbox1.thinkbroadband.com: ICMP echo reply, id 32865, seq 35957, length 8
11:22:00.242345 IP pingbox1.thinkbroadband.com > cpc1-xxxxxxx-x-x-xxxxxxx.xx-x.cable.virginm.net: ICMP echo request, id 32834, seq 40106, length 8
11:22:00.242453 IP cpc1-xxxxxxx-x-x-xxxxxxx.xx-x.cable.virginm.net > pingbox1.thinkbroadband.com: ICMP echo reply, id 32834, seq 40106, length 8
11:22:01.049995 IP static-87-xx-xx-xx.vodafonexdsl.co.uk > pingbox1.thinkbroadband.com: ICMP echo reply, id 32865, seq 35958, length 8
11:22:01.242320 IP pingbox1.thinkbroadband.com > cpc1-mapp13-2-0-cust242.12-4.cable.virginm.net: ICMP echo request, id 32834, seq 40107, length 8
11:22:01.242380 IP cpc1-xxxxxxx-x-x-xxxxxxx.xx-x.cable.virginm.net > pingbox1.thinkbroadband.com: ICMP echo reply, id 32834, seq 40107, length 8
11:22:02.049907 IP static-87-xx-xx-xx.vodafonexdsl.co.uk > pingbox1.thinkbroadband.com: ICMP echo reply, id 32865, seq 35959, length 8

I can see that the echo replies for the pppoe-wanb interface, are suddenly going over this WAN. I am using mwan3, because I am multihomed, but what can I do to ensure the ping reply goes over the correct interface?

2

It would seem doing:

/etc/init.d/firewall restart && mwan3 restart

Made the ping reply go through the right interface again, I wonder what rule had made it go through eth1.2.

3

I have tested another WAN interface, which is an L2TP tunnel, which provides a static IP as well. I can see the same problem happening, however reloading the firewall doesn't help. The echo replies always go out of eth1.2 which is my default interface.

# tcpdump -i l2tp-aaisp icmp
12:55:50.010781 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39717, length 8
12:55:51.011181 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39718, length 8
12:55:52.010585 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39719, length 8
12:55:53.010770 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39720, length 8
12:55:54.011068 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39721, length 8
12:55:55.010506 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39722, length 8
12:55:56.010506 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39723, length 8
12:55:57.010874 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39724, length 8
12:55:58.010599 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39725, length 8
12:55:59.010626 IP pingbox1.thinkbroadband.com > xxx.xxx.xxx.81.in-addr.arpa: ICMP echo request, id 32854, seq 39726, length 8

# tcpdump -i eth1.2 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1.2, link-type EN10MB (Ethernet), capture size 262144 bytes
12:56:42.010999 IP xxx.xxx.xxx.81.in-addr.arpa > pingbox1.thinkbroadband.com: ICMP echo reply, id 32854, seq 39769, length 8
12:56:43.010826 IP xxx.xxx.xxx.81.in-addr.arpa > pingbox1.thinkbroadband.com: ICMP echo reply, id 32854, seq 39770, length 8
12:56:44.010534 IP xxx.xxx.xxx.81.in-addr.arpa > pingbox1.thinkbroadband.com: ICMP echo reply, id 32854, seq 39771, length 8
12:56:45.010598 IP xxx.xxx.xxx.81.in-addr.arpa > pingbox1.thinkbroadband.com: ICMP echo reply, id 32854, seq 39772, length 8
12:56:46.011068 IP xxx.xxx.xxx.81.in-addr.arpa > pingbox1.thinkbroadband.com: ICMP echo reply, id 32854, seq 39773, length 8
12:56:47.010499 IP xxx.xxx.xxx.81.in-addr.arpa > pingbox1.thinkbroadband.com: ICMP echo reply, id 32854, seq 39774, length 8

Because I'm using mwan3 and policy routing, how can I stop this happening without breaking failover and existing policies?

Can I specifically specify ICMP to be handled in a certain way? It looks like a routing issue, but can I use the firewall or perhaps NAT rules to control the outbound device when ICMP is received on an interface?

4

you may try another version of mwan3 in x-wrt

it could replace mwan3 package, since it is another fork of mwan3 but some changes since.

5

Thanks, I'll stay on the mwan3 2.8 stable branch for now, because I've had my setup running on it for a while and know it works. I don't have a lot of time to debug any issues currently. There is a fair bit of development happening with the 2.10.x version currently but I'm keep an eye on the developments to move to in the future.

6

Did you ever solve this? I seem to be having the same problem - but without mwan. I have 3 active wan connections, on veth00, veth01, and veth02. These are all macvlans on physical eth0. Whatever interface I ping, the response is sent from veth00, which is "wan" and has the lowest metric. It seems like this shouldn't be default behaviour, as it makes troubleshooting harder.

7

Only that flushing conntrack on interface events i.e. ifup and ifupdate seemed to help. Although, I see you mentioned you weren't using mwan3. Without something steering the routing over each wan interface, I'm not sure if you are going to have any control over what packet an interface goes over.

2 Likes
8

Exactly.

I got an issue with one providers at work, so 2nd connection become a backup until issue been sorted.

I removed mwan3 and prioritised backup connection through lower metric over other.

As mentioned here Its not about mwan3 but lack of packet controls in OpenWrt on its own.

There is nothing like that, that when you have initiated connection through 2nd wan response is through 2nd as well. If 1st got lower metric, than it will go through there.

This going back to issue with wireguard handshake through wrong interface, but as well as ping and iperf3 that cannot be run over specified interface because of metrics that taking priority over anything else.

1 Like