Strange ARP requests to Amazon

SirToffski · November 5, 2019, 3:02am

Here's my setup:

ISP provided gateway, which also serves as a wireless AP for my devices. 192.168.2.0/24 subnet.
Edgerouter X running OpenWRT built from the master branch. WAN port aka vlan 2 connects to the ISP gateway and grabs a public address - think of it as a "poor-man's-bridge".
Edgerouter X has a 192.168.1.0/24 LAN subnet on VLAN 1 - this is where my home server hosting Pi-Hole resides.
In order to allow wireless devices on 192.168.2.0/24 to access the Pi-Hole, I've created a in interface, as below:

config interface 'ROS'
	option proto 'static'
	option ifname 'eth0.2'
	option ipaddr '192.168.2.250'
	option netmask '255.255.255.0'
	option gateway '192.168.2.1'

On the ISP gateway I've added a static route: 192.168.1.0 255.255.255.0 192.168.2.250

This setup has worked for a long time without an issue. Today my wireless devices stopped communicating with 192.168.1.0/24 subnet, so before powercycling everything I've tried to ping my home-server. This is what came back:

$ ping 192.168.1.156
PING 192.168.1.156 (192.168.1.156) 56(84) bytes of data.
From 192.168.2.1 icmp_seq=2 Redirect Host(New nexthop: 250.2.168.192)
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Now this is odd - why would the next hop be 250.2.168.192, aka the reversed version of 192.168.2.250...

Naturally I pulled out wireshark - there I saw some odd looking arp requests to what appears to be Amazon-owned addresses:

6e:8b:30:b9:5c:96	Broadcast	ARP	42	Who has 63.35.84.143? Tell 169.254.1.2
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: 6e:8b:30:b9:5c:96 (6e:8b:30:b9:5c:96)
    Sender IP address: 169.254.1.2
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 63.35.84.143

Whois returns the following:

$ whois 63.35.84.143

# start

NetRange:       63.32.0.0 - 63.35.255.255
CIDR:           63.32.0.0/14
NetName:        AMAZO-4
NetHandle:      NET-63-32-0-0-1
Parent:         NET63 (NET-63-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       
Organization:   Amazon.com, Inc. (AMAZO-4)

I guess this is a two-part question.

Has anyone ever seen the next hop address being reversed from 192.168.2.250 to 250.2.168.192?
What about ARP from what looks like APIPA 169.254.1.2 to Amazon-owned 63.35.84.143? I tried looking up OUI of 6e:8b:30:b9:5c:96, doesnt look like it belongs to a known vendor.

jeff · November 5, 2019, 3:28am

Self-assigned IPv4 address?

Any IoT devices on your net?

Edit: Isn’t 06: indicative of a locally administered MAC?

Amazon push-buttons or other around?

SirToffski · November 5, 2019, 3:35am

There's a Samsung Smart TV - which also connects to ERX, but I've isolated it to vlan 50 on 192.168.50.0/24 range; also blocked forwarding to/from it:

config interface 'IoT'
	option ifname 'eth0.50'
	option proto 'static'
	option ipaddr '192.168.50.1'
	option netmask '255.255.255.0'

config rule
	option src '*'
	option name 'BlockSamsungIn'
	option target 'DROP'
	option proto 'all'
	option dest 'IoT_Zone'

config rule
	option proto 'all'
	option name 'BlockSamsungOut'
	option target 'DROP'
	option dest 'lan'
	option src 'IoT_Zone'
	option src_ip '192.168.50.248'

Hmm I guess in the BlockSamsungOut I could put '*' for destination, since I dont really need it talking to WAN either. There's also Apple TV on 192.168.2.0/24 network - but then why would it send stuff to Amazon.

jeff · November 5, 2019, 3:37am

“Everybody” hosts IoT on AWS, so could be most anything.

SirToffski · November 5, 2019, 3:53am

I mean yeah you're right. Strange that it would send ARP though - why would it want to talk to AWS via layer 2? Unless there's some security feature when the device establishes a tunnel to the AWS server and then it can technically talk to it on L2. Maybe that's a stretch haha..

Actually.... there's a Roomba too - always forget about that sneaky vacuum - I bet that's what it is - though 6e:8b:30:b9:5c:96 doesn't show up on any of my networks. This one can be considered as solved - some IoT likely trying to phone home.

What about the From 192.168.2.1 icmp_seq=2 Redirect Host(New nexthop: 250.2.168.192) part? Here I'm clueless as to how this would even happen...

jeff · November 5, 2019, 4:44am

Check your ARP and routing tables. An ICMP redirect often indicates that the router believes the client has direct access to the next-hop router and should go direct.

lleachii · November 5, 2019, 4:55am

What is 192.168.2.1?

SirToffski · November 5, 2019, 4:59am

The redirect itself is not surprising since this is how I set everything up. Here's a quick diagram:

The ISP gateway has a static route:

ERX has a interface with an IP of 192.168.2.250/24 on the eth0.2:

My laptop is connected wirelessly to the ISP gateway and is grabbing 192.168.2.109 via DHCP. Normally when I send ICMP to a device in 192.168.1.0/24 network, I see the redirect messages like this:

$ ping 192.168.1.156
PING 192.168.1.156 (192.168.1.156) 56(84) bytes of data.
From 192.168.2.1 icmp_seq=2 Redirect Host(New nexthop: 192.168.2.250)

But today for the first time the redirect looked like below - notice the nexthop IP is backwards

$ ping 192.168.1.156
PING 192.168.1.156 (192.168.1.156) 56(84) bytes of data.
From 192.168.2.1 icmp_seq=2 Redirect Host(New nexthop: 250.2.168.192)

^ This had never happened before. ARP looks unremarkable:

$ arp -a
? (192.168.2.103) at xx:xx:xx:xx:xx:xx [ether] on wlan0
? (192.168.2.151) at xx:xx:xx:xx:xx:xx [ether] on wlan0
? (192.168.2.250) at xx:xx:xx:xx:xx:xx [ether] on wlan0
_gateway (192.168.2.1) at xx:xx:xx:xx:xx:xx [ether] on wlan0

Route from the Edgerouter:

root@Bifrost:~# ip route
default via x.x.x.x [public-ip] dev eth0.2 proto static src x.x.x.x [public-ip]
x.x.x.0/22 dev eth0.2 proto kernel scope link src x.x.x.x [public-ip]
# Wireguard peers 
172.30.4.5 dev wg0 proto static scope link 
172.30.40.2 dev wg0 proto static scope link 
172.30.40.3 dev wg0 proto static scope link 
172.30.40.4 dev wg0 proto static scope link 
172.30.40.6 dev wg0 proto static scope link 
172.30.40.7 dev wg0 proto static scope link 
172.30.40.8 dev wg0 proto static scope link 
172.30.40.156 dev wg0 proto static scope link 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
# Upstream ISP-provided gateway
192.168.2.0/24 dev eth0.2 proto kernel scope link src 192.168.2.250 
# IoT stuff
192.168.50.0/24 dev eth0.50 proto kernel scope link src 192.168.50.1

lleachii · November 5, 2019, 5:06am

To be clear, are you intentionally configuring your router to give out next hop ICMP messages?

SirToffski · November 5, 2019, 6:02am

I've intentionally configured the next hop itself via a static route yes, which I presume is the reason for nexthop ICMP messages. To clarify further, I did not explicitly configure the router to send nexthop ICMP replies.

I'm not concerned by the fact that nexthop ICMP messages are present, more-so by the fact that today these messages showed an incorrect hext-hop address aka 250.2.168.192 - which is just bizarre.. There were no config changes, so it's not like I've somehow fat-fingered the static route..

I was just curious to see if anyone had experienced anything like this before - I suspect it's the ISP modem screwing up due to a bug or something.

vgaetera · November 5, 2019, 10:36am

ip neigh | grep -e 6e:8b:30:b9:5c:96

That device has an incorrect direct route without gateway to 63.35.84.143.
Most likely, it did not manage to obtain a DHCP lease for some reason.
Restarting the device should fix the problem.

Check the routing table on 192.168.2.1.

SirToffski · November 6, 2019, 3:38am

Thanks! Routing table on 192.168.2.1 looks mostly unremarkable. There is an entry for

169.254.1.0 255.255.255.248 0.0.0.0 br0

On the ERX, I do see a stale arp entry for 6e:8b:30:b9:5c:96

~# ip nei | grep -e 6e:8b:30:b9:5c:96
fe80::6c8b:30ff:feb9:5c96 dev eth0.2 lladdr 6e:8b:30:b9:5c:96 STALE

It's on eth0.2 so it's coming from somewhere upstream.