External network 'unreachable' when OpenWRT only connected on LAN. Am I missing a route?

dlich · October 15, 2021, 8:36pm

I have openwrt installed on a TPLink Archer A7

Model	TP-Link Archer A7 v5
Architecture	Qualcomm Atheros QCA956X ver 1 rev 0
Firmware Version	OpenWrt 21.02.0 r16279-5cc0535800 / LuCI openwrt-21.02 branch git-21.231.26241-422c175
Kernel Version	5.4.143

I'm not yet ready to deploy it in my network. I'm still getting it set up.

I'm working on understanding routes and such, and getting ready for multiple LANs eventually.

Right now while I'm working on it, the Router's connected only on one LAN port.
And it's on a different subnet than the rest of my LAN.

The configuration looks like this:

INTERNET
 |
 |
 | enp1s0: WAN ip = XX.XX.XX.XX
MAIN_ROUTER
 | enp2s0: ip = 10.1.1.100, 10.199.199.100
 | 
SWITCH
    |
 |------------------------------ OPENWRT
 |                                  |- 'Internet'   empty
 | ip = 10.1.1.7, 10.199.199.7      |- 'Ethernet 1' ip = 10.199.199.1
DESKTOP                             |- 'Ethernet 2' empty
                                    |- 'Ethernet 3' empty
                                    |- 'Ethernet 4' empty

from my MAIN_ROUTER & DESKTOP everything works as normal.
I can get everywhere out on the net.
And I can

ping to OPENWRT @ 10.199.199.1
ssh  to OPENWRT @ 10.199.199.1
http to OPENWRT @ 10.199.199.1

From OPENWRT I can only ping internal network on the same LAN segment

ping to DESKTOP @ 10.199.199.7
ping to MAIN_ROUTER @ 10.199.199.100

But nowhere external. e.g.

ping 1.1.1.1
	PING 1.1.1.1 (1.1.1.1): 56 data bytes
	ping: sendto: Network unreachable

Checking routes on OPENWRT

ip route show
	default via 10.199.199.100 dev br-lan onlink
	10.199.199.0/24 dev br-lan scope link  src 10.199.199.1

ip route get 10.199.199.7
	10.199.199.7 dev br-lan  src 10.199.199.1
ip route get 10.199.199.100
	10.199.199.100 dev br-lan  src 10.199.199.1

ip route get 10.1.1.7
	10.1.1.7 via 10.199.199.100 dev br-lan  src 10.199.199.1
ip route get 10.1.1.100
	10.1.1.100 via 10.199.199.100 dev br-lan  src 10.199.199.1

ip route get 1.1.1.1
	1.1.1.1 via 10.199.199.100 dev br-lan  src 10.199.199.1

Looks ok to me.

Testing ping from OPENWRT

ping 1.1.1.1 -c1
	PING 1.1.1.1 (1.1.1.1): 56 data bytes

while watching on MAIN_ROUTER

tcpdump -i enp2s0 host 10.199.199.1 -vv -n
	16:05:57.757418 IP (tos 0x0, ttl 64, id 57580, offset 0, flags [DF], proto ICMP (1), length 84)
	    10.199.199.1 > 1.1.1.1: ICMP echo request, id 4050, seq 0, length 64
	16:05:57.757657 IP (tos 0xc0, ttl 64, id 48840, offset 0, flags [none], proto ICMP (1), length 112)
	    10.199.199.100 > 10.199.199.1: ICMP host 1.1.1.1 unreachable, length 92
	        IP (tos 0x0, ttl 63, id 57580, offset 0, flags [DF], proto ICMP (1), length 84)
	    10.199.199.1 > 1.1.1.1: ICMP echo request, id 4050, seq 0, length 64

The traffic is getting TO the MAIN_ROUTER, and it looks like it's getting out to 1.1.1.1 too.

But that

10.199.199.100 > 10.199.199.1: ICMP host 1.1.1.1 unreachable, length 92

looks like a problem.

What I'm confused by is that when testing from MAIN_ROUTER to OPENWRT it looks OK

ip route get 10.199.199.1
	10.199.199.1 dev enp2s0 src 10.199.199.100 uid 0
	    cache

ping -c1 10.199.199.1
	PING 10.199.199.1 (10.199.199.1) 56(84) bytes of data.
	64 bytes from 10.199.199.1: icmp_seq=1 ttl=64 time=0.355 ms

	--- 10.199.199.1 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 0.355/0.355/0.355/0.000 ms

I guess I'm missing a route somewhere.

Don't have a clue where though :-/

Any hints on what's missing, where? So I can get out to the net from the OPENWRT when only connected by the LAN?

psherman · October 15, 2021, 9:20pm

It appears you are using VLANs on your main router, is that correct? What is your main router and how is it configured? Do you have the trunk setup on your switch for the respective uplink port? And do you also have the correct VLAN setup for the port that connects to the OpenWrt box? Is it tagged or untagged on that switch port? Have you setup OpenWrt to match the tagged or untagged status?

Beyond all of those questions, does your main router allow access to the internet on the 10.199.199.0/24 network? (have you verified this using a computer on that VLAN)? If so, have you set the gateway (and DNS) on your OpenWrt LAN config?

mk24 · October 15, 2021, 9:38pm

Yes that seems to be the main issue.

Multiple IPs on one interface doesn't afford any security, it just makes things complicated. Anyone could just set their machine to an IP address on the more privileged network and access it. You have to use VLANs to actually have securely separated networks on the same cable.

psherman · October 15, 2021, 9:40pm

It is not clear to me how the OP has configured the main router -- is it just multiple addresses or is it with proper VLANs.

dlich · October 15, 2021, 9:46pm

hi

It appears you are using VLANs on your main router, is that correct?

not currently. Eventually I have plans for it. But not there yet.

What is your main router

it's a Linux box.

how is it configured?

not sure what you're asking here. it's got two real ethernet interfaces, one 'external', and one 'internal'.
it's working for the rest of my LAN ok for all inbound and outbound traffic.

Beyond all of those questions, does your main router allow access to the internet on the 10.199.199.0/24 network?

yes

I figured out how to bind ping to an interface/address

checking it from the MAIN_ROUTER

ping -c1 -I 10.1.1.100 1.1.1.1
	PING 1.1.1.1 (1.1.1.1) from 10.1.1.100 : 56(84) bytes of data.
	64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=21.3 ms

	--- 1.1.1.1 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 21.327/21.327/21.327/0.000 ms

ping -c1 -I 10.199.199.100 1.1.1.1
	PING 1.1.1.1 (1.1.1.1) from 10.199.199.100 : 56(84) bytes of data.
	64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=22.0 ms

	--- 1.1.1.1 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 21.956/21.956/21.956/0.000 ms

is OK

have you verified this using a computer on that VLAN?

from the DESKTOP, on the LAN behind the MAIN_ROUTER, just like OPENWRT is,

ping -c1 -I 10.1.1.7 1.1.1.1
	PING 1.1.1.1 (1.1.1.1) from 10.1.1.7 : 56(84) bytes of data.
	64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=36.4 ms

	--- 1.1.1.1 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 36.377/36.377/36.377/0.000 ms

ping -c1 -I 10.199.199.7 1.1.1.1
	PING 1.1.1.1 (1.1.1.1) from 10.199.199.7 : 56(84) bytes of data.
	From 10.199.199.100 icmp_seq=1 Destination Host Unreachable

	--- 1.1.1.1 ping statistics ---
	1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

the ping from the 10.199.199.X net fails outbound.

ah. so that sounds like a route is needed on the MAIN_ROUTER?

psherman · October 15, 2021, 9:49pm

Your main router needs to be setup to be a gateway for the second network. But you should be using VLANs if you are trying to create unique networks on the same interface. Simply putting multiple addresses on the same interface does zero for improving security and/or network efficiency.

Is your switch managed? You need to be using a managed switch if you plan to use VLANs.

dlich · October 15, 2021, 10:01pm

Your main router needs to be setup to be a gateway for the second network.

I added an explicit firewall rule on the MAIN_ROUTER allowing 10.199.199.0/24 out to the net. And now it's working.
I didn't see the explicit rule for my 10.1.1.0/24 LAN's access, and it works. Guess it is handled by a default. I need to look more closely at that :-/

But you should be using VLANs if you are trying to create unique networks on the same interface. Simply putting multiple addresses on the same interface does zero for improving security and/or network efficiency.

I understand that in principle. Having a hard time so far wrapping my head around how & where you actually 'tag' traffic so that it flows across everything.

That's one of the (several) reasons that this router isn't ready for primetime.
I was hoping to get this set up first -- so I can run 'opkg' to get install software and such.

THEN the plan was to get the VLAN stuff going one step at a time.

Is your switch managed? You need to be using a managed switch if you plan to use VLANs.

Not the current one.
My new one that's managed and should do the trick (https://www.tp-link.com/us/business-networking/managed-switch/tl-sg3428/) is arriving tomorrow.

psherman · October 15, 2021, 10:07pm

Glad things are working. But, keep in mind that you've setup your second network in a way that is not advisable and not really properly controllable.

You'll want to do some reading on VLANs in general and also with respect to the specific implementations in whatever OS/environment you are using for your main router. You'll want to try creating multiple networks on different physical interfaces as well as VLANs on a single interface using 802.1q tagging, and with the routing firewall features to understand how to properly route between the networks and how to block and/or selectively allow traffic to flow. You'll also need to work with your switch to understand how to configure the ports to be members of each VLAN. It takes a little while to wrap your head around this stuff, but when you look back at your current configuration, you'll realize why it is not a good solution.

dlich · October 15, 2021, 11:07pm

keep in mind that you've setup your second network in a way that is not advisable and not really properly controllable.

Sure. That's clear. It's only during setup. Seemed like the 'simplest' install into my existing network so I could work on it, without screwing up the rest of my network.

You'll want to do some reading ...

Already have stacks of it to catch up on

my eventual goal, at least currently on paper, looks like

INTERNET
  |
MODEM[*]
  |
  |wan/eth
OPENWRT1[*] ------------- WIFI1/AP (WLAN1), isolated
  |lan/eth   |----------- WIFI2/AP (WLAN2), isolated, minimum guaranteed bandwidth
  |
WIFI0/AP[*]~~~~~~~~~~~~~~~ SSID_02, prioritized bandwidth
    \ SSID01
    /  (long link)
    \
    /
WIFI0/CLIENT[*]
  |
  |
MAIN_ROUTER/FIREWALL
  |
  |
MANAGED_SWITCH[*]
  |----------------- OPENWRT2[*] ------- WIFI3/AP (WLAN3), isolated
  |                               |----- WIFI4/AP (WLAN4), LAN1 access
  |- admin01 (LAN1)
  |- desk01  (LAN1)
  |- desk02  (LAN1)
  |- ...

where all "[*]" labelled devices are 'on' the 10.199.199.0/24, accessible only from 'admin01' and MAIN_ROUTER machines.

I've got a ways to go.