Routing issues, can't ping or ssh, using VLAN and DNS Port Forwarding

Hello, I'm in the process of setting up my network and ran into the following problem(s), and have no clue on how to fix these issues.

  1. I'm unable to ping my DNS Server from within 192.168.0.0/24. Caddy cannot ping AGH, but I can ping AGH from any client that's routed via the Router...
  2. I cannot ssh from my client (192.168.1.50) to 192.168.0.5, but ssh to 192.168.1.5 works.

I use DNS Port Forwarding to redirect all DNS traffic, even forced nameservers, to my internal DNS. I deleted the 192.168.0.0/24 route on AGH as suggested by this tutorial video: https://youtu.be/Q-xDtCxnD0Q?t=1400 to make requests go over the router.

dns~# ip route
default via 192.168.0.1 dev eth0  metric 249 
-192.168.0.0/24 dev eth0 scope link  src 192.168.0.204

Removing this route causes my pings to AGH to fail. I have to add a custom route on caddy to route to 192.168.0.1 in order to make ping work again. Since AGH has a webui and caddy is my reverse proxy, only caddy really needs access to AGH. All traffic to AGH either goes through the router or to the reverse proxy. When I do this it works for a few minutes, maybe hours, but eventually fails. Sometimes it adds additional routes. Deleting the 192.168.0.0/24 route also works, but then all other 192.168.0.0/24 clients can't ping caddy either...

caddy:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.0/24 dev eth0 scope link  src 192.168.0.99
+192.168.0.204 via 192.168.0.1 dev eth0

Also, as soon as I add this route I'm unable to ping .lan hosts:

caddy:~# traceroute bleproxy.lan
traceroute: bad address 'bleproxy.lan'

In addition to this problem, and maybe even related, is the issue that I cannot ssh into 192.168.0.5 anymore from 192.168.1.50/24, but I can ssh from 192.168.1.1/24... I can ping 192.168.0.5 from 192.168.1.50 however.

This is the topology, simplified.

My router config:

~# ubus call system board
{
        "kernel": "5.15.134",
        "hostname": "OpenWrt",
        "system": "MediaTek MT7621 ver:1 eco:3",
        "model": "ASUS RT-AX53U",
        "board_name": "asus,rt-ax53u",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.0",
                "revision": "r23497-6637af95aa",
                "target": "ramips/mt7621",
                "description": "OpenWrt 23.05.0 r23497-6637af95aa"
        }
}
~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option packet_steering '1'
        option ula_prefix 'fdd0:a84f:5d28::/48'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'
        option peerdns '0'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix 'auto'
        option peerdns '0'

config device
        option type 'bridge'
        option name 'br-guest'

config interface 'guest'
        option device 'br-guest'
        option proto 'static'
        option ipaddr '192.168.255.1'
        option netmask '255.255.255.0'

config interface 'infra'
        option proto 'static'
        option ipaddr '192.168.0.1'
        option netmask '255.255.255.0'
        option device 'br-vlan10'

config interface 'lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option device 'br-vlan11'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'

config bridge-vlan
        option device 'br-lan'
        option vlan '10'
        list ports 'lan1:t'
        list ports 'lan2'

config bridge-vlan
        option device 'br-lan'
        option vlan '11'
        list ports 'lan1:t'
        list ports 'lan3'

config device
        option type 'bridge'
        option name 'br-vlan10'
        list ports 'br-lan.10'
        option bridge_empty '1'

config device
        option type 'bridge'
        option name 'br-vlan11'
        list ports 'br-lan.11'
        option bridge_empty '1'

config interface 'iot'
        option proto 'static'
        option ipaddr '192.168.3.1'
        option netmask '255.255.255.0'
        option device 'br-iot'

config device
        option type 'bridge'
        option name 'br-iot'
~# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path '1e140000.pcie/pci0000:00/0000:00:01.0/0000:02:00.0'
        option channel '1'
        option band '2g'
        option htmode 'HE20'
        option cell_density '0'
        option country 'AT'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option mode 'ap'
        option ssid 'ABC'
        option encryption 'psk2'
        option key 'XYZ'
        option network 'lan'
        option ieee80211r '1'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'

config wifi-device 'radio1'
        option type 'mac80211'
        option path '1e140000.pcie/pci0000:00/0000:00:01.0/0000:02:00.0+1'
        option channel '36'
        option band '5g'
        option htmode 'HE80'
        option cell_density '0'
        option country 'AT'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option mode 'ap'
        option ssid 'ABC'
        option encryption 'psk2'
        option key 'XYZ'
        option ieee80211r '1'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'lan'

config wifi-iface 'wifinet2'
        option device 'radio0'
        option mode 'ap'
        option ssid 'ABC-Guest'
        option key 'XYZ'
        option encryption 'psk2'
        option network 'guest'
        option isolate '1'
        option ieee80211r '1'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'

config wifi-iface 'wifinet3'
        option device 'radio1'
        option mode 'ap'
        option ssid 'ABC-Guest'
        option key 'XYZ'
        option encryption 'psk2'
        option isolate '1'
        option network 'guest'
        option ieee80211r '1'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'

config wifi-iface 'wifinet5'
        option device 'radio0'
        option mode 'ap'
        option ssid 'ABCIoT'
        option encryption 'psk2'
        option key 'XYZ'
        option ieee80211r '1'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'infra'

config wifi-iface 'wifinet6'
        option device 'radio1'
        option mode 'ap'
        option ssid 'ABC IoT'
        option encryption 'psk2'
        option key 'XYZ'
        option ieee80211r '1'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'infra'

~# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option rebind_protection '0'
        option local '/lan/'
        option domain 'lan'
        list server '192.168.0.204#53'
        option logqueries '1'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'guest'
        option interface 'guest'
        option start '100'
        option limit '150'
        option leasetime '12h'

# config host static leases truncated...

config dhcp 'infra'
        option interface 'infra'
        option start '100'
        option limit '150'
        option leasetime '12h'

config dhcp 'iot'
        option interface 'iot'
        option start '100'
        option limit '150'
        option leasetime '12h'

~# cat /etc/config/firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'DROP'
        option flow_offloading '1'
        option drop_invalid '1'

config zone
        option name 'guest'
        option output 'ACCEPT'
        list network 'guest'
        option input 'REJECT'
        option forward 'REJECT'

config zone
        option name 'iot'
        option output 'ACCEPT'
        list network 'iot'
        option forward 'REJECT'
        option input 'REJECT'

config zone
        option name 'infra'
        option input 'ACCEPT'
        option output 'ACCEPT'
        list network 'infra'
        option forward 'ACCEPT'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        option input 'DROP'
        option forward 'DROP'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option proto 'icmp'
        option family 'ipv4'
        option target 'ACCEPT'
        list icmp_type 'echo-request'
        option src '*'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option proto 'icmp'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        list icmp_type 'bad-header'
        list icmp_type 'destination-unreachable'
        list icmp_type 'echo-reply'
        list icmp_type 'echo-request'
        list icmp_type 'neighbour-advertisement'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'packet-too-big'
        list icmp_type 'router-advertisement'
        list icmp_type 'router-solicitation'
        list icmp_type 'time-exceeded'
        list icmp_type 'unknown-header-type'
        option src '*'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config redirect
        option target 'DNAT'
        option name 'Forward IoT DNS'
        option src 'iot'
        option src_dport '53'
        option dest 'infra'
        option dest_ip '192.168.0.204'
        option dest_port '53'
        option src_ip '!192.168.0.204'

config redirect
        option name 'Foward INFRA DNS'
        option src 'infra'
        option src_dport '53'
        option dest_ip '192.168.0.204'
        option dest_port '53'
        option dest 'infra'
        option src_ip '!192.168.0.204'

config redirect
        option name 'Forward LAN DNS'
        option src 'lan'
        option src_dport '53'
        option dest 'infra'
        option dest_ip '192.168.0.204'
        option dest_port '53'
        option src_ip '!192.168.0.204'

config forwarding
        option src 'lan'
        option dest 'wan'

config forwarding
        option src 'guest'
        option dest 'wan'

config forwarding
        option src 'infra'
        option dest 'lan'

config forwarding
        option src 'infra'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'infra'

config rule
        option name 'Guest DNS'
        option src 'guest'
        option dest_port '53'
        option target 'ACCEPT'

config rule
        option name 'Guest DHCP'
        list proto 'udp'
        option src 'guest'
        option target 'ACCEPT'
        option dest_port '67-68'

config rule
        option name 'Guest Block All'
        option src 'guest'
        option target 'DROP'

config forwarding
        option src 'iot'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'iot'

config forwarding
        option src 'infra'
        option dest 'iot'

On AGH the only interesting setting is the upstream DNS server to resolve .lan domains:

upstream_dns:
    - https://dns.quad9.net/dns-query
+   - '[/lan/]192.168.0.1'

You should not need any routes like this... it would be a tautology in the networking environment because the host is directly connected to that network and should be able to reach everything at L2 (no routing involved). 192.168.0.204 and 192.168.0.9 and 192.168.0.1 are all part of 192.168.0.0/24. I would recommend deleting them. Also, you probably don't need a metric on the default route.

I do see an issue here:

The above should be br-lan.10 and br-lan.11 respectively.

And these should be deleted (it would be a bridge within a bridge):

Try fixing those things, reboot your router, and try again.

I have removed my added ip route:

caddy:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.0/24 dev eth0 scope link  src 192.168.0.99
-192.168.0.204 via 192.168.0.1 dev eth0

I also removed the extraneous bridges and the extraneous ip route, and it worked for a minute, but now pings don't work anymore again. Funnily, while the ping to .204 was working, the ping to .lan didn't - now it's the opposite:

caddy:~# ping bleproxy.lan
ping: bad address 'bleproxy.lan'

caddy:~# ping 192.168.0.204
PING 192.168.0.204 (192.168.0.204): 56 data bytes
64 bytes from 192.168.0.204: seq=0 ttl=64 time=0.054 ms
64 bytes from 192.168.0.204: seq=1 ttl=64 time=0.066 ms

...

caddy:~# ping bleproxy.lan
PING bleproxy.lan (192.168.0.46): 56 data bytes
64 bytes from 192.168.0.46: seq=0 ttl=255 time=65.369 ms
...

caddy:~# ping 192.168.0.204
PING 192.168.0.204 (192.168.0.204): 56 data bytes
--- 192.168.0.204 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

Removing the other ip route causes .99 to be unpingable from within 192.168.0.0. However I can ping 192.168.0.204 from 192.168.0.99, while .lan still causes bad address

dns:~# ip route
default via 192.168.0.1 dev eth0  metric 202
- 192.168.0.0/24 dev eth0 scope link  src 192.168.0.99

The two routes I posted are automatically added, or rather I have them on all my LXCs. I did not add them by hand.

vpn:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.0/24 dev eth0 scope link  src 192.168.0.205

Ok... those routes look fine.

What is the IP address of caddy? Is it 192.168.0.99?
And what OS is on the host at 192.168.0.204?

caddy is 192.168.0.99
DNS/AGH is 192.168.0.204

Both are running alpine linux

caddy:~# uname -a
Linux caddy 6.5.11-6-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-6 (2023-11-29T08:32Z) x86_64 Linux

dns:~# uname -a
Linux dns 6.5.11-6-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-6 (2023-11-29T08:32Z) x86_64 Linux

EDIT: In the meantime things changed again. I can ping 192.168.0.204, but .lan is broken again. I also got more ip routes automatically added:

caddy:~# ping bleproxy.lan
<nothing>
^C

caddy:~# ping 192.168.0.204
PING 192.168.0.204 (192.168.0.204): 56 data bytes
64 bytes from 192.168.0.204: seq=0 ttl=64 time=0.072 ms
64 bytes from 192.168.0.204: seq=1 ttl=64 time=0.056 ms
^C
--- 192.168.0.204 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.056/0.064/0.072 ms

caddy:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.250 via 192.168.0.250 dev eth0
192.168.0.204 via 192.168.0.204 dev eth0
192.168.0.202 via 192.168.0.202 dev eth0

EDIT2: I fixed the ip routes again, they now look like this again:

caddy:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.0/24 dev eth0 scope link  src 192.168.0.99

I'm now able to ping 192.168.0.204. So that issue was probably caused by the nested bridges I just deleted. But bleproxy.lan is still unpingable, even though I can see the name is correctly resolved by 192.168.0.1 on AGH

This is a DNS issue. It has to do with your AGH configuration, not OpenWrt.

1 Like

Alright, I was including .lan in my observation in case it helps troubleshoot the issue. It just seemed so weird to me, that only either would work.

This is the state now, after I've removed the nested bridges:

  • ssh from 192.168.1.50 to 192.168.0.5 is still not possible
  • 192.168.0.99 (caddy), or any other client from 192.168.0.0/24, cannot ping 192.168.0.204 (dns)
  • Pinging .lan works fine from clients in 192.168.0.0/24
  • 192.168.0.204 (dns) can ping 192.168.0.99.
caddy:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.0/24 dev eth0 scope link  src 192.168.0.99

caddy:~# ping bleproxy.lan
PING bleproxy.lan (192.168.0.46): 56 data bytes
64 bytes from 192.168.0.46: seq=0 ttl=255 time=107.605 ms
64 bytes from 192.168.0.46: seq=1 ttl=255 time=49.113 ms
--- bleproxy.lan ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 49.113/78.359/107.605 ms

caddy:~# ping 192.168.0.204
PING 192.168.0.204 (192.168.0.204): 56 data bytes
--- 192.168.0.204 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
dns:~# ip route
default via 192.168.0.1 dev eth0  metric 202

Another client for reference:

vpn:~# ip route
default via 192.168.0.1 dev eth0  metric 202
192.168.0.0/24 dev eth0 scope link  src 192.168.0.205

vpn:~# ping bleproxy.lan
PING bleproxy.lan (192.168.0.46): 56 data bytes
64 bytes from 192.168.0.46: seq=0 ttl=255 time=28.679 ms
--- bleproxy.lan ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 28.679/74.709/120.739 ms

vpn:~# ping 192.168.0.204
PING 192.168.0.204 (192.168.0.204): 56 data bytes
--- 192.168.0.204 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

I can only ping 192.168.0.204 if either:

  • the client I ping from has a route to 192.168.0.204, or
  • 192.168.0.204 has a route to the client that's pinging it

This suggests a misconfiguration on the host. To be clear, OpenWrt (or any router) has no influence in connections between devices on the same subnet. Routing happens at L3 and that is what happens when you cross between subnets. When you ping a host in 192.168.0.0/24 from another host in 192.168.0.0/24, that is purely L2 (switching) and the router is not involved at all.