[SOLVED] Forwarding internet traffic (port 22 and port 80) to a KVM guest

I have a home network that I'm converting from an ASUS RT-N66U router (stock firmware) to a TP Link Archer C7 (AC 1750 - v4) running OpenWRT.

Flashing the firmware went smoothly using OpenWrt 18.06.1 r7258-5eb055306f / LuCI openwrt-18.06 branch (git-18.228.31946-f64b152). I had no problems setting up the internal network, the guest network, and all of the devices on those networks. Where I seem to be stuck is in how to forward internet traffic to two specific endpoints within my network.

The relevant portion of the internal network looks like this:

Router  (192.168.1.1) -> Debian KVM hypervisor (192.168.1.40) -> Debian KVM guest (192.168.2.2)

According to the diagnostics page in luci, the router can ping the KVM guest. Other devices on the internal network (including the hypervisor and Windows DHCP wifi clients) can connect to the KVM guest on port 22 and port 80. However, I haven't been able to forward any internet traffic through to the guest. (To eliminate confusion about what is internal and what is external, I have been testing connectivity from a Debian box that's completely independent of my home network.)

I've tried this a bunch of different ways. My initial hope was to forward wlan:80 directly to 192.168.2.2:80, but that doesn't work.

I then set up iptables forwarding on the hypervisor so it would accept traffic on 192.168.1.40:8080 and forward to 192.168.2.2:80, using the rules below. Then, I configured OpenWRT to forward traffic from wlan:80 to 192.168.1.40:8080. No dice.

iptables -I FORWARD -o virbr1 -d 192.168.2.2 -j ACCEPT
iptables -t nat -I PREROUTING -p tcp --dport 8080 -j DNAT --to 192.168.2.2:80
iptables -t nat -A OUTPUT -p tcp --dport 8080 -j DNAT --to 192.168.2.2:80

After a few more dead ends, I decided I might have a problem with NAT, and I tried the rewrite rules as suggested by Jo-Philipp Wich at the bottom of this bug. Same behavior.

The other suggestion in that bug was to change the default route, but I wasn't sure exactly where that would be done. The hypervisor has these routes:

default via 192.168.1.1 dev eth0 onlink
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.40
192.168.2.0/24 dev virbr1 proto kernel scope link src 192.168.2.1

The guest has these:

default via 192.168.2.1 dev eth0 onlink
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.2

I don't think I can change the guest to use 192.168.1.1 as the default route, since that network isn't reachable from the guest's virtual network.

The hypervisor is currently sitting on an eth0 wired connection using this virsh network:

<network connections='3'>
  <name>virtual</name>
  <forward mode="nat" dev="eth0"/>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:xx:yy:zz:a0:b9'/>
  <ip address='192.168.2.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.2.2' end='192.168.2.250'/>
      <host mac='52:xx:00:00:00:01' ip='192.168.2.2'/>
      <host mac='52:xx:00:00:00:02' ip='192.168.2.3'/>
      <host mac='52:xx:00:00:00:03' ip='192.168.2.4'/>
    </dhcp>
  </ip>
</network>

I've also tried it on a wireless wlan0 network with <forward mode='route'/>, which is my preference (if possible) because the room the hypervisor normally lives in doesn't have a wired connection right now.

I'm at a loss. Can someone point me at what I might be missing? After staring at this for too many hours, I'm not sure whether I'm just being dense and missing some obvious step, or something is broken. This whole setup used to work with the ASUS router, so it it seems like it should be possible.

@openwrt-router:

ip a; ip r; ip ru
uci show network
uci show firewall

root@etcetera:~# ip a; ip r; ip ru (with IP and MAC blanked out)

Summary
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether xx:84:c6:e5:b2:44 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::yy84:c6ff:fee5:b244/64 scope link
       valid_lft forever preferred_lft forever
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether xx:84:c6:e5:b2:44 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fe80::yy84:c6ff:fee5:b244/64 scope link
       valid_lft forever preferred_lft forever
6: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether xx:84:c6:e5:b2:44 brd ff:ff:ff:ff:ff:ff
7: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether xx:84:c6:e5:b2:44 brd ff:ff:ff:ff:ff:ff
    inet xx.yy.251.95/22 brd xx.yy.251.255 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::yy84:c6ff:fee5:b244/64 scope link
       valid_lft forever preferred_lft forever
8: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether yy:84:c6:e5:b2:43 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.1/24 brd 192.168.3.255 scope global br-guest
       valid_lft forever preferred_lft forever
    inet6 fe80::xx84:c6ff:fee5:b244/64 scope link
       valid_lft forever preferred_lft forever
9: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether xx:84:c6:e5:b2:44 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::yy84:c6ff:fee5:b244/64 scope link
       valid_lft forever preferred_lft forever
10: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether xx:84:c6:e5:b2:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::yy84:c6ff:fee5:b243/64 scope link
       valid_lft forever preferred_lft forever
11: wlan1-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-guest state UP qlen 1000
    link/ether yy:84:c6:e5:b2:44 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::xx84:c6ff:fee5:b244/64 scope link
       valid_lft forever preferred_lft forever
12: wlan0-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-guest state UP qlen 1000
    link/ether yy:84:c6:e5:b2:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::xx84:c6ff:fee5:b243/64 scope link
       valid_lft forever preferred_lft forever
default via xx.yy.248.1 dev eth0.2  src xx.yy.251.95
xx.yy.248.0/22 dev eth0.2 scope link  src xx.yy.251.95
192.168.1.0/24 dev br-lan scope link  src 192.168.1.1
192.168.2.0/24 via 192.168.1.40 dev br-lan
192.168.3.0/24 dev br-guest scope link  src 192.168.3.1
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

root@etcetera:~# uci show network

Summary
network.loopback=interface
network.loopback.ifname='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.globals=globals
network.globals.ula_prefix='fd3f:30a6:b622::/48'
network.lan=interface
network.lan.type='bridge'
network.lan.ifname='eth0.1'
network.lan.proto='static'
network.lan.ipaddr='192.168.1.1'
network.lan.netmask='255.255.255.0'
network.wan=interface
network.wan.proto='dhcp'
network.wan.ifname='eth0.2'
network.@switch[0]=switch
network.@switch[0].name='switch0'
network.@switch[0].reset='1'
network.@switch[0].enable_vlan='1'
network.@switch_vlan[0]=switch_vlan
network.@switch_vlan[0].device='switch0'
network.@switch_vlan[0].vlan='1'
network.@switch_vlan[0].ports='2 3 4 5 0t'
network.@switch_vlan[1]=switch_vlan
network.@switch_vlan[1].device='switch0'
network.@switch_vlan[1].vlan='2'
network.@switch_vlan[1].ports='1 0t'
network.@route[0]=route
network.@route[0].interface='lan'
network.@route[0].target='192.168.2.0'
network.@route[0].netmask='255.255.255.0'
network.@route[0].gateway='192.168.1.40'
network.guest=interface
network.guest.type='bridge'
network.guest.proto='static'
network.guest.ipaddr='192.168.3.1'
network.guest.netmask='255.255.255.0'

root@etcetera:~# uci show firewall

Summary
firewall.@defaults[0]=defaults
firewall.@defaults[0].syn_flood='1'
firewall.@defaults[0].input='ACCEPT'
firewall.@defaults[0].output='ACCEPT'
firewall.@defaults[0].forward='REJECT'
firewall.@zone[0]=zone
firewall.@zone[0].name='lan'
firewall.@zone[0].input='ACCEPT'
firewall.@zone[0].output='ACCEPT'
firewall.@zone[0].forward='ACCEPT'
firewall.@zone[0].network='lan'
firewall.@zone[1]=zone
firewall.@zone[1].name='wan'
firewall.@zone[1].input='REJECT'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].forward='REJECT'
firewall.@zone[1].masq='1'
firewall.@zone[1].mtu_fix='1'
firewall.@zone[1].network='wan wan6'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].src='lan'
firewall.@forwarding[0].dest='wan'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
firewall.@rule[0].proto='udp'
firewall.@rule[0].dest_port='68'
firewall.@rule[0].target='ACCEPT'
firewall.@rule[0].family='ipv4'
firewall.@rule[0].enabled='0'
firewall.@rule[1]=rule
firewall.@rule[1].name='Allow-Ping'
firewall.@rule[1].src='wan'
firewall.@rule[1].proto='icmp'
firewall.@rule[1].icmp_type='echo-request'
firewall.@rule[1].family='ipv4'
firewall.@rule[1].target='ACCEPT'
firewall.@rule[1].enabled='0'
firewall.@rule[2]=rule
firewall.@rule[2].name='Allow-IGMP'
firewall.@rule[2].src='wan'
firewall.@rule[2].proto='igmp'
firewall.@rule[2].family='ipv4'
firewall.@rule[2].target='ACCEPT'
firewall.@rule[2].enabled='0'
firewall.@rule[3]=rule
firewall.@rule[3].name='Allow-DHCPv6'
firewall.@rule[3].src='wan'
firewall.@rule[3].proto='udp'
firewall.@rule[3].src_ip='fc00::/6'
firewall.@rule[3].dest_ip='fc00::/6'
firewall.@rule[3].dest_port='546'
firewall.@rule[3].family='ipv6'
firewall.@rule[3].target='ACCEPT'
firewall.@rule[3].enabled='0'
firewall.@rule[4]=rule
firewall.@rule[4].name='Allow-MLD'
firewall.@rule[4].src='wan'
firewall.@rule[4].proto='icmp'
firewall.@rule[4].src_ip='fe80::/10'
firewall.@rule[4].icmp_type='130/0' '131/0' '132/0' '143/0'
firewall.@rule[4].family='ipv6'
firewall.@rule[4].target='ACCEPT'
firewall.@rule[4].enabled='0'
firewall.@rule[5]=rule
firewall.@rule[5].name='Allow-ICMPv6-Input'
firewall.@rule[5].src='wan'
firewall.@rule[5].proto='icmp'
firewall.@rule[5].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unk                                                                                                                                                                                       nown-header-type' 'router-solicitation' 'neighbour-solicitation' 'router-advertisement' 'neighbour-advertisement'
firewall.@rule[5].limit='1000/sec'
firewall.@rule[5].family='ipv6'
firewall.@rule[5].target='ACCEPT'
firewall.@rule[5].enabled='0'
firewall.@rule[6]=rule
firewall.@rule[6].name='Allow-ICMPv6-Forward'
firewall.@rule[6].src='wan'
firewall.@rule[6].dest='*'
firewall.@rule[6].proto='icmp'
firewall.@rule[6].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unk                                                                                                                                                                                       own-header-type'
firewall.@rule[6].limit='1000/sec'
firewall.@rule[6].family='ipv6'
firewall.@rule[6].target='ACCEPT'
firewall.@rule[6].enabled='0'
firewall.@rule[7]=rule
firewall.@rule[7].name='Allow-IPSec-ESP'
firewall.@rule[7].src='wan'
firewall.@rule[7].dest='lan'
firewall.@rule[7].proto='esp'
firewall.@rule[7].target='ACCEPT'
firewall.@rule[7].enabled='0'
firewall.@rule[8]=rule
firewall.@rule[8].name='Allow-ISAKMP'
firewall.@rule[8].src='wan'
firewall.@rule[8].dest='lan'
firewall.@rule[8].dest_port='500'
firewall.@rule[8].proto='udp'
firewall.@rule[8].target='ACCEPT'
firewall.@rule[8].enabled='0'
firewall.@include[0]=include
firewall.@include[0].path='/etc/firewall.user'
firewall.@zone[2]=zone
firewall.@zone[2].name='guest'
firewall.@zone[2].forward='REJECT'
firewall.@zone[2].output='ACCEPT'
firewall.@zone[2].network='guest'
firewall.@zone[2].input='REJECT'
firewall.@zone[2].family='ipv4'
firewall.@forwarding[1]=forwarding
firewall.@forwarding[1].dest='wan'
firewall.@forwarding[1].src='guest'
firewall.@rule[9]=rule
firewall.@rule[9].target='ACCEPT'
firewall.@rule[9].proto='tcp udp'
firewall.@rule[9].dest_port='53'
firewall.@rule[9].name='Guest DNS'
firewall.@rule[9].family='ipv4'
firewall.@rule[9].src='guest'
firewall.@rule[10]=rule
firewall.@rule[10].target='ACCEPT'
firewall.@rule[10].proto='udp'
firewall.@rule[10].dest_port='67-68'
firewall.@rule[10].name='Guest DHCP'
firewall.@rule[10].family='ipv4'
firewall.@rule[10].src='guest'
firewall.@redirect[0]=redirect
firewall.@redirect[0].target='DNAT'
firewall.@redirect[0].src='wan'
firewall.@redirect[0].dest='lan'
firewall.@redirect[0].proto='tcp'
firewall.@redirect[0].src_dport='80'
firewall.@redirect[0].name='http'
firewall.@redirect[0].dest_ip='192.168.1.40'
firewall.@redirect[0].dest_port='8080'
firewall.@redirect[1]=redirect
firewall.@redirect[1].target='DNAT'
firewall.@redirect[1].src='wan'
firewall.@redirect[1].dest='lan'
firewall.@redirect[1].proto='tcp'
firewall.@redirect[1].src_dport='22'
firewall.@redirect[1].name='ssh'
firewall.@redirect[1].dest_ip='192.168.1.40'
firewall.@redirect[1].dest_port='2222'

@openwrt-router:

iptables-save

@kvm-hypervisor, kvm-guest:

sudo iptables-save

Thanks for the help. It's 12:30am for me right now, so I am heading to bed and will check for anything else you need tomorrow morning.

Here's the result for the router.

root@etcetera:~# iptables-save

Summary
# Generated by iptables-save v1.6.2 on Sun Oct 14 05:20:37 2018
*nat
:PREROUTING ACCEPT [8180:723990]
:INPUT ACCEPT [347:33633]
:OUTPUT ACCEPT [587:45758]
:POSTROUTING ACCEPT [32:5376]
:postrouting_guest_rule - [0:0]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_guest_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_guest_postrouting - [0:0]
:zone_guest_prerouting - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
-A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
-A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
-A PREROUTING -i eth0.2 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i br-guest -m comment --comment "!fw3" -j zone_guest_prerouting
-A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
-A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
-A POSTROUTING -o eth0.2 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o br-guest -m comment --comment "!fw3" -j zone_guest_postrouting
-A zone_guest_postrouting -m comment --comment "!fw3: Custom guest postrouting rule chain" -j postrouting_guest_rule
-A zone_guest_prerouting -m comment --comment "!fw3: Custom guest prerouting rule chain" -j prerouting_guest_rule
-A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
-A zone_lan_postrouting -s 192.168.1.0/24 -d 192.168.1.40/32 -p tcp -m tcp --dport 8080 -m comment --comment "!fw3: http (reflection                                                                                                      )" -j SNAT --to-source 192.168.1.1
-A zone_lan_postrouting -s 192.168.1.0/24 -d 192.168.1.40/32 -p tcp -m tcp --dport 2222 -m comment --comment "!fw3: ssh (reflection)                                                                                                      " -j SNAT --to-source 192.168.1.1
-A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
-A zone_lan_prerouting -s 192.168.1.0/24 -d 73.62.251.95/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)"                                                                                                       -j DNAT --to-destination 192.168.1.40:8080
-A zone_lan_prerouting -s 192.168.1.0/24 -d 73.62.251.95/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -                                                                                                      j DNAT --to-destination 192.168.1.40:2222
-A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
-A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
-A zone_wan_prerouting -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http" -j DNAT --to-destination 192.168.1.40:8080
-A zone_wan_prerouting -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh" -j DNAT --to-destination 192.168.1.40:2222
COMMIT
# Completed on Sun Oct 14 05:20:37 2018
# Generated by iptables-save v1.6.2 on Sun Oct 14 05:20:37 2018
*mangle
:PREROUTING ACCEPT [173741:107090711]
:INPUT ACCEPT [5120:585836]
:FORWARD ACCEPT [167889:106285282]
:OUTPUT ACCEPT [10285:2313481]
:POSTROUTING ACCEPT [177986:108589965]
-A FORWARD -o eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to                                                                                                      -pmtu
COMMIT
# Completed on Sun Oct 14 05:20:37 2018
# Generated by iptables-save v1.6.2 on Sun Oct 14 05:20:37 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:forwarding_guest_rule - [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_wan_rule - [0:0]
:input_guest_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_wan_rule - [0:0]
:output_guest_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_wan_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_guest_dest_ACCEPT - [0:0]
:zone_guest_dest_REJECT - [0:0]
:zone_guest_forward - [0:0]
:zone_guest_input - [0:0]
:zone_guest_output - [0:0]
:zone_guest_src_REJECT - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
-A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
-A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
-A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
-A INPUT -i eth0.2 -m comment --comment "!fw3" -j zone_wan_input
-A INPUT -i br-guest -m comment --comment "!fw3" -j zone_guest_input
-A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
-A FORWARD -i eth0.2 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -i br-guest -m comment --comment "!fw3" -j zone_guest_forward
-A FORWARD -m comment --comment "!fw3" -j reject
-A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
-A OUTPUT -o eth0.2 -m comment --comment "!fw3" -j zone_wan_output
-A OUTPUT -o br-guest -m comment --comment "!fw3" -j zone_guest_output
-A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
-A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
-A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j R                                                                                                      ETURN
-A syn_flood -m comment --comment "!fw3" -j DROP
-A zone_guest_dest_ACCEPT -o br-guest -m comment --comment "!fw3" -j ACCEPT
-A zone_guest_dest_REJECT -o br-guest -m comment --comment "!fw3" -j reject
-A zone_guest_forward -m comment --comment "!fw3: Custom guest forwarding rule chain" -j forwarding_guest_rule
-A zone_guest_forward -m comment --comment "!fw3: Zone guest to wan forwarding policy" -j zone_wan_dest_ACCEPT
-A zone_guest_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_guest_forward -m comment --comment "!fw3" -j zone_guest_dest_REJECT
-A zone_guest_input -m comment --comment "!fw3: Custom guest input rule chain" -j input_guest_rule
-A zone_guest_input -p tcp -m tcp --dport 53 -m comment --comment "!fw3: Guest DNS" -j ACCEPT
-A zone_guest_input -p udp -m udp --dport 53 -m comment --comment "!fw3: Guest DNS" -j ACCEPT
-A zone_guest_input -p udp -m udp --dport 67:68 -m comment --comment "!fw3: Guest DHCP" -j ACCEPT
-A zone_guest_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_guest_input -m comment --comment "!fw3" -j zone_guest_src_REJECT
-A zone_guest_output -m comment --comment "!fw3: Custom guest output rule chain" -j output_guest_rule
-A zone_guest_output -m comment --comment "!fw3" -j zone_guest_dest_ACCEPT
-A zone_guest_src_REJECT -i br-guest -m comment --comment "!fw3" -j reject
-A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
-A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
-A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
-A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
-A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
-A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o eth0.2 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
-A zone_wan_dest_ACCEPT -o eth0.2 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_REJECT -o eth0.2 -m comment --comment "!fw3" -j reject
-A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
-A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
-A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
-A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
-A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
-A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
-A zone_wan_src_REJECT -i eth0.2 -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Sun Oct 14 05:20:37 2018

Here's the result for the hypervisor. Note that this does not include the rules I added above to accept traffic on :8080. I've since rebooted the hypervisor, and these are the defaults on the box. There are no iptables rules on the guest (iptables-save is empty).

sol:/root# iptables-save

Summary
# Generated by iptables-save v1.6.0 on Sun Oct 14 05:16:50 2018
*nat
:PREROUTING ACCEPT [1639:144788]
:INPUT ACCEPT [180:35391]
:OUTPUT ACCEPT [58:4048]
:POSTROUTING ACCEPT [1414:96371]
-A PREROUTING -p tcp -m tcp --dport 8080 -j DNAT --to-destination 192.168.2.2:80
COMMIT
# Completed on Sun Oct 14 05:16:50 2018
# Generated by iptables-save v1.6.0 on Sun Oct 14 05:16:50 2018
*mangle
:PREROUTING ACCEPT [275645:124995488]
:INPUT ACCEPT [1180:154956]
:FORWARD ACCEPT [274362:124823458]
:OUTPUT ACCEPT [698:109821]
:POSTROUTING ACCEPT [275060:124933279]
-A POSTROUTING -o virbr1 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sun Oct 14 05:16:50 2018
# Generated by iptables-save v1.6.0 on Sun Oct 14 05:16:50 2018
*filter
:INPUT ACCEPT [1181:154996]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [700:110397]
-A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.2.0/24 -o virbr1 -j ACCEPT
-A FORWARD -s 192.168.2.0/24 -i virbr1 -j ACCEPT
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
-A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
COMMIT
# Completed on Sun Oct 14 05:16:50 2018

https://www.libvirt.org/hooks.html

# cat /etc/libvirt/hooks/network
#!/bin/bash

if [ "$2" = "started" ]
then
	iptables -D "FORWARD" -i "virbr1" \
		-j "REJECT" --reject-with "icmp-port-unreachable"
	iptables -D "FORWARD" -o "virbr1" \
		-j "REJECT" --reject-with "icmp-port-unreachable"
fi

exit 0

Ok, I added /etc/libvirt/hooks/network as shown in your reply. I can see from iptables-save that both the the -i and -o REJECT rules were removed, but it didn't make any difference in behavior.

I've now fully re-tested this in three basic combinations, as described below. Except for problems forwarding port 80 and port 22 off the internet to my guest VM, everything seems to be working as expected.

Are there other diagnostics I can provide to help debug this further?

I haven't (yet) figured out how to watch what's happening when traffic hits the internet-facing port 22 or port 80. It seems like that's where something is getting lost...?


The three combinations I re-tested were:

  • Wired network - eth0 on 192.168.1.41, virsh network with <forward mode='route'/>
  • Wired network - eth0 on 192.168.1.41, virsh network with <forward mode="nat" dev="eth0"/>
  • Wireless Network - wlan0 on 192.168.1.41, virsh network with <forward mode='route'/>

Note the change from 192.168.1.40 (as in my original post) and 192.168.1.41 in this setup.

In all cases:

  • Hosts on 192.168.1 have connectivity to hosts on 192.168.1
  • Hosts on 192.168.1 have connectivity to hosts on 192.168.2
  • Hosts on 192.168.1 have internet connectivity
  • Hosts on 192.168.2 have connectivity to hosts on 192.168.1
  • Hosts on 192.168.2 have connectivity to hosts on 192.168.2
  • Hosts on 192.168.2 have internet connectivity
  • Router can successfully ping 192.168.2.2, nc 192.162.2.2 80 and ssh 192.168.2.2 -p 22
  • Router can successfully ping 192.168.1.41, nc 192.162.1.41 8080 and ssh 192.168.1.41 -p 2222
  • Internet traffic is NOT forwarded through regardless of whether I set up port forwards in OpenWrt to 192.168.2.2 or 192.168.1.41

For reference, these are the current libvirt hooks that are in place on the hypervisor:

#!/bin/bash
# /etc/libvirt/hooks/network

if [ "$2" = "started" ]
then
   iptables -D "FORWARD" -i "virbr1" -j "REJECT" --reject-with "icmp-port-unreachable"
   iptables -D "FORWARD" -o "virbr1" -j "REJECT" --reject-with "icmp-port-unreachable"
fi

exit 0
#!/bin/sh
# /etc/libvirt/hooks/qemu

if [ "$1" = "mercury" ]; then

   if [ "$2" = start ]; then

      # Forward traffic from sol to mercury, used for a port forward on the router
      iptables -I FORWARD -o virbr1 -d 192.168.2.2 -j ACCEPT
      iptables -t nat -I PREROUTING -p tcp --dport 8080 -j DNAT --to 192.168.2.2:80
      iptables -t nat -A OUTPUT -p tcp --dport 8080 -j DNAT --to 192.168.2.2:80
      iptables -t nat -I PREROUTING -p tcp --dport 2222 -j DNAT --to 192.168.2.2:22
      iptables -t nat -A OUTPUT -p tcp --dport 2222 -j DNAT --to 192.168.2.2:22

   elif [ "$2" = stopped ]; then

      # Remove the forwarding rules
      iptables -D FORWARD -o virbr1 -d 192.168.2.2 -j ACCEPT
      iptables -t nat -D PREROUTING -p tcp --dport 8080 -j DNAT --to 192.168.2.2:80
      iptables -t nat -D OUTPUT -p tcp --dport 8080 -j DNAT --to 192.168.2.2:80
      iptables -t nat -D PREROUTING -p tcp --dport 2222 -j DNAT --to 192.168.2.2:22
      iptables -t nat -D OUTPUT -p tcp --dport 2222 -j DNAT --to 192.168.2.2:22

  fi

fi

exit 0

This is the current result from iptables-save on the hypervisor:

# Generated by iptables-save v1.6.0 on Sun Oct 14 16:33:10 2018
*nat
:PREROUTING ACCEPT [169:17138]
:INPUT ACCEPT [12:1586]
:OUTPUT ACCEPT [44:3037]
:POSTROUTING ACCEPT [173:11846]
-A PREROUTING -p tcp -m tcp --dport 2222 -j DNAT --to-destination 192.168.2.2:22
-A PREROUTING -p tcp -m tcp --dport 8080 -j DNAT --to-destination 192.168.2.2:80
-A OUTPUT -p tcp -m tcp --dport 8080 -j DNAT --to-destination 192.168.2.2:80
-A OUTPUT -p tcp -m tcp --dport 2222 -j DNAT --to-destination 192.168.2.2:22
COMMIT
# Completed on Sun Oct 14 16:33:10 2018
# Generated by iptables-save v1.6.0 on Sun Oct 14 16:33:10 2018
*mangle
:PREROUTING ACCEPT [2074:242931]
:INPUT ACCEPT [335:37644]
:FORWARD ACCEPT [1702:198004]
:OUTPUT ACCEPT [287:70349]
:POSTROUTING ACCEPT [1989:268353]
-A POSTROUTING -o virbr1 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sun Oct 14 16:33:10 2018
# Generated by iptables-save v1.6.0 on Sun Oct 14 16:33:10 2018
*filter
:INPUT ACCEPT [336:37684]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [287:71063]
-A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.2.2/32 -o virbr1 -j ACCEPT
-A FORWARD -d 192.168.2.0/24 -o virbr1 -j ACCEPT
-A FORWARD -s 192.168.2.0/24 -i virbr1 -j ACCEPT
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
-A OUTPUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
COMMIT
# Completed on Sun Oct 14 16:33:10 2018

Is there specific reason to use virtual network, especially in NAT-mode?
The simplest way to make it work is setting up a network bridge.
You can do it with systemd-networkd or some other network manager.

I've had the hypervisor set up using a virtual network and <forward mode='route'/> for about two years. That was my preferred solution at the time because it was easy to manage the IP addresses assigned to the guests. Today, I tried the <forward mode="nat" dev="eth0"/> setup just for the sake of completeness - I wanted to make sure behavior was the same.

Do you think the virtual network is part of the problem? In my own debugging I had tentatively ruled it out, since the router can connect successfully to both the hypervisor and the guest on the ports I want to forward to.

Two other pieces of information I can provide:

  1. I've noticed that the packet counts for the zone_wan chains related to these ports are always zero. This makes me suspicious that traffic isn't even being accepted on those ports.
  2. I tried forwarding different internet-facing ports (8080 and 2222 rather than 80 and 22) to see whether that made a difference in behavior. It didn't.

Those (reflection) firewall rules on OpenWrt look suspicious.
Where are they coming from?
Remove them, use some nmap-online service in the internet and check iptables rule counters for nat and filter tables.


Not likely, but it requires more troubleshooting.


Anyway, I recommend to try network bridging, it really makes life easier.

The (reflection) rules appear to be coming from the port forward entries I set up under Network > Firewall > Port Forwards. When I remove the port forward entries, the reflection rules disappear. Maybe there's some other way that I should be setting up those forward rules?

Just to eliminate the virtual network from the equation, I attempted to forward internet traffic from port 22 to a different host on the internal network - a Raspberry Pi that's on the wifi (192.168.1.37:22). SSH requests don't get through to that device, either.

I see, trying to add port forwarding on my router also creates them.
But OpenWrt-18.06.1-x86_64 on VPS does not have those rules and it works just fine.
May be there's some difference in DNAT implementation for x86_64 and other platforms.
I assume it's standard behavior.

Let's temporary forget about port forwarding and test port opening on OpenWrt from the internet.
Better with service fingerprint to make sure it's really your service.
And also verify iptables rule counters change:

iptables-save -c | grep "RULE_NAME"

<sigh>

I figured out the root cause. Somewhere in the midst of all of this, my cable modem got a new IP address. This usually doesn't happen more than once every 12-18 months, but I rebooted the modem yesterday, and that apparently triggered it. I have external monitoring that would normally have warned me to change the DNS entry, but of course that had already started complaining when I took the old router offline and I ignored it.

Once I was actually testing against the correct public IP address, everything worked as I expected. I was able to set up forwarding directly to the guest on 192.168.2.2:22 and 192.168.2.2:80, and now HTTP and SSH work like they did before.

I'm rather embarrassed that I didn't think to check this sooner. I appreciate all of your help in debugging the problem, even if in the end it was entirely my fault.

In case anyone else is looking at this thread, the qemu and network hooks were no longer required once I figured out what was going on.

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.