Virtualbox Openwrt works fine but clients connecting to vm-router have no internet

moshem · August 11, 2020, 6:37am

opkg update was not working when I switched adapters as there was no internet

I am using latest virtualbox 6.0 on macos 10.15.5

my router is a netgear ea7500, pretty standard

not sure sophos is the problem because I did manage at one point to create another interface for my VPN provider and had clients connect to the vm-router and have internet, but on a new vm without vpn it doesn't work...

I followed the VPN guide here: https://support.surfshark.com/hc/en-us/articles/360010347939-How-to-set-up-a-router-with-OpenWRT

anon50098793 · August 11, 2020, 6:40am

i believe in the past your wan was 'vbox-natted' which we know works... ( but you must have pointed the clients to the laptop's ip? and the lan was vbox-natted too? )

try re-install
then ask about 'macos bridged airport not-working' in the virtualbox forums ( there are many posts about this, mostly from v5 and before )

once you get simple lan bridged opkg update working, we know the subsystems are good to proceed with anything openwrt... you could try changing (vm-lan) back to dhcp to exclude an ip conflict.

moshem · August 12, 2020, 4:09pm

Hey

So I did some more research, tried

reinstall virtualbox
remove anti virus
virtualbox on an second mac
virtualbox on a windows PC

all using the VM in the link above, very simple 1 nic, bridged.

all resulted in a working router vm, opkg update working, ping to self, ping to 192.168.1.1 working

all resulted in the same problem of no internet to various clients (including another windows PC) when setting its gateway address to the VM router bridged IP

all clients have working internet when the GW is set to the real router IP and traceroute shows a proper list via the gateway, all client stopped working and traceroute gave nothing once the gw was set to the VPN router

not sure what else can I do or try..

vgaetera · August 12, 2020, 4:48pm

You should probably enable masquerading for the LAN zone as well as disable bridging or set up bridge firewall, otherwise your LAN client traffic may bypass the VM router.

moshem · August 12, 2020, 5:50pm

I tried masquerade, had no effect
I tried normal eth0 interface without bridging, no dice

vgaetera · August 12, 2020, 5:57pm

You need to implement both simultaneously.

moshem · August 12, 2020, 6:19pm

done, still nothing

I am now using
eth0 bridged virtualbox adapter DHCP (gets ip from real router)
eth1 host only adpater DHCP (gets ip from virtualbox internal 192.168.56.1)

/etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd2c:fe7f:c7e6::/48'

config interface 'lan'
        option ifname 'eth0'
        option proto 'dhcp'

#config interface 'wan'
#       option ifname 'eth1'
#        option proto 'dhcp'

#config route 'route1'
#        option interface 'wan'
#        option target '192.168.1.0'
#        option netmask '255.255.255.0'
#        option gateway '192.168.56.1'

/etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config rule
        option src '*'
        option target 'ACCEPT'
        list proto 'all'

#config zone                                     
#        option network 'wanlan lan wan'         
#        option input 'ACCEPT'                   
#        option forward 'ACCEPT'                 
#        option masq '1'                         
#        option output 'ACCEPT'                  
#        option name 'wan'                       
                                                
config zone                                     
        option name 'lan'                       
        option input 'ACCEPT'                   
        option forward 'ACCEPT'                 
        option masq '1'                         
        option output 'ACCEPT'                  
        option network 'lan'

/etc/resolv.conf

search lan
nameserver 192.168.1.1
nameserver 8.8.8.8

client config (no internet)

Manual
IP 192.168.1.60
subnet 255.255.255.0
router/gateway 192.168.1.56 (virtual router)
dns 192.168.1.1/192.168.1.56 (real router/fake router, tried both)

vgaetera · August 12, 2020, 6:37pm

What is the purpose of the WAN interface?
It has a route which conflicts with the LAN interface.
Moreover, it will likely result in a collision of firewall rules, if you plan to keep WAN and LAN interfaces on the same network but assigned different firewall zones.

moshem · August 12, 2020, 6:43pm

I added it to utilize the virtualbox router, after giving up on a single bridged adapter.

I just dropped the wan interface again and got the same result

I updated the above post with the new updated files, commented out WAN and routes and FW rules

vgaetera · August 12, 2020, 6:50pm

# OpenWrt
opkg update
opkg install tcpdump
tcpdump -n -i any icmp

# Clients
ping -w 3 openwrt_vm_ip
ping -w 3 8.8.8.8

moshem · August 12, 2020, 7:23pm

thats great

ping -w 3 192.168.1.56

19:19:50.731666 IP 192.168.1.12 > 192.168.1.56: ICMP echo request, id 1, seq 21, length 40
19:19:50.731713 IP 192.168.1.56 > 192.168.1.12: ICMP echo reply, id 1, seq 21, length 40
19:19:51.754060 IP 192.168.1.12 > 192.168.1.56: ICMP echo request, id 1, seq 22, length 40
19:19:51.754096 IP 192.168.1.56 > 192.168.1.12: ICMP echo reply, id 1, seq 22, length 40
19:19:52.782945 IP 192.168.1.12 > 192.168.1.56: ICMP echo request, id 1, seq 23, length 40
19:19:52.782985 IP 192.168.1.56 > 192.168.1.12: ICMP echo reply, id 1, seq 23, length 40
19:19:53.813259 IP 192.168.1.12 > 192.168.1.56: ICMP echo request, id 1, seq 24, length 40
19:19:53.813293 IP 192.168.1.56 > 192.168.1.12: ICMP echo reply, id 1, seq 24, length 40

the client got reply from 192.168.1.56

ping -w 3 8.8.8.8
got nothing on tcp dump!

my windows 10 client config

IPv4 : on
IP manual
IP:192.168.1.12
subnet len: 24
gw: 192.168.1.56
dns: 192.168.1.56

moshem · August 12, 2020, 7:45pm

I then tried the same tcpdump monitoring on macos host and saw interesting stuff
it does report it, unlike the virtualbox router, but it seems to try to reach the real physical router (192.168.1.1) instead of the virtualbox one . probably why the virtualbox router (192.168.1.56) never gets it. not sure why it does that though

22:42:35.432146 IP 192.168.1.12 > 8.8.8.8: ICMP echo request, id 1, seq 55, length 72
22:42:40.637769 IP 192.168.1.35 > 192.168.1.1: ICMP 192.168.1.35 udp port 50439 unreachable, length 36
22:42:40.637957 IP 192.168.1.35 > 192.168.1.1: ICMP 192.168.1.35 udp port 56686 unreachable, length 36
22:42:41.672923 IP 192.168.1.12 > 8.8.8.8: ICMP echo request, id 1, seq 56, length 72
22:42:46.032769 IP 192.168.1.12 > 216.58.207.78: ICMP echo request, id 1, seq 57, length 40
22:42:47.430836 IP 192.168.1.12 > 216.58.207.78: ICMP echo request, id 1, seq 58, length 40
22:42:48.938315 IP 192.168.1.12 > 216.58.207.78: ICMP echo request, id 1, seq 59, length 40
22:42:50.424042 IP 192.168.1.12 > 216.58.207.78: ICMP echo request, id 1, seq 60, length 40
22:43:06.994946 IP 192.168.1.35 > 192.168.1.1: ICMP 192.168.1.35 udp port 62221 unreachable, length 36

ip map

192.168.1.1 physical router
192.168.1.12 windows client
192.168.1.35 macos host
192.168.1.56 virtualbox openwrt router bridged adapter

vgaetera · August 12, 2020, 8:29pm

It seems that the host redirects traffic to its own gateway.
Sorry, I'm not familiar with VirtualBox/MacOS enough to continue troubleshooting.

moshem · August 12, 2020, 9:27pm

I got some progress
I added one more adapter a host only adapter to virtualbox, now I get the traffic both on the mac tcpdump and on the vm, no redirection to the original network

ping -w 3 8.8.8.8

21:26:10.891918 IP 192.168.1.12 > 8.8.8.8: ICMP echo request, id 1, seq 137, length 40
21:26:11.867244 IP 192.168.1.12 > 8.8.8.8: ICMP echo request, id 1, seq 138, length 40
21:26:13.452588 IP 192.168.1.12 > 8.8.8.8: ICMP echo request, id 1, seq 139, length 40
21:26:14.887510 IP 192.168.1.12 > 8.8.8.8: ICMP echo request, id 1, seq 140, length 40
21:26:16.379542 IP 192.168.1.56 > 8.8.8.8: ICMP 192.168.1.56 udp port 28359 unreachable, length 149
21:26:16.379552 IP 192.168.1.56 > 8.8.8.8: ICMP 192.168.1.56 udp port 28359 unreachable, length 149
21:26:16.394879 IP 192.168.1.56 > 8.8.4.4: ICMP 192.168.1.56 udp port 28359 unreachable, length 149
21:26:16.394887 IP 192.168.1.56 > 8.8.4.4: ICMP 192.168.1.56 udp port 28359 unreachable, length 149

I still get no replies on the client

moshem · August 12, 2020, 9:30pm

the ideas on how to do so are from

/etc/config/network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd2c:fe7f:c7e6::/48'

config interface 'lan'                         
        option type 'bridge'                   
        option ifname 'eth0'                   
        option proto 'static'                  
        option ipaddr '192.168.1.56'
        option gateway '192.168.1.1'
        option netmask '255.255.255.0'
        option dns '192.168.1.1'      
        option hostname 'openwrt-mac'  
                                      
config interface 'wan'                
        option ifname 'eth1'          
        option proto 'dhcp'         
        option peerdns '0'          
        list dns '8.8.8.8'  
        list dns '8.8.4.4'  
                            
config interface 'wan6'    
        option ifname 'eth1'
        option proto 'dhcpv6'

/etc/config/firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option mtu_fix '1'
	option masq '0'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config include
	option path '/etc/firewall.user'

/etc/resolve.conf

search lan
nameserver 127.0.0.1

the router is up, has IPs, has internet access, has dns access, can ping 8.8.8.8

mk24 · August 12, 2020, 9:53pm

Your only connection from the Macbook to anything is its single wifi adapter running as a client then? I don't think that can be put in a bridge. A wifi client only holds one MAC / IP address, and that would be taken by OSX.

Probably you can forego OpenWrt and run OpenVPN directly on OSX, then have your other machines on the LAN configured to use the Macbook as their gateway to the Internet.

moshem · August 12, 2020, 10:15pm

my macbook (the host for the virtualbox openwrt vm) is connected to the physical router via wifi - yes

I am building the vm so I can run it on my mac for development & testing, but intend to use it on a windows machine and a linux box

not sure I understand why would a wifi adapter be limited like that and an ethernet adapter wouldn't, thats the idea behind virtualization isn't?

slh · August 12, 2020, 10:24pm

As mk24 suggested, your problem is not really the host OS (although that might provide some additional hurdles), but your chosen connection to the network. The IEEE 802.11 standards don't support bridging anything, it's simply not in the wireless standards (which only account for three MAC addresses in the frames, while you'd need four for bridging).

Linux and its nl80211 wireless stack has added an extension to the 802.11 standards called 4addr (often dubbed WDS), which does what the name implies - offering the ability to transport a forth MAC address to the wireless frames (via a clever combination of AP and STA interfaces). However this 4addr mode is specific to linux and mainline/ nl80211 based drivers, it's not available for MacOS or Windows (some proprietary wireless vendors have similar, but incompatible, features for their offerings). The easiest solution to your problem would probably to hard-wire the host running the OpenWrt VM via ethernet. If possible, I'd avoid the WDS/ 4addr route for the host networking, as this complicates your situation considerably (but it's possible, running linux on the host - and the main router providing the internet to this host).

moshem · August 15, 2020, 3:26pm

Ok,

maybe it was the wireless adapter...

tried on a wired windows host
created a new VM, fresh openwrt install, 19.07.3
only one bridged adapter
ip: 192.168.1.56

and.. it works, if I set another windows client windows 10 to use 192.168.1.56 (the vm router) as the gateway and dns it routes well and works

/etc/config/network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd81:2c66:2212::/48'

config interface 'lan'
	option ifname 'eth0'
	option proto 'static'
	option ipaddr '192.168.1.56'
	option netmask '255.255.255.0'
	option gateway '192.168.1.1'
	option dns '192.168.1.1'
	option ip6assign '60'

config interface 'wan'
	option ifname 'eth0'
	option proto 'dhcp'
	option auto '1'
	option peerdns '0'
	list dns '162.252.172.57'
	list dns '149.154.159.92'

config interface 'wan6'
	option ifname 'eth0'
	option proto 'dhcpv6'
	option reqprefix 'auto'
	option reqaddress 'try'
	option auto '1'

I still had random connection issues until I disabled the bridge

thank you everyone!

anon50098793 · August 15, 2020, 4:02pm

thanks for the update and thanks to @vgaetera @mk24 and @slh for really helping out ( pensive looks humbled to me )

never used bridged over wifi before... so suspected something macos related... apologies... will update the wiki with a note regarding this...

@moshem kudos to you also for sticking with it, and providing useful and relevant debugging info... generally you nailed most of this... so I wouldn't be too deterred from this experience... but yeah... wired 'hosts' seem to be the way to go...

fwiw... just for brevvity ( don't suggest you try this, but worth mentioning )... one could theoretically also use vboxNAT mode as a workaround... to overcome this 802.11 limitation... mostly applicable for a 'single lan-service' scenario... i suspect when you said that it was working in the past... this was how...