Only the first interface in br-lan is working properly with a VirtualBox x86_64 OpenWrt

Hello,

It seems that only the first interface mentioned in option ifname of /etc/config/network gets a proper connectivity within a VirtualBox x86_64 OpenWrt.

The OpenWrt router runs in a VirtualBox.
NIC1 is bridged with br1 (a local bridge). It is seen as eth0 by OpenWrt.
NIC2 is bridged with a physical interface (access to the outer lan) It is seen as eth1 by OpenWrt.
NIC3 is bridged with br2 (a local bridge). It is seen as eth2 by OpenWrt.

2 Alpine containers are respectively connected br1 and br2, all links are up. Both containers receive an IP address (192.168.1.122 and 192.168.1.154) upon DHCP request. However with

config interface 'lan'
        option force_link '1'
        option type 'bridge'
        option ifname 'eth0 eth2'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'

Only the container connected to eth0<->br1 is able to ping the router. With

config interface 'lan'
        option force_link '1'
        option type 'bridge'
        option ifname 'eth2 eth0'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'

Only the container connected to eth2<->br2 is able to ping the router.

I tried the following on both 19.07.2 and snapshot, with and without the firewall. The wrong checksum inherent to virtual bridged interfaces is addressed.

What puzzles me is that both get IP addresses through DCHP requests.
Arp requests confirms that the router sees both containers:

IP address       HW type     Flags       HW address            Mask     Device
192.168.1.154    0x1         0x2         00:16:3e:6c:5b:5b     *        br-lan
192.168.1.122    0x1         0x2         00:16:3e:97:f1:c2     *        br-lan
192.168.64.1     0x1         0x2         d8:58:d7:00:43:ad     *        eth1

and the routes seems ok

root@OpenWrt:/# ip r s
default via 192.168.64.1 dev eth1  src 192.168.64.173 
192.168.1.0/24 dev br-lan scope link  src 192.168.1.1 
192.168.64.0/21 dev eth1 scope link  src 192.168.64.173

as well as links and the bridge

root@OpenWrt:/# ip l sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP qlen 1000
    link/ether 08:00:27:66:be:81 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether 08:00:27:87:07:78 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP qlen 1000
    link/ether 08:00:27:ef:4e:23 brd ff:ff:ff:ff:ff:ff
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 08:00:27:ef:4e:23 brd ff:ff:ff:ff:ff:ff
root@OpenWrt:/# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          7fff.080027ef4e23       no              eth2
                                                        eth0

With both containers pinging the router
Below is a tcpdump within the router, where it can be seen that pings are received and answered to a single one, BUT there is an answered arp request from the other one.

root@OpenWrt:/# tcpdump -vv
18:11:34.691933 IP (tos 0x0, ttl 64, id 5950, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.154 > OpenWrt.lan: ICMP echo request, id 18689, seq 401, length 64
18:11:34.692025 IP (tos 0x0, ttl 64, id 32843, offset 0, flags [none], proto ICMP (1), length 84)
    OpenWrt.lan > 192.168.1.154: ICMP echo reply, id 18689, seq 401, length 64
18:11:35.239168 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has OpenWrt.lan tell 192.168.1.122, length 46
18:11:35.239207 ARP, Ethernet (len 6), IPv4 (len 4), Reply OpenWrt.lan is-at 08:00:27:ef:4e:23 (oui Unknown), length 28
18:11:35.692153 IP (tos 0x0, ttl 64, id 5987, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.154 > OpenWrt.lan: ICMP echo request, id 18689, seq 402, length 64
18:11:35.692239 IP (tos 0x0, ttl 64, id 32865, offset 0, flags [none], proto ICMP (1), length 84)
    OpenWrt.lan > 192.168.1.154: ICMP echo reply, id 18689, seq 402, length 64

Below is a tcpdump from the physical on both veth

root@bioman2:~# tcpdump -vvi veth45d893e1
tcpdump: listening on veth45d893e1, link-type EN10MB (Ethernet), capture size 262144 bytes
20:10:11.353934 IP (tos 0x0, ttl 64, id 28066, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.122 > biorouter.home.prog.in: ICMP echo request, id 17153, seq 311, length 64
20:10:12.354157 IP (tos 0x0, ttl 64, id 28214, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.122 > biorouter.home.prog.in: ICMP echo request, id 17153, seq 312, length 64
20:10:13.354318 IP (tos 0x0, ttl 64, id 28397, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.122 > biorouter.home.prog.in: ICMP echo request, id 17153, seq 313, length 64
root@bioman2:~# tcpdump -vvi vethbcdaf621
tcpdump: listening on vethbcdaf621, link-type EN10MB (Ethernet), capture size 262144 bytes
20:10:53.686524 IP (tos 0x0, ttl 64, id 1058, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.154 > biorouter.home.prog.in: ICMP echo request, id 18689, seq 360, length 64
20:10:53.687342 IP (tos 0x0, ttl 64, id 30781, offset 0, flags [none], proto ICMP (1), length 84)
    biorouter.home.prog.in > 192.168.1.154: ICMP echo reply, id 18689, seq 360, length 64
20:10:54.686716 IP (tos 0x0, ttl 64, id 1303, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.154 > biorouter.home.prog.in: ICMP echo request, id 18689, seq 361, length 64
20:10:54.687265 IP (tos 0x0, ttl 64, id 30815, offset 0, flags [none], proto ICMP (1), length 84)
    biorouter.home.prog.in > 192.168.1.154: ICMP echo reply, id 18689, seq 361, length 64

Any hint or idea would be greatly appreciated. Thank you.

Your post was a bit long for me to read it all but it does not sound VirtualBox specific. Paste a 'cat /etc/config/firewall' for more help with that or lookup the multitude of docs that cover additional network setup.

If your br2 client (Alpine) get's an ip as you say... then most likely you have not setup a proper firewall zone et.al.(and related dns,etc.) for that network.

I believe you can take a shortcut and add eth2 to the lan firewall zone.

( NOTE: You can replace the br in virtualbox to something else hostonly/vmnet/physicalif's if you are convinced that it is virtualbox and you need to rule it out, flipping stuff around in /etc/config/network is a very confusing [for most] way to go about troubleshooting in a virtual environment )

Hello, Thanks for your suggestions!
My /etc/config/firewall is almost vanilla. It is posted at the bottom of this message. I thought that the point of bridging the eth0 and eth2 was to have them addressed exactly the same way by the firewall, as a lan through the br-lan interface.
I tried to add eth2, and both eth2 and eth0 to the network in firewall.zone with no luck.

I found a way to have the second container ping the router. If an arping command is running, both the router and the container can ping each other. It stops about 2 second after the arping is killed.

root@OpenWrt:~# arping -I eth2 -s 192.168.1.1 192.168.1.154 &
root@OpenWrt:~# ARPING 192.168.1.154 from 192.168.1.1 eth2
root@OpenWrt:~# ping 192.168.1.154
PING 192.168.1.154 (192.168.1.154): 56 data bytes
64 bytes from 192.168.1.154: seq=0 ttl=64 time=0.664 ms
64 bytes from 192.168.1.154: seq=2 ttl=64 time=1.073 ms
64 bytes from 192.168.1.154: seq=3 ttl=64 time=0.893 ms

I will have to better understand how the frame layer is working exactly. I will try to create a second full blown zone per se as well.

root@OpenWrt:~# cat /etc/config/firewall
config defaults 
        option syn_flood        1
        option input            ACCEPT
        option output           ACCEPT
        option forward          REJECT
# Uncomment this line to disable ipv6 rules
#       option disable_ipv6     1

config zone
        option name             lan
        list   network          'lan'
        list   network          'eth0'
        list   network          'eth2'
        option input            ACCEPT
        option output           ACCEPT
        option forward          ACCEPT

config zone
        option name             wan
        list   network          'wan'
        list   network          'wan6'
        option input            REJECT
        option output           ACCEPT
        option forward          REJECT
        option masq             1
        option mtu_fix          1
config forwarding
        option src              lan
        option dest             wan

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
        option name             Allow-DHCP-Renew
        option src              wan
        option proto            udp
        option dest_port        68
        option target           ACCEPT
        option family           ipv4

# Allow IPv4 ping
config rule
        option name             Allow-Ping
        option src              wan
        option proto            icmp
        option icmp_type        echo-request
        option family           ipv4
        option target           ACCEPT

config rule
        option name             Allow-IGMP
        option src              wan
        option proto            igmp
        option family           ipv4
        option target           ACCEPT

# Allow DHCPv6 replies
# see https://dev.openwrt.org/ticket/10381
config rule
        option name             Allow-DHCPv6
        option src              wan
        option proto            udp
        option src_ip           fc00::/6
        option dest_ip          fc00::/6
        option dest_port        546
        option family           ipv6
        option target           ACCEPTconfig rule
        option name             Allow-MLD
        option src              wan
        option proto            icmp
        option src_ip           fe80::/10
        list icmp_type          '130/0'
        list icmp_type          '131/0'
        list icmp_type          '132/0'
        list icmp_type          '143/0'
        option family           ipv6
        option target           ACCEPT

# Allow essential incoming IPv6 ICMP traffic
config rule
        option name             Allow-ICMPv6-Input
        option src              wan
        option proto    icmp
        list icmp_type          echo-request
        list icmp_type          echo-reply
        list icmp_type          destination-unreachable
        list icmp_type          packet-too-big
        list icmp_type          time-exceeded
        list icmp_type          bad-header
        list icmp_type          unknown-header-type
        list icmp_type          router-solicitation
        list icmp_type          neighbour-solicitation
        list icmp_type          router-advertisement
        list icmp_type          neighbour-advertisement
        option limit            1000/sec
        option family           ipv6
        option target           ACCEPT

# Allow essential forwarded IPv6 ICMP traffic
config rule
        option name             Allow-ICMPv6-Forward
        option src              wan
        option dest             *
        option proto            icmp
        list icmp_type          echo-request
        list icmp_type          echo-reply
        list icmp_type          destination-unreachable
        list icmp_type          packet-too-big
        list icmp_type          time-exceeded
        list icmp_type          bad-header
        list icmp_type          unknown-header-type
        option limit            1000/sec
        option family           ipv6
        option target           ACCEPT

config rule
        option name             Allow-IPSec-ESP
        option src              wan
        option dest             lan
        option proto            esp
        option target           ACCEPT

config rule
        option name             Allow-ISAKMP
        option src              wan
        option dest             lan
        option dest_port        500
        option proto            udp
        option target           ACCEPT

# include a file with users custom iptables rules
config include  
        option path /etc/firewall.user

When I talked about bridging... I was referring to your host bridges...

It seems that the issue is due to the fact the promiscuous mode on the VirtualBox interfaces was set to "Deny" on all interfaces. I did not perform all the tests, but setting promiscuous mode to "Allow All" may save my afternoon.

This post put me on the right track.

I will have to review extensively the wiki page and review the security implications.

Despite the fact the VirtualBox box interface "Intel PRO/1000 MT Desktop" may not allow it, no error shows up in OpenWrt log

device eth2 entered promiscuous mode

Thank you.

1 Like

Use the other Network Adapter as LAN and in VirtualBox you select Bridged Adapter with Promiscuous Mode to

  • Deny if you don't want to expose bridged virtual networks to your physical network

Feel free to add a section to this page... it's my ( and bobafetathotmail ) WIP...

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.