Passing multiple network bridges in Proxmox to openWRT VM

Installing and Using OpenWrt
1

Hi,
I have OpenWRT in VM. I pass-through several hardware NICs including the one that is 'wan'. I also pass in one virtual bridge (vmbr1) which is part of 'lan' subnet. This way I can connect my other lxc or vm to 'lan'. I wanted to pass in second virtual bridge (vmbr2) for my second subnet 'sec'. But when I do that, all of the sudden my 'wan' stops working (doesn't get an ip) and thus I do not have internet connectivity and I can not use openWRT Luci interface.
This is a screenshot of the devices added to the openwrt vm.

1768077984636.png

Any ideas what could be the issues? I tried to add the vmbr2 (which is eth4 in openwrt) to a brige for 'sec' (same config for vmbr1 and 'lan' bridge). But it didn't help. I also set manually mac addresses for each device because several had same. But it also didn't help. Both bridges (vmbr1 and 2) has the same config:
1768077938891.png

In logs, when the issue occurs I see:
daemon.notice netifd: wan (2574): Command failed: ubus call network.interface notify_proto { "action": 0, "link-up": false, "keep": false, "interface": "wan" } (Permission denied)
I found an issue but I don't think its the same, adding the delay doesn't help.

2

Did you check the ethX assigned order didn't change inside openwrt after adding more virtual nics?

In my experience the ethX order as seen from openwrt doesn't really follow the net1/net2/net3 virtual interface order as seen from proxmox virtual machine config.

As a way a diagnosing this I'd suggest marking the virtual nics as disconnected on the vm config and seeing if openwrt rise up the interface (without carrier).

3

Hi, thank you for the reply. Where do I need to check the order? When I do ip a I see that the eth0 is still assigned the same mac and eth4 gets the new mac (the added bridge). Is this what you meant, that the eth's got scrambled to different numbers?

The second part I tried just now to disable both bridges on the vm and the issue persists (no IP, no internet).

I also tried to segment the vmbr1 with vlan ids and that worked, but I was quite sure that will work, because one bridge was already okay. But at least its and alternative way I can do this.

4

It’s hard with limited info. Thora is probably right about what the problem is.

It’s saying permission denied, which is strange to see if it’s a virtual machine and not a container.

After you make this change, does the device you assigned to ‘wan’ still exist when you run the command ‘ip link’ ? Is it a vlan interface by chance ?

You’ve posted a screenshot of the bridge config, can you post a screenshot of the nic config for the virtual machine ? Is there some kind of network name or number option and they’re both set to the same or something ?

5

Yes the wan device (eth1) still exists.

can you post a screenshot of the nic config for the virtual machine

You mean this config?

image
image962×410 31 KB

Feel free to ask for more, I actually don't know a lot so I don't know what can be useful. Also, I can not copy from the vm console for some reason, so it is hard for me to show result of that textually. Like e.g. the ip link result:

image
image910×413 64.8 KB

Here btw. you can see my new setup with VLANs. But the second bridge vmbr2 (eth4) still breaks it.
eth0 is the vmbr1. eth1 - eth3 are physical NICs and wlan1 is physical wifi card all 4 pcie devices are passed-through.

6

ok, i’m not sure why eth1 would stop working if it is the wan device

eth4 looks to be unconfigured, so openwrt hasn’t even brought it up

Are you passing the nics through using the pci passthrough or are you using macvtap direct ? Why does eth4 have a mac address so similar to the physical devices ?

A quick way you can potentially test if things have been switched around in the vm is to actually switch the cables around on the nics and see what happens, if that works then you know straight away what it is. Bedtime for me :grinning_face:

1 Like
7

This is slightly off topic and the last post i’ll make in this one so someone competent can take over.

If you aren’t using pci passthrough or macvtap direct, there is another way where you can just make bridges on the host side for every ethernet device you have, you then put the device into it’s bridge.

eg…

ip link add br-adapter1 type bridge
ip link set enp1s0 master br-adapter1

ip link set enp1s0 mtu 1500 up
ip link set br-adapter1 mtu 1500 up

So you would do this for each one. Then you would set your vm config to use the adapter1 bridge etc for that nic to be passed through.

If you are using vlans in the vm, you need to issue this directive from the host via this way or via changing/saving the sysctl:

echo 16384 > /sys/class/net/br-adapter1/bridge/group_fwd_mask

This will allow vlan packets to be forwarded.

Once you’ve done this kind of set up you dont need to make an extra adapter inside the vm, the host can connect through the bridge on it’s side. You can either just run directly off it or if you need to use vlan from inside the host you can make a veth pair for the virtual connection.

eg..

ip link add type veth
ip link set veth0 name ho-adapter1
ip link set veth1 name ho-lan

So then you make a vlan interface off ho-lan..

ip link add link ho-lan ho-lanv type vlan id 100

Then wire one side of the veth into the adapter1 bridge, and the newly created vlan interface into your normal host bridge. eg.. (I assume br-host is already made and the main bridge for the host)

ip link set master br-adapter1 ho-adapter1
ip link set master br-host ho-lanv
ip link set ho-adapter1 up
ip link set ho-lan up
ip link set ho-lanv up

This way of doing it has a higher overhead than using pci passthrough, but it works and you can use vlans any way you like and it keeps the configuration within the openwrt vms clean and easy to migrate between methods.

I’m using the raw commands here but there’s probably a way to do it within the proxmox configs, else you can enable an rc.local service and play around that way until you figure it out how you like.

8

Are you passing the nics through using the pci passthrough

I am using pci passthrough. Though the bridges vmbr1 and vmbr2 do not include any physical port or anything. Its just a way to use the various subnets that I set in openwrt by vm/lxc's

A quick way you can potentially test if things have been switched around in the vm is to actually switch the cables around

I tried to move the cable around but it didn't help.

9

This is slightly off topic and the last post i’ll make in this one so someone competent can take over.

I think I do not follow :grinning_face: As said I do pci passthrough. And the bridges are there only for the link between openwrt and any potential vm/lxc that I want to be part of a subnet on that router.

10

This does not sound right, if by wan you mean openwrt default wan interface, having no ip on that interface should not preclude you from accessing luci interface, which by default it’s only reachable from lan interface.

Having such a complex setup with virtual and physical nics, I'd suggest making a some sort of graphics scheme to show in detail how connections between openwrt interfaces, virtual nics, proxmox bridges, and physical links are set up, so someone more knowledgeable could understand and make more useful suggestions.

11

Hi, yeah I admit its complex, partly because I am working with what I could get cheap second hand. And so I have e.g. 3 separate 2.5 GB NICs instead of one with 3 ports.
This is my current setup in a table + the vmbr2 that makes things not work:

Physical port PVE interface OpenWRT device OpenWRT bridge VLAN Tagged OpenWRT interface
- vmbr1 eth0 br4 1 U LAN
2 T SEC
1 enp1s0 eth1 - - - WAN
2 enp2s0 eth2 br4 1 U LAN
3 enp3s0 eth3 br4 2 U SEC
4 wlp4s0 wlan1 br4 1 ? LAN
5 enp0s31f6 - - - - -
- vmbr2 eth4

Regarding the interface, by LAN you mean the subnet in openWRT or the outer network in which the operWRT is hooked (so from openWRT point of view the WAN). I have specifically added firewall rule to be able to access it from the outside (meaning my outer network really not the internet).

12

Don't bridge in both the host and in OpenWrt as that is a network loop.

Rather than PCI passthrough a physical port, it is probably more stable and performant to set up bridges in the host that bridge the physical port to a virtual port.