Layer 2 vxlan tunnel setup

Installing and Using OpenWrt Network and Wireless Configuration
21

I'm following your suggestion and setting it via the userland utils.

So far I've deleted the old bridge, vxlan and ip alias.

I'll write down what I've set up on box A, it is the same on box B otherwise will be specified.

brctl addbr vxl-br
brctl addif vxl-br eth1
ip link add vxlan0 type vxlan id 10101 dstport 4789 local 192.168.255.1 remote 192.168.255.2 | ip link add vxlan0 type vxlan id 10101 dstport 4789 local 192.168.255.2 remote 192.168.255.1
brctl addif vxl-br vxlan0

root@Device_A:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vxl-br          8000.3e45f3a8bc26       no              vxlan0 eth1
                                                        
root@Device_B:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vxl-br          8000.167acd535e5d       no              vxlan0 eth1

Set ip on vxlan0 interface

ip a a 192.168.255.1/24 dev vxlan0 | ip a a 192.168.255.2/24 dev vxlan0

Set the links up

ip link set up dev vxl-br
ip link set up dev vxlan0
ip link set up dev eth1

root@Device_A:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vxl-br          8000.3e45f3a8bc26       no              vxlan0
                                                        eth1

Set forwarding entries:

bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 192.168.255.1 | bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 192.168.255.2

Host routes are still installed

root@Device_A:~# ip r
default via 10.0.0.1 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.10 
192.168.255.0/24 dev vxlan0 proto kernel scope link src 192.168.255.1 
192.168.255.2 via 10.0.0.20 dev eth0 proto static 

root@Device_B:~# ip r
default via 10.0.0.1 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.20 
192.168.255.0/24 dev vxlan0 proto kernel scope link src 192.168.255.2 
192.168.255.1 via 10.0.0.10 dev eth0 proto static 
192.168.255.2 dev eth0 scope link

I'm now able to see incoming packets via eth1, visible on the vxl-br and on vxlan0, I also see the packets leaving via eth0 and being delivered to vxlan0, vxl-br then eth1

I have to connect another laptop to the lan port on device B, but it seems to be working.

EDIT1: added remote setting to ip link add vxlan
EDIT2: it is working with the kernel utils!

22

I'd like to set this up using uci.
Following your suggestion I've tried deleting the vxlan ip alias and adding that address to lo0, but it seems to retain the same behavior.

root@Device_A:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          7fff.6a786d6ad7ea       no              eth1
root@Device_B:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          7fff.6a786d6ad7ec       no              eth1

root@Device_A:~# brctl addif br-lan vxlan0
brctl: bridge br-lan: Resource busy
root@Device_B:~# brctl addif br-lan vxlan0
brctl: bridge br-lan: Resource busy

Thank you,
Pedro

EDIT: typing error

23

Hey @pedrocaetano !

I have found the time and tried to replicate and reproduce...

With UCI I'm NOT able to get it working. I've set up only the vxlan interface, not an alias interface. (I do not understand for what this should be used anyway?!)

I used the following minimal config stanza:

config interface 'vxlan0'
	option proto 'vxlan'
	option peeraddr <the other routers wan address>
	option port '4789'
	option vid <id>
	option tunlink <wan inteface>

Edit: And of course list port vxlan0 was added to the br-lan stanza.

And this does not work, just because the fdb entry for the other peer is missing. If you add it manually afterwards it works.

My impression is that the function in /lib/netifd/proto/vxlan.sh is not called at all, but don't ask me why.

For now you can just add a script to /etc/hotplug.d/iface/ where you ensure that the fdb entry is added and present.

To fix this issue in the long run, I would say, open an issue and ask the package maintainer for conformation and assistance. I can jump in, too, and add details, if you like. Maybe I find a little bit time the next days to debug, too.
I can understand your wish to do it with UCI and not with a script.

(Just out of curiosity: Do you have a reason for not using i.e. frr with BGP EVPN?)

24

Hi @_bernd,

I added the ip alias option according to the documentation on how to set up vxlan interfaces.

Regarding the vxlan0 configuration snippet you've posted, I presume the source address is added to the loopback address?

I've chosen to use vxlan tunneling over gre, for no good reason other than learning how it works (-:
I've also considered on using openvswitch to set this up, but that'd result in a more complex environment.

This is a means to tunnel ArtNet broadcasts over a mpls network on a pair of cheapish boxes.

I'll try setting this up on hotplug.d, I was considering setting it the 'dirty way' on rc.local.

25

I assumed so. My comment was more in a general sense: Because as far as I know vxlan, you have no address on the vxlan interface it self. MAYBE it's useful to have an address tied to this interface exclusively to provide i.e. a service or something.

No. I did not bother to check the default sysctl settings, but the host did not want to anser. So I just used to addresses set on br-lan.
(But in general I -personally- do prefer to have addresses on loopback and use a routing protocol to advertise the host addresses.)

Fair enough :slight_smile: But: As far as I know and have followed: vxlan implementaion on linux still lacks i.e. proper multicast support. You can use multicast on the virtual layer-2 just fine, but i.e. the bridge does not support igmp for vxlan. So if you have like 10 hosts each with a vtep, but only like 3 vms/container/whatsoever attach to this vxlan are multicast lister, nevertheless all hosts get the multicast traffic anyway. But most of the time this is no big deal.

I'm still unsure how good the vxlan openvswitch implementation is, because you need to do many things manually anyway.
So if you have only two or a handful of routers attach to this vxlan, then the UCI approch could still be sufficient, and yes, setting up frr with bgp would be total overkill if you have not yet experience with if. But for "small" networks routing protocols and daemons are not that resource hungry.

The downside of rc.local is that it is only executed on boot. The hotplug event for i.e. iface is triggered everytime the interfaces changes state via uci, so still somehow dirty but not muddy :sunglasses:

Have a nice sunday evening and just ping me if you need some help.

26

So, after adding this to rc.local and rebooting the boxes everything works properly!

I'll try to migrate this 'hack' to hotplug.d tomorrow.

Thank you for your guidance @_bernd !

27

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.