Layer 2 vxlan tunnel setup

pedrocaetano · September 23, 2022, 4:58pm

Hi,

I've been tinkering with a pair of nano pi r2s boxes to set up an vxlan L2 tunnel via a L3 network.

The topology is roughly drawn below:

vxlan0 lan
\ /
\ /
\ / <==== device A
/
br-lan
|
wan
|
|
|
wan
|
br-lan
/\
/ \ <==== device B
/
/
vxlan0 lan

I've been able to establish the tunnel, it is possible to see rx and tx packets on both ends, but there is an issue with arp bridging, which renders the communication unusable.

Running a ping towards a device on the remote end of the vxlan interface triggers an arp request that is seen leaving device A encapsulated on the vxlan via the wan interface. (as expected)

On the remote device the arp request is decapsulated from the vxlan interface, it is then bridged to the lan interface.

The issue is the arp reply never leaves the bridge interface on the remote device.

After some searching it seems the issue is related with BUM traffic, but I'm a bit stuck on advancing on this topic.

Thanks for any help!

PS: it is my first post, so please be gentle on the replies

_bernd · September 23, 2022, 5:15pm

Please post what you have done to setup the vxlans... And have a look at i.e. https://vincent.bernat.ch/en/blog/2017-vxlan-linux which gives a good overview how to do it without evpn.

pedrocaetano · September 23, 2022, 5:38pm

I've added the packages below to the base installation:
kmod-vxlan luci-proto-vxlan vxlan tcpdump ip-bridge

Below is the configuration.
Device A:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd91:e571:934c::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'vxlan0'

config device
        option name 'eth1'
        option macaddr '00:00:00:6a:d7:ea'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.1'

config device
        option name 'eth0'
        option macaddr '00:00:00:6a:d7:e9'

config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option ipaddr '10.0.0.10'
        option netmask '255.255.255.0'
        option gateway '10.0.0.1'
        option broadcast '10.0.0.255'
        option delegate '0'

config interface 'vxlan0'
        option proto 'vxlan'
        option peeraddr '10.0.1.10'
        option vid '10101'
        option peerdns '0'
        option delegate '0'
        option ipaddr '192.168.2.1'

Device B

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd91:e571:934c::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'vxlan0'

config device
        option name 'eth1'
        option macaddr '00:00:00:6a:d7:ea'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.2'

config device
        option name 'eth0'
        option macaddr '00:00:00:6a:d7:e9'

config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option netmask '255.255.255.0'
        option delegate '0'
        option ipaddr '10.0.1.10'
        option gateway '10.0.1.1'
        option broadcast '10.0.1.255'

config interface 'vxlan0'
        option proto 'vxlan'
        option vid '10101'
        option peerdns '0'
        option delegate '0'
        option peeraddr '10.0.0.10'
        option ipaddr '192.168.2.2'

_bernd · September 23, 2022, 6:36pm

https://openwrt.org/docs/guide-user/network/tunneling_interface_protocols#protocol_vxlan_vxlan_layer_2_virtualization_over_layer_3_network and have a look at vxlan and at the end of the page for an example.

https://openwrt.org/docs/guide-user/base-system/basic-networking mentions also ip-full.

What I don't see how the interface is configured. When you have installed ip-full, please share the output of IP -details foe the vxlan interface.

(Sorry for the formating but I'm on my Mobil phone)

pedrocaetano · September 23, 2022, 9:34pm

Hi,

After reviewing the links you've posted I've bound the vxlan interfaces to wan interface.

Below is the output from ip -d address from both boxes.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:e9 brd ff:ff:ff:ff:ff:ff permaddr 2e:a0:0c:c6:eb:3c promiscuity 0 minmtu 46 maxmtu 3712 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7e9/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9194 
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 800 hello_time 100 max_age 1000 ageing_time 30000 stp_state 0 priority 32767 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  211.13 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
    inet 192.168.2.1/24 brd 192.168.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd91:e571:934c::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7ea/64 scope link 
       valid_lft forever preferred_lft forever
6: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UNKNOWN group default qlen 1000
    link/ether 4e:33:1b:f4:0d:27 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535 
    vxlan id 10101 remote 10.0.0.20 srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx 
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8002 port_no 0x2 designated_port 32770 designated_cost 0 designated_bridge 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 








1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:e9 brd ff:ff:ff:ff:ff:ff permaddr 2e:a0:0c:c6:eb:3c promiscuity 0 minmtu 46 maxmtu 3712 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 
    inet 10.0.0.20/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7e9/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9194 
    bridge_slave state disabled priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
4: br-lan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 800 hello_time 100 max_age 1000 ageing_time 30000 stp_state 0 priority 32767 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  228.11 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
    inet 192.168.2.2/24 brd 192.168.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd91:e571:934c::1/60 scope global tentative noprefixroute 
       valid_lft forever preferred_lft forever
6: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d2:b1:4a:15:fe:3b brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    vxlan id 10101 remote 10.0.0.10 dev br-lan srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 6553

Thank you,
Pedro

_bernd · September 23, 2022, 11:25pm

OK so you can ping the vxlan IP of both endpoints? Do you see with bridge fdb show both vxlan IPs with the 00:00:00:00:00:00 entry? I assume you do not want to have static entries for clients?
From a poor man's perspective that's all you need. Reaching the vtep and have an entry for each vtep with the all zero Mac in the fdb, then it should just work.

As I could see both vxlan interfaces (vtep) are bridged to br-lan. That's good.
You should do not need proxy ARP and probably do not want it.

You should be able to reach clients on the other side.
If you are able to do so try to let tcpdump run to verify or spot issues. Just in case, maybe lower the MTU to 1420.

Edit: yeah lower the MTU of the vxlan interface otherwise packets do not fit in your wan interface.

pedrocaetano · September 24, 2022, 8:48am

Hi, unfortunately the setup is still not working.

I have created an ip alias for the vxlan interface and set it to layer 2, as stated here

But i'm unable to ping the local nor remote ip aliases of the vxlan interface on both devices.

Output from /etc/config/network, ip -d a and bridge fdb show on both devices

Device A:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd91:e571:934c::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'vxlan0'

config device
        option name 'eth1'
        option macaddr '6a:78:6d:6a:d7:ea'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.1'

config device
        option name 'eth0'
        option macaddr '6a:78:6d:6a:d7:e9'

config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option ipaddr '10.0.0.10'
        option netmask '255.255.255.0'
        option gateway '10.0.0.1'
        option broadcast '10.0.0.255'
        option delegate '0'

config interface 'vxlan0'
        option proto 'vxlan'
        option vid '10101'
        option peerdns '0'
        option delegate '0'
        option peeraddr '10.0.0.20'
        option tunlink 'lan'
        option force_link '1'

config interface 'vx0'
        option proto 'static'
        option device '@vxlan0'
        option ipaddr '192.168.2.61'
        option netmask '255.255.255.0'
        option broadcast '192.168.2.255'
        option layer '2'














Device B:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd91:e571:934c::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'vxlan0'
        option ipv6 '0'

config device
        option name 'eth1'
        option macaddr '6a:78:6d:6a:d7:ea'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.2'

config device
        option name 'eth0'
        option macaddr '6a:78:6d:6a:d7:e9'

config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option netmask '255.255.255.0'
        option delegate '0'
        option ipaddr '10.0.0.20'
        option gateway '10.0.0.1'
        option broadcast '10.0.0.255'

config interface 'vxlan0'
        option proto 'vxlan'
        option vid '10101'
        option peerdns '0'
        option delegate '0'
        option peeraddr '10.0.0.10'
        option tunlink 'lan'
        option force_link '1'

config interface 'vx0'
        option proto 'static'
        option device '@vxlan0'
        option netmask '255.255.255.0'
        option broadcast '192.168.2.255'
        option ipaddr '192.168.2.62'
        option layer '2'

Device A:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:e9 brd ff:ff:ff:ff:ff:ff permaddr 2e:a0:0c:c6:eb:3c promiscuity 0 minmtu 46 maxmtu 3712 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7e9/64 scope link dadfailed tentative 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9194 
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 800 hello_time 100 max_age 1000 ageing_time 30000 stp_state 0 priority 32767 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer   39.94 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
    inet 192.168.2.1/24 brd 192.168.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd91:e571:934c::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7ea/64 scope link 
       valid_lft forever preferred_lft forever
6: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 62:18:c8:df:dd:73 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    vxlan id 10101 remote 10.0.0.20 dev br-lan srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
    inet 192.168.2.61/24 brd 192.168.2.255 scope global vxlan0
       valid_lft forever preferred_lft forever





Device B:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:e9 brd ff:ff:ff:ff:ff:ff permaddr 2e:a0:0c:c6:eb:3c promiscuity 0 minmtu 46 maxmtu 3712 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 
    inet 10.0.0.20/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7e9/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9194 
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    3.91 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 800 hello_time 100 max_age 1000 ageing_time 30000 stp_state 0 priority 32767 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 7fff.6A:78:6D:6A:D7:EA designated_root 7fff.6A:78:6D:6A:D7:EA root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  145.50 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
    inet 192.168.2.2/24 brd 192.168.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
6: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ba:d6:f6:06:a5:bd brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    vxlan id 10101 remote 10.0.0.10 dev br-lan srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 
    inet 192.168.2.62/24 brd 192.168.2.255 scope global vxlan0
       valid_lft forever preferred_lft forever

33:33:00:00:00:01 dev eth0 self permanent
33:33:00:00:00:02 dev eth0 self permanent
01:00:5e:00:00:01 dev eth0 self permanent
33:33:ff:6a:d7:e9 dev eth0 self permanent
00:e0:4c:68:16:ee dev eth1 master br-lan 
6a:78:6d:6a:d7:ea dev eth1 vlan 1 master br-lan permanent
6a:78:6d:6a:d7:ea dev eth1 master br-lan permanent
33:33:00:00:00:01 dev eth1 self permanent
33:33:00:00:00:02 dev eth1 self permanent
01:00:5e:00:00:01 dev eth1 self permanent
33:33:00:00:00:01 dev br-lan self permanent
33:33:00:00:00:02 dev br-lan self permanent
01:00:5e:00:00:01 dev br-lan self permanent
33:33:ff:00:00:01 dev br-lan self permanent
33:33:ff:6a:d7:ea dev br-lan self permanent
33:33:ff:00:00:00 dev br-lan self permanent
00:00:00:00:00:00 dev vxlan0 dst 10.0.0.20 via br-lan self permanent


Device B:
33:33:00:00:00:01 dev eth0 self permanent
33:33:00:00:00:02 dev eth0 self permanent
01:00:5e:00:00:01 dev eth0 self permanent
33:33:ff:6a:d7:e9 dev eth0 self permanent
33:33:ff:00:00:00 dev eth0 self permanent
00:e0:4c:68:16:ee dev eth1 master br-lan 
6a:78:6d:6a:d7:ea dev eth1 vlan 1 master br-lan permanent
6a:78:6d:6a:d7:ea dev eth1 master br-lan permanent
33:33:00:00:00:01 dev eth1 self permanent
33:33:00:00:00:02 dev eth1 self permanent
01:00:5e:00:00:01 dev eth1 self permanent
33:33:00:00:00:01 dev br-lan self permanent
33:33:00:00:00:02 dev br-lan self permanent
01:00:5e:00:00:01 dev br-lan self permanent
33:33:00:01:00:02 dev br-lan self permanent
33:33:00:01:00:03 dev br-lan self permanent
00:00:00:00:00:00 dev vxlan0 dst 10.0.0.10 via br-lan self permanent

MTU is set at 1450 which should be enough given vlans are not being used.

Thanks,
Pedro

_bernd · September 24, 2022, 8:56am

What does ip route get <vtep ip> say?

pedrocaetano · September 24, 2022, 9:19am

Apologies for the incomplete report, I failed to specify where I was pinging from

Device A:
root@Device_A:~# ip r get 192.168.2.61
local 192.168.2.61 dev lo table local src 192.168.2.61 uid 0 
    cache <local> 
root@Device_A:~# ip r get 192.168.2.62
192.168.2.62 dev br-lan src 192.168.2.1 uid 0 
    cache

Device B:
root@Device_B:~# ip r get 192.168.2.61
192.168.2.61 dev br-lan src 192.168.2.2 uid 0 
    cache 
root@Device_B:~# ip r get 192.168.2.62
local 192.168.2.62 dev lo table local src 192.168.2.62 uid 0 
    cache <local>

From device A, I'm able to ping the local alias, but not the remote.

Device A:
root@Device_A:~# ping -c1 192.168.2.61; ping -c1 192.168.2.62
PING 192.168.2.61 (192.168.2.61): 56 data bytes
64 bytes from 192.168.2.61: seq=0 ttl=64 time=0.374 ms

--- 192.168.2.61 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.374/0.374/0.374 ms
PING 192.168.2.62 (192.168.2.62): 56 data bytes

--- 192.168.2.62 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: seq=0 ttl=64 time=0.381 ms

From device B, I'm able to ping the local alias, but not the remote.

root@Device_B:~# ping -c1 192.168.2.62 && ping -c1 192.168.2.61
PING 192.168.2.62 (192.168.2.62): 56 data bytes
64 bytes from 192.168.2.62: seq=0 ttl=64 time=0.377 ms

--- 192.168.2.62 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.377/0.377/0.377 ms
PING 192.168.2.61 (192.168.2.61): 56 data bytes

--- 192.168.2.61 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

From devices sitting on the lan network it is possible to ping the br-lan adress, but not the vxlan alias address:

Laptop connected to device A:
[pcaetano@fedora pcaetano]$ ping -c1 192.168.2.1; ping -c1 192.168.2.61; ping -c1 192.168.2.62
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.834 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.834/0.834/0.834/0.000 ms
PING 192.168.2.61 (192.168.2.61) 56(84) bytes of data.
From 192.168.2.101 icmp_seq=1 Destination Host Unreachable

--- 192.168.2.61 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 192.168.2.62 (192.168.2.62) 56(84) bytes of data.
From 192.168.2.101 icmp_seq=1 Destination Host Unreachable

--- 192.168.2.62 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Laptop connected to device B:
[pcaetano@fedora pcaetano]$ ping -c1 192.168.2.2; ping -c1 192.168.2.62; ping -c1 192.168.2.61
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.824 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.824/0.824/0.824/0.000 ms
PING 192.168.2.62 (192.168.2.62) 56(84) bytes of data.
From 192.168.2.101 icmp_seq=1 Destination Host Unreachable

--- 192.168.2.62 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 192.168.2.61 (192.168.2.61) 56(84) bytes of data.
From 192.168.2.101 icmp_seq=1 Destination Host Unreachable

--- 192.168.2.61 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

EDIT: clarify ping sources

_bernd · September 24, 2022, 9:35am

I just realized how you have choosen your IP addresses...

Choose a different IP space for the vtep/vxlan interfaces aka not from your LAN ip space... Like 172.16.0.0 and ensure/set a route via your wan interface and gateway.

(I prefer host-routes /32 for this use-case... )
10.0.0.1 is your gateway IP on eth0?
device A: ip route add 172.16.0.2/32 via 10.0.0.1
device B: `ip route add 172.16.0.1/32 via 10.0.0.1

Right now you want to reach the IP on your local lan, BUT the vtep is only reachable via your wan interface...

pedrocaetano · September 24, 2022, 9:54am

I do not have another device capable of routing nearby, so the wan interfaces are connected back to back.

If I understand your post clearly the configuration should be something like this:

vtep subnet: 192.168.255.0/24
Device A: 192.168.255.1
Device B: 192.168.255.2

root@Device_A:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:eb brd ff:ff:ff:ff:ff:ff permaddr 2e:a0:0c:c6:eb:3c
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7eb/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ea brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd91:e571:934c::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7ea/64 scope link 
       valid_lft forever preferred_lft forever
6: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 2e:a7:d8:1c:b0:69 brd ff:ff:ff:ff:ff:ff
    inet 192.168.255.1/24 brd 192.168.255.255 scope global vxlan0
       valid_lft forever preferred_lft forever

root@Device_B:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ed brd ff:ff:ff:ff:ff:ff permaddr 2e:a0:0c:c6:eb:3c
    inet 10.0.0.20/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6878:6dff:fe6a:d7ed/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ec brd ff:ff:ff:ff:ff:ff
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:78:6d:6a:d7:ec brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.2/24 brd 192.168.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
6: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 2e:90:e6:21:fc:26 brd ff:ff:ff:ff:ff:ff
    inet 192.168.255.2/24 brd 192.168.255.255 scope global vxlan0
       valid_lft forever preferred_lft forever

root@Device_A:~# ip r
default via 10.0.0.1 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.10 
192.168.2.0/24 dev br-lan proto kernel scope link src 192.168.2.1 
192.168.255.0/24 dev vxlan0 proto kernel scope link src 192.168.255.1 
192.168.255.2 via 10.0.0.20 dev eth0 proto static

root@Device_B:~# ip r
default via 10.0.0.1 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.20 
192.168.2.0/24 dev br-lan proto kernel scope link src 192.168.2.2 
192.168.255.0/24 dev vxlan0 proto kernel scope link src 192.168.255.2 
192.168.255.1 via 10.0.0.10 dev eth0 proto static

Thanks,
Pedro

_bernd · September 24, 2022, 10:28am

Yeah looks good. How does the fdb look and are you now able to reach the vtep itself and other hosts on the lan?

pedrocaetano · September 24, 2022, 10:57am

It is now possible to ping the vtep addresses from both ends:

root@Device_A:~# ping -c1 192.168.255.1; ping -c1 192.168.255.2
PING 192.168.255.1 (192.168.255.1): 56 data bytes
64 bytes from 192.168.255.1: seq=0 ttl=64 time=0.398 ms

--- 192.168.255.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.398/0.398/0.398 ms
PING 192.168.255.2 (192.168.255.2): 56 data bytes
64 bytes from 192.168.255.2: seq=0 ttl=64 time=1.427 ms

--- 192.168.255.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.427/1.427/1.427 ms

root@Device_B:~# ping -c1 192.168.255.2; ping -c1 192.168.255.1
PING 192.168.255.2 (192.168.255.2): 56 data bytes
64 bytes from 192.168.255.2: seq=0 ttl=64 time=0.389 ms

--- 192.168.255.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.389/0.389/0.389 ms
PING 192.168.255.1 (192.168.255.1): 56 data bytes
64 bytes from 192.168.255.1: seq=0 ttl=64 time=1.528 ms

--- 192.168.255.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.528/1.528/1.528 ms

Whilst possible to ping vtep addresses from each end, the traffic is not shown on tcpdump?

root@Device_A:~# tcpdump -ni vxlan0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

root@Device_B:~# tcpdump -ni vxlan0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

If the wan cable is unplugged, it is not possible to ping the remote vtep address, so it is surely being used.

Regarding the other devices, it is not possible to reach other devices via the bridge segment:

Lan client1:
[pcaetano@fedora pcaetano]$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp0s13f0u3      UP             192.168.2.101/24 fe80::c363:209c:ae32:b3d1/64 

[pcaetano@fedora pcaetano]$ ping -c 5 192.168.2.102
PING 192.168.2.102 (192.168.2.102) 56(84) bytes of data.
From 192.168.2.101 icmp_seq=1 Destination Host Unreachable
From 192.168.2.101 icmp_seq=2 Destination Host Unreachable
From 192.168.2.101 icmp_seq=3 Destination Host Unreachable
From 192.168.2.101 icmp_seq=4 Destination Host Unreachable
From 192.168.2.101 icmp_seq=5 Destination Host Unreachable

Output from tcpdump on both devices:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
11:47:58.940273 00:e0:4c:68:16:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.102 tell 192.168.2.101, length 46
11:47:59.958099 00:e0:4c:68:16:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.102 tell 192.168.2.101, length 46
11:48:00.983061 00:e0:4c:68:16:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.102 tell 192.168.2.101, length 46
11:48:02.006284 00:e0:4c:68:16:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.102 tell 192.168.2.101, length 46
11:48:03.029978 00:e0:4c:68:16:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.102 tell 192.168.2.101, length 46
11:48:04.053962 00:e0:4c:68:16:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.102 tell 192.168.2.101, length 46
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

root@Device_B:~# tcpdump -e -ni br-lan not port 80 and not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Output from bridge fdb show below:

root@Device_A:~# bridge fdb show
33:33:00:00:00:01 dev eth0 self permanent
33:33:00:00:00:02 dev eth0 self permanent
01:00:5e:00:00:01 dev eth0 self permanent
33:33:ff:6a:d7:eb dev eth0 self permanent
33:33:ff:00:00:00 dev eth0 self permanent
00:e0:4c:68:16:ee dev eth1 master br-lan 
6a:78:6d:6a:d7:ea dev eth1 vlan 1 master br-lan permanent
6a:78:6d:6a:d7:ea dev eth1 master br-lan permanent
33:33:00:00:00:01 dev eth1 self permanent
33:33:00:00:00:02 dev eth1 self permanent
01:00:5e:00:00:01 dev eth1 self permanent
33:33:00:00:00:01 dev br-lan self permanent
33:33:00:00:00:02 dev br-lan self permanent
01:00:5e:00:00:01 dev br-lan self permanent
33:33:ff:00:00:01 dev br-lan self permanent
33:33:ff:6a:d7:ea dev br-lan self permanent
33:33:ff:00:00:00 dev br-lan self permanent
00:00:00:00:00:00 dev vxlan0 dst 10.0.0.20 via br-lan self permanent

root@Device_B:~# bridge fdb show
33:33:00:00:00:01 dev eth0 self permanent
33:33:00:00:00:02 dev eth0 self permanent
01:00:5e:00:00:01 dev eth0 self permanent
33:33:ff:6a:d7:ed dev eth0 self permanent
33:33:ff:00:00:00 dev eth0 self permanent
e4:11:5b:29:8f:7e dev eth1 master br-lan 
6a:78:6d:6a:d7:ec dev eth1 vlan 1 master br-lan permanent
6a:78:6d:6a:d7:ec dev eth1 master br-lan permanent
33:33:00:00:00:01 dev eth1 self permanent
33:33:00:00:00:02 dev eth1 self permanent
01:00:5e:00:00:01 dev eth1 self permanent
33:33:00:00:00:01 dev br-lan self permanent
33:33:00:00:00:02 dev br-lan self permanent
01:00:5e:00:00:01 dev br-lan self permanent
33:33:00:01:00:02 dev br-lan self permanent
33:33:00:01:00:03 dev br-lan self permanent
00:00:00:00:00:00 dev vxlan0 dst 10.0.0.10 via br-lan self permanent

EDIT: wrong tcpdump output posted

_bernd · September 24, 2022, 11:06am

As it seams as a lab setup anyway could you just disable the firewall for now?

_bernd · September 24, 2022, 11:23am

Try to tcpdump on wan and see if you see vxlan packets.
Alternativ to disable the firewall allow the vxlan UDP port...

pedrocaetano · September 24, 2022, 11:40am

The firewall is already disabled.
All chains are set to ACCEPT and traffic is allowed to flow between all zones.
Am I missing something else?

_bernd · September 24, 2022, 11:52am

Then you should see at least

on eth1/br-lan the original packet
on eth0 (wan) the udp packet with the vxlan encapsulation...
and on deviceB the encap. vxlan packet, and then it should go to br-lan....

Did you just tried it to setup without UCI and just plain ip link add and bridge fdb append?
I'm asking because doing it "manually" is rather easy, and I personally prefer to do it at first the manual way and when I've ensured I got all the "moving parts", I then go from there on...
(If you do it manually you do not need to set peer on the vxlan interface. Only the local endpoint IP. And with bridge fdb append you set the other VTEPs. Just as a reminder, the vxlan interface needs to be enslaved on br-lan... Hope this helps you somehow.

pedrocaetano · September 24, 2022, 1:50pm

Hi again,

I had to step away from this for a bit.

I'll review the configuration thoroughly and report back.

@_bernd thank you for your guidance!

pedrocaetano · September 24, 2022, 2:18pm

I'm a bit puzzled by the following:

root@Device_A:~# cat /etc/config/network 

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd91:e571:934c::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'vxlan0'

config device
        option name 'eth1'
        option macaddr '6a:78:6d:6a:d7:ea'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.1'

config device
        option name 'eth0'
        option macaddr '6a:78:6d:6a:d7:eb'

config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option ipaddr '10.0.0.10'
        option netmask '255.255.255.0'
        option gateway '10.0.0.1'
        option broadcast '10.0.0.255'
        option delegate '0'

config interface 'vxlan0'
        option proto 'vxlan'
        option vid '10101'
        option peerdns '0'
        option delegate '0'
        option peeraddr '10.0.0.20'
        option tunlink 'lan'
        option force_link '1'

config interface 'vx0'
        option proto 'static'
        option device '@vxlan0'
        option netmask '255.255.255.0'
        option layer '2'
        option ipaddr '192.168.255.1'
        option broadcast '192.168.255.255'

config route
        option interface 'wan'
        option gateway '10.0.0.20'
        option target '192.168.255.2/32'

root@Device_A:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          7fff.6a786d6ad7ea       no              eth1
root@Device_A:~# brctl addif br-lan vxlan0
brctl: bridge br-lan: Resource busy

root@Device_B:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd91:e571:934c::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'vxlan0'
        option ipv6 '0'

config device
        option name 'eth1'
        option macaddr '6a:78:6d:6a:d7:ec'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.2'

config device
        option name 'eth0'
        option macaddr '6a:78:6d:6a:d7:ed'

config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option netmask '255.255.255.0'
        option delegate '0'
        option ipaddr '10.0.0.20'
        option gateway '10.0.0.1'
        option broadcast '10.0.0.255'

config interface 'vxlan0'
        option proto 'vxlan'
        option vid '10101'
        option peerdns '0'
        option delegate '0'
        option peeraddr '10.0.0.10'
        option tunlink 'lan'
        option force_link '1'

config interface 'vx0'
        option proto 'static'
        option device '@vxlan0'
        option netmask '255.255.255.0'
        option layer '2'
        option ipaddr '192.168.255.2'
        option broadcast '192.168.255.255'

config route
        option interface 'wan'
        option target '192.168.255.1/32'
        option gateway '10.0.0.10'

root@Device_B:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          7fff.6a786d6ad7ec       no              eth1
root@Device_B:~# brctl addif br-lan vxlan0
brctl: bridge br-lan: Resource busy

It seems the vxlan interface 'refuses' to be a member o br-lan bridge.

_bernd · September 24, 2022, 3:06pm

Ah because the vxlan interface has set an IP addr. But the vxlan gets only the local flag with an address set. If an interface has an address it can not be enslaved by a bridge. Like I said before: try to do it manually first. If that works try to put it into the UCI config.

Edit.: If you set only the local flag then you need to ensure the address is at least present on the loopback!