Last update, i've been spend quite hours if not days, and the set-up is done and working.
Here is the final scheme of the network (if it interest any):
Spend money: 600€
- Ubiquiti EdgeRouterX: 100€
- Unifi Long Range 6: 180€ each (360€ both)
- Adapter PoE+: 22€ each (44€ both)
- GS308Tv1: 47€ each (~100€ both)
Performance (pc wired on the switch, itself wired on the mesh node) - verified with proof.ovh.net:
- 300mbt/s download average
- 250mbt/s upload average
- Latence 14ms.
Feeling: very cool to have a complete open source set-up (openwrt on each node) but it can be quite expensive and have lower performance compare to already existing solution (thinking of Orbi rbk23 where I had 400mbt/s download/upload: the limit of the ISP. I'm pretty sure that locally it is even better than 400mbt/s). At least I have the full control of my network (e.g., vlan, add multiple ssid) that I haven't with Orbi rbk23. Note that it is possible to have this kind of features with others models (more expensive also).
After all of this I'm still having some questions to improve the set-up:
- Do the configurations include some nonsense instructions ? Is it well specified according to the new 21.02 version with DSA architecture ? (see Appendix.B)
- How would you optimize the network ? to approach the 400mbit/s download as done with Orbi rbk23. After some debug that I ran (see Appendix.A), it feels that the mesh network is might not be the limitation (maybe the switch ? the router ?). However, I'm loosing half of the bandwidth from batman to the IP part of the same device (736Mbps with batctl and 423 Mbps with iperf), is this normal ? Would you change some part of the mesh nodes configuration to improve the performance ? (e.g., channel, or tx power, or htmode ?) Looking for an expert guiding me to gain some time, because I spend too much of trying without understanding the impacts.
- The mesh nodes get an ip through DHCP (static lease). What are the risk/consequences? Is it a bad practice and I should be better to set-up static ip ? e.g., the lr6-node1 is up, but not the lr6-gw, I guess the node1 will not have any IP, and I will have no access to debug it easily (i.e., ssh).
- I thought to set-up a management SSID on each mesh node, in case the gateway is dead and I want an access to debug it. Is it a good practice ?
- I'm trying to disable ipv6, what is the best way ? (why ? because I do not understand well enough the protocol yet in terms of security and impacts. I want to take more time to understand how to well set-up the dhcpv6. I do not really like the fact that each device has a kind of static ipv6 visible on internet: I feel it ease the work for tracking website and so on). It feels repetitive to add
option ipv6 '0'
on each interface for each device (switch, mesh node, router). If I don't, I find an ipv6 everywhere (example on Appendix.C). Is this normal ?
- Adding a vlan feel to repetitive (adding on each device). Is there any known solution to say that in-between device accept any vlan ? Is
trunk
the right terms ? I'm thinking about the mesh nodes. E.g., Every new x
vlan need a new bat0.x
port.
- Even if the set-up work (followed the OpenWRT documentation), I'm a little bit confuse why the
bat0.x
are not tagged like eth0:t
. Why ?
- What's the good practice of configuration to allow any device in lan (vlan 100) to access/ping any home device (vlan 200). Only need a firewall rule on router right ? (see appendix.D)
- I guess since the router is the only routing part (layer 3), having firewall is worthless on the other devices (mesh nodes, switch), right ? it can be disable ?
- (bonus) LEDs are dead on GS308Tv1 (some post talk about fixing on v3). Where should I look on the code source to help make it works ? or understand why it does not work on v1.
I'm ready to pay 10$ worth LTC for a good answer and another 10$ worth LTC on the optimization part.. I'm not sure posting this here is the good way ? Should I make a new post on installing and configuring
topic ?
More ressources (personal notes, can be messy):
-- APPENDIX --
A. some commands to check performance between two node mesh (gw and node1)
root@lr6-node1:~# batctl tp lr6-gw
Test duration 10530ms.
Sent 969239016 Bytes.
Throughput: 87.78 MB/s (736.36 Mbps)
root@lr6-node1:~# iperf -c 192.168.2.156 # ip of mesh-lr6-gw
------------------------------------------------------------
Client connecting to 192.168.2.156, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.2.208 port 55738 connected with 192.168.2.156 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.10 sec 509 MBytes 423 Mbits/sec
B. Configurations
edgerouterx - /etc/config/network:
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option packet_steering '1'
option ula_prefix 'fd3b:fde2:a5d2::/48'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth1'
list ports 'eth2'
list ports 'eth3'
list ports 'eth4'
option ipv6 '0'
config interface 'wan'
option device 'eth0'
option proto 'dhcp'
config interface 'wan6'
option device 'eth0'
option proto 'dhcpv6'
option auto '0'
option reqaddress 'try'
option reqprefix 'auto'
# personal
config bridge-vlan
option device 'br-lan'
option vlan '100'
list ports 'eth1:u*'
list ports 'eth2:t'
list ports 'eth3:t'
list ports 'eth4:u*'
config interface 'lan'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option ipaddr '192.168.2.1'
list dns '192.168.2.227'
option device 'br-lan.100'
option delegate '0'
option ipv6 '0'
# services
config bridge-vlan
option device 'br-lan'
option vlan '150'
list ports 'eth1:u'
list ports 'eth2:t'
list ports 'eth3:t'
list ports 'eth4'
config interface 'services'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option ipaddr '192.168.4.1'
list dns '192.168.2.227'
option device 'br-lan.150'
option delegate '0'
option ipv6 '0'
# home
config bridge-vlan
option device 'br-lan'
option vlan '200'
list ports 'eth2:t*'
list ports 'eth3:t*'
list ports 'eth4'
config interface 'home'
option proto 'static'
option device 'br-lan.200'
option ipaddr '192.168.10.1'
option netmask '255.255.255.0'
# list dns '192.168.2.227' # disable for now, AdGuard mess the performance of different services (Amazon Video, Netflix)
option delegate '0'
# iot
config bridge-vlan
option device 'br-lan'
option vlan '300'
list ports 'eth2:t'
list ports 'eth3:t'
list ports 'eth4'
config interface 'iot'
option proto 'static'
option device 'br-lan.300'
option ipaddr '192.168.100.1'
option netmask '255.255.255.0'
list dns '192.168.2.227'
option delegate '0'
# office
config bridge-vlan
option device 'br-lan'
option vlan '400'
list ports 'eth2:t'
list ports 'eth3:t'
list ports 'eth4'
config interface 'office'
option proto 'static'
option device 'br-lan.400'
option ipaddr '192.168.110.1'
option netmask '255.255.255.0'
list dns '192.168.2.227'
option delegate '0'
mesh-lr6-gw - /etc/config/network:
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd35:4bb4:8223::/48'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0'
list ports 'bat0.100'
list ports 'bat0.200'
list ports 'bat0.300'
list ports 'bat0.400'
option ipv6 '0'
config interface 'bat0'
option proto 'batadv'
option routing_algo 'BATMAN_IV'
option aggregated_ogms 1
option ap_isolation 0
option bonding 0
option fragmentation 1
#option gw_bandwidth '10000/2000'
option gw_mode 'off'
#option gw_sel_class 20
option log_level 0
option orig_interval 1000
option bridge_loop_avoidance 1
option distributed_arp_table 1
option multicast_mode 1
option network_coding 0
option hop_penalty 30
option isolation_mark '0x00000000/0x00000000'
option ipv6 '0'
config interface 'nwi_mesh0'
option mtu '2304'
option proto 'batadv_hardif'
option master 'bat0'
# lan
config interface 'lan'
option device 'br-lan.100'
option proto 'dhcp'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '100'
list ports 'eth0:t'
list ports 'bat0.100'
# services
config interface 'services'
option device 'br-lan.150'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '150'
list ports 'eth0:t'
list ports 'bat0.150'
# home
config interface 'home'
option device 'br-lan.200'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '200'
list ports 'eth0:t'
list ports 'bat0.200'
# iot
config interface 'iot'
option device 'br-lan.300'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '300'
list ports 'eth0:t'
list ports 'bat0.300'
# office
config interface 'office'
option device 'br-lan.400'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '400'
list ports 'eth0:t'
list ports 'bat0.400'
mesh-lr6-gw - /etc/config/wireless:
config wifi-device 'radio1'
option type 'mac80211'
option path '1a143000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0'
option hwmode '11a'
option channel '36'
option band '5g'
option htmode 'HE80'
option disabled '0'
option country 'FR'
config wifi-iface 'mesh0'
option device 'radio1'
option ifname 'mesh0'
option network 'nwi_mesh0'
option mode 'mesh'
option mesh_fwding '0'
option mesh_id 'meshlr6'
option encryption 'sae'
option key 'xxx'
config wifi-device 'radio0'
option type 'mac80211'
option path 'platform/18000000.wmac'
option channel '1'
option band '2g'
option htmode 'HT20'
option disabled '0'
config wifi-iface 'lan_radio0'
option device 'radio0'
option network 'lan'
option mode 'ap'
option ssid 'Livebox-2090l'
option encryption 'psk-mixed'
option key 'xxx'
...
mesh-lr6-node1 - /etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd8f:77cb:b992::/48'
# Bridge
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0'
list ports 'bat0.100'
list ports 'bat0.200'
list ports 'bat0.300'
list ports 'bat0.400'
option ipv6 '0'
# Mesh network
config interface 'bat0'
option proto 'batadv'
option routing_algo 'BATMAN_IV'
option aggregated_ogms 1
option ap_isolation 0
option bonding 0
option fragmentation 1
#option gw_bandwidth '10000/2000'
option gw_mode 'off'
#option gw_sel_class 20
option log_level 0
option orig_interval 1000
option bridge_loop_avoidance 1
option distributed_arp_table 1
option multicast_mode 1
option network_coding 0
option hop_penalty 30
option isolation_mark '0x00000000/0x00000000'
option ipv6 '0'
config interface 'nwi_mesh0'
option mtu '2304'
option proto 'batadv_hardif'
option master 'bat0'
# lan
config interface 'lan'
option device 'br-lan.100'
option proto 'dhcp'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '100'
list ports 'eth0:t'
list ports 'bat0.100'
# services
config interface 'services'
option device 'br-lan.150'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '150'
list ports 'eth0:t'
list ports 'bat0.150'
# home
config interface 'home'
option device 'br-lan.200'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '200'
list ports 'eth0:t'
list ports 'bat0.200'
# iot
config interface 'iot'
option device 'br-lan.300'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '300'
list ports 'eth0:t'
list ports 'bat0.300'
# office
config interface 'office'
option device 'br-lan.400'
option proto 'none'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '400'
list ports 'eth0:t'
list ports 'bat0.400'
switch-gs308t-desk - /etc/config/network:
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fddc:4119:d273::/48'
# last port allow to manage switch
config device
option name 'management'
option type 'bridge'
list ports 'lan8'
config interface 'management'
option device 'management'
option proto 'static'
option netmask '255.255.255.0'
option ipaddr '192.168.20.1'
config device
option name 'switch'
option type 'bridge'
list ports 'lan1 lan2 lan3 lan4 lan5 lan6 lan7'
# lan
config interface 'lan'
option device 'switch.100'
option proto 'dhcp'
config bridge-vlan
option device 'switch'
option vlan '100'
list ports 'lan1:t'
list ports 'lan2:u'
list ports 'lan3:u'
list ports 'lan4:t'
# home
config bridge-vlan
option device 'switch'
option vlan '200'
list ports 'lan1:t'
list ports 'lan5:u'
list ports 'lan6:u'
# office
config bridge-vlan
option device 'switch'
option vlan '400'
list ports 'lan1:t'
list ports 'lan7:u*'
C. ipv6 presence inet6 fe80::2e0:4cff:fe00:0/64 scope link
root@gs308t:~# ip a
13: switch.100@switch: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:e0:4c:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.102/24 brd 192.168.2.255 scope global switch.100
valid_lft forever preferred_lft forever
inet6 fe80::2e0:4cff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
D. lan
(vlan 100) networking accessing home
(vlan 200)
edgerouterx - /etc/config/firewall
config forwarding
option src 'lan'
option dest 'home'