Figured I would do a little write up on my adventures with a Ubiquiti EdgeRouter X and OpenWRT (18.06.1, r7258-5eb055306f).
Last Updated 1/10/2019
Back story.
This whole process started when I was trying to upgrade from OpenWRT(VM) to something bare metal. The rest of my house is all Ubiquiti Unifi gear, but Ubiquiti's current Unifi Routers (USG) dont fit my needs/cost use case. After much googling and frustration I stumbled across the Ubiquiti EdgeRouter X (ERX). I was VERY pleased to find a nice peace of hardware with solid specs (2 core, 4 threads, 256/256 Storage/RAM), NO wireless, PoE (24V), 5X 1Gbps ports,and an all metal case. Was even more happy that I could just power the ERX right off my core switch.
How to get from the stock EdgeOS to OpenWRT.
The process is simple. EdgeOS --> custom version of OpenWRT --> stock OpenWRT 18.06.1. Firstly we need to build the "openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar" which will be used as a stepping stone to get to a stock version of OpenWRT. Here are my steps to build openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar. But if you are lazy, here is the link to my compiled version: http://batbuilds.com/openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar
git clone https://www.github.com/openwrt/openwrt -b openwrt-18.06
cd openwrt/
./scripts/feeds update -a
./scripts/feeds install -a
make menuconfig
Target System: "MediaTek Ralink MIPS"
Subtarget: "MT7621 based boards"
Target Profile: "Ubiquiti EdgeRouter X"
Target Images: [*] ramdisk ---> Compression (lzma)
save, exit
make -j16 #(Adjust the -jXX for the number of threads your compiler has access to)
cd bin/targets/ramips/mt7621
cat sha256sums | grep openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar #Please note, if you compile, your sum will differ
2239d14073c99cc871544eb897dfd63405a0d464226f8e5b3b9686a2557533d5 *openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar
scp openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar user@desktop:~Get OpenWRT installed on the EdgeRouter X.
By default EdgeOS comes up at 192.168.1.1 on the physical port labeled"eth0". Default creds are ubnt/ubnt.
scp openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar ubnt@192.168.1.1:/tmp
ssh ubnt@192.168.1.1
cd /tmp
add system image /tmp/openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar
Checking upgrade image...Done
Preparing to upgrade...Done
Copying upgrade image.../usr/bin/ubnt-upgrade: line 569: [: too many arguments
Done
Removing old image...Done
Checking upgrade image...Done
Copying config data...Done
Finishing upgrade...Done
Upgrade completed
show system image
The system currently has the following image(s) installed:
ramips r7627-753531d 18.06-SNAPSHOT (default boot)
v1.9.7+hotfix.3.5013617.170830.0227 (running image)
#A reboot is needed to boot default image, which is now are custom OpenWRT
root@ubnt:~# reboot
Proceed with reboot? [confirm]y
#Reboot time was rather fast. Less then 1m.Moving from my custom image to a stock OpenWRT image
First thing on my list is getting a "stock" install of OpenWRT in place. This assumes you have a "WAN" connection, with functional internet, on port eth0. You could also download the firmware to a local computer and scp it over.
Pro Tip: By default OpenWRT's LAN connection will come up on the physical port "eth1".
ssh root@192.168.1.1
cd /tmp
wget http://downloads.openwrt.org/releases/18.06.1/targets/ramips/mt7621/openwrt-18.06.1-ramips-mt7621-ubnt-erx-squashfs-sysupgrade.tar
sysupgrade /tmp/openwrt-18.06.1-ramips-mt7621-ubnt-erx-squashfs-sysupgrade.tar
Cannot save config while running from ramdisk.
Commencing upgrade. Closing all shell sessions.
Connection to 192.168.1.1 closed by remote host.
Connection to 192.168.1.1 closed.
#Reboot time was rather fast. Less then 1m.Moving the "WAN" connection to "eth4"
At this point we are running a stock version of OpenWRT. One tweak I like to make is moving the "LAN" connection back to "eth0" and bumping the "WAN" connection to "eth4". That is simply done by tweaking around the switch ports as seen below.
Pro Tip: Ports 0~3, with my tweak, all act as "LAN" ports. Use any of them, like you would a normal switch. Only the physical port labeled "eth0" can support 24V PoE in. (PoE out is covered in the Wiki, see references below)
root@OpenWrt:~# cat /etc/config/network
...
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '0 1 2 3 6t'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '4 6t'Performance test: LAN
At this point I have everything up and running and I am very happy to have internet up. New hardware is in play, but how fast is it? speedtest(dot)net puts my connection around 550/400 down/up, and the CPU on my ERX drops to less than 5% idle. I know my Centurylink(dot)net 1Gbps fiber connection can do 930/930. Lets look at what iperf says, for local traffic:
OpenWRT: OpenWRT CPU "45% idle", root@OpenWrt:~# iperf -c 10.1.1.30 -t 60 -P 1, [ 8] 0.0-60.0 sec 5.77 GBytes 825 Mbits/sec
Main Server: OpenWRT CPU "22% idle", root@bigiron:~# iperf -c 10.1.1.1 -t 60 -P 1, [ 3] 0.0-60.0 sec 5.94 GBytes 851 Mbits/sec
OpenWRT: OpenWRT CPU "33% idle", root@OpenWrt:~# iperf -c 10.1.1.30 -t 60 -P 4, [SUM] 0.0-60.1 sec 6.37 GBytes 911 Mbits/sec
Main Server: OpenWRT CPU "10% idle", root@bigiron:~# iperf -c 10.1.1.1 -t 60 -P 4, [SUM] 0.0-60.0 sec 6.55 GBytes 938 Mbits/sec NEED MORE SPEED!! Enabling hardware flow control:
BIG FAT WARNING! Hardware flow offloading and SQM are incompatible with each other.
Through the web interface, Network --> Firewall, "Software flow offloading, which will then allow you to enable "Hardware flow offloading". All other setting are unchanged/defaults.
SQM on the other hand is an installed package 'opkg install luci-app-sqm' Network --> SQM QoS.
1. SQM works with software flow offload, allowing throughput of up to 200mbps with +3ms bufferbloat. Any thing faster maxes the CPU.
2. HW flow offload resulting in 900mbps but the +200ms bufferbloat hurts
3. NO HW flow offload, NO SQM, resulting in 550+ mbps with +80ms bufferbloatFinal thoughts.
At a $50 USD price point, this thing is AMAZING! Getting to OpenWRT is almost painless, as it requires one extra step. Serial console on the PCB is a VERY nice fall back, for when I brick the ERX. Hardware acceleration is a major boost, if you can suffer a little buffer bloat.
CenturyLink fiber config
For any one who follows in my tracks, here is my full networking config, including VLAN 201 stuff required by CenturyLink fiber. Please note on the VLAN 201, both ports need to be tagged "option ports '4t 6t'"
cat /etc/config/network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
option ipv6 '0'
config interface 'lan'
option type 'bridge'
option ifname 'eth0.1'
option proto 'static'
option ipaddr '10.1.1.1'
option netmask '255.255.255.0'
option ipv6 '0'
config device 'lan_dev'
option name 'eth0.1'
option macaddr 'fc:ec:da:7e:b7:7e'
config interface 'wan'
option ifname 'eth0.201'
option proto 'pppoe'
option password 'password'
option ipv6 '0'
option username 'fuzzy@nugget.net'
config device 'wan_dev'
option name 'eth0.201'
option macaddr 'fc:ec:da:7e:b7:7f'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '0 1 2 3 6t'
config switch_vlan
option device 'switch0'
option vlan '201'
option ports '4t 6t'Many thanks for all the people who have come before me, and posted below, to lay the ground work for this post.
References:
https://oldwiki.archive.openwrt.org/doc/howto/sqm
https://oldwiki.archive.openwrt.org/toh/ubiquiti/ubiquiti_edgerouter_x_er-x_ka
https://openwrt.org/toh/ubiquiti/ubiquiti_edgerouter_x_er-x_ka?s[]=edgerouter
http://sector5d.org/openwrt-on-the-ubiquiti-edgerouter-x.html
https://an.undulating.space/post/180927-er_alternate_firmware_benchmarks/
