Ubiquiti EdgeRouter X, Loading OpenWrt and performance numbers


#1

Figured I would do a little write up on my adventures with a Ubiquiti EdgeRouter X and OpenWRT (18.06.1, r7258-5eb055306f).
Last Updated 1/10/2019

Back story.
This whole process started when I was trying to upgrade from OpenWRT(VM) to something bare metal. The rest of my house is all Ubiquiti Unifi gear, but Ubiquiti's current Unifi Routers (USG) dont fit my needs/cost use case. After much googling and frustration I stumbled across the Ubiquiti EdgeRouter X (ERX). I was VERY pleased to find a nice peace of hardware with solid specs (2 core, 4 threads, 256/256 Storage/RAM), NO wireless, PoE (24V), 5X 1Gbps ports,and an all metal case. Was even more happy that I could just power the ERX right off my core switch.

How to get from the stock EdgeOS to OpenWRT.
The process is simple. EdgeOS --> custom version of OpenWRT --> stock OpenWRT 18.06.1. Firstly we need to build the "openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar" which will be used as a stepping stone to get to a stock version of OpenWRT. Here are my steps to build openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar. But if you are lazy, here is the link to my compiled version: http://batbuilds.com/openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar

git clone https://www.github.com/openwrt/openwrt -b openwrt-18.06
cd openwrt/
./scripts/feeds update -a
./scripts/feeds install -a
make menuconfig
   Target System: "MediaTek Ralink MIPS"
   Subtarget: "MT7621 based boards"
   Target Profile: "Ubiquiti EdgeRouter X"
   Target Images: [*] ramdisk  ---> Compression (lzma)
   save, exit
make -j16 #(Adjust the -jXX for the number of threads your compiler has access to)
cd bin/targets/ramips/mt7621
cat sha256sums | grep openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar #Please note, if you compile, your sum will differ
    2239d14073c99cc871544eb897dfd63405a0d464226f8e5b3b9686a2557533d5 *openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar
scp openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar user@desktop:~

Get OpenWRT installed on the EdgeRouter X.
By default EdgeOS comes up at 192.168.1.1 on the physical port labeled"eth0". Default creds are ubnt/ubnt.

scp openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar ubnt@192.168.1.1:/tmp
ssh ubnt@192.168.1.1
cd /tmp
add system image /tmp/openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar
   Checking upgrade image...Done
   Preparing to upgrade...Done
   Copying upgrade image.../usr/bin/ubnt-upgrade: line 569: [: too many arguments
   Done
   Removing old image...Done
   Checking upgrade image...Done
   Copying config data...Done
   Finishing upgrade...Done
   Upgrade completed
show system image
   The system currently has the following image(s) installed:
   ramips                         r7627-753531d                  18.06-SNAPSHOT                 (default boot) 
   v1.9.7+hotfix.3.5013617.170830.0227 (running image) 

#A reboot is needed to boot default image, which is now are custom OpenWRT
root@ubnt:~# reboot
   Proceed with reboot? [confirm]y

#Reboot time was rather fast.  Less then 1m.

Moving from my custom image to a stock OpenWRT image
First thing on my list is getting a "stock" install of OpenWRT in place. This assumes you have a "WAN" connection, with functional internet, on port eth0. You could also download the firmware to a local computer and scp it over.
Pro Tip: By default OpenWRT's LAN connection will come up on the physical port "eth1".

ssh root@192.168.1.1
cd /tmp
wget http://downloads.openwrt.org/releases/18.06.1/targets/ramips/mt7621/openwrt-18.06.1-ramips-mt7621-ubnt-erx-squashfs-sysupgrade.tar
sysupgrade /tmp/openwrt-18.06.1-ramips-mt7621-ubnt-erx-squashfs-sysupgrade.tar
   Cannot save config while running from ramdisk.
   Commencing upgrade. Closing all shell sessions.
   Connection to 192.168.1.1 closed by remote host.
   Connection to 192.168.1.1 closed.

#Reboot time was rather fast.  Less then 1m.

Moving the "WAN" connection to "eth4"
At this point we are running a stock version of OpenWRT. One tweak I like to make is moving the "LAN" connection back to "eth0" and bumping the "WAN" connection to "eth4". That is simply done by tweaking around the switch ports as seen below.
Pro Tip: Ports 0~3, with my tweak, all act as "LAN" ports. Use any of them, like you would a normal switch. Only the physical port labeled "eth0" can support 24V PoE in. (PoE out is covered in the Wiki, see references below)

root@OpenWrt:~# cat /etc/config/network 
...
config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '0 1 2 3 6t'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '4 6t'

Performance test: LAN
At this point I have everything up and running and I am very happy to have internet up. New hardware is in play, but how fast is it? speedtest(dot)net puts my connection around 550/400 down/up, and the CPU on my ERX drops to less than 5% idle. I know my Centurylink(dot)net 1Gbps fiber connection can do 930/930. Lets look at what iperf says, for local traffic:

OpenWRT: OpenWRT CPU "45% idle", root@OpenWrt:~# iperf -c 10.1.1.30 -t 60 -P 1, [  8]  0.0-60.0 sec  5.77 GBytes   825 Mbits/sec
Main Server: OpenWRT CPU "22% idle", root@bigiron:~# iperf -c 10.1.1.1 -t 60 -P 1, [  3]  0.0-60.0 sec  5.94 GBytes   851 Mbits/sec

OpenWRT: OpenWRT CPU "33% idle", root@OpenWrt:~# iperf -c 10.1.1.30 -t 60 -P 4, [SUM]  0.0-60.1 sec  6.37 GBytes   911 Mbits/sec 
Main Server: OpenWRT CPU "10% idle", root@bigiron:~# iperf -c 10.1.1.1 -t 60 -P 4, [SUM]  0.0-60.0 sec  6.55 GBytes   938 Mbits/sec 

NEED MORE SPEED!! Enabling hardware flow control:
BIG FAT WARNING! Hardware flow offloading and SQM are incompatible with each other.
Through the web interface, Network --> Firewall, "Software flow offloading, which will then allow you to enable "Hardware flow offloading". All other setting are unchanged/defaults.
SQM on the other hand is an installed package 'opkg install luci-app-sqm' Network --> SQM QoS.

1. SQM works with software flow offload, allowing throughput of up to 200mbps with +3ms bufferbloat. Any thing faster maxes the CPU.
2. HW flow offload resulting in 900mbps but the +200ms bufferbloat hurts
3. NO HW flow offload, NO SQM, resulting in 550+ mbps with +80ms bufferbloat

Final thoughts.
At a $50 USD price point, this thing is AMAZING! Getting to OpenWRT is almost painless, as it requires one extra step. Serial console on the PCB is a VERY nice fall back, for when I brick the ERX. Hardware acceleration is a major boost, if you can suffer a little buffer bloat.

CenturyLink fiber config
For any one who follows in my tracks, here is my full networking config, including VLAN 201 stuff required by CenturyLink fiber. Please note on the VLAN 201, both ports need to be tagged "option ports '4t 6t'"

 cat /etc/config/network
config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option ipv6 '0'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.1'
        option proto 'static'
        option ipaddr '10.1.1.1'
        option netmask '255.255.255.0'
        option ipv6 '0'

config device 'lan_dev'
        option name 'eth0.1'
        option macaddr 'fc:ec:da:7e:b7:7e'

config interface 'wan'
        option ifname 'eth0.201'
        option proto 'pppoe'
        option password 'password'
        option ipv6 '0'
        option username 'fuzzy@nugget.net'

config device 'wan_dev'
        option name 'eth0.201'
        option macaddr 'fc:ec:da:7e:b7:7f'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '0 1 2 3 6t'

config switch_vlan
        option device 'switch0'
        option vlan '201'
        option ports '4t 6t'

Many thanks for all the people who have come before me, and posted below, to lay the ground work for this post.
References:

https://oldwiki.archive.openwrt.org/doc/howto/sqm
https://oldwiki.archive.openwrt.org/toh/ubiquiti/ubiquiti_edgerouter_x_er-x_ka
https://openwrt.org/toh/ubiquiti/ubiquiti_edgerouter_x_er-x_ka?s[]=edgerouter
http://sector5d.org/openwrt-on-the-ubiquiti-edgerouter-x.html
https://an.undulating.space/post/180927-er_alternate_firmware_benchmarks/

#2

Thanks for taking the time to write up your experiences! I added a link to this thread on the current device page, https://openwrt.org/toh/ubiquiti/ubiquiti_edgerouter_x_er-x_ka


#3

mt7621 has support for hardware-flowoffloading, which might get you far enough.


#4

Can confirm. This will give you 1 gbps speeds with 90-95% CPU idle. Highly recommended :slight_smile:


#5

But hardware flow offload is not compatible with SQM as far as I know.

Maybe doesn't matter in your case, but the gamers and voipers out there should know. My ATT fiber line still can easily saturate and have several hundred ms of bufferbloat without SQM.


#6

Any one willing to share some links/"how to"/patch set on how to enable hardware-flow offloading? I am happy to give it a test and update my original post.
Thanks!


#7

WOW! That made a huge difference!!!
"Network -> Firewall -> Software Flow Offloading:
When checked it reveals Hardware Flow Offloading, saying "Requires hardware NAT support. Implemented at least for mt7621"."

Testing
With OUT hardware flow offloading: 550Mbps, average ping time of "min/avg/max/mdev = 1.288/2.063/3.868/0.402 ms" CPU maxes out.
WITH hardware flow offloading: 900Mbps+, average ping time of "min/avg/max/mdev = 1.751/2.273/8.150/0.989 ms" CPU is 99% idle.

Note: Ping test was carried out with 60 pings, from my home network to the centurylink gateway. Trying to use the shorted possible hop, while still NAT'ing through the ERX.


#8

Can you run a dslreports speed test with bufferbloat test using hardware offload. You will probably want to set preferences to say 16 streams.


#9

According to DSLReports speed test @ 32 stream
With OUT hardware off load, around +30ms Bufferbloat (A rating)
WITH hardware off load, around +200ms Bufferbloat (C rating)


#10

@dingbatca

Hello

I'm a total newbie here and just flashed this. So far internet works on port 4 and lan works on ports 0-3. How did you get the web interface to show up? I type in 192.168.1.1 and I don't see the luci web interface.

I tried:

opkg update
opkg install luci
/etc/init.d/uhttpd start
/etc/init.d/uhttpd enable

but it was no good.

I also want to know how to revert back to stock firmware. It was easy to get to openWRT is it easy to get back? Can you point me into the right direction for reverting?

Thanks for helping me out!


#11

Its odd that the default ip doesnt redirect you.

But try http://192.168.1.1/cgi-bin/luci which should work


#12

@cavanaug Thanks man that worked for me. Now to figure out how to go back to stock haha, but I'll probably stick with openWRT for a bit since the web GUI works now.


#13

That is very strange that the stock web page did not load. I wonder if something cached in your browser. Can you try using a privet/incognito session to http://192.168.1.1?


#14

Funny you should mention that.

Instructions to restore EdgeOS on EdgeRouter X.

Just in case you need it.


#16

Fantastic article, it went flawlessly for my brand new ERX thanks to it.
I used the serial console for the install and upgrade to stock 18.06.1
Just out of curiosity, has anyone tried updating from EdgeOS' web using the compiled tar ?


#17

I can't believe this guy hasn't received more likes...

thank you for your precompiled openwrt file, finally was able to flash this BAD BAD BOYOpenEdgeX


#18

So for clarification, based on your two posts, I think you are saying:

  1. SQM works with software flow offload, allowing throughput of 550 mbps with +30ms bloat.
  2. HW flow offload is not compatible with SQM, resulting in 900+ mbps with +200ms bloat

Is that right? That's actually pretty good. Out of this relatively cheap device, you can nearly max out a 1 Gbps connection with HW offload (but with bloat) or you can get roughly half that with funcitonal SQM and minimal bloat.


#19

I'd like some clarity on this as well, as far as I can tell from reading back, the 550mbps might have been without software offload or SQM. I'd be very surprised if this device can do SQM at 550Mbps that's x86 territory


#20

Thanks for the comments/feedback. I edited/updated my original post to make this MUCH clearer.


#21

Can you also put the speeds you achieve and the SQM settings you set for these results? And was the With OUT hardware off load version also using software offload or not? Thanks!