ERLite3 offloading (DebWRT kernel?)

767fgh · May 16, 2024, 2:45am

I noticed that after install on ERLite3 only the IPv6 DHCP server is running. I had to replace that with the IPv4 capable DHCP server. Why is that?
When wired in, I only get 10% of the speed I got with the Ubiquity firmware, which is the same speed as I get from another OpenWRT device. Is this normal? Is there some hardware on the ERLite that is unsupported? Is there something I need to enable?

The guide I used:
https://openwrt.org/toh/ubiquiti/edgerouter_lite

edit:
It would be useful if I could test the speed past 200mb/s on the router itself

edit2:
Perhaps I spoke too soon. The slowness is due to lack of proprietary offloading support. This has been brought over into DebWRT, but AFAIK that's discontinued. Basically, if this router goes EOL, it'll either be 10x slower or be insecure. Food for thought. Next time I buy something, I need to dig into the chipsets to see how open they are to figure out how long a device will work securely for! What a PITA.

psherman · May 16, 2024, 3:34am

An IPv4 server should be active by default on all official OpenWrt images.

We'd need to see your config.

But as you mention, yes, they do have some secret sauce. That said, if you don't bridge the physical interfaces, that may help considerably. Again, lets see the config.

As far as the vendor firmware, it does seem that UI is working on a major update to the EdgeMax firmware. Check the EA software releases for them and you'll see. I have no idea how good it will be and how committed they are to continuing to support that line, but I was surprised when I saw an entirely new firmware in the works.

brada4 · May 16, 2024, 6:08am

There are many edges of "performance"
For what offload is concerned it applies to forwarded traffic and tcp+udp ie nat and routing, not io to device or bridging.
Test via forwarding:

But you can install iperf3 or netperf on a wired machine on different subnet from test client and asses speed at home.

Whichever test you choose:
Install htop, f2 -> unhide kernel threads, enable cpu detail -> f10
Now check router under test - red and lilac bars should be +/-10% equal on both cores.
If not - enable packet steering
....
expecting waveform test links from you and a screenshot.

767fgh · May 24, 2024, 12:27pm

Thanks. I didn't think I could get much more speed without offloading. Glad to hear that it might be possible.
How do I make an anonymised config file and post it here? Like this?

opkg install bandwidthd
opkg install base-files
opkg install bmon
opkg install busybox
opkg install ca-bundle
opkg install cgi-io
opkg install confuse
opkg install curl
opkg install dnsmasq
opkg install dropbear
opkg install e2fsprogs
opkg install firewall
opkg install fstools
opkg install fwtool
opkg install getrandom
opkg install ip6tables
opkg install iptables
opkg install iptables-mod-conntrack-extra
opkg install jshn
opkg install jsonfilter
opkg install kernel
opkg install kmod-ifb
opkg install kmod-ip6tables

config bandwidthd
	option dev		br-lan
	option subnets		"192.168.1.0/24"
	option skip_intervals	0
	option graph_cutoff	1024
	option promiscuous	true
	option output_cdf	false
	option recover_cdf	false
	option filter		ip
	option graph		true
	option meta_refresh	150

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv6 'server'
	option ra 'server'
	option ra_management '1'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'eth2'
	option start '100'
	option leasetime '12h'
	option limit '150'
	option interface 'eth2'


config dropbear
	option PasswordAuth 'on'
	option Port '22'
	option Interface 'lan'


config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config include
	option path '/etc/firewall.user'


config core 'main'
	option lang 'auto'
	option mediaurlbase '/luci-static/bootstrap'
	option resourcebase '/luci-static/resources'
	option ubuspath '/ubus/'

config extern 'flash_keep'
	option uci '/etc/config/'
	option dropbear '/etc/dropbear/'
	option openvpn '/etc/openvpn/'
	option passwd '/etc/passwd'
	option opkg '/etc/opkg.conf'
	option firewall '/etc/firewall.user'
	option uploads '/lib/uci/upload/'

config internal 'languages'

config internal 'sauth'
	option sessionpath '/tmp/luci-sessions'
	option sessiontime '3600'

config internal 'ccache'
	option enable '1'

config internal 'themes'
	option Bootstrap '/luci-static/bootstrap'

config internal 'apply'
	option rollback '90'
	option holdoff '4'
	option timeout '5'
	option display '1.5'

config internal 'diag'
	option dns 'openwrt.org'
	option ping 'openwrt.org'
	option route 'openwrt.org'


config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd2c:94e1:bb52::/48'

config interface 'lan'
	option type 'bridge'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option igmp_snooping '1'
	option delegate '0'
	option ifname 'eth0 eth2'
	list dns 'Cad7da.dns.nextdns.io'

config interface 'wan'
	option ifname 'eth1'
	option proto 'dhcp'

config interface 'wan6'
	option ifname 'eth1'
	option proto 'dhcpv6'


config nextdns 'main'
	option enabled '1'
	option setup_router '1'
	option report_client_info '1'
	option log_queries '0'
	option cache_size '0'
	option use_hosts '1'
	option max_inflight_requests '256'
	option detect_captive_portals '0'
	option bogus_priv '1'
	option cache_max_age '0s'
	option max_ttl '0s'
	option timeout '5s'
	option control '/var/run/nextdns.sock'
	list listen 'localhost:53'
	option auto_activate '1'
	option config 'cad7da'
	option hardened_privacy '1'
	option log_query '0'

config rpcd
	option socket /var/run/ubus.sock
	option timeout 30

config login
(redacted)

config system
	option hostname 'OpenWrt'
	option timezone 'UTC'
	option ttylogin '0'
	option log_size '64'
	option urandom_seed '0'

config timeserver 'ntp'
	option enabled '1'
	option enable_server '0'
	list server '0.openwrt.pool.ntp.org'
	list server '1.openwrt.pool.ntp.org'
	list server '2.openwrt.pool.ntp.org'
	list server '3.openwrt.pool.ntp.org'

package trafficshaper

config globals 'globals'
	option mark_mask '0xFF'

config wan 'wan'
	option downlink '20000'
	option uplink '20000'

config wan 'wanb'
	option downlink '15000'
	option uplink '5000'

config class 'corp'
	list network '192.168.1.0/24'
	list network 'fdc8:1234:1234:1::/64'
	option reserved_downlink '50%'
	option reserved_uplink '35%'
	option allowed_downlink '100%'
	option allowed_uplink '100%'

config class 'vpn'
	list network '192.168.2.0/24'
	list network 'fdc8:1234:1234:2::/64'
	option reserved_downlink '25%'
	option reserved_uplink '50%'
	option allowed_downlink '100%'
	option allowed_uplink '100%'

config class 'guest'
	list network '192.168.3.0/24'
	list network 'fdc8:1234:1234:3::/64'
	option reserved_downlink '25%'
	option reserved_uplink '15%'
	option allowed_downlink '25%'
	option allowed_uplink '15%'

config class 'default'
	option reserved_downlink '1000'
	option reserved_uplink '1000'
	option allowed_downlink '100%'
	option allowed_uplink '100%'

config network
	option init 'network'
	list affects 'dhcp'
	list affects 'radvd'

config wireless
	list affects 'network'

config firewall
	option init 'firewall'
	list affects 'luci-splash'
	list affects 'qos'
	list affects 'miniupnpd'

config olsr
	option init 'olsrd'

config dhcp
	option init 'dnsmasq'
	list affects 'odhcpd'

config odhcpd
	option init 'odhcpd'

config dropbear
	option init 'dropbear'

config httpd
	option init 'httpd'

config fstab
	option exec '/sbin/block mount'

config qos
	option init 'qos'

config system
	option init 'led'
	option exec '/etc/init.d/log reload'
	list affects 'luci_statistics'
	list affects 'dhcp'

config luci_splash
	option init 'luci_splash'

config upnpd
	option init 'miniupnpd'

config ntpclient
	option init 'ntpclient'

config samba
	option init 'samba'

config tinyproxy
	option init 'tinyproxy'

config nextdns
	option init 'nextdns'

# Server configuration
config uhttpd main

	# HTTP listen addresses, multiple allowed
	list listen_http	0.0.0.0:80
	list listen_http	[::]:80

	# HTTPS listen addresses, multiple allowed
	list listen_https	0.0.0.0:443
	list listen_https	[::]:443

	# Redirect HTTP requests to HTTPS if possible
	option redirect_https	1

	# Server document root
	option home		/www

	# Reject requests from RFC1918 IP addresses
	# directed to the servers public IP(s).
	# This is a DNS rebinding countermeasure.
	option rfc1918_filter 1

	# Maximum number of concurrent requests.
	# If this number is exceeded, further requests are
	# queued until the number of running requests drops
	# below the limit again.
	option max_requests 3

	# Maximum number of concurrent connections.
	# If this number is exceeded, further TCP connection
	# attempts are queued until the number of active
	# connections drops below the limit again.
	option max_connections 100

	# Certificate and private key for HTTPS.
	# If no listen_https addresses are given,
	# the key options are ignored.
	option cert		/etc/uhttpd.crt
	option key		/etc/uhttpd.key

	# CGI url prefix, will be searched in docroot.
	# Default is /cgi-bin
	option cgi_prefix	/cgi-bin

	# List of extension->interpreter mappings.
	# Files with an associated interpreter can
	# be called outside of the CGI prefix and do
	# not need to be executable.
#	list interpreter	".php=/usr/bin/php-cgi"
#	list interpreter	".cgi=/usr/bin/perl"

	# List of prefix->Lua handler mappings.
	# Any request to an URL beneath the prefix
	# will be dispatched to the associated Lua
	# handler script. Lua support is disabled when
	# no handler mappings are specified. Lua prefix
	# matches have precedence over the CGI prefix.
	list lua_prefix		"/cgi-bin/luci=/usr/lib/lua/luci/sgi/uhttpd.lua"

	# Specify the ubus-rpc prefix and socket path.
#	option ubus_prefix	/ubus
#	option ubus_socket	/var/run/ubus.sock

	# CGI/Lua timeout, if the called script does not
	# write data within the given amount of seconds,
	# the server will terminate the request with
	# 504 Gateway Timeout response.
	option script_timeout	60

	# Network timeout, if the current connection is
	# blocked for the specified amount of seconds,
	# the server will terminate the associated
	# request process.
	option network_timeout	30

	# HTTP Keep-Alive, specifies the timeout for persistent
	# HTTP/1.1 connections. Setting this to 0 will disable
	# persistent HTTP connections.
	option http_keepalive	20

	# TCP Keep-Alive, send periodic keep-alive probes
	# over established connections to detect dead peers.
	# The value is given in seconds to specify the
	# interval between subsequent probes.
	# Setting this to 0 will disable TCP keep-alive.
	option tcp_keepalive	1

	# Basic auth realm, defaults to local hostname
#	option realm	OpenWrt

	# Configuration file in busybox httpd format
#	option config	/etc/httpd.conf

	# Do not follow symlinks that point outside of the
	# home directory.
#	option no_symlinks	0

	# Do not produce directory listings but send 403
	# instead if a client requests an url pointing to
	# a directory without any index file.
#	option no_dirlists	0

	# Do not authenticate any ubus-rpc requests against
	# the ubus session/access procedure.
	# This is dangerous and should be always left off
	# except for development and debug purposes!
#	option no_ubusauth	0

	# For this instance of uhttpd use the listed httpauth
	# sections to require Basic auth to the specified
	# resources.
#	list httpauth prefix_user


# Defaults for automatic certificate and key generation
config cert defaults

	# Validity time
	option days		730

	# key type: rsa or ec
	option key_type		rsa

	# RSA key size
	option bits		2048

	# EC curve name
	# Curve names vary between mbedtls/px5g and openssl
	# P-256 or P-384 are guaranteed to work
	option ec_curve		P-256

	# Location
	option country		ZZ
	option state		Somewhere
	option location		Unknown

	# Common name
	option commonname	'OpenWrt'

# config httpauth prefix_user
#	option prefix /protected/url/path
#	option username user
#	option password 'plaintext_or_md5_or_$p$user_for_system_user'

brada4 · May 24, 2024, 12:41pm

You were asked for links to waveform tests with offload on and off and a screenshot from htop while tests are running. If you have problems with other packages, not overall forwarding performance you have to open new topics.

psherman · May 24, 2024, 2:53pm

Things look quite unusual here. I'm guessing this isn't an official OpenWrt build?

What is the output of

ubus call system board

767fgh · May 24, 2024, 2:54pm

OK! Got it. Sorry, got mixed up. By testing with and without offloading, I can only use offloading with the original Ubiquity firmware, so I'll need to plug that back in to test that. I don't quite understand that part of processing other than to confirm bufferbloat with offloading?

Anyway,
This is behind another OpenWRT router with wifi5 set to AP mode: WaveForm bufferbloat test
And this is set to wired: https://www.waveform.com/tools/bufferbloat?test-id=57136c3a-2181-4366-a928-4fb7b4387746
Both grade C:

This was taken during the bufferbloat test:

So if I understand this correctly, because both CPU cores are not equal, I should setup packet steering via:
`#opkg install shortcut-fe
config flow
option src 'wan'
option dest 'lan'
option proto 'tcp'
option helper 'shortcut-fe-tcp'

config flow
option src 'lan'
option dest 'wan'
option proto 'tcp'
option helper 'shortcut-fe-tcp'

#/etc/init.d/shortcut-fe enable
#/etc/init.d/shortcut-fe start`

767fgh · May 24, 2024, 2:55pm

 OpenWrt 19.07.8, r11364-ef56c85848
 -----------------------------------------------------
root@OpenWrt:~# ubus call system board
{
	"kernel": "4.14.241",
	"hostname": "OpenWrt",
	"system": "UBNT_E100 (CN5020p1.0-500-SCP)",
	"model": "UBNT_E100 (CN5020p1.0-500-SCP)",
	"board_name": "erlite",
	"release": {
		"distribution": "OpenWrt",
		"version": "19.07.8",
		"revision": "r11364-ef56c85848",
		"target": "octeon/generic",
		"description": "OpenWrt 19.07.8 r11364-ef56c85848"
	}
}
root@OpenWrt:~#

brada4 · May 24, 2024, 2:57pm

Looks like you need to upgrade to openwrt 23.05.3 and make measurements under generic configuration.
fe-lite (aka turboacc) from immortalwrt et al defeats any QoS by pushing packets through stack ahead of any QoS queues.

psherman · May 24, 2024, 2:59pm

19.07 is really old. It has been EOL for several years and has many known security vulnerabilities, and it is also unsupported. Please upgrade to the latest (23.05). During the upgrade, do not keep settings as they are not compatible.

767fgh · May 24, 2024, 3:06pm

Thanks for noticing that.
I hadn't chosen to install that version, so I've no idea how it got on there. I got the firmware from
https://openwrt.org/toh/ubiquiti/edgerouter_lite and I have no record of downloading that older version.
I'll use firmware-selector.openwrt.org this time....

brada4 · May 24, 2024, 3:12pm

Redo measurements without any additional QoS configuration, and compare wired/wireless. We cannot clean your air from radio waves, but we can maximize speed to LAN and consequentially reduce CPU load while on wifi.

767fgh · May 26, 2024, 2:37pm

Testing with the wired ethernet needs me to enable a new ethernet port and give it internet access. When I add this eth0 port to br-lan, I lose DHCP access to clients and connectivity on this in-use network, so after a lot of wrangling I'm just testing behind the wifi AP. Hopefully this testing is good enough?

After upgrading to
"version": "23.05.3",
"revision": "r23809-234f1a2efa", "kernel": "5.15.150", "target": "octeon/generic",
I got about 80mb/s, down from ~150mb/s. Thanks to your advice, I enabled PACKET STEERING and now throughput is up to nearly 200mb/s.

Bufferbloat is still grade C though:

Along the way, I installed NextDNS CLI and this setting wasn't passed onto DHCP clients initially. So I set 192.168.1.1 as the custom DNS in DHCP and now after a lot of restarts clients are working again.

CPU load between the cores varies under load from ~30-50%.

edit: Just got a bufferbloat grade B result after a few retests, for what it's worth

brada4 · May 26, 2024, 3:36pm

Enable soft offload but sneaky way:

Also check that in network/interfaces/global packet steering checkbox is enabled.
The load should be +/-10% between cores and certainly not hitting 100% on one core.
Current source of latency is few packets dropped and retransmitted due to unavailable CPU time.

brada4 · May 26, 2024, 3:39pm

While at soft offload you can try one of SQM qdiscs, though you need to switch to lighter one (noqueue<bfifo<pfifo_fast<fq_codel<pie<cake) if CPU starts hitting 100% when SQM qdisc is enabled.

767fgh · May 26, 2024, 11:47pm

Thanks! Great patch!
Load came down to ~60% under load with offloading enabled after the patches.
However, bufferbloat is still grade C.

I thought SQM is incompatible with offloading though? I only have cake and fq_codel available to try in the interface at the moment. I'm not sure if I should applying QoS to the bridge interface, or just the LAN or WAN. I'll read up on SQM.

Just to confirm, checking the hardware offloading box won't do anything, since hardware is unsupported? But it also won't lock me out of the router?

brada4 · May 27, 2024, 4:36am

You have to make br-wan and use sqm at physical interface.

brada4 · May 28, 2024, 3:40pm

Let me expand on br-wan

create bridge interface call it br-wan, txqueue 0, MAC address from WAN interface,"bring up empty bridge" save+apply
assign both WAN interfaces to this device (no save and apply)
add physical WAN port to br-wan, save and apply
sadly UCI applies all settings in alphabetical order, not typing order,so you have to restart WAN interfaces to acquire IP addresses back.

now you can qdisc on physical WAN interface undisturbed by firewall.

brada4 · June 2, 2024, 10:11am

You have more latency with upload, which means that with limited cpu you may set download bandwidth to zero disabling ingress shaping and halving CPU hunger of qdiscs.

767fgh · June 6, 2024, 3:52am

I think it can't be called br-wan, because the dash isn't allowed, so I could use br-wan
Why the need to apply SQM at the physical interface? What's the drawback of just applying to the bridge that's already in place, even just for testing?