Luci Web Interface slows down massively 2-3 days after setup

Installing and Using OpenWrt
1

Hi,

I've been googleing this issue for a while, but either I'm not getting the right keywords, or there is not much on this topic to find. The few posts that come up are talking about different situations or issues mostly. Trying those solutions, that seemed somewhat similar to my issue, did not help.
Honestly, I have no idea what causes the issue. And it is really annoying to test, because it takes days to take effect. I am hoping that someone here has some idea what could cause the issue (and maybe even have a sollution).

The Problem:
When I install a brand new installation of OpenWRT on my Linksys WRT32x, everything is perfect. I update the packages, install the few I need, etc. place the settings for my network and everything is just as it should be.
But after 2-3 days, even without touching anything on Luci, the web page is almost dead. Sometimes I'm lucky and the selected page loads, but most of the time the browser keeps loading and either gives me "Bad Gateway - The process did not produce any response", or Luci gives me "Error XHR request timed out".
It seems like something is overloading the process, like idk, too many connections or something. The page definitly is active and sometimes works. But most of the time, responsiveness is pretty much zero.
My internet Connection keeps working perfectly. I get my advertised speed, low latency, no package loss. SSH is no problem either. As far as I can tell, the effect is limited to Luci only.

One thing that I have tried is to schedule a reboot every 24h, hoping a reboot would fix the slow down. But it doesn't. When I select Reboot on the Webpage, it loads a few seconds, then puts me back to the page. So at least I can tell that the device itself does not cycle. It seems to be software only.

When I eventually get to the Flash Firmware page and set up everything from scratch, it's perfect again. For 2-3 days. But that's not good enough for me.
My second Router / AP ASUS AX53U doesn't have this issue. It runs and runs and runs. It works when I set it up almost identically to my Linksys (except WAN internet connection). It works when I run it as a Dumb Access Point.

Looking at the Connection List, I can not see anything unusual or exessively repeating. From what I can tell, both routers are pretty much in the same boat, so I'm confused to what even causes this behaviour.
I'm hesitant to post this over on the Github Bug Report, because I can't tell much on the issue, so I'm trying my luck here first.

2

You should never ever update packages.

Start from scratch, don't update, see if it helps.

3 Likes
3

Wow ... ok ... ?

But installing new ones is fine, right?

4

Yes, installing new packages (i.e. ones that don't already exist in the current installation) is fine.

1 Like
6

Ok thanks, I will try this and see what happens! :+1:

1 Like
7

Please check top via SSH to verify the CPU load. Broken DNS on the OpenWrt system can also slow down various parts of LuCI.

1 Like
8

image
image1356×1894 380 KB

Looks pretty safe to me. 99% idle should be enough headroom.

Unfortunately Luci got stuck again. It barely reacts to input and mostly throws errors and loading screens at my face. Like every other time, it‘s just the web interface that takes a hit. As far as I can tell, networking works perfectly fine otherwise.

I have noticed something on the boot drives. The alternate partition is listed as „Linksys/Unknown (Compressed)“. Could this be causing something? It does boot to it and then list it as OpenWrt, so I guess probably not.

9

What are you using your router for, considering you got clamav running...

What would happen if you'd only restart uhttpd?

1 Like
10

Nothing really. I‘m just thought rather-safe-than-sorry, but I don‘t need it really.

Restarting httpd did not help.

11

Hm, dnsmasq, hostapd, rpcd all shown in D state, which means they're stuck in some uninterruptible I/O operation. This could either be due to a broken storage (less likely) or clamav massively slowing down filesystem operations.

2 Likes
12

Hm, must have been a moment on the screenshot. The only service showing Stat D continuously is hostapd, run by user network.

Edit: I have uninstalled clamav, no improvement yet.

13

Although it usually manifests faster then 2-3 days, the description is similar to what occurs when attempting to enable WPA3 on the rango or venom devices.

3 Likes
14
  • You have something called UDPSpeeder installed
  • Despite top showing 99% idle CPU, the Load is ~3

Load is a per-core number, so e.g. if your device has:

  • 1 core, it's 3x loaded
  • 2 cores, it's 1.5x loadedd
  • 4 cores, it's only 0.75x (75%) loaded

So something is using your CPU.

15

Ok, now the system is starting to see ghosts.

grafik
grafik827×292 14.5 KB

I'll re-flash the entire thing from scratch, reset everything and see what happens. No Clamav, No UDPSpeeder this time.
This is what I run over SSH to install additional packages (plus the dependencies). I add nothing else:

opkg remove wpad-basic-wolfssl
opkg install luci-app-advanced-reboot diffutils luci-app-uhttpd iperf3 nano openssh-sftp-server wpad-wolfssl luci-app-vnstat2 luci-app-sqm

Is any of those a possible culprit?

Edit: Since I said, that those are all packages, I might want to add that I am interested in adding batman-adv to my network.

16

So, today I logged back into the web interface to see if it is still running. But it has slowed down again.
Here is a cut out of the result when I try 'top' in CLI:

grafik
grafik1252×1244 106 KB

rpcd and hostapd are the two services, that are consistently tagged with the Stat D. Load is pretty much as low as it gets. I have installed only those packages I have mentioned in the last post.

Is there some other command I should post the result of?

17

If you're still having the problem, maybe reset to default and don't install any packages... then test in ~2 days and if it's okay, install one package and then wait a few days, test again, repeat.

1 Like
18

sigh I guess that's the safest way to find the culprit.
I just hope it's not the firmware, or one of my network settings :sweat_smile:

19

Post your configs and we can review that first.

Please copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall
20
  • what version
  • maybe try building an image in the firmware selector with all requisite bits from the get-go.
  • try nginx instead.
  • try openssl instead, will probably be more performant on that device anyways.
21

what version

ATM, 22.03.2. I have noticed that 03.3 is in the works. Gonna try that too.

maybe try building an image in the firmware selector with all requisite bits from the get-go.

I did. But either I also need to add dependencies, or something else is missing. Because the flashed custom image does not run propperly. On one try it crashed my router so bad, I had to get into recovery mode to flash back an original. Not sure what was wrong though. i.e on the pre-set list of packages for custom builds, if I add the packages I mentioned earlier, it won't even give me a web interface. So I settled with original firmware only for now, as it's easy enough to install the few things post flash.

try nginx instead.

Not sure how to switch that. There are a lot of nginx packages to install. Any recommendations on which to get and/or in which order?

try openssl instead, will probably be more performant on that device anyways.

I switched to wolfssl because it claims to be more performant and up-to-date, as it was designed some time after openssl. Also, the firmware comes with wolfssl-base by default.
I can try it though.

Post your configs and we can review that first.

Alright. Just so you know, I learned most of the setup I use from a Youtube Channel, OneMarcFifty, plus some alternations to fit i.e device names or Vlan IDs to my liking. So some things may seem out-of-place or excessive, but I keep them mostly as testing platforms so they won't be affecting my actual lan too much, or even simply out of not-knowing-better.
As far as I can tell, everything is doing what it is supposed to though.

root@OpenWrt-Linksys:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'br-vlan.1'
        option bridge_empty '1'
        option stp '1'
        option igmp_snooping '1'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.1.250'
        list dns '192.168.1.32'   (= pi-hole)
        option ip6hint '1'
        option ip6ifaceid '::250'
        option ip6assign '64'

config device
        option name 'wan'
        option macaddr

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config device
        option type 'bridge'
        option name 'br-guest'
        option bridge_empty '1'
        option stp '1'
        option igmp_snooping '1'
        list ports 'bat0.30'
        list ports 'br-vlan.30'

config device
        option type 'bridge'
        option name 'br-iot'
        option bridge_empty '1'
        option stp '1'
        option igmp_snooping '1'
        list ports 'bat0.20'
        list ports 'br-vlan.20'

config device
        option type 'bridge'
        option name 'br-bat'
        option bridge_empty '1'
        option stp '1'
        option igmp_snooping '1'
        list ports 'br-vlan.10'

config interface 'guest'
        option proto 'static'
        option device 'br-guest'
        option ipaddr '192.168.30.250'
        option netmask '255.255.255.0'
        option ip6assign '64'
        option ip6hint '30'
        list dns '192.168.1.250'
        option ip6ifaceid '::250'

config interface 'iot'
        option proto 'static'
        option device 'br-iot'
        option ipaddr '192.168.20.250'
        option netmask '255.255.255.0'
        option ip6assign '64'
        option ip6hint '20'
        option ip6ifaceid '::250'

config device
        option type 'bridge'
        option name 'br-vlan'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'
        option bridge_empty '1'
        option stp '1'
        option igmp_snooping '1'

config bridge-vlan
        option device 'br-vlan'
        option vlan '1'
        list ports 'lan1:u*'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config bridge-vlan
        option device 'br-vlan'
        option vlan '2'
        option local '0'
        list ports 'lan1:t'

config bridge-vlan
        option device 'br-vlan'
        option vlan '3'
        option local '0'
        list ports 'lan1:t'

config bridge-vlan
        option device 'br-vlan'
        option vlan '10'
        list ports 'lan1:t'

config bridge-vlan
        option device 'br-vlan'
        option vlan '20'

config bridge-vlan
        option device 'br-vlan'
        option vlan '30'

config interface 'bat0'
        option proto 'batadv'
        option bridge_loop_avoidance '1'
        option gw_mode 'server'
        option hop_penalty '30'
        option ip6assign '64'
        option ip6hint '10'
        option ip6ifaceid '::250'
        option routing_algo 'BATMAN_IV'

config interface 'batwire'
        option proto 'batadv_hardif'
        option device 'br-bat'
        option master 'bat0'
        option ip6assign '64'
        option ip6hint '10'
        option ip6ifaceid '::250'

config interface 'batmesh'
        option proto 'batadv_hardif'
        option master 'bat0'
        option ip6assign '64'
        option ip6hint '10'
        option ip6ifaceid '::250'

root@OpenWrt-Linksys:~# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'soc/soc:pcie/pci0000:00/0000:00:01.0/0000:01:00.0'
        option band '5g'
        option txpower '15'
        option country
        option cell_density '1'
        option htmode 'VHT80'
        option channel '48'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option macaddr
        option ssid
        option dtim_period '3'
        option key
        option ieee80211r '1'
        option mobility_domain '123F'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        list maclist
        option macfilter 'allow'
        option encryption 'psk2+ccmp'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'soc/soc:pcie/pci0000:00/0000:00:02.0/0000:02:00.0'
        option band '2g'
        option htmode 'HT20'
        option channel '4'
        option txpower '13'
        option country
        option cell_density '1'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option mode 'ap'
        option macaddr
        option ssid
        option dtim_period '3'
        option encryption 'sae-mixed'
        option key
        option ieee80211r '1'
        option mobility_domain '20FF'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option disabled '1'
        option macfilter 'allow'
        list maclist
        option network 'IoT iot'

config wifi-device 'radio2'
        option type 'mac80211'
        option path 'platform/soc/soc:internal-regs/f10d8000.sdhci/mmc_host/mmc0              /mmc0:0001/mmc0:0001:1'
        option channel '34'
        option band '5g'
        option htmode 'VHT80'
        option disabled '1'

config wifi-iface 'wifinet3'
        option device 'radio1'
        option mode 'ap'
        option ssid
        option encryption 'sae-mixed'
        option dtim_period '3'
        option key
        option ieee80211r '1'
        option mobility_domain '12FF'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option network 'lan'
        list maclist
        option macfilter 'allow'

config wifi-iface 'wifinet4'
        option device 'radio0'
        option mode 'ap'
        option ssid
        option encryption 'sae-mixed'
        option isolate '1'
        option dtim_period '3'
        option key
        option ieee80211r '1'
        option mobility_domain '30FF'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option disabled '1'
        option network 'Guest guest'

config wifi-iface 'wifinet5'
        option device 'radio1'
        option mode 'mesh'
        option encryption 'sae'
        option mesh_id
        option mesh_fwding '0'
        option mesh_rssi_threshold '0'
        option key
        option disabled '1'
        option dtim_period '3'
        option network 'batmesh'

root@OpenWrt-Linksys:~# cat /etc/config/dhcp

config host
        option name 'OpenWRT-Linksys'
        option dns '1'
        option mac
        option ip '192.168.1.250'
        option leasetime '1440'

config domain
        option name 'OpenWRT-Linksys'
        option ip '192.168.1.250'
...
...
...
config dhcp 'IoT'
        option interface 'IoT'
        option start '100'
        option leasetime '12h'
        option limit '50'

config dhcp 'Guest'
        option interface 'Guest'
        option start '100'
        option leasetime '12h'
        option limit '50'
        list dhcp_option '6,192.168.1.32,1.1.1.1'

config dhcp 'guest'
        option interface 'guest'
        option start '100'
        option leasetime '12h'
        option limit '50'
        list dhcp_option '6,192.168.1.250,1.1.1.1'

config dhcp 'iot'
        option interface 'iot'
        option start '100'
        option leasetime '12h'
        option limit '50'

root@OpenWrt-Linksys:~# cat /etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'
        option flow_offloading '1'
        option flow_offloading_hw '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        option input 'DROP'
        option forward 'DROP'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option family 'ipv4'
        list icmp_type 'echo-request'
        option target 'DROP'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name
        option src 'wan'
        option src_dport
        option dest_ip
        option dest_port

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name
        option src 'wan'
        option src_dport
        option dest_ip
        option dest_port

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name
        option src 'wan'
        option src_dport
        option dest_ip
        option dest_port

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name
        option src 'wan'
        option src_dport
        option dest_ip
        option dest_port

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name
        option src 'wan'
        option src_dport
        option dest_port
        option dest_ip

config zone
        option name 'IoT'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'IoT'
        list network 'iot'

config forwarding
        option src 'lan'
        option dest 'IoT'

config zone
        option name 'Guest'
        option output 'ACCEPT'
        option forward 'REJECT'
        option input 'REJECT'
        list network 'Guest'
        list network 'guest'

config forwarding
        option src 'Guest'
        option dest 'wan'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name
        option src 'wan'
        option src_dport
        option dest_ip
        option dest_port

config rule
        option name 'Guest DNS DHCP'
        option src 'Guest'
        option dest_port '53 67 68'
        option target 'ACCEPT'

config rule
        option name 'IoT DNS DHCP'
        option src 'IoT'
        option dest_port '53 67 68'
        option target 'ACCEPT'