Raspberry Pi 4 | System is freezing up

Hi,

OpenWrt has been rock solid stable for a few months. I've recently switched to a 1 Gbps WAN connection and for the last couple of days the system has been locking up and crashing. This seems to happen a few minutes after reboot.

The netdata application starts to have gaps in the CPU graphs:

Luci becomes unreponsive.

Any ideas how I can troubleshoot this issue and find out what's causing it?

What version of OpenWRT?

Do you have a single (the onboard) ethernet for both WAN and LAN, or do you have a second device - i.e. some USB Ethernet device - if so, what make and model is it?

Are you using the wifi on the Pi?

you are exceeding the capabilities of the device(or bad disk);

  • reduce io or
  • upgrade to a more powerful device (x64)

I am using a Raspberry Pi 4 with a TP-Link UE300 USB device.

Hmmm, well, I know what not to use if/when I get gigabit

You never mentioned the version of OpenWRT

I don't think it is the Raspberry Pi 4. I'm starting to think it is the irqbalance package.

Perhaps it is messing with the core threads?

I've uninstalled it for now and will monitor the situation.

What are your temps when this is going on? Are you using a heatsink and fan? With those speeds you are probably hitting the thermal safety limit of 85C if you don't have good cooling. Probably overheating the usb controller or something too. Try running with the top open assuming you even have a case, that is.

The Raspberry Pi 4 is in a FLIRC Raspberry Pi 4 Case which acts as a heatsink.

I don't know how to pull a temperature reading from OpenWrt, but I don't think temperatures are an issue. The case is barely getting warm.

The wide spike in the middle of this graph is a speedtest running with SQM.

Now I have uninstalled the irqbalance package, the system appears to have returned to normal.

I am getting some CPU interupts during speedtests.

Interesting. Then you probably don't have overheating issue.

FWIW I use irqbalance on my pi4b too and do not have issues with it but I don't have a gig connection!.

You can also try packet steering (if you haven’t already), I’m not sure of the differences between it and irqbalance, but it serves the same purpose (network>interfaces>global network options).

System

Hostname OpenWrt
Model Raspberry Pi 4 Model B Rev 1.4
Architecture ARMv8 Processor rev 3
Firmware Version OpenWrt 21.02.0-rc2 r16122-c2139eef27 / LuCI openwrt-21.02 branch git-21.148.49484-14511e5
Kernel Version 5.4.119
Local Time 2021-10-25 10:15:35
Uptime 10h 19m 33s
Load Average 0.40, 0.45, 0.43

Are there any commands that I can run via the CLI that will help to diagnose/isolate this issue.

The router has been stable since I removed irqbalance, but I'm not sure if that is just a coincidence.

The problem first occurred when I enabled both 'packet steering' and irqbalance. Should both things be run at the same time?

try disabling one, then the other

Which is 'better'?

The one that works without issue :smiley:

I've read both are rather ineffective anyway

Without either Packet Steering or irqbalance enabled, the download speed maxes out at around 520Mbps with SQM enabled, so they are definitely effective.


I think I have resolved the issue.

The syslog contained a lot of errors relating to the 'wan' connection, which is my previous PPPoE broadband. I had unplugged the cable, but left the configuration alone. Adblock was also running very slowly and may have been experiencing conflicts.

I've now completely deleted the PPPoE interface and luci is very responsive and appears to be stable with both irqbalance and packet steering enabled.

SQM Enabled

SQM Disabled

Good for you! For any future readers looking for Pi help, I run packet steering and irqbalance and have not had a problem with them on my Pi4B, however as always YMMV.

I personally found packet steering to slightly hinder performance on the Pi 4. irqbalance is good (and necessary!) but can be achieved through manual pinning of interrupts as well without risking them shifting around later.

Do you have a link to a guide that explains how to do this?

Thanks.

  1. Disable or uninstall irqbalance :slight_smile:
  2. Add the following to /etc/rc.local (which just does what irqbalance does, except as a one-shot versus having them potentially shift later):
echo 2 > /proc/irq/31/smp_affinity
echo 8 > /proc/irq/32/smp_affinity
  1. Execute the above lines manually or reboot

Thanks.

I've added that to my startup options and it seems to be working.

image