How to compile OpenWrt with pcie_aspm=off kernel options?

For Developers
1

Hey friends, how are you doing.
Currently i am encountering some issues with my network cards and the 1000e intel Gbe driver. It's throwing messages about a hanging unit, causing packetloss.

Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   TDH                  <e>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   TDT                  <f5>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   next_to_use          <f5>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   next_to_clean        <c>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140] buffer_info[next_to_clean]:
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   time_stamp           <10111ee40>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   next_to_watch        <e>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   jiffies              <10111f0b8>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140]   next_to_watch.status <0>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140] MAC Status             <40080083>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140] PHY Status             <796d>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140] PHY 1000BASE-T Status  <3800>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140] PHY Extended Status    <3000>
Fri Jan  7 20:50:34 2022 kern.err kernel: [72110.433140] PCI Status             <10>
Fri Jan  7 20:50:36 2022 user.info mwan3track[3241]: Check (ping: latency=999999ms loss=100%) failed for target "8.8.4.4" on interface wan (bonding-i0.2100). Current score:

Some people on serverfault.com suggested to use pcie_aspm=off
as kernel flag or option. But i think it's not that easy to add that on OpenWrt. What's the best-practice way to apply this kernel flag, do i have to compile a new image? And if so, where do i add that?
Or is there a way to do so for my already running OpenWrt x86 build?

Thanks in advance!

2

You don't need to recompile to set this for x86, just edit /boot/grub/grub.cfg on the boot partition accordingly.

3

An easier method that works for me™ is to install ethtool and add ethtool -K eth0 tx off rx off in /etc/rclocal.d. Been using this for over a week now and my system / kernel log is clean and no more drop-outs in connectivity.

nitroshift

1 Like
4

Well that works as well, and the performance is quite satisfying. I don't see any performance issues though.

Is there a way to create a custom grub.cfg when compiling a new image? For every change i make i'd like to have a way to put it into a new image when i re-compile everything.

5

Just include ethtool in the build. Contents of /etc/rclocal.d survive sysupgrades.

nitroshift

6

The grub.cfg isn't part of the Linux kernel/openwrt image, it's used by the boot loader, whatever you add to it, will stay there until removed...