Users needed to test Wi-Fi stability on Linksys WRT3200ACM & WRT32X on OpenWrt 21.02

TLDR

I have a recipe, based on @arinc9's instructions that will produce a kernel package with the same magic number as the official release, and does not require patches:

git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git checkout v21.02.1
rm -rf package/kernel/{mac80211,ath10k-ct,mt76,rtl8812au-ct}
git checkout d1100c76b33ff68c6db0f5fa31a26532bdbb15c4 -- package/kernel/{mac80211,ath10k-ct,mt76,rtl8812au-ct}
rm -f package/kernel/mac80211/patches/ath/551-ath9k_ubnt_uap_plus_hsr.patch
./scripts/feeds update -a -f && ./scripts/feeds install -a -f
./scripts/feeds uninstall batman-adv
wget 'https://downloads.openwrt.org/releases/21.02.1/targets/mvebu/cortexa9/config.buildinfo'
cp config.buildinfo .config
make defconfig
make -j8 download && make -j8 BUILD_LOG=1

Detailed Explanation regarding the kernel vermagic hash

batman-adv fails to compile because of header mixup involving a mac80211-installed header. It could be trivially patched, but the purpose of the build is to allow openwrt package installation, so I don't imagine it would be much of a problem.

You can customize the configuration with make menuconfig, but beware that some packages will change the kernel config, changing the vermagic number. The value is computed at the beginning of make target/compile. So if you want to check if the configuration changes its value, run make target/compile, and wait a bit for the sources to be unpacked, and the .config files to be generated. It is saved to build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/linux-mvebu_cortexa9/linux-5.4.154/.vermagic.

Inclusion or exclusion of a package may change the hash. Notably, excluding the ath10k-ct, and other wireless driver packages will change the hash. The same happens if you don't install a feed. Luckily, omitting batman-adv did not alter the hash, so I took the shortcut and just removed it.

I'm just speculating, as my wrt3200s are used in production, and I can't just test things at will, but it may be possible to use the official image, and then just force-downgrade the mac80211 packages (kmod-cfg80211, kmod-mwifiex-sdio, kmod-mac80211) from a local build that shares the same magicver.

Edit: I'm adding kmod-mwlwifi to the list of packages that need to come from the local build. One of the functions has apparently changed name, judging by the difference when running strings /lib/modules/5.4.154/mwlwifi.ko:

--- strings    	2021-11-17 16:07:23.206678891 -0300
+++ strings.new	2021-11-17 16:07:48.576678480 -0300

... minor binary stuff skipped.

@@ -934,10 +934,10 @@
 __dev_kfree_skb_any
 devm_ioremap_resource
 filp_close
+ieee80211_channel_to_frequency
 ieee80211_unregister_hw
 cancel_work_sync
 ieee80211_beacon_get_tim
-ieee80211_channel_to_freq_khz
 skb_copy
 __aeabi_uidivmod
 ieee80211_hdrlen
3 Likes

If this proves to be possible, this would be absolutely phenomenal. This would definitely make troubleshooting this issue quicker and easier. If this proves successful, this could potentially be extended to WRT1200AC/WRT1900AC* devices too.

1 Like

I have edited the post above to change the order of the commands. The config.buildinfo file must be copied to .config after running the feeds commands. Otherwise, luci will be left out of the image, since the feeds command run make defconfig themselves, before the feeds are setup, so their packages are left out of the image. Nonetheless, the kernel magic version hash has not changed.

I've tested the differences between installed packages that reside under targets/mvebu/cortexa9/packages using my recipe:

$ for f in $(basename -a -s .control ~/src/test/openwrt/build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/linux-mvebu_cortexa9/target-dir-70ea5744/usr/lib/opkg/info/*.control | awk '{print $1;}'); do (VANILLA="$(grep -h ${f}_ sha256sums | awk '{print $1;}')"; NEW="$(grep ${f}_ bin/targets/mvebu/cortexa9/sha256sums | awk '{print $1;}')"; echo VANILLA=$VANILLA NEW=$NEW >/dev/null; [ "${VANILLA}" = "${NEW}" ] || echo $f) done
kmod-cfg80211
kmod-mac80211
kmod-mwifiex-sdio
kmod-mwlwifi

The first three packages are part of mac80211 that we changed, so 100% expected. kmod-mwlwifi was not on my initial list, since it matched my local vanilla 21.02.1 build. However, the hash did not match the official repository. I redid the vanilla build, and it matched the official one this time. I checked the strings diff and found an apparent function name change between the them:

$ diff -u strings strings.new
--- strings    	2021-11-17 16:07:23.206678891 -0300
+++ strings.new	2021-11-17 16:07:48.576678480 -0300
@@ -1,5 +1,5 @@
 Linux
-YE"4P
+YE"0P
 mwl_fwcmd_set_slot_time
 strnlen
 strlen
@@ -504,9 +504,9 @@
 _193
 _182
 _120
+_121
 _122
 _123
-_121
 _104
 _139
 _140
@@ -934,10 +934,10 @@
 __dev_kfree_skb_any
 devm_ioremap_resource
 filp_close
+ieee80211_channel_to_frequency
 ieee80211_unregister_hw
 cancel_work_sync
 ieee80211_beacon_get_tim
-ieee80211_channel_to_freq_khz
 skb_copy
 __aeabi_uidivmod
 ieee80211_hdrlen

Here's the full list of changed packages inside of targets/mvebu/cortexa9/packages:

$ for f in $(sed -ne '/packages/{s!.*\*packages/\([^_]\+\).*!\1!; p}' sha256sums); do (VANILLA="$(grep -h ${f}_ sha256sums | awk '{print $1;}')"; NEW="$(grep ${f}_ bin/targets/mvebu/cortexa9/sha256sums | awk '{print $1;}')"; echo VANILLA=$VANILLA NEW=$NEW >/dev/null; [ "${VANILLA}" = "${NEW}" ] || echo $f) done
kmod-adm8211
kmod-ar5523
kmod-ath10k-ct-smallbuffers
kmod-ath10k-ct
kmod-ath10k
kmod-ath5k
kmod-ath6kl-sdio
kmod-ath6kl-usb
kmod-ath6kl
kmod-ath9k-common
kmod-ath9k-htc
kmod-ath9k
kmod-ath
kmod-b43
kmod-b43legacy
kmod-batman-adv
kmod-brcmfmac
kmod-brcmsmac
kmod-brcmutil
kmod-carl9170
kmod-cfg80211
kmod-dahdi
kmod-hermes-pci
kmod-hermes-plx
kmod-hermes
kmod-hwmon-gpiofan
kmod-hwmon-it87
kmod-hwmon-lm75
kmod-hwmon-pwmfan
kmod-hwmon-tmp102
kmod-ikconfig
kmod-ipw2100
kmod-ipw2200
kmod-iwl-legacy
kmod-iwl3945
kmod-iwl4965
kmod-iwlwifi
kmod-jool
kmod-lib80211
kmod-libertas-sdio
kmod-libertas-spi
kmod-libertas-usb
kmod-libipw
kmod-mac80211-hwsim
kmod-mac80211
kmod-macremapper
kmod-mt76-connac
kmod-mt76-core
kmod-mt76-usb
kmod-mt7601u
kmod-mt7603
kmod-mt7615-common
kmod-mt7615-firmware
kmod-mt7615e
kmod-mt7663-firmware-ap
kmod-mt7663-firmware-sta
kmod-mt7663-usb-sdio
kmod-mt7663s
kmod-mt7663u
kmod-mt76
kmod-mt76x0-common
kmod-mt76x02-common
kmod-mt76x02-usb
kmod-mt76x0e
kmod-mt76x0u
kmod-mt76x2-common
kmod-mt76x2
kmod-mt76x2u
kmod-mt7915e
kmod-mt7921e
kmod-mwifiex-pcie
kmod-mwifiex-sdio
kmod-mwl8k
kmod-mwlwifi
kmod-owl-loader
kmod-p54-common
kmod-p54-pci
kmod-p54-usb
kmod-rsi91x-sdio
kmod-rsi91x-usb
kmod-rsi91x
kmod-rt2400-pci
kmod-rt2500-pci
kmod-rt2500-usb
kmod-rt2800-lib
kmod-rt2800-mmio
kmod-rt2800-pci
kmod-rt2800-usb
kmod-rt2x00-lib
kmod-rt2x00-mmio
kmod-rt2x00-pci
kmod-rt2x00-usb
kmod-rt61-pci
kmod-rt73-usb
kmod-rtl8180
kmod-rtl8187
kmod-rtl8192c-common
kmod-rtl8192ce
kmod-rtl8192cu
kmod-rtl8192de
kmod-rtl8192se
kmod-rtl8723bs
kmod-rtl8812au-ct
kmod-rtl8821ae
kmod-rtl8xxxu
kmod-rtlwifi-btcoexist
kmod-rtlwifi-pci
kmod-rtlwifi-usb
kmod-rtlwifi
kmod-rtw88
kmod-thermal
kmod-usb-serial-dmx
kmod-wil6210
kmod-wl12xx
kmod-wl18xx
kmod-wlcore
kmod-zd1211rw
libstdcpp6

libstdcpp6 stood out from the crowd, but the diff is just the build path stored in /usr/lib/libstdc++.so.6.0.25-gdb.py. Most of the remaining packages are wireless drivers. I truly expect that, unless there are other wireless drivers installed, the image changes are restricted to the four packages I mentioned above.

I've highlighted the fact that I was checking packages under target (aka openwrt_core in /etc/opkg/distfeeds.conf) since they are easy to match. The other repositories are moving targets, so it will be a lot harder to reach a conclusion, but as you move farther away from the kernel, the differences should matter less.

1 Like

Would you be able to upload the modified kmod-cfg80211, kmod-mac80211, kmod-mwifiex-sdio, and kmod-mwlwifi, or even the image you compiled, for others to test?

@cotequeiroz Is it the lack of matching vermagic that would have been the cause of the reboot loops in previous testing for installing kmod-mac80211 and kmod-cfg80211 a few days back on this thread?

The vermagic mismatch by itself may or may not have caused it.
If the ABI difference is just a different function name, like I' ve shown above--they are easily visible--then the module would not load. However, if you call a function with a pointer to a struct that changed from one version to another as parameter, then it may easily cause trouble.

Certainly using drivers from different kernel versions--like mwlwifi from plain 21.02.2 and mac80211 from a different backport version is a lot riskier than loading modules with slightly different build options. As I've shown above, you can get mismatching modules even if the kernel magic version matches.

I'm doing some more comparisons to point out the dangerous kmods. Most of them in the list I've posted above have a different package version--the ones built by mac80211--so those would be easily visible. I haven't finished my analisys yet, but it seems there are only a handful of them that would be silently installed--mwlwifi is notably in this short list.

1 Like

Here are the four kmods, the images (wrt3200acm & wrt32x), and a tarball with all four kmods:
https://drive.google.com/drive/folders/1qOFN0bt2XQTGeC-9LhWfOLqf93wOsiv3?usp=sharing

1 Like

thanks for that, can youuuuu post the solution for wpa3? new dev?

Nice work here @cotequeiroz !

Interesting that you found a way that avoid the need for patching the Makefile.

I'm going to test out your build script using my new Podman/Docker setup, which replicates that Openwrt build fleet.

I've created instructions for how to build everything using a Dockerfile + custom build script (I used rootless Podman), which you can find here: https://austindw.com/wrt3200acm-wrt32x-builds/#podman-docker-instructions

Based on your research/script, it should be easy to modify my custom-build.sh script to build everything reliably. We can either use that to create community images, or just vermagic-compatible kmod packages, as you seem to be discovering.

Edit: Oh, and I did manage to use this approach to create an image + fully built repo that we can use as a community. But I'd like to see if @cotequeiroz 's approach can be used to install packages on a stock Openwrt image, in which case my fully built image + repo won't be needed.

Edit2: I was able to reproduce @cotequeiroz 's build using my Dockerfile. I can confirm that the vermagic value matches the official OpenWrt build. Might have to test a stock image install + custom kmod install on my home router next...

2 Likes

There is no solution to WPA3. There never will be, unless by some miracle the driver + firmware blobs get updated by the source company.

In short, for everyone in the community - please stop asking for WPA3 on these devices. It's not going to happen.

I just installed @cotequeiroz 's kmods over my existing stock image and it looks like everything is working so far. At least wireless is working. Not much time to test anything else at the moment.

3 Likes

So just for confirmation sake, we also need swap out kmod-mwlwifi for the kmod-mwlwifi version that was specifically compiled under mac80211 5.7.5?

It is absolutely great seeing the progress toward (potentially) being able to use official OpenWrt stable release images with the ability to downgrade mac80211 5.10.x to mac80211 5.7.5 via opkg packages.

You’ll need to use the following packages, built with the older mac80211 version:

  • kmod-cfg80211
  • kmod-mac80211
  • kmod-mwifiex-sdio
  • kmod-mwlwifi

I’ve uploaded them here:
https://drive.google.com/drive/folders/1qOFN0bt2XQTGeC-9LhWfOLqf93wOsiv3?usp=sharing

If this turns out to be definitive, then it will be a good idea to bump the version to a very high number, so that opkg will not offer an upgrade to the stock version.

If the packages can be hosted reliably somewhere, then a repo can be added to /etc/opkg/customfeeds.conf, and opkg can pick them up from there.

1 Like

@cotequeiroz Thank you so much for your effort and contribution toward this issue. Your effort and time is greatly appreciated. And your attention to detail, of course.

Yes, this would be likely the most convenient method for users. I assume that the repo would also need to handle future releases as well (21.02.2, 21.02.3, etc.) and each would have it's own unique vermagic number.

I will install stock OpenWrt 21.02.1 tonight and do some testing of these driver packages as well.

EDIT: I believe that this method will also benefit WRT1200AC, WRT1900AC, and WRT1900ACS users too because, as far as I understand, the kmod-mwlwifi driver is the same as used for WRT3200ACM/WRT32X, with the only difference being the firmware blob which is in a separate package anyway.

1 Like

EDIT: I believe that this method will also benefit WRT1200AC, WRT1900AC, and WRT1900ACS users too because, as far as I understand, the kmod-mwlwifi driver is the same as used for WRT3200ACM/WRT32X, with the only difference being the firmware blob which is in a separate package anyway.

Yes please. I've got a WRT19200ACS v2 and OpenWrt 21.02.1 is unusable for me due to intermittent connectivity problems. I'm not sure if it's the same issues you guys have been reporting but it never hurts to check.

To @slh's point though, this can't be a long-term solution. Ultimately, we need to isolate the problem and fix it at the source so everyone can go back to using official builds.

1 Like

@cowwoc, regarding the WRT1900ACS, I thought discussions above showed evidence that rolling back to mac80211 5.7.5 might not be necessary. Theoretically, even if the driver is the same, there might be something in their differing firmware that doesn't result in the same buggy behavior. Did you try the following suggestions?

@onetwofour presented the following tweaks regarding his WRT1900ACS, and tested with an iPhone:

  1. removed wpad-basic-wolfssl and installed wpad-basic, forced wpa2 auth
  2. disabled ipv6 (that I do every time I install/upgrade new version as it's a mess)
    Put these three lines into /etc/sysctl.conf
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
  3. turned on Disassociate On Low Acknowledgement - not sure if that even works or if that's turned on by default.

@petersmith also mentioned issues with Apple devices on his WRT1900ACS, and said the following worked for him:

try adding to /etc/rc.local

echo "0" >> /sys/kernel/debug/ieee80211/phy0/mwlwifi/tx_amsdu && echo "0" >> /sys/kernel
1 Like

It'll try it eventually. As you can imagine, my entire family gets a bit testy when the internet is down or choppy for a couple of hours :slight_smile: It doesn't help that the symptoms are intermittent. I wish there was a more deterministic way to check for a problem.

After further analysis of the kmod differences, here are my conclusions:

  1. Compatible packages:

    • kmod-thermal: empty package in offical repo; not built in custom
    • differences are due to leakage of build directories:
      • kmod-dahdi
      • kmod-jool
      • kmod-usb-serial-dmx_usb_module
      • libstdcpp
    • difference is restricted to a dependency on the empty-package kmod-thermal, which is not present in the custom image (in reality, no differences):
      • kmod-hwmon-gpiofan_5.4.154-1_arm_cortex-a9_vfpv3-d16.ipk
      • kmod-hwmon-it87_5.4.154-1_arm_cortex-a9_vfpv3-d16.ipk
      • kmod-hwmon-lm75_5.4.154-1_arm_cortex-a9_vfpv3-d16.ipk
      • kmod-hwmon-pwmfan_5.4.154-1_arm_cortex-a9_vfpv3-d16.ipk
      • kmod-hwmon-tmp102_5.4.154-1_arm_cortex-a9_vfpv3-d16.ipk
  2. Won't cause trouble, but the package should come from the same source as the running image.

    • kmod-ikconfig: the purpose of this module is to show the actual .config used to build the kernel. The .config file is different, so for the package to fulfill its purpose, it would need to match the source of the running image. That means if you're running a stock image, then you'd need to install the stock kmod-ikconfig; if you're running an image that was built using any of the recipes used here so far, you have to use that recipe's kmod-ikconfig, or /proc/config.gz will show a different configuration than the running kernel.
  3. kmod-macremapper: I'm just not sure about this one, but I would bet it causes no trouble. The differences are restricted to 2 identical blocks of strings being in different positions. If I sort the output of strings then they match 100%.

  4. Will probably cause trouble: wireless packages that appear to show different symbol names, probably due to the difference in mac80211, but that did not show a different version here compared to the official package. I since they are wireless drivers that depend on mac80211, use only the custom packages:

    • kmod-mwlwifi
    • kmod-rtl8812au-ct
  5. Packages that have changed version because of the downgrade to mac80211, ath10k-ct, mt76. These should only be installed using the custom packages:

    • kmod-adm8211
    • kmod-ar5523
    • kmod-ath10k-ct-smallbuffers
    • kmod-ath10k-ct
    • kmod-ath10k
    • kmod-ath5k
    • kmod-ath6kl-sdio
    • kmod-ath6kl-usb
    • kmod-ath6kl
    • kmod-ath9k-common
    • kmod-ath9k-htc
    • kmod-ath9k
    • kmod-ath
    • kmod-b43
    • kmod-b43legacy
    • kmod-batman-adv
    • kmod-brcmfmac
    • kmod-brcmsmac
    • kmod-brcmutil
    • kmod-carl9170
    • kmod-cfg80211
    • kmod-hermes-pci
    • kmod-hermes-plx
    • kmod-hermes
    • kmod-ipw2100
    • kmod-ipw2200
    • kmod-iwl-legacy
    • kmod-iwl3945
    • kmod-iwl4965
    • kmod-iwlwifi
    • kmod-lib80211
    • kmod-libertas-sdio
    • kmod-libertas-spi
    • kmod-libertas-usb
    • kmod-libipw
    • kmod-mac80211-hwsim
    • kmod-mac80211
    • kmod-mt76-connac
    • kmod-mt76-core
    • kmod-mt76-usb
    • kmod-mt7601u
    • kmod-mt7603
    • kmod-mt7615-common
    • kmod-mt7615-firmware
    • kmod-mt7615e
    • kmod-mt7663-firmware-ap
    • kmod-mt7663-firmware-sta
    • kmod-mt7663-usb-sdio
    • kmod-mt7663s
    • kmod-mt7663u
    • kmod-mt76
    • kmod-mt76x0-common
    • kmod-mt76x02-common
    • kmod-mt76x02-usb
    • kmod-mt76x0e
    • kmod-mt76x0u
    • kmod-mt76x2-common
    • kmod-mt76x2
    • kmod-mt76x2u
    • kmod-mt7915e
    • kmod-mt7921e
    • kmod-mwifiex-pcie
    • kmod-mwifiex-sdio
    • kmod-mwl8k
    • kmod-owl-loader
    • kmod-p54-common
    • kmod-p54-pci
    • kmod-p54-usb
    • kmod-rsi91x-sdio
    • kmod-rsi91x-usb
    • kmod-rsi91x
    • kmod-rt2400-pci
    • kmod-rt2500-pci
    • kmod-rt2500-usb
    • kmod-rt2800-lib
    • kmod-rt2800-mmio
    • kmod-rt2800-pci
    • kmod-rt2800-usb
    • kmod-rt2x00-lib
    • kmod-rt2x00-mmio
    • kmod-rt2x00-pci
    • kmod-rt2x00-usb
    • kmod-rt61-pci
    • kmod-rt73-usb
    • kmod-rtl8180
    • kmod-rtl8187
    • kmod-rtl8192c-common
    • kmod-rtl8192ce
    • kmod-rtl8192cu
    • kmod-rtl8192de
    • kmod-rtl8192se
    • kmod-rtl8723bs
    • kmod-rtl8821ae
    • kmod-rtl8xxxu
    • kmod-rtlwifi-btcoexist
    • kmod-rtlwifi-pci
    • kmod-rtlwifi-usb
    • kmod-rtlwifi
    • kmod-rtw88
    • kmod-wil6210
    • kmod-wl12xx
    • kmod-wl18xx
    • kmod-wlcore
    • kmod-zd1211rw
      Edited to make it clear that packages in "4" above are incompatible.
      .. and again to remove kmod-thermal from the list, as it is an empty package not present in the custom build.
1 Like

I can also confirm that running stock OpenWrt 21.02.1 on my WRT3200ACM and downgrading mac80211 (and related packages) from 5.10.x to the 5.7.5 packages from @cotequeiroz is successful and working as expected.

For anyone that tries this, make sure that you have the firmware blob package (mwlwifi-firmware-88w8964_2020-02-06-a2fd00bb-2_arm_cortex-a9_vfpv3-d16.ipk) installed before you reboot your router. Also keep in mind that the firmware blob would be different for WRT1200AC, WRT1900AC, and WRT1900ACS users.