Any kernel module package you enable/disable on the menuconfig will produce kernel with different hash when compiled. Therefore, the kernel dependency on the package will have a different hash. I don't know how the buildbot builds the packages but I suppose you need to keep the menuconfig unchanged and compile the packages without building a whole image.
I had just my iPhone X connected to the 5GHz radio. No other device was connected to any radios.
I've run download and upload speed tests for 20 seconds using iperf3 without touching the bluetooth kernel modules.
iperf3 server is run on the router, iperf3 client is run on an iPhone X using iSH app. Logs are taken from the router side.
I recorded no fluctuations.
Download:
Accepted connection from 10.10.10.214, port 61661
[ 5] local 10.10.10.1 port 5201 connected to 10.10.10.214 port 61662
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 68.8 MBytes 577 Mbits/sec 0 2.02 MBytes
[ 5] 1.00-2.00 sec 75.0 MBytes 628 Mbits/sec 0 2.02 MBytes
[ 5] 2.00-3.00 sec 72.5 MBytes 610 Mbits/sec 0 2.02 MBytes
[ 5] 3.00-4.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 4.00-5.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 5.00-6.00 sec 75.0 MBytes 629 Mbits/sec 0 2.02 MBytes
[ 5] 6.00-7.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 7.00-8.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 8.00-9.00 sec 72.5 MBytes 608 Mbits/sec 0 2.02 MBytes
[ 5] 9.00-10.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 10.00-11.00 sec 72.5 MBytes 608 Mbits/sec 0 2.02 MBytes
[ 5] 11.00-12.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 12.00-13.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 13.00-14.00 sec 73.8 MBytes 619 Mbits/sec 0 2.02 MBytes
[ 5] 14.00-15.00 sec 73.8 MBytes 618 Mbits/sec 0 2.02 MBytes
[ 5] 15.00-16.00 sec 72.5 MBytes 609 Mbits/sec 0 2.02 MBytes
[ 5] 16.00-17.00 sec 75.0 MBytes 629 Mbits/sec 0 2.02 MBytes
[ 5] 17.00-18.00 sec 72.5 MBytes 608 Mbits/sec 0 2.02 MBytes
[ 5] 18.00-19.00 sec 72.5 MBytes 608 Mbits/sec 0 2.02 MBytes
[ 5] 19.00-20.00 sec 75.0 MBytes 629 Mbits/sec 0 2.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.01 sec 1.43 GBytes 615 Mbits/sec 0 sender
Upload:
Accepted connection from 10.10.10.214, port 61664
[ 5] local 10.10.10.1 port 5201 connected to 10.10.10.214 port 61665
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 60.3 MBytes 506 Mbits/sec
[ 5] 1.00-2.00 sec 67.8 MBytes 569 Mbits/sec
[ 5] 2.00-3.00 sec 68.8 MBytes 577 Mbits/sec
[ 5] 3.00-4.00 sec 68.7 MBytes 576 Mbits/sec
[ 5] 4.00-5.00 sec 67.0 MBytes 562 Mbits/sec
[ 5] 5.00-6.00 sec 65.2 MBytes 547 Mbits/sec
[ 5] 6.00-7.00 sec 66.1 MBytes 554 Mbits/sec
[ 5] 7.00-8.00 sec 67.8 MBytes 568 Mbits/sec
[ 5] 8.00-9.00 sec 66.2 MBytes 555 Mbits/sec
[ 5] 9.00-10.00 sec 66.0 MBytes 553 Mbits/sec
[ 5] 10.00-11.00 sec 66.6 MBytes 559 Mbits/sec
[ 5] 11.00-12.00 sec 65.9 MBytes 553 Mbits/sec
[ 5] 12.00-13.00 sec 67.1 MBytes 562 Mbits/sec
[ 5] 13.00-14.00 sec 66.6 MBytes 559 Mbits/sec
[ 5] 14.00-15.00 sec 65.8 MBytes 551 Mbits/sec
[ 5] 15.00-16.00 sec 66.0 MBytes 554 Mbits/sec
[ 5] 16.00-17.00 sec 65.8 MBytes 552 Mbits/sec
[ 5] 17.00-18.00 sec 66.8 MBytes 560 Mbits/sec
[ 5] 18.00-19.00 sec 66.8 MBytes 561 Mbits/sec
[ 5] 19.00-20.00 sec 68.2 MBytes 572 Mbits/sec
[ 5] 20.00-20.01 sec 540 KBytes 475 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-20.01 sec 1.30 GBytes 558 Mbits/sec receiver
Could you test this on your side as well?
guys. why are you having wifi problem? since 21.02.1, i never have disconnect issue. so weird.
TLDR
I have a recipe, based on @arinc9's instructions that will produce a kernel package with the same magic number as the official release, and does not require patches:
git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git checkout v21.02.1
rm -rf package/kernel/{mac80211,ath10k-ct,mt76,rtl8812au-ct}
git checkout d1100c76b33ff68c6db0f5fa31a26532bdbb15c4 -- package/kernel/{mac80211,ath10k-ct,mt76,rtl8812au-ct}
rm -f package/kernel/mac80211/patches/ath/551-ath9k_ubnt_uap_plus_hsr.patch
./scripts/feeds update -a -f && ./scripts/feeds install -a -f
./scripts/feeds uninstall batman-adv
wget 'https://downloads.openwrt.org/releases/21.02.1/targets/mvebu/cortexa9/config.buildinfo'
cp config.buildinfo .config
make defconfig
make -j8 download && make -j8 BUILD_LOG=1
Detailed Explanation regarding the kernel vermagic hash
batman-adv fails to compile because of header mixup involving a mac80211-installed header. It could be trivially patched, but the purpose of the build is to allow openwrt package installation, so I don't imagine it would be much of a problem.
You can customize the configuration with make menuconfig, but beware that some packages will change the kernel config, changing the vermagic number. The value is computed at the beginning of make target/compile. So if you want to check if the configuration changes its value, run make target/compile, and wait a bit for the sources to be unpacked, and the .config files to be generated. It is saved to build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/linux-mvebu_cortexa9/linux-5.4.154/.vermagic.
Inclusion or exclusion of a package may change the hash. Notably, excluding the ath10k-ct, and other wireless driver packages will change the hash. The same happens if you don't install a feed. Luckily, omitting batman-adv did not alter the hash, so I took the shortcut and just removed it.
I'm just speculating, as my wrt3200s are used in production, and I can't just test things at will, but it may be possible to use the official image, and then just force-downgrade the mac80211 packages (kmod-cfg80211, kmod-mwifiex-sdio, kmod-mac80211) from a local build that shares the same magicver.
Edit: I'm adding kmod-mwlwifi to the list of packages that need to come from the local build. One of the functions has apparently changed name, judging by the difference when running strings /lib/modules/5.4.154/mwlwifi.ko:
--- strings 2021-11-17 16:07:23.206678891 -0300
+++ strings.new 2021-11-17 16:07:48.576678480 -0300
... minor binary stuff skipped.
@@ -934,10 +934,10 @@
__dev_kfree_skb_any
devm_ioremap_resource
filp_close
+ieee80211_channel_to_frequency
ieee80211_unregister_hw
cancel_work_sync
ieee80211_beacon_get_tim
-ieee80211_channel_to_freq_khz
skb_copy
__aeabi_uidivmod
ieee80211_hdrlen
3 Likes
If this proves to be possible, this would be absolutely phenomenal. This would definitely make troubleshooting this issue quicker and easier. If this proves successful, this could potentially be extended to WRT1200AC/WRT1900AC* devices too.
1 Like
I have edited the post above to change the order of the commands. The config.buildinfo file must be copied to .config after running the feeds commands. Otherwise, luci will be left out of the image, since the feeds command run make defconfig themselves, before the feeds are setup, so their packages are left out of the image. Nonetheless, the kernel magic version hash has not changed.
I've tested the differences between installed packages that reside under targets/mvebu/cortexa9/packages using my recipe:
$ for f in $(basename -a -s .control ~/src/test/openwrt/build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/linux-mvebu_cortexa9/target-dir-70ea5744/usr/lib/opkg/info/*.control | awk '{print $1;}'); do (VANILLA="$(grep -h ${f}_ sha256sums | awk '{print $1;}')"; NEW="$(grep ${f}_ bin/targets/mvebu/cortexa9/sha256sums | awk '{print $1;}')"; echo VANILLA=$VANILLA NEW=$NEW >/dev/null; [ "${VANILLA}" = "${NEW}" ] || echo $f) done
kmod-cfg80211
kmod-mac80211
kmod-mwifiex-sdio
kmod-mwlwifi
The first three packages are part of mac80211 that we changed, so 100% expected. kmod-mwlwifi was not on my initial list, since it matched my local vanilla 21.02.1 build. However, the hash did not match the official repository. I redid the vanilla build, and it matched the official one this time. I checked the strings diff and found an apparent function name change between the them:
$ diff -u strings strings.new
--- strings 2021-11-17 16:07:23.206678891 -0300
+++ strings.new 2021-11-17 16:07:48.576678480 -0300
@@ -1,5 +1,5 @@
Linux
-YE"4P
+YE"0P
mwl_fwcmd_set_slot_time
strnlen
strlen
@@ -504,9 +504,9 @@
_193
_182
_120
+_121
_122
_123
-_121
_104
_139
_140
@@ -934,10 +934,10 @@
__dev_kfree_skb_any
devm_ioremap_resource
filp_close
+ieee80211_channel_to_frequency
ieee80211_unregister_hw
cancel_work_sync
ieee80211_beacon_get_tim
-ieee80211_channel_to_freq_khz
skb_copy
__aeabi_uidivmod
ieee80211_hdrlen
Here's the full list of changed packages inside of targets/mvebu/cortexa9/packages:
$ for f in $(sed -ne '/packages/{s!.*\*packages/\([^_]\+\).*!\1!; p}' sha256sums); do (VANILLA="$(grep -h ${f}_ sha256sums | awk '{print $1;}')"; NEW="$(grep ${f}_ bin/targets/mvebu/cortexa9/sha256sums | awk '{print $1;}')"; echo VANILLA=$VANILLA NEW=$NEW >/dev/null; [ "${VANILLA}" = "${NEW}" ] || echo $f) done
kmod-adm8211
kmod-ar5523
kmod-ath10k-ct-smallbuffers
kmod-ath10k-ct
kmod-ath10k
kmod-ath5k
kmod-ath6kl-sdio
kmod-ath6kl-usb
kmod-ath6kl
kmod-ath9k-common
kmod-ath9k-htc
kmod-ath9k
kmod-ath
kmod-b43
kmod-b43legacy
kmod-batman-adv
kmod-brcmfmac
kmod-brcmsmac
kmod-brcmutil
kmod-carl9170
kmod-cfg80211
kmod-dahdi
kmod-hermes-pci
kmod-hermes-plx
kmod-hermes
kmod-hwmon-gpiofan
kmod-hwmon-it87
kmod-hwmon-lm75
kmod-hwmon-pwmfan
kmod-hwmon-tmp102
kmod-ikconfig
kmod-ipw2100
kmod-ipw2200
kmod-iwl-legacy
kmod-iwl3945
kmod-iwl4965
kmod-iwlwifi
kmod-jool
kmod-lib80211
kmod-libertas-sdio
kmod-libertas-spi
kmod-libertas-usb
kmod-libipw
kmod-mac80211-hwsim
kmod-mac80211
kmod-macremapper
kmod-mt76-connac
kmod-mt76-core
kmod-mt76-usb
kmod-mt7601u
kmod-mt7603
kmod-mt7615-common
kmod-mt7615-firmware
kmod-mt7615e
kmod-mt7663-firmware-ap
kmod-mt7663-firmware-sta
kmod-mt7663-usb-sdio
kmod-mt7663s
kmod-mt7663u
kmod-mt76
kmod-mt76x0-common
kmod-mt76x02-common
kmod-mt76x02-usb
kmod-mt76x0e
kmod-mt76x0u
kmod-mt76x2-common
kmod-mt76x2
kmod-mt76x2u
kmod-mt7915e
kmod-mt7921e
kmod-mwifiex-pcie
kmod-mwifiex-sdio
kmod-mwl8k
kmod-mwlwifi
kmod-owl-loader
kmod-p54-common
kmod-p54-pci
kmod-p54-usb
kmod-rsi91x-sdio
kmod-rsi91x-usb
kmod-rsi91x
kmod-rt2400-pci
kmod-rt2500-pci
kmod-rt2500-usb
kmod-rt2800-lib
kmod-rt2800-mmio
kmod-rt2800-pci
kmod-rt2800-usb
kmod-rt2x00-lib
kmod-rt2x00-mmio
kmod-rt2x00-pci
kmod-rt2x00-usb
kmod-rt61-pci
kmod-rt73-usb
kmod-rtl8180
kmod-rtl8187
kmod-rtl8192c-common
kmod-rtl8192ce
kmod-rtl8192cu
kmod-rtl8192de
kmod-rtl8192se
kmod-rtl8723bs
kmod-rtl8812au-ct
kmod-rtl8821ae
kmod-rtl8xxxu
kmod-rtlwifi-btcoexist
kmod-rtlwifi-pci
kmod-rtlwifi-usb
kmod-rtlwifi
kmod-rtw88
kmod-thermal
kmod-usb-serial-dmx
kmod-wil6210
kmod-wl12xx
kmod-wl18xx
kmod-wlcore
kmod-zd1211rw
libstdcpp6
libstdcpp6 stood out from the crowd, but the diff is just the build path stored in /usr/lib/libstdc++.so.6.0.25-gdb.py. Most of the remaining packages are wireless drivers. I truly expect that, unless there are other wireless drivers installed, the image changes are restricted to the four packages I mentioned above.
I've highlighted the fact that I was checking packages under target (aka openwrt_core in /etc/opkg/distfeeds.conf) since they are easy to match. The other repositories are moving targets, so it will be a lot harder to reach a conclusion, but as you move farther away from the kernel, the differences should matter less.
1 Like
Would you be able to upload the modified kmod-cfg80211, kmod-mac80211, kmod-mwifiex-sdio, and kmod-mwlwifi, or even the image you compiled, for others to test?
@cotequeiroz Is it the lack of matching vermagic that would have been the cause of the reboot loops in previous testing for installing kmod-mac80211 and kmod-cfg80211 a few days back on this thread?
The vermagic mismatch by itself may or may not have caused it.
If the ABI difference is just a different function name, like I' ve shown above--they are easily visible--then the module would not load. However, if you call a function with a pointer to a struct that changed from one version to another as parameter, then it may easily cause trouble.
Certainly using drivers from different kernel versions--like mwlwifi from plain 21.02.2 and mac80211 from a different backport version is a lot riskier than loading modules with slightly different build options. As I've shown above, you can get mismatching modules even if the kernel magic version matches.
I'm doing some more comparisons to point out the dangerous kmods. Most of them in the list I've posted above have a different package version--the ones built by mac80211--so those would be easily visible. I haven't finished my analisys yet, but it seems there are only a handful of them that would be silently installed--mwlwifi is notably in this short list.
1 Like
Here are the four kmods, the images (wrt3200acm & wrt32x), and a tarball with all four kmods:
https://drive.google.com/drive/folders/1qOFN0bt2XQTGeC-9LhWfOLqf93wOsiv3?usp=sharing
1 Like
thanks for that, can youuuuu post the solution for wpa3? new dev?
Nice work here @cotequeiroz !
Interesting that you found a way that avoid the need for patching the Makefile.
I'm going to test out your build script using my new Podman/Docker setup, which replicates that Openwrt build fleet.
I've created instructions for how to build everything using a Dockerfile + custom build script (I used rootless Podman), which you can find here: https://austindw.com/wrt3200acm-wrt32x-builds/#podman-docker-instructions
Based on your research/script, it should be easy to modify my custom-build.sh script to build everything reliably. We can either use that to create community images, or just vermagic-compatible kmod packages, as you seem to be discovering.
Edit: Oh, and I did manage to use this approach to create an image + fully built repo that we can use as a community. But I'd like to see if @cotequeiroz 's approach can be used to install packages on a stock Openwrt image, in which case my fully built image + repo won't be needed.
Edit2: I was able to reproduce @cotequeiroz 's build using my Dockerfile. I can confirm that the vermagic value matches the official OpenWrt build. Might have to test a stock image install + custom kmod install on my home router next...
2 Likes
There is no solution to WPA3. There never will be, unless by some miracle the driver + firmware blobs get updated by the source company.
In short, for everyone in the community - please stop asking for WPA3 on these devices. It's not going to happen.
I just installed @cotequeiroz 's kmods over my existing stock image and it looks like everything is working so far. At least wireless is working. Not much time to test anything else at the moment.
3 Likes
So just for confirmation sake, we also need swap out kmod-mwlwifi for the kmod-mwlwifi version that was specifically compiled under mac80211 5.7.5?
It is absolutely great seeing the progress toward (potentially) being able to use official OpenWrt stable release images with the ability to downgrade mac80211 5.10.x to mac80211 5.7.5 via opkg packages.
You’ll need to use the following packages, built with the older mac80211 version:
- kmod-cfg80211
- kmod-mac80211
- kmod-mwifiex-sdio
- kmod-mwlwifi
I’ve uploaded them here:
https://drive.google.com/drive/folders/1qOFN0bt2XQTGeC-9LhWfOLqf93wOsiv3?usp=sharing
If this turns out to be definitive, then it will be a good idea to bump the version to a very high number, so that opkg will not offer an upgrade to the stock version.
If the packages can be hosted reliably somewhere, then a repo can be added to /etc/opkg/customfeeds.conf, and opkg can pick them up from there.
1 Like
@cotequeiroz Thank you so much for your effort and contribution toward this issue. Your effort and time is greatly appreciated. And your attention to detail, of course.
Yes, this would be likely the most convenient method for users. I assume that the repo would also need to handle future releases as well (21.02.2, 21.02.3, etc.) and each would have it's own unique vermagic number.
I will install stock OpenWrt 21.02.1 tonight and do some testing of these driver packages as well.
EDIT: I believe that this method will also benefit WRT1200AC, WRT1900AC, and WRT1900ACS users too because, as far as I understand, the kmod-mwlwifi driver is the same as used for WRT3200ACM/WRT32X, with the only difference being the firmware blob which is in a separate package anyway.
1 Like
EDIT: I believe that this method will also benefit WRT1200AC, WRT1900AC, and WRT1900ACS users too because, as far as I understand, the
kmod-mwlwifidriver is the same as used for WRT3200ACM/WRT32X, with the only difference being the firmware blob which is in a separate package anyway.
Yes please. I've got a WRT19200ACS v2 and OpenWrt 21.02.1 is unusable for me due to intermittent connectivity problems. I'm not sure if it's the same issues you guys have been reporting but it never hurts to check.
To @slh's point though, this can't be a long-term solution. Ultimately, we need to isolate the problem and fix it at the source so everyone can go back to using official builds.
1 Like
@cowwoc, regarding the WRT1900ACS, I thought discussions above showed evidence that rolling back to mac80211 5.7.5 might not be necessary. Theoretically, even if the driver is the same, there might be something in their differing firmware that doesn't result in the same buggy behavior. Did you try the following suggestions?
@onetwofour presented the following tweaks regarding his WRT1900ACS, and tested with an iPhone:
- removed wpad-basic-wolfssl and installed wpad-basic, forced wpa2 auth
- disabled ipv6 (that I do every time I install/upgrade new version as it's a mess)
Put these three lines into /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1- turned on Disassociate On Low Acknowledgement - not sure if that even works or if that's turned on by default.
@petersmith also mentioned issues with Apple devices on his WRT1900ACS, and said the following worked for him:
try adding to /etc/rc.local
echo "0" >> /sys/kernel/debug/ieee80211/phy0/mwlwifi/tx_amsdu && echo "0" >> /sys/kernel
1 Like