I havent had chance to flash your image yet. Maybe later today
Ok, I had 20 minutes so thought I'd flash the 5.11 release on your page
Speed is better, but still not what I was expecting
Everything else looks good so far
I ran this:
root@OpenWrt:~# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
Should it not be BBR?
Yeah, odd. I'm rather certain I set BBR as the default. Can you change it and see what happens?
You know what I'm going to ask now right? 
I did a search but couldnt find the command
Yup.
sysctl net.ipv4.tcp_congestion_control=bbr
root@OpenWrt:~# sysctl net.ipv4.tcp_congestion_control=bbr
sysctl: write error: No such file or directory
OK you must be using an old image. BBR is not enabled on the system you're using. Can you flash the image through CFE if you haven't done already?
1 Like
Yup. Ages ago I managed to get a kernel commit applied to backports, but it was a bit of a pain in the ****
The backports git repo appears to be the git repo for the backporting tool itself, while OpenWRT is working with an already-processed backports tarball.
The commit logs for the kernels in question are:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/net/wireless/broadcom/brcm80211/brcmfmac?h=v5.12.19
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/net/wireless/broadcom/brcm80211/brcmfmac?h=v5.11.22
Unfortunately some of the commits are reordered so some analysis needs to be done there...
My initial guess would be that some part of the broadcom driver wasn't properly backported for https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.12.19&id=2fe8ef106238b274c505c480ecf00d8765abf0d8 or https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.12.19&id=a05829a7222e9d10c416dd2dbbf3929fe6646b89
Since these appear to be wide-reaching changes that would likely cross the boundaries between a newer and older kernel that would affect backports, and that seems to be the most likely place where a commit would be broken in backports but not mainline... On the other hand, everything outside of brcm that changed appears to be self-contained within a backports package?
I haven't the slightest clue how to revert these out of backports though...
https://gist.github.com/Entropy512/da22b4d5e8a292fae819df4c28f71aec - diff between backports-5.11.22-1 and 5.12.19-1, limited to just brcm changes
Rebuilding with kernel ubsan enabled, maybe that will help...
Edit: I did confirm that the oddball USB errors I see are not directly a part of the sysupgrade process, but are a problem in the reboot process that causes the Pi to do a full shutdown instead of a reboot. These strange USB port errors only occur with "broken" backports...
Edit 2: UBSAN with alignment checks is waaaaaaaaaaayyyy too spammy... Time for another pair of rebuilds...
Just an FYI but I just ran this on the August release and its reporting cubic?
root@OpenWrt:~# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
Yes, because as I said before, bbr was never used before. I don't want to work on this anymore for two reasons:
It was always cubic being used.
Trying bbr was just for testing, there's no way we can get it on OpenWrt trunk. So I don't want to pour more time into this.
You're going to have to figure out CFE and this yourself I'm afraid.
I don't want to spend any more time on this as I don't know how to work on the backports project and it's not my responsibility to look over the broken code which was put in without proper testing.
For now, I'm going to use images compiled by myself with reverts. I suggest these backports to be reverted on OpenWrt trunk as well until the problematic part of the code is identified and fixed.
P.S. I'm open to run any sort of tests on my router to fix the problem.
I have a bit more time on this, after all it's nearly impossible for volunteers to test all platforms at once. I'd definitely be a bit more annoyed about the situation if it had been the backports from an RC kernel that broke things, but it wasn't... On the other hand, being asked to state exactly what is breaking when I had already stated exactly what was breaking is frustrating.
That said - I'm running out of runway on this one. I have absolutely no idea how to proceed other than to get a syslog showing how the backports package is causing strange widespread system instability beyond just the fact that every networking tool causes the kernel to silently hang (either that or somehow panics never get printed even over serial console on Raspberry Pi, but I can't think of anything that would result in that behavior???). I've tried enabling UBSAN and it just results in piles of misalignment spam on both working and nonworking kernels - I have yet to find anything that causes any meaningful errors or diagnostics to get emitted other than the fact that somehow this issue causes USB port resets to misbehave during a reboot attempt, even though I'm 90% certain the Broadcom WLAN chip is on SDIO on a Pi 4.
The only thing I can think of is to somehow learn the backports tool, keep on running it against varying states of kernel tree, feed the results to OpenWRT and test - but I can't think of any efficient way to do this and as I said - I AM running out of runway before I need to go back to other tasks on my plate because I've lost too much time on this.
I have noticed that for some reason, even with backports-5.11, sometimes WLAN takes FOREVER to connect to the network, but that may be due to the fact that I've fired up too many fresh installs and various security monitors on our network are unhappy about seeing one IP constantly cycling MACs as I switch hardware, and also seeing MACs hop between two IP addresses, temporarily dumping said MACs into a blacklist...
Based on trying to forward-port patches: It isn't https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/net/wireless/broadcom/brcm80211/brcmfmac?h=v5.12.19&id=7dd56ea45a6686719a9d05c3e3f946a85809d322 - ifconfig works OK even with it present
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/patch/?id=2fe8ef106238b274c505c480ecf00d8765abf0d8 will apply but the end result will not compile and I don't have time to determine why, I assume it needs to be applied to the tree that goes into the backport tool and not afterwards
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/patch/?id=a05829a7222e9d10c416dd2dbbf3929fe6646b89 will not apply
Well, nobody is getting paid to do this so I can't blame anyone for not paying complete attention.
Things went well for me today, I think I'm going to take a look at this in the upcoming days.
For now, here's what I understand and how I plan to test stuff:
The backports tarball that includes drivers is pulled from https://cdn.kernel.org/pub/linux/kernel/projects/backports/stable/, the version is defined on the makefile of mac80211 package
-
Make my own tarball using 5.12.0 drivers to see if it works
- If yes, use newer versions
- If no, use older versions
-
Download 5.12.0 tarball
-
Replace drivers, include & net directories with 5.12.0 tarball's on backports-5.11 tarball
-
Host it on Github
-
Modify mac80211 package Makefile to retrieve my tarball instead
I'm attempting to use https://backports.wiki.kernel.org/index.php/Documentation/backports/hacking to generate tarballs off of modified kernel trees.
But I don't know of a way to point OpenWRT to use those local tarballs that isn't extremely painful.
I also really shouldn't be doing this since @hauke is basically asking us to download 6GB+ of stuff required by the backports tool... But today is surprisingly a much quieter one than expected because a dentist appointment forced work-from-home, so I can tell anyone who wants me to be physically present in the lab to wait until tomorrow.
WHAT THE HELL
https://backports.wiki.kernel.org/index.php/Documentation/backports/hacking - I had assumed that "path to your release" would be where it puts the tarball.
But gentree.py will complain if the directory is not empty and bomb out - AFTER MAKING THE DIRECTORY EMPTY. There is absolutely NO WARNING in the documentation that it will delete the entire output directory!
I just nuked my entire directory of git repos trying to do @hauke s job for him
Good thing I didn't try to output the tarball to ~
I think you need to calm down. This is community work. If things were perfect, we wouldn’t even be here discussing this issue. I’m pretty sure @hauke would want getting this fixed soon as much as we do.
1 Like
Yeah except he asked us to start bisecting things, which requires using the backports tool, which has NO WARNING WHATSOEVER that it'll delete all contents the output directory - and the tool won't issue a warning or prompt if the output directory is not empty. Sure, it'll bomb out if the directory is not empty, but NOT AFTER DELETING IT. The tool and its documentation are dangerous as hell, since anyone reading the documentation is not going to realize that whatever you set as the output directory is going to get nuked.
Fortunately most of what was in that directory that isn't already pushed to a git repository was recently sent to a coworker whom I was trying to get up to speed as far as working with openwrt, but still - I have a ton of repos to re-clone and properly set up remotes for, and I had an EXTREMELY close call that I didn't try to output a tarball to my home directory - I was considering that, and had I done that, the gentree script would have nuked ~
@hauke recently pushed a commit solving the system hangup issue.
@jonlad1 the snapshot builds on the firmware selector does not contain this commit yet. Now it does. Could you see if everything works fine on EA9500?
1 Like
Only just seen this - I will give the latest snap a try later today.....