Current default is Cubic, and yes we can indeed change the default TCP congestion control to BBR. However, BBR was disabled all along on the generic kernel config. I'm going to upload new images to see if it makes any difference.
1 Like
It's ready on the same releases page.
Run sysctl net.ipv4.tcp_congestion_control on terminal to make sure bbr is used instead of cubic.
You can also try cubic again to compare them. sysctl net.ipv4.tcp_congestion_control=cubic
Hi, Could someone please post a full bootlog of a system with the problem in the brcmfmac driver caused by the mac80211 update and one with the mac80211 backports reverted to backports-5.11 where it was last working.
I'm working on this at the moment - this does not just affect EA9500, Raspberry Pi 4B also appears to be affected. Currently rebuilding, the only error I have seen is one that appears, based on looking at commits, to be spurious/noncritical (the result of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/broadcom/brcm80211?h=v5.15&id=5ff013914c62c493c206d70554cfb1d311ea481a )
The driver loads successfully, but fails to bring up the link, and any attempt to run ifconfig or iw from the serial console hangs the system hard.
Give me an hour or two to rebuild with everything reverted. Trying to do clean rebuilds to avoid any possible oddities - unless you're OK with a build from circa October 20, which appears to be mac80211 5.10.68 per the git log, not 5.11
Edit: I have a complete log from the October 20 build, but pasting it in a message exceeds the message limit, it appears that only images can be uploaded as attachments unless I'm missing something, and pastebin is blocked from where I currently am... Any suggestions?
The commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/broadcom/brcm80211?h=v5.15&id=5ff013914c62c493c206d70554cfb1d311ea481a was only added in kernel 5.15-rc1. If this causes the problem backports 5.11 should still be ok.
It should also be fine if you just post about 10 lines before the brcmfmac driver starts till about 10 lines after it stopped from the working and the broken case.
Yup - as I mentioned, that commit is causing an error to be printed, but I am fairly certain that specific error is spurious/noise and not actual root cause of the problem, since the fallback behavior does appear to be working.
I'm almost done rebuilding a version reverted to 5.11, will post log excerpts once I finish that and run it.
October 20 build:
[ 8.275807] usbcore: registered new interface driver cdc_acm
[ 8.281476] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 8.289964] Loading modules backported from Linux version v5.10.68-0-g4d8524048a35
[ 8.297575] Backport generated by backports.git v5.10.68-1-0-ga4f9ba32
[ 8.304737] usbcore: registered new interface driver ems_usb
[ 8.310881] usbcore: registered new interface driver esd_usb2
[ 8.317301] usbcore: registered new interface driver gs_usb
[ 8.325894] usbcore: registered new interface driver kvaser_usb
[ 9.344435] mcp251x spi0.0: MCP251x didn't enter in conf mode after reset
[ 9.351377] mcp251x spi0.0: Probe failed, err=110
[ 9.359357] mcp251x: probe of spi0.0 failed with error -110
[ 9.397682] usbcore: registered new interface driver peak_usb
[ 9.403884] slcan: serial line CAN interface driver
[ 9.408790] slcan: 10 dynamic interface channels.
[ 9.415306] usbcore: registered new interface driver ums-alauda
[ 9.421595] usbcore: registered new interface driver ums-cypress
[ 9.427942] usbcore: registered new interface driver ums-datafab
[ 9.434207] usbcore: registered new interface driver ums-freecom
[ 9.440617] usbcore: registered new interface driver ums-isd200
[ 9.446881] usbcore: registered new interface driver ums-jumpshot
[ 9.453250] usbcore: registered new interface driver ums-karma
[ 9.459437] usbcore: registered new interface driver ums-sddr09
[ 9.465695] usbcore: registered new interface driver ums-sddr55
[ 9.471911] usbcore: registered new interface driver ums-usbat
[ 9.478293] usbcore: registered new interface driver usb_8dev
[ 9.485077] usbcore: registered new interface driver usbhid
[ 9.490652] usbhid: USB HID core driver
[ 9.496413] usbcore: registered new device driver usbip-host
[ 9.503400] usbcore: registered new interface driver usbserial_generic
[ 9.509970] usbserial: USB Serial support registered for generic
[ 9.516324] vcan: Virtual CAN interface driver
[ 9.521587] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 9.527524] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 3
[ 9.534773] vhci_hcd: created sysfs vhci_hcd.0
[ 9.539298] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[ 9.547569] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 9.554792] usb usb3: Product: USB/IP Virtual Host Controller
[ 9.560531] usb usb3: Manufacturer: Linux 5.10.72 vhci_hcd
[ 9.566014] usb usb3: SerialNumber: vhci_hcd.0
[ 9.570629] hub 3-0:1.0: USB hub found
[ 9.574395] hub 3-0:1.0: 8 ports detected
[ 9.578727] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 9.584658] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 4
[ 9.591974] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[ 9.600122] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
[ 9.608387] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 9.615611] usb usb4: Product: USB/IP Virtual Host Controller
[ 9.621350] usb usb4: Manufacturer: Linux 5.10.72 vhci_hcd
[ 9.626832] usb usb4: SerialNumber: vhci_hcd.0
[ 9.631418] hub 4-0:1.0: USB hub found
[ 9.635190] hub 4-0:1.0: 8 ports detected
[ 9.650604] xt_time: kernel timezone is -0000
[ 9.687717] usbcore: registered new interface driver cp210x
[ 9.693332] usbserial: USB Serial support registered for cp210x
[ 9.700497] usbcore: registered new interface driver ftdi_sio
[ 9.706313] usbserial: USB Serial support registered for FTDI USB Serial Device
[ 9.724849] usbcore: registered new interface driver pl2303
[ 9.730451] usbserial: USB Serial support registered for pl2303
[ 9.749160] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 9.967654] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 9.985887] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 23 2020 02:20:01 version 7.45.206 (r725000 CY) FWID 01-febaba43
[ 10.027828] usbcore: registered new interface driver brcmfmac
[ 10.034048] kmodloader: done loading kernel modules from /etc/modules.d/*
[ 11.433123] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 11.441813] bcmgenet fd580000.ethernet eth0: Link is Down
[ 11.447446] br-lan: port 1(eth0) entered blocking state
[ 11.452680] br-lan: port 1(eth0) entered disabled state
[ 11.458181] device eth0 entered promiscuous mode
[ 12.609598] netlink: 'iw': attribute type 302 has an invalid length.
[ 16.942550] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Latest git - driver loads, but link never comes up, and attempting to run ifconfig hangs the system:
[ 8.367110] usbcore: registered new interface driver cdc_acm
[ 8.372770] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 8.381152] Loading modules backported from Linux version v5.15-rc6-0-g519d81956ee2
[ 8.388823] Backport generated by backports.git v5.15-rc6-1-0-gd44432d6
[ 8.395875] usbcore: registered new interface driver ems_usb
[ 8.401905] usbcore: registered new interface driver esd_usb2
[ 8.410525] usbcore: registered new interface driver kvaser_usb
[ 8.442865] usbcore: registered new interface driver peak_usb
[ 8.449127] slcan: serial line CAN interface driver
[ 8.454007] slcan: 10 dynamic interface channels.
[ 8.460228] usbcore: registered new interface driver ums-alauda
[ 8.466789] usbcore: registered new interface driver ums-cypress
[ 8.473129] usbcore: registered new interface driver ums-datafab
[ 8.479455] usbcore: registered new interface driver ums-freecom
[ 8.485771] usbcore: registered new interface driver ums-isd200
[ 8.492005] usbcore: registered new interface driver ums-jumpshot
[ 8.498392] usbcore: registered new interface driver ums-karma
[ 8.504795] usbcore: registered new interface driver ums-sddr09
[ 8.511009] usbcore: registered new interface driver ums-sddr55
[ 8.517262] usbcore: registered new interface driver ums-usbat
[ 8.523718] usbcore: registered new interface driver usb_8dev
[ 8.530472] usbcore: registered new interface driver usbhid
[ 8.536056] usbhid: USB HID core driver
[ 8.541661] usbcore: registered new device driver usbip-host
[ 8.548685] usbcore: registered new interface driver usbserial_generic
[ 8.555270] usbserial: USB Serial support registered for generic
[ 8.561773] vcan: Virtual CAN interface driver
[ 8.567208] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 8.573137] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 3
[ 8.580399] vhci_hcd: created sysfs vhci_hcd.0
[ 8.584937] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[ 8.593197] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.600422] usb usb3: Product: USB/IP Virtual Host Controller
[ 8.606169] usb usb3: Manufacturer: Linux 5.10.76 vhci_hcd
[ 8.611647] usb usb3: SerialNumber: vhci_hcd.0
[ 8.616292] hub 3-0:1.0: USB hub found
[ 8.620053] hub 3-0:1.0: 8 ports detected
[ 8.624400] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 8.630316] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 4
[ 8.637582] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[ 8.645735] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
[ 8.653994] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.661214] usb usb4: Product: USB/IP Virtual Host Controller
[ 8.666961] usb usb4: Manufacturer: Linux 5.10.76 vhci_hcd
[ 8.672439] usb usb4: SerialNumber: vhci_hcd.0
[ 8.677042] hub 4-0:1.0: USB hub found
[ 8.680800] hub 4-0:1.0: 8 ports detected
[ 8.696146] xt_time: kernel timezone is -0000
[ 8.735522] usbcore: registered new interface driver cp210x
[ 8.741137] usbserial: USB Serial support registered for cp210x
[ 8.748035] usbcore: registered new interface driver ftdi_sio
[ 8.753812] usbserial: USB Serial support registered for FTDI USB Serial Device
[ 8.772297] usbcore: registered new interface driver pl2303
[ 8.777955] usbserial: USB Serial support registered for pl2303
[ 8.796737] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 8.805964] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.bin failed with error -2
[ 8.817647] brcmfmac mmc1:0001:1: Falling back to sysfs fallback for: brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.bin
[ 9.042507] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 9.060938] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 23 2020 02:20:01 version 7.45.206 (r725000 CY) FWID 01-febaba43
[ 9.102143] usbcore: registered new interface driver brcmfmac
[ 9.108417] kmodloader: done loading kernel modules from /etc/modules.d/*
[ 12.143347] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 12.151987] bcmgenet fd580000.ethernet eth0: Link is Down
[ 12.152375] br-lan: port 1(eth0) entered blocking state
[ 12.162672] br-lan: port 1(eth0) entered disabled state
[ 12.168103] device eth0 entered promiscuous mode
Revert to 5.11:
[ 8.274771] usbcore: registered new interface driver cdc_acm
[ 8.280432] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 8.288784] Loading modules backported from Linux version v5.11.22-0-gef8aed2fffe0
[ 8.296365] Backport generated by backports.git v5.11.22-1-0-g49432781
[ 8.303323] usbcore: registered new interface driver ems_usb
[ 8.309610] usbcore: registered new interface driver esd_usb2
[ 8.318262] usbcore: registered new interface driver kvaser_usb
[ 8.351311] usbcore: registered new interface driver peak_usb
[ 8.357684] slcan: serial line CAN interface driver
[ 8.362568] slcan: 10 dynamic interface channels.
[ 8.368603] usbcore: registered new interface driver ums-alauda
[ 8.374971] usbcore: registered new interface driver ums-cypress
[ 8.381271] usbcore: registered new interface driver ums-datafab
[ 8.387584] usbcore: registered new interface driver ums-freecom
[ 8.393874] usbcore: registered new interface driver ums-isd200
[ 8.400118] usbcore: registered new interface driver ums-jumpshot
[ 8.406486] usbcore: registered new interface driver ums-karma
[ 8.412652] usbcore: registered new interface driver ums-sddr09
[ 8.418870] usbcore: registered new interface driver ums-sddr55
[ 8.425108] usbcore: registered new interface driver ums-usbat
[ 8.431414] usbcore: registered new interface driver usb_8dev
[ 8.438154] usbcore: registered new interface driver usbhid
[ 8.443728] usbhid: USB HID core driver
[ 8.449157] usbcore: registered new device driver usbip-host
[ 8.456172] usbcore: registered new interface driver usbserial_generic
[ 8.462719] usbserial: USB Serial support registered for generic
[ 8.469085] vcan: Virtual CAN interface driver
[ 8.474355] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 8.480282] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 3
[ 8.487544] vhci_hcd: created sysfs vhci_hcd.0
[ 8.492302] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[ 8.500571] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.507794] usb usb3: Product: USB/IP Virtual Host Controller
[ 8.513532] usb usb3: Manufacturer: Linux 5.10.76 vhci_hcd
[ 8.519016] usb usb3: SerialNumber: vhci_hcd.0
[ 8.523634] hub 3-0:1.0: USB hub found
[ 8.527404] hub 3-0:1.0: 8 ports detected
[ 8.531747] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 8.537676] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 4
[ 8.545028] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[ 8.553171] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
[ 8.561436] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.568656] usb usb4: Product: USB/IP Virtual Host Controller
[ 8.574404] usb usb4: Manufacturer: Linux 5.10.76 vhci_hcd
[ 8.579882] usb usb4: SerialNumber: vhci_hcd.0
[ 8.584493] hub 4-0:1.0: USB hub found
[ 8.588250] hub 4-0:1.0: 8 ports detected
[ 8.603504] xt_time: kernel timezone is -0000
[ 8.642216] usbcore: registered new interface driver cp210x
[ 8.647866] usbserial: USB Serial support registered for cp210x
[ 8.654781] usbcore: registered new interface driver ftdi_sio
[ 8.660562] usbserial: USB Serial support registered for FTDI USB Serial Device
[ 8.679272] usbcore: registered new interface driver pl2303
[ 8.684887] usbserial: USB Serial support registered for pl2303
[ 8.700055] kmodloader: done loading kernel modules from /etc/modules.d/*
[ 8.710489] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 8.928184] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 8.949766] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 23 2020 02:20:01 version 7.45.206 (r725000 CY) FWID 01-febaba43
[ 8.992812] usbcore: registered new interface driver brcmfmac
[ 11.695257] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 11.703911] bcmgenet fd580000.ethernet eth0: Link is Down
[ 11.706939] br-lan: port 1(eth0) entered blocking state
[ 11.714676] br-lan: port 1(eth0) entered disabled state
[ 11.720088] device eth0 entered promiscuous mode
BusyBox v1.34.1 (2021-11-01 15:34:36 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt SNAPSHOT, r17781+152-a889dcd3f2
-----------------------------------------------------
root@Automation-pibridge164:/# [ 37.031759] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Note that it took a LONG time for the link to come up, and WLAN performance is extremely poor (to the point where LuCi is unusable due to XHR timeouts). There is a possibility that is due to lacking a Raspberry Pi specific backport from the Pi kernel repos that I have been procrastinating on upstreaming here. ifconfig and iw don't hang the system
Edit: Performance issues seem to be either a need to reboot at least once with a "sane" configuration, or due to lacking AC mode on Raspberry Pi 4 WiFi - #37 by Entropy512
Edit 2: I mentioned on github that sysupgrade, even of a locally obtained file (such as from USB storage) was failing with the "bad" brcmfmac - somehow, the bad driver was causing the kernel to behave EXTREMELY strangely - in addition to ifconfig and iw hanging the kernel silently, a sysupgrade would cause the console to spam a bunch of USB port errors, an mmcblk0 error, then the Pi would completely power off. The USB port errors would not occur during a transfer - manually copying an image from USB to /tmp would work, but then sysupgrading would send the system out to lunch, and I was able to tell from uname that sysupgrade didn't flash anything (the system had been running a kernel prior to the 5.10.76 bump, and a sysupgrade success should have taken it to 5.10.76)
Bootlog from Master branch at f3a28b6bcf984dc55ec2d675ba75f0d3bc794eb9 + https://github.com/arinc9/openwrt/commit/c8edd791ed3690efd25b92fcb96ec25e67b8ac90
Bootlog from Master branch reverted up to but not including backports-5.11 + https://github.com/arinc9/openwrt/commit/c8edd791ed3690efd25b92fcb96ec25e67b8ac90
I haven't yet confirmed that 5.12 breaks on Pi - but for reference as far as identifying which patch might be causing the issue - is backports-5.11 based on 5.11 or 5.11.22, or somewhere in between? Same for 5.12 - 5.12, or 5.12.19?
Edit: I shot myself in the foot and reassembled my test unit when things were working, so I can't serial console this unit without major headache. Building with mac80211 at backports-5.12, will debug it tomorrow, I have another pi that needs serial console love for other reasons in another facility and will probably be schlepping up there tomorrow morning.
Thank you for the logs, sadly I do not see a problem in them. the logging about the not found firmware file should not cause problems, it is just searching for an additional name now.
What exactly is not working?
@Entropy512 The older backports versions like backports-5.12 are using the latest stable version from that branch like 5.12.19.
There are not so many changes between 5.11.22 and 5.12.19 in brcmfmac, someone could try to bisect these changes and find the commit which broke it. I would start with backports-5.12.19 and then revert all the brcmfmac changes back to 5.11.22 and changed if it works, then just revert half of the brcmfmac changes and try to narrow it down to the kernel commit which broke it.
@rmilecki Do you have an idea what could be the problem?
Um... I'm a bit frustrated with this. I described EXACTLY what was not working. I also outright told you that the one log item about missing firmware is not expected to be root cause, AND I EVEN LINKED THE RELEVANT COMMIT FOR THAT ERROR and explained WHY it was not a problem.
Again, exactly what is not working is:
Despite no errors in kernel log, WLAN never connects - note utter lack of "link becomes ready" for wlan0 in affected kernels
As I already described, TWICE, ifconfig and iw hang the system silently - nothing printed even to serial console. CRs appear to be echoed, but that may be local echo... Need to run a few "serial disconnected" experiments for this one.
"strange" behavior in sysupgrade - I'll get a log for this one tomorrow, but piles of USB port errors even when running a sysupgrade from /tmp - manually copying a 100MB file from USB to /tmp issues no error. Strange that this is somehow related to a brcmfmac SDIO error, but it does NOT occur when reverting to 5.11-backports or earlier. Affects multiple platforms - executing sysupgrade on a system with "broken" brcmfmac breaks EA9500 and Raspberry Pi 4B - which are different base platforms. (Technically both BRCM... But in general Pis tend to be vastly better supported than anything else from BRCM...)
Unfortunately, beyond getting a "bad" sysupgrade log tomorrow, I'm going to be limited in what additional debugging I can do without incurring the wrath of upper management... (At least until I can get more approval to participate in more "upstream" activities from IT and legal, which I AM fighting for... But such things take time...)
Confirmed. Wireless LEDs also won't lit up.
Confirmed. ip commands also hang.
Confirmed. I don't even need to enable an SSID. The moment I boot the image with broken brcmfmac, sysupgrade won't flash any other images. Router reboots back to the same firmware. Only way is to flash via CFE Recovery Mode.
I'm only a user but really appreciate all the work everyone is putting! Thank you
2 Likes
I'd like to do this but I can't figure out how to. I checked the commit log and I can't figure out how this project pulls the drivers from the (specified?) version of the Linux kernel.
Then, I suppose I need to change mac80211 Makefile to use that commit of backports so it will build into the OpenWrt image.
Were you able to do a speedtest with bbr?
I havent had chance to flash your image yet. Maybe later today
Ok, I had 20 minutes so thought I'd flash the 5.11 release on your page
Speed is better, but still not what I was expecting
Everything else looks good so far
I ran this:
root@OpenWrt:~# sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
Should it not be BBR?
Yeah, odd. I'm rather certain I set BBR as the default. Can you change it and see what happens?
You know what I'm going to ask now right? 
I did a search but couldnt find the command
Yup.
sysctl net.ipv4.tcp_congestion_control=bbr
root@OpenWrt:~# sysctl net.ipv4.tcp_congestion_control=bbr
sysctl: write error: No such file or directory