Qualcommax & ath11k board calibration & BDF data woes

For Developers
1

Happy New Year everyone!

Howdy! So, recently I got miself a shiny new Buffalo WXR-6000AX12P, which is basically the same thing as https://openwrt.org/toh/buffalo/wxr-5950ax12. Buffalo tends to release identical hardware by slightly changed model names every year or so - likely just for marketing reasons.

Anyways, the WXR-5950AX12 got initial support about a year ago (thanks @musashino), so, an acquaintance and I got one device each, holiday discounts and everything...

Luckily, the 23.05 images TFTP booted and installed just fine on our WXR-6000AX12P routers, but that is where all the fun started. On both our devices, the 5GHz wifi would not recognize any of the clients, unless they were like 20cm away from the router. The clients "see" the AP just fine, with the great TX signal strength from the router, but even at 10cm from the router the RX signal was a disaster, as if no RX antenna existed at all. 2.4GHz radio worked just fine. After some tinkering it turned out that the board pq-wifi-buffalo_wxr-5950ax12 package that comes pre-installed in the official images and contains the API2 variant of athk11k board calibration data BDF was the culprit. Removing the pre-packaged calibration data file and manually extracting what seems to be a board calibration data + BDF from the device mtd (0:art), totally fixed the 5GHz wifi, on both devices.

It also happens that mtd calibration data on two WXR-6000AX12P of identical hardware revision is slightly different, suggesting that calibration data is not only device / revision specific, but it may even be specific to different production batches of identical firmware. Would it make sense that an updated calibration data for a slightly different antenna setups may be beneficial? I guess only Qualcomm has the answer...

So, my argument here is that pre-installing board calibration BDF package on qualcommax may not be an optimal approach. Do all of the currently supported devices in the target do come with the model/revision/batch specific calibration + BDF data in the mtd, and does the extraction with caldata.sh work as is to get it? There may be some minor issues with /etc/hotplug.d/firmware/11-ath11k-caldata and the ath11k firmware callbacks, for which I am still not sure yet if they are specific to target or device I am looking at. So for now the calibration data + BDF extraction has to be done manually, in case the pre-installed package is removed; details in https://openwrt.org/toh/buffalo/wxr-5950ax12#potential_issueslimitations

Anyway, I am about to propose some small changes in a PR, such that the pq-wifi-buffalo_wxr-5950ax12 is no longer part of the images anymore (IMHO, the packaged caldata should only be as an optional fallback instead), and the mtd art API1 data is used by default. The bonus benefit of this approach is that the wifi would work in uImage as well, which is currently not the case. But first I would like to hear some opinions from people who worked on the target supoprt and those that seem to own the same device.

Anyways, many thanks for all the OpenWRT awesomeness & hard work invested in this target m(_ _)m.

^ @robimarko @Ansuel @fakemanhk @musashino

EDIT:

This thread went to a pretty wrong direction, likely due to my misunderstanding of separation of board calibration data and BDF. The fact that the data from ART mtd partition does have very similar structure as the real BDF, and that driver successfully loads it and the radios just work as it was a real BDF, made me wrongly believe that BDF is the same thing as calibration data.

Lesson learned.

I am still baffled about the reason why the data in my device's ART can work as both BDF and calibration data at the same time. If someone can shed some light on that mystery, beers are on me.

2 Likes
2

I think you might be mixing BDF and calibration data. The calibration data is unique to each individual device, and Openwrt does not package any calibration data as that is being read and applied for each device during boot. The BDF however is only unique to a device type, hence that is part of the Openwrt package and the source is usually the commercial firmware itself (as it is a proprietary closed format by Qualcomm). And the radios does not function without the BDF.

that comes pre-installed in the official images and contains the API2 variant of athk11k board calibration data was the culprit

Again, the cal data is not packaged, and is read and applied every boot. If you delete it from the filesystem, it will get back at next reboot. If you deleted the BDF, that would make the wifi not work at all, so you are mixing up stuff here.

To be clear:

root@XAX6:/lib/firmware/ath11k/IPQ8074/hw2.0# ls -la
drwxr-xr-x    1 root     root           248 Dec 21 11:26 .
drwxr-xr-x    1 root     root           224 Dec 21 11:26 ..
-rw-r--r--    1 root     root        131172 Dec 21 11:26 board-2.bin
-rw-r--r--    1 root     root        131072 Dec 21 11:26 cal-ahb-c000000.wifi.bin

The "board-2.bin" is the BDF
The "cal-ahb-c000000.wifi.bin" is the unique cal data

3

What you are doing by dropping the BDF package is falling back to the generic one, that is incorrect as each board has a BDF that configures a lot of things that are specific to it and you cannot just reuse the generic one.

4

Happy New Year @dchard

I think you might be mixing BDF and calibration data.

The "board-2.bin" is the BDF

I admit the terminology is very new to me, but are you absolutely sure about this? My understanding is that what you call "BDF" is nothing more than some cal-ahb-c000000.bin or board.bin cal data file(s) wrapped in some metadata header, which is doable by ath11k-bdencoder. I actually tried that few days ago, created my own board-2.bin directly from from 0:art, and it all just worked. The ath11k kernel driver code suggest that the only difference between board.bin (aka cal-<bus>-<id>.bin) and boad-2.bin it the caldata firmware loading API, API1 vs API2. No?

Openwrt does not package any calibration data as that is being read and applied for each device during boot.

Well, that depends. On this particular device in question, the board-2.bin "BDF" is indeed part of the ipq-wifi-buffalo_wxr-5950ax12 package, evidence here and here. And again, in ath11k code itself, API2 has precedence over API1, so if packaged board-2.bin exists (which does seem to contain caldata), the caldata extracted from mtd art is simply ignored. The API1 is used as failback if API2 caldata is not available.

5

Ahh did not realized there is a fallback. Last time I deleted the BDF ath11k simply failed to start because of missing BDF. The rest is clear.

What I cant figure out is what is the actual issue, and what he actually did to fix it as that part is not detailed enough.

6

True, but failback does not work if one simply removes the package from the image, due to two reasons:

  1. Missing /lib/firmware/ath11k/IPQ8074/hw2.0 path
  2. Some weird ath11k firmware callback ordering logic

@robimarko can you please look into this? Here some evidence for 2), on how ath11k is doing firmware callbacks. For the sake of debugging this, I modded the /etc/hotplug.d/firmware/11-ath11k-caldata on my device into this:

#!/bin/sh

echo "$(date +%T.%4N) I was called about $FIRMWARE" >> /tmp/log/11-ath11k-caldata.log
[ -e /lib/firmware/$FIRMWARE ] && exit 0

. /lib/functions/caldata.sh

board=$(board_name)

case "$FIRMWARE" in
"ath11k/IPQ8074/hw2.0/cal-ahb-c000000.wifi.bin")
	case "$board" in
	arcadyan,aw1000|\
	buffalo,wxr-5950ax12|\
	compex,wpq873|\
	dynalink,dl-wrx36|\
	edgecore,eap102|\
	edimax,cax1800|\
	netgear,rax120v2|\
	netgear,wax218|\
	netgear,wax620|\
	netgear,wax630|\
	qnap,301w|\
	redmi,ax6|\
	xiaomi,ax3600|\
	xiaomi,ax9000|\
	yuncore,ax880|\
	zte,mf269|\
	zyxel,nbg7815)
		echo "$(date +%T.%4N) I am asked to dump caldata to /lib/firmware/$FIRMWARE" >> /tmp/log/11-ath11k-caldata.log
		mkdir -p /lib/firmware/${FIRMWARE%/*}
		caldata_extract "0:art" 0x1000 0x20000
		;;
	prpl,haze)
		caldata_extract_mmc "0:ART" 0x1000 0x20000
		;;
	esac
	;;
"ath11k/QCN9074/hw1.0/cal-pci-0000:01:00.0.bin"|\
"ath11k/QCN9074/hw1.0/cal-pci-0001:01:00.0.bin")
	case "$board" in
	prpl,haze)
		caldata_extract_mmc "0:ART" 0x26800 0x20000
		;;
	xiaomi,ax9000)
		caldata_extract "0:art" 0x26800 0x20000
		;;
	esac
	;;
*)
	exit 1
	;;
esac

This is is what gets logged in /tmp/log/11-ath11k-caldata.log

root@OpenWrt:~# cat /tmp/log/11-ath11k-caldata.log 
10:21:20. I was called about ath11k/IPQ8074/hw2.0/board-2.bin
10:21:20. I was called about ath11k/IPQ8074/hw2.0/board-2.bin
10:21:20. I was called about ath11k/IPQ8074/hw2.0/board.bin
10:21:20. I was called about ath11k/IPQ8074/hw2.0/board.bin

and in kernel:

root@OpenWrt:~# dmesg | grep ath11k
[    9.299641] ath11k c000000.wifi: ipq8074 hw2.0
[    9.299674] ath11k c000000.wifi: FW memory mode: 0
[   10.351861] ath11k c000000.wifi: qmi ignore invalid mem req type 3
[   10.359319] ath11k c000000.wifi: chip_id 0x0 chip_family 0x0 board_id 0xff soc_id 0xffffffff
[   10.359367] ath11k c000000.wifi: fw_version 0x290104a5 fw_build_timestamp 2023-08-02 20:32 fw_build_id WLAN.HK.2.9.0.1-01890-QCAHKSWPL_SILICONZ-1
[   10.447148] ath11k c000000.wifi: failed to fetch board data for bus=ahb,qmi-chip-id=0,qmi-board-id=255,variant=Buffalo-WXR-5950AX12 from ath11k/IPQ8074/hw2.0/board-2.bin
[   10.447210] ath11k c000000.wifi: failed to fetch board data for bus=ahb,qmi-chip-id=0,qmi-board-id=255 from ath11k/IPQ8074/hw2.0/board-2.bin
[   10.461363] ath11k c000000.wifi: failed to fetch board data for bus=ahb,qmi-chip-id=0,qmi-board-id=255 from ath11k/IPQ8074/hw2.0/board-2.bin
[   10.474023] ath11k c000000.wifi: failed to fetch board.bin from IPQ8074/hw2.0
[   10.486583] ath11k c000000.wifi: qmi failed to fetch board file: -12
[   10.493566] ath11k c000000.wifi: failed to load board data file: -12

So, since pre-packaged broken board-2.bin ain't there no more, the driver does API1 fail-back callback for board.bin but not cal-<bus>-<id>.bin at all! WTF?

And here it goes weird; after adding an extra case on top of to /etc/hotplug.d/firmware/11-ath11k-caldata like this and rebooting:

"ath11k/IPQ8074/hw2.0/board.bin")
	echo "$(date +%T.%4N) I am asked to dump caldata to /lib/firmware/$FIRMWARE" >> /tmp/log/11-ath11k-caldata.log
	mkdir -p /lib/firmware/${FIRMWARE%/*}
	caldata_extract "0:art" 0x1000 0x20000
	;;

this gets logged:

root@OpenWrt:~# cat /tmp/log/11-ath11k-caldata.log 
10:27:09. I was called about ath11k/IPQ8074/hw2.0/board-2.bin
10:27:09. I was called about ath11k/IPQ8074/hw2.0/board-2.bin
10:27:09. I was called about ath11k/IPQ8074/hw2.0/board.bin
10:27:09. I am asked to dump caldata to /lib/firmware/ath11k/IPQ8074/hw2.0/board.bin
10:27:09. I was called about ath11k/IPQ8074/hw2.0/board.bin
10:27:09. I was called about ath11k/IPQ8074/hw2.0/cal-ahb-c000000.wifi.bin
10:27:09. I am asked to dump caldata to /lib/firmware/ath11k/IPQ8074/hw2.0/cal-ahb-c000000.wifi.bin
10:27:09. I was called about ath11k/IPQ8074/hw2.0/cal-ahb-c000000.wifi.bin

kernel:

root@OpenWrt:~# dmesg | grep ath11k
[    9.348901] ath11k c000000.wifi: ipq8074 hw2.0
[    9.348934] ath11k c000000.wifi: FW memory mode: 0
[   10.400754] ath11k c000000.wifi: qmi ignore invalid mem req type 3
[   10.408217] ath11k c000000.wifi: chip_id 0x0 chip_family 0x0 board_id 0xff soc_id 0xffffffff
[   10.408263] ath11k c000000.wifi: fw_version 0x290104a5 fw_build_timestamp 2023-08-02 20:32 fw_build_id WLAN.HK.2.9.0.1-01890-QCAHKSWPL_SILICONZ-1
[   10.890618] ath11k c000000.wifi: htt event 48 not handled

and all good from there.

So, cal-<bus>-<id>.bin callback only happens if board.bin exists! Really?

7

The BDF and the CAL data are not the same, although they have a similar structure. The CAL data cannot be replaced by the BDF, as the CAL part is unique to each individual device as manufacturer calibrates each device during manufacturing. The BDF is part of Openwrt as stated before, as the BDF is controlling the features and function of a specific device type and is not unique to each individual device.

If you take the CAL file and repackage it as a BDF, that might or might not work. We have seen that the CAL files content for feature control (which is the main purpose of the BDF) can be incorrect, as the vendor usually only pays attention to the part of the file which is interesting for the purpose: the CAL files feature flags section was completely incorrect compared to the vendor BDF. So dont get mislead that the files have a similar structure: these files are not 1:1 replaceable for sure just because it loads.

8

You are confusing caldata with BDF, these are not the same.
By not including the ipq-wifi metapackage you are not including the BDF at all.

You cannot load the BDF from ART as its simply not there, only the caldata which already is extracted from there is loaded

9

the CAL part is unique to each individual device as manufacturer calibrates each device during manufacturing

But of course, and in this post I am claiming that packaged caldata is bad and art caldata is good, hence give it to me.

The BDF is part of Openwrt as stated before, as the BDF is controlling the features and function of a specific device type and is not unique to each individual device.

Where in OpenWRT is BDF for my device, if not in the ipq-wifi-buffalo_wxr-5950ax12?

@robimarko

You are confusing caldata with BDF, these are not the same. By not including the ipq-wifi metapackage you are not including the BDF at all.

OK, please help me clear that confusion m(_ _)m. The ipq-wifi is not a mere metapackage, it actually contains a real board-2.bin data which breaks my 5G wifi. And my assumption is that it is due to wrong caldata.

You cannot load the BDF from ART as its simply not there, only the caldata which already is extracted from there is loaded

My intention here is not to load BDF, just the working caldata.

OK, but where can one see evidence that ipq-wifi package contains BDF and not only caldata for my device? Which part of the blob contains the BDF? What the heck is this BDF thing supposed to contain anyway? Is caldata supposed to be a subset of BDF?

From what my eyes have seen so far on devices and in the code and in hex dumps, the board-2.bin is just one or more instances of board.bin plus some fancy header. Maybe this "BDF" thing exists in some other firmware packages and not on my device? In my case the packaged board-2.bin and stuff stored in art has identical 128K size (if one ignores a small header) and seemingly almost identical structure. In fact, when when looking into hex diffs and entropy, the art data has a few data structures more than what is in board-2.bin. So the evidence I have access to does not support that the claim that board-2.bin has "BDF" and art doesn't. They both do seem to contain caldata, and that is what matters to me; the art caldata works, theipq-wifi packaged caldata doesn't.

Anyways, these blobs are all evil, please help me understand them better, if you can...

10

You are still mixing things up.
board-2.bin is the BDF, not the caldata loaded which gets loaded on the fly via the /etc/hotplug.d/firmware/11-ath11k-caldata script.

In ART you only have caldata stored, the structure used to encode caldata and BDF is almost the same but their content is not the same, its not even similar.

Again, ipq-wifi DOES NOT PROVIDE CALDATA, I dont know what evidence you need

11

You are still mixing things up. board-2.bin is the BDF

In ART you only have caldata stored, the structure used to encode caldata and BDF is almost the same but their content is not the same, its not even similar.

That is not what I am seeing here. I am seeing very similar content. And everything just works. So, if ART is only caldata, and board-2.bin is BDF (with or without caladata), as you guys say, why does this work then:

rm -rf /lib/firmware/ath11k/IPQ8074/hw2.0/*
. /lib/functions/caldata.sh
FIRMWARE=ath11k/IPQ8074/hw2.0/board.bin caldata_extract "0:art" 0x1000 0x20000
FIRMWARE=ath11k/IPQ8074/hw2.0/cal-ahb-c000000.wifi.bin caldata_extract "0:art" 0x1000 0x20000
md5sum /lib/firmware/ath11k/IPQ8074/hw2.0/*
95d3e6413c0610874fb1d4585f72e061  /lib/firmware/ath11k/IPQ8074/hw2.0/board.bin
95d3e6413c0610874fb1d4585f72e061  /lib/firmware/ath11k/IPQ8074/hw2.0/cal-ahb-c000000.wifi.bin
reboot

and voilà

root@OpenWrt:~# dmesg | grep ath11k
[    9.423673] ath11k c000000.wifi: ipq8074 hw2.0
[    9.423707] ath11k c000000.wifi: FW memory mode: 0
[   10.661348] ath11k c000000.wifi: qmi ignore invalid mem req type 3
[   10.668819] ath11k c000000.wifi: chip_id 0x0 chip_family 0x0 board_id 0xff soc_id 0xffffffff
[   10.668868] ath11k c000000.wifi: fw_version 0x290104a5 fw_build_timestamp 2023-08-02 20:32 fw_build_id WLAN.HK.2.9.0.1-01890-QCAHKSWPL_SILICONZ-1
[   11.031884] ath11k c000000.wifi: htt event 48 not handled

root@OpenWrt:~# ip a show dev phy0-ap0
9: phy0-ap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 68:e1:dc:87:48:e0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6ae1:dcff:fe87:48e0/64 scope link 
       valid_lft forever preferred_lft forever

So, my brain may be tired and weary from all the festive goodness, and concerned with all the shite happening in here these days, but screw me if this is not an evidence that "BDF" blob is the very same shite as caldata blob. The very same thing happens if I package board-2.bin from the very same 0:art data with ath11k-bdencoder and remove board.bin.

Someone please tell me that I ain't crazy?

12

That is not evidence as the driver does no verification of the binary you pass, il bet the FW only checks the checksum and that's it.

So, unless the original device support author packaged the caldata as BDF they should not be the same.

13

As Robi stated, it is not the same. If it would be the same, you would not need two different files. The fact it loads and/or it kinda seems it is working does not prove anything. We dealt a lot with BDF and CAL files and I can assure you: they are not the same.

But you know what? Lets focus on your actual issue. So you state that the BDF comes with Openwrt results in poor RF performance on your device, correct? Why dont you try and extract the BDF from the latest official firmware for your device and try with that? Or try to compare the BDF extracted from the original firmware and the one packaged in Openwrt?

I still think that removing the packaged board-2.bin and simply extract both board.bin and cal-ahb-c000000.wifi.bin from ART is much healthier approach. Thoughts?

The BDF is not on the ART partition, only the CAL data. You need to extract the original BDF from the vendor firmware.

14

Yeah, but the hard evidence is that everything just works as expected. If the ath11k firmware is happy with loading the correct data, I'd call it a day. In the ideal world we would have access to the firmware code or documentation and would know better what is correct and what is not, but until that is the case I'd take whatever ain't ultimately broken, as of today. The only thing I can think of what is happening here is that the 0:art for this device simply contains both BDF and caldata in the very same blob, and the firmware knows where in the blog to get each of them, no matter they are technically different files.

IMHO, the fact that both BDF and caldata may come packaged together is kinda convenient; the driver does not complain, the athk11 firmware loads both BDF and caldata, and the world can be a happy place again... I suspect very much that the same thing would work for other devices in the qualcommax target.

But, back to the actual issue; I guess I could reword it as "The ipq-wifi-buffalo_wxr-5950ax12 packaged BDF does not work with ART extracted caldata". In that regard, what do you suggest as a solution? I still think that removing the packaged board-2.bin and simply extract both board.bin and cal-ahb-c000000.wifi.bin from ART is much healthier approach. Thoughts?

15

Not poor performance, 5GHz RX is completely broken. Since BDF already seems to be part of ART in this device, that seems like a waste of time really. I'd rather just use what is in ART, it is proven to work well.

16

It is not. But you can believe whatever you want.

17

Care to share the knowledge about how these things are different? Was there any attempt to understand / document the structure of these blobs? I only know what my eyes can see, if the packaged board-2.bin is truly a BDF, then what is in my device's ART is also a BDF. The bits and bytes do not lie. I'll be happy to share the binaries with you guys...

18

Nobody will share proprietary NDA controlled items so you can extract and understand the actual structures in these files. The fact some of us may (or may not) have access to tools that can do this, or if you know what to search for might able to find these tools (or at least very outdated versions of them) on the internet is not up to us. So the only thing you can do is accept what we say about it.

You have been given instructions about what you can try, feel free to do so, or dont. The ART file is not interesting at all. The original fimrware's bootlog is, and the latest extract of the BDF from the vendor's latest firmware to do comparison with the BDF packaged with Openwrt.

19

So, you are saying they are 100% binary identical?

20

Well that sucks. So not even a hint or tiny bit of shared knowledge on how to distinguish between the blobs? How about simple answer to this question; is it possible that caldata can be a subset within the BDF blob?

Anyway, here is what is on the vendor rootfs:

ls -alh squashfs-root/lib/firmware/IPQ8074/*bin
lrwxrwxrwx 1 root root   15 11月 16  2022 squashfs-root/lib/firmware/IPQ8074/bdwlan.bin -> /tmp/bdwlan.bin
-rw-r--r-- 1 root root 128K 11月 16  2022 squashfs-root/lib/firmware/IPQ8074/bdwlan_rev1.bin
-rwxr-xr-x 1 root root 128K 11月 16  2022 squashfs-root/lib/firmware/IPQ8074/bdwlan_rev2.bin
-rwxr-xr-x 1 root root 128K 11月 16  2022 squashfs-root/lib/firmware/IPQ8074/bdwlan_rev2_sky85772.bin
lrwxrwxrwx 1 root root   24 11月 16  2022 squashfs-root/lib/firmware/IPQ8074/caldata.bin -> /tmp/IPQ8074/caldata.bin

Are these bdwlan* files BDF blobs? Unfortunately, the caldata.bin is a symlink to something on temp, and I would need to revert to stock to check that, which I would like to avoid unless absolutely necessary.