R7500v2 (ath10k 9980) netperf observations, firmware/board-2.bin files, and other wifi issues

For Developers
2

Regarding the netperf behavior, I'm currently trying to disable "airtime fairness" (based on this suggestion - see also this) on master to see if that has an affect. My first attempt didn't work. It's slow going as the AP is in almost continuous use and I have limited opportunities to flash new images (even a simple reboot can be disruptive). Assuming 19.07.4 does not have airtime fairness enabled, its likely faster to try a test with that...

One other suggestion is to try a non ct ath10k firmware/driver. For this device, I've found that the ct driver firmware work well (stable and as fast as I need) and if problems come up with these, the dev is responsive.

By contrast, the non-ct ath10k firmware used by openwrt for the 9980 has not been updated in 5 years, I don't know it's origin beyond this, and I don't think the non-ct ath10k driver is supported or updated by openwrt at this point. When I've tired the non-ct driver/firmware in the past they cause issues for me. I have yet to try the non-ct ath10k driver again - its on my to-do list.

Regarding ath10k firmware (and board files) for 99x0 devices, there are other options (i.e. the stock netgear r7500v2 images which are still getting updates) but until now I didn't understand how to try those. I'll get into details on this "side show" in my next post.

3

A worked example to create your own 99X0 ath10k "firmware-5.bin" and "board-2.bin" files from "stock Netgear" firmware.

First a few caveats. At the time this post was written, it seems there is no "redistribution licence" for a "board-2.bin" one might create themselves.

As I'm not aware that anything has changed since then and I expect the same is true for a "firmware-5.bin," I will not "redistribute" any such files here or elsewhere. I do not recommend anyone else redistribute any such files they might make for their own use unless an applicable redistribution license for such files exits. (The 99X0 board-2.bin file currently in use by openwrt comes from here - and is possibly covered by this license.)

I consider this example moderately difficult so if you are new to openwrt/linux, consider giving yourself some time to get acclimated before trying this. Also this is not likely to improve your wifi performance - it's "just for fun" I'm using Ubuntu 20.04.1 LTS for this example. That said, the risk that you will irreversibly break something here is small.

Download a recent Netgear image:
wget https://www.downloads.netgear.com/files/GDC/R7500v2/R7500v2-V1.0.3.48.zip

Unzip the firmware and use binwalk to extract the image:

$ unzip R7500v2-V1.0.3.48.zip 
Archive:  R7500v2-V1.0.3.48.zip
  inflating: R7500v2-V1.0.3.48.img   
  inflating: R7500v2-V1.0.3.48_Release_Notes.html  

$ binwalk -e R7500v2-V1.0.3.48.img 

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
128           0x80            uImage header, header size: 64 bytes, header CRC: 0x4197B40F, created: 2020-08-17 07:42:56, image size: 2152600 bytes, Data Address: 0x41508000, Entry Point: 0x41508000, data CRC: 0xC47E45C9, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.4.103"
192           0xC0            Linux kernel ARM boot executable zImage (little-endian)
15124         0x3B14          xz compressed data
15356         0x3BFC          xz compressed data
2228288       0x220040        uImage header, header size: 64 bytes, header CRC: 0xC2A155AD, created: 2020-08-17 07:43:04, image size: 23547904 bytes, Data Address: 0x40908000, Entry Point: 0x40908000, data CRC: 0x69A6711, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: lzma, image name: "Linux-3.4.103"
2228352       0x220080        Squashfs filesystem, little endian, version 4.0, compression:xz, size: 23546438 bytes, 3631 inodes, blocksize: 262144 bytes, created: 2020-08-17 07:43:04

Change to the directory holding the ath10k firmware and board files:

$ cd _R7500v2-V1.0.3.48.img.extracted/squashfs-root/lib/firmware/AR900B/hw.2/
$ ls -al
total 1228
drwxr-xr-x 2 ul ul   4096 Aug 17 03:39 .
drwxr-xr-x 3 ul ul   4096 Aug 17 03:39 ..
-rw-r--r-- 1 ul ul 355234 Oct 20  2016 athwlan.bin
-rw-r--r-- 1 ul ul 106493 Oct 20  2016 athwlan.codeswap.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boarddata_0.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boarddata_1.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS238_5GMipiHigh_v2_004.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS238_5GMipiHigh_v2_CTL.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS238_5GMipiMed_v2_003.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS239_5G_3x3_v2_001.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boardData_AR900B_CUS239_5G_v2_001.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boardData_AR900B_CUS239_5G_v2_CE_001.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS239_5G_v2_ch144.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS239_5G_v2_CTL.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boardData_AR900B_CUS239_5G_v2_MAX_001.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boardData_AR900B_CUS239_5G_v2_NEW_FCC_001.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS239_negative_pwr_offset_5G_v2_007.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS240_2GMipiHigh_v2_006.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS240_2GMipiHigh_v2_CTL.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS240_2GMipiMed_v2_005.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS260_2G_3x3_v2_002.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boardData_AR900B_CUS260_2G_v2_002.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS260_2G_v2_CTL.bin
-rwxr-xr-x 1 ul ul  12064 Aug 17 03:25 boardData_AR900B_CUS260_2G_v2_MAX_002.bin
-rwxr-xr-x 1 ul ul  12064 Oct 20  2016 boardData_AR900B_CUS260_negative_pwr_offset_2G_v2_008.bin
-rwxr-xr-x 1 ul ul    374 Oct 20  2016 .filenames
-rw-r--r-- 1 ul ul  11065 Oct 20  2016 otp.bin
-rw-r--r-- 1 ul ul 303932 Oct 20  2016 utf.bin
-rw-r--r-- 1 ul ul 100417 Oct 20  2016 utf.codeswap.bin
-rw-r--r-- 1 ul ul  98701 Oct 20  2016 waltest.codeswap.bin

To create a "firmware-x.bin" and/or a "board-2.bin" container file from these files, you will need the ath10k-fwencoder and ath10k-bdencoder python scripts from the qca-swiss-army-knife.

You will also need to know which of the files above to use. In this case, reading the output from ath10k-fwencoder --help and ath10k-bdencoder --help is informative. From the ath10k-fwencoder script help output, it looks like I need the athwlan.bin, athwlan.codeswap.bin, and the otp.bin files to create a functional firmware. Also really helpful and fun is the ath10k-fwencoder --dump-cmdline /path/to/ct-firmware-5.bin (what exactly is in that ct firmware anyway? Could I mix and match otp.bin files and still have a functional firmware? Probalby not...).

Note the 2016 date of the athwlan*.bin and opt.bin files above... newer than 5 years old but not a great improvment. It looks like these files stoped changing after Netgear's R7500v2-V1.0.3.16.zip firmware. Note, the crc32 checksum reported by ath10k-fwencoder for the individual *.bin files are not the same "inside" a *.bin file vs extracted. Extract the files from a firmware-x.bin file and use md5sum on each extracted file if you want to check for changes (bdencoder will give you consistent md5sum checksums regardless of extraction).

Create a "firmware-5.bin" container:

$ ath10k-fwencoder --create \
--firmware-version=12.34.5-6 \
--features=no-p2p \
--firmware=athwlan.bin \
--otp=otp.bin \
--set-wmi-op-version=10.4 \
--set-htt-op-version=10.4 \
--firmware-codeswap=athwlan.codeswap.bin
firmware-4.bin created: 472904 B
$ mv firmware-4.bin firmware-5.bin

So now what about those board files? Looking around at firmware from dd-wrt, openwrt, notes from ct firmware www site, and most importantly dmesg output, my guess is that only boardData_AR900B_CUS239_5G_v2_001.bin is needed for the 5 GHz band and boardData_AR900B_CUS260_2G_v2_002.bin is needed for the 2 GHz band. Using the openwrt board-2.bin file to create a "board.json" via
ath10k-bdencoder --extract board-2.bin and editing the resulting *.json file to look like:

$ cat board-2.json
[
    {
        "data": "boardData_AR900B_CUS239_5G_v2_001.bin", 
        "names": [
            "bus=pci,bmi-chip-id=1,bmi-board-id=1"
        ]
    }, 
    {
        "data": "boardData_AR900B_CUS260_2G_v2_002.bin", 
        "names": [
            "bus=pci,bmi-chip-id=1,bmi-board-id=2"
        ]
    }
]

I made guesses about the bmi-chip-id, bmi-board-id, and which wifi band corresponds to which board file from dmesg output on my r7500v2 (left as an exercise for the reader). Note many of the Netgear board files where updated August 17, 2020...

$ ath10k-bdencoder --create board-2.json 
board binary file 'board-2.bin' is created
$ ath10k-bdencoder --info board-2.bin
FileSize: 24268
FileCRC32: cb100cc3
FileMD5: b05b549a62e246b70f54354c82ddca53
BoardNames[0]: 'bus=pci,bmi-chip-id=1,bmi-board-id=1'
BoardLength[0]: 12064
BoardCRC32[0]: a0a1ad69
BoardMD5[0]: e8b77aec3f913f592b44360d5545e032
BoardNames[1]: 'bus=pci,bmi-chip-id=1,bmi-board-id=2'
BoardLength[1]: 12064
BoardCRC32[1]: aa572ad1
BoardMD5[1]: f495c23de5be19a81aab70ecb757ebd8

So what about those other board files? Well that could be interesting. I can only speculate but the file names might be revealing. One guess is that the propietary ath10k drivers can load a board file based on what they think is needed... It could be fun to experiment here (i.e. want to tune for 5 GHz 3x3 clients?).

BTW don't mess with your "pre-calibration" data (not otherwise mentioned in this post). This is likely specific to your device and if you lose it, you probably will permanently lose wifi. Nothing in this example will change the pre-calibration data that is used in combination with the board files.

At this point, I copied my DIY firmware-5.bin and board-2.bin files to my router and tried them out. First I tried the board-2.bin file with the ath10k-ct driver and ct firmware on a snapshot image:

r7500v2 # cp ~/tmp/board-2.bin /lib/firmware/ath10k/QCA99X0/hw2.0/board-2.bin
r7500v2 # reboot

After a reboot, check your dmesg to see that the new board-2.bin loaded ok and wifi functions... mine did. Note the original board-2.bin file can be restored via
r7500v2 # cp /rom/lib/firmware/ath10k/QCA99X0/hw2.0/board-2.bin /lib/firmware/ath10k/QCA99X0/hw2.0/board-2.bin

I didn't do extensive testing, there might have been a slight improvement but nothing likley to make you think it was worth the effort above. Your doing this for the fun of it, right? And maybe one of those other board files will make a difference...

Next I tried the DIY firmware-5.bin file (in combination with the ath10k-ct driver) with the DIY board-2.bin file:

r7500v2 # cp ~/tmp/firmware-5.bin /lib/firmware/ath10k/QCA99X0/hw2.0/ct-firmware-5.bin
r7500v2 # reboot

Again check your dmesg to see that your DIY firmware got loaded. Note the original ct-firmware-5.bin file can be restored via
r7500v2 # cp /rom/lib/firmware/ath10k/QCA99X0/hw2.0/ct-firmware-5.bin /lib/firmware/ath10k/QCA99X0/hw2.0/ct-firmware-5.bin

I ran with this DIY combo for about 12 hrs. Nothing crashed and wifi did function - just not nearly as well as the original ct-firmware. Throughput dropped on both bands by about 2-3x and I still observed the same netperf behavior for which I started this thread. Perhaps using such a DIY firmware will help someone else or using the non-ct ath10k driver with a DIY firmware will function better. Regardless, it looks like I did this for the fun of it.

EDIT: I'll likely be making more edits to this post - I just need to be careful about what button I push.

3 Likes
4

Would be great to create a wiki page since these are a lots of info for these mistery bin...

Correct me if i'm wrong but from your speculation, the proprietary firmware should be able to load different boardData based on some settings? Why for example we use the 001 instead of others? (why use the pain 001 instead of MAX?)

5

I'm not certain at this point if it would be the firmware (specifically something in otp.bin?), driver, or both needed for changing the board files on the fly. There are few clues in the ath10k mailing list - I'd need to spend more time reading there.

As to why one board file is used over the other, I don't have an answer and I get the impression others (outside of industry insiders) know much either. Experimentation is always an option. As I'm not the only one to try this, I expect there may not be much benefit.

1 Like
6

wonder if this changes if i use a unlocked reg domain... Since there is a negative pwr offset i wonder if this actually unlock the driver to go beyond the 20 db limit

7

I haven't looked into that. I'd avoid trying to bypass a regulatory limit as that may not be legal (or considerate of other nearby wifi users).

There are the utf*.bin files... googling it seems these can be used to create a "testing" firmware-x.bin file (possible with the waltest.codeswap.bin). I haven't tried those either, but perhaps some of the board files are for testing.

I do recall a recent thread here in which it was observed that the stock r7800 firmware seems better able to cope with wifi interference on the 2.4 GHz band.

Perhaps some of the other board files are used to help with that.

8

Could be that they use the dfs data and chose the right board based on the noise near the radio...

No noise you can pump the wifi... very noisy, you reduce the power since a far wifi wouldn't work anyway due to interference... That's a lot of logic in the driver... I'm impressed if it does work like that...

1 Like
9

If you find some discussion about using different board on ath10k mailing list pls link here... i'm very curious if they even tried to push this kind of thing

10

I saw at least one casual comment about "proprietary" ath10k drivers but I cant find the url ATM.

you don't need that I think

$ cd _R7500v2-V1.0.3.48.img.extracted/squashfs-root/lib/modules/3.4.103
$ strings -f *.ko | grep board
...
umac.ko: boardData_AR900B_CUS239_5G_v2_MAX_001.bin
...

a few other refs to umac.ko and "boardData*.bin" files in that output.

A quick google got me to this (from 2008) but you get the idea...

[  274.247192] umac: module license 'Proprietary' taints kernel.

some other umac.ko? or the same one, idk.

silly of me...

$ strings umac.ko | grep license
license=Dual BSD/GPL

so it should be out there - I'll look more later.

For reference:

$ strings -f *.ko | grep Proprietary
adf.ko: license=Proprietary
asf.ko: license=Proprietary
ath_dev.ko: license=Proprietary
ath_dfs.ko: license=Proprietary
ath_hal.ko: license=Proprietary
ath_rate_atheros.ko: license=Proprietary
ath_spectral.ko: license=Proprietary
clbr.ko: license=Proprietary
hst_tx99.ko: license=Proprietary
NetUSB.ko: license=Proprietary
11

so it's umac that decide what to load... Also it should be GPL but i can't really find any source code... Should be in the GPL tar of r7500v2

Ok i have the source of umac from the qsdk 10

Interesting it looks like there are tools to parse the tx power from a boardData

12

well maybe... (or likely more annoying, yes and no) I suspect some of that is in the drivers and firmware (otp) - at least based on the board-2.bin loading functanlity (which is discussed in the ath10k mailing list). I don't yet know what this is for... the utf*.bin stuff? What I'm looking for i.e. changing board files on the fly? Something else... actually it be nice to not find what I'm looking for. If it turns out to be an interesting idea, then I think its more worthwhile to work on it.

Could be GPL but not pushed anywhere obvious, or the license string could be an error (and it is "Proprietary" like what i found from 2008).

the umac module gets loaded from lib/wifi/qcawifi.sh* and looks "test" related.

13

the package where these boardfile are defined is the athtestcmd

1 Like
15

it looks like the ath10k-ct driver possibly supports using different calibration files through "calname" in its fwcfg api... I have not tried that yet tho

EDIT this looks related to the "pre-cal" files and not the board data files. Might still be useful for utilizing backups of pre-cal files in the event the partition with the pre-cal data (used to generate the pre-cal filee) became corrupted.

there is a "bname" key which does look related to the board files tho

16

Yep looking at the code

if (strcasecmp(filename, "calname") == 0) {
			sz = sizeof(ar->fwcfg.calname);
			strncpy(ar->fwcfg.calname, val, sz);
			ar->fwcfg.calname[sz - 1] = 0; /* ensure null term */
		}
		else if (strcasecmp(filename, "bname") == 0) {
			sz = sizeof(ar->fwcfg.bname);
			strncpy(ar->fwcfg.bname, val, sz);
			ar->fwcfg.bname[sz - 1] = 0; /* ensure null term */
		}
		else if (strcasecmp(filename, "bname_ext") == 0) {
			sz = sizeof(ar->fwcfg.bname_ext);
			strncpy(ar->fwcfg.bname_ext, val, sz);
			ar->fwcfg.bname_ext[sz - 1] = 0; /* ensure null term */
		}
		else if (strcasecmp(filename, "fwname") == 0) {
			sz = sizeof(ar->fwcfg.fwname);
			strncpy(ar->fwcfg.fwname, val, sz);
			ar->fwcfg.fwname[sz - 1] = 0; /* ensure null term */
		}
		else if (strcasecmp(filename, "fwver") == 0) {
			if (kstrtol(val, 0, &t) == 0) {
				ar->fwcfg.fwver = t;
				ar->fwcfg.flags |= ATH10K_FWCFG_FWVER;
			}
		}
		else if (strcasecmp(filename, "vdevs") == 0) {
			if (kstrtol(val, 0, &t) == 0) {
				ar->fwcfg.vdevs = t;
				ar->fwcfg.flags |= ATH10K_FWCFG_VDEVS;
			}
		}

And the function

ath10k_core_fetch_board_data_api_1

(code from ath10k-ct)

17

for ath10k-ct, it looks like the board data is "downloaded" to the SOC via the bootloader messaging interface (bmi). If that's the case, I suspect the concept I'm thinking about likely isn't possible or at the least a misconception on my part.

It was hoping it might be possible to have multiple calibrations in use, possibly selected by station or some other criteria. Calibration is a somewhat vague term. I tend to think about calibration from the perspective of analytical chemistry (for example, an equation relating light absorbance - the "signal" - to concentration of a chemical compound - the data).

It's possible calibration here might mean something that aides in conditioning the wifi "signal" in a more direct sense like something that is adjusting the electronics hardware to obtain a "better" signal rather than translating it.

If the "calibration data" is downloaded to a SOC (possibly limited by constraints such a memory size, proprietary firmware, etc) it would be difficult (or not physical) to execute my concept even if you had access to the firmware code as candella does.

18

Regarding my netperf observations: running a 19.07.4 image (I believe this does not have airtime fairness capability) does not change the results.

If I could invent a phrase, I would describe my issue as "throughput jitter." Wifi throughput (regardless of the magnitude) changes rapidly. For example, 180 Mbps to 90 mbps back to 180 mbps over the time span of a few seconds (as reported by netperf). Add a second netperf session from a "location challenged client" and throughput on that client seems to completely stop (but not all connections are stoped as I'm ssh'ed into the client looking at irtt results which go on). Thinking maybe I should try the "dma_burst=0" fwcfg item for the ct firmware.

Wifi "quality" in general remains poor - testing via dsl reports from a client with a direct view from the AP less than 3 m away can give dismal results (C's and D's throughput erratically changing), while repeating the test over the wire look normal - even great (A's and expected throughput given light use from other users). My spouse a few hours ago asked our kids to stay off the "internet" so her conference call wouldn't be interrupted.

Wifi interference is always a possibility and seems more likely at this point - my neighbors are not close (the closest AP is more than 30 m away and 2.4 and 5 GHz for the most part run on different channels than I use). The closest AP on 5 GHz on the same channel 36 is ~ 100 m away. It does look that neighbor started using a mesh system recently.

Before going back to master, I will try the non ct driver and firmware and after that, I'll look at wpad variants...

It's also possible that AP and client location is the real culprit here - back when "wifi was good" (late 2019 through perhaps early May this year) clients were generally closer. Now clients are farther from the AP (which did not move) more polarized (located in opposite directions relative to the AP). In general, the signal does have to pass through more walls.

I just put the AP in a less obstructed position but I can still reproduce the netperf behavior with a more "location challenged" client.

19

Sorry for the stupid question... Does the pre-cal bin is device specific? I mean is created when the router is created or is generic and equal for every router of that type..
Example:
Both we have a r7500v2. Is my pre-cal file different than yours?

20

yes and I think its a good question. Obviously, I'm just expressing the impression I get from the reading I've done - I've taken my own advice and avoided messing with these.

Apparently the data is stored on one of the partitions (ART?) and the pre-calibrations files (/lib/firmware/ath10k/pre-cal-pci-000[01]:01:00.0.bin) are generated from that. I'd have to go back to find a reference and more specifics.

21

The pre-cal bin is just copied with a simple dd (in fact i'm pushing a patch to make the driver read directly from it instead of extract and read from file, the main idea is clean the hotplug.d script)

I had to ask this since there is an option to provide the cal data directly from the dts as an array of char... This wouldn't make sense if the cal data is different from one device to another...

22

I recall a mailing list comment about dd not always working (I'd share it but I've got a few hundred links in my browser history on this topic I'd have to sift through)... but if your just replacing a dd cmd in a hotplug script you should be OK - I just mentioned it in the event there are "edge" cases (apparently it can be read from EEPROM for some devices nvmnd, EEPROM is an alias for the ART partition).