After looking over /lib/netifd/wireless/mac80211.sh, it isn't obvious to me that editing /etc/config/wireless to set options like [BF-ANTENNA-3] would work (at least like I suggested).
Rather than try that, I did
# edit /var/run/hostapd-phy0.conf vht_capab parameter to look like:
r7500v2 # cat /var/run/hostapd-phy0.conf | grep vht_capab
vht_capab=[RXLDPC][SHORT-GI-80][TX-STBC-2BY1][SU-BEAMFORMER][MU-BEAMFORMER][SU-BEAMFORMEE][RX-ANTENNA-PATTERN][TX-ANTENNA-PATTERN][RX-STBC-1][MAX-MPDU-11454][MAX-A-MPDU-LEN-EXP7][BF-ANTENNA-2][SOUNDING-DIMENSION-2][BF-ANTENNA-3][SOUNDING-DIMENSION-3]
r7500v2 # kill -HUP $(pidof hostapd) # reload hostapd config
To see if my hostapd changes were accepted, I used:
r7500v2 # hostapd_cli -i wlan0 status | grep vht_cap
vht_caps_info=338b79b2
i'm pretty sure vht_caps_info is a hex number for the vht_capab bit mask in <path_to_hostapd-wpad-full_src>/hostapd/config_file.c
Between that and the defs in <path_to_hostapd-wpad-full_src>/src/common/ieee802_11_defs.h and a little python to check bits:
In[36]: int(bin(0x339b79b2)[2:]) & 1 << 13
Out[36]: 8192
In[37]: int(bin(0x339b79b2)[2:]) & 1 << 14
Out[37]: 16384
In[38]: int(bin(0x339b79b2)[2:]) & 1 << 15
Out[38]: 32768
I can see that the bits got set. The above python command using the vht_caps_info value before the changes returned "0" for bits 13, 14, and 15 and other bits are consistent with the vht_capab parameters in /var/run/hostapd-phy0.conf.
I think one can compare the hostapd vht_capab value(s) with the VHT capabilities advertised by the ath10k-ct firmware via
r7500v2 # iw phy phy0 info | grep VHT\ Capab
VHT Capabilities (0x339b79b2):
and comparing the bits making up 0x339b79b2 with those advertised by hostapd (0x338b79b2).
I disconnected/connected a few clients after that and... well everything still works but I have yet to see a meaningful difference. So far I've found limited information about setting these bits but there is some hints on the ath10k mailing list. At least I learning a little.
I would not have figured any of this out without your hint:
Thank you.