Add rtl8852au driver to kernel module but ignored when building

I want to add a new kernel module to support my usb wifi nic.
Here is the driver source code in github: https://github.com/lwfinger/rtl8852au.
It was successfully built in ubuntu/debian with linux kernel headers.

Here is my steps to adding it.
Get the source code from github and make a new Makefile.

git clone https://github.com/openwrt/openwrt.git -b openwrt-22.03
cd openwrt
./scripts/feeds update -a
./scripts/feeds install -a
mkdir package/kernel/rtl8852au
cp package/kernel/rtl8812au-ct/Makefile package/kernel/rtl8852au/Makefile

Then modify the Makefile. The following is what it looks like after modifications.

include $(TOPDIR)/rules.mk

PKG_NAME:=rtl8852au
PKG_RELEASE=1

PKG_LICENSE:=GPLv2
PKG_LICENSE_FILES:=

PKG_SOURCE_URL:=https://github.com/lwfinger/rtl8852au.git
PKG_MIRROR_HASH:=405f6e2b60027fa47fb44af724e8b49107a7f8622cd093d265a1ab1b527e2f19
PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2023-02-25
PKG_SOURCE_VERSION:=cb319bee1375ccf37eabe71dad6876a2894bb155

#PKG_MAINTAINER:=Ben Greear <greearb@candelatech.com>
PKG_BUILD_PARALLEL:=0
#PKG_EXTMOD_SUBDIRS:=rtl8852au-ct

STAMP_CONFIGURED_DEPENDS := $(STAGING_DIR)/usr/include/mac80211-backport/backport/autoconf.h

include $(INCLUDE_DIR)/kernel.mk
include $(INCLUDE_DIR)/package.mk

define KernelPackage/rtl8852au
  SUBMENU:=Wireless Drivers
  TITLE:=Driver for Realtek 8852 AU devices
  DEPENDS:=+kmod-cfg80211 +kmod-usb-core +@DRIVER_11N_SUPPORT +@DRIVER_11AC_SUPPORT
  FILES:=\
	  ${LINUX_DIR}/drivers/net/wireless/realtek/rtw89/8852.ko
  AUTOLOAD:=$(call AutoProbe,rtl8852au)
  PROVIDES:=kmod-rtl8852au
endef

NOSTDINC_FLAGS := \
	$(KERNEL_NOSTDINC_FLAGS) \
	-I$(PKG_BUILD_DIR) \
	-I$(PKG_BUILD_DIR)/include \
	-I$(STAGING_DIR)/usr/include/mac80211-backport \
	-I$(STAGING_DIR)/usr/include/mac80211-backport/uapi \
	-I$(STAGING_DIR)/usr/include/mac80211 \
	-I$(STAGING_DIR)/usr/include/mac80211/uapi \
	-include backport/backport.h

NOSTDINC_FLAGS+=-DCONFIG_IOCTL_CFG80211 -DRTW_USE_CFG80211_STA_EVENT -DBUILD_OPENWRT

define Build/Compile
	+$(MAKE) $(PKG_JOBS) -C "$(LINUX_DIR)" \
		$(KERNEL_MAKE_FLAGS) \
		M="$(PKG_BUILD_DIR)" \
		NOSTDINC_FLAGS="$(NOSTDINC_FLAGS)" \
		modules
endef

$(eval $(call KernelPackage,rtl8852au))

Then I select it in menuconfig config.

But it failed when try to build it.

make[3]: Entering directory '/home/debian/openwrt/package/kernel/rtl8852au'
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.prepared_7f35634bf13af35f486399935b2990ee_6664517399ebbbc92a37c5bb081b5c53_check
. /home/debian/openwrt/include/shell.sh; xzcat /home/debian/openwrt/dl/rtl8852au-2023-02-25-cb319bee.tar.xz | tar -C /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.. -xf -
[ ! -d ./src/ ] || cp -fpR ./src/. /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.prepared_7f35634bf13af35f486399935b2990ee_6664517399ebbbc92a37c5bb081b5c53
rm -f /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.configured_*
rm -f /home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/stamp/.rtl8852au_installed
(cd /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/./; if [ -x ./configure ]; then find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.guess | xargs -r chmod u+w; find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.guess | xargs -r -n1 cp --remove-destination /home/debian/openwrt/scripts/config.guess; find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.sub | xargs -r chmod u+w; find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.sub | xargs -r -n1 cp --remove-destination /home/debian/openwrt/scripts/config.sub; AR="aarch64-openwrt-linux-musl-gcc-ar" AS="aarch64-openwrt-linux-musl-gcc -c -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee=rtl8852au-2023-02-25-cb319bee -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=aarch64-openwrt-linux-musl-ld NM="aarch64-openwrt-linux-musl-gcc-nm" CC="aarch64-openwrt-linux-musl-gcc" GCC="aarch64-openwrt-linux-musl-gcc" CXX="aarch64-openwrt-linux-musl-g++" RANLIB="aarch64-openwrt-linux-musl-gcc-ranlib" STRIP=aarch64-openwrt-linux-musl-strip OBJCOPY=aarch64-openwrt-linux-musl-objcopy OBJDUMP=aarch64-openwrt-linux-musl-objdump SIZE=aarch64-openwrt-linux-musl-size CFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee=rtl8852au-2023-02-25-cb319bee -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee=rtl8852au-2023-02-25-cb319bee -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/usr/include -I/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/include/fortify -I/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/include " LDFLAGS="-L/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/usr/lib -L/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/lib -znow -zrelro "   ./configure --target=aarch64-openwrt-linux --host=aarch64-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  ; fi; )
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.configured_68b329da9893e34099c7d8ad5cb9c940
rm -f /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.built
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.built_check
make -j1 -C "/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176" KCFLAGS="-fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl=target-aarch64_cortex-a72_musl -fno-caller-saves " HOSTCFLAGS="-O2 -I/home/debian/openwrt/staging_dir/host/include -I/home/debian/openwrt/staging_dir/hostpkg/include -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/host/include -Wall -Wmissing-prototypes -Wstrict-prototypes" CROSS_COMPILE="aarch64-openwrt-linux-musl-" ARCH="arm64" KBUILD_HAVE_NLS=no KBUILD_BUILD_USER="" KBUILD_BUILD_HOST="" KBUILD_BUILD_TIMESTAMP="Mon Mar 27 16:59:53 2023" KBUILD_BUILD_VERSION="0" HOST_LOADLIBES="-L/home/debian/openwrt/staging_dir/host/lib" KBUILD_HOSTLDLIBS="-L/home/debian/openwrt/staging_dir/host/lib" CONFIG_SHELL="bash" V=1  cmd_syscalls= KBUILD_EXTRA_SYMBOLS="/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers/mac80211.symvers" KERNELRELEASE=5.10.176 M="/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee" NOSTDINC_FLAGS="-nostdinc  -isystem /home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/lib/gcc/aarch64-openwrt-linux-musl/11.2.0/include -I/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee -I/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/include -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211-backport -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211-backport/uapi -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211 -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211/uapi -include backport/backport.h -DCONFIG_IOCTL_CFG80211 -DRTW_USE_CFG80211_STA_EVENT -DBUILD_OPENWRT" modules
make[4]: Entering directory '/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176'
test -e include/generated/autoconf.h -a -e include/config/auto.conf || (		\
echo >&2;							\
echo >&2 "  ERROR: Kernel configuration is invalid.";		\
echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are missing.";\
echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix it.";	\
echo >&2 ;							\
/bin/false)
make -f ./scripts/Makefile.build obj=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee \
single-build= \
need-builtin=1 need-modorder=1
  {  :; } | awk '!x[$0]++' - > /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/modules.order
make -f ./scripts/Makefile.modpost
  sed 's/ko$/o/' /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/modules.order | scripts/mod/modpost     -o /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers -e -i Module.symvers -i /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers/mac80211.symvers   -T -
make -f ./scripts/Makefile.modfinal
make[4]: Leaving directory '/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176'
for subdir in .; do realdir=$(readlink -f /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee); grep -F /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/$subdir/Module.symvers >> /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers.tmp; [ "/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee" = "$realdir" ] || grep -F $realdir /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/$subdir/Module.symvers >> /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers.tmp; done; sort -u /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers.tmp > /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers; mkdir -p /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers; mv /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers/rtl8852au.symvers
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.built
rm -rf /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au.installed /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au
mkdir -p /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au
ERROR: module '/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176/drivers/net/wireless/realtek/rtw89/8852.ko' is missing.
make[3]: *** [Makefile:56: /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au.installed] Error 1
make[3]: Leaving directory '/home/debian/openwrt/package/kernel/rtl8852au'
time: package/kernel/rtl8852au/compile#0.88#0.44#1.19
    ERROR: package/kernel/rtl8852au failed to build.
make[2]: *** [package/Makefile:116: package/kernel/rtl8852au/compile] Error 1
make[2]: Leaving directory '/home/debian/openwrt'
make[1]: *** [package/Makefile:110: /home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/debian/openwrt'
make: *** [/home/debian/openwrt/include/toplevel.mk:230: world] Error 2

Why the compiler bypass it? How to fix this.

if you want to save time, and avoid pulling your hair out, get a better ( = non RTL) usb wifi device.

Buy another one which is supported by openwrt is definitely the best choice.
But number of devices supported by openwrt is small.
I know that mt7921u with features WIFI6 80MHz ap mode and supported by openwrt. But I see lots of negative comments of its stability.
Could you recommend some other usb wifi nic which supports WIFI6 80MHz ap mode and supported by openwrt?

these should work https://github.com/morrownr/USB-WiFi

if your plan for the RTL was an AP, it probably would have been a dead end anyway.

But I see lots of negative comments of its stability.

That is not my experience. I use a mt7921au based adapter on 22.03.3. Maybe the comments are old and the driver has been improved since the comments were made...or maybe those making the comments were doing something that I am not. The mt7921u driver gets an lot of ongoing work to add features and improve function.

Hi. I bought a cf-953ax(mt7921au) yesterday.

I compiled the firmware of mt7921u mt7921e successfully and install it on my rpi 4B.

The WIFI5 80MHz ap works fine. But when it comes to WIFI6, the AP only works in short distance, about 4~5m. If I go farther, it disconnects.
I notice that the rx rate of AP is low when I far away 4m from AP.

Someone else also have encoutered this. https://github.com/openwrt/mt76/issues/713
Have you encountered such a situation?

Did you set a country for the radio ?

Sure, here is the config of the ap.

wireless.radio1=wifi-device
wireless.radio1.type='mac80211'
wireless.radio1.path='scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3'
wireless.radio1.band='5g'
wireless.radio1.channel='153'
wireless.radio1.txpower='30'
wireless.radio1.cell_density='0'
wireless.radio1.htmode='HE80'
wireless.radio1.country='CN'

In WIFI6(HE 80), the problem occurs.
In WIFI5(VHT 80), it works fine.

I also tried to change the country to US, but nothing seems to help.

I built openwrt from the master branch instead of the 22.03 and flash it to the rpi4b.
After that WIFI6 ap works fine with mt7921au.
So, buy another wifi nic is the best choice.

I am working on adding this out of tree driver to OpenWRT.
It builds and loads, but upon device recognition, the kernel crashes. Right now I am rebuilding it all with more debug info, as this is all I got: https://autkin.net/tmp/rtl8852au/boot-and-crash/2023-12-07_23:39:08.dmesg.txt

To build it my way, prepend this to your feeds.conf.default:
src-git rtl8852au_feed https://github.com/andrey-utkin/rtl8852au-feed

you should make a patch to solve that problem

OK, now with function names: (kallsyms had to be enabled):

https://autkin.net/tmp/rtl8852au/boot-and-crash/2023-12-09_14:38:28.dmesg.txt

I am trying to build this out-of-tree module the naive way, linkage fails but I have some hope to build it and see how the build runs comparing to the in-openwrt build.

12:16:54 Sun 10 Dec openwrt-dev@homeserver /mnt/raid0/openwrt-dev/rtl8852au
 $ make V=1 CROSS_COMPILE="aarch64-openwrt-linux-musl-" ARCH="arm64" KSRC=/mnt/raid0/openwrt-dev/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-6.1.64  modules -j13
#rm -f .symvers.8852au
make ARCH=arm64 CROSS_COMPILE=aarch64-openwrt-linux-musl- -C /mnt/raid0/openwrt-dev/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-6.1.64 M=/mnt/raid0/openwrt-dev/rtl8852au  modules
make -f ./scripts/Makefile.build obj=/mnt/raid0/openwrt-dev/rtl8852au need-builtin=1 need-modorder=1
sh ./scripts/modules-check.sh /mnt/raid0/openwrt-dev/rtl8852au/modules.order
make -f ./scripts/Makefile.modpost
   sed 's/ko$/o/'  /mnt/raid0/openwrt-dev/rtl8852au/modules.order | scripts/mod/modpost      -o /mnt/raid0/openwrt-dev/rtl8852au/Module.symvers -n -e -i Module.symvers -T -
ERROR: modpost: "__cfg80211_alloc_event_skb" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "wiphy_register" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_remain_on_channel_expired" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_vendor_cmd_reply" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_del_sta_sinfo" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "wiphy_unregister" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_ch_switch_notify" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_ch_switch_started_notify" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "ieee80211_get_channel_khz" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!                                                                                                                  ERROR: modpost: "cfg80211_ready_on_channel" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!                                                                                                                  WARNING: modpost: suppressed 19 unresolved symbol warnings because there were too many)
make[2]: *** [scripts/Makefile.modpost:126: /mnt/raid0/openwrt-dev/rtl8852au/Module.symvers] Error 1
make[1]: *** [Makefile:1966: modpost] Error 2
make: *** [Makefile:639: modules] Error 2
[ERR=2]
12:17:25 Sun 10 Dec openwrt-dev@homeserver /mnt/raid0/openwrt-dev/rtl8852au
 $ make V=1 CROSS_COMPILE="aarch64-openwrt-linux-musl-" ARCH="arm64" KSRC=/mnt/raid0/openwrt-dev/openwrt/build_dir/target-aarch64_cortex-a72_musl/openwrt-sdk-SNAPSHOT-bcm27xx-bcm2711_gcc-12.3.0_musl.Linux-x86_64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-6.1.64/ modules -j13
#rm -f .symvers.8852au
make ARCH=arm64 CROSS_COMPILE=aarch64-openwrt-linux-musl- -C /mnt/raid0/openwrt-dev/openwrt/build_dir/target-aarch64_cortex-a72_musl/openwrt-sdk-SNAPSHOT-bcm27xx-bcm2711_gcc-12.3.0_musl.Linux-x86_64/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-6.1.64/ M=/mnt/raid0/openwrt-dev/rtl8852au  modules
make -f ./scripts/Makefile.build obj=/mnt/raid0/openwrt-dev/rtl8852au need-builtin=1 need-modorder=1
sh ./scripts/modules-check.sh /mnt/raid0/openwrt-dev/rtl8852au/modules.order
make -f ./scripts/Makefile.modpost
   sed 's/ko$/o/'  /mnt/raid0/openwrt-dev/rtl8852au/modules.order | scripts/mod/modpost      -o /mnt/raid0/openwrt-dev/rtl8852au/Module.symvers -n -e -i Module.symvers -T -
ERROR: modpost: "__cfg80211_alloc_event_skb" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "wiphy_register" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_remain_on_channel_expired" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_vendor_cmd_reply" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_del_sta_sinfo" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "wiphy_unregister" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_ch_switch_notify" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_ch_switch_started_notify" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "ieee80211_get_channel_khz" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
ERROR: modpost: "cfg80211_ready_on_channel" [/mnt/raid0/openwrt-dev/rtl8852au/8852au.ko] undefined!
WARNING: modpost: suppressed 19 unresolved symbol warnings because there were too many)
make[2]: *** [scripts/Makefile.modpost:126: /mnt/raid0/openwrt-dev/rtl8852au/Module.symvers] Error 1
make[1]: *** [Makefile:1966: modpost] Error 2
make: *** [Makefile:639: modules] Error 2
[ERR=2]
12:17:59 Sun 10 Dec openwrt-dev@homeserver /mnt/raid0/openwrt-dev/rtl8852au
 $ env | grep STAGING_DIR
STAGING_DIR=/mnt/raid0/openwrt-dev/openwrt/staging_dir/

Got the module to build and link out of tree, by sticking KBUILD_EXTRA_SYMBOLS variable into that make invocation, the value was taken from the logs of openwrt build system. The value is a space separated list of *.symvers files, most notably mac80211.symvers.
Still got exactly the same runtime behaviour as before - kernel warnings, probably an oops (my netconsole setup broke so I didn't see it) and a reboot.

lwfinger (the maintainer of the driver repo) is not interested in dealing with backported kernel and seems to have a suspicion of backporting issue https://github.com/lwfinger/rtl8852au/issues/71#issuecomment-1849026150

I have given up on building rtl8852au driver.
Now I am working with mt7921au and it is pretty well.

I've retried building openwrt from master with the same custom package code. It fails similarly, and I am convinced it won't happen without backports being in the picture. According to forums, one "can't" just switch away from backport wifi support modules and use just the kernel, unless you are willing to do a lot of extra packaging and patching work. I am not going to do that, so I give up on this project for now.
The code I have shared theoretically might start working at some point in future, so don't lose hope!

@autkin hello greetings.

Any progress so far? I think the kernel patches wee need are not that big. Are you interessted in creating a Repo?

Greetings,
H.