Add rtl8852au driver to kernel module but ignored when building

I want to add a new kernel module to support my usb wifi nic.
Here is the driver source code in github: https://github.com/lwfinger/rtl8852au.
It was successfully built in ubuntu/debian with linux kernel headers.

Here is my steps to adding it.
Get the source code from github and make a new Makefile.

git clone https://github.com/openwrt/openwrt.git -b openwrt-22.03
cd openwrt
./scripts/feeds update -a
./scripts/feeds install -a
mkdir package/kernel/rtl8852au
cp package/kernel/rtl8812au-ct/Makefile package/kernel/rtl8852au/Makefile

Then modify the Makefile. The following is what it looks like after modifications.

include $(TOPDIR)/rules.mk

PKG_NAME:=rtl8852au
PKG_RELEASE=1

PKG_LICENSE:=GPLv2
PKG_LICENSE_FILES:=

PKG_SOURCE_URL:=https://github.com/lwfinger/rtl8852au.git
PKG_MIRROR_HASH:=405f6e2b60027fa47fb44af724e8b49107a7f8622cd093d265a1ab1b527e2f19
PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2023-02-25
PKG_SOURCE_VERSION:=cb319bee1375ccf37eabe71dad6876a2894bb155

#PKG_MAINTAINER:=Ben Greear <greearb@candelatech.com>
PKG_BUILD_PARALLEL:=0
#PKG_EXTMOD_SUBDIRS:=rtl8852au-ct

STAMP_CONFIGURED_DEPENDS := $(STAGING_DIR)/usr/include/mac80211-backport/backport/autoconf.h

include $(INCLUDE_DIR)/kernel.mk
include $(INCLUDE_DIR)/package.mk

define KernelPackage/rtl8852au
  SUBMENU:=Wireless Drivers
  TITLE:=Driver for Realtek 8852 AU devices
  DEPENDS:=+kmod-cfg80211 +kmod-usb-core +@DRIVER_11N_SUPPORT +@DRIVER_11AC_SUPPORT
  FILES:=\
	  ${LINUX_DIR}/drivers/net/wireless/realtek/rtw89/8852.ko
  AUTOLOAD:=$(call AutoProbe,rtl8852au)
  PROVIDES:=kmod-rtl8852au
endef

NOSTDINC_FLAGS := \
	$(KERNEL_NOSTDINC_FLAGS) \
	-I$(PKG_BUILD_DIR) \
	-I$(PKG_BUILD_DIR)/include \
	-I$(STAGING_DIR)/usr/include/mac80211-backport \
	-I$(STAGING_DIR)/usr/include/mac80211-backport/uapi \
	-I$(STAGING_DIR)/usr/include/mac80211 \
	-I$(STAGING_DIR)/usr/include/mac80211/uapi \
	-include backport/backport.h

NOSTDINC_FLAGS+=-DCONFIG_IOCTL_CFG80211 -DRTW_USE_CFG80211_STA_EVENT -DBUILD_OPENWRT

define Build/Compile
	+$(MAKE) $(PKG_JOBS) -C "$(LINUX_DIR)" \
		$(KERNEL_MAKE_FLAGS) \
		M="$(PKG_BUILD_DIR)" \
		NOSTDINC_FLAGS="$(NOSTDINC_FLAGS)" \
		modules
endef

$(eval $(call KernelPackage,rtl8852au))

Then I select it in menuconfig config.

But it failed when try to build it.

make[3]: Entering directory '/home/debian/openwrt/package/kernel/rtl8852au'
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.prepared_7f35634bf13af35f486399935b2990ee_6664517399ebbbc92a37c5bb081b5c53_check
. /home/debian/openwrt/include/shell.sh; xzcat /home/debian/openwrt/dl/rtl8852au-2023-02-25-cb319bee.tar.xz | tar -C /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.. -xf -
[ ! -d ./src/ ] || cp -fpR ./src/. /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.prepared_7f35634bf13af35f486399935b2990ee_6664517399ebbbc92a37c5bb081b5c53
rm -f /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.configured_*
rm -f /home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/stamp/.rtl8852au_installed
(cd /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/./; if [ -x ./configure ]; then find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.guess | xargs -r chmod u+w; find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.guess | xargs -r -n1 cp --remove-destination /home/debian/openwrt/scripts/config.guess; find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.sub | xargs -r chmod u+w; find /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/ -name config.sub | xargs -r -n1 cp --remove-destination /home/debian/openwrt/scripts/config.sub; AR="aarch64-openwrt-linux-musl-gcc-ar" AS="aarch64-openwrt-linux-musl-gcc -c -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee=rtl8852au-2023-02-25-cb319bee -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=aarch64-openwrt-linux-musl-ld NM="aarch64-openwrt-linux-musl-gcc-nm" CC="aarch64-openwrt-linux-musl-gcc" GCC="aarch64-openwrt-linux-musl-gcc" CXX="aarch64-openwrt-linux-musl-g++" RANLIB="aarch64-openwrt-linux-musl-gcc-ranlib" STRIP=aarch64-openwrt-linux-musl-strip OBJCOPY=aarch64-openwrt-linux-musl-objcopy OBJDUMP=aarch64-openwrt-linux-musl-objdump SIZE=aarch64-openwrt-linux-musl-size CFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee=rtl8852au-2023-02-25-cb319bee -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee=rtl8852au-2023-02-25-cb319bee -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/usr/include -I/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/include/fortify -I/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/include " LDFLAGS="-L/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/usr/lib -L/home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/lib -znow -zrelro "   ./configure --target=aarch64-openwrt-linux --host=aarch64-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  ; fi; )
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.configured_68b329da9893e34099c7d8ad5cb9c940
rm -f /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.built
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.built_check
make -j1 -C "/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176" KCFLAGS="-fmacro-prefix-map=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl=target-aarch64_cortex-a72_musl -fno-caller-saves " HOSTCFLAGS="-O2 -I/home/debian/openwrt/staging_dir/host/include -I/home/debian/openwrt/staging_dir/hostpkg/include -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/host/include -Wall -Wmissing-prototypes -Wstrict-prototypes" CROSS_COMPILE="aarch64-openwrt-linux-musl-" ARCH="arm64" KBUILD_HAVE_NLS=no KBUILD_BUILD_USER="" KBUILD_BUILD_HOST="" KBUILD_BUILD_TIMESTAMP="Mon Mar 27 16:59:53 2023" KBUILD_BUILD_VERSION="0" HOST_LOADLIBES="-L/home/debian/openwrt/staging_dir/host/lib" KBUILD_HOSTLDLIBS="-L/home/debian/openwrt/staging_dir/host/lib" CONFIG_SHELL="bash" V=1  cmd_syscalls= KBUILD_EXTRA_SYMBOLS="/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers/mac80211.symvers" KERNELRELEASE=5.10.176 M="/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee" NOSTDINC_FLAGS="-nostdinc  -isystem /home/debian/openwrt/staging_dir/toolchain-aarch64_cortex-a72_gcc-11.2.0_musl/lib/gcc/aarch64-openwrt-linux-musl/11.2.0/include -I/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee -I/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/include -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211-backport -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211-backport/uapi -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211 -I/home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/usr/include/mac80211/uapi -include backport/backport.h -DCONFIG_IOCTL_CFG80211 -DRTW_USE_CFG80211_STA_EVENT -DBUILD_OPENWRT" modules
make[4]: Entering directory '/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176'
test -e include/generated/autoconf.h -a -e include/config/auto.conf || (		\
echo >&2;							\
echo >&2 "  ERROR: Kernel configuration is invalid.";		\
echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are missing.";\
echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix it.";	\
echo >&2 ;							\
/bin/false)
make -f ./scripts/Makefile.build obj=/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee \
single-build= \
need-builtin=1 need-modorder=1
  {  :; } | awk '!x[$0]++' - > /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/modules.order
make -f ./scripts/Makefile.modpost
  sed 's/ko$/o/' /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/modules.order | scripts/mod/modpost     -o /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers -e -i Module.symvers -i /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers/mac80211.symvers   -T -
make -f ./scripts/Makefile.modfinal
make[4]: Leaving directory '/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176'
for subdir in .; do realdir=$(readlink -f /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee); grep -F /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/$subdir/Module.symvers >> /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers.tmp; [ "/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee" = "$realdir" ] || grep -F $realdir /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/$subdir/Module.symvers >> /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers.tmp; done; sort -u /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers.tmp > /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers; mkdir -p /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers; mv /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/Module.symvers /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/symvers/rtl8852au.symvers
touch /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.built
rm -rf /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au.installed /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au
mkdir -p /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au
ERROR: module '/home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/linux-5.10.176/drivers/net/wireless/realtek/rtw89/8852.ko' is missing.
make[3]: *** [Makefile:56: /home/debian/openwrt/build_dir/target-aarch64_cortex-a72_musl/linux-bcm27xx_bcm2711/rtl8852au-2023-02-25-cb319bee/.pkgdir/kmod-rtl8852au.installed] Error 1
make[3]: Leaving directory '/home/debian/openwrt/package/kernel/rtl8852au'
time: package/kernel/rtl8852au/compile#0.88#0.44#1.19
    ERROR: package/kernel/rtl8852au failed to build.
make[2]: *** [package/Makefile:116: package/kernel/rtl8852au/compile] Error 1
make[2]: Leaving directory '/home/debian/openwrt'
make[1]: *** [package/Makefile:110: /home/debian/openwrt/staging_dir/target-aarch64_cortex-a72_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/debian/openwrt'
make: *** [/home/debian/openwrt/include/toplevel.mk:230: world] Error 2

Why the compiler bypass it? How to fix this.

if you want to save time, and avoid pulling your hair out, get a better ( = non RTL) usb wifi device.

Buy another one which is supported by openwrt is definitely the best choice.
But number of devices supported by openwrt is small.
I know that mt7921u with features WIFI6 80MHz ap mode and supported by openwrt. But I see lots of negative comments of its stability.
Could you recommend some other usb wifi nic which supports WIFI6 80MHz ap mode and supported by openwrt?

these should work https://github.com/morrownr/USB-WiFi

if your plan for the RTL was an AP, it probably would have been a dead end anyway.

But I see lots of negative comments of its stability.

That is not my experience. I use a mt7921au based adapter on 22.03.3. Maybe the comments are old and the driver has been improved since the comments were made...or maybe those making the comments were doing something that I am not. The mt7921u driver gets an lot of ongoing work to add features and improve function.

Hi. I bought a cf-953ax(mt7921au) yesterday.

I compiled the firmware of mt7921u mt7921e successfully and install it on my rpi 4B.

The WIFI5 80MHz ap works fine. But when it comes to WIFI6, the AP only works in short distance, about 4~5m. If I go farther, it disconnects.
I notice that the rx rate of AP is low when I far away 4m from AP.

Someone else also have encoutered this. https://github.com/openwrt/mt76/issues/713
Have you encountered such a situation?

Did you set a country for the radio ?

Sure, here is the config of the ap.

wireless.radio1=wifi-device
wireless.radio1.type='mac80211'
wireless.radio1.path='scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.3'
wireless.radio1.band='5g'
wireless.radio1.channel='153'
wireless.radio1.txpower='30'
wireless.radio1.cell_density='0'
wireless.radio1.htmode='HE80'
wireless.radio1.country='CN'

In WIFI6(HE 80), the problem occurs.
In WIFI5(VHT 80), it works fine.

I also tried to change the country to US, but nothing seems to help.

I built openwrt from the master branch instead of the 22.03 and flash it to the rpi4b.
After that WIFI6 ap works fine with mt7921au.
So, buy another wifi nic is the best choice.