rtl8812AU and/or rtl8814AU drivers

..about missing symbols

How would I compile it against that module? Is there a way to update https://github.com/nazar-pc/lede-rtl8814au to compile it that way?

I haven't done anything with the rtl8812/4, but you could do something like the acx-mac80211, mt76 or mwlwifi package.
https://github.com/lede-project/source/blob/master/package/kernel/acx-mac80211/Makefile

I've added NOSTDINC_FLAGS as following:

include $(TOPDIR)/rules.mk
include $(INCLUDE_DIR)/kernel.mk

PKG_NAME:=rtl8814au
PKG_VERSION=2017-09-30-$(PKG_SOURCE_VERSION)
PKG_RELEASE:=1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/nazar-pc/rtl8814AU.git
PKG_SOURCE_VERSION:=7b129626b334c8df1d1e87643d9b04bd33f86af4

include $(INCLUDE_DIR)/package.mk

define KernelPackage/rtl8814au
  SUBMENU:=Wireless Drivers
  TITLE:=Realtek RTL8814AU support
  DEPENDS:= @USB_SUPPORT +kmod-usb-core +kmod-mac80211
  FILES:= $(PKG_BUILD_DIR)/8814au.ko
#  AUTOLOAD:=$(call AutoLoad,rtl8814au)
endef

NOSTDINC_FLAGS = \
	-I$(PKG_BUILD_DIR) \
	-I$(STAGING_DIR)/usr/include/mac80211-backport/uapi \
	-I$(STAGING_DIR)/usr/include/mac80211-backport \
	-I$(STAGING_DIR)/usr/include/mac80211/uapi \
	-I$(STAGING_DIR)/usr/include/mac80211 \
	-include backport/backport.h

define Build/Compile
	$(MAKE) $(KERNEL_MAKEOPTS) M=$(PKG_BUILD_DIR) \
		USER_EXTRA_CFLAGS="-D_LINUX_BYTEORDER_SWAB_H -DCONFIG_IOCTL_CFG80211 -DRTW_USE_CFG80211_STA_EVENT" \
		CONFIG_RTL8814A=y CONFIG_RTL8814AU=m \
		NOSTDINC_FLAGS="$(NOSTDINC_FLAGS)" \
		modules
endef

$(eval $(call KernelPackage,rtl8814au))

Unfortunately, now it fails to compile:

  CC [M]  /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.o
In file included from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/drv_types.h:35:0,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:22:
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/wifi.h:28:16: error: expected identifier before '(' token
 #define BIT(x) (1 << (x))
                ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:72:23: note: in expansion of macro 'BIT'
 #define WLAN_STA_AUTH BIT(0)
                       ^
/lede/staging_dir/target-x86_64_musl/usr/include/mac80211/sta_info.h:77:2: note: in expansion of macro 'WLAN_STA_AUTH'
  WLAN_STA_AUTH,
  ^
In file included from /lede/staging_dir/target-x86_64_musl/usr/include/mac80211-backport/asm-generic/bug.h:3:0,
                 from ./arch/x86/include/asm/bug.h:35,
                 from ./include/linux/bug.h:4,
                 from /lede/staging_dir/target-x86_64_musl/usr/include/mac80211-backport/linux/bug.h:3,
                 from ./include/linux/thread_info.h:11,
                 from ./arch/x86/include/asm/preempt.h:6,
                 from ./include/linux/preempt.h:59,
                 from ./include/linux/spinlock.h:50,
                 from ./include/linux/seqlock.h:35,
                 from ./include/linux/time.h:5,
                 from /lede/staging_dir/target-x86_64_musl/usr/include/mac80211-backport/linux/time.h:3,
                 from ./include/linux/stat.h:18,
                 from ./include/linux/module.h:10,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/basic_types.h:81,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/drv_types.h:31,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:22:
/lede/staging_dir/target-x86_64_musl/usr/include/mac80211/sta_info.h: In function 'sta_info_pre_move_state':
/lede/staging_dir/target-x86_64_musl/usr/include/mac80211/sta_info.h:612:34: error: 'WLAN_STA_INSERTED' undeclared (first use in this function)
  WARN_ON_ONCE(test_sta_flag(sta, WLAN_STA_INSERTED));
                                  ^
./include/asm-generic/bug.h:117:27: note: in definition of macro 'WARN_ON_ONCE'
  int __ret_warn_once = !!(condition);   \
                           ^
/lede/staging_dir/target-x86_64_musl/usr/include/mac80211/sta_info.h:612:34: note: each undeclared identifier is reported only once for each function it appears in
  WARN_ON_ONCE(test_sta_flag(sta, WLAN_STA_INSERTED));
                                  ^
./include/asm-generic/bug.h:117:27: note: in definition of macro 'WARN_ON_ONCE'
  int __ret_warn_once = !!(condition);   \
                           ^
In file included from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/drv_types.h:112:0,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:22:
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/rtw_mlme_ext.h: At top level:
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/rtw_mlme_ext.h:481:34: error: 'NUM_STA' undeclared here (not in a function)
  struct FW_Sta_Info  FW_sta_info[NUM_STA];
                                  ^
In file included from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:22:0:
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/drv_types.h:945:18: error: field 'stapriv' has incomplete type
  struct sta_priv stapriv;
                  ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c: In function 'rtw_setstakey_cmd':
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1697:40: error: 'struct sta_info' has no member named 'hwaddr'
  _rtw_memcpy(psetstakey_para->addr, sta->hwaddr,ETH_ALEN);
                                        ^
In file included from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/drv_types.h:82:0,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:22:
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/rtw_security.h:260:26: error: 'struct sta_info' has no member named 'dot118021XPrivacy'
     encry_algo =(u8) psta->dot118021XPrivacy;\
                          ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1702:3: note: in expansion of macro 'GET_ENCRY_ALGO'
   GET_ENCRY_ALGO(psecuritypriv, sta, psetstakey_para->algorithm, _FALSE);
   ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1709:42: error: 'struct sta_info' has no member named 'dot118021x_UncstKey'
   _rtw_memcpy(&psetstakey_para->key, &sta->dot118021x_UncstKey, 16);
                                          ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c: In function 'rtw_clearstakey_cmd':
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1769:50: error: 'struct sta_info' has no member named 'hwaddr'
   while ((cam_id = rtw_camid_search(padapter, sta->hwaddr, -1, -1)) >= 0) {
                                                  ^
In file included from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/drv_types.h:36:0,
                 from /lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:22:
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:27: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                           ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:41: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                         ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:55: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                       ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:69: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                                     ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:83: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                                                   ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:97: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                                                                 ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:27: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                           ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:41: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                         ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:55: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                       ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:69: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                                     ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:83: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                                                   ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:88: error: 'struct sta_info' has no member named 'hwaddr'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
                                                                                        ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/include/ieee80211.h:1381:97: note: in definition of macro 'MAC_ARG'
 #define MAC_ARG(x) ((u8*)(x))[0],((u8*)(x))[1],((u8*)(x))[2],((u8*)(x))[3],((u8*)(x))[4],((u8*)(x))[5]
                                                                                                 ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1770:4: note: in expansion of macro 'DBG_871X_LEVEL'
    DBG_871X_LEVEL(_drv_always_, "clear key for addr:"MAC_FMT", camid:%d\n", MAC_ARG(sta->hwaddr), cam_id);
    ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:1802:41: error: 'struct sta_info' has no member named 'hwaddr'
   _rtw_memcpy(psetstakey_para->addr, sta->hwaddr, ETH_ALEN);
                                         ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c: In function 'rtw_chk_hi_queue_hdl':
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:3207:13: error: implicit declaration of function 'rtw_get_bcmc_stainfo' [-Werror=implicit-function-declaration]
  psta_bmc = rtw_get_bcmc_stainfo(padapter);
             ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:3207:11: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
  psta_bmc = rtw_get_bcmc_stainfo(padapter);
           ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:3219:13: error: 'struct sta_info' has no member named 'sleepq_len'
  if(psta_bmc->sleepq_len==0)
             ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:3225:16: error: dereferencing pointer to incomplete type 'struct sta_priv'
    if (pstapriv->tim_bitmap & BIT(0))
                ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c: In function 'rtw_setstaKey_cmdrsp_callback':
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:4140:26: error: implicit declaration of function 'rtw_get_stainfo' [-Werror=implicit-function-declaration]
  struct sta_info* psta = rtw_get_stainfo(pstapriv, psetstakey_rsp->addr);
                          ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:4140:26: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c: In function 'rtw_setassocsta_cmdrsp_callback':
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:4166:26: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
  struct sta_info* psta = rtw_get_stainfo(pstapriv, passocsta_parm->addr);
                          ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:4176:6: error: 'struct sta_info' has no member named 'aid'
  psta->aid = psta->mac_id = passocsta_rsp->cam_id;
      ^
/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.c:4176:18: error: 'struct sta_info' has no member named 'mac_id'
  psta->aid = psta->mac_id = passocsta_rsp->cam_id;
                  ^
cc1: some warnings being treated as errors
scripts/Makefile.build:293: recipe for target '/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.o' failed
make[5]: *** [/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/core/rtw_cmd.o] Error 1
Makefile:1491: recipe for target '_module_/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4' failed
make[4]: *** [_module_/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4] Error 2
make[4]: Leaving directory '/lede/build_dir/target-x86_64_musl/linux-x86_64/linux-4.9.52'
Makefile:38: recipe for target '/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/.built' failed
make[3]: *** [/lede/build_dir/target-x86_64_musl/linux-x86_64/rtl8814au-2017-09-30-7b129626b334c8df1d1e87643d9b04bd33f86af4/.built] Error 2
make[3]: Leaving directory '/lede/package/kernel/rtl8814au'
package/Makefile:109: recipe for target 'package/kernel/rtl8814au/compile' failed
make[2]: *** [package/kernel/rtl8814au/compile] Error 2
make[2]: Leaving directory '/lede'
package/Makefile:105: recipe for target '/lede/staging_dir/target-x86_64_musl/stamp/.package_compile' failed
make[1]: *** [/lede/staging_dir/target-x86_64_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/lede'
Build failed - please re-run with -j1 to see the real error message
/lede/include/toplevel.mk:207: recipe for target 'world' failed
make: *** [world] Error 1

I'm not a kernel developer, not even C developer, just trying to make this thing to work. Could you take a closer look at what is going wrong here?

That doesn't look good. The expected identifier before '(' token error in this place is likely related to a error in one of the header files. This is usually tricky to trace down.

In any case, Hauke just updated the compat-wireless / backport package based on 4.14-rc2.

I'm on master branch and just pulled all of the recent changes. I'm still getting the same error when trying to load compiled module and the same compilation errors with NOSTDINC_FLAGS.
Do you know LEDE kernel developer whom can I ask for help here. I'm clearly not capable of handling this myself.

You could ask hauke (I don't think he has a forum account thought?).
He made a package for the out-of-tree rtl8189es driver, that he uploaded to his repository:
https://git.lede-project.org/?p=lede/hauke/staging.git;a=commit;h=239f9cdc685088454dccc93888c9ba76a1660c27

you can see, that he had to patch the sources for the driver quite a bit to get it to compile.
And every new release might break something, so it will need constant updates as well.

That was extremely helpful, thanks! I've looked at code there and updated my fork and now it builds against mac80211 from backport.
After some more tweaks (and updates to driver's sources) I've got something:

[ 4181.211970] RTL871X: module init start
[ 4181.212439] RTL871X: rtl8814au v4.3.21_17997.20160531
[ 4181.212943] usbcore: registered new interface driver rtl8814au
[ 4181.213467] RTL871X: module init ret=0

So driver compiles against LEDE's master branch and loads fine.
It scans networks without issues, but doesn't yet want to work in AP mode for whatever reason:

[  428.687323] RTL871X: cfg80211_rtw_set_wiphy_params
[  428.693752] RTL871X: cfg80211_rtw_add_virtual_intf wiphy:phy0, name:wlan0, type:3
[  428.695460] RTL871X: cfg80211_rtw_add_virtual_intf ndev:          (null), ret:-19
[  429.968790] RTL871X: cfg80211_rtw_add_virtual_intf wiphy:phy0, name:tmp.radio0, type:2
[  429.969679] RTL871X: cfg80211_rtw_add_virtual_intf ndev:          (null), ret:-19
[  429.971877] RTL871X: cfg80211_rtw_add_virtual_intf wiphy:phy0, name:tmp.radio0, type:2
[  429.972730] RTL871X: cfg80211_rtw_add_virtual_intf ndev:          (null), ret:-19

@nazar-pc (or others)- can your current git repository (your fork) just copy the downloaded rtl8814au driver source to packages/kernel and compile it into the image - and does it work (though as a client)?

EDIT:
Compile firmware with your driver, install firmware, connect my dongle rtl8814au to USB3 (WNDR4700) and...

(...)
Wed Oct  4 11:33:12 2017 daemon.info dnsmasq-dhcp[5570]: read /etc/ethers - 0 addresses
Wed Oct  4 11:37:53 2017 kern.info kernel: [  586.826286] usb 2-1: new high-speed USB device number 3 using xhci_hcd
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.960546] RTL871X: 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.960546] usb_endpoint_descriptor(0):
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.966656] RTL871X: bLength=7
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.969804] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.973549] RTL871X: bEndpointAddress=81
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.977461] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.981280] RTL871X: bInterval=0
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.984495] RTL871X: RT_usb_endpoint_is_bulk_in = 1
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.989350] RTL871X: 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.989350] usb_endpoint_descriptor(1):
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.995423] RTL871X: bLength=7
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  586.998465] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.002198] RTL871X: bEndpointAddress=2
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.006016] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.009835] RTL871X: bInterval=0
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.013050] RTL871X: RT_usb_endpoint_is_bulk_out = 2
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.017991] RTL871X: 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.017991] usb_endpoint_descriptor(2):
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.024066] RTL871X: bLength=7
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.027106] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.030839] RTL871X: bEndpointAddress=3
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.034657] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.038476] RTL871X: bInterval=0
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.041691] RTL871X: RT_usb_endpoint_is_bulk_out = 3
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.046633] RTL871X: 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.046633] usb_endpoint_descriptor(3):
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.052708] RTL871X: bLength=7
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.055748] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.059481] RTL871X: bEndpointAddress=4
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.063300] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.067118] RTL871X: bInterval=0
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.070333] RTL871X: RT_usb_endpoint_is_bulk_out = 4
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.075297] RTL871X: 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.075297] usb_endpoint_descriptor(4):
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.081368] RTL871X: bLength=7
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.084416] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.088148] RTL871X: bEndpointAddress=85
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.092053] RTL871X: wMaxPacketSize=64
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.095785] RTL871X: bInterval=1
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.099001] RTL871X: RT_usb_endpoint_is_int_in = 5, Interval = 1
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.104980] RTL871X: nr_endpoint=5, in_num=2, out_num=3
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.104980] 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.111658] RTL871X: USB_SPEED_HIGH
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.115133] RTL871X: CHIP TYPE: RTL8814
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.518807] RTL871X: read_chip_version_8814a SYS_CFG(0xF0)=0x044411b5 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.525337] RTL871X: rtw_hal_config_rftype RF_Type is 6 TotalTxPath is 3 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.532120] RTL871X: Chip Version Info: CHIP_8814A_Normal_Chip_SMIC_B_CUT_3T3R_RomVer(0)
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.540185] RTL871X: _ConfigChipOutEP_8814 OutEpQueueSel(0x07), OutEpNumber(3) 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.547477] RTL871X: ====> Hal_InitEfuseVars_8814A 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.552346] RTL871X: ====>pEfuseHal->WordUnit 4 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.556948] RTL871X: ====>pEfuseHal->OOBProtectBytes 15 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.562252] RTL871X: ====>pEfuseHal->ProtectBytes 16 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.567292] RTL871X: ====>pEfuseHal->MaxSecNum_WiFi 64 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.572500] RTL871X: ====>pEfuseHal->PhysicalLen_WiFi 1024 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.578054] RTL871X: ====>pEfuseHal->LogicalLen_WiFi 512 
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.583459] RTL871X: Hal_InitEfuseVars_8814A <====
Wed Oct  4 11:37:54 2017 kern.warn kernel: [  587.661252] RTL871X: Boot from EFUSE, Autoload OK !
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.799360] RTL871X: VID = 0x0BDA, PID = 0x8813
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.803902] RTL871X: Customer ID: 0x00, SubCustomer ID: 0xCD
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.809568] RTL871X: EEPROMRegulatory = 0x1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.813800] RTL871X: EEPROM RF set 4T4R
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.817636] RTL871X: default rf type: 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.821463] RTL871X: Final rf_config: 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.825282] RTL871X: mlmepriv.ChannelPlan = 0x7f
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.829887] RTL871X: CrystalCap: 0x b
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.833534] RTL871X: ThermalMeter = 0x21
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.837448] RTL871X: hal_ReadRemoteWakeup_8814A...bSupportRemoteWakeup(0)
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.844212] RTL871X: SWAS: bHwAntDiv = 0, TRxAntDivType = ff
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.849855] RTL871X: pHalData->ExternalPA_2G = 0, pHalData->ExternalPA_5G = 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.856965] RTL871X: pHalData->ExternalLNA_2G = 0, pHalData->ExternalLNA_5G = 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.864248] RTL871X: pHalData->TypeGPA = 0x0, pHalData->TypeAPA = 0x0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.870668] RTL871X: pHalData->TypeGLNA = 0x0, pHalData->TypeALNA = 0x0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.877260] RTL871X: RFE Type: 0x 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.880741] RTL871X: PID= 0x8813, VID=  bda
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.884905] RTL871X: Customer ID: 0x 0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.888640] RTL871X: hal_GetRxGainOffset_8814A(): AutoloadFail = 0!!
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.894973] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_2G = 0xffff!!
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.902256] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_5GL = 0xffff!!
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.909626] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_5GM = 0xffff!!
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.917019] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_5GH = 0xffff!!
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.924390] RTL871X: registrypriv.RegRfKFreeEnable = 2
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.929506] RTL871X: kfree flag:0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.932809] RTL871X: rtw_hal_read_chip_info in 1385 ms
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.938995] RTL871X: init_channel_set((null)) ChannelPlan ID:0x7f, ch num:37
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.946655] RTL871X: NR_RECVBUFF: 8
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.950159] RTL871X: MAX_RECVBUF_SZ: 32768
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.954252] RTL871X: NR_PREALLOC_RECV_SKB: 8
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.958828] RTL871X: rtw_alloc_macid((null)) if1, hwaddr:ff:ff:ff:ff:ff:ff macid:1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.966435] RTL871X: can't get autopm: 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.970275] RTL871X: rtw_macaddr_cfg mac addr:40:a5:ef:d7:fd:8c
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.976177] RTL871X: bDriverStopped:True, bSurpriseRemoved:False, bup:0, hw_init_completed:0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.989257] RTL871X: reg 0x5, usb write 1 fail, status:-71 value=0x80a5efd7, vendorreq_times:1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  588.998336] RTL871X: reg 0x5, usb write 1 fail, status:-71 value=0x80a5efd7, vendorreq_times:2
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.007362] RTL871X: reg 0x5, usb write 1 fail, status:-71 value=0x80a5efd7, vendorreq_times:3
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.016459] RTL871X: reg 0x5, usb write 1 fail, status:-71 value=0x80a5efd7, vendorreq_times:4
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.025486] RTL871X: reg 0x5, usb write 1 fail, status:-71 value=0x80a5efd7, vendorreq_times:5
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.034106] RTL871X: [dvobj:ce174800][ERROR] continual_io_error:5 > 4
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.040544] RTL871X: ### Clean STA_(1) info ###
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.045070] RTL871X: rtw_dev_unload: bup==_FALSE
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.049684] RTL871X: +r871xu_dev_remove, hw_init_completed=0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.055648] RTL871X: 4free_recv_skb_queue not empty, 8
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.060922] RTL871X: rtw_wdev_free(wdev=  (null))
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.065620] RTL871X: usb attached..., try to reset usb device
Wed Oct  4 11:37:56 2017 kern.info kernel: [  589.176459] usb 2-1: USB disconnect, device number 3
Wed Oct  4 11:37:56 2017 kern.info kernel: [  589.395252] usb 2-1: new high-speed USB device number 4 using xhci_hcd
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.533388] RTL871X: 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.533388] usb_endpoint_descriptor(0):
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.539487] RTL871X: bLength=7
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.542536] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.546276] RTL871X: bEndpointAddress=81
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.550186] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.554004] RTL871X: bInterval=0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.557267] RTL871X: RT_usb_endpoint_is_bulk_in = 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.562134] RTL871X: 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.562134] usb_endpoint_descriptor(1):
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.568214] RTL871X: bLength=7
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.571259] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.574991] RTL871X: bEndpointAddress=2
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.578810] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.582641] RTL871X: bInterval=0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.585864] RTL871X: RT_usb_endpoint_is_bulk_out = 2
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.590815] RTL871X: 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.590815] usb_endpoint_descriptor(2):
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.596889] RTL871X: bLength=7
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.599935] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.603668] RTL871X: bEndpointAddress=3
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.607488] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.611314] RTL871X: bInterval=0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.614528] RTL871X: RT_usb_endpoint_is_bulk_out = 3
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.619472] RTL871X: 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.619472] usb_endpoint_descriptor(3):
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.625545] RTL871X: bLength=7
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.628596] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.632431] RTL871X: bEndpointAddress=4
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.636267] RTL871X: wMaxPacketSize=512
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.640093] RTL871X: bInterval=0
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.643307] RTL871X: RT_usb_endpoint_is_bulk_out = 4
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.648250] RTL871X: 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.648250] usb_endpoint_descriptor(4):
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.654323] RTL871X: bLength=7
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.657365] RTL871X: bDescriptorType=5
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.661097] RTL871X: bEndpointAddress=85
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.665003] RTL871X: wMaxPacketSize=64
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.668735] RTL871X: bInterval=1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.671950] RTL871X: RT_usb_endpoint_is_int_in = 5, Interval = 1
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.677937] RTL871X: nr_endpoint=5, in_num=2, out_num=3
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.677937] 
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.684615] RTL871X: USB_SPEED_HIGH
Wed Oct  4 11:37:56 2017 kern.warn kernel: [  589.688090] RTL871X: CHIP TYPE: RTL8814
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.924521] RTL871X: read_chip_version_8814a SYS_CFG(0xF0)=0x044411b5 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.931046] RTL871X: rtw_hal_config_rftype RF_Type is 6 TotalTxPath is 3 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.937815] RTL871X: Chip Version Info: CHIP_8814A_Normal_Chip_SMIC_B_CUT_3T3R_RomVer(0)
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.945986] RTL871X: _ConfigChipOutEP_8814 OutEpQueueSel(0x07), OutEpNumber(3) 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.953280] RTL871X: ====> Hal_InitEfuseVars_8814A 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.958140] RTL871X: ====>pEfuseHal->WordUnit 4 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.962738] RTL871X: ====>pEfuseHal->OOBProtectBytes 15 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.968025] RTL871X: ====>pEfuseHal->ProtectBytes 16 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.973054] RTL871X: ====>pEfuseHal->MaxSecNum_WiFi 64 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.978263] RTL871X: ====>pEfuseHal->PhysicalLen_WiFi 1024 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.983811] RTL871X: ====>pEfuseHal->LogicalLen_WiFi 512 
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  589.989186] RTL871X: Hal_InitEfuseVars_8814A <====
Wed Oct  4 11:37:57 2017 kern.warn kernel: [  590.006063] RTL871X: Boot from EFUSE, Autoload OK !
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.111027] RTL871X: VID = 0x0BDA, PID = 0x8813
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.115568] RTL871X: Customer ID: 0x00, SubCustomer ID: 0xCD
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.121240] RTL871X: EEPROMRegulatory = 0x1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.125491] RTL871X: EEPROM RF set 4T4R
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.129329] RTL871X: default rf type: 1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.133155] RTL871X: Final rf_config: 1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.136974] RTL871X: mlmepriv.ChannelPlan = 0x7f
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.141579] RTL871X: CrystalCap: 0x b
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.145225] RTL871X: ThermalMeter = 0x21
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.149154] RTL871X: hal_ReadRemoteWakeup_8814A...bSupportRemoteWakeup(0)
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.155917] RTL871X: SWAS: bHwAntDiv = 0, TRxAntDivType = ff
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.161557] RTL871X: pHalData->ExternalPA_2G = 0, pHalData->ExternalPA_5G = 1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.168668] RTL871X: pHalData->ExternalLNA_2G = 0, pHalData->ExternalLNA_5G = 1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.175950] RTL871X: pHalData->TypeGPA = 0x0, pHalData->TypeAPA = 0x0
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.182369] RTL871X: pHalData->TypeGLNA = 0x0, pHalData->TypeALNA = 0x0
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.188952] RTL871X: RFE Type: 0x 1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.192426] RTL871X: PID= 0x8813, VID=  bda
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.196598] RTL871X: Customer ID: 0x 0
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.200333] RTL871X: hal_GetRxGainOffset_8814A(): AutoloadFail = 0!!
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.206664] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_2G = 0xffff!!
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.213939] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_5GL = 0xffff!!
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.221304] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_5GM = 0xffff!!
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.228673] RTL871X: hal_GetRxGainOffset_8814A(): RegRxGainOffset_5GH = 0xffff!!
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.236042] RTL871X: registrypriv.RegRfKFreeEnable = 2
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.241163] RTL871X: kfree flag:0
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.244466] RTL871X: rtw_hal_read_chip_info in 1291 ms
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.375302] RTL871X: init_channel_set((null)) ChannelPlan ID:0x7f, ch num:37
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.391621] RTL871X: NR_RECVBUFF: 8
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.395123] RTL871X: MAX_RECVBUF_SZ: 32768
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.399223] RTL871X: NR_PREALLOC_RECV_SKB: 8
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.416301] RTL871X: rtw_alloc_macid((null)) if1, hwaddr:ff:ff:ff:ff:ff:ff macid:1
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.423894] RTL871X: can't get autopm: 
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.427740] RTL871X: rtw_macaddr_cfg mac addr:40:a5:ef:d7:fd:8c
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.433646] RTL871X: bDriverStopped:True, bSurpriseRemoved:False, bup:0, hw_init_completed:0
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.470197] RTL871X: rtw_wiphy_alloc(phy2)
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.474393] RTL871X: rtw_wdev_alloc(padapter=d65bc000)
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.479532] RTL871X: rtw_wiphy_register(phy2)
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.483870] RTL871X: Register RTW cfg80211 vendor cmd(0x67) interface 
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.490778] RTL871X: rtw_ndev_init(wlan0) if1 mac_addr=40:a5:ef:d7:fd:8c
Wed Oct  4 11:37:58 2017 kern.warn kernel: [  591.498156] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  606.952580] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  606.978540] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.021952] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.058641] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.086650] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.102574] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.114522] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.130546] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.143845] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.166375] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.182602] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:14 2017 kern.warn kernel: [  607.218588] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:19 2017 kern.warn kernel: [  612.796241] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:19 2017 kern.warn kernel: [  612.815557] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:19 2017 kern.warn kernel: [  612.835368] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.867108] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.885539] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.897425] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.904774] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.912199] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.919418] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.929425] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.936763] RTL871X: cfg80211_rtw_get_txpower
Wed Oct  4 11:38:20 2017 kern.warn kernel: [  612.954034] RTL871X: cfg80211_rtw_get_txpower
(...)

...what next ?
Pleas instruction "step by step"
:smiley:

Are you compiling it for LEDE? If not, try proper DKMS version for desktops instead: https://github.com/nazar-pc/RTL8814AU
For LEDE I've only tested with upstream master branch, but didn't succeed with use much besides being able to search for available APs under some circumstances.
So no, I didn't succeed using it as client on LEDE, but it does work with DKMS driver under my Ubuntu 17.04 x64 with the same performance as Windows driver from Realtek.

no - i compile and install LEDE master with this driver:

Luci view only "Generic MAC80211 802.11abg (radio2)" (???) interface...unfortunately client mode not connected (80MHz ac, 5 GHz)...
###############
To my ubuntu-16.04.3 (kernel hwe: 4.10.0-35) work fine (very fine to linux; windows oryginal driver to w10 very...shit - hehe) use this dkms driver:

It changes to abgnac when you try to scan for networks (if booted with disabled interface in UI), but otherwise I see exactly the same behavior - can't run in AP mode, can't connect in client mode.
I've reported this issue to the person that knows much more than me here: https://github.com/ulli-kroll/rtl8814au/issues/5
Hopefully he'll find some time to look at it soon. Looks like there is some fundamental issue after which driver refuses to create virtual interfaces and you can't really do anything about it:(

root@WNDR4700:~# iw list
Wiphy phy2
	max # scan SSIDs: 9
	max scan IEs length: 2304 bytes
	max # sched scan SSIDs: 0
	max # match sets: 0
	max # scan plans: 1
	max scan plan interval: -1
	max scan plan iterations: 0
	Retry short limit: 7
	Retry long limit: 4
	Coverage class: 1 (up to 450m)
	Available Antennas: TX 0 RX 0
	Supported interface modes:
		 * IBSS
		 * managed
		 * AP
		 * P2P-client
		 * P2P-GO
	Band 1:
		Frequencies:
			* 2412 MHz [1] (30.0 dBm)
			* 2417 MHz [2] (30.0 dBm)
			* 2422 MHz [3] (30.0 dBm)
			* 2427 MHz [4] (30.0 dBm)
			* 2432 MHz [5] (30.0 dBm)
			* 2437 MHz [6] (30.0 dBm)
			* 2442 MHz [7] (30.0 dBm)
			* 2447 MHz [8] (30.0 dBm)
			* 2452 MHz [9] (30.0 dBm)
			* 2457 MHz [10] (30.0 dBm)
			* 2462 MHz [11] (30.0 dBm)
			* 2467 MHz [12] (disabled)
			* 2472 MHz [13] (disabled)
			* 2484 MHz [14] (disabled)
	Band 2:
		Frequencies:
			* 5170 MHz [34] (disabled)
			* 5180 MHz [36] (23.0 dBm)
			* 5190 MHz [38] (23.0 dBm)
			* 5200 MHz [40] (23.0 dBm)
			* 5210 MHz [42] (23.0 dBm)
			* 5220 MHz [44] (23.0 dBm)
			* 5230 MHz [46] (23.0 dBm)
			* 5240 MHz [48] (23.0 dBm)
			* 5260 MHz [52] (23.0 dBm) (radar detection)
			* 5280 MHz [56] (23.0 dBm) (radar detection)
			* 5300 MHz [60] (23.0 dBm) (radar detection)
			* 5320 MHz [64] (23.0 dBm) (radar detection)
			* 5500 MHz [100] (23.0 dBm) (radar detection)
			* 5520 MHz [104] (23.0 dBm) (radar detection)
			* 5540 MHz [108] (23.0 dBm) (radar detection)
			* 5560 MHz [112] (23.0 dBm) (radar detection)
			* 5580 MHz [116] (23.0 dBm) (radar detection)
			* 5600 MHz [120] (23.0 dBm) (radar detection)
			* 5620 MHz [124] (23.0 dBm) (radar detection)
			* 5640 MHz [128] (23.0 dBm) (radar detection)
			* 5660 MHz [132] (23.0 dBm) (radar detection)
			* 5680 MHz [136] (23.0 dBm) (radar detection)
			* 5700 MHz [140] (23.0 dBm) (radar detection)
			* 5745 MHz [149] (30.0 dBm)
			* 5765 MHz [153] (30.0 dBm)
			* 5785 MHz [157] (30.0 dBm)
			* 5805 MHz [161] (30.0 dBm)
			* 5825 MHz [165] (30.0 dBm)
			* 5920 MHz [184] (disabled)
			* 5940 MHz [188] (disabled)
			* 5960 MHz [192] (disabled)
			* 5980 MHz [196] (disabled)
			* 6000 MHz [200] (disabled)
			* 6020 MHz [204] (disabled)
			* 6040 MHz [208] (disabled)
			* 6060 MHz [212] (disabled)
			* 6080 MHz [216] (disabled)
	interface combinations are not supported
(...)

Anyway thank you for any concrete progress in the very creation of the installing and building driver, who, as I also have a quiet hope, will soon start working "full mouth" under LEDE.

Hey guys some update about add AP mode on RTL8814AU

This approach is crazy (as hell ?)

Looking trough the code I have not just to add AP mode functionality into
cfg80211_rtw_add_virtual_intf()
including open/close for AP mode, which I can be borrowed form
rtw_cfg80211_add_monitor_if()
There is (maybe) also some code missing to use this "virtual" interface.
i.e. send the right information Elements (for SSID HT/VHT capabilities),
There is absolutely no "sample" code I can (re)use here.

The same applies on RTL8821AU which uses v5.1.5.
No AP mode at all :frowning:
and no STA mode in cfg80211.
Unfortunately I didn't test the drivers with cfg80211

So I've decided to reactivate my RTL8821AU work, with my USB3 issue to test.
Which shows some interesting findings.

First log of my RTL8821AU driver with fast USB3 switch using a RTL8813 device

[   19.520094] usb 2-2: new high-speed USB device number 3 using xhci_hcd
[   19.702909] usb 2-2: New USB device found, idVendor=0bda, idProduct=8813
[   19.709835] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   19.717377] usb 2-2: Product: 802.11ac NIC
[   19.721647] usb 2-2: Manufacturer: Realtek
[   19.725915] usb 2-2: SerialNumber: 123456
[   62.893080] rtl8821au: loading out-of-tree module taints kernel.
[   62.903618] usb 2-2: rtl8812au_test_usb3_switch : 0bda:8813
[   62.909427] usb 2-2: try switch from USB2 to USB3
[   62.914246] usb 2-2: device running in USB2 mode
[   62.919560] usb 2-2: switch to USB3
[   62.924799] usb 2-2: do reconnect
[   62.928668] usb 2-2: USB disconnect, device number 3
[   62.928994] reg 0x5, usbctrl_vendorreq TimeOut! status:0xffffffed value=0x80
[   62.929027] usb 2-2: try settle down
[   62.950054] rtl8821au: probe of 2-2:1.0 failed with error -11
[   62.956157] usbcore: registered new interface driver rtl8821au
[   63.320273] usb 3-2: new SuperSpeed USB device number 5 using xhci_hcd
[   63.352408] usb 3-2: Int endpoint with wBytesPerInterval of 512 in config 1 interface 0 altsetting 0 ep 133: setting to 64
[   63.365276] usb 3-2: New USB device found, idVendor=0bda, idProduct=8813
[   63.372185] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   63.379590] usb 3-2: Product: 802.11ac NIC
[   63.383934] usb 3-2: Manufacturer: Realtek
[   63.388169] usb 3-2: SerialNumber: 123456

note here high-speed vs. SuperSpeed
And now with a RTL8812AU device, with the same driver

[   46.740073] usb 2-1.4.2: new high-speed USB device number 9 using xhci_hcd
[   46.873471] usb 2-1.4.2: New USB device found, idVendor=0bda, idProduct=8812
[   46.880862] usb 2-1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   46.888565] usb 2-1.4.2: Product: 802.11n NIC
[   46.893042] usb 2-1.4.2: Manufacturer: Realtek
[   46.897728] usb 2-1.4.2: SerialNumber: 123456
[   46.904782] usb 2-1.4.2: rtl8812au_test_usb3_switch : 0bda:8812
[   46.911037] usb 2-1.4.2: try switch from USB2 to USB3
[   46.916354] usb 2-1.4.2: device running in USB2 mode
[   46.921863] usb 2-1.4.2: switch to USB3
[   46.927466] usb 2-1.4.2: do reconnect
[   46.937629] reg 0x5, usbctrl_vendorreq TimeOut! status:0xffffffb9 value=0x80
[   46.944862] usb 2-1.4.2: try settle down
[   46.970117] rtl8821au: probe of 2-1.4.2:1.0 failed with error -11
[   46.979218] usb 2-1.4.2: USB disconnect, device number 9
[   47.380073] usb 2-1.4.2: new high-speed USB device number 10 using xhci_hcd
[   47.513465] usb 2-1.4.2: New USB device found, idVendor=0bda, idProduct=8812
[   47.520746] usb 2-1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   47.528398] usb 2-1.4.2: Product: 802.11n NIC
[   47.532861] usb 2-1.4.2: Manufacturer: Realtek
[   47.537524] usb 2-1.4.2: SerialNumber: 123456
[   47.544122] usb 2-1.4.2: rtl8812au_test_usb3_switch : 0bda:8812
[   47.550420] usb 2-1.4.2: try switch from USB2 to USB3
[   47.555554] usb 2-1.4.2: device running in USB2 mode
[   47.561104] usb 2-1.4.2: switch to USB3
[   47.566717] usb 2-1.4.2: do reconnect
[   47.576862] reg 0x5, usbctrl_vendorreq TimeOut! status:0xffffffb9 value=0x80
[   47.584118] usb 2-1.4.2: try settle down
[   47.610071] rtl8821au: probe of 2-1.4.2:1.0 failed with error -11
[   47.619094] usb 2-1.4.2: USB disconnect, device number 10
[   48.020038] usb 2-1.4.2: new high-speed USB device number 11 using xhci_hcd
[   48.153467] usb 2-1.4.2: New USB device found, idVendor=0bda, idProduct=8812
[   48.160745] usb 2-1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3

you see an endless loop

yup, i pulled last night and failed to compile but with a small patch i worked up. i got it back up and running again. here is the patch (please dont expect this to patch, my driver isn't stock).
add this one the ones you are currently using if you pull trunk lately and it fails on cfg80211_roamed having the wrong arguments.

---
 os_dep/linux/ioctl_cfg80211.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/os_dep/linux/ioctl_cfg80211.c b/os_dep/linux/ioctl_cfg80211.c
index b11aca4..6cb6f61 100644
--- a/os_dep/linux/ioctl_cfg80211.c
+++ b/os_dep/linux/ioctl_cfg80211.c
@@ -802,21 +802,33 @@ check_bss:
 		struct ieee80211_channel *notify_channel;
 		u32 freq;
 		u16 channel = cur_network->network.Configuration.DSConfig;
+		struct cfg80211_roam_info roam_info = {};
 
 		freq = rtw_ch2freq(channel);
 		notify_channel = ieee80211_get_channel(wiphy, freq);
 		#endif
 
 		DBG_871X(FUNC_ADPT_FMT" call cfg80211_roamed\n", FUNC_ADPT_ARG(padapter));
+		roam_info.channel = notify_channel;
+		roam_info.bssid = cur_network->network.MacAddress;
+		roam_info.req_ie =
+		pmlmepriv->assoc_req+sizeof(struct rtw_ieee80211_hdr_3addr)+2;
+		roam_info.req_ie_len =
+		pmlmepriv->assoc_req_len-sizeof(struct rtw_ieee80211_hdr_3addr)-2;
+		roam_info.resp_ie =
+		pmlmepriv->assoc_rsp+sizeof(struct rtw_ieee80211_hdr_3addr)+6;
+		roam_info.resp_ie_len =
+		pmlmepriv->assoc_rsp_len-sizeof(struct rtw_ieee80211_hdr_3addr)-6;
 		cfg80211_roamed(padapter->pnetdev
-			#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 39) || defined(COMPAT_KERNEL_RELEASE)
-			, notify_channel
-			#endif
-			, cur_network->network.MacAddress
-			, pmlmepriv->assoc_req+sizeof(struct rtw_ieee80211_hdr_3addr)+2
-			, pmlmepriv->assoc_req_len-sizeof(struct rtw_ieee80211_hdr_3addr)-2
-			, pmlmepriv->assoc_rsp+sizeof(struct rtw_ieee80211_hdr_3addr)+6
-			, pmlmepriv->assoc_rsp_len-sizeof(struct rtw_ieee80211_hdr_3addr)-6
+			//#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 39) || defined(COMPAT_KERNEL_RELEASE)
+			//, notify_channel
+			//#endif
+			, &roam_info
+			//, cur_network->network.MacAddress
+			//, pmlmepriv->assoc_req+sizeof(struct rtw_ieee80211_hdr_3addr)+2
+			//, pmlmepriv->assoc_req_len-sizeof(struct rtw_ieee80211_hdr_3addr)-2
+			//, pmlmepriv->assoc_rsp+sizeof(struct rtw_ieee80211_hdr_3addr)+6
+			//, pmlmepriv->assoc_rsp_len-sizeof(struct rtw_ieee80211_hdr_3addr)-6
 			, GFP_ATOMIC);
 	}
 	else
---

edit: glad to see you back here @ElektromAn :slight_smile:

edit two (what this patch is for 4.12) see here:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/diff/include/net/cfg80211.h?h=v4.12.6&id=29ce6ecbb83c9185d76e3a7c340c9702d2a54961 and https://patchwork.kernel.org/patch/9714419/

edit three (similar to this): https://github.com/Mange/rtl8192eu-linux-driver/pull/47 also they seem to have done something else here wiphy->max_sched_scan_reqs = 1; going to check this out. most likely something i missed..

@Cutepally
I will take the patch, if you want to demand on your authorship please to a pull request as @nazar-pc has done it.

If this applies (needed) I will use this patch on rtl8821au also.

Currently I do a check on rtl8821au before I do any push on my repositories.
I've made some small mistakes in designing the switch to mac80211, which causes me to tip on my own toes.

And no there is actually no mac80211 support, I'm far away from testing

ah, yeah.. i just looked at his repo and see it in there :slight_smile: cool! its also required for rtl8812au in trunk currently.
you'll have to actually drop the LINUX_VERSION_CODE >= KERNEL_VERSION(4,12,0) check maybe though because, at least for my router, is 4.4.91 current and required here (for LEDE building of course) nevermind this part, his activate_all_backport_code.patch already doing it, great !

thanks for the update though, i know how much it would take to get this where most want it. i personally couldn't get it there nor do i really have the time to so. it's sad but at least we know that it can run under LEDE with hostapd or udhcpc, and wpa_supplicant

agreed. lol
but again thanks the update. my post above was much the same just an update because i ran into an issue and wanted to share. i hadn't checked out @nazar-pc repo just then but it's in there too, which is great. :slight_smile:

have a nice day :blossom:

edit: i read more and seen the link to hauke's repo, thanks for that! so im going to go with his patch :slight_smile: https://git.lede-project.org/?p=lede/hauke/staging.git;a=commit;h=239f9cdc685088454dccc93888c9ba76a1660c27

I about to buy one on lighning deal https://www.amazon.com/dp/B019SRBUNG
is is compatible rtl8812au driver? are in repo now?

I have one of the same, can't be used without manual setup so I don't recommend..