i believe that was already answered in rtl8812AU and/or rtl8814AU drivers "I only quickly build tested my Makefile on x86, maybe it is different in this regard. As a quick hack, try passing USER_EXTRA_CFLAGS="-D_LINUX_BYTEORDER_SWAB_H" to make which should inhibit the redefinitions."
i'll repeat myself though "even when you do get it loaded, you'd have to run manual commands to get it to actually do anything." it can't be controlled by luci at all (other than scanning for an AP).
(it also seems you are trying to use little_endian and big_endian which, you can't do. if that is the case, this will cause a compile error after you fix the byteorder thing. if that isn't the case, it might be that you need to use big_endian instead of little_endian). i'm thinking this because i see both and i'm quite sure, seeing both in that log looks wrong to me.
anyways, please share your makefile and the driver you're trying to compile, if you want more help.
this is my makefile (well a slightly older one but this one should still build, if the driver is patched with these https://github.com/dl12345/rtl8812au/tree/master/patches and assuming your building for mips_24kc, unless you are on trunk. if you want more info about that just click the link then read the comments from the link in this gist, more toward the bottom): https://gist.github.com/kodewdle/3537b45981392dc4e403cbe005374957
~ on another note ~
as of right now, i have a little script I'm working on it's not 100% and it's probably not the best thing but it works pretty decent for me in client mode. http://i.imgur.com/UVJzJfo.png
the main goal of it is to start the module, initialize wpa_supplicant, and initialize udhcpc (for client mode) with some options (while putting udhcpc in the background and also printing out some info about the radio/dhcp info). right now, it works off of a set of preconfigured configuration files. all you have to do is power the router on then use the script to bring the card up and connect it with one of the preconfigured configuration files. this makes it easy because you just send one little line that's easily typed (/etc/config/wpad/start.sh wpa). also this can be added into the startup script for automatic purpose.
i plan on adding (what i stated above is already in-place) AP support into this when i have more time to test what actually works in AP mode and if it has some hard limit to the amount of clients that can connect to it (i really don't know because i haven't tested this mode any further than getting the AP to actually broadcast).
i haven't even tried to attempt to figure out why luci hates it so much but these might sum it up https://github.com/dl12345/rtl8812au/issues/1#issuecomment-284224662 and https://github.com/dl12345/rtl8812au/issues/2#issuecomment-284225158
also, i wasn't trying to be rude before when i said don't waste your time, i was just being honest. im surprised i even got it running at all, much less some-what-stable.
i hope you have a nice day~ 