I would like to start playing around with mesh on my home network for learning purpose.
Some APs are indoor/outdoor, some have ethernet and others don't have ethernet.
Is there a mesh protocol allowing authentication of nodes with support of both wired/wireless?
What packages should I install? BATMAN-adv and bmx6?
I've just started playing around with a mesh network as well, but I've started to get things working. Here's my thoughts so far.
802.11s is the official standard for mesh networking. If your hardware (and drivers, and even firmware, if necessary) support it, it is probably the easiest solution to set up. All of the above affect performance and encryption--if the OpenWRT driver/firmware (and so on) are only partially supported on your device, you'll have varying results.
11s doesn't require any additional packages, as long as your hardware and driver support it. Once you have it working, you may choose to encrypt your mesh network (suggested), and that will require the authsae package on each router.
For example, on my router, the 5 GHz radio will only do mesh at VHT20 -- 40 or 80 fail, and VHT20 is slow, so I'm currently meshing over 2.4 GHz (which limits overall network throughput.) YMMV.
I would start by trying to set up unencrypted 11.s on 2.4 GHz and get that working, and then go from there.
Have a look at the code that creates the config scripts for the mesh. There was a difference for the description of the 20/40M channel between the /etc/wireless config file description for wide bandwidth and what the mesh config was using. I forget the exact details but one would accept HT40 and the other would only accept HT40+/- or vis versa.
I'll see if I can find it as the same issue may be present for the VHT20/40/80
 found this https://github.com/lede-project/source/commit/fd718c50256536ed082852d735dc7690d7604418 might give you a starting point to look at
[edit#2] look in /lib/wifi/authsae.sh
Good finds, greybeard. mac80211.sh doesn't appear to support "HT40" alone, but just "HT40+" and "HT40-". If HT40 is the recommended value to use, why wouldn't it be supported?
Further, authsae.sh also appears to only support HT20/HT40+/HT40-. Why isn't there HT40, and VHT values?
Apparently, 802.11s should support WAP supplicant:
Currently we are working on adding mesh support (including open, authenticated and encrypted mesh networks) within the widely adopted wireless controller wpa_supplicant tool.
We are currently preparing patches for open-source submission, until the patches are accepted and integrated you can fork the latest development code from our repository.
Apparently, there is no support in Luci to setup mesh network password.