I got it recently working with the newest snapshot on my 2x Archer C2600, fully encrypted together with APs on 5 ghz. You only need wpad-mesh-openssl and bridge the mesh (with option mesh_fwding '1') into your LAN. Meanwhile my APs are running with WPA3/WPA2 mixed.
Oh well: you can only use channel 36-48 on 5ghz for 802.11s.
Beware: Luci in the snapshot is for today a bit buggy, so you should set this up manually.
Here is my config:
config wifi-device 'radio0'
option type 'mac80211'
option hwmode '11a'
option path 'soc/1b500000.pci/pci0000:00/0000:00:00.0/0000:01:00.0'
option htmode 'VHT80'
option country 'DE'
option txpower '23'
option channel '48'
config wifi-iface 'mesh'
option device 'radio0'
option network 'lan'
option mode 'mesh'
option mesh_id 'yourmeshid'
option mesh_fwding '1'
option key 'yourpw'
option mesh_rssi_threshold '0'
option encryption 'sae'
config wifi-device 'radio1'
option type 'mac80211'
option hwmode '11g'
option path 'soc/1b700000.pci/pci0001:00/0001:00:00.0/0001:01:00.0'
option htmode 'HT40'
option txpower '20'
option country 'DE'
option legacy_rates '0'
option channel '13'
config wifi-iface 'wifinet1'
option device 'radio0'
option key 'yourpw'
option network 'lan'
option mode 'ap'
option ssid 'yourssid'
option encryption 'sae-mixed'
option ieee80211w '1'
config wifi-iface 'wifinet2'
option encryption 'sae-mixed'
option device 'radio1'
option key 'yourpw'
option network 'lan'
option mode 'ap'
option ssid 'yourssid'
option ieee80211w '1'
Edit 1: Checked WPA3 on Mesh again. Its really working! Deleted the wrong assumption of WPA2 fallback.
Edit 2: Added option ieee80211w '1' on both APs since its a requirement for WPA3, but only optional for WPA2 because of compatibility.