Wifi Bridge - WDS, relayd, or something else?

I want to build something that works similarly to the Ubiquiti NanoStation wireless point-to-point bridge, but with regular Atheros hardware. I have something working with relayd, but the performance isn't as good as I would have hoped. I have two identical devices. Connecting directly via 5Ghz wifi to one I can get 100Mbps download from my 100Mbps internet connection, but connected via the relayd bridge (2 identical units in a relayd configuration over 5GHz wifi) I can only get around 60Mbps. There is no interference, I'm the only 5GHz station in the area, used channel 36 for my test. The devices were tested at 3m and 30m apart, same results.

Is that just the way it is, or is there a better way to do this? I know Ubiquiti uses a proprietary TDMA protocol (Airmax), but isn't there something else I can do to improve performance?

You haven't said what hardware you are using for OpenWrt. I'm afraid if you want 'best' performance, you have to use OEM firmware particularly if you have a 'slow' device.

eg. relayd on Home Hub 5A maxes out at 70 mbps on 5GHz 2x2. Linksys EA6350v3 is better. I think I tested 190 mbps with 5GHz 2x2 80 MHz link, but don't quote me as it was a long time ago, but still inferior to OEM firmware which is capable of up to 420 mbps in wireless bridge mode.

I'm using 2x ath79 target devices, CF-E313AC. I'll try the OEM firmware and see if it is better.

You should use a 802.11s based mesh-setup.
see
https://openwrt.org/docs/guide-user/network/wifi/mesh/80211s