For what it's worth neighbor password is still not documented.
So maybe it is not implemented yet properly. The easiest way to troubleshoot is to debug the packets or capture them with tcpdump.
In any case there is nothing OpenWrt can do here, as quagga is just a package without any special instructions or configuration files.
You could do a search in the available packages for the bgp keyword. This seems to support it, but I have never used it.
bird2 - 2.0.7-3 - BIRD is an internet routing daemon which manages TCP/IP routing tables
with support of modern routing protocols, easy to use configuration
interface and powerful route filtering language. It is lightweight and
efficient and therefore appropriate for small embedded routers.
BIRD supports OSPFv2, RIPv2, Babel and BGP protocols for IPv4 and
OSPFv3, RIPng, Babel and BGP protocols for IPv6.
In BGP, BIRD supports communities, multiprotocol extensions, MD5
authentication, 32bit AS numbers and could act as a route server or a
route reflector. BIRD also supports multiple RIBs, multiple kernel
routing tables and redistribution between the protocols with a powerful
configuration syntax.
This is the 2.0 branch of Bird which integrates support for IPv4 and IPv6
into a single branch, and also adds support for the Babel routing protocol.
I assume both. They both need this implemented on the TCP stack level. But as long as that is in place, then it's not much code on the userplane side.
BGP auth works fine with quagga on Linux. Using it on plenty of servers peering with mostly Juniper MXes. I haven't tried it on OpenWrt, but that's "just another" Linux distro so it should definitely work.
I have no experience with BIRD, but I'd be very surprised if it didn't support this as well. It is pretty much a prerequisite for any BGP peering nowadays.