Hello, if there's no way to change the config so that it needs less RAM maybe create a Linux swap file somewhere on a file system (USB HDD or something). That should get mysql_install_db going, so that it can finish. Hopefully the server doesn't need to swap once it's running (or at least not that much).
In my experience MySQL is still a better solution than MariaDB. I have not dug through all the history and change logs, but on a limited RAM system I have always seen better performance with MySQL. That's just my opinion, as mentioned try enabling some swap to let the operation finish and see how things go. For me I was able to get MariaDB running with swap on a USB drive, but the performance was horribly slow and for basic database usage that had worked fine with MySQL 3 days of fighting just to get that far was enough for me.
You can still build MySQL on modern OpenWRT, you just need to port the make file and compile it yourself.
FYI - when I tried to compile against trunk using the last makefile I have, I get this error:
configure: error: Could not find system readline or libedit libraries
Use --with-readline or --with-libedit to use the bundled
versions of libedit or readline
Makefile:185: recipe for target '/build_dir/target-mipsel_24kc_musl/mysql-5.1.73/.configured_68b329da9893e34099c7d8ad5cb9c940' failed