Usage of Git in OpenWRT with dropbear

I'm trying to version control my whole /etc folder but I'm having trouble with ssh access. Help appreciated.
The git repo resides in a gitea server, I don't know if this could be an issue but maybe worth mentioning. Ssh login with rsa key works flawlessly to another computer, although I have to manually select the identify file. In my attempts to ssh to the git server, from other hosts I get a successful authentication and then immediately I'm logged out. When trying the same from OpenWRT I get connection refused. This is, to clarify using dropbear and not git at this stage (just to verify the authentication). I am specifying the identify file.
When using Git I am guessing the identify file is not used by default as I'm assuming dropbear is being used in the background. I have tried setting the Git environment variables both for the system with GIT_SSH_COMMAND="ssh -i /root/.ssh/id_rsa" and for the repo with git config core.sshCommand "ssh -i /root/.ssh/id_rsa -F /dev/null"