So I was continuing playing with IPv6 and found that I am not able to ssh to my oracle cloud instance via ssh (port is not standard 22, I changed it for security reasons)
Then I disable IPv6 completely in openwrt following this, then I was able to ssh.
Any logical explanation for this?