Well, having access to SSH from the outside world has saved me several times. Especially when I'm somewhere that I wouldn't have a key with me.
Meh, pick a strong password, rotate it regularly and your 90% there.
Create a non-privileged user, install sudo, disable root logins, and you're 99% of the way there.
Edit: I personally prefer OpenSSH over dropbear for both perceived security, as well as the ability to better handle command-bound keys and other advanced SSH techniques.
In my opinion, that last 1% isn't worth the time of either setting it up, or the frustration when you can't get access to your systems when you really need to.
(I also don't believe that using a non-standard port does much more than reducing log spam, if you've followed even the 90% recommendation.)
There are other things that will gain you more in terms of security for that time that, in my opinion, are more valuable.
(that's a link to my recent post on your other thread)