yeah im in the same boat, except im looking for SSH+SFTP, its nice to be able to download files from the server (or upload something) via ssh.
have you looked at ppcssh? i haven't tried it but from what i was told it supports scp, sftp, ssh.. if it indeed supports sftp then it should also utilize private keys.
see:
http://www.irongeek.com/i.php?page=security/ppcssh
if you decide to try it, kindly post your review and opinion about it (i may try it if it works out for you)
cheers.