If you read http://www-10.lotus.com/ldd/lcwiki.nsf/dx/Sending_mail_from_a_dedicated_mail_server_ic301
, you will see you have to define other custom properties for SMTPS : mail.smtp.socketfactory.port, mail.smtp.socketfactory.class and mail.smtp.socketFactory.fallback. Maybe this is the clue.
If this is not solved after adding these, try to enable SMTP debug (ISC > Resources > Mail Sessions > lcnotification, check "Enable debug mode").
I hope this will help you.