Okay... The problem is explained quite simple. We are using the smtps protocol. That means, that the custom settings need to me "mail.smtps.***" and not "mail.smtp.***"... Now the mails are send over SSL and with authentication :) I wish the IBM documentation would have a little hint for this...