Introduction

In the modern messaging world Transport Layer Security has become the norm and given the ease at which Microsoft Exchange is able to achieve this out of the box, organisations quite rightly demand this when they are on Coexistence mode between IBM Domino and Microsoft Exchange.

Up to Domino 8.5.x Domino does not actually do genuine TLS – it actually does HTTPS v3 for Message Transport. Apparently Domino 9 will do TLS natively.

Below I have outlined the steps to Configure Domino 8.5.3 and Exchange 2010 for TLS.

Domino Server Configuration

• Create Domino CSR
• Create Key Ring
• Create Certificate Request
• Sent to CA
• Install trusted Root Certificate into Key Ring
• View & Edit Key Rings
• View Certificate Request Log
• Install Certificate Into Key Ring
• Configure Domino
• Enable SSL on Domino
• Enable TLS for SMTP Outbound mail
• Enable TLS for SMTP Inbound mail
• Configure notes.ini
• Replicate changes to Domino Environment
• Restart The Router and SMTP tasks

Exchange Server Configuration

• Configure Receive Connectors
• CHECK Config
• Configure Send Connectors
• CHECK Config

Domino Server Configuration

1. Create Domino CSR

1. Create Key Ring

2. Create Certificate Request

—–BEGIN NEW CERTIFICATE REQUEST—–
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
—–END NEW CERTIFICATE REQUEST—–

Sent to CA

3. Install trusted Root Certificate into Key Ring

*

—–BEGIN CERTIFICATE—–

***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
***************************************************************************
—–END CERTIFICATE—–

4. View & Edit Key Rings

5. View Certificate Request Log

6. Install Certificate Into Key Ring

Configure Domino

Enable SSL on Domino

• Create SSL Cert Request, send to EnTrust, Retrieve CER file, Complete Cert Request
  • Save SSL key file name file to Domino Data directory
• Server Document: Server Ports, Internet Ports, SSL key File Name, set SSL ciphers to RC$ 128-bit only (MD5 and SHA-1)
• Restart HTTP Service
• Don’t need HTTPS running for SSL

Enable TLS for SMTP Outbound mail

• Server Document: Server Ports, Internet Ports, Set TCPIP port status to Negotiated SSL

Enable TLS for SMTP Inbound mail

• Server Configuration Document: Router\SMTP, Advanced, Commands and Extensions set SSL negotiated over TCP/IP port: to ENABLED

Configure notes.ini

Check /or set the following notes.ini settings

HTTPJVMMaxHeapSize=64M

• HTTPJVMMaxHeapSizeSet=1
• HTTPDisableMethods=TRACE
• SSL_USE_ADDSESSION2=1
• SSL_SESSION_SIZE=8192
• SSLCipherSpec=0405

Debug Settings
If Anything Goes wrong, use these:

;SMTPClientDebug=0
;SMTPDebug=0
;SMTPDebugIO=0
;SMTPSaveOutboundToFile=1
;SMTPSaveImportErrors=0

;HTTPDebugLogDirectpry=<path>
;HTTPEnabledThreadDebug=1
;HTTPEnablePostDataLogging=1
;HTTPEnableResponseContentLogging=1
;DEBUG_SSL_ALL=1
;TraceSSLHandshake=1
;ReportSSLHandshakeErrors=1

Replicate changes to Domino Environment

• Replicate changes to Domino affected server if necessary

Restart The Router and SMTP tasks

• Stop and start the Domino Router and SMTP tasks

Exchange Server Configuration

Configure Receive Connectors

Get-ReceiveConnector | ? { $_ -like ‘*domino*’ } | Set-ReceiveConnector -RequireTLS $true -ProtocolLoggingLevel Verbose

Get-ReceiveConnector | ? { $_ -like ‘*domino’ } | Set-ReceiveConnector -AuthMechanism Tls -PermissionGroups AnonymousUsers

Get-ReceiveConnector | ? { $_ -like ‘*domino’ } | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient”

CHECK Config

Get-ReceiveConnector | ? { $_ -like ‘*domino*’ } | FL Identity,RequireTLS,ProtocolLoggingLevel

Configure Send Connectors

Get-SendConnector | ? { $_ -like ‘*domino*’ } | Set-SendConnector -TlsAuthLevel $null -IgnoreSTARTTLS $false -RequireTLS $true -ProtocolLoggingLevel Verbose

CHECK Config

Get-SendConnector | ? { $_ -like ‘*domino*’ } | FL Identity,TlsAuthLevel,RequireTLS,IgnoreSTARTTLS,ProtocolLoggingLevel

Restart Transport Service

Restart-Service MSExchangeTransport