Messages where Forced TLS policy rules are in place can fail for any number of reasons.  The most typical reason has to do with certificates that are issued from Certificate Authorities that are not trusted by the FOPE service.  In order to test connections to SMTP TLS MTA's you can make use of Win32 OpenSSL s_client functionality.

Downloads for Win32 OpenSSL can be found at http://www.slproweb.com/products/Win32OpenSSL.html
The man page for s_client for openssl can be found at http://www.openssl.org/docs/apps/s_client.html

To test connectivity to a TLS enabled SMTP server run the following command

openssl s_client -connect mail.messaging.microsoft.com:25 -showcerts -starttls smtp

 In the example shown we are establishing a connection to the FOPE inbound hostname on port 25 and establishing a TLS connection and showing the certificates back to the client that the server provides.

The return text will be in the following format:

• error regarding local issuer certificate (ignore we aren't using a local ca bundle in the example and doesn't matter)
• Certificate Chain
  • First Certificate in chain i.e. the certificate issued to/installed on the server we are connecting to
    • Number indicating location in chain for first one will be 0
    • s: subject name that this cert is issued to
    • i: issuer that issued this certificate
    • The actual certificate, this lies between the ----BEGIN CERTIFICATE---- and ----END CERTIFICATE-----
  • Second Certificate in chain i.e. the CA that issued the first certificate this could either be a root CA or possibly an intermediary
    • Number indicating location in chain for the second will be 1
    • s: subject name that this cert is issued to
    • i: issuer that issued this certificate
    • The actual certificate, this lies between the ----BEGIN CERTIFICATE---- and ----END CERTIFICATE-----
• Summary with Server certificate subject and issuer subject
• Line regarding whether client certificates were used (we are not in this example)
• Line regarding SSL handshake data
• Cipher Suite chosen for the connection

 From this you have most of the data you need to troubleshoot connections some common scenarios for failure are:

• Self-issued certificates. Check the "issuer" certificate if this is issued by the same server or something that looks like it is local to the recipient ie ca.contoso.com just issued this to mail.contoso.com and the chain does not go any higher than ca.contoso.com then that is a problem.  You have to have a certificate issued that chains up to a FOPE trusted root CA in order for the TLS connection to occur.
• Expired certificates. To find the certificate lifetime you will need to copy all the data for the certificate between --BEGIN CERTIFICATE--- and ---END CERTIFICATE-- and save this as a .crt file.  Once you have this .crt file you can open this in windows and look at the Valid from and to dates to see if this is possibly an issue with the certificate being expired.
• Certificates issued from untrusted root CA's.  This is pretty much the same as the Self-issued certiificates issue you need to examine the root CA issuing the certificate and make sure it is on the list of  FOPE trusted root CA's.
• Complete certificate chain not offered from the SMTP TLS server. Example would be:
  • Certificate chain
     0 s:/C=US/ST=Statename/L=County/O=ORGNAME./OU=INTERNALORGNAME/OU=Terms of use at www.ver
    isign.com/rpa (c)05/CN=mail.contoso.com
       i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
  • Since VeriSign Class 3 Secure Server CA - G2 is not an explicitly trusted root Certificate by the FOPE service this will fail.  The TLS spec RFC 5246 basically states the following in 7.4.2
    • "Each following certificate MUST directly certify the one preceding it.  Because certificate validation requires that root keys be distributed
            independently, the self-signed certificate that specifies the root certificate authority MAY be omitted from the chain, under the
            assumption that the remote end must already possess it in order to validate it in any case.
  • So it's a little vague but basically the certificate chain needs to be shown all the way up to the self-signed certificate or Root.  Typically non-root or intermediary CA's are not part of the trusted roots.
  • A good certificate chain would look like the followingCertificate chain note the 2nd cert in the chain that indicates the intermediary does chain up to what is normally a trusted root by most crypto systems.
     0 s:/C=US/ST=Statename/L=County/O=ORGNAME./OU=INTERNALORGNAME/OU=Terms of use at www.ver
    isign.com/rpa (c)05/CN=mail.contoso.com
       i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
    1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
    i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority - G2/OU=(c) 1998 VeriSign, Inc. - For authorized use only/OU=VeriSign Trust Network