451 4.4.0 DNS query failed

Problem: Emails to a particular domain were not going through. Emails to all other domains were being received fine.

dcswa-ex01 = Edge Transport
dcswa-ex02 = Hub Transport

domain.com = sending companies email suffix.
example.com = recieving companies email suffix.

Symptoms

Emails sent to example.com are not arriving to the destination. Doriseng.com emails are being passed from the hub transport server dcswa-ex02 to the edge transport server dcswa-ex01 successfully. The emails then sit in the edge transport queue for example.com with the following error:

LastError : 451 4.4.0 DNS query failed



Dcswa-ex01 is resolving DNS correctly for example.com, we can verify this by using the nslookup utility.



The edge transport server Dcswa-ex01 is able to open TCP25 connections to example.com SMTP servers.



The Exchange 2010 connectivity logs shows that the DNS queries exchange is generating for example.com were timing out on the edge transport server:

2010-12-22T02:26:15.708Z,08CD6F05CD2DBD8F,SMTP,example.com,+,DnsConnectorDelivery afb8a1d5-3e6f-4e6b-8bab-17e38b9d7bad;QueueLength=1
2010-12-22T02:27:21.458Z,08CD6F05CD2DBD8F,SMTP,example.com,>,DNS server returned ErrorTimeout reported by 0.0.0.0
2010-12-22T02:27:21.458Z,08CD6F05CD2DBD8F,SMTP,example.com,-,Messages: 0 Bytes: 0 (The DNS query for 'DnsConnectorDelivery':'example.com':'afb8a1d5-3e6f-4e6b-8bab-17e38b9d7bad' failed with error : ErrorTimeout)

Resolution

On the Edge Transport server dcswa-ex01 set the external DNS servers to use for external mail relay.



On the hub transport server dcswa-ex01 configure "Use the External DNS Lookup setting on the transport server" for all send connectors configured for edge sync with dcswa-ex01.



Force an Edge Sync on the hub transport server dcswa-ex02:



Restart the Microsoft Exchange Transport service on the edge transport server to utilize the new DNS settings.

After making this change it took 5-10 minutes for the email to successfully leave the queue!

Looking at the queue for example.com again:



The email was successfully delivered as it no longer resides in the queue.

If we look at the SMTP send log on the edge transport server dcswa-ex01 we can see that the email transferred correctly. One interesting thing I found about example.com was they are digitally encrypting all email communication traffic, I don't see how this would cause DNS to fail but I want to point that out.

2010-12-22T03:59:00.740Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,0,,81.80.156.146:25,*,,attempting to connect
2010-12-22T03:59:01.052Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,1,202.160.101.139:2546,81.80.156.146:25,+,,
2010-12-22T03:59:01.365Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,2,202.160.101.139:2546,81.80.156.146:25,<,220 example.com [ESMTP Server] service ready;DORIS ENGINEERING Email Gateway ok; 12/22/10 04:57:30, 2010-12-22T03:59:01.365Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,3,202.160.101.139:2546,81.80.156.146:25,>,EHLO dcswa-ex01.cloud.dcswa,
2010-12-22T03:59:01.677Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,4,202.160.101.139:2546,81.80.156.146:25,<,250-example.com, 2010-12-22T03:59:01.677Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,5,202.160.101.139:2546,81.80.156.146:25,<,250-SIZE 13631488, 2010-12-22T03:59:01.677Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,6,202.160.101.139:2546,81.80.156.146:25,<,250-8BITMIME, 2010-12-22T03:59:01.677Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,7,202.160.101.139:2546,81.80.156.146:25,<,250 STARTTLS, 2010-12-22T03:59:01.677Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,8,202.160.101.139:2546,81.80.156.146:25,>,STARTTLS,
2010-12-22T03:59:01.974Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,9,202.160.101.139:2546,81.80.156.146:25,<,220 Ready to start TLS, 2010-12-22T03:59:01.974Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,10,202.160.101.139:2546,81.80.156.146:25,*,,Sending certificate 2010-12-22T03:59:01.974Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,11,202.160.101.139:2546,81.80.156.146:25,*,CN=dcswa-ex01,Certificate subject 2010-12-22T03:59:01.974Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,12,202.160.101.139:2546,81.80.156.146:25,*,CN=dcswa-ex01,Certificate issuer name 2010-12-22T03:59:01.974Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,13,202.160.101.139:2546,81.80.156.146:25,*,67E29A29EDE76AAF4BDBC5340D3185F0,Certificate serial number 2010-12-22T03:59:01.974Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,14,202.160.101.139:2546,81.80.156.146:25,*,2A3B56F723AD7056F9372E486B3192E0EF877C6D,Certificate thumbprint 2010-12-22T03:59:01.990Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,15,202.160.101.139:2546,81.80.156.146:25,*,dcswa-ex01;dcswa-ex01.cloud.dcswa,Certificate alternate names 2010-12-22T03:59:02.646Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,16,202.160.101.139:2546,81.80.156.146:25,*,,Received certificate 2010-12-22T03:59:02.646Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,17,202.160.101.139:2546,81.80.156.146:25,*,B6CDD7D2A3CAC50AB653830A828037EC0D0B3901,Certificate thumbprint 2010-12-22T03:59:02.646Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,18,202.160.101.139:2546,81.80.156.146:25,>,EHLO dcswa-ex01.cloud.dcswa,
2010-12-22T03:59:02.958Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,19,202.160.101.139:2546,81.80.156.146:25,<,250-example.com, 2010-12-22T03:59:02.958Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,20,202.160.101.139:2546,81.80.156.146:25,<,250-SIZE 13631488, 2010-12-22T03:59:02.958Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,21,202.160.101.139:2546,81.80.156.146:25,<,250 8BITMIME, 2010-12-22T03:59:02.958Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,22,202.160.101.139:2546,81.80.156.146:25,*,29980,sending message 2010-12-22T03:59:02.958Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,23,202.160.101.139:2546,81.80.156.146:25,>,MAIL FROM: SIZE=5233,
2010-12-22T03:59:03.271Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,24,202.160.101.139:2546,81.80.156.146:25,<,250 Sender OK,
2010-12-22T03:59:03.271Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,25,202.160.101.139:2546,81.80.156.146:25,>,RCPT TO:,
2010-12-22T03:59:03.583Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,26,202.160.101.139:2546,81.80.156.146:25,<,250 Recipient OK,
2010-12-22T03:59:03.583Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,27,202.160.101.139:2546,81.80.156.146:25,>,RCPT TO:,
2010-12-22T03:59:03.896Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,28,202.160.101.139:2546,81.80.156.146:25,<,250 Recipient OK,
2010-12-22T03:59:03.896Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,29,202.160.101.139:2546,81.80.156.146:25,>,RCPT TO:,
2010-12-22T03:59:04.208Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,30,202.160.101.139:2546,81.80.156.146:25,<,250 Recipient OK,
2010-12-22T03:59:04.208Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,31,202.160.101.139:2546,81.80.156.146:25,>,DATA,
2010-12-22T03:59:04.521Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,32,202.160.101.139:2546,81.80.156.146:25,<,354 Start mail input; end with .,
2010-12-22T03:59:05.146Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,33,202.160.101.139:2546,81.80.156.146:25,<,250 OK: <1f3f0b7e00007399@example.com>,
2010-12-22T03:59:05.146Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,34,202.160.101.139:2546,81.80.156.146:25,>,QUIT,
2010-12-22T03:59:05.458Z,EdgeSync - Perth to Internet,08CD6F9DD3D5A300,35,202.160.101.139:2546,81.80.156.146:25,<,221 [ESMTP Server] service closing transmission channel,

There is another workaround that has been documented on the internet to add the external MX servers to the local "hosts" file on the edge transport server. I tested this and it does work however I do not recommend it. If another domain does fail, it will be a manual exercise on a case by case basis.