I had a customer experiencing extensive delays receiving email through the Exchange Transport stack both for internal and external mail relay.
When we ran the Get-Queue or Get-Message cmdlets, the following error was experienced.
Exchange can't connect to the Exchange Transport service on computer "localhost". Verify that the service is started.
Interestingly, sometimes when we ran the Get-Queue and Get-Message command, the correct output was returned so this issue was intermittent.
During the issue, I noticed that the ExchangeTransport.exe service enters a suspended state.
Looking at the Application Logs on the server, I noticed the Transport ESE Databases (the Sender Reputation and IP Filtering) were constantly being rebuilt.
Enabling verbose logging on the Exchange server for the Transport Stack we saw that the issue was being caused by third party transport agents.
Get-EventLogLevel "MSExchangeTransport" | Set-EventLogLevel -Level Expert
Log Name: Application
Source: MSExchangeTransport
Date: 23/04/2018 3:43:10 PM
Event ID: 10003
Task Category: PoisonMessage
Level: Error
Keywords: Classic
User: N/A
Computer: COMPUTERNAME
Description:
The transport process failed during message processing with the following call stack: System.Runtime.InteropServices.COMException (0x800706BE): Creating an instance of the COM component with CLSID {9DE1008F-1451-4E2E-8767-A1DAF9BE3560} from the IClassFactory failed due to the following error: 800706be The remote procedure call failed. (Exception from HRESULT: 0x800706BE).
at ContentSecurity.ExchangeAgents.MailRouteItemSourceContext.processItem(String sender, ArrayList recipients, IStream pRawData, UInt32& pFlags)
at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.ProcessMailRouteMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e, Boolean& bDeleteFromSource)
at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.SubmittedMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e)
at Microsoft.Exchange.Data.Transport.Routing.RoutingAgent.Invoke(String eventTopic, Object source, Object e)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
When we ran the Get-Queue or Get-Message cmdlets, the following error was experienced.
Exchange can't connect to the Exchange Transport service on computer "localhost". Verify that the service is started.
Interestingly, sometimes when we ran the Get-Queue and Get-Message command, the correct output was returned so this issue was intermittent.
During the issue, I noticed that the ExchangeTransport.exe service enters a suspended state.
Looking at the Application Logs on the server, I noticed the Transport ESE Databases (the Sender Reputation and IP Filtering) were constantly being rebuilt.
- edgetransport (34164) Sender Reputation Database: The database engine is initiating recovery steps.
- edgetransport (34164) Sender Reputation Database: The database engine has begun replaying logfile C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\SenderReputation\trn.log.
- edgetransport (34164) Sender Reputation Database: The database engine has successfully completed recovery steps.
- edgetransport (34164) Sender Reputation Database: The database engine started a new instance (2). (Time=0 seconds)
Enabling verbose logging on the Exchange server for the Transport Stack we saw that the issue was being caused by third party transport agents.
Get-EventLogLevel "MSExchangeTransport" | Set-EventLogLevel -Level Expert
Log Name: Application
Source: MSExchangeTransport
Date: 23/04/2018 3:43:10 PM
Event ID: 10003
Task Category: PoisonMessage
Level: Error
Keywords: Classic
User: N/A
Computer: COMPUTERNAME
Description:
The transport process failed during message processing with the following call stack: System.Runtime.InteropServices.COMException (0x800706BE): Creating an instance of the COM component with CLSID {9DE1008F-1451-4E2E-8767-A1DAF9BE3560} from the IClassFactory failed due to the following error: 800706be The remote procedure call failed. (Exception from HRESULT: 0x800706BE).
at ContentSecurity.ExchangeAgents.MailRouteItemSourceContext.processItem(String sender, ArrayList recipients, IStream pRawData, UInt32& pFlags)
at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.ProcessMailRouteMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e, Boolean& bDeleteFromSource)
at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.SubmittedMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e)
at Microsoft.Exchange.Data.Transport.Routing.RoutingAgent.Invoke(String eventTopic, Object source, Object e)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()