Third Party Transport Agents delaying Emails and Crashing the Transport Databases

I had a customer experiencing extensive delays receiving email through the Exchange Transport stack both for internal and external mail relay.

When we ran the Get-Queue or Get-Message cmdlets, the following error was experienced.

Exchange can't connect to the Exchange Transport service on computer "localhost". Verify that the service is started.

Interestingly, sometimes when we ran the Get-Queue and Get-Message command, the correct output was returned so this issue was intermittent.

During the issue, I noticed that the ExchangeTransport.exe service enters a suspended state.

Looking at the Application Logs on the server, I noticed the Transport ESE Databases (the Sender Reputation and IP Filtering) were constantly being rebuilt.

• edgetransport (34164) Sender Reputation Database: The database engine is initiating recovery steps. 
• edgetransport (34164) Sender Reputation Database: The database engine has begun replaying logfile C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\SenderReputation\trn.log. 
• edgetransport (34164) Sender Reputation Database: The database engine has successfully completed recovery steps. 
• edgetransport (34164) Sender Reputation Database: The database engine started a new instance (2). (Time=0 seconds) 

Enabling verbose logging on the Exchange server for the Transport Stack we saw that the issue was being caused by third party transport agents.

Get-EventLogLevel "MSExchangeTransport" | Set-EventLogLevel -Level Expert

Log Name:      Application
Source:        MSExchangeTransport
Date:          23/04/2018 3:43:10 PM
Event ID:      10003
Task Category: PoisonMessage
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTERNAME
Description:
The transport process failed during message processing with the following call stack: System.Runtime.InteropServices.COMException (0x800706BE): Creating an instance of the COM component with CLSID {9DE1008F-1451-4E2E-8767-A1DAF9BE3560} from the IClassFactory failed due to the following error: 800706be The remote procedure call failed. (Exception from HRESULT: 0x800706BE).
   at ContentSecurity.ExchangeAgents.MailRouteItemSourceContext.processItem(String sender, ArrayList recipients, IStream pRawData, UInt32& pFlags)
   at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.ProcessMailRouteMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e, Boolean& bDeleteFromSource)
   at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.SubmittedMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e)
   at Microsoft.Exchange.Data.Transport.Routing.RoutingAgent.Invoke(String eventTopic, Object source, Object e)
   at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session)
   at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Disabling the Exchange Transport Agents for GFI and restarting the MSExchangeTransport service resolved the issue.

If anyone experiences these symptoms, please make sure you check your third party Transport Agents!