Customer with a single Exchange 2010 completely down. The following error was spammed throughout the event log:
Log Name: Application
Source: MSExchange ActiveSync
Date: 18/01/2017 7:31:46 PM
Event ID: 1016
Task Category: Server
Level: Error
Keywords: Classic
User: N/A
Computer: EXCHANGE2010
Description:
Exchange ActiveSync has encountered repeated failures when it tries to access data on Mailbox server [EXCHANGE2010.domain.local]. It will temporarily stop making requests to the Mailbox server for [60] seconds to reduce load on that server. This delay may occur if the Mailbox server is overloaded. If this event is logged frequently, review the Application log on this server and the Mailbox server noted above for other events that could indicate the root cause of performance problems.
Additional information:
"serverFQDN=EXCHANGE2010.domain.local
Error 0:
ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=EXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=User.
Exception level: 0
Exception stack trace: at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
......
Lid: 13720 dwParam: 0x6D9 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam: 0x6D9 Msg: EEInfo: NumberOfParameters: 4
Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[2]: Long val: -545057711
Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[3]: Long val: 382312662
Lid: 45169 StoreEc: 0x824
Lid: 44273
Lid: 59431 EMSMDB.EcDoConnectEx called [length=140]
Lid: 34855 EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
Lid: 59505 StoreEc: 0x824
Lid: 25964 StoreEc: 0x824
Lid: 36081
Lid: 52465 StoreEc: 0x80040115
Lid: 60065
Lid: 33777 StoreEc: 0x80040115
Lid: 59805
Lid: 52209 StoreEc: 0x80040115
Lid: 56583
Lid: 52487 StoreEc: 0x80040115
Lid: 19778
Lid: 27970 StoreEc: 0x80040115
Lid: 17730
Lid: 25922 StoreEc: 0x80040115
Exception level: 1
Exception stack trace: at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 1:
ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=sfso.
Exception level: 0
Exception stack trace: at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
......
Lid: 13720 dwParam: 0x6D9 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam: 0x6D9 Msg: EEInfo: NumberOfParameters: 4
Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[2]: Long val: -545057711
Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[3]: Long val: 382312662
Lid: 45169 StoreEc: 0x824
Lid: 44273
Lid: 59431 EMSMDB.EcDoConnectEx called [length=133]
Lid: 34855 EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
Lid: 59505 StoreEc: 0x824
Lid: 25964 StoreEc: 0x824
Lid: 36081
Lid: 52465 StoreEc: 0x80040115
Lid: 60065
Lid: 33777 StoreEc: 0x80040115
Lid: 59805
Lid: 52209 StoreEc: 0x80040115
Lid: 56583
Lid: 52487 StoreEc: 0x80040115
Lid: 19778
Lid: 27970 StoreEc: 0x80040115
Lid: 17730
Lid: 25922 StoreEc: 0x80040115
Exception level: 1
Exception stack trace: at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 2:
ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=sfso.
Exception level: 0
Exception stack trace: at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
......
Lid: 13720 dwParam: 0x6D9 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam: 0x6D9 Msg: EEInfo: NumberOfParameters: 4
Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[2]: Long val: -545057711f
Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[3]: Long val: 382312662
Lid: 45169 StoreEc: 0x824
Lid: 44273
Lid: 59431 EMSMDB.EcDoConnectEx called [length=133]
Lid: 34855 EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
Lid: 59505 StoreEc: 0x824
Lid: 25964 StoreEc: 0x824
Lid: 36081
Lid: 52465 StoreEc: 0x80040115
Lid: 60065
Lid: 33777 StoreEc: 0x80040115
Lid: 59805
Lid: 52209 StoreEc: 0x80040115
Lid: 56583
Lid: 52487 StoreEc: 0x80040115
Lid: 19778
Lid: 27970 StoreEc: 0x80040115
Lid: 17730
Lid: 25922 StoreEc: 0x80040115
Exception level: 1
Exception stack trace: at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
errorCount=3, backingOff=True".
I know there are a few causes of this error however in my instance it was due to Microsoft Exchange Replication Service not started. The service had been misconfigured to run as a service account with invalid credentials.
Setting it back to local system resolved the issue and getting the customer back online.
The Active Sync service queries "Active Manager" which is part of the Exchange Replication Service to determine where the users active mailbox copy in a DAG resides - and still follows this model even if your not using DAG's in your environment. Hence if Active Manager is not available, no one can locate their Active Mailbox.