Diagnosing Performance Issues in Exchange

This blog post looks into diagnosing performance issues with Exchange 2010.  Like most applications, Exchange performance can be effected by numerous bottlenecks within a system including:

• Storage bottlenecks
• CPU Utilisation
• Anti-Virus applications
• Memory shortage
• Rouge applications installed on the Exchange server
• Global Catalog whether the Global Catalog servers are to busy or Exchange is using a Global Catalog server in a remote site due to AD sites and services being configured incorrectly.
• An individual user causing high amounts of load on the Exchange infrastructure, not so common now days due to Exchange Throttling Policies, however if the default Throttling Policy is modified always a possibility.
• Network traffic, who knows a broadcast storm may be occurring on your network?
• Internet Spam or Denial of Service (DOS) attacks, your Exchange environment may be trying to process large quantities of inbound SMTP email is known as junk email.  As a result the SMTP queues may be building up.
• Backup applications running through the day causing significant server load.
• Monitoring applications with to many data collector sets causing significant server load.

Exchange Server User Monitor (ExMon)

The Exchange Server User Monitor also known as ExMon amongst Exchange professionals has been a great tool from Microsoft for tracking user session based utilisation of an Exchange server.  For a summary of ExMon along with a screenshot please visit the following blog post:

http://clintboessen.blogspot.com/2009/07/exchange-user-monitor.html

Unfortunately with the introduction of the RPC Client Access Service, ExMon now experiences difficulties in an Exchange 2010 environment.  If you have an Exchange 2010 mailbox server not running the Client Access Role, ExMon should work without problems.  However if the Client Access and Mailbox server roles are installed on the Exchange server, your going to experience:

Log Name:      Application

Source:        Application Error

Date:          16/07/2012 11:21:51 AM

Event ID:      1000

Task Category: (100)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      Troy-Exch-2010

Description:

Faulting application name: ExMon.exe, version: 14.2.247.0, time stamp: 0x4e929c00

Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec49b8f

Exception code: 0xc0000005

Fault offset: 0x00033ab5

Faulting process id: 0x216c

Faulting application start time: 0x01cd63021e952eca

Faulting application path: C:\Program Files (x86)\Exchange User Monitor\ExMon.exe

Faulting module path: C:\Windows\SysWOW64\ntdll.dll

Report Id: 61f27c3a-cef5-11e1-acb2-005056880002

According to Xiu Zhang from Microsoft, this bug is by design when running ExMon on an Exchange 2010 server that has Client Access and Mailbox on the same server.

http://social.technet.microsoft.com/Forums/en-US/exchangesvrgeneral/thread/4ebba22b-d280-44dd-af06-c8e40257ad7c/

Exchange Trouble Shooting Assistant

Exchange Troubleshooting Assistant known as ExTRA can identify potential performance issues on an Exchange server such as mail flow and database mounting issues on computers running Microsoft Exchange Server. The tool automates specialized troubleshooting steps for identified symptoms.

Network Monitor

To examine potential performance issues related to high network utilization, Microsoft Network Monitor can be utilised to view the traffic on the network interface.  Network Monitor will allow Administrators to view the following information:

• The source address of the computer that sent a frame to the network (this address is a unique hexadecimal (or base-16) number that identifies that computer on the network).
• The destination address of the computer that received the frame.
• The protocols used to send the frame.
• The data or a portion of the message being sent.

Performance Monitor

Performance Monitor is one of the best tools monitoring performance issues as it provides an insight into potential storage bottlenecks, CPU utilisation, memory shortages, RPC MAPI latency, Exchange Queues, Global Catalog traffic and much more.  For ease of configuration, Microsoft have created a PowerShell script known as ExPerfwiz which can be downloaded from the following site:

http://code.msdn.microsoft.com/ExPerfwiz

This script automatically creates a Data Collector Set which captures all the important information required for diagnosing Exchange performance issues.  

To run the Script you must first digitally sign the script or set the ExecutionPolicy to Unrestricted.  This can be down using the following powershell command:

Set-ExecutionPolicy Unrestricted

Run the script by navigating to the script directory and executing the following powershell command.

.\ExPerfwiz.ps1

This creates a data collector set called Exchange_perfwiz and samples the data every 30 seconds.  This data can be generated to a report which can be viewed either in a graph or in a report.  To get to the Data Collector Set run perfmon.msc from Start --> Run.  Create a report from the Data Collector Set by right clicking on it and selecting Create Report.

Your also able to view components from the data collector set through line and bar graphs.

Microsoft Whitepaper

Microsoft has put together a whitepaper for diagnosing Exchange performance issues.  Whilst written for Exchange 2003 many components are still relevant today.  I encourage you to have a read, it can be downloaded from the following link.

http://www.microsoft.com/en-us/download/confirmation.aspx?id=19125

Directory Services Performance

To troubleshoot performance issues related to Global Catalog two performance counters are of relevance:

MSExchange ADAccess

• LDAP Read Time
• LDAP Search Time

Both of these counters should never exceed 100ms and for normal activity should remain under 50ms.

Another blog post which is very helpful for additional information is the one below by Dougg Owans:

http://blogs.msdn.com/b/douggowans/archive/2007/01/02/a-very-quick-guide-to-monitoring-the-performance-of-your-exchange-server.aspx