Weird Exchange Performance Issues

This post is interesting as it is one of the weirdest Exchange 2010/2013 performance problems I have seen in my career for a while.  In writing this blog post, hopefully other organisations with the same bizarre issue can find this post hopefully resolve their Exchange performance problem.

This post will cover the following topics:

• Symptoms of the problem
• Troubleshooting Steps
• Issue Resolution

My customer was running a single Exchange 2010 SP3 CU3 multi-role server however it needs to be noted this issue can also occur for Exchange 2013 based on what I have read.

Symptoms

RPC Response Times

The Outlook RPC Response times we were seeing from clients connecting on the same subnet as the Exchange server were abnormally high.  We were seeing response times in the 10,000 - 20,000 facinity consistently across multiple clients.  Response times should generally be below 50 and sometimes higher for remote clients in branch offices or clients connecting using RPC over HTTP as shown in the following screenshot.

Autodiscover Requests Timing Out

Autodiscover requests were randomly timing out as the Exchange server was taking to long to respond.  The following error was being generated in the Test E-mail AutoConfiguration tool:

GetLastError = 12002; httpStatus=0
Autodiscover internet timeout against URL https://exchangeserver.com/autodiscover/autodiscover.xml

Mail Tips Issues

Users were complaining Mail Tips was failing across multiple outlook clients.  When composing new emails instead of receiving a mail tip, in the same area where the mail tips are displayed users would receive an error indicating there was a problem receiving mail tips from the server.  This is due to the server not responding to the mail tips request in a timely fashion.

Outlook Web App Performance Problems

Loading the Outlook Web App page and logging in was extremely slow taking approximately 45 seconds for the to completely load.  Navigating the mailbox, composing new emails and accessing features such as Exchange Control Panel were also extremely slow and barely usable.  Sometimes the server did not respond fast enough and the web page would simply timeout.

Microsoft Outlook Load Times

Launching the Microsoft Outlook client was extremely slow and took approximately 21 seconds across all clients instead of the snappy 2-3 second load time.

Troubleshooting Performed

I was contracted to come in and resolve the performance problems for the customer.  During my initial 4 hours troubleshooting the performance issues I went through and looked at a large number of items which generally contribute to Exchange performance issues.

General Server Performance

The general server performance was looked at including items such as physical disk, memory, cpu and network utilisation.  Items examined included:

• Number of hard page faults
• Disk queue length
• Disk Time
• CPU Utilisation
• Memory utilisation
• Network number of packets sent/received
• Network bandwidth utilisation 

A few other related counters in performance monitor were also examined, all within normal readings.

Network Stack Tweaking

For testing the following changes were made to the Windows network stack:

netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled
netsh int tcp set global taskoffload=disabled
netsh int tcp set global autotuninglevel=disabled

These had no impact and changes were reverted.

Exchange User Monitor

The Exchange User Monitor was ran on the server to verify that no significant load was placed on the server from one user session from virus or bad Outlook profile.  Readings from ExMon were normal.

Anti-Virus or Third Party Applications

AntiVirus or Third Party Applications were investigated.  No real time file scanning AV is installed on the Exchange server (as recommended by Microsoft).

The only third party application installed is Exclaimer Outlook Signature which is found not to be causing an issue.

Network Connectivity Test

A network connectivity test was performed between the Exchange server and a client on the same subnet suffering performance issues using a bandwidth measuring tool known as iPerf.  The network connectivity test showed the client was almost able to achieve gigabit speeds to the Exchange 2010 server over the local network segment.

IIS Application Pools

IIS Application pools were flushed with an IISReset to rule out App Pool Recycling and memory usage/limit/leaking related issues which can significantly impact performance of web based applications on an IIS server.  In the event it was related to an IISApp pool issue, an IISReset would generally restore performance for a short period of time which did not happen.

Certificate CRL Checking

When using public certificates, the service in which the certificate is associated with must be able to contact the public certificate revocation list.  In the event it cannot, this can cause web based applications to be sluggish as we are waiting for CRL timeouts to occur on the backend.  I tested this from the Exchange server and was able to contact the CRL lists meaning there was no firewall blocking this communication.

Exchange Troubleshooting Assistant

The Microsoft Exchange Troubleshooting Assistant also known as ExTRA can be used to identify performance related issues on an Exchange 2010 server.  This was run and the report flagged nothing of interest.

Process Monitor

Microsoft system internals process monitor (procmon.exe) was run on the Exchange 2010 server to identify application loops or unwanted activity which may be related to slow websites loading.  Output from procmon was normal.

Exchange Performance Counters

Some core Exchange Performance Counters were checked on Exchange including things such as:
- MSExchangeIS\RPC Requests (should be below 30)
- MSExchangeIS\RPC Averaged Latency (should be below 50ms)

These were all normal. 

Exchange Event Log Level

I increased the event log level for OWA for testing purposes.  No problems could be identified relating to performance problems from OWA Event Logs.

IPv6 Disabled

IPv6 can be known to cause performance issues with Microsoft Exchange.  As a result for testing I advised the customer to turn of IPv6 by disabling it on the network interface adapter on the Exchange 2010 server and by putting in place the DisabledComponents registry DWORD value as required by Microsoft KB929852.  This had no effect and the issue continued to occur.

Second Exchange 2010 Server

After all troubleshooting performed above I advised my customer to build a new Exchange 2010 server as it was they had a simple environment with only a single Exchange 2010 server.  We performed the following steps:

• Performed a fresh install of Windows 2008 R2 with all latest updates
• Installed Exchange 2010 with SP1 multi role deployment
• Installed Exchange 2010 SP3 update
• Installed Exchange 2010 SP3 CU3
• Performed minor core configuration tasks such as configuring public folder replication, OAB Distribution, Enabling Outlook Anywhere, Configuring the Digital Certificate and a few other things.
• Moved a few test users to the new Exchange 2010 server for testing purposes.

The test users experienced the exact same performance issues on the new server as those on the old server despite it being a fresh installation of Windows Server and Microsoft Exchange.

Resolution

My customer stumbled across the following webpage where another company was having a similar issue with their Exchange server performance:

http://www.outlookforums.com/threads/91195-exchange-2013-outlook-2010-slow-startups/

This other company with the similar issue resolved it by Disabling IPv6.  Instead of manually disabling IPv6 like I did in my step above instead used the Microsoft FitIt tool to disable IPv6 which is available on the same knowledge base article, KB929852.

http://support.microsoft.com/kb/929852

After running the Microsoft FixIt tool, the customer expressed to me that the issue is resolved and Exchange is no longer under performing.

What puzzles me however is the Microsoft FixIt tool as documented simply creates the DisabledComponents registry key, something which I created manually during troubleshooting.  This is documented under "For more information" in the manual section of KB929852, the same KB I used to create my manual registry key.  Perhaps it makes another change which I am unaware of and is not documented on the knowledge base article?  It might be worth while testing this theory by running an application such as RegSnap before and after the tool is run to identify exactly what changes it has made.

The other thing which is important to note, Exchange generally should be deployed with IPv6.  I have many customers running Exchange 2010/2013 servers with IPv6 enabled and no performance issues.  This particular customer did express to me that around the time the issue started occurring, they also had an upgrade on their core switch.  This network infrastructure change should not be ruled out as possibly effecting the performance of clients connecting to Exchange.

This was definitely one of the most puzzling problems I have dealt with in a while.  If you do have the same problem as the symptoms expressed in this post, please do try running the Microsoft FixIt tool as documented above.  If this also resolves your problem, please do comment as I am looking forward to hearing your feedback.

Thanks for reading.

Making DNS GlobalNames Zone Available to Other Forests

In large complex forests with multiple domains, the DNS Suffix search list can be extensive and can result in DNS name resolution failing.  Companies with a large number of domains often used WINS to provide single name resolution across all domain infrastructure along side DNS.  This is because it was not possible to provide a single name resolution technology which you could span across your entire Active Directory infrastructure using DNS.

Starting from Windows Server 2008, Microsoft introduced a new DNS concept known as GlobalNames which was a move to permanently decommission the legacy WINS technology to enable single name resolution.

When deploying GlobalNames you first enable it on all DNS servers in your domain using dnscmd.exe:

dnscmd ServerName /config /enableglobalnamessupport 1

You then create the GlobalNames zone file and ensure it replicates to all DNS servers in your Forest (best practice) or alternatively create a separate application partition to hold the GlobalNames zone which gives you the flexibility of choosing which DNS servers in your forest hold this zone.  The whole point of GlobalNames is generally to allow DNS servers in other Active Directory domains (within the same forest) to all share the single name resolution hence removing the need for complex DNS Suffix Search Lists.

But what if you want DNS Servers in other forests to use the GlobalNames zone?

This is a little more complicated as we cant simply replicate the GlobalNames DNS zone to DNS servers in other domains due to the fact it is another Active Directory forest, it runs under a complete different set of security identifies and does not adhere to Ticket Granting Tickets (TGTs) handed out by Kerberos KDC's from another forest.

What Microsoft has done to allow other forests to share a GlobalNames DNS zone is to provide companies the ability to create a service location (SRV) resource records in the remote forests DNS zone.  This is done as follows:

_globalnames._msdcs.remotedomain.local which must point to an FQDN of a DNS server that hosts the GlobalNames zone in the main forests.

Fore more information please refer to the following TechNet article:

http://technet.microsoft.com/en-us/library/cc731744.aspx

I hope this post has been helpful.

What is the krbtgt account in Active Directory?

The KRBTGT account is a user account which resides inside the domain users container in every Active Directory domain.  This service account is critical and is required by the Kerberos Key Distribution Centre (KDC).

This account must never be deleted.
This account must never be renamed.

Before reading this post, I hope you have an understanding of how the Kerberos authentication protocol works.  If not I encourage you to watch the following video entitled "How Kerberos Works" by Don Jones, a CBT Nuggets instructor.  This can be found on the following link:

https://www.youtube.com/watch?v=kp5d8Yv3-0c

From watching this video you will understand that all users and devices on a network must be assigned a Ticket Granting Ticket (TGT) from the KDC which then enables them the ability to access resources on the network.

The KRBTGT account is critical to this process as the KDC uses the password derived from the KRBTGT account to encrypt each TGT assigned to users and devices on the network.  All KDC servers on the network have the KRBTGT account password because all domain controllers have this account.  The KRBTGT account is created upon the promotion of the first domain controller in a new Active Directory domain.

If you haven't already guessed, KRBTGT stands for "Kerberos Ticket Generating Ticket Account".

Read Only Domain Controllers

There are a few things to be aware of with the KRBTGT account when dealing with Read Only Domain Controllers.  RODC's also act as a KDC for branch offices and as a result require a KRBTGT account.  However, RODC's do not contain the passwords of all accounts in an Active Directory domain, only passwords specified by an administrator defined in the Password Replication Policy (PRP) - for more information see http://technet.microsoft.com/en-us/library/cc730883.aspx

Now because the password associated with the KRBTGT account is so sensitive, we do not want this residing at branch sites as the whole point of an RODC is to implement it when physical security to the server is low.  To ensure that the KRBTGT of a compromised RODC can't be leveraged to request tickets to other domain controllers, each RODC has a special local KRBTGT account. This account has the format KRBTGTXXX, where "XXX" is a string of random numbers. This random string uniquely identifies the RODC and is generated when an RODC is installed.

The accounts generated for each RODC appear in the users container on all Active Directory domain controllers in the domain.  As a result all writeable domain controllers also keep a copy of the KRBTGT password hash.

Lastly, if an RODC receives a session ticket request based on a TGT that isn't valid, a Kerberos error will occur asking the client computer to request a new TGT.  If the RODC does not have a copy of the users password hash, the RODC will forward the TGT request to a writable domain controller.

Moving Mailbox - Active Directory operation failed

When attempting to move a users mailbox from Exchange 2003 to Exchange 2010, the following error was experienced.

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:00:00

Username
Failed
Error:
Active Directory operation failed on DC.domain.local. This error is not retriable. Additional information: Insufficient access rights to perform the operation.
Active directory response: 00002098: SecErr: DSID-03150BC1, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

The user has insufficient access rights.
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.3.169.1&t=exchgf1&e=ms.exch.err.Ex6AE46B
Warning:
When an item can't be read from the source database or it can't be written to the destination database, it will be considered corrupted. By specifying a non-zero BadItemLimit, you are requesting that Exchange not copy such items to the destination mailbox. At move completion, these corrupted items won't be available in the destination mailbox.

Exchange Management Shell command attempted:
'domain.local\users\username' | New-MoveRequest -TargetDatabase 'MailboxDatabase4' -BadItemLimit '10'
Elapsed Time: 00:00:00

The problem was caused by incorrect permissions on the AD user account.  Re-enabling "inherit permissions" on the user account resolved the issue.

 

 

"Try Next Closest Site" Group Policy Setting

In a previous post I wrote about how the DC Locator component of Windows locates a domain controller.  This post can be found on the following URL:

http://clintboessen.blogspot.com.au/2010/05/how-clients-locate-domain-controllers.html

In this post we are going to look at a feature called "Try Next Closest Site" which is enabled on Windows Vista upwards clients via Group Policy.

In Windows 2000/XP/2003 when all domain controllers in an Active Directory site fail, there is a chance workstations may failover to another Active Directory site at a higher cost then the most preferable one as defined in Active Directory Sites and Services.  Changes have been made to the DC Locator algorithm starting from Windows Vista/2008 Server onwards which improves the DC Locator algorithm to ensure workstations always communicate with the next closest Active Directory site as defined in Sites and Services.

I strongly recommend this setting always be configured if your workstations are Windows Vista or higher.  To enable this setting perform the following.

1.Click Start, click Administrative Tools, and then click Group Policy Management.

2. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

3.Double-click Forest:forest_name, double-click Domains, and then double-click domain_name.

4.Right-click Default Domain Policy, and then click Edit.

5.In Group Policy Management Editor, in the console tree, go to Computer Configuration/Policies/Administrative Templates/System/Netlogon/DC Locator DNS Records.

6.In the details pane, double-click Try Next Closest Site, click Enabled, and then click OK.
For additional reading please see the following URLs:

http://technet.microsoft.com/en-us/library/cc733142(v=ws.10).aspx
http://technet.microsoft.com/en-us/library/cc772592(v=ws.10).aspx

Note: With "Try Next Closest Site" it will never try a remote site which contains a read only domain controller as read only domain controllers generally only store passwords for the users at the specific remote site for security reasons.

Distribution Groups - The properties on this object have invalid data.

I am in the process of carrying out another Exchange 2003 to Exchange 2010 migration for a customer with 800 users.  On the Exchange 2010 server, when attempting to open the properties of a distribution group which was created on Exchange 2003, the following message appears.

"The properties on this object have invalid data. If you click OK, the default values will be used instead and will be saved if you do not change them before hitting Apply or OK on the properties page. If you click cancel, the object will be displayed read-only and corrupted values will be retained."

This message can appear in a number of circumstances for any corrupt information on an Exchange 2010 distribution group.  To compare what is corrupt a trick is to do this:

1. Create a new distribution group on Exchange 2010
2. Look at the properties of the distribution group by typing Get-DistributionGroup "New Distribution Group" | fl
3. Compare this against the properties of the bad distribution group Get-DistributionGroup "Bad Distribution Group" | fl
4. Fix whatever invalid information exists on the bad distribution group.

In my case, the Distribution Group had spaces in the Alias.  The alias attribute must not contain spaces, after removing the spaces the error now longer displayed.

 

What are Scoped Send Connectors?

A topic which frequently confuses Exchange Administrators is the concept of "Scoped Send Connectors".  So what are they?

When you mark a send connector as scoped, this means it can only be used by Exchange 2007/2010 hub transport or Exchange 2013 mailbox servers in the same Active Directory site as the send connector.  If not selected, the connector can be used by all transport servers in the Exchange environment.

How do you determine what servers are in the Active Directory Site?  A Send Connector is not bound to any specific Active Directory site Clint, they are merely an object in Active Directory!

True, however say you have an Active Directory site called "Contoso HQ" and the site has 4 transport servers whether they be Exchange 2010 Hub Transport servers or Exchange 2013 mailbox servers.  Two of these servers are marked as source servers a of the scoped send connector and two aren't.  The two servers which are a member as a source server of the send connector can obviously use the send connector.  The other two transport servers in the same Active Directory site which are not source servers can also use the scoped send connector, all other transport servers in other sites - bad luck!

Unmarking it as scoped means transport servers in other sites can relay through the send connector if there is no closer option for external mail relay.

Hope this blog post brings clarification to this topic.

How to Enable Group Policy Debugging on Windows 7 / 8 Clients

In Windows XP / 2003 there was a very useful log file called "Userenv.log" which was located under %Systemroot%\Debug\UserMode\Userenv.log.  This log file was extremely happy when diagnosing issues relating to Group Policy.

This log file no longer exists in Windows 7 / Windows 8 and Microsoft has moved majority of the Group Policy logging to the new "Applications and Services Logs" under Group Policy\Operational.  The only caveat however from my experience is these logs are no where near as verbose as the logs provided under the legacy Userenv.log.

You can however enable verbose logging on Windows 7 / Windows 8 computers by adding an additional registry key, this procedure however has not been documented officially by Microsoft on TechNet and should be used for advanced troubleshooting only as a last resort.

The registry key which turns on advanced verbose logging is as follows.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Diagnostics]
"GPSvcDebugLevel"=dword:00030002

The resulting log file “gpsvc.log” can be found %WINDIR%\debug\usermode.
Hope this post assists you in whatever group policy troubleshooting your currently performing!

Direct Access Client Connectivity Issue

I have implemented Microsoft Direct Access for a customer running Windows 7 Enterprise and Windows 8 Enterprise workstations.  Direct Access was tested in a working order however the customer has a number of laptops which are unable to connect to the Direct Access service.

The following commands were run to gather diagnostic information about the issue:

netsh interface httpstunnel show interfaces

Role: client
URL: https://vpn.example.com:443/IPHTTPS
Last Error Code: 0x2af9
Interface Status: failed to connect to the IPHTTPS server.  Waiting to reconnect

Get-DAConnectionStatus

Status: Error
Substatus: CouldNotContactDirectAccessServer

This error was caused by a proxy server set on the workstations in question in Internet Explorer.  When a proxy is set, Direct Access will attempt to create the IPHTTPS connection through the proxy server.  If you want to leave proxy servers intact in Internet Explorer properties, what you can do is add vpn.example.com (the connection URL) as a proxy exception in Internet Options.

ODBC SQL Server Driver - Cannot Generate SSPI context

On the weekend I responded to an emergency callout regarding a custom Microsoft Access front end application not being able to talk to the backend SQL database.  The error which was being received was as follows.

Connection failed:
SQL State:'S1000'
SQL Server Error 0
[Microsoft][ODBC SQL Server Driver]Cannot generate SSPI context

After investigation this was caused by an incorrect time set on the SQL server.  The Kerberos time tolerance by default is set 5 minutes however the SQL server clock had drifted 8 minutes out.  The customer did not have their Active Directory time hierarchy configured correctly.

What is Name Resolution Policy Table (NRPT)

The NRPT is a new feature which was first introduced in Windows Server 2008 R2 that allows a client to assign a DNS server address to particular namespaces rather than to particular interfaces. The NRPT essentially stores a list of name resolution rules that are applied to clients through Group Policy.  Each rule defines a DNS namespace and DNS client behaviour for that namespace.

This feature is essential to support new features of Windows such as Microsoft Direct Access.

When a DirectAccess client is on the Internet, each name query request is compared against the namespace rules stored in the NRPT. If a match is found, the request is processed according to the settings in the NRPT rule. The settings determine the DNS servers to which each request will be sent.

For more information about NRPT, please view the following TechNet website:

http://technet.microsoft.com/en-us/library/ee649241.aspx

Licensing Windows Server 2012 or Windows 8 with KMS

In previous releases of Windows such as 2008 and Windows 7 you can install Windows without entering a product key.  After the installation is complete, the computer would automatically find the KMS through DNS and license/activate itself against the KMS server.

In Windows Server 2012, it asks for a product key as part of the installation process as shown in the following screenshot.

The only way to progress this screen is to enter a MAK key obtained of a companies Microsoft licensing portal.

So what happens if you want the Windows 2012 Server or Windows 8 machine to automatically license itself against a KMS upon completion of the installation?  You can't enter a product key here as this means the machine will become a MAK license.

The answer is to use a KMS Client Setup Key.  This isn't the KMS key itself, but a key which tells the Windows Server 2012 or Windows 8 machine that once it finishes installation to go off and license itself against a KMS.

Microsoft has published a list of KMS Client Setup Keys on the following TechNet article.  Simply enter the correct KMS Client Setup Key to complete the installation in which Windows will go off and automatically activate against the appropriate KMS server through DNS or Active Directory discovery.

http://technet.microsoft.com/en-us/library/jj612867.aspx

Hope this post has been helpful!

Windows 8.1 Internet Explorer 11 - This page can't be displayed for Google

A customer of mine flagged an interesting issue on Windows 8.1 running IE11.  Whenever they attempted to access "google.com" or any Google related page such as Google Maps they would receive an error stating "This page can't be displayed".  Upon refreshing the page, the page would then load normally.

Generally these symptoms relate to one of two issues:

• The Maximum Transmission Unit (MTU) being set to something too high on the core switch/router
• A DNS Issue where the DNS server is configured with a forwarder which is not responding in sufficient time or simply delaying on DNS resolution attempts.

These issues however effect ALL websites.  This issue was only isolated to Google related websites.

A little research and I stumped across the following website which illustrates the exact issue:

http://betanews.com/2013/10/19/google-is-broken-in-ie11-on-windows-8-1/

This website mentions that Google has recently made code changes to its website which effects the Internet Explorer 11 web browser.

The Work Around

Until Google or Microsoft sort these issues out between each other, the work around is to disable "Enhanced Protect Mode" through Internet Properties.  To do this perform the following procedure:

1. Click the Tools icon in Internet Explorer.
2. Go to Advanced tab and select the  Security section and uncheck the checkbox for Enable Enhanced Protected Mode (requires restarting Internet Explorer).
3. Click on Apply, and then click OK.
4. Close all open Internet Explorer windows, and then restart Internet Explorer.

 

Windows Server 2012 DHCP Failover Not Adhering to Scope Times

When setting up Windows Server 2012 DHCP Failover you may notice the lease duration provided to clients may not adhere to the lease duration configured on the DHCP scope.  This is by design and a normal behaviour in a DHCP failover relationship.

By default when a lease is first issued to a client, it adheres to the Maximum Client Lead Time (MCLT) value, not the value on the DHCP scope.  This is considered a temporary lease period given by a failover server to a new client.  In addition to providing a temporary lease to client, it specifies the amount of time for which a DHCP lease may be renewed by either failover peer without contacting the other.  It also specifies the amount of time that either DHCP server will wait in a “partner down” state before assuming control of the entire IP address range within the scope.  ( default = 1 hour ).

Maximum Client Lead Time (MCLT) can be found when configuring DHCP failover in the following location in the setup wizard.

 

After the initial lease, when the client attempts to renew its IP address it will then inherit the default scope lease time.

 

Key points to take away:

• Windows Server 2012 DHCP Failover, an initial lease always = the MCLT (Maximum Client Lead Time)
• Afterwards the same client will renew and get whatever is defined in the scope definition.

So if you see clients receiving 1 hour leases from DHCP, relax its by design!  They will receive a full lease when they next attempt to renew DHCP.

Sources:

http://technet.microsoft.com/en-us/library/dn338973.aspx
http://tools.ietf.org/html/draft-ietf-dhc-failover-12#section-5.2
http://support.microsoft.com/kb/2831920

Naming Information cannot be located because: The directory or file cannot be created.

I just added the first Windows Server 2012 R2 domain controller to a 2008 Active Directory domain at a customer.  As soon as I added the first 2012 R2 domain controller to the new domain, when opening any of the Active Directory MMC Management consoles the following error was experienced.

Naming information cannot be located because:
The directory or file cannot be created.
Contact your system administrator to verify that your domain is properly configured and is currently online.

When performing a netdom query dc on the new domain controller it complained time is out.

Fixing up the time to ensure it is within 5 minutes of the PDC Emulator (as required for Kerberos) resolved the problem.

Easy one but had me stumped for a little while (I didn't glance at my system tray to check the clock until a while).

Configure failover failed. Error: 20114: The DHCP Failover relationship already exists.

This post relates to the new Windows Server 2012 Failover technology.  I was diagnosing DHCP Issues at a customer when I was required to remove my DHCP scopes from failover configuration then re-add them.  After attempting to re-add the scopes I received the following error.

Configure failover failed.  Error: 20114: The DHCP Failover relationship already exists.

Despite removing the scopes from DHCP failover the partner server still believes the DHCP failover relationship was in existence.

After messing around a little longer I found out a work around.  The error was occurring because I was using the same Relationship Name as before.  Changing the Relationship Name to something else resolves the issue.

 

Exchange 2003 to Exchange 2010 Mailbox Move Failing

When attempting to perform mailbox moves between Exchange 2003 to Exchange 2010 the following error was experienced.

Mailbox Name
Failed

Error:
Mailbox database 'b70397e5-c347-4f7b-8a15-575172b89820' is offline.

MapiExceptionLogonFailed: Unable to make connection to the server. (hr=0x80040111, ec=1010)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: cpaexch01.cpawa.com.au
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=99]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x3F2][length=56][latency=0]
    Lid: 56945 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=99]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x3F2][length=56][latency=15]
    Lid: 59505   StoreEc: 0x3F2    
    Lid: 52465   StoreEc: 0x3F2    
    Lid: 60065 
    Lid: 33777   StoreEc: 0x3F2    
    Lid: 59805 
    Lid: 52209   StoreEc: 0x3F2    
    Lid: 56583 
    Lid: 52487   StoreEc: 0x3F2    
    Lid: 19778 
    Lid: 27970   StoreEc: 0x3F2    
    Lid: 17730 
    Lid: 25922   StoreEc: 0x3F2    
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.3.169.1&t=exchgf1&e=ms.exch.err.ExC2B9A8
Exchange Management Shell command attempted:
domain.local/Townsend/Users/Mailbox Name | New-MoveRequest -TargetDatabase 'MailboxDatabase2'

Elapsed Time: 00:00:01

After some investigation it turned out that the Exchange 2010 server did not have permissions to the Exchange 2003 mailbox databases.  To fix the issue I simply went to the properties of the Exchange 2003 mailbox database in System Manager, Added the Exchange 2010 computer object and granted it full control permissions.

After making this change mailbox moves worked successfully.

80090328 - The token supplied to the function is invalid

In this post I am going to show you how to resolve the below error in Exchange 2003 System Manager when attempting to browse the public folder certificate store.

The token supplied to the function is invalid.

ID no: 80090328
Exchange System Manager

To resolve this error perform the following steps.

Navigate to Internet Information Services on the Exchange 2003 server, right click Exadmin and select properties.

 

Under secure communications click Edit.

 

 

Untick require secure channel then click OK, followed by OK.

 

 

Make sure you perform an IISRESET after making the changes documented in this blog post.

 

80090328 - The received certificate has expired

In this post I am going to show you how to resolve the below error in Exchange 2003 System Manager when attempting to browse the public folder certificate store.

The received certificate has expired.

ID no: 80090328
Exchange System Manager

This error occurs when the certificate linked to the Exchange 2003 default website has expired.

Simply replace the certificate linked to the default website in Internet Information Services (IIS) and this will resolve the problem.

Make sure you perform an IISRESET after making the changes documented in this blog post.


Note if you perform these steps and get the following error "The token supplied to the function is invalid", please refer to the following blog post:

http://clintboessen.blogspot.com.au/2013/12/80090328-token-supplied-to-function-is.html

Enterprise Certificate Authority Pushing Old Certificates through Active Directory

A new public key infrastructure was deployed on Windows Server 2012 R2 consisting of two certificate authorities.  For security reasons and to adhere to Microsoft best practice, we deployed a new stand alone offline certificate authority and a subordinate enterprise certificate authority which is Active Directory integrated and will be responsible for issuing certificates to all devices, users and service accounts on the domain.

The following image is an overview of the deployed solution.

When I setup the subordinate certificate authority, I issued a certificate to the subordinate from the root certificate authority to validate its identity and authorise it to issue certificates on behalf of the root.  Only after I issued this certificate, I found out the default issuing time for certificates on stand alone certificate authorities in Windows Server 2012 R2 is only 1 year.  This period of time is far to small for a certificate that is assigned to another certificate authority.

As a result we increased the time that the root certificate authority issues certificates for which was performed with the following command:

certutil -setreg ca\ValidityPeriodUnits "10"

This changed it from 1 year to 10 years.
We then issued a new certificate from the root authority to the subordinate certificate authority for the 10 year period.

The Problem

After issuing a new certificate to the subordinate certificate authority I realised the subordinate certificate authority is pushing out the new 10 year certificate to the intermediate certificate store on workstations as well as the old 1 year certificate.  This is shown in the following screenshot below on a domain member machine.  The two certificates highlighted were old certificates which use to be assigned to the enterprise subordinate certificate authority.  There are two because we issued two (we forgot to restart the certificate authority service on the root CA after running the certutil command above to extend the validity period so we had to repeat the process).

I don't want the Active Directory certificate authority pushing out these invalid certificates to all domain joined devices.

How to Remove Certificates from Active Directory Deployment

To remove certificates from Active Directory deployment, you must open an application called pkiview.msc on an Enterprise Certificate Authority.

 

Right click Enterprise PKI and select Managed AD Containers.

 

 

Enterprise root certificate authorities are located in the "Certification Authorities Container" tab and enterprise subordinate certificate authorities are located under the "NTAuthCertificates" and "AIA Container" tab. View the certificates by looking at expiry date or thumb print to ensure you find the correct certificate.  Remove the unwanted certificates which are no longer required by your workstations.

 

 

After removing the unwanted certificates performing a gpupdate /force on a domain member automatically removes the unwanted certificates from the intermediate store and root store.  This is shown in the following screenshot taken of the computers certificate store in a mmc snapin.

A big thankyou to River Mei from Microsoft who assisted me with this issue.

Routing Group Connector - The mailbox recipient does not have a mailbox database

If you are a regular follower of my blog, you will have noticed I have been writing about different problems you may experience whilst in co-existence between Exchange 2003 and Exchange 2010 using Routing Group Connectors.  Previously I wrote about error "There is currently no route to the mailbox database" and how to overcome it which can be found on the following page:

http://clintboessen.blogspot.com.au/2013/12/routing-group-connectors-there-is.html

 Today we are going to look at another issue using Routing Group Connectors

The mailbox recipient does not have a mailbox database.

This error can too be viewed using the "Get-Message | fl" command by looking at the last error attribute.



 As before, we have a valid routing group connector between a single Exchange 2010 and Exchange 2003 server setup correctly:



However mail is not passing between the servers for select users only!

Resolution

When installing Exchange 2010 into an existing Exchange 2003 organisation, the Exchange 2010 infrastructure has a new list of security groups it creates to assign permissions over Active Directory objects.  These permissions extend over user account objects to Active Directory configuration partition exchange related objects.

The error message "The mailbox recipient does not have a mailbox database" is being shown because Exchange 2010 does not have permissions to see the user account in Active Directory.  It does not know the Exchange 2003 legacy mailbox exists and the Exchange 2010 server definitely does not have the mailbox hence it is complaining that the recipient has no mailbox database!

This can happen when user accounts have broken inheritance in Active Directory.  To fix this, open Active Directory Users and Computers, find the user account in question and navigate to the account properties.  Under security, advanced make sure that inheritance is enabled for the user in question.  This ensures the Exchange 2010 environment has access to the user object and can see that it is a mailbox.

Note: If you cannot see the security tab on the user account in Active Directory, you will need to enable Advanced Features from the view menu.

Websense Appliance - Internal Error - Server Connection Terminated

When setting up a Websense V5000 G2 appliance 7.8.1, in Gateway Manager I enabled integrated Windows Authentication, joined the appliance to the domain then enabled SOCKS.  After making these changes, when navigating to websites from clients the following error was experienced:

internal error - server connection terminated

The Internal Error generally occurs when SOCKS has been enabled but not configured.  After disabling SOCKS in the Websense Gateway Manager interface, the issue was resolved.

 

Routing Group Connectors - There is currently no route to the mailbox database

When performing an Exchange 2003 to Exchange 2010 migration, mail flowing between the two servers using a routing group connector was not working.  When relaying an email to the Exchange 2010 server for a user mailbox residing on Exchange 2003, the email would simply sit in the queue and not send.

The following error was experienced:

There is currently no route to the mailbox database

This can be seen in the message log with "Get-Message | fl"

 The messages simply sat in the Unreachable queue on the Exchange 2010 server.

This error generally means no routing group connector was made between the two servers.  However I confirmed the routing group connector was created by Exchange Setup automatically.

Next I removed these routing groups and recreated them with new ones through PowerShell:

After recreating the routing group connectors you must restart the "Simple Mail Transfer Protocol" service on Exchange 2003 and the "Microsoft Exchange Transport" service on Exchange 2010.

This still did not resolve the problem.

Resolution

Next I started checking permissions of Exchange container objects in the configuration partition within Active Directory against a default install of Exchange 2003 in a test lab.  I noticed that permissions on the Exchange 2003 object in Active Directory was no longer inheriting from the Administrative Group/Servers Container.

After re-enabling inheritance using ADSI Edit over the Exchange 2003 server object, this resolved the issue.

Secure Email, The Next Generation Today

The following post is a guest post by Andy Campbell.

Secure Email, The Next Generation Today

Out With the Old In With the New
If you exchange data regularly with colleagues, business partners and clients, which probably describes all of us, then no doubt you will have at some point considered the risks to that data being leaked or compromised in some way. This will have led you to investigate encryption as a tool to safeguard both you and your recipient’s data. Many of you might have elected not to proceed with an encryption solution because of the burden this would place on normal business practices. Well if that’s you, it’s time you looked again. This year’s winner of the UK IT Industries Cloud Provider of the Year, Egress Software Technologies, have an email encryption product called Switch that just might have the answer.

Sending Secure Emails
This can be done via either a desktop app, smartphone/tablet app or web interface, or if you are a Microsoft Outlook or Lotus Notes user from within your email client. Plus you don’t need to worry whether your recipient is a current user of Switch or has even heard of it. We are running this demonstration on a MacBook Pro using OS X version 10.7.5.

 Fig. 1 Welcome Screen

From the “Welcome” screen (Fig. 1) you can choose to create a package, create a secure message or view the packages you have created earlier. Creating a package is the term used here that describes how you can wrap encryption around any form of data to be exchanged by USB stick, CD ROM, FTP file transfer or DVD.

However since we are talking about secure email let’s look more closely at that aspect of this product.

On clicking the “Create Message” icon your browser will be opened at Egress’s web mail page where you can compose your email and add attachments as required (Fig. 2). If you are using a Windows PC it will integrate with either Microsoft Outlook or Lotus Notes if available.

 Fig. 2 Secure Email

Clicking “Send Secure” will automatically encrypt and send your email to your recipient, you will get a confirmation screen (Fig. 3). It couldn’t be easier!

 Fig. 3 Confirmation Screen

Your recipient will receive an email as shown in Fig. 4.

 Fig. 4 Secure Email Arrives

It is no problem if your recipient has never used Switch before, they will most likely click on the “read this secure email” link, which will take them to the account sign in screen (Fig. 5) where they can create a free Switch account for themselves.

Fig. 5 Sign In Screen

Once you have created an account and signed in you will have automatic access to the decrypted email and attachment (Fig. 6)

Fig. 6 Decrypted Email

From here you can obviously read the email and any attachment, but and this is where it gets quite cleaver, both the email and attachment are “water-marked” with the recipients email address. This extra layer of protection is designed to encourage users not to share the contents of your email with others without your permission. The copying of and attachment has been made more difficult by making the contents appear opaque when the window in which they appear is no longer the focus (Fig. 7).

 Fig. 7 Opaque Content

Should the user try to forward this email securely to someone else they will not be able to open it even if they have a Switch account in their name unless you have added them to the list of approved recipients of this email. Does that make sense? In other words you keep control of where your data can be viewed. This makes Switch so much more than just a secure email product. You can even time when access is to be granted.

How Secure Is Switch?
Switch offers pretty strong encryption, using AES at 256 bits and coming with a stamp of approval from CESG and FIPS, you can be assured that the algorithm has been properly implemented. And since each communication is encrypted with fresh keys any possible future breach would effect only the single communication not previous ones or future ones. Hopefully I told you enough for you to go and take a look for yourselves.


About Andy Campbell
Andy Campbell has over 40 years experience in the computer business, 20 of those years were spent at Reflex Magnetics Ltd a UK developer of security software. As managing director he was instrumental in forging a close relationship with the MoD and CESG, Reflex's Disknet and Data Vault products being used extensively by both the UK's armed forces and NATO. More recently he has been involved as a director and investor in Reflect Digital a web marketing agency.