Exchange 2013 FAST Search Technology Failed

I had a customer where Exchange 2013 FAST Search Technology failed and rebuilding the search indexes for the databases did not resolve the problem on a single Exchange 2013 SP1 server.

In this blog post I'm going to share with you the symptoms of my issue along with the resolution.

Symptoms of Issues

Users on the network were unable to search their mailbox from Outlook Web App or using Microsoft Outlook in online mode.

In addition to not being able to search in Outlook Web App or Microsoft Outlook, the Test-ExchangeSearch cmdlet failed with "Time out for test thread".

Despite this issue, the ContentIndexState on the databases remained in a Healthy report status with the Get-MailboxDatabaseCopyStatus cmdlet as shown below:

In the Server Application logs the following error was logged numerous times:

Log Name:      Application
Source:        MSExchangeIS
Date:          4/12/2014 1:54:06 PM
Event ID:      1012
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ExchangeServer.domain.local
Description:
Exchange Server Information Store has encountered an error while executing a full-text index query ("eDiscovery search query execution on database a0976351-948a-4625-8840-f649f8b98e0e failed."). Error information: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0821592. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847

Checking the size of the catalog folder was only around 16MB when the database size was 150GB.  From my experience a FAST index for a 150GB database is generally between 5GB and 10GB in size depending on how small the items are inside the database.

Issue Resolution

To resolve the issue the following steps were taken in order.

1. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller service.
 
2. Remove all the files under “Fsis” file:

C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis
We moved these files to another folder for backup purposes.

3. Open EMS and “Run as Administrator"

4. Navigate to directory C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer

5. Launch the script and run the command below:

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.

6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):

.\installconfig.ps1 -action U -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

Then again installed Search component with below command :

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2013 server with repadmin /syncall /APeD.

8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatus and review the ContentIndexState.

When the Content Index process has finished crawling we can see the index size of the folder is now 8GB and searching is now working as normal.

 

Filtering Exchange MessageTrackingLog Output from Get-MessageTrackingLog Cmdlet

In todays blog I am going to show you how to filter Message Tracking Log output from the Get-MessageTrackingLog command.  I had a customer from an organisation example.com who wanted to see all external recipients who received an email with the message subject "Staffing Update".

Without using some PowerShell filtering, you can view all tracking log related entries for this against all Transport Servers in your organisation by running:

Get-TransportService | Get-MessageTrackingLog -MessageSubject "Staffing Update"

Note: If your running Exchange 2007 or 2010 replace Get-TransportService with Get-TransportServer.

This command will return all messages relayed in the last 30 days by default which is the amount of time the message tracking logs hang around for by default.

However if we want to meet my customers requirement, we want the output to not contain any recipients for "*@example.com" so that we can focus on emails leaving the company.  When I say external users or "leaving the company", I mean users who do not have an email address for "*@example.com", you may have an Exchange Organisation where you have multiple accepted domains setup for multi-tenancy!

To do this we need to do some filtering with PowerShell.  This can be achieved using the where{ } command as follows:

Get-TransportService | Get-MessageTrackingLog -MessageSubject "Staffing Update" | where{$_.recipients -notlike "*@example.com"}

But say we only want to see which users received this message with a @example.com email address.  Easy this can be done by reverting the -notlike to a -like.

Get-TransportService | Get-MessageTrackingLog -MessageSubject "Staffing Update" | where{$_.recipients -like "*@example.com"}

You can also easily filter senders by replacing the $_.recipients line with $_.senders like shown below:

where{$_.sender -like "*@example.com"}

Happy Filtering!

Microsoft PST Capture 2.0 Not Supported on Exchange 2013

The PST Capture 2.0 tool is not supported with Exchange 2013 despite being documented as supported under the following Microsoft Exchange Product Team blog post:

http://blogs.technet.com/b/exchange/archive/2013/02/22/time-to-go-pst-hunting-with-the-new-pst-capture-2-0.aspx

I have used this tool successfully with previous versions of Exchange Server however with Exchange Server 2013 this tool does not work.  I have tested PST Capture 2.0 with Exchange Server 2013 SP1 and Update Rollup 6, both unsuccessful.  Testing has been performed both in my lab environment and at a customer site.

I logged this issue with Microsoft under a professional services support case in which Microsoft were able to reproduce the issue in their lab environment.  This issue was then escalated within Microsoft and an issue identified with the tool with no fix provided.  Microsoft explained they are under no obligation to develop a hotfix for this tool as the End User Licensing Agreement (EULA) states that the software is provided "as is" and they are not under obligation to support it should issues or bugs be flagged with the software.  This is shown in my screenshot below.

When attempting to import a mailbox with PST Capture 2.0, the following error message is flagged in the tool:

Import error: Error opening mailbox

This issue is generally caused by one of the following issues as documented on the following website:

http://www.sysadminsblog.com/microsoft/pst-capture-errors/

However with Exchange 2013, none of these resolve the problem.

The PST Capture tool has a more advanced logging available for errors which can be found under the following location:

"C:\ProgramData\Microsoft\Exchange\PST Capture\Logs\MapiComServer 15"

The technical error which is generated when attempting to import to Exchange 2013 is as follows:

Failed to open default store - HR 8004011d

Hopefully Microsoft does get around to developing a fix/work around for this product as it will be a shame to purchase this tool from Red Gate Software and then not maintain it hence making it redundant.

How to Change Network Profile in Registry for 2008 Server

In a previous post I wrote about how to change the network location or "firewall profile" in Windows Server 2012 using PowerShell as you could no longer do this from the GUI like you could in previous versions of Windows Server. This article can be found under the following URL:

http://clintboessen.blogspot.com.au/2013/11/how-to-change-your-network-profile-in.html

However yesterday I had a problem at a customer running Windows Server 2008 where I was unable to change the network location from the GUI using Network and Sharing Centre and as a result I had to revert to the registry.  Tracking down the registry key responsible for the network profile was a difficult task so I decided to blog this one.

All network profiles are located under the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList\Profiles

Each network card will be represented below in a sub key with a GUID.  The Network Location is controlled by a DWORD value called Category.

Public = 0
Home = 1
Domain = 2

As you see my category is set to 2 which means this network adapter is using the Domain Firewall profile.

Remove Power Shell and Server Manager Pinned Icons from Start Menu Server 2012

In Windows Server 2012 and Windows Server 2012 R2 by default all users have a Power Shell icon and a Server Manager icon pinned to their start menu.  When setting up a Remote Desktop Session Host with Windows Server 2012 Remote Desktop Services you may not want users to have the Server Manager icon or Power Shell icon especially when desktop access is enabled.

 

In previous versions of Windows Server there was a Group Policy setting called "Remove pinned programs list from the Start Menu" which administrators could use to remove these pinned applications from the start menu from Remote Desktop Session Hosts.

 

 

This policy no longer works on Windows Server 2012 and there is no well documented method available for doing this on the Internet.

 

In Windows Server 2012, the three default pinned applications on the start menu are located under the following location:

 

C:\Users\username\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\TaskBar

 

 

The User Pinned\Taskbar  folder in the Quick Launch directory does not exist in the Default profile in Windows Server 2012 as shown below.

So how do these icons get created in the users profile and where do they come from?

This process appears to be hardcoded into Windows Server 2012 and executed upon creating a new user profile.  The user profile creation process creates Quick Launch\User Pinned\TaskBar folder as part of the login then copies a bunch of "lnk" shortcut files from the "All Users Profile".

Server Manager Pinned Icon

The "Server Manager.lnk" file gets copied by the user profile creation From:

"C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Administrative Tools\Server Manager.lnk"

To:

"C:\Users\username\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\TaskBar\Server Manager.lnk"

Windows PowerShell Pinned Icon

The "Windows PowerShell.lnk" file gets copied by the user profile creation From:

"C:\ProgramData\Microsoft\Windows\Start Menu\Programs\System Tools\Windows PowerShell.lnk"

To:

"C:\Users\username\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\TaskBar\Windows PowerShell.lnk"

How to Prevent the icons from getting Populated to User Profiles

To stop the "Windows PowerShell.lnk" and "Server Manager.lnk" shortcuts from getting pinned to the start menu on all users, simply delete these icons from the All Users profile under "C:\ProgramData" from the following locations:

"C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Administrative Tools\Server Manager.lnk"

and

"C:\ProgramData\Microsoft\Windows\Start Menu\Programs\System Tools\Windows PowerShell.lnk"

 

Now when new users login for the first time, the User Profile creation process is unable to copy the "Server Manager.lnk" and "Windows PowerShell.lnk" shortcuts as they no longer exist as shown in the following screenshot.