Wednesday, December 24, 2014

Exchange 2013 FAST Search Technology Failed

I had a customer where Exchange 2013 FAST Search Technology failed and rebuilding the search indexes for the databases did not resolve the problem on a single Exchange 2013 SP1 server.

In this blog post I'm going to share with you the symptoms of my issue along with the resolution.

Symptoms of Issues

Users on the network were unable to search their mailbox from Outlook Web App or using Microsoft Outlook in online mode.

In addition to not being able to search in Outlook Web App or Microsoft Outlook, the Test-ExchangeSearch cmdlet failed with "Time out for test thread".


Despite this issue, the ContentIndexState on the databases remained in a Healthy report status with the Get-MailboxDatabaseCopyStatus cmdlet as shown below:


In the Server Application logs the following error was logged numerous times:

Log Name:      Application
Source:        MSExchangeIS
Date:          4/12/2014 1:54:06 PM
Event ID:      1012
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ExchangeServer.domain.local
Description:
Exchange Server Information Store has encountered an error while executing a full-text index query ("eDiscovery search query execution on database a0976351-948a-4625-8840-f649f8b98e0e failed."). Error information: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0821592. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847


Checking the size of the catalog folder was only around 16MB when the database size was 150GB.  From my experience a FAST index for a 150GB database is generally between 5GB and 10GB in size depending on how small the items are inside the database.

Issue Resolution

To resolve the issue the following steps were taken in order.

1. Stop the Microsoft Exchange Search and Microsoft Exchange Search Host Controller service.
 
2. Remove all the files under “Fsis” file:

C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis
We moved these files to another folder for backup purposes.

3. Open EMS and “Run as Administrator"

4. Navigate to directory C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer

5. Launch the script and run the command below:


.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.

6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):

.\installconfig.ps1 -action U -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

Then again installed Search component with below command :

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"


7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2013 server with repadmin /syncall /APeD.

8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatus and review the ContentIndexState.


When the Content Index process has finished crawling we can see the index size of the folder is now 8GB and searching is now working as normal.

 

6 comments:

  1. Clint - thank you for this information. We are suffering through the exact same issue but it seems to be confined to one database. We checked the catalogue folder size and each one is anywhere between 13 - 15 gb so not sure if this is the issue though. When we test using Test-ExchangeSearch we get a time out on test thread for mailboxes on this database only. Thoughts?

    ReplyDelete
    Replies

    1. windows xp home edition activation key , windows 10 serial key number , office project 2013 keys , windows 10 product key checker , windows 7 pro key sale , windows 10 product key tool , home premium product key free , windows 7 home premium sp1 activation key free download , sWzjoY

      windows 10 pro key

      office 2013 pro key sale

      cheap visual studio key buy

      Delete
  2. Thank you very much. Works smoothly.

    ReplyDelete
  3. you sir, just saved my evening, now my exchange server are indexing again, thank you a lot !

    ReplyDelete
  4. Unfortunately it did no work. When I try to reinstall the search, it only creates AdminNode1 and then aborts after a certain time (timeout). we could not manage to get the index running again. It also tries to get a noderunner task up and running, but it seems to fail (timeout). It seems like a permission issue if we try to reinstall search (with elevated rights, as admin).

    any other Idea what we should be looking at?

    ReplyDelete
  5. Not seeing your comment I'd probably never tried uninstalling after performing reconfig for ten times at least. But it really worked! Thanks.

    ReplyDelete