Exchange 2010 Outlook Anywhere Connection Randomly Drops Out

One of my customers experienced an issue where Outlook clients randomly lost their HTTPS connection to the Exchange server.  All Outlook clients at my customer connect to the Exchange server using http/RPC rather then TCP (MAPI) both internally and externally.  Randomly once a day the Outlook HTTP connection would break and fall back to TCP (internally) or break completely for external users.

Running iisreset would fix the problem but the problem would always re-emerge.

To ensure that Outlook clients retain their connection with the Exchange server using HTTP and not TCP both "On fast networks, conect using HTTP first" and "On slow networks, connect using HTTP first" must be selected.

Activesync and webmail continued to work ok and were not effected by this issue.

This issue was caused by the RPC web application using te Default Application Pool (DefaultAppPool) which is configured to recycle worker processes every 1740 minutes (29 hours).  During the recycling process, IIS allows active worker threads an additional 90 seconds to finish servicing requests before IIS terminates the active threads.

Because RPC over HTTP uses long-running connections, the connections may not finish within an additional 90 secosd that were given to the worker threads.  In this scenario, the connections are terminated.  Therefore Outlook loses connectivity with IIS.  When this action occurs, Outlook immediately tries to reconnect.  If many Outlook clients are disconnected at the same time, the large number of concurrent reconnections may overwhelm the server.

To resolve this problem create a new Application Pool dedicated to the RPC over HTTP web application with a larger HTTP sys que limit.  Please refer to the following TechNet article with instructions on how to perform this procedure:

http://technet.microsoft.com/en-us/library/dd421855.aspx

An Insight into Stellar Phoenix Outlook PST Repair tool

In this article we are going to look at how to repair a corrupt PST file using the Outlook PST Repair tool created by Stellar Phoenix.

Before we dive into Outlook PST Repair lets quickly cover Scanpst.exe.  Scanpst.exe is a free tool shipped with Microsoft Outlook 2003/2007/2010 which lets you repair corrupt PST files.

On my Office 2010 installation, ScanPST can be found in the following path:

C:\Program Files\Microsoft Office\Office14


Below is a screenshot of ScanPST.

Stellar Phoenix labs performed testing with ScanPST and from their testing they discovered that the free PST repair tool is capable of repairing PST files with only minor structural errors.  PST files with severe correction or PST files where the indexing table is completely removed, ScanPST will not repair the file.

Stellar Phoenix claim that their tool Outlook PST Repair v4.5 can repair a corrupt PST file and bring it back to a consistent state regardless how severe.  I questioned this with Stellar Phoenix as 100% of corruption is a big claim however the company was confident to back it.  All content within the corrupt the PST file which is in its valid state can be recovered.  Data which has been lost due to corruption is gone, no tool will be able to recover this.

Outlook PST Repair v4.5 has been designed to look like Microsoft Outlook to provide users and administrators with a familiar user experience.  When a corrupt PST loaded, all content which is still readable inside the corrupt PST file will be displayed.  Companies have the flexibility of recovering individual emails, attachments, sub folders or entire PST files.

Below is a screenshot of the the Outlook PST Repair tool:

To begin using the tool simply the Open an Outlook File to Repair.

Select the location of the PST file which is corrupt.  In my case I have a corrupt PST file called test.pst.

Hit the Start button and Outlook PST Repair will go through and scan for all recoverable content.

The tool displays all data which is now recoverable.  The user is able to browse mail items, calendars, contacts, tasks, notes everything which can be displayed in Outlook using the Outlook PST Repair tool.

The user is able to do the following things once a corrupt PST file has been loaded in Outlook PST Repair v4.5:

• Export all content which is readable within the corrupt PST into a new PST file.
• Export select content from a corrupt PST file by selecting what content they wish to export.
• Extract attachments from emails
• Export individual emails to MSG or EML format

Outlook PST Repair v4.5 comes in a demo version and a full version.  What is the difference between the demo version and the full version?

The demo version allows you to see all items which can be discovered, read email and look at calendar items however it does not allow you to extract any information out of the corrupt PST file including attachments, individual items or folders.

The full version allows you to browse a corrupt PST file and export content from a corrupt PST file to a new PST.

There are two licencing versions for purchasing the full version of Outlook PST Repair 4.5.  Both licences come are lifetime and come with 24/5 technical support free with the purchase.

• Single User Licence ($129 USD).  Users receive a key which they use to activate the Outlook PST Repair tool.  Once activated the key will only ever work on the Windows instance in which Outlook PST Repair tool was activated.  In the event the user purchases a new computer or re-installs Windows, the user must contact support to transfer the licence.
• Technician Licence (299 USD).  The technician licence can be used unlimited times on different workstations.  However a USB key must be connected to the machine to activate the licence and perform the recovery.  Only one recovery can be performed at a time.  One technician licence must be purchased per office.  Stellar Phoenix ship the USB key to the customer upon purchase.

Note: All pricing is subject to change, to get the latest pricing please visit www.stellarinfo.com

Summary

The Outlook PST Repair 4.5 tool is a fantastic tool for fixing corrupt PST files.  If Scanpst.exe fails to recover a corrupt PST file or you need to perform granular recovery from a corrupt PST file I encourage you to give Stellar Phoenix Outlook PST Repair a shot.

For more information or to obtain a copy of Stellar Phoenix Outlook PST Repair please visit the following website:

http://www.stellaroutlooktools.com/scan/pst-repair.php

How do I find out if an Email Address exists in Exchange

You want to determine if an Email address has been already configured on an Exchange server.  To do this you need to use the following cmdlet:

Get-Recipient

For Example:

Get-Recipient clint.boessen@avantgardetechnologies.com.au

You cannot use the Get-Mailbox cmdlet as remember you can configure email addresses on more then just user accounts.  Email addresses can be configured on groups, contacts even public folders.

Note: If you use Get-Mailbox with the -an switch it will only search the primary SMTP addresses.

Cisco - Port Two Public IP Addresses to the Same Internal Address

Today we required the ability to port forward two public IP addresses both listening on TCP25 to the same internal IP address listening on TCP25.  By default a Cisco router will not let you do this.  However there is an extenable option which you can put on the end of your command to allow you to do this.

 

For example to allow TCP25 from both 3.3.3.3 and 3.3.3.4 to 10.1.1.40 on TCP25 we would do the following:

 

perth-router(config)#do show run | in ip nat
ip nat inside source static tcp 10.0.8.10 25 3.3.3.3 25 extendable
ip nat inside source static tcp 10.0.8.10 25 3.3.3.4 25 extendable

Hope this has been helpful. 

HTTP Attack Resulted in RBL Listing

Today one of my customers was listed on the SpamHaus XBL list.  The Spamhaus Exploits Block List (XBL) is a realtime database of IP addresses of hijacked PCs infected by illegal 3rd party exploits, including open proxies (HTTP, socks, AnalogX, wingate, etc), worms/viruses with built-in spam engines, and other types of trojan-horse exploits.

My customer had all client workstations access the Internet from the same public IP address as what the Exchange 2010 server relayed email from.  Workstations did not connect to the Internet through a proxy, just sandard network address translation (NAT).

My customer did block TCP25 Outbound (SMTP Traffic) from all hosts on the network but the internal IP address of there Exchange 2010 server.  Despite this my customer was still added to the XBL.SpamHaus.org blocklist and as a result had difficulties sending and receiving email from many companies especially because SpamHaus is one of the more popular blocklists.

This was because a few workstations on their network was infected with the Pushdo trojan which was performing denial of service (DOS) attacks against target web servers.

Below is the reason why we were RBLed extracted from the SpamHaus.org website:

To get around this problem we changed the outgoing IP address of email, ensured a PTR record exists for the new IP address, updated the Sender Policy Framework (SPF)  TXT record on the DNS zone.  Finally we updated the port forward on the router and MX records to ensure all mail relay went through a dedicated email.

So what did we learn from this?

• If possible always use a dedicated public IP address for relaying mail (if possible)
• Use a proxy server for your users to surf the net and block HTTP/HTTPS and other ports if possible outbound to the Internet.

Regarding the Pushdo botnet, we got around to cleaning that up too to ensure my customers network was not used to DOS innocent web servers on the net.

The Windows Backup engine could not be contacted. Retry the operation.

Today when attempting to perform a System State backup on a Domain Controller I received the following error message:

The Windows Backup engine could not be contacted. Retry the operation.
The RPC server is unavailable.

I also noticed the following event errors appearing in Event Viewer.

Log Name:      Application
Source:        VSS
Date:          13/03/2013 10:48:41 AM
Event ID:      12292
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {06d8e136-56f6-4048-93fb-a5943e949375} [0x80040154, Class not registered
].
Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Get Shadow Copy Properties
Context:
   Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}
   Class ID: {06d8e136-56f6-4048-93fb-a5943e949375}
   Snapshot Context: -1
   Snapshot Context: -1
   Execution Context: Coordinator

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 10:48:50 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x5888
Faulting application start time: 0x01ce1f9517234ddc
Faulting application path: C:\Windows\system32\wbengine.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 880e1970-8b88-11e2-aefa-005056a2000b

The above event error 12292 it provided us the Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}.  Looking in the registery under HKLM\System\CurrentControlSet\services\VSS\Providers\{5fdb6ef5-6ead-4610-995b-401c88626115} it shows this provider as the Backup Exec VSS Provider.



For some reason WBAdmin is trying to use the Backup Exec VSS Provider instead of the Microsoft VSS Provider.

I added the registry DWORD UseMicrosoftProvider to HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore with a value of "1" which is meant to force the backup to use the Microsoft provider.

This key had no effect, the backup still attempted to use the Symantec VSS Provider.  Next I used the following Symantec article 130940 to completely remove the Symantec backup exec agent from the server including removing registry keys.

http://www.symantec.com/business/support/index?page=content&id=TECH130940

After removing the Symantec backup exec agent I ran a test backup and the backup failed again with the same error.  Running a "vssadmin list providers" revealed that the Symantec VSS Provider was still in place despite following Symantec article 130940 which was meant to completely remove backup exec from a windows server.

Again we see same GUID of the Symantec provider which was presented in the event error and the registry, {5fdb6ef5-6ead-4610-995b-401c88626115}.

I then followed Symantec article 77585 to completely remove the Backup Exec VSS Provider by deleting the {5fdb6ef5-6ead-4610-995b-401c88626115} key from the following location in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\

http://www.symantec.com/business/support/index?page=content&id=TECH77585

After restarting the VSS service we see the Backup Exec VSS Provider is no longer available.

I then rebooted the server.  After a reboot I attempted another backup with wbadmin.  We got further this time but it still crashed out.

Some new event logs exist now:

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 2:47:07 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2

Log Name:      Application
Source:        VSS
Date:          13/03/2013 2:47:11 PM
Event ID:      8193
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...).  hr = 0x80070005, Access is denied.
.
Operation:
   Initializing Writer
Context:
   Writer Class Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
   Writer Name: NPS VSS Writer
   Writer Instance ID: {37bef355-a711-4241-a2bc-91f1181c845b}

 

VSS Event ID 8193 says that the VSS provider was denied access when opening a registry key under the security context of SYSTEM

 

SYSTEM\CurrentControlSet\Services\VSS\Diag,...). 

 

Damn it cut off!  We could use Sysinternals ProcMon to get the full path however lets just force FULL access for tye System account from the DIAG key downwards.

 

 

 After making this change I then tested another wbadmin.  Made no difference. :-(

 

I searched the entire registry for the GUID of the Backup Exec VSS Provider to ensure nothing was missed.  My search found nothing.  Whilst I have isolated the problem to the VSS Provider provided by Symantec, a change made by the Symantec Backup Exec agent remains and as a result wbadmin will not function.

If there is someone out there who has fixed this issue can you please comment below with your resolution to ensure others with this issue have a fix as this is not documented anywhere on the Internet.

The backup storage location is invalid. You cannot use a volume that is included in the backup as a storage location.

I was about to do a large number of Active Directory changes on a domain controller and needed to grab a system state backup before proceeding.  I took a system state backup using the wbadmin utility on a 2008 R2 SP1 domain controller by using the following command from a command prompt:

wbadmin start systemstatebackup -backupTarget:c:

When running the command I received the following error message:

The backup storage location is invalid. You cannot use a volume that is included in the backup as a storage location.

The resolution for this problem was adding a new Key and DWORD value to the system registry.  Create the following:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wbengine.
Create a key called "SystemStateBackup"
Set the value of this entry as follows:
Name: AllowSSBToAnyVolume
Data type: DWORD
Value data: 1

After creating the registry key the system state backup now completed successfully. 

The FTP site cannot be started. Another FTP site may be using the same port.

I found what appears to be a bug today with Microsoft FTP on IIS 7.5 running on Windows Server 2008 R2 where I would get a false error message.

"The FTP site cannot be started.  Another FTP site may be using the same port."

This occurs when right clicking the FTP site and trying to start it.

Now I know that I do not have any services listening on TCP21 on my server.  A netstat shows this.

I did however have a test service running on TCP21 for a small period of time to verify the firewall portforwarding rules are setup correctly.

When attempting to start a FTP site on IIS7.5, it does not check that TCP21 is free, it checks prior at some stage.  To get it to recheck properally you need to restart the "Microsoft FTP Service"

Ahh now all is good... this bug had me scratching my head for a second!

NSClient++ Clients Report Arguments not Enabled

I am currently in the process of deploying the open source Nagios monitoring platform for a customer to provide them the ability to monitor their Windows machines, network infrastructure and virtual environment.

There are a couple of Nagios agents for Windows out there such as NSClient++, NC_Net and WINRPE which all do a great job of extracting Event Logs, Disk Utilization, Process Status, Service Status, Schedule Tasks, Windows Update Status, Anti Virus protection and much more.

In my deployment I chose to implement the NSClient++ on my Windows Server infrastructure.  The copy of NSClient++ I'm using is NSCP-0.4.1.90-x64.msi which I downloaded from the downloads page:

http://nsclient.org/nscp/downloads

When installing the application I chose to enable both Check_NT and Check_NRPE.

Check_NT is the legacy method for monitoring Windows hosts which only provides basic reporting functionality.

Check_NRPE is the more advanced reporting functionality which provides endless functionality including the ability to execute bat, vbs and powershell scripts on hosts monitored.

After NSClient++ was installed on my servers I tested it using the following command from my Nagios server:

[root@Nagios objects]# /usr/lib/nagios/plugins/check_nrpe -H 10.1.1.40

I (0,4,1,90 2013-02-04) seem to be doing fine...

Great all is well.

Now I went to shoot a test command to check my drive size on C:\ however when running this command I got the following error:

Exception processing request: Request contained arguments (not currently allowed, check the allow arguments option).

To Resolve this problem I needed to allow arguments in the nsclient.ini file on my Windows workstation by adding the following code:

[/settings/NRPE/server]
allow arguments=1

I added this configuration just below the allowed_hosts section of the default nsclient.ini file as shown in the following screenshot:

After making this change and restarting the NSClient++ Service all was well, I was able to successfully run Check_NRPE commands against my Nagios server.

Hope this blog post helps someone.