Monday, April 28, 2014

Problems Removing WSUS from SBS 2008

Once upon a time (approximately 4 years ago) I built an SBS 2008 server for a small business,  everything worked, everything was automated and the customer was very happy.  I was engaged to build this server, deploy the workstations and setup the network infrastructure but not maintain it.  Four years later the customer tracks me down by and calls me back with a mountain of problems.  The integrator here in Perth Western Australia (which will remain nameless) turned my perfect SBS deployment into a nightmare of problems due to unexperienced engineers performing server work - something which is far to common here in Australia.

One of the problems I faced was with the Windows Software Update Services, the service which "deploys updates" to computers on the network.

I found the "Update Services" service in a disabled state, the Windows Internal Database (WID) SQL Instance was completely missing from the server.  As a result the Windows Server Update Services mmc console would simply not launch and crash.  Starting the Update Services service also did not help improve the situation due to the missing SQL WID database.

Running a "wsusutil reset" simply failed due to the lack of database!

Fatal Error: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

 
Attempting to uninstall WSUS resulted in the following error messages:
 
The Windows Server Update Services 3.0 SP2 was not removed successfully because an error occurred.
 
 
Attempt to un-install Windows Server Update Services failed with error code 0x80070643.  Fatal error during installation
 
 
These uninstall errors were being caused because the SQL database no longer exists.  To remove a WSUS server which is missing its SQL database I found that you need to modify the following registry key:
 
 HKLM\Software\Microsoft\Update Services\Server\Setup
 
DWORD: wYukonInstalled
 
Set the decimal value to 0
 
 
Repeat the uninstall process (make sure you select not to remove the SQL database) as part of the uninstall and it will complete successfully.
 
 
After performing the uninstall, a simple server reboot was required before I re-installed WSUS using Windows Server 2008 Server Manager.

Changing Mailbox Permissions in Exchange Not Working

Over the years I have had numerous customers contact me and complain that when they change mailbox permissions such as "Manage Full Access Permissions" in the GUI or by changing permissions in Exchange Management Shell, the permissions do not apply to end users.  Other times administrators remove permissions to a mailbox and complain that the user still has access to the mailbox.

I always end up explaining to my customer "the change will work, just wait longer".  I want to take a few seconds to look into what is happening.

When you change permissions on a mailbox, the permission is not being set on the mailbox but in fact in Active Directory.  It takes time for the Exchange server to commit the change made to the information store.  This is due to the MBI cache which runs on the Exchange information store.

The information store caches information contained in the directory store and by default re-reads it every 120 minutes.  Any change made to Active Directory such as a mailbox permission change is not read by the information store for at least 2 hours.  It is also important to note that if the information store is idol (not busy) for over 15 minutes it can update its MBI cache faster from Active Directory.

Is there a way I can force the permission change made to the Exchange server?  Yes there are two ways of achieving this:
  • Reboot the Exchange Server
  • Restart the Information Store service
Both of these methods will result in down time.

It is also possible to tweak how often the MRI cache is updated by the information store service by modifying the registry.  The MRI cache is controlled by two registry settings, "Mailbox Cache Age Limit" and "Mailbox Cache Idle Limit".  The default for these settings are as follows:

Mailbox Cache Age Limit = 120 minutes
Mailbox Cache Idle Limit = 15 minutes

These registry values are REG_DWORD and are located under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem

NOTE: This registry entry is not a switch, it is a setting. If it is set to 1 the server rereads the cache every minute; if it is set to 2 the server rereads the cache every 2 minutes, and so forth.

If you change these registry keys you must restart the information store service for the change to take effect.

For more information about these registry values please see:

http://technet.microsoft.com/en-us/library/aa996988.aspx

As a general rule of thumb I would not recommend companies modify these default values, there is no need for it.  Just be patient and your permission change will automatically take effect.

In regards to Exchange 2013 I'm not sure if it follows the same behaviour as previous versions of Exchange such as 2007/2010.  In Exchange 2013 the information store has been overhauled and has been divided into separate worker processes responsible for each database.  Due to these architecture changes, the MBI cache may work/update differently.  If you are reading this and have experience with MRI cache and Exchange 2013, please feel free to comment below and share the knowledge!

Tuesday, April 22, 2014

Recursively Remove Mailbox Folder Permissions

You have been assigned a task to recursively remove mailbox permissions from a user mailbox in Exchange 2010/2013 using Exchange Server management tools.  How would you go about doing this?

Well there is a PowerShell command in Exchange called "Remove-MailboxFolderPermission" but this command only allows you to remove permissions from one folder at a time.  This cmdlet does not have a -Recursive option to allow you to propagate this against all sub folders.  The Set-MailboxFolderPermission, Remove-MailboxFolderPermission and Get-MailboxFolderPermission commands can be run by users who are a member of any of the following groups:
  • Organization Management
  • Recipient Management
  • Help Desk
I can run these commands as a member of the Organization Management role group against any user in my environment as shown in the following screenshot:


Now if we want to recursively remove mailbox folder permissions from all folders within a mailbox, we still need to use the Remove-MailboxFolderPermission command however we need to pipe this command into another command allowing us to recursively move through all the folders in the mailbox as the Remove-MailboxFolderPermission does not support the recursive action.  This can be done with the following command:

Get-MailboxFolder -Identity Bugs.Bunny:\FolderName -Recurse | Remove-MailboxFolderPermission -User username

At this point however you may hit another problem.  The Get-MailboxFolder cmdlet must be run under the user context who owns the mailbox.  For example, I want to use the Get-MailboxFolder cmdlet which is required in the above command to view the Inbox folder of my bugs.bunny test user.  To do this I run the following PowerShell command under my Administrator account which is a member of "Organization Management" but I get the following error:

Get-MailboxFolder Bugs.Bunny:\Inbox

The specified mailbox "bugs.bunny" doesn't exist.   
+ CategoryInfo          : NotSpecified: (0:Int32) [Get-MailboxFolder], ManagementObjectNotFoundException   
+ FullyQualifiedErrorId : 2DCA0FEB,Microsoft.Exchange.Management.StoreTasks.GetMailboxFolder


If I run the same PowerShell command under the security context of bugs.bunny however, we will not have any issues.  For example in the following screenshot I have the Exchange Management tools installed on a Windows 7 PC.  I am going to run the Exchange Management Shell as the bugs.bunny user account and run the same command.

 
Now that my Exchange Management Shell is running as bugs.bunny, when I run the command I will receive no error.
 
Get-MailboxFolder Bugs.Bunny:\Inbox
 
 
 
Important: The Get-MailboxFolder command must be utilised under the security context of the user whom owns the mailbox.

Now that we understand the limitations with the Get-MailboxFolder cmdlet, we understand that in order to recursively remove mailbox permissions from a mailbox, we must run the following command under the security context of the user account in order for the Get-MailboxFolder recursive command to work successfully.

Get-MailboxFolder -Identity Bugs.Bunny:\FolderName -Recurse | Remove-MailboxFolderPermission -User username

This method is not efficient as it involves the administrator contacting the user to gain access to their personal password.

There is however an alternative method for recursively removing permissions on a user mailbox using ExFolders, the new version of PFDavAdmin.  Download the ExFolders tool from the following TechNet URL:

http://gallery.technet.microsoft.com/Exchange-2010-SP1-ExFolders-e6bfd405

Once downloaded follow the instructions in the readme.txt file to install the tool and copy it to the Exchange bin folder.  ExFolders allows you to browse permissions on any of the sub folders of a users mailbox by simply right clicking on the folder and navigating to permissions.


If you want to recursively change permissions, simply right click on the folder tree you want to recursively remove the permissions from and click "Propagate folder ACEs".

 
Select the user which you want to remove from all sub folders and click Remove.
 
 
This will go through every sub folder in the hierarchy and remove the permissions.
 
 
Note: If the user you want to remove has been granted rights to only specific sub folders throughout the hierarchy, add the user to the root folder first so you are able to select the user.  For example if my Hulk Hogan user account had permissions to the alerts folder and one of my MVP folders, I would not be able to select the user from the root of the Inbox folder as it does not exist at this level.  By adding it to the root of the Inbox folder first, I am able to select the user for a recursive remove.
 
Important: Before you can use ExFolders you must have full mailbox access of the recipient for which you are looking to modify.  You can grant this using Exchange Management Console (EMC) by selecting "Manage Full Access Permission".  See in the screenshot I provided the AT\Administrator account rights which is the account I used to run ExFolders.
 
 

Tuesday, April 15, 2014

Delete all folders under a sub folder older then X days

Below is a simple PowerShell script which deletes all folders under D:\test older then 2 days.  Have fun!

$Now = Get-Date
$Days = "2"
$TargetFolder = "D:\test"
$LastWrite = $Now.AddDays(-$Days)
d:
cd "D:\test"
$Folders = get-childitem -path $TargetFolder | Where {$_.psIsContainer -eq $true} | Where {$_.LastWriteTime -le "$LastWrite"}

   
foreach ($Folder in $Folders)
   
{
   
write-host "Deleting $Folder" -foregroundcolor "Red"
   
Remove-Item $Folder -recurse -Confirm:$false
   
}
   

Thursday, April 3, 2014

Export List of Remote Desktop Service Logons from an RD Session Host

Sometimes as an administrator you may be required to create a detailed report of all Remote Desktop session logon's to a RD Session Host for auditing purposes.  All logon requests to a terminal server are recorded to TerminalServices-LocalSessionManager in the event log.

 
Simply tap into this with the Get-WinEvent PowerShell cmdlet.  The following PowerShell command will create you a spreadsheet in CSV format :
 
Get-WinEvent -LogName Microsoft-Windows-TerminalServices-LocalSessionManager/Operational | select TimeCreated,Message | Export-Csv c:\output.csv -NoTypeInformation

Outlook Meeting Request Randomly Sending Twice

I responded to a weird Outlook calendaring issue today involving an executive assistant to a CEO.  When the executive assistant send a meeting invite to a distribution group in particular "All Staff", the meeting request would send once and then again approximately 10 minutes later.  The customer was running Microsoft Exchange 2010 SP3 UR5 in a 2 node DAG (multi-role) with Outlook 2010 clients.

When told about it, I found this issue hard to comprehend so I went to the users computer and sent a test calendar invite out to All Staff and the Executive Group called "Please Ignore - IT Testing (please delete).  Needed to see it happen for myself!

The calendar invite went out first at 9:50am, then 13 minutes later resent at 10:53am without the user clicking a button.


After spending some time examining what was going on I checked message headers.  Whenever an email is relayed internally within an Exchange environment sent from OWA or Outlook, it will never have a message header.  The first email at 9:50am as expected had no header.  This screenshot is shown below.


Examining the second message which was sent at 10:53am does have a message header!


In the header notice under Content-Type it has "Apple-Mail".

Subject: Please Ignore - IT Testing (please delete)
From: ea@company.com
Content-Type: multipart/alternative;
               boundary="Apple-Mail-BE8A5DA5-12DD-4165-80F3-CB2112C97133"
Message-ID: 781CC260-A1EA-4FB2-B82F-0624F349F9AE@company.com
Date: Wed, 2 Apr 2014 10:53:03 +0800
To: "ea@company.com" ea@company.com,
Michael Smith mdto@company.com,
Executive Group ExecutiveGroup@company.com,
All Staff AllStaff@company.com
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0 (1.0)


After some more digging I found out the executive assistant has an Apple iPhone connected with Active Sync to her Exchange mailbox.  Her iPhone 4 was running Apple IOS 5.1.1 (9B206) where the latest version at the time was 7.1 (11D167), her phone was well out of date.

We upgraded her phone to the latest version 7.1 and it resolved the issue.  Very weird problem!

Outlook Error - Your server administrator has limited the number of items you can open simultaneously.

One of my customers running a custom line of business application called Office Automator by Office Automation which integrates with Microsoft Outlook experienced problems were experienced when sending emails.  The Outlook client was configured to use a direct RPC to the Exchange server.  The error users were experiencing was as follows:

Error No: -2147220731

Your server administrator has limited the number of items you can open simultaneously. Try closing messages you have opened or removing attachments and images from unsent messages you are composing.


The customer was running a single Exchange Server 2010 SP1 multi role in the backend. It is important to note that the user had multiple Outlook sessions open having Outlook running locally on their desktop computer and a second session initiated as a RemoteApp from a remote desktop session host server.

In addition to the error being generated on the Outlook client, the following error was also being generated in the Exchange 2010 application logs on the mailbox server role.

Event ID: 9646
Source: MSExchangeIS

Mapi session 'GUID' exceeded the maximum of 250 objects of type "objtMessage".


This error was resolved by increasing the amount of maximum objtMessage calls a single MAPI session for a mailbox can create from 250 to 500.  This was done by making the following registry key changes:
  1. On the Microsoft Exchange Mailbox Server exhibiting the error, open the Registry Editor.
  2. Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem
  3. Right-click ParametersSystem, point to New, and then click Key.
  4. Type MaxObjsPerMapiSession, and then press ENTER to name the new subkey.
  5. Right-click MaxObjsPerMapiSession, click New, and then click DWORD Value.
  6. Type objtMessage, and then press ENTER to name the object.
  7. Right-click objtMessage, and then click Modify.
  8. In the Value data box, type the number of objects to limit this entry to, and then click OK.
  9. Close the Registry Editor.
As you see below I increased the limit to 500.


 

For more information on tweaking Exchange Store Limits in Exchange 2010 please review the following TechNet article:

http://technet.microsoft.com/en-us/library/ff477612(v=exchg.141).aspx

Hope this post has been helpful!

Tuesday, April 1, 2014

Cannot Delegate Control - Only Shows Contacts!

I had a problem at a customer where users could not delegate other users access to their Exchange mailbox.  When a user opens up their delegates and clicks Add they cannot see the Global Address List, and hence cannot delegate access.

 
It only showed the All Contacts list, and of course contacts are not accounts and do not have access to network resources so they cannot be used for permission delegation.
 
 
This Exchange environment had just undergone cross-forest migration from an Exchange resource forest so there are a number of quirks which are popping up needing resolution.

After investigating I saw on the Default Offline Address Book that it included the Global Address List and the following "All Contacts" address list.

 
I removed the All Contacts Address List:
 
 
Then forced the OAB to regenerate with:
 
Get-OfflineAddressBook | Update-OfflineAddressBook
 
Then redistributed it to the Client Access Servers with:
 
Update-FileDistributionService CASNAME
 
For more information on this process refer to:
 http://clintboessen.blogspot.com.au/2009/05/how-oab-distribution-works.html

Then re-downloaded the OAB with Outlook

 
The issue was then fixed:
 
 
Having an address list defined on the Offline Address Book does not generally cause problems in Outlook with the ability to see the GAL.  I'm not sure what happened in this environment as when I re-added the All Contacts address list I could not reproduce the problem.