The Windows Backup engine could not be contacted. Retry the operation.

Today when attempting to perform a System State backup on a Domain Controller I received the following error message:

The Windows Backup engine could not be contacted. Retry the operation.
The RPC server is unavailable.

I also noticed the following event errors appearing in Event Viewer.

Log Name:      Application
Source:        VSS
Date:          13/03/2013 10:48:41 AM
Event ID:      12292
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {06d8e136-56f6-4048-93fb-a5943e949375} [0x80040154, Class not registered
].
Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Get Shadow Copy Properties
Context:
   Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}
   Class ID: {06d8e136-56f6-4048-93fb-a5943e949375}
   Snapshot Context: -1
   Snapshot Context: -1
   Execution Context: Coordinator

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 10:48:50 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x5888
Faulting application start time: 0x01ce1f9517234ddc
Faulting application path: C:\Windows\system32\wbengine.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 880e1970-8b88-11e2-aefa-005056a2000b

The above event error 12292 it provided us the Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}.  Looking in the registery under HKLM\System\CurrentControlSet\services\VSS\Providers\{5fdb6ef5-6ead-4610-995b-401c88626115} it shows this provider as the Backup Exec VSS Provider.



For some reason WBAdmin is trying to use the Backup Exec VSS Provider instead of the Microsoft VSS Provider.

I added the registry DWORD UseMicrosoftProvider to HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore with a value of "1" which is meant to force the backup to use the Microsoft provider.

This key had no effect, the backup still attempted to use the Symantec VSS Provider.  Next I used the following Symantec article 130940 to completely remove the Symantec backup exec agent from the server including removing registry keys.

http://www.symantec.com/business/support/index?page=content&id=TECH130940

After removing the Symantec backup exec agent I ran a test backup and the backup failed again with the same error.  Running a "vssadmin list providers" revealed that the Symantec VSS Provider was still in place despite following Symantec article 130940 which was meant to completely remove backup exec from a windows server.

Again we see same GUID of the Symantec provider which was presented in the event error and the registry, {5fdb6ef5-6ead-4610-995b-401c88626115}.

I then followed Symantec article 77585 to completely remove the Backup Exec VSS Provider by deleting the {5fdb6ef5-6ead-4610-995b-401c88626115} key from the following location in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\

http://www.symantec.com/business/support/index?page=content&id=TECH77585

After restarting the VSS service we see the Backup Exec VSS Provider is no longer available.

I then rebooted the server.  After a reboot I attempted another backup with wbadmin.  We got further this time but it still crashed out.

Some new event logs exist now:

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 2:47:07 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2

Log Name:      Application
Source:        VSS
Date:          13/03/2013 2:47:11 PM
Event ID:      8193
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...).  hr = 0x80070005, Access is denied.
.
Operation:
   Initializing Writer
Context:
   Writer Class Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
   Writer Name: NPS VSS Writer
   Writer Instance ID: {37bef355-a711-4241-a2bc-91f1181c845b}

 

VSS Event ID 8193 says that the VSS provider was denied access when opening a registry key under the security context of SYSTEM

 

SYSTEM\CurrentControlSet\Services\VSS\Diag,...). 

 

Damn it cut off!  We could use Sysinternals ProcMon to get the full path however lets just force FULL access for tye System account from the DIAG key downwards.

 

 

 After making this change I then tested another wbadmin.  Made no difference. :-(

 

I searched the entire registry for the GUID of the Backup Exec VSS Provider to ensure nothing was missed.  My search found nothing.  Whilst I have isolated the problem to the VSS Provider provided by Symantec, a change made by the Symantec Backup Exec agent remains and as a result wbadmin will not function.

If there is someone out there who has fixed this issue can you please comment below with your resolution to ensure others with this issue have a fix as this is not documented anywhere on the Internet.

The backup storage location is invalid. You cannot use a volume that is included in the backup as a storage location.

I was about to do a large number of Active Directory changes on a domain controller and needed to grab a system state backup before proceeding.  I took a system state backup using the wbadmin utility on a 2008 R2 SP1 domain controller by using the following command from a command prompt:

wbadmin start systemstatebackup -backupTarget:c:

When running the command I received the following error message:

The backup storage location is invalid. You cannot use a volume that is included in the backup as a storage location.

The resolution for this problem was adding a new Key and DWORD value to the system registry.  Create the following:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wbengine.
Create a key called "SystemStateBackup"
Set the value of this entry as follows:
Name: AllowSSBToAnyVolume
Data type: DWORD
Value data: 1

After creating the registry key the system state backup now completed successfully. 

The FTP site cannot be started. Another FTP site may be using the same port.

I found what appears to be a bug today with Microsoft FTP on IIS 7.5 running on Windows Server 2008 R2 where I would get a false error message.

"The FTP site cannot be started.  Another FTP site may be using the same port."

This occurs when right clicking the FTP site and trying to start it.

Now I know that I do not have any services listening on TCP21 on my server.  A netstat shows this.

I did however have a test service running on TCP21 for a small period of time to verify the firewall portforwarding rules are setup correctly.

When attempting to start a FTP site on IIS7.5, it does not check that TCP21 is free, it checks prior at some stage.  To get it to recheck properally you need to restart the "Microsoft FTP Service"

Ahh now all is good... this bug had me scratching my head for a second!

NSClient++ Clients Report Arguments not Enabled

I am currently in the process of deploying the open source Nagios monitoring platform for a customer to provide them the ability to monitor their Windows machines, network infrastructure and virtual environment.

There are a couple of Nagios agents for Windows out there such as NSClient++, NC_Net and WINRPE which all do a great job of extracting Event Logs, Disk Utilization, Process Status, Service Status, Schedule Tasks, Windows Update Status, Anti Virus protection and much more.

In my deployment I chose to implement the NSClient++ on my Windows Server infrastructure.  The copy of NSClient++ I'm using is NSCP-0.4.1.90-x64.msi which I downloaded from the downloads page:

http://nsclient.org/nscp/downloads

When installing the application I chose to enable both Check_NT and Check_NRPE.

Check_NT is the legacy method for monitoring Windows hosts which only provides basic reporting functionality.

Check_NRPE is the more advanced reporting functionality which provides endless functionality including the ability to execute bat, vbs and powershell scripts on hosts monitored.

After NSClient++ was installed on my servers I tested it using the following command from my Nagios server:

[root@Nagios objects]# /usr/lib/nagios/plugins/check_nrpe -H 10.1.1.40

I (0,4,1,90 2013-02-04) seem to be doing fine...

Great all is well.

Now I went to shoot a test command to check my drive size on C:\ however when running this command I got the following error:

Exception processing request: Request contained arguments (not currently allowed, check the allow arguments option).

To Resolve this problem I needed to allow arguments in the nsclient.ini file on my Windows workstation by adding the following code:

[/settings/NRPE/server]
allow arguments=1

I added this configuration just below the allowed_hosts section of the default nsclient.ini file as shown in the following screenshot:

After making this change and restarting the NSClient++ Service all was well, I was able to successfully run Check_NRPE commands against my Nagios server.

Hope this blog post helps someone.

An insight into Exchange 2013 Safety Net

Saftey Net is the new version of Transport Dumpster which was first introduced in Exchange 2007 and was continued in Exchange 2010.  I wrote about Transport Dumpster back in May 2011, please refer to the following blog post URL:

http://clintboessen.blogspot.com/2011/05/continuous-replication-block-mode-vs.html

However lets do a quick recap...

Transport Dumpster resides on all Hub Transport roles on both Exchange 2007/2010.  All messages which get delivered to users mailboxes is routed through a hub transport server and stored in Transport Dumpster.  In the event an email is sent from one user to another user on the same mailbox server, the mailbox server routes the email to a hub transport server and back again through MAPI.  This is to ensure that things such as journaling rules, transport rules and any other transport agents take effect, the message is filtered for malware/viruses (if configured), the message is trackable using message tracking logs and the message is copied in transport dumpster for a small period of time.

The following diagram shows what happens when Joe sent an email to Bob on the same mailbox server, the message goes to a hub transport server in the same AD site then back again to the mailbox server.

In Exchange 2010 the Transport Dumpster is controlled using the Set-TransportConfig cmdlet is configured to 15MB per database per default.  This means for every mailbox database the transport dumpster will always hold the last 15MB of email delivered to the  mailbox server.

What is the point?

In a database availability group (DAG) environment your active copy ships transaction logs to your passive copies.  What happens if suddenly your active mailbox server was to fail?  The passive copy may have not received the last transaction log, this will result in mail loss (assuming file mode replication is used).  After a mailbox database failover in a DAG environment, the new active copy will check for any non-replicated emails in the transport dumpster.  In the event it requires additional email, it will retrive the missing content from the dumpster.

Exchange 2013 Safety Net

Now that we have done a quick recap of Exchange Transport Dumpster which existed in Ex2010/2007 for DAG/CCR environments, lets look at whats new in Exchange 2013 Safety Net.

Unlike Transport Dumpster, Safety Net you cannot configure how many MB of messages to store, only how long you want to store messages with the default being 2 days.  This is because by setting a limit on the amount of data can result in data loss during a failover in the event a large amount of data had not replicated to the passive database copy.  Microsoft wanted to design Safety Net as a lossless solution hence this design change.

Message resubmissions from Safety Net are initiated by the Active Manager component of the Microsoft Exchange Replication service that manages DAGs and mailbox database copies. No manual actions are required to resubmit messages from Safety Net.

Safety Net is a queue that's associated with the Transport service on a Mailbox server. This queue stores copies of messages that were successfully processed by the server.  Safety Net uses the mail.que database, the same database which is used to store messages in queue.  As by default Safety Net will keep the last 2 days worth of email in this queue, expect the mail.que database to be larger then previous versions of Exchange.

The mail.que database file uses the Extensive Storage Engine (ESE), the same database technology which is used by the mailbox databases themselves.

Another improvement with Exchange 2013 Safety Net over Transport Dumpster is redundancy.  Safety Net itself is now redundant, and is no longer a single point of failure. This introduces the concept of the Primary Safety Net and the Shadow Safety Net. If the Primary Safety Net is unavailable for more than 12 hours, resubmit requests become shadow resubmit requests, and messages are re-delivered from the Shadow Safety Net.

With Safety Net being redundant, you can now feel confident in configuring the database mount dial setting to a more relaxed setting other then lossless and still feel confident that email will not be lost during failover.

msDS- Attributes in Active Directory

This is a short post to explain what msDS- attributes are in Active Directory.  As an administrator responsible for maintaining your companies Active Directory environment at some stage you have probably seen a bunch of msDS attributes linked to class objects such as user accounts.

What are msDS- attributes and how are they different to other attributes?

msDS- attributes are designed to hold data for Microsoft applications.  As best practice Microsoft recommends administrators never modify msDS- attributes as it can cause issues with applications.

Any attribute beginning with msDS- reference it, but do not modify it unless the changes is made through the application to avoid issues in your environment.

Group Policy Scripts in Windows Server 2008/2008R2/2012

With the release of Group Policy Management Console (GPMC) on server 2003, Group Policy had many sample scripts which were very handy when working with Group Policy.  These scripts included:

BackupAllGPOs.wsf
BackupGPO.wsf
CopyGPO.wsf
CreateEnvironmentFromXML.wsf
CreateGPO.wsf
CreateMigrationTable.wsf
CreateXMLFromEnvironment.wsf
DeleteGPO.wsf
DumpGPOInfo.wsf
DumpSOMInfo.wsf
FindDisabledGPOs.wsf
FindDuplicateNamedGPOs.wsf
FindGPOsByPolicyExtension.wsf
FindGPOsBySecurityGroup.wsf
FindGPOsWithNoSecurityFiltering.wsf
FindOrphanedGPOsInSYSVOL.wsf
FindSOMsWithExternalGPOLinks.wsf
FindUnlinkedGPOs.wsf
GetReportsForAllGPOs.wsf
GetReportsForGPO.wsf
GrantPermissionOnAllGPOs.wsf
ImportAllGPOs.wsf
ImportGPO.wsf
Lib_CommonGPMCFunctions.js
ListAllGPOs.wsf
ListSOMPolicyTree.wsf
QueryBackupLocation.wsf
RestoreAllGPOs.wsf
RestoreGPO.wsf
SampleEnvironment.xml
SampleMigrationTable.migtable
ScriptingReadme.rtf
SetGPOCreationPermissions.wsf
SetGPOPermissions.wsf
SetGPOPermissionsBySOM.wsf
SetSOMPermissions.wsf

These scripts are no longer packaged with Group Policy Management console in Server 2008/2008R2 or Server 2012.  Microsoft however has released these scripts for download to be used with any of the later versions of the Windows Operating system.  To download these scripts please view the following TechNet website:

http://www.microsoft.com/en-us/download/details.aspx?id=14536

Converting Basic Disk to Dynamic Disk Stripped SMB Shares

We experienced a very interesting problem on a Windows Server 2003 file server in which all SMB shares were removed from the file server upon reboot.  We were in the process of moving a data volume from a Raw Device Mapping (RDM) to a VMDK virtual disk file in VMware.  To achieve this we were going to perform the following tasks within the virtual machine:

1. Present a new VMDK to the Virtual Machine
2. Convert the NTFS Volume from Basic to Dynamic
3. Create a Software RAID1 Mirror between the RDM and VMDK volumes
4. Allow for Windows Software Raid to replicate the data
5. Remove the RAID1 Mirror for the RDM volume.

Upon converting the basic volume to a dynamic volume, Windows Server 2003 asked to reboot the server.  Upon reboot, all SMB shares on the file server automatically removed themselves.  There were hundreds of shares and as a result users could not get access to resources.

To recover the shares we simply restored the system registry which can be achieved through performing a system state restore.  Windows stores all SMB shares under the following location in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares

We found the following Symantec article very helpful during this procedure:

http://www.symantec.com/business/support/index?page=content&id=TECH159845

Your computer can't connect to the remote computer because an error occurred on the remote computer that you want to connect to

I experienced an issue at a customer site with with a new Remote Desktop Services deployment on Windows Server 2008 R2 when building a Server Farm.

 

When Windows 7 PC's accessed a RemoteApp or attempted create a remote desktop session using the Microsoft Terminal Services Client (MSTSC.exe) they were able to connect to the farm without problems.

 

When an Windows XP PC accessed the remote desktop farm, the following error was experienced:

 

"Your computer can't connect to the remote computer because an error occurred on the remote computer that you want to connect to.  Contact your network administrator for assistance."

 

After researching the issue it turned out that the RD Session Hosts needed to be configured to use RDP Security as the Security Layer.  After installing a custom trusted certificate to the RDP-Tcp connection to ensure users connecting to the session hosts do not receive RDP Certificate not trusted warnings the issue started occuring.

These configuration options can be found under "Remote Desktop Session Host Configuration"

 

By default the Security layer was set to Negotiate.

 

Set all servers to RDP Security Layer in your farm to ensure both XP and Windows 7 clients can connect.

WSUS Not Working

I had an issue at a customer today where a new WSUS server was not patching clients.  I went through the WSUS configuration, all was correct.

Problem Symptoms

The symptoms of this problem are as follows:

Windows 7/2008 clients recieved the following message when attempting to receive updates from the WSUS server.

Windows could not check for updates automatically. (Important)

 

For these workstations, they report in WSUS with Last Status Report as Not yet reported.

 

 

On member servers and workstations experiancing the update issue, the following was observed in the WindowsUpdate.log file found under %WINDIR%\WindowsUpdate.log

WARNING: Search callback failed, result = 0x80072EFD
WARNING: Failed to find updates with error code 80072EFD

Here is the full Log:
2013-01-30 15:21:29:317  772 a54 Agent *************
2013-01-30 15:21:29:317  772 a54 Agent ** START **  Agent: Finding updates [CallerId = AutomaticUpdates]
2013-01-30 15:21:29:317  772 a54 Agent *********
2013-01-30 15:21:29:317  772 a54 Agent   * Online = Yes; Ignore download priority = No
2013-01-30 15:21:29:317  772 a54 Agent   * Criteria = "IsInstalled=0 and DeploymentAction='Installation' or IsPresent=1 and DeploymentAction='Uninstallation' or IsInstalled=1 and DeploymentAction='Installation' and RebootRequired=1 or IsInstalled=0 and DeploymentAction='Uninstallation' and RebootRequired=1"
2013-01-30 15:21:29:317  772 a54 Agent   * ServiceID = {3DA21691-E39D-4DA6-8A4B-B43877BCB1B7} Managed
2013-01-30 15:21:29:317  772 a54 Agent   * Search Scope = {Machine}
2013-01-30 15:21:29:491  772 a54 Setup Checking for agent SelfUpdate
2013-01-30 15:21:29:539  772 a54 Setup Client version: Core: 7.6.7600.256  Aux: 7.6.7600.256
2013-01-30 15:21:31:552  772 a54 Misc WARNING: Send failed with hr = 80072efd.
2013-01-30 15:21:31:557  772 a54 Misc WARNING: SendRequest failed with hr = 80072efd. Proxy List used: <(null)> Bypass List used : <(null)> Auth Schemes used : <>
2013-01-30 15:21:31:557  772 a54 Misc WARNING: WinHttp: SendRequestUsingProxy failed for <http://ang-pth-upd1/selfupdate/wuident.cab>. error 0x80072efd
2013-01-30 15:21:31:557  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation MakeRequest failed. error 0x80072efd
2013-01-30 15:21:31:557  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x80072efd
2013-01-30 15:21:31:557  772 a54 Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x80072efd
2013-01-30 15:21:49:646  772 a54 Misc WARNING: Send failed with hr = 80072efd.
2013-01-30 15:21:49:646  772 a54 Misc WARNING: SendRequest failed with hr = 80072efd. Proxy List used: <(null)> Bypass List used : <(null)> Auth Schemes used : <>
2013-01-30 15:21:49:646  772 a54 Misc WARNING: WinHttp: SendRequestUsingProxy failed for <http://ang-pth-upd1/selfupdate/wuident.cab>. error 0x80072efd
2013-01-30 15:21:49:646  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation MakeRequest failed. error 0x80072efd
2013-01-30 15:21:49:646  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x80072efd
2013-01-30 15:21:49:646  772 a54 Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x80072efd
2013-01-30 15:21:53:899  772 a54 Misc WARNING: Send failed with hr = 80072efd.
2013-01-30 15:21:53:899  772 a54 Misc WARNING: SendRequest failed with hr = 80072efd. Proxy List used: <(null)> Bypass List used : <(null)> Auth Schemes used : <>
2013-01-30 15:21:53:899  772 a54 Misc WARNING: WinHttp: SendRequestUsingProxy failed for <http://ang-pth-upd1/selfupdate/wuident.cab>. error 0x80072efd
2013-01-30 15:21:53:899  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation MakeRequest failed. error 0x80072efd
2013-01-30 15:21:53:899  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x80072efd
2013-01-30 15:21:53:899  772 a54 Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x80072efd
2013-01-30 15:21:58:153  772 a54 Misc WARNING: Send failed with hr = 80072efd.
2013-01-30 15:21:58:153  772 a54 Misc WARNING: SendRequest failed with hr = 80072efd. Proxy List used: <(null)> Bypass List used : <(null)> Auth Schemes used : <>
2013-01-30 15:21:58:153  772 a54 Misc WARNING: WinHttp: SendRequestUsingProxy failed for <http://ang-pth-upd1/selfupdate/wuident.cab>. error 0x80072efd
2013-01-30 15:21:58:153  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation MakeRequest failed. error 0x80072efd
2013-01-30 15:21:58:153  772 a54 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x80072efd
2013-01-30 15:21:58:153  772 a54 Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x80072efd
2013-01-30 15:21:58:153  772 a54 Misc WARNING: DownloadFileInternal failed for http://ang-pth-upd1/selfupdate/wuident.cab: error 0x80072efd
2013-01-30 15:21:58:153  772 a54 Setup WARNING: SelfUpdate check failed to download package information, error = 0x80072EFD
2013-01-30 15:21:58:153  772 a54 Setup FATAL: SelfUpdate check failed, err = 0x80072EFD
2013-01-30 15:21:58:153  772 a54 Agent   * WARNING: Skipping scan, self-update check returned 0x80072EFD
2013-01-30 15:21:58:213  772 a54 Agent   * WARNING: Exit code = 0x80072EFD
2013-01-30 15:21:58:213  772 a54 Agent *********
2013-01-30 15:21:58:213  772 a54 Agent **  END  **  Agent: Finding updates [CallerId = AutomaticUpdates]
2013-01-30 15:21:58:213  772 a54 Agent *************
2013-01-30 15:21:58:213  772 a54 Agent WARNING: WU client failed Searching for update with error 0x80072efd
2013-01-30 15:21:58:213  772 a0c AU >>##  RESUMED  ## AU: Search for updates [CallId = {44F20E45-61DE-4968-9DE7-32BD9B0ACFB0}]
2013-01-30 15:21:58:213  772 a0c AU   # WARNING: Search callback failed, result = 0x80072EFD
2013-01-30 15:21:58:213  772 a0c AU   # WARNING: Failed to find updates with error code 80072EFD
2013-01-30 15:21:58:213  772 a0c AU #########
2013-01-30 15:21:58:213  772 a0c AU ##  END  ##  AU: Search for updates [CallId = {44F20E45-61DE-4968-9DE7-32BD9B0ACFB0}]
2013-01-30 15:21:58:213  772 a0c AU #############

Resolution

The resolution to these problems in my case was upgrading my WSUS 3.0 server to Service Pack 2 by applying KB2720211.  This can be found under the following URL:

http://support.microsoft.com/kb/2720211

I then wanted all computers in my domain to attempt to detect new updates using the wuauclt.exe /discovernow, to ensure the WSUS console populated correctly and all machines started downloading new updates.  To do this I populated a list of all computers in the domain using the DSQUERY tool.

DSQUERY COMPUTER “DC=mydomain,DC=com” -o rdn -limit 1000 > c:\machines.txt

I then used Excel to construct the following psexec statement in mass wrapping the "psexec \\" and " wuauclt.exe /detectnow" strings around the computer name.

psexec \\COMPUTER1 wuauclt.exe /detectnow
psexec \\COMPUTER2 wuauclt.exe /detectnow
psexec \\COMPUTER3 wuauclt.exe /detectnow

I copyed this content from Excel to a batch script.

I then downloaded psexec from the following sysinternals website on TechNet:

http://technet.microsoft.com/en-au/sysinternals/bb897553.aspx

Ran the batch script and forced every computer on my domain to discover for updates instead of having to wait for them to do it automatically.

Group Policy Software Installation Not Working

Today a customer had a few Windows 7 workstations which were  not receiving software which is pushed through group policy.  RSOP.msc and gpresult had confirmed that the group policy instructing the software installation was successfully applied to the workstation.  Despite group policy applying, the assigned application would not install.

I then found out that the following events had occurred:

1. The application had installed through Group Policy in the past.
2. An Administrator uninstalled the application through Add/Remove Programs in Control Panel
3. The administrator ran gpupdate /force assuming the application would re-install automatically.

This is normal behavior  if an Administrator manually removes an assigned application, it will not automatically redeploy upon computer reboot.  This is because the Software Installation Engine which runs on the Workstation does not know the assigned application has been removed, hence it does not attempt to reinstall.

How do you force the application to be re-installed?

All applications which are assigned through Group Policy get added to the computers registry under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\AppMgmt

The applications will appear under this policy as a bunch of GUID's.

If you select one of the GUID's it it will have a string value called GPO Name which will display the name of the Group Policy Object.

Delete the GUID of the application which is not deploying.

Run a GPUpdate /Force

Reboot the workstation.

The application will now install

Exchange 2003 and Windows Server 2012 Domain Controller

A customer asked me today if they can upgrade to their domain controllers to Windows Server 2012 whilst running Exchange Server 2003.

Exchange 2003 does not support Windows Server 2012 Domain Controllers.

This not only includes the 2012 Forest Funtional Level (FFL) and 2012 Domain Functional Level (DFL) but the Windows Server 2012 DC's.  If you have a Windows Server 2012 Domain Controller running 2003 FFL/DFL, this is still not supported.

There is no support planned for Windows Server 2012 Domain Controllers with Exchange 2003.

Windows 8 Won't Shutdown

One of my friends purchased a Windows 8 computer and complained to me that no matter what they do the machine will not shutdown through use of the Start Portal, Task Manager or even using the shutdown command from command prompt such as:

shutdown.exe -s -f

After some research it turns out that the issue was caused by the Windows 8 new dynamic ticking feature.  This feature is aimed at helping you increase your devices battery life by varying the speed of your device's processor based on its workload.  For some reason this feature on this computer prevented shutdown all together.

The fix was to use BCDEdit to disable the BCDEdit feature using the following command from an elevated command prompt.

bcdedit /set disabledynamictick yes

During my research I also saw complaints about the dynamic ticking feature causing Windows 8 to crash and freeze with older processors and older mainboards.  For more information about these problems other people have had please refer to following blog post:

http://www.oscarcao.com/blog/2012/08/21/fix-for-windows-8-hanging-and-freezing

Dell Wireless Driver

I had a bit of a drama finding a driver for the following hardware ID today on a customers laptop.

PCI\VEN_168C&DEV_002B&SUBSYS_02051028&REV_01
PCI\VEN_168C&DEV_002B&SUBSYS_02051028
PCI\VEN_168C&DEV_002B&CC_028000
PCI\VEN_168C&DEV_002B&CC_0280
It turned out to be a Dell Wireless 1702 802.11b/g/n adapter which can be found under R301705.exe.  Download this driver from the following website:

http://www.dell.com/support/drivers/au/en/aubsd1/DriverDetails?driverId=R301705

Hope this post has been helpful.

Citrix Gateway 3.0 - The server certifiate specified is unusable

Today I attempted to replace a certificate on a Citrix Gateway.  When replacing the certificate in the Secure Gateway Configuration Wizard I received the following error message:

"The server certificate specified is unusable"

Citrix has the following knowledge base article on this problem however it did not resolve my issue.  This knowledge base article can be found under the following address.

http://support.citrix.com/article/CTX118548

I created the certificate on a Windows 2008 server using MMC console using a procedure similar to the following:

http://techontip.wordpress.com/tag/san-certificate-request-from-windows-mmc/

I exported the certificate with private key and imported it onto my Citrix Gateway server running Windows 2003 server.

After some further investigation it turns out that Citrix Gateway does not support certificates which were exported from one server then imported.  The Certificate signing request MUST be created on the Citrix Gateway server otherwise Citrix will complain about the certificate.  After recreating the certificate on the Citrix Gateway server I did not have any issues.

I was using Citrix Gateway version 3.0.0

Hope this post has been helpful.

Whats this new In-Place buzz word?

Some of my customers have been asking me lately about some of the new In-Place features of Exchange 2013.  In-Place is a new buzz word which has been adopted by the Exchange product team when naming some of the features of Exchange 2013.

For example we have:

In-Place Archive (formally Personal Archive in Exchange 2010)
In-Place Hold (formally Legal Hold in Exchange 2010)
In-Place Discovery (formally Multi-Mailbox Search in Exchange 2010)

While these features were in Exchange 2010, they have been improved and revamped in Exchange 2013 with new interfaces and closer interaction with other products such as SharePoint 2013 and Lync 2013.

For more information on In-Place Archive please visit:

http://technet.microsoft.com/en-us/library/dd979800.aspx

For more information on In-Place Hold please visit:

http://technet.microsoft.com/en-us/library/ff637980.aspx

For more information on In-Place Discovery please visit:

http://technet.microsoft.com/en-us/library/dd298021.aspx

A quick look at Exchange 2013 Managed Availability

Microsoft Exchange 2013 has a new monitoring and alerting engine built into the product called Managed Availability.  Managed Availability detects, alerts and recovers problems as they occur within the product.

In previous versions of Exchange such as 2007/2010, Microsoft recommended Administrators used System Center Operations Manager (SCOM) to monitor an Exchange environment.  In Exchange 2013, the product now has its own monitoring engine which companies can leverage to provide an insite into their email infrastructure.

Note: SCOM Intergration with Exchange 2013 will still be supported

The Managed Availability platform was designed to provide a monitoring solution for a single server deployment of Exchange through to the largest deployments of Exchange in the world.  Microsoft leveraged its experiance with the Office 365 and Exchange Online over the past 6 years to determine which alerts from the SCOM management pack are useful and which alerts are not.  From the 1100 alerts in the management pack, 150 were seen as useful.

For common re-occuring issues which Microsoft experianced in the Office 365 environment, an automated recovery process was put in place to automatically resolve issues to ensure administrative intervention was not required. These automated recovery processes are not available in Exchange Server 2010.  In Exchange 2013, Microsoft has brought the recovery workflow engine based on its learnings from Office 365 to an on-premises environment so companies can benefit in automatic recovery of Exchange related issues.  In my opinion this is a significant selling point between Microsoft Exchange 2013.

To ensure you have a firm understanding on the Exchange 2013 Managed Availability engine, I will run through the core components below.

Probes

Probes essentially probe the environment to identify portential problems with the environment.  They are similar to the test cmdlets in past releases of Exchange in the way they measure the perception of services by executing end-to-end user transactions against core services.

Monitors

Data collected by probes is fed into Monitors.  Monitors look at the results of probes and come to a conclution based on a number of additional checks programmed into each monitor.  The conclution of a monitor is either the service is healthy or unhealthy.

The correlation between Probes an Monitors is Many to One where Many Probes can be fed into a single monitor.

Responders

Responders only execute in the event a monitor is marked in an unhealthy state.  Depending on the monitor which entered an unhealthy state, there are severa responders available to respond to the monitor:

• Restart Responder  Terminates and restarts service
• Reset AppPool Responder  Cycles IIS application pool
• Failover Responder  Takes an Exchange 2013 Mailbox server out of service
• Bugcheck Responder  Initiates a bugcheck of the server
• Offline Responder  Takes a protocol on a machine out of service (in the event a load balanced clustered environment is available, this the faulty service will not disrupt services)
• Escalate Responder  escalates an issue
• Specialized Component Responders  

For more information about Exchange 2013 Managed Availability please see the following blogpost put together by Ross Smith IV which can be found on the following website:

http://blogs.technet.com/b/exchange/archive/2012/09/21/lessons-from-the-datacenter-managed-availability.aspx

Schemus Mail Configuration hanging on Loading

I was configuring the Schemus Mail Configuration tool to synchronise recipient information from an Active Directory forest to Symantec Cloud but ran into an issue with the Synchronisation tool.

I entered the configuration for an Active Directory domain controller into the tool along with a username and password as follows:

When clicking next the wizard would get stuck on Loading and not progress any further.

After clicking cancel and closing the wizard then selecting View --> Logs I noticed multiple entries of "Cannot access the LDAP server. Malformed reply from SOCKS server"

 

Can't access the LDAP server. Malformed reply from SOCKS server

com.metanate.schemus.data.source.SourceException: Malformed reply from SOCKS server
 at com.metanate.schemus.data.source.LdapSource.setContext(Unknown Source)
 at com.metanate.schemus.data.source.LdapSource.ensureOpen(Unknown Source)
 at com.metanate.schemus.data.source.LdapSource.getSearchBases(Unknown Source)
 at com.metanate.schemus.config.gui.wizard.swing.source.SearchSettingsPanel$2.construct(Unknown Source)
 at com.metanate.schemus.config.gui.wizard.swing.SwingWorker$2.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Caused by: javax.naming.CommunicationException: 10.100.8.91:389 [Root exception is java.net.SocketException: Malformed reply from SOCKS server]
 at com.sun.jndi.ldap.Connection.(Unknown Source)
 at com.sun.jndi.ldap.LdapClient.(Unknown Source)
 at com.sun.jndi.ldap.LdapClient.getInstance(Unknown Source)
 at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
 at com.sun.jndi.ldap.LdapCtx.(Unknown Source)
 at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)
 at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)
 at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)
 at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)
 at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
 at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)
 at javax.naming.InitialContext.init(Unknown Source)
 at javax.naming.ldap.InitialLdapContext.(Unknown Source)
 ... 6 more
Caused by: java.net.SocketException: Malformed reply from SOCKS server
 at java.net.SocksSocketImpl.readSocksReply(Unknown Source)
 at java.net.SocksSocketImpl.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at java.net.Socket.(Unknown Source)
 at java.net.Socket.(Unknown Source)
 at com.sun.jndi.ldap.Connection.createSocket(Unknown Source)

 

SOCKS is a proxy technology.  After checking Internet Options it turns out that a proxy server was configured on the server.

Removing the proxy server from Internet Options resolved the problem.

Changing Password on Administrator Accounts - Performing an Audit

This article addresses a common task which many administrators have to address within their career as as an IT professional - changing the password on a core administrator account.

Scenario

It is well known that Administrators should always create dedicated service accounts with appropriate access to be used by network applications on a Microsoft network.  However there is always a case of a lazy administrator in the past who could not be bothered to create dedicated service accounts so they use the default domain admin account "Domain\Administrator" for applications and services to use.  So what happens in the event when there are applications and services across the network using a default domain account, everyone including previous employees, current employees and end users know the password to this account and you don't know exactly applications are using the account?  This article addresses exactly this situation.

Solution

The only way to identify all applications using an account for authentication is to revert to audit logs on domain controllers and identify the IP addresses in which the authentication attempts have been initiated from.  Once you have the IP addresses as an administrator your able to dig down into the servers configuration and identify what applications are installed and figure out what is making the authentication attempts from the account.  No application will be able to tell you exactly what program is performing the authentication request because all applications are different.  For example some applications may store the domain administrator credentials in a text configuration file, others might store the credentials in some type of database table and others might simply store it in a service or scheduled task.  No audit application understand the inter workings of every single application made, at best they can only look for where applications "usually" store credentials and return results based on that.

Another thing to note is each domain controller stores audit logs for authentication requests made against the individual DC.  There is no place where you can look at all authentication requests against domain controllers on a domain wide level without using additional software.  To gain inside into what authentication requests are being made on your network I recommend a product such as Snare Server.  Snare is seen by many as the industry standard for capturing and filtering audit and event log data.  Snare Server will pull audit logs from all domain controllers in your organisation and allow you to quickly identify exactly what servers in your organisation are using a specific account.

After ringing them for pricing, they are very cost effective compared to other audit collection tools on the market.  Its priced based on geographical region, so you will need to contact them to get pricing for your country.

Check out the following video which goes through Snare Server in detail:

http://www.intersectalliance.com/Contact.html?Video=SnareServer

Discovery VBS Script

As mentioned above the only way to perform a thorough audit of what applications in your environment are using a specific account is to revert to audit logs.  However below I will show you a handy VBS script which is able scan through all computer accounts in your domain and check if they are using an administrator account for a service or scheduled task.  This script can be downloaded from the following location, just rename it to *.vbs.

https://sites.google.com/site/cbblogspotfiles/ScanForUserID.txt

This script requires you to modify two fields:

• The strSearchFor field is the account in which you want to find.  For example, Administrator.
• The strExclude field are computers or locations within Active Directory in which you want to exclude.  If you want to exclude nothing you can leave this field as ""

strSearchFor = "Administrator"
strExclude = "dx-iren,dx-iren2,OU=Computers"

When you run the script it will display output in a webpage.  It pings each machine before performing the scan to ensure it is online so ensure ICMP is enabled on your windows firewall.  This can be done with group policy.

Ensure you launch it using cscript from a command prompt running as Administrator "Run As Administrator" to get around User Account Control (UAC) restrictions.

The audit results get pushed out to a CSV file under C:\results.csv and can be opened in Excel.

 Hope this post has been helpful and goodluck.

Identify number of items in folder Outlook Cached Exchange Mode and Exchange

This blog post shows you how to identify if a folder within your Outlook OST file has become out of sync with your Exchange server for Outlook clients which are running under cached Exchange mode.  To do this we will look at the item count both of the cached outlook client and the Exchange server.

To get the item count of a folder in Outlook, this can be done in two ways.  The first method is by using a VBS script such as the one from David Lee's blog:

http://techniclee.wordpress.com/2012/02/29/exporting-outlook-message-counts-to-excel/

A copy of this script can be found below.

'Declare some variables
Dim olkApp, olkSes

'Connect to Outlook
Set olkApp = CreateObject("Outlook.Application")
Set olkSes = olkApp.GetNamespace("MAPI")
olkSes.Logon olkApp.DefaultProfileName

'Call the export process once for each folder count to be exported
'Format is ExportMessageCountToExcel <Path to Outlook Folder>, <Path and filename of the Excel file to export to>, <Number of the sheet the count goes on>
'The following lines are examples.  Edit them as needed.  Add additional lines as desired.
ExportMessageCountToExcel "Mailbox - Doe, John\Inbox", "C:\Message_Counts.xlsx", 1
ExportMessageCountToExcel "Personal Folders\Projects", "C:\Message_Counts.xlsx", 2

'Disconnect from Outlook
olkSes.Logoff
Set olkSes = Nothing
Set olkApp = Nothing
WScript.Quit

Sub ExportMessageCountToExcel(strFolder, strWorkbook, intSheet)
    Const EXCEL_COL = 1
    Dim olkFld, excApp, excWkb, excWks, lngRow
    Set olkFld = OpenOutlookFolder(strFolder)
    Set excApp = CreateObject("Excel.Application")
    Set excWkb = excApp.Workbooks.Open(strWorkbook)
    Set excWks = excWkb.Worksheets(intSheet)
    lngRow = excWks.UsedRange.Rows.Count
    If lngRow = 1 Then 
        If excWks.Cells(lngRow,1) <> "" Then
            lngRow = lngRow + 1
        End If
    Else
        lngRow = lngRow + 1
    End If
    excWks.Cells(lngRow, EXCEL_COL) = olkFld.Items.Count
    Set excWks = Nothing
    excWkb.Close True
    Set excWkb = Nothing
    excApp.Quit
    Set excApp = Nothing
    Set olkFld = Nothing
End Sub

Function OpenOutlookFolder(strFolderPath)
    Dim arrFolders, varFolder, bolBeyondRoot
    On Error Resume Next
    If strFolderPath = "" Then
        Set OpenOutlookFolder = Nothing
    Else
        Do While Left(strFolderPath, 1) = "\"
            strFolderPath = Right(strFolderPath, Len(strFolderPath) - 1)
        Loop
        arrFolders = Split(strFolderPath, "\")
        For Each varFolder In arrFolders
            Select Case bolBeyondRoot
                Case False
                    Set OpenOutlookFolder = olkApp.Session.Folders(varFolder)
                    bolBeyondRoot = True
                Case True
                    Set OpenOutlookFolder = OpenOutlookFolder.Folders(varFolder)
            End Select
            If Err.Number <> 0 Then
                Set OpenOutlookFolder = Nothing
                Exit For
            End If
        Next
    End If
    On Error GoTo 0
End Function

This script will output the results to an Excel spreadsheet as shown in the following screenshot:

The second method is by selecting all items within the Outlook folder by pressing "CTRL + A" then pressing "ENTER".  Make sure you select NO!

Now to check the number of items in my Sent Items on the Exchange server.  This is done with the following PowerShell command:

Get-MailboxFolderStatistics -Identity "clint" -FolderScope "SentItems"

This shows the number of items on my Exchange server matches my Outlook client meaning my Outlook Cached Copy is indeed in sync!

Exchange ActiveSync Error (500) Internal Server Error

One of my customers had an issue on their Exchange 2010 server regarding Exchange ActiveSync.  The mobile device would Autodiscover the settings however it would fail to synchronise email.  After investigating the issue I quickly isolated the issue to the users Active Directory user account.  This was determined after I recreated the users mailbox by exporting to PST, disabling the mailbox, creating a new mailbox then reimporting the mailbox content.

I used the following command in PowerShell to test the users ActiveSync configuration status.

Test-ActiveSyncConnectivity -MailboxCredential (Get-Credential domain\username) -UseAutodiscoverForClientAccessServer

The command failed.

I then piped the output to a format list function to provide more details about the error received.

The error received was as follows:

"[System.Net.WebException]: The remote server returned an error: (500) Internal Server Error."

As I knew the problem was not to do with the mailbox contents as I had recently recreated the users mailbox, and this user was the only user effected by the issue, it had to be a problem with the Active Directory user account itself.

After investigating the user account I noticed the account was no longer inheriting permissions in Active Directory.  After re-enabling inheritable permissions, this resolved the problem.

Troubleshooting Fun with the Test-ActiveSyncConnectivity cmdlet

The Test-ActiveSyncConnectivity cmdlet is a great command for testing Active Sync connectivity with a device.  This can be used to test Active Sync connectivity with any user in your domain.  You may have issues using this command however and the example commands provided in the Get-Help in powershell are wrong.  For example the powershell help instructs administrators to enter the users UPN or DomainNetBIOS\Username format for the MailboxCredential parameter.

Test-ActiveSyncConnectivity -UseAutodiscoverForClientAccessServer $true -URL "http://contoso.com/mail" -MailboxCredential pauls@contoso.com

However the MailboxCredential parameter requires the full credentials including username and password of the user account in which your attempting to test.  Not doing so will result in the following error.

Cannot process argument transformation on parameter 'MailboxCredential'. Cannot convert the "jag" value of type "System.String" to type "System.Management.Automation.PSCredential".
    + CategoryInfo          : InvalidData: (:) [Test-ActiveSyncConnectivity], ParameterBindin...mationException
    + FullyQualifiedErrorId : ParameterArgumentTransformationError,Test-ActiveSyncConnectivity

To specify both the username and password along with the Test-ActiveSyncConnectivity cmdlet you can use the Get-Credential cmdlet, for example:

Test-ActiveSyncConnectivity -MailboxCredential (Get-Credential domain\username) -UseAutodiscoverForClientAccessServer

This command above will work.  What I also noticed is in the powershell help it mentions to use the $true switch for the UseAutodiscoverForClientAccessServer parameter.  You must not use the $true switch when running the command, if you do the command does not process correctly and you will receive this error:

Cannot bind positional parameters because no names were given.
    + CategoryInfo          : InvalidArgument: (:) [Test-ActiveSyncConnectivity], ParameterBindingException
    + FullyQualifiedErrorId : AmbiguousPositionalParameterNoName,Test-ActiveSyncConnectivity

So in summary, whenever you need to test Active Sync for an individual user within your domain, this is the command you need.

Test-ActiveSyncConnectivity -MailboxCredential (Get-Credential domain\username) -UseAutodiscoverForClientAccessServer

Delete Files older then X Days BATCH SCRIPT

To delete files older then X Days use the following script:

forfiles.exe /p D:\Files /s /m *.* /d -7 /c "cmd /c del @file"

/p	This parameter specifies the path that contain the files I wish to delete.
/s	This parameter tells the program to recurse into any subfolders to look for additional files.
/m	If you want to specify a specific file type, this parameter will allow you to limit the search to specific files, such as *.doc for Word documents. In my case, I looked for all files (*.*).
/d	This one is the key parameter – it specifies the last modified date value. In my example I specify “-7″ which indicates that the files need to have a modified date 7 days less than the current date.
/c	This is the command that I execute on the files found by the program. The delete command is executed in a command window for each file.

VB Script - Remove Files from being Read Only

The below VB Code uses scripting.filesystemobject to go through a bunch of files and remove the read only parameter.  I found it quite handy for a scheduled task when dealing with a rouge application.

Dim fl As File
If fso.FileExists(FileName) Then
    Set fl = fso.GetFile(FileName)
    If (fl.Attributes And ReadOnly) Then
      fl.Attributes = fl.Attributes - ReadOnly
    End If
End If

Hope this code snippet helps someone else!