The Windows Backup engine could not be contacted. Retry the operation.

Today when attempting to perform a System State backup on a Domain Controller I received the following error message:

The Windows Backup engine could not be contacted. Retry the operation.
The RPC server is unavailable.

I also noticed the following event errors appearing in Event Viewer.

Log Name:      Application
Source:        VSS
Date:          13/03/2013 10:48:41 AM
Event ID:      12292
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {06d8e136-56f6-4048-93fb-a5943e949375} [0x80040154, Class not registered
].
Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Get Shadow Copy Properties
Context:
   Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}
   Class ID: {06d8e136-56f6-4048-93fb-a5943e949375}
   Snapshot Context: -1
   Snapshot Context: -1
   Execution Context: Coordinator

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 10:48:50 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x5888
Faulting application start time: 0x01ce1f9517234ddc
Faulting application path: C:\Windows\system32\wbengine.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 880e1970-8b88-11e2-aefa-005056a2000b

The above event error 12292 it provided us the Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}.  Looking in the registery under HKLM\System\CurrentControlSet\services\VSS\Providers\{5fdb6ef5-6ead-4610-995b-401c88626115} it shows this provider as the Backup Exec VSS Provider.



For some reason WBAdmin is trying to use the Backup Exec VSS Provider instead of the Microsoft VSS Provider.

I added the registry DWORD UseMicrosoftProvider to HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore with a value of "1" which is meant to force the backup to use the Microsoft provider.

This key had no effect, the backup still attempted to use the Symantec VSS Provider.  Next I used the following Symantec article 130940 to completely remove the Symantec backup exec agent from the server including removing registry keys.

http://www.symantec.com/business/support/index?page=content&id=TECH130940

After removing the Symantec backup exec agent I ran a test backup and the backup failed again with the same error.  Running a "vssadmin list providers" revealed that the Symantec VSS Provider was still in place despite following Symantec article 130940 which was meant to completely remove backup exec from a windows server.

Again we see same GUID of the Symantec provider which was presented in the event error and the registry, {5fdb6ef5-6ead-4610-995b-401c88626115}.

I then followed Symantec article 77585 to completely remove the Backup Exec VSS Provider by deleting the {5fdb6ef5-6ead-4610-995b-401c88626115} key from the following location in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\

http://www.symantec.com/business/support/index?page=content&id=TECH77585

After restarting the VSS service we see the Backup Exec VSS Provider is no longer available.

I then rebooted the server.  After a reboot I attempted another backup with wbadmin.  We got further this time but it still crashed out.

Some new event logs exist now:

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 2:47:07 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2

Log Name:      Application
Source:        VSS
Date:          13/03/2013 2:47:11 PM
Event ID:      8193
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...).  hr = 0x80070005, Access is denied.
.
Operation:
   Initializing Writer
Context:
   Writer Class Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
   Writer Name: NPS VSS Writer
   Writer Instance ID: {37bef355-a711-4241-a2bc-91f1181c845b}

 

VSS Event ID 8193 says that the VSS provider was denied access when opening a registry key under the security context of SYSTEM

 

SYSTEM\CurrentControlSet\Services\VSS\Diag,...). 

 

Damn it cut off!  We could use Sysinternals ProcMon to get the full path however lets just force FULL access for tye System account from the DIAG key downwards.

 

 

 After making this change I then tested another wbadmin.  Made no difference. :-(

 

I searched the entire registry for the GUID of the Backup Exec VSS Provider to ensure nothing was missed.  My search found nothing.  Whilst I have isolated the problem to the VSS Provider provided by Symantec, a change made by the Symantec Backup Exec agent remains and as a result wbadmin will not function.

If there is someone out there who has fixed this issue can you please comment below with your resolution to ensure others with this issue have a fix as this is not documented anywhere on the Internet.