VSS Snapshot error. The maximum number of snapshots for this volume has been reached.

Today on a customers SBS 2008 server still running Backup Exec 2010, backups started failing with the following error.

 - AOFO: Initialization failure on: "\\JCC-SBS\Microsoft Information Store\First Storage Group". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11234 - VSS Snapshot error. The maximum number of snapshots for this volume has been reached.  Microsoft Volume Shadow Copy Service (VSS)  will not allow any more snapshots.

The Advanced Open File Option was set to Automatically select open file technology - I changed this to use "System - Use Microsoft Software Shadow Copy Provider".

After making this change, I tested another backup and it completed successfully.

Interestingly, if you read the error it was already using the Microsoft Volume Shadow Provider yet it failed.  Hardcoding it in the settings of the backup job did the trick.

Exchange Database file: '-546 The log file sector size does not match the sector size of the current volumn

When attempting to backup an Exchange 2007 server running on Small Business Server 2008 with Backup Exec 2010 R2, the Exchange backup component of the selection failed with the following error in the job log.

Backup- \\SERVER\Microsoft Information Store\First Storage GroupV-79-57344-918 - Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-546 The log file sector size does not match the sector size of the current volumn. '
Backup- \\SERVER\Microsoft Information Store\Second Storage GroupV-79-57344-918 - Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-546 The log file sector size does not match the sector size of the current volumn. '

 This error occurred when backing up to a HP Tape Library attached to the server via SCSI.  When I attempted to do a disk based backup with Backup Exec 2010 R2 as a test, it worked successfully.

After leasing with Symantec, it turned out when backing up to tape, Backup Exec requires a temporary staging directory to be use during the backup job.  By default, Backup Exec selects the largest volume with the most available free space.  This cannot be a Removable USB Drive!

Backing up to disk does not require this staging area.

I had a 4TB USB Drive attached to the SBS 2008 server which was causing this issue.  To resolve the issue you need to manually specify what drive to use for staging by following these steps:

1. On the Exchange Server, click Start -> Run and type "regedit".  Click OK.
2. Within the Registry Editor, navigate to HKLM\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Exchange
3. Right click and add a new String Value, "OnHostTemp".
4. Right click and set the value of OnHostTemp to "C:\Temp"
5. Restart the Backup Exec Remote Agent for Windows Systems service on the Exchange Servers.
6. Run tape backup job of Exchange Resources from media server.

I selected H:\Temp as it had the most available free space on my SBS 2008 server.

After putting this RegKey in place it resolved the issue.

V-79-10000-11226 - VSS Snapshot error Microsoft Exchange 2007

An SBS Customer of mine running Exchange 2007 on Microsoft Windows Server SBS 2008 with Backup Exec 2010 R2 ran into a backup issue where their Exchange 2007 backups with GRT began failing.  The backups had been operational for over 3 years and suddenly started failing with the following errors.

Before I cover of the errors we were experiencing, it is important to note that during this time we also had disk problems with disks in a RAID5 array failing and requiring replacing.  The failing disks also resulted in some slight disk corruption and I needed to repair the Exchange database with eseutil /p.

The following errors were experienced:

- AOFO: Initialization failure on: "\\SERVER\Microsoft Information Store\First Storage Group". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).
V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

In addition to the above Backup Exec error, the following Windows Application Event Logs were logged:

Log Name:      Application

Source:        VSS
Date:          9/18/2014 7:45:11 PM
Event ID:      12293
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server.domain.local
Description:
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details EndPrepareSnapshots({2c88e07d-a06c-4f6f-b826-b6e8bbfd3e10}) [hr = 0x8000ffff].

Operation:

   Executing Asynchronous Operation

Context:

   Current State: DoSnapshotSet

After researching VSS EventID 12293 it brought me to http://support.microsoft.com/kb/924262 which said the resolution was:

"To resolve this issue, you can delete a Snapshot copy that lets you continue with your present backup."

I went and deleted all Exchange VSS backups with diskshadow.exe by using the following commands:

unexposed e:

delete shadows all

Note: E:\ is the volume containing my Microsoft Exchange databases.

After cleaning up the existing shadow copies, Exchange backups resumed to normal.  I believe the problem was introduced due to the disk corruption problems we experienced.

V-79-57344-38260 - Unable to create a snapshot of the virtual machine

In this post I am going to shed some light on a very common error in Backup Exec 2010, 2012 and 2014 when performing VM backups from a VMware Hypervisor.  The error experienced is:

V-79-57344-38260 - Unable to create a snapshot of the virtual machine. The virtual machine no longer exists or may be too busy to quiesce to take the snapshot.

This error is extremely generic and can be one of 10 possible problems.  For a list of all possible problems see the following knowledge base article from Symantec:

http://www.symantec.com/business/support/index?page=content&id=TECH146397

The most common cause which is not documented heavily on the Internet is the lack of the Storage APIs.  The Storage APIs are only available in VMware vSphere Standard Edition and higher.  If your using the free version of ESXi, you will not have the Storage API and hence your backups will not work.  Check the licensing page on your ESX host to see if you have the Storage API in your license key:

 If you don't have it, upgrade your license key and success will be yours :).

Backup Exec 2010 and Exchange 2010 Recovery Problems

Yesterday I responded to an emergency callout to a customer with 800 users running a single Exchange 2010 SP3 UR3 multi role server running on Windows Server 2008 R2 SP1.  The server server would not boot and was simply blue screening.  We were not able to access Windows in anyway even by booting into safe mode and had no indication as to why the failure occurred as we could not access the server event logs.  The Exchange 2010 SP3 server was running on top of a VMware vSphere 5.1 clustered environment hosted on shared storage.

This server was one I setup a couple of years back and as a result it followed my standard multi-role Exchange server build which consists of two or more NTFS volumes.

• Volume 1 (SYSTEM) consists of Operating System, Page File and Exchange System Files.
• Volume 2 (Logs  + Database) which contain the Exchange 2010 database and log files

Due to the changes in Disk I/O it is no longer a requirement to separate transaction logs from the Exchange database as I/O is no longer an issue.  Remember Exchange 2010 has a 90% disk I/O reduction over Exchange 2003.

Note: This server only had two volumes however additional volumes can exist in the event additional databases are required.

As the system volume was corrupt and no longer booting, this needed to either rebuilt or restored from backup.  The database/log volume was assumed to be fine and the plan was to simply re-attach the database/log files after the system volume containing Windows and Exchange server was restored.  We did have a full backup of the Exchange 2010 server through Backup Exec 2010 R3 SP3 which was taken on the weekend of both the system volume and system state.  As a result we had two methods for recovering this Exchange 2010 SP3 server bringing it back online:

• Recover the servers system volume and system state using the last backup taken with Backup Exec and relink it to the database/log volume.
• Recover the Server by performing an Exchange Recover Server installation to reconnect a newly installed Exchange server to the existing configuration stored in the Active Directory configuration partition using the Setup /m:RecoverServer switch.

It was deemed at the time, the best course of action would be to recover the server through Backup Exec as this would ensure things such as the digital certificate and Exchange Web service URL addresses would all be restored back to their original state.

Backup Exec 2010 R3 SP3 Issues

I have been working with Backup Exec for over 8 years now back when it was owned by a company named Veritas.  Every time I have had to perform a full system restore of a failed server, it has always been a cumbersome process aligned with multiple challenges.  After the so many years of having this product on the market, you think the functionality of the "Backup" and "Restore" processes would be completely ironed out and bullet proof, after the primary purpose of this product is to backup and restore data.  However, this my friends is what makes Backup Exec "special", the ability to cause companies pain by not performing these tasks.

Despite having used Backup Exec to recover servers in the past, from the history experienced with the product, to give myself the best chance for a successful restore I put what I knew aside and followed the Symantec online documentation exactly.  To restore a server running Windows Server 2008 or Windows Server 2008 R2, Symantec has published a knowledge base article on their support page to restore a remote Windows 2008 computer which has completely failed.  This is the most important support page article they could publish online as it documents the steps to restore a failed Windows Server, again the whole point of a backup and restore program.  As a result you think the instructions documented on this particular article would be 100% accurate and reviewed carefully by the Symantec support team.  Unfortunately, this is not the case and the below article does have technical mistakes to performing a successful restore of a Windows 2008 server.

http://www.symantec.com/business/support/index?page=content&id=TECH86323

In the instructions taken from the above article entitled "Disaster Recovery restore steps for a remote Windows 2008 computer" it states to:

• Build a new Windows Server 2008 computer by performing a fresh install
• Provide it the same computer name as before
• Ensure it has the same disk configuration as the previous system
• Do not join the new computer to an Active Directory domain, instead leave it as Workgroup.

Following these instructions exactly, I was unable to proceed with the restore documented further down in step 10 in the documentation.  At this point we raised a support case with Symantec Gold Support to assist us with restoring the server.  The Symantec support engineer after taking the case advised us that the instructions documented online were incorrect and the server needed to be joined to the domain.

After joining the computer to the domain I was able to proceed with performing the restore task.  The task proceeded successfully until it got to 91% where the following error was generated.

V-79-57344-782
The job failed with the following error: Unable to swap out active registry hive with new data

 

After rebooting the server being restored, Windows Boot Manager was unable to boot the server as it was left in a corrupt state.

 

 

At this point the Symantec engineer asked us to retry the process.  It failed again, exact same experience.  At this point the company had been without email for over 12 hours due to the lengthy timeframe Backup Exec takes to restore data.

Symantec then ran a tool on the Backup Exec media server to collect a bunch of logs for analysis and advised us that they require 48 hours to review why the server restore is failing.  When we asked them the question "so you expect us to be without email for 48 hours", their response was yes there is nothing we can do.

Exchange 2010 Recover Server Installation

At this point I advised my customer that we need to forget about Backup Exec and proceed doing a native Exchange 2010 Recover Server installation using the Setup /m:RecoverServer, something I had faith in working.  We rebuilt the Windows 2008 R2 server, provided it the same host name, re-joined it to the domain, installed all windows updates along with Exchange 2010 pre-requisites.

One thing I noticed is the Exchange 2010 SP3 media does not allow you to run Setup /m:RecoverServer, it errors out.  You must run the Recover Server installation from Exchange 2010 SP2 media and then after the install proceed to installing Exchange 2010 SP3 followed by the latest update rollup.

We had the server up within 2 hours of starting this procedure.

Summary

There is no denying that the restore procedure documented under http://www.symantec.com/business/support/index?page=content&id=TECH86323 should have worked for servers backed up using a Backup Exec agent.

This being said, it is important to note that Backup Exec has better methods for backing up and recovering servers which this customer is not currently following.  In virtual environments running VMware ESX it is recommended that companies present the VMware datastores running Virtual Machine File System (VMFS) to the Backup Exec media server.  Whilst Windows cannot read VMFS, Backup Exec can allowing it to directly backup the Virtual Machine hard disks (VMDKs) and configuration files.  This backup method provides faster backup speeds and reliable restores as Backup Exec no longer needs to rely on Backup Exec agents, instead it can communicate directly with vCentre to snapshot virtual machines for backup purposes and restore entire virtual machines back to their original state.

In addition to virtual machine level backups, Symantec Backup Exec 2012 also offers an alternative way for recovering servers backed up using a Backup Exec agent.  Backup Exec 2012 provides a recovery media for companies to boot off providing the ability to directly restore a server taken from an agent based backup.  This means companies no longer need to:

• Install Windows Server from Microsoft Installation Media
• Provide the server the same host name and drive letters
• Join the server to the domain
• Deploy the Backup Exec Agent
• Start the Recovery Process

The Backup Exec recovery media significantly reduces the time required to perform a bare metal restore of a failed Windows Server.

For companies which have already made significant investment in Backup Exec, before throwing away the investment with Symantec, it is advised that customers review their backup strategy to ensure it suits the infrastructure within their company.

The Windows Backup engine could not be contacted. Retry the operation.

Today when attempting to perform a System State backup on a Domain Controller I received the following error message:

The Windows Backup engine could not be contacted. Retry the operation.
The RPC server is unavailable.

I also noticed the following event errors appearing in Event Viewer.

Log Name:      Application
Source:        VSS
Date:          13/03/2013 10:48:41 AM
Event ID:      12292
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {06d8e136-56f6-4048-93fb-a5943e949375} [0x80040154, Class not registered
].
Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Get Shadow Copy Properties
Context:
   Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}
   Class ID: {06d8e136-56f6-4048-93fb-a5943e949375}
   Snapshot Context: -1
   Snapshot Context: -1
   Execution Context: Coordinator

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 10:48:50 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x5888
Faulting application start time: 0x01ce1f9517234ddc
Faulting application path: C:\Windows\system32\wbengine.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 880e1970-8b88-11e2-aefa-005056a2000b

The above event error 12292 it provided us the Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}.  Looking in the registery under HKLM\System\CurrentControlSet\services\VSS\Providers\{5fdb6ef5-6ead-4610-995b-401c88626115} it shows this provider as the Backup Exec VSS Provider.



For some reason WBAdmin is trying to use the Backup Exec VSS Provider instead of the Microsoft VSS Provider.

I added the registry DWORD UseMicrosoftProvider to HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore with a value of "1" which is meant to force the backup to use the Microsoft provider.

This key had no effect, the backup still attempted to use the Symantec VSS Provider.  Next I used the following Symantec article 130940 to completely remove the Symantec backup exec agent from the server including removing registry keys.

http://www.symantec.com/business/support/index?page=content&id=TECH130940

After removing the Symantec backup exec agent I ran a test backup and the backup failed again with the same error.  Running a "vssadmin list providers" revealed that the Symantec VSS Provider was still in place despite following Symantec article 130940 which was meant to completely remove backup exec from a windows server.

Again we see same GUID of the Symantec provider which was presented in the event error and the registry, {5fdb6ef5-6ead-4610-995b-401c88626115}.

I then followed Symantec article 77585 to completely remove the Backup Exec VSS Provider by deleting the {5fdb6ef5-6ead-4610-995b-401c88626115} key from the following location in the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\

http://www.symantec.com/business/support/index?page=content&id=TECH77585

After restarting the VSS service we see the Backup Exec VSS Provider is no longer available.

I then rebooted the server.  After a reboot I attempted another backup with wbadmin.  We got further this time but it still crashed out.

Some new event logs exist now:

Log Name:      Application
Source:        Application Error
Date:          13/03/2013 2:47:07 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Faulting application name: wbengine.exe, version: 6.1.7601.17514, time stamp: 0x4ce79951
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2

Log Name:      Application
Source:        VSS
Date:          13/03/2013 2:47:11 PM
Event ID:      8193
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DomainController
Description:
Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...).  hr = 0x80070005, Access is denied.
.
Operation:
   Initializing Writer
Context:
   Writer Class Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
   Writer Name: NPS VSS Writer
   Writer Instance ID: {37bef355-a711-4241-a2bc-91f1181c845b}

 

VSS Event ID 8193 says that the VSS provider was denied access when opening a registry key under the security context of SYSTEM

 

SYSTEM\CurrentControlSet\Services\VSS\Diag,...). 

 

Damn it cut off!  We could use Sysinternals ProcMon to get the full path however lets just force FULL access for tye System account from the DIAG key downwards.

 

 

 After making this change I then tested another wbadmin.  Made no difference. :-(

 

I searched the entire registry for the GUID of the Backup Exec VSS Provider to ensure nothing was missed.  My search found nothing.  Whilst I have isolated the problem to the VSS Provider provided by Symantec, a change made by the Symantec Backup Exec agent remains and as a result wbadmin will not function.

If there is someone out there who has fixed this issue can you please comment below with your resolution to ensure others with this issue have a fix as this is not documented anywhere on the Internet.

Reset Backup Exec 2012 to Factory Defaults

Symantec has published an article on how to restore Backup Exec to factory defaults using BEUtility.exe however this article only works for  Backup Exec 11D/12.0/12.5/2010/2010R2/2010R3.  This article can be found here:

http://www.symantec.com/business/support/index?page=content&id=TECH66780

When trying to perform this procedure on Backup Exec 2012 it fails with the following error message:

Error: Unable to drop database

This is a bug with Backup Exec 2012 however there is a work around, to restore the Database to Factory Defaults perform the following procedure:

1. Stop all backup exec services.

2. Go to C:\program files\symantec\backup exec\data

3. Rename the current database file bedb_dat.mdf to something else.

4. Rename the current log file database  bedb_log.ldf to something else

5. Now use bedb_dat.bak file and rename it to bedb_dat.mdf

6. Now use the bedb_log.bak file and rename it to bedb_log.ldf

7. Now restart all backup exec services.

vssadmin - Microsoft Exchange Writer is Missing

I had a server that had the "Microsoft Exchange Writer" missing. When typing the following command from a command prompt all the VSS (Volume Shadow Copy) writers would come up besides the "Microsoft Exchange Writer".

"vssadmin list writers"

This is what the Microsoft Exchange Writer looks like should it come up:



If it does not come up in the list... to enable it navigate to the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem

Modify the "Disable Exchange Writer" registry key to a value of 0.

After this restart the "Exchange Information Store" service.

"vssadmin list writers" should now display the writer correctly.

Backup Exec 12.5 with Windows Server 2008 R2

If you are running Symantec Backup Exec 12.5 and you are wishing to implement 2008 R2, there is a bug your going to come across. This is a must read article written by one of my work colleges:

http://blog.samkendall.net/2010/04/01/backup-exec-12-5-windows-server-2008-r2-v-79-57344-65225-a-failure-occurred-accessing-the-writer-metadata/

Enable Backup Exec Agent Debug Logging

If you have a weird backup error on the symantec backup server exec server you may want to turn on debug logging on the backup exec agent for the server in question to help you diagnose the problem.

To do this first stop the "Backup Exec Remote Agent for Windows Systems" service on the server with the problem.

Edit the properties of the service and int he start parameters enter -debug



Next you need to enable debugging in the registry:

HKLM\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\Logging

Change the DWORD CreateDebugLog to "1" to enable debug logging:



Next start the "Backup Exec Remote Agent for Windows Systems" service again.

The logs will be located in this folder:

C:\Program Files\Symantec\Backup Exec\RAWS\logs

Start the backup and review the client log file for additional information as to what caused it to error out.

Backup Exec 12.5 - 0x80070057 - One or more arguments are invalid

I have a robotic HP LTO Ultrium-3 Tape Drive connected to a LSI Adapter, Ultra320 SCSI 2000 series (w/1020/1030)(StorPort) in a HP server. Was recieving the following error in backup exec:



In event viewer I was recieving:
The driver detected a controller error on \Device\RaidPort0.



The problem turned out to be disconnecting the SCSI terminator from the tape drive then plugging it back on. This caused the green light on the terminator to light up... weird.

Physical Volume Library Media not found

In Backup Exec every backup job was failing with this error:

Final Error Code: a000810d HEX (0xa000810d HEX) or e000810d HEX (0xe000810d HEX)Final Error Description: "Physical Volume Library Media not found"Final Error Category: Backup Media ErrorsError Text In Job Log: "Media mount failed."

To fix the problem i did the following:
1. In Backup Exec, go to Tools Options
2. Select the Catalog option under Settings
3. Clear the Use storage media-based catalogs check box
4. Re-catalog the tape