Monday, June 23, 2014

PowerShell - Nightly DFS-R Database Backups

Windows Server 2012 R2 provides a new feature allowing customers to export and import the DFS-R database located in "System Volume Information" for any volumes with folders partaking in DFS-R replication.  For more information about this new feature, please see the following TechNet article which I strongly recommend a read over:

The ability to Export and Import the DFS-R database provides the following advantages:
  • Significantly reduces the amount of time the initial sync process takes when pre-staging new file servers to partake in DFS-R replication as the DFS-R database can be exported and imported into the new server.  This process also requires the data be copied through robocopy or a backup product (such as ntbackup or wbadmin) and ensure the data is exactly the same as the source server.
  • Provide the ability to restore a corrupt DFS-R database which can be caused by incorrect shutdown of a Windows Server running DFS-R.  When a DFS-R database goes corrupt the server automatically kicks of self recovery by default which involves cross checking the file hashes against every file for the replication groups volume against the other DFS-R servers in the cluster in order to repair the state of the database.  This process can take a long time, sometimes as long as the initial sync process backlogging all new replication traffic.
Some DFS-R environments consist of a single hub server and up to 100 spoke servers with large volumes of data sometimes exceeding 10TB with over 10 million files.  In this scenario if the DFS-R database suffered corruption on the hub server, this would result in the entire DFS-R replication backlogging for weeks while the self recovery process rechecks all the files across the environment!

I have a customer with a DFS-R environment similar to the example provided above.  As a result I put in place measures to recover the DFS-R database in the event corruption occurred.  A PowerShell script was created to automatically backup the database on a nightly basis using the new Export-DfsrClone cmdlet introduced in Windows Server 2012 R2 which runs on the hub server.  In the event corruption occurs, we can simply import the database using the new Import-DfsrClone cmdlet.

This PowerShell Script performs the following:
  • Creates backups under C:\DfsrDatabaseBackups
  • Each backup is placed in a folder labelled YYYY-MM-DD HH-mm"
  • The script automatically deletes any database backups older then 14 days by default to ensure old backups are cleaned up.

#DFSR Database Backup Script - Created by Clint Boessen 15/04/2014
$basefolder = "C:\DfsrDatabaseBackups"
$datefolder = get-date -format "yyyy-MM-dd HH-mm"
$backuplocation = $basefolder + "\" + $datefolder
New-Item -ItemType directory $backuplocation
Export-DfsrClone -Volume E: -Path $backuplocation -Force

#Remove Databases older then 14 Days
$Now = Get-Date
$Days = "14"
$LastWrite = $Now.AddDays(-$Days)

cd $basefolder

$Folders = get-childitem -path $basefolder |
Where {$_.psIsContainer -eq $true} |
Where {$_.LastWriteTime -le "$LastWrite"}

foreach ($Folder in $Folders)

write-host "Deleting $Folder" -foregroundcolor "Red"
Remove-Item $Folder -recurse -Confirm:$false

Add the above script to a PowerShell script "ps1" file and create a scheduled task on your DFS-R file server to run the script according to a schedule in which you want DFS-R database backups to occur.  Once configured, you will see DFS-R database backups occurring on a regular basis according to your schedule with old backups automatically being cleaned automatically!

I scheduled my script to run at 5am on weekdays.  Please not the backup process can take hours, in my environment due to large amount of files the export is taking a total of 3 hours finishing around 9am which you can see by the date modified timestamp.

It is important to note, DFS-R replication will not work when a database backup is occurring.  As a result please ensure the backups are scheduled at a time when replication can be paused.  Replication automatically resumes after the export process is completed.

PowerShell - Locate Missing SYSTEM Permissions from Folder Structure

I am in the middle of a DFS-R project for a customer where I'm provisioning new Windows Server 2012 R2 file servers and migrating the data across to the new server.  To perform the migration I initially performed the pre-sync of the data with robocopy in backup mode "/b" then added the new servers to the DFS-R replication group/namespace.  Once the initial DFS-R sync had completed which took a few days, I enabled the namespace for the new servers and disabled the old servers.

Upon cutting the users across, many users complained the data was approximately 7 days old which is the approximate time I did the initial robocopy.  After further investigation it appeared DFS-R was not keeping the data in sync and many directories had not been replicated.  These files and folders which were not replicated also did not appear in the backlog count under DFS-R Health Reports which were run to verify replication status.

It turned out the cause of this issue was because the "SYSTEM" permissions were missing from many directories in the file server structure.  As the DFS-R service runs under the "SYSTEM" account, it must have access to the data in order to perform replication.  Robocopy was however able to move this data as it was running in backup mode which uses VSS to snapshot the data.

This directory structure utilised block policy inheritance numerous times throughout the folder structure and as a result finding directories which did not have SYSTEM permissions configured correctly was a challenging task.  As a result I wrote a PowerShell script which performs an audit against a directory structure and returns all folders which are missing the "SYSTEM" permission so that an Administrator can manually add the missing permission at all folder levels with inheritance broken.

This is a handy script and I posted it online for everyone as I recommend running it against any directory structure on file servers to ensure the SYSTEM account has full control over all data, a recommended Microsoft best practice.

$OutFile = "C:\Permissions.csv"

$Folders = dir $RootPath -recurse | where {$_.psiscontainer -eq $true}
foreach ($Folder in $Folders){
       $ACLs = get-acl $Folder.fullname | ForEach-Object { $_.Access  }
     $Found = $False
       Foreach ($ACL in $ACLs){
    if ($ACL.IdentityReference -eq "NT AUTHORITY\SYSTEM")
            $Found = $True
             if ($Found -ne $True)
             $OutInfo = $Folder.FullName
             Add-Content -Value $OutInfo -Path $OutFile

I hope this PowerShell script helps other people out there!

Monday, June 9, 2014

V-79-57344-38260 - Unable to create a snapshot of the virtual machine

In this post I am going to shed some light on a very common error in Backup Exec 2010, 2012 and 2014 when performing VM backups from a VMware Hypervisor.  The error experienced is:

V-79-57344-38260 - Unable to create a snapshot of the virtual machine. The virtual machine no longer exists or may be too busy to quiesce to take the snapshot.

This error is extremely generic and can be one of 10 possible problems.  For a list of all possible problems see the following knowledge base article from Symantec:

The most common cause which is not documented heavily on the Internet is the lack of the Storage APIs.  The Storage APIs are only available in VMware vSphere Standard Edition and higher.  If your using the free version of ESXi, you will not have the Storage API and hence your backups will not work.  Check the licensing page on your ESX host to see if you have the Storage API in your license key:
 If you don't have it, upgrade your license key and success will be yours :).