Hyper-V
Important
Remember to read the Best Practices chapter common for all of our hypervisor plugins.
Hyper-V Plugins to Cover All Backup Needs
Microsoft has created various technologies for backing up Hyper-V virtual machines. Thus, multiple Bacula Plugins have been designed to maximize the benefits of each solution.
Hyper-V is equipped with a VSS writer on all compatible versions of Windows Server. This VSS writer enables developers to utilize the existing VSS infrastructure for backing up virtual machines to Bacula using the Bacula Enterprise VSS Plugins. This technology is supported by the original Bacula Hyper-V plugin. Although it doesn’t allow incremental and differential backups and other more granular VM-based options, it can still cover any Hyper-V version. Therefore, it is highly recommended for small standalone Hyper-V servers (single node, no Failover Cluster) and older Hyper-V versions that do not offer Virtual Disk Service or Hyper-V WMI API.
The Virtual Disk Service (VDS) is a service provided by Microsoft Windows that handles query and configuration tasks upon request from end users, scripts, and applications. This service is compatible with Windows Server 2003, Windows Vista, and newer versions. The Hyper-V Winapi Plugin utilizes this technology to backup and restore virtual machines. It supports incremental and differential backups, making it the recommended solution for more intricate Hyper-V servers, such as Failover Clusters with multiple nodes, where local disk space is a critical resource. Depending on the relocation of virtual machines within the Cluster, it may be necessary to migrate the backed-up VMs across specific nodes.
Starting in Windows 8 and Windows Server 2012, Hyper-V supports backup via the Hyper-V WMI API. This feature enables individual Guest VMs to be backed up separately and incrementally, offering a more scalable solution compared to using VSS in the host. The Bacula Enterprise Hyper-V WMI Plugin uses this technology for backup and restore to/from Bacula. It backups/restores VM in the recommended Microsoft format. By utilizing the Microsoft snapshot format from/to disk, it is essential to have sufficient disk space available for the process to proceed smoothly. In scenarios where disk space may be limited due to busy configurations, the Hyper-V Winapi Plugin can be utilized as an alternative solution.
Backups created using the Hyper-V WMI Plugin, Hyper-V Winapi Plugin and Hyper-V Plugin are not compatible with each other.
Overview
This white paper presents how to use the Microsoft Hyper-V Server plugin when backing up with Bacula Enterprise version 8.2. These solutions are not applicable to prior versions. This document is intended to be used by Bacula Enterprise administrators.
Bacula Windows Hyper-V Plugin
Bacula Systems provides a single plugin for Bacula Enterprise
named vss-fd.dll that permits you to backup a number of
different components on Windows machines. One of those components is
Microsoft Hyper-V Server, which is the subject of this white paper.
Backing up and restoring Hyper-V virtual machine is supported with Full
level backups. It is not possible to do Incremental or Differential
backups because Microsoft does not support that backup level for the
Hyper-V Server product. Use of the Global Endpoint Deduplication plugin
and the bothsides FileSet option permits to minimize the data
transfer and the storage.
Installation and Configuration
To activate the Hyper-V component you have to put the following into the Include section of the File Set which will be used to back up the Hyper-V Server:
Plugin = "vss:/@HYPERV/"
This will back up all Hyper-V Virtual Machine. The plugin directive must be specified exactly as shown above. A Job may have one or more of the vss plugin components specified.
You must ensure that the vss-fd.dll plugin is in the plugins
directory on the FD doing the backup (done by default with the
installer), and that the Plugin Directory directive line is present
and enabled in the FD’s configuration file bacula-fd.conf.
An example of the FD configuration file is shown in the screenshot below:
File Daemon Configuration Excerpt with “Plugin Directory” Line
The status output of a client with the VSS plugin enabled:
*status client=wsb-sql08-fd
Connecting to Client wsb-sql08-fd at wsb-sql08:9102
wsb-sql08-fd Version: 8.2.0 (02 Feb 2015) VSS Linux Cross-compile Win64
Daemon started 20-Apr-12 13:14. Jobs: run=15 running=0.
Microsoft Windows Server 2008 R2 Standard Edition Service Pack 1 (build 7601), 64-bit
Heap: heap=0 smbytes=1,061,455 ...
Sizes: boffset_t=8 size_t=8 debug=0 ...
Plugin: vss-fd.dll
Backup
If everything is set up correctly as above then the backup will include the Hyper-V server data. The Hyper-V server data files backed up will appear in a bconsole or bat restore like:
/@HYPERV/
...
etc
A complete example of a FileSet and Job resource for Hyper-V Server data is
shown below. As for all VSS-enabled components, it
is the administrator’s responsibility to make sure that the required VSS
snapshots are created by explicitly mentioning at least one file or
directory for each drive where data that is handled by the plugin is
stored. In the example, we use the file c:/backmeup to ensure
this.
FileSet {
Name = HYPERV
Include {
Options {
Signature = SHA1
Dedup = bothsides
}
File = C:/backmeup
Plugin = "vss:/@HYPERV/"
}
}
Job {
Name = HYPERV08
Accurate = Yes
File Set = HYPERV
Client = wsb-hyp08-fd
Job Defs = DefaultJob
Level = Full
}
Note in the example above that C:/backmeup is explicitly included, which is
required to ensure that Bacula creates the required VSS snapshot of that
Windows drive letter. If Hyper-V Server data is also stored on other
partitions, you need to create similar File =-lines for these drives,
too.
Note
Starting with Bacula Enterprise version 12.6, the explicit include of a dummy
file (see File = C:/backmeup in the fileset example above) is not
mandatory anymore
File Set {
Name = HYPERV-TestVM
Include {
Options {
Signature = SHA1
}
File = C:/backmeup
# backup only TestVM on the server
Plugin = "vss:/@HYPERV/ cinclude=\"Host Component\" cinclude=*/TestVM cexclude=*"
}
}
Hyper-V uses one of two mechanisms to back up each VM. The default backup mechanism is called the “Saved State” or “Offline” method, where the VM is put into a saved state during the processing of the PrepareForSnapshot event, snapshots are taken of the appropriate volumes, and the VM is returned to the previous state during the processing of the PostSnapshot event.
The other backup mechanism is called the “Child VM Snapshot” or “Online” method, which uses VSS inside the child VM to participate in the backup. For the “Child VM Snapshot” method to be supported, all of the following conditions must be met:
Backup (volume snapshot) Integration Service is installed and running in the child VM. The service name is “Hyper-V Volume Shadow Copy Requestor”.
The child VM must be in the running state.
The Snapshot File Location for the VM is set to be the same volume in the host operating system as the VHD files for the VM.
All volumes in the child VM are basic disks and there are no dynamic disks.
All disks in the child VM must use a file system that supports snapshots (for example, NTFS).
To know if your VMs are “Offline” or “Online”, it is possible to use the following windows command on Windows 2012 R2:
C:/> echo list writers > t.txt
C:/> diskshadow /s t.txt | find "Caption: O"
- Caption: Offline/2012
- Caption: Offline/windows
- Caption: Online/centos
On Windows 2012 and 2008
C:/> echo list writers > t.txt
C:/> diskshadow /s t.txt | find /i "Caption: Backup Using"
For Offline backups: Backup Using Saved State/VMname1
For Online backups: Backup Using Child Partition Snapshot/VMname2
Restore
Restoring the VMs is done entirely by the host operating system; the VSS writers in the child VMs are not involved.
During the processing of the PreRestore event, the Hyper-V VSS writer turns off and deletes any VMs that are about to be restored.
After all VSS writers have processed the PreRestore event, the files are restored.
For each VM that was restored, the Hyper-V VSS writer registers the VM with the Hyper-V management service. If the VM is restored to a nondefault location, a symbolic link is created in the default location linking to that location.
For each VHD that was restored, the location is compared with the one specified for that VM. If the location is different, then the configuration is updated with the proper location.
The network configuration is updated. If the virtual switches that the VM was connected to when it was backed up still exit, new ports are created and connected to the VM.
When restoring a “Offline” VM, the VM will not be re-created by Microsoft Hyper-V vss driver. It is possible to run “New-VM” powershell command to re-create the VM.
New-VM -VMName centos -VHDPath C:/VM/centos.vhdx -MemoryStartupBytes 512MB -SwitchName VMNetwork
without_vss
With Bacula Enteprise 8.2, it is possible to restore VSS
files directly on disk without using the VSS restore framework. In the
restore menu, it is possible to configure Plugin Options menu and
set the without_vss option to “true”.
Run Restore job
JobName: RestoreFiles
Bootstrap: /opt/bacula/working/trusty-amd64-dir.restore.9.bsr
Where: c:/tmp
Replace: Always
FileSet: Full Set
Backup Client: hyperv
Restore Client: hyperv
Storage: dedup
When: 2015-03-03 06:50:22
Catalog: MyCatalog
Priority: 10
Plugin Options: *None* <------------- Plugin Options menu
Example
We assume that a correct backup of Hyper-V data exists and you start the
restore with option 5 of the bconsole restore command, mark the
complete tree of data backed up by the Hyper-V component of the VSS
plugin, then finally do lsmark @HYPERV to show all the files
selected to be restored:
$ mark *
31 files marked.
$ lsmark
*@HYPERV/
*Microsoft Hyper-V VSS Writer/
*Host Component/
*:component_info_5215da3c
*c:/
*programdata/
*microsoft/
*windows/
*hyper-v/
*initialstore.xml
*resource types/
*06ff76fa-2d58-4baf-9f8d-455773824f37.xml
*118c3be5-0d31-4804-85f0-5c6074abea8f.xml
*146c56a0-3546-469b-9737-fcbcf82428f4.xml
*dacdcf3f-6f67-4eb8-a4d0-5d93b48a2468.xml
*f6293891-f32f-4930-b2db-1a8961d9cb75.xml
*Offline/
*ubuntu/
*:component_info_5215da3c
*c:/
*programdata/
*microsoft/
*windows/
*hyper-v/
*virtual machines/
*690f5094-ff23-411e-92c0-639fc7ebc598/
*690f5094-ff23-411e-92c0-639fc7ebc598.bin
*690f5094-ff23-411e-92c0-639fc7ebc598.vsv
*690f5094-ff23-411e-92c0-639fc7ebc598.xml
*vm/
*ubuntu.vhdx
$ lsmark
*@HYPERV/
*Microsoft Hyper-V VSS Writer/
*Host Component/
*:component_info_5216cf46
*c:/
*programdata/
*microsoft/
*windows/
*hyper-v/
*initialstore.xml
*resource types/
*06ff76fa-2d58-4baf-9f8d-455773824f37.xml
*118c3be5-0d31-4804-85f0-5c6074abea8f.xml
*Online/
*centos/
*:component_info_5216cf46
*c:/
*programdata/
*microsoft/
*windows/
*hyper-v/
*snapshots/
*acc145fb-9566-402d-9434-04f1e325a75f-backupsnapshot.xml
*virtual machines/
*acc145fb-9566-402d-9434-04f1e325a75f.xml
*vm/
*centos-childvhd.avhdx
*centos.vhdx
File Level Restore
To restore a set of files from a Hyper-V VM backup without re-importing
the entire VM, it is possible to restore VHD files in a directory using
the without_vss plugin restore option (See sec without_vss) and
mount them in the system with the Powershell command Mount-VHD (or the
Server Manager console (see Fig Attach/Mount Option in Server Manager below). Once mounted, the
VHD image is accessible like other physical disks on the system.
Mount-VHD –Path c:\test\testvhdx.vhdx -ReadOnly
More information about the Mount-VHD command can be found here:
https://technet.microsoft.com/en-us/library/hh848551.aspx
Attach/Mount Option in Server Manager
Detach/Unmount Option in Server Manager
We advise to restore VHD files on a different system to avoid
operational problems during the restore. If the without_vss option
is not properly set, the original VM would be deleted by Hyper-V
automatically during the restore.
Single Item Restore
Plugin Notes
Windows VSS Plugin Items to Note
One file from each drive needed by the plugins must be explicitly listed in File Set used. This is to ensure that the main Bacula code does a snapshot of all the required drives. At a later time, we will find a way to accomplish this automatically.
When doing a backup that is to be used for Bare Metal Recovery, do not use the VSS plugin.
General Plugin Items to Note
The
estimatecommand does not handle plugins. When estimating a job that uses plugins, an error message regarding the plugin will be displayed. However, backup jobs will use the plugin.The File Set Include Option
CheckFileChanges = Yesdoes not work with plugin-generated data. Thus, you must not use that Option in the Include section of the FileSet where you specify using the Hyper-V plugin.When an Offline virtual machine is currently backed up, it is not possible to start it (Hyper-V limitation).
The Bacula
replaceflag is not respected by the Hyper-V plugin. Virtual machines will be always overwritten during restore.Microsoft Hyper-V vss interface doesn’t support Differential and Incremental backup. Use of the Global Endpoint Deduplication plugin and the
bothsidesFileSet option permits to minimize the data transfer and the storage.When trying to restore Incremental or Differential jobs, Hyper-V VSS writer will print errors on PostRestore and PreRestore events. The virtual disk image (VHDX) should be restored despite these errors.
Note
You can download this article as a PDF
Best Practices
While it is technically possible to backup multiple VMs in one Bacula hypervisor plugin backup job (VMware, Hyper-V, RHV, Proxmox, etc), this is not necessarily the best way to perform VM backups. It is strongly recommended that one backup Job is created for each VM being backed up for the following reasons:
By default, if one of your VMs fails to backup in a “multi-VM” backup job, the main Bacula job will terminate “Backup OK – with warnings.” The JobStatus for jobs that terminate “Backup OK” and “Backup OK – with warnings” are not differentiated in the catalog. They are both ‘T’, so this means that you will have to carefully monitor your backup job logs in case some VM backups fail and pay attention to the JobErrors field in the job summaries.
To address this issue, there is a plugin option called “abort_on_error” in each of the Bacula hypervisor plugins, which causes Bacula to immediately fail the job as soon as an error is detected while backing up a VM. However, if you use this option, and the backup of VM number 11 in a list of 50 VMs fails, then the whole job will be failed, and VMs 12-50 will not be backed up during that job’s run.
A 1:1 configuration (one VM backed up per job) means that the “abort_on_error” option will make more sense to enable in each job so you will immediately know when a VM fails to backup since the Bacula job will terminate with a “Backup failed” message and ‘f’ in the catalog for the job.
With a 1:1 VM/Job configuration, re-running a specific VM backup job is simple to do after the cause of the failure is investigated and fixed.
In the example about the 50 VMs, without a 1:1 configuration, there is no way to re-run a backup of just the one VM that failed to backup.
Additionally, with a 1:1 VM/Job configuration, job metrics will have more meaning because each VM will be one job, and you will know to expect a specific number of jobs each night with each job representing one VM.
With a multi-VM per job configuration, each VM will be backed up “serially”, one at a time, disk by disk, VM by VM. A 1:1 configuration will allow several VM backups to be run concurrently which will reduce the overall time to perform the VM backups. Of course, you will need to pay close attention to SD and ESXi storage and networking resources, and adjust the number of concurrent jobs accordingly.
For some hypervisors (VMware, Proxmox, etc) Bacula provides automation scripts (eg: scan_datacenter.pl for VMware). These scripts are designed so that they will create 1:1 VM/Job configurations. If you plan to make use of these automation scripts, it is a good idea to already be thinking this way, and having your hypervisor plugin backup configurations in a 1:1 configuration from the beginning.