Backup

The general backup operation consists on the following procedure:

Find all selected target instances.
Find all selected target disks for each instance and do for each disk.
Cleaning previous snapshots.
Check for base snapshots (Incremental and Differential backups).
Launch a snapshot without predecessor (Full) or using a found predecessor (Incremental and Differential) for all the target disks at the same time.
Download block by block (EBS Direct API) and in parallel, all the blocks belonging to each disk.

Backup File Structure

One instance with one single volume will usually produce the following contents during a backup operation:

/@amazon-ec2/i-xxxxxx/i-xxxxxx.json

/@amazon-ec2/i-xxxxxx/vol-yyyyyy-zzzz-wG.json

/@amazon-ec2/i-xxxxxx/vol-yyyyyy-zzzz-wG.bimg

/@amazon-ec2/i-xxxxxx/vol-yyyyyy.bmp

/@amazon-ec2/i-xxxxxx/vol-yyyyyy.abmp

Being i-xxxxxx an instance id, all the information of each backed up instance will be contained into a folder with that name. Inside files are explained below:

/@amazon-ec2/i-xxxxxx/i-xxxxxx.json

It is a JSON file containing all the configuration data of the instance, such as device mapping, tags or networking among many others.

/@amazon-ec2/i-xxxxxx/vol-yyyyyy-zzzz-wG.json

Being vol-yyyyyy the EBS volume id, it is a JSON file containing all the configuration data of the associated volume, such as volume type, iops or throughput among many others. The name will contain also the name of the device inside the instance in ‘zzzz’ (sda for instance) and the size in Gigabytes in ‘w’.

/@amazon-ec2/i-xxxxxx/vol-yyyyyy-wG.bimg

It contains the associated volume data in the instant the backup snapshot was done. This file contains headers with offset and size.

/@amazon-ec2/i-xxxxxx/vol-yyyyyy.bmp

/@amazon-ec2/i-xxxxxx/vol-yyyyyy.abmp

Contains all the headers (offset and size) of the associated disk (.bmp), as well as the headers of the empty blocks with the same format (.abmp).

Changed Block Tracking

The Amazon EC2 API provides a diff function to compare changes at the block level between two Snapshots. That function is what this plugin uses to perform Incremental or Differential backups. As a result, Bacula will need to keep one snapshot at least (so changes can be compared later) in the cloud.

By default, Full backups will keep the snapshot just created during backup. Differential backups won’t keep any snapshot. Incremental backups will keep also the last snapshot, but will remove the previous one (also if it’s from a Full backup). This makes the combination of Incremental and Differential backups not possible in the same backup chain. If you wish to combine then, then you need to use the parameter: keep_full_snapshot=yes.

Backup with bec2

In addition to a regular backup job, this plugin allows to invoke the same functions through the command line, using the provided ‘bec2’ script. This script will accept the same parameters as the plugin command line and will store the backup in a local folder. Below an example on how to use it:

BEC2 backup example

bec2 --operation=backup  --region=us-east-1 --access_key=ABCXXXEXAMPLEKEY --secret_key=xxdrperefkafasfdExampleSecret123 --instances=my-instance --bitmap_headers=false

That invocation will download a backup of the ‘my-instance’ instance into the working directory.