Restore
Restore Procedure
A restore operation of this plugin will typically involve the following steps:
For each involved volume, a new empty snapshot will be created
Data will be uploaded to those empty snapshots in an incremental way (Full first, Incremental data after)
Snapshots will be completed once everything was uploaded
Instance will be created mixing original instance configuration and user specified information during restore session
The creation operation will already include the creation of any associated EBS Volume coming from the restored snapshots (except boot volume)
Instance will be switched off
If boot volume is being restored, default boot volume of created instance (coming from base AMI) will be de-attached and removed
Restored snapshot of boot volume will be used to create a new volume that will be attached to the restored instance
Instance will be switched on if necessary and all the snapshots used by the restore will be cleaned up
Restore Parameters
Amazon EC2 Plugin is able to do raw restore to any local filesystem mounted over the host where the File Daemon is
running, or to the Amazon EC2 environment. The restore method is selected based on the value of the where
parameter at restore time:
Empty or ‘/’ (example:
where=/) → Amazon EC2 restore will be triggeredAny other path for where (example:
where=/tmp) → Raw restore to the local file system will be triggered.
This principle generally applies to other Bacula plugins as well. Nevertheless, in this specific plugin, it is advised against using the raw restore method by the File Daemon. This is because the raw restore method retains the headers generated during the backup process in the volume disk files. If you want to restore the filesystem, you can trigger an Amazon EC2 restore using the where=/ parameter. However, you should utilize the restore variable to_local_path and specify the desired destination. By doing so, the mentioned headers will be automatically removed, and the resulting disk image will be suitable for any future purposes.
When using the Amazon EC2 restore method, the following parameters are available to control the restore behavior under ‘Plugin Options’ menu during a BConsole restore session:
Option |
Required |
Default |
Values |
Example |
Description |
|---|---|---|---|---|---|
instance_name |
No |
Original backup value |
Existing email address on the target EC2 service |
Set the name for a new restored instance, or specify an existing name or id where you want to attach the volumes |
|
instance_switchon |
No |
no |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Start the instance just after restoring it |
instance_type |
No |
Original backup value |
Accepted instance type in Amazon EC2. Defined in: https://aws.amazon.com/ec2/instance-types/ |
c7g.medium |
Set the instance type (c1.medium, a1.large…) |
instance_tags |
No |
Original backup value |
List of key value strings: key1=value1, key2=value2 |
restored=yes,testing=yes |
Add extra tags to the restored instance(s) (tag1key:tag1value,tag2key:tag2value…) |
instance_key_name |
No |
Original backup value |
The name of the access key pair that should be configured to allow SSH connections to the VM |
my-key-pair1 |
|
instance_generate_new_key |
No |
no |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
|
instance_no_network |
No |
no |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Do not generate any network configuration for the restore instance(s) |
instance_keep_ip_addresses |
No |
no |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Keep original IPv4 or IPv6 addresses of the instance(s). By default, they won’t be kept |
instance_security_groups |
No |
Original backup value |
Id of an existing Security Group in the destination region |
sg-23423adsfdfExampleSG |
Set a new set of security groups (ids separated by ‘,’) |
instance_no_licenses |
No |
no |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Do not restore the licenses configuration to the restored instance(s) |
instance_licenses |
No |
no |
List of license key strings separated by ‘,’ |
license1,license2 |
Set a new set of licenses to the restored instance(s) |
instance_placement_tenancy |
No |
Original backup value |
shared ; dedicated ; host ; default |
host |
Set up the tenancy value for the destination placement of the restored instance(s) |
instance_placement_affinity |
No |
Original backup value |
default ; host |
host |
Set up the affinity value for the destination placement of the restored instance(s) |
instance_placement_groupid |
No |
Original backup value |
String |
MyPlacementGroup |
Set up the group value for the destination placement of the restored instance(s) |
instance_image_id |
No |
Original backup value |
Id of an existing image in the destination AWS EC2 region |
ami-xx234ddfaSampleAMI,ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 |
Set a different base imageId. It supports specific image id, or ssm service tag: https://docs.aws.amazon.com/autoscaling/ec2/userguide/using-systems-manager-parameters.html |
instance_root_volume_id |
No |
Original backup value |
Id of the volume being restored (or file name) that should be root in the restored instance |
vol-12adf3SampleVol |
Set the volume that should be used as the root volume of the restore instance to boot it. Use special character ‘*’ to indicate to use the first restored volume |
volume_type |
No |
Original backup value |
Accepted volume type in Amazon EBS. Defined in: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html |
gp3 |
Set up the volume type for the restored disk image(s) |
volume_encrypted |
No |
Original backup value |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Encrypt the restored volume(s) |
volume_iops |
No |
Original backup value |
Positive integer |
3000 |
Iops value for the restored volume(s) |
volume_throughput |
No |
Original backup value |
Positive integer |
125 |
Throughput value for ther restored volume(s) |
availability_zone |
No |
Original backup value |
String: availability zone existing in: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html |
us-east-1a |
Set the destination availability zone for the instance(s) and its/their volume(s) |
image_generate_new |
No |
No |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
If restoring a root volume, generate the instance from a new image instead of root volume substitution |
image_name |
No |
String |
myCustomAmi |
Image name for the new image |
|
image_architecture |
No |
Original backup value or instance type value |
i386 ; x86_64 ; arm64 |
x86_64 |
Image architecture: i386, x86_64 or arm64 |
image_boot_mode |
No |
Original backup value |
uefi ; legacy-bios ; uefi-preferred |
legacy-bios |
Image boot mode: uefi, legacy-bios, or uefi-preferred |
image_virtualization_type |
No |
Original backup value or instance type value |
hvm ; paravirtual |
hvm |
Image virtualization type: hvm or paravirtual |
image_ena_support |
No |
Yes |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Image support for ENA (Elastic Network Adapter) network devices |
image_imds_support |
No |
Original backup value |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Image support for IMDS (Instance Metadata Service) |
image_tpm_support |
No |
Original backup value |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
yes |
Image support for TPM (Trusted Platform Module) |
access_key |
No |
Original backup value |
String |
AKIAIOSFODNN7EXAMPLE |
Set a different access key to access to the destination |
secret_key |
No |
Original backup value |
String |
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY |
Set a different secret key to access to the destination |
region |
No |
Original backup value |
String: region code existing in: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html |
us-east-2 |
Set the destination region |
to_local_path |
No |
Accessible and existing local path |
/tmp/my-local-restore |
Local path to restore the information, including the disk images without bitmap headers |
|
from_local_path |
No |
Accessible and existing local path containing all the files of an instance backup |
/tmp/my-local-backup |
Local path to restore the information from it, instead of the Bacula job |
|
dry_run |
No |
no |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
true |
Do not actually create any object in Amazon EC2 (just for running tests) |
debug |
No |
0 |
0, 1, 2 ,3, 4, 5, 6, 7, 8, 9 |
8 |
Change debug level |
Restore Use Cases
The following restore scenarios are supported, and steps to execute them are described:
Restore an instance to Amazon EC2, to its original location, and keeping all its configuration :
Run a restore session selecting appropriate backup jobs
Select all the contents of the instance directory (named with the instance id: i-xxxxxxxx/)
Use
Where=/
Important: Amazon EC2 Plugin uses the name (the value of the tag ‘Name’) to decide if an instance is existing. It means that for a) case to happen it is needed that the original instance with the original name is no more there or to have a different name.
Restore an instance to Amazon EC2 with a different name:
Follow previously described ‘a’ scenario.
Set
instance_namewith the value of the desired new name
Restore an instance to Amazon EC2, but generate a new access key for it:
Follow previously described ‘a’ or ‘b’ scenarios
Before confirming the restore operation, set
instance_generate_new_keywithyesRun the restore. The joblog will show the associated secret to the generated new key
Restore an instance to Amazon EC2, but adjust any configuration parameter of it (advanced):
Follow previously described ‘a’, ‘b’ or ‘c’ scenarios.
Adjust any desired parameter among below list:
Underlying hardware parameters:
instance_type,instance_placement_tenancy,instance_placement_affinity,instance_placement_groupidNetworking and security:
instance_security_groups,instance_key_name,instance_keep_ip_addressesCustomization of the instance:
image_id,instance_tags,instance_no_licensesPlease, note that the configuration you set here will be applied as you put it. Therefore, the consistency of that configuration will depend on everything being correct (existing and consistent) at Amazon EC2 side for your particular infrastructure
Restore an instance to Amazon EC2, but to a different location:
Follow previously described ‘a’, ‘b’, ‘c’ or ‘d’ scenarios.
Adjust
region,access_key,secret_keywith the destination values
Restore an instance to Amazon EC2, but adjust any configuration parameter of the selected EBS volumes (advanced):
Follow previously described ‘a’, ‘b’, ‘c’, ‘d’ or ‘e’ scenarios.
Adjust
volume_type,volume_encrypted,volume_iops,volume_throughputwith the desired valuesPlease, note that the configuration you set here will be applied as you put it. Therefore, the consistency of that configuration will depend on everything being correct (existing and consistent) at Amazon EC2 side for your particular infrastructure
Restore a volume to an existing instance:
Run a restore session selecting appropriate backup jobs
Select the desired volume yyyyy.bimg file(s) inside a given i-xxxxxxx/ folder
Use
Where=/Set
instance_namewith the value of an existing instance in the destination region that will hold the restored volumes
Restore instance files or volume files to a local directory:
Run a restore session selecting appropriate backup jobs
Select the desired files (or all of them) inside a given i-xxxxxxx/ folder
Use
Where=/Adjust
to_local_pathto the desired path
Restore just an instance without any disk (use image root disk):
Run a restore session selecting appropriate backup jobs
Select the just the i-xxxxx.json file
Use
Where=/
Restore a complete instance from a local directory with bec2 script:
Run bec2 command using a local folder containing a full instance backup like the example below, and using the desired name for the restored instance:
bec2 –operation=restore –region=us-east-1 –access_key=ABCXXXEXAMPLEKEY –secret_key=xxdrperefkafasfdExampleSecret123 –from_local_path=/my-local-path/amazon-ec2/i-xxxxxxx__my-instance –instance_name=RestoredInstanceName
Restore Example Session
In the following restore example session, we retrieve all the emails from the backup and restore them back to the original mailbox, instance_no_network inside the ‘restored’ folder.
Note
It is also possible to run backup or restore operations from any of the Bacula Graphical User Interfaces.
restore jobid=1 Client=127.0.0.1-fd where="/"
Using Catalog "MyCatalog"
You have selected the following JobId: 1
Building directory tree for JobId(s) 1 ...
5 files inserted into the tree.
You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.
cwd is: /
$ cd "/@amazon-ec2/"
cwd is: /@amazon-ec2/
$ dir
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-ec2/i-0c762632709fa24fa__REGRESS_20231115155818/
$ mark *
5 files marked.
$ done
Bootstrap records written to /tmp/regress/working/127.0.0.1-dir.restore.2.bsr
The Job will require the following (*=>InChanger):
Volume(s) Storage(s) SD Device(s)
===========================================================================
TEST-2023-11-15:0 File FileStorage
Volumes marked with "*" are in the Autochanger.
5 files selected to be restored.
Using Catalog "MyCatalog"
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.2.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2023-11-15 16:04:18
Catalog: MyCatalog
Priority: 10
Plugin Options: *None*
OK to run? (Yes/mod/no): mod
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Restore Client
6: When
7: Priority
8: Bootstrap
9: Where
10: File Relocation
11: Replace
12: JobId
13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : amazon-ec2: region="us-east-1" access_key="AKIAQVXBC4DMSVXJERBX" secret_key="mETUHxdz8vjA8DyWLmzFSaJDudXz8Nyh1J4FW3vP" instances="REGRESS_20231115155818" debug=6
Plugin Restore Options
Option Current Value Default Value
instance_name: *None* (*None*)
instance_switchon: *None* (*None*)
instance_type: *None* (*none*)
instance_security_groups: *None* (*none*)
instance_tags: *None* (*none*)
instance_key_name: *None* (*none*)
instance_generate_new_key: *None* (*none*)
instance_no_network: *None* (*none*)
instance_keep_ip_addresses: *None* (*none*)
instance_no_licenses: *None* (*none*)
instance_placement_tenancy: *None* (*none*)
instance_placement_affinity: *None* (*none*)
instance_placement_groupid: *None* (*none*)
instance_image_id: *None* (*none*)
volume_type: *None* (*none*)
volume_encrypted: *None* (*none*)
volume_iops: *None* (*none*)
volume_throughput: *None* (*none*)
availability_zone: *None* (*None*)
access_key: *None* (*None*)
secret_key: *None* (*None*)
region: *None* (*None*)
to_local_path: *None* (*None*)
from_local_path: *None* (*None*)
dry_run: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (Yes/mod/no): mod
You have the following choices:
1: instance_name (Set the name for a new restored instance, or specify an existing name or id where you want to attach the volumes)
2: instance_switchon (Start the instance just after restoring it)
3: instance_type (Set the instance type (c1.medium, a1.large...))
4: instance_security_groups (Set a new set of security groups (ids separated by ','))
5: instance_tags (Add extra tags to the restored instance(s) (tag1key:tag1value,tag2key:tag2value...))
6: instance_key_name (Set the name of the access key to access the new created instance(s))
7: instance_generate_new_key (Generate a new access key to access the new created instance(s))
8: instance_no_network (Do not generate any network configuration for the restore instance(s))
9: instance_keep_ip_addresses (Keep original IPv4 or IPv6 addresses of the instance(s))
10: instance_no_licenses (Do not restore the licenses configuration to the restored instance(s))
11: instance_placement_tenancy (Set up the tenancy value for the destination placement of the restored instance(s))
12: instance_placement_affinity (Set up the affinity value for the destination placement of the restored instance(s))
13: instance_placement_groupid (Set up the group value for the destination placement of the restored instance(s))
14: instance_image_id (Set a different base imageId)
15: volume_type (Set up the volume type for the restored disk image(s))
16: volume_encrypted (Encrypt the restored volume(s))
17: volume_iops (Iops value for the restored volume(s))
18: volume_throughput (Throughput value for ther restored volume(s))
19: availability_zone (Set the destination availability zone for the instance(s) and its/their volume(s))
20: access_key (Set a different access key to access to the destination)
21: secret_key (Set a different secret key to access to the destination)
22: region (Set the destination region)
23: to_local_path (Local path to restore the information, including the disk images without bitmap headers)
24: from_local_path (Local path to restore the information from it, instead of the Bacula job)
25: dry_run (Do not actually create any object in Amazon EC2)
26: debug (Change debug level)
Select parameter to modify (1-26): 1
Please enter a value for instance_name: REGRESS_20231115160418
Plugin Restore Options
Option Current Value Default Value
instance_name: REGRESS_20231115160418 (*None*)
instance_switchon: *None* (*None*)
instance_type: *None* (*none*)
instance_security_groups: *None* (*none*)
instance_tags: *None* (*none*)
instance_key_name: *None* (*none*)
instance_generate_new_key: *None* (*none*)
instance_no_network: *None* (*none*)
instance_keep_ip_addresses: *None* (*none*)
instance_no_licenses: *None* (*none*)
instance_placement_tenancy: *None* (*none*)
instance_placement_affinity: *None* (*none*)
instance_placement_groupid: *None* (*none*)
instance_image_id: *None* (*none*)
volume_type: *None* (*none*)
volume_encrypted: *None* (*none*)
volume_iops: *None* (*none*)
volume_throughput: *None* (*none*)
availability_zone: *None* (*None*)
access_key: *None* (*None*)
secret_key: *None* (*None*)
region: *None* (*None*)
to_local_path: *None* (*None*)
from_local_path: *None* (*None*)
dry_run: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (Yes/mod/no): mod
You have the following choices:
1: instance_name (Set the name for a new restored instance, or specify an existing name or id where you want to attach the volumes)
2: instance_switchon (Start the instance just after restoring it)
3: instance_type (Set the instance type (c1.medium, a1.large...))
4: instance_security_groups (Set a new set of security groups (ids separated by ','))
5: instance_tags (Add extra tags to the restored instance(s) (tag1key:tag1value,tag2key:tag2value...))
6: instance_key_name (Set the name of the access key to access the new created instance(s))
7: instance_generate_new_key (Generate a new access key to access the new created instance(s))
8: instance_no_network (Do not generate any network configuration for the restore instance(s))
9: instance_keep_ip_addresses (Keep original IPv4 or IPv6 addresses of the instance(s))
10: instance_no_licenses (Do not restore the licenses configuration to the restored instance(s))
11: instance_placement_tenancy (Set up the tenancy value for the destination placement of the restored instance(s))
12: instance_placement_affinity (Set up the affinity value for the destination placement of the restored instance(s))
13: instance_placement_groupid (Set up the group value for the destination placement of the restored instance(s))
14: instance_image_id (Set a different base imageId)
15: volume_type (Set up the volume type for the restored disk image(s))
16: volume_encrypted (Encrypt the restored volume(s))
17: volume_iops (Iops value for the restored volume(s))
18: volume_throughput (Throughput value for ther restored volume(s))
19: availability_zone (Set the destination availability zone for the instance(s) and its/their volume(s))
20: access_key (Set a different access key to access to the destination)
21: secret_key (Set a different secret key to access to the destination)
22: region (Set the destination region)
23: to_local_path (Local path to restore the information, including the disk images without bitmap headers)
24: from_local_path (Local path to restore the information from it, instead of the Bacula job)
25: dry_run (Do not actually create any object in Amazon EC2)
26: debug (Change debug level)
Select parameter to modify (1-26): 2
Please enter a value for instance_switchon: true
Plugin Restore Options
Option Current Value Default Value
instance_name: REGRESS_20231115160418 (*None*)
instance_switchon: true (*None*)
instance_type: *None* (*none*)
instance_security_groups: *None* (*none*)
instance_tags: *None* (*none*)
instance_key_name: *None* (*none*)
instance_generate_new_key: *None* (*none*)
instance_no_network: *None* (*none*)
instance_keep_ip_addresses: *None* (*none*)
instance_no_licenses: *None* (*none*)
instance_placement_tenancy: *None* (*none*)
instance_placement_affinity: *None* (*none*)
instance_placement_groupid: *None* (*none*)
instance_image_id: *None* (*none*)
volume_type: *None* (*none*)
volume_encrypted: *None* (*none*)
volume_iops: *None* (*none*)
volume_throughput: *None* (*none*)
availability_zone: *None* (*None*)
access_key: *None* (*None*)
secret_key: *None* (*None*)
region: *None* (*None*)
to_local_path: *None* (*None*)
from_local_path: *None* (*None*)
dry_run: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (Yes/mod/no): yes
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.2.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2023-11-15 16:04:18
Catalog: MyCatalog
Priority: 10
Plugin Options: User specified
OK to run? (Yes/mod/no): yes
Job queued. JobId=3
llist joblog jobid=3
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-dir JobId 3: Start Restore Job RestoreFiles.2023-11-15_16.04.18_09
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-dir JobId 3: Restoring files from JobId(s) 1
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-dir JobId 3: Connected to Storage "File" at 127.0.0.1:8103 with TLS
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-dir JobId 3: Using Device "FileStorage" to read.
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-dir JobId 3: Connected to Client "127.0.0.1-fd" at 127.0.0.1:8102 with TLS
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-fd JobId 3: Connected to Storage at 127.0.0.1:8103 with TLS
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-sd JobId 3: Ready to read from volume "TEST-2023-11-15:0" on Dedup device "FileStorage" (/tmp/regress/tmp).
time: 2023-11-15 16:04:20
logtext: 127.0.0.1-sd JobId 3: Forward spacing Volume "TEST-2023-11-15:0" to addr=268
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Plugin log of this job available in: /tmp/regress/working/amazon-ec2/amazon-ec2-debug-0.log
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Jar Version: 1.0.0
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Starting backend restore process
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Restoring to Amazon EC2 service
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Restoring from Bacula stream
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Instance name: REGRESS_20231115160418
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Instance type was not set. We will use the original one from each instance of the backup
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Instance switch-on: enabled
time: 2023-11-15 16:04:21
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Availability Zone was not set. We will use the original one from each volume
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Volume Encrypted not set, we will use the value from each original volume
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Volume Iops was not set. We will use the value from each original volume
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Volume Throughput was not set. We will use the value from each original volume
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Volume Type was not set. We will use the value from each original volume
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Generate New Key: disabled
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Key Name was not set. We will use the value from each original instance
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Image Id was not set. We will use the value from each original instance
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: No network: disabled
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: No licenses: disabled
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Placement Affinity was not set. We will use the value from each original instance
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Placement Group Id was not set. We will use the value from each original instance
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Placement Tenancy was not set. We will use the value from each original instance
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Security Groups were not set. We will use the values from each original instance
time: 2023-11-15 16:04:22
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: No extra tags will be added to restored instances
time: 2023-11-15 16:04:36
logtext: 127.0.0.1-sd JobId 3: Elapsed time=00:00:16, Transfer rate=110.3 M Bytes/second
time: 2023-11-15 16:04:36
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Restoring disk:vol-0fcff2bdcafb6cdf5 data to new snapshot
time: 2023-11-15 16:04:36
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Started snapshot:snap-0efed219eca86c531 for original volume:vol-0fcff2bdcafb6cdf5
time: 2023-11-15 16:04:36
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Sending data to destination for original volume:vol-0fcff2bdcafb6cdf5 ...
time: 2023-11-15 16:04:36
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: 3365 blocks were uploaded to volume:vol-0fcff2bdcafb6cdf5
time: 2023-11-15 16:04:42
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Completing snapshot:snap-0efed219eca86c531 with checksum:EzDgqUll+NZrc9sE8GivEP8l4Nj+C/fQAKGoAKvwSpE= ...
time: 2023-11-15 16:05:13
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Snapshot snap-0efed219eca86c531: is completed
time: 2023-11-15 16:05:58
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Finalizing volume restore for original volume:vol-0fcff2bdcafb6cdf5 ...
time: 2023-11-15 16:05:58
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Creating new instance from instance:i-0c762632709fa24fa ...
time: 2023-11-15 16:06:30
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Instance i-0a267ee7035475860: is running
time: 2023-11-15 16:06:30
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Stopping instance:i-0a267ee7035475860 ...
time: 2023-11-15 16:07:02
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Successfully stopped instance: i-0a267ee7035475860
time: 2023-11-15 16:07:02
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Dettaching and deleting automatically generated root volume for instance:i-0a267ee7035475860 ...
time: 2023-11-15 16:07:02
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Root volume of instance:i-0a267ee7035475860 is:vol-0f5649a3d860669ce
time: 2023-11-15 16:07:13
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Successfully deleted volume: vol-0f5649a3d860669ce
time: 2023-11-15 16:07:13
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Creating and attaching root volume from snapshot:snap-0efed219eca86c531 to instance:i-0a267ee7035475860
time: 2023-11-15 16:07:19
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Volume vol-0192896b074dad9e2: is available
time: 2023-11-15 16:07:19
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Cleaning created snapshots during the restore process
time: 2023-11-15 16:07:19
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Deleting snapshot:snap-0efed219eca86c531 ...
time: 2023-11-15 16:07:20
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Starting instance:i-0a267ee7035475860...
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: Successfully started instance: i-0a267ee7035475860
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-fd JobId 3: amazon-ec2: No more items to restore. Restore ended
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-dir JobId 3: Bacula 127.0.0.1-dir 16.0.7 (05Oct23):
Build OS: x86_64-pc-linux-gnu ubuntu 22.04
JobId: 3
Job: RestoreFiles.2023-11-15_16.04.18_09
Restore Client: "127.0.0.1-fd" 16.0.7 (05Oct23) x86_64-pc-linux-gnu,ubuntu,22.04
Where: /
Replace: Always
Start time: 15-nov-2023 16:04:20
End time: 15-nov-2023 16:07:51
Elapsed time: 3 mins 31 secs
Files Expected: 5
Files Restored: 5
Bytes Restored: 1,764,313,223 (1.764 GB)
Rate: 8361.7 KB/s
FD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Restore OK
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-dir JobId 3: Begin pruning Jobs older than 6 months .
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-dir JobId 3: No Jobs found to prune.
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-dir JobId 3: Begin pruning Files.
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-dir JobId 3: No Files found to prune.
time: 2023-11-15 16:07:51
logtext: 127.0.0.1-dir JobId 3: End auto prune.
jobid: 3
job: RestoreFiles.2023-11-15_16.04.18_09
name: RestoreFiles
purgedfiles: 0
type: R
level: F
clientid: 1
clientname: 127.0.0.1-fd
jobstatus: T
jobstatuslong: Completed successfully
schedtime: 2023-11-15 16:04:18
starttime: 2023-11-15 16:04:20
endtime: 2023-11-15 16:07:51
realendtime: 2023-11-15 16:07:51
realstarttime: 2023-11-15 16:04:20
jobtdate: 1,700,060,871
volsessionid: 3
volsessiontime: 1,700,060,507
jobfiles: 5
jobbytes: 1,764,313,223
readbytes: 1,764,312,106
joberrors: 0
jobmissingfiles: 0
poolid: 0
poolname:
priorjobid: 0
priorjob:
filesetid: 0
fileset:
hascache: 0
comment:
reviewed: 0
isvirtualfull: 0
rate: 8361.7
compressratio: 0
statusinfo:
writestorage:
writedevice:
lastreadstorage: File
lastreaddevice: FileStorage
Cross Hypervisor Restore and Virtual Machine Migration
Bacula Amazon EC2 Plugin is designed to provide also generic restore operations using RAW data from disk images. Those data disks can come from different Bacula Enterprise plugins working at disk image level, this is IaaS or Hypervisor Plugins, as well as any other technology producing disks in this RAW format.
Bacula Amazon EC2 Plugin works at the block level, so it is capable of reading and uploading just the blocks where the real information resides. As a result, migrations done with it are much more faster than any other done through the import method described by Amazon, where all the data needs to be uploaded first to a S3 bucket, before the import operation is called. More information: https://aws.amazon.com/ec2/vm-import/
A migration operaton is just a restore operation calling the bec2 script, where the parameter ‘from_local_path’ points to the folder or file where the data disks are placed. Then, using all the instance_* and image_* parameters, the migrated instance can be configured. Below an example:
/opt/bacula/bin/bec2 --operation=restore --region=us-east-1 --access_key=XXXXXX --secret_key=YYYYYYY --from_local_path=disk-0.bvmdk --instance_name=FromVMware1 --instance_switchon=true --instance_security_groups=sg-XXXXX --instance_root_volume_id=disk-0.bvmdk --image_generate_new=true
BEC2 will try to use files with ‘.img’ or ‘.bvmdk’ extensions. Contents must be in RAW format. If the format is different, normally it’s possible to convert it using tools like ‘qemu-img convert’: https://qemu-project.gitlab.io/qemu/tools/qemu-img.html
If you plan to use bec2 tool to migrate Virtual Machines coming from other virtualization systems, some planning and preparation is needed:
The safest restore method for migration is to restore instances in Amazon EC2 using supported images and default root/boot volumes that come with them. After, data to be migrated can be attached as a new volume. To use this method, bec2 restore operation needs to leave emtpy the parameter ‘instance_root_volume_id’.
When it comes to root volumes, it is possible to find inconveniences with booting or the networking after the instance was created in Amazon EC2. Amazon EC2 has some prerequsites to have Virtual Machines correctly working when they are created outside the service, detailed here: https://docs.aws.amazon.com/vm-import/latest/userguide/prerequisites.html
On the other hand, the internal configuration of the Virtual Machine has also to comply with some conditions, detailed here: https://docs.aws.amazon.com/vm-import/latest/userguide/prepare-vm-image.html
It is key to understand that if you plan to use your backups from a different environment as something to migrate to Amazon EC2, you must prepare the target Virtual Machines before the backup is performed.
We recommend to check those requirements as they can be changed by Amazon anytime. We list here an example on the actions that could be done on a given Linux VM to be migrated:
1. Check fstab entries are using UUID=xxx
2. Check dhcp client is enabled
3. Disable predictable Network interfaces (isn0...):
In /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="net.ifnames=0 biosdevname=0"
In /etc/sysconfig/network-scripts
cp ifcfg-ens192 ifcfg-eth0
-> Adapt the new script with the proper network interface name (eth0 here)
4. Check grub config exists and it's done with UUID
grep root /boot/grub2/grub.cfg
5. Disable any antivirus
6. Uninstall VMWare tools
7. Add xen, nvme and ena modules to initram
cat /etc/dracut.conf
# additional kernel modules to the default
add_drivers+="xen-blkfront xen-netfront nvme-core nvme ena virtio"
Update grub and initram
grub2-mkconfig -o /boot/grub2/grub.cfg
dracut -f -v
8. Enable access for non-root user with public ssh keys
ssh-keygen -t rsa -b 4096
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
# Enable SSH service
# Copy .ssh/id_rsa to source machine from which we want to connect
...
If a Virtual Machine was not prepared and you are still trying to migrate it with bec2, it is recommended to not use the original boot disk as root volume and just use a native boot volume from a standard Amazon AMI while attaching the other disks to it. Once connecting to the new instance, data can be moved manually to the new boot disk. Another alternative would be to try to modify the data contained in the RAW disk before attempting the restore with a tool like ‘virt-customize’ (https://libguestfs.org/virt-customize.1.html).
Once you have your virtual machines correctly prepared, a migration operation will typically consist on:
Restore your Virtual Machine to the local filesystem (in general, use where=/local/fs/path).
Convert the data with qemu-img to RAW format. This is not always needed (for instance, VMWare Plugin restores will be directly in RAW)
Run bec2 tool pointing to the restored disks
Image(AMI) and root volume options
For Virtual Machine migration operations root volume and image management options are more relevant compared to regular restores coming from Amazon EC2 Plugin backups.
The different options to restore images are described below:
Use an existing image in Amazon EC2 and attach migrated data disks to it:
Use ‘instance_root_volume_id’ and ‘instance_image_id’ options.
Use ‘instance_name’ with a non existing instance name in Amazon EC2
Create a new image in Amazon EC2, using one of the volumes as root disk, while attaching the others after to the generated instance:
Use ‘instance_root_volume_id’, enable ‘instance_generate_new’ and any ohter ‘instance_*’ desired option.
Use ‘instance_name’ with a non existing instance name in Amazon EC2
Use one of the data disks as root disk, while attaching the others, but do not create a new image:
Use ‘instance_root_volume_id’ with the volume that should be root
Use ‘instance_name’ with a non existing instance name in Amazon EC2
Use an existing instance and attach our data disks to without modifying root volume
Use ‘instance_name’ with the existing instance name in Amazon EC2
Use an existing instance and attach our data disks to while replacing root volume
Use ‘instance_name’ with the existing instance name in Amazon EC2
Use ‘instance_root_volume_id’ with the volume that should be root
Connection key options
When restoring a root disk, all the information contained for booting is holded by the data coming from the disk image provided. Therefore, in all associated restore scenarios, any option related with the SSH keys won’t take any effect (instance_key_name, instance_generate_new_key).
Networking options
Bacula Amazon EC2 Plugin will take backup values by default, but they can be fully personalized. Similarly, when using a RAW disk without any other metadata, default values will be used for networking, such as default subnet and default security group for the selected destination region. If you don’t specify your own values, please be aware that normally default Security Groups have no SSH or any other access rule allowed, so you may find difficulties to connect to your new instance.
Go back to the Operations article.
Go back to the Amazon EC2 plugin main page.
Go back to the main Dedicated Backup Solutions page.