Backup
The overall backup operation involves the following steps:
Identify all selected target database instances or clusters.
Trigger a snapshot of the selected targets. This snapshot is inherently incremental compared to the previous one (if applicable).
If export is enabled, trigger an export operation of the selected tables to S3 (to a bucket configured in the plugin).
If download is enabled, download the exported files and store them in Bacula (send them to the Storage Daemon).
If export delete option is enabled, delete the exported files from the S3 bucket.
Remove old snapshots according to the established snapshot retention policy (which also includes deleting associated export files if the
export_delete
is enabled).
Backup File Structure
A single database backup with export and download enabled will produce contents in the backup similar to the following ones:
$ dir
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/
-rw-r----- 1 nobody nogroup 645 2024-06-06 14:38:30 /@amazon-rds/jg-pg-aws/export_info_plugintest-20240606-135513-03.json
-rw-r----- 1 nobody nogroup 72801 2024-06-06 14:38:30 /@amazon-rds/jg-pg-aws/export_tables_info_plugintest-20240606-135513-03_from_1_to_38.json
-rw-r----- 1 nobody nogroup 3192 2024-03-28 11:27:41 /@amazon-rds/jg-pg-aws/jg-pg-aws.i.json
-rw-r----- 1 nobody nogroup 1103 1970-01-01 00:59:59 /@amazon-rds/jg-pg-aws/plugintest-20240606-135513-03.isnap.json
$ pwd
cwd is: /@amazon-rds/jg-pg-aws/
$ cd bacula
cwd is: /@amazon-rds/jg-pg-aws/bacula/
$ dir
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.basefiles/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.cdimages/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.client/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.counters/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.device/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.events/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.file/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.fileevents/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.filemedia/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.fileset/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.job/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.jobhisto/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.jobmedia/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.location/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.locationlog/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.log/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.malwaremd5/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.malwaresha256/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.media/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.mediatype/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.metaattachment/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.metaemail/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.object/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.path/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.pathhierarchy/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.pathvisibility/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.pool/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.restoreobject/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.snapshot/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.status/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.storage/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.tagclient/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.tagjob/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.tagmedia/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.tagobject/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.unsavedfiles/
---------- 0 root root 0 1970-01-01 01:00:00 /@amazon-rds/jg-pg-aws/bacula/public.version/
cd public.status/1
cwd is: /@amazon-rds/jg-pg-aws/bacula/public.status/1/
$ dir
-rw-r----- 1 nobody nogroup 1421 2024-06-06 14:36:49 /@amazon-rds/jg-pg-aws/bacula/public.status/1/part-00000-e8ed76c2-8a29-49c7-9e0f-771a666ebe74-c000.gz.parquet
The JSON files represent information about the database instance, the snapshot and the export operation. Additionally, there exists a directory for each table of the database, wherein one can find an Apache Parquet file containing the data of that specific table.
Snapshot and Exports Livecycle
The Bacula Enterprise Amazon RDS Plugin controls the whole livecycle of the snapshots created within the Amazon Web Services Cloud, as well as the livecycle of the data exported to S3.
Amazon RDS Snapshots are actually Amazon EBS Snapshots containing also certain transaction logs. The EBS layer is transparent to the user and cannot be directly controlled.
The database snapshots are sent to an S3 bucket, which is also not directly controllable. When an RDS instance or cluster snapshot is created, Amazon automatically manages these details in the background, creating the appropriate EBS device snapshot and storing the data in S3. Similarly, when a snapshot is removed from RDS, the corresponding data in S3 will be automatically removed.
Bacula Enterprise Amazon RDS Plugin will remove exported files from S3 after each backup if the export_delete
flag is enabled.
Snapshots that exceed the snapshots_retention
value will be deleted. Their associated exports, if they still exist, will also be deleted if export_delete
flag is enabled.
Go back to: Operations.