Backup

The overall backup operation involves the following steps:

Identify all selected target database instances or clusters.
Trigger a snapshot of the selected targets. This snapshot is inherently incremental compared to the previous one (if applicable).
If export is enabled, trigger an export operation of the selected tables to S3 (to a bucket configured in the plugin).
If download is enabled, download the exported files and store them in Bacula (send them to the Storage Daemon).
If export delete option is enabled, delete the exported files from the S3 bucket.
Remove old snapshots according to the established snapshot retention policy (which also includes deleting associated export files if the export_delete is enabled).

Backup File Structure

A single database backup with export and download enabled will produce contents in the backup similar to the following ones:

Files protected with the backup

 $ dir
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/
 -rw-r-----   1 nobody   nogroup          645  2024-06-06 14:38:30  /@amazon-rds/jg-pg-aws/export_info_plugintest-20240606-135513-03.json
 -rw-r-----   1 nobody   nogroup        72801  2024-06-06 14:38:30  /@amazon-rds/jg-pg-aws/export_tables_info_plugintest-20240606-135513-03_from_1_to_38.json
 -rw-r-----   1 nobody   nogroup         3192  2024-03-28 11:27:41  /@amazon-rds/jg-pg-aws/jg-pg-aws.i.json
 -rw-r-----   1 nobody   nogroup         1103  1970-01-01 00:59:59  /@amazon-rds/jg-pg-aws/plugintest-20240606-135513-03.isnap.json

 $ pwd
 cwd is: /@amazon-rds/jg-pg-aws/

 $ cd bacula
 cwd is: /@amazon-rds/jg-pg-aws/bacula/

 $ dir
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.basefiles/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.cdimages/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.client/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.counters/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.device/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.events/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.file/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.fileevents/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.filemedia/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.fileset/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.job/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.jobhisto/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.jobmedia/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.location/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.locationlog/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.log/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.malwaremd5/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.malwaresha256/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.media/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.mediatype/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.metaattachment/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.metaemail/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.object/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.path/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.pathhierarchy/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.pathvisibility/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.pool/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.restoreobject/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.snapshot/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.status/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.storage/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.tagclient/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.tagjob/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.tagmedia/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.tagobject/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.unsavedfiles/
 ----------   0 root     root               0  1970-01-01 01:00:00  /@amazon-rds/jg-pg-aws/bacula/public.version/

 cd public.status/1
 cwd is: /@amazon-rds/jg-pg-aws/bacula/public.status/1/

 $ dir
 -rw-r-----   1 nobody   nogroup         1421  2024-06-06 14:36:49  /@amazon-rds/jg-pg-aws/bacula/public.status/1/part-00000-e8ed76c2-8a29-49c7-9e0f-771a666ebe74-c000.gz.parquet

The JSON files represent information about the database instance, the snapshot and the export operation. Additionally, there exists a directory for each table of the database, wherein one can find an Apache Parquet file containing the data of that specific table.

Snapshot and Exports Livecycle

The Bacula Enterprise Amazon RDS Plugin controls the whole livecycle of the snapshots created within the Amazon Web Services Cloud, as well as the livecycle of the data exported to S3.

Amazon RDS Snapshots are actually Amazon EBS Snapshots containing also certain transaction logs. The EBS layer is transparent to the user and cannot be directly controlled.

The database snapshots are sent to an S3 bucket, which is also not directly controllable. When an RDS instance or cluster snapshot is created, Amazon automatically manages these details in the background, creating the appropriate EBS device snapshot and storing the data in S3. Similarly, when a snapshot is removed from RDS, the corresponding data in S3 will be automatically removed.

Bacula Enterprise Amazon RDS Plugin will remove exported files from S3 after each backup if the export_delete flag is enabled.

Snapshots that exceed the snapshots_retention value will be deleted. Their associated exports, if they still exist, will also be deleted if export_delete flag is enabled.