Retention Period Configuration
CommunityEnterpriseYou can define several retention times in your pools, but also for each clients. The first rule is that the file retention must be lower or equal to job retention which should be lower or equal to the volume retention. Normally you should set up Retention so they match the following rule:
File Retention <= Job Retention <= Volume Retention
The longer you keep file entries in the Catalog, the bigger the catalog. As a rule of thumb you can expect at least 150 bytes per file or object stored in the catalog.
Once the File Retention and the Job Retention are over for all Jobs stored in a Volume, this volume can be reused at any time by Bacula for next backup Jobs.
Retention times are used by Bacula to recycle your volume so your tapes and disk Volumes can be re-used. That’s why it is important to set them correctly so volumes are not recycled “too early” (see also figure Example for a simple schedule with 7 days of retention for Full and Incremental backups in section Schedules and Retentions). These retention settings must match your requirements.
In most cases, you should define File Retention and Job Retention in the Client{} resource, while the Volume Retention should be
defined in the Pool{} resource.
In a Client resource:
Client {
Neme = HRserv-fd # Human Ressources server File Daemon
...
File Retention = 94608000 # 3 years in seconds
Job Retention = 103680000 # 3 years and 4 months in seconds
}
In a Pool:
Pool {
Name = 3years-pool
Description = long term archiving Pool
...
File Retention = 94608000 # 3 years in seconds
Job Retention = 103680000 # 3 years and 4 months in seconds
Volume Retention = 106272000 # 3 years and 5 months in seconds
}
Note
If File retention is lower but not equal to Job Retention, you will, during the “gap” between those times, still be able to query the catalog for the Job but not directly for a file of this Job.
Pay great attention to retention times to avoid unwanted recycling or non-recycling.
Recycling Volumes
Let’s assume, as noted above, that you want to be able to restore a given file for at least a month on a daily basis. Generally you do so by running an Incremental backup once a day (usually at night) with occasional Full backups. If you write all your Incremental Jobs to the same Volume, and you have a Volume retention period of 30 days, and you started with your first Incremental Job written to Vol001 on the first day, then on the 31st day, you begin writing to Vol002, you will not be able to recover the space used by the Job written to Vol001 on the first day until 61 days after it was written. This is because the 30 day retention period on Vol001 begins counting down only after the last time it was written.
If instead of writing 30 days of Incremental data to a single Volume, you had written a new Volume every day, when your retention has expired, you will be able to recycle a bit each day rather than waiting an additional 30 days and recycling a big volume. In order to have 30 days of backup, you will need to keep your Full backups for at least 30 + 30 days so that you can always recover your system back 30 days. For Incremental Volumes you will need to keep them for 30 + 29 days (the one day of difference being the second Full which can substitute for an Incremental. This means that your retention period as specified in Bacula must be at least twice the period you want to be able to restore to on a day by day basis.
Thus the smaller the period of time that you write into a given Volume the faster the Volume can be recycled, but the more Volumes you will need. As an example, if you backup 1,000 clients a night, and all the data for a single day goes into one Volume, then at the end of 30 days, you will have 30 Volumes. If you write each client into a separate Volume, at the end of the 30 days, you will have 30,000 Volumes. It should be obvious that for Bacula to manage 30,000 Volumes or more is more compute intensive than if it is managing only 30 Volumes. However, the size of the Volumes for a given amount of data will be very different.
Often times, you will want to make some compromise to balance Volume size versus the number of Volumes. For example, you might limit the number of Jobs to 100 for a given Volume. Doing so with the above example, will mean that you will need 10 Volumes per night to store 1,000 Jobs or 300 for 30 days. Depending on your data size this may be a good compromise.
The Recycle Oldest Volume = yes record in the Pool resource tells Bacula to Prune the oldest volume in the Pool, and if all files were pruned to recycle this volume and use it.
The Recycle Current Volume = yes record in the Pool resource tells Bacula to Prune the currently mounted volume in the Pool, and if all files were pruned to recycle this volume and use it.
The Purge Oldest Volume = yes record in the Pool resource permits a forced recycling of the oldest Volume when a new one is needed. N.B. This record ignores retention periods! We highly recommend not to use this record, but instead use Recycle Oldest Volume
The Maximum Volumes record can also be used in conjunction with the Volume Retention period to limit the total number of archive Volumes (files) that Bacula will create. By setting an appropriate Volume Retention period, a Volume will be purged just before it is needed and thus Bacula can cycle through a fixed set of Volumes. Cycling through a fixed set of Volumes can also be done by setting Recycle Oldest Volume = yes or Recycle Current Volume = yes. In this case, when Bacula needs a new Volume, it will prune the specified volume.
Pruning Volumes
Catalog file record retention periods, the File Retention = ttt record in the Client resource.
Catalog job record retention periods, the Job Retention = ttt record in the Client resource.
The AutoPrune = yes record in the Client resource to permit application of the above two retention periods.
The first three directives (File Retention, Job Retention, and AutoPrune) determine the amount of time that Job and File records will remain in your Catalog, and they are discussed in detail in Volume Recycling.
Since automatic catalog pruning can be time-consuming, for maximum performance, you may want to avoid automatic pruning of old jobs at the end of each backup Job. To do that, set the directive AutoPrune = No in all Client resources, ensuring that the Job-level Prune Jobs and Prune Files directives are not enabled. Alternatively, to disable automatic pruning globally with a single directive, set AutoPrune = No in the Director resource. This [Director-level setting](https://docs.baculasystems.com/BETechnicalReference/Director/DirectorResourceTypes/DirectorResource/index.html#director-director-autoprune) overrides the settings in all other resources, eliminating the need to configure automatic pruning separately in each Client or Pool resource. Pruning should then be performed manually or through a [scheduled administrative Job](https://docs.baculasystems.com/BEManagement/StorageSpaceManagement/Pruning/DisablingAutomaticPruning/index.html#disabling-automatic-pruning) to prevent continuous Catalog growth.
New Prune “Expired” Volume Command
It is now possible to prune all volumes (from a pool, or globally) that are “expired”. This option can be scheduled after or before the backup of the Catalog and can be combined with the Truncate On Purge option.
This new Expired option to the Prune commands can be scheduled in a RunScript to explicitly handle the pruning. The following two variations are accepted:
* prune expired volumes
* prune expired volumes pool=FullPool
To schedule this option automatically, it can be added as a Runscript to the BackupCatalog job definition.
Job {
Name = CatalogBackup
...
RunScript {
Console = "prune expired volume yes"
RunsWhen = Before
}
}
Truncate Volume on Purge
By default, when Bacula purges a volume, it just changes the volume status in the catalog. It doesn’t physically touch your Volume or its data. Thus you can still recover data after the retention period has expired, but the Volume continues to occupy disk space. This is generally what one wants providing one has sufficient disks space. For tape based backup, this default behavior is also very useful because you may want to avoid extra mount operations to just truncate volumes.
However, by truncating the Volume when it is purged, you can recover the disk space allocated. To do this, you will need to add the directive:
Action On Purge = Truncate
to all your Pool resources, and you will need to schedule a console
RunScript to execute the purge during a period when Bacula is not
backing up files. (Note: if you add this directive, you may need to run
an update command to apply Pool’s configuration changes to the
Volumes already created in the catalog).
Below, we show an example of executing the truncate in the Job that runs the nightly catalog backup.
Job {
Name = CatalogBackup
...
RunScript {
RunsWhen=After
RunsOnClient=No
Console = "truncate volume allpools storage=File"
}
}
See also
Previous articles:
Next articles:
Go back to: Backup Policy Configuration.