FileSet Resource

The FileSet resource defines what files are to be included or excluded in a backup job. A FileSet resource is required for each backup Job. It consists of a list of files or directories to be included, a list of files or directories to be excluded and the various backup options such as compression, encryption, and signatures that are to be applied to each file.

Any change to the list of the included files will cause Bacula to automatically create a new FileSet (defined by the name and an MD5 checksum of the Include/Exclude contents). Each time a new FileSet is created, Bacula will ensure that the next backup is always a Full save.

Bacula is designed to handle most character sets of the world, US ASCII, German, French, Chinese, … However, it does this by encoding everything in UTF-8, and it expects all configuration files (including those read on Windows machines) to be in UTF-8 format. UTF-8 is typically the default on Linux machines, but not on all Unix machines, nor on Windows, so you must take some care to ensure that your locale is set properly before starting Bacula. On most modern Operating Systems, you can edit the conf files and choose output encoding UTF-8.

To ensure that Bacula configuration files can be correctly read including foreign characters the LANG environment variable must end in .UTF-8. A full example is en_US.UTF-8. The exact syntax may vary a bit from OS to OS, and exactly how you define it will also vary.

Bacula assumes that all filenames are in UTF-8 format on Linux and Unix machines. On Win32 they are in Unicode (UTF-16), and will be automatically converted to UTF-8 format.

FileSet Start of the FileSet resource. One FileSet resource must be defined for each Backup job.

Name

Name = <name> The name of the FileSet resource. This directive is required.

Ignore Fileset Changes

Ignore FileSet Changes = <yes|no> Normally, if you modify the FileSet Include or Exclude lists, the next backup will be forced to a Full so that Bacula can guarantee that any additions or deletions are properly saved.

We strongly recommend against setting this directive to yes, since doing so may cause you to have an incomplete set of backups.

If this directive is set to yes, any changes you make to the FileSet Include or Exclude lists, will not force a Full during subsequent backups. Note that any changes to Options resources in the FileSet are not considered by this directive. You can use the Accurate mode for this to be treated correctly, or schedule a new Full backup manually.

The default is no, in which case, if you change the Include or Exclude lists, Bacula will force a Full backup to ensure that everything is properly backed up.

Enable VSS

EnableVSS = <yes|no> If this directive is set to yes the File daemon will be notified that the user wants to use a VSS backup for this Job. The default is yes.

This directive is effective only for VSS enabled Win32 File daemons. It permits a consistent copy of open files to be made for cooperating writer applications, and for applications that are not VSS aware, Bacula can at least access open files.

As of Bacula version 10.2, only the VSS snapshot will be used to identify files to back up. In prior versions, files were searched in the “regular” file system and backed up from the snapshot.

In earlier versions of Bacula, a VSS snapshot was only created on Windows drives that were explicitly mentioned in a File List; in many cases, this is no longer necessary.

Enable Snapshot

EnableSnapshot = <yes|no> If this directive is set to yes the File Daemon will be notified that the user wants to use the Snapshot Engine for this job. The default is no. This directive is effective only for Snapshot enabled Unix File Daemons. It permits a consistent copy of open files to be made for cooperating applications. The bsnapshot tool should be installed on the Client.

Include

Include {Options {<file-options>}; <file-list> }

Exclude

Exclude {<file-list>}

The Include resource must contain a list of directories and/or files to be processed in the backup job. Normally, all files found in all subdirectories of any directory in the Include File list will be backed up. Note, see below for the definition of the file-list. The Include resource may also contain one or more Options resources that specify options such as compression to be applied to all or any subset of the files found when processing the file-list for backup. Please see below for more details concerning Options resources.

There can be any number of Include resources within the FileSet, each having its own list of directories or files to be backed up and the backup options defined by one or more Options resources. The file-list consists of one file or directory name per line. Directory names should be specified without a trailing slash with Unix path notation.

Windows users, please take note to specify directories (even c:/…) in Unix path notation. If you use Windows conventions, you will most likely not be able to restore your files due to the fact that the Windows path separator was defined as an escape character long before Windows existed, and Bacula adheres to that convention (i.e. means the next character appears as itself).

You should always specify a full path for every directory and file that you list in the FileSet. In addition, on Windows machines, you should always prefix the directory or filename with the drive specification (e.g. c:/xxx) using Unix directory name separators (forward slash). The drive letter itself can be upper or lower case (e.g. c:/xxx or C:/xxx).

Bacula’s default for processing directories is to recursively descend in the directory saving all files and subdirectories. Bacula will not by default cross filesystems (or mount points in Unix parlance). This means that if you specify the root partition (e.g. /), Bacula will save only the root partition and not any of the other mounted filesystems. Similarly on Windows systems, you must explicitly specify each of the drives you want saved (e.g. c:/ and d:/…). In addition, at least for Windows systems, you will most likely want to enclose each specification within double quotes particularly if the directory (or file) name contains spaces. The df command on Unix systems will show you which mount points you must specify to save everything. See below for an example.

Take special care not to include a directory twice or Bacula will backup the same files two times wasting a lot of space on your archive device. Including a directory twice is very easy to do. For example:

Include {
    Options {compression=GZIP}
    File = /
    File = /usr
    }

on a Unix system where /usr is a subdirectory (rather than a mounted filesystem) will cause /usr to be backed up twice.

Please take note of the following items in the FileSet syntax:

There is no equal sign (=) after the Include and before the opening brace ({). The same is true for the Exclude.
Each directory (or filename) to be included or excluded is preceded by a File =. Previously they were simply listed on separate lines.
The options that previously appeared on the Include line now must be specified within their own Options resource.
The Exclude resource does not accept Options.
When using wild-cards or regular expressions, directory names are always terminated with a slash (/) and filenames have no trailing slash.

The Options resource is optional, but when specified, it will contain a list of keyword=value options to be applied to the file-list. See below for the definition of file-list. Multiple Options resources may be specified one after another. As the files are found in the specified directories, the Options will applied to the filenames to determine if and how the file should be backed up. The wildcard and regular expression pattern matching parts of the Options resources are checked in the order they are specified in the FileSet until the first one that matches. Once one matches, the compression and other flags within the Options specification will apply to the pattern matched.

A key point is that in the absence of an Option or no other Option is matched, every file is accepted for backing up. This means that if you want to exclude something, you must explicitly specify an Option with an exclude = yes and some pattern matching.

Once Bacula determines that the Options resource matches the file under consideration, that file will be saved without looking at any other Options resources that may be present. This means that any wild cards must appear before an Options resource without wild cards.

If for some reason, Bacula checks all the Options resources to a file under consideration for backup, but there are no matches (generally because of wild cards that don’t match), Bacula as a default will then backup the file. This is quite logical if you consider the case of no Options clause is specified, where you want everything to be backed up, and it is important to keep in mind when excluding as mentioned above.

However, one additional point is that in the case that no match was found, Bacula will use the options found in the last Options resource. As a consequence, if you want a particular set of “default” options, you should put them in an Options resource after any other Options.

It is a good idea to put all your wild-card and regex expressions inside double quotes to prevent conf file scanning problems.

This is perhaps a bit overwhelming, so there are a number of examples included below to illustrate how this works.

You find yourself using a lot of Regex statements, which will cost quite a lot of CPU time, we recommend you simplify them if you can, or better yet convert them to Wild statements which are much more efficient.

The directives within an Options resource may be one of the following:

compression=GZIP All files saved will be software compressed using the GNU ZIP compression format. The compression is done on a file by file basis by the File Daemon. If there is a problem reading the tape in a single record of a file, it will at most affect that file and none of the other files on the tape. Normally this option is not needed if you have a modern tape drive as the drive will do its own compression. In fact, if you specify software compression at the same time you have hardware compression turned on, your files may actually take more space on the volume.

Software compression is very important if you are writing your Volumes to a file, and it can also be helpful if you have a fast computer but a slow network, otherwise it is generally better to rely your tape drive’s hardware compression. As noted above, it is not generally a good idea to do both software and hardware compression.

Specifying ``GZIP`` uses the default compression level 6 (i.e. ``GZIP`` is identical to GZIP6). If you want a different compression level (1 through 9), you can specify it by appending the level number with no intervening spaces to ``GZIP``. Thus compression=GZIP1 would give minimum compression but the fastest algorithm, and compression=``GZIP``9 would give the highest level of compression, but requires more computation. According to the GZIP documentation, compression levels greater than six generally give very little extra compression and are rather CPU intensive.

You can overwrite this option per Storage resource with Allow Compression option.
compression=LZO All files saved will be software compressed using the LZO compression format. The compression is done on a file by file basis by the File Daemon. Everything else about GZIP is true for LZO.

LZO provides much faster compression and decompression speed but lower compression ratio than GZIP. If your CPU is fast enough you should be able to compress your data without making the backup duration longer.

Note

Bacula only use one compression level LZO1X-1 specified by LZO.

You can overwrite this option per Storage resource with Allow Compression option.

Signature

Signature=SHA1 An SHA1 signature will be computed for all The SHA1 algorithm is purported to be some what slower than the MD5 algorithm, but at the same time is significantly better from a cryptographic point of view (i.e. much fewer collisions, much lower probability of being hacked.) It adds four more bytes than the MD5 signature. We strongly recommend that either this option or MD5 be specified as a default for all files. Note, only one of the two options MD5 or SHA1 can be computed for any file.

Signature=SHA256 A SHA256 signature will be computed for all files saved. Adding this option generates about 5% extra overhead for each file saved. In addition to the additional CPU time, the SHA256 signature adds 44 more bytes per file to your catalog.

Signature=SHA512 A SHA256 signature will be computed for all files saved. Adding this option generates about 5% extra overhead for each file saved. In addition to the additional CPU time, the SHA512 signature adds 87 more bytes per file to your catalog.

Signature=MD5 An MD5 signature will be computed for all files saved. Adding this option generates about 5% extra overhead for each file saved. In addition to the additional CPU time, the MD5 signature adds 16 more bytes per file to your catalog. We strongly recommend that this option or the SHA1 option be specified as a default for all files.

Dedup

Dedup=none||storage|bothsides The Dedup Fileset option can use the following values:

storage All the deduplication work is done on Storage Daemon side if the device type is dedup (default value).
none Force File Daemon and Storage Daemon to not use deduplication even if the device type is dedup.
bothsides The deduplication work is done on the FD and the SD if the device type is dedup.

Basejob

Basejob=<options> The options letters specified are used when running a Backup Level=Full with BaseJobs. The options letters are the same as in the accurate= option below.

Accurate=<options> The options letters specified are used when running a Backup Level=Incremental/Differential in Accurate mode. The options letters are the same as in the verify= option below with the addition of the following options:

A Always backup files

M Look mtime/ctime like normal incremental backup

o Check signature if other options would cause a backup, and if the signature is unchanged, back up only file metadata.

The ’o’ option should be used in conjunction with one of the signature checking options (1, 2, 3, or 5). When the option is specified, the signature is computed only for files that have one of the other accurate options specified triggering a backup of the file (for example an inode change, a permission change, etc…). In cases where only the file’s metadata has changed (ie: the signature is identical), only the file’s attributes will be backed up. If the file’s data has been changed (hence a different signature), the file will be backed up in the usual way.

This can save considerable space when backing up files which have only metadata part changed (e.g. changed permissions). The ’pinug2o’ options could be used, which will then cause the File Daemon to compare permision bits, inotes, number of links, owner user and group and if any of it changes it will also compute the file’s singature to verify if only metadata need to be backed up or is it needed to back up the file’s contents.

Verify

verify=<options> The options letters specified are used when running a Verify Level=Catalog as well as the DiskToCatalog level job. The options letters may be any combination of the following:

i compare the inodes

p compare the permission bits

n compare the number of links

u compare the user id

g compare the group id

s compare the size

a compare the access time

m compare the modification time (st_mtime)

c compare the change time (st_ctime)

d report file size decreases

5 compare the MD5 signature

1 compare the SHA1 signature

2 compare the SHA256 signature

3 compare the SHA512 signature

o Save only the metadata of a file when possible (available since Bacula Enterprise 12.8.2)

The ‘o’ option should be used in conjunction with one of the signature checking options (1, 2, 3, or 5). When the option is specified, the signature is computed only for files that have one of the other accurate options specified triggering a backup of the file (for example an inode change, a permission change, etc…). In cases where only the file’s metadata has changed (ie: the signature is identical), only the file’s attributes will be backed up. If the file’s data has been changed (hence a different singature), the file will be backed up in the usual way

A useful set of general options on the Level=Catalog or Level=DiskToCatalog verify is pins5 i.e. compare permission bits, then inodes, number of links, size, and finally MD5 changes.

To save some space when backing up files which have only metadata part changed (e.g. changed permissions) the ‘pino5’ options could be used (or some other combination with the ‘o’ flag). It will then compare permision bits, inotes, number of links and if any of it changes it will also compute file’s singature to verify if only metadata need to be backed up or is it needed to update file’s contents.

Onefs

Onefs=<yes|no> If set to yes (the default), Bacula will remain on a single file system. That is it will not backup file systems that are mounted on a subdirectory. If you are using a *nix system, you may not even be aware that there are several different filesystems as they are often automatically mounted by the OS (e.g. /dev, /net, /sys, /proc …). It will inform you when it decides not to traverse into another filesystem. This can be very useful if you forgot to backup a particular partition. An example of the informational message in the job report is:

bacula-fd: /misc is a different filesystem. Will not descend from / into /misc
bacula-fd: /net is a different filesystem. Will not descend from / into /net
bacula-fd: /var/lib/nfs/rpc_pipefs is a different filesystem. Will not descend from /var/lib/
bacula-fd: /selinux is a different filesystem. Will not descend from / into /selinux
bacula-fd: /sys is a different filesystem. Will not descend from / into /sys
bacula-fd: /dev is a different filesystem. Will not descend from / into /dev
bacula-fd: /home is a different filesystem. Will not descend from / into /home

Note

In older versions of Bacula, the above message was of the form:

Filesystem change prohibited. Will not descend into /misc

If you wish to backup multiple filesystems, you can explicitly list each filesystem you want saved. Otherwise, if you set the onefs option to no, Bacula will backup all mounted file systems (i.e. traverse mount points) that are found within the FileSet. Thus if you have NFS or Samba file systems mounted on a directory listed in your FileSet, they will also be backed up. Normally, it is preferable to set onefs=yes and to explicitly name each filesystem you want backed up. Explicitly naming the filesystems you want backed up avoids the possibility of getting into a infinite loop recursing filesystems. Another possibility is to use onefs=no and to set fstype=ext4, …. See the example below for more details.

If you think that Bacula should be backing up a particular directory and it is not, and you have onefs=no set, before you complain, please do:

stat /
stat <filesystem>

where you replace filesystem with the one in question. If the Device: number is different for / and for your filesystem, then they are on different filesystems. E.g.

stat /
    File: ‘/’
    Size: 4096          Blocks: 16      IO Block: 4096    directory
Device: 302h/770d       Inode: 2         Links: 26
Access: (0755/drwxr-xr-x) Uid: (0/ root) Gid: (0/root)
Access: 2005-11-10 12:28:01.000000000 +0100
Modify: 2005-09-27 17:52:32.000000000 +0200
Change: 2005-09-27 17:52:32.000000000 +0200

stat /net
    File: ‘/home’
    Size: 4096         Blocks: 16       IO Block: 4096    directory
Device: 308h/776d      Inode: 2         Links: 7
Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/root)
Access: 2005-11-10 12:28:02.000000000 +0100
Modify: 2005-11-06 12:36:48.000000000 +0100
Change: 2005-11-06 12:36:48.000000000 +0100

Also be aware that even if you include /home in your list of files to backup, as you most likely should, you will get the informational message that “/home is a different filesystem” when Bacula is processing the / directory. This message does not indicate an error. This message means that while examining the File = referred to in the second part of the message, Bacula will not descend into the directory mentioned in the first part of the message. However, it is possible that the separate filesystem will be backed up despite the message. For example, consider the following FileSet:

File = /
File = /var

where /var is a separate filesystem. In this example, you will get a message saying that Bacula will not decend from / into /var. But it is important to realise that Bacula will descend into /var from the second File directive shown above. In effect, the warning is bogus, but it is supplied to alert you to possible omissions from your FileSet. In this example, /var will be backed up. If you changed the FileSet such that it did not specify /var , then /var will not be backed up.

Honor nodump flag

Honor nodump flag=<yes|no> If your file system supports the nodump flag (e.g. most BSD-derived systems and many Linux file systems) Bacula will honor the setting of the flag when this option is set to yes. Files having this flag set will not be included in the backup and will not show up in the catalog. For directories with the nodump flag set recursion is turned off and the directory will be listed in the catalog. If the honor nodump flag option is not defined or set to no every file and directory will be eligible for backup.

Portable

Portable=<yes|no> If set to yes (default is no), the Bacula File Daemon will backup Win32 files in a portable format, but not all Win32 file attributes will be saved and restored. By default, this option is set to no, which means that on Win32 systems, the data will be backed up using Windows BackupRead API calls and all the security and ownership attributes will be properly backed up (and restored). However this format is not portable to other systems – e.g. Unix, Win95/98/Me. When backing up Unix systems, this option is ignored, and unless you have a specific need to have portable backups, we recommend accept the default (no) so that the maximum information concerning your Windows files is saved.

Note

The portable directive shouldn’t be required in recent Windows/Linux versions as recent Linux versions can handle files with win32 format.

Recurse

Recurse=<yes|no> If set to yes (the default), Bacula will recurse (or descend) into all subdirectories found unless the directory is explicitly excluded using an exclude definition. If you set recurse=no, Bacula will save the subdirectory entries, but not descend into the subdirectories, and thus will not save the files or directories contained in the subdirectories. Normally, you will want the default (yes).

Sparse

Sparse=<yes|no> Enable special code that checks for sparse files such as created by ndbm. The default is no, so no checks are made for sparse files. You may specify sparse=yes even on files that are not sparse file. No harm will be done, but there will be a small additional overhead to check for buffers of all zero, and if there is a 32K block of all zeros (see below), that block will become a hole in the file, which may not be desirable if the original file was not a sparse file.

Restrictions: Bacula reads files in 64K buffers. If the whole buffer is zero, it will be treated as a sparse block and not written to the target Volume. However, if any part of the buffer is non-zero, the whole buffer will be written, possibly including large ranges of zeroes. As a consequence, Bacula’s detection of sparse blocks is in 64K increments rather than the system block size. If anyone considers this to be a real problem, please send in a request for change with the reason.

If you are not familiar with sparse files, an example is say a file where you wrote 512 bytes at address zero, then 512 bytes at address 1 million. The operating system will allocate only two blocks, and the empty space or hole will have nothing allocated. However, when you read the sparse file and read the addresses where nothing was written, the OS will return all zeros as if the space were allocated, and if you backup such a file, a lot of space will be used to write zeros to the Volume. Worse yet, when you restore the file, all the previously empty space will now be allocated using much more disk space. By turning on the sparse option, Bacula will specifically look for empty space in the file, and any empty space will not be written to the Volume, nor will it be restored. The price to pay for this is that Bacula must search each block it reads before writing it. On a slow system, this may be important. If you suspect you have sparse files, you should benchmark the difference or set sparse for only those files that are really sparse.

You probably should not use this option on files or raw disk devices that are not really sparse files (i.e. have holes in them).

Read Fifo

ReadFifo=<yes|no> If enabled, tells the Client to read the data on a backup and write the data on a restore to any FIFO (pipe) that is explicitly mentioned in the FileSet. In this case, you must have a program already running that writes into the FIFO for a backup or reads from the FIFO on a restore. This can be accomplished with the RunBeforeJob directive. If this is not the case, Bacula will hang indefinitely on reading/writing the FIFO. When this is not enabled (no, the default), the Client simply saves the directory entry for the FIFO.

Unfortunately, when Bacula runs a RunBeforeJob, it waits until that script terminates, and if the script accesses the FIFO to write into the it, the Bacula job will block and everything will stall. However, Vladimir Stavrinov as supplied tip that allows this feature to work correctly. He simply adds the following to the beginning of the RunBeforeJob script:

exec > /dev/null

No Atime

NoAtime=<yes|no> If enabled, and if your Operating System supports the O_NOATIME file open flag, Bacula will open all files to be backed up with this option. It makes it possible to read a file without updating the inode atime (and also without the inode ctime update which happens if you try to set the atime back to its previous value). It also prevents a race condition when two programs are reading the same file, but only one does not want to change the atime. It’s most useful for backup programs and file integrity checkers (and bacula can fit on both categories).

This option is particularly useful for sites where users are sensitive to their MailBox file access time. It replaces both the keepatime option without the inconveniences of that option (see below).

If your Operating System does not support this option, it will be silently ignored by Bacula.

Mtime Only

MtimeOnly=<yes|no> If enabled, tells the Client that the selection of files during Incremental and Differential backups should based only on the st_mtime value in the stat() packet. The default is no which means that the selection of files to be backed up will be based on both the st_mtime and the st_ctime values. In general, it is not recommended to use this option.

Keep atime

KeepAtime=<yes|no> The default is no. When enabled, Bacula will reset the st_atime (access time) field of files that it backs up to their value prior to the backup. This option is not generally recommended as there are very few programs that use st_atime, and the backup overhead is increased because of the additional system call necessary to reset the times. However, for some files, such as mailboxes, when Bacula backs up the file, the user will notice that someone (Bacula) has accessed the file. In this, case keepatime can be useful.

Note

If you use this feature, when Bacula resets the access time, the change time (st_ctime) will automatically be modified by the system, so on the next incremental job, the file will be backed up even if it has not changed. As a consequence, you will probably also want to use mtimeonly = yes as well as keepatime.

Check File Changes

CheckFileChanges=<yes|no> If enabled, the Client will check size, age of each file after their backup to see if they have changed during backup. If time or size mismatch, an error will raise. The default value is no.

bacula-fd: Client1.2023-03-31_09.46.21 Error: /tmp/test mtime changed during backup.

In general, it is recommended to use this option to backup regular files. It is not compatible with fifo or plugin streams.

Hardlinks

Hardlinks=<yes|no> When enabled (yes, the default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once. The process of keeping track of the hard links can be quite expensive if you have lots of them (tens of thousands or more). This doesn’t occur on normal Unix systems, but if you use a program like BackupPC, it can create hundreds of thousands, or even millions of hard links. Backups become very long and the File daemon will consume a lot of CPU power checking hard links. In such a case, set hardlinks=no and hard links will not be backed up. Note, using this option will most likely backup more data and on a restore the file system will not be restored identically to the original.

Wild

Wild=<string> Specifies a wild-card string to be applied to the filenames and directory names. Note, if Exclude is not enabled, the wild-card will select which files are to be included. If Exclude=yes is specified, the wild-card will select which files are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched.

You may want to test your expressions prior to running your backup by using the bwild program. Please see the Utilities chapter of Bacula Enterprise for more information. You can also test your full FileSet definition by using the in the estimate command. It is recommended to enclose the string in double quotes.

Enhanced Wild

EnhancedWild=<yes|no> The EnhancedWild directive controls how path name matching with wildcards is done.

The default is no, which makes wildcards not match match path separators. If set to yes, an asterisk, question mark, or a bracketed slash will also be matched by a slash. In other words, wildcards will then span path hierarchy.

Wild Dir

WildDir=<string> Specifies a wild-card string to be applied to directory names only. No filenames will be matched by this directive. Note, if Exclude is not enabled, the wild-card will select directories to be included. If Exclude=yes is specified, the wild-card will select which directories are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched.