InitCatalog/Catalog Verification

The one shortcoming of using DiskToCatalog verification, if integrity checking of important files is desired, is that the verification process shown above only looks at files actually being saved. This approach may fail when a changed file is not actually being backed up, which will happen if

  1. A file is modified in a way that does not change its ctime or mtime metadata,

  2. Full backups are not run against the interesting set of files 9, and

  3. if Differential or Incremental backups are run, those are not set up to not use accurate mode with settings capturing those files.

In this situation, there simply may not exist a backup which includes the modified file, and accordingly, that file would not be verified.

The simple solution would be to run Full backups of the interesting files daily (or even more often). This would require more backup jobs to exist than required per an organizations backup policy, and it will also waste storage space. For those reasons, Bacula Enterprise offers another solution to this challenge: InitCatalog / Catalog verification.

Essentially, a Verify Job at level InitCatalog goes through a File Set, records all the metadata as configured, but does not actually back up any data. The result is that all the needed information to verify exists in the catalog. The next step then is to run a Verify Job against that original job. This is done with Verify Jobs at level Catalog.

Job {
  Name = Verify-RH1-sbin-Job-Catalog
  Type = Verify
  Level = Catalog
  Client = s-rh1-fd
  File Set = sbin-fs
  JobDefs = DefaultJob
}

A Job to implement this might look like the one shown in figure above.

We first generate the initial information in the catalog by issuing a command like run level=InitCatalog job=Verify-RH1-sbin-Job-Catalog yes to bconsole. The resulting Job Report should look similar to the one shown in figure fig:cvinit.

main-dir JobId 4: Start Verify JobId=4 Level=InitCatalog Job=Verify-RH1-sbin-Job-Catalog.2010-10-17_22.28.10_12
main-dir JobId 4: Bacula Enterprise main-dir 4.0.4 (04Sep10):
  Build:                  x86_64-unknown-linux-gnu suse 11.1
  JobId:                  24489
  Job:                    Verify-RH1-sbin-Job-Catalog.2010-10-17_22.28.10_12
  FileSet:                sbin-fs
  Verify Level:           InitCatalog
  Client:                 s-rh1-fd
  Verify JobId:           0
  Verify Job:
  Start time:             17-Oct-2010 22:28:12
  End time:               17-Oct-2010 22:28:14
  Files Examined:         279
  Non-fatal FD errors:    0
  FD termination status:  OK
  Termination:            Verify OK

Running the actual verification job can be done manually, through bconsole, BAT, or BWeb, or by scheduling it as usual (typically outside the backup window, or using priorities to prevent it from real backup jobs running in time). Note that there’s no need to schedule InitCatalog levels of this job – when verifying files that are expected to be static this would severely reduce the ability to detect file changes!

A Job Report for a Catalog verification without any changes detected is shown in figure fig:vcok, and one with discrepancies is shown in figure fig:vcnok.

main-dir JobId 29: Verifying against JobId=28 Job=Verify-RH1-sbin-Job-Catalog.2010-10-17_22.28.10_12
main-dir JobId 29: Start Verify JobId=29 Level=Catalog Job=Verify-RH1-sbin-Job-Catalog.2010-10-17_22.39.47_14
main-dir JobId 29: Bacula Enterprise main-dir 4.0.4 (04Sep10):
  Build:                  x86_64-unknown-linux-gnu suse 11.1
  JobId:                  24490
  Job:                    Verify-RH1-sbin-Job-Catalog.2010-10-17_22.39.47_14
  FileSet:                sbin-fs
  Verify Level:           Catalog
  Client:                 s-rh1-fd
  Verify JobId:           24489
  Verify Job:
  Start time:             17-Oct-2010 22:39:49
  End time:               17-Oct-2010 22:39:50
  Files Examined:         279
  Non-fatal FD errors:    0
  FD termination status:  OK
  Termination:            Verify OK

main-dir JobId 29: Verifying against JobId=28 Job=Verify-RH1-sbin-Job-Catalog.2010-10-17_22.28.10_12
main-dir JobId 29: Start Verify JobId=29 Level=Catalog Job=Verify-RH1-sbin-Job-Catalog.2010-10-17_22.47.10_15
main-dir JobId 29: File: /sbin/grub-install
main-dir JobId 29:       st_ino   differ. Cat: 229424 File: 229651
main-dir JobId 29:       st_size  differ. Cat: 18521 File: 18538
main-dir JobId 29:       st_mtime differs
main-dir JobId 29:       st_ctime differs
main-dir JobId 29:       MD5 digest differs. File=U8f9+qfyGoNZY9Fmf0j9hQ Cat=7cLXtYBAGwQNMU4yz+UFoQ
main-dir JobId 29: New file: /sbin/ls
main-dir JobId 29: File: /sbin/
main-dir JobId 29:       st_mtime differs
main-dir JobId 29:       st_ctime differs
main-dir JobId 29: Warning: The following files are in the Catalog but not on disk:
main-dir JobId 29:       /sbin/mkinitrd
main-dir JobId 29: Bacula Enterprise main-dir 4.0.4 (04Sep10):
  Build:                  x86_64-unknown-linux-gnu suse 11.1
  JobId:                  29
  Job:                    Verify-RH1-sbin-Job-Catalog.2010-10-17_22.47.10_15
  FileSet:                sbin-fs
  Verify Level:           Catalog
  Client:                 s-rh1-fd
  Verify JobId:           28
  Verify Job:
  Start time:             17-Oct-2010 22:47:12
  End time:               17-Oct-2010 22:47:13
  Files Examined:         279
  Non-fatal FD errors:    0
  FD termination status:  OK
  Termination:            Verify Differences

Only after explicitly allowed changes to the files covered by those Verify Jobs the job has to be run at InitCatalog level again. This would be the case after configuration changes or system updates. Notifying system or application administrators about this requirement is easily done and adds only minimal overhead to your existing maintenance procedures.

This sort of jobs allows to verify data integrity of files not being backed up at all, or not backed up recently. For security-aware organizations, this functionality may be essential. Seeing that Bacula can be one tool to do this, the question is: Why use Bacula for File Integrity checking?

There are several good reasons for it:

  • Bacula Enterprise is based on open source, and as such may be more trusted than a purely closed source solution.

  • You may already use Bacula Enterprise for backups, and it may be desirable not to install and maintain another software package.

  • Bacula Enterprise keeps its metadata, which it uses for integrity checking, off the checked systems, which is the only reliable way to prevent tampering with the baseline information once a system is compromised.

  • As a backup software is security relevant, a Bacula Enterprise installation, its data (like the catalog) and access to it should be closely controlled and well audited anyway; not having to maintain the environment of another software system under tight control may be a noticeable reduction of management effort.

  • Verification reports from Bacula Enterprise can be easily fed into monitoring or ticketing systems, which may not be easily possible for alternative integrity scanners. 10

  • Bacula Enterprise is highly customizable; it may allow you more flexibility and more detailed checks than other integrity solutions.

Bacula Systems accordingly believes that Bacula may be a good choice of a software package to use for file integrity scanning.

See also

Go back to:

Go to:

Go back to the main Advanced Features Usage page.

9

Which is quite common for files expected to be mostly static; many organizations prefer not to back up system files at all, as re-installing them from the original source may be easier than recovering from a backup. Regular full backups of static files is also, in a sense, a waste of backup space.

10

We will not discuss these items in this document; refer to Bacula Systems if you want to get more information about these aspects of Bacula Enterprise integration.