Migration to New Major Version of Bacula

Overview
- Scope
Upgrading Bacula
Catalog Upgrade
Notes on Directives and Commands
Checklist
- Critical Items
Examples
Upgrading a Major PostgreSQL Version

Overview 

This document demonstrates how to upgrade Bacula Enterprise from previous versions to version 18.0. If you are upgrading from one Bacula version to another you should first carefully read the ReleaseNotes of all major versions between your current version and the version to which you are upgrading. For many upgrades, especially for minor patch upgrades (e.g. between 18.0.0 and 18.0.1), there will be no database upgrade so the process is to simply install the new software version. However, in many of the major version upgrades there are also changes made to the Bacula Catalog database which require the additional steps of running one or more database upgrade scripts.

Contact Bacula Systems Support if you have any questions about the upgrade procedure.

Scope 

This document is specifically written for Bacula Systems Enterprise version 18.0. The upgrade procedures are very similar from one version to another, however this paper is directly applicable only to upgrading to Bacula Enterprise 18.0.x and the outlined procedures must be modified to be applicable to other versions.

Upgrading Bacula 

File Daemon 

As always, in any new release we attempt to support the prior File daemon version and usually even older File daemons can be supported. This avoids the need to do a simultaneous upgrade of many machines. For precisely which older versions of the File daemon are supported, please see the ReleaseNotes for the new version.

Linux/Unix File Daemon Deployment

On Linux/Unix, you can upgrade the File daemon by simply copying the new version (rpm, deb or tar) to the server, then install it using the package manager native to that system.

Tools such as CFEngine or Puppet may also be used for automatic distribution and upgrade. It is even possible to create a Bacula Restore job that will put the Bacula binaries on the remote server, then execute a ClientRunAfterJob runscript at the end of the job to deploy the new version.

Microsoft Windows File Daemon Deployment

On Windows, it is possible to use Samba tools to deploy your new version remotely from the Director. First, use the net command to stop the bacula-fd service, then copy new files on the server using the smbclient or smbtar tool and the restart the bacula-fd service.

$ net -U administrator%passwd -S your-server rpc service stop bacula-fd
$ smbclient //your-server/c$ '' -U administrator%passwd -N -Tx bacula-win32.tar
$ net -U administrator%passwd -S your-server rpc service start bacula-fd

Director and Storage Daemons 

You must always upgrade both the Director and the Storage Daemon(s) at the same time, and you must also upgrade any File daemon that is running on the same machine as a Director or a Storage daemon. Note: In general if only the patch number changes (the third digit of the Bacula version), you will not need to upgrade Storage daemons that are on separate machines. However, for major version changes, for example from 6.4.x to 18.0.x, you must also upgrade all of your Storage daemons on all machines.

BWeb Management Suite Catalog Upgrade 

When upgrading BWeb Management Suite from 6.x, 8.0, 8.2, 8.4, 8.6, you must update the BWeb SQL tables with the script upgrade-6.0-8.8_postgresql.sql or upgrade-6.0-8.8_mysql.sql.

For MySQL users, when upgrading BWeb Management Suite from 8.8, 10.0 you must update the BWeb SQL tables with the script upgrade-8.8-10.2_mysql.sql.

BCloud Service Upgrade 

The major release 12.2 of BCloud Service uses a new catalog format. We provide a SQL script that converts 12.0.x and earlier catalog versions (2) to the new 12.2 format (3).

% psql -U bacula bacula < /opt/bacula/bcloud/www/protected/conf/upgrade-10.0_12.2_postgresql.sql

After applying the SQL script, please go to Admin => Directors page on the BCloud Service interface and please fill for each defined Director the Address and Port fields.

Upgrade Process Overview 

This release of Bacula uses a new catalog format.

We provide a set of scripts that convert earlier Catalog versions to the new 18.0.0 format (database version 1027). You can utilize the scripts provided in order to perform the manual Catalog upgrade process as described in the Section 2. We strongly recommend that you save a copy of your existing database before upgrading.

Bacula Systems Support can assist if you have questions prior to or during the upgrade process.

In the case where you did not perform the manual Catalog upgrade prior to the installation of the Bacula upgrade binaries, the upgrade process will be carried out automatically. Please note that as a first step of the automatic Catalog upgrade process, a dump of the Catalog database will be placed in the /opt/bacula/working directory. Please make sure that this directory has enough free space available to hold this Catalog dump. Once the automatic upgrade process is complete, the database backup file will not be deleted automatically.

Generally, we recommend the automatic Catalog upgrade approach. However, in scenarios where the size of the Catalog database is relatively large and/or that there is not enough free space on the partition that holds the /opt/bacula/working directory we recommend performing the manual Catalog upgrade process or to stop the Catalog service prior to the upgrade.

The 1027 Catalog format (for Bacula version 18.0.x) is a quick and simple schema upgrade.

[upgradeoverview] General Bacula upgrade process:

Stop any current version of Bacula from running.
To skip the automatic Catalog upgrade, stop the Catalog service.
Install the 18.0.x version of Bacula.
If you have not performed the manual Catalog upgrade, the install will upgrade your database automatically.
If you have stopped the Catalog service, restart it and run the update_bacula_tables script.
Start the new Bacula daemons. If everything works correctly, no error messages should be printed.

Catalog Upgrade 

Catalog Backup 

We strongly advise you to backup your Bacula catalog database before making any changes to it. To do so, you can use the make_catalog_backup.pl script provided with Bacula (see your catalog backup job for the exact command). Keeping the SQL result file on your local disk rather than in a Bacula volume or on tape can speed up recovery if something goes wrong.

Bacula Enterprise Binaries 

If you are using the Bacula Enterprise binaries, you will need the update scripts that are discussed in the following sections. These scripts are contained in a special updatedb package. For rpms you should install:

bacula-enterprise-updatedb-x.y.z.rpm

where the x and y vary depending on what rpm and version you are installing. Normally this rpm will install the scripts in the /opt/bacula/scripts/updatedb directory. If this is not the case please take note that some of the paths shown in subsequent sections will need to be adjusted accordingly.

For systems that use the .deb packages, these upgrade scripts will already be installed in the /opt/bacula/scripts directory.

Scripts may be extracted from packages using a tool such a mc, or the Bacula Systems support team can provide the latest version of the migration scripts on demand. If the database server is stopped during the installation, automatic backup and update procedures are skipped.

Upgrade Procedure 

Upgrading the catalog is normally done after Bacula is installed by:

cd /opt/bacula/scripts       (or where the scripts can be found)
./update_bacula_tables

If there are several database upgrades between your version and the version to which you are upgrading (see table below), you will need to run the database upgrade script for each version. Version 12, 13, 14, 1014, 1015, 1017, 1018, 1019, 1020, 1025, 1026, 1027 catalogs are converted using the update_bacula_tables script. For your convenience, you can find all the old upgrade scripts in the upgradedb package for RPMs and in /opt/bacula/scripts. If you don’t find these scripts, please contact the Bacula Systems support team. You will need to edit the scripts to correspond to your system configuration. The final upgrade script can be applied as noted above.

Catalog Versions 

Bacula version	Catalog version
1.38.x	9
2.0.x	10
2.2.x	10
2.4.x	10
2.6.x	11*
3.0.x	11*
5.0.x	12
4.0.x	13
5.2.x	14
6.0.x	1014
6.4.x	1015
6.6.x	1016*
8.0.x	1016*
8.2.x	1017
8.6.x	1018
8.8.x	1019
8.10.x	1019
10.0.x	1020
10.2.x	1020
12.0.x	1021
12.2.x	1021
12.4.x	1022
12.6.x	1023
12.8.x	1024
14.0.x	1025
16.0.x	1026
18.0.x	1027

The time and storage space required for the database upgrades with “” in the table above depends on the number of files that your catalog has stored and the capacity of your hardware. For versions listed without the “*”, upgrades should proceed very quickly.*

How to Verify the Current Version Number 

$ bconsole
*sql
Entering SQL query mode.
Terminate each query with a semicolon.
Terminate query mode with a blank line.
Enter SQL query: SELECT VersionId FROM Version;
+-----------+
| VersionId |
+-----------+
|      1027 |
+-----------+

Important Note on 2.4.x to 3.0.x Upgrade (Catalog Versions 10 to 11)

In this upgrade, we changed the size of the FileId field from 32 bits (integer) to 64 bits (bigint). If you have already done this prior to this upgrade, you need to remove the ALTER TABLE File command in the script update_postgresql_tables_10_to_11 or update_mysql_tables_10_to_11 (see below), otherwise the script may fail. If you have questions about this, or the script fails, please contact Bacula Systems support team.

On PostgreSQL:

$ psql -U bacula bacula
bacula=> \d File
...
fileid     | integer        (can be integer or bigint)

Or on MySQL:

$ mysql -u bacula bacula
mysql> describe File;
...
| FileId     | bigint(20) unsigned

Important Note on 6.0, 6.2, 6.4 to 6.6, 8.x,10.x,12.0 Upgrade (Catalog Version 1016 and 1015)

For a PostgreSQL database, the upgrade from 1015 to 1016 will create a large temporary file in the local directory (where the script is executed). Make sure that the filesystem is big enough to create the file (up to 20% of the database size).

For a MySQL database, or when using the Advanced Upgrade Procedure for PostgreSQL, the upgrade from 1015 to 1016 will create a copy of the File table in the database, please make sure that the database filesystem is big enough to create the table copy (up to 80% of the database size).

For more information, see Section myupgrade, and Section pgupgrade.

When installing RPMs or Debs, the package installer will try to dump the old catalog and upgrade the catalog automatically. This means that you must have the space available for a dump, and during a major upgrade such as moving from 6.6.x to 12.8.0, the database upgrade procedure will require additional disk space equal to the size of your database. Thus in the worst case, unless you off-load the dump, you will need additional disk of two times the size of your existing database.

In the case of extremely large databases where you do not have sufficient disk space, it is possible to perform the database upgrade prior to doing the Bacula package upgrade. This will prevent the package upgrade from detecting an old database version and the database dump will be skipped, thereby removing the requirement of additional disk space of two times the size of your existing database.

MySQL Upgrade Process

Stop any current version of Bacula from running.
Save a copy of your existing database.
Ensure that there is enough free space on the catalog filesystem to hold a copy of the File table. The upgrade process will probably require 50% of the catalog size of free disk space before starting the upgrade process.
If the database is large, it is possible to upgrade the catalog prior the package installation (see the next section).
Install the 12.8.x version of Bacula.
If the install didn’t upgrade your database automatically, run the upgrade script by reading section runupgrade section of this document.
Start the new Bacula daemons. If everything worked correctly, no error messages should be printed.

The MySQL tmpdir directory (usually /tmp) must have adequate space for the upgrade. This may be as large as many gigabytes for catalogs with many files. You must also check the innodb_data_file_path setting. For example, the following might cause the upgrade to fail if more than 512MB is required:

innodb_data_file_path = ibdata1:10M:autoextend:max:512M

This should be changed to something like:

ibdata1:10M:autoextend

to allow unlimited space.

PostgreSQL Upgrade Process

The following parameters can be set to avoid extra work for the database. It will require a restart of the PostgreSQL (9.x) database.

archive_mode = off
wal_level = minimal

If you are using PostgreSQL replication, we advise you to stop it during the migration process, and re-synchronize your PostgreSQL slave instance after the migration. Don’t forget to change your settings back to their original values once the upgrade is finished.

When using PostgreSQL, it is possible to choose between two migration processes depending on the size of the catalog.

Default PostgreSQL Upgrade Procedure

The default method of upgrading the database requires the Director to be shut down during the upgrade. If your database is large and it is not possible to have a long downtime, please read the next section (advancedpg) about the advanced upgrade procedure, then contact the Bacula Systems support team for additional help.

Install the pigz or the lzop compression program. This step is not mandatory, but using a parallel compression program (like pigz or pbzip2) will speed up the process. By default, the upgrade script will detect if pigz, lzop or pbzip2 are installed. If neither are available, the upgrade script will use gzip.
Ensure you have enough free space on the catalog filesystem to hold a compressed copy of the File table (around 20% of the database size).
Stop any current version of Bacula from running.
Make sure you have a copy of your existing database.
Install the 12.8.x version of Bacula.
If the install didn’t upgrade your database automatically, run the upgrade script described in the runupgrade section of this document.
Start the new Bacula daemons. If everything worked correctly, no error messages should be printed.

In testing, a server with 16GB of RAM, RAID5 storage, and a 5GB File table with 17 million records produced 1.3GB of compressed dump and took 25 minutes to migrate the catalog format from 1015 to 1016.

The Bacula Systems installation packages (rpm, deb, …) will try to dump the Catalog before the actual upgrade, so if you don’t want to run the dump step, you can manually upgrade the catalog before installing the packages then skip the post-install step, which normally does the catalog upgrade when installing packages. (Possible with RPMs using the –nopost option).

Advanced Upgrade Procedure from 1015 to 1016

The advanced upgrade procedure permits running the biggest part of the catalog upgrade 1015 to 1016 while Bacula backup and restore jobs are running. Please, contact the Bacula Systems support team to evaluate your needs and to prepare the migration with you.

If upgrading from 6.2.x, 6.0.x or a previous version, first upgrade to 6.4.x.
Install the perl DBI module and the perl DBD::Pg (perl-DBI and perl-DBD-Pg on Redhat or libdbi-perl and libdbd-pg-perl on Debian/Ubuntu).
Ensure you have enough free space on the catalog filesystem to hold a copy of the File table (you can use \d+ psql command to display table size). The upgrade process will probably require between 50% and 100% of the catalog size of free space where the database is stored before to start.
Save a copy of your existing database.
Follow the section advpg2 to run the migration script
Install the 12.8.0 version of Bacula.
If not done automatically, upgrade the catalog form from 1016 to 1020
Start the new Bacula daemons. If everything worked correctly, no error messages should be printed.

Notes on Directives and Commands 

Deprecated Directives 

The “Allow Higher Duplicates” directive is now deprecated. It did not work as documented and was confusing.

The “Dedup Index Directory” File Daemon directive is no longer used to store information. Instead, the new “Client Rehydration” directive enables the “restore cache” feature. Please see the Global Endpoint Deduplication user’s guide for more information.

Changed Default Directives 

Since version 6.0.0, the default for “Allow Duplicate Jobs” has been changed from no to yes.

Since version 8.6.4, the default for “Heartbeat Interval” has been changed from 0 (disabled) to 300 seconds.

Since version 8.8.0, restricted Console users must modify their existing Console resources to specify DirectoryACL=*all* and UserIdAcl=*all* in order to continue to use the restore command. If these two directives are not set, the restore command will return an error.

Since version 12.0, all network communication are encrypted by default. To disable encryption, use TLS Enabled=no.

Changed Command Option 

The default for the “setbandwidth limit” option has been changed from kb/s to b/s. The option now accepts a speed suffix such as “kb/s”, “mb/s”.

Suse Enterprise Linux Changes 

On Suse Linux, the Director and the Storage Daemons are now started with the unix account “bacula”. To upgrade a Suse system, the ownership of the Bacula volumes and the configuration files must be changed to “bacula:bacula” and the PostgreSQL/MySQL configuration may also have to be adapted.

chown -R bacula:bacula /opt/bacula/archives
chown -R bacula:bacula /opt/bacula/working

# If needed
chown bacula:bacula /opt/bacula/etc/bacula-dir.conf
chown bacula:bacula /opt/bacula/etc/bacula-sd.conf

Package Changes 

Since version 16.0, The bacula-enterprise-client RPM package does not provide the bconsole program anymore. The program is available separatly in the bacula-enteprise-console RPM package.

Since version 10.2.0, Cloud packages are distributed with specific packages for each driver (S3, Google, Oracle, and Azure). To upgrade from a previous version, it is recommended to remove the previous bacula-enterprise-cloud-storage package and install the specific cloud type packages required.

Checklist 

Whether you are migrating to Bacula Enterprise from a different product or a prior, possibly community Bacula version, we recommend that you take your time before implementing any production Bacula backup system.

If you haven’t already done so, we recommend that you read Disk Backup Design article.

If you follow the instructions in this chapter, you will have covered most of the major problems that can occur. It goes without saying that if you ever find that we have left out an important point, please inform us, so that we can document it to the benefit of everyone.

Critical Items 

The following assumes that you have installed Bacula, you more or less understand it, you have at least worked through the tutorial or have equivalent experience, and that you have set up a basic production configuration. If you haven’t done the above, please do so and then come back here. The following is a sort of checklist. In most cases, you will find additional details elsewhere in the manual. The order is more or less the order you would use in setting up a production system (if you already are in production, please use the checklist anyway).

If you have a Bacula Systems subscription, we encourage you to contact us prior to upgrading so that we may schedule your upgrade when our support personnel are available. Most of us are in the CET timezone, and upgrading is best not done on a Friday. By scheduling it, you can be sure we will be available in case you have any unusual problems.
If you are upgrading, follow the instructions at the beginning of this white paper (upgradeoverview).
If you use tapes for backup, walk through all of the steps in the “Testing Your Tape Drive With Bacula” chapter of the Problem Resolution Guide. It may take you a bit of time, but it will eliminate any surprises.
Do at least one restore of files. If you backup multiple OS types (Linux, Solaris, HP, MacOS, FreeBSD, Win32, …), restore files from each system type.
Write a bootstrap file to a separate system for each backup job. The “Write Bootstrap” directive is described in the manual, and more details are available in the “The Bootstrap File” chapter of the Bacula Main Reference Guide. Also, the default bacula-dir.conf comes with a “Write Bootstrap” directive defined. This allows you to recover the state of your system as of the last backup.
Backup your catalog. An example of this is found in the default bacula-dir.conf file. The backup script is installed by default and should handle any database, though you may want to make your own local modifications.
Write a bootstrap file for the catalog. An example of this is found in the default bacula-dir.conf file. This will allow you to quickly restore your catalog in the event it is wiped out – otherwise it may take many hours of work to recover.
Make a copy of the bacula-dir.conf, bacula-sd.conf, and bacula-fd.conf files that you are using on your server. Put them it in a safe place (on another machine) as these files can be difficult to reconstruct if they are lost or damaged.
Decide whether or not you need a bare metal recover, then either do it yourself or see about getting the Bacula Enterprise Bare Metal Restore programs for Linux and Windows.
Bacula assumes all filenames are in UTF-8 format. This is important when saving the filenames to the catalog. For Win32 machines, Bacula will automatically convert from Unicode to UTF-8, but on Unix, Linux, *BSD, and Apple OS X machines, you must explicitly ensure that your locale is set properly. Typically this means that the bf LANG environment variable must end in .UTF-8. An full example is en_US.UTF-8. The exact syntax may vary a bit from OS to OS, and exactly how you define it may also vary.

On most modern Win32 machines, you can edit the conf files with notepad and choose output encoding UTF-8.

Examples 

You should use the following commands only after installing the new version of Bacula, unless you want to explicitly skip the automatic backup and upgrade procedure due to a complex setup, for example.

On Redhat with PostgreSQL catalog backend, you must run the update scripts as the postgres Unix user.

# su - postgres
$ cd /opt/bacula/scripts
$ ./update_bacula_tables
$ ./grant_bacula_privileges
...

On Debian/Ubuntu, scripts can run as bacula Unix user. Further it is necessary to specify the database credentials through the shell variables such as db_user and sometimes editing the script to set the variable db_password.

# su - bacula
$ cd /opt/bacula/scripts
$ ./update_bacula_tables
$ db_user=bacula ./grant_bacula_privileges
...

The update_bacula_tables script handles multiple catalog versions in a single execution beginning with catalog version 12. For example a single execution of the script will permit upgrading from Bacula Enterprise 6.0.x to 16.0.

Example : Upgrade from 10.x to 18.0.x 

From the version table shown above, we can see that 10.0 and 10.2 uses catalog version 1020 and version 18.0 is 1027. The following procedure will upgrade your MySQL or PostgreSQL catalog:

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_bacula_tables
$ ./grant_bacula_privileges

Example : Upgrade from 8.10.x to 18.0.x 

From the version table shown above, we can see that 8.10 uses catalog version 1019 and version 18.0 is 1027. The following procedure will upgrade your MySQL or PostgreSQL catalog:

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_bacula_tables
$ ./grant_bacula_privileges

Example : Upgrade from 6.4.x to 18.0.x 

From the version table shown above, we can see that 6.4.x uses catalog version 1015, and version 18.0.x is 1027. The following procedure will upgrade your Postgresql catalog by applying 4 separate upgrade scripts:

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_bacula_tables
$ ./grant_bacula_privileges

The time and the space needed to run the upgrade from version 1015 to 1016 depends on your catalog size. As Bacula must be shutdown during this upgrade, you may want to perform it during non-working hours.

Typical output from the ./update_bacula_tables script will look like the following:

./update_bacula_tables
Altering postgresql tables

This script will update a Bacula PostgreSQL database from version
 12-14,1014-1026 to 1027 which is needed to convert from
 Bacula Enterprise version 4.0.x to 18.0.x
Dumping File table to /opt/bacula/scripts/file1016.data.
The process may fail if the current user
 doesn't have write permission on the current directory,
 or if the system doesn't have enough space to store a
 compressed export of the File table
BEGIN
DROP TABLE
DROP TABLE
CREATE TABLE
COMMIT
Loading the File table from /opt/bacula/scripts/file.3341.data...
date
Creation of indexes and PK on the File table...
SET
BEGIN
CREATE INDEX
CREATE INDEX
psql:-:6: NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "file_pkey"
for table "file"
ALTER TABLE
CREATE SEQUENCE
ALTER SEQUENCE
 setval
-----------
230909040
(1 row)
ALTER TABLE
ANALYZE
ALTER TABLE
ALTER TABLE
psql:-:29: NOTICE:  CREATE TABLE will create implicit sequence "snapshot_snapshotid_seq"
for serial column "snapshot.snapshotid"
psql:-:29: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "snapshot_pkey"
for table "snapshot"
CREATE TABLE
CREATE INDEX
UPDATE 1
COMMIT
BEGIN
CREATE TABLE
CREATE INDEX
UPDATE 1
COMMIT
Upgrade of the File table succeeded.
SET

Example : Upgrade from 6.0.x to 18.0.x 

From the version table shown above, we can see that 6.0.x uses catalog version 1014, and version 18.0.x is 1027. The following procedure will upgrade your Postgresql catalog by applying three separate upgrade scripts:

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_bacula_tables
$ ./grant_bacula_privileges

The time and the space needed to run the upgrade from version 1015 to 1016 depends on your catalog size. As Bacula must be shutdown during this upgrade, you may want to run it during non-working hours.

Example : Upgrade from 2.6.x to 18.0.x 

In the above table, we can see that 2.6.x uses catalog version 11, and version 18.0.x uses 1028. The following procedure will upgrade your Postgresql catalog in two steps.

$ cd /opt/bacula/scripts
$ ./update_postgresql_tables_11_to_12
$ ./update_postgresql_tables
$ ./grant_bacula_privileges

During the upgrade to database version 12, you may see errors about file_fp_idx and file_jpfid_idx indexes, but they can be ignored.

You may change the script parameters (or edit it) to fit your security or installation. (All extra command line parameters are passed to the database interpreter). In this example, the database server is located on host “192.168.0.1”

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_postgresql_tables_11_to_12 -h 192.168.0.1
$ cd /opt/bacula/scripts    # or possibly bacula/src/cats
$ ./update_postgresql_tables -h 192.168.0.1
$ ./grant_bacula_privileges -h 192.168.0.1

Example : Upgrade from 2.6.x to 18.0.x with MySQL 

For a MySQL database, the procedure is very similar the PostgreSQL upgrade described above:

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_mysql_tables_11_to_12
$ cd /opt/bacula/scripts    # or possibly bacula/src/cats
$ ./update_mysql_tables -u bacula
$ ./grant_bacula_privileges

Example : Upgrade from 2.0.x, 2.2.x or 2.4.x to 18.0.x 

From the version table shown above, we can see that 2.4.x uses catalog version 10, and version 18.0.x is 1027. The following procedure will upgrade your Postgresql catalog by applying four separate upgrade scripts:

$ cd /opt/bacula/scripts    # or possibly bacula/updatedb
$ ./update_postgresql_tables_10_to_11
$ ./update_postgresql_tables_11_to_12
$ cd /opt/bacula/scripts    # or possibly bacula/src/cats
$ ./update_postgresql_tables
$ ./grant_bacula_privileges

The time needed to run the upgrade from version 10 to 11 and 1015 to 1027 depends on your catalog size. As Bacula must be shutdown during this upgrade, you may want to perform it during non-working hours.

Example: Advanced PostgreSQL Upgrade from 6.4.x to 18.0.x 

The advanced migration script is designed to convert a 6.4.x schema to 18.0.x. You may need to convert your catalog from a previous version first up to 1015, please see Section stop1015.

As the upgrade from 6.4.x (1015) to 6.6.x (1016) can take quite long time, we advise you to use run the command from a session started with “screen”. In case of a network disconnection, the migration script will continue to run, and it is possible to re-attach the screen session using screen -r option.

$ screen
$ su - postgres
$ cd /opt/bacula/scripts
$ ./update_large_catalog_1015_1016 -D bacula
$ ./update_postgresql_tables
$ ./grant_bacula_privileges

The update_large_catalog_1015_1016 script is designed to be interactive and it is possible to stop the process at almost anytime using Ctrl-C (of course, if the break is done in a middle of a query, the script will have to do the work again at the next run).

$ su - postgres
$ /opt/bacula/scripts/update_large_catalog_1015_1016 -D bacula
16:23:14 INFO: Connexion to the catalog OK
16:23:14 INFO: Found catalog version 1015
16:23:14 INFO: Your catalog is already pretty big.
       Make sure to stay in touch with Bacula Systems Support team
       during the catalog migration

       The best way to proceed with your Catalog is to create a second
       File table while your Bacula is running. This method requires
       storing at least 2 times the size of your database during the
       migration.

       PLEASE MAKE SURE YOUR CATALOG FILESYSTEM IS BIG ENOUGH!!

Filesystem              Size  Used Avail Use% Mounted on
/dev/md0                 70G   31G  34.7G 43% /
/dev/md1                978G  887G   42G  96% /home
/dev/sdb3               493G   14G  454G   3% /boot

  -- Press enter key to continue or press Ctrl-C to cancel ---

If your system doesn’t have enough space to hold two copies of the File table, you will need to add more temporary space (for example using tablespace). If it is not possible to add extra space, the default migration procedure will be a better option. However, it will require stopping the Bacula Director during the migration (see Section Example : Upgrade from 6.4.x to 18.0.x).

16:33:16 INFO: Creating the temporary table 'nb_file_temp'
16:33:16 INFO: Creating the temporary File1016 table
16:33:16 INFO: Will update the temp File table with records
16:33:16 INFO: 2427 records have been inserted in the new table
16:33:16 INFO: Creating indexes if not already created
16:33:16 INFO: Now it's time to STOP bacula and start the final part of
               the upgrade

         Everything was committed to the database, so you can stop here
         and restart the process later.

  -- Press enter key to continue or press Ctrl-C to cancel ---

At this point, most of the work is done, the new table and all indexes are created. It is possible to stop here and re-run the script later, the script will ask if you want to continue the migration or restart from scratch.

For large databases, the first part of the migration can take up to 20 hours, it is advised to stop here and run the script again to reduce the final downtime.

Once the Bacula Director is stopped, press enter in the terminal and let the script finish the offline part of the migration.

39:34 INFO: Do a last update from file to temporary file table
39:34 INFO: 10 records have been inserted in the new table
39:34 INFO: Copying GRANT information
39:34 INFO: Dropping original File table
39:34 INFO: Restoring GRANT information for File sequence
39:34 INFO: Deleting orphan records
39:34 INFO: 0 records deleted
39:34 INFO: You can now upgrade bacula itself with your package manager

Once done, install the new 8.2.x version of bacula using your package manager.

Example: Advanced Upgrade from 6.0.x to 18.0.x with PostgreSQL 

To use the advanced migration script starting from a Catalog format prior to 1015 (6.4.x), it is possible to use the special –stop1015 option of the update_postgresql_tables script. The –stop1015 option will convert the catalog from an old revision up to the 1015 schema.

$ cd /opt/bacula/scripts
$ ./update_postgresql_tables --stop1015   # other options...
$ ./grant_bacula_privileges

Once done, follow instructions described in Section advpg2.

Upgrading a Major PostgreSQL Version 

Standard Procedure 

Note, this section applies to upgrading PostgreSQL rather than the Bacula table format.

Please note that the Bacula Systems QA team tests Bacula Enterprise with the default PostgeSQL version that comes with the RHEL, CentOS, Debian, SLES, and Ubuntu platforms. If you wish to run a more recent version of PostgreSQL or install the Catalog on another distribution, please contact the Bacula Systems Support for advice.

The standard procedure to upgrade PostgreSQL is described in https://www.postgresql.org/docs/10/static/upgrading.html It consists of making a SQL dump of your database, stopping it, installing the new version and reloading the SQL dump. This procedure can take a lot of time during which the database is unavailable. You can speed up the loading phase by changing the postgresql.conf file:

fsync = no
checkpoint_segments = 1G
checkpoint_timeout = 3000
checkpoint_completion_target = 0.9
autovacuum = off

These settings are not suitable for production. Please do not forget reset your production settings when the upgrade is finished.

At the end of the procedure, run the “ANALYZE” command.

Advanced Procedure 

PostgreSQL has the capability to restore a database using multiple connections at the same time. To use this feature, the dump should be done using pg_dump and the custom format.

First, you need to dump all roles:

postgres$ pg_dumpall --globals > /tmp/roles.sql

Then, you can to dump all databases:

postgres$ psql -AtU postgres -c "SELECT datname FROM pg_database WHERE NOT datistemplate"| \
while read f;
   do pg_dump -Upostgres --format=c --file=/tmp/$f.sqlc $f;
done;

In this example, dumps will be generated in /tmp. Feel free to change this location. Now you can install and start the new PostgreSQL version, and restore roles:

postgres$ psql -f /tmp/roles.sql

Finally, you can restore each database with the pg_restore command, ex:

postgres$ pg_restore -j 4 -C -d postgres /tmp/bacula.sqlc

Using the pg_upgrade Tool 

This method is very interesting because the migration is very fast compared to other methods.

pg_upgrade (or pg_migrator supports upgrades from/to 8.3.X and 8.4.X, including snapshot and alpha releases. For upgrading to 9.0.X or later, use the pg_upgrade tool that is part of /contrib. To use pg_upgrade, you need to install it from the postgresql contrib directory.

$ cd <postgresql source>/contrib/pg_upgrade
$ make install
$ cd ../pg_upgrade_support
$ make install

To continue, you need to define some variables about your installation.

postgres$ export NEWBINDIR=/opt/pg9.0/bin
postgres$ export NEWDATADIR=/opt/pg9.0/data
postgres$ export OLDBINDIR=/opt/pg8.4/bin
postgres$ export OLDDATADIR=/opt/pg8.4/data

Your catalog should be stopped to run the upgrade. The pg_upgrade tool will guide you if something is misconfigured.

postgres$ /opt/pg9.0/bin/pg_upgrade
Performing Consistency Checks
-----------------------------
Checking old data directory (/var/lib/postgres/data)        ok
Checking new data directory (/tmp/pg9/DATA)                 ok
Checking for /contrib/isn with bigint-passing mismatch      ok
... skip ...
Setting next oid for new cluster                            ok
Creating script to delete old cluster                       ok
Checking for large objects                                  ok

Upgrade complete

Important Note After Upgrading

In all cases, you need to run the ANALYZE command after the upgrade. Be sure to adapt the new postgresql.conf file to your previous tuning.