Percona XtraDB Cluster in Kubernetes

This document outlines the procedure for utilizing the Bacula MySQL Plugin running in a container to perform backups of the Percona XtraDB Cluster.

For additional details regarding Percona XtraDB Cluster, visit: https://www.percona.com

The software versions used in this use case:

• Percona XtraDB Cluster: Server version: 8.0.36-28.1 Percona XtraDB Cluster (GPL), Release rel28, Revision bfb687f, WSREP version 26.1.4.3

• MySQL Client: mysql Ver 8.0.37 for Linux on x86_64 (MySQL Community Server - GPL)

• Percona XtraBackup: xtrabackup version 8.0.35-31 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 55ec21d7)

Bacula File Daemon and MySQL Plugin Kubernetes Deployment

Dockerfile

Use the following Dockerfile to build a Bacula File Daemon with the MySQL Plugin installed image.

# cat Dockerfile
FROM debian:bullseye
LABEL maintainer="Bacula Systems SA"
LABEL version="18.0.5"
LABEL name="Bacula Enterprise Edition Client"
LABEL vendor="BACULA SYSTEMS SA"
LABEL summary="This is a Bacula File Daemon with the MySQL plugin"
LABEL description="This image contains a Bacula File Daemon which allows connection between this pod resources and the Bacula Director."
# Update image
RUN apt update
# install required dependencies for the mysql client and Percona XtraBackup
RUN apt install lz4 ca-certificates curl gnupg lsb-release wget -y
# install the Bacula File Daemon and the MySQL Plugin
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://qa.baculasystems.com/dl/QA_SL_2-adsfJLU783jklAKjd667aJKNyX/BaculaSystems-Public-Signature-08-2017.asc -o /etc/apt/keyrings/bacula.asc
RUN chmod a+r /etc/apt/keyrings/bacula.asc
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/bacula.asc] https://qa.baculasystems.com/dl/QA_SL_2-adsfJLU783jklAKjd667aJKNyX/debs/bin/18.0.5/bullseye-64/ bullseye main" > /etc/apt/sources.list.d/Bacula-Enterprise-Edition.list
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/bacula.asc] https://qa.baculasystems.com/dl/QA_SL_2-adsfJLU783jklAKjd667aJKNyX/debs/mysql/18.0.5/bullseye-64/ bullseye mysql" > /etc/apt/sources.list.d/Bacula-Enterprise-Edition-mysql-plugin.list
RUN apt-get update
RUN apt-get install -y bacula-enterprise-client bacula-enterprise-mysql-plugin
# download and install the mysql client for Percona/MySQL 8.0
RUN wget https://repo.mysql.com/mysql-apt-config_0.8.29-1_all.deb
RUN DEBIAN_FRONTEND=noninteractive dpkg -i mysql-apt-config_0.8.29-1_all.deb
RUN apt-get update
RUN apt -y install mysql-client
RUN apt list --installed | grep mysql > /tmp/apt_mysql.txt
# download and install Percona XtraBackup 8.0
RUN wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
RUN DEBIAN_FRONTEND=noninteractive dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
RUN percona-release enable pxb-80 release
RUN apt -y install percona-xtrabackup-80
RUN apt autoremove
# use the bacula-fd.conf file previously configured for a specific Bacula Director / configuration
RUN rm /opt/bacula/etc/bacula-fd.conf
COPY bacula-fd.conf /opt/bacula/etc/bacula-fd.conf
# copy required my.cnf and .my.cnf files to the bacula-fd with the specific MySQL configuration needed to access the Percona cluster
COPY my.cnf /opt/bacula/etc
COPY .my.cnf /opt/bacula/etc
#  expose bacula-fd port
EXPOSE 9102
USER root
# Start the Bacula File Daemon service
CMD ["/opt/bacula/bin/bacula-fd", "-f"]

The .my.cnf file used in this use case:

~/bacula-fd-mysql# cat .my.cnf
[client]
host=10.43.198.239 <--- The ClusterIP address of the pxc-db-haproxy
user=root
password=2R7D9D.6V1I+5Z,F$ <--- root password get from the `pxc-db-secrets`

# kubectl get service -n pxc
NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
bacula-fd                         ClusterIP   10.43.80.87     10.0.97.201   9102/TCP                                5d22h
my-db-pxc-db-haproxy              ClusterIP   10.43.4.195     <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   12d
my-db-pxc-db-haproxy-replicas     ClusterIP   10.43.54.231    <none>        3306/TCP                                12d
my-db-pxc-db-pxc                  ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP            12d
my-db-pxc-db-pxc-unready          ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP            12d
percona-xtradb-cluster-operator   ClusterIP   10.43.248.174   <none>        443/TCP                                 12d
pxc-db-haproxy                    ClusterIP   10.43.198.239   <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   12d

# kubectl -n pxc get secrets pxc-db-secrets -o jsonpath="{.data.root}" | base64 --decode

Build an image using the Dockerfile, and tag/push it to a local registry in the Kubernetes Cluster.

Important Notes

• The mysql-client and the Percona XtraBackup for MySQL versions installed must be compatible with the Percona XtraDB Cluster MySQL version.

• Have a .my.cnf file with user-specific options to allow the connection from the MySQL Plugin to the Percona XtraDB Cluster.

• Have the my.cnf file used by the Percona XtraDB Cluster, or use a Kubernetes configmap.

• Have the bacula-fd.conf file previously configured (the Director the File Daemon will use, for example), or use a Kubernetes configmap for a valid bacula-fd.conf file.

• It is possible to use any base image - Debian Bullseye is used in this use case - but dependencies and external programs versions could change.

Create bacula-fd Deployment in Kubernetes Cluster

After creating the bacula-fd deployment in the Kubernetes Cluster, attach one of the datadir persistent volumes, used by the Percona Cluster, to the bacula-fd pod, and mount the volumes in the bacula-fd container.

It is important to provide an external access for the communication between the Bacula File Daemon in the Kubernetes Cluster and both the Director and the Storage Daemon, when the services run outside the Kubernetes Cluster.

In this use case, the EXTERNAL-IP configured to the bacula-fd service is the ip address of one of the Kubernetes Master nodes:

# kubectl get service -n pxc
NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
bacula-fd                         ClusterIP   10.43.80.87     10.0.97.201   9102/TCP                                5d22h

An External Load Balancer or Ingress can be used as well.

Backup and Restore using bacula-fd Service in Kubernetes Cluster

Using the MySQL Plugin, both DUMP and Binary modes can be used to backup the Percona XtraDB Cluster.

For details about the MySQL Plugin backup configuration, refer to the main MySQL Plugin page.

Fileset Configuration

The Fileset configuration used in this use case for DUMP and Binary mode backups:

Fileset {
    Name = "percona-dump-fileset"
    Include {
      Options {
        IgnoreCase = yes
        OneFs = no
        Signature = Md5
      }
      Plugin = "mysql: debug verbose abort_on_error mycnf_dir=\"/opt/bacula/etc/\" config_file=\"/opt/bacula/etc/my.cnf\""
    }
    Exclude {
    }
}

Fileset {
    Name = "percona-binary-fileset"
    Include {
      Options {
        IgnoreCase = yes
        OneFs = no
        Signature = Md5
      }
      Plugin = "mysql: debug abort_on_error mode=\"binary\" mycnf_dir=\"/opt/bacula/etc\" config_file=\"/opt/bacula/etc/my.cnf\" backup_software=\"xtrabackup\""
    }
    Exclude {
    }
}

Restore - BINARY mode - Complete Server - without preparation

To allow a restore of the data in the datadir Percona Cluster persistent volumes, one of the datadir persistent volumes must be mounted to the bacula-fd container as ReadWrite Access Mode.

It is recommended to stop the Percona XtraDB Cluster to perform the complete server restore:

# kubectl -n pxc patch --type=merge --patch='{ "spec": { "pause": true } }' pxc pxc-db
perconaxtradbcluster.pxc.percona.com/pxc-db patched

And remove all, if any, existent files from the current /var/lib/mysql directory in the datadir persistent volume. This operation can be done from the bacula-fd container.

Restore Files into Local Directory

Run a restore by selecting either Full or Incremental job corresponding to the specific point in time when the MySQL server should be restored. The files will be restored in a local directory of the bacula-fd container:

root@bacula-fd-86bfd77786-98qxm:/opt/bacula/archive/bacula-restores# ls -lR
.:
total 8
drwxrwsr-x 3 root 1001 4096 Nov 20 14:43 @MYSQL
drwxr-xr-x 3 root root 4096 Nov 21 09:05 opt

./@MYSQL:
total 4
drwxrwsr-x 2 root 1001 4096 Nov 20 14:43 main

./@MYSQL/main:
total 82764
-rw-r----- 1 root 1001  2020282 Nov 20 14:41 all-databases.34206973.xbstream
-rw-r----- 1 root 1001  1954746 Nov 20 14:42 all-databases.34208728.xbstream
-rw-r----- 1 root 1001 80760446 Nov 20 14:40 all-databases.xbstream
-rw-r--r-- 1 root root        9 Nov 15 14:13 my.cnf
-rw-r--r-- 1 root root      536 Nov 20 14:43 mysql.dat

This is an example to restore one Full and two Incremental jobs using BWeb:

Procedure using xtrabackup to Restore Data into datadir in Cluster Persistent Volumes

1. cd /opt/bacula/archive/bacula-restores to create a directory for the Full, and one for each Incremental job:

   • mkdir full

   • mkdir incr1

   • mkdir incr2

2. cd @MYSQL/main/ in the directory you have restored the files.

3. decompress the .xbstream files in the corresponding full, incr1 and incr2 directories:

   • cd /opt/bacula/archive/bacula-restores/full && xbstream -x < /opt/bacula/archive/bacula-restores/@MYSQL/main/all-databases.xbstream (Full job)

   • cd /opt/bacula/archive/bacula-restores/incr1 && xbstream -x < /opt/bacula/archive/bacula-restores/@MYSQL/main/all-databases.34206973.xbstream (first Incremental job)

   • cd /opt/bacula/archive/bacula-restores/incr1 && xbstream -x < /opt/bacula/archive/bacula-restores/@MYSQL/main/all-databases.34208728.xbstream (second and latest Incremental job)

4. Prepare Full and the Incremental jobs:

   • Prepare the Full job using the –apply-log-only option, in the Full level job directory:

     xtrabackup --prepare --apply-log-only --target-dir=/opt/bacula/archive/bacula-restores/full

   • Apply the Incremental backup to the Full backup using the –apply-log-only option:

     xtrabackup --prepare --apply-log-only --incremental-dir=incr1 --target-dir=/opt/bacula/archive/bacula-restores/full

   • Apply the latest Incremental backup to the Full backup not using the –apply-log-only option:

     xtrabackup --prepare --incremental-dir=incr2 --target-dir=/opt/bacula/archive/bacula-restores/full

5. Copy the backup to the datadir of the server/cluster:

   xtrabackup --datadir=/var/lib/mysql --copy-back --target-dir=/opt/bacula/archive/bacula-restores/full

Start Percona XtraDB Cluster

# kubectl -n pxc patch --type=merge --patch='{ "spec": { "pause": false } }' pxc pxc-db
perconaxtradbcluster.pxc.percona.com/pxc-db patched

Go back to the Use Cases page.

Go back to the MySQL Plugin page.