Etcd Backup

Starting with Bacula Enterprise version 18.2.0, the plugin includes support for backing up etcd. It also introduces new parameters (prefixed with etcd) specifically for etcd backups. Both single-node etcd deployments and high-availability (HA) configurations are supported.

What is Etcd?

Etcd is a reliable and highly available key-value store that serves as the backend for all cluster data in Kubernetes. Its main role in a Kubernetes cluster is to provide central storage for the cluster’s state, tracking all resources like Pods, Services, configuration objects and more.

When you make changes to the configuration in Kubernetes, such as deploying a new service or removing a Pod, these updates are recorded in etcd. If etcd fails or is lost, Kubernetes loses all information regarding the current cluster state and configuration. Consequently, if etcd becomes corrupted, the Kubernetes cluster may become unusable. In a production environment, this could lead to the loss of essential applications and data, significantly affecting service availability and business continuity.

Why is Backing Up Important?

Etcd acts as the “source of truth” for your Kubernetes cluster. Regular backups provide:

Disaster Recovery: Restore the cluster after data loss, corruption, or a major cluster failure.
Business Continuity: Reduce data loss and recovery time, ensuring that services remain operational.

Accessing Etcd Resources

Typically, etcd resides in the kube-system namespace called. You can access this database remotely using etcd certificates.

To obtain these certificates, you need to know their location on one of the etcd nodes (nodes with role etcd). You can find this information by checking the etcd Pod YAML file to see where the certificates are mounted on the host. You can then access it remotely to retrieve the certificates.

One way to get the Pods with etcd service is:

kubectl -n kube-system get pods | grep etcd
etcd-rke2-master01-bck                       1/1     Running     2              33d
etcd-rke2-master02-bck                       1/1     Running     1              33d
etcd-rke2-master03-bck                       1/1     Running     1              33d

Next, identify which configuration file is used by etcd:

kubectl -n kube-system get pods etcd-rke2-master01-back -o yaml | grep "\-file"
- --config-file=/var/lib/rancher/rke2/server/db/etcd/config

Now you can access an etcd node or an etcd Pod, and locate the certificate paths referenced in that configuration file:

root@rke2-master01-bck:~# cat /var/lib/rancher/rke2/server/db/etcd/config | grep file
cert-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.crt
key-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.key
trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
cert-file: /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt
key-file: /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.key
trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt

The important values of keys are: cert-file: server-client.crt, key-file:server-client.key and trusted-ca-file:server-ca.crt . These files allow you to access the etcd database. In this example, they are:

cert-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.crt
key-file: /var/lib/rancher/rke2/server/tls/etcd/server-client.key
trusted-ca-file: /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt

Import these files to your host and store them in a Kubernetes Secret:

Note

By default, the certificates expire after one year.

root@kubeadm-master01-bck:~# kubectl cp <etcd-pod-name>:/path/to/server-ca.crt ./server-ca.crt -n kube-system
root@kubeadm-master01-bck:~# kubectl cp <etcd-pod-name>:/path/to/server-client.crt ./server-client.crt -n kube-system
root@kubeadm-master01-bck:~# kubectl cp <etcd-pod-name>:/path/to/server-client.key ./server-client.key -n kube-system

The recommended method for importing these certificates into the Kubernetes Plugin is to create a Kubernetes Secret in the namespace where the etcd Pod is located. You can do this by following these steps:

root@kubeadm-master01-bck:~# kubectl create secret generic etcd-certificates \
--from-file=server-ca.crt=./server-ca.crt \
--from-file=server-client.crt=./server-client.crt \
--from-file=server-client.key=./server-client.key \
-n kube-system

Note

The key names in the Secret must be: server-ca.crt, server-client.crt and server-client.key.

Note

The Secret must be created in the kube-system namespace.

The final step is to add the etcd_secret parameter to the Bacula Fileset.

Alternatively, you can store the certificates in the Bacula File Daemon and include the parameters etcd_ca_cert, etcd_tls_cert and etcd_tls_key in the Bacula Fileset.

You must also set etcd_endpoints in the Fileset. You can set this parameter using the following command:

root@rke2-master01-bck:~# kubectl get pod -n kube-system -o wide | grep etcd
etcd-rke2-master01-bck                       1/1     Running   0              19h    10.0.97.201   rke2-master01-bck   <none>           <none>
etcd-rke2-master02-bck                       1/1     Running   0              19h    10.0.97.202   rke2-master02-bck   <none>           <none>
etcd-rke2-master03-bck                       1/1     Running   0              19h    10.0.97.203   rke2-master03-bck   <none>           <none>

Based on this output, you can set: etcd_endpoints=https://10.0.97.201:2379,https://10.0.97.202:2379,https://10.0.97.203:2379. You can also use hostnames, but ensure DNS resolution is working correctly.

In the future, the plugin will create this Kubernetes Secret with a .query command.