RHV Plugin: Troubleshooting

Bacula Enterprise Only

This solution is only available for Bacula Enterprise. For subscription inquiries, please reach out to sales@baculasystems.com.

The following chapter describes the troubleshooting of the RHV Plugin.

Bacula Environment

This plugin stores the logs depending of the number of paralel executions it has. In other words, if the operations are only performed synchronously, it will always be sotred in the same file, under the suffix of ’-0.log’. However, when a backup is running and another backup(or a restore) is started, the first backup will continue to store the log in the same file(under suffix of ’-0.log’), but the one just started will be stored in the file under the suffix of ’-1.log’.

The plugin indicates the log file in a message in job log. By default, the location of log file will be: /opt/bacula/working/rhv/rhv-debug-X.log, but the user can modify this default value in file /opt/bacula/rhv_backend, or in each job with param log.

It’s recommended that the log file always be in /opt/bacula/working/rhv/.

RHV Environment

The RHV Manager can return errors in some situations. The RHV REST API return code and messages do not always precisely reflect the cause of the problem. Given that general HTTP codes indicating rather broad error conditions are used, the plugin can not always react in a perfect manner. Instead, the plugin returns all available information about errors and warnings, but in many situations it will be necessary to check RHV logs to understand the cause of an operational error.

The most informative logs of RHV are usually vdc-log.txt and /var/log/ovirt-engine/engine.log (for RHV Manager) and /var/log/vdsm/vdsm.log (for virtualization hosts) since those reflect the main flow of the application and if any error occurs, it will generally be tracked in those logs.

Once the exact component that failed is known, it is possible to check the specific log file. The full list of RHV logs is described at redhat.com/…/17587#RHEV_Logs_Overview

Below we present some common errors and associated actions to resolve them.

Failed Transfer

If a disconnection happens when a disk image is being transferred by the ImageTransferService, the virtual disk remains in the state “Transfering via API” even if the backup job has terminated. This state blocks snapshot deletion. If a backup has suffered from non recoverable connection loss to the RHV Manager and this state remains for a virtual disk, a manual action on the RHV Manager can be performed to unblock the situation:

# log in to the RHV Manager shell, as root user
su postgres
psql -U engine -d engine
SELECT command_id FROM image_transfers;
# Get the command_id that is aborted
DELETE FROM image_transfers WHERE command_id='<UUID from above>';
# The state 'Transfering via API' will change to 'Locked' (2).
# Query the images table for locked images
SELECT image_group_id FROM images WHERE imagestatus = '2';
# Identify which one needs to be unlocked
UPDATE images SET imagestatus = '1' WHERE imagestatus = '2';

This will make the virtual disk inoperative, but it will allow either delete or commits the snapshots created previously.

In the RHV GUI, navigate to “Virtual Machines”, select the faulty one, select “Snapshots”, and perform a preview of a stable snapshot. Afterwards, “Commit” the action.

The plugin will automatically clear this situation with the following backup. This manual intervention is only necessary if there is no option of running a subsequent backup.

DSM <host> command ExtendImageTicketVDS Failed

The full message provided would be similar to

DSM <host> command ExtendImageTicketVDS failed: Image daemon request
failed: u'
status=404, code=404,
title=Not Found,
explanation=The resource could not be found.,
detail=No such ticket: a035901e-c3d6-4033-b0fa-8e127d422ccf,
reason=Not Found'

This error can be solved manually in the same way as the previous problem, and the plugin will also clear this situation with the following backup.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException

Here, the full error message would look like

javax.net.ssl.SSLHandshakeException:sun.security.validator.ValidatorException:
  PKIX path building failed:
    sun.security.provider.certpath.SunCertPathBuilderException:
      unable to find valid certification path to requested target
at
sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at
sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
at
sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
...

This error is related to the certificates used by RHV. A possible solution is shown here: stackoverflow.com/…/pkix-path-building-failed.

org.apache.http.NoHttpResponseException: <RHVM>:443 failed to respond

The full error is

org.apache.http.NoHttpResponseException: rhv.example.com:443 failed to respond
org.ovirt.engine.sdk4.Error
org.ovirt.engine.sdk4.Error: Failed to send request at
org.ovirt.engine.sdk4.internal.HttpConnection.send(HttpConnection.java:213)
at org.ovirt.engine.sdk4.internal.services.
  EventsServiceImpl\$AddRequestImpl.send(EventsServiceImpl.java:83)
...

This error happens when no certificate is present. The solution can be found in the SSLHandshakeException problem resolution.

HTTP Response Code is 409

The full error is:

org.ovirt.engine.sdk4.Error: HTTP response code is "409".
  HTTP response message is "Conflict".
at org.ovirt.engine.sdk4.internal.services.ServiceImpl.
  throwError(ServiceImpl.java:113)
at org.ovirt.engine.sdk4.internal.services.ServiceImpl.
  checkFault(ServiceImpl.java:40)

This can happen if a snapshot could not correctly get deleted.

As explained in section ch:fixlockedsnapshot, this can be manually resolved by commiting a snapshot preview in the RHV GUI.

The plugin will solve this situation manually with the following backup. A manual intervention is only necessary if there is no option of running that following backup.

HTTP response is 401. HTTP response message is “Unauthorized”

This error happens when RHVM credentials are incorrect.

HTTP 503 Code at Disk Download

This happens if the connection between RHV Manager and RHV virtualization host daemons is not working correctly. Usually this indicates a firewalling issue.

The solution is to ensure that the firewall service (firewalld is active and correctly configured. For RHV operations, two ports–54321 for vdsm and 54322 for ovirt-imageio– need to be accessible. To ensure this, execute the follow commands on RHV virtualization hosts:

systemctl start firewalld
firewall-cmd --permanent --add-port=54321/tcp
firewall-cmd --permanent --add-port=54322/tcp

To check if this command has been effective:

ssystemctl restart firewalld
firewall-cmd --list-ports

The ports 54321/tcp and 54322/tcp must appear in the list of open ports.

In a production environment, it is reasonable to limit access to the daemons affected only from trusted hosts or network segments. Please refer to RHEL documentation of the firewall-cmd program, which is available using man firewall-cmd.

VDSM hostTest command GetCapabilitiesVDS failed: Vds timeout occurred

This error can happen if a host is not registered. It should not impact operations on other hosts.

To fix this situation it is usually necessary to confirm the host has been rebooted or shutdown and put it into maintenance mode after.

Can’t create any snapshot in a specific vm, but the vm can run

The origin of this error is unknown, this case is reported here. The simple explanation is that an attempted snapshot removal (which we have no logs of), corrupted the chain.

Querying RHV

Assuming the Java environment is properly configured, the general approach is as follows:

java -jar /opt/bacula/lib/bacula-rhv-plugin.jar --server=<RHVM-DNS> \
--truststore_file=<path-to-truststore> --truststore_password=<password> \
--auth=<auth> --profile=<profile> \
--user=<user-name> --password=<password> \
--operation=list --path=<RHVM structure>

Results will be returned in JSON format.

The required options indicated above are directly representing the plugin options as described in table pluginoptions.

The following list shows all the available RHVM structure, by default “/vms”:

It is also possible to list RHV information using using the .ls-command of bconsole. Below, some examples are provided.

The meaning of the required and optional parameters are the same as shown in table pluginoptions and above for the query functionality.

Note also that bconsole commands need to be entered without any line breaks.

Verifying compatible backups of VMs than they starts with vm1*:

*.ls client=RHV-fd plugin="rhv:server=myrhv.com \
insecure=true password=rhvPass123" path=/vms/vm1*

#Answer:
# -rw-r----- 0 root root 0 1970-01-01 01:00:00 4f5f9abd-f407-4da6-ba39-272aed709c80->vm1\
                        [vm] [BACKUP_CLONE,BACKUP_SNAP]
# -rw-r----- 0 root root 0 1970-01-01 01:00:00 a8bf6b18-a116-4067-b884-172777fec446->vm11\
                        [vm] [BACKUP_SNAP]

Listing all disks attached to “vm1”:

.ls client=RHV-fd plugin="rhv:server=myrhv.com \
insecure=true user=bacula password=alsogood" path=/vms/vm1/disks/

# Answer:
#-rw-r----- 0 root root 0 1970-01-01 01:00:00 c43a1865-13ba-483e-b38b-6b929d6b4653->\
                        Preassigned_Disk1 [disk]
#-rw-r----- 0 root root 1 1970-01-01 01:00:00 b0b349ea-72d3-405f-9429-c1cb009f30ce->\
                        Preassigned_Disk2 [disk]
#-rw-r----- 0 root root 2 1970-01-01 01:00:00 3f993fb1-3157-4916-aad1-2f323a3cdd4f->\
                        Preassigned_Disk3 [disk]
#-rw-r----- 0 root root 3 1970-01-01 01:00:00 e9ac9bd8-920f-4eb8-b4d0-b405d675b16f->\
                        Preassigned_Disk4 [disk]

To list all datacenters:

.ls client=RHV-fd plugin="rhv:server=RHV.example.com \
insecure=true user=bacula password=alsogood" path=/datacenters/

# Answers:
# -rw-r----- 0 root root 0 1970-01-01 01:00:00 3e54cf31-7a9a-46ad-8a8d-7843b6d7a145->\
                        dcTest [datacenter]
# -rw-r----- 0 root root 1 1970-01-01 01:00:00 4cdcc978-73af-11e8-835d-b827eb7e5e47->\
                        Default [datacenter]

To list all options in datacenter named “dcTest”:

.ls client=RHV-fd plugin="rhv:server=RHV.example.com \
insecure=true user=bacula password=alsogood" path=/datacenters/dcTest/

# Answer:
# -rw-r----- 0 root root 0 1970-01-01 01:00:00 clusters
# -rw-r----- 0 root root 1 1970-01-01 01:00:00 disks
# -rw-r----- 0 root root 2 1970-01-01 01:00:00 hosts
# -rw-r----- 0 root root 3 1970-01-01 01:00:00 storages
# -rw-r----- 0 root root 4 1970-01-01 01:00:00 templates
# -rw-r----- 0 root root 5 1970-01-01 01:00:00 vms

Check if the Cluster “clusterTest” is compatible with the Storage Domain “iSCSIStorage”

.ls client=RHV-fd plugin="rhv:server=RHV.example.com \
user=bacula password=alsogood" insecure=true \
path=/api/compatibility/clone_storage/clusterTest[Name_cluster]/iSCSIStorage[Name_storage]
# Answer:
# -rw-r----- 0 root root 0 1970-01-01 01:00:00 true

Get version of RHVM:

.ls client=RHV-fd plugin="rhv:server=RHV.example.com \
user=bacula password=alsogood insecure=true" path=/api/version
# Answer:
# -rw-r----- 0 root root 0 1970-01-01 01:00:00 4.2.0