Note

You can download this article as a PDF

Microsoft 365 Plugin

Overview

This whitepaper presents how to protect the most relevant elements of Microsoft 365 services using Bacula Enterprise.

Features

The Bacula Enterprise Microsoft 365 Plugin is a very easy to deploy and configure plugin supporting the following M365 services:

  • OneDrive

  • Emails

  • Mailbox settings

  • Sharepoint Online

  • Calendars

  • Contacts

  • OneNote

  • Tasks

  • Teams

  • Chats

It is shipped with advanced parallelization, resiliency, and flexibility features in addition to covering most of the possible M365 use case scenarios. A full feature list is presented below:

  • Common features

    • Microsoft Graph API based backups

    • Multi-service backup in the same task

    • Multi-service parallelization capabilities

    • Multi-thread single service processes

    • Automatic parallelization of fetching processes

    • Generation of user-friendly report for restore operations

    • Network resiliency mechanisms

    • Latest Microsoft Authentication mechanisms

    • Discovery/List/Query capabilities

    • Restore objects to Microsoft 365

      • To original entity

      • To any other entity

      • To a different tenant (cross-tenant restore)

    • Restore any object to filesystem

    • Owner data protection feature:

      • Notify data owner about restore actions of his data

      • Do not proceed further until he enters his M365 credentials

    • Incremental & Differential backups

      • Advanced delta function for improved performance (for selected services)

  • Backup and Restore of Exchange Online Mailboxes

    • Mailfolder, message and attachment granularity for restore

    • Email addresses and mailfolders selection capabilities for backup

    • Mailbox settings protection

    • Folder rules protection

    • Restore objects to Microsoft 365 or to any file-system

    • Restore MIME messages to any filesystem

    • Incremental & Differential backup

    • Support for user mailboxes and shared mailboxes

    • User categories protection

    • Fully indexed information into Bacula Catalog

    • Advanced search capabilities for restore operations

    • Ability to filter input data by date

    • Ability to exclude message fields from backup or from index

    • Exclude private or spam messages through powerful filtering capability

    • Export to PST format

  • Backup and Restore of OneDrive for Business & Sharepoint Document libraries

    • Backup and Restore of User drives

    • Backup and Restore of Group drives

    • Backup and Restore of Sharepoint document libraries

      • Include/Exclude system libraries

      • Include/Exclude hidden libraries

    • Backup main entity drive unit, but also any other unit

    • Advanced selection capabilities

      • Target entities (Users/Groups/Sites)

        • List Include/List Exclude/Regex include/Regex Exclude…

      • Folder selection capabilities for backup

        • List Include/List Exclude

      • File selection capabilities for backup

        • Regex include/Regex Exclude…

      • Drive unit selection capabilities

    • Folder and file granularity for restore

    • Computed hash check at backup and restore time

    • Backup and restore of permissions shares

    • Backup and restore of shared elements to each entity

    • Backup and restore of OneDrive file versions

  • Backup and Restore of Sharepoint Sites

    • Backup of Site Objects (MS Graph Object)

      • Backup Site Sharing permissions

    • Backup of full Site Templates (PnP Powershell Provisioning):

      • Site metadata

      • Lists metadata

      • ListItems metadata

      • WebPages metadata

      • DocumentLibraries metadata

      • Support for Sub-Sites

    • Restore backed up sites as new sites using Site Template (PnP Powershell Provisioning)

    • Backup/Restore Lists Objects (MS Graph Object)

    • Backup/Restore ListItems (MS Graph Object)

    • Backup/Restore Document Libraries Drive Items

  • Backup and Restore of Contacts/People

    • Backup/restore Contacts

    • Backup/restore groups of contacts

    • Backup Organizational contacts (Read-only)

  • Backup and Restore of Tasks

    • Backup/restore User todo tasks

    • Backup/restore Teams Planner tasks

  • Backup and Restore of Calendars

    • Backup/Restore user calendars

    • Backup/Restore user groups of calendars

    • Backup/Restore groups calendar

      • Calendar permissions

    • Backup/Restore Events

      • Support for Attachments

        • File Attachments

        • Reference Attachments

        • Item Attachments (including backup of MIME objects)

  • Backup and Restore of Onenote

    • Backup/Restore of Notebooks

      • User notebooks

      • Site notebooks

      • Group notebooks

    • Backup/Restore of SectionGroups/Sections

    • Backup/Restore of Pages

      • Support for Page resources: Images and files

  • Backup and Restore of Teams

    • Backup/Restore of Team

      • Public and Private Teams

      • Team entity

      • Team members and roles

      • Team installed apps

    • Backup/Restore of Team Channels

      • Public and Private Channels

      • Channel entity

      • Channel members

      • Channel tabs

      • Channel messages

      • Channel messages hosted contents

  • Backup and Restore of Chats

    • Backup/Restore of Chat

      • Chat entity

      • Chat installed apps

      • Chat tabs

      • Chat messages

      • Chat messages hosted contents

Requirements

Microsoft 365 Personal, Family, Microsoft Home & Student subscriptions are not supported for backup/restore purposes.

It is necessary to have full administrative access to the target Tenant to protect in order to provide the required permissions to the Azure Application linked to this Bacula Enterprise Microsoft 365 Plugin.

Currently the plugin must be installed on a Linux based OS (RH, Debian, Ubuntu, SLES ..) where a Bacula Enterprise File Daemon is installed. Bacula Systems will address support for running this plugin on a Windows platform in a future version.

The OS where the File Daemon is installed must have installed Java version 8 or above.

If the Sharepoint module is going to be used, the OS where File Daemon is installed must also have the following packages installed:

  • PowerShell v7.2.1 or above

  • PnP Powershell v1.9.0 or above

Memory and computation requirements completely depend on the usage of this plugin (parallelization, environment size, etc). However, it is expected to have a minimum of 4GB RAM in the server where the File Daemon is running. By default, every job could end up using up to 512Mb of RAM in demanding scenarios (usually it will be less). However, there can be particular situations where this could be higher. This memory limit can be adjusted internally (see Out of Memory). Refer to the Scope section below for any service specific requirements.

Why Protect Microsoft 365?

This is a common question that arises frequently among IT and Backup professionals, so it is important to have a clear picture of it. It is true that Microsoft offers some services intended to prevent data loss:

  • As with any cloud data, Microsoft 365 data is geo-replicated using Azure cloud to several destinations automatically and transparently. Therefore, complete data loss because of hardware failures are very unlikely to happen.

  • Data Loss Protection service: Policy based services capable of detecting filtered content and act upon it encrypting it or modifying it in order to protect it (remove headers, etc). This is not a backup tool, is a service to prevent undesired actions to the content stored in Microsoft 365 (for example sharing confidential information with the wrong people).

  • Retention policies of Microsoft 365: Microsoft retains a maximum of 30 days of deleted information from active subscriptions. Therefore it is possible to recover accidental deleted items inside that period. For more information:

There are no other protection mechanisms for data protection. Below is a listing of challenges not covered:

  • No Ransomware protection: If data suffers an attack and becomes encrypted, data is lost.

  • No malicious attacker protection: If data is deleted permanently, data is lost.

  • No real point-in-time recovery for Exchange 365, and recoveries of partially deleted files are limited to 30 days.

  • Point in time recovery for OneDrive/Sharepoint limited to 30 days from deletion.

  • It is not possible to align data protection of Microsoft 365 services to general retention periods or policies longer than 30 days.

  • No automated way to extract any data from the cloud to save it in external places (this could lead to eventual compliance problems)

Scope

Bacula Enterprise Microsoft 365 Plugin is applicable on environments using any enterprise plan where target services of this plugin are included:

  • Office 365 platform

  • Exchange Online

  • OneDrive for Business

  • Sharepoint Online

The lack of any of the services in your subscription could lead to authentication problems (See troubleshooting section App Registration Error).

For more details about Microsoft 365 plans, visit these links:

This paper presents solutions for Bacula Enterprise version 12.8 and later, and is not applicable to prior versions.

Note

Important considerations

Before using this plugin, please carefully read the elements discussed in this section.

Onedrive Recycle Bin

Onedrive Recycle Bin cannot be protected with this plugin. This is a Microsoft limitation coming from their exposed REST APIs, where it is not possible to access Recycle Bin information. If future versions of Microsoft REST APIs officially include this function, Bacula Systems will include it as a new feature of this plugin.

Empty files

In general, empty files (files with 0 byte contents) are simply not backed up by Microsoft 365 plugin. In particular, email attachments or onedrive files will show a message in the joblog to inform about empty files detected and so not processed.

Backup of Attachments and Files

In general, this plugin backups two types of information:

  • Objects

  • Files

Objects are elements representing some entity in Microsoft 365 such as a calendar event, a contact, an email, a team, etc.

Files are attachments, hosted contents or OneDrive files.

While objects are directly streamed from memory to the backup engine, files need to be downloaded to the FD host before being sent. This is done in order to make some metadata checks and to improve overall performance, as this way operations can be parallelized. Every file is removed just after being completely downloaded and sent to the backup engine.

The path used for this purpose is established by the ‘path’ plugin variable, that usually is set up in the m365_backend script with the value: /opt/bacula/working

Inside the path variable, a ‘spool’ directory will be created and used for those temporary download processes.

Therefore, it is necessary to have at least enough disk space available for the size of the largest file in the backup session. If you are using concurrency between jobs or through the same job (by default this is the case through the concurrent_threads=5 parameter), you would need at least that size for the largest file multiplied by the number of operations in parallel you run.

Accurate Mode and Virtual Full Backups

Accurate mode and Virtual Full backups are not supported. These features will be addressed in future versions of this plugin.

Details on the Sharepoint backup/restore

The M365 plugin overcomes the current limitations of the MS Graph API on backup/restore of SharePoint sites by using the Powershell PnP opensource project:

This project has been designed to abstract many complexities of the Sharepoint Online APIs and provides all sorts of automation processes. Among them, site template provisioning, which is the main part of the method Bacula Enterprise Microsoft 365 Plugin uses in order to provide backup/restore capabilities for Sharepoint Online.

Site Template Provisioning

Microsoft can modify at any time (and they do it frequently) the properties, structure or limitations of any element belonging to a site kind (team site, communication site, project site, etc) whicn can cause the restore provision process to fail. It is even common to see native structures of Sharepoint Online trigger errors with particular items/values and this error comes from the Sharepoint Online API.

As a result, is not possible to guarantee that a given template is going to work in a provision process. Many times, the template might need to be partially modified before being applied (removing some particular item, removing some particular column..). That part of the process is not possible to predict or automate, so we strongly recommend frequent testing of site restore processes to detect any need of manual restore processes before a real restore is required.

Particular List/ListItems Restores

It is possible to select and restore a particular list or listItems, without restoring an entire Sharepoint Site, and pointing the operation to an existing Site. This is done using the Graph API, but as we have previously exposed, this API has many limitations with the Sharepoint API. One of the limitations is that lists or list items with advanced elements (location, images, etc) are not supported. Therefore this feature should only be used for user created lists implying simple elements like texts, checks, dates, etc.

Site Backups In Parallel

Backing up sites in parallel from the same FD is not supported.

Email Limitations

The following items associated to Outlook elements (Exchange Online service) are not supported due to the fact they are currently not accessible using the APIs this plugin relies on (MS Graph API and PnP Powershell):

  • Sharing permissions associated to MailFolders

  • Public Folders

Onenote Limitations

The following items associated to Onenote elements are not supported due to the fact they are currently not accessible using the APIs this plugin relies on (MS Graph API and PnP Powershell):

  • Sharing permissions associated to Notebooks

  • Shared elements from one entity to another one. To protect them it is necessary to make it always from the source entity.

Notebook 5000 files limitation

Onenote notebooks can only store up to 5000 different items into their associated OneDrive Libraries, this a M365 limitation.

When they surpass that limit they become unaccessible, not only for backup but also for accessing them through any other Microsoft native interface.

When a job finds this kind of notebooks, an error message will be generated and the notebook won’t be included in the backup.

You can get more information about this limit and how to solve it in the following Microsoft documentation link:

External Personal Calendars

External calendars (as Google Calendar) connected to Microsoft accounts are not supported. They are external elements not exposed through MS APIs, so M365 Plugin cannot protect that information.

Calendar Events icons

Little icons that appear on the left of a calendar event title (usually auto-generated by the Outlook service) cannot be backed up, as they are not exposed through the MS Graph API the plugin is relying on.

Chats types

For the chats module, only types ‘group’ or ‘one_on_one’ can be backed up. Other types like: ‘meeting’ or ‘unknownFutureValue’ will be ignored.

MS Cloud APIs General Disclaimer

MS Cloud APIs are Microsoft property and they can change or evolve at any time. In particular, the Graph API is actively developed, containing new features every week, even if the version number of the service (1.0) is not changed as a result of any of those additions:

This situation is significantly different from traditional on-premise software, where each update is clearly numbered and controlled for a given server, so applications consuming that software, can clearly state what is offered and what are the target supported versions.

Microsoft is committed to trying to not break any existing functionality that could affect external applications. However, this situation can happen and therefore, cause some ocasional problems with this plugin. Bacula Systems controls this with an advanced automatic monitoring system which is always checking the correct behavior of existing features, and will react quickly to that hypothetical event, but please be aware of the nature and implications of this kind of cloud technologies.

Architecture

Bacula Enterprise Microsoft 365 Plugin is using the Microsoft Graph API to perform almost all of its operations. Therefore, the plugin is working at the maximum granularity that the service provides.

Microsoft Graph

Microsoft Graph

All the information is obtained using secure and encrypted HTTPS queries to Microsoft 365 from the File Daemon where the plugin is installed. All the requests are performed over the following endpoints:

To get more information about Graph API, visit: https://learn.microsoft.com/en-us/graph/overview

The plugin will contact an Azure registered app named bacula-m365-plugin and will use it as a bridge to download the required data or objects during the time of a backup and send them to the Storage Daemon. Conversely, the plugin will receive them from an SD and perform uploads as needed during a restore.

The implementation is done through a Java Daemon, therefore Java is a requirement in the FD host. For more information about the bacula-m365-plugin, please, consult Authorization section.

Below is a simplified vision of the architecture of this plugin inside a generic Bacula Enterprise deployment:

M365 Plugin Architecture

M365 Plugin Architecture

Listed below is the information that can be protected using this plugin:

  • Email

    • Common MailFolders (Inbox, Deleted Items, Drafts..)

    • User MailFolders & SubFolders

    • Messages & EventMessages

    • Attachments (ItemAttachments, FileAttachments and ReferenceAttachments)

    • Mailbox settings

    • Folder Rules

  • OneDrive

    • Onedrive for Business of Users, for each drive unit

      • Folders

      • Files

      • File Versions

    • Group libraries, for each drive unit

      • Folders

      • Files

      • File Versions

    • Sharepoint site libraries, for each drive unit

      • Folders

      • Files

      • File Versions

    • Shared permissions (direct access, share links, expiration times..)

    • SharedWithMe Objects

  • Sharepoint

    • Pnp Site template

      • Site metadata

      • Lists metadata

      • ListItems metadata

      • WebPages metadata

    • Site Object

      • Site sharing permissions

    • Lists Objects

    • ListItem Objects

  • Contacts/People

    • Contact object

    • Name of Groups of contacts

    • Organizational contact object

  • Tasks

    • User Todo lists

    • User Todo tasks

  • Calendars

    • Calendar objects

    • Calendar group objects

      • Calendar permissions

    • Events objects

    • Attachments (ItemAttachments, FileAttachments and ReferenceAttachments)

      • MIME objects where possible

  • Notebooks

    • Notebook objects

    • Section objects

    • SectionGroup objects

    • Pages

      • Page contents (Html formatted) - Page resources

        • Page image files

        • Page object files (any other file apart from images)

  • Teams

    • Team objects

    • Team settings

    • Team members and associated roles

    • Team installed apps

    • Channel objects

      • Channel tabs

      • Channel chat messages

        • Chat messsages hosted contents

  • Chat

    • Chat objects

    • Chat installed apps

    • Chat tabs

    • Channel chat messages

      • Chat messsages hosted contents

All the information of each object is stored in JSON format (except for Pnp site template, which is stored in XML), preserving all their original values. When the plugin works with objects containing additional data (MIME files for messages, data for attachments and files of OneDrive, etc), that data is also backed up.

Services and Features

In this section we will dig into how this plugin behaves for each particular service, describing special features and and behaviors that require an extended description.

Special features

In the following section, special features and behaviors are detailed.

Installation

The Bacula File Daemon and the Microsoft 365 Plugin need to be installed on the host that is going to connect to the cloud based services. The plugin is implemented over a Java layer, therefore it can be deployed on the platform better suited for your needs among any of the officially supported platforms of Bacula Enterprise (RHEL, SLES, Debian, Ubuntu, Windows, etc).

Please, note that you may want to deploy your File Daemon and the plugin on a virtual machine directly deployed in Azure Cloud in order to reduce the latency between it and the Microsoft Graph API and experience modest performance gains. However, this option is only recommended in case of having a very stable connection between the File Daemon and the Storage Daemon, which means a special, dedicated connection with Azure or when the Storage Daemon is deployed in the cloud as well. This is not usually the case - thus the data needs to traverse the Internet with standard and shared connections from the File Daemon to the Storage Daemon. Disconnections while transmitting data between these two daemons may make jobs fail and cause large timeouts that are difficult to manage and stabilize. The best strategy strategy for this kind of scenarios is to deploy the File Daemon and the Plugin to the same host as the destination Storage Daemon is installed. This way, disconnections between the two daemons will not happen, while disconnections between the FD and M365 will be transparently recovered (when possible), so jobs will finish successfully.

The system must have Java >= 8 installed (openjdk-8-jre for example) and the Java executable should be available in the system PATH.

The Sharepoint module depends on the Powershell and the PnP Powershell modules. Therefore, they also need to be installed before installing the Bacula packages (see section PnP.Powershell below).

PnP.Powershell

Install PowerShell

In order to install PowerShell it is necessary to follow the instructions for the particular OS involved which may be found in the github site of the project:

For example, if using Debian, these are the instructions:

The procedure is shown below, some dependencies are installed, a repository is added, and then the apt package manager is used to install the tool:

Install PowerShell
# Install system components
sudo apt-get update
sudo apt-get install -y curl gnupg apt-transport-https

# Import the public repository GPG keys
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -

# Register the Microsoft Product feed
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch
main" > /etc/apt/sources.list.d/microsoft.list'

# Update the list of products
sudo apt-get update

# Install PowerShell
sudo apt-get install -y powershell

# Start PowerShell
pwsh

Install PnP.Powershell

In order to install the PnP.Powershell module, once Powershell is already installed, we simply need to run the command:

  • Install-Module -Name “PnP.PowerShell”

Inside a Powershell session. Below we provide an example:

Install PnP.PowerShell
yourworkstation:~$ pwsh
PowerShell 7.2.0
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS /home/john> Install-Module -Name "PnP.PowerShell"

Bacula Packages

We are taking Debian Buster as the example base system to proceed with the installation of the Bacula Enterprise Microsoft 365 Plugin. In this system, the installation is most easily done by adding the repository file suitable for the existing subscription and the Debian version utilized. An example would be /etc/apt/sources.list.d/bacula.list with the following content:

APT
# Bacula Enterprise
deb https://www.baculasystems.com/dl/@customer-string@/debs/bin/@version@/buster-64/ buster main
deb https://www.baculasystems.com/dl/@customer-string@/debs/m365/@version@/buster-64/ buster m365

After that, a run of apt update is needed:

APT install
apt update

Then, the plugin may be installed using:

APT install
apt install bacula-enterprise-m365-plugin

The plugin has two different packages implied that should be installed automatically with the command shown:

  • bacula-enterprise-m365-plugin

  • bacula-enterprise-m365-plugin-libs

Alternately, manual installation of the packages may be done after downloading the poackages from your Bacula Systems provided download area, and then using the package manager to install. An example:

APT install
dpkg -i bacula-enterprise-*

The package will install the following elements:

  • Jar libraries in /opt/bacula/lib (such as bacula-m365-plugin-x.x.x.jar and bacula-m365-plugin-libs-x.x.x.jar). Please note that the version of those jar archives is not aligned with the version of the package. However, that version will be shown in the joblog in a message like ‘Jar version:X.X.X’.

Note

Version in Jar Name

Version is included in the name of .jar files from Bacula Enterprise version 14.0.4. Before that, libraries were composed by: bacula-m365-plugin.jar, bacula-meta-plugin-1.0.0.jar and bacula-m365-plugin-libs-1.0.0.jar

  • Plugin connection file (m365-fd.so) in the plugins directory (usually /opt/bacula/plugins)

  • Backend file (m365_backend) that invokes the jar files in /opt/bacula/bin. This backend file searches for the most recent bacula-m365-plugin-x.x.x.jar file in order to launch it, even thought usually we should have only one file.

  • A collection of powershell files used in the Sharepoint module in /opt/bacula/bin.

Configuration

Authorization

The first step in order to use the Bacula Enterprise Microsoft 365 Plugin is to authorize it to handle data of the target tenant to backup.

There are two possible strategies in order to allow the communication between the Bacula Enterprise Plugin and your tenant:

  • Method A (DEPRECATED): Common app model
    • Register the pre-existing Bacula Systems bacula-m365-plugin Azure AD app into your tenant.

    • The communication will happen through this multi-tenant application.

    • Application Id and associated secrets are internal to the plugin.

    • Microsoft Graph limits associated to an application are common for everyone using this application (multi-customer).

    • For future new permissions you just need to click on ‘Grant permissions’ from Azure AD enterprise apps section

  • Method B (RECOMMENDED): Standalone app model
    • Register the pre-existing Bacula Systems bacula-m365-registratror Azure AD app into your tenant. Then add a standalone application in your tenant.

    • Then add your own standalone application in your tenant calling the appropriate automatic command from bconsole

    • A bacula-m365-plugin-standalone Azure AD application will be created specifically for your tenant. The communication will happen through it.

    • Application Id and associated secrets need to be correctly set and they can be managed by you.

    • Microsoft Graph limits associated to an application are specific to your standalone application

    • For future new permissions you need to re-run the bconsole add-app command (which uses bacula-m365-registrator) or do it yourself manually

To walk through the process described in Method B with the help of a video, click on the image below:

M365 Video

Note

Authentication Method B is only available from Bacula Enterprise 12.8.2

The first method is simpler and faster to setup, however it is only advised for testing purposes. The second method needs a few extra variables to manage and it is recommended for medium or large environments. It is more secure and it can offer better performance.

Backup and restore operations will be using in general the ‘Application permissions’ model, where the application has enough privileges to perform all the operations without impersonating any user. However, some specific modules need to employ ‘Delegated permissions’. To know more about them, please go to Delegated permissions

Note

Starting from Bacula Enterprise version 14.0 you can also perform these authorization tasks directly using BWeb, to see more details, please go to section BWeb Management Console

The sections below will show how to use both methods. For any of them, the first step is to find your Tenant ID:

How to find the Tenant ID

In order to find the Tenant ID you only need to login to the Azure portal (portal.azure.com) and take a look at the ‘Overview’ page of the service Azure Active Directory. Just click in the service as the below image shows:

Azure Active Directory

Once there, you will find the Tenant ID in the box highlighted in the below image:

Tenant ID

Authorization Method A: Common app model with ‘bacula-m365-plugin’ (DEPRECATED)

Bacula Systems has a registered application in Azure AD named bacula-m365-plugin. This section will show how to authorize it to perform backup and restore operations over your target tenant. All required steps to complete this authorization process are presented below.

Please, use this connection method for testing purposes as it is deprecated. For production systems, please go ahead with Authorization method B: Authorization Method B: Standalone app model with ‘bacula-m365-registrator’ (RECOMMENDED).

Most of the procedures described in this section must be done by a tenant administrator. A tenant administrator is a user who has been assigned the Azure AD role Global administrator.

Below we show an schema of how this authorization method works:

Authorization method A
A-1. Authorize bacula-m365-plugin

A tenant administrator must run the following query in a web browser, replacing {tenantId} with the value obtained in the first step:

Authorization URL
https://login.microsoftonline.com/{tenantId}/adminconsent?client_id=14a0b71a-d9ca-496c-b4c0-76a3cbb5dc33&state=12345&redirect_uri=https://www.baculasystems.com/m365-plugin-auth/common

You can get the exact same URL from the plugin command line itself, once you have installed it in a client named {your_client_name} using the following special Query command.

Authorization URL
*.query plugin="m365:" client={your_client_name} parameter=register:{tenantId}

Here is an execution example where the instructions are displayed and you need to open the provided URI:

Query command for tenant URL
*.query plugin="m365:" client=127.0.0.1-fd parameter=register:57uia43-d107-17a2-a2g2-aa53c10tdahc
console=---- M365 PLUGIN REGISTER COMMAND ----
console=-------------------------------------
console=-------------------------------------
info=Open the following URI in a browser with your tenant admin credentials
console=-------------------------------------
console=-------------------------------------
uri=https://login.microsoftonline.com/57uia43-d107-17a2-a2g2-aa53c10tdahc/adminconsent?client_id=14a0b71a-d9ca-496c-b4c0-76a3cbb5dc33&state=12345&redirect_uri=https://www.baculasystems.com/m365-plugin-auth/common
console=-------------------------------------
console=-------------------------------------
info=Once you have accepted the provided permissions to the bacula-m365-plugin app..
info=You can get your ObjectId using the command below
console=-------------------------------------
console=-------------------------------------
command=.query plugin="m365: tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc" client={your_client_name} parameter=objectid
console=-------------------------------------

When opening the URI, Microsoft 365 will ask for credentials. You need to authenticate as a tenant admin:

Login

…and once they are correctly provided, the app will ask for all the required permissions to backup and restore all of the supported elements:

Confirm Permissions

The application needs all of the listed permissions to be able to work. If any of them is missing, backup or restore operationms will fail.

The image and list shown here are illustrative and some additional permissions may be needed as the plugin evolves. However, we provide also here a text list of current permissions needed:

  • Graph

    • Read and write user chat messages → Backup/Restore of Team Channels and Chats

    • Read and write user and shared calendars → Backup/Restore of group Calendars

    • Read and write all groups → Creation of Teams

    • Send user chat messages → Restore of Chats

    • Create, read, update and delete user’s tasks and task lists → Backup/Restore of Tasks

    • Read and write tags in Teams → For future support of Team tags

    • Read organization information → Find tenant name

    • Read and write files in all sites collections → Backup/Restore OneDrive and Sharepoint

    • Read and write all chat messages → Backup/Restore of Team Channels

    • Read all users full profiles → Find users to Backup/Restore

    • Read all channel messages → Backup of Team Channels

    • Read and write all user mailbox settings → Outlook categories and more future mailbox information

    • Read and write contacts in all mailboxes → Backup/Restore contacts

    • Read directory data → Security checks of objectid, list groups, etc

    • Read and write calendars in all mailboxes → Backup/Restore User Calendars

    • Read organizational contacts → Backup organizational contacts

    • Read and write tabs in Microsoft Teams → Backup of Microsoft Teams tabs

    • Read and write all OneNote notebooks → Backup/Restore OneNote service*

    • Have full control of all site collections → Backup/Restore Sharepoint

    • Add and remove members from all Teams → Backup/Restore of Microsoft Teams members

    • Create chat and channel messages with anyone’s identity and with any timestamp

    • Manage Teams apps for all chats → Backup of Microsoft Teams apps

    • Manage Teams apps for all teams → Backup of Chats apps

    • Read and write the names, descriptions, and settings of all channels of all Teams → Backup of Microsoft Teams Channels

    • Add and remove members from all channels → Backup of Microsoft Teams Channels

    • Read and write managed metadata → Security checks of objectid, list groups, etc

    • Sign in and read user profile → Ability to connect to the tennant

    • etc.

  • Sharepoint Online

    • Full management of all site collections → Backup/Restore Sharepoint (PnP)

    • Full management of Term Store → Backup/Restore Sharepoint (PnP)

Once it is confirmed, the browser will use the ‘redirect_uri’, which is a page on baculasystems.com that will confirm the result of the registration process.

Registration confirmation

The generated URI contains the parameter admin_consent=True if the action was successful, and you will see a confirmation message in that case. Otherwise, the operation may have not been successful for some reason and you will see an error message.

Once that action is done, the tenant where our app now has permissions will show the plugin in the Enterprise Apps section:

Plugin app in a new tenant

Clicking on the app, the tenant admin can always see the permissions assigned:

Plugin app permissions in new tenant
A-2. Grab Object ID

The plugin needs a final parameter that is unique to each tenant and the plugin app. This is ObjectID, and may be obtained from the Overview app page, once step 2 has been completed:

ObjectId of app inside a tenant

The plugin can also obtain it from the command line using another special Query command. You can see that this exact command is also suggested in the command that shows the register URL:

Authorization URL
*.query plugin="m365: tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc" client={your_client_name} parameter=objectid

Here is an execution example:

Authorization URL
*.query plugin="m365: tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc" client=127.0.0.1-fd parameter=objectid
console=---- M365 PLUGIN OBJECTID COMMAND ----
console=-------------------------------------
objectid=56ddf1h9-eb5d-42nf-bac7-7b019fd284g5
console=-------------------------------------

Check permissions

It is possible to check if the service principal of the target tenant (this is shown under ‘enterprise applications’ in Azure AD) has all the needed permissions to perform plugin operations.

You just need to run the following command:

Check permissions
.query client={your-client} plugin="m365: tenant={yourTenantId} objectid={yourobjectid}" parameter=permissions

The ouput will report if everything is ok around permissions, or if there is any missing one:

Check permissions: ok output
.query client={your-client} plugin="m365: tenant={yourTenantId} objectid={yourobjectid}" parameter=permissions
info=All permissions are correctly set
Check permissions: some is missing
.query client={your-client} plugin="m365: tenant={yourTenantId} objectid={yourobjectid}" parameter=permissions
error=Delegated permission Chat.ReadWrite was not found in your Azure service principal 8cee7605-0d5e-42b6-b269-c39f3411b5b9
error=Please review your app permissions

In case you are using the standalone application, you will need to run the command with appid and secret too:

Check permissions
.query client={your-client} plugin="m365: tenant={yourTenantId} objectid={yourobjectid} appid={yourappid} secret={yoursecret}" parameter=permissions

Please, run this command at any time you have doubts about the correct app permissions.

Please note though, that the command is unable to check if you have activated or not ‘Protected APIs’ for Teams and Chats. It will check the presence of those permissions, but it is up to Microsoft to enable those particular ones and they do not show (at the time of writing) any flag indicating if they are enabled or not.

Migration from Authenticaton Method A to Authentication Method B

Perhaps you started to use this plugin using Method A and now you want to switch to Method B in order to enjoy its advantadges. This section will show the steps needed in order to complete such migration.

Migration - Step 1: Remove bacula-m365-plugin from your tenant

The first step is to remove the common app ‘bacula-m365-plugin’ from your tenant. In order to do it, you need to go to your Azure portal and open the ‘Azure Active Directory’ application as usual.

From there, please go to ‘Enterprise applications’ and locate bacula-m365-plugin app. You need to select the application and enter into ‘Properties’ section. From there, you need to click on the ‘Delete’ button:

Delete common app
Migration - Step 2: Perform Authentication Method B

Once you have removed the common application, you simply need to follow all the steps explained in this documment for authentication method B. In case you are using BWeb and you have already registered you tenant, it is recommended to remove it first, and then add it also using method B. Otherwise, for manual configuration, simply follow the steps mentioned in this document: Authorization Method B: Standalone app model with ‘bacula-m365-registrator’ (RECOMMENDED).

Migration - Step 3: Update your filesets

Once you have your new app in your tenant, you need to update all your filesets and: - Replace objectid with the new value associated to your app in your tenant - Add appid and secret associated to your app in your tenant

Fileset Configuration

Once the plugin is successfully authorized, it is possible to define regular filesets for backup jobs in Bacula, where we need to include a line similar to the one below, in order to call the M365 Plugin:

Fileset M365
FileSet {
   Name = FS_M365
   Include {
      Options {
        signature = MD5
        ...
      }
      Plugin = "m365: <m365-parameter-1>=<m365-value-1> <m365-parameter-2>=<m365-value-2> ..."
   }
}

It is strongly recommended to use only one ‘Plugin’ line in every fileset. The plugin offers the needed flexibility to combine different modules backup inside the same plugin line. Different tenants, in case of existing, should be using different filesets and different jobs.

Below sub-sections list all the parameters you can use to control M365 Plugin behavior.

In this plugin, any parameter allowing a list of values can be assigned with a list of values separated by ‘,’.

Note

Starting with version 14.0 of Bacula Enterprise you can also perform fileset configuration directly using a dedicated BWeb M365 Wizard. Please go to section BWeb Management Console to see more details.

Common parameters

These parameters are common and applicable to all the modules of the M365 Plugin.

Option

Required

Default

Values

Example

Description

abort_on_error

No

No

No, Yes

Yes

If set to Yes: Abort job as soon as any error is found with any element. If set to No: Jobs can continue even if it they found a problem with some elements. They will try to backup or restore the other and only show a warning

config_file

No

The path pointing to a file containing any combination of plugin parameters

/opt/bacula/m365.settings

Allows to define a config file where configure any parameter of the plugin. Therefore you don’t need to put them directly in the Plugin line of the fileset

log

No

/opt/bacula/working/m365/m365-debug.log

An existing path with enough permissions for File Daemon to create a file with the provided name

/tmp/m365.log

Generates additional log in addition to what is shown in job log. This parameter is included in the backend file, so, in general, by default the log is going to be stored in the working directory.

debug

No

0

0, 1, 2, 3, 4, 5, 6, 7, 8, 9

Debug level. Greater values generate more debug information

Generates the working/m365/m365-debug.log* files containing debut information which is more complete with a greater debug number

path

No

/opt/bacula/working

An existing path with enough permissions for File Daemon to create any internal plugin file

/mnt/my-vol/

Uses this path to store metadata and temporary files

tenant

Yes

A valid tenant id string

57uia43-d107-17a2-a2g2-aa53c10tdahc

The tenant ID where the plugin will connect to in order to run backups or restores. Please, check section 5.1 of this paper for more information

objectid

Yes

String representing the objectid related to bacula-m365-plugin Azure app and the provided tenant id

56ddf1h9-eb5d-42nf-bac7-7b019fd284g5

The object ID of the plugin app of the plugin in Azure, once this is registered in the target tenant. Please, check section 5.1 of this paper for more information

appid

No

String representing the appid associated to bacula-m365-plugin-standalone Azure app registered in the configured tenant id

89tt4hu7-r4h7-kied-56gu-0895kf94jr9d

A valid appid string associated to bacula-m365-plugin-standalone Azure app registered in the configured tenant id. Please, check section 5.1 of this paper for more information

secret

No

String representing the associated appid secret

Jn8.lU-B.3P5gIRTGY6M.Xl3e29oQ6Xaf~

The secret associated to the m365 Azure app corresponding the configured appid string. Please, check section 5.1 of this paper for more information

token_cache_file

No

token_cache.json

A file name located in a valid existing path (it’s possible to put only the name of the file and ‘working’ dir will be used)

my_cache_file.json

The path that will be used to store the login cache for the device code flow authenticated users, wich is relative to the working tenant folder (working/tenant-name/token_cache.json)

service

No

email, drive, sharepoint, contact, calendar, onenote, tasks, teams, chat (list parameter: it can contain 0, 1 or more elements separated by ‘,’)

drive

Establish the service or services that will be backed up. If this is not set, the plugin will try to backup all supported services. It is recommended to split the work among different jobs when several services need to be applied. Therefore, even if this field is not required, it is strongly recommended to use it in every backup job.

owner_restore_protection

No

No

Yes, No

Yes

Enable owner restore protection feature, where the owner of the data being restored will be notified with an email and the restore job will be blocked until he/she enters a code and his/her M365 credentials to approve the operation

proxy_host

No

String representing DNS Name or IP address of the http(s) proxy

myproxy.example.com

Set up a proxy to make any plugin HTTP connection

proxy_port

No

Integer

3981

Set up the proxy port

proxy_user

No

String of proxy user

admin

Set up the proxy user

proxy_password

No

String of proxy password

myPass123

Set up the proxy user password

Note

The path option is adjustable while M365 jobs are in progress. The running jobs will continue to use the current spool area, and new jobs will use the new spool area defined by the path option.

If specifying more than one service in the fileset, please note that they will run in parallel. Because of that, we recommend to decrease a little the concurrency.

Below some multi-services fileset examples:

Fileset for all data belonging to a user
FileSet {
   Name = fs-m365-adelev-user
   Include {
      Options { signature = MD5 }
      Plugin = "m365: service=drive,email,calendar,contact,onenote,tasks,chat tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
   42nf-bac7-7b019fd284g5 user=adelev@baculaenterprise.onmicrosoft.com concurrent_threads=2"
   }
}
Fileset for all data belonging to a group
FileSet {
   Name = fs-m365-devteam-group
   Include {
      Options { signature = MD5 }
      Plugin = "m365: service=drive,calendar,onenote,tasks,teams tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
   42nf-bac7-7b019fd284g5 group=DevTeam concurrent_threads=2"
   }
}
Fileset for all data belonging to a site
FileSet {
   Name = fs-m365-mysite-site
   Include {
      Options { signature = MD5 }
      Plugin = "m365: service=sharepoint,drive,onenote tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
   42nf-bac7-7b019fd284g5 site=MySite sharepoint_include_drive_items=no sharepoint_system_include=yes drive_system_include=yes concurrent_threads=2"
   }
}

Note

Activating proxy mody will route all the requests through the proxy. However, it is needed DNS resolution to be working separately. Hence, the client where the FD is running will need to have a proper and working DNS Server configured. Other option is to setup the /etc/hosts file manually with the IP addresses of Microsoft authentication servers, but they could change over time. At the time of writing they are: 40.126.31.2 login.microsoftonline.com 20.190.160.131 login.microsoft.com

Advanced common parameters

Following parameters are common to all M365 modules (and even with some other plugins), but are advanced ones. They should not be modified in most common use cases.

Option

Required

Default

Values

Example

Description

stream_sleep

No

1

Positive integer (1/10 secconds)

5

Time to sleep when reading header packets from FD and not having a full header available

stream_max_wait

No

120

Positive integer (seconds)

360

Max wait time for FD to answer packet requests

time_max_last_modify_log

No

86400

Positive integer (seconds)

43200

Maximum time to wait to ovewrite a debug log that was marked as being used by other process

logging_max_file_size

No

50MB

String size

300MB

Maximum size of a single debug log fileGenerates the working/m365/m365-debug.log* files containing debut information which is more complete with a greater debug number

logging_max_backup_index

No

25

Positive integer (number of files)

50

Maximum number of log files to keep

log_rolling_file_pattern

No

m365.log.%d{dd-MMM}.log.gz”

No, Yes

Yes

Log patter for rotated log files

split_config_file

No

=

Character

:

Character to be used in config_file parameter as separator for keys and values

opener_queue_timeout_secs

No

1200

Positive integer (seconds)

3600

Timeout when internal object opener queue is full

publisher_queue_timeout_secs

No

1200

Positive integer (seconds)

3600

Timeout when internal object publisher queue is full

The internal plugin logging framework presents some relevant features that we are going to describe:

  • The “.log” files are rotated automatically. Currently each file can be 50Mb at maximum and the plugin will keep 25 files.

    • This behavior can be changed using the internal advanced parameters: logging_max_file_size and logging_max_backup_index

  • The “.err” file can show contents even if no real error happened in the jobs. It can show contents too even if debug is disabled. This file is not rotated, but it is expected to be a small file in general. If you still need to rotate it, you can include it in a general rotating tool like ‘logrotate’.

  • Backups in paralel and also failed backups will generate several log files. For example: m365-debug-0.log, m365-debug-1.log…

Tuning parameters

These set of parameters are common to all modules and they are advanced ones. They should not be modified in general. They can be used to tune the behavior of the plugin to be more flexible in particular bad network environments or when significant job concurrency is happening, etc.

Option

Required

Default

Values

Example

Description

backup_queue_size

No

100

0-500

1

Number of maximum enqueued internal operations between service static internal threads (there are 3 communicating through queues with the set size: service fetcher, service opener and general publisher to bacula core). This could potentially affect graph api concurrent requests and consequently, Graph throttling. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel

concurrent_threads

No

10

0-100

1

Number of maximum concurrent backup threads running in parallel in order to fetch or open data for running download actions. This means every service fetcher and service opener will open this number of child concurrent threads. This will affect graph api concurrent requests. Graph API can throttle requests depending on a variety of circumstances, but this parameter impacts it directly. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel. If you want to have a precise control of your parallelization through different jobs, please set up this value to 1. Please be careful also with the memory requirements, multi-threaded increases very significantly memory consumption per job

concurrent_listing_threads

No

5

0-20

1

Number of maximum concurrent backup page listing threads running in parallel in order to fetch sets of data for some modules. Currently it’s only used in the email module. This parameter will also affect graph api concurrent requests. Graph API can throttle requests depending on a variety of circumstances, but this parameter impacts it directly. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel. If you want to have a precise control of your parallelization through different jobs, please set up this value to 1. Please be careful also with the memory requirements, multi-threaded increases very significantly memory consumption per job

graph_list_page_size

No

200

1-500

350

Number of maximum elements got from Graph API for each page of objects. Higher number implies less requests, but more memory and more time for each request

graph_timeout

No

9000

Positive integer (milliseconds)

60000

Graph call timeout inside HttpClient

graph_read_timeout

No

300

Positive integer (milliseconds)

30000

Graph read timeout inside HttpClient

graph_retries

No

5

Positive integer (number of retries)

10

Graph number of retries for retry-candidate requestsInclude some stats information in the joblog. Useful to measure task times

graph_retry_delay

No

5

Positive integer (seconds)

10

Graph delay between retries

general_network_retries

No

5

Positive integer (number of retries)

10

Number of retries for the general M365 external retry mechanism

general_network_delay

No

50

Positive integer (seconds)

100

General M365 Plugin delay between retries

throttled_wait_time

No

300

Positive integer (seconds)

600

Extra wait time for throttled situations where the timeout is not provided or not got from MS API. In onenote module this is multiplied by 2. Once MS throttles a request is better to not retry too soon or the changes to continue with rejected requests will increase significantly

stats

No

No

No, Yes

Yes

Include some stats information in the joblog. Useful to measure task times

Note

graph_list_page_size had default value 500 before BEE 14.0.7. A higher value for this parameter can improve the performance at it reduces the number of API calls done to M365 service. However, the service can also be overloaded and return more HTTP 503 errors (Bad Gateway), especially for the email module. Starting from version 16.0.3, default values for backup_queue_size and concurrent_threads have been increased, also the allowed ranges.

Entity parameters

The following list of parameters are commonly shared through any module used into the same fileset line and are intended to select the target entities to backup. Every module subsection mentions what entities are supported too.

Option

Required

Default

Values

Example

Services

Description

user

No

Valid email addresses of existing users on the selected tenant separated by ‘,’

AlexW@yourdomain.com, LeeY@yourdomain.com

email, drive, contact, calendar, onenote, chat, tasks

Backup mailboxes, drive unit spaces, categories, or whatever service is selected of this list of users. If no user is provided: - The backup will be done for all accessible users of the given tenant if no other entry has value either (group or site parameters)

user_exclude

No

Valid email addresses of existing users on the selected tenant separated by ‘,’

LauraG@yourdomain.com, AmandaT@yourdomain.com

email, drive, contact, calendar, onenote, chat, tasks

Exclude selected mailboxes or onedrive spaces. If this is the only parameter found for selection, all elements will be included and this list will be excluded

user_regex_include

No

Valid regex

.*@management\.mydomain.com

email, drive, contact, calendar, onenote, chat, tasks

Backup matching user mailboxes or onedrive spaces. Please, only provide list parameters (user + user_exclude) or regex ones. But do not try to combine them

user_regex_exclude

No

Valid regex

.*@guests\.mydomain.com

email, drive, contact, calendar, onenote, chat, tasks

Exclude matching user mailboxes or onedrive spaces from the selection. Please, only provide list parameters (user + user_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded

site

No

Valid names of existing user sharepoint sites on the selected tenant separated by ‘,’

Communication site, Dev site

drive, sharepoint, onenote

Backup onedrive site library space (OneDrive service) belonging to this list of sharepoint sites or the site itself (Sharepoint service). If no site name is provided: - OneDrive: user and site variables will be checked. If no one has value either, backup will be done for all accessible users, groups or sites of the given tenant - Sharepoint: all accessible sites of the tenant will be backed up

site_exclude

No

Valid names of existing user sharepoint sites on the selected tenant separated by ‘,’

Test site

drive, sharepoint, onenote

Exclude listed sites from backup. If this is the only parameter found for selection, all elements will be included and this list will be excluded

site_regex_include

No

Valid regex

.*site

drive, sharepoint, onenote

Backup matching sites. Please, only provide list parameters (site + site_exclude) or regex ones. But do not try to combine them

site_regex_exclude

No

Valid regex

Test.*

drive, sharepoint, onenote

Exclude matching sites from the selection. Please, only provide list parameters (site + site_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded

group

No

Valid names of existing user groups on the selected tenant separated by ‘,’

Dev Team, Bacula Users

drive, onenote, calendar, teams, tasks

Backup onedrive sharepoint library spaces or notebooks of this list of groups. If no group is provided, user and site variables will be checked. If no one has value either, backup will be done for all accessible users, groups or sites of the given tenant

group_exclude

No

Valid names of existing user groups on the selected tenant separated by ‘,’

Monitoring, External

drive, onenote, calendar, teams, tasks

Exclude selected group. If this is the only parameter found for selection, all elements will be included and this list will be excluded

group_regex_include

No

Valid regex

.*Management

drive, onenote, calendar, teams, tasks

Backup matching group onedrive spaces. Please, only provide list parameters (group + group_exclude) or regex ones. But do not try to combine them

group_regex_exclude

No

Valid regex

.*External

drive, onenote, calendar, teams, tasks

Exclude matching groups from the selection. Please, only provide list parameters (group + group_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded

Backup parameters

Please, check the specific module pages in order to see backup parameters that are applicable only to each of them:

Deprecated parameters

The following parameters have been deprecated since version 12.8.3, in order to provide more flexibility. General shared parameters have been transformed in specific ones for each service
  • files -> email_files, drive_files, calendar_files, contact_files

  • files_exclude -> email_files_exclude, drive_files_exclude, calendar_files_exclude, contact_files_exclude

  • files_regex_include -> email_files_regex_include, drive_files_regex_include, calendar_files_regex_include, contact_files_regex_include

  • files_regex_exclude -> email_files_regex_exclude, drive_files_regex_exclude, calendar_files_regex_exclude, contact_files_regex_exclude

  • multilevel_attach -> email_multilevel_attach, calendar_multilevel_attach

  • exclude_attachments -> email_exclude_attachments, calendar_exclude_attachments

  • system_include -> drive_system_include, sharepoint_system_include

  • version_history -> drive_version_history, sharepoint_version_history

  • backup_threads -> backup_queue_size

Restore parameters

The plugin is able to restore to the local file system on the server where the File Daemon is running or to the Microsoft 365 environment. The method is selected based on the value of the where parameter at restore time:

  • Empty or ‘/’ (example: where=/) → Microsoft 365 restore will be triggered

  • Any other path for where (example: where=/tmp) → Local file system restore will be triggered

When using the Microsoft 365 restore option, the following parameters may be modified by selecting ‘Plugin Options’ during the bconsole restore session:

Option

Required

Default

Values

Example

Services

Description

destination_user

No

Existing email address on the target Azure AD Tenant

AlexW@yourdomain.com

email, drive, contact, calendar, onenote, tasks, chat

Destination User where restore data will be uploaded. If no user is set, every selected file will be restored in the original account

destination_group

No

Existing group name address on the target Azure AD Tenant

Marketing

drive, calendar, onenote

Destination Group where restore data will be uploaded. If no group is set, every selected file will be restored in the original group

destination_site

No

Existing site name/display name on the target Azure AD Tenant

live @ contoso

drive, sharepoint, onenote

Destination Site where restore data will be uploaded. If no site is set, every selected file will be restored in the original site

destination_path

No

Existing path on the selected user (mailfolder path or onedrive folder path)

Inbox

email, drive, contact, calendar, sharepoint

Destination folder where all selected files to restore will be restored. If no path is set: - If no user is set either, every element will go to its original location - If a user is set using the variable destination_user: - Elements belonging to destination_user will be restored in their original location - Elements belonging to different users than destination_user will be restored in a new folder using the email address of the original user of the element

destination_drive

No

Existing drive name or drive id in the destination entity

documents

drive

Destination drive where restore data will be uploaded. If no drive is set, every selected file will be restored in the original drive

send_report

No

0

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

1

email, drive, contact, calendar, sharepoint, tasks

Send a report to the user where every restore action is listed (generated emails, generated onedrive files..). - In the email service this will be an email in the user Inbox - In onedrive service this will generate a new text file in the top restore folder

allow_duplicates

No

1

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

0

email, contact, calendar

In Email services indicates if we want to allow to create duplicate emails or if we want to skip emails that are already in place. Important: This is done using the Microsoft Message ID. Therefore, non intuitive situations can happen. For example, if the original message is deleted and we restore several times using this option, we will see duplicate emails anyway

drive_skip_sharedwitme

No

0

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

1

drive

Skip restoring shared with me elements even if they are selected.

skip_versions

No

1

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

0

drive

Skip restoring former file versions (tagged with ‘###date’) even if they are selected. Important: Notice that this parameter is enabled by default, as we consider not restoring file versions the most common case. You need to disable it in order to have this kind of files restored

restore_share_permissions

No

0

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

1

drive, calendar

Restore share permissions of every element in order to regenerate sharing information as allowed identities, shared links, etc. Please note that this parameter is disabled by default.**Important**: Notice that this parameter is disabled by default, as we consider not restoring sharing permissions the most common case. You need to enable it in order to haver shared permissions restored

drive_send_invitations

No

0

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

1

drive

Send email invitations for restored OneDrive shares - Note you need to enable drive_restore_share_permissions in order to this parameter to have any effect

drive_invitations_message

No

String

I’m sharing with you automatically restored shares

drive

Set invitations message for restored OneDrive shares - Note you need to enable drive_restore_share_permissions in order to this parameter to have any effect

calendar_name

No

String

RestoredCalendar

calendar

Set the name of restored(s) calendar(s)

sharepoint_list_name

No

String

documentsCopy

sharepoint

Set the name of restored(s) sharepoint list(s)

sharepoint_newsite_name

No

String

Restored Site

sharepoint

Set the name of restored sharepoint new site

sharepoint_newsite_owner

No

String

bill@your.tennantname.com

sharepoint

Set the owner of the new restored sharepoint site

sharepoint_local_template_path

No

String

/tmp/my-restores/company meeting.site.template.xml

sharepoint

Set the local path of the PnP Sharepoint Site template you want to be using instead of the one contained in the backup itself.

notebook_name

No

String

myRestoredNotes

onenote

Set the owner of the new restored sharepoint site

notesection_name

No

String

Technology

onenote

Set the name of restored onenote notebook

notesection_group_name

No

String

ForWork

onenote

Set the name of restored onenote section

team_name

No

String

MyRestoredTeam

teams

Set the name of restored team (it will create also a new associated group)

team_channel_name

No

String

MyRestoredChannel

teams

Set the name of restored channel

team_private_channels_mode

No

DELEGATED

DELEGATED, PUBLIC, SKIP

PUBLIC

teams

Set the private channels restore mode

team_guest_members_enable

No

0

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

1

teams

Enalbe restore of team members with role ‘guest’. That kind of external user needs delegated permissions to be added, so potentially the restore could ask for user interaction (of first team owner found)

chat_topic

No

String

MyRestoredChat

chat

Set the topic of restored chat (supported only for group chats)

tasklist_name

No

String

ProjectA

tasks

Set the name of restored task list

tasklist_skip_sharedwithme

No

1

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

0

tasks

Skip shared with me tasks in the restore process

plan_name

No

String

RestoredPlan

tasks

Set the name of restored planner plan

plan_create_tab

No

1

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

0

tasks

Create or not an associated tab inside the associated team

local_path_json_objects

No

{path}/bacula-restores

String

/home/me/restores

onenote, sharepoint

Set local path to restore objects that cannot be restored directly into the M365 service

foreign_container_generation

No

1

0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on

0

email, drive, contact, calendar

Generate a general container (usually a folder) to put inside restored objects coming from different entities. For example, if we restore emails from user a@tenant.com into Mailbox of user b@tenant.com, this option enabled will generate an automatic folder a@tenant.com inside the destination restore folder used over destination user a@tenant.com

tenant

No

A valid tenant id string where some bacula m365 plugin app is registered

57uia43-d107-17a2-a2g2-aa53c10tdahc

email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat

Set the destination tenant id for a cross-tenant restore

appid

No

Appid of bacula-m365-plugin-standalone app registered in the provided tenant

56ddf1h9-eb5d-42nf-bac7-7b019fd284g5

email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat

Set the destination appid of bacula-m365-plugin-standalone for a cross-tenant restore

objectid

No

Object id associated to set up appid

89tt4hu7-r4h7-kied-56gu-0895kf94jr9d

email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat

Set the objectid associated to the appid in a cross-tenant restore

secret

No

Valid secret associated to appid

Jn8.lU-B.3P5gIRTGY6M.Xl3e29oQ6Xaf~

email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat

Set the secret associated to the appid set in the above parameter

debug

No

0

0, 1, 2 ,3, 4, 5, 6, 7, 8, 9

3

email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat

Change debug level

Automatic Objects Integration

Since Bacula version 16.0.7, a new solution has been introduced, so that each object can be backed up separately with different Jobs to maximize the throughput and the resiliency. It is highly recommended to use this new solution for that purpose - Automatic Object Integration (Scan Plugin). See an example for Microsoft 365Example.

Operations

To learn about backup and restore from a video, click on the image below:

M365 Video

Backup

Microsoft 365 plugin backup configurations currently have one specific requirement in the Job resource. Below we show some examples.

Job Example

The only special requirement with M365 jobs is that Accurate mode backups must be disabled, as this feature is not supported at this time.

Job Example
Job {
Name = m365-mytenant-backup
FileSet = fs-m365-email-all
Accurate = no
...
}

FileSet Examples

The plugin supports enough flexibility to configure almost any type of desired backup. Multiple Plugin= lines should not be spoecified in he Include section of a FileSet for the M365 Plugin.

Fileset examples for every supported service are linked below. For common purposes, the following two examples show how to configure an external config file or configure the number of threads:

Setup external config file:

Fileset Example
FileSet {
   Name = FS_M365_DRIVE
   Include {
      Options {
        signature = MD5
      }
      Plugin = "m365: config_file=/opt/bacula/etc/m365.settings service=drive"
   }
}
Settings file
$ cat /opt/bacula/etc/m365.settings
tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc
objectid=56ddf1h9-eb5d-42nf-bac7-7b019fd284g5

Increase number of threads:

Fileset Example
FileSet {
   Name = fs-m365-email-adelev
   Include {
      Options {
        signature = MD5
      }
      Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
   42nf-bac7-7b019fd284g5 user=adelev@baculaenterprise.onmicrosoft.com backup_threads=10"
   }
}

More fileset examples for:

Restore

Restore operations are done using standard Bacula Enterprise bconsole commands.

The where parameter controls if the restore will be done locally to the File Daemon’s file system or to the Microsoft 365 service:

  • where=/ or empty value → Restore will be done over M365

  • where=/any/other/path → Restore will be done locally to the File Daemon file system

Restore options are described in the Restore parameters section of this document, so here we are going to simply show an example restore session, particularly this example is about OneDrive service:

Restore Drive Bconsole Session
*restore where=/

First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.

To select the JobIds, you have the following choices:
1: List last 20 Jobs run
2: List Jobs where a given File is saved
3: Enter list of comma separated JobIds to select
4: Enter SQL list command
5: Select the most recent backup for a client
6: Select backup for a client before a specified time
7: Enter a list of files to restore
8: Enter a list of files to restore before a specified time
9: Find the JobIds of the most recent backup for a client
10: Find the JobIds for a backup for a client before a specified time
11: Enter a list of directories to restore for found JobIds
12: Select full restore to a specified Job date
13: Select object to restore
14: Cancel
Select item: (1-14): 5
Automatically selected Client: 127.0.0.1-fd
Automatically selected FileSet: FS_M365_DRIVE
+-------+-------+----------+-------------+---------------------+-------------------+
| jobid | level | jobfiles | jobbytes    | starttime           | volumename        |
+-------+-------+----------+-------------+---------------------+-------------------+
| 11    | F     | 190      | 332,978,505 | 2021-01-22 10:39:34 | TEST-2021-01-22:0 |
| 12    | I     | 1        | 550         | 2021-01-22 10:43:05 | TEST-2021-01-22:0 |
+-------+-------+----------+-------------+---------------------+-------------------+
You have selected the following JobIds: 11,12

Building directory tree for JobId(s) 11,12 ... +++++++++++++++++++++++++++++++++++++++++++++++
188 files inserted into the tree.

You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.

cwd is: /
$ cd /@m365/baculaenterprise/adelev@baculaenterprise.onmicrosoft.com/drive/root:
cwd is: /@m365/baculaenterprise/adelev@baculaenterprise.onmicrosoft.com/drive/root:/
$ ls
Docs/
pluginTest.drive.deltaLink
sharedWithMe/
test4###v1.0_2020-11-25_153507.html
test4###v2.0_2020-11-25_153507.html
test4###v3.0_2020-12-02_180612.html
test4###v4.0_2020-12-02_180612.html
test4###v5.0_2020-12-03_134422.html
test4###v6.0_2020-12-02_180612.html
test4.html
testlink123###v1.0_2020-11-26_123244.url
testlink123###v2.0_2020-11-26_123244.url
testlink123###v3.0_2020-12-02_180613.url
testlink123###v4.0_2020-12-02_180613.url
testlink123.url
$ mark *
189 files marked.
$ done
Bootstrap records written to /tmp/regress/working/127.0.0.1-dir.restore.3.bsr

The Job will require the following (*=>InChanger):
Volume(s) Storage(s) SD Device(s)
===========================================================================

TEST-2021-01-22:0 File FileStorage

Volumes marked with "*" are in the Autochanger.
189 files selected to be restored.

Using Catalog "MyCatalog"
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2021-01-22 11:54:55
Catalog: MyCatalog
Priority: 10
Plugin Options: *None*
OK to run? (yes/mod/no): mod
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Restore Client
6: When
7: Priority
8: Bootstrap
9: Where
10: File Relocation
11: Replace
12: JobId
13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : m365: service=drive tenant="574dda43-d107-48e2-a7f2-aa51c10bdaec" objectid="31ddf4b1-
ed5d-432f-bac7-7b946fd23394" user="adelev@baculaenterprise.onmicrosoft.com" drive_version_history=yes debug=3
Plugin Restore Options
Option Current Value Default Value
destination_user: *None* (*None*)
destination_path: *None* (*None*)
send_report: *None* (0)
email_allow_duplicates: *None* (1)
drive_skip_sharedwithme: *None* (0)
drive_skip_versions: *None* (1)
drive_restore_share_permissions: *None* (0)
drive_send_invitations: *None* (0)
drive_invitations_message: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (yes/mod/no): mod
You have the following choices:
1: destination_user (Destination User)
2: destination_path (Destination Path in M365)
3: send_report (Send report of the restore operation to the affected user)
4: email_allow_duplicates (Allow Duplicate Emails)
5: drive_skip_sharedwithme (Skip restoring shared with me elements even if they are selected)
6: drive_skip_versions (Skip restoring file former versions (tagged with '###date') even if they are selected)
7: drive_restore_share_permissions (Restore share permissions of items so they are shared if they originally
were)
8: drive_send_invitations (Send email invitations for restored OneDrive shares)
9: drive_invitations_message (Set invitations message for restored OneDrive shares)
10: debug (Change debug level)
Select parameter to modify (1-10): 2
Please enter a value for destination_path: MY_RESTORE_PATH
Plugin Restore Options
Option Current Value Default Value
destination_user: *None* (*None*)
destination_path: MY_RESTORE_PATH (*None*)
send_report: *None* (0)
email_allow_duplicates: *None* (1)
drive_skip_sharedwithme: *None* (0)
drive_skip_versions: *None* (1)
drive_restore_share_permissions: *None* (0)drive_send_invitations: *None* (0)
drive_invitations_message: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (yes/mod/no): yes
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2021-01-22 11:54:55
Catalog: MyCatalog
Priority: 10
Plugin Options: User specified
OK to run? (yes/mod/no): yes
Job queued. JobId=14

Restore by service

In this section some example restore configurations will be shown:

Cross tenant restore

You can perform cross-tenant restores using the restore variables: - tenantid - appid - objectid - secret

Obviously, it is needed to set up the destination tenant values, where one of the authentication methods should be applied first.

If you are using the standalone app authetication model, you will need to use the four values in your restore session.

If you want to use the common app authentication model, you won’t need to put the secret. On the other hand, please, ask the bacula enterprise edition support team in order to get the correct value for appid.

List

It is possible to list information using the bconsole .ls command and providing a path. In general, we need to provide the service parameter and a path representing a folder or object (like calendars).

There are some general commands (user, groups, sites), the rest of the commands need to point to the correct service.

Below some examples:

List general info: Users, groups, sites

Here we are showing these 3 commands using the bconsoel .ls command, but notice you may also use them with the query interface (keep your variable values, but apply something like: .query plugin=”…” client=xxxx parameter=xxx). Note that the referenced ‘config_file’ should contain the connection parameters (tenant, objectid, appid and secret).

List example: General information
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd path=/user
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
ptcomm: Starting Plugin Job
ptcomm: Finished reading Plugin Params
ptcomm: Backend connection to M365 stablished
ptcomm: Connected to tenant: baculaenterprise
ptcomm: Starting ListingStart
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /AdeleV@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /AlexW@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /DiegoS@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /jane@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /GradyA@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /HenriettaM@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /IsaiahL@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /JohannaL@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /JoniS@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /john@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /LeeG@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /LidiaH@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /LynneR@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /MeganB@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /MiriamG@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /NestorW@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /PattiF@baculaenterprise.onmicrosoft.com
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /PradeepG@baculaenterprise.onmicrosoft.com
2000 OK estimate files=18 bytes=0

*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf"
client=127.0.0.1-fd path=/group
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
ptcomm: Starting Plugin Job
ptcomm: Finished reading Plugin Params
ptcomm: Backend connection to M365 stablished
ptcomm: Connected to tenant: baculaenterprise
ptcomm: Starting ListingStart
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Contoso Team
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Dev Team
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /baculaenterprise
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /All Company
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Bacula Systems Team Site
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Dev people Group
2000 OK estimate files=6 bytes=0

*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf"
client=127.0.0.1-fd path=/site
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
ptcomm: Starting Plugin Job
ptcomm: Finished reading Plugin Params
ptcomm: Backend connection to M365 stablished
ptcomm: Connected to tenant: baculaenterprise
ptcomm: Starting ListingStart
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Communication site
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Adele Vance
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Alex Wilber
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Diego Siciliani
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Grady Archie
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Henrietta Mueller
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Isaiah Langer
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Johanna Lorenz
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Joni Sherman
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Lidia Holloway
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Lynne Robbins
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Megan Bowen
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Pradeep Gupta
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Bacula Systems Team Site
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Work @ Contoso
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Contoso Team
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Live @ Contoso
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Give @ Contoso
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Dev Team
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Dev people Group
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Team Site
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /John Doe
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Jane Doe
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Patti Fernandez
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Lee Gu
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Miriam Graham
-rw-r--r--    1 root    root        -1 2106-02-07 07:28:15 /Nestor Wilke
2000 OK estimate files=28 bytes=0

# Query equivalents follow
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=user
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=group
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=site

# Other similar query function
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=team

List Onedrive contents

List example: General information
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: baculaenterprise
m365: Backend connection to M365 stablished
m365: Starting ListingStart
drwxr-xr-x   1 nobody    nogroup            26800469 2020-11-08 01:34:35 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/root/
drwxr-xr-x   1 nobody    nogroup            9876293 2020-12-03 15:28:31 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/
-rw-r-----   1 nobody    nogroup            217873 2021-04-13 13:29:56 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Hats.JPG
-rw-r-----   1 nobody    nogroup   1914266 2021-02-01 17:45:34 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/lombok.jar
-rw-r-----   1 nobody    nogroup            785043 2021-02-01 17:45:56 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/noShared.dat
-rw-r-----   1 nobody    nogroup            4 2021-02-01 17:45:35 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/test2.doc
-rw-r-----   1 nobody    nogroup            4 2020-11-25 15:35:07 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/test4.html
-rw-r-----   1 nobody    nogroup            42 2020-11-26 12:32:44 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/testlink123.url
drwxr-xr-x   1 nobody    nogroup            -1 2106-02-07 07:28:15 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/sharedWithMe/
2000 OK estimate files=23 bytes=53,600,938
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf
service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/sharedWithMe
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: baculaenterprise
m365: Backend connection to M365 stablished
m365: Starting ListingStart
-rw-r-----   1 nobody    nogroup               11722 2021-04-13 17:42:06 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Aeque.ppt
-rw-r-----   1 nobody    nogroup               4039 2021-04-13 17:42:02 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Voluptatibus.txt
-rw-r-----   1 nobody    nogroup               7697 2021-04-13 17:41:59 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Nullam.jpeg
-rw-r-----   1 nobody    nogroup               21031 2021-04-13 17:41:55 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Graeci.ppt
-rw-r-----   1 nobody    nogroup               21417 2021-04-13 17:41:52 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Habeo.txt
-rw-r-----   1 nobody    nogroup               15354 2021-04-13 17:41:47 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Atomorum.jpeg
-rw-r-----   1 nobody    nogroup               22217 2021-04-13 17:41:43 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Faucibus.ppt
-rw-r-----   1 nobody    nogroup               15714 2021-04-12 11:13:52 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Sodales.ppt
-rw-r-----   1 nobody    nogroup               14371 2021-04-12 11:13:48 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Cum.txt
-rw-r-----   1 nobody    nogroup               11059 2021-04-12 11:13:44 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Sapien.jpeg
-rw-r-----   1 nobody    nogroup               14144 2021-04-12 11:13:41 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Habitasse.ppt
-rw-r-----   1 nobody    nogroup               12594 2021-04-12 11:13:37 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Deserunt.txt
-rw-r-----   1 nobody    nogroup               14923 2021-04-12 11:13:32 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Postulant.jpeg
-rw-r-----   1 nobody    nogroup                21307 2021-04-12 11:13:29 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Tation.ppt
drwxr-xr-x   1 nobody    nogroup                0 2021-01-08 14:32:20 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/john@baculaenterprise.onmicrosoft.
com/JohnFolder/
-rw-r-----   1 nobody    nogroup                1442312 2021-01-08 11:57:09 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/john@baculaenterprise.onmicrosoft.
com/developers.pdf
-rw-r-----   1 nobody    nogroup                630174 2021-01-08 11:57:08 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/john@baculaenterprise.onmicrosoft.
com/console.pdf
2000 OK estimate files=17 bytes=2,280,075*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf
service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Docs
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: baculaenterprise
m365: Backend connection to M365 stablished
m365: Starting ListingStart
drwxr-xr-x   1 nobody    nogroup                9876293 2020-12-03 15:28:31 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/
drwxr-xr-x   1 nobody    nogroup                793475 2020-11-25 15:34:49 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/Simple Sub Folders/
-rw-r-----   1 nobody    nogroup                6383460 2020-12-08 10:13:11 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/E16WKAMS.10E.zip
-rw-r-----   1 nobody    nogroup                1914266 2020-12-08 10:13:32 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/lombok.jar
-rw-r-----   1 nobody    nogroup                785043 2021-01-12 10:12:04 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/noShared.dat
-rw-r-----   1 nobody    nogroup
24 2020-11-25 15:35:05 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test1.txt
-rw-r-----   1 nobody    nogroup                13 2021-01-11 15:39:57 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test11.txt
-rw-r-----   1 nobody    nogroup                4 2020-11-25 15:35:05 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test2.doc
-rw-r-----   1 nobody    nogroup                4 2020-11-25 15:35:05 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test3.pdf
-rw-r-----   1 nobody    nogroup                4 2020-12-03 16:52:32 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test4.html
2000 OK estimate files=10 bytes=19,752,586

Other list examples

Please note that the following examples are not available in .query mode, they are only for .ls command.

List example: General information
# List contents of Inbox folder of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=email user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Inbox

# List contents of Sent folder of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=email user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Sent

# List all contacts of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=contacts user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/

# List contacts of user adelev@baculaenterprise.onmicrosoft.com and folder MyContactsDir
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=contacts user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyContactsDir

# List calendar events of user adelev@baculaenterprise.onmicrosoft.com and her calendar MyCalendar
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=calendar user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyCalendar

# List onenote pages of user adelev@baculaenterprise.onmicrosoft.com and her section secA included in section Group groupA
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=onenote user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyNotebook/sectionGroups/groupA/sections/secA

# List contents of OneDrive folder 'folderA' of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/folderA

Performance command

This plugin provides a query command allowing to check the performance of a given setup. The process consist on uploading some randomly generated files of a customizable size (and a customizable number of them) to a target tenant, then download them using the same method the backup itself employs. Upload and download times are measured. An average is calculated and all the information is presented to the user.

Please, note that this command is currently working in single-threaded mode, while the plugin is running with multiple threads (configurable through concurrent_threads variable), so you could have better results with the actual backup process.

The usage of this command is:

Query perf
*.query client=<your-client> plugin="m365: tenant=<tenantId> objectid=<objectId> user=<your-user> service=<service>" parameter=perf

Currently, theh plugin supports two types of performance commands: perf for onedrive (default) and perf for email. So the value service should be: drive or email.

The behavior of this service from performance perspective can differ significantly. Below we detail some important aspects:

  • On Onedrive and other services implying user files, it is needed to request a ‘download session’ for each file.

    • Once we have it, the download starts. This requires significantly more time than just retrieving simple objects.

  • In general, objects are not retrieved one by one. We request for lists of objects, that later on are processed.

    • For objects that are not pointing to any other file (as emails without attachments), the time needed per object is very low.

Drive perf command

For Onedrive service, in addition to user, you can also use group or site (site name or siteId) for this command. The final output will be similar to the following:

Query perf drive output
...
date=2021-09-06 17:25:56
tenant_name=johndoe.onmicrosoft.com
upload-speed-average=3.3 MiB/s
download-speed-average=32.2 MiB/s
latency-average=482.00 ms
console=---- Test END ----

You can control the size and the number of files used using the following query parameters: - q_perf_size (in bytes) - q_perf_files (number of files)

By default, their values are: - q_perf_size = 209715200 - q_perf_files = 3

Below we provide a usage example:

Query perf drive sample
*.query client=127.0.0.1-fd plugin="m365:
tenant=f421f256-748a-4a80-90da-bf47y66fa166 objectid=90173rea-c144-46ye-8f93-4uh87bcd6d47 site=\"johndoe.sharepoint.com,fda66478-aab5-4896-a6ff-e0fb02af2a40,c024ee54-dcc0-4f1e-ab76-671f6eb87fda\" service=drive q_perf_files=5 q_perf_size=314572800" parameter=perf
Email perf command

For Email service, specifying a user is mandatory. The command can generate simple emails, but also emails with one attachment. The final output can be similar to the following:

Query perf drive output
...
console=---- Summary ----
date=2021-11-11 17:05:18
tenant_name=johndoe.onmicrosoft.com
upload-speed-average=44.6 KiB/s
download-speed-average=220.5 KiB/s
latency-average-per-list-call=510.83 ms
latency-average-per-message=36.18 ms
latency-average-attachments=617.64 ms
console=---- Test END ----

The behavior of the email perf command is a little different from the Onedrive one. Instead of uploading/downloading one message at a time, here everything is uploaded first. After that everything is downloaded. The reason is here the backup process uses lists of objects and, unless there are attachments implied, that’s all, the objects are simply stored from that list.

Similarly to the onedrive perf command, you can control the size and the number of emails. However, there is an extra parameter to control the size of the attachments:

  • q_perf_size (in bytes)

  • q_perf_size_attachments (in bytes)

    • If not specified, no attachment will be generated

    • If specified, one simple text attachment will be generated together with each email

  • q_perf_files (number of files)

By default, their values are:

  • q_perf_size = 10240

  • q_perf_size_attachments = 0

  • q_perf_files = 50

Below we provide a usage example:

Query perf email sample
*.query client=127.0.0.1-fd plugin="m365:
tenant=f421f256-748a-4a80-90da-bf47y66fa166 objectid=90173rea-c144-46ye-8f93-4uh87bcd6d47 site=\"johndoe.sharepoint.com,fda66478-aab5-4896-a6ff-e0fb02af2a40,c024ee54-dcc0-4f1e-ab76-671f6eb87fda\" service=email q_perf_size_attachment=204800" parameter=perf

Other query commands

Sometimes there is an element that causes some error while fetching it in the email service and it’s difficult to identify it by its reported debug id. For this situation, there is a decoding command:

As the example shows, the query parameter is ‘query=decode|{url}’. Url should be the URL until the element we are interested into (so the attachment here).

Another useful command shows all the options the plugin can accept as parameters, categorized by section:

The output of this command will return a json structure with all the available options.

Best practices

Jobs Distribution

It is recommended, to split the target backup between different groups of entities or even having one job per entity (user, group, site, etc). This way errors in one job will not invalidate a whole backup cycle where some entities have been successful and some others had errors. This also makes easier to identify the cause of the error.

Concurrency

Microsoft public APIs impose a variety of boundaries that need to be considered. If a boundary is crossed, the corresponding API call will fail and the application will need to wait some amount of time to retry, which is different depending on the boundary crossed.

It is crucial to plan an adequate strategy to backup all the elements without reaching API boundaries. A single job implements some parallelism which can be reduced until a point, if necessary, using the variable backup_queue_size (default value is 30). This variable controls the size of the internal queues communicating the internal threads, that are designed to fetch, open and send every item to Bacula core. Reducing its size will produce, ultimately (with a value of 1 for example), an execution very similar to a single threaded process. On the othere hand the plugin has concurrent_threads which controls the number of simultaneous processes fetching and downloading data (default value is 5).

If you are going to launch different jobs in parallel it is recommended to configure different services for each of them (protect in parallel email, drive, contacts…). However, be careful with the concurrency over the same service (in general, it is recommended a maximum of 4-5 jobs working with the same service) and plan a step-by-step testing scenario before putting it into production. Other important point is the timing schedule, as some boundaries are related to time-frames (number of request per 10 minutes or 1 hour, for example). If you detect you reach boundaries when running all your backups during a single day of the week, please try to use 2 or 3 days and spread the load through them in order to achieve better performance results.

More information about Microsoft 365 Graph API boundaries may be found here:

https://docs.microsoft.com/graph/throttling

Disk Space

It is necessary to have at least enough disk space available for the size of the largest file in the backup session. If you are using concurrency between jobs or through the same job (by default this is the case through the concurrent_threads=5 parameter), you would need at least that size for the largest file multiplied by the number of operations in parallel you run.

Read more details in the Backup of Attachments and Files section.

Performance

The performance of this plugin is highly dependent on many external factors:

  • ISP latency and bandwidth

  • Network infrastructure

  • FD Host hardware

  • FD Load

In summary, it is not possible to establish an exact reference about how much time a backup will need to complete.

As reference values, in testing environments and using sequential calls we had the following results:

  • 3000 emails in the same folder of a single mailbox, with emails ranging from 500 to 3000 words - about 10K to 30K in size

  • The total time required to back them up was: 88243 ms.

  • That implies a total time per email of about 30ms

Concurrency has also been tested. For example, when using single-threaded mode (concurrent_threads=1) with emails, good values where found for 4-6 concurrent jobs that allowed us to backup around 200 emails per second (sizes between 20K-30K) when no attachments were implied. As more attachments were implied, the number of emails will decrease very significantly, while the overall size speed will be increased. This can be generalized:

  • Many little objects to protect -> More objects per second, but less speed (MB/s)

  • Big files to protect -> Less objects per second, but greater speed (MB/s)

It is recommended to benchmark your own environment in base to your requirements and needs.

The automatic parallelization mechanism (using concurrent_threads=x, default is 5) should work well for most scenarios, however, fine tune is possible if we define one job per entity and we control how many of them run in pararllel, together to decrease the concurrent_threads value in order to avoid throttling from MS Graph API.

There are many different possible strategies to use this plugin, so please, study what is best suiting for your needs before deploying the jobs for your entire environment, so you can get best possible results:

  • You can have a job per entity (users, groups, sites…) and all services

  • You can have inside a job multiple entities and only some services

  • You can split your workload through an schedule, or try to run all your jobs together.

  • You can run jobs in parallel or take advantadge of concurrent_threads and so run less jobs in parallel

  • You can select what services to backup or backup them all

  • You can backup whole services to backup or select precisely what elements you really need inside each service (folders, paths, exclussions…)

  • For delegated permissions services, you can proceed with the login previously to backups (recommended) or you can wait for the job to ask for what it needs

  • etc.

Best practices by service

Each Microsoft 365 service presents its own particularties. Moreover each deployment and each environment can be protected through different strategies depending on the size, number of entities and other parameters. However, there are some general considerations that can be helpful to consider for a selection of services:

Sharepoint

Sharepoint Online is the service where the target data varies the most among the M365 services. Sites can be structured very differently and there exists many different kinds of Sharepoint lists. In addition to that, some of them could have very large document libraries, while others could be very sparse.

For these reasons, it is recommended to split Sharepoint jobs to be as atomic as possible.

Ideally jobs should be split up to a single site backup job. This will help to monitor the health of the backups, to keep good backups for all sites presenting no issues and but to quickly troubleshoot any site presenting any kind of inconvenience. Usually, this is especially needed when Sharepoint service are intensively used.

If sites are created and removed often, another option to avoid a frequent configuration change is to employ the site_regex_include plugin variable and use the naming patterns to set up the jobs, so they will also include future created sites (possible examples: .*Team, .*Web, Dev.* … ).

In order to facilitate the job split, it is possible to use the listing or query commands described in List general info: Users, groups, sites to have the site names list.

Another best practice is to split Sharepoint sites into two parts with two different jobs: - Pure Sharepoint job: Use sharepoint service (service=sharepoint site=SiteName) and the parameter sharepoint_include_drive_items=no. - Drive part job: Use drive service (service=drive site=SiteName)

Sharepoint jobs, when using the associated powershell commands, cannot be run in parallel. One will wait for any other session to finish, before executing the corresponding commands. Using the above strategy makes Sharepoint jobs to be much more quick and light, so it’s a lot easier to control such concurrency. Other ways to control the concurrency in Bacula Enterprise are:

  • Designing separated time windows through scheduling

  • Sending all Sharepoint jobs to the same pool and the associated storage resource (both of them dedicated to Sharepoint backups) and limiting ConcurrentJobs to the desired value in that particular storage

  • Having 1 specific client resource for Sharepoint and use ConcurrentJobs parameter there.

In general, it’s not needed to enable system lists, hidden lists or even the versioning of the elements. All those options will impact the performance of the jobs directly, so you should plan carefully the usage of such parameters.

Some Sharepoint sites present an issue with the hashcheck of the files contained in some of its document libraries. If the hash is not correct at M365 side, it does not match the one that is calculated by the plugin locally, while the file is correct. This situation causes a lot of error messages around the hash and also many extra calls to the service to re-check this hash. When this situation happens, it is recommended to disable the hashcheck mechanism for the affected site using the parameter ‘drive_disable_hashcheck=true’.

Onenote

Onenote is the most sensitive module to throttling limits in Microsoft 365 APIs: https://learn.microsoft.com/en-us/graph/throttling

It is recommended to be selective with the information to be protected and make one job for each selected entity: that is one job per user, group or site.

In case throttling problems appears with this strategy, jobs will need to be spread into larger time windows.

It is not recommended to include onenote for entity focused strategies where there is one job per user including all services (email, drive, contact, calendar ..). It is recomended to have specific onenote jobs.

Chats and Todo

This service uses delegated permissions. They can be protected together, but we recomend to run the ‘login’ query command for each individual user before running the actual jobs.

Email

Email service allows you to protect not only email and attachments, but also settings, user categories and an additional MIME copy for each email with its attachments. MIME files are very useful if you are planning to migrate your data from M365 to any other Email tool, however, the cost of getting each MIME file is very high and it duplicates backup time. MIME files are not needed to perform a local restore or a restore over M365. Therefore, it is recommended to disable MIME backup for a general basis and use it only in case of planning a complete migration of the data.

Similarly to other services, it is recommended to split the backup set into small sets of users (ideally one backup per mailbox, but it’s also fine to group them by 5 or 10), specially if there is a large number of mailboxes and they contain thousands of emails. To make this easier, the plugin offers regular expression selection parameters (user_regex_include), as well as listing and query commands to analyze the backup target (List general info: Users, groups, sites).

Update app permissions

Bacula Enterprise Microsoft 365 Plugin is an application that is under continous evolution. In consequence it may happen that new features require new permissions to function properly.

Bacula Systems will add the required new permissions to the central bacula-m365-plugin app registered in Azure AD. If you are already using the plugin, you will need to update the permissions of your own app (usually bacula-m365-plugin app itself, available under ‘enterprise applications’ option in your Azure Active Directory application).

To refresh and allow any new permission to your app. Please apply the following procedure:

  1. Access the Azure Portal with a Tenant admin user

  2. Locate your bacula-m365-plugin app under ‘Enterprise applications’ and click on it.

m365-plugin-app in enterprise applications
  1. In the left menu, click on ‘Permissions’ and then click the big blue button showing ‘Grant admin consent for <yourtenantname>’:

grant permissions
  1. Wait for a few minutes

  2. Refresh the page

  3. See the new permissions in the list

New releases needing any new permissions will properly reflect that information with the specific permissions list.

Delegated permissions

The great majority of services of the Bacula Enterprise M365 plugin are using the model of ‘Application permissions’. With this model, the plugin application can directly access the implied services once a tenant admin has aproved it.

On the contrary, some other services can require the model of ‘Delegated permissions’ where the plugin needs to impersonate a given user in order to be capable of accessing and restoring the data inside the service. Although we will support more in the future, currently, the only service requesting delegated permissions is:

  • Calendar: If any group is included as an entity to protect

  • Chat: In order to be able to backup all chat messages belonging to a certain user the user in question will need to delegate the permissions as described below.

Delegated permissions or application permissions are automatically applied when they are needed. However, delegated permissions need user interaction the first time they are invoked, or if the local information about access tokens has been lost for some reason.

The user interaction can be manually triggered with a special query command that may be run as follows:

query-login
.query plugin="m365: tenant=e421f055-748a-4a80-90da-bf48b66fa166 objectid=9017f87a-c164-488e-8f93-47139bcd6d47" client=127.0.0.1-fd parameter=login
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: johndoe
m365: Backend connection to M365 stablished
m365: Starting QueryStart
m365: We don't have a valid token. Please, To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CQ7LETVFT to authenticate. You need to do it with an administrator user

We only need to follow the instructions, open a browser, enter the code and use the correct user to login (which means a user with access to the resource we want to backup).

For example, if we are working with the calendar of a group, we will need a user with access to that calendar (in general, to protect calendars associated to groups, it will be necessary to create a special backup user with access to all of them and perform the login with, which will avoid the need of performing the login process for every different group).

m365-devicelogin-2
m365-devicelogin-3

Here, if any MFA mechanism is enabled, it will be normally required and the user must complete the associated process. Finally Microsoft will ask to confirm if we want the plugin app to proceed with the operation, and we need to confirm:

device-code-flow-confirm

Once the login is complete, the plugin will automatically end the operation and show something like:

query-login-2
info=We have a valid token got from account: johndoe@johndoe.onmicrosoft.com
info=It will be automatically refreshed when needed. But the current expiration date is: 2021-06-16T16:18:16Z
info=Backups using services with delegated permissions won't ask for intervention

It will also store the token information locally in a special file that it is stored in the working directory inside a folder with the tenant name. For example: /opt/bacula/working/m365/tennantname/bacula-m365-plugin-standalone-token_cache.json

This file is also backed up in the backup operation itself, so it can be restored manually, in case it was lost and in case you don’t want to run login operations again

In case the query method is not invoked prior to any job needing it, the job itself will show the message in the joblog and it will be blocked until a user performs the needed interaction described above. Once this happens, the job will automatically continue.

Note

Starting with version 14.0 of Bacula Enterprise you can also see logged in users directly using BWeb, where you can also manually add logged-in users. In order to see more details, please go to section: BWeb Console<./m365-bweb-console>.

BWeb Management Console

Bacula Enterprise Microsoft 365 Plugin can be managed from a dedicated user friendly Web console, specifically designed to facilitate tasks as the authentication process for new tenants or the fileset configuration process.

Note

This feature is available starting with version 14.0 of Bacula Enteprise.

Connection

The purpose of this screen is to show a list of the currently connected tenants that are already registered. The presence of each line in the table means that a tenant has been configured at some point.

Tenant list

In the upper right corner, it is possible to switch the FD at any time, so the information is based on the currently selected FD. After the first time a FD is selected, the information will be remembered using cookies for the next time and for any other M365 Console windows.

The table shows tenant name, tenant id and the connection state. The connection state can indicate a possible incomplete registration operation (because of some error), otherwise it will indicate a successful connection and the type of connection: through the common app or through the standalone app.

The big button placed at the bottom of the table allows us to configure the connection to a new tenant, so it will open the ‘New tenant wizard’ that is described in the section below.

Other actions that can be performed from this tenant list window are:

  • Retry operation: It allows to retry a possible incomplete registration, so re-launch the proper internal query command (add-app or the objectid command) using a selected tenant.

  • Delete registrator: It allows to delete the registrator app inside the selected tenant. It is not possible to delete a registrator if registration is incomplete . In this case, you must retry operation or delete tenant. This is a possibility only offered for security concerns, as the registrator app has some elevated permissions.

  • Delete tenant: It removes the tenant from the internal BWeb configuration.

New tenant wizard

When opening new tenant wizard, we can select the registration model to be used: Standalone or Common.

Details of each model are shown in the images and you can also read more information in this whitepaper under the Authorization section.

It is needed to provide the tenant id, which can be found following the whitepaper instructions placed under the same Authorization section.

It is recommended to use the Standalone model (first one to appear in the web interface) and be using the common model only for testing purposes.

Once we proceed with one of the models, an URI of Microsoft website will be suggested and you need to provide the needed permissions, then, Microsoft will redirect you to a Bacula Systems page (same as shown in the Authorization section).

BWeb will automatically guide you until the end of the process where you will see your tenant correctly added into the tenant list screen.

Logged in users

In this screen a list of logged in users is shown.

Logged users list

Logged in users are particular users that have approved the Azure AD plugin app to make operations over their accounts that need delegated permissions.

Examples of operations needing this permissions model are:

  • Backup of calendar of groups

  • Todo Tasks module

  • Chats module

In addition to show the users list, it is also possible to force an user login. This function is offered to avoid having this kind of requests at backup time.

Users need to login only once, afterwards, the local cache will automatically be used to use saved tokens or to refresh them as needed.

For more information about delegated permissions, please take a look to <./m365-delegated-permissions>

FileSets

The purpose of the FileSets page is to control all the filesets of Microsoft 365 Plugin, so we will see them listed in the main screen:

Fileset list

From there it is possible to create a new M365 Fileset in an user friendly way:

We need to select the tenant to work with as a first step:

Fileset wizard select tenant

The second step is to select the entities to protect: they can be users, groups or sites. All the services of M365 are ‘owned’ by any of those entities:

Fileset wizard select entities

It is possible to dinamically add or remove them, the wizard will store your final selection.

The third step allows us to select the services to protect in a similar way than the entities step, with multiselection allowed, and where we can remove or add services at will:

Fileset wizard select services

After selecting the services, we will have one independent screen for each selected service where we will enter the particular configuration for each of the services implied. In the screenshot below, we can see the Sharepoint services:

Fileset wizard setup sharepoint

As a final step, it is possible to setup some advanced parameters and then finish the configuration:

Fileset wizard advanced and confirm

From this final screen is possible to go directly to the job configuration wizard and finish a backup configuration or to return to the previous fileset list page.

Please, note that this Wizard contains the most common parameters and it is intended to help to configure the most common use cases. In case you need to setup more advanced parameters or to edit existing filesets, BWeb will allow you to do it using the regular Fileset editor where you will find all plugin parameters.

  • The restart command has limitations with plugins, as it initiates the Job from scratch rather than continuing it. Bacula determines whether a Job is restarted or continued, but using the restart command will result in a new Job.

Troubleshooting

Listed in this section are some scenarios that are known to cause issues.

Pwsh: The specified program requires a newer version of windows

If a Sharepoint backup shows the message referenced in the title, it means the powershell session of the job has conflicted with another open Powershell session.

Solution: Be sure no other Powershell sessions are open and try again

M365 DataPack Onedrive Hashes

If you test the plugin with the developer Datapack of M365 you may find problems with hash checking of OneDrive autogenerated files. You could see hash failures.

The reason is those auto-generated files have incorrect hashes.

Solution: Use your own uploaded files to test the plugin.

Sharepoint PnP Template restore problems

As stated in section Details on the Sharepoint backup/restore, templates of Sharepoint can present different problems when they are applied over new generated sites. To troubleshoot these issues, the following steps can be taken:

  • Restore the site template locally

  • Detect the exact part of the template that is causing problems and remove that section from the xml

  • Manually apply the site template using PnP.Powershell

Suppose we are experiencing this problem and we have already restored our template in /tmp/mytemplate.xml and it is ready to be tested. Below we show how to run manually PnP actions:

  1. The first thing to do is run Powershell. Just run the following command in your terminal:

pwsh
pwsh
  1. Import the Pnp Module:

pwsh
Import-Module -Name PnP.PowerShell
  1. Connect to your main site, which is https://tenantname.sharepoint.com

    1. We will use the DeviceLogin method, as this method allows Multifactor Authentication. Just run the command shown with your main site:

      ../../../../_images/DeviceLogin-1.png
    2. Follow the instructions and go to a browser pointing to the provided URL. You will be asked for credentials and you need to login with a tenant administrator user:

      ../../../../_images/DeviceLogin-3.png
    3. If MFA is enabled, complete the process:

      ../../../../_images/DeviceLogin-4.png
    4. Then put the code shown in the terminal in the first step and click Next (in this example code would be D56MS8244):

    ../../../../_images/DeviceLogin-2.png
    1. Approve the required permissions for PnP.Powershell:

    ../../../../_images/DeviceLogin5.png
    1. Get back to the terminal and check that you are connected:

    ../../../../_images/DeviceLogin-6.png
    1. You can check the correct result of the command Get-PnPSite:

    pwsh
    Get-PnPSite
    Url CompatibilityLevel
    --- ------------------            https://baculaenterprise.sharepoint.com 15
    
  1. Create your new site using the following commands. Depending on the site kind, you will need to utilize one command or another.

    1. Site kind is determined by @BaseSiteTemplate value which you can find in your template

    2. Replace $var with appropriate values:

      1. newSiteName → NonExistingSiteName in your tenant

      2. newSiteUrl → https://tenantname.sharepoint.com/sites/newSiteName

      3. newSiteOwner → anyadminuser@yourtenant.com

      4. newSiteTimeZone → RegionalSettings > TimeZone attribute value from your template

      5. newSiteLocale → RegionalSettings > LocaleId attribute value from your template

      6. newSiteKind → Commented BaseSiteTemplate attribute value from your template

    3. TeamSite

    pwsh
    New-PnPSite -Type TeamSite -Title $newSiteName -Alias $newSiteName -Description "-" -Owners
    $newSiteOwner -Wait
    
    1. CommunicationSite

    pwsh
    New-PnPSite -Type CommunicationSite -Title $newSiteName -Url $newSiteUrl -Owner $newSiteOwner
    
    1. Other:

    pwsh
    New-PnPTenantSite -Title $newSiteName -Url $newSiteUrl -Owner $newSiteOwner -TimeZone
    $newSiteTimeZone -Lcid $newSiteLocale -Template $newSiteKind
    
  2. Now you need to connect to your newly created site. Therefore, repeat step 1, but using your new site URL: https://tenantname.sharepoint.com/sites/newSiteName

  3. Once connected to your target tenant, apply the template with these 2 commands:

pwsh
Set-PnPSite -NoScriptSite:$false
Invoke-PnPSiteTemplate -Path /tmp/mytemplate.xml
  1. Once the process ends, if everything was OK, you can disconnect with:

pwsh
Disconnect-PnPOnline
exit

Out of Memory

If you ever face OutOfMemory errors of the Java daemon (you will find them in the m365-debug.err file),

you are very likely dealing with sub-attachments containing big files or you are using a high level of concurrency through internal concurrent_threads parameter and/or parallel jobs. To overcome this situation you can:

  1. Disable the backup of multi-level attachments setting multi_attach=no

  2. Reduce concurrent_threads parameter

  3. Reduce the number of jobs running in parallel

  4. If you cannot do that or you are not using multi-level attachments you should increase JVM memory.

To increase JVM memory that you will need to:

Create a this file: ‘/opt/bacula/etc/m365_backend.conf’.

Below, an example of the contents:

|M365_JVM_MIN=2G |M365_JVM_MAX=8G

Those values will define the MIN (M365_JVM_MIN) and MAX (M365_JVM_MAX) memory values assigned to the JVM Heap size. In this example we are setting 2Gb for the minimum, and 8Gb for the maximum. In general, those values should be more than enough. Please, be careful if you are running jobs in parallel, as very big values and several jobs at a time could quickly eat all the memory of your host.

The ‘/opt/bacula/etc/m365_backend.conf’ won’t be modified through package upgrades, so your memory settings will be persistent.

App Registration Error

When trying to register the bacula-m365-plugin app in a given tenant, the following error can appear in the URL of a resulting page placed on baculasystems.com:

Registration error
Registration error
https://www.baculasystems.com/m365-plugin-auth/common?error=access_denied&error_description=AADSTS650052%
3a+The+app+needs+access+to+a+service+(%27https%3a%2f%2f*.dps.mil%2f%27)+that+your+organization+%27cd77bbc0-999e-
44c5-af9d-c11bea178407%27+has+not+subscribed+to+or+enabled.
+Contact+your+IT+Admin+to+review+the+configuration+of+your+service+subscriptions.%0d%0aTrace+ID%3a+afdc0b36-
bad6-4d1f-8d24-9213439c0a01%0d%0aCorrelation+ID%3a+c73a3631-5573-41ee-8220-cc281ea4882f%0d%0aTimestamp%3a+2021-
04-27+09%3a21%3a55Z&error_uri=https%3a%2f%2flogin.microsoftonline.com%2ferror%3fcode%
3d650052&admin_consent=True&state=12345

If we look carefully into the error contents we will notice the following message:

  • AADSTS650052: The app needs access to a service (https://*.dps.mil.com) that your organization cd77bbc0-999e-44c5-af9d-c11bea178407 has not subscribed to or enabled. Contact your IT Admin to review the configuration of your service subscriptions.

Even if it is not a very clear message, the error is pointing us to the fact that our tenant is missing some kind of access or subscription in order to utilize bacula-m365-plugin app. The contents of the error can vary depending on the specific situation of the target tenant.

However, in general, and particularly for the example shown, the problem comes from the fact that target tenant has no license to utilize Sharepoint Online service. Without this license/permission, the plugin cannot be registered.

Empty Items in Group/Team Site Libraries

When working with the OneDrive module and Group entities you may experience a view of a list of apparently empty elements if navigating through them using Sharepoint views:

Empty file names

This is not an error of this plugin. This is a problem with some kind of views available in the Sharepoint Online service, where list items corresponding to drive items are shown. As here we are dealing with drive items, those list items have no title attached.

In the particular example shown above, a ‘list view’ of a document library is selected. However if we select any other view as ‘Titles’, ‘All Documents’, etc:

View options

We will be able to see folders and files with their corresponding names:

Titles view

Empty Chat messages after restoring Teams

After a successful restore operation with the Teams module, it can happen that accessing the data using the web interface of the M365 services will show the associated channels with empty contents. This is usually associated to some bad functions of implied cookies with the web browsers.

If you experience this situation we recommend to open the information using a ‘private window/tab’ or removing previous cookies, as the information is actually there. In case this strategy does not work, please use the Teams ‘desktop app’ for your OS, where the behavior is much more solid.

Chat messages with not displayed contents in Teams

There are several kind of auto-generated messages that rely on other Microsoft Cloud internal data structure. Examples of this are:

  • Scheduled meetings through chat actions, not associated to calendar (they belong to ‘onlineMeetings’ module, which is not included today in Teams backup)

  • Messages showing that a Tab was added

  • Messages showing contents of external apps

This kind of messages will be restored exactly with the same form that they were got during backup and text contents will be visible. However, as the information linked is attached in the external applications to the old team, that visual content may not be visible as it was in the original message.

Error restoring teams app

Teams apps are external entities controlled by an undetermined number of external provides. When adding an app to a team, a process belonging to that provider takes place. That process is invoked through the plugin restore process when restoring teams with apps. Sometimes that external process can return unexpected results.

The M365 Plugin will show those situations with a propper message in the joblog that is not treated as error, and the restore will succeed in most cases.

As this is not managed by Bacula Systems or MS Graph API, the approach is just to show the result as it is, when it is not successful. However, please note that the app itself,

in most cases, will be also successfully restore even if that situation happens.

Error with Teams App after restoring a Planner plan

Teams desktop applications use cache data that can interfere sometimes with the real data. Restoring a planner plan can be an example. When you try to open a restored planner tab you may see an error message. However, please just click on the reload button and you should see your data without issue.

Orphaned Java Processe After Job Failure

In some circumstances, java processes belonging to the plugin can be left behind. They may be using noticeable CPU and memory resources. It is safe to remove them by sending an appropriate signal.

Such orphaned processes should not occur with Bacula Jobs finishing without problems.

A way to identify such processes would be to look for java processes running the plugin .jar file either when no such job is actually active, or when no such job of a similar age exists. For example

Identify Old Java Processes
[root@bsys-demo-m365] ~ # ps -o pid,stime,etime,args -wwC java | grep -E 'bacula-m365-plugin-.*'
 370270 Jul04 50-04:48:36 java -Xms512M -Xmx4G -jar /opt/bacula/lib/bacula-m365-plugin-4.4.9.jar --mode=bacula --path=/opt/bacula/working --log=/opt/bacula/working/m365/m365-debug.log
 370403 Jul04 50-04:42:10 java -Xms512M -Xmx4G -jar /opt/bacula/lib/bacula-m365-plugin-4.4.9.jar --mode=bacula --path=/opt/bacula/working --log=/opt/bacula/working/m365/m365-debug.log

If no m365 job is running since July 04, it would be safe to kill -9 370270 370403.

In newer versions of the plugin, the actual Bacula Job running one of the java processes will be indicated in a command line option of the java process, and the Job report will indicate the process id of the java process started.