Note
You can download this article as a PDF
Microsoft 365 Plugin
Overview
This whitepaper presents how to protect the most relevant elements of Microsoft 365 services using Bacula Enterprise.
Features
The Bacula Enterprise Microsoft 365 Plugin is a very easy to deploy and configure plugin supporting the following M365 services:
OneDrive
Emails
Mailbox settings
Sharepoint Online
Calendars
Contacts
OneNote
Tasks
Teams
Chats
It is shipped with advanced parallelization, resiliency, and flexibility features in addition to covering most of the possible M365 use case scenarios. A full feature list is presented below:
Common features
Microsoft Graph API based backups
Multi-service backup in the same task
Multi-service parallelization capabilities
Multi-thread single service processes
Automatic parallelization of fetching processes
Generation of user-friendly report for restore operations
Network resiliency mechanisms
Latest Microsoft Authentication mechanisms
Discovery/List/Query capabilities
Restore objects to Microsoft 365
To original entity
To any other entity
To a different tenant (cross-tenant restore)
Restore any object to filesystem
Owner data protection feature:
Notify data owner about restore actions of his data
Do not proceed further until he enters his M365 credentials
Incremental & Differential backups
Advanced delta function for improved performance (for selected services)
Backup and Restore of Exchange Online Mailboxes
Mailfolder, message and attachment granularity for restore
Email addresses and mailfolders selection capabilities for backup
Mailbox settings protection
Folder rules protection
Restore objects to Microsoft 365 or to any file-system
Restore MIME messages to any filesystem
Incremental & Differential backup
Support for user mailboxes and shared mailboxes
User categories protection
Fully indexed information into Bacula Catalog
Advanced search capabilities for restore operations
Ability to filter input data by date
Ability to exclude message fields from backup or from index
Exclude private or spam messages through powerful filtering capability
Export to PST format
Backup and Restore of OneDrive for Business & Sharepoint Document libraries
Backup and Restore of User drives
Backup and Restore of Group drives
Backup and Restore of Sharepoint document libraries
Include/Exclude system libraries
Include/Exclude hidden libraries
Backup main entity drive unit, but also any other unit
Advanced selection capabilities
Target entities (Users/Groups/Sites)
List Include/List Exclude/Regex include/Regex Exclude…
Folder selection capabilities for backup
List Include/List Exclude
File selection capabilities for backup
Regex include/Regex Exclude…
Drive unit selection capabilities
Folder and file granularity for restore
Computed hash check at backup and restore time
Backup and restore of permissions shares
Backup and restore of shared elements to each entity
Backup and restore of OneDrive file versions
Backup and Restore of Sharepoint Sites
Backup of Site Objects (MS Graph Object)
Backup Site Sharing permissions
Backup of full Site Templates (PnP Powershell Provisioning):
Site metadata
Lists metadata
ListItems metadata
WebPages metadata
DocumentLibraries metadata
Support for Sub-Sites
Restore backed up sites as new sites using Site Template (PnP Powershell Provisioning)
Backup/Restore Lists Objects (MS Graph Object)
Backup/Restore ListItems (MS Graph Object)
Backup/Restore Document Libraries Drive Items
Backup and Restore of Contacts/People
Backup/restore Contacts
Backup/restore groups of contacts
Backup Organizational contacts (Read-only)
Backup and Restore of Tasks
Backup/restore User todo tasks
Backup/restore Teams Planner tasks
Backup and Restore of Calendars
Backup/Restore user calendars
Backup/Restore user groups of calendars
Backup/Restore groups calendar
Calendar permissions
Backup/Restore Events
Support for Attachments
File Attachments
Reference Attachments
Item Attachments (including backup of MIME objects)
Backup and Restore of Onenote
Backup/Restore of Notebooks
User notebooks
Site notebooks
Group notebooks
Backup/Restore of SectionGroups/Sections
Backup/Restore of Pages
Support for Page resources: Images and files
Backup and Restore of Teams
Backup/Restore of Team
Public and Private Teams
Team entity
Team members and roles
Team installed apps
Backup/Restore of Team Channels
Public and Private Channels
Channel entity
Channel members
Channel tabs
Channel messages
Channel messages hosted contents
Backup and Restore of Chats
Backup/Restore of Chat
Chat entity
Chat installed apps
Chat tabs
Chat messages
Chat messages hosted contents
Requirements
Microsoft 365 Personal, Family, Microsoft Home & Student subscriptions are not supported for backup/restore purposes.
It is necessary to have full administrative access to the target Tenant to protect in order to provide the required permissions to the Azure Application linked to this Bacula Enterprise Microsoft 365 Plugin.
Currently the plugin must be installed on a Linux based OS (RH, Debian, Ubuntu, SLES ..) where a Bacula Enterprise File Daemon is installed. Bacula Systems will address support for running this plugin on a Windows platform in a future version.
The OS where the File Daemon is installed must have installed Java version 8 or above.
If the Sharepoint module is going to be used, the OS where File Daemon is installed must also have the following packages installed:
PowerShell v7.2.1 or above
PnP Powershell v1.9.0 or above
Memory and computation requirements completely depend on the usage of this plugin (parallelization, environment size, etc). However, it is expected to have a minimum of 4GB RAM in the server where the File Daemon is running. By default, every job could end up using up to 512Mb of RAM in demanding scenarios (usually it will be less). However, there can be particular situations where this could be higher. This memory limit can be adjusted internally (see Out of Memory). Refer to the Scope section below for any service specific requirements.
Why Protect Microsoft 365?
This is a common question that arises frequently among IT and Backup professionals, so it is important to have a clear picture of it. It is true that Microsoft offers some services intended to prevent data loss:
As with any cloud data, Microsoft 365 data is geo-replicated using Azure cloud to several destinations automatically and transparently. Therefore, complete data loss because of hardware failures are very unlikely to happen.
Data Loss Protection service: Policy based services capable of detecting filtered content and act upon it encrypting it or modifying it in order to protect it (remove headers, etc). This is not a backup tool, is a service to prevent undesired actions to the content stored in Microsoft 365 (for example sharing confidential information with the wrong people).
Retention policies of Microsoft 365: Microsoft retains a maximum of 30 days of deleted information from active subscriptions. Therefore it is possible to recover accidental deleted items inside that period. For more information:
There are no other protection mechanisms for data protection. Below is a listing of challenges not covered:
No Ransomware protection: If data suffers an attack and becomes encrypted, data is lost.
No malicious attacker protection: If data is deleted permanently, data is lost.
No real point-in-time recovery for Exchange 365, and recoveries of partially deleted files are limited to 30 days.
Point in time recovery for OneDrive/Sharepoint limited to 30 days from deletion.
It is not possible to align data protection of Microsoft 365 services to general retention periods or policies longer than 30 days.
No automated way to extract any data from the cloud to save it in external places (this could lead to eventual compliance problems)
Scope
Bacula Enterprise Microsoft 365 Plugin is applicable on environments using any enterprise plan where target services of this plugin are included:
Office 365 platform
Exchange Online
OneDrive for Business
Sharepoint Online
The lack of any of the services in your subscription could lead to authentication problems (See troubleshooting section App Registration Error).
For more details about Microsoft 365 plans, visit these links:
This paper presents solutions for Bacula Enterprise version 12.8 and later, and is not applicable to prior versions.
Note
Important considerations
Before using this plugin, please carefully read the elements discussed in this section.
Onedrive Recycle Bin
Onedrive Recycle Bin cannot be protected with this plugin. This is a Microsoft limitation coming from their exposed REST APIs, where it is not possible to access Recycle Bin information. If future versions of Microsoft REST APIs officially include this function, Bacula Systems will include it as a new feature of this plugin.
Empty files
In general, empty files (files with 0 byte contents) are simply not backed up by Microsoft 365 plugin. In particular, email attachments or onedrive files will show a message in the joblog to inform about empty files detected and so not processed.
Backup of Attachments and Files
In general, this plugin backups two types of information:
Objects
Files
Objects are elements representing some entity in Microsoft 365 such as a calendar event, a contact, an email, a team, etc.
Files are attachments, hosted contents or OneDrive files.
While objects are directly streamed from memory to the backup engine, files need to be downloaded to the FD host before being sent. This is done in order to make some metadata checks and to improve overall performance, as this way operations can be parallelized. Every file is removed just after being completely downloaded and sent to the backup engine.
The path used for this purpose is established by the ‘path’ plugin variable, that usually is set up in the m365_backend script with the value: /opt/bacula/working
Inside the path variable, a ‘spool’ directory will be created and used for those temporary download processes.
Therefore, it is necessary to have at least enough disk space available for the size of the largest file in the backup session. If you are using concurrency between jobs or through the same job (by default this is the case through the concurrent_threads=5 parameter), you would need at least that size for the largest file multiplied by the number of operations in parallel you run.
Accurate Mode and Virtual Full Backups
Accurate mode and Virtual Full backups are not supported. These features will be addressed in future versions of this plugin.
Site Template Provisioning
Microsoft can modify at any time (and they do it frequently) the properties, structure or limitations of any element belonging to a site kind (team site, communication site, project site, etc) whicn can cause the restore provision process to fail. It is even common to see native structures of Sharepoint Online trigger errors with particular items/values and this error comes from the Sharepoint Online API.
As a result, is not possible to guarantee that a given template is going to work in a provision process. Many times, the template might need to be partially modified before being applied (removing some particular item, removing some particular column..). That part of the process is not possible to predict or automate, so we strongly recommend frequent testing of site restore processes to detect any need of manual restore processes before a real restore is required.
Particular List/ListItems Restores
It is possible to select and restore a particular list or listItems, without restoring an entire Sharepoint Site, and pointing the operation to an existing Site. This is done using the Graph API, but as we have previously exposed, this API has many limitations with the Sharepoint API. One of the limitations is that lists or list items with advanced elements (location, images, etc) are not supported. Therefore this feature should only be used for user created lists implying simple elements like texts, checks, dates, etc.
Site Backups In Parallel
Backing up sites in parallel from the same FD is not supported.
Email Limitations
The following items associated to Outlook elements (Exchange Online service) are not supported due to the fact they are currently not accessible using the APIs this plugin relies on (MS Graph API and PnP Powershell):
Sharing permissions associated to MailFolders
Public Folders
Onenote Limitations
The following items associated to Onenote elements are not supported due to the fact they are currently not accessible using the APIs this plugin relies on (MS Graph API and PnP Powershell):
Sharing permissions associated to Notebooks
Shared elements from one entity to another one. To protect them it is necessary to make it always from the source entity.
Notebook 5000 files limitation
Onenote notebooks can only store up to 5000 different items into their associated OneDrive Libraries, this a M365 limitation.
When they surpass that limit they become unaccessible, not only for backup but also for accessing them through any other Microsoft native interface.
When a job finds this kind of notebooks, an error message will be generated and the notebook won’t be included in the backup.
You can get more information about this limit and how to solve it in the following Microsoft documentation link:
External Personal Calendars
External calendars (as Google Calendar) connected to Microsoft accounts are not supported. They are external elements not exposed through MS APIs, so M365 Plugin cannot protect that information.
Calendar Events icons
Little icons that appear on the left of a calendar event title (usually auto-generated by the Outlook service) cannot be backed up, as they are not exposed through the MS Graph API the plugin is relying on.
Chats types
For the chats module, only types ‘group’ or ‘one_on_one’ can be backed up. Other types like: ‘meeting’ or ‘unknownFutureValue’ will be ignored.
MS Cloud APIs General Disclaimer
MS Cloud APIs are Microsoft property and they can change or evolve at any time. In particular, the Graph API is actively developed, containing new features every week, even if the version number of the service (1.0) is not changed as a result of any of those additions:
This situation is significantly different from traditional on-premise software, where each update is clearly numbered and controlled for a given server, so applications consuming that software, can clearly state what is offered and what are the target supported versions.
Microsoft is committed to trying to not break any existing functionality that could affect external applications. However, this situation can happen and therefore, cause some ocasional problems with this plugin. Bacula Systems controls this with an advanced automatic monitoring system which is always checking the correct behavior of existing features, and will react quickly to that hypothetical event, but please be aware of the nature and implications of this kind of cloud technologies.
Architecture
Bacula Enterprise Microsoft 365 Plugin is using the Microsoft Graph API to perform almost all of its operations. Therefore, the plugin is working at the maximum granularity that the service provides.
All the information is obtained using secure and encrypted HTTPS queries to Microsoft 365 from the File Daemon where the plugin is installed. All the requests are performed over the following endpoints:
Graph API to manage all the services (except Sharepoint Online): https://graph.microsoft.com
Login endpoint: https://login.microsoftonline.com/
Sharepoint Online service endpoint: https://{tenantname}.sharepoint.com/
To get more information about Graph API, visit: https://learn.microsoft.com/en-us/graph/overview
The plugin will contact an Azure registered app named bacula-m365-plugin and will use it as a bridge to download the required data or objects during the time of a backup and send them to the Storage Daemon. Conversely, the plugin will receive them from an SD and perform uploads as needed during a restore.
The implementation is done through a Java Daemon, therefore Java is a requirement in the FD host. For more information about the bacula-m365-plugin, please, consult Authorization section.
Below is a simplified vision of the architecture of this plugin inside a generic Bacula Enterprise deployment:
Listed below is the information that can be protected using this plugin:
Email
Common MailFolders (Inbox, Deleted Items, Drafts..)
User MailFolders & SubFolders
Messages & EventMessages
Attachments (ItemAttachments, FileAttachments and ReferenceAttachments)
Mailbox settings
Folder Rules
OneDrive
Onedrive for Business of Users, for each drive unit
Folders
Files
File Versions
Group libraries, for each drive unit
Folders
Files
File Versions
Sharepoint site libraries, for each drive unit
Folders
Files
File Versions
Shared permissions (direct access, share links, expiration times..)
SharedWithMe Objects
Sharepoint
Pnp Site template
Site metadata
Lists metadata
ListItems metadata
WebPages metadata
Site Object
Site sharing permissions
Lists Objects
ListItem Objects
Contacts/People
Contact object
Name of Groups of contacts
Organizational contact object
Tasks
User Todo lists
User Todo tasks
Calendars
Calendar objects
Calendar group objects
Calendar permissions
Events objects
Attachments (ItemAttachments, FileAttachments and ReferenceAttachments)
MIME objects where possible
Notebooks
Notebook objects
Section objects
SectionGroup objects
Pages
Page contents (Html formatted) - Page resources
Page image files
Page object files (any other file apart from images)
Teams
Team objects
Team settings
Team members and associated roles
Team installed apps
Channel objects
Channel tabs
Channel chat messages
Chat messsages hosted contents
Chat
Chat objects
Chat installed apps
Chat tabs
Channel chat messages
Chat messsages hosted contents
All the information of each object is stored in JSON format (except for Pnp site template, which is stored in XML), preserving all their original values. When the plugin works with objects containing additional data (MIME files for messages, data for attachments and files of OneDrive, etc), that data is also backed up.
Services and Features
In this section we will dig into how this plugin behaves for each particular service, describing special features and and behaviors that require an extended description.
Special features
In the following section, special features and behaviors are detailed.
Installation
The Bacula File Daemon and the Microsoft 365 Plugin need to be installed on the host that is going to connect to the cloud based services. The plugin is implemented over a Java layer, therefore it can be deployed on the platform better suited for your needs among any of the officially supported platforms of Bacula Enterprise (RHEL, SLES, Debian, Ubuntu, Windows, etc).
Please, note that you may want to deploy your File Daemon and the plugin on a virtual machine directly deployed in Azure Cloud in order to reduce the latency between it and the Microsoft Graph API and experience modest performance gains. However, this option is only recommended in case of having a very stable connection between the File Daemon and the Storage Daemon, which means a special, dedicated connection with Azure or when the Storage Daemon is deployed in the cloud as well. This is not usually the case - thus the data needs to traverse the Internet with standard and shared connections from the File Daemon to the Storage Daemon. Disconnections while transmitting data between these two daemons may make jobs fail and cause large timeouts that are difficult to manage and stabilize. The best strategy strategy for this kind of scenarios is to deploy the File Daemon and the Plugin to the same host as the destination Storage Daemon is installed. This way, disconnections between the two daemons will not happen, while disconnections between the FD and M365 will be transparently recovered (when possible), so jobs will finish successfully.
The system must have Java >= 8 installed (openjdk-8-jre for example) and the Java executable should be available in the system PATH.
The Sharepoint module depends on the Powershell and the PnP Powershell modules. Therefore, they also need to be installed before installing the Bacula packages (see section PnP.Powershell below).
PnP.Powershell
Install PowerShell
In order to install PowerShell it is necessary to follow the instructions for the particular OS involved which may be found in the github site of the project:
For example, if using Debian, these are the instructions:
The procedure is shown below, some dependencies are installed, a repository is added, and then the apt package manager is used to install the tool:
# Install system components
sudo apt-get update
sudo apt-get install -y curl gnupg apt-transport-https
# Import the public repository GPG keys
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
# Register the Microsoft Product feed
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-debian-stretch-prod stretch
main" > /etc/apt/sources.list.d/microsoft.list'
# Update the list of products
sudo apt-get update
# Install PowerShell
sudo apt-get install -y powershell
# Start PowerShell
pwsh
Install PnP.Powershell
In order to install the PnP.Powershell module, once Powershell is already installed, we simply need to run the command:
Install-Module -Name “PnP.PowerShell”
Inside a Powershell session. Below we provide an example:
yourworkstation:~$ pwsh
PowerShell 7.2.0
Copyright (c) Microsoft Corporation.
https://aka.ms/powershell
Type 'help' to get help.
PS /home/john> Install-Module -Name "PnP.PowerShell"
Bacula Packages
We are taking Debian Buster as the example base system to proceed with the installation of the Bacula Enterprise Microsoft 365 Plugin. In this system, the installation is most easily done by adding the repository file suitable for the existing subscription and the Debian version utilized. An example would be /etc/apt/sources.list.d/bacula.list with the following content:
# Bacula Enterprise
deb https://www.baculasystems.com/dl/@customer-string@/debs/bin/@version@/buster-64/ buster main
deb https://www.baculasystems.com/dl/@customer-string@/debs/m365/@version@/buster-64/ buster m365
After that, a run of apt update is needed:
apt update
Then, the plugin may be installed using:
apt install bacula-enterprise-m365-plugin
The plugin has two different packages implied that should be installed automatically with the command shown:
bacula-enterprise-m365-plugin
bacula-enterprise-m365-plugin-libs
Alternately, manual installation of the packages may be done after downloading the poackages from your Bacula Systems provided download area, and then using the package manager to install. An example:
dpkg -i bacula-enterprise-*
The package will install the following elements:
Jar libraries in /opt/bacula/lib (such as bacula-m365-plugin-x.x.x.jar and bacula-m365-plugin-libs-x.x.x.jar). Please note that the version of those jar archives is not aligned with the version of the package. However, that version will be shown in the joblog in a message like ‘Jar version:X.X.X’.
Note
Version in Jar Name
Version is included in the name of .jar files from Bacula Enterprise version 14.0.4. Before that, libraries were composed by: bacula-m365-plugin.jar, bacula-meta-plugin-1.0.0.jar and bacula-m365-plugin-libs-1.0.0.jar
Plugin connection file (m365-fd.so) in the plugins directory (usually /opt/bacula/plugins)
Backend file (m365_backend) that invokes the jar files in /opt/bacula/bin. This backend file searches for the most recent bacula-m365-plugin-x.x.x.jar file in order to launch it, even thought usually we should have only one file.
A collection of powershell files used in the Sharepoint module in /opt/bacula/bin.
Configuration
Fileset Configuration
Once the plugin is successfully authorized, it is possible to define regular filesets for backup jobs in Bacula, where we need to include a line similar to the one below, in order to call the M365 Plugin:
FileSet {
Name = FS_M365
Include {
Options {
signature = MD5
...
}
Plugin = "m365: <m365-parameter-1>=<m365-value-1> <m365-parameter-2>=<m365-value-2> ..."
}
}
It is strongly recommended to use only one ‘Plugin’ line in every fileset. The plugin offers the needed flexibility to combine different modules backup inside the same plugin line. Different tenants, in case of existing, should be using different filesets and different jobs.
Below sub-sections list all the parameters you can use to control M365 Plugin behavior.
In this plugin, any parameter allowing a list of values can be assigned with a list of values separated by ‘,’.
Note
Starting with version 14.0 of Bacula Enterprise you can also perform fileset configuration directly using a dedicated BWeb M365 Wizard. Please go to section BWeb Management Console to see more details.
Common parameters
These parameters are common and applicable to all the modules of the M365 Plugin.
Option |
Required |
Default |
Values |
Example |
Description |
---|---|---|---|---|---|
abort_on_error |
No |
No |
No, Yes |
Yes |
If set to Yes: Abort job as soon as any error is found with any element. If set to No: Jobs can continue even if it they found a problem with some elements. They will try to backup or restore the other and only show a warning |
config_file |
No |
The path pointing to a file containing any combination of plugin parameters |
/opt/bacula/m365.settings |
Allows to define a config file where configure any parameter of the plugin. Therefore you don’t need to put them directly in the Plugin line of the fileset |
|
log |
No |
/opt/bacula/working/m365/m365-debug.log |
An existing path with enough permissions for File Daemon to create a file with the provided name |
/tmp/m365.log |
Generates additional log in addition to what is shown in job log. This parameter is included in the backend file, so, in general, by default the log is going to be stored in the working directory. |
debug |
No |
0 |
0, 1, 2, 3, 4, 5, 6, 7, 8, 9 |
Debug level. Greater values generate more debug information |
Generates the working/m365/m365-debug.log* files containing debut information which is more complete with a greater debug number |
path |
No |
/opt/bacula/working |
An existing path with enough permissions for File Daemon to create any internal plugin file |
/mnt/my-vol/ |
Uses this path to store metadata and temporary files |
tenant |
Yes |
A valid tenant id string |
57uia43-d107-17a2-a2g2-aa53c10tdahc |
The tenant ID where the plugin will connect to in order to run backups or restores. Please, check section 5.1 of this paper for more information |
|
objectid |
Yes |
String representing the objectid related to bacula-m365-plugin Azure app and the provided tenant id |
56ddf1h9-eb5d-42nf-bac7-7b019fd284g5 |
The object ID of the plugin app of the plugin in Azure, once this is registered in the target tenant. Please, check section 5.1 of this paper for more information |
|
appid |
No |
String representing the appid associated to bacula-m365-plugin-standalone Azure app registered in the configured tenant id |
89tt4hu7-r4h7-kied-56gu-0895kf94jr9d |
A valid appid string associated to bacula-m365-plugin-standalone Azure app registered in the configured tenant id. Please, check section 5.1 of this paper for more information |
|
secret |
No |
String representing the associated appid secret |
Jn8.lU-B.3P5gIRTGY6M.Xl3e29oQ6Xaf~ |
The secret associated to the m365 Azure app corresponding the configured appid string. Please, check section 5.1 of this paper for more information |
|
token_cache_file |
No |
token_cache.json |
A file name located in a valid existing path (it’s possible to put only the name of the file and ‘working’ dir will be used) |
my_cache_file.json |
The path that will be used to store the login cache for the device code flow authenticated users, wich is relative to the working tenant folder (working/tenant-name/token_cache.json) |
service |
No |
email, drive, sharepoint, contact, calendar, onenote, tasks, teams, chat (list parameter: it can contain 0, 1 or more elements separated by ‘,’) |
drive |
Establish the service or services that will be backed up. If this is not set, the plugin will try to backup all supported services. It is recommended to split the work among different jobs when several services need to be applied. Therefore, even if this field is not required, it is strongly recommended to use it in every backup job. |
|
owner_restore_protection |
No |
No |
Yes, No |
Yes |
Enable owner restore protection feature, where the owner of the data being restored will be notified with an email and the restore job will be blocked until he/she enters a code and his/her M365 credentials to approve the operation |
proxy_host |
No |
String representing DNS Name or IP address of the http(s) proxy |
myproxy.example.com |
Set up a proxy to make any plugin HTTP connection |
|
proxy_port |
No |
Integer |
3981 |
Set up the proxy port |
|
proxy_user |
No |
String of proxy user |
admin |
Set up the proxy user |
|
proxy_password |
No |
String of proxy password |
myPass123 |
Set up the proxy user password |
Note
The path option is adjustable while M365 jobs are in progress. The running jobs will continue to use the current spool area, and new jobs will use the new spool area defined by the path option.
If specifying more than one service in the fileset, please note that they will run in parallel. Because of that, we recommend to decrease a little the concurrency.
Below some multi-services fileset examples:
FileSet {
Name = fs-m365-adelev-user
Include {
Options { signature = MD5 }
Plugin = "m365: service=drive,email,calendar,contact,onenote,tasks,chat tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 user=adelev@baculaenterprise.onmicrosoft.com concurrent_threads=2"
}
}
FileSet {
Name = fs-m365-devteam-group
Include {
Options { signature = MD5 }
Plugin = "m365: service=drive,calendar,onenote,tasks,teams tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 group=DevTeam concurrent_threads=2"
}
}
FileSet {
Name = fs-m365-mysite-site
Include {
Options { signature = MD5 }
Plugin = "m365: service=sharepoint,drive,onenote tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 site=MySite sharepoint_include_drive_items=no sharepoint_system_include=yes drive_system_include=yes concurrent_threads=2"
}
}
Note
Activating proxy mody will route all the requests through the proxy. However, it is needed DNS resolution to be working separately. Hence, the client where the FD is running will need to have a proper and working DNS Server configured. Other option is to setup the /etc/hosts file manually with the IP addresses of Microsoft authentication servers, but they could change over time. At the time of writing they are: 40.126.31.2 login.microsoftonline.com 20.190.160.131 login.microsoft.com
Advanced common parameters
Following parameters are common to all M365 modules (and even with some other plugins), but are advanced ones. They should not be modified in most common use cases.
Option |
Required |
Default |
Values |
Example |
Description |
---|---|---|---|---|---|
stream_sleep |
No |
1 |
Positive integer (1/10 secconds) |
5 |
Time to sleep when reading header packets from FD and not having a full header available |
stream_max_wait |
No |
120 |
Positive integer (seconds) |
360 |
Max wait time for FD to answer packet requests |
time_max_last_modify_log |
No |
86400 |
Positive integer (seconds) |
43200 |
Maximum time to wait to ovewrite a debug log that was marked as being used by other process |
logging_max_file_size |
No |
50MB |
String size |
300MB |
Maximum size of a single debug log fileGenerates the working/m365/m365-debug.log* files containing debut information which is more complete with a greater debug number |
logging_max_backup_index |
No |
25 |
Positive integer (number of files) |
50 |
Maximum number of log files to keep |
log_rolling_file_pattern |
No |
m365.log.%d{dd-MMM}.log.gz” |
No, Yes |
Yes |
Log patter for rotated log files |
split_config_file |
No |
= |
Character |
: |
Character to be used in config_file parameter as separator for keys and values |
opener_queue_timeout_secs |
No |
1200 |
Positive integer (seconds) |
3600 |
Timeout when internal object opener queue is full |
publisher_queue_timeout_secs |
No |
1200 |
Positive integer (seconds) |
3600 |
Timeout when internal object publisher queue is full |
The internal plugin logging framework presents some relevant features that we are going to describe:
The “.log” files are rotated automatically. Currently each file can be 50Mb at maximum and the plugin will keep 25 files.
This behavior can be changed using the internal advanced parameters: logging_max_file_size and logging_max_backup_index
The “.err” file can show contents even if no real error happened in the jobs. It can show contents too even if debug is disabled. This file is not rotated, but it is expected to be a small file in general. If you still need to rotate it, you can include it in a general rotating tool like ‘logrotate’.
Backups in paralel and also failed backups will generate several log files. For example: m365-debug-0.log, m365-debug-1.log…
Tuning parameters
These set of parameters are common to all modules and they are advanced ones. They should not be modified in general. They can be used to tune the behavior of the plugin to be more flexible in particular bad network environments or when significant job concurrency is happening, etc.
Option |
Required |
Default |
Values |
Example |
Description |
---|---|---|---|---|---|
backup_queue_size |
No |
100 |
0-500 |
1 |
Number of maximum enqueued internal operations between service static internal threads (there are 3 communicating through queues with the set size: service fetcher, service opener and general publisher to bacula core). This could potentially affect graph api concurrent requests and consequently, Graph throttling. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel |
concurrent_threads |
No |
10 |
0-100 |
1 |
Number of maximum concurrent backup threads running in parallel in order to fetch or open data for running download actions. This means every service fetcher and service opener will open this number of child concurrent threads. This will affect graph api concurrent requests. Graph API can throttle requests depending on a variety of circumstances, but this parameter impacts it directly. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel. If you want to have a precise control of your parallelization through different jobs, please set up this value to 1. Please be careful also with the memory requirements, multi-threaded increases very significantly memory consumption per job |
concurrent_listing_threads |
No |
5 |
0-20 |
1 |
Number of maximum concurrent backup page listing threads running in parallel in order to fetch sets of data for some modules. Currently it’s only used in the email module. This parameter will also affect graph api concurrent requests. Graph API can throttle requests depending on a variety of circumstances, but this parameter impacts it directly. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel. If you want to have a precise control of your parallelization through different jobs, please set up this value to 1. Please be careful also with the memory requirements, multi-threaded increases very significantly memory consumption per job |
graph_timeout |
No |
9000 |
Positive integer (milliseconds) |
60000 |
Graph call timeout inside HttpClient |
graph_read_timeout |
No |
300 |
Positive integer (milliseconds) |
30000 |
Graph read timeout inside HttpClient |
graph_retries |
No |
5 |
Positive integer (number of retries) |
10 |
Graph number of retries for retry-candidate requestsInclude some stats information in the joblog. Useful to measure task times |
graph_retry_delay |
No |
5 |
Positive integer (seconds) |
10 |
Graph delay between retries |
general_network_retries |
No |
5 |
Positive integer (number of retries) |
10 |
Number of retries for the general M365 external retry mechanism |
general_network_delay |
No |
50 |
Positive integer (seconds) |
100 |
General M365 Plugin delay between retries |
throttled_wait_time |
No |
300 |
Positive integer (seconds) |
600 |
Extra wait time for throttled situations where the timeout is not provided or not got from MS API. In onenote module this is multiplied by 2. Once MS throttles a request is better to not retry too soon or the changes to continue with rejected requests will increase significantly |
stats |
No |
No |
No, Yes |
Yes |
Include some stats information in the joblog. Useful to measure task times |
Note
graph_list_page_size had default value 500 before BEE 14.0.7. A higher value for this parameter can improve the performance at it reduces the number of API calls done to M365 service. However, the service can also be overloaded and return more HTTP 503 errors (Bad Gateway), especially for the email module. Starting from version 16.0.3, default values for backup_queue_size and concurrent_threads have been increased, also the allowed ranges.
Entity parameters
The following list of parameters are commonly shared through any module used into the same fileset line and are intended to select the target entities to backup. Every module subsection mentions what entities are supported too.
Option |
Required |
Default |
Values |
Example |
Services |
Description |
---|---|---|---|---|---|---|
user |
No |
Valid email addresses of existing users on the selected tenant separated by ‘,’ |
email, drive, contact, calendar, onenote, chat, tasks |
Backup mailboxes, drive unit spaces, categories, or whatever service is selected of this list of users. If no user is provided: - The backup will be done for all accessible users of the given tenant if no other entry has value either (group or site parameters) |
||
user_exclude |
No |
Valid email addresses of existing users on the selected tenant separated by ‘,’ |
email, drive, contact, calendar, onenote, chat, tasks |
Exclude selected mailboxes or onedrive spaces. If this is the only parameter found for selection, all elements will be included and this list will be excluded |
||
user_regex_include |
No |
Valid regex |
.*@management\.mydomain.com |
email, drive, contact, calendar, onenote, chat, tasks |
Backup matching user mailboxes or onedrive spaces. Please, only provide list parameters (user + user_exclude) or regex ones. But do not try to combine them |
|
user_regex_exclude |
No |
Valid regex |
.*@guests\.mydomain.com |
email, drive, contact, calendar, onenote, chat, tasks |
Exclude matching user mailboxes or onedrive spaces from the selection. Please, only provide list parameters (user + user_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded |
|
site |
No |
Valid names of existing user sharepoint sites on the selected tenant separated by ‘,’ |
Communication site, Dev site |
drive, sharepoint, onenote |
Backup onedrive site library space (OneDrive service) belonging to this list of sharepoint sites or the site itself (Sharepoint service). If no site name is provided: - OneDrive: user and site variables will be checked. If no one has value either, backup will be done for all accessible users, groups or sites of the given tenant - Sharepoint: all accessible sites of the tenant will be backed up |
|
site_exclude |
No |
Valid names of existing user sharepoint sites on the selected tenant separated by ‘,’ |
Test site |
drive, sharepoint, onenote |
Exclude listed sites from backup. If this is the only parameter found for selection, all elements will be included and this list will be excluded |
|
site_regex_include |
No |
Valid regex |
.*site |
drive, sharepoint, onenote |
Backup matching sites. Please, only provide list parameters (site + site_exclude) or regex ones. But do not try to combine them |
|
site_regex_exclude |
No |
Valid regex |
Test.* |
drive, sharepoint, onenote |
Exclude matching sites from the selection. Please, only provide list parameters (site + site_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded |
|
group |
No |
Valid names of existing user groups on the selected tenant separated by ‘,’ |
Dev Team, Bacula Users |
drive, onenote, calendar, teams, tasks |
Backup onedrive sharepoint library spaces or notebooks of this list of groups. If no group is provided, user and site variables will be checked. If no one has value either, backup will be done for all accessible users, groups or sites of the given tenant |
|
group_exclude |
No |
Valid names of existing user groups on the selected tenant separated by ‘,’ |
Monitoring, External |
drive, onenote, calendar, teams, tasks |
Exclude selected group. If this is the only parameter found for selection, all elements will be included and this list will be excluded |
|
group_regex_include |
No |
Valid regex |
.*Management |
drive, onenote, calendar, teams, tasks |
Backup matching group onedrive spaces. Please, only provide list parameters (group + group_exclude) or regex ones. But do not try to combine them |
|
group_regex_exclude |
No |
Valid regex |
.*External |
drive, onenote, calendar, teams, tasks |
Exclude matching groups from the selection. Please, only provide list parameters (group + group_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded |
Backup parameters
Please, check the specific module pages in order to see backup parameters that are applicable only to each of them:
Deprecated parameters
- The following parameters have been deprecated since version 12.8.3, in order to provide more flexibility. General shared parameters have been transformed in specific ones for each service
files -> email_files, drive_files, calendar_files, contact_files
files_exclude -> email_files_exclude, drive_files_exclude, calendar_files_exclude, contact_files_exclude
files_regex_include -> email_files_regex_include, drive_files_regex_include, calendar_files_regex_include, contact_files_regex_include
files_regex_exclude -> email_files_regex_exclude, drive_files_regex_exclude, calendar_files_regex_exclude, contact_files_regex_exclude
multilevel_attach -> email_multilevel_attach, calendar_multilevel_attach
exclude_attachments -> email_exclude_attachments, calendar_exclude_attachments
system_include -> drive_system_include, sharepoint_system_include
version_history -> drive_version_history, sharepoint_version_history
backup_threads -> backup_queue_size
Restore parameters
The plugin is able to restore to the local file system on the server where the File Daemon is running or to the Microsoft 365 environment. The method is selected based on the value of the where parameter at restore time:
Empty or ‘/’ (example: where=/) → Microsoft 365 restore will be triggered
Any other path for where (example: where=/tmp) → Local file system restore will be triggered
When using the Microsoft 365 restore option, the following parameters may be modified by selecting ‘Plugin Options’ during the bconsole restore session:
Option |
Required |
Default |
Values |
Example |
Services |
Description |
---|---|---|---|---|---|---|
destination_user |
No |
Existing email address on the target Azure AD Tenant |
email, drive, contact, calendar, onenote, tasks, chat |
Destination User where restore data will be uploaded. If no user is set, every selected file will be restored in the original account |
||
destination_group |
No |
Existing group name address on the target Azure AD Tenant |
Marketing |
drive, calendar, onenote |
Destination Group where restore data will be uploaded. If no group is set, every selected file will be restored in the original group |
|
destination_site |
No |
Existing site name/display name on the target Azure AD Tenant |
live @ contoso |
drive, sharepoint, onenote |
Destination Site where restore data will be uploaded. If no site is set, every selected file will be restored in the original site |
|
destination_path |
No |
Existing path on the selected user (mailfolder path or onedrive folder path) |
Inbox |
email, drive, contact, calendar, sharepoint |
Destination folder where all selected files to restore will be restored. If no path is set: - If no user is set either, every element will go to its original location - If a user is set using the variable destination_user: - Elements belonging to destination_user will be restored in their original location - Elements belonging to different users than destination_user will be restored in a new folder using the email address of the original user of the element |
|
destination_drive |
No |
Existing drive name or drive id in the destination entity |
documents |
drive |
Destination drive where restore data will be uploaded. If no drive is set, every selected file will be restored in the original drive |
|
send_report |
No |
0 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
1 |
email, drive, contact, calendar, sharepoint, tasks |
Send a report to the user where every restore action is listed (generated emails, generated onedrive files..). - In the email service this will be an email in the user Inbox - In onedrive service this will generate a new text file in the top restore folder |
allow_duplicates |
No |
1 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
0 |
email, contact, calendar |
In Email services indicates if we want to allow to create duplicate emails or if we want to skip emails that are already in place. Important: This is done using the Microsoft Message ID. Therefore, non intuitive situations can happen. For example, if the original message is deleted and we restore several times using this option, we will see duplicate emails anyway |
drive_skip_sharedwitme |
No |
0 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
1 |
drive |
Skip restoring shared with me elements even if they are selected. |
skip_versions |
No |
1 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
0 |
drive |
Skip restoring former file versions (tagged with ‘###date’) even if they are selected. Important: Notice that this parameter is enabled by default, as we consider not restoring file versions the most common case. You need to disable it in order to have this kind of files restored |
restore_share_permissions |
No |
0 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
1 |
drive, calendar |
Restore share permissions of every element in order to regenerate sharing information as allowed identities, shared links, etc. Please note that this parameter is disabled by default.**Important**: Notice that this parameter is disabled by default, as we consider not restoring sharing permissions the most common case. You need to enable it in order to haver shared permissions restored |
drive_send_invitations |
No |
0 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
1 |
drive |
Send email invitations for restored OneDrive shares - Note you need to enable drive_restore_share_permissions in order to this parameter to have any effect |
drive_invitations_message |
No |
String |
I’m sharing with you automatically restored shares |
drive |
Set invitations message for restored OneDrive shares - Note you need to enable drive_restore_share_permissions in order to this parameter to have any effect |
|
calendar_name |
No |
String |
RestoredCalendar |
calendar |
Set the name of restored(s) calendar(s) |
|
sharepoint_list_name |
No |
String |
documentsCopy |
sharepoint |
Set the name of restored(s) sharepoint list(s) |
|
sharepoint_newsite_name |
No |
String |
Restored Site |
sharepoint |
Set the name of restored sharepoint new site |
|
sharepoint_newsite_owner |
No |
String |
sharepoint |
Set the owner of the new restored sharepoint site |
||
sharepoint_local_template_path |
No |
String |
/tmp/my-restores/company meeting.site.template.xml |
sharepoint |
Set the local path of the PnP Sharepoint Site template you want to be using instead of the one contained in the backup itself. |
|
notebook_name |
No |
String |
myRestoredNotes |
onenote |
Set the owner of the new restored sharepoint site |
|
notesection_name |
No |
String |
Technology |
onenote |
Set the name of restored onenote notebook |
|
notesection_group_name |
No |
String |
ForWork |
onenote |
Set the name of restored onenote section |
|
team_name |
No |
String |
MyRestoredTeam |
teams |
Set the name of restored team (it will create also a new associated group) |
|
team_channel_name |
No |
String |
MyRestoredChannel |
teams |
Set the name of restored channel |
|
team_private_channels_mode |
No |
DELEGATED |
DELEGATED, PUBLIC, SKIP |
PUBLIC |
teams |
Set the private channels restore mode |
team_guest_members_enable |
No |
0 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
1 |
teams |
Enalbe restore of team members with role ‘guest’. That kind of external user needs delegated permissions to be added, so potentially the restore could ask for user interaction (of first team owner found) |
chat_topic |
No |
String |
MyRestoredChat |
chat |
Set the topic of restored chat (supported only for group chats) |
|
tasklist_name |
No |
String |
ProjectA |
tasks |
Set the name of restored task list |
|
tasklist_skip_sharedwithme |
No |
1 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
0 |
tasks |
Skip shared with me tasks in the restore process |
plan_name |
No |
String |
RestoredPlan |
tasks |
Set the name of restored planner plan |
|
plan_create_tab |
No |
1 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
0 |
tasks |
Create or not an associated tab inside the associated team |
local_path_json_objects |
No |
{path}/bacula-restores |
String |
/home/me/restores |
onenote, sharepoint |
Set local path to restore objects that cannot be restored directly into the M365 service |
foreign_container_generation |
No |
1 |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
0 |
email, drive, contact, calendar |
Generate a general container (usually a folder) to put inside restored objects coming from different entities. For example, if we restore emails from user a@tenant.com into Mailbox of user b@tenant.com, this option enabled will generate an automatic folder a@tenant.com inside the destination restore folder used over destination user a@tenant.com |
tenant |
No |
A valid tenant id string where some bacula m365 plugin app is registered |
57uia43-d107-17a2-a2g2-aa53c10tdahc |
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat |
Set the destination tenant id for a cross-tenant restore |
|
appid |
No |
Appid of bacula-m365-plugin-standalone app registered in the provided tenant |
56ddf1h9-eb5d-42nf-bac7-7b019fd284g5 |
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat |
Set the destination appid of bacula-m365-plugin-standalone for a cross-tenant restore |
|
objectid |
No |
Object id associated to set up appid |
89tt4hu7-r4h7-kied-56gu-0895kf94jr9d |
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat |
Set the objectid associated to the appid in a cross-tenant restore |
|
secret |
No |
Valid secret associated to appid |
Jn8.lU-B.3P5gIRTGY6M.Xl3e29oQ6Xaf~ |
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat |
Set the secret associated to the appid set in the above parameter |
|
debug |
No |
0 |
0, 1, 2 ,3, 4, 5, 6, 7, 8, 9 |
3 |
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat |
Change debug level |
Automatic Objects Integration
Since Bacula version 16.0.7, a new solution has been introduced, so that each object can be backed up separately with different Jobs to maximize the throughput and the resiliency. It is highly recommended to use this new solution for that purpose - Automatic Object Integration (Scan Plugin). See an example for Microsoft 365Example.
Operations
To learn about backup and restore from a video, click on the image below:
Backup
Microsoft 365 plugin backup configurations currently have one specific requirement in the Job resource. Below we show some examples.
Job Example
The only special requirement with M365 jobs is that Accurate mode backups must be disabled, as this feature is not supported at this time.
Job {
Name = m365-mytenant-backup
FileSet = fs-m365-email-all
Accurate = no
...
}
FileSet Examples
The plugin supports enough flexibility to configure almost any type of desired backup. Multiple Plugin= lines should not be spoecified in he Include section of a FileSet for the M365 Plugin.
Fileset examples for every supported service are linked below. For common purposes, the following two examples show how to configure an external config file or configure the number of threads:
Setup external config file:
FileSet {
Name = FS_M365_DRIVE
Include {
Options {
signature = MD5
}
Plugin = "m365: config_file=/opt/bacula/etc/m365.settings service=drive"
}
}
$ cat /opt/bacula/etc/m365.settings
tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc
objectid=56ddf1h9-eb5d-42nf-bac7-7b019fd284g5
Increase number of threads:
FileSet {
Name = fs-m365-email-adelev
Include {
Options {
signature = MD5
}
Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 user=adelev@baculaenterprise.onmicrosoft.com backup_threads=10"
}
}
More fileset examples for:
Restore
Restore operations are done using standard Bacula Enterprise bconsole commands.
The where parameter controls if the restore will be done locally to the File Daemon’s file system or to the Microsoft 365 service:
where=/ or empty value → Restore will be done over M365
where=/any/other/path → Restore will be done locally to the File Daemon file system
Restore options are described in the Restore parameters section of this document, so here we are going to simply show an example restore session, particularly this example is about OneDrive service:
*restore where=/
First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.
To select the JobIds, you have the following choices:
1: List last 20 Jobs run
2: List Jobs where a given File is saved
3: Enter list of comma separated JobIds to select
4: Enter SQL list command
5: Select the most recent backup for a client
6: Select backup for a client before a specified time
7: Enter a list of files to restore
8: Enter a list of files to restore before a specified time
9: Find the JobIds of the most recent backup for a client
10: Find the JobIds for a backup for a client before a specified time
11: Enter a list of directories to restore for found JobIds
12: Select full restore to a specified Job date
13: Select object to restore
14: Cancel
Select item: (1-14): 5
Automatically selected Client: 127.0.0.1-fd
Automatically selected FileSet: FS_M365_DRIVE
+-------+-------+----------+-------------+---------------------+-------------------+
| jobid | level | jobfiles | jobbytes | starttime | volumename |
+-------+-------+----------+-------------+---------------------+-------------------+
| 11 | F | 190 | 332,978,505 | 2021-01-22 10:39:34 | TEST-2021-01-22:0 |
| 12 | I | 1 | 550 | 2021-01-22 10:43:05 | TEST-2021-01-22:0 |
+-------+-------+----------+-------------+---------------------+-------------------+
You have selected the following JobIds: 11,12
Building directory tree for JobId(s) 11,12 ... +++++++++++++++++++++++++++++++++++++++++++++++
188 files inserted into the tree.
You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.
cwd is: /
$ cd /@m365/baculaenterprise/adelev@baculaenterprise.onmicrosoft.com/drive/root:
cwd is: /@m365/baculaenterprise/adelev@baculaenterprise.onmicrosoft.com/drive/root:/
$ ls
Docs/
pluginTest.drive.deltaLink
sharedWithMe/
test4###v1.0_2020-11-25_153507.html
test4###v2.0_2020-11-25_153507.html
test4###v3.0_2020-12-02_180612.html
test4###v4.0_2020-12-02_180612.html
test4###v5.0_2020-12-03_134422.html
test4###v6.0_2020-12-02_180612.html
test4.html
testlink123###v1.0_2020-11-26_123244.url
testlink123###v2.0_2020-11-26_123244.url
testlink123###v3.0_2020-12-02_180613.url
testlink123###v4.0_2020-12-02_180613.url
testlink123.url
$ mark *
189 files marked.
$ done
Bootstrap records written to /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
The Job will require the following (*=>InChanger):
Volume(s) Storage(s) SD Device(s)
===========================================================================
TEST-2021-01-22:0 File FileStorage
Volumes marked with "*" are in the Autochanger.
189 files selected to be restored.
Using Catalog "MyCatalog"
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2021-01-22 11:54:55
Catalog: MyCatalog
Priority: 10
Plugin Options: *None*
OK to run? (yes/mod/no): mod
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Restore Client
6: When
7: Priority
8: Bootstrap
9: Where
10: File Relocation
11: Replace
12: JobId
13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : m365: service=drive tenant="574dda43-d107-48e2-a7f2-aa51c10bdaec" objectid="31ddf4b1-
ed5d-432f-bac7-7b946fd23394" user="adelev@baculaenterprise.onmicrosoft.com" drive_version_history=yes debug=3
Plugin Restore Options
Option Current Value Default Value
destination_user: *None* (*None*)
destination_path: *None* (*None*)
send_report: *None* (0)
email_allow_duplicates: *None* (1)
drive_skip_sharedwithme: *None* (0)
drive_skip_versions: *None* (1)
drive_restore_share_permissions: *None* (0)
drive_send_invitations: *None* (0)
drive_invitations_message: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (yes/mod/no): mod
You have the following choices:
1: destination_user (Destination User)
2: destination_path (Destination Path in M365)
3: send_report (Send report of the restore operation to the affected user)
4: email_allow_duplicates (Allow Duplicate Emails)
5: drive_skip_sharedwithme (Skip restoring shared with me elements even if they are selected)
6: drive_skip_versions (Skip restoring file former versions (tagged with '###date') even if they are selected)
7: drive_restore_share_permissions (Restore share permissions of items so they are shared if they originally
were)
8: drive_send_invitations (Send email invitations for restored OneDrive shares)
9: drive_invitations_message (Set invitations message for restored OneDrive shares)
10: debug (Change debug level)
Select parameter to modify (1-10): 2
Please enter a value for destination_path: MY_RESTORE_PATH
Plugin Restore Options
Option Current Value Default Value
destination_user: *None* (*None*)
destination_path: MY_RESTORE_PATH (*None*)
send_report: *None* (0)
email_allow_duplicates: *None* (1)
drive_skip_sharedwithme: *None* (0)
drive_skip_versions: *None* (1)
drive_restore_share_permissions: *None* (0)drive_send_invitations: *None* (0)
drive_invitations_message: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (yes/mod/no): yes
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2021-01-22 11:54:55
Catalog: MyCatalog
Priority: 10
Plugin Options: User specified
OK to run? (yes/mod/no): yes
Job queued. JobId=14
Restore by service
In this section some example restore configurations will be shown:
Cross tenant restore
You can perform cross-tenant restores using the restore variables: - tenantid - appid - objectid - secret
Obviously, it is needed to set up the destination tenant values, where one of the authentication methods should be applied first.
If you are using the standalone app authetication model, you will need to use the four values in your restore session.
If you want to use the common app authentication model, you won’t need to put the secret. On the other hand, please, ask the bacula enterprise edition support team in order to get the correct value for appid.
List
It is possible to list information using the bconsole .ls command and providing a path. In general, we need to provide the service parameter and a path representing a folder or object (like calendars).
There are some general commands (user, groups, sites), the rest of the commands need to point to the correct service.
Below some examples:
List general info: Users, groups, sites
Here we are showing these 3 commands using the bconsoel .ls command, but notice you may also use them with the query interface (keep your variable values, but apply something like: .query plugin=”…” client=xxxx parameter=xxx). Note that the referenced ‘config_file’ should contain the connection parameters (tenant, objectid, appid and secret).
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd path=/user
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
ptcomm: Starting Plugin Job
ptcomm: Finished reading Plugin Params
ptcomm: Backend connection to M365 stablished
ptcomm: Connected to tenant: baculaenterprise
ptcomm: Starting ListingStart
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /AdeleV@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /AlexW@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /DiegoS@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /jane@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /GradyA@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /HenriettaM@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /IsaiahL@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /JohannaL@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /JoniS@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /john@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /LeeG@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /LidiaH@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /LynneR@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /MeganB@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /MiriamG@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /NestorW@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /PattiF@baculaenterprise.onmicrosoft.com
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /PradeepG@baculaenterprise.onmicrosoft.com
2000 OK estimate files=18 bytes=0
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf"
client=127.0.0.1-fd path=/group
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
ptcomm: Starting Plugin Job
ptcomm: Finished reading Plugin Params
ptcomm: Backend connection to M365 stablished
ptcomm: Connected to tenant: baculaenterprise
ptcomm: Starting ListingStart
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Contoso Team
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Dev Team
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /baculaenterprise
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /All Company
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Bacula Systems Team Site
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Dev people Group
2000 OK estimate files=6 bytes=0
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf"
client=127.0.0.1-fd path=/site
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
ptcomm: Starting Plugin Job
ptcomm: Finished reading Plugin Params
ptcomm: Backend connection to M365 stablished
ptcomm: Connected to tenant: baculaenterprise
ptcomm: Starting ListingStart
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Communication site
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Adele Vance
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Alex Wilber
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Diego Siciliani
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Grady Archie
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Henrietta Mueller
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Isaiah Langer
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Johanna Lorenz
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Joni Sherman
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Lidia Holloway
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Lynne Robbins
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Megan Bowen
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Pradeep Gupta
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Bacula Systems Team Site
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Work @ Contoso
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Contoso Team
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Live @ Contoso
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Give @ Contoso
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Dev Team
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Dev people Group
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Team Site
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /John Doe
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Jane Doe
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Patti Fernandez
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Lee Gu
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Miriam Graham
-rw-r--r-- 1 root root -1 2106-02-07 07:28:15 /Nestor Wilke
2000 OK estimate files=28 bytes=0
# Query equivalents follow
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=user
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=group
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=site
# Other similar query function
*.query plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf" client=127.0.0.1-fd parameter=team
List Onedrive contents
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: baculaenterprise
m365: Backend connection to M365 stablished
m365: Starting ListingStart
drwxr-xr-x 1 nobody nogroup 26800469 2020-11-08 01:34:35 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/root/
drwxr-xr-x 1 nobody nogroup 9876293 2020-12-03 15:28:31 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/
-rw-r----- 1 nobody nogroup 217873 2021-04-13 13:29:56 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Hats.JPG
-rw-r----- 1 nobody nogroup 1914266 2021-02-01 17:45:34 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/lombok.jar
-rw-r----- 1 nobody nogroup 785043 2021-02-01 17:45:56 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/noShared.dat
-rw-r----- 1 nobody nogroup 4 2021-02-01 17:45:35 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/test2.doc
-rw-r----- 1 nobody nogroup 4 2020-11-25 15:35:07 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/test4.html
-rw-r----- 1 nobody nogroup 42 2020-11-26 12:32:44 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/testlink123.url
drwxr-xr-x 1 nobody nogroup -1 2106-02-07 07:28:15 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/sharedWithMe/
2000 OK estimate files=23 bytes=53,600,938
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf
service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/sharedWithMe
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: baculaenterprise
m365: Backend connection to M365 stablished
m365: Starting ListingStart
-rw-r----- 1 nobody nogroup 11722 2021-04-13 17:42:06 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Aeque.ppt
-rw-r----- 1 nobody nogroup 4039 2021-04-13 17:42:02 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Voluptatibus.txt
-rw-r----- 1 nobody nogroup 7697 2021-04-13 17:41:59 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Nullam.jpeg
-rw-r----- 1 nobody nogroup 21031 2021-04-13 17:41:55 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Graeci.ppt
-rw-r----- 1 nobody nogroup 21417 2021-04-13 17:41:52 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Habeo.txt
-rw-r----- 1 nobody nogroup 15354 2021-04-13 17:41:47 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Atomorum.jpeg
-rw-r----- 1 nobody nogroup 22217 2021-04-13 17:41:43 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Faucibus.ppt
-rw-r----- 1 nobody nogroup 15714 2021-04-12 11:13:52 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Sodales.ppt
-rw-r----- 1 nobody nogroup 14371 2021-04-12 11:13:48 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Cum.txt
-rw-r----- 1 nobody nogroup 11059 2021-04-12 11:13:44 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Sapien.jpeg
-rw-r----- 1 nobody nogroup 14144 2021-04-12 11:13:41 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Habitasse.ppt
-rw-r----- 1 nobody nogroup 12594 2021-04-12 11:13:37 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Deserunt.txt
-rw-r----- 1 nobody nogroup 14923 2021-04-12 11:13:32 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Postulant.jpeg
-rw-r----- 1 nobody nogroup 21307 2021-04-12 11:13:29 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/SharePoint App/Tation.ppt
drwxr-xr-x 1 nobody nogroup 0 2021-01-08 14:32:20 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/john@baculaenterprise.onmicrosoft.
com/JohnFolder/
-rw-r----- 1 nobody nogroup 1442312 2021-01-08 11:57:09 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/john@baculaenterprise.onmicrosoft.
com/developers.pdf
-rw-r----- 1 nobody nogroup 630174 2021-01-08 11:57:08 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/sharedWithMe/john@baculaenterprise.onmicrosoft.
com/console.pdf
2000 OK estimate files=17 bytes=2,280,075*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf
service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Docs
Connecting to Client 127.0.0.1-fd at 127.0.0.1:8102
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: baculaenterprise
m365: Backend connection to M365 stablished
m365: Starting ListingStart
drwxr-xr-x 1 nobody nogroup 9876293 2020-12-03 15:28:31 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/
drwxr-xr-x 1 nobody nogroup 793475 2020-11-25 15:34:49 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/Simple Sub Folders/
-rw-r----- 1 nobody nogroup 6383460 2020-12-08 10:13:11 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/E16WKAMS.10E.zip
-rw-r----- 1 nobody nogroup 1914266 2020-12-08 10:13:32 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/lombok.jar
-rw-r----- 1 nobody nogroup 785043 2021-01-12 10:12:04 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/noShared.dat
-rw-r----- 1 nobody nogroup
24 2020-11-25 15:35:05 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test1.txt
-rw-r----- 1 nobody nogroup 13 2021-01-11 15:39:57 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test11.txt
-rw-r----- 1 nobody nogroup 4 2020-11-25 15:35:05 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test2.doc
-rw-r----- 1 nobody nogroup 4 2020-11-25 15:35:05 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test3.pdf
-rw-r----- 1 nobody nogroup 4 2020-12-03 16:52:32 /baculaenterprise/users
/adelev@baculaenterprise.onmicrosoft.com/drives/onedrive/root:/Docs/test4.html
2000 OK estimate files=10 bytes=19,752,586
Other list examples
Please note that the following examples are not available in .query mode, they are only for .ls command.
# List contents of Inbox folder of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=email user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Inbox
# List contents of Sent folder of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=email user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Sent
# List all contacts of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=contacts user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/
# List contacts of user adelev@baculaenterprise.onmicrosoft.com and folder MyContactsDir
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=contacts user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyContactsDir
# List calendar events of user adelev@baculaenterprise.onmicrosoft.com and her calendar MyCalendar
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=calendar user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyCalendar
# List onenote pages of user adelev@baculaenterprise.onmicrosoft.com and her section secA included in section Group groupA
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=onenote user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyNotebook/sectionGroups/groupA/sections/secA
# List contents of OneDrive folder 'folderA' of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/folderA
Performance command
This plugin provides a query command allowing to check the performance of a given setup. The process consist on uploading some randomly generated files of a customizable size (and a customizable number of them) to a target tenant, then download them using the same method the backup itself employs. Upload and download times are measured. An average is calculated and all the information is presented to the user.
Please, note that this command is currently working in single-threaded mode, while the plugin is running with multiple threads (configurable through concurrent_threads variable), so you could have better results with the actual backup process.
The usage of this command is:
*.query client=<your-client> plugin="m365: tenant=<tenantId> objectid=<objectId> user=<your-user> service=<service>" parameter=perf
Currently, theh plugin supports two types of performance commands: perf for onedrive (default) and perf for email. So the value service should be: drive or email.
The behavior of this service from performance perspective can differ significantly. Below we detail some important aspects:
On Onedrive and other services implying user files, it is needed to request a ‘download session’ for each file.
Once we have it, the download starts. This requires significantly more time than just retrieving simple objects.
In general, objects are not retrieved one by one. We request for lists of objects, that later on are processed.
For objects that are not pointing to any other file (as emails without attachments), the time needed per object is very low.
Drive perf command
For Onedrive service, in addition to user, you can also use group or site (site name or siteId) for this command. The final output will be similar to the following:
...
date=2021-09-06 17:25:56
tenant_name=johndoe.onmicrosoft.com
upload-speed-average=3.3 MiB/s
download-speed-average=32.2 MiB/s
latency-average=482.00 ms
console=---- Test END ----
You can control the size and the number of files used using the following query parameters: - q_perf_size (in bytes) - q_perf_files (number of files)
By default, their values are: - q_perf_size = 209715200 - q_perf_files = 3
Below we provide a usage example:
*.query client=127.0.0.1-fd plugin="m365:
tenant=f421f256-748a-4a80-90da-bf47y66fa166 objectid=90173rea-c144-46ye-8f93-4uh87bcd6d47 site=\"johndoe.sharepoint.com,fda66478-aab5-4896-a6ff-e0fb02af2a40,c024ee54-dcc0-4f1e-ab76-671f6eb87fda\" service=drive q_perf_files=5 q_perf_size=314572800" parameter=perf
Email perf command
For Email service, specifying a user is mandatory. The command can generate simple emails, but also emails with one attachment. The final output can be similar to the following:
...
console=---- Summary ----
date=2021-11-11 17:05:18
tenant_name=johndoe.onmicrosoft.com
upload-speed-average=44.6 KiB/s
download-speed-average=220.5 KiB/s
latency-average-per-list-call=510.83 ms
latency-average-per-message=36.18 ms
latency-average-attachments=617.64 ms
console=---- Test END ----
The behavior of the email perf command is a little different from the Onedrive one. Instead of uploading/downloading one message at a time, here everything is uploaded first. After that everything is downloaded. The reason is here the backup process uses lists of objects and, unless there are attachments implied, that’s all, the objects are simply stored from that list.
Similarly to the onedrive perf command, you can control the size and the number of emails. However, there is an extra parameter to control the size of the attachments:
q_perf_size (in bytes)
q_perf_size_attachments (in bytes)
If not specified, no attachment will be generated
If specified, one simple text attachment will be generated together with each email
q_perf_files (number of files)
By default, their values are:
q_perf_size = 10240
q_perf_size_attachments = 0
q_perf_files = 50
Below we provide a usage example:
*.query client=127.0.0.1-fd plugin="m365:
tenant=f421f256-748a-4a80-90da-bf47y66fa166 objectid=90173rea-c144-46ye-8f93-4uh87bcd6d47 site=\"johndoe.sharepoint.com,fda66478-aab5-4896-a6ff-e0fb02af2a40,c024ee54-dcc0-4f1e-ab76-671f6eb87fda\" service=email q_perf_size_attachment=204800" parameter=perf
Other query commands
Sometimes there is an element that causes some error while fetching it in the email service and it’s difficult to identify it by its reported debug id. For this situation, there is a decoding command:
As the example shows, the query parameter is ‘query=decode|{url}’. Url should be the URL until the element we are interested into (so the attachment here).
Another useful command shows all the options the plugin can accept as parameters, categorized by section:
The output of this command will return a json structure with all the available options.
Best practices
Jobs Distribution
It is recommended, to split the target backup between different groups of entities or even having one job per entity (user, group, site, etc). This way errors in one job will not invalidate a whole backup cycle where some entities have been successful and some others had errors. This also makes easier to identify the cause of the error.
Concurrency
Microsoft public APIs impose a variety of boundaries that need to be considered. If a boundary is crossed, the corresponding API call will fail and the application will need to wait some amount of time to retry, which is different depending on the boundary crossed.
It is crucial to plan an adequate strategy to backup all the elements without reaching API boundaries. A single job implements some parallelism which can be reduced until a point, if necessary, using the variable backup_queue_size (default value is 30). This variable controls the size of the internal queues communicating the internal threads, that are designed to fetch, open and send every item to Bacula core. Reducing its size will produce, ultimately (with a value of 1 for example), an execution very similar to a single threaded process. On the othere hand the plugin has concurrent_threads which controls the number of simultaneous processes fetching and downloading data (default value is 5).
If you are going to launch different jobs in parallel it is recommended to configure different services for each of them (protect in parallel email, drive, contacts…). However, be careful with the concurrency over the same service (in general, it is recommended a maximum of 4-5 jobs working with the same service) and plan a step-by-step testing scenario before putting it into production. Other important point is the timing schedule, as some boundaries are related to time-frames (number of request per 10 minutes or 1 hour, for example). If you detect you reach boundaries when running all your backups during a single day of the week, please try to use 2 or 3 days and spread the load through them in order to achieve better performance results.
More information about Microsoft 365 Graph API boundaries may be found here:
Disk Space
It is necessary to have at least enough disk space available for the size of the largest file in the backup session. If you are using concurrency between jobs or through the same job (by default this is the case through the concurrent_threads=5 parameter), you would need at least that size for the largest file multiplied by the number of operations in parallel you run.
Read more details in the Backup of Attachments and Files section.
Performance
The performance of this plugin is highly dependent on many external factors:
ISP latency and bandwidth
Network infrastructure
FD Host hardware
FD Load
…
In summary, it is not possible to establish an exact reference about how much time a backup will need to complete.
As reference values, in testing environments and using sequential calls we had the following results:
3000 emails in the same folder of a single mailbox, with emails ranging from 500 to 3000 words - about 10K to 30K in size
The total time required to back them up was: 88243 ms.
That implies a total time per email of about 30ms
Concurrency has also been tested. For example, when using single-threaded mode (concurrent_threads=1) with emails, good values where found for 4-6 concurrent jobs that allowed us to backup around 200 emails per second (sizes between 20K-30K) when no attachments were implied. As more attachments were implied, the number of emails will decrease very significantly, while the overall size speed will be increased. This can be generalized:
Many little objects to protect -> More objects per second, but less speed (MB/s)
Big files to protect -> Less objects per second, but greater speed (MB/s)
It is recommended to benchmark your own environment in base to your requirements and needs.
The automatic parallelization mechanism (using concurrent_threads=x, default is 5) should work well for most scenarios, however, fine tune is possible if we define one job per entity and we control how many of them run in pararllel, together to decrease the concurrent_threads value in order to avoid throttling from MS Graph API.
There are many different possible strategies to use this plugin, so please, study what is best suiting for your needs before deploying the jobs for your entire environment, so you can get best possible results:
You can have a job per entity (users, groups, sites…) and all services
You can have inside a job multiple entities and only some services
You can split your workload through an schedule, or try to run all your jobs together.
You can run jobs in parallel or take advantadge of concurrent_threads and so run less jobs in parallel
You can select what services to backup or backup them all
You can backup whole services to backup or select precisely what elements you really need inside each service (folders, paths, exclussions…)
For delegated permissions services, you can proceed with the login previously to backups (recommended) or you can wait for the job to ask for what it needs
etc.
Best practices by service
Each Microsoft 365 service presents its own particularties. Moreover each deployment and each environment can be protected through different strategies depending on the size, number of entities and other parameters. However, there are some general considerations that can be helpful to consider for a selection of services:
Onenote
Onenote is the most sensitive module to throttling limits in Microsoft 365 APIs: https://learn.microsoft.com/en-us/graph/throttling
It is recommended to be selective with the information to be protected and make one job for each selected entity: that is one job per user, group or site.
In case throttling problems appears with this strategy, jobs will need to be spread into larger time windows.
It is not recommended to include onenote for entity focused strategies where there is one job per user including all services (email, drive, contact, calendar ..). It is recomended to have specific onenote jobs.
Chats and Todo
This service uses delegated permissions. They can be protected together, but we recomend to run the ‘login’ query command for each individual user before running the actual jobs.
Email
Email service allows you to protect not only email and attachments, but also settings, user categories and an additional MIME copy for each email with its attachments. MIME files are very useful if you are planning to migrate your data from M365 to any other Email tool, however, the cost of getting each MIME file is very high and it duplicates backup time. MIME files are not needed to perform a local restore or a restore over M365. Therefore, it is recommended to disable MIME backup for a general basis and use it only in case of planning a complete migration of the data.
Similarly to other services, it is recommended to split the backup set into small sets of users (ideally one backup per mailbox, but it’s also fine to group them by 5 or 10), specially if there is a large number of mailboxes and they contain thousands of emails. To make this easier, the plugin offers regular expression selection parameters (user_regex_include), as well as listing and query commands to analyze the backup target (List general info: Users, groups, sites).
Update app permissions
Bacula Enterprise Microsoft 365 Plugin is an application that is under continous evolution. In consequence it may happen that new features require new permissions to function properly.
Bacula Systems will add the required new permissions to the central bacula-m365-plugin app registered in Azure AD. If you are already using the plugin, you will need to update the permissions of your own app (usually bacula-m365-plugin app itself, available under ‘enterprise applications’ option in your Azure Active Directory application).
To refresh and allow any new permission to your app. Please apply the following procedure:
Access the Azure Portal with a Tenant admin user
Locate your bacula-m365-plugin app under ‘Enterprise applications’ and click on it.
In the left menu, click on ‘Permissions’ and then click the big blue button showing ‘Grant admin consent for <yourtenantname>’:
Wait for a few minutes
Refresh the page
See the new permissions in the list
New releases needing any new permissions will properly reflect that information with the specific permissions list.
Delegated permissions
The great majority of services of the Bacula Enterprise M365 plugin are using the model of ‘Application permissions’. With this model, the plugin application can directly access the implied services once a tenant admin has aproved it.
On the contrary, some other services can require the model of ‘Delegated permissions’ where the plugin needs to impersonate a given user in order to be capable of accessing and restoring the data inside the service. Although we will support more in the future, currently, the only service requesting delegated permissions is:
Calendar: If any group is included as an entity to protect
Chat: In order to be able to backup all chat messages belonging to a certain user the user in question will need to delegate the permissions as described below.
Delegated permissions or application permissions are automatically applied when they are needed. However, delegated permissions need user interaction the first time they are invoked, or if the local information about access tokens has been lost for some reason.
The user interaction can be manually triggered with a special query command that may be run as follows:
.query plugin="m365: tenant=e421f055-748a-4a80-90da-bf48b66fa166 objectid=9017f87a-c164-488e-8f93-47139bcd6d47" client=127.0.0.1-fd parameter=login
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: johndoe
m365: Backend connection to M365 stablished
m365: Starting QueryStart
m365: We don't have a valid token. Please, To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CQ7LETVFT to authenticate. You need to do it with an administrator user
We only need to follow the instructions, open a browser, enter the code and use the correct user to login (which means a user with access to the resource we want to backup).
For example, if we are working with the calendar of a group, we will need a user with access to that calendar (in general, to protect calendars associated to groups, it will be necessary to create a special backup user with access to all of them and perform the login with, which will avoid the need of performing the login process for every different group).
Here, if any MFA mechanism is enabled, it will be normally required and the user must complete the associated process. Finally Microsoft will ask to confirm if we want the plugin app to proceed with the operation, and we need to confirm:
Once the login is complete, the plugin will automatically end the operation and show something like:
info=We have a valid token got from account: johndoe@johndoe.onmicrosoft.com
info=It will be automatically refreshed when needed. But the current expiration date is: 2021-06-16T16:18:16Z
info=Backups using services with delegated permissions won't ask for intervention
It will also store the token information locally in a special file that it is stored in the working directory inside a folder with the tenant name. For example: /opt/bacula/working/m365/tennantname/bacula-m365-plugin-standalone-token_cache.json
This file is also backed up in the backup operation itself, so it can be restored manually, in case it was lost and in case you don’t want to run login operations again
In case the query method is not invoked prior to any job needing it, the job itself will show the message in the joblog and it will be blocked until a user performs the needed interaction described above. Once this happens, the job will automatically continue.
Note
Starting with version 14.0 of Bacula Enterprise you can also see logged in users directly using BWeb, where you can also manually add logged-in users. In order to see more details, please go to section: BWeb Console<./m365-bweb-console>.
BWeb Management Console
Bacula Enterprise Microsoft 365 Plugin can be managed from a dedicated user friendly Web console, specifically designed to facilitate tasks as the authentication process for new tenants or the fileset configuration process.
Note
This feature is available starting with version 14.0 of Bacula Enteprise.
Connection
The purpose of this screen is to show a list of the currently connected tenants that are already registered. The presence of each line in the table means that a tenant has been configured at some point.
In the upper right corner, it is possible to switch the FD at any time, so the information is based on the currently selected FD. After the first time a FD is selected, the information will be remembered using cookies for the next time and for any other M365 Console windows.
The table shows tenant name, tenant id and the connection state. The connection state can indicate a possible incomplete registration operation (because of some error), otherwise it will indicate a successful connection and the type of connection: through the common app or through the standalone app.
The big button placed at the bottom of the table allows us to configure the connection to a new tenant, so it will open the ‘New tenant wizard’ that is described in the section below.
Other actions that can be performed from this tenant list window are:
Retry operation: It allows to retry a possible incomplete registration, so re-launch the proper internal query command (add-app or the objectid command) using a selected tenant.
Delete registrator: It allows to delete the registrator app inside the selected tenant. It is not possible to delete a registrator if registration is incomplete . In this case, you must retry operation or delete tenant. This is a possibility only offered for security concerns, as the registrator app has some elevated permissions.
Delete tenant: It removes the tenant from the internal BWeb configuration.
New tenant wizard
When opening new tenant wizard, we can select the registration model to be used: Standalone or Common.
Details of each model are shown in the images and you can also read more information in this whitepaper under the Authorization section.
It is needed to provide the tenant id, which can be found following the whitepaper instructions placed under the same Authorization section.
It is recommended to use the Standalone model (first one to appear in the web interface) and be using the common model only for testing purposes.
Once we proceed with one of the models, an URI of Microsoft website will be suggested and you need to provide the needed permissions, then, Microsoft will redirect you to a Bacula Systems page (same as shown in the Authorization section).
BWeb will automatically guide you until the end of the process where you will see your tenant correctly added into the tenant list screen.
Logged in users
In this screen a list of logged in users is shown.
Logged in users are particular users that have approved the Azure AD plugin app to make operations over their accounts that need delegated permissions.
Examples of operations needing this permissions model are:
Backup of calendar of groups
Todo Tasks module
Chats module
In addition to show the users list, it is also possible to force an user login. This function is offered to avoid having this kind of requests at backup time.
Users need to login only once, afterwards, the local cache will automatically be used to use saved tokens or to refresh them as needed.
For more information about delegated permissions, please take a look to <./m365-delegated-permissions>
FileSets
The purpose of the FileSets page is to control all the filesets of Microsoft 365 Plugin, so we will see them listed in the main screen:
From there it is possible to create a new M365 Fileset in an user friendly way:
We need to select the tenant to work with as a first step:
The second step is to select the entities to protect: they can be users, groups or sites. All the services of M365 are ‘owned’ by any of those entities:
It is possible to dinamically add or remove them, the wizard will store your final selection.
The third step allows us to select the services to protect in a similar way than the entities step, with multiselection allowed, and where we can remove or add services at will:
After selecting the services, we will have one independent screen for each selected service where we will enter the particular configuration for each of the services implied. In the screenshot below, we can see the Sharepoint services:
As a final step, it is possible to setup some advanced parameters and then finish the configuration:
From this final screen is possible to go directly to the job configuration wizard and finish a backup configuration or to return to the previous fileset list page.
Please, note that this Wizard contains the most common parameters and it is intended to help to configure the most common use cases. In case you need to setup more advanced parameters or to edit existing filesets, BWeb will allow you to do it using the regular Fileset editor where you will find all plugin parameters.
The
restart
command has limitations with plugins, as it initiates the Job from scratch rather than continuing it. Bacula determines whether a Job is restarted or continued, but using therestart
command will result in a new Job.
Troubleshooting
Listed in this section are some scenarios that are known to cause issues.
Pwsh: The specified program requires a newer version of windows
If a Sharepoint backup shows the message referenced in the title, it means the powershell session of the job has conflicted with another open Powershell session.
Solution: Be sure no other Powershell sessions are open and try again
M365 DataPack Onedrive Hashes
If you test the plugin with the developer Datapack of M365 you may find problems with hash checking of OneDrive autogenerated files. You could see hash failures.
The reason is those auto-generated files have incorrect hashes.
Solution: Use your own uploaded files to test the plugin.
Out of Memory
- If you ever face OutOfMemory errors of the Java daemon (you will find them in the m365-debug.err file),
you are very likely dealing with sub-attachments containing big files or you are using a high level of concurrency through internal concurrent_threads parameter and/or parallel jobs. To overcome this situation you can:
Disable the backup of multi-level attachments setting multi_attach=no
Reduce concurrent_threads parameter
Reduce the number of jobs running in parallel
If you cannot do that or you are not using multi-level attachments you should increase JVM memory.
To increase JVM memory that you will need to:
Create a this file: ‘/opt/bacula/etc/m365_backend.conf’.
Below, an example of the contents:
|M365_JVM_MIN=2G |M365_JVM_MAX=8G
Those values will define the MIN (M365_JVM_MIN) and MAX (M365_JVM_MAX) memory values assigned to the JVM Heap size. In this example we are setting 2Gb for the minimum, and 8Gb for the maximum. In general, those values should be more than enough. Please, be careful if you are running jobs in parallel, as very big values and several jobs at a time could quickly eat all the memory of your host.
The ‘/opt/bacula/etc/m365_backend.conf’ won’t be modified through package upgrades, so your memory settings will be persistent.
App Registration Error
When trying to register the bacula-m365-plugin app in a given tenant, the following error can appear in the URL of a resulting page placed on baculasystems.com:
https://www.baculasystems.com/m365-plugin-auth/common?error=access_denied&error_description=AADSTS650052%
3a+The+app+needs+access+to+a+service+(%27https%3a%2f%2f*.dps.mil%2f%27)+that+your+organization+%27cd77bbc0-999e-
44c5-af9d-c11bea178407%27+has+not+subscribed+to+or+enabled.
+Contact+your+IT+Admin+to+review+the+configuration+of+your+service+subscriptions.%0d%0aTrace+ID%3a+afdc0b36-
bad6-4d1f-8d24-9213439c0a01%0d%0aCorrelation+ID%3a+c73a3631-5573-41ee-8220-cc281ea4882f%0d%0aTimestamp%3a+2021-
04-27+09%3a21%3a55Z&error_uri=https%3a%2f%2flogin.microsoftonline.com%2ferror%3fcode%
3d650052&admin_consent=True&state=12345
If we look carefully into the error contents we will notice the following message:
AADSTS650052: The app needs access to a service (https://*.dps.mil.com) that your organization cd77bbc0-999e-44c5-af9d-c11bea178407 has not subscribed to or enabled. Contact your IT Admin to review the configuration of your service subscriptions.
Even if it is not a very clear message, the error is pointing us to the fact that our tenant is missing some kind of access or subscription in order to utilize bacula-m365-plugin app. The contents of the error can vary depending on the specific situation of the target tenant.
However, in general, and particularly for the example shown, the problem comes from the fact that target tenant has no license to utilize Sharepoint Online service. Without this license/permission, the plugin cannot be registered.
Empty Items in Group/Team Site Libraries
When working with the OneDrive module and Group entities you may experience a view of a list of apparently empty elements if navigating through them using Sharepoint views:
This is not an error of this plugin. This is a problem with some kind of views available in the Sharepoint Online service, where list items corresponding to drive items are shown. As here we are dealing with drive items, those list items have no title attached.
In the particular example shown above, a ‘list view’ of a document library is selected. However if we select any other view as ‘Titles’, ‘All Documents’, etc:
We will be able to see folders and files with their corresponding names:
Empty Chat messages after restoring Teams
After a successful restore operation with the Teams module, it can happen that accessing the data using the web interface of the M365 services will show the associated channels with empty contents. This is usually associated to some bad functions of implied cookies with the web browsers.
If you experience this situation we recommend to open the information using a ‘private window/tab’ or removing previous cookies, as the information is actually there. In case this strategy does not work, please use the Teams ‘desktop app’ for your OS, where the behavior is much more solid.
Chat messages with not displayed contents in Teams
There are several kind of auto-generated messages that rely on other Microsoft Cloud internal data structure. Examples of this are:
Scheduled meetings through chat actions, not associated to calendar (they belong to ‘onlineMeetings’ module, which is not included today in Teams backup)
Messages showing that a Tab was added
Messages showing contents of external apps
This kind of messages will be restored exactly with the same form that they were got during backup and text contents will be visible. However, as the information linked is attached in the external applications to the old team, that visual content may not be visible as it was in the original message.
Error restoring teams app
Teams apps are external entities controlled by an undetermined number of external provides. When adding an app to a team, a process belonging to that provider takes place. That process is invoked through the plugin restore process when restoring teams with apps. Sometimes that external process can return unexpected results.
The M365 Plugin will show those situations with a propper message in the joblog that is not treated as error, and the restore will succeed in most cases.
- As this is not managed by Bacula Systems or MS Graph API, the approach is just to show the result as it is, when it is not successful. However, please note that the app itself,
in most cases, will be also successfully restore even if that situation happens.
Error with Teams App after restoring a Planner plan
Teams desktop applications use cache data that can interfere sometimes with the real data. Restoring a planner plan can be an example. When you try to open a restored planner tab you may see an error message. However, please just click on the reload button and you should see your data without issue.
Orphaned Java Processe After Job Failure
In some circumstances, java processes belonging to the plugin can be left behind. They may be using noticeable CPU and memory resources. It is safe to remove them by sending an appropriate signal.
Such orphaned processes should not occur with Bacula Jobs finishing without problems.
A way to identify such processes would be to look for java processes running the plugin .jar file either when no such job is actually active, or when no such job of a similar age exists. For example
[root@bsys-demo-m365] ~ # ps -o pid,stime,etime,args -wwC java | grep -E 'bacula-m365-plugin-.*'
370270 Jul04 50-04:48:36 java -Xms512M -Xmx4G -jar /opt/bacula/lib/bacula-m365-plugin-4.4.9.jar --mode=bacula --path=/opt/bacula/working --log=/opt/bacula/working/m365/m365-debug.log
370403 Jul04 50-04:42:10 java -Xms512M -Xmx4G -jar /opt/bacula/lib/bacula-m365-plugin-4.4.9.jar --mode=bacula --path=/opt/bacula/working --log=/opt/bacula/working/m365/m365-debug.log
If no m365 job is running since July 04, it would be safe to kill -9 370270 370403.
In newer versions of the plugin, the actual Bacula Job running one of the java processes will be indicated in a command line option of the java process, and the Job report will indicate the process id of the java process started.