The Bacula EnterpriseMicrosoft 365 Plugin is a very easy to deploy and configure plugin
supporting the following M365 services:
OneDrive
Emails
Mailbox settings
Sharepoint Online
Calendars
Contacts
OneNote
Tasks
Teams
Chats
It is shipped with advanced parallelization, resiliency, and flexibility
features in addition to covering most of the possible M365 use case
scenarios. A full feature list is presented below:
Common features
Microsoft Graph API based backups
Multi-service backup in the same task
Multi-service parallelization capabilities
Multi-thread single service processes
Automatic parallelization of fetching processes
Generation of user-friendly report for restore operations
Network resiliency mechanisms
Latest Microsoft Authentication mechanisms
Discovery/List/Query capabilities
Restore objects to Microsoft 365
To original entity
To any other entity
To a different tenant (cross-tenant restore)
Restore any object to filesystem
Owner data protection feature:
Notify data owner about restore actions of his data
Do not proceed further until he enters his M365 credentials
Incremental & Differential backups
Advanced delta function for improved performance (for selected
services)
Backup and Restore of Exchange Online Mailboxes
Mailfolder, message and attachment granularity for restore
Email addresses and mailfolders selection capabilities for backup
Mailbox settings protection
Folder rules protection
Restore objects to Microsoft 365 or to any file-system
Restore MIME messages to any filesystem
Incremental & Differential backup
Support for user mailboxes and shared mailboxes
User categories protection
Fully indexed information into Bacula Catalog
Advanced search capabilities for restore operations
Ability to filter input data by date
Ability to exclude message fields from backup or from index
Exclude private or spam messages through powerful filtering capability
Export to PST format
Backup and Restore of OneDrive for Business & Sharepoint Document
libraries
Backup and Restore of User drives
Backup and Restore of Group drives
Backup and Restore of Sharepoint document libraries
Include/Exclude system libraries
Include/Exclude hidden libraries
Backup main entity drive unit, but also any other unit
Advanced selection capabilities
Target entities (Users/Groups/Sites)
List Include/List Exclude/Regex include/Regex Exclude…
Folder selection capabilities for backup
List Include/List Exclude
File selection capabilities for backup
Regex include/Regex Exclude…
Drive unit selection capabilities
Folder and file granularity for restore
Computed hash check at backup and restore time
Backup and restore of permissions shares
Backup and restore of shared elements to each entity
Backup and restore of OneDrive file versions
Backup and Restore of Sharepoint Sites
Backup of Site Objects (MS Graph Object)
Backup Site Sharing permissions
Backup of full Site Templates (PnP Powershell Provisioning):
Site metadata
Lists metadata
ListItems metadata
WebPages metadata
DocumentLibraries metadata
Support for Sub-Sites
Restore backed up sites as new sites using Site Template (PnP Powershell Provisioning)
Backup/Restore Lists Objects (MS Graph Object)
Backup/Restore ListItems (MS Graph Object)
Backup/Restore Document Libraries Drive Items
Backup and Restore of Contacts/People
Backup/restore Contacts
Backup/restore groups of contacts
Backup Organizational contacts (Read-only)
Backup and Restore of Tasks
Backup/restore User todo tasks
Backup/restore Teams Planner tasks
Backup and Restore of Calendars
Backup/Restore user calendars
Backup/Restore user groups of calendars
Backup/Restore groups calendar
Calendar permissions
Backup/Restore Events
Support for Attachments
File Attachments
Reference Attachments
Item Attachments (including backup of MIME objects)
Microsoft 365 Personal, Family, Microsoft Home & Student
subscriptions are not supported for backup/restore purposes.
It is necessary to have full administrative access to the target Tenant to protect in order to provide the required permissions to the
Azure Application linked to this Bacula Enterprise Microsoft 365 Plugin.
Currently the plugin must be installed on a Linux based OS (RH,
Debian, Ubuntu, SLES ..) where a Bacula Enterprise File Daemon is installed.
Bacula Systems will address support for running this plugin on a Windows platform in a future version.
The OS where the File Daemon is installed must have installed Java version 8 or above.
If the Sharepoint module is going to be used, the OS where File Daemon is
installed must also have the following packages installed:
PowerShell v7.2.1 or above
PnP Powershell v1.9.0 or above
Memory and computation requirements completely depend on the usage of this plugin
(parallelization, environment size, etc). However, it is expected to have a minimum
of 4GB RAM in the server where the File Daemon is running. By default, every job
could end up using up to 512Mb of RAM in demanding scenarios (usually it
will be less). However, there can be particular situations where this could be higher.
This memory limit can be adjusted internally (see Out of Memory). Refer to the Scope section below for any service specific requirements.
This is a common question that arises frequently among IT and Backup
professionals, so it is important to have a clear picture of it. It is
true that Microsoft offers some services intended to prevent data loss:
As with any cloud data, Microsoft 365 data is geo-replicated using Azure
cloud to several destinations automatically and transparently.
Therefore, complete data loss because of hardware failures are very
unlikely to happen.
Data Loss Protection service: Policy based services capable of detecting
filtered content and act upon it encrypting it or modifying it in order
to protect it (remove headers, etc). This is not a backup tool, is a
service to prevent undesired actions to the content stored in
Microsoft 365 (for example sharing confidential information with the
wrong people).
Retention policies of Microsoft 365: Microsoft retains a maximum
of 30 days of deleted information from active subscriptions. Therefore
it is possible to recover accidental deleted items inside that
period. For more information:
Onedrive Recycle Bin cannot be protected with this plugin. This is a
Microsoft limitation coming from their exposed REST APIs, where it is
not possible to access Recycle Bin information. If future versions of
Microsoft REST APIs officially include this function, Bacula Systems
will include it as a new feature of this plugin.
In general, empty files (files with 0 byte contents) are simply not backed up by Microsoft 365 plugin.
In particular, email attachments or onedrive files will show a message in the joblog to inform about
empty files detected and so not processed.
In general, this plugin backups two types of information:
Objects
Files
Objects are elements representing some entity in Microsoft 365 such as a calendar event, a contact, an email, a team, etc.
Files are attachments, hosted contents or OneDrive files.
While objects are directly streamed from memory to the backup engine, files need to be downloaded to the FD host before
being sent. This is done in order to make some metadata checks and to improve overall performance, as this way operations can be
parallelized. Every file is removed just after being completely downloaded and sent to the backup engine.
The path used for this purpose is established by the ‘path’ plugin variable, that usually is set up in the m365_backend script with the value:
/opt/bacula/working
Inside the path variable, a ‘spool’ directory will be created and used for those temporary download processes.
Therefore, it is necessary to have at least enough disk space available for the size of the largest file in the backup session. If you are using
concurrency between jobs or through the same job (by default this is the case through the concurrent_threads=5 parameter), you would need at least
that size for the largest file multiplied by the number of operations in parallel you run.
The M365 plugin overcomes the current limitations of the MS Graph API on
backup/restore of SharePoint sites by using the Powershell PnP opensource project:
This project has been designed to abstract many complexities of
the Sharepoint Online APIs and provides all sorts of automation processes.
Among them, site template provisioning, which is the main part of the
method Bacula Enterprise Microsoft 365 Plugin uses in order to provide
backup/restore capabilities for Sharepoint Online.
Microsoft can modify at any time (and they do it frequently) the
properties, structure or limitations of any element belonging to a site
kind (team site, communication site, project site, etc) whicn can cause the
restore provision process to fail. It is even common to see native
structures of Sharepoint Online trigger errors with particular
items/values and this error comes from the Sharepoint Online API.
As a result, is not possible to guarantee that a given template is going
to work in a provision process. Many times, the template might need to
be partially modified before being applied (removing some particular
item, removing some particular column..). That part of the process is
not possible to predict or automate, so we
strongly recommend frequent testing of site restore processes to detect
any need of manual restore processes before a real restore is required.
It is possible to select and restore a particular list or listItems,
without restoring an entire Sharepoint Site, and pointing the operation
to an existing Site. This is done using the Graph API, but as we have
previously exposed, this API has many limitations with the Sharepoint API.
One of the limitations is that lists or list items with advanced
elements (location, images, etc) are not supported. Therefore this
feature should only be used for user created lists implying simple
elements like texts, checks, dates, etc.
The following items associated to Outlook elements (Exchange Online
service) are not supported due to the fact they are currently not accessible
using the APIs this plugin relies on (MS Graph API and PnP Powershell):
The following items associated to Onenote elements are not supported due to the fact they
are currently not accessible using the APIs this plugin relies on (MS Graph API and PnP Powershell):
Sharing permissions associated to Notebooks
Shared elements from one entity to another one. To protect them it is necessary to make it always from the source entity.
External calendars (as Google Calendar) connected to Microsoft accounts are not supported. They are external elements not exposed through MS APIs, so M365 Plugin cannot protect that information.
Little icons that appear on the left of a calendar event title (usually
auto-generated by the Outlook service) cannot be backed up, as they are not
exposed through the MS Graph API the plugin is relying on.
MS Cloud APIs are Microsoft property and they can change or evolve
at any time. In particular, the Graph API is actively developed, containing new
features every week, even if the version number of the service (1.0) is
not changed as a result of any of those additions:
This situation is significantly different from traditional on-premise
software, where each update is clearly numbered and controlled for a
given server, so applications consuming that software, can clearly state
what is offered and what are the target supported versions.
Microsoft is committed to trying to not break any existing
functionality that could affect external applications. However, this
situation can happen and therefore, cause some ocasional problems
with this plugin. Bacula Systems controls this with an advanced
automatic monitoring system which is always checking the correct
behavior of existing features, and will react quickly to that
hypothetical event, but please be aware of the nature and implications
of this kind of cloud technologies.
Bacula Enterprise Microsoft 365 Plugin is using the Microsoft Graph API
to perform almost all of its operations. Therefore, the plugin is working at the
maximum granularity that the service provides.
All the information is obtained using secure and encrypted HTTPS queries to Microsoft 365 from the
File Daemon where the plugin is installed. All the requests are performed over the following endpoints:
The plugin will contact an Azure registered app named
bacula-m365-plugin and will use it as a bridge to download the required
data or objects during the time of a backup and send them to the Storage Daemon.
Conversely, the plugin will receive them from an SD and perform uploads as needed
during a restore.
The implementation is done through a Java Daemon, therefore Java is a
requirement in the FD host. For more information about the
bacula-m365-plugin, please, consult Authorization section.
Below is a simplified vision of the architecture of this plugin
inside a generic Bacula Enterprise deployment:
Attachments (ItemAttachments, FileAttachments and
ReferenceAttachments)
MIME objects where possible
Notebooks
Notebook objects
Section objects
SectionGroup objects
Pages
Page contents (Html formatted)
- Page resources
Page image files
Page object files (any other file apart from images)
Teams
Team objects
Team settings
Team members and associated roles
Team installed apps
Channel objects
Channel tabs
Channel chat messages
Chat messsages hosted contents
Chat
Chat objects
Chat installed apps
Chat tabs
Channel chat messages
Chat messsages hosted contents
All the information of each object is stored in JSON format (except for Pnp
site template, which is stored in XML), preserving all their original
values. When the plugin works with objects containing additional data
(MIME files for messages, data for attachments and files of OneDrive,
etc), that data is also backed up.
In this section we will dig into how this plugin behaves for each
particular service, describing special features and and behaviors that
require an extended description.
The Bacula File Daemon and the Microsoft 365 Plugin need to be installed
on the host that is going to connect to the cloud based services. The
plugin is implemented over a Java layer, therefore it can be deployed
on the platform better suited for your needs among any of the
officially supported platforms of Bacula Enterprise (RHEL, SLES, Debian,
Ubuntu, Windows, etc).
Please, note that you may want to deploy your File
Daemon and the plugin on a virtual machine directly deployed in
Azure Cloud in order to reduce the latency between it and the
Microsoft Graph API and experience modest performance gains. However, this option is only recommended in case of having a very stable
connection between the File Daemon and the Storage Daemon, which means a special, dedicated connection with Azure or when the Storage
Daemon is deployed in the cloud as well. This is not usually the case - thus the
data needs to traverse the Internet with standard and shared connections from the File Daemon
to the Storage Daemon. Disconnections
while transmitting data between these two daemons may make jobs fail and cause large timeouts that are difficult
to manage and stabilize. The best strategy strategy for this kind of scenarios is to deploy the File Daemon and the Plugin to the
same host as the destination Storage Daemon is installed. This way, disconnections between the two daemons will not happen, while
disconnections between the FD and M365 will be transparently recovered (when possible), so jobs will finish successfully.
The system must have Java >= 8 installed (openjdk-8-jre for example)
and the Java executable should be available in the system PATH.
The Sharepoint module depends on the Powershell and the PnP Powershell modules.
Therefore, they also need to be installed before installing the Bacula
packages (see section PnP.Powershell below).
In order to install PowerShell it is necessary to follow the instructions
for the particular OS involved which may be found in the github site of
the project:
yourworkstation:~$ pwsh
PowerShell 7.2.0
Copyright (c) Microsoft Corporation.
https://aka.ms/powershell
Type 'help' to get help.
PS /home/john> Install-Module -Name "PnP.PowerShell"
We are taking Debian Buster as the example base system to proceed with
the installation of the Bacula Enterprise Microsoft 365 Plugin. In this
system, the installation is most easily done by adding the repository
file suitable for the existing subscription and the Debian version
utilized. An example would be /etc/apt/sources.list.d/bacula.list with
the following content:
# Bacula Enterprise
deb https://www.baculasystems.com/dl/@customer-string@/debs/bin/@version@/buster-64/ buster main
deb https://www.baculasystems.com/dl/@customer-string@/debs/m365/@version@/buster-64/ buster m365
The plugin has two different packages implied that should be installed
automatically with the command shown:
bacula-enterprise-m365-plugin
bacula-enterprise-m365-plugin-libs
Alternately, manual installation of the packages may be done after downloading
the poackages from your Bacula Systems provided download area, and then
using the package manager to install. An example:
Jar libraries in /opt/bacula/lib (such as bacula-m365-plugin-x.x.x.jar and bacula-m365-plugin-libs-x.x.x.jar). Please note that the version of those jar archives is not aligned with the version of the package. However, that version will be shown in the joblog in a message like ‘Jar version:X.X.X’.
Note
Version in Jar Name
Version is included in the name of .jar files from Bacula Enterprise version 14.0.4. Before that, libraries were composed by: bacula-m365-plugin.jar, bacula-meta-plugin-1.0.0.jar and bacula-m365-plugin-libs-1.0.0.jar
Plugin connection file (m365-fd.so) in the plugins directory (usually /opt/bacula/plugins)
Backend file (m365_backend) that invokes the jar files in /opt/bacula/bin. This backend file searches for the most recent bacula-m365-plugin-x.x.x.jar file in order to launch it, even thought usually we should have only one file.
A collection of powershell files used in the Sharepoint module in /opt/bacula/bin.
The first step in order to use the Bacula Enterprise Microsoft 365
Plugin is to authorize it to handle data of the target tenant to backup.
There are two possible strategies in order to allow the communication between
the Bacula Enterprise Plugin and your tenant:
Method A (DEPRECATED): Common app model
Register the pre-existing Bacula Systems bacula-m365-plugin Azure AD app into your tenant.
The communication will happen through this multi-tenant application.
Application Id and associated secrets are internal to the plugin.
Microsoft Graph limits associated to an application are common for everyone using this application (multi-customer).
For future new permissions you just need to click on ‘Grant permissions’ from Azure AD enterprise apps section
Method B (RECOMMENDED): Standalone app model
Register the pre-existing Bacula Systems bacula-m365-registratror Azure AD app into your tenant. Then add a standalone application in your tenant.
Then add your own standalone application in your tenant calling the appropriate automatic command from bconsole
A bacula-m365-plugin-standalone Azure AD application will be created specifically for your tenant. The communication will happen through it.
Application Id and associated secrets need to be correctly set and they can be managed by you.
Microsoft Graph limits associated to an application are specific to your standalone application
For future new permissions you need to re-run the bconsole add-app command (which uses bacula-m365-registrator) or do it yourself manually
To walk through the process described in Method B with the help of a video, click on the image below:
Note
Authentication Method B is only available from Bacula Enterprise 12.8.2
The first method is simpler and faster to setup, however it is only advised for testing purposes.
The second method needs a few extra variables to manage and it is recommended for medium or large environments. It is
more secure and it can offer better performance.
Backup and restore operations will be using in general the ‘Application permissions’ model, where the application
has enough privileges to perform all the operations without impersonating any user. However, some specific modules
need to employ ‘Delegated permissions’. To know more about them, please go to Delegated permissions
Note
Starting from Bacula Enterprise version 14.0 you can also perform these authorization tasks directly using BWeb,
to see more details, please go to section BWeb Management Console
The sections below will show how to use both methods. For any of them, the first
step is to find your Tenant ID:
In order to find the Tenant ID you only need to login to the Azure portal
(portal.azure.com) and take a look at the ‘Overview’ page of the service
Azure Active Directory. Just click in the service as the below image
shows:
Once there, you will find the Tenant ID in the box highlighted in the
below image:
Authorization Method A: Common app model with ‘bacula-m365-plugin’ (DEPRECATED)
Bacula Systems has a registered application in Azure AD named
bacula-m365-plugin. This section will show how to authorize it to
perform backup and restore operations over your target tenant. All required
steps to complete this authorization process are presented below.
Most of the procedures described in this section must be done by a tenant
administrator. A tenant administrator is a user who has been assigned the
Azure AD role Global administrator.
Below we show an schema of how this authorization method works:
You can get the exact same URL from the plugin command line itself, once
you have installed it in a client named {your_client_name} using the
following special Query command.
*.query plugin="m365:" client=127.0.0.1-fd parameter=register:57uia43-d107-17a2-a2g2-aa53c10tdahc
console=---- M365 PLUGIN REGISTER COMMAND ----
console=-------------------------------------
console=-------------------------------------
info=Open the following URI in a browser with your tenant admin credentials
console=-------------------------------------
console=-------------------------------------
uri=https://login.microsoftonline.com/57uia43-d107-17a2-a2g2-aa53c10tdahc/adminconsent?client_id=14a0b71a-d9ca-496c-b4c0-76a3cbb5dc33&state=12345&redirect_uri=https://www.baculasystems.com/m365-plugin-auth/common
console=-------------------------------------
console=-------------------------------------
info=Once you have accepted the provided permissions to the bacula-m365-plugin app..
info=You can get your ObjectId using the command below
console=-------------------------------------
console=-------------------------------------
command=.query plugin="m365: tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc" client={your_client_name} parameter=objectid
console=-------------------------------------
When opening the URI, Microsoft 365 will ask for credentials. You need to
authenticate as a tenant admin:
…and once they are correctly provided, the app will ask for all the
required permissions to backup and restore all of the supported elements:
The application needs all of the listed permissions to be able to work. If any of them is missing, backup or restore operationms will fail.
The image and list shown here are illustrative and some additional permissions
may be needed as the plugin evolves. However, we provide also here a text list of
current permissions needed:
Graph
Read and write user chat messages → Backup/Restore of Team Channels and Chats
Read and write user and shared calendars → Backup/Restore of group Calendars
Read and write all groups → Creation of Teams
Send user chat messages → Restore of Chats
Create, read, update and delete user’s tasks and task lists → Backup/Restore of Tasks
Read and write tags in Teams → For future support of Team tags
Read organization information → Find tenant name
Read and write files in all sites collections → Backup/Restore OneDrive and Sharepoint
Read and write all chat messages → Backup/Restore of Team Channels
Read all users full profiles → Find users to Backup/Restore
Read all channel messages → Backup of Team Channels
Read and write all user mailbox settings → Outlook categories and more future mailbox information
Read and write contacts in all mailboxes → Backup/Restore contacts
Read directory data → Security checks of objectid, list groups, etc
Read and write calendars in all mailboxes → Backup/Restore User Calendars
Read and write tabs in Microsoft Teams → Backup of Microsoft Teams tabs
Read and write all OneNote notebooks → Backup/Restore OneNote service*
Have full control of all site collections → Backup/Restore Sharepoint
Add and remove members from all Teams → Backup/Restore of Microsoft Teams members
Create chat and channel messages with anyone’s identity and with any timestamp
Manage Teams apps for all chats → Backup of Microsoft Teams apps
Manage Teams apps for all teams → Backup of Chats apps
Read and write the names, descriptions, and settings of all channels of all Teams → Backup of Microsoft Teams Channels
Add and remove members from all channels → Backup of Microsoft Teams Channels
Read and write managed metadata → Security checks of objectid, list groups, etc
Sign in and read user profile → Ability to connect to the tennant
etc.
Sharepoint Online
Full management of all site collections → Backup/Restore
Sharepoint (PnP)
Full management of Term Store → Backup/Restore Sharepoint (PnP)
Once it is confirmed, the browser will use the ‘redirect_uri’,
which is a page on baculasystems.com that will confirm the result of
the registration process.
The generated URI contains the parameter admin_consent=True if the action
was successful, and you will see a confirmation message in that case. Otherwise,
the operation may have not been successful for some reason and you will see an
error message.
Once that action is done, the tenant where our app now has permissions
will show the plugin in the Enterprise Apps section:
Clicking on the app, the tenant admin can always see the permissions
assigned:
The plugin needs a final parameter that is unique to each tenant and
the plugin app. This is ObjectID, and may be obtained from the
Overview app page, once step 2 has been completed:
The plugin can also obtain it from the command line using another
special Query command. You can see that this exact command is also
suggested in the command that shows the register URL:
Authorization Method B: Standalone app model with ‘bacula-m365-registrator’ (RECOMMENDED)
Bacula Systems has a registered application in Azure AD named
bacula-m365-registrator. This section will show how to authorize it to
perform application management operations over your target tenant.
Once you authorize it, you will be able to add a new bacula-m365-plugin-standalone
specifically for your tenant that you can directly control and that will be used only
in your environment.
This is the recommended authorization method for any production environment.
Below we show an schema of how this authorization method works:
You can get the exact same URL from the plugin command line itself, once
you have installed it in a client named {your_client_name} using the
following special Query command.
*.query client=127.0.0.1-fd plugin="m365:" parameter=register-registrator:57uia43-d107-17a2-a2g2-aa53c10tdahc
console=---- M365 PLUGIN REGISTER COMMAND ----
console=-------------------------------------
console=-------------------------------------
info=Open the following URI in a browser with your tenant admin credentials
console=-------------------------------------
console=-------------------------------------
uri=https://login.microsoftonline.com/57uia43-d107-17a2-a2g2-aa53c10tdahc/adminconsent?client_id=8ed4fad3-3f63-44fb-b7e0-e324895d7df9&state=12345&redirect_uri=https://www.baculasystems.com/m365-plugin-auth/standalone
console=-------------------------------------
When opening the URI, Microsoft 365 will ask for credentials. You need to
authenticate as a tenant admin:
…and once they are correctly provided, the app will ask for all the
required permissions to backup and restore all of the supported elements:
The application needs all of the listed permissions to be able to work.
If any of them is missing, the final command to add the application will fail.
The image shown here is illustrative and some additional permission may be
needed as the plugin evolves. To facilitate that information, we are
listing the permissions in text alongside the reason for their need.
Graph
- Sign in and read user profile: Connect to the tenant
- Read and write all groups: Necessary to manage some permission grants (delegated form)
- Read directory data: Find your tenant information, check service principals (delegated form)
- Manage all delegated permission grants: Generate delegated permissions grants for the end app
- Read and write all applications: Create/Update the new application (client credentials)
- Read organization information: Find your tenant information and show user-friendly progress
- Read and write all applications: Create/Update the new application (delegated form)
- Read and write all groups: Necessary to manage some permission grants (client credentials)
- Read directory data: Find your tenant information, check service principals (client credentials)
- Manage app permission grants and app role assignments: Add necessary plugin permissions
Once it is confirmed, the browser will use the ‘redirect_uri’,
which is a page on baculasystems.com that will confirm the result of
the registration process.
The generated URI contains the parameter admin_consent=True if the action
was successful, so you will see a confirmation message in that case. Otherwise,
the operation may have not been successful for some reason and you will see an
error message.
Once that action is done, the tenant where our app now has permissions
will show the plugin in the Enterprise Apps section:
Clicking on the app, the tenant admin can always see the permissions
assigned:
B-2. Add bacula-m365-plugin-standalone to your tenant
Once the registrator app is successfully registered into the target tenant, it
is possible to add the bacula-m365-plugin-standalone application as a
‘registered app’ using the following bconsole command:
This command needs to run operations with ‘client credential’ permissions, but
also with ‘delegated permissions’ where an admin user must be implied.
Therefore, at some point during its execution it will ask for the user interaction
in order to complete the authentication process using an special URL and a given
code. Once introduced, we will be asked about confirming the sign-in of
bacula-m365-registrator:
The process must be performed by a tenant administrator. On the other hand,
Microsoft Azure service needs some time in order to present newly created
objects. Therefore, the process will be paused a couple of times for about one
minute.
You need to wait 2-3 minutes (minimum!) after the first registration command
has been completed. If it is executed too soon it could fail, as the registrator
may not be completely available yet. If this happens, please, just run the command again.
*.query client=127.0.0.1-fd plugin="m365: tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc" parameter=add-app
console=---- M365 ADD APP ----
info=Creating destination config file in: /opt/bacula/working/m365/baculaenterprise/bacula_m365_config.conf
info=Creating application: bacula-m365-plugin-standalone in tenant: baculaenterprise ...
info=Waiting a bit so MS Azure cache has enough time to show our new Application...
info=Adding certificate key credential ...
info=Adding password credential ...
info=Application successfully created
info=We don't have a valid token. Please:
info=To sign in: use a web browser to open the page https://microsoft.com/devicelogin and enter the code CG2QYSK8Y to authenticate. You need to do it with an administrator user
info=Adding owner: johndoe@johndoe.onmicrosoft.com to the application ...
info=Owner successfully added
info=Creating service principal for application ...
info=Service principal successfully created
info=Waiting a bit so MS Azure cache has enough time to show our new ServicePrincipal...
info=Granting admin consent for permissions ...
info=Admin consent granted
info=Storing access configuration to the new app in file: /tmp/regress/working/m365/johndoe/bacula_m365_config.conf
info=Access configuration successfully stored
console=---- ADD APP COMPLETED SUCCESSFULLY ----
console=-------------------------------------
info=Use the generated configuration file in your m365 filesets. Just add the property config_file=/your/path/bacula_m365_config.conf to the line Plugin="m365: ..."
info=Sample: Plugin = "m365: config_file=/opt/bacula/etc/m365/mytenant/bacula_m365_config.conf service=drive"
console=-------------------------------------
console=-------------------------------------
In case the add-app command fails for any reason, please, wait a bit and just
run it again.
The result of the add-app command is the generation of a configuration file that
is intended to be referenced in any future fileset, as the output of the command
is indicating.
We recomend to copy that file to a convenience common location (for instance
/opt/bacula/etc/filesets) and set the same diretive in any M365 fileset:
#Thu Jul 01 12:30:34 CEST 2021 -> Generation Date
secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx -> Password secret auto-generated
appid=1587ba33-8eer-4512-bgb1-6d65fd393f51 -> Azure AD AppID of your bacula-m365-plugin-standalone
objectid=8jki8705-010p-4ee6-b2aa-c3klmn1b5b9 -> Service principal ID of your Application
tenant=e4wsdf55-de8a-4a83-945a-bfaabb6fa166 -> Your tenant id
Please, note that once this application is generated in your tenant, you have
full control of it. As a result, you can manually generate a new secret and set
the value yourself at your convenience whenever this is needed.
The add-app command discussed in section B-3 is prepared to automatically
regenerate a new password secret if it detects that the current ones are
expired or close to expire (6 months). Therefore, you could simply call it
again in case you need to renew the secret key which has a maximum life span
of 24 months in Azure. It would be possible to do this automatically using a
properly configured Admin Job.
The registrator application has elevated permissions, as without them it would
not be possible to manage applications and assoicated permissions. It is handy
to have it if you plan to run automatically the add-app command at some point,
as suggested in B-4.
However, if you want to remove it from your tenant after completing the add-app
process you could do it manually or you could call the following bconsole command:
Please, be aware that once you do that, you won’t be able to run the add-app
command. If you need to run it again after deletion, you will need to call
again the register-registrator command.
It is possible to check if the service principal of the target tenant (this is shown under ‘enterprise applications’
in Azure AD) has all the needed permissions to perform plugin operations.
.query client={your-client} plugin="m365: tenant={yourTenantId} objectid={yourobjectid}" parameter=permissions
error=Delegated permission Chat.ReadWrite was not found in your Azure service principal 8cee7605-0d5e-42b6-b269-c39f3411b5b9
error=Please review your app permissions
In case you are using the standalone application, you will need to run the command with appid and secret too:
Please, run this command at any time you have doubts about the correct app permissions.
Please note though, that the command is unable to check if you have activated or not ‘Protected APIs’ for Teams and Chats.
It will check the presence of those permissions, but it is up to Microsoft to enable those particular ones and they do not show
(at the time of writing) any flag indicating if they are enabled or not.
Migration from Authenticaton Method A to Authentication Method B
Perhaps you started to use this plugin using Method A and now you want to switch to Method B in order to enjoy its advantadges. This section
will show the steps needed in order to complete such migration.
Migration - Step 1: Remove bacula-m365-plugin from your tenant
The first step is to remove the common app ‘bacula-m365-plugin’ from your tenant. In order to do it, you need to go to your Azure portal and open
the ‘Azure Active Directory’ application as usual.
From there, please go to ‘Enterprise applications’ and locate bacula-m365-plugin app. You need to select the application and enter into ‘Properties’
section. From there, you need to click on the ‘Delete’ button:
Once you have removed the common application, you simply need to follow all the steps explained in this documment for authentication method B. In
case you are using BWeb and you have already registered you tenant, it is recommended to remove it first, and then add it also using method B. Otherwise,
for manual configuration, simply follow the steps mentioned in this document: Authorization Method B: Standalone app model with ‘bacula-m365-registrator’ (RECOMMENDED).
Once you have your new app in your tenant, you need to update all your filesets and:
- Replace objectid with the new value associated to your app in your tenant
- Add appid and secret associated to your app in your tenant
Once the plugin is successfully authorized, it is possible to define regular filesets for backup jobs in Bacula,
where we need to include a line similar to the one below, in order to call the M365 Plugin:
It is strongly recommended to use only one ‘Plugin’ line in every fileset. The plugin offers the needed flexibility to combine different modules backup inside the same plugin line. Different tenants, in case of existing, should be using different filesets and different jobs.
Below sub-sections list all the parameters you can use to control M365 Plugin behavior.
In this plugin, any parameter allowing a list of values can be assigned with a list of values separated by ‘,’.
Note
Starting with version 14.0 of Bacula Enterprise you can also perform fileset configuration directly using a dedicated BWeb
M365 Wizard. Please go to section BWeb Management Console to see more details.
These parameters are common and applicable to all the modules of the M365 Plugin.
Option
Required
Default
Values
Example
Description
abort_on_error
No
No
No, Yes
Yes
If set to Yes: Abort job as soon as any error is found with any element. If set to No: Jobs can continue even if it they found a problem with some elements. They will try to backup or restore the other and only show a warning
config_file
No
The path pointing to a file containing any combination of plugin parameters
/opt/bacula/m365.settings
Allows to define a config file where configure any parameter of the plugin. Therefore you don’t need to put them directly in the Plugin line of the fileset
log
No
/opt/bacula/working/m365/m365-debug.log
An existing path with enough permissions for File Daemon to create a file with the provided name
/tmp/m365.log
Generates additional log in addition to what is shown in job log. This parameter is included in the backend file, so, in general, by default the log is going to be stored in the working directory.
debug
No
0
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
Debug level. Greater values generate more debug information
Generates the working/m365/m365-debug.log* files containing debut information which is more complete with a greater debug number
path
No
/opt/bacula/working
An existing path with enough permissions for File Daemon to create any internal plugin file
/mnt/my-vol/
Uses this path to store metadata and temporary files
tenant
Yes
A valid tenant id string
57uia43-d107-17a2-a2g2-aa53c10tdahc
The tenant ID where the plugin will connect to in order to run backups or restores. Please, check section 5.1 of this paper for more information
objectid
Yes
String representing the objectid related to bacula-m365-plugin Azure app and the provided tenant id
56ddf1h9-eb5d-42nf-bac7-7b019fd284g5
The object ID of the plugin app of the plugin in Azure, once this is registered in the target tenant. Please, check section 5.1 of this paper for more information
appid
No
String representing the appid associated to bacula-m365-plugin-standalone Azure app registered in the configured tenant id
89tt4hu7-r4h7-kied-56gu-0895kf94jr9d
A valid appid string associated to bacula-m365-plugin-standalone Azure app registered in the configured tenant id. Please, check section 5.1 of this paper for more information
secret
No
String representing the associated appid secret
Jn8.lU-B.3P5gIRTGY6M.Xl3e29oQ6Xaf~
The secret associated to the m365 Azure app corresponding the configured appid string. Please, check section 5.1 of this paper for more information
token_cache_file
No
token_cache.json
A file name located in a valid existing path (it’s possible to put only the name of the file and ‘working’ dir will be used)
my_cache_file.json
The path that will be used to store the login cache for the device code flow authenticated users, wich is relative to the working tenant folder (working/tenant-name/token_cache.json)
service
No
email, drive, sharepoint, contact, calendar, onenote, tasks, teams, chat (list parameter: it can contain 0, 1 or more elements separated by ‘,’)
drive
Establish the service or services that will be backed up. If this is not set, the plugin will try to backup all supported services. It is recommended to split the work among different jobs when several services need to be applied. Therefore, even if this field is not required, it is strongly recommended to use it in every backup job.
owner_restore_protection
No
No
Yes, No
Yes
Enable owner restore protection feature, where the owner of the data being restored will be notified with an email and the restore job will be blocked until he/she enters a code and his/her M365 credentials to approve the operation
proxy_host
No
String representing DNS Name or IP address of the http(s) proxy
myproxy.example.com
Set up a proxy to make any plugin HTTP connection
proxy_port
No
Integer
3981
Set up the proxy port
proxy_user
No
String of proxy user
admin
Set up the proxy user
proxy_password
No
String of proxy password
myPass123
Set up the proxy user password
Note
The path option is adjustable while M365 jobs are in progress. The running jobs will continue to use the current spool area, and new jobs will use the new spool area defined by the path option.
If specifying more than one service in the fileset, please note that they will run in parallel. Because of that, we recommend to decrease a little the concurrency.
Activating proxy mody will route all the requests through the proxy. However, it is needed DNS resolution to be working separately. Hence, the client where the FD is running will need to have a proper and working DNS Server configured. Other option is to setup the /etc/hosts file manually with the IP addresses of Microsoft authentication servers, but they could change over time.
At the time of writing they are:
40.126.31.2 login.microsoftonline.com
20.190.160.131 login.microsoft.com
Following parameters are common to all M365 modules (and even with some other plugins), but are advanced ones. They should not be modified in most common use cases.
Option
Required
Default
Values
Example
Description
stream_sleep
No
1
Positive integer (1/10 secconds)
5
Time to sleep when reading header packets from FD and not having a full header available
stream_max_wait
No
120
Positive integer (seconds)
360
Max wait time for FD to answer packet requests
time_max_last_modify_log
No
86400
Positive integer (seconds)
43200
Maximum time to wait to ovewrite a debug log that was marked as being used by other process
logging_max_file_size
No
50MB
String size
300MB
Maximum size of a single debug log fileGenerates the working/m365/m365-debug.log* files containing debut information which is more complete with a greater debug number
logging_max_backup_index
No
25
Positive integer (number of files)
50
Maximum number of log files to keep
log_rolling_file_pattern
No
m365.log.%d{dd-MMM}.log.gz”
No, Yes
Yes
Log patter for rotated log files
split_config_file
No
=
Character
:
Character to be used in config_file parameter as separator for keys and values
opener_queue_timeout_secs
No
1200
Positive integer (seconds)
3600
Timeout when internal object opener queue is full
publisher_queue_timeout_secs
No
1200
Positive integer (seconds)
3600
Timeout when internal object publisher queue is full
The internal plugin logging framework presents some relevant features that we are going to describe:
The “.log” files are rotated automatically. Currently each file can be 50Mb at maximum and the plugin will keep 25 files.
This behavior can be changed using the internal advanced parameters: logging_max_file_size and logging_max_backup_index
The “.err” file can show contents even if no real error happened in the jobs. It can show contents too even if debug is disabled. This file is not rotated, but it is expected to be a small file in general. If you still need to rotate it, you can include it in a general rotating tool like ‘logrotate’.
Backups in paralel and also failed backups will generate several log files. For example: m365-debug-0.log, m365-debug-1.log…
These set of parameters are common to all modules and they are advanced ones. They should not be modified in general. They can be used to tune the behavior of the plugin to be more flexible in particular bad network environments or when significant job concurrency is happening, etc.
Option
Required
Default
Values
Example
Description
backup_queue_size
No
100
0-500
1
Number of maximum enqueued internal operations between service static internal threads (there are 3 communicating through queues with the set size: service fetcher, service opener and general publisher to bacula core). This could potentially affect graph api concurrent requests and consequently, Graph throttling. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel
concurrent_threads
No
10
0-100
1
Number of maximum concurrent backup threads running in parallel in order to fetch or open data for running download actions. This means every service fetcher and service opener will open this number of child concurrent threads. This will affect graph api concurrent requests. Graph API can throttle requests depending on a variety of circumstances, but this parameter impacts it directly. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel. If you want to have a precise control of your parallelization through different jobs, please set up this value to 1. Please be careful also with the memory requirements, multi-threaded increases very significantly memory consumption per job
concurrent_listing_threads
No
5
0-20
1
Number of maximum concurrent backup page listing threads running in parallel in order to fetch sets of data for some modules. Currently it’s only used in the email module. This parameter will also affect graph api concurrent requests. Graph API can throttle requests depending on a variety of circumstances, but this parameter impacts it directly. It is only needed to modify this parameter, in general, if you are going to run different jobs in parallel. If you want to have a precise control of your parallelization through different jobs, please set up this value to 1. Please be careful also with the memory requirements, multi-threaded increases very significantly memory consumption per job
graph_timeout
No
9000
Positive integer (milliseconds)
60000
Graph call timeout inside HttpClient
graph_read_timeout
No
300
Positive integer (milliseconds)
30000
Graph read timeout inside HttpClient
graph_retries
No
5
Positive integer (number of retries)
10
Graph number of retries for retry-candidate requestsInclude some stats information in the joblog. Useful to measure task times
graph_retry_delay
No
5
Positive integer (seconds)
10
Graph delay between retries
general_network_retries
No
5
Positive integer (number of retries)
10
Number of retries for the general M365 external retry mechanism
general_network_delay
No
50
Positive integer (seconds)
100
General M365 Plugin delay between retries
throttled_wait_time
No
300
Positive integer (seconds)
600
Extra wait time for throttled situations where the timeout is not provided or not got from MS API. In onenote module this is multiplied by 2. Once MS throttles a request is better to not retry too soon or the changes to continue with rejected requests will increase significantly
stats
No
No
No, Yes
Yes
Include some stats information in the joblog. Useful to measure task times
Note
graph_list_page_size had default value 500 before BEE 14.0.7. A higher value for this parameter can improve the performance at it reduces the number of API calls done to M365 service. However, the service can also be overloaded and return more HTTP 503 errors (Bad Gateway), especially for the email module.
Starting from version 16.0.3, default values for backup_queue_size and concurrent_threads have been increased, also the allowed ranges.
The following list of parameters are commonly shared through any module used into the same fileset line and are intended to select the target entities to backup. Every module subsection mentions what entities are supported too.
Option
Required
Default
Values
Example
Services
Description
user
No
Valid email addresses of existing users on the selected tenant separated by ‘,’
email, drive, contact, calendar, onenote, chat, tasks
Backup mailboxes, drive unit spaces, categories, or whatever service is selected of this list of users. If no user is provided: - The backup will be done for all accessible users of the given tenant if no other entry has value either (group or site parameters)
user_exclude
No
Valid email addresses of existing users on the selected tenant separated by ‘,’
email, drive, contact, calendar, onenote, chat, tasks
Exclude selected mailboxes or onedrive spaces. If this is the only parameter found for selection, all elements will be included and this list will be excluded
email, drive, contact, calendar, onenote, chat, tasks
Backup matching user mailboxes or onedrive spaces. Please, only provide list parameters (user + user_exclude) or regex ones. But do not try to combine them
email, drive, contact, calendar, onenote, chat, tasks
Exclude matching user mailboxes or onedrive spaces from the selection. Please, only provide list parameters (user + user_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded
site
No
Valid names of existing user sharepoint sites on the selected tenant separated by ‘,’
Communication site, Dev site
drive, sharepoint, onenote
Backup onedrive site library space (OneDrive service) belonging to this list of sharepoint sites or the site itself (Sharepoint service). If no site name is provided: - OneDrive: user and site variables will be checked. If no one has value either, backup will be done for all accessible users, groups or sites of the given tenant - Sharepoint: all accessible sites of the tenant will be backed up
site_exclude
No
Valid names of existing user sharepoint sites on the selected tenant separated by ‘,’
Test site
drive, sharepoint, onenote
Exclude listed sites from backup. If this is the only parameter found for selection, all elements will be included and this list will be excluded
site_regex_include
No
Valid regex
.*site
drive, sharepoint, onenote
Backup matching sites. Please, only provide list parameters (site + site_exclude) or regex ones. But do not try to combine them
site_regex_exclude
No
Valid regex
Test.*
drive, sharepoint, onenote
Exclude matching sites from the selection. Please, only provide list parameters (site + site_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded
group
No
Valid names of existing user groups on the selected tenant separated by ‘,’
Dev Team, Bacula Users
drive, onenote, calendar, teams, tasks
Backup onedrive sharepoint library spaces or notebooks of this list of groups. If no group is provided, user and site variables will be checked. If no one has value either, backup will be done for all accessible users, groups or sites of the given tenant
group_exclude
No
Valid names of existing user groups on the selected tenant separated by ‘,’
Monitoring, External
drive, onenote, calendar, teams, tasks
Exclude selected group. If this is the only parameter found for selection, all elements will be included and this list will be excluded
group_regex_include
No
Valid regex
.*Management
drive, onenote, calendar, teams, tasks
Backup matching group onedrive spaces. Please, only provide list parameters (group + group_exclude) or regex ones. But do not try to combine them
group_regex_exclude
No
Valid regex
.*External
drive, onenote, calendar, teams, tasks
Exclude matching groups from the selection. Please, only provide list parameters (group + group_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded
The following parameters have been deprecated since version 12.8.3, in order to provide more flexibility. General shared parameters have been transformed in specific ones for each service
The plugin is able to restore to the local file system on the server where the File Daemon is
running or to the Microsoft 365 environment. The method is
selected based on the value of the where parameter at restore time:
Empty or ‘/’ (example: where=/) → Microsoft 365 restore will be triggered
Any other path for where (example: where=/tmp) → Local file system restore will be triggered
When using the Microsoft 365 restore option, the following
parameters may be modified by selecting ‘Plugin Options’ during the bconsole restore
session:
Option
Required
Default
Values
Example
Services
Description
destination_user
No
Existing email address on the target Azure AD Tenant
email, drive, contact, calendar, onenote, tasks, chat
Destination User where restore data will be uploaded. If no user is set, every selected file will be restored in the original account
destination_group
No
Existing group name address on the target Azure AD Tenant
Marketing
drive, calendar, onenote
Destination Group where restore data will be uploaded. If no group is set, every selected file will be restored in the original group
destination_site
No
Existing site name/display name on the target Azure AD Tenant
live @ contoso
drive, sharepoint, onenote
Destination Site where restore data will be uploaded. If no site is set, every selected file will be restored in the original site
destination_path
No
Existing path on the selected user (mailfolder path or onedrive folder path)
Inbox
email, drive, contact, calendar, sharepoint
Destination folder where all selected files to restore will be restored. If no path is set: - If no user is set either, every element will go to its original location - If a user is set using the variable destination_user: - Elements belonging to destination_user will be restored in their original location - Elements belonging to different users than destination_user will be restored in a new folder using the email address of the original user of the element
destination_drive
No
Existing drive name or drive id in the destination entity
documents
drive
Destination drive where restore data will be uploaded. If no drive is set, every selected file will be restored in the original drive
send_report
No
0
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
1
email, drive, contact, calendar, sharepoint, tasks
Send a report to the user where every restore action is listed (generated emails, generated onedrive files..). - In the email service this will be an email in the user Inbox - In onedrive service this will generate a new text file in the top restore folder
allow_duplicates
No
1
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
0
email, contact, calendar
In Email services indicates if we want to allow to create duplicate emails or if we want to skip emails that are already in place. Important: This is done using the Microsoft Message ID. Therefore, non intuitive situations can happen. For example, if the original message is deleted and we restore several times using this option, we will see duplicate emails anyway
drive_skip_sharedwitme
No
0
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
1
drive
Skip restoring shared with me elements even if they are selected.
skip_versions
No
1
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
0
drive
Skip restoring former file versions (tagged with ‘###date’) even if they are selected. Important: Notice that this parameter is enabled by default, as we consider not restoring file versions the most common case. You need to disable it in order to have this kind of files restored
restore_share_permissions
No
0
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
1
drive, calendar
Restore share permissions of every element in order to regenerate sharing information as allowed identities, shared links, etc. Please note that this parameter is disabled by default.**Important**: Notice that this parameter is disabled by default, as we consider not restoring sharing permissions the most common case. You need to enable it in order to haver shared permissions restored
drive_send_invitations
No
0
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
1
drive
Send email invitations for restored OneDrive shares - Note you need to enable drive_restore_share_permissions in order to this parameter to have any effect
drive_invitations_message
No
String
I’m sharing with you automatically restored shares
drive
Set invitations message for restored OneDrive shares - Note you need to enable drive_restore_share_permissions in order to this parameter to have any effect
Set the local path of the PnP Sharepoint Site template you want to be using instead of the one contained in the backup itself.
notebook_name
No
String
myRestoredNotes
onenote
Set the owner of the new restored sharepoint site
notesection_name
No
String
Technology
onenote
Set the name of restored onenote notebook
notesection_group_name
No
String
ForWork
onenote
Set the name of restored onenote section
team_name
No
String
MyRestoredTeam
teams
Set the name of restored team (it will create also a new associated group)
team_channel_name
No
String
MyRestoredChannel
teams
Set the name of restored channel
team_private_channels_mode
No
DELEGATED
DELEGATED, PUBLIC, SKIP
PUBLIC
teams
Set the private channels restore mode
team_guest_members_enable
No
0
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
1
teams
Enalbe restore of team members with role ‘guest’. That kind of external user needs delegated permissions to be added, so potentially the restore could ask for user interaction (of first team owner found)
chat_topic
No
String
MyRestoredChat
chat
Set the topic of restored chat (supported only for group chats)
tasklist_name
No
String
ProjectA
tasks
Set the name of restored task list
tasklist_skip_sharedwithme
No
1
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
0
tasks
Skip shared with me tasks in the restore process
plan_name
No
String
RestoredPlan
tasks
Set the name of restored planner plan
plan_create_tab
No
1
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
0
tasks
Create or not an associated tab inside the associated team
local_path_json_objects
No
{path}/bacula-restores
String
/home/me/restores
onenote, sharepoint
Set local path to restore objects that cannot be restored directly into the M365 service
foreign_container_generation
No
1
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on
0
email, drive, contact, calendar
Generate a general container (usually a folder) to put inside restored objects coming from different entities. For example, if we restore emails from user a@tenant.com into Mailbox of user b@tenant.com, this option enabled will generate an automatic folder a@tenant.com inside the destination restore folder used over destination user a@tenant.com
tenant
No
A valid tenant id string where some bacula m365 plugin app is registered
57uia43-d107-17a2-a2g2-aa53c10tdahc
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat
Set the destination tenant id for a cross-tenant restore
appid
No
Appid of bacula-m365-plugin-standalone app registered in the provided tenant
56ddf1h9-eb5d-42nf-bac7-7b019fd284g5
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat
Set the destination appid of bacula-m365-plugin-standalone for a cross-tenant restore
objectid
No
Object id associated to set up appid
89tt4hu7-r4h7-kied-56gu-0895kf94jr9d
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat
Set the objectid associated to the appid in a cross-tenant restore
secret
No
Valid secret associated to appid
Jn8.lU-B.3P5gIRTGY6M.Xl3e29oQ6Xaf~
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat
Set the secret associated to the appid set in the above parameter
debug
No
0
0, 1, 2 ,3, 4, 5, 6, 7, 8, 9
3
email, drive, contact, calendar, sharepoint, onenote, tasks, teams, chat
Since Bacula version 16.0.7, a new solution has been introduced, so that
each object can be backed up separately with different Jobs to maximize the
throughput and the resiliency. It is highly recommended to use this new solution for that purpose -
Automatic Object Integration (Scan Plugin).
See an example for Microsoft 365Example.
The plugin supports enough flexibility to configure almost any type of
desired backup. Multiple Plugin= lines should not be spoecified in
he Include section of a FileSet for the M365 Plugin.
Fileset examples for every supported service are linked below.
For common purposes, the following two examples show how to configure
an external config file or configure the number of threads:
Restore operations are done using standard Bacula Enterprise bconsole commands.
The where parameter controls if the restore will be done locally to the
File Daemon’s file system or to the Microsoft 365 service:
where=/ or empty value → Restore will be done over M365
where=/any/other/path → Restore will be done locally to the File Daemon file system
Restore options are described in the Restore parameters section of this document, so here
we are going to simply show an example restore session, particularly
this example is about OneDrive service:
*restore where=/
First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.
To select the JobIds, you have the following choices:
1: List last 20 Jobs run
2: List Jobs where a given File is saved
3: Enter list of comma separated JobIds to select
4: Enter SQL list command
5: Select the most recent backup for a client
6: Select backup for a client before a specified time
7: Enter a list of files to restore
8: Enter a list of files to restore before a specified time
9: Find the JobIds of the most recent backup for a client
10: Find the JobIds for a backup for a client before a specified time
11: Enter a list of directories to restore for found JobIds
12: Select full restore to a specified Job date
13: Select object to restore
14: Cancel
Select item: (1-14): 5
Automatically selected Client: 127.0.0.1-fd
Automatically selected FileSet: FS_M365_DRIVE
+-------+-------+----------+-------------+---------------------+-------------------+
| jobid | level | jobfiles | jobbytes | starttime | volumename |
+-------+-------+----------+-------------+---------------------+-------------------+
| 11 | F | 190 | 332,978,505 | 2021-01-22 10:39:34 | TEST-2021-01-22:0 |
| 12 | I | 1 | 550 | 2021-01-22 10:43:05 | TEST-2021-01-22:0 |
+-------+-------+----------+-------------+---------------------+-------------------+
You have selected the following JobIds: 11,12
Building directory tree for JobId(s) 11,12 ... +++++++++++++++++++++++++++++++++++++++++++++++
188 files inserted into the tree.
You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.
cwd is: /
$ cd /@m365/baculaenterprise/adelev@baculaenterprise.onmicrosoft.com/drive/root:
cwd is: /@m365/baculaenterprise/adelev@baculaenterprise.onmicrosoft.com/drive/root:/
$ ls
Docs/
pluginTest.drive.deltaLink
sharedWithMe/
test4###v1.0_2020-11-25_153507.html
test4###v2.0_2020-11-25_153507.html
test4###v3.0_2020-12-02_180612.html
test4###v4.0_2020-12-02_180612.html
test4###v5.0_2020-12-03_134422.html
test4###v6.0_2020-12-02_180612.html
test4.html
testlink123###v1.0_2020-11-26_123244.url
testlink123###v2.0_2020-11-26_123244.url
testlink123###v3.0_2020-12-02_180613.url
testlink123###v4.0_2020-12-02_180613.url
testlink123.url
$ mark *
189 files marked.
$ done
Bootstrap records written to /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
The Job will require the following (*=>InChanger):
Volume(s) Storage(s) SD Device(s)
===========================================================================
TEST-2021-01-22:0 File FileStorage
Volumes marked with "*" are in the Autochanger.
189 files selected to be restored.
Using Catalog "MyCatalog"
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2021-01-22 11:54:55
Catalog: MyCatalog
Priority: 10
Plugin Options: *None*
OK to run? (yes/mod/no): mod
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Restore Client
6: When
7: Priority
8: Bootstrap
9: Where
10: File Relocation
11: Replace
12: JobId
13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : m365: service=drive tenant="574dda43-d107-48e2-a7f2-aa51c10bdaec" objectid="31ddf4b1-
ed5d-432f-bac7-7b946fd23394" user="adelev@baculaenterprise.onmicrosoft.com" drive_version_history=yes debug=3
Plugin Restore Options
Option Current Value Default Value
destination_user: *None* (*None*)
destination_path: *None* (*None*)
send_report: *None* (0)
email_allow_duplicates: *None* (1)
drive_skip_sharedwithme: *None* (0)
drive_skip_versions: *None* (1)
drive_restore_share_permissions: *None* (0)
drive_send_invitations: *None* (0)
drive_invitations_message: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (yes/mod/no): mod
You have the following choices:
1: destination_user (Destination User)
2: destination_path (Destination Path in M365)
3: send_report (Send report of the restore operation to the affected user)
4: email_allow_duplicates (Allow Duplicate Emails)
5: drive_skip_sharedwithme (Skip restoring shared with me elements even if they are selected)
6: drive_skip_versions (Skip restoring file former versions (tagged with '###date') even if they are selected)
7: drive_restore_share_permissions (Restore share permissions of items so they are shared if they originally
were)
8: drive_send_invitations (Send email invitations for restored OneDrive shares)
9: drive_invitations_message (Set invitations message for restored OneDrive shares)
10: debug (Change debug level)
Select parameter to modify (1-10): 2
Please enter a value for destination_path: MY_RESTORE_PATH
Plugin Restore Options
Option Current Value Default Value
destination_user: *None* (*None*)
destination_path: MY_RESTORE_PATH (*None*)
send_report: *None* (0)
email_allow_duplicates: *None* (1)
drive_skip_sharedwithme: *None* (0)
drive_skip_versions: *None* (1)
drive_restore_share_permissions: *None* (0)drive_send_invitations: *None* (0)
drive_invitations_message: *None* (*None*)
debug: *None* (*None*)
Use above plugin configuration? (yes/mod/no): yes
Run Restore job
JobName: RestoreFiles
Bootstrap: /tmp/regress/working/127.0.0.1-dir.restore.3.bsr
Where: /
Replace: Always
FileSet: Full Set
Backup Client: 127.0.0.1-fd
Restore Client: 127.0.0.1-fd
Storage: File
When: 2021-01-22 11:54:55
Catalog: MyCatalog
Priority: 10
Plugin Options: User specified
OK to run? (yes/mod/no): yes
Job queued. JobId=14
You can perform cross-tenant restores using the restore variables:
- tenantid
- appid
- objectid
- secret
Obviously, it is needed to set up the destination tenant values, where one of the authentication methods
should be applied first.
If you are using the standalone app authetication model, you will need to use the four values in your restore session.
If you want to use the common app authentication model, you won’t need to put the secret. On the other hand, please,
ask the bacula enterprise edition support team in order to get the correct value for appid.
It is possible to list information using the bconsole .ls command and providing a path. In general, we need to provide the service parameter and a path representing a folder or object (like calendars).
There are some general commands (user, groups, sites), the rest of the commands need to point to the correct service.
Here we are showing these 3 commands using the bconsoel .ls command, but notice
you may also use them with the query interface (keep your
variable values, but apply something like: .query plugin=”…” client=xxxx parameter=xxx). Note that the
referenced ‘config_file’ should contain the connection parameters (tenant, objectid, appid and secret).
# List contents of Inbox folder of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=email user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Inbox
# List contents of Sent folder of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=email user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/Sent
# List all contacts of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=contacts user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/
# List contacts of user adelev@baculaenterprise.onmicrosoft.com and folder MyContactsDir
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=contacts user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyContactsDir
# List calendar events of user adelev@baculaenterprise.onmicrosoft.com and her calendar MyCalendar
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=calendar user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyCalendar
# List onenote pages of user adelev@baculaenterprise.onmicrosoft.com and her section secA included in section Group groupA
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=onenote user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/MyNotebook/sectionGroups/groupA/sections/secA
# List contents of OneDrive folder 'folderA' of user adelev@baculaenterprise.onmicrosoft.com
*.ls plugin="m365: config_file=/opt/bacula/etc/m365/exaple.tenant.conf service=drive user=adelev@baculaenterprise.onmicrosoft.com" client=127.0.0.1-fd path=/folderA
This plugin provides a query command allowing to check the performance of a given setup. The process consist on uploading some randomly generated files of a customizable size (and a customizable number of them) to a target tenant,
then download them using the same method the backup itself employs. Upload and download times are measured. An average is calculated and all the information is presented to the user.
Please, note that this command is currently working in single-threaded mode, while the plugin is running with multiple threads (configurable through concurrent_threads variable), so you could have better results with the actual backup process.
Currently, theh plugin supports two types of performance commands: perf for onedrive (default) and perf for email. So the value service should be: drive or email.
The behavior of this service from performance perspective can differ significantly. Below we detail some important aspects:
On Onedrive and other services implying user files, it is needed to request a ‘download session’ for each file.
Once we have it, the download starts. This requires significantly more time than just retrieving simple objects.
In general, objects are not retrieved one by one. We request for lists of objects, that later on are processed.
For objects that are not pointing to any other file (as emails without attachments), the time needed per object is very low.
For Onedrive service, in addition to user, you can also use group or site (site name or siteId) for this command. The final output will be similar to the following:
...
date=2021-09-06 17:25:56
tenant_name=johndoe.onmicrosoft.com
upload-speed-average=3.3 MiB/s
download-speed-average=32.2 MiB/s
latency-average=482.00 ms
console=---- Test END ----
You can control the size and the number of files used using the following query parameters:
- q_perf_size (in bytes)
- q_perf_files (number of files)
By default, their values are:
- q_perf_size = 209715200
- q_perf_files = 3
For Email service, specifying a user is mandatory. The command can generate simple emails, but also emails with one attachment. The final output can be similar to the following:
...
console=---- Summary ----
date=2021-11-11 17:05:18
tenant_name=johndoe.onmicrosoft.com
upload-speed-average=44.6 KiB/s
download-speed-average=220.5 KiB/s
latency-average-per-list-call=510.83 ms
latency-average-per-message=36.18 ms
latency-average-attachments=617.64 ms
console=---- Test END ----
The behavior of the email perf command is a little different from the Onedrive one. Instead of uploading/downloading one message at a time, here everything is uploaded first. After that
everything is downloaded. The reason is here the backup process uses lists of objects and, unless there are attachments implied, that’s all, the objects are simply stored from that list.
Similarly to the onedrive perf command, you can control the size and the number of emails. However, there is an extra parameter to control the size of the attachments:
q_perf_size (in bytes)
q_perf_size_attachments (in bytes)
If not specified, no attachment will be generated
If specified, one simple text attachment will be generated together with each email
Sometimes there is an element that causes some error while fetching it in the email service and it’s difficult to identify it by its reported debug id. For
this situation, there is a decoding command:
As the example shows, the query parameter is ‘query=decode|{url}’. Url should be the URL until the element we are interested into (so the attachment here).
Another useful command shows all the options the plugin can accept as parameters, categorized by section:
The output of this command will return a json structure with all the available options.
It is recommended, to split the target backup between
different groups of entities or even having one job per entity (user,
group, site, etc). This way errors in one job will not invalidate a whole
backup cycle where some entities have been successful and some others
had errors. This also makes easier to identify the cause of the error.
Microsoft public APIs impose a variety of boundaries that need to be
considered. If a boundary is crossed, the corresponding API call will
fail and the application will need to wait some amount of time to retry, which is
different depending on the boundary crossed.
It is crucial to plan an adequate strategy to backup all the elements
without reaching API boundaries. A single job implements some parallelism which can be reduced until a point, if necessary,
using the variable backup_queue_size (default value is 30). This variable controls the
size of the internal queues communicating the internal threads, that are designed to fetch, open and send every item
to Bacula core. Reducing its size will produce, ultimately (with a value of 1 for example), an
execution very similar to a single threaded process. On the othere hand the plugin has concurrent_threads which controls the number
of simultaneous processes fetching and downloading data (default value is 5).
If you are going to launch different jobs in parallel it is recommended to configure different services
for each of them (protect in parallel email, drive, contacts…). However, be careful with
the concurrency over the same service (in general, it is recommended a maximum of 4-5 jobs working with the same service)
and plan a step-by-step testing scenario before putting it into production. Other important point is the timing schedule, as some
boundaries are related to time-frames (number of request per 10 minutes or 1 hour, for example). If you detect you reach boundaries
when running all your backups during a single day of the week, please try to use 2 or 3 days and spread the load through them in order
to achieve better performance results.
More information about Microsoft 365 Graph API boundaries may be found here:
It is necessary to have at least enough disk space available for the size of the largest file in the
backup session. If you are using concurrency between jobs or through the same job (by default this
is the case through the concurrent_threads=5 parameter), you would need at least that size for the
largest file multiplied by the number of operations in parallel you run.
The performance of this plugin is highly dependent on many external factors:
ISP latency and bandwidth
Network infrastructure
FD Host hardware
FD Load
…
In summary, it is not possible to establish an exact reference about how
much time a backup will need to complete.
As reference values, in testing environments and using sequential calls
we had the following results:
3000 emails in the same folder of a single mailbox, with emails
ranging from 500 to 3000 words - about 10K to 30K in size
The total time required to back them up was: 88243 ms.
That implies a total time per email of about 30ms
Concurrency has also been tested. For example, when using single-threaded mode (concurrent_threads=1) with emails, good values where found for 4-6 concurrent jobs
that allowed us to backup around 200 emails per second (sizes between 20K-30K) when no attachments
were implied. As more attachments were implied, the number of emails will decrease very significantly, while
the overall size speed will be increased. This can be generalized:
Many little objects to protect -> More objects per second, but less speed (MB/s)
Big files to protect -> Less objects per second, but greater speed (MB/s)
It is recommended to benchmark your own environment in base to your requirements and needs.
The automatic parallelization mechanism (using concurrent_threads=x, default is 5) should
work well for most scenarios, however, fine tune is possible if we define one job per entity and we control how many of them run in pararllel, together to decrease the concurrent_threads
value in order to avoid throttling from MS Graph API.
There are many different possible strategies to use this plugin, so please, study what is best suiting for your needs before deploying the jobs for your entire environment, so you can get best possible results:
You can have a job per entity (users, groups, sites…) and all services
You can have inside a job multiple entities and only some services
You can split your workload through an schedule, or try to run all your jobs together.
You can run jobs in parallel or take advantadge of concurrent_threads and so run less jobs in parallel
You can select what services to backup or backup them all
You can backup whole services to backup or select precisely what elements you really need inside each service (folders, paths, exclussions…)
For delegated permissions services, you can proceed with the login previously to backups (recommended) or you can wait for the job to ask for what it needs
Each Microsoft 365 service presents its own particularties. Moreover each deployment and each environment can be protected through different strategies depending on the size, number of entities and other parameters. However, there are some general considerations that can be helpful to consider for a selection of services:
Sharepoint Online is the service where the target data varies the most among the M365 services. Sites can be structured very differently and there exists many different kinds of Sharepoint lists. In addition
to that, some of them could have very large document libraries, while others could be very sparse.
For these reasons, it is recommended to split Sharepoint jobs to be as atomic as possible.
Ideally jobs should be split up to a single site backup job. This will help to monitor the health of the backups, to keep good backups for all sites presenting no issues and but to quickly troubleshoot any site presenting
any kind of inconvenience. Usually, this is especially needed when Sharepoint service are intensively used.
If sites are created and removed often, another option to avoid a frequent configuration change is to employ the site_regex_include plugin variable and use the naming patterns to
set up the jobs, so they will also include future created sites (possible examples: .*Team, .*Web, Dev.* … ).
In order to facilitate the job split, it is possible to use the listing or query commands described in List general info: Users, groups, sites to have the site names list.
Another best practice is to split Sharepoint sites into two parts with two different jobs:
- Pure Sharepoint job: Use sharepoint service (service=sharepoint site=SiteName) and the parameter sharepoint_include_drive_items=no.
- Drive part job: Use drive service (service=drive site=SiteName)
Sharepoint jobs, when using the associated powershell commands, cannot be run in parallel. One will wait for any other session to finish, before executing the corresponding commands. Using the above strategy makes Sharepoint jobs to be much more quick and light, so it’s a lot easier to control such concurrency.
Other ways to control the concurrency in Bacula Enterprise are:
Designing separated time windows through scheduling
Sending all Sharepoint jobs to the same pool and the associated storage resource (both of them dedicated to Sharepoint backups) and limiting ConcurrentJobs to the desired value in that particular storage
Having 1 specific client resource for Sharepoint and use ConcurrentJobs parameter there.
In general, it’s not needed to enable system lists, hidden lists or even the versioning of the elements. All those options will impact the performance of the jobs directly, so you should plan carefully the usage of such parameters.
Some Sharepoint sites present an issue with the hashcheck of the files contained in some of its document libraries. If the hash is not correct at M365 side, it does not match the one that is calculated by the plugin locally, while the file is correct. This situation causes a lot of error messages around the hash and also
many extra calls to the service to re-check this hash. When this situation happens, it is recommended to disable the hashcheck mechanism for the affected site using the parameter ‘drive_disable_hashcheck=true’.
It is recommended to be selective with the information to be protected and make one job for each selected entity: that is one job per user, group or site.
In case throttling problems appears with this strategy, jobs will need to be spread into larger time windows.
It is not recommended to include onenote for entity focused strategies where there is one job per user including all services (email, drive, contact, calendar ..). It is recomended to have specific onenote jobs.
This service uses delegated permissions. They can be protected together, but we recomend to run the ‘login’ query command for each individual user before running the actual jobs.
Email service allows you to protect not only email and attachments, but also settings, user categories and an additional MIME copy for each email with its attachments. MIME files are very useful if you are planning to migrate your data from M365 to any other Email tool, however, the cost of getting each MIME file is very high and it duplicates backup time. MIME files are not needed to perform a local restore or a restore over M365. Therefore, it is recommended to disable MIME backup for a general basis and use it only in case of planning a complete migration of the data.
Similarly to other services, it is recommended to split the backup set into small sets of users (ideally one backup per mailbox, but it’s also fine to group them by 5 or 10), specially if there is a large number of mailboxes and they contain thousands of emails. To make this easier, the plugin offers regular expression selection parameters (user_regex_include), as well as listing and query commands to analyze the backup target (List general info: Users, groups, sites).
Bacula Enterprise Microsoft 365 Plugin is an application that is under continous evolution. In consequence it may happen that new features require new permissions to function properly.
Bacula Systems will add the required new permissions to the central bacula-m365-plugin app registered in Azure AD. If you are already using the plugin, you will need to update the permissions of your own app (usually bacula-m365-plugin app itself, available under ‘enterprise applications’ option in your Azure Active Directory application).
To refresh and allow any new permission to your app. Please apply the following procedure:
Access the Azure Portal with a Tenant admin user
Locate your bacula-m365-plugin app under ‘Enterprise applications’ and click on it.
In the left menu, click on ‘Permissions’ and then click the big blue button showing ‘Grant admin consent for <yourtenantname>’:
Wait for a few minutes
Refresh the page
See the new permissions in the list
New releases needing any new permissions will properly reflect that information with the specific permissions list.
The great majority of services of the Bacula Enterprise M365 plugin are using the model of ‘Application permissions’. With this model, the plugin application can directly access the implied services once a tenant admin has aproved it.
On the contrary, some other services can require the model of ‘Delegated permissions’ where the plugin needs to impersonate a given user in order to be capable of accessing and restoring the data inside the service. Although we will support more in the future, currently, the only service requesting delegated permissions is:
Calendar: If any group is included as an entity to protect
Chat: In order to be able to backup all chat messages belonging to a certain user the user in question will need to delegate the permissions as described below.
Delegated permissions or application permissions are automatically applied when they are needed. However, delegated permissions need user interaction the first time they are invoked, or if the local information about access tokens has been lost for some reason.
The user interaction can be manually triggered with a special query command that may be run as follows:
.query plugin="m365: tenant=e421f055-748a-4a80-90da-bf48b66fa166 objectid=9017f87a-c164-488e-8f93-47139bcd6d47" client=127.0.0.1-fd parameter=login
m365: Starting Plugin Job
m365: Finished reading Plugin Params
m365: Connected to tenant: johndoe
m365: Backend connection to M365 stablished
m365: Starting QueryStart
m365: We don't have a valid token. Please, To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CQ7LETVFT to authenticate. You need to do it with an administrator user
We only need to follow the instructions, open a browser, enter the code and use
the correct user to login (which means a user with access to the resource we
want to backup).
For example, if we are working with the calendar of a group, we will need a user
with access to that calendar (in general, to protect calendars associated to
groups, it will be necessary to create a special backup user with access to all
of them and perform the login with, which will avoid the need of performing the
login process for every different group).
Here, if any MFA mechanism is enabled, it will be normally required and the user must complete the associated process. Finally Microsoft will ask to confirm if we want the plugin app to proceed with the operation, and we need to confirm:
Once the login is complete, the plugin will automatically end the operation and show something like:
info=We have a valid token got from account: johndoe@johndoe.onmicrosoft.com
info=It will be automatically refreshed when needed. But the current expiration date is: 2021-06-16T16:18:16Z
info=Backups using services with delegated permissions won't ask for intervention
It will also store the token information locally in a special file that it is stored in the working directory inside a folder with the tenant name. For example:
/opt/bacula/working/m365/tennantname/bacula-m365-plugin-standalone-token_cache.json
This file is also backed up in the backup operation itself, so it can be restored
manually, in case it was lost and in case you don’t want to run login operations again
In case the query method is not invoked prior to any job needing it, the job itself will show the message in the joblog and it will be blocked until a user performs the needed interaction described above. Once this happens, the job will automatically continue.
Note
Starting with version 14.0 of Bacula Enterprise you can also see logged in users directly using BWeb, where you can also manually add logged-in users.
In order to see more details, please go to section: BWeb Console<./m365-bweb-console>.
Bacula Enterprise Microsoft 365 Plugin can be managed from a dedicated user friendly Web console,
specifically designed to facilitate tasks as the authentication process for new tenants or the fileset
configuration process.
Note
This feature is available starting with version 14.0 of Bacula Enteprise.
The purpose of this screen is to show a list of the currently connected tenants that are already registered. The presence of each line in the table means that a tenant has been configured at some point.
In the upper right corner, it is possible to switch the FD at any time, so the information is based on the currently selected FD. After the first time a FD is selected, the information will be
remembered using cookies for the next time and for any other M365 Console windows.
The table shows tenant name, tenant id and the connection state. The connection state can indicate a possible incomplete registration operation (because of
some error), otherwise it will indicate
a successful connection and the type of connection: through the common app or
through the standalone app.
The big button placed at the bottom of the table allows us to configure the
connection to a new tenant, so it will open the ‘New tenant wizard’ that is
described in the section below.
Other actions that can be performed from this tenant list window are:
Retry operation: It allows to retry a possible incomplete registration, so
re-launch the proper internal query command (add-app or the objectid
command) using a selected tenant.
Delete registrator: It allows to delete the registrator app inside the
selected tenant. It is not possible to delete a registrator if registration
is incomplete . In this case, you must retry operation or delete tenant.
This is a possibility only offered for security concerns, as the registrator
app has some elevated permissions.
Delete tenant: It removes the tenant from the internal BWeb configuration.
When opening new tenant wizard, we can select the registration model to be used: Standalone or Common.
Details of each model are shown in the images and you can also read more information in this whitepaper under the Authorization section.
It is needed to provide the tenant id, which can be found following the whitepaper instructions placed under the same Authorization section.
It is recommended to use the Standalone model (first one to appear in the web interface) and be using the common model only for testing purposes.
Once we proceed with one of the models, an URI of Microsoft website will be suggested and you need to provide the needed permissions, then, Microsoft will redirect you to
a Bacula Systems page (same as shown in the Authorization section).
BWeb will automatically guide you until the end of the process where you will see your tenant correctly added into the tenant list screen.
In this screen a list of logged in users is shown.
Logged in users are particular users that have approved the Azure AD plugin app to make operations over their accounts that need delegated permissions.
Examples of operations needing this permissions model are:
Backup of calendar of groups
Todo Tasks module
Chats module
In addition to show the users list, it is also possible to force an user login. This function is offered to avoid having this kind of requests at backup time.
Users need to login only once, afterwards, the local cache will automatically be used to use saved tokens or to refresh them as needed.
For more information about delegated permissions, please take a look to <./m365-delegated-permissions>
The purpose of the FileSets page is to control all the filesets of Microsoft 365 Plugin, so we will see them listed in the main screen:
From there it is possible to create a new M365 Fileset in an user friendly way:
We need to select the tenant to work with as a first step:
The second step is to select the entities to protect: they can be users, groups or sites. All the services of M365 are ‘owned’ by any of those
entities:
It is possible to dinamically add or remove them, the wizard will store your final selection.
The third step allows us to select the services to protect in a similar way than the entities step, with multiselection allowed, and where we can
remove or add services at will:
After selecting the services, we will have one independent screen for each selected service where we will enter the particular configuration for
each of the services implied. In the screenshot below, we can see the Sharepoint services:
As a final step, it is possible to setup some advanced parameters and then finish the configuration:
From this final screen is possible to go directly to the job configuration wizard and finish a backup configuration or to return to the previous
fileset list page.
Please, note that this Wizard contains the most common parameters and it is intended to help to configure the most common use cases. In case you need to
setup more advanced parameters or to edit existing filesets, BWeb will allow you to do it using the regular Fileset editor where you will find all plugin
parameters.
The restart command has limitations with plugins, as it initiates the Job from
scratch rather than continuing it. Bacula determines whether a Job is restarted or
continued, but using the restart command will result in a new Job.
Listed in this section are some scenarios that are known to cause issues.
Pwsh: The specified program requires a newer version of windows
If a Sharepoint backup shows the message referenced in the title, it
means the powershell session of the job has conflicted with another open
Powershell session.
Solution: Be sure no other Powershell sessions are open and try again
If you test the plugin with the developer Datapack of M365 you may find
problems with hash checking of OneDrive autogenerated files. You could
see hash failures.
The reason is those auto-generated files have incorrect hashes.
Solution: Use your own uploaded files to test the plugin.
As stated in section Details on the Sharepoint backup/restore, templates of Sharepoint can present
different problems when they are applied over new generated sites. To
troubleshoot these issues, the following steps can be taken:
Restore the site template locally
Detect the exact part of the template that is causing problems and remove that section from the xml
Manually apply the site template using PnP.Powershell
Suppose we are experiencing this problem and we have already restored
our template in /tmp/mytemplate.xml and it is ready to be tested. Below
we show how to run manually PnP actions:
The first thing to do is run Powershell. Just run the following command in your terminal:
We will use the DeviceLogin method, as this method allows Multifactor Authentication. Just run the command shown with your main site:
Follow the instructions and go to a browser pointing to the
provided URL. You will be asked for credentials and you need to login with a tenant administrator user:
If MFA is enabled, complete the process:
Then put the code shown in the terminal in the first step and click Next (in this example code would be D56MS8244):
Approve the required permissions for PnP.Powershell:
Get back to the terminal and check that you are connected:
You can check the correct result of the command Get-PnPSite:
If you ever face OutOfMemory errors of the Java daemon (you will find them in the m365-debug.err file),
you are very likely dealing with sub-attachments containing big files or you are using a high level of concurrency
through internal concurrent_threads parameter and/or parallel jobs. To overcome this situation you can:
Disable the backup of multi-level attachments setting multi_attach=no
Reduce concurrent_threads parameter
Reduce the number of jobs running in parallel
If you cannot do that or you are not using multi-level attachments you should increase JVM memory.
To increase JVM memory that you will need to:
Create a this file: ‘/opt/bacula/etc/m365_backend.conf’.
Those values will define the MIN (M365_JVM_MIN) and MAX (M365_JVM_MAX) memory values assigned to the JVM Heap size. In
this example we are setting 2Gb for the minimum, and 8Gb for the maximum. In general, those values should be more than enough.
Please, be careful if you are running jobs in parallel, as very big values and several jobs at a time could quickly eat all
the memory of your host.
The ‘/opt/bacula/etc/m365_backend.conf’ won’t be modified through package upgrades, so your memory settings will be persistent.
When trying to register the bacula-m365-plugin app in a given tenant,
the following error can appear in the URL of a resulting page placed on baculasystems.com:
If we look carefully into the error contents we will notice the following message:
AADSTS650052: The app needs access to a service (https://*.dps.mil.com) that your organization cd77bbc0-999e-44c5-af9d-c11bea178407 has not subscribed to or enabled. Contact your IT Admin to review the configuration of your service subscriptions.
Even if it is not a very clear message, the error is pointing us to the fact that our tenant is missing some kind of access or subscription in order to utilize bacula-m365-plugin app. The contents of the error can vary depending on the specific situation of the target tenant.
However, in general, and particularly for the example shown, the problem comes from the fact that target tenant has no license to utilize Sharepoint Online service. Without this license/permission, the plugin cannot be registered.
When working with the OneDrive module and Group entities you may
experience a view of a list of apparently empty elements if
navigating through them using Sharepoint views:
This is not an error of this plugin. This is a problem with some kind of
views available in the Sharepoint Online service, where list items
corresponding to drive items are shown. As here we are dealing with
drive items, those list items have no title attached.
In the particular example shown above, a ‘list view’ of a document
library is selected. However if we select any other view as ‘Titles’, ‘All Documents’, etc:
We will be able to see folders and files with their corresponding names:
After a successful restore operation with the Teams module, it can happen that accessing the data using the web
interface of the M365 services will show the associated channels with empty contents. This is usually associated
to some bad functions of implied cookies with the web browsers.
If you experience this situation we recommend to open the information using a ‘private window/tab’ or removing previous cookies,
as the information is actually there. In case this strategy does not work, please use the Teams ‘desktop app’ for your OS, where
the behavior is much more solid.
Chat messages with not displayed contents in Teams
There are several kind of auto-generated messages that rely on other Microsoft Cloud internal data structure. Examples of this are:
Scheduled meetings through chat actions, not associated to calendar (they belong to ‘onlineMeetings’ module, which is not included today in Teams backup)
Messages showing that a Tab was added
Messages showing contents of external apps
This kind of messages will be restored exactly with the same form that they were got during backup and text contents will be visible. However, as the information linked is
attached in the external applications to the old team, that visual content may not be visible as it was in the original message.
Teams apps are external entities controlled by an undetermined number of external provides. When adding an app to a team, a process
belonging to that provider takes place. That process is invoked through the plugin restore process when restoring teams with apps. Sometimes that external process can return unexpected results.
The M365 Plugin will show those situations with a propper message in the joblog that is not treated as error, and the restore will succeed in most cases.
As this is not managed by Bacula Systems or MS Graph API, the approach is just to show the result as it is, when it is not successful. However, please note that the app itself,
in most cases, will be also successfully restore even if that situation happens.
Error with Teams App after restoring a Planner plan
Teams desktop applications use cache data that can interfere sometimes with the real data. Restoring a planner plan can be an example. When you try to open a restored planner tab you may see an
error message. However, please just click on the reload button and you should see your data without issue.
In some circumstances, java processes belonging to the plugin can be left behind.
They may be using noticeable CPU and memory resources. It is safe to remove them
by sending an appropriate signal.
Such orphaned processes should not occur with Bacula Jobs finishing without
problems.
A way to identify such processes would be to look for java processes running the
plugin .jar file either when no such job is actually active, or when no such job
of a similar age exists. For example
If no m365 job is running since July 04, it would be safe to kill -9 370270 370403.
In newer versions of the plugin, the actual Bacula Job running one of the java
processes will be indicated in a command line option of the java process, and the
Job report will indicate the process id of the java process started.
