Email / Mailboxes
Bacula Enterprise Microsoft 365 Plugin can protect M365 Mailboxes associated to users. It is possible to utilize advanced selection methods to decide exactly what is backed up (folders included/excluded, users included/excluded), as well as control precisely which messages or attachments to restore and where (original user’s account or another user’s account). The information protected with this service is:
Folders (Inbox, Deleted Items, Sent…)
Messages (Metadata and contents)
Attachments
File Attachments
Item Attachments
Reference Attachments
Folder Rules
Mailbox Settings
Outlook Categories
Public folders are not supported and folder sharing permissions are not protected (see Email Limitations). On the other hand, folders are backed up always from their original accounts:
This means if a folder ‘folderA’ is shared from user ‘U1’ to user ‘U2’, if you only backup the entire mailbox of U2, contents of folderA will not be protected. In order to protect folderA you will need to add some additional job protecting U1 and folderA into the backup set or simply add the entire mailbox of U1.
User mailboxes and shared mailboxes are supported, it is only required to setup the correct address in user* parameters (shared mailboxes are treated as users). For groups, please note that they have no real mailbox, therefore it is not possible to backup the emails belonging to a given group. However, protecting emails of any of the users that is member of those groups will result in protecting all the received emails of that group.
Mailbox backup includes the following features:
Incremental backup with Delta function:
Delta function is applied for each folder individually
MIME object backup:
Based on the fileset parameter email_mime it is possible to get mime messages as well as the M365 objects. These kind of objects can be useful to have if there is any plan of using the emails outside the M365 service.
At restore time, if the restore operation is done via M365 services and not to a local FileSystem, mime_objects are automatically ignored.
Multilevel-attachments:
M365 supports having messages or event objects as the attachment of a given message object. These attachments can have additional attachments. This situation is what M365 Plugin considers multilevel-attachments. Some particular objects such as Files, when present in this multilevel way, are got embedded in the metadata of the message attached. This implies that the plugin must get them in a single HTTP call and load them entirely in memory. If there is not enough memory in the system, the variable multilevel_attach can be disabled and those objects will be ignored.
Multilevel-attachments are backed up at the same level. This means that a message with this structure:
Base Message has 1 Message-Attachment, Message-Attachment has 1 more internal Message-Attachment 2, and Message-Attachment 2 has a File attachment
When restored, the result will be all Attachments being restored to the same level, not one inside the another as they were originally. For example:
Base Message
Message-Attachment 1
Message-Attachment 2
File Attachment
Messages will be formatted in the catalog in order to not include sensitive information and will be included in a path like this:
/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 09.58.52..J-123.10.msgWhere the message name is composed as: messageDate..J-JobId.JobIndex.msg
messageDate corresponds with the receivedDate
JobIndex is an internal index relative to a backlup job execution
The extension .msg corresponds to a file containing an email
Mime messages will have the extra word ‘mime’ in their extension. For example:
/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 15.54.06..J-2021-04-19_15.54.45_03.10.mime.msg
/@m365/tenant.name/users/user@tenant.com/email/mailbox\_settings.mail.setMailbox Settings
/@m365/tenant.name/users/user@tenant.com/email/rules/nameRule.mail.ruleMailbox Rules
/@m365/tenantname/users/usermail/outlook\_category_name.olk.cat/
Mailbox Categories
Attachments will be stored together with message objects:
They include their original name (file name)
They have an extension about their type (.file.att, .ref.att or .item.att)
The first part of the attachment name is the name of the parent message
Here are a few attachment examples:
/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 15.54.06..J-123.10.Aliquid.gen.1.file.att/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 15.54.06..J-123.10.Dolor.gen.4.ref.att/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 15.54.06..J-123.10.Sapien Nostrum Aperiri Unum - t.3.item.att/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 15.54.06..J-123.10.Veri.gen.2.file.att/@m365/tenant.name/users/user@tenant.com/email/foldername/2021-04-19 15.54.06..J-123.10.msg → Parent messageBackup parameters
The list below shows the specific backup parameters that can be set up in order to control the behavior of the email module.
In order to select the email module, the common service parameter must be equals or be containing the value email.
Entities that can include mailboxes are: users.
Option |
Required |
Default |
Values |
Example |
Description |
|---|---|---|---|---|---|
email_files |
No |
Strings representing existing mailfolders for the given users or groups separated by ‘,’ |
Inbox, Sent |
Backup only specified mailfolders belonging to the selected users |
|
email_files_exclude |
No |
Strings representing existing mailfoders for the given users or groups separated by ‘,’ |
Archive, Personal |
Exclude selected mailfolders belonging to the selected users |
|
email_files_regex_include |
No |
Valid regex |
.*Company |
Backup matching mailfolders. Please, only provide list parameters (files + files_exclude) or regex ones. But do not try to combine them. |
|
email_files_regex_exclude |
No |
Valid regex |
.*Plan |
Exclude matching mailfolders from the selection. Please, only provide list parameters (files + files_exclude) or regex ones. But do not try to combine them. If this is the only parameter found for selection, all elements will be included and this list will be excluded. |
|
email_multilevel_attach |
No |
Yes |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
No |
Include multilevel attachments in backup. A multilevel attachment is, for example, a message attached to a message containing more elements attached inside. Under certain circumstances, multilevel attachments could imply higher memory needs, as some of them are embedded in the metadata of the objects to backup, so they need to be caught in a single call and be loaded entirely in memory. This parameter is intended to control the behavior against that situation. |
email_exclude_attachments |
No |
No |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
Yes |
Exclude any attachment from backup |
email_mime |
No |
No |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
Yes |
Backup raw MIME file of every email, in addition to the Message object itself |
email_settings |
No |
No |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
Yes |
Backup mailbox settings of included users |
email_rules |
No |
No |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
Yes |
Backup mailbox folder rules of included users |
email_categories |
No |
No |
0, no, No, false, FALSE, false, off ; 1, yes, Yes, TRUE, true, on |
Yes |
Backup outlook categories of included users |
email_messages_exclude_expr |
No |
String representing a valid Boolean Javascript expression regarding email message fields |
emailSubject.includes(‘private’) && !emailIsRead |
Exclude from backup all messages that match the provided expression |
|
email_messages_exclude_index_expr |
No |
String representing a valid Boolean Javascript expression regarding email message fields |
/.*private.com/.test(emailFrom) |
Exclude only from indexing (catalog email tables) messages matching the provided expression |
|
email_fields_exclude |
No |
String representing a list of email message fields |
emailTo, emailBodyPreview |
Do not store into the backup the provided list of message fields |
|
email_fields_exclude_index |
No |
String representing a list of email message fields |
emailFrom, emailSubject |
Do not store into the index (catalog email tables) the provided list of message fields |
|
email_filter_received_from |
No |
Date in format: yyyy-MM-dd HH:mm:ss |
2020-06-01 00:00:00 |
Request messages for backup starting only from the provided date |
Note
In previous versions, instead of email_files* parameters, it was possible to use files_* parameters. They are deprecated parameters now as we recommend to use module specific parameters in order to have better control and more possibilities inside a single fileset. However, jobs using files_* parameters will still work. The following fields: email_messages_exclude_expr, email_messages_exclude_index_expr, email_fields_exclude, email_fields_exclude_index, email_filter_received_from are available from Bacula Enterprise version 14.0.
Restore
The list below shows the subset of restore parameters that can be used to control the behavior of email module restore operations:
destination_user, destination_path, send_report, allow_duplicates, debug, foreign_container_generation
Use cases
The following restore scenarios are supported:
Restore directories, emails, or attachments to original user or to a different user
Restore parameters implied: destination_user
Restore directories, emails, or attachments to original path or to a different path
Restore parameters implied: destination_path
Restore files, directories, or file versions to local file system (general restore where parameter must be set to a path)
It is possible to control whether or not duplicate elements are allowed (based on file id):
Restore parameters implied: allow_duplicates
Particularities:
If no destination_user is set, every message will be restored into its original mailbox
If no destination_path is set, every message will be restored into its original path
If the selection contains messages from several users:
Original user messages will be restored in their original location
For other users, a special folder will be created with the email address of each of them, containing the full path and messages of the restored objects, unless the parameter foreign_container_generation is disabled
Example:
Restore of emails from 2 different users over a third mailbox without destination_path result in auto-generated Restore_date folder containing those 2 foreign users with the restored folder inside of them
Restored elements will be duplicated by default, unless allow_duplicates variable is disabled
Even when disabling that variable, messages will be checked by id. So if there is an element with the same information but different ID, it will not be considered to be a duplicate
For more details about the behavior of each parameter, please check the general section of restore parameters.
Messages exclude expressions
Bacula Systems is aware about one of many privacy concerns that may arise when tools like this M365 Plugin enables the possibility to backup and restore data coming from different users, so the backup administrator can restore potentially private data at his will. Moreover, emails are usually one of the most critical items in terms of privacy.
One of many strategies this plugin offers in order to deal with that problem is the possibility to exclude messages. This is a very powerfull feature where it is possible to use quite flexible expressions that allow to select a subset of messages and simply exclude them from the backup:
email_messages_exclude_expr new fileset parameter
Or only from the index (from the catalog)
email_messages_exclude_index_expr new fileset parameter
Not only messages can be excluded but also select only a subset of email fields to be included in the protected information. It is possible to exclude fields from the backup:
email_fields_exclude new fileset parameter
Or only from the index (from the catalog):
email_fields_exclude_index new fileset parameter
Please, be aware that if the fields are excluded from backup, the restore operation to M365 can fail easily, since for instance emails cannot exist in M365 without From field, Subject field, etc.
All four discussed expressions are based on an internal structure of fields to work with. Below you can see the entire list of fields that you can use:
emailTags
emailSubject
emailFolderName
emailFrom
emailTo
emailCc
emailBodyPreview
emailImportance
emailTime
emailIsRead
emailIsDraft
Please note that it is very important to write the fields exactly as written above.
These fields can be used in a comma separated list in the ‘email_fields_exclude’ parameter and also ‘email_fields_exclude_index’ parameter.
Then, for ‘email_messages_exclude_expr’ and ‘email_messages_exclude_index_expr’ use them in a valid boolean expression in Javascript language syntax. Some examples are provided below:
emailSubject.includes('private')
!emailIsRead && (emailIsDraft || emailFolderName == 'Private')
!emailTime < Date.parse('2012-11-01')
/.*private.com/.test(emailFrom)
Note
This feature is available since Bacula Enterprise version 14.0
Expression tester
This expression mechanism can sometimes be uncertain for end users, where they can have doubts about the correct behavior of their prepared expressions. In order to help with that, M365 Plugin presents a query method that allows to test those expressions against a static pre-loaded set of data.
There are two commands available:
Show command: It will show the static data in json format, so it is possible to see the contents to adapt the expressions to test
Test command: It will apply the expression parameters to the pre-loaded static data
The test command has the following format:
.query client=<your-fd-client> plugin="m365: tenant=<your-tenant-id>" parameter=email-expr-show
The show command has the following fomat
.query client=<your-fd-client> plugin="m365: tenant=<your-tenant-id> email_messages_exclude_expr = \"<your-js-expression>\"" parameter=json|email-expr-test
// Or
.query client=<your-fd-client> plugin="m365: tenant=<your-tenant-id> email_messages_exclude_index_expr = \"<your-js-expression>\"" parameter=json|email-expr-test
Please, not that you need to provide a valid tenantId, even if it’s not really used to process any data.
The test command produces some JSON output with objects with the exact format that is received from Microsoft and, consequently the same format that is stored in backup. Please not the ‘total’ value at the end, where the value of 12 total pre-loaded messages is shown
.query client=<your-fd-client> plugin="m365: tenant=<your-tenant-id>" parameter=json|email-expr-show
....
"email-12": {
"body": {
"content": "These are the contents in text format of the 12 email of test data. It has the following categories:orange, black, white, purpleYou can try to filter this body using any JS method like /.*12.*/.test(emailBody) or emailBody.includes(12)",
"contentType": "TEXT"
},
"ccRecipients": [
{
"emailAddress": {
"address": "danny@other.com"
}
},
{
"emailAddress": {
"address": "lucas@other.com"
}
},
{
"emailAddress": {
"address": "terese@other.com"
}
}
],
"from": {
"emailAddress": {
"address": "elon@other.com"
}
},
"hasAttachments": false,
"isDraft": false,
"isRead": false,
"replyTo": [
{
"emailAddress": {
"address": "elon@other.com"
}
}
],
"sentDateTime": {
"dateTime": {
"date": {
"year": 2021,
"month": 12,
"day": 5
},
"time": {
"hour": 11,
"minute": 30,
"second": 0,
"nano": 0
}
},
"offset": {
"totalSeconds": 0
}
},
"subject": "This is private subject 12",
"toRecipients": [
{
"emailAddress": {
"address": "laura@other.com"
}
},
{
"emailAddress": {
"address": "jack@other.com"
}
},
{
"emailAddress": {
"address": "john@other.com"
}
}
],
"categories": [
"orange",
"black",
"white",
"purple"
]
}
},
{
"total": "12"
}
The test command, on its side will produce two different outputs. The first part presents the same format than the show format, and those are the messages that would be included in the backup. The second part presents a different format, so an output like:
.query client=<your-fd-client> plugin="m365: tenant=<your-tenant-id>" parameter=json|email-expr-show
....
{
"meta-email-12": {
"EmailId": "",
"EmailOwner": "test@test.com",
"EmailTenant": "johndoe.onmicrosoft.com",
"EmailTags": "orange,black,white,purple",
"EmailSubject": "This is private subject 12",
"EmailFolderName": "/",
"EmailFrom": "elon@other.com",
"EmailTo": "laura@other.com,jack@other.com,john@other.com",
"EmailCc": "danny@other.com,lucas@other.com,terese@other.com",
"EmailInternetMessageId": "",
"EmailBodyPreview": "",
"EmailImportance": "",
"EmailConversationId": "",
"EmailSize": 235,
"EmailIsRead": 0,
"EmailIsDraft": 0,
"EmailHasAttachment": 0,
"Type": "EMAIL",
"Version": 1,
"Plugin": "m365"
}
},
{
"total-backup": "12"
},
{
"total-index": "12"
}
That part represents the information that would be indexed in the backup (included into the catalog). You can also see the total entries at the end, that are very useful to quickly compare with the original 12 value and so, knowing if our expression is filtering the expected data or not. Below we provide an example where some filtering is applied to the backup, but also to the index:
.query client=127.0.0.1-fd plugin="m365: tenant=xxxx-xx-xxxx-xxx-xxxx email_messages_exclude_expr=\"emailFrom == 'elon@other.com'\" email_messages_exclude_index_expr=\"emailSubject.includes('private')\"" parameter=json|email-expr-test
...
{
"meta-email-4": {
"EmailId": "",
"EmailOwner": "test@test.com",
"EmailTenant": "johndoe.onmicrosoft.com",
"EmailTags": "orange,black,white,purple",
"EmailSubject": "This is orange subject 8",
"EmailFolderName": "/",
"EmailFrom": "bob@company.com",
"EmailTo": "laura@company.com,jack@company.com,john@company.com",
"EmailCc": "danny@company.com,lucas@company.com,terese@company.com",
"EmailInternetMessageId": "",
"EmailBodyPreview": "",
"EmailImportance": "",
"EmailConversationId": "",
"EmailSize": 232,
"EmailIsRead": 0,
"EmailIsDraft": 0,
"EmailHasAttachment": 0,
"Type": "EMAIL",
"Version": 1,
"Plugin": "m365"
}
},
{
"total-backup": "6"
},
{
"total-index": "4"
}
- In case your expression is not valid, the plugin will also inform about that with the following message:
error=Error listing elements. Cause: Predicate test error!! Review your query ….
Fileset examples
Backup Full MailBox of some users, but excluding some folders:
FileSet {
Name = fs-m365-drive-adjon-users-notemp
Include {
Options { signature = MD5 }
Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 user=\"adelev@baculaenterprise.onmicrosoft.com,jonis@baculaenterprise.onmicrosoft.com\"
email_files_exclude=\"*.temporary\""
}
}
Backup all MailBoxes:
FileSet {
Name = fs-m365-email-all
Include {
Options { signature = MD5 }
Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5"
}
}
Backup only the Inbox folder of some users:
FileSet {
Name = fs-m365-email-2user-inbox
Include {
Options { signature = MD5 }
Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 user="peter@mycompany.com,john@mycompany.com" email_files=inbox"
}
}
Backup some users and include MIME messages:
FileSet {
Name = fs-m365-email-2user-mime
Include {
Options { signature = MD5 }
Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 user="peter@mycompany.com,miriam@mycompany.com" email_mime=yes"
}
}
Disable backup of multi-level attachments:
FileSet {
Name = fs-m365-email-multilevel-attach
Include {
Options { signature = MD5 }
Plugin = "m365: service=email tenant=57uia43-d107-17a2-a2g2-aa53c10tdahc objectid=56ddf1h9-eb5d-
42nf-bac7-7b019fd284g5 user="peter@mycompany.com,miriam@mycompany.com" multilevel_attach=no"
}
}
Well known folders
Microsoft 365 can present the folders information in local languages to the user.
In general, there is no ‘multilanguage’ support, in the sense that folders must be included with their original name. For example, if you create a folder named ‘books’, you cannot expect it to be backed up if you use something like ‘livres’ or ‘libros’ from other languages. You need to use the real name that was used to create such folder.
There is one very important special case though, which is ‘well known folders’. Well known folders are folders like ‘inbox’, ‘outbox’, ‘archive’… A full list can be found here: https://docs.microsoft.com/en-us/graph/api/resources/mailfolder?view=graph-rest-1.0
That kind of folders can be ‘found’ by the plugin using their ‘well known name’, instead their internal id, as it’s the general case. Therefore, for them it is possible to get the folder using their English well known word even if the user sees the folder with a translated word.
For example, to backup inbox it is needed to use ‘inbox’ even if for some user it is ‘Posteingang’ or ‘boîte de réception’. Microsoft 365 Plugin will recognize this special words and will query the information through them.
To summarize:
Well known folders -> Use English word
Other user folders -> Use original name