Backup in Exchange EWS Plugin
Backup jobs in Exchange EWS plugin behave as any other backup job in Bacula Enterprise once the fileset has been created, as described in the configuration section. Below, some special features of the plugin that happen at backup time are described, as well as the file structure that a backup creates.
Backup File Structure
Items are formatted in the backup catalog in order to not include sensitive information. They are included in a path in the following format:
/@e2ws/domain.name/users/user@domain.name/foldername/shortId_itemDate.itemExtension
Depending on the type of the item, itemExtension wil be:
Message: .msg
Appointment: .pp
Task: .task
Contact: .con
Contact group: .con.gr
Mime files will have the extra word ‘mime’ in their extension. For example:
/@e2ws/testlab.local/users/ex-admin@testlab.local/regress_20230417125041/AAPfLdTrAAA=_r20230417-125325.mime.msg
Attachments will be stored together with item objects:
They include their original name (file name)
They have a special extension “.att”
They include the attachment type (file or item)
The first part of the attachment name is the name of the parent message.
Here is an example of an attachment :
/@e2ws/testlab.local/users/ex-admin@testlab.local/inbox/AAPfLdTuAAA=_r20230417-125329.msg.Prompta.gen.file.att/@e2ws/testlab.local/users/ex-admin@testlab.local/inbox/AAPfLdTuAAA=_r20230417-125329.msg → Parent messageMIME Objects Backup
Based on the fileset parameter mime, it is possible to get mime formatted items as well as regular objects (which are in json format). This kind of objects can be useful to get if there is any plan of using the information outside the Exchange service (e.g. for migration purposes).
Note
Activating this option has a performance penalty, and the backup time will be significantly higher, as for every email the information will be requested twice (one for the regular format, so .json message file plus attachments; another one for mime format, so to get a unified .mime.msg file containing the message and attachments).
At restore time, if the restore operation is done via EWS services and not to any local filesystem, selected mime objects are automatically ignored. It means if only those .mime.msg files were manually selected during a restore session over Exchange, the restore won’t restore any file. While doing the same over a Local filesystem, destination will end up with those .mime.msg files restored. Usually, the selection would include both kind of files (a folder, a whole backup). In that situation, the restore will be simply successful, while those .mime.msg won’t be used.
Email Privacy Filters
Bacula Systems is aware of one of many privacy concerns that may arise when tools like the Exchange EWS Plugin enables the possibility to backup and restore data coming from different users, so the backup administrator can restore potentially private data at his will. Moreover, emails are usually one of the most critical items in terms of privacy.
One of many strategies the plugin offers in order to deal with that problem is the possibility to exclude messages. This is a very powerful feature where it is possible to use quite flexible expressions that allow to select a subset of messages and simply exclude them from the backup:
email_exclude_exprfileset parameter will exclude completely the selected messagesemail_exclude_index_exprfileset parameter will exclude the selected messages from the index (MetaEmail catalog table).
Not only messages can be excluded, but also select only a subset of email fields to be included in
the indexed information using email_fields_exclude_index fileset parameter.
All three discussed expressions are based on an internal structure of fields to work with. Below, you can see the entire list of fields that you can use:
emailTags
emailSubject
emailFolderName
emailFrom
emailTo
emailCc
emailBodyPreview
emailImportance
emailTime
emailIsRead
emailIsDraft
Note
It is very important to write the fields exactly as written above.
These fields can be used in a comma separated list in the email_fields_exclude_index parameter.
Then, for email_exclude_index_expr and email_exclude_expr, use them in a valid boolean
expression in Javascript language syntax. Some examples are provided below:
emailSubject.includes('private')
!emailIsRead && (emailIsDraft || emailFolderName == 'Private')
!emailTime < Date.parse('2012-11-01')
/.*private.com/.test(emailFrom)
Note
This feature is available since Bacula Enterprise version 14.0
Expression Tester
This expression mechanism can sometimes be uncertain for end users as they can have doubts about the correct behavior of their prepared expressions. In order to help with that, Exchange EWS Plugin presents a query method that allows to test those expressions against a static preloaded set of data.
There are two commands available:
Show command
Test command
The show command will show the static data in json format, so it is possible to see the contents to adapt the expressions to test command - it will apply the expression parameters to the preloaded static data.
The test command has the following format:
.query client=<your-fd-client> plugin="e2ws: endpoint=<ews-endpoint> admin_user=<username> admin_password=<password>" parameter=email-expr-show
The show command has the following format:
.query client=<your-fd-client> plugin="e2ws: endpoint=<ews-endpoint> admin_user=<username> admin_password=<password> email_exclude_expr = \"<your-js-expression>\"" parameter=json|email-expr-test
// Or
.query client=<your-fd-client> plugin="e2ws: endpoint=<ews-endpoint> admin_user=<username> admin_password=<password> email_exclude_index_expr = \"<your-js-expression>\"" parameter=json|email-expr-test
Note
You need to provide a valid endpoint and user credentials, even if it’s not really used to process any data.
The test command produces JSON output with objects with the exact format that is received from Microsoft and, consequently, the same format that is stored in backup. Note that ‘total’ value at the end, where the value of 12 total preloaded messages is shown.
.query client=<your-fd-client> plugin="e2ws: endpoint=<ews-endpoint> admin_user=<username> admin_password=<password>" parameter=json|email-expr-show
....
"email-12": {
"body": {
"content": "These are the contents in text format of the 12 email of test data. It has the following categories:orange, black, white, purple. You can try to filter this body using any JS method like /.*12.*/.test(emailBody) or emailBody.includes(12)",
"contentType": "TEXT"
},
"ccRecipients": [
{
"emailAddress": {
"address": "danny@other.com"
}
},
{
"emailAddress": {
"address": "lucas@other.com"
}
},
{
"emailAddress": {
"address": "terese@other.com"
}
}
],
"from": {
"emailAddress": {
"address": "elon@other.com"
}
},
"hasAttachments": false,
"isDraft": false,
"isRead": false,
"replyTo": [
{
"emailAddress": {
"address": "elon@other.com"
}
}
],
"sentDateTime": {
"dateTime": {
"date": {
"year": 2021,
"month": 12,
"day": 5
},
"time": {
"hour": 11,
"minute": 30,
"second": 0,
"nano": 0
}
},
"offset": {
"totalSeconds": 0
}
},
"subject": "This is private subject 12",
"toRecipients": [
{
"emailAddress": {
"address": "laura@other.com"
}
},
{
"emailAddress": {
"address": "jack@other.com"
}
},
{
"emailAddress": {
"address": "john@other.com"
}
}
],
"categories": [
"orange",
"black",
"white",
"purple"
]
}
},
{
"total": "12"
}
The test command on its side will produce two different outputs. The first part presents the same format as the show format, and those are the messages that would be included in the backup. The second part presents a different format, so an output like:
.query client=<your-fd-client> plugin="e2ws: endpoint=<ews-endpoint> admin_user=<username> admin_password=<password>" parameter=json|email-expr-show
....
{
"meta-email-12": {
"EmailId": "",
"EmailOwner": "test@test.com",
"EmailTenant": "ews.test",
"EmailTags": "orange,black,white,purple",
"EmailSubject": "This is private subject 12",
"EmailFolderName": "/",
"EmailFrom": "elon@other.com",
"EmailTo": "laura@other.com,jack@other.com,john@other.com",
"EmailCc": "danny@other.com,lucas@other.com,terese@other.com",
"EmailInternetMessageId": "",
"EmailBodyPreview": "",
"EmailImportance": "",
"EmailConversationId": "",
"EmailSize": 235,
"EmailIsRead": 0,
"EmailIsDraft": 0,
"EmailHasAttachment": 0,
"Type": "EMAIL",
"Version": 1,
"Plugin": "e2ws"
}
},
{
"total-backup": "12"
},
{
"total-index": "12"
}
That part represents the information that would be indexed in the backup (included into the Catalog). You can also see the total entries at the end, which are very useful to quickly compare with the original 12 value and so, knowing if our expression is filtering the expected data or not. Below, we provide an example where some filtering to the backup is applied, but also it is applied to the index:
.query client=127.0.0.1-fd plugin="e2ws: endpoint=<ews-endpoint> admin_user=<username> admin_password=<password> email_exclude_expr=\"emailFrom == 'elon@other.com'\" email_exclude_index_expr=\"emailSubject.includes('private')\"" parameter=json|email-expr-test
...
{
"meta-email-4": {
"EmailId": "",
"EmailOwner": "test@test.com",
"EmailTenant": "ews.test",
"EmailTags": "orange,black,white,purple",
"EmailSubject": "This is orange subject 8",
"EmailFolderName": "/",
"EmailFrom": "bob@company.com",
"EmailTo": "laura@company.com,jack@company.com,john@company.com",
"EmailCc": "danny@company.com,lucas@company.com,terese@company.com",
"EmailInternetMessageId": "",
"EmailBodyPreview": "",
"EmailImportance": "",
"EmailConversationId": "",
"EmailSize": 232,
"EmailIsRead": 0,
"EmailIsDraft": 0,
"EmailHasAttachment": 0,
"Type": "EMAIL",
"Version": 1,
"Plugin": "e2ws"
}
},
{
"total-backup": "6"
},
{
"total-index": "4"
}
In case your expression is not valid, the plugin will also inform about that with the following message:
``error=Error listing elements. Cause: Predicate test error!! Review your query .....
Delta Backup
The Microsoft EWS API provides a Delta function to track changes of some objects. Bacula Enterprise Exchange EWS Plugin uses this function in order to speed up Incremental/Differential processes.
Delta function has the following important characteristics:
Delta tokens can expire at some point, or even become invalid due to internal Microsoft issues. If that happens, the plugin will try to start a new Delta cycle.
There are two delta types of tokens implied: one for the folder structure, another for every folder that has changes inside.
Any situation where the Delta function cannot be used will trigger a regular Full/Inc/Diff, where every element is listed and selected or discarded according to the item dates.
The Delta backup cycle is described below:
Full backup: All entity elements are backed up. A token (token_1) is generated and the token is stored locally by the FD.
Incremental 1 backup: token_1 is used to retrieve changes since token_1’s generation, so every change is backed up. A new token is generated and stored locally by the FD.
Incremental 2 backup: token_2 is used to retrieve changes since token_2’s generation, so every change is backed up. A new token is generated and stored locally by the FD.
And so on…
Tokens are stored in a file placed in a path defined by the path
parameter of the plugin. The name is: jobname.deltaLink.
The file stores tokens required for every execution, and it is renewed (emptied) during every Full backup execution.
This file is also backed up in the backup itself, so it can be restored manually, before an Incremental/Differential execution in case it was lost and in case you don’t want to run a Full backup again.
Here, we can see an example of the contents of the file, with one execution and one user entity involved. The structure is tree-based, so it is easy to understand what would be generated in case of backing up other folders or users:
{
"deltaServices" : {
"e2ws" : {
"entities" : {
"ex-admin@testlab.local" : {
"containers" : {
"AAMkADkwMWYyMWQwLWZjZmMtNDU3NS1iMmM3LWVmMTRkNTQ0MjVjYQAuAAAAAAD2ghxkrOnXTJN4mEgvv12nAQAtUQmYk+IgRb3KJtCcyy+/AAPfLblCAAA=" : {
"deltaEntries" : [
{
"date" : "Apr 14, 2023, 10:14:42 AM",
"delta" : "H4sIAAAAAAAEAGNgYGcAAotqE0tHE2NTA0ddZ3NHC10TR2djXSdnJ2ddNyMnC2cnczdLU1OD2vBgveDKvOTgksSSVOfEvMSiSgYr0nW65eekpBZ5pjBYkq43LLWoODM/j8GaaK3+QMuKS4JSk1Mzy1JTQjJzU0nwrU9icYlnXnFJYl5yqncqKb71zS9K9SxJzS32zwtOLSpLLSLByXDfhgNxUW5iUTYklrgYGISA0tDwAxkOUskgCJQyAGI9kJpvTTIpa15e95lcMcNDf3/sckYG3UDOGZMfKbjuPaV2Yc5p/f0MzPd1dzoxMDAy8DEwg7RwM9jViBcesLntwSAEFOUFYqB1rIwMDL6OAZ6+jn4gRQxupm5hYOVooB2I5ZD4S9H4MHAYiM2Q+OjOYmQIAgkzMt/Xj0sKYLgXwr5brZ7Dd7L+v6q5Sj/2soE1YfomCCoO0paMRRbo2b1aYNIYr94knDYyMTA0AAAccVKZJwMAAA==",
"job" : "pluginTest.2023-04-14_10.14.42_03"
}
],
"description" : "regress_20230414101303",
"id" : "AAMkADkwMWYyMWQwLWZjZmMtNDU3NS1iMmM3LWVmMTRkNTQ0MjVjYQAuAAAAAAD2ghxkrOnXTJN4mEgvv12nAQAtUQmYk+IgRb3KJtCcyy+/AAPfLblCAAA="
},
"MailboxFolders" : {
"deltaEntries" : [
{
"date" : "Apr 14, 2023, 10:14:42 AM",
"delta" : "H4sIAAAAAAAEAK2be1ST5x3HX6HeQJCgwqzIXaVqJEhCiEBpLsSkBpRytG6oECBcLAaXAkqFeQFXT6TUiU5AgWCptlQp6Oqsh4qXOVfdBitjHMTWTaerVK1YN3W2ZYk9cvbH8rjne37vOTknB/h83t/veZ/3uSMIowX7Fb1BqlBKI2USpVgtV0aLpUp1pFilVqnF2vmqaLVKrlXIZJKyV1PmpZSYM1MKjYUmtdFstJQIC/hJbUF+lsmizxIU/Owyk+X1vAKzIFHn55nMhU9+lqaWSzQRURKpWKJIkIul0YpssSoiWiXWqCIj5SqFMlIr0QpR6iJLislSbLIkGs152abXC/8/7kn5uAmCjz2MHyLX5ZksRktmbokjKkFk/7XE/pnn+Lt/bfHLOnSzz1C9vkYX3rny4ChBnDy+pvpqYMInF2Z277sY3un4q1Hj7B/B0/6xX/bvQqXjC5fFtU/fM0UQXPgpH4jyhSg/QXDlp6ZDlD9EBUBUIEQFQVQwRIVAVChEzYComRA1C6LCIOoFiJoNUXO43y9HO+IBUQboXZ4I3cuTuzQcFH+NclD8NUoIbFP6QM/rR1CE/DXKEeEUiJoMRYi1AJMgij9CBzUVop6HqGlQGYY4enduyo2bcu0aMgVDFBbhRIji780dVCx0rwkQ5SUIY/gpoOTtNWoVRKVDlBGisiEqByp5+9h5ND/lDtX5IIgKhagZEDUTomZBVBhEvQBRsyFqDkTN5ad6XWzfQ9QwQjUJwPulHVoEUQaISoRaAC+IEkGUN0QBdcNOiSFqHkSFQ5QEoiIgaj5ERUKUFKJk0EgvCqLk0EwqGqIUgvAcP7UAol6ESj4eol6CKCVEaSAqAaK0ELUQonQQpYcopCfS9yA9kb5nCUQlQ9QKiNoIURVQC7AVupcVutd26F6/gKidEFUNUU0QtR+iDkMl38p/ryvij1VQhBkQlQlRWRCVC1F5ELUaoswQVQBRayGqEKKKIKoYouohqgGiGqGVivEQ5Q2Nvjz5KfvMF5in2GejoyDKBSoN/tGXg+IfETko/jGbg3oyZuPfjBWebun62L/9j0e6KFpUWj18nr8mTGFKfaR91wr4pT5M6TRRW/sW4vQN6UP/HGjij9SPKTW+W77nDrU0Q9T1HWeZPjP9VVnLVxdkUEtNB1ed30YtzTY/fNTMX6ZBTGnOnbo/fUYtzT3R/fa31OnnfSwXfsIfaShTurrs7cV66khfC72oXE8szbCpA3qXEEvXhLSM+VUctTT04dW014ilFkum3Pcjaun6B2GH91NKHdde/0e33uOXerCkXjebZz88RCm1pz+r8drPjzbQlunLj2/8UXf/U36pJyt991c81j43SC1dGXNkFNCbMqSBbbrTva31LZRS+4OKdb+aqwDeKHb61pzSTi11+rve9MrfTS2t6GxP+j11+tqBQB3nsOeZDUrshIy27uv8fdQkptTjXnH2PupIY86OWarkj3QqS6qJy72pmM4vfZ4pfbHneOo54kbaPUbRn+pJ3fLX1GmswINyc55+1z1b9paqtG7+GWcwS1pTNH9WFumg13HVx1V/STqWsl9uf28P23ycWDp5IOXS1+uo36iQ0bb3DNRl2hDTOW4mdeX3LdnWkkabvuHH1/9Q7tiAJJUaH4tNtcP8DUo6S5oh3Drp0UgcqXlX3e7fcg7QRg70OJXWJsR4Ax0fu0o1+ja8JaOW1lpT2/xJy7Tr3o6yfxQZf0cstf307spbocTS5rr3C9bc5W+kZzCljeaCSs41lGdG2rJp++V1JmJp6/tuwYGf86cfxpS2Hq8+Qjvh7bp3rPxbzUvU9fTEId+BA5yD3pFDVE6l7eM8vJeRRtrreqMs/Xq+B7F08EBs/tI3udepfzih5VT6wVd7z5A2fa59OlncwqkXgFk0UxqXlDMhj3Oxa+T4l1NpyqDKnTjSRXOaRB/UWYml4Rdqd1sX83d8Iqa069wS8zjiSEs11tZdHxJLN61ZXnX5KH/6YqbU0lJxu5840h1bj+0fWEos3dXfIJsPrPYwpbWRYS6/OUMsrU/LmnvsKrG0adtR246z/E8/kil9643Hnb7EkXZ4nQqqCqGWin79TSD1flSHT9JSaSqx9JR+xW1JPLH0nObmh7HbiaWffuL97rIOYuml8tXNw7OJpV8c+uJGeCV/5Y9nSttl8X+mrlLfNOVPfMS5w/ssqcG18sSXa4lHKIaxe1LE5YX8ZZrAlO4bqlLTTiT6DO5HldP1xBsyhoknNwSuAA5l6JjSs5+P7RURR+p7Xap5MJk/0kVM6WBaevzPiCMNzUi//eox/kiXMKU5m385uZM40uQTy9qLaac8fYaW/G75ke+IpSfbB2fsBHbNK1jSM4cTo+4Q91GGPjdVcm0Ff6RWlnRgzJWK0gXEkd4V2S5t1BFL7/tZ4zZxbnCPnHp1Kg2apu6roo00MahM69IDPKgmpnSLvi0A2ONjRpp4auHKawf4Iz3MkiZ3HDzdSjvmvxK+avHy4neA3UhmlcpWJ13bzbvY9fQ4r1Op7ut1EVv5pZlMaeLQRfKFbstlUcM7j/kjzWVK/zbUhKSfx5TeaOsXUQ/QNq79S8YmzlZq5CCyU2nRQDCSfgFT+obfcNlm4vQrx5+v7K/nj7SQKfWs8ZAA6RcxpZNSQ8+WEjd9U+c2zM2O5Y+0nimNGKs9+G9+aQNTGhWwfKeNepdnT+9p/xJqqe2vzQHAv/h7s2Z8ITv9ai5uIF7nP9WifBBFu23c63rn/rSvPuPdOnp6Ntyp9NFH1bIVxA9qwt7c76uB2YmGKe144PUQ2I5jSw+UJ00CljvY0nZznf9WYbzwX9d/AO8yNqGhTQAA",
"job" : "pluginTest.2023-04-14_10.14.42_03"
}
],
"description" : "mailboxfolders",
"id" : "MailboxFolders"
}
},
"id" : "ex-admin@testlab.local",
"name" : "ex-admin@testlab.local"
}
}
}
},
"jobName" : "pluginTest"
}
See also
Go to List & Query
Go back to the Operations article.