Concurrency

When using Exchange EWS APIs, it is possible to find a variety of boundaries that need to be considered. We highlight some of them below:

Exchange EWS Throttling: https://learn.microsoft.com/en-us/exchange/client-developer/exchange-web-services/ews-throttling-in-exchange
Capabilities of the host serving the Exchange Service
Usage of the service during the backup window
Internet Information Server (IIS) limits: https://learn.microsoft.com/en-us/exchange/architecture/client-access/client-message-size-limits?view=exchserver-2019

If a boundary is crossed, the corresponding request will usually fail. Bacula Exchange EWS Plugin is prepared to wait some amount of time and retry it, so it has a certain level of resiliency. However, it is crucial to plan an adequate strategy to backup all the elements without needing to reach any boundary on a regular basis. This means to control how many concurrent requests are done during the backup window.

A single job implements some parallelism which can be reduced until a point, if necessary, using the following parameters:

backup_queue_size - this variable controls the size of internal queues communicating internal threads, that are designed to fetch, open and send every item to Bacula core. Reducing its size will produce, ultimately (with a value of 1 for example), an execution very similar to a single threaded process.
concurrent_threads which controls the number of simultaneous processes fetching and downloading data. This can be reduced or increased to directly affect the concurrency level of a single job.
concurrent_listing_threads controls a different pool of threads intended only to fetch information from the API. It can be reduced to 1, but increasing it over the default values won’t change significantly the behavior of the plugin.

The recommended strategy to backup a new environment is to plan a step-by-step testing scenario before putting it into production, where the number of users and the concurrency of the jobs are increased progressively. Other important point is the timing schedule as some boundaries are related to time-frames (number of request per amount of time). If you detect you reach boundaries when running all your backups during a single day of the week, try to increase the time window, and spread the load through it in order to achieve better performance results.