Concurrency

When using Amazon EC2 APIs, it is possible to find a variety of boundaries that need to be considered. We highlight some of them below:

Amazon EC2 Throttling: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html
Capabilities of the host serving the EC2 Service
Usage of the service during the backup window

If a boundary is crossed, the corresponding request will usually fail. Bacula Amazon EC2 Plugin is prepared to wait some amount of time and retry it, so it has a certain level of resiliency. However, it is crucial to plan an adequate strategy to backup all the elements without needing to reach any boundary on a regular basis. This means to control how many concurrent requests are done during the backup window.

A single job implements parallelism which can be reduced or increased if necessary, using the following parameter:

concurrent_threads which controls the number of simultaneous processes fetching and downloading data (EBS blocks). This can be reduced or increased to directly affect the concurrency level of a single job.

The recommended strategy to backup a new environment is to plan a step-by-step testing scenario before putting it into production, where the number of instances and the concurrency of the jobs are increased progressively. Other important point is the timing schedule as some boundaries are related to time-frames (number of request per amount of time). If you detect you reach boundaries when running all your backups during a single day of the week, try to increase the time window, and spread the load through it in order to achieve better performance results.

Note that from an architectural and AWS service point of view, you can also consider to:

Run your File Daemon directly in the cloud (if your SD is also in the cloud)

Run your Storage Daemon and File Daemon in the same host, so you skip one network hop in the process (recommended)

Use a dedicated AWS connection (https://aws.amazon.com/directconnect/)

Increase throttling limits in AWS (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html#throttling-increase)