Storage Load Balancing

Storage providers limit how many uploads one account can run at a time. A busy rule that sends every file through one storage account can hit those limits and slow down. This page explains how mxHERO paces uploads, how to spread them across a pool of accounts, and how to cut the number of uploads in the first place.

How mxHERO paces uploads

mxHERO handles each recipient of a message as a separate operation. A message to 10 recipients is 10 operations, and each may upload the same attachments.

For each storage account, mxHERO allows a fixed number of uploads at the same time:

Further uploads wait in a queue and resume on their own when a slot is free. Nothing is lost, but delivery of the email waits for its upload. If the provider itself returns a rate-limit error, mxHERO waits for the time the provider asks, then retries.

Reduce the number of uploads

Before adding accounts, check whether the rule needs one upload per recipient.

On the rule's Target step, choose Send the same copy of the content and attachment links to all recipients. The first recipient's operation uploads the files, and the others reuse the same links. A message to 500 recipients then makes 1 upload instead of up to 500.

Trade-offs of the same-copy option:

• Every recipient gets the same links, so per-recipient link types (recipient access links) and per-recipient folders don't apply.
• Between isn't available; the rule uses From.
• Some storage options, such as Collaborate Subfolder and store by user account, are turned off.

Balance uploads across accounts

A pool spreads uploads over several storage accounts, so each stays under its limit.

Before you begin

• The rule must use an organization connection (for example Box for Organizations) or Dropbox. Individual accounts, mxHERO Shares, Drag and Drop and Zip Attachments rules can't use a pool.
• Choose up to 100 storage accounts. Each must exist in your storage service.
• Choose a destination folder that every account in the pool can write to, such as a shared folder. Otherwise uploads by some accounts fail.

Steps

1. Open the rule and go to the Cloud Storage step.

2. Choose the organization connection in Cloud Storage Account.

3. In Select which storage accounts are used:, choose Balance uploads using a list of accounts.

   

4. In Storage accounts, enter each account's address and press Enter after each.

5. On the Storage Filepath step, choose a folder all the accounts can reach.

6. Save the rule.

How accounts are picked

For each message, mxHERO picks an account at random from those in the pool with a free upload slot. It isn't strict round-robin, even though the tooltip describes taking turns. An account that is busy is skipped for a few seconds. If every account is busy, the message waits and is retried, by default after about a minute.

For mailbox rules, mxHERO also sets aside, for one hour, a pool account that keeps failing, and sends the warning notification "Storage user marked unhealthy" (STORAGE_USER_WITH_ERROR). The other accounts carry on.

Each message goes to a random pool account that has a free upload slot.

Verify

1. Send several test messages through the rule.
2. In Reports › Email Flow Logs, open the messages and check the storage events. The store_file_uploaded events should show different storage accounts.
3. Watch for storage_account_concurrent_lock events. A few are normal at peaks; many mean the pool is too small.

Troubleshooting

Related pages

• Storage overview
• Create a rule
• Storage upload failures
• Supported providers and limits