Skip to content
Dispatch Dispatch Dispatch

Adaptive concurrency control

Dispatch uses an algorithm called adaptive concurrency control to automatically adapt the throughput of request execution to the capacity and availability of the destinations.

Benefits

Remote systems typically throttle the request throughput. However, due to the linear relationship between throughput, latency, and concurrency, measuring the number of inflight requests allows Dispatch to dynamically adapt to rate limits, system capacity, downtime, or any combination of issues and limits that arise in distributed systems.

Dispatch offers the benefits of adaptive concurrency control with no need for upfront configuration.

Achieving fairness in multi-tenant systems

Instead of varying the transmission window size, Dispatch uses the number of inflight requests as a measure to continuously adapt the number of requests sent to each destination.

Individual nodes in the underlying Dispatch infrastructure do not share any state. Still, just like TCP, the algorithm always converges towards fairly sharing resources between tenants, preventing unhealthy links from damaging the overall quality of service.

Relationship between concurrency and throughput

Using concurrency as the metric works because distributed systems obey Little’s Law, stating that the amount of work in a system is proportional to the average rate of incoming work multiplied by the average time to perform the work:

L = λW

In our case, L is the number of inflight requests, λ the average request rate, and W is the average request latency.

AIMD Algorithm

Adaptive concurrency control is derived from the Congestion Control algorithm in network protocols like TCP.

Dispatch groups requests using a composite key, including the source organization and destination hostname. It starts each group with a default concurrency allocation and then adapts that allocation based on the result of each request:

  • On successful requests, the concurrency allocation is increased
  • On failed requests (e.g., 5xx, timeouts), the concurrency allocation is decreased

This algorithm is known as AIMD, which stands for Additive Increase Multiplicative Decrease. The increase is an addition by a constant factor, and the decrease is a multiplication by a factor smaller than 1.