Capacity Is a Chain of Budgets

A service can sit at 40% CPU and still time out because its database connection pool is full. Adding application replicas can make the outage worse: each new process brings another pool. That is why I distrust capacity estimates that stop at requests per second.

Capacity is a chain. People create requests. Requests consume application time. Some reads hit a cache; others reach a database. Writes create durable work. Background jobs use the same resources at different times. Autoscaling one link may increase pressure on the next. A plan that stops at requests per second is a partial plan.

From People to Requests

Take a document-sharing product with 50,000 daily active users. Each makes 20 requests on an average day. That is one million requests per day, or about 12 per second across the full day. If the busy hour runs at eight times that average, the application sees about 93 requests per second.

The eightfold multiplier carries a lot of weight. Office users may pile into the same working hours. A document preview may be cheap; a generated report may occupy a worker for minutes. Traffic logs, endpoint mix, payload size, and CPU time tell us whether 93 requests per second resembles the busy hour we actually need to serve.

The arithmetic is simple:

average requests/s = daily users × requests/user/day ÷ 86,400
peak requests/s    = average requests/s × peak multiplier

Now follow those 93 requests into the rest of the system.

Follow Each Request to the Next Budget

At peak, 70% of the requests are reads. A cache serves 80% of them, leaving about 13 reads per second for the database. The other 30% contribute about 28 writes per second. Counting one database operation per request for now, the database sees roughly 41 operations per second.

The database’s actual work depends on the queries behind those numbers. One unindexed search can cost more than a hundred key lookups. A write can update several indexes or hold a lock while other requests wait. An 80% hit rate averaged over a day may be much lower during the busy hour.

Lose the cache during that peak and all 65 reads reach the database. The database jumps from roughly 41 to 93 operations per second while incoming traffic stays exactly the same. Work moved along the chain.

Autoscaling Can Move the Outage

Say a load test puts sustainable application throughput at 100 requests per second per instance. At a 60% utilisation target, each instance gets a working budget of 60 requests per second. Two instances cover the 93-request peak. The database pool has its own ceiling.

If every instance may open 20 database connections, two instances can claim 40. Add ten reserved worker connections and the possible total is 50. Against a database budget of 80, that leaves headroom. Set the application to autoscale to ten instances and its possible connection claim becomes 200 before workers join it. Every application instance can look healthy while the shared database refuses new connections.

The configured upper bound is the number to budget:

possible connections = max app instances × pool ceiling
                     + worker pools
                     + scheduled jobs
                     + operational reserve

A connection pool does not create database capacity. It controls how much capacity a process can request. Sometimes a smaller pool and bounded queue protect the database better than adding more application instances.

The application and workers meet at the same database ceiling:

flowchart TB
    people[Peak requests] --> apps[Application instances]
    apps --> cache[Cache]
    cache -- misses --> apppool[Application DB pools]
    apps -- writes --> apppool
    apppool --> db[(Shared database ceiling)]
    jobs[Queued jobs] --> workers[Workers]
    workers --> workerpool[Worker DB pools]
    workerpool --> db

Autoscaling adds application pools. More workers add worker pools. A change that looks local to either tier can consume the connections the other tier needs. I would cap both sides against the same database budget and leave room for migrations, administrative work, and failure recovery.

Latency Multiplies Work in Flight

Connection counts are only part of the picture. At steady throughput, the number of requests in flight is roughly the request rate multiplied by response time. At 93 requests per second and a 200 ms response time, that is about 19 concurrent requests. If a slow dependency pushes response time to two seconds at the same arrival rate, it becomes about 186.

requests in flight ≈ requests per second × response time in seconds

Those requests occupy memory, request slots, and possibly pool waiters. The CPU graph may stay calm while latency and queueing climb. Adding instances can accept more of the incoming work, but the slow dependency still sets the completion rate. This is where timeouts, admission limits, and backpressure become capacity tools.

A short burst and a sustained peak also need different responses. Spare instances can absorb a brief spike; a queue can smooth work whose completion may lag the request. Neither helps if the average arrival rate exceeds the rate at which the slowest shared resource completes work.

Queues Have Capacity Too

Move report generation out of a request and the response gets faster. The work still has to finish. A queue with 100,000 messages and workers completing 20 per second holds more than an hour of work even if no new message arrives. When arrivals outpace completion, the backlog keeps growing.

New arrivals change the drain time sharply. At ten new jobs per second, the net drain rate is ten per second, so that 100,000-job backlog takes almost three hours to clear. At twenty arrivals per second, it never shrinks. Adding worker concurrency may help until the workers reach a shared database or provider limit; then extra workers make the bottleneck busier without draining the queue faster.

Queue depth alone conceals that delay. Arrival rate, completion rate, task duration, retries, and the age of the oldest unfinished job show what users are waiting for. A retry storm sends extra work to the service already struggling. A durable queue can preserve every job while missing the product’s deadline for all of them.

The same reasoning applies to storage. A write rate multiplied by payload size and retention tells only the logical floor. Indexes, replicas, versions, logs, and backups add physical cost. Retention policy is a capacity decision, not merely a legal setting.

Design With Headroom and Evidence

A useful capacity plan puts measured ceilings next to assumed traffic. It shows a normal peak, a burst, a dependency slowdown, a cache outage, and a worker retry surge. For each run, it names the first resource to saturate and the margin left before that happens.

Then the next decision is specific: optimise a query, limit queue dispatch, add replicas, change a cache policy, or split a workload. The Capacity Planner Lab lets you move those budgets and see which limit moves next.