GPU economics

GPU Request Economics Calculator

GPU request cost is the cost of enough complete model replicas to serve a workload, divided by completed requests. The replica count rounds upward: crossing a capacity boundary can add a whole node even when demand increases only slightly. This calculator uses measured request goodput, explicit capacity headroom, whole-replica rates and operating costs, then compares the same request counts with uncached input and output API pricing. It does not assume that all GPUs, models or service levels are interchangeable.

Open the live GPU Request Economics calculator - Cost per request and replica-band break-even →

Example scenario

Original illustrative scenario: one million requests, 1,000 input and 200 output tokens each, with hypothetical API rates of $2 and $8 per million tokens, cost $3,600. A complete replica costs $2 per hour for 730 hours plus $100 in extras. At two good requests per second and 70% allocated capacity it holds 3,679,200 requests. One replica, $200 shared overhead and $0.0001 per request cost $1,860. These numbers are assumptions, not measured vendor performance or current prices.

What the inputs mean

Completed requests in the window: Enter requests. The initial value is illustrative, not an observed price or performance measurement.
Input tokens per request: Enter tokens. The initial value is illustrative, not an observed price or performance measurement.
Billed output tokens per request: Enter tokens. The initial value is illustrative, not an observed price or performance measurement.
API input rate: Enter USD / million input tokens. The initial value is illustrative, not an observed price or performance measurement.
API output rate: Enter USD / million output tokens. The initial value is illustrative, not an observed price or performance measurement.
Measured goodput per whole replica: Enter successful requests / second. The initial value is illustrative, not an observed price or performance measurement.
Allocated share of measured capacity: Enter %. The initial value is illustrative, not an observed price or performance measurement.
Provisioned hours per replica: Enter hours. The initial value is illustrative, not an observed price or performance measurement.
Whole replica hourly price: Enter USD / replica-hour. The initial value is illustrative, not an observed price or performance measurement.
Extras per replica in the window: Enter USD / replica. The initial value is illustrative, not an observed price or performance measurement.
Shared operating costs in the window: Enter USD. The initial value is illustrative, not an observed price or performance measurement.
Additional GPU-side cost per request: Enter USD / request. The initial value is illustrative, not an observed price or performance measurement.

What the result means

A negative GPU-minus-API result is lower GPU spend in this scenario, not a quality-equivalence or latency guarantee. The band crossing is not a global break-even threshold. Bursty traffic, redundancy, minimum replicas and uneven routing can require more capacity than average-volume sizing.

Assumptions

Use goodput measured per complete replica at the same model revision, quantization, prompt/output distribution, cache state, concurrency and latency objectives. The allocation percentage is extra planning headroom, not measured GPU utilization. Rates are uncached flat rates; failed attempts are not modeled here. Add their costs explicitly and use separate scenarios for disjoint cached/reasoning/tool charges. Average-volume sizing cannot prove peak capacity or equivalent quality.
All inputs must describe the same comparison window. Unknown costs are not automatically zero.
Default numbers are original scenarios; this calculator makes no network requests.

Where the prices come from

Use the GPU explorer to find source-attributed observations, then check purchase unit, region and terms. Rates here are editable assumptions and are never silently replaced.

Formula and methodology

Capacity per replica = floor(measured good requests/second x 3,600 x provisioned hours x allocated fraction). Replicas = ceil(requests/capacity), with zero replicas at zero demand. GPU cost = replicas x (hours x whole-replica hourly price + per-replica extras) + shared costs + requests x variable GPU-side request cost. API cost = requests x (input tokens x input rate + output tokens x output rate) / 1,000,000. A break-even candidate is reported only inside the current integer-replica capacity band, and only when the API unit cost exceeds the variable self-hosted unit cost.

Interpretation guide

Compare alternatives with the same workload assumptions.
Stress-test output-heavy, retry-heavy, cache-miss, and power-user cases before committing budget.
Verify source links and production logs before using the estimate for billing decisions.

Limitations before production billing decisions

Treat ByteCosts calculations as planning estimates, not final billing totals. Real invoices can differ because token mix, retry rate, cache hit rate, rate limits, taxes, gateway fees, regional pricing, and negotiated discounts change the effective cost.

Verify the provider source before production billing decisions, then compare the estimate with your own logs or invoice once production traffic is live.

GPU Request Economics Calculator. ByteCosts. https://bytecosts.com/tools/gpu-request-economics/