GPU Hours to Token and Request Cost

Start with the workload, not the GPU name

An inference comparison needs an explicit unit of useful work. Record the model revision, quantization, input and output length distribution, cache state, concurrency and service-level targets before recording a speed. A measurement from another workload cannot establish your deployment’s capacity. The vLLM serving benchmark documentation distinguishes throughput from request goodput under configured latency objectives. Goodput is the more useful sizing input when requests must finish within a target.

Keep the measured configuration intact. A result from an eight-GPU replica is not a one-GPU result multiplied by eight. Likewise, aggregate output tokens per second are not the streaming speed seen by each user. The first determines aggregate capacity; the second affects the experience of one request. A cheap deployment that misses the required service level is not an equivalent alternative.

Converting measured output into an effective unit cost

For an illustrative deployment billed for H hours at a complete-replica rate P, suppose the measured output rate Q is sustainable during a productive share U of that billed time. The effective compute-only cost per million output tokens is:

compute cost per million output tokens = P * 1,000,000 / (3,600 * Q * U)

This is a derived scenario, not a hardware constant. Q must describe the same input distribution, cache behavior and concurrency as production. U represents useful production relative to the stated measured capacity, not a percentage read from a GPU activity monitor. If Q was already measured across idle periods, applying the same idle discount again would count that penalty twice.

An output-only denominator still includes the deployment cost of processing prompts. Do not compare it with an API output rate alone. Include the corresponding API input charge, billable output, caching categories and other fees. Different models can also produce different output lengths or require different numbers of attempts for the same accepted task.

Whole replicas make break-even discontinuous

The GPU Request Economics Calculator starts with requests rather than an implied token conversion. It computes each replica’s capacity from measured good requests per second, provisioned hours and a separate planning allocation. It then rounds the number of replicas upward before adding per-replica and shared expenses.

In an original illustrative example, one million requests each contain 1,000 input and 200 output tokens. Hypothetical API prices of $2 and $8 per million input and output tokens yield $3,600. One hypothetical replica costs $2 per hour for 730 hours plus $100 of replica-specific extras. Two good requests per second at 70% allocated capacity provide room for 3,679,200 requests in that window. Adding $200 shared overhead and $0.0001 per request produces $1,860 of deployment cost.

Those numbers are editable assumptions, not current provider prices or a GPU benchmark. At 3,679,201 requests, this simplified model requires another complete replica. Consequently, a crossing calculated for one replica is not a permanent threshold beyond which self-hosting always wins.

Capacity, acceptance and the purchasing unit

Average volume does not prove peak capacity. Test arrival bursts, routing imbalance, minimum replicas and redundancy separately. Preserve failed-attempt costs when reporting cost per accepted result; the existing accepted-task calculator handles that different denominator. The request calculator assumes completed requests without an additional failure-rate model.

A source can advertise a per-GPU rate while selling a multi-GPU instance. The Azure ND H100 v5 specification describes an eight-GPU configuration. Dividing its instance rate by eight is useful for display, but does not establish that one eighth of the instance is independently purchasable.

Use the GPU explorer for source-attributed observations, the VRAM tool for a separate memory feasibility estimate, and the burst billing explanation for paid time that does not produce useful work. Re-measure after a material workload or serving configuration change rather than carrying a convenient conversion factor forward.