AI Fundamentals
LLM Latency vs Throughput: TTFT, TPOT, TPS, and RPS Explained
LLM latency measures the time required to return part or all of a response, while throughput measures completed work per unit of time. Low latency improves an individual request; high throughput improves system capacity, and optimizing one can worsen the other. A production benchmark should report both under realistic prompt lengths, output lengths, concurrency, and quality constraints.
Latency is not one number
A streamed language-model response has several useful latency measurements.
Time to first token (TTFT) is the time from sending the request until the first content token arrives. It can include network time, queueing, tokenization, prompt processing, and initial generation.
Time per output token (TPOT), also called inter-token latency (ITL), describes the average delay between generated tokens after the first token.
End-to-end latency is the time from request submission until the final response token arrives.
A simplified relationship is:
end-to-end latency ≈ TTFT + generation time
For a response with more than one output token, a simplified average is:
TPOT ≈ (end-to-end latency - TTFT) ÷ (output tokens - 1)
Benchmark tools can define boundaries differently, so compare metric definitions before comparing values.
Throughput measures system capacity
Common throughput metrics include:
Tokens per second (TPS): the number of generated tokens completed per second across a system or, in some reports, for one request.
Requests per second (RPS): the number of completed requests per second.
Input-token throughput: prompt tokens processed per second during prefill.
Output-token throughput: generated tokens produced per second during decoding.
A server can report high aggregate TPS while each user experiences slower token delivery because many requests share the same capacity. Always distinguish system-wide throughput from per-user generation speed.
Why latency and throughput trade off
Batching multiple requests can improve hardware utilization and aggregate throughput. It can also make requests wait for a batch or compete for memory and compute. As concurrency rises, total throughput may increase until the system saturates, while queueing and individual latency continue to rise.
This tradeoff is workload-dependent. Interactive chat usually values low TTFT and steady token delivery. Offline document processing may tolerate longer latency in exchange for higher batch throughput and lower unit cost.
A benchmark should therefore begin with a service objective, not a single leaderboard number.
Prompt length affects TTFT
During LLM inference, the prefill stage processes the input sequence. Longer prompts generally require more prefill work and more memory. They can increase TTFT even when output length remains unchanged.
Compare systems with the same input-token distribution. A benchmark using 128 input tokens does not predict the experience of a RAG application that sends 20,000 tokens.
Prompt caching can reduce repeated-prefix work when the provider or serving engine supports it. It does not eliminate queueing, network latency, or all prompt processing.
Output length affects end-to-end latency
Autoregressive generation emits output sequentially. More output tokens require more decoding steps. Two systems can have the same TTFT but very different end-to-end latency when one produces tokens more slowly.
Maximum-output settings also affect capacity planning. Even if average responses are short, concurrent requests that generate near the limit can occupy memory and serving slots longer than expected.
Hold output policy constant when measuring model or hardware performance.
A benchmark matrix that answers real questions
Test several combinations rather than one prompt:
| Dimension | Example test points |
|---|---|
| Input length | short, median, 95th percentile |
| Output length | short answer, normal response, upper tail |
| Concurrency | 1, expected steady state, peak |
| Request rate | below capacity, near saturation |
| Streaming | enabled and disabled if both apply |
| Quality | fixed task and acceptance threshold |
For each point, report distributions, not only averages:
- TTFT p50, p90, and p95
- TPOT or ITL p50 and p95
- End-to-end latency p50 and p95
- Aggregate output TPS
- Completed RPS
- Error and timeout rate
- Input and output token counts
- Hardware and software configuration
Percentiles reveal queueing and tail behavior hidden by means.
Cost per token depends on acceptable latency
For rented compute, a simplified unit-cost equation is:
cost per 1M output tokens = hourly compute cost ÷ output tokens per hour × 1,000,000
But “output tokens per hour” must be measured under the latency and quality constraints of the product. Driving the server to maximum throughput may violate interactive response targets. Reserving headroom improves latency and reliability but raises effective unit cost.
The open-model token cost calculator and self-host versus API calculator require measured throughput for this reason.
Streaming changes perceived latency, not total work
Streaming lets the user see partial output before the response is complete. It can improve perceived responsiveness because reading begins while generation continues. It does not necessarily reduce end-to-end latency or compute use.
A streamed application should monitor both TTFT and completion time. Optimizing only completion time can leave users staring at an empty interface. Optimizing only TTFT can produce a quick first token followed by an unacceptably slow stream.
Common benchmark mistakes
Reporting “tokens per second” without scope. State whether it is per request, per user, or aggregate system throughput.
Comparing different sequence lengths. Input and output length materially affect performance.
Ignoring concurrency. Single-user speed does not establish multi-user capacity.
Omitting queueing. Server-only measurements can look better than end-user latency.
Using peak throughput as deployable throughput. Production systems need headroom for bursts, failures, and variance.
Ignoring quality changes. A faster quantized model or shorter response is not equivalent unless it meets the same acceptance criteria.
Frequently asked questions
Is tokens per second a latency or throughput metric?
It can describe either per-user generation speed or aggregate system throughput, depending on the report. The benchmark must state the scope and calculation.
What is a good time to first token?
There is no universal threshold. The acceptable TTFT depends on whether the workload is interactive, asynchronous, batch, or agentic, and on the product’s user-experience target.
Why does throughput increase while user latency gets worse?
Higher concurrency and batching can keep hardware busier and raise aggregate output, while each request waits longer or receives a smaller share of capacity.
Should I optimize TTFT or TPOT first?
Use the product experience. A chatbot needs a responsive first token and readable streaming speed. A batch pipeline may prioritize total completion throughput and unit cost.
Sources
LLM Latency vs Throughput: TTFT, TPOT, TPS, and RPS Explained. ByteCosts. Updated 2026-06-21. https://bytecosts.com/blog/llm-latency-vs-throughput/