What Is LLM Inference?

Inference is not training

Training changes model parameters by optimizing them over data. Inference uses the parameters that already exist to calculate an output for a new input. Fine-tuning is also a training process because it updates some model weights or added parameters. Calling a model through an API or serving a downloaded checkpoint is inference.

This distinction changes the economics. Training cost is driven by optimization steps, training tokens, activations, gradients, optimizer states, and hardware time. Inference cost is driven by request volume, input and output lengths, model size, serving efficiency, concurrency, and the price of the API or compute capacity.

A model can be expensive to train yet economical to call, or inexpensive to download yet expensive to serve at low utilization.

The stages of a text-generation request

A production request can pass through several stages:

Request preparation: the application builds messages, tools, retrieved context, and parameters.
Queueing: the request waits for available serving capacity.
Tokenization: text is converted into token IDs.
Prefill: the model processes the input sequence and creates attention state for it.
Decoding: the model predicts and emits new tokens iteratively.
Detokenization and streaming: token IDs become text and may be sent incrementally.
Post-processing: the application validates structured output, handles tool responses, stores traces, or retries.

Different providers and serving engines can combine or optimize these stages, but separating them helps diagnose performance.

Prefill and decoding behave differently

During prefill, the model processes the prompt. Longer input sequences generally increase prompt-processing work and time to first token. During decoding, the model generates output one token at a time. Longer responses therefore require more sequential generation steps.

This produces two user-visible performance questions:

How long until the first useful token appears?
How quickly do the remaining tokens arrive?

They correspond to metrics such as time to first token and inter-token latency. The latency versus throughput guide defines these measurements and explains why one tokens-per-second number is insufficient.

What determines inference cost

For a hosted API, cost may include:

Input tokens
Cached-input categories
Output tokens
Batch or priority processing
Feature-specific or modality-specific charges
Failed calls and retries
Secondary model calls in routing or fallback flows

For self-hosted inference, cost may include:

GPU or accelerator rental
CPU, RAM, and storage
Idle capacity
Data transfer
Autoscaling headroom
Observability
Engineering and operations
Redundancy and regional deployment

The per-token cost of a self-hosted system depends heavily on utilization. A fast GPU that remains idle most of the month can have a poor effective unit cost. A highly utilized system can be economical but may expose users to queueing when demand peaks.

Use the self-host versus API calculator to model the crossover with measured throughput and realistic utilization rather than hardware peak claims.

Throughput, batching, and concurrency

Serving engines can batch work from multiple requests to use hardware more efficiently. Higher concurrency may increase total system throughput, but it can also increase queueing and per-user latency. The relationship is not unlimited. Throughput eventually saturates when compute, memory bandwidth, memory capacity, or another resource becomes the bottleneck.

This creates a service-level tradeoff. Maximizing tokens per second across the server is not the same as minimizing latency for one user.

Benchmark with the input and output distributions of the target application. A result measured with short prompts and short outputs cannot be transferred directly to a long-document workload.

Memory used during inference

Model weights are only one part of the memory budget. Inference can also require:

KV cache for processed tokens
Temporary activations
Runtime workspaces
Tokenizer and framework overhead
Batching buffers
Memory fragmentation allowance
Multiple replicas or adapters

Long context and high concurrency can make the KV cache a major constraint. Quantization reduces weight memory, but it does not automatically reduce every other category by the same factor. Read what VRAM means for LLMs and what model quantization is before selecting hardware.

Inference quality is part of the system

Performance optimization should not invalidate the output. Changes to precision, quantization method, sampling, batching, prompt truncation, or model routing can affect quality. A meaningful comparison holds the task, input distribution, output policy, and quality threshold constant.

A cheap response that fails validation and triggers a retry may cost more than a more capable first attempt. Multi-step workflows amplify this effect because one user action can produce several model calls and repeated processing stages.

Hosted inference versus local inference

Hosted APIs transfer most serving responsibility to a provider. They offer simple scaling and usage-based billing but expose the product to provider prices, limits, and terms.

Local or self-hosted inference gives more control over model, data path, runtime, and capacity. It also makes the operator responsible for deployment, performance, reliability, and utilization.

Neither approach is universally cheaper. The correct comparison is a workload model with the same quality, latency, availability, and volume requirements.

Frequently asked questions

What is the difference between inference and generation?

Inference is the broader process of running the trained model. Generation is the iterative production of output tokens in a generative language-model request.

What is prefill in LLM inference?

Prefill is the stage where the model processes the input prompt and prepares attention state before iterative output generation begins. Longer input sequences generally increase prefill work.

Why is LLM inference memory-intensive?

Large models require memory for weights, and serving also uses memory for KV cache, activations, workspaces, and concurrent requests. The exact mix depends on architecture and runtime.

Is inference cost the same as API price?

No. API price is one commercial way to charge for hosted inference. Self-hosted inference has compute and operating costs, while API workflows can also add retries, retrieval, tools, and other services.

What Is LLM Inference? Prefill, Decoding, Latency, and Cost