Trust & method
AI cost glossary: tokens, caching, margins, and more
This glossary defines the 23 terms that drive AI application cost - from input and output tokens to prompt caching, cache hit rate, batch vs realtime APIs, RAG, model routing, power users, gross margin, and COGS - and explains why each one moves your bill. The recurring theme: per-token list price is rarely what decides cost. Output volume, cache hit rate, retry rate, and the heavy tail of power users move an AI bill far more than the headline rate. Each term links to the calculator where it's modeled, so you can turn a definition into a number for your own workload. Definitions are plain-language and vendor-neutral; the cost notes reflect how the ByteCosts engine treats each variable.
Glossary summary
The terms are grouped around the cost variables that most often change an AI product's bill:
- Token volume: input tokens, output tokens, context window, and per-request cost explain the base model bill.
- Caching and batching: prompt caching, cache hit rate, and batch API discounts explain why effective rates can differ from list rates.
- Product economics: power users, gross margin, COGS, monthly active users, and model routing explain why a flat plan can lose money even when the average user looks profitable.
- Retrieval systems: RAG and embeddings separate indexing, query, storage, reranking, and generation costs so the pipeline does not collapse into one vague line item.
Input token
- Definition: A unit of text fed into a model - the prompt, system instructions, retrieved context, and conversation history.
- Why it matters for cost: Billed at the input rate, usually several times cheaper than output. Input-heavy workloads (RAG, classification) are where a higher-quality model can still be cost-effective.
- Model it: AI cost calculator → /tools/ai-cost-calculator
Output token
- Definition: A unit of text the model generates in its response.
- Why it matters for cost: Almost always the most expensive token class - often 3–5× the input rate. Long, verbose, or reasoning outputs dominate most AI bills, so output volume moves cost far more than the headline input price.
- Model it: AI cost calculator → /tools/ai-cost-calculator
Context window
- Definition: The maximum number of tokens (input + output) a model can consider in a single request.
- Why it matters for cost: A larger window lets you stuff in more retrieved context, but every token you send is billed - a big window is a capacity, not a discount. Workloads near the limit pay for it on every call.
Prompt caching
- Definition: Reusing a previously-processed prompt prefix (system prompt, tool defs, shared context) so it isn't re-billed at the full input rate on every request.
- Why it matters for cost: Cache reads are ~0.1× the input rate; writes cost a premium (1.25× for 5-minute, 2× for 1-hour). Caching only saves money above a break-even reuse count - cold prefixes pay the write premium for nothing.
- Model it: Prompt cache calculator → /tools/cache-savings
Cache hit rate
- Definition: The fraction of requests whose cacheable prefix is served from a warm cache rather than re-processed.
- Why it matters for cost: The single biggest lever on a cached workload's bill. A high hit rate turns expensive input into cheap cache reads; a low hit rate means you keep paying write premiums and the cache loses money.
- Model it: Prompt cache calculator → /tools/cache-savings
Batch API
- Definition: An asynchronous processing mode where requests are submitted as a batch and returned within a window (often up to 24 hours) in exchange for a discount.
- Why it matters for cost: Typically ~50% off realtime rates. Cheaper for latency-tolerant work (summarization, classification, evals) but unusable for interactive features where a user is waiting.
- Model it: Batch vs realtime calculator → /tools/batch-vs-realtime
Realtime API
- Definition: The standard synchronous mode where a request is answered immediately, suitable for interactive, user-facing features.
- Why it matters for cost: Priced at full list rates with no batch discount. You pay the realtime premium for latency, so reserve it for work where a user is actually waiting on the response.
- Model it: Batch vs realtime calculator → /tools/batch-vs-realtime
RAG
- Definition: Retrieval-augmented generation - retrieving relevant documents and passing them into the prompt as context so the model answers from grounded data.
- Why it matters for cost: Input-token heavy: the retrieved context is billed on every query. Cost scales with how much you retrieve, making chunk size, top-k, and prompt caching the main levers.
- Model it: RAG cost calculator → /tools/rag-cost-calculator
Embedding
- Definition: A numeric vector representation of text used for semantic search and retrieval.
- Why it matters for cost: Cheap per token and usually a one-time cost per document, but re-embedding a large or frequently-changing corpus adds up. Embedding is a separate billing lane from chat/completion tokens.
Vector database
- Definition: A datastore that indexes embeddings for fast nearest-neighbor search, the retrieval layer behind RAG.
- Why it matters for cost: A recurring infra cost outside model tokens - priced on stored vectors, dimensions, and query volume. Easy to forget when budgeting a RAG pipeline whose model cost looks small.
- Model it: RAG cost calculator → /tools/rag-cost-calculator
Inference provider
- Definition: A company that serves model inference over an API - a first-party lab (OpenAI, Anthropic) or a hosting/aggregator platform that serves open-weight models.
- Why it matters for cost: The same open-weight model can be priced very differently across providers, so the provider you pick - not just the model - sets your per-token cost.
- Model it: Provider pricing index → /tools/ai-provider-pricing
LLM gateway
- Definition: A proxy layer in front of one or more model providers that adds routing, retries, caching, observability, and unified billing.
- Why it matters for cost: Convenient, but it adds a percentage fee or flat cost on top of raw model spend. Budget the gateway markup separately - it isn't in any provider's list price.
- Model it: Scenario Studio → /tools/scenario-studio
Model routing
- Definition: Sending each request to the cheapest model that can handle it - e.g. a small model for easy turns and a flagship only when needed.
- Why it matters for cost: One of the largest cost levers available: shifting even a fraction of traffic to a cheaper model can cut spend sharply without a quality hit on easy requests.
- Model it: Scenario Studio → /tools/scenario-studio
Retry rate
- Definition: The share of requests that are re-sent after a failure, timeout, tool error, or low-quality result.
- Why it matters for cost: Each retry re-bills the entire context, so a single failed agent step can double a call's cost. Retries are a hidden multiplier most back-of-envelope estimates ignore.
- Model it: Scenario Studio → /tools/scenario-studio
Power user
- Definition: A small subset of users (often the top 1%) whose usage is many times the median.
- Why it matters for cost: Power users can dominate total spend and erase a flat plan's margin single-handedly. Cost-per-user averages hide them - model the heavy tail explicitly or a few accounts will run you negative.
- Model it: Scenario Studio → /tools/scenario-studio
Gross margin
- Definition: Revenue minus the cost of goods sold (COGS), as a percentage of revenue.
- Why it matters for cost: For AI SaaS, model/API cost is the dominant COGS line. A plan with thin gross margin can flip negative the moment power users or long outputs push token cost above the subscription price.
- Model it: AI SaaS margin calculator → /tools/scenario-studio
COGS
- Definition: Cost of goods sold - the direct cost of delivering the product. For an AI app this is mostly inference/API spend, plus gateway, vector DB, and egress.
- Why it matters for cost: The denominator of unit economics. If per-user COGS exceeds per-user revenue, growth loses money faster - which is why token cost, not headcount, is the number to watch.
- Model it: Scenario Studio → /tools/scenario-studio
Price confidence
- Definition: A grade (A+ to E) ByteCosts assigns each official price row based on how it was sourced.
- Why it matters for cost: Tells you how much to trust a number before you build a budget on it. A+/A rows were read from an official page; lower grades may be stale or scraped and should be re-verified.
- Model it: Methodology → /methodology
Source evidence
- Definition: The link and last-checked date attached to a price, pointing to the provider page it was read from.
- Why it matters for cost: Makes a price auditable - you can re-verify the current rate yourself before committing budget, rather than trusting an aggregator blindly.
- Model it: Sources → /sources
Token normalization
- Definition: Converting every provider's native billing unit (per-1K, per-request, per-character) into one comparable unit - USD per 1M tokens.
- Why it matters for cost: Without it, you can't compare two providers honestly. Normalization is what lets ByteCosts rank models on price across providers that quote differently.
- Model it: Methodology → /methodology
Per-user cost
- Definition: The monthly model/API cost attributable to a single active user.
- Why it matters for cost: The unit that decides whether a per-seat price is profitable. It's an average that hides power users, so always pair it with a heavy-tail view before setting a price.
- Model it: AI cost calculator → /tools/ai-cost-calculator
Per-request cost
- Definition: The cost of a single model call - input tokens × input rate plus output tokens × output rate, normalized.
- Why it matters for cost: The atomic unit all the math builds on. Multiply by requests-per-user-per-day and active users to get monthly spend; small per-request changes compound across millions of calls.
- Model it: AI cost calculator → /tools/ai-cost-calculator
Monthly active users
- Definition: The count of distinct users active in a month (MAU) - the volume driver for a usage-based bill.
- Why it matters for cost: Total cost scales roughly with MAU × requests-per-user × per-request cost. Growth that looks free can become the biggest line item once MAU and power users are priced in.
- Model it: AI cost calculator → /tools/ai-cost-calculator
Frequently asked questions
What is input token in AI cost?
A unit of text fed into a model - the prompt, system instructions, retrieved context, and conversation history. Billed at the input rate, usually several times cheaper than output. Input-heavy workloads (RAG, classification) are where a higher-quality model can still be cost-effective.
What is output token in AI cost?
A unit of text the model generates in its response. Almost always the most expensive token class - often 3–5× the input rate. Long, verbose, or reasoning outputs dominate most AI bills, so output volume moves cost far more than the headline input price.
What is context window in AI cost?
The maximum number of tokens (input + output) a model can consider in a single request. A larger window lets you stuff in more retrieved context, but every token you send is billed - a big window is a capacity, not a discount. Workloads near the limit pay for it on every call.
What is prompt caching in AI cost?
Reusing a previously-processed prompt prefix (system prompt, tool defs, shared context) so it isn't re-billed at the full input rate on every request. Cache reads are ~0.1× the input rate; writes cost a premium (1.25× for 5-minute, 2× for 1-hour). Caching only saves money above a break-even reuse count - cold prefixes pay the write premium for nothing.
What is cache hit rate in AI cost?
The fraction of requests whose cacheable prefix is served from a warm cache rather than re-processed. The single biggest lever on a cached workload's bill. A high hit rate turns expensive input into cheap cache reads; a low hit rate means you keep paying write premiums and the cache loses money.
What is batch api in AI cost?
An asynchronous processing mode where requests are submitted as a batch and returned within a window (often up to 24 hours) in exchange for a discount. Typically ~50% off realtime rates. Cheaper for latency-tolerant work (summarization, classification, evals) but unusable for interactive features where a user is waiting.
What is realtime api in AI cost?
The standard synchronous mode where a request is answered immediately, suitable for interactive, user-facing features. Priced at full list rates with no batch discount. You pay the realtime premium for latency, so reserve it for work where a user is actually waiting on the response.
What is rag in AI cost?
Retrieval-augmented generation - retrieving relevant documents and passing them into the prompt as context so the model answers from grounded data. Input-token heavy: the retrieved context is billed on every query. Cost scales with how much you retrieve, making chunk size, top-k, and prompt caching the main levers.
What is embedding in AI cost?
A numeric vector representation of text used for semantic search and retrieval. Cheap per token and usually a one-time cost per document, but re-embedding a large or frequently-changing corpus adds up. Embedding is a separate billing lane from chat/completion tokens.
What is vector database in AI cost?
A datastore that indexes embeddings for fast nearest-neighbor search, the retrieval layer behind RAG. A recurring infra cost outside model tokens - priced on stored vectors, dimensions, and query volume. Easy to forget when budgeting a RAG pipeline whose model cost looks small.
What is inference provider in AI cost?
A company that serves model inference over an API - a first-party lab (OpenAI, Anthropic) or a hosting/aggregator platform that serves open-weight models. The same open-weight model can be priced very differently across providers, so the provider you pick - not just the model - sets your per-token cost.
What is llm gateway in AI cost?
A proxy layer in front of one or more model providers that adds routing, retries, caching, observability, and unified billing. Convenient, but it adds a percentage fee or flat cost on top of raw model spend. Budget the gateway markup separately - it isn't in any provider's list price.
What is model routing in AI cost?
Sending each request to the cheapest model that can handle it - e.g. a small model for easy turns and a flagship only when needed. One of the largest cost levers available: shifting even a fraction of traffic to a cheaper model can cut spend sharply without a quality hit on easy requests.
What is retry rate in AI cost?
The share of requests that are re-sent after a failure, timeout, tool error, or low-quality result. Each retry re-bills the entire context, so a single failed agent step can double a call's cost. Retries are a hidden multiplier most back-of-envelope estimates ignore.
What is power user in AI cost?
A small subset of users (often the top 1%) whose usage is many times the median. Power users can dominate total spend and erase a flat plan's margin single-handedly. Cost-per-user averages hide them - model the heavy tail explicitly or a few accounts will run you negative.
What is gross margin in AI cost?
Revenue minus the cost of goods sold (COGS), as a percentage of revenue. For AI SaaS, model/API cost is the dominant COGS line. A plan with thin gross margin can flip negative the moment power users or long outputs push token cost above the subscription price.
What is cogs in AI cost?
Cost of goods sold - the direct cost of delivering the product. For an AI app this is mostly inference/API spend, plus gateway, vector DB, and egress. The denominator of unit economics. If per-user COGS exceeds per-user revenue, growth loses money faster - which is why token cost, not headcount, is the number to watch.
What is price confidence in AI cost?
A grade (A+ to E) ByteCosts assigns each official price row based on how it was sourced. Tells you how much to trust a number before you build a budget on it. A+/A rows were read from an official page; lower grades may be stale or scraped and should be re-verified.
What is source evidence in AI cost?
The link and last-checked date attached to a price, pointing to the provider page it was read from. Makes a price auditable - you can re-verify the current rate yourself before committing budget, rather than trusting an aggregator blindly.
What is token normalization in AI cost?
Converting every provider's native billing unit (per-1K, per-request, per-character) into one comparable unit - USD per 1M tokens. Without it, you can't compare two providers honestly. Normalization is what lets ByteCosts rank models on price across providers that quote differently.
What is per-user cost in AI cost?
The monthly model/API cost attributable to a single active user. The unit that decides whether a per-seat price is profitable. It's an average that hides power users, so always pair it with a heavy-tail view before setting a price.
What is per-request cost in AI cost?
The cost of a single model call - input tokens × input rate plus output tokens × output rate, normalized. The atomic unit all the math builds on. Multiply by requests-per-user-per-day and active users to get monthly spend; small per-request changes compound across millions of calls.
What is monthly active users in AI cost?
The count of distinct users active in a month (MAU) - the volume driver for a usage-based bill. Total cost scales roughly with MAU × requests-per-user × per-request cost. Growth that looks free can become the biggest line item once MAU and power users are priced in.
ByteCosts AI cost glossary. ByteCosts. Updated July 9, 2026. https://bytecosts.com/glossary/