AI inference is the moment a trained model stops learning and starts working, turning its knowledge into real-world results, according to Google Cloud‘s documentation. It takes in new data and produces an instant output, such as a prediction or a decision. This entry covers that lifecycle stage, not the formal-logic or statistical senses of the word.
Inference is the process where a trained AI model generates new outputs by reasoning and making predictions on new data, per NVIDIA’s glossary. The model applies learned knowledge in real time. Serving is a neighboring term. It is the process of deploying and managing the model for inference, and it often involves setting up an API endpoint, per Google Cloud.
Key Takeaways
- Inference is the execution phase, and it uses the trained and fine-tuned model to make fast predictions on new, unseen data, according to Google Cloud. The process is a single, fast forward pass of new data.
- Each individual prediction is far less computationally demanding than training.
- Inference runs in two phases with opposite hardware profiles, per NVIDIA’s inference-optimization documentation. Prefill effectively saturates GPU utilization, and decode is a memory-bound operation.
- Anthropic prices Claude Opus 5 at $5 per million base input tokens, $0.50 per million tokens on cache hits and refreshes, and $25 per million output tokens.
- OpenAI lists gpt-5.6-sol at $5.00 short context input, $0.50 short context cached input, and $30.00 short context output.
- An attacker can cause an integrity violation by mounting an evasion attack at deployment time or a poisoning attack at training time, according to NIST.
How Does AI Inference Work?
AI inference involves three steps that turn new data into a useful output: input data preparation, model execution, and output generation, according to Google Cloud. The analysis step is called a forward pass, a read-only step where the model applies its knowledge without learning anything new.
Inside that forward pass, a language model splits the work in two.
1. The Input Is Prepared
New data is provided first, for instance a photo you have just submitted. It is then prepped for the model, which might mean simply resizing it to the exact dimensions it was trained on.
2. Prefill Reads the Prompt
In the prefill phase, the LLM processes the input tokens to compute the intermediate states (keys and values), per NVIDIA. Those states are used to generate the first new token. The prefill phase performs a matrix-matrix operation that is highly parallelized and effectively saturates GPU utilization.
Prefill is the model reading a whole page at a glance. Everything is available at once, so the hardware has plenty to chew on.
3. Decode Writes the Answer One Token at a Time
In the decode phase, the LLM generates output tokens autoregressively one at a time, until a stopping criterion is met. Each sequential output token needs to know all the previous iterations’ output states. That is like a matrix-vector operation that underutilizes the GPU compute ability compared to the prefill phase.
The speed at which the data is transferred to the GPU from memory dominates the latency rather than how fast the computation actually happens. Decode is therefore a memory-bound operation.
4. KV Caching Stops Decode Redoing Work
One common optimization for the decode phase is KV caching, per NVIDIA. The decode phase generates a single token at each time step, but each token depends on the key and value tensors of all previous tokens.
The KV cache works like keeping earlier pages of a transcript open on the desk instead of pulling the file again for every sentence.
Time to first token (TTFT) measures how long you wait before seeing the model’s output. It is the time from query submission to the first received token, according to NVIDIA’s NIM benchmarking documentation. Inter-token latency (ITL) is the average time between consecutive tokens, also known as time per output token.
| Phase | What happens | Bottleneck | Metric that tracks it |
|---|---|---|---|
| Prefill | The model processes the input tokens to compute the keys and values used to generate the first new token | Compute, since the matrix-matrix operation effectively saturates GPU utilization | Time to first token (TTFT) |
| Decode | The model generates output tokens autoregressively one at a time until a stopping criteria is met | Memory, since data transfer to the GPU dominates the latency | Inter-token latency (ITL) |
Source: NVIDIA developer documentation, NVIDIA NIM benchmarking documentation
The Split Shows Up on the Price List
The asymmetry is visible somewhere readers can check it. Anthropic charges $5 per million base input tokens on Claude Opus 5 against $25 per million output tokens. It prices a cache read (hit) at 0.1x the base input price. OpenAI lists the same three-way split on gpt-5.6-sol: $5.00 short context input, $0.50 short context cached input, and $30.00 short context output.
Prompt caching reduces costs and latency by reusing previously processed portions of a prompt across API calls. The API reads from cache at a fraction of the standard input price, according to Anthropic.
The cache-read discount is the prefill phase being skipped, priced. Two independent vendors publish the same shape: reading is cheap, writing is expensive, and re-reading something already processed is cheapest of all. We track model-by-model rates in our AI model price tiers, and the input-to-output gap holds across the tiers.
AI Inference vs Training vs Serving
AI training is the foundational learning phase, according to Google Cloud. It is a computationally intensive process where a model analyzes a massive dataset to learn patterns and relationships. It requires powerful hardware accelerators like GPUs and TPUs and can take anywhere from hours to weeks.
AI fine-tuning is a shortcut to training. It takes a powerful, pre-trained model and adapts it to a more specific task using a smaller, specialized dataset.
| Stage | Objective | Process | Business focus |
|---|---|---|---|
| Training | Create an accurate and knowledgeable model | Iteratively learns from a large dataset | Model accuracy and capability |
| Fine-tuning | Adapt a pre-trained model to a more specific task | Refines an existing model with a smaller dataset | Efficiency and customization |
| Inference | Make fast predictions on new, unseen data | A single, fast forward pass of new data | Speed (latency), scale, and cost-efficiency |
| Serving | Deploy and manage the model for inference | Package the model and expose it as an API | Reliability, scalability, and manageability of the inference endpoint |
Source: Google Cloud documentation
Predictive machine learning involves a training stage in which a model is learned, according to NIST. It also involves a deployment stage in which the model is deployed on new, unlabeled data samples to generate predictions. Standards language and vendor language land in the same place. The deployment stage is where inference runs.
What Is the Difference Between AI Training and AI Inference?
Training teaches the model, and inference uses what it learned. Training iteratively learns from a large dataset and is computationally intensive. Inference makes fast predictions on new, unseen data through a single, fast forward pass, per Google Cloud. Each individual prediction is far less computationally demanding than training.
Types of AI Inference
Deployment shape is how the chip-vendor documentation organizes the category.
- Batch inference combines multiple user requests to maximize GPU usage, providing high throughput for many users, per NVIDIA.
- Real-time inference processes data instantly as it arrives, essential for applications needing immediate decisions, like autonomous driving or video analysis.
- Distributed inference runs inference across multiple devices to parallelize computations for large models.
- Disaggregated inference divides the inference process into two stages, analysis and response generation, on specialized systems.
Disaggregated inference is the prefill and decode split from the mechanics above, moved onto separate hardware pools. Accelerator vendor share and shipment figures sit in our AI accelerator market data.
Batching is priced as well as engineered. The Batch API allows asynchronous processing of large volumes of requests with a 50% discount on both input and output tokens, according to Anthropic. OpenAI lists a separate batch tier for gpt-5.6-sol at $2.50 for short-context input and $15.00 for short-context output.
One request’s prefill phase can overlap with another request’s generation phase, per NVIDIA’s NIM benchmarking documentation. That overlap is what lets a single accelerator serve many conversations without stalling on any one of them.
Why Does AI Inference Matter?
Inference is where AI delivers business value, according to Google Cloud. For anyone building with AI, understanding how to make inference fast, scalable, and cost-effective is the key to creating successful solutions.
Delivering millions of predictions in real-time requires a highly optimized and scalable infrastructure, even though each individual prediction is far less computationally demanding than training.
Model quality gets argued in training terms while almost every operational constraint a team actually hits lives in inference. Our AI benchmark coverage tracks capability rankings that turn over within a couple of update cycles. The latency and cost numbers a deployment team watches barely move with them.
Time to first token measures how long you wait before seeing the model’s output, per NVIDIA’s NIM benchmarking documentation. That number is what a user experiences as the pause before an answer appears.
Output quality sits on a separate axis from speed and cost, and it is measured separately in model hallucination rates.
Pros, Cons, and Risks of AI Inference
Advantages
- Each individual prediction is far less computationally demanding than training, though delivering millions of predictions in real time requires highly optimized, scalable infrastructure.
- The phase is optimized for speed and efficiency, often using techniques like speculative decoding, quantization, pruning, and layer fusion to enhance accuracy, per NVIDIA.
- A cache read (hit) is priced at 0.1x the base input price. The API reads from cache instead of reprocessing the same large system prompt, document, or conversation history on every request.
Trade-offs and Risks
- Decode underutilizes the GPU compute ability compared to the prefill phase. The speed at which the data is transferred to the GPU from memory dominates the latency.
- Longer prompts increase TTFT because the attention mechanism uses the full input sequence to create the KV cache before generation begins.
- Evasion attacks require the modification of testing samples to create adversarial examples that are misclassified by the model, according to NIST. Those adversarial examples often remain stealthy and imperceptible to humans.
- Availability attacks may be initiated at training or deployment time, although their impacts are typically experienced at deployment time. They can be mounted as an energy-latency attack via query access.
NIST classifies these attacks along four dimensions, per its adversarial machine learning taxonomy. The first is the learning method and stage of the learning process when the attack is mounted. The rest are attacker goals and objectives, attacker capabilities, and attacker knowledge of the learning process.
An attack that inflates decode work is an attack on the invoice as much as on the service. Scoping what a deployed model may read, and capping how much work a single query can trigger, helps reduce that exposure without removing it. Refusal behavior under adversarial prompting is tracked separately in jailbreak attempt data.
Real-World Applications of AI Inference
Metered API Inference
Anthropic publishes per-million-token rates that separate base input, cache hits, and output. It states that introductory pricing of $2/$10 per million input/output tokens is in effect through August 31, 2026. The standard pricing of $3/$15 per million input/output tokens takes effect after that.
| Model | Base input, per million tokens | Cache hits and refreshes, per million tokens | Output, per million tokens |
|---|---|---|---|
| Claude Opus 5 | $5 | $0.50 | $25 |
| Claude Sonnet 5, through August 31, 2026 | $2 | $0.20 | $10 |
| Claude Haiku 4.5 | $1 | $0.10 | $5 |
Source: Anthropic platform pricing documentation, accessed July 2026
Claude Haiku 4.5 sits lower again, at $1 per million base input tokens and $5 per million output tokens. Rates move, so treat the table as the published documentation on the capture date rather than a permanent price.
Benchmarking a Deployment
Total tokens per second (TPS) per system represents total output token throughput across all simultaneous requests, according to NVIDIA’s NIM benchmarking documentation.
Three numbers describe a running deployment: the first-token wait, the gap between tokens, and the system-wide throughput. Agent workloads that call a model in loops multiply all three, a pattern visible in autonomous agent adoption data.
Scenario: One Chat Request, End to End
A prompt arrives at the endpoint and prefill processes the whole thing at once, building the key and value tensors it needs. The first token comes back, and that gap is the time to first token.
Decode then emits the rest one token at a time until the stopping criteria is met. The spacing between those tokens is the inter-token latency. The bill splits the same way the hardware did. The prompt is charged at the input rate, the answer at the output rate, and any repeated prefix at the cache rate.
Is AI Inference More Expensive Than Training?
Per prediction, no. Each individual prediction is far less computationally demanding than training, according to Google Cloud. Training is a computationally intensive process that requires powerful hardware accelerators and can take anywhere from hours to weeks.
In aggregate, the picture inverts. Delivering millions of predictions in real-time requires a highly optimized and scalable infrastructure. Training runs on a schedule; inference repeats for every request, which is why the published per-million-token rates are the number that compounds.
Conclusion
Inference runs as two phases with opposite hardware profiles: prefill effectively saturates GPU utilization, and decode is a memory-bound operation, per NVIDIA’s inference-optimization documentation. Anthropic’s published rate for Claude Opus 5 prices that split directly, at $5 per million base input tokens against $25 per million output tokens. A cache read (hit) costs 0.1x the base input price. Holding those two facts together is what lets a team estimate a workload’s cost before running it.
Making inference fast, scalable, and cost-effective is the key to creating successful solutions, according to Google Cloud. The same stage carries the exposure, since an attacker can cause an integrity violation by mounting an evasion attack at deployment time. Availability attacks can be mounted as an energy-latency attack via query access, per NIST. Cost control and threat modeling meet at the same phase, which makes inference an operating discipline in its own right.
