An AI hallucination is generative AI output that confidently presents erroneous or false content in response to a prompt, including output that diverges from the prompt or contradicts a previously generated statement in the same context. NIST calls the phenomenon confabulation and lists “hallucinations” and “fabrications” as its colloquial names.
The term here covers generative AI systems only, since the clinical use of “hallucination” in medicine describes an unrelated condition. The definition is the easy part. The mechanism underneath it is where readers get stuck. A system with no concept of truth still produces falsehoods in the same steady register it produces facts.
Key Takeaways
- Confabulation covers two failures at once, according to NIST. It spans content that is erroneous or false, plus generated outputs that diverge from the prompts or contradict previously generated statements in the same context.
- Confabulations are a natural result of the way generative models are designed. Those models generate outputs that approximate the statistical distribution of their training data, and LLMs predict the next token or word in a sentence.
- Training and evaluation procedures reward guessing over acknowledging uncertainty, and models optimized to be good test-takers find that guessing when uncertain improves test performance, according to Kalai and co-authors.
- Across 576,000 generated code samples from 16 popular LLMs, the average percentage of hallucinated packages was at least 5.2% for commercial models and 21.7% for open-source models.
- Hallucination cannot be eliminated in LLMs, per Xu and co-authors, who define it as inconsistency between a computable LLM and a computable ground truth function.
How Does an AI Hallucination Happen?
A hallucination arrives through normal operation rather than through a fault. The model applies its ordinary process to a question it cannot answer from what it absorbed during training.
1. Prediction Replaces Retrieval
Confabulations are a natural result of the way generative models are designed, according to NIST. Those models generate outputs that approximate the statistical distribution of their training data, and LLMs predict the next token or word in a sentence or phrase. Picture an autocomplete that was never given permission to stop typing. It has no lookup table to consult and no empty result to return, so it keeps producing the most statistically comfortable continuation available.
2. Plausible Text Beats True Text When Both Fit the Pattern
Statistical prediction can produce factually accurate and consistent outputs, and it can also produce outputs that are factually inaccurate or internally inconsistent. Kalai and co-authors argue that hallucinations originate simply as errors in binary classification. If incorrect statements cannot be distinguished from facts, hallucinations in pretrained language models will arise through natural statistical pressures.
Another comparison helps. Think of a witness rebuilding a crime scene from the statistics of every scene they have watched, instead of from a recording of that room. The reconstruction will be coherent and detailed. Whether it matches the actual room is a question the witness has no way to check.
3. Nothing in the Pipeline Flags the Gap
The dynamic is particularly relevant, per NIST, for open-ended prompts calling for long-form responses and in domains which require highly contextual or domain expertise. Deep-learning-based generation is prone to hallucinating unintended text, according to Ji and co-authors, which degrades system performance and fails to meet user expectations in many real-world scenarios.
The security consequence follows from the mechanism. A plausible-looking software package name is as easy to emit as a plausible-looking sentence, and a package name is a registrable artifact that somebody else can claim. Package hallucinations represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain. One study catalogued 205,474 unique examples of hallucinated package names.
| Step in the exchange | What the model actually does | What readers commonly assume |
|---|---|---|
| Answering a question | Predicts the next token from patterns in its training data | Retrieves a stored fact and repeats it |
| Signaling confidence | Produces fluent text in one register whether or not support exists | Hedges audibly when support is thin |
| Offering a citation | Can generate a reference that matches the shape of a real one | Only names documents it has verified |
| Reporting an error | Emits no separate failure signal | Raises a warning when something breaks |
Source: NIST AI 600-1, Generative AI Profile
Why Do AI Models Guess Instead of Admitting Uncertainty?
The mechanism explains how a wrong answer forms. It does not explain why the model answers at all rather than declining, and that gap is where most explanations stop.
Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. The analogy belongs to the paper’s authors, and it carries a sharper point than it looks like it does. Hallucinations persist due to the way most evaluations are graded, because language models are optimized to be good test-takers and guessing when uncertain improves test performance.
The authors call this epidemic of penalizing uncertain responses addressable only through socio-technical mitigation. The fix they propose is modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards, rather than introducing additional hallucination evaluations.
That places the fix upstream of any prompt a reader can write. A model that scores worse for saying “I do not know” will keep saying something else.
Is an AI Hallucination the Same as a Software Bug?
No. A bug is a defect in code that produces a deterministic wrong result, and patching the defect removes it. A confabulation follows from the design instead, because generative models are built to produce outputs that approximate the statistical distribution of their training data. Nothing was miscoded when a model invents a citation, which is why the fix looks like changed incentives rather than a patch.
Why Does an AI Hallucination Matter?
Risks from confabulations may arise when users believe false content, often due to the confident nature of the response, leading users to act upon or promote the false information. GAI outputs may also include confabulated logic or citations that purport to justify or explain the system’s answer, which may further mislead humans into inappropriately trusting the system’s output.
The defining property of this failure is the missing warning light. Classical software fails loudly, through a stack trace, a red status, or a timeout.
A confabulating model fails in the same register it succeeds in. A fabricated citation therefore does more damage than a fabricated fact. The citation is what a careful reader reaches for when they want to verify the claim above it.
Exposure widens when models run outside any review process, which is the pattern visible in ungoverned AI tool adoption inside companies. Our AI coverage keeps surfacing the same lag. Capability rankings shift every few months, while public assumptions about what a model can verify move far more slowly.
The practical consequence for a reader is narrow. Confidence in the output carries no information about whether that output is grounded. Trust has to come from the checking process rather than from the tone of the answer.
Pros, Cons, and Risks
Advantages
- The same statistical prediction that produces confabulations also produces factually accurate and consistent outputs.
- Sampling variation gives drafting and brainstorming tools their range, because a model that always returned one fixed continuation would generate far fewer usable alternatives.
- The behavior is measurable at the task level, so teams can benchmark it rather than argue about it.
Trade-offs and Risks
- Xu and co-authors show that LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers.
- Risks of confabulated content may be especially important to monitor when integrating GAI into applications involving consequential decision-making.
- Hallucination sits alongside, and is distinct from, adversarial failures such as prompt injection attack data, where the trigger is hostile input rather than an error internal to the model.
- Model-facing attack research such as LLM jailbreak attempt data tracks a separate category again, in which the output is accurate but the guardrail was bypassed.
- Mitigation strategies have been shown to significantly reduce the number of package hallucinations while maintaining code quality. Reduction is the honest word here, and no documented technique removes the behavior.
Types of AI Hallucination
NIST splits confabulation into two branches. The first is content that is erroneous or false. The second is output that diverges from the prompts or contradicts previously generated statements in the same context.
The research literature then subdivides both branches by task.
Ji and co-authors survey hallucination across six generation tasks: abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. Their survey also covers hallucination in large language models specifically.
One further type arises from fact-conflicting errors when generating code with LLMs. The package-hallucination research classifies it as a package confusion attack against the software supply chain.
| Type | What it looks like | Where it shows up | Why it is hard to catch |
|---|---|---|---|
| False content | A stated fact that is simply wrong | Open-ended prompts, long-form answers | Arrives in the same confident register as a correct answer |
| Context contradiction | Output that diverges from the prompt or contradicts an earlier turn | Long conversations, multi-step agents | The contradicted statement sits several turns back |
| Fabricated citation | A reference, quote, or link that does not exist | Research summaries, legal and clinical drafting | The citation is formatted exactly like a real one |
| Task-level hallucination | Unintended text inserted into a generated summary | Summarization and translation tools | The output is fluent and the source document is long |
| Package hallucination | A dependency name absent from any registry | Code generation in Python and JavaScript | The name looks plausible and installation is automated |
Sources: NIST, ACM Computing Surveys, USENIX Security Symposium
Real-World Applications
Hallucinated Software Packages and the Software Supply Chain
Researchers used 16 popular LLMs for code generation and two unique prompt datasets. They generated 576,000 code samples in two programming languages, then analyzed them for package hallucinations. The average percentage of hallucinated packages was at least 5.2% for commercial models and 21.7% for open-source models, including 205,474 unique examples of hallucinated package names.
The exposure comes from the reliance of popular programming languages such as Python and JavaScript on centralized package repositories and open-source software. Those registries are the Python Package Index and npm, and both let anyone claim an unused name. A hallucinated dependency is therefore not a dead end for an attacker; it is a vacant address with a queue of developers already walking toward it.
- The failure is silent at generation time, because the invented name reads like a real one.
- Automated install steps in CI pipelines can resolve the name before a human reads the diff.
- Repeat prompts tend to produce the same invented names, which makes the target predictable.
Related exposure appears in wider security defects in AI-generated code, where the generated artifact compiles cleanly and still carries risk.
Consequential Decisions in Regulated Settings
NIST illustrates the stakes with healthcare, where a confabulated summary of patient information reports could cause doctors to make incorrect diagnoses or recommend the wrong treatments. The example is a standards-body illustration of risk rather than clinical guidance.
NIST published the Generative AI Profile in 2024 as a companion to its AI Risk Management Framework, and confabulation is one of the named risks inside it. Regulated buyers now cite that document in procurement questionnaires, which turns an abstract accuracy concern into a compliance line item.
Benchmark Scoring and Model Evaluation
Researchers including authors affiliated with OpenAI argue that the fix is a change to how existing benchmarks score uncertainty, since those benchmarks are misaligned but dominate leaderboards. That reframes the problem as an industry-level one. A reader comparing releases on a model release and capability tracker is reading scores produced under exactly the grading scheme the paper criticizes.
Can AI Hallucinations Be Eliminated?
No, and there is a formal argument for why. Xu and co-authors define a formal world in which hallucination is inconsistency between a computable LLM and a computable ground truth function. Using results from learning theory, they show that LLMs cannot learn all the computable functions and will inevitably hallucinate if used as general problem solvers.
Because that formal world is part of a much more complicated real world, the authors argue hallucinations are also inevitable for real-world LLMs. Grounding, retrieval, and human review help reduce the rate. None of them closes it.
How Often Do AI Models Hallucinate?
Rates depend on the task, the model, and how the output is scored, so a single number would mislead. One measured example: across 576,000 generated code samples, hallucinated packages averaged at least 5.2% for commercial models and 21.7% for open-source models. Benchmark-level frequency across general question answering is tracked separately in measured LLM hallucination rates, which compare models rather than tasks.
Conclusion
Hallucinated packages hit at least 5.2% for commercial models and 21.7% for open-source models across 576,000 generated code samples, a measured floor for how often these models recommend a package erroneously. The property behind the number matters more. Output arrives with no signal separating a verified fact from a statistically comfortable guess, so review has to sit outside the model rather than inside its tone.
The mitigation the literature points to is socio-technical. It means modifying the scoring of existing benchmarks that dominate leaderboards, so that admitting uncertainty stops costing a model points. That fix belongs to the labs and the benchmark maintainers, well upstream of the person writing the prompt. Until it lands, any team shipping generated text or generated code should treat fluency as a formatting property, with verification as a separate step that a human or a deterministic tool owns.
