Hallucination
In artificial intelligence, hallucination refers to output generated by a model that is fluent and confident but factually incorrect. The model does not know it is wrong. It produces the most statistically plausible continuation of its input, and sometimes that continuation describes things that do not exist: citations that were never published, APIs that were never built, legal precedents that were never decided.
The term is borrowed loosely from psychology, but the mechanism is different. An AI model does not experience a perceptual breakdown. It generates text based on patterns in its training data, and those patterns do not always map to accurate facts. When the model encounters a gap between what it was trained on and what it is being asked, it fills that gap with plausible-sounding content rather than uncertainty.
Why Hallucination Is a Security Risk
In a casual context, hallucination is an inconvenience. In a security context, it is a risk with real consequences.
Insecure output handling is the OWASP GenAI Security Project risk most directly connected to hallucination. When an application passes LLM output to a downstream system without validation, a hallucinated response becomes an injection vector. A fabricated function name passed to a code executor, a non-existent endpoint passed to an API caller, or a malformed query passed to a database can each cause downstream failures or exploitable behavior.
Dependency confusion attacks have been demonstrated directly against hallucinating models. Security researchers have shown that LLMs reliably hallucinate plausible-sounding package names. An attacker who registers those names in a public package repository can cause developers who trust AI-generated code to unknowingly install malicious dependencies.
Eroded trust in automated workflows is a subtler but compounding risk. As organizations use LLMs to summarise threat intelligence, triage alerts, or generate incident reports, undetected hallucinations introduce errors that propagate through downstream decisions. A hallucinated indicator of compromise, a fabricated CVE reference, or a misattributed threat actor can misdirect a security investigation at a cost.
What Organizations Can Do
Treat all LLM output as untrusted until verified. Ground models in authoritative sources using retrieval-augmented generation (RAG) where accuracy is required. Validate outputs before passing them to any downstream system. Apply the OWASP GenAI Security Project guidance on insecure output handling as a baseline control for any application built on a generative model.
What is AI hallucination?
Hallucination is when an AI model generates output that is confident and coherent but factually incorrect. The model is not lying. It is producing statistically plausible text that is not true.
Why do large language models hallucinate?
LLMs generate output by predicting the most likely next token given their input. They do not retrieve facts from a verified database. When asked about something outside or at the edges of their training data, they produce plausible-sounding content rather than acknowledging the gap.
What is the security risk of AI hallucination?
The primary risk is insecure handling of output: passing unverified LLM output to downstream systems that act on it. Hallucinated package names, function calls, or query strings can introduce exploitable behavior. Hallucinated security data can misdirect investigations and incident response.
How can organizations reduce hallucination risk?
Validate all LLM output before it reaches a downstream system. Use retrieval-augmented generation to ground the model in verified sources. Apply the OWASP GenAI Security Project Top 10 for LLM Applications, particularly the guidance on insecure output handling, to any workflow that acts on AI-generated content.
