What is AI hallucination?

An AI hallucination is a response generated by an artificial intelligence system that contains false, fabricated, or misleading information presented as fact. The AI does not know the output is wrong — it produces the response because the underlying model predicts it as a statistically plausible continuation of the conversation, not because it has verified the content against reality.

The term is borrowed from psychology, where hallucinations describe perceptions with no basis in external reality. In AI, the analogy holds: the system confidently outputs something that isn't there. Unlike a human who might say "I'm not sure," most large language models present hallucinated content with the same tone and apparent certainty as accurate content — which is what makes the phenomenon particularly consequential in high-stakes applications like customer service.

This page covers what AI hallucination is, why it happens, what types appear most often in customer-facing AI, why hallucination carries unique risk in customer service, how organizations reduce it, and the honest limits of current mitigation approaches.

AI hallucination in one sentence

An AI hallucination is a confident, plausible-sounding output that is factually wrong.

Why AI systems hallucinate

AI hallucination is a structural consequence of how large language models work, not a bug that can be patched out. Understanding why requires a basic picture of how these models generate responses.

Large language models (LLMs) are trained on vast amounts of text. Through that training, they learn the statistical patterns of language — which words, phrases, and ideas tend to appear together in which contexts. When asked a question, the model generates a response by predicting the most statistically probable continuation of the text, token by token, based on everything it has seen.

This means the model is optimizing for plausibility, not truth. It does not look things up. It does not check its output against a source. It produces what sounds like the right answer given the pattern of the conversation — and what sounds right is very often correct, but not always.

Three specific failure modes drive most hallucinations:

Pattern completion over verification. When asked about a company's return policy, an LLM generates text that looks like a return policy — because it has seen thousands of return policies in its training data. It does not verify that the generated policy matches the actual policy. It produces the pattern.

Confidence without knowledge. LLMs do not have an internal uncertainty meter the way a human expert does. A doctor who doesn't know something says "I'm not sure, let me check." An LLM typically doesn't — it generates an answer with equal syntactic confidence whether it is certain or completely fabricating. The output reads the same either way.

Training data gaps. Every LLM has knowledge limits defined by its training data. When a model is asked about something not well-represented in that data — a brand-new product, an obscure company policy, a recent event — it fills in the gap with statistically plausible content drawn from similar contexts. The output is coherent. It is not accurate.

Types of AI hallucination

Researchers and practitioners distinguish several categories, which appear in different patterns depending on the application.

Factual hallucination is the most discussed type: the model confidently states something false. This includes inventing sources ("according to a 2024 study by Stanford..."), fabricating statistics, creating biographical details about real people that never happened, or citing legal cases that do not exist. The output is false but coherent — it reads like a real fact.

Contextual hallucination occurs when the model's output is accurate in general but wrong for the specific situation. A return policy that is technically common in the industry but doesn't apply to this company. A product feature that many products have but this product does not. The AI has not made something up wholesale — it has applied a real pattern to the wrong context.

Instruction drift happens in longer conversations or complex agentic workflows when the model loses track of the original constraint and begins operating outside it. Earlier in the conversation it was told to only discuss products the company sells; later, it invents a product. The hallucination is not about one bad fact — it's about the model gradually departing from its configured boundaries.

In customer service specifically, the most operationally damaging hallucinations tend to be contextual: the model produces something that sounds like plausible company information but contradicts the actual policy, product spec, or procedure.

Why hallucination is especially consequential in customer service

AI hallucination carries different stakes in different applications. In creative writing, generating content that wasn't in the source material might be exactly what the user wants. In a customer service context, the same behavior produces misinformation that customers may act on — and that companies may be held legally responsible for.

The most cited example is the Air Canada chatbot case. A customer asked the airline's AI about bereavement fares and was told he could apply for a discounted fare after his trip. The policy didn't exist. When the customer tried to claim the discount, Air Canada refused. A Canadian court ruled that the company was bound by its chatbot's hallucinated promise — Air Canada had to honor the invented policy and pay.

The legal precedent established a principle customer-facing AI teams need to understand: a hallucinated output from company-deployed AI can be treated by courts as a company statement. The AI's technical inaccuracy is not a defense.

The operational stakes beyond liability are equally significant. In customer service, hallucinations most often take these forms:

Policy invention. The AI creates shipping timelines, return windows, warranty terms, or discount structures that don't exist. Customers attempt to act on them and then encounter an agent or system that contradicts what the AI said, producing confusion and distrust.

Product fabrication. The AI describes product features, specifications, or compatibility that are not real. Customers make purchase decisions based on the AI's answer. The product arrives without the feature the AI described.

Procedure misrepresentation. The AI explains how to complete a return, exchange, escalation, or technical process incorrectly. The customer follows the instructions, the process fails, and the customer contacts support again — often frustrated that the company's AI gave them wrong information.

Promotional hallucination. The AI generates discount codes, sale prices, or promotional terms that aren't valid. Customers try to use them at checkout and can't, producing frustration at the exact moment of purchase conversion.

Each of these erodes the customer's trust in the brand — not just in the AI. When the AI says something wrong, the company said something wrong.

How organizations reduce AI hallucination

Hallucination cannot be fully eliminated from current AI systems, but its frequency and impact can be substantially reduced through architecture and design choices.

Retrieval-augmented generation (RAG). The most widely adopted mitigation approach. Rather than relying on the model's training data to answer questions, RAG-based systems retrieve relevant content from a defined knowledge source first, then generate a response grounded in that retrieved content. For a customer service AI, this means the model answers from the company's actual policy documents, product catalog, and knowledge base — not from its general training. It substantially reduces policy and product hallucinations because the AI is drawing on verified source material rather than pattern-completing from memory.

Knowledge grounding and source constraints. Related to RAG: configuring the AI to operate only within a defined information boundary, and to decline or escalate rather than guess when a question falls outside that boundary. A well-grounded customer service AI says "I don't have information on that, but a team member can help" rather than inventing an answer. This requires explicit configuration — models don't do this by default.

Confidence thresholds and escalation routing. Some systems implement mechanisms that detect when the model's internal confidence is low and route those interactions to human team members rather than presenting an uncertain answer as fact. The calibration of this is imperfect, but it reduces the damage from the model's worst-confidence outputs.

Human review for high-stakes interactions. For categories of interaction where a hallucinated response would carry serious risk — complex refund disputes, warranty claims, sensitive account actions — keeping a human in the loop to review AI-drafted responses before they reach the customer remains the most reliable backstop.

Regular auditing. Systematic comparison of AI responses against known-correct answers, particularly for high-traffic questions about policy, product specs, and procedures. Hallucination patterns are often detectable: if the AI gives different answers to the same question across sessions, or if customers are consistently contacting agents to say "your AI told me X and that's wrong," those are diagnostic signals.

For a deeper look at how these mitigations work in a customer service context, see the Gladly guide on AI hallucinations in customer experience.

The honest limits of current mitigation

RAG and grounding reduce hallucination substantially but do not eliminate it. A few important limitations to understand:

RAG doesn't solve everything. If the knowledge source itself is incomplete, out of date, or contains errors, the model will retrieve and repeat those errors. The quality of the knowledge base directly determines the quality of the grounded output. RAG shifts the problem from model-internal fabrication to knowledge-base maintenance — which is a better problem to have, but still a problem.

Hallucination in agentic workflows is harder to catch. When AI is not just generating text but taking actions — processing returns, updating accounts, sending emails — a hallucination in the middle of a multi-step workflow can propagate into real-world consequences before any review happens. The higher the autonomy of the AI, the more consequential the hallucination risk.

Current models cannot fully express their own uncertainty. An AI that appears confident in its response may be hallucinating; an AI that hedges may be perfectly accurate. Confidence expressed in language is not a reliable signal of factual accuracy in LLMs, which means human reviewers and auditing systems cannot rely on hedging language as a hallucination detector.

The goal is trustworthy AI, not perfect AI. Customers can accept AI limitations if those limitations are handled gracefully — an AI that says "I don't have that information, let me connect you with someone who does" builds more trust than an AI that confidently provides the wrong answer. Managing hallucination risk is as much an experience design question as a technical one.