LLM Hallucination Detection
LLM Hallucination Detection Methods: 5 Ways to Catch AI Errors
Date
Author
Andrew Zheng
Confident hallucinations are the hardest to catch. When a model delivers a made-up statistic with certainty, it sails past reviewers and reaches users. That's the hallucination problem at scale. Detection won't prevent these errors from being generated, but it catches them before they ship.
How Hallucination Detection Works
Detection means evaluating the model's output against a reference source and flagging responses that don't match. In RAG systems, the reference is obvious: the document you retrieved. In customer support, it might be your knowledge base. Either way, if something contradicts the source or invents facts, you catch it.
The gap most teams don't realize: catching a hallucination is only half the problem. Knowing what to do when you find one is the other half.
When a hallucination is flagged, you have three choices:
Block it: Return an error. Users see errors constantly if your hallucination rate is above 2-3%. Trust dies.
Fallback message: Tell the user you're not confident. Better UX, but they still get no answer.
Reroute to a different model: Discard the bad response, send the request to a more reliable model, give the user a real answer.
Option 3 wins on both user experience and accuracy. It requires your gateway to catch the flag, pick a fallback model, and retry automatically. Building that per application means rebuilding it for every AI feature you add. Building it once at the gateway level covers everything.
Three Types of Hallucinations
1. Factual Hallucinations
The model invents facts not in the source. Example: A chatbot answers a product question with specs that don't exist.
Easy to catch. Any method comparing output against a source document will flag it.
2. Contextual Hallucinations
The model contradicts its source directly. Example: A RAG system retrieves "Feature X is not available" but answers "You can use Feature X by doing Y."
Also easy. The contradiction is explicit.
3. Extrinsic Hallucinations
The model adds details that aren't in the source and aren't contradicted either. Example: "We're running a 20% discount this week" when the source never mentions a discount.
Hardest to catch. The added fact sounds plausible and the response seems coherent. LLM-as-a-Judge handles this better than entailment checkers, because it can reason about what's missing, not just what's present.
The 5 Main Detection Methods Explained
1. LLM-as-a-Judge
Send the output, query, and source to a second LLM. Give it a rubric: "Are all named entities correct? Are any statistics fabricated? Does the response stay within the document?" This rubric-based approach outperforms a simple "is this correct?" because it forces the model to reason about specific failure modes.
How it works: You pair your application model with a smaller, dedicated judge model. The judge sees the same context and output, evaluates against your rubric, and returns a score. Responses below a threshold get flagged.
Pros: High accuracy. Handles complex outputs, extrinsic hallucinations, and nuanced edge cases. Doesn't require labeled training data.
Cons: Adds one extra inference call (2-4 seconds of latency). Cost multiplies at scale. Judge models can hallucinate too.
Use for: Accuracy-critical responses where users depend on correctness: financial advice, medical information, legal documentation, customer support escalations.
2. Semantic Similarity
Embed the source and the response using the same embedding model, then measure cosine similarity. A score below 0.6-0.7 signals drift from the source.
How it works: You convert both the retrieved document and the generated response into vectors, then compute how similar those vectors are. Simple threshold crossing does the flagging.
Pros: Fast (milliseconds, no extra inference calls), cheap, handles long contexts well.
Cons: Medium accuracy. Misses localized errors (one false fact buried in correct content). Fails on extrinsic hallucinations (added details that don't contradict the source). Doesn't understand specific claims, just overall meaning.
Use for: High-volume RAG pipelines where you want to catch obvious drift fast. Product recommendation systems, knowledge base search.
3. Self-Consistency Sampling
Ask the model the same question 3-5 times. If answers diverge significantly, the model is uncertain, and uncertainty correlates with hallucination risk.
How it works: Generate multiple responses in parallel and measure agreement. You can check for exact matching on key facts, semantic overlap, or token-level probability convergence.
Pros: Requires no external reference or labeled data. Works across domains without retraining.
Cons: Very expensive (3-5x inference calls per query). Latency multiplies dramatically. Doesn't catch confident hallucinations (model hallucinating the same wrong answer consistently).
Use for: Offline evaluation, high-stakes analysis where latency budgets are large, internal auditing of hallucination rates.
4. Entailment Checking
Use a trained NLI (natural language inference) model. Feed it the source as a premise and individual claims as hypotheses. It returns: entailed (follows from source), neutral (neither confirmed nor contradicted), or contradicted (conflicts with source).
How it works: Traditional NLI models like RoBERTa are fine-tuned on datasets like SNLI. Newer approaches like AlignScore handle longer passages and multi-claim reasoning better, making them more practical for real documents. You can either manually split the model's response into individual sentences or claims, or use a secondary lightweight model to extract them automatically.
Pros: More precise than similarity (operates at the claim level). Faster and cheaper than LLM-as-a-Judge. Good balance of accuracy and cost.
Cons: Struggles with long, multi-claim passages (some models). Doesn't catch extrinsic hallucinations (can't reason about what's absent). Threshold tuning required per domain.
Use for: Fact-checking in knowledge bases, legal document Q&A, academic paper summarization, structured compliance checking.
5. Token Confidence Scoring
Some model APIs expose token-level log probabilities. Low probability on factual tokens (numbers, dates, proper nouns) signals the model was uncertain when generating that token. When a model is uncertain about a fact, it's more likely to hallucinate.
How it works: No extra calls needed. When your model generates a response, check the probability of tokens in factual positions. A named entity with 0.3 probability is a warning sign. The model essentially said "I'm not confident about this name or number," which correlates with increased hallucination risk.
Pros: Zero cost (probabilities come free). Zero latency. Excellent first-pass triage.
Cons: Low-medium accuracy on its own. Only available from select providers. Needs domain-specific calibration (a number matters more in a price context than a narrative).
Use for: Fast triage layer before expensive detectors, lightweight signal in high-volume pipelines.
Quick Reference: How the Methods Compare
Method | Accuracy | Latency | Cost | Best For |
|---|---|---|---|---|
LLM-as-a-Judge | High | High | High | Accuracy matters most |
Semantic Similarity | Medium | Low | Low | High-volume, broad drift |
Self-Consistency | Medium-High | Very High | Very High | Offline, high-stakes |
Entailment | Medium-High | Low-Medium | Low-Medium | Fact-checking, documents |
Token Confidence | Low-Medium | None | None | First-pass triage |
The Layered Approach: How to Do This in Practice
Token confidence first (if available): Costs nothing, catches obvious cases.
Semantic similarity: Check if response drifted from source. Narrows down what needs closer inspection.
Entailment or LLM-as-a-Judge: For the subset that failed earlier steps, run a more expensive detector.
This cuts cost while keeping accuracy where it matters.
Setting Detection Thresholds: The Hardest Part
The typical RAG pipeline is straightforward: retrieve documents, generate a response, run detection, flag or reroute anything suspicious. The hard part is setting the threshold where you actually flag something.
Set your threshold too high, you flag accurate responses as hallucinations (false positives). Users see "not confident" messages for questions the model actually answered correctly. Set it too low, errors pass through uncaught (false negatives).
There's no universal threshold. Calibrate against your own production data:
Grab 200-500 responses from your live system. Manually label each as "hallucination" or "clean."
Run your detector on all of them. Record scores at different threshold values.
Measure precision (how many flagged responses were actually hallucinations) and recall (how many actual hallucinations you caught).
Choose the threshold where precision and recall match your business priorities.
If false positives cost you (angry users seeing error messages), raise the threshold. If false negatives cost you (hallucinations reaching customers), lower it. Generic benchmarks are starting points. Your data is ground truth.
Real example: You start at 0.7 confidence threshold and measure 15% false positives. Adjust to 0.8 and re-measure. Keep iterating until you hit the tradeoff that works for your use case.
Which Method Should You Actually Use?
It depends. Here's how to pick based on what matters most to you:
Latency critical (need response in under 1 second): Start with semantic similarity or token confidence. Both add almost no delay. Use entailment checking as a second layer if you need higher accuracy.
Accuracy critical (financial, legal, medical advice): Start with LLM-as-a-Judge. Yes, it's slower. Yes, it costs more. Accuracy matters more than latency here.
High-volume, moderate accuracy (e-commerce, customer support): Layer semantic similarity (fast first pass) plus entailment checking for higher-confidence cases. Catches most hallucinations without the latency overhead of LLM-as-a-Judge.
Budget constrained: Token confidence (if available) and semantic similarity are essentially free. That's your starting point. Add entailment checking as you scale.
Most production teams use a layered approach:
Token confidence scores every response (free).
Semantic similarity narrows down suspects (cheap).
Entailment or LLM-as-a-Judge investigates the rest (expensive).
This significantly reduces costs compared to running every response through the most expensive detector.
Building Detection That Scales: The Gateway Approach
A lot of teams implement detection inside each application. Feature A gets its own detection logic. Feature B gets rebuilt from scratch. Feature C again. It burns engineers and scales badly.
[Infron's SEAR framework] takes a different approach: move detection to the gateway layer. Define your rules once and they apply to every request across all models and features automatically. When something fails quality checks, the system retries with a fallback model. The app never sees the detection logic or the retry, it just gets a clean response. This matters because a gateway sees all traffic, understands patterns across features, and can route based on real hallucination rates that individual apps can't measure.
What changes:
Build detection once, cover all features (saves rebuilding per app)
Automatic rerouting without application code changes
Observability: see which models hallucinate on which domains, optimize routing rules based on real patterns
For teams with multiple AI features, gateway-level detection reduces engineering overhead and hallucination rates faster because you're optimizing based on live production data, not guesses.
FAQ
What's a reasonable hallucination rate?
Even top models hallucinate occasionally. Focus on catching them before users see them, not eliminating hallucinations entirely.
Which LLM Hallucination Detection method should we start with?
If you have latency constraints, start with semantic similarity or token confidence. If accuracy is critical, start with LLM-as-a-Judge or entailment checking. Most teams layering methods do best.
How long does detection add to latency?
LLM-as-a-Judge adds 2-4 seconds per response. Entailment checking adds 100-500ms. Semantic similarity adds milliseconds. Token confidence adds nothing.
What's the hallucination rate for modern models?
According to recent benchmarks, top models in 2026 have reduced hallucination rates significantly. Citation hallucinations have dropped from 4-8% (2024) to 0.1-0.7% (2026). But even 1% means 100 bad responses per 10,000 queries, so detection remains important regardless of which models you use.
Ready to move detection to the gateway?
Read how SEAR (Schema-Based Evaluation and Routing) brings hallucination detection to the gateway level: SEAR: Schema-Based Evaluation and Routing for LLM Gateways.
