
You’ve Built Something That Works. Then It Says Something That Isn’t True.
I’ve seen this story play out dozens of times. A team launches an AI system that looks great in testing. It answers questions, it retrieves documents, it seems reliable. Then a customer calls. The system just told them something confidently wrong. Maybe it invented a product feature. Maybe it cited a policy that doesn’t exist. Maybe it gave them a price that’s completely made up.
That’s a hallucination, and it happens because large language models are fundamentally prediction machines. They’re trained to guess the next token based on patterns in their training data. They’re not consulting a database. They’re not looking something up. They’re generating text that feels right based on statistical patterns. When the model doesn’t have good information, or when it’s confused, it doesn’t say “I don’t know.” It makes something up. Confidently.
The scary part? Your system might be doing this right now, and you might not know it yet.
I’m going to walk you through what hallucinations actually are, why they happen, how to spot them before your customers do, and what to build so they don’t destroy your business.
What’s Actually Happening When a Model Hallucinates
A hallucination isn’t a bug in the traditional sense. It’s the model doing exactly what it was trained to do: predict the next most likely token. The problem is that “most likely” and “actually true” are not the same thing.
Imagine you ask an LLM about a specific company policy from last quarter. The model was trained on public data, maybe some internal documents, but the policy changed last month and the training data is stale. The model doesn’t know this. It sees the context, it sees patterns from similar policies, and it generates something that sounds plausible. It fills in the gap.
Or imagine you’re using retrieval-augmented generation (RAG). You’ve built a system that retrieves documents and feeds them to the model. The retrieval works fine. But the documents themselves are incomplete or outdated. The model reads them, finds gaps, and fills those gaps with hallucinations.
Or the model is just confused. It’s trying to be helpful. It doesn’t have enough context to answer accurately, so it generates something that sounds right.
The core issue is this: language models are not fact-checking systems. They’re pattern-matching systems. And when the patterns don’t give them a clear answer, they invent one.
Why Detection Is Harder Than You Think
You can’t just ask the model if it’s hallucinating. It won’t know. It will confidently defend its answer.
You can’t rely on your users to catch everything. Some hallucinations are subtle. Some are in domains where your users aren’t experts. Some are in edge cases that rarely come up. By the time a customer notices, the damage is done.
You can’t just add a fact-checking layer. Fact-checking requires knowing what the ground truth is. If you knew the ground truth for everything, you wouldn’t need an LLM in the first place.
I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean. They have dashboards showing “hallucination score,” but when you ask what that score actually measures, they can’t tell you. Is it measuring consistency? Is it measuring whether the model cites sources? Is it measuring whether the output matches the retrieved documents? Those are all different things, and they all matter.
Detection requires three things working together: monitoring what the model outputs, comparing it to reliable sources of truth, and understanding what you’re actually measuring.
Building Detection Into Your System
Here’s what actually works. Start by being specific about what “true” means in your context.
If you’re using RAG, the model should only answer questions based on the retrieved documents. That’s your source of truth. You can measure this: does the model’s answer match the content of the documents? Does it cite them? Does it admit when the documents don’t contain the answer? This is measurable and automated.
If you’re answering questions about structured data (products, policies, pricing), you can compare the model’s output against the actual data. Did it get the price right? Did it describe the feature accurately? This is also measurable.
If you’re generating text in a domain where you have domain experts, you can sample outputs and have experts review them. This is slower and more expensive, but it catches hallucinations that automated systems miss. I’ve seen teams where this is the only thing that actually works, because the hallucinations are subtle and domain-specific.
The key is to measure what matters for your business. Not generic hallucination scores. Real, specific things: does the output match the source data, does it cite sources, does it admit uncertainty, does it stay within scope.
Prevention: Fixing the Problem Upstream
Detection is important, but prevention is better. Here’s what I’ve seen work.
Fix your retrieval layer first. I’ve seen vector databases deployed without any thought to data freshness, permissions, or accuracy. I’ve seen retrieval systems where the documents were weeks out of date, and nobody noticed until a customer asked why the system was giving them outdated information. Before you blame the model, make sure your data is clean, current, and complete. Can you trace where the data came from? I’ve seen vector databases where nobody could explain how documents got into them. Fix that first.
Be explicit about scope. Tell the model what it should and shouldn’t answer. “Answer only based on the provided documents. If the documents don’t contain the answer, say so.” This doesn’t eliminate hallucinations, but it makes them less likely and more obvious when they happen.
Use retrieval as a constraint, not just context. Don’t just feed the model retrieved documents and hope it stays on topic. Measure whether the model’s answer actually comes from those documents. If it doesn’t, flag it. If the model is inventing information that isn’t in the retrieval results, that’s a hallucination you can catch automatically.
Add a verification step. For high-stakes answers, add a second model or a human to verify. This is expensive, so you need to be strategic about when you use it. But for things that matter on your P&L, it’s worth it.
Monitor consistency. I’ve seen systems where the same question returned different answers depending on which server handled the request. That’s a hallucination you can detect automatically. If the model gives different answers to the same question, something is wrong.
The Tiered Approach That Actually Works
In my experience, teams usually end up with a tiered approach. Not everything needs the same level of verification.
For low-risk questions (things that don’t affect business decisions or customer trust), you might just monitor for consistency and let the model answer freely.
For medium-risk questions (things that matter but aren’t critical), you retrieve documents, constrain the model to stay within scope, and measure whether it actually uses the retrieved documents.
For high-risk questions (things that directly affect revenue or compliance), you add human verification or a fact-checking step.
This is more work than just deploying a model and hoping for the best. But it’s less work than trying to detect hallucinations after they’ve damaged your reputation.
What Your Monitoring Dashboard Should Actually Show
This sounds obvious, but I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean.
Your dashboard should show:
- Retrieval quality. Did the system retrieve relevant documents? You can measure this by comparing retrieved documents to what an expert would have retrieved.
- Answer grounding. Does the model’s answer come from the retrieved documents, or is it making things up? You can measure this by checking whether key claims in the answer appear in the source documents.
- Consistency. Does the system give the same answer to the same question? If not, investigate why.
- User feedback. Are customers telling you the answers are wrong? This is the most important signal. Set up a way for users to flag hallucinations.
- Domain-specific metrics. If you’re answering questions about pricing, measure accuracy of prices. If you’re answering questions about policies, measure accuracy of policies.
Each of these should have a clear definition, a clear threshold for what’s acceptable, and a clear escalation path when something goes wrong.
Access Control and Audit Trails Matter More Than You Think
Here’s where I’ve seen the most creative chaos. I’ve seen teams where anyone with database access could retrain the model, which meant that a data analyst could accidentally (or intentionally) change the model’s behaviour without anyone knowing. I’ve seen other teams where only one person could deploy models, which meant that a single person leaving the company could paralyse the entire operation. I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks.
Hallucinations are often a symptom of a deeper problem: you don’t know who changed what, when, or why. You can’t trace a hallucination back to its source because you don’t have audit trails.
Build this in from the start. Who can change the retrieval data? Who can update the model? Who can change the prompts? Who can deploy to production? All of this should be logged. All of it should be auditable. Not because you’re paranoid, but because when something goes wrong (and it will), you need to know what happened.
When to Call in Help
If you’re building this yourself, you need people who understand both the model side and the data side and the monitoring side. That’s rare. Most teams have either great ML engineers or great data engineers, but not both. And they’re not thinking about production reliability the way ops engineers do.
We work with teams that have already built something and discovered that it’s hallucinating more than they expected. We help them understand where the hallucinations are coming from (is it the model, the retrieval layer, the data, or something else?), build detection into their system, and fix the root causes.
If you want to move fast, we offer a Diagnostic Sprint: two weeks to a clear picture of what you have, what you need, and what the business case looks like. We talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome. We can audit what you have, identify the gaps between what you’re doing and what actually matters for your business, and build a roadmap to close them.
You can also reach out directly at hello@aartian.com. We reply within a day.
FAQ
Can I eliminate hallucinations completely?
No. But you can make them rare enough that they don’t damage your business. The goal isn’t zero hallucinations. It’s hallucinations that are caught before they reach customers.
Should I use a smaller model to reduce hallucinations?
Not necessarily. Smaller models hallucinate too. Sometimes more. The issue isn’t the model size. It’s whether you’re constraining the model to stay within reliable sources of truth.
How often should I check for hallucinations?
Continuously. Every answer the model generates should be checked against your criteria. If you’re only spot-checking, you’ll miss things.
What if my source data is wrong?
Then the model will confidently repeat the wrong information. This is why data quality matters more than model quality. Fix your data first.
Can I use another LLM to detect hallucinations?
You can try, but it’s unreliable. One hallucinating model checking another hallucinating model doesn’t solve the problem. Use automated checks against ground truth, and use humans for things that require judgment.
The Path Forward
Hallucinations are not a model problem. They’re a system problem. They happen when you deploy a model without building the infrastructure to keep it honest.
Start by understanding what’s actually happening in your system. What are you measuring? What are you not measuring? Where are the gaps? Then build detection and prevention into your system from the start, not as an afterthought.
If you’re already in production and discovering hallucinations, don’t panic. This is fixable. But it requires understanding the root cause, and that usually means looking at the retrieval layer, the data quality, the monitoring, and the access control.
If you want help figuring out where to start, reach out. We’ve seen this pattern enough times to know what questions to ask and where the problems usually hide.