You’ve Built the RAG System. Now It’s Giving You Wrong Answers.

This is the conversation I have almost every week. A team has deployed a retrieval-augmented generation system. It works in demos. It works in testing. Then it goes to production and starts confidently telling customers things that are incomplete, outdated, or just plain wrong.

The frustrating part: the model itself isn’t broken. The retrieval layer is.

I’ve watched this play out dozens of times, and it’s almost always the same story. Teams focus on the model, on prompt engineering, on the latest inference optimization. They treat the retrieval infrastructure like plumbing: necessary, but not interesting. Then they discover that their vector database has no idea when documents were last updated, or who has permission to change them, or whether the source data is even still accurate.

Here’s what I’ve learned from working through this with real teams: hallucinations in RAG systems aren’t usually a model problem. They’re a data problem. And data problems are infrastructure problems.

Let me walk you through how to build this right from the start.

Start with Your Data Sources, Not Your Vector Database

This is where most teams get it backwards. They pick a vector database first. They think about embeddings. They start chunking documents. Then they realize they have no idea where the documents came from, when they were added, or whether they’re still accurate.

Start here instead: what are your actual source-of-truth data systems?

For a financial services team, that might be your regulatory filing database, your client account system, and your compliance documentation store. For a healthcare organization, it might be your electronic health record system and your clinical guidelines repository. For a product company, it might be your knowledge base, your API documentation, and your customer support tickets.

Write down the answers to these questions for each source:

I’ve seen vector databases where nobody could explain how documents got into them. That’s a sign that you skipped this step.

Build Your Freshness Pipeline Before You Build Your Index

Once you know your sources, you need to decide how to keep your retrieval system in sync with them.

There are basically three patterns I’ve seen work:

Pattern 1: Continuous sync. Your retrieval pipeline watches your source systems and updates the index whenever something changes. This works well if your sources publish change events (most modern databases do). The downside: you need monitoring to know if the sync breaks.

Pattern 2: Scheduled refresh. You run a batch job every hour, every day, or every week that pulls the latest data from your sources and rebuilds relevant parts of your index. This is simpler to reason about. The downside: you’re always a little bit out of date.

Pattern 3: Hybrid. You do continuous sync for critical data and scheduled refresh for everything else. This is what I’ve seen work best in practice.

Here’s the part that matters: you need to know which pattern you’re using, and you need to be able to answer the question “how stale is my data right now?” in production. I’ve seen retrieval systems where the documents were weeks out of date, and nobody noticed until a customer asked why the system was giving them outdated information.

Chunk Strategically, Not Automatically

Chunking is where your retrieval quality lives or dies.

Most teams chunk by size: take a document, split it into 512-token chunks, embed each chunk, done. This works sometimes. It also creates a lot of orphaned context. A question about “what is our refund policy for digital goods?” might retrieve a chunk that says “…for digital goods, customers have 30 days to request a refund. Refunds are issued to the original payment method. For physical goods…” The model sees that chunk and hallucinates the rest.

Here’s what I’ve seen work better: chunk by semantic boundary whenever you can.

If you have structured data (a FAQ, a set of policies, a product catalog), chunk by that structure. Each FAQ entry is one chunk. Each policy section is one chunk. Each product is one chunk. This gives your retrieval system a fighting chance of getting the whole picture.

For unstructured documents, you have more work to do. Read the document. Understand its structure. Chunk by sections, by topics, by the natural breaks in the content. This takes longer than automatic chunking. It’s worth it.

I’ve also seen teams use a two-stage approach: retrieve a chunk, then expand to the full section or document before passing it to the model. This catches a lot of hallucinations that happen because the model only saw part of the picture.

Set Up Evaluation Before You Go to Production

You need to know whether your retrieval system is actually working.

This is where I’ve seen the most creative chaos. Teams deploy retrieval systems without any way to measure whether the retrieved documents are actually relevant to the question. They rely on user complaints to find problems.

Build an evaluation harness. Start small. Pick 20 or 30 questions that matter to your business. For each question, write down what documents should be retrieved. Then, every time you change your chunking strategy, your embedding model, your index, or your freshness pipeline, run those 20 questions through the system and see if you still get the right documents.

This sounds obvious, but I’ve seen teams with sophisticated monitoring systems where nobody actually knows whether retrieval quality is going up or down.

You’ll also want to measure end-to-end accuracy: does the model give the right answer when given the right documents? This is different from retrieval accuracy. I’ve seen systems where retrieval was perfect but the model still hallucinated because it misunderstood the retrieved context.

Governance: Know What Your System Can Access

Here’s the question I get asked most often: “Can you guarantee my data stays in my tenant?”

The second question is usually: “How do I know what documents my retrieval system actually has access to?”

This matters because I’ve seen vector databases deployed without any thought to data freshness, permissions, or accuracy. More importantly, I’ve seen teams where anyone with database access could change what documents the system retrieves, which meant that a data analyst could accidentally (or intentionally) change the model’s behaviour without anyone knowing.

You need to answer these questions:

I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks. I’ve also seen teams where nobody understood the access control system well enough to actually use it, so they just gave everyone full access and hoped for the best.

The right approach usually sits in the middle: clear, documented access policies that actually match your business needs. Not perfect security theater, but real accountability.

Monitoring and Alerts That Actually Matter

Once you’re in production, you need to know when something breaks.

I’ve seen teams with hundreds of alerts and alert fatigue so bad that they ignore all of them. I’ve also seen teams with so few alerts that they don’t notice problems until customers complain.

Here’s what I’ve seen work: monitor the things that affect your business outcome.

But also: make sure someone actually responds to these alerts. I’ve seen teams with perfect monitoring and terrible incident response because nobody knew what to do when an alert fired.

The Audit Trail: Trace Everything

Here’s the third question I get asked: “If something goes wrong, can you show me exactly what happened?”

This is where I’ve seen the most creative chaos: the audit trail.

I’ve seen systems where the audit trail is so sparse that you can’t figure out what happened when something goes wrong. I’ve also seen audit logs that are so detailed that they’re useless.

You need to log:

I’ve seen teams where one person understands the entire system, and that’s a disaster waiting to happen. When that person leaves, nobody else knows how to debug problems. The audit trail is your insurance policy.

Bringing It Together: A Production-First Approach

Here’s what I’ve seen work: teams that start with their business outcome, work backwards to figure out what the system needs to do, and then build the infrastructure to support that.

They don’t start with the latest vector database. They don’t start with the most sophisticated embedding model. They start with a question: “What does this system need to do to be valuable to our business?”

Then they work backwards. If the system needs to give accurate answers about our policies, what data sources do we need? How often do those sources change? How stale can our retrieval system be before it starts giving wrong answers? What evaluation harness do we need to know whether it’s working?

Then they build the infrastructure. And they do it in production, with real data, with real users, with real consequences if something breaks. Not in a demo environment where everything works.

This is slower at first. It’s faster in the long run.

FAQ

Q: Should we use a managed vector database or build our own?

A: That depends on your data freshness requirements, your compliance requirements, and your tolerance for vendor lock-in. I’ve seen teams get locked into a vendor’s way of doing things, only to discover that their workflow doesn’t fit the platform’s assumptions. I’ve also seen teams build lean, focused stacks that run like clockwork. This is worth thinking through carefully. We can help you evaluate this decision.

Q: How do we know if our retrieval system is hallucinating?

A: You need an evaluation harness. Start with 20 or 30 questions that matter to your business. For each question, write down what documents should be retrieved. Then measure whether your system retrieves them. Also measure whether the model gives the right answer when given the right documents. That’s where you’ll find hallucinations that your retrieval system didn’t cause.

Q: What embedding model should we use?

A: The one that works for your data and your queries. There’s no universal answer. I’ve seen teams get great results with open models and great results with proprietary models. I’ve also seen teams waste months evaluating embedding models when their real problem was data freshness or chunking strategy. Fix your retrieval layer first.

Q: How do we handle compliance and audit requirements?

A: You need to know what your compliance requirements actually are. Then you need to audit what you have, identify the gaps, and build a roadmap to close them. Many teams skip this step and discover compliance problems in production. That’s expensive and stressful. We can help you get this right.

Let’s Talk About Your System

If you’re building a RAG system right now, or if you have one in production that’s not working the way you hoped, I’d like to talk to you. Not to sell you something, but to understand what you’re trying to do and where the problems are hiding.

Most teams don’t need a new vector database or a new embedding model. They need to fix their data pipeline, their freshness strategy, their evaluation harness, or their governance layer. We’ve seen these patterns play out enough times to know what questions to ask and where the problems usually hide.

If you want a quick assessment of where you stand, we offer a Diagnostic Sprint: two weeks to a clear picture of what you have, what you need, and what the business case looks like. We talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome.

Then we give you a prioritized build plan based on what’s actually holding you back.

Get in touch. Let’s figure this out together.