I’ve seen this story play out dozens of times.
A team builds a retrieval-augmented generation system. It works beautifully in the demo. The model answers questions. The vector database hums along. Everyone’s excited.
Then it hits production.
Suddenly the system is returning stale data, or hallucinating with confidence, or collapsing under load, or nobody can explain why it’s behaving differently than it did last week. The team calls us in, and we find the same five problems, over and over.
Here’s what I’ve learned from watching this pattern repeat.
Mistake 1: Treating the Retrieval Layer Like an Afterthought
I’ve seen vector databases deployed without any thought to data freshness, permissions, or accuracy. The team gets excited about the language model, builds the RAG wrapper around it, and assumes the retrieval part will just work.
It doesn’t.
I’ve watched teams spend months building beautiful data pipelines, then discover the data changed. The vector database still held the old embeddings. The system returned confident answers from documents that were weeks out of date. Nobody noticed until a customer asked why the system was giving them outdated information.
Here’s what actually matters:
- Data freshness. How often does your source data update? Daily? Hourly? Real-time? Your retrieval infrastructure needs to match that cadence, not lag behind it.
- Source traceability. Can you trace where each document came from? I’ve seen vector databases where nobody could explain how documents got into them. That’s a liability, not a system.
- Accuracy at retrieval time. The model can only work with what it retrieves. If your retrieval is pulling irrelevant or incomplete documents, the model’s output quality doesn’t matter.
- Consistency. I’ve seen systems where the same question returned different answers depending on which server handled the request. That’s chaos dressed up as scale.
Fix your retrieval layer first. Everything else depends on it.
Mistake 2: No Monitoring That Actually Tells You What’s Happening
This sounds obvious, but I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean.
They’re tracking latency. They’re tracking token counts. They’re tracking vector database query times. But they’re not tracking whether the system is actually giving correct answers.
I’ve seen teams with perfect model metrics and terrible business outcomes because they were optimizing for the wrong thing. The model was fast. The retrieval was efficient. The system was reliable. And it was confidently giving customers wrong information.
What you actually need to monitor:
- Retrieval quality. Are the documents you’re pulling actually relevant to the question? This usually requires human review, at least for a sample.
- Hallucination rate. How often does the model generate information that isn’t in the retrieved documents? You need to catch this before your customers do.
- Business outcome. Whatever you built this system to do, are you actually doing it? If you built it to reduce support tickets, are support tickets going down? If you built it to improve customer satisfaction, is satisfaction improving? If you’re not measuring this, you’re flying blind.
- Data staleness. When was the last time the source data updated? When did the retrieval index last refresh? Are they in sync?
Monitoring is only useful if it tells you something you can actually act on.
Mistake 3: Access Control That’s Either Nonexistent or Paralyzing
I’ve seen teams where anyone with database access could retrain the model, which meant that a data analyst could accidentally (or intentionally) change the model’s behaviour without anyone knowing.
I’ve seen other teams where only one person could deploy models, which meant that a single person leaving the company could paralyse the entire operation.
I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks.
The right answer is usually somewhere in the middle, but finding it requires thinking about what actually needs to be controlled.
In my experience, teams usually end up with a tiered approach, and it’s the right one:
- Data access. Who can read, write, or modify the source data? This should be tightly controlled, with audit trails for everything.
- Model retraining. Who can retrain the model? This should be a small group, with approval workflows and rollback procedures.
- Deployment. Who can deploy to production? This should be documented and auditable, but not so restrictive that you can’t respond to incidents.
- Monitoring and alerting. Who gets notified when something goes wrong? This should be broad enough that problems don’t get missed, but focused enough that people actually pay attention.
The goal isn’t perfect security. It’s accountability. You need to know who did what, when, and why.
Mistake 4: No Clear Picture of What You’re Actually Running
I’ve seen teams where one person understands the entire system, and that’s a disaster waiting to happen. But I’ve also seen teams where nobody understands the whole picture, and that’s worse.
One team member leaves, and suddenly nobody knows how the data pipeline works. Another person gets promoted, and nobody can explain why the model behaves the way it does. A third person goes on vacation, and the system breaks because only they know how to fix it.
Here’s where I’ve seen the most creative chaos: teams where the data pipeline, the model training, the retrieval infrastructure, and the deployment are all maintained by different people, with no documentation connecting them. When something breaks, they spend weeks figuring out which piece failed and why.
You need someone who understands the whole system. Not one person who can do everything, but at least one person who can explain how the pieces fit together.
You also need documentation. Not the kind that nobody reads. The kind that actually answers questions like:
- How does data get from the source system into the vector database?
- What happens if the data pipeline fails?
- How do you roll back a model deployment?
- What do you do if the retrieval system starts returning irrelevant results?
- Who do you call if the system goes down at 3 AM?
Mistake 5: Measuring the Wrong Things
I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean. I’ve seen teams with perfect model metrics and terrible business outcomes because they were optimizing for the wrong thing.
Here’s what I’ve learned: the metrics that matter are the ones that connect to your business outcome.
If you built the system to reduce support tickets, measure support tickets. If you built it to improve customer satisfaction, measure satisfaction. If you built it to reduce time-to-answer, measure time-to-answer. If you built it to reduce hallucinations, measure hallucinations.
But also measure the thing that actually matters: did the business outcome improve?
I’ve seen teams where the model got better, but the business outcome got worse. The system was faster, but less accurate. The retrieval was more efficient, but less relevant. The deployment was more reliable, but less useful.
The metrics you optimize for are the outcomes you’ll get. Pick the right ones.
What Actually Works
Here’s what I’ve seen work: teams that start with their business outcome, work backwards to figure out what the system needs to do, and then build the infrastructure to support that.
They think about retrieval first, not as an afterthought. They measure what actually matters, not what’s easy to measure. They document how the system works, not as an afterthought, but as part of building it. They set up access controls that make sense for their organization, not copy-pasted from somewhere else. And they monitor the things that tell them whether the system is actually doing what they built it to do.
They also get help early. Not after the system breaks, but before it goes to production. We’ve seen the patterns play out enough times to know what questions to ask and where the problems usually hide.
How We Help
If you’re not sure whether your RAG deployment has these problems, or you’re worried you might be building one that does, we can help.
Our Diagnostic Sprint is a two-week engagement where we talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome. We audit what you have, identify the gaps between what you’re measuring and what actually matters, and build a roadmap to fix it.
Or if you’re still in the planning phase, we can help you design the system right the first time. We focus on retrieval infrastructure, model operations, governance, and the business outcomes that actually matter.
We work as one team, accountable end-to-end. Not a handoff chain where each person optimizes their piece and hopes the whole thing works. Production-first, not demo-first. Business outcomes, not technology.
FAQ
How do I know if my RAG system has these problems?
Ask yourself: Can I explain how data gets into my retrieval system? Can I trace where it came from? Do I know when it was last updated? Can I explain what my system is optimizing for? Can I point to a business outcome that’s improved because of it? If you’re hesitating on any of these, you probably have at least one of these problems.
What’s the difference between fixing these problems now and fixing them later?
Now is cheaper. Later is more expensive, and you’ve already disappointed customers or regulators or your board.
Do I need to rebuild my entire system?
Usually not. Most of these problems are fixable without a complete rebuild. But you need to understand what you have first.
What’s the first thing I should fix?
Your retrieval layer. If the data going in is stale or inaccurate or untraced, nothing else matters. Fix that first, then fix monitoring, then fix access control.
Let’s Talk
If you’re building a RAG system or you’ve already built one and you’re worried it might have these problems, reach out. We reply within a day.
Email us directly: hello@aartian.com
Or if you want to move faster, we offer a Diagnostic Sprint. Two weeks to a clear picture of what you have, what you need, and what the business case looks like. No obligation, no long-term contract. Just a clear-eyed assessment and a roadmap forward.
We’ve seen these patterns play out. We know where the problems hide. And we know how to fix them.