Why Most Teams Monitor the Wrong Things
I’ve watched teams spend months building beautiful monitoring dashboards, then discover that the metrics they’re tracking have nothing to do with whether the system is actually working.
Here’s the pattern I see: a team deploys a model, they set up monitoring for accuracy, precision, recall, F1 score. All the standard ML metrics. The dashboards look great. The numbers look good. And then a customer calls and says the system is giving them useless answers.
Turns out the model’s accuracy was fine. The problem was that the data feeding into the model had changed, or the retrieval system was returning outdated documents, or the system was hallucinating in ways that the accuracy metric couldn’t catch.
Monitoring is not about having the most data. It’s about knowing what to look at when something breaks.
Start with Your Business Outcome
Every monitoring strategy I’ve seen that actually works starts the same way: what does success look like for the business?
That’s harder to measure. And it’s the only thing that matters.
When I help teams build monitoring, we start by asking: what would it look like if this system was failing? Not failing technically. Failing at its job.
For a customer support RAG system, that might be: customers are getting answers that are outdated, or incomplete, or contradictory. For a financial analysis system, it might be: the system is pulling data from the wrong time period, or making calculations based on stale market data. For a code generation system, it might be: the generated code doesn’t compile, or it introduces security vulnerabilities.
Once you know what failure looks like, you can figure out what to monitor.
The Four Layers of Monitoring
In my experience, teams usually end up with a tiered approach, and it’s the right one.
Layer 1: Data and Retrieval Quality
I’ve watched vector databases deployed without any thought to data freshness, permissions, or accuracy. Then the system started returning outdated information, and nobody knew why.
If you’re running a RAG system, this is where most of your problems live. You need to know:
- Is the data in your retrieval system actually up to date? I’ve seen retrieval systems where the documents were weeks out of date, and nobody noticed until a customer asked why the system was giving them outdated information.
- Can you trace where the data came from? I’ve seen vector databases where nobody could explain how documents got into them.
- Is the retrieval system actually returning relevant documents? This sounds obvious, but I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean.
- Are there documents that should be in the system but aren’t?
This is where I’ve seen the most creative chaos: teams where the data pipeline, the model training, the retrieval infrastructure, and the deployment are all maintained by different people, with no documentation connecting them.
Start here. Fix your retrieval layer first. I’ve seen teams with perfect model metrics and terrible business outcomes because they were optimizing for the wrong thing.
Layer 2: Model Behavior
Once your data is solid, you need to monitor what the model is actually doing.
For most AI systems, this means watching for drift. The model was trained on data from six months ago. The world has changed. The patterns the model learned might not apply anymore. You need to know when that’s happening.
But drift is subtle. I’ve seen teams where the drift detection system was so sensitive that it alerted on every small fluctuation, and teams ignored all the alerts. I’ve also seen teams where the drift detection was so loose that they missed real problems.
What I’ve seen work: start with your business outcome again. If the system is supposed to answer questions about current events, you need tighter drift detection than if it’s supposed to answer questions about historical facts. If the system is supposed to generate code, you need to watch for hallucinations. If it’s supposed to make financial recommendations, you need to watch for recommendations that contradict each other or violate compliance rules.
Layer 3: System Performance
I’ve watched teams spend months evaluating data center options, comparing latency profiles and availability zones, only to discover that their actual bottleneck is somewhere completely different.
You need to monitor:
- Latency: how long does it take to get an answer? I’ve seen systems where the same question returned different answers depending on which server handled the request.
- Throughput: can the system handle the load? I’ve seen systems that worked fine with 100 requests a day completely collapse at 1,000.
- Cost: how much are you spending per request? I’ve seen teams where the cost per request was so high that the business case fell apart, but nobody noticed until the bill arrived.
- Errors: what’s actually failing? I’ve seen systems where the error rate was so high that the system was useless, but the alerts were so noisy that nobody noticed.
The trick here is not to monitor everything. Monitor the things that would make the system unusable if they broke. If latency goes from 200ms to 500ms, does that matter? Depends on your use case. If it goes from 500ms to 5 seconds, it probably does.
Layer 4: Governance and Compliance
I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks. I’ve also seen teams where only one person could deploy models, which meant that a single person leaving the company could paralyse the entire operation.
You need to monitor:
- Who changed what, and when? I’ve seen systems where the audit trail is so sparse that you can’t figure out what happened when something goes wrong.
- Are people following the access control policies? I’ve seen teams where the approval workflow is so complicated that people just approve everything without reading it.
- Is the system behaving in ways that violate compliance requirements? I’ve seen teams where one person understands the entire system, and that’s a disaster waiting to happen.
Here’s where I’ve seen the most creative chaos: the audit trail. I’ve seen audit logs that are so detailed that they’re useless, and audit logs that are so sparse that they’re useless in a different way.
What works: log the decisions, not the noise. Log when someone deploys a model. Log when someone changes an access control policy. Log when someone runs a test. Don’t log every internal function call.
What to Actually Alert On
I’ve watched teams build sophisticated monitoring systems that nobody actually used. The alerts were so noisy that people just turned them off. The dashboards were so complicated that nobody looked at them.
Here’s what I’ve seen work: one alert per failure mode. If your retrieval system returns no results, that’s a failure mode. Alert on it. If your model starts giving answers that contradict each other, that’s a failure mode. Alert on it. If your system is spending more than your budget allows, that’s a failure mode. Alert on it.
Then, for each alert, write down: what does this alert mean? What should I do about it? Who should I tell?
I’ve seen teams with perfect alert definitions and terrible runbooks. I’ve also seen teams with terrible alert definitions but great runbooks that made up for it.
What matters is that when an alert fires, someone knows what to do.
Building a Monitoring Stack That Works
You don’t need an expensive platform. I’ve seen teams buy expensive platforms and still end up with chaos. I’ve also seen teams build lean, focused stacks that run like clockwork.
Here’s what I’ve seen work:
- Start with what you can measure right now. Don’t wait for the perfect monitoring system. Start with logs, metrics, traces. Get the data flowing.
- Pick one place to look. I’ve seen teams where monitoring data was scattered across five different tools, and nobody could get a complete picture. Pick one tool, or at least one place where all the data flows.
- Build dashboards for humans, not for stakeholders. A dashboard should answer the question: is the system working? Not: how many requests did we handle today?
- Automate the boring parts. You don’t need a human to notice that latency spiked. You need a human to decide what to do about it.
- Test your alerts. I’ve seen teams with alerts that never fired because the condition was impossible, or alerts that fired constantly because the threshold was wrong.
This matters because I’ve seen teams where vector databases were deployed without any thought to data freshness, permissions, or accuracy. This sounds obvious, but I’ve seen teams that ignored this and paid for it later when they needed to switch implementations.
The Monitoring Checklist
When I help teams set up monitoring, we go through a checklist:
- Can you see what data is in your retrieval system, and when it was last updated?
- Can you detect when the model’s behavior changes?
- Can you see how long requests take, and whether they’re getting slower?
- Can you see when the system is making mistakes, and what kind of mistakes?
- Can you see who changed what, and when?
- Can you trace a customer complaint back to a specific failure?
- Do you have alerts for the things that would make the system unusable?
- Do you have runbooks that explain what to do when each alert fires?
- Does someone actually look at the alerts, or do they just go to a channel that nobody reads?
If you can’t answer yes to most of these, you don’t have monitoring. You have data collection.
When to Bring in Help
I’ve seen teams paralysed by the complexity of agent monitoring itself. They know they need to monitor something, but they don’t know what, or how, or where to start.
That’s where a diagnostic sprint makes sense. We talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome. We audit what you have, identify the gaps between what you’re measuring and what actually matters, and build a roadmap to fix it.
We’ve seen these patterns play out enough times to know what questions to ask and where the problems usually hide.
Or if you want to move faster, we offer a Diagnostic Sprint – two weeks to a clear picture of what you have, what you need, and what the business case looks like.
FAQ
Q: How often should I check my monitoring dashboards?
A: If you’re checking them constantly, something is wrong. Either your system is too unstable, or your alerts are too noisy. If you’re never checking them, that’s also a problem. Aim for: you check them once a day, and you see that everything is normal. When something breaks, the alert tells you before you check the dashboard.
Q: Should I monitor every metric, or just the important ones?
A: Monitor the important ones. I’ve seen teams with so many metrics that nobody could tell which ones mattered. Start with five metrics. If you find yourself ignoring one of them, remove it. If you find yourself wishing you had another one, add it.
Q: What’s the difference between monitoring and observability?
A: Monitoring tells you that something is wrong. Observability helps you figure out why. You need both. Start with monitoring. Once you have monitoring working, add observability.
Q: How do I know if my monitoring is working?
A: When something breaks, your alerts catch it before your customers do. That’s the test.
Let’s Talk About Your System
If you’re running an AI system in production, you’re probably monitoring something. The question is whether you’re monitoring the right things.
I’ve seen the patterns play out enough times to know what questions to ask. If you want to talk through what monitoring actually looks like for your system, or if you want to audit what you have and build a plan to fix it, reach out. We can help you figure out what matters, and what to do about it when it breaks.
Get in touch at /contact/ and let’s talk about what your system actually needs.