The Question I Hear Every Week
You’ve deployed an AI system to improve a process. It’s working. The model accuracy is good. The latency is acceptable. But when your CFO asks “what’s the actual return on this?”, you realize you don’t have a clean answer.
Maybe you’re measuring things like model accuracy, inference speed, or the number of documents your retrieval system can handle. Those are real metrics. But they’re not ROI. They’re infrastructure metrics. They tell you the system works. They don’t tell you whether it was worth building.
I’ve watched this play out dozens of times. Teams spend months optimizing metrics that have almost nothing to do with business outcomes. Then they’re shocked when the CFO asks why the company should keep funding the project.
The problem isn’t that you’re measuring the wrong things. The problem is that you’re measuring too many things, and none of them connect to money.
What Actually Matters: The P&L Connection
Let me be direct. ROI means return on investment. It means money in, money out, and the difference between them. Everything else is a supporting metric.
For an AI process improvement, that usually means one of three things:
First, you’re saving labor. A process that used to take a person eight hours now takes two hours. That’s a real saving. You can calculate it: hours saved multiplied by loaded cost per hour, minus the cost to run the AI system. That’s ROI.
Second, you’re enabling volume. A process that used to be a bottleneck can now handle ten times the throughput. If that throughput has a margin attached to it, you can calculate the incremental revenue. Subtract the cost to run the system. That’s ROI.
Third, you’re reducing errors or risk. A compliance process that used to miss things now catches them. A customer-facing process that used to frustrate people now delights them. These are harder to quantify, but they’re not impossible. You can measure the cost of the errors you’re preventing, or the customer lifetime value you’re protecting.
Everything else is a vanity metric. Model accuracy, latency, throughput in requests per second, the size of your vector database, the number of tokens processed, the cost per inference. These are all real metrics. But they’re only useful if they connect to one of those three outcomes.
The Vanity Metrics Trap
Here’s where teams get stuck. They optimize for metrics that sound impressive but don’t move the needle.
I’ve seen teams celebrate a 15% improvement in model accuracy. It’s a vanity metric.
I’ve seen teams obsess over latency. They cut response time from 800 milliseconds to 200 milliseconds. But if the process is asynchronous and nobody’s waiting for the answer in real time, that improvement doesn’t matter. It costs money to achieve and delivers zero value.
I’ve seen teams measure “documents processed” or “tokens consumed”. Those are activity metrics. They tell you the system is busy. They don’t tell you whether being busy is making money.
The trap is that these metrics are easy to measure. You can pull them from your logs. You can track them over time. You can show improvement. But improvement in a vanity metric is not the same as improvement in business outcome.
How to Connect Your Metrics to Money
Start with the business outcome. Not the AI outcome. The business outcome.
If you’re improving a process, someone owns that process. Their job is measured on something. Maybe it’s cost. Maybe it’s speed. Maybe it’s quality or customer satisfaction. Whatever it is, that’s your north star.
Now work backwards. What would need to be true about your AI system for that business outcome to improve?
If the owner’s metric is labor cost, then you need to measure time saved. Not model accuracy. Time saved. How long did the process take before? How long does it take now? What’s the difference? That’s your number.
If the owner’s metric is throughput, then you need to measure volume handled. Not requests per second. Volume that actually moved through the process and generated revenue. That’s your number.
If the owner’s metric is error rate, then you need to measure errors prevented. Not false positives in your validation set. Actual errors that would have made it to a customer or a regulator. That’s your number.
Once you have that number, you can calculate ROI. Take the business value of the improvement. Subtract the cost to build and run the system. Divide by the cost to build and run the system. That’s your return.
The Hidden Costs Nobody Measures
Here’s where most ROI calculations fall apart. Teams measure the benefit side accurately but ignore half the cost side.
Yes, the AI system has a direct cost. Compute, storage, maybe licensing fees. But that’s usually the smaller cost.
The bigger costs are invisible if you’re not looking for them.
There’s the cost of keeping the system running. Monitoring. Alerting. On-call support. Incident response when something breaks at 2 AM. I’ve seen teams deploy a model and then discover they need to hire an MLOps engineer just to keep it alive. That’s a real cost. It needs to be in your ROI calculation.
There’s the cost of maintaining data quality. If your system depends on fresh data, someone has to manage that pipeline. If the data goes stale, the system breaks. That’s a real cost.
There’s the cost of governance and compliance. If your system touches regulated data or makes decisions that affect customers, you need audit trails, access control, testing. That’s a real cost. I’ve seen teams spend more on governance than on the model itself.
There’s the cost of keeping the system accurate. Models drift. Data changes. Competitors change. If you’re not monitoring for drift and retraining when needed, your system will slowly get worse. That’s a real cost.
Most teams measure the benefit side accurately but miss half the cost side. Then they’re shocked when the ROI turns out to be lower than they expected.
The Baseline Problem
Here’s another trap. You need a baseline to measure against.
If you’re automating a process, what was the cost of doing it manually? If you don’t know, you can’t calculate savings. You need to measure the current state before you deploy the system.
I’ve seen teams deploy a system and then try to estimate what the old cost was. That never works. You end up with a number that sounds plausible but isn’t real. Your CFO will smell it.
Measure the baseline first. How many hours does this process take? How many errors does it make? How much does it cost to run? Write it down. Then deploy the system. Then measure again.
The difference is your ROI.
If you’re improving an existing process rather than automating it, the baseline is the current state. Measure it before you change anything.
When the ROI is Negative
Sometimes the math doesn’t work out. The cost to build and run the system is higher than the benefit.
That’s not a failure. That’s useful information.
I’ve seen teams discover that automating a process costs more than paying someone to do it manually. That tells you something. Maybe the process is too complex. Maybe the volume is too low. Maybe the errors are rare enough that preventing them isn’t worth the cost.
That’s when you need to make a decision. Do you accept the negative ROI because there’s a strategic reason to automate? Or do you kill the project and move on?
But you can only make that decision if you know the ROI is negative. If you’re measuring vanity metrics, you won’t know.
Building a Measurement Framework
Here’s what I’ve seen work. Start simple.
Identify one business metric that your AI system is supposed to improve. Just one. Make sure it’s something that connects to money or risk.
Measure that metric before you deploy the system. Write it down.
Deploy the system.
Measure that metric again after the system has been running for a month, or a quarter, or whatever timeframe makes sense for your business.
Calculate the improvement. Multiply by the value per unit. That’s your benefit.
Add up all the costs: compute, storage, labor, governance, everything. That’s your cost.
Subtract cost from benefit. Divide by cost. That’s your ROI.
Done.
Everything else is supporting detail. You can measure model accuracy and latency and throughput if you want. Those metrics can help you understand why the business metric improved or didn’t. But they’re not the ROI. They’re diagnostics.
The Ongoing Measurement Problem
ROI isn’t a one-time calculation. It changes over time.
When you first deploy a system, the cost is high and the benefit might be low. ROI is bad.
As you optimize the system and learn how to use it better, the benefit goes up and the cost per unit of benefit goes down. ROI improves.
But as time goes on, the system might drift. Data might change. Competitors might change. The benefit might go down. ROI gets worse.
You need to measure ROI regularly. Monthly, quarterly, whatever makes sense. Track it over time. If it’s trending down, you need to know why and fix it.
I’ve seen teams deploy a system with great ROI, then ignore it for a year while the ROI slowly became negative. By the time they noticed, they’d wasted a lot of money.
FAQ
Q: What if I can’t measure the business outcome directly?
A: You can usually find a proxy. If you can’t measure labor cost saved, measure time saved and use an industry standard for loaded cost per hour. If you can’t measure error cost, measure error rate and estimate the cost per error based on similar processes in your industry.
Q: Should I include sunk costs in my ROI calculation?
A: No. ROI is forward-looking. If you spent money building the system, that’s done. Your ROI calculation should include only the ongoing cost to run the system and the ongoing benefit it generates.
Q: What’s a good ROI for an AI system?
A: That depends on your business. A 2x return (100% ROI) in the first year is solid. A 3x return is great. A 5x return is exceptional. But the bar is different for every company and every process.
Q: How long should I wait before calculating ROI?
A: Long enough for the system to reach steady state. If you’re automating a process, that might be a month. If you’re improving something that changes seasonally, that might be a year. Don’t calculate ROI before the system has had time to prove itself.
Let’s Talk About Your Numbers
If you’re trying to calculate ROI on an AI process improvement and the math isn’t working, we can help. We’ve seen this enough times to know where the problems usually hide.
Often it’s not that the system is bad. It’s that you’re measuring the wrong thing, or you’re missing a cost, or the baseline was wrong.
We can audit what you have, understand where your measurement is breaking down, and build a plan for calculating ROI in a way that actually makes sense to your CFO.
Or if you’re still in the planning stage and want to figure out whether an AI process improvement is worth building in the first place, we offer a Diagnostic Sprint. Two weeks to understand what you have, what you need, and what the business case actually looks like.
Get in touch. Let’s talk about your numbers.