
You’ve Built Something, and It’s Not Working
You’ve got an AI system in production. Maybe it’s a retrieval system pulling answers from your documents. Maybe it’s a model making decisions about your business. Maybe it’s all of the above, layered together in ways that made sense six months ago but now feel fragile.
And somewhere along the way, it stopped feeling like progress. The metrics look good on paper. The model accuracy is solid. But the business outcome is stuck. Or the system is too expensive to run. Or it’s so complicated that only one person understands it, and that person is thinking about leaving.
So you start looking for help. And suddenly you’re swimming in options. There are consultancies that will sell you platforms. There are firms that will tell you to rebuild everything from scratch. There are freelancers who will optimize your model for vanity metrics. There are vendors with impressive case studies that don’t quite match your situation.
How do you know who to actually trust?
I’ve been on both sides of this conversation. I’ve watched teams hire the wrong consultancy and waste months. I’ve also watched teams find the right partner and move their business forward in weeks. The difference isn’t always obvious until you’re deep into the engagement.
Let me share what I’ve learned from doing this work every day.
The First Question: What Are They Optimizing For?
This is the one that matters most, and it’s the one most consultancies won’t answer directly.
Some consultancies optimize for billable hours. They’ll find problems, scope them broadly, and keep you on retainer for as long as possible. The work is real, but the incentive is to keep things complex.
Some optimize for technology. They want to show you their platform, their architecture, their latest tool. The tool might be genuinely useful, but it might also be solving a problem you don’t have.
Some optimize for demos. They’ll build something beautiful that works in a controlled environment, hand it off, and move on. When it hits production and the real data doesn’t cooperate, you’re on your own.
The consultancy you want is optimizing for your business outcome. That means they care about what shows up on your P&L, not what looks impressive in a presentation. It means they’re willing to say “you don’t need a new system, you need to fix the one you have.” It means they scope their work around results, not around billable hours.
How do you tell the difference? Ask them directly: “How do you measure success?” If the answer is about your business metrics, you’re on the right track. If it’s about technology metrics or platform adoption, keep looking.
What Does Their Process Actually Look Like?
I’ve watched teams hire consultancies that disappeared into a room for three months and came back with a 200-page document that nobody read.
I’ve also watched consultancies that talked to everyone but never wrote anything down, so six weeks later nobody could remember what was decided.
The right process is messy and specific. It involves talking to the people who built your system, the people who maintain it, the people who depend on it, and the people who own the business outcome. Not in separate meetings where everyone tells a different story, but in a way that surfaces the contradictions and resolves them.
Then it involves actually looking at what you have. Not a PowerPoint summary of your architecture, but the actual code, the actual data pipelines, the actual monitoring setup. Because the real problems are never in the executive summary. They’re in the details.
A good consultancy will also be transparent about what they don’t know. They’ll ask questions instead of assuming. They’ll challenge your assumptions, but they’ll do it respectfully, because they know that you understand your business better than they do.
And they’ll write things down. Not a novel, but something concrete that you can act on after they leave.
Red Flag: They’re Selling You a Platform
I’ve seen this play out dozens of times. A consultancy comes in, does a quick assessment, and suddenly they’re recommending a platform they happen to have a partnership with.
Now, sometimes a platform is the right answer. But it’s rarely the first answer, and it’s almost never the only answer.
The problem is that platforms come with assumptions built in. They assume your data looks a certain way. They assume your compliance requirements fit their model. They assume your team structure matches their design. And when your situation doesn’t fit those assumptions, you either have to contort your business to fit the platform, or you end up with an expensive system that doesn’t quite work.
I’ve watched teams get locked into a vendor’s way of doing things, only to discover that their workflow doesn’t fit the platform’s assumptions. I’ve also seen teams buy expensive platforms and still end up with chaos because nobody actually implemented the features they paid for.
A consultancy that’s worth your time will do a real assessment before recommending anything. They’ll ask about your build-vs-buy decision. They’ll evaluate whether you should buy, build, or integrate multiple tools. And they’ll be honest about the tradeoffs.
If they’re pushing a platform before they understand your situation, that’s a sign they’re optimizing for their commission, not your outcome.
The Diagnostic Sprint: A Way to Test the Waters
You don’t have to commit to a six-month engagement to find out if a consultancy is worth working with.
A good consultancy will offer a time-boxed diagnostic engagement. Two weeks. In that time, they audit what you have, talk to the people who understand it, identify the gaps between where you are and where you need to be, and build a business case for fixing them.
At the end, you have three things: a clear picture of what’s actually holding you back, a prioritized roadmap for fixing it, and a cost estimate so you know what you’re signing up for.
This is valuable for two reasons. First, you get real insight into your situation without committing to a long engagement. Second, you get to see how the consultancy actually works. Do they ask good questions? Do they talk to the right people? Do they deliver something actionable, or something vague?
If the diagnostic sprint is good, you know whether to move forward. If it’s not, you’ve only invested two weeks.
Who’s Actually Accountable?
I’ve seen consultancies that hand you a design and disappear. I’ve seen teams that hire consultancies to “help with implementation” and end up doing all the work themselves while the consultancy watches from the sidelines.
The consultancy you want is accountable end to end. That means the same team that designed your system is the same team that helped you deploy it and the same team that’s helping you operate it. Because if they designed something they didn’t have to live with, they might have missed the operational reality.
This matters because the gap between “looks good in a design document” and “actually works in production” is where most AI systems fail. The consultancy that’s accountable for both is motivated to bridge that gap.
It also matters for knowledge transfer. If one team designs and a different team implements, the knowledge doesn’t transfer cleanly. You end up with a system that nobody fully understands.
What Happens After They Leave?
The real test of a good consultancy is what your team can do after they’re gone.
A bad consultancy leaves you dependent on them. They’ve built something so complex that you can’t maintain it without calling them back. Or they’ve left you with so little documentation that you can’t change anything without breaking it.
A good consultancy leaves you with a system you understand, a team that can operate it, and a clear roadmap for what to do next. They’ve built things in a way that your team can maintain and extend. They’ve documented not just what they did, but why they did it that way.
Before you hire someone, ask them: “What does handoff look like? How do you make sure our team can operate this after you’re gone?” The answer tells you a lot about whether they’re thinking long-term or short-term.
The Questions to Ask
When you’re evaluating consultancies, here are the questions that matter:
On their approach: How do you measure success? Who do you talk to? What does your process look like? How long does a typical engagement take?
On their experience: What kinds of systems have you worked on? What problems do you see most often? What’s the most common mistake you see teams make?
On their team: Who would actually be working on my engagement? How much of their time? What’s their background?
On the work: What does the output look like? How much documentation do you provide? How do you handle knowledge transfer?
On the business side: How do you price engagements? Are you willing to scope work around results instead of hours? What happens if we need to adjust the plan?
If they’re evasive on any of these, that’s a signal.
FAQ
How long does a typical engagement take?
It depends on what you need. A diagnostic sprint is two weeks. A full systems architecture project might be two to four months. Model operations support can be ongoing. The key is that the timeline should be driven by your business needs, not by a standard package.
Should I hire a consultancy or hire full-time staff?
Both, probably. A consultancy is good for getting you unstuck quickly, setting up the right patterns, and making sure your team knows how to operate the system. Full-time staff is good for ongoing maintenance and evolution. The best approach is usually to use a consultancy to get you to a stable state, then hire staff to maintain it.
How do I know if they’re actually good, or just good at selling?
Ask for references from people in your industry who’ve worked with them. Ask what they did, what the outcome was, and whether they’d hire them again. And do a diagnostic sprint before committing to a big engagement. Two weeks will tell you a lot.
What if they recommend something I disagree with?
That’s actually a good sign. You want a consultancy that challenges your assumptions, not one that just tells you what you want to hear. The question is whether they can explain their reasoning clearly and whether they’re willing to listen if you have a different perspective.
The Bottom Line
Choosing the right consultancy is about finding someone who understands that AI systems are business systems first and technology systems second. Someone who’s willing to do the unglamorous work of understanding your actual situation before recommending a solution. Someone who’s accountable for results, not just for effort.
It’s also about finding someone who’s been doing this long enough to recognize the patterns. I’ve watched teams spend months building beautiful data pipelines, then discover the data changed. I’ve seen vector databases deployed without any thought to data freshness, permissions, or accuracy. I’ve watched RAG systems generate confident answers from retrieved documents that were incomplete or outdated.
I’ve also seen teams get this right. They start with their business outcome, work backwards to figure out what the system needs to do, and then build the infrastructure to support that. They move faster than teams with bigger budgets because they’re not optimizing for the wrong thing.
If you’re looking for a consultancy that works that way, let’s talk. We’ll start with a conversation about what you’re actually trying to solve. No pressure, no sales pitch. Just a straightforward discussion about whether we can help.
If we can, we’ll propose a diagnostic sprint. Two weeks, and you’ll have a clear picture of what you have, what you need, and what it’s going to take to get there.
If we can’t, we’ll tell you that too.