
The Question Everyone Gets Wrong
You’re sitting across from an AI consultant. Their LinkedIn is impressive. They’ve worked with Fortune 500 companies. They speak confidently about vector databases, fine-tuning, and RAG systems.
Then you ask them about your production system, and something shifts. The answers become vague. They talk around the question. They pivot to what they’d do differently, rather than what they’d actually do with what you have.
I’ve watched this happen dozens of times, and it’s the moment I know whether someone has spent real time in production or just in the sales cycle.
The problem is that AI consulting has become a credential sport. People list certifications, past clients, and technical knowledge. But credentials don’t tell you whether someone has actually shipped a system that stayed running when the business depended on it. They don’t tell you whether someone knows what happens at 2 AM when your model starts hallucinating answers that look correct but aren’t.
Hiring the wrong consultant costs you months. Hiring the right one saves you from mistakes that would have cost you years.
Here’s what actually matters when you’re evaluating an AI consultant.
Ask About Their Last Production Failure
This is the question that separates the real from the performative.
Ask: “Tell me about a production system that broke. What went wrong? How did you find it? What did you do?”
Listen carefully to the answer. Someone with real production experience will give you specifics. They’ll talk about the actual symptom they saw first. They’ll describe the debugging process. They’ll explain what they learned that they now do differently on every project.
Someone without production experience will give you a generic answer. They’ll talk about best practices. They’ll describe a problem in abstract terms. They might even describe someone else’s failure as if it were their own.
The real answer sounds like a story. It has texture. It has the kind of detail that only comes from actually living through it.
I’ve watched teams deploy vector databases without any thought to data freshness, permissions, or accuracy. I’ve seen retrieval systems generate confident answers from documents that were weeks out of date, and nobody noticed until a customer asked why the system was giving them outdated information. I’ve seen systems where the same question returned different answers depending on which server handled the request. These aren’t theoretical problems. They’re the kind of thing that happens when you’re running a real system at scale.
If a consultant can’t tell you a specific story about something that broke and how they fixed it, that’s a red flag.
Ask About Their Worst Architectural Decision
Every system has one. A choice that seemed right at the time but created problems down the line.
Ask: “What’s the worst architectural decision you’ve seen a team make? How would you have done it differently?”
Again, listen for specifics. Someone with real experience will describe a real tradeoff. They’ll explain why the decision made sense at the time. They’ll talk about what changed that made it a problem. They’ll describe how they’d approach it differently now.
This matters because it tells you whether the consultant understands that every system is a series of tradeoffs. There’s no perfect architecture. There’s only the architecture that works for your constraints, your team, and your business outcome.
I’ve watched teams spend months evaluating data center options, comparing latency profiles and availability zones, only to discover that their actual bottleneck was somewhere completely different. I’ve seen teams buy expensive platforms and still end up with chaos. I’ve also seen teams build lean, focused stacks that run like clockwork.
The consultant who understands tradeoffs is the one who’ll help you make the right choice for your situation, not the choice that looks best on paper.
Ask Them to Audit Your System in Real Time
Don’t hire a consultant based on a conversation. Ask them to spend a few hours actually looking at what you have.
Here’s what matters: can they ask good questions? Do they talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome? Do they understand that the system you have is the system that actually matters, not the system you wish you had?
I’ve seen consultants walk into a room and immediately start talking about what they’d build from scratch. I’ve seen other consultants spend time understanding what you actually have, what’s working, what’s broken, and what matters to your business.
The second kind of consultant is the one worth paying for.
A good consultant will ask questions like: Can you trace where the data came from? Who has permission to change it? When was it last updated? How do you know if your retrieval system is giving you accurate answers? If something goes wrong, can you show me exactly what happened?
These aren’t theoretical questions. They’re the questions that reveal whether your system is actually ready for production, or whether it’s one bad day away from a crisis.
Ask About Their Team Structure
One person who understands everything is a disaster waiting to happen. But a team where nobody understands the whole picture is worse.
Ask: “How do you structure your team? Who’s accountable for what? What happens when someone leaves?”
The right answer involves one team that’s accountable from architecture through production operations end to end. It involves clear ownership, clear handoffs, and documentation that actually connects the pieces.
I’ve seen teams where one person understands the entire system, and that person leaving the company would paralyse the operation. I’ve seen other teams where nobody understands the whole picture, and every problem becomes a mystery that takes weeks to solve. I’ve seen teams where the model got better, but the business outcome got worse, because the people building the model weren’t talking to the people who owned the business outcome.
The consultant who understands team structure is the one who’ll help you build a system that doesn’t depend on heroics.
Ask How They Measure Success
This is where a lot of consultants get uncomfortable.
Ask: “How do you know if the engagement was successful? What metrics matter? What would success actually look like for my business?”
Listen for whether they’re talking about technology metrics or business metrics. Are they talking about model accuracy? Or are they talking about the business outcome that accuracy is supposed to drive?
I’ve seen teams celebrate a 15% improvement in model accuracy, only to discover that the business outcome didn’t improve at all. I’ve seen teams with perfect model metrics and terrible business outcomes because they were optimizing for the wrong thing.
The consultant who starts with your business outcome and works backwards to figure out what the system needs to do is the one who’ll actually move the needle.
Ask About Governance and Compliance
This is where I’ve seen the most creative chaos.
Ask: “How do you think about access control? How do you build audit trails? How do you know that your system is actually compliant with what your board cares about?”
I’ve seen teams where anyone with database access could retrain the model, which meant that a data analyst could accidentally change the model’s behaviour without anyone knowing. I’ve seen teams where only one person could deploy models, which meant that a single person leaving the company could paralyse the entire operation. I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks.
I’ve also seen teams where the audit trail is so sparse that you can’t figure out what happened when something goes wrong. And I’ve seen audit logs that are so detailed that they’re useless.
The consultant who understands governance is the one who can help you build something that’s actually compliant, actually auditable, and actually usable.
Ask What They’d Do in Your First Two Weeks
This is the practical question that reveals whether someone is thinking about your actual situation.
Ask: “If I hired you tomorrow, what would you do in the first two weeks? What would you need to understand? Who would you talk to? What would you be trying to figure out?”
The answer should involve talking to people, understanding what you actually have, and building a clear picture of what needs to happen next. It should not involve immediately proposing solutions or starting to build things.
I’ve watched this play out dozens of times, and the teams that move fastest are not the ones that jump straight into building. They’re the ones that start with a clear understanding of what they’re actually trying to solve.
What to Watch Out For
There are a few things that should make you pause.
First: if a consultant can’t explain their thinking in plain language, they probably don’t understand it well enough. Complexity is easy. Clarity is hard.
Second: if a consultant is selling you a specific technology before they understand your problem, they’re selling you the technology, not solving your problem. I’ve seen teams get locked into a vendor’s way of doing things, only to discover that their workflow doesn’t fit the platform’s assumptions. I’ve also seen teams where one person understands the entire system, and that’s a disaster waiting to happen.
Third: if a consultant is talking about billable hours instead of results, they’re not aligned with your success. The consultant you want is the one who cares whether you actually ship something that works, not whether they can keep billing you for the next quarter.
Fourth: if a consultant hasn’t actually spent time in production, they can’t help you with production. They can help you build a demo. They can help you understand the theory. But when something breaks at 2 AM, you need someone who’s been there before.
The Diagnostic Sprint: A Way to Test Drive
If you’re not sure whether a consultant is the right fit, ask about a short engagement. We offer a Diagnostic Sprint: two weeks to a clear picture of what you have, what you need, and what the business case looks like.
This is a way to work together without a massive commitment. You get to see how they think. They get to understand your situation. By the end, you both know whether it’s worth going deeper.
FAQ
Q: Should I prioritize consultants with specific certifications?
Certifications tell you that someone passed a test. Production experience tells you that someone has actually solved problems. I’d take production experience over certifications every time.
Q: How do I know if a consultant is overselling their past work?
Ask for specifics. Ask them to describe their role in a project, not just the outcome. Ask them what they’d do differently if they did it again. The consultant who can be honest about what worked and what didn’t is the one telling you the truth.
Q: What if I don’t have a production system yet?
Then you want a consultant who understands what production actually requires, and who can help you build it right the first time instead of having to rebuild it later. The questions are the same. You’re just asking them about systems they’ve helped other teams build.
Q: How long should I expect a consulting engagement to take?
That depends on what you’re trying to solve. A short audit might take two weeks. A full system redesign might take months. The consultant who scopes the engagement around results rather than billable hours is the one you want.
Q: Should I hire a consultant for the whole project or just the architecture phase?
The best consultant is the one who stays accountable from architecture through production operations. I’ve seen too many projects where the consultant designed something beautiful, then handed it off to the team to implement, and the team discovered problems that the consultant never had to deal with. The consultant who stays involved through production is the one who’ll actually make sure it works.
Get in Touch
If you’re evaluating consultants and you want to talk through your situation, reach out. We can spend a few hours understanding what you have, what you’re trying to solve, and what it would actually take to get there. No pitch. No sales process. Just a conversation with someone who’s spent a lot of time in production and knows what questions to ask.
Email us, and we’ll get back to you within a business day.