The Infrastructure Question Nobody’s Asking

Every week, I talk to teams wrestling with the same underlying worry: “Are we building on the right infrastructure? Should we be planning for massive scale? Are the big cloud providers going to lock us in?”

Meanwhile, the headlines are full of data center announcements. OpenAI is rebranding its data centers as “AI compute campuses,” and major consulting firms are racing to build out infrastructure advisory practices. It’s easy to feel like infrastructure is the bottleneck.

Here’s what I’ve learned from working through dozens of AI projects: infrastructure is almost never your real problem. Not yet, anyway.

The problem is almost always upstream. It’s your data. It’s your retrieval layer. It’s knowing what you’re actually trying to measure. It’s access control so tangled that nobody can move. It’s audit trails so sparse you can’t figure out what happened when something breaks.

I’m going to walk you through what actually matters when you’re thinking about AI infrastructure, and why the conversation most teams should be having is very different from the one they’re having.

Why Data Center Scale Feels Urgent (But Isn’t)

When you read that OpenAI is expanding its compute footprint, or that major infrastructure players are consolidating, it creates a sense of urgency. Shouldn’t you be planning for that? Shouldn’t you be locking in capacity?

The answer, in most cases, is no.

I’ve watched teams spend months evaluating data center options, comparing latency profiles and availability zones, only to discover that their actual bottleneck is somewhere completely different. They’re not hitting compute limits. They’re hitting data freshness problems. They’re hitting retrieval accuracy problems. They’re hitting governance problems that make it impossible to retrain or redeploy anything.

The teams that are actually hitting real infrastructure constraints? They’re the ones with proven, production systems that are already handling serious volume. And they’re usually the ones who figured out their data and retrieval layer first.

Start there. Not with the data center.

The Real Bottleneck: Your Retrieval Layer

This is where I’ve seen the most creative chaos.

I’ve seen vector databases deployed without any thought to data freshness, permissions, or accuracy. I’ve seen retrieval systems where the documents were weeks out of date, and nobody noticed until a customer asked why the system was giving them outdated information. I’ve seen systems where the same question returned different answers depending on which server handled the request.

I’ve seen teams with sophisticated infrastructure and terrible retrieval quality. They had the compute. They didn’t have the data pipeline.

Here’s what I see when teams get this right: they start with their business outcome. What question are you actually trying to answer? Where does the source data live? How often does it change? Who has permission to change it? How do you know if your retrieval is working?

Then they build backwards to the infrastructure that supports that.

I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks. I’ve seen other teams where only one person could deploy models, which meant that a single person leaving the company could paralyse the entire operation. I’ve seen teams where anyone with database access could retrain the model, which meant that a data analyst could accidentally (or intentionally) change the model’s behaviour without anyone knowing.

None of that is an infrastructure problem. It’s a retrieval and governance problem. And it’s what’s actually stopping you from moving fast.

What to Actually Evaluate in Your Infrastructure Decisions

When you are thinking about infrastructure – and eventually you will need to – here’s what matters.

Can you trace your data?

I’ve seen vector databases where nobody could explain how documents got into them. Can you trace where the data came from? Do you know when it was last updated? Do you know who changed it? If you can’t answer those questions, your infrastructure choice doesn’t matter. You’ve already lost.

Can you monitor what actually matters?

This sounds obvious, but I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean. You need to know: Is your retrieval accuracy holding up? Is your data getting stale? Are your costs going up? Are your latencies creeping up? Are your failure rates increasing?

Not because the infrastructure vendor’s dashboard told you so. Because you measured it against your business outcome.

Can you change it without breaking everything?

I’ve seen systems where the same question returned different answers depending on which server handled the request. I’ve seen systems that worked fine with 100 requests a day completely collapse at 1,000. I’ve seen systems where nobody understood how the deployment pipeline actually worked.

You need infrastructure that you can actually understand, test, and change. That usually means less complexity, not more.

The Build vs. Buy Question (And Why It’s Not What You Think)

One of the questions we get asked most often is: should we build our own infrastructure, or buy from a vendor?

Here’s what I’ve learned: the answer depends on what you’re actually trying to do. If you’re trying to prove that AI can solve a business problem, you should buy. Use managed services. Get to production fast. Learn what actually matters.

If you’re trying to build something that needs to scale to serious volume, with serious reliability requirements, and with serious governance constraints – then you might need to build. But not until you’ve already proven the business case.

I’ve watched teams spend months building beautiful data pipelines, then discover the data changed. I’ve watched teams build sophisticated monitoring systems that nobody actually used. I’ve watched teams build access control frameworks that were so complicated that they just gave everyone full access and hoped for the best.

Start with buy. Move to build only when you have a clear reason.

Governance and Audit: The Infrastructure Most Teams Ignore

Here’s where I’ve seen the most creative chaos: the audit trail.

I’ve seen systems where nobody could trace where the data came from, or who had permission to change it, or when it was last updated. I’ve seen audit logs that are so sparse that you can’t figure out what happened when something goes wrong. I’ve seen audit logs that are so detailed that they’re useless.

I’ve seen teams paralysed by the complexity of access control itself, where the system is so complicated that nobody wants to use it, so they just give everyone full access and hope for the best. And I’ve seen access control so complicated that teams needed five approvals and three weeks just to give someone permission to read a log file.

This is infrastructure. It’s not glamorous. But it’s what determines whether you can actually operate at scale, whether you can satisfy compliance requirements, and whether you can move fast without breaking things.

What’s Changed in the Industry (And What Hasn’t)

TechCrunch recently reported that OpenAI is considering rebranding its data centers as “AI compute campuses,” signalling a shift in how the industry thinks about infrastructure as a strategic asset. At the same time, major consulting and real estate firms are building out dedicated AI infrastructure advisory practices.

What this tells me is that infrastructure is becoming a commodity. The big players are competing on scale, on reliability, on geographic distribution. That’s good for you. It means you have options. It means you don’t need to build your own.

What hasn’t changed is the fact that infrastructure is not your constraint. Your constraint is knowing what to build, how to measure it, and how to operate it reliably.

Where to Start: The Diagnostic Sprint

If you’re sitting with an AI system that’s not quite working the way you hoped, or you’re trying to figure out what to build next, here’s what I recommend.

We offer a Diagnostic Sprint – two weeks to a clear picture of what you have, what you need, and what the business case looks like. We talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome. We audit what you have, identify the gaps between what you’re doing and what actually matters, and build a roadmap to close them.

We look at your data. We look at your retrieval layer. We look at your governance and access control. We look at your monitoring and alerting. We look at your deployment pipeline. We look at your costs. We look at your risks.

Then we tell you, in plain language, what’s working, what’s not, and what to fix first.

Most of the time, the answer is not “build a bigger data center.”

FAQ

Should we be worried about vendor lock-in with cloud infrastructure?

Not yet. Lock-in happens when you’ve built something so specific to one vendor’s service that it’s expensive to move. That usually doesn’t happen until you’re at serious scale. Until then, focus on building something that works. You can always move it later.

Do we need to plan for massive scale right now?

No. Plan for the scale you need today, plus maybe 10x. If you’re already at 10x, then you can start thinking about the next tier. Most teams are not there yet.

What’s the difference between a data center and an “AI compute campus”?

Marketingly, it’s a rebrand. Functionally, it’s the same thing – a facility with power, cooling, and networking infrastructure optimized for compute-heavy workloads. The terminology shift reflects how the industry is positioning AI infrastructure as a strategic asset. For you, it doesn’t change anything.

How do we know if our infrastructure is the problem?

Ask yourself: Can I deploy a change in less than a day? Can I roll back if something breaks? Do I know what my costs are? Do I know if my system is getting slower? Do I know if my data is getting stale? If you can’t answer those questions, your problem is not infrastructure. It’s observability and governance.

Let’s Talk About What You Actually Need

I’ve watched this play out dozens of times, and the teams that move fastest are not the ones with the biggest data centers. They’re the ones with the clearest picture of what they’re trying to do, the most reliable data pipelines, and the most straightforward governance model.

If you’re thinking about AI infrastructure – whether that’s your first system or your fifth – let’s talk. We can audit what you have, identify what’s actually holding you back, and build a plan that gets you moving.

Reach out. Let’s figure out what you actually need.

This article was prompted by news from techcrunch.com: OpenAI might rebrand its data centers as ‘AI compute campuses’.