The Question I Hear Every October
It’s mid-Q4, and suddenly your board is asking about compliance. Or your legal team surfaces a regulation you didn’t know applied to you. Or a customer asks for proof that you’re actually following the security practices you promised.
Then you realize: nobody has a clear answer.
I’ve watched this play out dozens of times. Teams have built impressive AI systems. They’re running in production. They’re delivering business value. But when someone asks, ‘Can you show me your access control logs? Who can retrain the model? What happens if the data source changes?’ the answers get fuzzy fast.
This is the compliance crunch. And Q4 is when it hits hardest.
Why Q4 Is Different
End of year brings three things together at once.
First, budget cycles. Finance wants to understand what they’re actually paying for, and what the risks are. That means someone has to audit the system.
Second, regulatory pressure. New rules land. Existing rules get enforced. Your industry might have published new guidance. Your customers might have new requirements. The pressure builds through the year and peaks in Q4.
Third, time. You’ve got a window between now and the new year to fix things before the next cycle starts. If you wait until January, you’re already behind.
The teams that move now have a huge advantage. They can identify the gaps while they still have budget and headroom to fix them. They can walk into 2025 with a clear picture of what they have, what they’re missing, and what it costs to close the gap.
What ‘Compliance’ Actually Means for Your AI System
Compliance isn’t a single thing. It’s not a checkbox. It’s a set of requirements that vary depending on your industry, your customers, and what your system actually does.
But I’ve seen enough systems to know where the gaps usually hide.
Access control is the first one. I’ve seen teams where anyone with database access could retrain the model, which meant that a data analyst could accidentally (or intentionally) change the model’s behaviour without anyone knowing. I’ve seen other teams where only one person could deploy models, which meant that a single person leaving the company could paralyse the entire operation. Both extremes are problems.
Audit trails are the second. I’ve seen systems where the audit trail is so sparse that you can’t figure out what happened when something goes wrong. I’ve also seen audit logs that are so detailed that they’re useless. Neither helps when a regulator asks, ‘Who changed what, and when?’
Data freshness and source tracking is the third. I’ve seen vector databases where nobody could explain how documents got into them. I’ve seen retrieval systems where the source data was weeks out of date. I’ve seen systems where you couldn’t trace a single answer back to its original source. That’s a compliance nightmare.
Monitoring and alerting is the fourth. This sounds obvious, but I’ve seen teams with sophisticated monitoring systems where nobody actually knows what the alerts mean. I’ve seen teams with hundreds of alerts and alert fatigue so bad that they ignore all of them. When something goes wrong, nobody notices until a customer complains.
Documentation and runbooks is the fifth. I’ve seen teams where one person understands the entire system, and that’s a disaster waiting to happen. But I’ve also seen teams where nobody understands the whole picture, and that’s worse. Compliance requires someone to be able to explain how the system works, what it’s supposed to do, and what happens when it breaks.
The Diagnostic Sprint: What We Actually Do
When teams come to us in Q4, we typically start with a Diagnostic Sprint. Two weeks. We talk to the people who built the system, the people who maintain it, the people who depend on it, and the people who own the business outcome.
We audit what you have. We identify the gaps between what you’re doing and what compliance actually requires. We build a roadmap to close them.
Here’s what that looks like in practice.
We start by mapping your system. Not the architecture diagram on the wall. The actual system as it runs today. Who has access to what? How does data flow from source to model to output? What happens when something breaks? What alerts fire, and who responds?
Then we compare that to your compliance requirements. This is where the gaps show up. You might have access control, but it doesn’t cover all the places where someone can change the model’s behaviour. You might have audit logs, but they don’t capture the things that actually matter. You might have monitoring, but it’s monitoring the wrong metrics.
Finally, we build a prioritized plan. Not everything needs to happen at once. Some things are quick wins. Some things are structural and take longer. We help you understand the tradeoffs, the costs, and the business case for each one.
At the end of two weeks, you have three things: a clear picture of what you have, a clear picture of what you’re missing, and a cost and risk exposure report that you can take to your board.
Where I’ve Seen the Most Creative Chaos
Let me walk through the places where things usually break down.
Access Control That Sounds Good But Doesn’t Work
I’ve seen teams paralysed by the complexity of access control itself, where every change requires five approvals and takes three weeks. I’ve also seen teams where nobody wants to use the system, so they just give everyone full access and hope for the best.
The right answer is usually somewhere in the middle. You need enough control to prevent accidents and catch intentional misuse. You need enough simplicity that people actually use the system instead of working around it.
In my experience, teams usually end up with a tiered approach, and it’s the right one. Different people need different levels of access depending on their role. A data analyst might be able to read logs and query the database, but not retrain the model. An ML engineer might be able to retrain the model, but only after passing a code review. A compliance officer might be able to audit everything, but not change anything.
Audit Trails That Nobody Can Actually Use
Here’s where I’ve seen the most creative chaos: the audit trail. I’ve seen systems where the approval workflow is so complicated that people just approve everything without reading it. I’ve also seen the opposite: audit logs so verbose that they’re useless.
The thing is, you’re not auditing for yourself. You’re auditing for a regulator, or a customer, or your board. They need to be able to look at the log and answer a specific question: ‘Who changed this, when, and why?’ If the log is too sparse, you can’t answer the question. If it’s too verbose, you can’t find the answer.
Here’s what I’ve seen work: start with the questions you need to answer. Then build the audit trail to answer those questions. Don’t try to log everything. Log the things that matter.
Data Freshness and Source Tracking
I’ve seen vector databases deployed without any thought to data freshness, permissions, or accuracy. I’ve seen retrieval systems where the documents were weeks out of date, and nobody noticed until a customer asked why the system was giving them outdated information.
This matters because compliance often requires you to know where your data came from, who has permission to change it, and when it was last updated. If you can’t answer those questions, you’ve got a problem.
Here’s what I’ve seen work: start with your data sources. Know where they are. Know who owns them. Know how often they change. Then build your retrieval infrastructure around that. If your data source changes daily, your retrieval system needs to refresh daily. If it changes monthly, you can refresh monthly. But you need to know, and you need to be able to prove it.
Monitoring That Actually Matters
I’ve seen systems where the same question returned different answers depending on which server handled the request. I’ve seen systems that worked fine with 100 requests a day completely collapse at 1,000. I’ve seen systems where the model got better, but the business outcome got worse.
Monitoring isn’t about collecting data. It’s about knowing when something is wrong and being able to respond. I’ve watched teams build sophisticated monitoring systems that nobody actually used. The alerts fire, but nobody knows what they mean, so nobody responds.
Here’s what I’ve seen work: start with your business outcomes. What matters? Revenue? Customer satisfaction? Accuracy? Then work backwards to figure out what you need to monitor. If accuracy matters, monitor accuracy. If latency matters, monitor latency. Then set up alerts that people actually respond to. Not hundreds of alerts. A few alerts that mean something.
Documentation and Runbooks
I’ve seen teams where one person understands the entire system. I’ve seen teams where nobody understands the whole picture. Both are disasters.
Compliance requires documentation. Not because regulators love paperwork. Because when something goes wrong, someone needs to be able to figure out what happened and how to fix it. If that information only exists in one person’s head, you’re in trouble.
Here’s what I’ve seen work: document the critical paths. How does data flow from source to model to output? What happens when each step breaks? Who do you call? What’s the runbook? You don’t need to document everything. You need to document the things that matter for compliance, reliability, and incident response.
Why This Matters Right Now
If you wait until January, you’re starting the year behind. Budget is tight. Headroom is limited. You’re already planning for the next cycle.
If you move now, you can:
Identify the gaps while you still have budget to fix them. Understand the cost and risk of each gap. Walk into 2025 with a clear plan. Start the year ahead instead of behind.
Plus, you’ve got time to think about this without the pressure of a regulatory deadline or a customer audit breathing down your neck.
How We Approach This
We work differently from most consultancies.
We don’t come in with a template and fill in your details. We talk to your people. We understand your system as it actually runs, not as it’s supposed to run. We understand your business outcomes. We understand your constraints.
Then we build a plan that actually works for you, not a generic compliance checklist that looks good on paper but doesn’t fit your reality.
We focus on business outcomes measurable on your P&L, not on technology for its own sake. We scope engagements around results, not billable hours. We put one team accountable from architecture through production operations end to end.
We’ve seen these patterns play out enough times to know what questions to ask and where the problems usually hide.
Getting Started
If you want to understand what you have, what you’re missing, and what it costs to close the gap, we can help. A Diagnostic Sprint takes two weeks. You get a full system audit, a cost and risk exposure report, and a prioritized build plan.
That’s enough to take to your board, your legal team, or your customers. It’s enough to start 2025 with a clear picture of where you stand.
FAQ
What if we’ve already done a compliance audit?
Most compliance audits focus on policy and process. They don’t dig into the actual system. We focus on the system itself. What’s actually running? Who can actually change what? What actually happens when something breaks? That’s usually where the gaps are.
How long does a Diagnostic Sprint take?
Two weeks. We spend time with your team, audit the system, and build the plan. At the end, you have a clear picture and a roadmap.
What if we don’t have time for a two-week engagement?
We can work with your schedule. A Diagnostic Sprint is the fastest way to get a complete picture, but we can also work in shorter sprints or focus on specific areas if that’s what you need.
What happens after the Diagnostic Sprint?
That’s up to you. You have a roadmap. You can execute it yourself. You can ask us to help. You can prioritize and tackle it over time. We’re here to help however makes sense for your situation.
Do we need to have everything documented already?
No. Most teams don’t. That’s part of what we do. We talk to your people, we look at your systems, we figure out what’s actually happening. Documentation comes later.
Let’s Talk
If you’re feeling the Q4 compliance crunch, you’re not alone. But you don’t have to figure it out alone either.
We’ve worked with teams in your situation. We know what to look for. We know how to turn a compliance audit into a business case. We know how to help you move faster.
Get in touch. Let’s talk about what you have, what you need, and what the path forward looks like.