One does not simply use AI under the gaze of the FDA
The bots are scurrying and the eye is watching.
Howdy friends,
A few weeks ago I wrote an issue called "Blood in the Water," right after the first FDA warning letter citing inappropriate use of generative AI bubbled to the surface of all our feeds. My prediction at the time was simple: every talking head on LinkedIn was about to crown themselves an AI expert or start wagging a finger going "see, this is where AI gets you." And here we are. The feeding frenzy played out more or less exactly like that, and you guess it, I waded in too.
Now that the water's calmed down a bit, I want to discuss it a bit more, because I think the most interesting thing about that letter isn't the part everyone fixated on.
The AI angle got all the eyeballs. Three short paragraphs about a manufacturer using AI agents to write compliance documents with no human checking the output, and someone telling investigators they didn't know a requirement existed because the AI never mentioned it. That's a genuinely useful cautionary tale, and if you're using ChatGPT like it's Google and passing the answers around your team unvetted, please take it as your wakeup call. But I'd bet good money that company wasn't special. If one firm got caught doing this, plenty of others are doing the same thing right now and simply haven't had FDA in the building for unrelated reasons. The letter wasn’t exposing a one-off but rather putting a common habit on the record for the first time.
What I want everyone to pay attention to is something else. FDA is rolling out AI to expand its own reach, and I think that's the real reason warning letters are climbing.
The agency has always been resource-constrained. Bigger fish to fry, not enough hours in the day, and a whole tail of smaller operations that stayed under the radar simply because nobody had time to look. That is all changing. FDA jumped to 303 drug warning letters in FY25, up roughly 59% from the year before, and the agency's own throughput tools are part of why they can suddenly cover more ground. The companies that have been quietly flying under the radar for years are about to find out the radar got an upgrade.
There’s a lot about this I don’t know, and invite anyone with insider knowledge to share. I have no idea how well FDA's staff is trained on these tools, or how this plays out over the next couple of years. There's a real irony sitting in plain sight here, which is that the same overreliance risk that chomped a manufacturer in the behind could just as easily show up in an agency racing to deploy. I'm not so much predicting that, I'm just noting that nobody is exempt from the thing the letter is actually about.
Which is the whole point, and it cuts the same way for all of us. AI doesn't fix a shaky process, it scales whatever you already have. If your underlying work is solid, AI makes you faster. If it's held together with duct tape, AI just produces more duct tape, faster and in cleaner formatting.
This is exactly why the rush to drop AI into deviation and CAPA writing makes me a little nervous. On paper it's the perfect use case, since investigations are slow, repetitive, and inconsistent across sites, and a model can crank out a tidy report in minutes. But a lot of what ails deviation management isn't typing speed, it's the depth of the root cause analysis and the judgment behind it. Point a drafting tool at a shallow investigation process and you don't get fewer recurring deviations, you get more of them, written way more persuasively. Now an auditor is looking at a pattern of weak CAPAs in beautiful prose, which is arguably a worse place to be than where you started. The tool is only as good as the framework you put around it before you turn it on.
So here's where I land, and it's not the hottest take in your inbox this week. Be deliberate. Map the process before you automate it and train everyone on AI literacy. But deliberate doesn't mean dragging your feet, and I do want a little pep in everyone's step here, because this stuff is useful when it's done with some care.
I know that runs against the marketing drumbeat and the breathless press releases, and I get why they sound the way they do. We're all running businesses, those businesses need to make money, and right now the money is in AI. I'm not above any of that. I'd just rather you move thoughtfully and keep your name off next year's warning letters than move fast and end up as someone else's LinkedIn cautionary tale.
Okay, enough from me. Thanks for reading!
Alexa
News
When everyone can build an agent, everyone does
What's New
The Wall Street Journal ran a piece this month on a problem that's surfacing inside companies that went all-in on AI agents: they now have too many of them. Reporting from Isabelle Bousquette describes firms like Lyft, DaVita, GitLab, and FICO trying to rein in a flood of redundant bots without killing the enthusiasm that created them. The industry has a name for it already: "agent sprawl." One detail that says it all is that the parent company of Ben & Jerry's, Magnum Ice Cream, is among the businesses wrestling with it, so this clearly isn't just a Silicon Valley problem.
How It Works
The root cause is how easy agent creation has become. Low-code and no-code platforms, including Anthropic's Claude Cowork, mean a non-technical employee can spin up an independent agent in an afternoon, so they do. Sales builds one, marketing builds one, support builds five, and nobody checks whether half of them already exist somewhere else in the building. The result is duplication, ballooning compute bills, and a real security headache, since each ungoverned agent is another thing with access to data and the ability to act on it. Gartner put some numbers around the trajectory, projecting that the average Fortune 500 company will run more than 150,000 agents by 2028, up from fewer than 15 in 2025, while only about 13% of organizations feel they have governance solid enough to handle it.
Why It Matters
We've watched this movie before. It's the same pattern as the SaaS sprawl that buried IT departments a decade ago (and is still happeing), when every team bought its own software and nobody had a master list. Agents raise the stakes because they don't just sit there storing data, they take actions, and an agent that was never properly permissioned can reach into systems it shouldn't or produce a slightly different answer than the four other agents doing the same job. For anyone in a regulated space, that last part should set off an alarm. An untracked agent making determinations inside a quality or manufacturing workflow is exactly the kind of thing that turns into an audit finding nobody can explain after the fact.
My Take
Look past the scary headline on this one. A swarm of redundant bots sounds like a disaster, but a lot of it is just what learning looks like when a powerful new tool gets cheap and easy, and honestly, we could all stand to crack a few eggs and figure out what these things are actually good for. I'd rather see a team experimenting and making a mess than sitting frozen waiting for permission. The caveat is about where you're cracking those eggs. Experimenting on your marketing copy is one thing, and experimenting in a GxP environment with no oversight is a different animal entirely, because there the mess has your name on a record an inspector can pull. Play freely in the low-stakes sandboxes, and build the guardrails before you let anything loose anywhere it touches compliance.
The quality system nobody uses (and the AI we want to build on it)
What's New
A survey out this month from MasterControl put a number on something a lot of us have felt in our bones for years. Polling 300 quality and manufacturing leaders across life sciences, they found that 94% of pharma quality leaders call poor employee adoption of their quality system a significant pain point. Biotech came in a little lower at 83%, but the takeaway is the same across the board: the systems people are counting on to power their AI ambitions are systems their own staff already avoid using. The report's own conclusion is the part worth sitting with, which is that AI's success in pharma won't come down to the technology so much as whether companies can close the human and system-level gaps that are already there.
How It Works
The logic is hard to argue with. An AI assistant pointed at your quality system is only as good as what's actually in that system, and if people are working around it, keeping records in spreadsheets, or quietly reverting to paper, then the data the AI feeds on is patchy from the start. Layer an agent on top of that and you haven't fixed the disengagement, you've just automated on top of it and added a confident-sounding narrator. The survey also surfaced where leaders expect the wins, with 44% of manufacturing leaders pointing to better traceability and compliance through automated data capture and 30% hoping AI will predict and prevent quality issues before they happen. Both of those depend entirely on people actually using the system in the first place.
Why It Matters
This is the whole theme of this issue with a data point attached. Everything we've talked about, from the warning letter to agent sprawl, comes back to the same idea, which is that these tools amplify whatever process you already have. A survey like this makes it concrete: if 94% of quality leaders can't get consistent engagement with the system today, then automating deviation handling or CAPA writing on top of that foundation isn't going to produce better investigations, just faster ones built on the same shaky inputs. There's even a few news terms making the rounds for the slow version of this failure, where teams gradually stop doing their own analysis because deferring to the AI is easier. The fix isn't a better model but getting people to trust and use the system before you ask a machine to run on it.
My Take
Two things can be true here. First, consider the source, because MasterControl sells quality and manufacturing software, the survey was commissioned through a research firm on their behalf, and the report lands neatly on the conclusion that you should build on a unified platform like, well, theirs. The sample is also a fairly modest 300 people, and the data was collected back in November, so read the number as a directional signal rather than gospel. But second, and this is why I'm including it anyway, it matches what I've seen with my own eyes. I've worked in quality systems where getting people to engage with the documentation was a genuine uphill battle, and it was almost always worst in the smaller organizations. The irony is brutal, because those are exactly the shops that would get the most lift from AI-assisted functions, and they're also the ones still running manual documentation, because automation costs money they'd rather not spend. So the companies that need the help most are the least set up to receive it, and a survey from a software vendor isn't going to change that math on its own. Worth knowing, worth a grain of salt.
Get the next edition in your inbox
Clear, practical takes on what matters for your CMC, QA, and regulatory work, once a week.
No spam. Unsubscribe anytime.
