The "illusion of control" (now in fresh AI scent!)

Howdy friends,

A few years back I did a quality audit for a device manufacturer, and I walked in pretty optimistic. The SOPs I’d reviewed ahead of time were good. Clear, well organized, and the kind of thing you’d be proud to sign off on.

About three-quarters of the way through the facility tour, it was pretty clear nobody there was following them. (I realized it when I found myself staring at backup generator able to supply 2 hours of emergency power when the SOP said 48. Hmm…)

As it turned out, a lot of that SOP language had been borrowed from another facility. If you’ve been in this industry for more than five minutes, you know this happens all the time. Someone leaves a company to start a new one, and wouldn’t you know it, they just happen to have a few old SOPs lying around. The words were great. They just weren’t theirs. Nobody had built the actual practice the documents described.

This can be called “the illusion of control”. Documentation is supposed to be evidence that the work is under control, and most of the time we treat it that way. It’s also exactly why inspectors don’t just read your binder and go home. They walk the floor.

The other version of this I’ve seen more times than I can count (and I’m not the only one, as basically every quality person I’ve talked to has a story) goes something like: “We just need to get this molecule to the clinic. We’ll deal with all the quality stuff later, closer to commercial.”

To be fair, phase-appropriate GMP is a real thing, and early programs should have lighter controls. The problem isn’t light controls. It’s controls that only exist on paper. Light-but-real scales up just fine. Thorough-looking-but-fake has to get torn down and rebuilt, usually right when a pre-approval inspection or a partner’s due diligence shows up and everyone suddenly cares a lot. By then the habits are years old and the records are already written.

So why am I bringing this up in an AI newsletter?

Because I’m watching the exact same thing happen with AI, just faster.

It’s very en vogue right now for commentators online to say AI is ruining our lives and quality processes because of hallucinations. And sure, if you’re using an old or free model, you’ll still see some. But with current tools, made-up facts aren’t the main way things go wrong anymore. What AI is really doing is amplifying whatever system it lands in. Strong systems get stronger, and weak ones get weaker, only now they’re producing ten times the documents and they all look great.

AI output is polished and confident, and guess what!?!? That’s the exact thing (even before AI was touching things) that made those borrowed SOPs look so good during the desk review. Polish makes gaps harder to spot, not easier.

And then there’s the industry’s favorite new phrase: “human in the loop.” It’s going into AI policies everywhere, and it sounds like a control. Someone signs off, box checked. But does that signature mean “I verified this” or “I looked at it”? Ask around and I bet you’ll find the answer depends heavily on who’s doing the reviewing and how close it is to 5pm on a Friday. In a lot of places, human in the loop is quietly turning into the new borrowed SOP: language that sounds like control, standing in for practice that was never really defined.

And it’s not just me side-eyeing this. In the EMA Annex 22 workshop report published earlier this month (more on that in the news below), regulators asked how you’d monitor the person doing the oversight, and the conclusion was basically that symbolic human-in-the-loop doesn’t cut it. Oversight has to be meaningful, and you have to be able to prove it.

Same goes for “adopt now, govern later,” which is just “we’ll deal with it closer to commercial”. Except AI habits form in weeks, not years, and the development reports, deviation summaries, and tech transfer docs your team is drafting with AI today become the history you’ll be defending later. You can’t go back and reconstruct who checked what.

Long-time readers might remember back in July when I said small pharma has a big AI advantage because there’s less to undo. I still think that’s true! But it’s only an advantage if you don’t build it wrong in the first place.

But it’s not all bad. If AI amplifies weak systems, it also shows you the weak spots sooner. Think of it as a stress test you get to run on yourself before somebody else runs it for you.

So my challenge for this week: go walk the floor on your own AI use. Pick one AI-assisted document your team produced recently and ask the person who reviewed it to show you exactly what they checked. Not what the procedure says they checked. What they actually did.

If the answer is great, congrats, you have a real control. If it’s a little fuzzy... well, better you find out now than in an inspection.

Thanks for reading!

-Alexa

News

EMA’s Annex 22 drafting group is rethinking its stance on generative AI

What’s New: In early October, EMA published the report from a 30 June workshop where the Annex 22 drafting group asked industry experts a pointed question: can a risk-based approach actually control generative AI, LLMs, and adaptive models in GMP? This matters because the current draft of Annex 22 excludes those models entirely and says they shouldn’t be used in critical GMP applications. Public consultation comments pushed back hard on that. The report says the drafting group is now discussing a scope that would allow them, if they’re backed by a fully documented, risk-based control strategy. Industry associations presented a consolidated position across six topics, and regulators asked questions.

How It Works:

• No bans by technology label. Industry argued that acceptability should depend on intended use, decision consequence, model influence, uncertainty, complexity, and detectability, not on whether the model is generative or adaptive. A valid risk assessment outcome can still be “don’t use it here,” as long as that conclusion is documented.

• Detectability gets a seat at the table. Regulators pointed out that with generative AI, not being able to see a failure may be the central risk. That pulls detectability back into the risk assessment, even though ICH Q9 defines risk mainly by severity and probability.

• Layered controls, also called the “Swiss cheese” model. Controls are stacked at the input (out-of-distribution checks, context-of-use verification), inside the model (retrieval with citation binding, uncertainty scoring, self-consistency checks), and at the output (confidence gates, trend monitoring, human gates, fallback to a validated state). The presenter’s caveat: controls built for known failure modes can’t be assumed to catch unknown ones.

• Human oversight is not the same as human in the loop. Oversight means a qualified person can understand, challenge, override, or escalate across the lifecycle. Human in the loop is one specific pattern where the workflow waits for a person. Regulators asked how you’d monitor the reviewer’s performance and manage automation bias. The answers included training, qualification, audit trails of AI outputs and human edits, and monitoring how the human and the AI perform together.

• Validation is a lifecycle, not an event. Test data must stay independent from training data, which gets tricky when models keep retraining. Monitoring, change control, CAPA, and supplier oversight are all part of validation.

• Your vendor’s model is still your problem. If a third-party API can’t give you the logs, outputs, or evidence you need to control it, you can’t justify using it. Annex 11 Chapter 7 (outsourced activities) was considered sufficient at the principle level, so Annex 22 will likely cross-reference it rather than duplicate it.

• Terminology debate. Participants suggested replacing “guardrails” with “control mechanisms” in the main annex and moving the fast-changing technical examples into appendices or Q&As that are easier to update.

Why It Matters: If the drafting group follows through, the biggest question for QA teams changes from “are we allowed to use an LLM here?” to “can we show an inspector a control strategy that actually works?” That’s a higher bar. Nearly every item in the report comes back to inspectable evidence: mapped risk scenarios, controls linked to specific failure modes, thresholds, monitoring history, and proof that the reviewers can actually challenge the output. It’s a familiar QMS toolkit (QRM, lifecycle validation, change control, supplier qualification), now aimed at a technology whose failures are hard to see. For anyone already using AI for deviation support, APR data pulls, or risk assessments, the report is a preview of the questions an inspector is likely to ask.

My Take: Vindication! Okay, I’m doing a small victory lap here, because I knew EMA would walk this back. When the draft excluded generative AI and LLMs from critical GMP applications, it never seemed like a position that would last. For one, it put Europe noticeably out of step with FDA, whose approach asks whether you can demonstrate credibility for a specific context of use, not which type of model you picked. For another... come on. People in GMP settings are already using these tools every day, approved or not. Putting your head in the sand doesn’t make that go away. It just means it happens without controls. A risk-based framework at least gives everyone something real to build on. To be fair, this is a workshop report and not a revised draft, so nothing is final yet. And the human oversight discussion makes it clear the bar is going up, not down. “A human reviews it” won’t be enough. You’ll need to show what that review actually catches. Still, I’ll take it.

Source: EMA: Report from the multistakeholder workshop on expert contributions to AI guidance development (Annex 22), workshop held 30 June 2026

An Annex 22 readiness checklist you can start before the ink dries (Candela Labs)

What’s New: Candela Labs, an AI consultancy, published a practical guide on 9 October on where EU GMP Annex 22 stands and what GMP sites can do now. Annex 22 is still a draft and not in force, more than a year after the July–October 2025 consultation, which drew over a thousand comments. Until something changes, the 2011 version of Annex 11 remains the binding text for computerised systems, including AI. The post walks through what the current draft actually requires, explains how that differs from ordinary software validation, and gives a readiness checklist that’s useful under Annex 11 today and likely required in some form once Annex 22 is final.

How It Works:

• What the current draft covers. Annex 22 applies to AI models used in critical GMP applications, meaning those with a direct impact on patient safety, product quality, or data integrity. As written, it only allows static models with deterministic outputs (same input, same output, no learning during use). Generative AI is allowed only in non-critical work, with a qualified person responsible for each output.

• The process SME is in charge. The draft makes the process subject matter expert, not the data scientist, responsible for the intended use and the acceptance criteria. Both are documented and approved before testing begins.

• The current process sets the bar. The model has to perform at least as well as the process it replaces. As the author points out, many sites have never actually measured how well their current manual process performs.

• The test set becomes a controlled GMP asset. It has to cover the full range of inputs, have verified labels, and stay strictly separate from training data, with access controls and an audit trail. Anyone who has seen the test data shouldn’t help train the same model unless they’re paired with a colleague who hasn’t (four-eyes principle). Using generative AI to create test data or labels is discouraged.

• Explainability and confidence. During testing, critical models should record which input features drove each decision and log a confidence score. Low-confidence results can be marked “undecided” and sent to a person instead of producing a shaky prediction.

• Validation continues after go-live. Configuration control, change control (including vendor model updates), and drift monitoring are all part of keeping the model in a validated state.

• The checklist groups the work into six areas: inventory and classification (including AI hidden inside vendor systems), intended use and acceptance criteria, test data, explainability and confidence, operation, and suppliers.

• On generative AI for 2026–2027: use it for non-critical work (SOP drafts, deviation summaries for a human investigator, document search) and treat the output as a draft. Keep it out of batch disposition, release testing, and anything that becomes a GMP record without human verification. Build review steps and full logging in now so you’re ready either way.

Why It Matters: Most of these items are things QA teams already understand from Annex 11 and GAMP 5. The new parts are what will catch people out. Statistical acceptance criteria replace pass/fail test cases. A test dataset becomes a GMP record. You need a performance baseline for a manual process nobody ever measured. And the first checklist item, a complete inventory of where AI is used, is harder than it sounds once you include AI built into vendor systems that nobody formally “adopted.” None of this happens overnight, and the post estimates months of work for most sites.

My Take: Paired with the EMA report, this is a clear signal to stop waiting for the final text. The inventory and the baseline measurements cost almost nothing to start, and you’ll need them no matter where Annex 22 ends up. What I like most is the point about the current process setting the bar. It’s a polite way of saying you can’t prove the AI is good enough if you never measured whether the humans were. That’s the illusion of control from this week’s note again: plenty of processes we assume are working have never been put to the test. Knowing your own baseline is useful well beyond AI.

Source: Candela Labs — EU GMP Annex 22 draft: an AI readiness checklist for GMP sites (October 9, 2026)

Generative AI declaration: Heck yeah I use gen AI to brainstorm, draft, and edit my newsletter sections (would you even trust me if I didn’t?). My stories and phenomenal sense of humor are my own.  However, if that bothers you, looks like you signed up for the wrong free newsletter and you are entitled to a full refund. -A