The pharma industry's new 'Children of the Magenta'

Howdy friends,

The more I talk to life science professionals about AI, the more curious I get about how other industries grew up after a big technological leap. AI is incredibly powerful and it has touched nearly everyone who works with a computer (plus a few industries you'd never expect to benefit from a computer brain), but this is not the first time a workforce has had to fumble around and learn from its mistakes after everything suddenly got better.

I recently watched a training lecture from the American Airlines Flight Academy, given in the '90s by Captain Warren Vanderburgh. He was speaking at a time when pilot dependence on the fully automated flight management system (FMS) kept showing up as either the cause of or a contributing factor in commercial flight mishaps, the hangover from the digitization push of the '70s and '80s. American had actually tasked him with figuring out what was going on, and when he and his team went through the accidents, incidents, and violations, they found that 68% of them came back to automation being mismanaged rather than automation failing. He coined the phrase "children of the magenta" for the pilots trained to follow the pre-programmed flight path drawn as a magenta line across the display instead of flying the plane themselves.

The point that really stood out to me was that over 15 or 20 years, pilot training had shifted from teaching people to fly the plane with their hands to teaching them to fly it with the computer. So in an emergency, those pilots would go straight to the computer and try to correct the problem through the digital system while the manual controls sat right next to them. If a situation needed correcting in a hurry, you could just… turn the automation off for a bit.

That immediately reminded me of a talk I gave at a regulatory meeting, where an audience member asked how to get an AI chat to write a justification for a process change exactly the way he wanted it. The first draft would come out decent, but he could never get the chatbot to land the final polished version, and the more turns he spent prompting, the worse it got. My answer was, "it is perfectly okay to use your fingers and type the words you want." I like to think Captain Vanderburgh would agree.

The best part of the lecture was what he proposed doing about all this. These days, when a new technology like AI has any hand in a mishap, the loudest voices immediately call for getting rid of the technology without knowing the full story. Most of the embarrassing AI headlines follow a similar pattern as that 68%, where the automation worked as designed and the people around it weren't watching closely enough. That makes sense to me, because we are all still working out what it even looks like to monitor something that didn't exist a few years ago. Vanderburgh's read was that the pilots who were never taught to hand-fly first had been done a disservice in flight school, and the alternative he laid out fits GMP work nicely. His position was to adopt levels of automation rather than trusting the automation in every situation.

So when people in life sciences ask "what should we automate, and what do we need to invest in to take advantage of this?", I'd rather answer a different question: where is your organization today, and which level of automation is right for the situation in front of you? Vanderburgh gave three, and I've loosely translated them to CMC work.

Full coupling — "write me a deviation summary."

Partial — you supply the evidence, the written regulation, and the structure, and the system works inside your boundaries.

Hand-flying — you write it first, and maybe the system critiques it after. (My favorite way to write anyway.)

Once the levels are defined, the flexibility is the whole point. When it gets weird, click down a notch. People's instinct is the opposite, where the output looks off so they re-prompt, and re-prompt, and re-prompt, which is the same head-down-fighting-the-FMS behavior that got crews into trouble. Just grab the controls. It's okay.

And think about where your hand-flying is worth the most. I'd skip the defensive exercise of cataloguing what AI can't replace about you, because I suspect it can do more of it than any of us would like. Stalling adoption won't get us anywhere, and neither will handing the whole thing over. The middle ground is where the organizations that figure this out are going to win.

If you have 25 minutes, the lecture is worth watching in full on YouTube.

Thanks for reading!

-Alexa

News

RAG Gets a Validation Framework (Sort Of)

What's New: Bioprocess Online published a Q&A on September 1 with data integrity consultant Ulrich Köllisch of UMK Consulting, previewing a talk he's giving with Körber Pharma's Pradeep Ravindra at the ISPE Annual Meeting on validating retrieval-augmented generation systems in GxP environments. The framing is that RAG is the practical bridge between probabilistic generative AI and an industry built specifically to eliminate variability, because the model answers out of a controlled knowledge base and cites the records it pulled from. Köllisch's position is that nobody needs a new framework for this, since GAMP 5, ICH Q9(R1), and existing data integrity expectations already stretch across the entire RAG life cycle, from ingestion and embedding through retrieval, model versioning, audit trails, and operational monitoring.

How It Works:

• Probabilistic systems don't return identical output for identical input, so testing against fully predefined expected results is off the table. Köllisch points out this problem already exists in GMP: visual inspection has inherent variability and we validate it with statistical methods, so the same thinking applies to a model.

• Validation has to cover the whole pipeline rather than the model alone, meaning data selection, ingestion, preprocessing, embedding, retrieval, and generation all fall in scope.

• A versioned model registry functions like controlled software release with model-specific information attached. A new version gets triggered by anything that could shift behavior or performance, including a change to the underlying model, the ingestion or embedding pipeline, or a preprocessing step.

• The attack surface grows once you're connecting multiple data sources, so least privilege and unique accounts matter more, and cloud hosting doesn't move accountability off the manufacturer.

• The agent operates inside the authenticated user's role-based access permissions, so it reaches only what that person could reach on their own.

• ALCOA++ traceability comes from an audit record capturing user identity, timestamp, the query, RBAC context, system configuration, which source records and versions were retrieved, the generated response, and what the user did next.

• On autonomy, Köllisch points to the draft EU GMP Annex 22 from July 2025, which explicitly prohibits probabilistic models from independently making decisions on critical GMP activities. He notes the draft never defines what "critical GMP activities" means.

Why It Matters: This is the most concrete public answer I've seen to the question every QA and CSV group is sitting on, which is what qualifying one of these things actually looks like on paper. The controls being described are ones regulated teams already run under different names: change control, access management, vendor oversight, audit trail review, and periodic performance monitoring. The part worth internalizing is that the audit record is what separates a defensible AI answer from a chat window, because when an investigator asks where a claim came from, "the model said so" is not a response and a retrieved source identifier with a version number is.

My Take: I’ve got a couple of thoughts on this. Number one, no two RAG systems are the same, and the performance you get out of one says almost nothing about the next. Chunking strategy, embedding model, how retrieval gets ranked, and what the knowledge base was built from all change the answer, and our documents are some of the harder material to retrieve against because so much of the meaning sits in tables, appendices, revision histories, and cross-references to other controlled documents. A validated pipeline doesn't fix that.

Number two, the Annex 22 line about probabilistic models not making decisions on critical GMP activities has gotten a lot of attention, and I think it's close to moot. There is no situation in GMP today where a Quality Unit isn't looking at the output before it means anything, and there won't be one soon. Removing that review would be a bad idea on its own merits, even if regulators were to suddenly lose their minds (by an alien brain-washing virus or whatever) and say it’s ok. The interesting part is that the fancy new technology has a good chance of pointing straight back at the document control problem we've had since long before any of this existed. ALCOA principles haven't changed. They just picked up a couple more plus signs and a much more expensive way to violate them.

Source: Controls For Retrieval-Augmented Generation In GxP — Bioprocess Online, September 1, 2026

When To Trust And When To Verify AI-Enabled Validation Systems

What's New: A companion piece to the RAG interview, published two days later in the same ISPE preview series. Jon O'Connell talks to Louie Rayal, VP of governance, risk, and compliance at GSK UK, who's presenting at the ISPE Annual Meeting with Compliance Group's Khaled Moussally. Rayal's argument is that FDA's 2022 computer software assurance guidance opened the door for AI to work inside the validation life cycle, because once the emphasis shifts from documenting every activity to demonstrating confidence in the high-risk areas, most of what AI is good at (drafting, summarizing, finding gaps, writing scripts) lands squarely on the administrative half of the job. The pitch is that validation teams are being asked to cover more complex digital ecosystems with leaner headcount while regulatory expectations hold steady or climb, and CSA plus AI is how you close that arithmetic.

How It Works:

• What gets cut is redundant narrative documentation that duplicates information already captured elsewhere, like lengthy test scripts and traceability write-ups. Rayal's confidence check is that risk assessments, objective test results, approvals, and traceability all survive intact.

• AI drafts the risk assessment, generates test scenarios, flags traceability gaps, summarizes results, and produces the validation report. Rayal calls these activities time-consuming but largely administrative.

• On FMEA, he says AI can surface failure modes a team might overlook by drawing on historical data and similar implementations, but the cross-functional SMEs still decide which risks are credible and what controls apply.

• Modern test automation tools can generate and execute scripts with minimal setup, so the human work concentrates at the end of the cycle: reviewing the AI output, confirming defects were actually remediated, and checking nothing slipped before sign-off.

• Human sign-off stays mandatory on anything touching patient safety, product quality, compliance, or data integrity. His list includes intended use, system classification, risk assessments, residual risk acceptance, deviation disposition, validation conclusions, and release.

• For inspection readiness, he says inspectors want transparency over perfection: an AI-generated rationale should identify the inputs used, explain how it got to its conclusion, trace back to requirements and risks and testing and approvals, and show that qualified people reviewed and approved the final call.

Why It Matters: Validation is the function where AI adoption in GMP is going to be decided, because it's the one place where the deliverable is already an argument about evidence rather than the evidence itself. It's also worth noticing what these two ISPE previews have in common. One says qualify the retrieval pipeline, the other says qualify the reasoning, and both land on the same closing move, which is that a qualified human signs.

My Take: The FMEA answer is right when the historical data actually exists. Where you have years of clean, retrievable records on similar equipment and similar products, a model proposing failure modes is a real gift. Not everyone does. Where you don't, the experienced people need to be in the room while the analysis happens rather than reviewing the output at the end.

I agree with him completely on the human sign-off thing, but it's also the easiest thing in our industry to say. A signature is a control only when the person signing has been trained to review this specific kind of output.  What happens when review volume has gone through the roof because automation, and the outputs look amazing at first glance? We would like to believe it works the same as before. But does it?

Source: When To Trust And When To Verify AI-Enabled Validation Systems — Bioprocess Online, September 3, 2026

Generative AI declaration: Heck yeah I use gen AI to brainstorm, draft, and edit my newsletter sections (would you even trust me if I didn’t?). My stories and phenomenal sense of humor are my own.  However, if that bothers you, looks like you signed up for the wrong free newsletter and you are entitled to a full refund. -A