Staying in your lane: the case for an "AI License"

Hey friends,

I've been thinking this week about the case for an "AI license."

OK, so not a real credential but more like an honesty test. Here’s my idea (it’s great, as most of my ideas are): Anyone can use generative AI for general knowledge. Anyone can (and should) have AI explain something well established, summarize an dense article, or help with brainstorming. It’s awesome, so go nuts.

But for niche, high-risk, regulated work (QMS, Reg, GMP), AI should only be in the hands of someone who can recognize when it's wrong. If you're not the domain expert, you shouldn't be the one generating the deliverable. Stay in your lane, because I am certainly going to stay in mine.

The more accurate term is probably domain authority. Authority over a domain is what permits AI use in that domain. But "AI license" captures how it should feel. Like something you either have or you don't.

It might sound restrictive but it really isn't. It's just being honest about what you can and can't verify.

We're in the "everyone has access to AI" phase right now. Information has never been more available, while the judgment to use it correctly has not budged. Which means domain expertise is more valuable now, not less, even though most of the AI conversation seems to be pointed in the opposite direction.

Why is that, though? It's worth digging into for a second, because I think people are missing the underlying mechanic. So here are my thoughts:

The first reason is that verification is harder than generation. LLMs are extremely good at producing plausible text. They are not good at evaluating whether that text is correct in a specific context, high-stakes or not. Verification requires knowing what should be there, not just what could be there. That capability lives in the domain expert, even to this day.

The second reason is tacit knowledge. A huge chunk of what any expert knows isn't written down anywhere AI can train on.

If you ever have the pleasure of attending a social event with a crowd of CMC consultants, you’ll soon find out that we’ve seen some sh*t.

We have war stories and seen so much stuff go wrong. But more importantly, we usually know what matters in a given situation versus what's just procedurally true. And if we don’t know, we almost always know another guy who does.

AI is trained on the explicit corpus. The experts hold the implicit one.

The third reason is calibrated uncertainty. An expert knows the shape of their own ignorance. They can say "I'm pretty sure on this, less sure on that, and on this third thing I have no idea and would never claim to." LLMs produce uniformly confident output regardless of whether they're on solid ground or completely hallucinating. The expert provides knowledge, and ideally appropriately bounded knowledge.

That last one is a biggie for me. I know where my knowledge gaps are and I know where I can go to fill them. AI does not. Or at least, it can't tell you with any confidence. Asking AI to flag what it doesn't know is like asking someone to spell a word by only telling you the letters that aren't in it, then write a book with that process. You maybe can, but…should you?

At least for today, AI output still looks good to people who can't tell when it's bad. That's not a knock on anyone, it's a structural thing. You don't know what you don't know, and the AI doesn't know either, and now you've got two parties confidently producing something that nobody in the conversation is actually qualified to verify

A version of this happened to me recently. On a lot of my projects I work alongside consultants from other departments (finance, usually), who own the client slide deck while I provide the bioprocess content. One time the finance team got ambitious and added a bunch of pharma product details before I had a chance to weigh in. The terms and vocab sounded great. They were also completely hallucinated. It was clear they weren't familiar with what they were writing about (what a conjugated vaccine is, what an adjuvant does, what a "presentation" even means in this context). I get it. They're finance people. They meant well. But next time just leave it alone because now I have to redo this slide.

Which brings me to cGMP. I've been deep in FDA warning letters lately for some research, and 99% of the time, the agency's recommendation is the same: hire a cGMP consultant. That was true before AI. It's still true now. One thing that may have changed is that some sponsors now think AI is a substitute. AI knows what a consultant knows, so we'll just use AI. What a cost savings!

Speaking as both a CMC consultant and an AI power user: no. I can offload plenty of tasks to AI. I do, every day. But the specific intersection of what my clients need and what I know is still too specific for AI to navigate on its own. I am, frankly, not worried about being replaced.

Will this change in the future? At the edges, definitely. Better models, better retrieval, and better tooling will keep raising the floor of what AI can handle without an expert in the loop. What feels like "domain expertise required" today will be "general knowledge" in a few years. But as the floor rises, the bar for what counts as expertise rises with it. The expert of 2030 isn't doing the same work as the expert today. They're working on harder (or maybe just different) problems, more ambiguous cases, and more integrated decisions. The gap doesn't necessarily close, it just shifts up.

And the calibrated uncertainty part, the one about knowing what you don't know, isn't getting solved any time soon. Until it does, there has to be someone in the loop who can know what they don't know. That's the domain expert, and they will still be the one who actually knows where the new bar is.

You all know me, none of what I’m saying here is anti-AI. I'm pro-AI, every day, all the time. But as cliche as it may sound at this point, your expertise is the thing AI can't replicate, and the goal isn't to outsource it. Use your AI license to amplify it.

Speaking of which…that's the whole premise of AI-Proof Your Practice, my next live workshop on May 29 (half-day virtual). It's about figuring out where AI fits in your work, with your expertise. And just as importantly, where it doesn't.

Thanks for reading!

Alexa

News

FDA Goes from Chatbot to Workspace in 11 Months

What's New: The FDA announced May 6th that Elsa, the agency's internal AI tool, has been upgraded to version 4.0. The update brings custom agents, document generation, quantitative data analysis with charts, secure web search, voice-to-text, OCR, and "optimized search" across large document repositories. Alongside the upgrade, the agency consolidated 40+ separate application and submission data systems into a single platform called HALO (Harmonized AI & Lifecycle Operations for Data). Elsa now sits on top of HALO. As Chief AI Officer Jeremy Walsh put it, "previously, FDA staff would bring data to Elsa. Now, Elsa sits on top of our data."

How It Works:

• Elsa 1.0 launched in June 2025 as a relatively basic LLM tool for reading, writing, and summarizing

• Elsa 4.0 (today) is closer to a full workspace with agents, document generation, data visualization, and search across the agency's data

• HALO is the data backbone underneath. Forty disparate application and submission systems consolidated into one platform. That's a years-long IT project most agencies never finish.

• The integration means staff can now query agency data and build workflows without manually uploading documents into chat

• Built on a FedRAMP High Google Cloud environment. Does not train on inputs or industry-submitted data. Web search pulls from refreshed secure web data, not the live internet.

• FDA explicitly states that human subject matter experts verify all inputs, analytic processes, and outputs

Why It Matters: The speed is the story. Eleven months from chatbot to workspace, inside a federal agency, is genuinely unusual. For life sciences professionals, two things to internalize. First, your submission is no longer a static document landing in an inbox. It's something that can be queried, compared, and cross-referenced against the agency's full submission history in seconds. How you structure submissions is going to start mattering differently (clarity, internal consistency, traceability). Second, FDA is now operating with a more sophisticated AI stack than most of the small-to-mid biotechs submitting to it. Time will tell if that becomes a problem.

My Take: OK, some of this feels a little flashy. OCR? Voice-to-text? That's not exactly cutting-edge in 2026, and listing it next to "custom agents" makes the announcement feel like it's trying a little hard. The whole rollout has the energy of making an old boyfriend jealous (sorry Anthropic) and making sure everyone sees how great things are going with the new BF after a bad breakup with the US government. That said, it really is good to see a traditionally slow and clunky agency take AI adoption this seriously. The architectural shift (Elsa sitting on top of HALO instead of staff pasting things into Elsa) is the genuinely meaningful part. I just hope they're putting equivalent effort into training staffers to use these tools well, because the gap between "an agency has AI" and "an agency uses AI competently" is exactly the one I keep writing about.

Source: https://www.fda.gov/news-events/press-announcements/fda-expands-ai-capabilities-and-completes-data-platform-consolidation