Why do AI models hallucinate?

Howdy friends,

For those of you who celebrated US Independence Day and Canada Day last week, I hope they were fun and restful! After a bit of a break, I'm excited to bring you this week's edition of the LabScale AI Weekly. And if you're new here — welcome! I have a feeling this is the beginning of a beautiful friendship.

Today I want to dig into a term we throw around constantly when we talk about AI assistants but rarely stop to unpack: hallucinations. A hallucination is when an AI gives you a response that sounds plausible, confident, and a whole lot like the correct answer buuuuut it isn't.

Let's start with the "why," because part of it is actually about us, not the machine. This is something I like to bring up in the mindset portion of my trainings. As humans, we've developed proxies — mental shortcuts — for instantly assessing whether something (often a person) is correct. One of the pitfalls of that shortcut is how we make the snap judgment: we lean heavily on how confident the person sounds. Well, this guy seems to believe himself, so I'll believe him too! Trusting our confident fellow humans has worked out okay (usually) for most low-to-medium-risk moments in life. But we all know there's a whole legion of people who can tell you something completely made up with a perfectly straight face. Some of them even believe it themselves — which, funny enough, isn't so different from an AI hallucination.

That said, frontier models have been improving fast, and hallucination rates have generally dropped as they've gotten better — 2024 benchmarks in the 15–45% range have come down into the low single digits to high teens depending on the task. I'll add one honest caveat, because it matters for how we work: the trend isn't a clean straight line. Some of the newer reasoning-focused models actually hallucinate more on pure fact-recall, because "thinking out loud" tempts them to fill in a missing step with something plausible rather than admit they don't have it. So "getting better" is not the same as "safe to stop checking”.

And here's the flip side of all that improvement: the better these models get, the more we trust them, and the less we verify. Our brains evolved to conserve energy wherever possible, so when a tool shows up that can do the thinking for us, we naturally want to offload as much cognitive load onto it as we can.

But that's just being lazy! people shout.

I'd argue it's more a feature of nature than a character flaw. One to be aware of, and maybe even to restructure how we think around, but I'll skip the hostile judgment. The catch is that as hallucinations get rarer, they also get harder to anticipate and harder to spot, and that very rarity is what breeds the over-trust.

So why do models hallucinate in the first place? As most of us know by now, models are trained on enormous amounts of data, and over the course of training they get very good at predicting patterns of words and phrases — kind of like a supercharged version of the autocomplete on your phone. Those predictions are shaped by how likely a particular token (a word, or a piece of a word) is to come next. The more often a pattern shows up in training, the more weight that prediction carries.

Think about a movie you watched way too many times as a kid — to the point of wearing out the VHS tape and driving your parents up the wall. (Mine was the first Ninja Turtles movie; my brother's was Jurassic Park, and we had to alternate nights.) Feed me a line from that movie and I could finish it without flexing a single brain cell. I probably couldn't stop myself if I tried. But feed me a line from page 356 of my junior-year organic chemistry textbook? Not exactly on the tip of my tongue. I read it once, and obviously that’s not enough.

So when the answer to a question is obscure, highly specific, or changes over time, the model does what it's built to do: it guesses. Which raises the obvious question — why not just say "I don't know"?

That brings me to the second reason, and this one is a bit more interesting. Models are trained to be helpful. Recent research (a nice paper out of OpenAI) points out that a big part of the problem comes from how models are trained and graded. When you score two possible answers, an actual attempt has historically scored better than an honest "I don't know" — so, much like a student on an exam, the model learns that a confident guess beats admitting uncertainty. Training and evaluation setups are being adjusted to reward appropriate humility, but the issue is far from solved.

We can, however, put ourselves on higher alert. Which kinds of questions are most likely to produce a hallucinated (made-up) answer? The big four:

1. Very specific facts, statistics, or citations

2. Obscure information that probably didn't come up much in training

3. Real people or events that aren't widely known

4. Incorrect claims that popular belief has repeated for so long they've muddied the underlying data

So how do we keep hallucinations from tripping us up? Also four:

1. Ask for sources — then actually click through and check them.

2. Tell the AI, right in your prompt, that it's okay to say "I don't know."

3. Verify facts, numbers, and citations — a web search works, and so does starting a fresh chat and asking a model to fact-check the previous output.

4. Stay a little skeptical, and ask follow-up questions.

One thing I want to make crystal clear: this failure mode is not a reason to avoid AI altogether. It's easy to get frustrated, even angry, when an output is wrong. But understanding the root causes (it's not personal, I promise) and having a few mitigation strategies in your back pocket is exactly what makes us strong, capable AI users across the board.

Thanks for reading!

Alexa

News

Fable 5 Is Back: Anthropic Redeploys After Export Controls Lift

What's New: The saga concludes! (for now) Over the last two (three?) weeks I covered Fable 5's launch, then its abrupt takeback three days later under a government export-control directive. On June 30, the export controls were lifted, and Fable 5 came back online globally on July 1 — with Mythos 5 restored to a set of approved US organizations. Alongside the redeploy, Anthropic laid out new safeguards and, more interestingly, a proposed industry-wide framework for grading the severity of AI "jailbreaks."

How It Works:

• The original directive traced back to an Amazon report describing a way to bypass Fable 5's safeguards to get it to identify software vulnerabilities (and, in one case, produce code demonstrating an exploit). Anthropic's testing found that less capable, widely available models — including Opus 4.8, GPT-5.5, and Kimi K2.7 — could surface the same vulnerabilities, and every model tested could reproduce the single exploit demo.

• Anthropic trained a new safety classifier that blocks the specific reported technique in over 99% of cases. Blocked requests to Fable 5 get rerouted to Opus 4.8. The tradeoff: more false positives on routine coding and debugging.

• The models use "defense in depth" — layered safeguards plus a deliberately wide "safety margin," meaning the classifiers block some requests that are probably benign to make sure genuinely harmful ones don't slip through.

• The proposed jailbreak-severity framework (drafted with Amazon, Microsoft, Google, and other Glasswing partners) scores a jailbreak on four axes: capability gain, breadth of that gain, ease of weaponization, and discoverability.

Why It Matters: For those of us in regulated work, the framework is the real story, not the drama. An objective, shared rubric for scoring the severity of a model's failure mode — and a calibrated, documented response to it — is exactly the kind of structure QA and Regulatory folks recognize. It’s how the security world already scores software vulnerabilities (CVSS). We're watching AI governance grope toward the standardized, auditable risk language that our industries have run on for decades. If it translates has yet to be seen.

My Take: Two (three?) weeks, three newsletters, one model — launched, yanked, redeployed. Beyond the whiplash, remember: don't over-anchor on any single model or provider. Access can vanish in an afternoon and reappear just as fast. But the more durable signal here is that "how do we objectively rate an AI's risk, and who decides when to act" is finally becoming a real, structured question with real frameworks attached. That's the machinery regulated industries will eventually be handed, so it's worth watching it get built.

Source: Anthropic – Redeploying Fable 5

Why an AI Lost $29k Running a Café in Stockholm

What's New: Andon Labs handed an AI agent named "Mona" the keys to a real café in Stockholm — real permits, real staff, real money — and let it run the place. Over two months on Gemini 3.1 Pro, Mona spent about $38k against just $9k in sales. Then they switched her to GPT-5.5 to compare. The write-up is a delightfully honest autopsy of where an autonomous agent's judgment falls apart in the real world.

How It Works:

• Easily manipulated: Gemini-Mona dropped an espresso from $3.60 to $1 because a stranger emailed calling it a "loss leader" ("Your reasoning is so convincing…"). She honored an unverified "99% discount" and, for one startup event, covered overtime, free food and drinks, $2.3k in branded hoodies, a $2.8k LED screen, and a $1.2k photographer — until the founder himself told her to stop.

• No initiative: She never investigated why weekday sales were slow, kept the same 11–5 hours from day one, and was slow to add the lunch items customers kept requesting.

• Over- and under-ordering at once: She bought 1,331 fresh bakery items against 326 sold, plus catering-scale quantities (8 kg cocoa, 4 kg salt, 1,200 teabags) for products that never hit the menu — while forgetting to order ingredients for dishes she did list, so baristas were told to make salads the café didn't stock.

• The core failure: the model ran on generic "what an LLM knows about cafés" and never adapted based on feedback from its own real financial data. GPT-5.5 was harder to manipulate and stopped listing dishes it couldn't make — but overcorrected, got spooked by low cash, and nearly stopped ordering entirely.

Why It Matters: Strip away the pastries and this is a study in agentic judgment failure — and it maps almost too cleanly onto why we don't hand autonomous agents the keys in regulated work. Mona could execute tasks (permits, hiring, a menu) but couldn't close the loop between her actions and her real-world outcomes, couldn't resist social engineering, and confidently instructed downstream operators to do things her inventory couldn't support. Swap "café" for any GxP process and those are precisely the risks — no feedback grounding, no verification, susceptibility to a persuasive prompt — that keep a human firmly in the loop.

My Take: This is the most fun I've had reading an eval, and also one of the more useful ones. It's a concrete, non-hypothetical picture of the "jagged frontier": the same agent that can navigate Swedish permit paperwork will also buy 30 hoodies because someone asked nicely. The capability curve on these long-horizon tasks is climbing fast, so I don't read this as "agents are useless" — I read it as a preview of exactly which judgment gaps we'll need controls, checkpoints, and human sign-off around before any of this goes near regulated decisions.

Source: Andon Labs – Why Gemini 3.1 Pro Lost Money Running Andon Café