In April, the Alabama Supreme Court sanctioned an attorney for filing legal briefs with citations to cases that did not exist. After being caught, he promised it wouldn't happen again. Then he cited more nonexistent cases at the end of the very next sentence.

This is not a story about a bad lawyer. It's a story about what happens to smart, responsible people when they outsource their thinking to a machine that makes things up.

The Scale of the Problem

A database maintained by Damien Charlotin, a senior research fellow at HEC Paris, lists more than 1,400 court decisions addressing AI errors in the past three years. The growth was exponential. It's now leveled off at roughly 350 to 400 decisions per quarter — a permanent undercurrent, not a spike.

The problem isn't limited to law. Journalists have published fabricated quotes. Software developers have shipped code based on invented APIs. Academic researchers have cited non-existent studies. Government consultants have made policy recommendations backed by data that was never there.

And it keeps happening despite everyone knowing the risk.

Cognitive Surrender

Researchers at the University of Pennsylvania's Wharton School have a name for this: "cognitive surrender." It's what happens when people stop verifying AI outputs because the cognitive cost of doubt feels higher than the cost of trust.

In one study, participants completed image classification tasks with AI-provided guidance. The guidance was wrong 50% of the time — regardless of whether it came from AI or humans. But participants with positive attitudes toward AI performed significantly worse when told the advice came from a machine. They knew it could be wrong. They trusted it anyway.

More alarming: a drone warfare simulation by Penn State's Alan Wagner found that participants — most making accurate civilian/combatant classifications — reversed their views in most cases where a robot disagreed with them. The robot's feedback was random. The participants' assessments were initially sound. The combination of the two produced catastrophe.

"I think that's the context in which those findings have to be interpreted," said Colin Holbrook of UC Merced. "These people were really trying. They thought that it mattered. And if the scenario had been real, they would have killed a lot of innocent people."

Why Warnings Don't Work

Researchers at Boston University ran an inoculation study: they warned students explicitly that ChatGPT produces inaccurate summaries of academic sources and struggles with complex math. Results were mixed and revealing.

Warning significantly increased verification on source-summary tasks. But on math problems, there was no significant effect — verification rates stayed low. Some participants said they already trusted AI's math abilities regardless. Time constraints (mimicking real deadlines) also reduced verification frequency.

"Our findings suggest that awareness alone isn't enough," said Chi B. Vu, a graduate student in human-AI interaction at Boston University.

The forces working against verification are structural, not just behavioral. Advertising highlights AI's potential. Workplaces pressure people to use AI to save time. As models improve, users become less inclined to double-check — deepening false confidence precisely when the stakes are highest.

The Dangerous Cycle

The core problem is that people who trust bad AI outputs never encounter the error. They file the brief. It gets sanctioned. But they don't learn the lesson the way they'd learn it from a human colleague who caught the mistake.

"They don't ever get to the ground truth," said Sophie Nightingale of Lancaster University. "They don't have any reason to question it because they carry on in their lives thinking that AI tool is correct — because 'Why wouldn't it be?'"

Charlotin has built a reference checker called Pelaikan to address the citation problem specifically. But tools only work for people who remember to use them — and cognitive surrender by definition means people stop checking.

The Alabama attorney didn't miss one citation. He missed dozens. He then repeated the mistake immediately after being formally warned. That pattern — over-trust, no correction, repeat — is the actual story. It's not about a bad actor. It's about what happens when the incentive structure of legal work meets a technology that is confidently wrong.

Source: Scientific American, May 2026. Research cited: Nature Scientific Reports (February 2025); University of Pennsylvania Wharton; Boston University Human-AI Interaction Lab; Penn State Aerospace Engineering.