BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

CASE-34 · Case Study — The Lawyer Who Cited Six Cases ChatGPT Made Up

What happened

In a routine personal-injury suit against the airline Avianca, a New York attorney filed a legal brief citing several court decisions to support his argument. There was one problem: at least six of the cited cases did not exist. The lawyer had used ChatGPT to do his legal research, and the model had fabricated the cases — complete with plausible names, docket numbers, quotes and internal citations. When opposing counsel and the judge couldn't find them, the attorney even asked ChatGPT to confirm the cases were real; it cheerfully insisted they were. In June 2023 the judge sanctioned the lawyers and their firm.

See it — fabrication all the way down

flowchart LR
  A["Lawyer asks ChatGPT<br/>for supporting cases"] --> B["Model invents 6 cases<br/>+ fake quotes & citations"]
  B --> C["Filed in federal court<br/>unverified"]
  C --> D["Asks ChatGPT 'are these real?'<br/>→ 'yes, they are'"]
  D --> E["Sanctions + national<br/>embarrassment"]

The root cause

A large language model predicts plausible text; it does not consult a database of ground truth. Asked for supporting case law it had none of, it produced the statistically likely shape of case law — real-sounding names and citations that were pure fiction. This is hallucination, and it is not a bug to be patched but an inherent property of ungrounded generation. The compounding error was human: the lawyer treated a fluent generator as an authoritative research tool, performed no independent verification, and — fatally — tried to check the AI's work with the same AI, which happily confirmed its own fabrications because agreeing was, again, the plausible next text.

The fix that would have prevented it

Treat every LLM factual claim as unverified until checked against a primary source — for law, that means the citation must resolve in an actual reporter or legal database before it goes in a brief. Use grounded, retrieval-based tools for research (systems that return real documents with links) rather than free-form generation. Never verify an AI's output with the same AI; the check must be independent and external. And recognise the professional-accountability principle the court applied: delegating research to a tool does not delegate responsibility — the human who signs the work owns every word.

The blast radius

The lawyers were fined and publicly sanctioned, the case became international news, and courts across the US began issuing standing orders requiring disclosure of AI use and certification that citations were verified. It is now the canonical cautionary tale taught in law schools and AI-literacy trainings alike, and it crystallised for a mass audience the single most important limitation of LLMs: they are fluent, not factual, and confidence is not evidence.

The lesson

An LLM will state a falsehood with exactly the same fluency and confidence as a truth, and it cannot reliably audit itself. Anywhere accuracy carries stakes, generation must be paired with grounding and independent verification, and the human in the loop remains fully accountable for the output. "The AI told me" is not a defence — it is an admission that no one checked.

Where it rhymes