When AI Is Confidently Wrong

AI tools can produce answers that are completely wrong — invented facts, made-up sources, outdated rules, wrong figures — while sounding exactly as fluent and confident as when they're right. This module covers that risk (often called 'hallucination'), walks through a real case of a professional caught out by fabricated references, and includes a hands-on exercise where you find the error yourself. The rule it builds: AI drafts, a named human verifies anything that leaves the firm.

A ~2-minute walk-through. Prefer to read? The full lesson is below. Captions are on.

Listen instead (audio only) — good for a commute or patchy signal

In one line

This is the single most common way AI causes real damage in an ordinary organization — not through some dramatic hack, but through a routine, believable-sounding mistake that nobody caught before it went out the door. AI language tools do not know when they don't know something. When they lack a real answer, they don't say 'I'm not sure' — they generate the most statistically plausible-sounding answer, complete with invented details, fake references, and confident phrasing. There is no warning sign in the tone of the text. A wrong answer reads exactly as smoothly as a right one. For a small, cautious organization, the risk isn't that AI will refuse to help — it's that it will help, fluently and pleasantly, with something that is quietly false. Level 1 (drafting and summarizing) and Level 2 (using AI answers directly in your work) are exactly where this risk lives, because that's where AI output is closest to becoming something a customer, regulator, or colleague actually reads.

The short version — what to remember

AI language tools don't know when they don't know something — when they lack a real answer, they generate a plausible-sounding one instead of saying 'I'm not sure,' with no change in tone or confidence.
The most common hallucinations in ordinary business use are invented sources or citations, wrong or fabricated figures, and outdated rules stated as current.
Fabricated details are often the most specific-sounding parts of the text — a precise law section, an exact figure, a named study — because specificity is what makes writing feel authoritative, whether or not it's true.
The rule for Levels 1-2: AI drafts, a named human verifies anything that leaves the firm — checked against a real, independent source, not by asking the AI to check itself.
If you can't quickly verify a specific claim, soften the language or leave it out rather than sending it as stated fact.

A real (anonymized) example

In 2023, a New York attorney representing a client in a personal injury case against an airline used an AI chatbot to help research supporting legal precedent for a court filing. The resulting brief cited roughly six prior court cases, complete with realistic case names, docket numbers, and quoted judicial reasoning, all woven into a well-argued, professionally written filing. None of those cases existed — the AI tool had fabricated them entirely, generating exactly the kind of citations a real legal brief would contain, without any actual basis in real case law. The attorney did not check the citations against a legal research database before filing, because the text gave no indication anything was wrong. Opposing counsel and the presiding judge were unable to locate any of the cited cases; when the court ordered the attorney to produce copies, the fabrication was exposed. The judge sanctioned the attorney and his firm, the case drew international press coverage, and the attorney's professional standing was seriously damaged. The failure wasn't a lack of legal skill — it was treating fluent, confident AI output as pre-verified fact. A single independent check against a real case database, the kind of check that takes minutes, would have caught it before it ever reached a courtroom.

Reflect

A few open questions — nothing to trip on

These are to think through, not a test — there's no score and no wrong answer. Jot a line if it helps, then open the note to see what a careful answer usually considers. Nothing is saved or shown to your admin.

Question 1 of 4

An AI tool gives you a confident, specific-sounding answer that includes a citation or exact figure you've never seen before. What does that confidence tell you about whether it's true?

See what a careful answer considers

A careful answer usually lands on: Nothing — AI sounds equally confident whether an answer is correct or completely fabricated. AI language tools produce fluent, confident text as their normal output style, regardless of whether the underlying fact is real. Confidence and correctness are not linked — that's exactly what makes hallucinations hard to spot by tone alone.

Question 2 of 4

In the exercise paragraph about fire safety obligations, which sentence was most likely to be the fabricated one?

See what a careful answer considers

A careful answer usually lands on: The specific citation to 'Section 4.2 of the National Fire Safety Amendment Act of 2019' with an exact deadline. The most specific-sounding, citation-like detail is often the fabricated one — precise names, section numbers, and deadlines make text feel authoritative whether or not they're real, which is exactly why they deserve the first check.

Question 3 of 4

What does the rule 'AI drafts, a named human verifies anything that leaves the firm' actually require in practice?

See what a careful answer considers

A careful answer usually lands on: Checking specific facts, figures, and claims against a real, independent source before anything with those details goes out under the firm's name. Asking the AI to re-check itself just produces another confident-sounding answer from the same source. Verification means checking against something independent — the real regulation, the real document, or a person who knows.

Question 4 of 4

In the real case discussed in this module, what actually went wrong?

See what a careful answer considers

A careful answer usually lands on: Fluent, confident AI-generated legal citations were filed in court without being checked against a real case database first. The AI tool fabricated realistic-sounding court cases, and the failure was that no one checked those citations against an independent, real source before they went into an official filing — the same everyday mistake this module's rule is built to prevent.

Read the full written lesson (~9 min)

Why AI gets things wrong without acting like it

It helps to understand, in plain terms, what's actually happening. AI language tools generate text by predicting what words are likely to come next, based on patterns learned from enormous amounts of writing. They are built to produce fluent, confident-sounding sentences — that is the entire skill. They were not built with a reliable way to check whether what they're saying is actually true.

This means an AI tool can, in the same tone of voice, correctly summarize a real document and invent a completely fictional one. It can quote a regulation accurately, or state a rule that expired two years ago as if it were current. It can cite a real study, or a study that was never written, with a title, author, and date that all sound entirely plausible. There is nothing in how the sentence is written that tells you which situation you're in — no hedge, no stumble, no change in confidence.

This behavior has a name: 'hallucination.' It doesn't mean the tool is broken or malfunctioning — it's producing exactly the kind of text it was designed to produce. It just isn't designed to guarantee that text is true. The three most common shapes this takes in ordinary business use are: invented sources or citations (a report, law, or study that doesn't exist), wrong or outdated figures (a number that sounds precise but is fabricated or stale), and outdated rules (a regulation, code, or policy stated as current when it has since changed).

Real case: the lawyer who filed fake court cases

In 2023, a practicing New York lawyer used an AI chatbot to help research and draft a legal brief for a client's case. The brief cited around six court cases to support the legal argument — complete with case names, docket numbers, and quoted legal reasoning. The writing was polished, well-structured, and read exactly like a normal, competent legal filing.

There was one problem: none of those six court cases existed. The AI tool had invented them entirely — realistic-sounding names, plausible citations, quotes that fit the argument perfectly — because that is what a fluent, plausible-sounding legal brief looks like, whether or not the underlying facts are real. The lawyer did not independently check the citations against a real legal database before filing, because nothing about the text looked suspicious.

The opposing lawyers and the judge could not find any of the cited cases. When the court asked the lawyer to produce them, the fabrication came to light. The judge sanctioned the lawyer and his firm, the case became international news, and the lawyer's professional reputation was seriously damaged — for citing sources that had never been checked, not for anything an intelligent, careful person would normally have thought to double-check on sight.

Where this shows up in ordinary Level 1–2 work

You don't need to be a lawyer for this risk to apply to you. It shows up anywhere AI is used to draft, summarize, or directly answer a question whose result might be read, acted on, or sent onward by someone else. A few realistic, everyday examples for a small, non-technical firm:

Drafting a client or tenant letter: you ask an AI tool to draft a message referencing 'the relevant fire safety regulation,' and it states a rule confidently and specifically — except the rule it names was updated last year, or doesn't apply to your type of building at all.
Summarizing a supplier contract or report: you ask AI to summarize a long document, and the summary includes a number, date, or clause that sounds exactly like something that would be in the document — but isn't, because the AI filled a gap in its understanding with something plausible.
Answering a factual question directly: you ask 'what's the current minimum requirement for X,' and the AI gives you a specific, confident-sounding figure or rule — with no indication of whether it's pulling from something current, something outdated, or nothing real at all.
Quoting a source to sound authoritative: you ask AI to help make a proposal or report sound more credible, and it adds a supporting statistic or a named study to back up a point — one that sounds exactly right for the argument and was invented for that reason.

None of these require anything unusual or technical to go wrong. They are the normal, everyday uses of AI at Level 1 and Level 2 — which is exactly why this risk deserves a routine habit, not a one-time warning.

Hands-on exercise: find the error the AI made

Below is a short paragraph, written in the fluent, confident style typical of an AI drafting tool, as if answering: 'Summarize our building's fire safety inspection obligations for the tenant handbook.' It contains one fabricated or wrong detail planted among true-sounding, reasonable statements — exactly the way a real hallucination would appear in your own work. Read it once, the way you normally would, and see if anything catches your attention before reading the answer below it.

"Under current fire safety regulations, commercial buildings of this size must undergo a full fire risk assessment every 12 months, conducted by a qualified assessor and documented in a fire safety log. Emergency lighting should be tested monthly, and all fire extinguishers require an annual service check by a certified provider. Per Section 4.2 of the National Fire Safety Amendment Act of 2019, any building with more than three floors must additionally install a secondary smoke ventilation system by the end of this year, regardless of prior compliance status."

Where's the problem? The first three sentences are the kind of general, plausible fire-safety practice you'd expect to find in a real handbook — routine assessments, monthly lighting checks, annual extinguisher service. But the last sentence — 'Section 4.2 of the National Fire Safety Amendment Act of 2019,' with a specific, invented-sounding mandatory deadline — is exactly the hallucination pattern: a precise-sounding citation, a real-sounding law name, and a confident deadline, none of which you (or the AI) actually verified against a real regulation. It reads as the most authoritative sentence in the paragraph — and it's the one that was fabricated. This is the trap: the fake detail is usually the most specific-sounding one, not the vaguest.

The exercise habit to take from this: when you read AI output, don't scan for what sounds wrong — scan for the parts that sound most official, most specific, or most citation-like, and check those first. That's where a hallucination hides best, because specificity is what makes text sound trustworthy, whether or not it's true.

The rule: AI drafts, a named human verifies anything that leaves the firm

This is the one habit to carry out of this module: AI drafts. A named human verifies anything that leaves the firm — before it goes.

In practice, this means:

Any number, date, rule, or citation an AI tool gives you gets checked against a real, independent source before it's used — the actual regulation text, the actual contract, the actual supplier document, a quick search, or a colleague who knows. Not by asking the AI to double-check itself, which just produces another confident-sounding answer.
'Leaves the firm' is a useful test. A quick internal note you'll re-read yourself carries low risk. A letter to a tenant, a client-facing report, a compliance document, or anything with your organization's name on it needs a named person to say 'I checked this and it's right' before it goes out.
The check takes less time than it feels like it should. You are not re-doing the AI's work — you are spot-checking the specific facts, figures, and claims, especially the ones that sound most precise or official, against something independent.
If you can't verify something quickly, that's your answer: soften the language (say 'you may want to confirm current requirements with...' instead of stating a specific rule as fact), or leave it out until you can check it.

This one habit — verify anything specific before it leaves the building — would have stopped the fabricated court citations, and it stops the everyday version of the same mistake in ordinary drafting, summarizing, and answering.

This lesson is written and reviewed by named humans. Content current as of 21 July 2026. See the Trust Center for our review process and AI-assistance disclosure.