You paste an AI answer into an email, and a colleague says, "That isn't what we offer." Or a customer rings about a deadline you never set. These slips come in a few familiar shapes, and once you know them they're easier to spot.
Made-up sources and references
This is the most documented kind. A 2023 study in a medical science journal looked at 178 references cited by GPT-3. Of those, 69 returned an incorrect or nonexistent digital object identifier (DOI), the code that points to a paper online.1
In a well-known 2023 US court case, lawyers filed a motion that contained numerous fake legal cases involving fictitious airlines, with fabricated quotations and internal citations.2 A business version is a "reference" in a proposal or blog post that looks real and goes nowhere. The full routine for this is in Checking Quotes, Links and Sources.
Wrong answers to simple-looking questions
Some failures are odd. The Australian privacy regulator gives an example: without protective measures, a language model asked how many "b"s are in "banana" will generally say two or three. The tool works with chunks of words, not letters, and its training data is full of similar questions about other letters.3
So counting letters or words deserves a second look, even when the question seems easy.
Out-of-date answers
A tool learns from text collected up to a certain point in time. That training data eventually becomes outdated.3 Prices, rules, opening hours, software menus and product ranges all change, and the tool may describe an old version as current.
Wrong policy or terms
In 2024 a Canadian tribunal heard a case about an airline's chatbot. It told a customer he could buy a full-price ticket and claim a refund of the price difference by applying within 90 days. The airline's own website said bereavement fares could not be applied retroactively to completed flights.4
The lesson for a small business is simple. If an AI drafts your returns policy, warranty wording or booking terms, the draft may say something you don't actually offer.
Plausible but unsupported claims
The OAIC notes that the way generative AI predicts text can lead to outputs that are inaccurate but appear credible.3 This covers the soft errors: a "fact" with no source, a confident comparison, a made-up trend. They're hard to catch because nothing looks wrong.
A quick scan list
- Names of people, firms, products and places
- Figures, prices, percentages and totals
- Dates, deadlines and time limits
- Quotes, links and "according to" statements
- Claims about what a law, rule or policy says
- Anything that sounds specific and comes with no source
Each of these has its own page of fixes. Start with Checking Names, Numbers and Dates if you're short on time, and read Why AI Makes Things Up if you want the cause.
Where the facts come from
- Hallucination (artificial intelligence), Wikipedia
- Mata v. Avianca, Inc., Wikipedia
- Guidance on privacy and the use of commercially available AI products, Office of the Australian Information Commissioner (OAIC)
- Moffatt v. Air Canada, Wikipedia
Keep checking
Why AI Makes Things UpWhy an AI tool can state a wrong answer with total confidence, what causes it, and what that means for checking its work.
How to Check an AI Answer Step by StepA simple routine for catching AI slips before you use an answer, from marking claims to confirming them at the source.
Checking Names, Numbers and DatesWhich small details AI tools most often get wrong, such as names, figures, prices and deadlines, and how to verify each one.