Checking Quotes, Links and Sources

The draft quotes an expert, names a report, or ends with a tidy list of links. You click one and get an error page, or find a real page that never says what the draft claims. That happens often enough that every citation needs opening.

Why sources fail

A tool that predicts likely words can produce a likely-looking reference. It doesn't need the reference to exist. The Australian privacy regulator notes that the way generative AI predicts text can produce outputs that are inaccurate but appear credible.¹

Researchers have counted it. A 2023 medical journal study checked the codes (called DOIs) that should lead to papers. For 69 of the 178 references GPT-3 cited, the code was incorrect or didn't exist.²

A 2024 University of Mississippi study of student work found that many of the citations that students submitted were partially or completely fabricated.² A reference list can look finished and still be wrong.

Fake sources in practice

In the US case Mata v. Avianca, a motion written with a chatbot cited numerous fake legal cases.³ The other side's lawyers told the court they were "unable to locate" a few of the cited cases, and the court could not find them either. The court ordered the plaintiff's lawyers to provide copies. They supplied documents purportedly containing all but one of the cases, after ChatGPT assured them the cases "indeed exist".³

So never use the same tool to confirm its own source.

Checking a link

• Click it, or paste it into a browser. Don't rely on how it looks.
• If it fails, search for the page title on the publisher's own site.
• If the site can't be found, or the page doesn't exist, drop the source.
• If the link works, read the page. A working link to an unrelated page is still a bad source.

Checking a quote

A quote should appear word for word on the page it's attributed to.

• Copy a short run of the quoted words and search for them inside the page. Use your browser's find tool.
• If you only find a loose match, the quote has been reworded. Either use the real words or describe the idea without quote marks.
• If the quote is attributed to a person, find it in an interview, transcript, filing or the person's own publication.
• If you can't find it anywhere, don't use it. Don't paraphrase a quote you couldn't confirm and still attribute it.

Checking a named report, book or paper

• Search the exact title along with the author or publisher.
• Look for a record in a library catalogue, the publisher's site or a government site.
• For a DOI, paste it after doi.org/ in your browser and see where it lands.
• Check the year, the authors and the title together. If any one of them doesn't match, treat the reference as unconfirmed.

What counts as a good source

Prefer the original: a government page, a regulator, a standards body, a maker's own manual or a university. A blog post that repeats a fact isn't a source for it. Find where the blog got it.

If you can't confirm it

Cut the claim, or rewrite it without the unconfirmed source. If the point matters, ask a person who knows the field.

For the full routine, including numbers and dates, see the checking routine. To see which jobs shouldn't go out on one person's glance, read about getting a second reader.