Why LLMs Invent Plausible‑Looking but False Citations

The little X thread was right to be annoyed: “And then there is this problem,” and a screenshot of a perfectly formatted bibliography that points to nothing. That moment — when a model hands you a bibliography that looks real but turns up zero results in Google Scholar — is not a mysterious failure mode of some rogue intelligence. It’s the natural behavior of a system trained to do one job extremely well and then asked to do something it was never taught to do: verify truth. (nature.com) Modern LLMs are trained as autoregressive next‑token predictors. In plain terms, during pretraining the model learns to maximize the probability of the next word (or token) given the preceding text; that objective is what gives a model fluency, style-transfer, and the ability to imitate scholarly prose. The canonical large‑scale example is GPT‑3, an autoregressive model trained on enormous text corpora to predict next tokens, and that framing explains both why models are useful and why they confidently produce plausible‑sounding text that can be false. The model learns shapes: how authors’ surnames, year numbers, article titles, and journal names are arranged — not which specific DOIs or PubMed IDs correspond to real studies. (arxiv.org) At generation time the model faces a decoding decision: which token to emit next? Popular decoding strategies include greedy selection, beam search, and stochastic sampling (top‑k or nucleus/top‑p). These strategies are all about producing locally probable continuations, and none of them magically perform a database lookup. Decoding amplifies the problem because the model will happily “fill in blanks” with high‑probability tokens that make the citation look coherent. This is the same mechanism that causes a model to invent a plausible but nonexistent restaurant name when asked for local recommendations: it knows the linguistic pattern of such names and completes the sequence with tokens that maximize local likelihood and fluency, not with tokens grounded in external reality. Work on decoding dynamics — notably the paper that diagnosed neural text degeneration and proposed nucleus sampling — shows how sampling choices dramatically alter hallucination and fluency behavior. (arxiv.org) There’s another, subtler training mismatch that explains why small lies compound into believable fabrications. During training the model is “teacher forced”: it always conditions on the true previous token from the dataset, so it never learns to recover from its own mistakes. At inference time, however, every token it emits becomes the context for the next prediction; a single fabricated author surname or truncated title biases subsequent tokens toward more fabrication. This exposure bias (and the suite of methods developed to mitigate it) is why a bibliography that begins with a near‑random surname and a plausible journal name can grow into a fully plausible but non‑existent citation within a few tokens. Engineers have proposed fixes such as scheduled sampling to reduce exposure bias, but they address one aspect of the mismatch — they don’t equip the model with the ability to check whether a generated DOI exists. (bengio.abracadoudou.com) The empirical data is blunt. In controlled experiments where ChatGPT variants were instructed to produce short literature reviews, more than half of the GPT‑3.5 citations were fabricated; GPT‑4 improved the picture but still produced fabricated citations at nontrivial rates (55% fabricated for GPT‑3.5 versus 18% for GPT‑4 in one study). On a larger scale, recent audits of published literature show the phenomenon bleeding into the scholarly record: one large‑scale analysis of thousands of papers estimates an accelerating rise in invalid or fabricated citations tied to AI usage, and a 2026 preprint that built an open citation verifier found hallucination rates across models ranging from roughly 14% to 95% depending on the model and domain, and detected an uptick of invalid citations in conference proceedings in 2025. Those are not edge‑case anecdotes; they are measurable, reproducible artifacts of how these systems are trained and used. (nature.com) If this sounds damning, it shouldn’t — the failure is comprehensible and therefore fixable. Treating the problem as an engineering constraint yields practical remedies. The most effective is to change the architecture of the generation pipeline so the model is forced to ground its claims in retrieved documents: Retrieval‑Augmented Generation (RAG) mixes a non‑parametric retrieval step with the generator so that when the model emits a citation it is constrained to point to a document that was explicitly retrieved and can be verified. RAG and dense retriever approaches have repeatedly shown that coupling retrieval to generation materially reduces factual errors in knowledge‑intensive tasks. Other useful interventions are hard: constrain decoding so citation tokens must match entries in a vetted bibliography, reject outputs that include fabricated DOI patterns, or post‑hoc verify every cited identifier with an external API before returning the result to a user. These are boring but effective engineering controls. (arxiv.org) Instruction‑tuning and RLHF (the methods that make assistants “follow instructions”) help too — they push models to be truthful and cautious — but they’re orthogonal. RLHF changes the reward landscape so the model prefers responses that humans judge useful, which lowers some hallucinations; it doesn’t, however, give the model a reliable mechanism for checking whether a paper with a particular title actually exists. That requires external evidence. The upshot is clear: hallucinated citations are not a failure of will or an adversarial trick, they are a byproduct of an optimization objective plus unconstrained decoding. Once you accept that, the path forward is engineering. (arxiv.org) The practical implications are immediate. For researchers and editors: assume every AI‑generated citation is unverified until you or a tool checks it. For platform builders: don’t ask the model to invent references; instead provide a retrieval oracle, restrict outputs to fetched IDs, and run automated verifiers as a last mile. For conference organizers and publishers: add automated citation verification at submission. If you let models write references and then accept them without verification, you will pollute the literature and then feed that polluted literature back into future models — a self‑reinforcing loop. The engineering work is straightforward, measurable, and urgent; the alternatives are not. Models will continue to get better at surface plausibility. That won’t by itself stop fake citations because what we want is not plausibility but provenance. The fix is not polishing the model’s prose; it’s building systems that force provenance into the generation loop. Do that, and the phenomenon illustrated by the X screenshot stops being a puzzle and becomes a solved integration task: reliable retrieval, constrained decoding, and verification pipelines. That’s not a philosophical victory. It’s the only practical one that protects the integrity of scholarship.