Why "Don't Make Up Numbers" Doesn't Work as an LLM Instruction

Tell a model "only state numbers you're certain about" and it will comply exactly as well as it complies with every other instruction embedded in a prompt: most of the time, until the one time it doesn't, with no way to tell which response that was without checking. An instruction is a request the model can silently ignore. A grounding guard is code the model's output has to pass through regardless of what the model decided to do, and that difference is the entire reason one of them is a security property and the other is a hope.

The failure mode isn't obviously wrong

The dangerous hallucination isn't the model claiming something absurd — that gets caught immediately. It's the model narrating a real, structured result and getting one figure inside it wrong: a percentage off by a plausible margin, a count close enough to the right one that nobody double-checks it, a total that reads exactly like something the system would say if it were true. Fluent, structured, contextually appropriate, and unverified — that's the output a grounding guard exists to catch, because it's the one a human reviewer is least likely to.

What a grounding guard actually validates

Before generation, the deterministic engine — the part of the system that actually queried the database, ran the calculation, or fetched the record — produces its own fact set: every number and named quantity it can stand behind, computed the ordinary way, with no model involved. After the model drafts its response, the guard extracts every numeric claim in that draft and checks each one against the fact set the engine actually produced for this exact request. A number that matches passes through. A number that doesn't — one the model introduced, rounded past an acceptable tolerance, or attached to the wrong quantity — fails, and the response doesn't ship as drafted.

Why a percentage and a raw count can't share a bucket

The naive version of this check flattens every numeric token into one pool and asks "does this number appear anywhere in the facts." That's how a hallucinated percentage slips through. Take a concrete case: the deterministic engine computed a raw count — say, twelve — somewhere in the response, the model wrote "12%" instead of plain "12," and a check that isn't unit-aware sees the digits match and calls it grounded, when the engine never computed that quantity as a percentage of anything at all. Every extracted claim has to carry its unit — percentage, currency, plain count, ratio — and get checked only against the subset of the fact set computed in that same unit. A raw count of twelve and a percentage of twelve are different claims about the world, and treating them as interchangeable because they share a digit is exactly the gap a careless implementation leaves open.

The fallback is deterministic copy, not a second roll of the dice

When a claim fails the check, re-prompting the model and hoping for a better answer just spends latency on the same failure mode — nothing about a second generation makes the model more likely to know a number it didn't know the first time. The guard instead substitutes deterministic fallback copy: a template sentence built directly from the fact set the engine already computed, with no model-generated text anywhere near the numeric claim. The user still gets an accurate answer; it's assembled by code that can't hallucinate instead of prose that might.

Keeping the CI check honest

A grounding guard is only as good as the assumption that the thing verifying it in CI actually matches the thing running in production, and that assumption breaks quietly the moment someone patches the runtime extraction logic — to handle a new currency format, say — and forgets a CI fixture test was its own hand-rolled copy of the same regex. The fix is to make that impossible rather than disciplined: the CI evaluation suite imports the exact extraction and matching functions the runtime guard calls, and runs them against a fixture set of adversarial model outputs — plausible wrong numbers, right numbers in the wrong unit, rounding at the edge of tolerance — on every change to either. There's no second implementation to drift out of sync with the first, because there's only one implementation.

What this means for an LLM security audit

An auditor asking "how do you know the model didn't make that up" wants a mechanism, not an assurance. The answer is the extraction and matching code itself, the fact set it checked against for a specific logged request, and the CI fixtures proving the check catches the failure modes it claims to catch — evidence that exists independently of anyone's confidence in the model, because none of it relies on the model at all.