Making an AI assistant quote only real numbers: the faithfulness check behind our in-app helper
Table of Contents
When an AI assistant answers "which post did best this month", the worst failure is not a dull answer. It is a wrong view count. For Xiaomai, the in-app helper in MindThread 2.0, we built four layers: code decides what to query and the model never gets tools; data is computed before the model sees it; once the answer is back, code checks every number, quote, @handle and URL against the data; anything that does not match goes back for one retry, and if it still fails, the assistant says plainly "I couldn't find that number". Every answer also carries a collapsible "Sources" section listing which data it looked at.
This post walks through each layer, why it is there, and what it still cannot catch. Xiaomai is still on the 2.0 preview site and not in production yet, so every number below comes from preview testing.
Why isn't the prompt enough?
The obvious first step is to write "use the numbers from the data" in the system prompt. We did, in detail: copy numbers digit for digit, never compute totals, ratios, percentages or multiples, never round, never turn 12034 into "about 12k", and say so when something is missing.
A prompt is a request, not a guarantee. Models love to help with arithmetic. Give one two view counts and it will add "that's twice the average". Give it 12034 and it decides "about 12k" reads better. Each one is well meant, and each one is a number the user has no way to verify. The prompt is the first line; code has to stand behind it.
Layer 1: the model never gets tools
Xiaomai has six data sources, each a fixed read-only query: connected accounts, the monthly report, post performance for the last 30 days, patrol activity for the last 7 days, the schedule for the next 14 days, and comments received in the last 30 days.
Which ones run is not up to the model. A pure function called pickTools decides by keyword. "Which post", "best" or "what time" pulls post performance; "schedule" or "queued" pulls the schedule; "comments" pulls comments. The account list always comes along, and when nothing matches we default to the monthly report plus 30-day post performance.
We skipped function calling on purpose. The question comes from the user and the posts and comments come from third parties, so letting the model choose calls lets that text reach the queries. With keyword routing, no instruction in the question can trigger a tool. Every query also runs only with the user ID from the verified login; team members query the workspace owner's data. A test statically checks that every query is scoped to that user and that the tool layer contains no writes.
The downside is less flexibility. A question like "how did @someaccount do in October" is beyond v1 until we add a tool for it.
Layer 2: compute first, let the model read
Models are weakest at arithmetic, so we do not ask them to do it. Engagement rate, average views, and the breakdown by hour and weekday are computed on the server and placed in the data. The model copies them.
The monthly report tool calls the same function the in-app report page uses, so Xiaomai's numbers match what members see when they open the report themselves. No "the assistant says 3000, the report says 2986".
Posts and comments in the data are third-party text and may hide "ignore all previous instructions". A fenceSafe helper strips boundary markers like </data> and triple backticks, the question gets the same treatment, and the system prompt says to ignore any instruction inside the data. Even if the model is fooled, it has no ability to write anything.
Layer 3: audit the answer, item by item
This is the core. The function is answerIssues(answer, allowed), where allowed is the exact data we gave the model plus the user's question. Everything in the answer must be found there. An empty array means the answer is faithful.
It catches six kinds of things:
- Arabic numerals, normalized first: 1,234 equals 1234, and 3.0 equals 3.
- Chinese multiples and fractions such as 七倍 (seven times) or 九成 (ninety percent). The data only uses Arabic numerals, so these mean the model did its own math.
- Large numbers written in Chinese characters, such as 一千二百 (1,200).
- Quoted excerpts that are still 6 characters or longer after spaces and quote marks are stripped. They must appear in the data exactly, commas and periods included.
- @handles.
- External URLs. Xiaomai's own page links are relative paths and do not count.
Two relaxations keep normal answers from being blocked: integers of 12 or less are skipped, because "top 3 posts" and "November" (month 11) are everywhere; and Chinese idioms that happen to contain number characters are not treated as numbers.
Here is a trimmed version you can use as is. The checks are the same as the ones we run; only the parts tied to our product are left out:
// answer-issues.mjs: things in the answer that the data cannot back up; [] = faithful
const norm = (s) => String(s || '').normalize('NFKC').toLowerCase()
const squash = (s) => norm(s).replace(/[\s「」『』"'“”‘’…]+/g, '')
const numsOf = (s) => [...norm(s).matchAll(/\d+(?:[.,]\d+)*/g)].map((m) => m[0].replace(/,/g, ''))
const URL_G = /https?:\/\/[^\s<>"'）)】」』，。！？]+/gi
const CN_RATIO_G = /[零〇一二三四五六七八九十百千萬兩半]+\s*(?:倍|成|%|％)/g
const CN_BIG_G = /[零〇一二三四五六七八九兩]?[十百千萬億]+[零〇一二三四五六七八九十百千萬兩]*/g
const IDIOMS = new Set(['千萬', '萬一', '萬萬', '百分百', '十萬火急'])
const MENTION_G = /@[A-Za-z0-9_.]{2,30}/g
const QUOTE_G = /[「『“"]([^」』”"\n]{6,200})[」』”"]/g
const SMALL_INT_FREE = 12
export function answerIssues(answer, allowed) {
const out = []
const add = (x) => { const v = x.trim(); if (v && !out.includes(v)) out.push(v) }
const src = String(allowed || ''), srcN = norm(src), srcSq = squash(src)
const srcNums = new Set(numsOf(src))
const body = String(answer || '').replace(URL_G, ' ')
for (const u of String(answer || '').match(URL_G) || []) if (!srcN.includes(norm(u))) add(u)
for (const n of numsOf(body)) {
if (srcNums.has(n)) continue
if (/^\d+$/.test(n) && Number(n) <= SMALL_INT_FREE) continue
if ([...srcNums].some((s) => Number(s) === Number(n))) continue // 3.0 matches 3
add(n)
}
for (const m of body.matchAll(CN_RATIO_G)) {
const after = body.slice(m.index + m[0].length, m.index + m[0].length + 2)
if (!/^不變/.test(after) && !srcSq.includes(squash(m[0]))) add(m[0])
}
for (const m of body.match(CN_BIG_G) || []) if (m.length >= 2 && !IDIOMS.has(m) && !srcSq.includes(squash(m))) add(m)
for (const m of body.match(MENTION_G) || []) if (!srcN.includes(norm(m))) add(m)
for (const m of body.matchAll(QUOTE_G)) {
const q = squash(m[1])
if (q.length >= 6 && !srcSq.includes(q)) add(`「${m[1].slice(0, 24)}」`)
}
return out
}
If your assistant only answers in English, you can drop the two Chinese patterns and the idiom set; the number, quote, handle and URL checks work the same. Note that English multiples like "twice" or "3x" are not caught by this version (3 is under the small-integer cutoff), so add a pattern for them if your model likes to write those.
We ran it on fake data (one post with 2380 views and a 4.6 engagement rate, 12034 total views). A faithful answer returns an empty array. "3,100 views" is caught as 3100. "Seven times higher" written in Chinese is caught. "About 4.7%" is caught as 4.7. "About 1.2 萬" (12k) is caught as 1.2. An invented quote and someone else's @handle are both caught. A sentence made of Chinese idioms plus "the top 3 posts" returns an empty array.
One detail matters: earlier turns of the conversation are given to the model as context only and are never added to allowed. Otherwise someone could forge a previous turn and launder its numbers into "valid" ones.
Layer 4: retry once, then say "not found"
If answerIssues returns anything, the list goes back to the model for one retry, roughly: "These items from your last answer are not in the data: 3100, 七倍. Use only numbers and posts that appear in the data exactly; say you could not find anything else." At most 8 items are passed back.
If the retry still has unmatched items, we drop the model's answer and return a fixed line: "I couldn't find that number. Try asking another way, or open Insights for the full numbers." For now the server only records the unmatched items with console.warn. Measuring which tools each question picked and how often retries happen is the next step.
A retry costs one more model call, but only when something is wrong. We would rather pay for that call than let a member make a decision based on a wrong number.
The Sources section: let users check for themselves
Under every answer there is a collapsible "Sources" section listing which queries ran, the range of each, for example "Post performance, last 30 days (10 posts with data)" or "Patrol, last 7 days", and which ones came back empty.
It serves the user and it serves us. A user who sees that Xiaomai only looked at the schedule knows it is not talking about post performance. When we saw a question hit the wrong tool during testing, we knew which routing keywords to add.
What did testing catch?
There are 80 unit tests, run before every build, covering tool selection, action detection, faithfulness and data isolation. On the preview site we logged in with a test account and asked a full round of questions: 24 checks, all passing. A spot check: the five 9 p.m. posts had 2380, 4120, 3560, 2890 and 1980 views, and the average is the 2986 in the data. Engagement (96 + 14) ÷ 2380 is about 4.62%, the data stores 4.6, and Xiaomai copied 4.6.
Three problems were fixed during those live questions. Markdown asterisks showed up raw in the panel, so answers are now cleaned to plain text before display. "What should I post next week" got "not found" because the data had no "advice" field, so the prompt now asks the model to derive advice from strong and weak posts, with any cited number still checked. Comments left by browser patrol on the member's behalf were described as "you replied manually", so the data now carries field notes.
What can't it catch?
At least three things, honestly:
- Integers of 12 or less are skipped, so a wrong "3 posts" gets through.
- Paraphrased post content without quotation marks gets through.
- Bad reasoning gets through. Every number can be right while the conclusion is off, like generalizing one strong post to a whole time slot.
So the faithfulness check stops fabrication, not bad judgment. For the latter we rely on the Sources section for people to see for themselves, and on sampling later.
If you are building RAG or a data assistant, RAG's worst failure is a confident wrong citation covers the other side of the same problem: a citation that looks authoritative but does not match the source.
Xiaomai is in preview in MindThread 2.0. v1 is read-only; actions like scheduling or changing a persona wait for v2, where each one needs explicit confirmation. The faithfulness check is not tied to our product, so the code above should drop straight into your own assistant.
FAQ
How do I stop an AI assistant from making up numbers when it answers data questions?
Do not rely on the prompt alone. We use four layers: code decides what to query by keyword, so the model never gets tools; data is computed before the model sees it, so the model only interprets; after the answer comes back, code checks every number, quoted excerpt, @handle and URL against the exact data we gave the model; anything that does not match goes back to the model as a list for one retry, and if it still does not match, the assistant replies "I couldn't find that number".
Why not let the model pick its own queries with function calling?
The question is written by the user, and the data contains posts and comments written by third parties. If the model decides what to call, that text gets a chance to steer the queries. With keyword routing, nothing written in the question can trigger a tool, and every tool only queries with the user ID from the verified login. The cost is flexibility: when no keyword matches, we default to the monthly report plus the last 30 days of post performance.
Won't the faithfulness check block perfectly good answers?
It will unless you relax it in a few places. We skip integers of 12 or less, because post counts, ordinals and months come up constantly; Chinese idioms that contain number characters do not count as numbers; 1,234 and 1234, or 3.0 and 3, are treated as the same value. The price is that a wrong "3 posts" slips through, and we accept that trade-off.