Rephrase the Question and the Citation Disappears: The Hardest RAG Failure to Find

Table of Contents

1. Same question, different wording, opposite answer
2. Why the test missed it
3. The fix: stop long articles from diluting themselves
4. A bonus find: the evaluation harness was lying to us
5. Spend a few minutes auditing your own retrieval
6. What we actually changed
7. Postscript: the sentinel was fake too
8. Checklist
9. Four sentences for this chapter

Our legal question answering has a regression suite: ten sentences an advisor would actually type, run after every retrieval change, checking that the correct statute lands in the top six. It had been green for weeks.

One day I aligned the test script's parameters with what production actually does. It went red immediately, on the highest frequency question we have.

Digging in, the reason it had been green was uncomfortable: the test had been measuring a more permissive world than the one our users live in.

#Same question, different wording, opposite answer

One of the most common questions advisors ask is whether a life insurance death benefit counts toward the taxable estate. Two ways to phrase it:

Phrasing	Rank of the key statute	Statutes in the top 12
Death benefit with a named beneficiary, does it count toward the estate?	5th	3
After the client dies, does the death benefit the beneficiary receives get taxed as estate?	21st	0

Under the second phrasing the other relevant statute sat at rank 35.

The ranking itself is not the point. What matters is what happens after it falls out.

Our retrieval has a floor: at least two of the top six slots are reserved for statutes, and if fewer are present we backfill from the twelve nearest candidates. Under the second phrasing there were no statutes at all in the top twelve, so the floor had nothing to draw on.

The model therefore saw six reference documents, every one of them a tax authority interpretation, and every one of those interpretations describes a case where the court ruled the benefit was taxable.

Under our rule that the model may only cite what the tools returned, this flips the answer: the exception gets presented as the rule. The advisor asks whether it is taxable and receives a well sourced looking answer that only shows the taxable side.

#Why the test missed it

The header comment of our regression script says it plainly: if the parameters drift from production, you are not measuring production. We wrote that down. Two parameters had drifted anyway.

The damaging one was the backfill range. Production backfills only from the twelve nearest candidates and would rather leave a slot empty than reach further, a limit that exists specifically to stop an unrelated statute being dragged in from far away to fill a quota. The test script backfilled from the entire candidate pool, all forty eight.

So in the test, the statute at rank twenty one got pulled in anyway and the check passed. In production nothing was within reach and the advisor got a reversed answer.

Same retrieval logic, looser test version, permanently optimistic results.

The second drift: this script's embedding call had no retry for network level errors. Calling an external API from a laptop fails intermittently, and one failure killed the whole suite on the first case, looking exactly like broken retrieval. That one does not create false green, but it does make people stop running the suite, which amounts to the same thing.

#The fix: stop long articles from diluting themselves

Once diagnosed, the real cause was vector dilution.

The estate tax statute in question has thirteen subsections covering donations, cultural artefacts, copyright, household items, professional tools, protected forest, insurance proceeds, land opened for public passage, and more. Encode all of that as one vector and you get the average of thirteen unrelated topics. Ask about insurance proceeds and the average is not close enough.

We split qualifying long articles into subsection level chunks: each chunk is the lead in sentence plus a single subsection, embedded separately. The parent chunk keeps its full text but its embedding text becomes "lead in plus subsection headings," so it still matches "what is this article about" without competing for subsection specific queries.

After the split, the subsection that actually says "a life insurance benefit payable to a named beneficiary on the insured's death" moved from rank thirty five to rank five. I separately verified the hit was that subsection itself rather than a lucky match on a neighbouring one.

But a whole set of long articles we deliberately do not split, and one exclusion rule alone accounts for eighteen of them.

That rule counts how many times the first subsection marker appears in the text. More than once means the article has multiple paragraphs each carrying its own subsection run, and splitting would number them wrongly. A wrong subsection number is worse than none, because advisors copy it verbatim.

The two longest articles in our entire corpus fall exactly here. One runs over three thousand characters with the marker appearing six times, and it turns out to be a category structure rather than a subsection structure. The other is two thousand characters with the marker twice. The two longest articles are the ones we must not split, and that is the correct outcome.

There are four exclusion rules in total. The other three: non consecutive subsection numbering, a lead in sentence over three hundred characters (split those and every child is still dominated by the lead in, which achieves nothing), and fewer than three subsections or under five hundred characters overall. Together the four rules excluded sixty seven articles; forty were actually split.

#A bonus find: the evaluation harness was lying to us

After the fix I ran our exam benchmark to check for regressions. The score dropped by 0.6 points. Tracing it, one question showed "retrieval empty after the split."

I assumed the split had broken retrieval. Then I checked: the statute that question needed was sitting at rank one with a distance of 0.2086, and that law was not even in the split candidate set.

The real cause was that the benchmark's embedding call had no retry, and the calling code's catch block was empty. One network hiccup silently degraded that question to answering with no reference material at all, and the result went straight into the score.

The comment even said "on retrieval failure, degrade to baseline and mark it in the result." The marking half was never implemented.

So it was not a product regression. It was my measurement tool making the product look worse, while handing me a perfectly normal looking score every time. On a rerun the score returned to its original value with every subject identical.

#Spend a few minutes auditing your own retrieval

01 Ask the same question three different ways

Not just the phrasing you happened to use when writing the test. Use what a user would actually type, a casual version, and a formal version. Compare where the key source lands.

# Write each phrasing of the same intent as its own case, then run the suite
# (ours are hard coded in the script's CASES array, not passed as a flag)
node scripts/check-law-retrieval.cjs --verbose

Red flag: one phrasing pushes a key source out of range and you have never tested that phrasing.

02 Reconcile every parameter in the test script against production

Not by reading the comment claiming they are in sync. Compare the constants one by one: top-k, candidate pool size, per source caps, relevance thresholds, backfill range. Any one of them looser in the test and you are testing a system that does not exist.

# Put both sets of retrieval constants side by side
grep -nE 'TOP_K|MULTIPLIER|MAX_PER|MIN_|GATE|slice\(0,' api/agent.ts
grep -nE 'TOP_K|MULTIPLIER|MAX_PER|MIN_|GATE|slice\(0,' scripts/check-law-retrieval.cjs

Red flag: any number differs, or the test uses the full pool while production uses a truncated one.

03 Check whether your long documents are diluting themselves

Pull the longest chunks in your knowledge base and look at whether each one spans several unrelated topics. Those are your candidates for "cannot be found when asked about one of them."

# Longest chunks and how many subsections each contains
node -e "…sort by text.length, print top 20 with subsection marker counts"

Red flag: your longest chunks are multi topic statutes or document sections, and retrieval encodes each as a single vector.

04 Does your evaluation harness degrade silently on failure

Find every empty catch in the benchmark scripts, especially any path that falls back to "no retrieval." Those make your product look worse in your own measurements.

# Empty catches, and whether failures are actually counted
grep -nE 'catch\s*(\([^)]*\))?\s*\{\s*/\*|catch\s*\{\s*\}' scripts/*.cjs
grep -nE 'retrievalError|failedCalls|degraded' scripts/bench-*.cjs

Red flag: a comment promises marking that the code never does, or failure counts never appear in the report.

#What we actually changed

Three separate things.

The retrieval hole was closed with subsection level chunking: forty long articles produced two hundred and ninety three child chunks, plus a new cap of two slots per statute so that an article split into thirteen pieces cannot flood the top six on its own. The regression suite went from ten cases to eleven, the new one being a permanent sentinel for the rephrasing failure.

What happened to that sentinel is the next section.

The test script drift was fixed in both places: backfill range aligned with production, and the embedding call given a retry that covers network level exceptions, not just HTTP status codes.

The silent degradation in the benchmark now counts failures and prints them, and we deliberately do not remove them from the denominator. Dropping them would be selecting away unfavourable samples. The honest version keeps them and discloses that some questions answered with no reference material.

#Postscript: the sentinel was fake too

With this piece written and ready to publish, I had an independent set of reviewers go through the repository and check every number in it against the code. They found something.

That "rephrasing" sentinel case had a q string identical to case one. Byte for byte identical, with the same expected statutes, differing only in a label that read "rephrasing sentinel".

So the regression suite did go from ten cases to eleven, but the eleventh was a copy paste of the first. Eleven cases covering ten phrasings, and that intent still tested exactly one way. One extra embedding call, zero extra coverage.

And the report printed "11 of 11 advisor phrasings retrieve the correct source in the top 6" all the same.

The script's own comment on line 80 read "keep one case per phrasing: leaving only one will miss the rephrase it and it breaks class of problem again." The comment was right. The code did not do it.

An article about how testing one phrasing of an intent means testing one angle, whose headline fix commits exactly that error. Committed while I already knew the trap well enough to build a whole article around it.

The fix has two layers. First, change case one's q to a genuinely different phrasing. Second, and this is the real point: a comment asking for "these two must differ" does not stop a paste, so the suite now runs a structural check first, scanning the case list and failing loudly if any two q values match, naming which two.

[fatal] duplicate phrasings in the case list, the coverage is fake:
  - case 1 and case 9 have the same q
  CASES has 11 entries but only 10 distinct phrasings.

After the fix: 11 cases / 11 distinct phrasings, then 11 of 11 green. And the two phrasings retrieve different statutes: the formal wording surfaces the Insurance Act article, the conversational wording surfaces the Estate and Gift Tax Act article. Same intent, different phrasing, genuinely different sources come back. That is the whole thesis of this piece, evidenced this time by our own test going wrong.

#Checklist

Run this against your own retrieval:

• How many phrasings of the same intent have you actually tested
• Are any two cases in your suite secretly the same case, and would anything tell you
• When did you last reconcile the test script's retrieval constants against production, line by line
• How many unrelated topics does your longest chunk cover
• When retrieval fails, does your evaluation report say so
• Do you have a rule like "rather leave a slot empty than reach further" that the test version does not enforce

The hardest failures are not the ones where something breaks. They are the ones that only break from the angle you never tested. The index is intact, retrieval runs, scores hold, tests stay green, and the citation quietly disappears when someone phrases it differently.

#Four sentences for this chapter

• Broken retrieval does not always mean nothing is found. It can mean nothing is found when phrased that way. Testing one wording of an intent tests one angle.
• A test script looser than production returns permanently optimistic results. Reconcile the constants one by one rather than trusting a comment that claims they are in sync.
• An article covering thirteen unrelated subjects, encoded as a single vector, is the average of thirteen topics. Only subsection level chunks make any one of them findable.
• The longest documents may be the ones you must not split. The test is whether the subsection marker repeats, because a wrong subsection number is worse than none when readers copy it verbatim.
• Test suites break too. Our new rephrasing sentinel was a copy paste of case one: eleven cases, ten phrasings, and a report that still printed 11 of 11. A comment demanding the two differ did not hold; a structural check did.

---

Source location: the splitter at scripts/split-long-articles.cjs (four exclusion rules: repeated subsection markers, non consecutive numbering, lead in over 300 characters, fewer than 3 subsections or under 500 characters; the run excluded 67 articles and split 40); the regression suite at scripts/check-law-retrieval.cjs (eleven cases across eleven distinct phrasings, including the rephrasing sentinel and a duplicate-case check, with constants that must be reconciled against the retrieval constants in api/agent.ts); and the exam benchmark at scripts/bench-moex.cjs, where retrieval failures are now counted and printed, and deliberately not removed from the denominator.

This is part of the Agentic Design Patterns × Lobster Fleet series. We systematise a solo company's AI agent fleet chapter by chapter, then run an adversarial audit on ourselves. Every chapter we claim to have implemented gets verified again, and the investigation and the fix are written up as steps you can run. The credibility of this series comes from our willingness to publish our own failures.