A Wrong Authoritative Memory Is More Dangerous Than No Memory: How to Check Your AI Agent's Knowledge Anchors
Table of Contents
A few months ago, in Multi-Agent Coordination Architecture: How to Keep 4 AI Agents from Stepping on Each Other, I held up "use markdown files as shared memory" as the most central design in the whole architecture. This post exposes a hole in that memory layer itself: the authoritative definition built specifically to suppress hallucination, and injected at the highest priority, had recorded UltraProbe's 25 defense vectors as 12. A wrong authoritative memory is more dangerous than no memory at all.
Chapter 8 of Agentic Design Patterns covers memory management. The core idea, in one line: an agent should actively maintain its own long-term memory and detect the gaps in it, not passively store a pile of conversations and call it done.
We read it and nodded. The public Q&A of our Lobster agent fleet has a "knowledge anchor" layer: hard definitions for high-risk terms, injected into the prompt at the highest priority, forcing the model to copy the authoritative definition instead of making things up. Memory management: implemented. Then we audited that memory layer itself. The truth of this chapter: the one memory built specifically to fix hallucination had got a number wrong itself. And a wrong authoritative memory is more dangerous than no memory.
How this anchor came about
On 2026-06-30, someone asked Lobster "what is AVS". It answered with complete confidence: AVS = Agent-based Virtual System. The whole line was invented. The correct answer is AI Visibility Score (SEO×0.35 + AEO×0.35 + AAO×0.30). That day, retrieval hit a blog passage that mentioned AVS, but the passage only mentioned it and never defined it, so the model filled in an expansion that sounded about right. The fallback rule "if you don't know, say you don't know" could not hold back the model's urge to force out a plausible answer.
The fix was a concept anchor: inject "AVS = AI Visibility Score" into the prompt at the highest priority, with an added line saying "do not use general LLM knowledge or infer from sources". From then on the AVS entry answered correctly, and it looked clean and tidy.
The day we audited the anchors themselves
We laid out the whole anchor file and checked each entry against the facts. The AVS entry was right. The problem was the entry next to it. The anchor said "UltraProbe = AI security scanner (12 defense vectors)", but another authoritative record of our own said 25 (12 LLM-era vectors + 13 agent-era vectors). Both carried the label "authoritative", one had roughly double the number of the other for the same fact, and working out which was right needed a separate check.
The truly frightening part is not the size of the gap. It is where the number was placed. It was not "reference material". It was a hard injection at the highest priority in the prompt, bound to "do not use general knowledge or infer from sources". Right or wrong, the model is forced to copy it and forbidden to doubt it. You meant it to suppress hallucination, but if the number is wrong, you have in effect issued a hallucination with an official stamp on it, and switched off whatever hesitation the model had. With no memory, the model at least says it is unsure. With one wrong authoritative memory, it repeats the wrong one with total certainty.
(The AVS story also came up in Chapter 19, but that time the LLM judge hallucinated on its own and marked a correct answer as wrong. This time the memory layer froze a number into an "authoritative definition". Same acronym, two different diseases in two different layers.)
Spend a few minutes checking your own authoritative memory
01 Has every number in your anchors been checked against the official source? Pull every "numeric fact" out of your knowledge base: vector counts, prices, account counts, version numbers, and check each one against its single source. The question is not "is this memory there", it is "is this memory right".
# Pull the numeric definitions out of the anchor file and check each one against its source
grep -noE '[0-9]+ ?(個|防禦向量|向量|帳號|模板|%|元)' lib/rag-concept-anchors.js
Red flag: the same fact has different numbers in two "authoritative" files, and you cannot say which one is right.
02 Is your memory the "injected at highest priority, doubt forbidden" kind? Then it has to be right even more Memory comes in two kinds: one the model can consult and overrule, and one that is hard-injected with explicit orders to copy it. The second kind is more lethal when it is wrong, because it switches off the model's doubt along with everything else. Go through your prompts for sections like "highest priority, no inference". Every word there must have been checked.
# Find high-authority injected sections in prompts that say "must follow this, no doubting"
grep -niE '最高優先|highest.?priority|禁止|must use|do not infer' lib/*.js prompts/*
Red flag: there is an injected section saying "must use this definition, no doubting", but nobody has ever checked its content word by word.
03 Is anything detecting which high-risk terms have no authoritative definition at all? Memory management is not only about storing the right things. It also means detecting gaps. The root cause of the AVS hallucination back then was that the term had "no anchor" at the time. List every term where a wrong answer is costly, check each one for an authoritative definition, and fill in the missing ones (hand that to a human, and do not let the system change live memory automatically).
# Ask the real lookup function whether each high-risk term has an anchor, and list the gaps
node -e 'const{getAnchorDefns}=require("./lib/rag-concept-anchors.js");
["AVS","UltraProbe","定價","退費"].forEach(t=>getAnchorDefns(t)||console.log("缺 anchor:",t))'
Red flag: you can list ten terms where "a wrong answer means trouble", but nothing checks whether all of them have an authoritative definition.
What we actually did
We wrote memory-reflect.sh, strictly read-only, which does two things on every run. First, it uses the real lookup function to ask whether each of 17 high-risk terms has an anchor, and lists the gaps. That is how, on 4 July, UltraGrowth, Quartz, DropPin and Atlas, terms that had been running without any definition, got their anchors. Second, it reads the semantic heartbeat score from Chapter 19 and treats a drop as a signal of knowledge decay. The deliberate design is that gaps are only listed and handed to a human to fill. It never changes the live RAG automatically, because letting a system rewrite its own memory amounts to letting hallucination reproduce itself.
As for the 12-or-25 entry, the honest answer: reflection can catch whether an anchor is missing, but not whether it is right. Checking numbers back to their source still takes a person sitting down and going through the entries one by one. That is the hardest half of memory management, and the half most easily skipped.
Authoritative memory health checklist
Run this against your own memory or knowledge base:
- Has every number in your anchors (vector counts, prices, versions) been checked against the official source?
- Is the same fact scattered across several "authoritative" files, with numbers that conflict?
- Is there a "highest priority, no inference" injection whose content has never been checked word by word?
- Is anything detecting whether high-risk terms are missing an authoritative definition?
- Does new memory go in only after human review, or does the system quietly change live memory on its own?
- Can you tell that "is this memory there" and "is this memory right" are two different things?
This chapter in four sentences
- Memory management is not just storing conversations properly. It is actively maintaining correct memory and detecting gaps. A gap makes the model make things up; a wrong memory makes it make things up with confidence.
- A wrong authoritative memory is more dangerous than no memory. Without memory the model hesitates; with a wrong memory it is categorical.
- Memory injected at the highest priority, with the model forbidden to doubt it, is the most powerful kind and the most lethal when wrong. Every number there must be checked.
- "Is this memory there" and "is this memory right" are two different things. Reflection systems can usually only check the first; the second needs a human to check by hand.
Source location: ~/.openclaw/scripts/lib/rag-concept-anchors.js (the AVS concept anchor + the gaps filled by memory-reflect on 2026-07-04), ~/.openclaw/scripts/memory-reflect.sh (read-only memory reflection: checks anchor coverage + reads the Chapter 19 heartbeat score, with gaps handed to a human to fill). When this post was written (2026-07-06), the UltraProbe entry said 12 defense vectors, our own vector ledger said 25, and the two "authoritative" records did not reconcile. On 2026-09-26 the anchor was changed to 25.
This is part of the Agentic Design Patterns × Lobster Fleet series. We follow the book to systematise a solo company's AI agent fleet, then run an adversarial audit on ourselves. Every chapter we claim to have implemented gets verified again, and the investigation and the fix are written up as steps you can run. The credibility of this series comes from our willingness to publish our own failures.
FAQ
Why is a wrong authoritative memory more dangerous than no memory?
Because authoritative memory is usually injected into the prompt at the highest priority, bound to 'do not use general knowledge or infer from sources'. Right or wrong, the model is forced to copy it and forbidden to doubt it. With no memory, the model at least says it is unsure. With one wrong authoritative memory, it repeats the wrong one with total certainty.
How do I check whether the numbers in my AI agent's knowledge anchors are right?
Pull every numeric fact out of your knowledge base, such as vector counts, prices, account counts and version numbers, and check each one against its single source. The question is not 'is this memory there', it is 'is this memory right'. The red flag is the same fact having different numbers in two 'authoritative' files, and you cannot say which one is right.
What is a knowledge anchor (concept anchor)?
A knowledge anchor is a hard definition for a high-risk term, injected into the prompt at the highest priority to force the model to copy the authoritative definition instead of making things up. For example, a model once invented 'AVS = Agent-based Virtual System'. The fix was to inject 'AVS = AI Visibility Score' with an added line saying 'do not use general LLM knowledge or infer from sources'.
How do I detect which high-risk terms have no authoritative definition?
List every term where a wrong answer is costly, use the real lookup function to check each one for an anchor, and hand the gaps to a human to fill instead of letting the system change live memory automatically. We do this with memory-reflect.sh, a strictly read-only script that checks whether each of 17 high-risk terms has an anchor.
Can an automated reflection system catch wrong numbers in memory?
Usually not. Reflection can catch whether an anchor is missing, but not whether its content is right. Checking numbers back to their source still takes a person going through the entries one by one, which is the hardest half of memory management and the half most easily skipped.