rot
a note contradicts itself, a citation points at a source that is gone, or a body drifts past its own changelog.
self_contradiction · dangling_citation · duplicate_value · temporal_mismatch · stale_body
open spec · mit · v1.1 · mechanical, no llm judge
rotbench
locomo and longmemeval measure whether your agent remembers. rotbench measures whether what it remembered can be trusted: that it was not corrupted, poisoned, or silently rewritten.
why this exists
the risk has a name. it does not have a number.
owasp put memory poisoning in its 2026 top 10 for agentic applications as ASI06, memory & context poisoning. owasp's own agent memory guard project describes itself as "the reference implementation that the risk definition currently lacks."
so the category is named and the attack classes are enumerated, but there is no accepted way to score a store. without a number, "our agent memory is protected" is not a falsifiable claim. every existing memory benchmark measures recall or qa against an answer key people keep disputing. rotbench does not read against a key at all. it checks the store against itself.
threat model
three ways memory stops being trustworthy
tamper
a note is edited after the store was attested. a post-write rewrite, not a legitimate update.
provenance_integrity (detached ed25519 signature over the canonical markdown state)
poisoning
untrusted input injects a memory with no real source. an agent writes a distilled fact carrying no resolving citation.
uncited_claim (cite-or-drop)
untrustworthy, note, not merely incomplete. a store missing a fact is a recall problem. a store confidently holding a fact that was never true is a different and worse problem, and it is the one nobody scores. that is the whole subject of memory integrity: detecting agent memory poisoning, tampering and rot in a store you already have, rather than hoping the write path caught it.
mapping
rotbench against owasp ASI06
| ASI06 attack vector | what it means | rotbench checks |
|---|---|---|
| direct injection | a malicious or buggy agent writes false information straight into shared memory. | uncited_claim, duplicate_value |
| indirect injection | an agent processes untrusted external data and stores the result as trusted memory. owasp notes detection here requires provenance tracking. | uncited_claim + dangling_citation, which are provenance checks by construction |
| gradual erosion | the "sleeper agent": behave normally to build trust, then inject poisoned memories later, so the payload is decoupled in time from the write. | temporal_mismatch, stale_body, updated_ahead, citation_source_stale, stale-signature warn |
where rotbench does not overlap ASI06
ASI06's mitigation guidance is mostly preventive and runtime: encryption at rest, least-privilege access, session isolation via partition keys, not persisting raw tool results, immutable system prompts, cache ttls. rotbench does none of that. it is detection over a store at rest, and it is scored rather than enforced.
these are complementary layers, not substitutes. agent memory guard blocks a poisoned write as it happens. rotbench tells you whether what is already stored can be trusted. it also uses a different primitive: agent memory guard validates with sha-256 baselines, rotbench uses a detached ed25519 signature over the canonical markdown state. same goal, different mechanism, and a store using either should be able to report a score.
the contract
the score is a formula, not an opinion
score = max(0, round(100 * clean_notes / total_notes) - warn_penalty)
warn_penalty = min(15, round(100 * warns / total * 0.3))
verdict INTACT iff (no fails) AND score >= 85a note is unclean if it carries any fail-level finding. a store with any fail is stamped ROT DETECTED regardless of score, because one contradicting note is still rot. exit code is part of the contract: nonzero means rot, so it gates ci and cron like a test suite.
every check is a deterministic predicate over the store's own bytes. no llm judge, no api key, no trust required. an llm asked to judge integrity can hallucinate integrity.
reproduce it
run it on your own memory in two lines
pip install homestead-memory
hsm verify --deep --json /path/to/your/memory
the checks assume only a folder of markdown notes with yaml frontmatter, optional
## Changelog lines, and optional (source: rel/path.md)
citations on extracted claims. deliberately minimal, so most markdown memory and
pkm layouts qualify with trivial adaptation.
to score a tool that is not ours, either export its memory to that layout and run the reference scorer, or implement the check families and the exact formula above and emit the same json shape. both count. the spec carries the full json schema and all 18 check families.
break it. we dare you.
a score is only credible if it survives adversaries
build a store that is obviously rotten to a human but scores INTACT, or an intact one that false-positives, and open an issue with the fixture. we merge the fixture and fix the check. the public break-it scoreboard is part of the repo.
trust you can inspect, and break, is the one thing a closed vendor cannot ship.
the scoreboard →our own numbers, unflattering parts included
we report all three, not just the good one
99.4 / 100
rotbench integrity
self-scored on our own vaults. so break it, the harness is open.
85%
retrieval recall@k
full 500-question LongMemEval set. reader-independent.
52.8%
qa accuracy
honest and mid. we publish the failed runs too.
recall and qa are honest but mid, and we say so. rotbench is the number that is
actually ours, because nobody else scores the integrity of the store itself. the
reporting convention we would like to see adopted:
RotBench <score>/100 (<fails> fail / <warns> warn, n=<notes>, v1.1, deep=<bool>),
published alongside any recall number. our full run
history is public, failed runs included.