Your agent stack doesn’t tell you when it’s wrong
I run an AI coding setup I built myself over the last two years. Memory server, code graph, spec checks, the whole thing. This month I finally did what I’d been putting off: I tested it head to head against the stack people actually install from GitHub, on the same four repos, with the same questions, and had separate verification agents check every single answer against the source with grep.
Both stacks gave wrong or outdated answers. Neither warned the user. That’s the finding. Who “won” matters less than how each one failed, because the failures were opposite, and both were invisible from inside a session.
The test
Stack A was assembled from the GitHub registry: a code-graph MCP server, a semantic code toolkit, a document knowledge graph, a notes vault, a session memory plugin. Stack B was mine.
The corpus: four real repositories. A Python codebase with 1,138 files, a Rust graph engine, a mixed Python/JS repo, and one repo that is mostly Shell and Markdown. I picked that last one on purpose because code-graph tools say they don’t cover it, and I wanted to see what happens when a tool gets asked about territory it doesn’t map.
Each stack ingested all four repos. Then, in fresh headless sessions, an agent had to answer three questions per repo using only what the stack had stored. Reading the source directly was forbidden; the point is to measure the stores, not the model’s ability to grep on the spot. The questions: what are the entry points, which file has the highest fan-in, and what do the docs say this project is for. Afterward, independent read-only agents scored every claim by checking it against the live repo.
One protocol rule turned out to matter more than the rest: smoke-test every component before measuring anything. My first full run produced garbage on both sides. A tool pinned to the wrong project. A pre-built graph that was empty. A server that wasn’t registered in the session at all. That was all setup, not capability. I kept the data, labeled it as the default-config experience, and re-ran after fixing the setups.
Wrong, with citations
The registry stack’s code graph answered the fan-in questions confidently. On two of four repos it was flat wrong.
On the Shell/Markdown repo, the actual most-referenced files were a coding-standards document (173 references) and a shell tool (171 references). The graph named a Python file instead, off by a factor of about 17, because a code-only index literally cannot see references written in markdown or shell. On the Rust repo it named the wrong module and contradicted a note it had itself written during ingestion.
Nothing in those answers hinted at a problem. They came with file paths and structured citations. If you didn’t already know the repo, you’d accept them. That’s the part that should bother the millions of people using these tools daily: the failure doesn’t look like a failure.
Right, but months old
My stack failed differently. Its answers traced back to real documents. In the most heavily audited session, every quote it produced was verified real, zero fabrication. The problem: the store was serving snapshots of the project docs that were two to four months behind the repo’s HEAD.
It cited a module count that the repo’s own docs had retracted nine days before the test. It described a storage default that had changed months earlier. All of it once true, none of it current, and nothing in the output said so. No timestamp, no age, no “captured on”. A stale fact and a fresh one look identical when neither carries a date.
One thing my stack did that I want to keep: when a backend was unreachable mid-session, it said so, reported the zero count, and refused to guess. An honest “I can’t answer that” was the rarest behavior in the whole experiment.
The scores
| Measure | Stack A (registry) | Stack B (mine) |
|---|---|---|
| Retrieval probes (12) | 8 correct, 2 partial, 2 wrong | 2 correct, 8 partial, 1 wrong, 1 honest refusal |
| Fabrication | none detected in repo probes | 1 fabricated quote in one session; zero in the strictest-scored one |
| Component probes | 4.5 / 5 | 4.5 / 5 |
| Session health | first run invalidated by config defects | 2 of 4 probe sessions ran degraded (flagged, kept) |
So yes, on the raw matrix my own stack scored worse, and I’m publishing that number anyway. Two caveats belong next to it. Two of its four sessions ran with a missing backend or an unregistered server, which measures session health rather than what the store knows. And the report deliberately refuses to name an overall winner, because averaging capability with session luck would produce a number that means nothing. I didn’t build this rig to protect my stack from its own data. If the test couldn’t hurt me, it couldn’t tell me anything either.
If you run an agent stack yourself
Four things I’d take from this even if you never look at my tools.
- Smoke-test the actual session before trusting it. Most of what went wrong in this benchmark, on both sides, was assembly: dropped tool names, mispinned projects, absent servers. Nobody tests the assembly.
- If a tool serves pre-built knowledge, ask how old it is. If it can’t answer, assume stale. A store without a freshness policy loses to plain grep, and grep is free.
- Ask your code-intel tool what it did NOT index. Coverage blindness produces the worst answers in the dataset: wrong, specific, and cited.
- Prefer tools that refuse over tools that guess. The single best behavior I observed was a disclosed blind spot.
What I fixed on my side
The staleness was my failure, so the fixes are mine. Every injected memory now carries its age and source (the data was already being fetched, then dropped at render, which stings). File-derived memories get a content hash, and a post-commit hook re-verifies them: if the source moved, the memory is superseded or demoted instead of served. And doc prose went into the search index, because my own earlier benchmark had already shown grep beating my recall there, 1.00 to 0.83, and I published that loss too.
The coverage blindness belongs to the other stack’s architecture. Their maintainers can decide what to do with it; the protocol, probe definitions, per-run JSON and scoring transcripts are public, so anyone can re-run the whole thing against their stack or mine. That’s the point: a benchmark the vendor can’t lose doesn’t tell you anything.