Your Coding Agent Has Memory. Does It Actually Arrive?

I measured my own memory system. Then I caught it lying to me. Twice.

Across 1,454 eligible coding-agent edits, LayerView delivered stored file-specific knowledge on 357 of them: a 24.6% delivery rate. Only 49 of 368 edited files (13%) ever triggered a delivery, and 80% of all deliveries came from 17 files.

Both of those numbers are the corrected ones. The first version of this piece reported a sample 42% larger, because the instrument was double-counting. Before that, a path-resolution bug had made 56% of the telemetry unmatchable, and it took ten days to notice. These measure delivery and coverage, not whether the delivered context improved any outcome.

One operator, four active projects, two weeks, 52% of the data from a single project. Dataset and script.

There is a great deal of writing about giving AI coding agents memory. There is almost none about whether the memory arrives.

The same gap sits one layer up. Written instructions do not always land either, which is why Claude Code skips rules in your CLAUDE.md. Writing knowledge down and knowledge arriving are two separate events, and only the first one usually gets checked.

Those are different questions, and the second one is trivially measurable, which makes the silence odd. So I measured mine. What I did not expect was that the measuring would turn out to be the interesting part: two separate bugs in my own instrument each produced a number that looked perfectly publishable. One of them I published.

The metric

memory delivery rate = deliveries / eligible retrieval opportunities

An eligible retrieval opportunity is one edit to one file inside an instrumented project, logged whether or not anything was delivered.

That last clause is the entire method. A log that records only successful retrievals has no denominator and cannot produce a rate. It can only tell you how often you got lucky, in units of nothing. Log the attempts and the rate falls out with no sampling and no inference.

The denominator is wider than it first appears, deliberately. A row is written for every edited file inside an instrumented project, including files the system has never heard of. Not “edits to files I have records for.” That makes 24.6% an unflattering number rather than a flattering one, and it is what makes the 13% coverage figure mean anything, because both are measured against everything the agent actually touched.

None of this is a new technique. It is exposure logging, structurally identical to a cache hit rate or a feature-flag exposure log, and any vendor shipping a memory feature is almost certainly computing it internally. My narrow claim is only that I could not find it published for an agent-memory system.

What LayerView stores, and how it arrives

Per-file records hold knowledge that is expensive to rediscover: incidents (“this broke in prod on the 4th, don’t revert it”), decisions with tradeoffs, cross-file constraints (“if you change A, B must change too”), external API quirks. Explicitly not documentation restating what the code already says, because a doc that restates the code goes stale and then lies to you. I later checked how many of these notes were still true against the code, and the ones about open bugs mostly weren’t.

A PreToolUse hook fires before the agent’s every Edit or Write, resolves the file to its record, and injects any warnings into the agent’s context unprompted, because an agent that has to remember to ask mostly doesn’t. Then it appends one JSON row: timestamp, project, resolved key, whether it warned, which components fired.

Results

Distribution is lopsided, and any reading should account for it:

The two projects with the highest rates have the least data. The headline figure is effectively A and B. The 7%–49% daily swing is not measurement noise; it tracks which project was busy that day.

What arrived, of 357 deliveries: gotchas 67.5%, co-change edges 62.2%, constraints 61.1%. 80 deliveries, 22.4% of the total, were co-change edges only. No prose at all, just “this file moves with that one.”

The finding that matters more than 25%

Only 49 of 368 edited files ever produced a delivery. That is 13%. And 80% of all deliveries came from 17 files.

The memory speaks on roughly one file in eight, and most of its activity is concentrated in about seventeen. That is not a comprehensive index of a codebase. It is a small hot core, and whatever value it has sits in a handful of places a person could probably name from memory.

Whether that is an indictment or the whole point is a real question. A system that fires on everything would be noise. A system that fires on seventeen files might be precision, or might be a rounding error dressed as a product. The delivery rate cannot tell you which, and that limit is the honest edge of what this method can do.

The instrument was lying and I did not know

The ledger held 3,327 rows after deduplication. 1,873 of them, 56%, were garbage, each recording a delivery of nothing.

The hook resolved a file to its record by walking up the directory tree to find the project root. But agents dispatched in parallel run inside git worktrees, and a worktree lives inside the repo it belongs to while carrying no project marker of its own. The walk sailed past it to the main root, and the key came out as .claude/worktrees/agent-a1284f2f.../worker/api.js instead of worker/api.js. Different key. Nothing ever matched.

Zero of those 1,873 edits were warned. Not “rarely”. Zero, for ten days.

Two things here matter more than the headline number.

First, the failure was concentrated exactly where the mechanism should matter most. Worktrees are how you run parallel agents, and a worktree agent is the thinnest-context actor in the system: freshly spawned, no session history, editing files it has never read. It was the one participant receiving nothing.

Second, and this is the part that should make anyone quoting a retrieval metric uneasy: the bug made the system look worse and made the metric look fine. Raw, the ledger reported 6%. That reads as “this barely fires”: a plausible, boring, actionable-looking finding. The true rate on files the system could address was four times that. A broken retrieval path and a genuinely low retrieval rate are indistinguishable from the outside. Guess wrong and you spend a month tuning a corpus when the plumbing was severed.

I caught it only because the ledger stored the resolved key, so the phantom paths were visible once someone looked.

If you build one of these: log the key you resolved, not just the outcome.

That is the most portable sentence here. A retrieval metric without the resolved key cannot distinguish “nothing to say” from “asked the wrong question.”

Erratum, 2026-08-16: a second instrumentation error

This piece originally reported 2,065 eligible edits, 515 deliveries, a 24.9% rate, 65% concentration in one project, and 63% phantom telemetry. Those numbers were inflated. The corrected figures are above; the originals are preserved in the dataset under v1-original/.

The pre-edit hook was registered twice for some projects, once machine-wide and once per-project, with command strings that differed only by path (/usr/local/bin/lv-before-edit versus $CLAUDE_PROJECT_DIR/bin/lv-before-edit). The harness deduplicates identical command strings but runs both when they differ. So a single edit wrote two identical ledger rows.

39.6% of the raw ledger was duplicates.

The evidence is not subtle, which is itself the point. It was sitting in the data the whole time:

• Duplicate groups are size exactly 2 in 2,456 of 2,458 cases. Mechanical, not organic; a human re-editing a file within one second would scatter across all projects.
• A natural control existed. One project had the same double registration, but its two command strings were identical, and it has zero duplicates across 597 edits.
• The onset is a step function pinned to a config commit: 0% duplication the day the second registration was added, 62% the next day, ~91% every day after.

What changed and what did not

The rate barely moved, because duplication removed from numerator and denominator alike. Per-project rates shift by at most 0.4pp. What was materially wrong is the sample size, overstated by 42%, and the concentration claim. A fair reading: the effect is unchanged, the confidence you should place in it is smaller.

The correction is a lower bound. Timestamps have one-second resolution and the hook takes ~340ms, so a double-fire straddling a second boundary still escapes the deduplication key. A wider two-second window gives 1,396 eligible and 24.4%. The true figure lies between 1,396 and 1,454, and I publish the conservative end.

Why this belongs in the piece rather than a quiet edit

This article’s thesis is that a ledger with a real denominator makes instrumentation failures findable. It already documented one: worktree paths that could never match. Here is the same ledger convicting itself a second time, in a completely different way, four weeks later.

That is not a reason to trust the number less. It is the argument. An instrument nobody audits reports whatever it reports; the failures do not announce themselves, and both of these were invisible until someone looked at the resolved keys and then at the duplicate groups. I found the first one and published anyway, carrying the second. Anyone running this method should assume they are currently carrying one too.

The fix, for anyone building the same thing: an idempotency guard at the point of write. Refuse to log the same key twice inside a short window. Configuration drifts and registration is easy to duplicate by accident; the ledger is the measurement instrument, so it should defend itself rather than trust its own wiring. It is the same failure mode as trusting a job’s own success counter, which I have written about before in don’t assume AI fixes things properly.

Delivery is not effect

Every number above measures delivery. None measures effect. 357 warnings arrived. I have no evidence that one of them changed an edit. A delivery is not a save.

The best-controlled study I found points the other way. Do Context Files Help Coding Agents? (Khatri, arXiv 2607.27250) ran 288 evaluated runs over 17 tasks against Claude Code and Codex, and found context strategy “does not measurably move correctness on either agent,” bounded to ≤10–15pp by equivalence testing. Its manipulation probe found context files never converted near-miss failures into passes.

The obvious objection is that the study used open-source repositories with no production incident history, so the knowledge LayerView carries (this broke in prod, that decision was deliberate, this cap is self-imposed) largely did not exist in its corpus. That objection is real, and it only covers part of my data: 62.2% of deliveries contained co-change edges and 22.4% were edges alone. Structural “these files move together” knowledge is much closer to what the ablation tested than to operational scar tissue. For that slice, the null result lands squarely.

One more caveat worth owning: silence is not safety. At 13% coverage, an agent that learns to trust the warnings may read no warning as nothing to worry about. That is an active harm mode, not merely a missed benefit. Nothing here measures it.

The transferable part

If you are building agent memory, the finding is not my 24.6%. Your number will differ, and mine is four projects over two weeks with half the data from one of them.

The finding is that the number is obtainable at all, and cheap. One JSONL row at the point of delivery, written on every attempt rather than every success, converts “is anyone using this?” from an unanswerable question into a denominator. It cost about twenty lines and 0.026 ms against a 340 ms operation.

I ran LayerView without it for months and had opinions. I ran it with instrumentation for two weeks and learned that 56% of my data was fictional, that real coverage sat in seventeen files, and that a fifth of what shipped was a bare edge list with no prose in it.

None of that was guessable. All of it was one log line away.

Related:

• Why Claude Code Ignores Your CLAUDE.md (and the Fix)
• Don’t Assume AI Fixes Things Properly
• AI Trusts Your Docs. That’s the Problem.

Sources

• Prakhar Khatri, Do Context Files Help Coding Agents? (arXiv 2607.27250): 288 evaluated runs, 17 tasks, Claude Code and Codex. The equivalence-tested null result this piece defers to on effect.
• Claude Code docs, Hooks: the PreToolUse hook that fires before every Edit and Write, which is where the ledger row gets written.
• LayerView: the memory system being measured. I build it, which is a conflict worth stating.
• Dataset and script: compute.py and the aggregates behind every figure here. CC0.

Data and reproduction: /research/agent-memory-delivery-2026-08/ holds compute.py (the exact script behind every figure), summary.json, per-day, per-project, payload-composition and concentration CSVs, with exclusions documented. CC0. Raw rows are withheld because they contain private repository paths; the aggregates preserve every claim made here.

Method: pre-edit hook ledger, snapshot 2026-08-15 18:00 UTC, window 2026-08-01 → 2026-08-15, four active projects of fourteen instrumented. A row is written for every edited file inside an instrumented project, registered or not. Malformed worktree rows are excluded from the rate and reported separately; fetch failures are counted in the denominator, since an edit where the lookup broke is still an edit where knowledge failed to arrive. The ledger grows during analysis, including from the edits that produced this piece, so figures are frozen at the snapshot rather than “current.”