# Agent memory delivery rate — dataset, 2026-08

Aggregate data behind **https://launchwithben.com/learn/research/agent-memory-delivery-rate**

Snapshot: **2026-08-15 18:00 UTC** · Window: **2026-08-01 → 2026-08-15**

> **⚠ CORRECTED 2026-08-16.** The figures first published here were inflated by duplicate
> ledger rows: the hook was registered twice with command strings differing only by path,
> and the harness ran both. `compute.py` now deduplicates. **The figures as first published
> are preserved unchanged in `v1-original/`** — this dataset is CC0 and may already have been
> copied, so nothing is rewritten in place. See the article's erratum for the causal timeline.

## The metric

```
memory delivery rate = deliveries / eligible retrieval opportunities
```

An **eligible retrieval opportunity** is one edit to one file inside an instrumented project, recorded **whether or not anything was delivered**. That is the entire method: a log that records only successful retrievals has no denominator and cannot produce a rate.

This is exposure logging — structurally the same as a cache hit rate or a feature-flag exposure log. Nothing about the technique is novel. What appears to be unpublished is applying it to agent memory.

## Headline

| | |
|---|---|
| Eligible opportunities | 1,454 (true value bounded [1,396, 1,454] — see below) |
| Deliveries | 357 |
| **Memory delivery rate** | **24.6%** |
| Distinct files edited | 368 |
| Distinct files that ever received a delivery | 49 (13%) |
| Duplicate rows removed | 2,189 of 5,516 in-window rows |
| Rows excluded as malformed | 1,873 (56.3% of deduplicated rows), 0 of which ever delivered |
| Active projects | 4 (of 14 instrumented) |

## Files

| File | Contents |
|---|---|
| `compute.py` | The exact script that produces every published figure. Run against any ledger in the documented row format. |
| `summary.json` | Headline figures, machine-readable |
| `daily.csv` | Per-day opportunities, deliveries, rate |
| `by-project.csv` | Per-project, anonymised A–D by volume |
| `payload-composition.csv` | Share of deliveries containing each component |
| `delivery-concentration.csv` | Rank vs deliveries, with cumulative share |

## Reproduce

```bash
python3 compute.py ~/.claude/lv-hook.log
```

Every number in the article and in these files comes from that command. It takes a ledger and prints the rate, the per-project split, payload composition, coverage, concentration, and the daily series.

## Exclusions, and why each exists

1. **Duplicate rows** (2,189 in-window). One edit logged twice by a double-registered
   hook. Deduplicated on `(ts, project, node, warned)` — deliberately conservative:
   timestamps have 1-second resolution, so a double-fire straddling a boundary still
   escapes the key. A 2-second window gives 1,396 eligible. The true value is a RANGE.
2. **Malformed worktree keys** (1,873 rows after deduplication). A path-resolution bug meant edits inside git worktrees resolved to keys like `file:.claude/worktrees/<agent>/src/api.js` instead of `file:src/api.js`. Those keys could never match stored memory, so those rows were *structurally incapable* of delivering — including them would report a mechanical failure as a low hit rate. They are excluded from the rate, retained in the ledger, counted, and reported separately, because they are themselves a finding.
3. **Snapshot cutoff.** The ledger grows continuously during analysis, including from the edits that write the analysis. All figures are frozen at the snapshot timestamp rather than computed against "now."

**Fetch failures are NOT excluded.** An edit where the lookup errored is still an edit where knowledge failed to arrive; removing them would flatter the rate. There was 1 in this window.

## What is not published, and why

**Raw ledger rows.** They contain file paths from private repositories. The aggregates here preserve every published claim — including the concentration curve, which is why `delivery-concentration.csv` gives rank and count with paths withheld rather than omitting the distribution.

**Project names**, anonymised A–D by volume. A is 52% of all opportunities (65% before deduplication); any reading of the headline rate should account for that.

Because the raw data cannot be published, a hash of it would not be verifiable by anyone else, so none is offered. The reproducibility claim here is deliberately narrow: **the script is the method, and you can run it on your own ledger.** These aggregates are one operator's output from it, not a corpus anyone can independently re-derive.

## Scope

One operator. Four active projects. Two weeks. 52% of data from a single project.

The transferable claim is the **method** — log every attempt, log the resolved key. The 24.9% is what the method returned when run once, and should be cited as such.

## Licence

Public domain (CC0). Cite, quote, or reuse freely, with or without attribution.