Agent Notes About Open Bugs Go False. Fix Stories Don't.

I checked my agents’ notes against the code. The ones about open bugs were mostly wrong.

I shut down LayerView on October 15. It was the tool that kept the notes my coding agents wrote about files: a warning, a constraint, the reason behind a weird line. Before I handed the notes back to each repo, I wanted to know how many were still true.

I figured a few would be stale. It was worse than that, and which ones went bad was easy to predict.

One agent read each note next to the current code and said whether it was still true. I’ll call that the first reader. Later a second agent re-read 67 of them without seeing the first one’s verdict. Each square below is one note. A filled square means the code showed the note was false.

Notes about a bug that was still open (46 notes)

Notes about a bug that was already fixed (41 notes, and the second reader re-read 21 of them)

Notes about an open bug were false more often than not on both readings. Notes about a bug that was already fixed almost never were.

What I checked

886 notes, written by agents in 12 codebases between April and October. 427 of them were warnings (gotchas and constraints). I dropped 35 that pointed at files that no longer exist, and 15 that were notes on LayerView’s own files, which I didn’t check. That left 377, across 11 codebases.

One agent read each note next to the current file and said whether the note was still true. 51 weren’t. About one in seven.

Then I had a second agent sort all 377 by what kind of note they were. It only saw the text. It didn’t know which ones had turned out wrong.

The notes about things still broken went false the most. The ones telling the story of a fix almost never did. Notes about open bugs, snapshots and procedures were a quarter of all the notes and two thirds of the wrong ones.

That’s a clean result from one agent’s opinion. So I tried to break it.

First test: would a second labeller sort them the same way?

I had another agent label a random 60 of the notes, blind. They matched on the exact kind 87% of the time (Cohen’s kappa 0.80). On the split that matters, perishable versus durable, they matched 97% (kappa 0.93). Every disagreement was between neighbours: five notes one agent called a rule and the other called a fix story, two a rule and a procedure, one a snapshot and an open bug.

Second test: would a second reader of the code agree?

This is the one I was nervous about. I gave different agents the notes with no verdict attached and asked one thing: is the main claim still true, now false, or unclear? Cite a file and a line. I sent them all 46 open-bug notes, plus 21 of the 41 fix stories. That covered 26 of the 51 notes the first pass had called false. The other 25 (rules, snapshots and procedures) I did not re-read.

I expected them to confirm the first pass. On the 26 false calls they re-read, they mostly did: 23 of 26. Two were still true and one was unclear.

The surprise went the other way. Of the 41 notes I sent back that the first pass had kept (21 open-bug, 20 fix-story), 12 were wrong. Eleven of those were open-bug notes.

So the first agent was too forgiving, and the two readers disagreed on 14 of 67 notes, with one more they couldn’t settle. I don’t have one number. I have a range. On the two readings, open-bug notes were false 54% and 74% of the time, and with the uncertainty on a sample this size each reading’s interval is wide. Fix stories came out at 1 of 41 on the first read and 1 of 21 on the second, with an interval of up to 23%. Putting the two readings’ intervals together, open bugs span 40% to 84% and fix stories 0.4% to 23%. They don’t touch.

My guess at why: fixing the bug is what makes the note lie

I didn’t measure the mechanism. The data says open-bug notes go false more often, not why. But the likely story is that a note holds two different things in one field. A rule and its reason stay true as long as the design does. A status, like “this guard is missing”, is true until someone adds the guard. Fixing the bug is the exact event that makes the note false, and nothing in a note can notice.

Here’s the one case I traced by hand. (The second reader couldn’t settle this one, so the hand check is mine.) SerpDelta’s AuthController.php had a note from May 13: “No try/catch around tenantManager->createDatabase() at lines 218-222.” The code now wraps it, at AuthController.php:218-229. It logs the failure and redirects to a retry page. Anyone trusting the note would go looking for a bug that’s gone.

The ones that survived look different. A note on a mahjong game’s App.tsx says: “The key prop on GameCanvas was causing shuffle/booster actions to reset the game (Date.now() generated new key on every App re-render). Fixed by using stable keys. If anyone adds Date.now() back to the key, the same bug will return.” The second reader confirmed the fix is still there. The note also warns that if Date.now() goes back into the key, the bug returns, so it isn’t maintenance-free. But it describes something that already happened, which is probably why it held.

What I wouldn’t claim

• Nobody ran the code. Both readers were agents that read it and cited file and line, and they disagreed on 14 of 67 notes. I trust the direction. I wouldn’t quote 54% or 74% as the number.
• 13.5% is rough. The first reader was too forgiving on open-bug notes (it kept 21, and 11 were false) but about right on fix stories (it kept 20 of the ones I re-read, and 1 was false). About 285 other kept notes, mostly rules and fix stories, were read once. Using the second reading wherever it exists gives 60 of 377, or 15.9%.
• Age could be a confound. Open-bug notes skew early: 37 of the 46 were written in April to June. Inside that quarter alone, 21 of 37 open-bug notes were false against 1 of 19 fix stories, so the gap holds with age roughly held constant. A quarter is a coarse control, though.
• The fix-story number is a sample. 21 of 41 were read twice and one was false. The interval is wide.
• The second readers weren’t told the first verdict, but I batched them by it. One set got only the notes the first reader called false and the other only the ones it called kept. Where a verdict was “main claim true, a detail now false”, I scored it true.
• It’s one setup. One person’s agents and notes, and about a third of the notes (133 of 377) came from a single project, an affiliate site. Treat this as a case study of that setup, not a finding about agent memory in general.
• My first draft led with 54% against 2.4%. That was one reader and a cell of one note in 41. I took the single rates out of the headline, and the 74% above is also one reader.

Run it yourself

The data and script are in a public repo: agent-notes-study. The table has one row per note and no note text, project names or file paths, and the dates are coarsened to a quarter. One command prints the rates, counts and intervals above that come from the table, and checks the two random draws:

python3 reproduce.py

Related: Your Coding Agent Has Memory. Does It Actually Arrive? is the earlier measurement of the same memory system. Claude Code Repeats the Same Mistakes is why I stopped trusting notes to change behavior. AI Trusts Your Docs. That’s the Problem. is the docs version of the same trap.