A trading bot crashed every ten minutes for two days, and the health check said 0 errors
My trading bot’s executor shipped with four runtime bugs. It crashed every ten minutes for two days. The health check reported “0 errors” the whole time, because it watched for exceptions it could see, not for whether the job produced any output. A rule in the project file said “prove the real path runs”. It changed nothing (what 17 retrospectives taught me).
A written rule is advice. The agent can read it and still skip it. I treat “done” as an interface problem: what can the agent claim, who checks the claim, and what happens before a risky action runs.
Only two of my five rules test the claim itself: a verify command and a review. The other three do a different job. They limit what an agent may do before anyone has checked anything: which commands are refused, which role may touch what, and when I get asked. I count all five, because how much checking a “done” needs depends on how much damage a wrong one can do.
The checks below do not all have the same reach. The Stop hook and the shell guardrail are global on this Mac, so they apply to every session I run here, across the 16 projects I have registered. The Stop hook only does anything during an unattended run. The role hooks exist in the five projects that have the role kit installed. The rest is prose. This is what I built, which parts are enforced, what I measured, and where the evidence stops.
What I built
- The agent
- What is supposed to stand in its way
- says a subtask is done, in an unattended run
- a Stop hook runs that subtask’s verify command, and it passes only if the command exits 0
- hands over finished work
- two reviewers, one from another vendor, neither told what the author concluded
- issues a shell command in a blocked category
- a guardrail refuses it before it runs
- is handed work in one of five projects
- a named role: interface, everything else, or concepts only
- reaches one of six boundaries: money, legal, a first public release, anything irreversible, an outside party, a real strategy fork
- I get asked
- does anything cheap and quick to undo
- it does it and reports afterwards
Each row is something the agent does. The right column is what is supposed to stand in its way. The rules below say which of these code enforces. The first two rows test "done". The rest limit what an agent may do unchecked, and the last row does not wait for me.
By my own standard, a rule only counts as enforced if code decides it. So each rule says which kind it is.
1. A subtask advances on an exit status, not on the agent’s word. Enforced, in unattended runs only. When I leave a list of tasks to run unattended, a Stop hook fires each time the agent tries to end its turn. The agent claims a subtask, the hook runs that subtask’s verify command, and the subtask is marked passed only if the command exits 0. After three failed verifies it is parked as failed, and the run goes on to the next one. It is not absolute: a verify command that checks too little still gives a false green, the agent can park a subtask as blocked if it gives a reason, and outside those runs nothing stops an agent saying done. A short public version, verify-before-done.sh, is in the hooks post. It illustrates the idea and is not my production hook.
2. Review comes from two reviewers, one of them another vendor’s model. Written. Neither is told what the author concluded, and neither sees the other’s answer. I use another vendor to reduce the chance of shared blind spots. I have not measured whether that works. This is an instruction my agents are given, not a check that code enforces. I have not counted how often they follow it, how many reviews of either kind have run, or how many defects either one catches.
3. Some shell commands are refused before they run. Enforced. A guardrail sits in front of every shell command an agent issues and refuses a short list of categories, such as printing a .env file or sending a secret to an outside host. It is a floor against accidents. Its own header says a determined injection can get around it, and file edits do not pass through it.
4. Roles limit who touches what. Partly enforced. Five projects have the role kit installed: a lead that scopes and delegates, one agent for interface work, one for everything else, and one that only draws concepts. The lead stays in the main checkout. The interface and everything-else roles each work in a separate copy of the repository. The concept role does not, and it is told not to touch production components. A hook refuses a dispatch that hands repository edits to a generic worker and points it at a named role. Which named role gets which work is still an instruction, and so is the rule that the lead does not type bulk edits itself.
5. A human is asked only at named boundaries. Written. Six of them, the ones in the figure. Everything cheap and quick to undo is done and reported afterwards. A check confirms that a request for my attention names its boundary. It cannot confirm that the agent raised everything it should have.
The one check I have measured
One of my written rules said “do not end a session by asking me whether to continue”. I counted: 26 of 212 sessions that had a final message ended that way anyway, 12%.
I turned the sentence into a check that runs when an agent tries to stop. It sends the agent back in two cases: the agent ends by asking permission, or it names its own next step and stops anyway. The 12% covers only the first case. I have no before number for the second.
This check is not one of the five. It sits next to rule 5: the agent should not hand the work back to me when nothing needs me.
In the last 30 days it fired 158 times, in 66 sessions: 21 times on asking, 137 on naming a next step and stopping. One firing has no transcript I can read. For the other 157 I counted tool calls after the block, and stopped counting at my next message or at the next firing in the same session, whichever came first:
- 105 were followed by at least three tool calls, 80 of them inside five minutes
- 28 were followed by one or two
- 24 were followed by none
By shape, that is 11 of the 21 asks and 94 of the 136 scoreable next-step stops. By session, in 30 of 65 every firing was followed by three or more calls, in 25 some were, and in 10 none was.
So about two in three blocks were followed by more activity before I said anything. That is all the count shows. It does not show the block caused the calls, that I wanted the work, or that the work was good. It also does not count the blocks that were wrong: the 24 with no tool call may include stops that were correct. And “my next message” is a rule of thumb applied to the transcript, not a field in it.
This paragraph has been wrong twice. The first version said 157 of 157. Two reviewers found that my report script read UTC timestamps as local time and counted calls made before a block as after it. The second said 154 of 158, with 118 inside five minutes. That version counted any later tool call in the session, including work I had asked for myself, and it matched a text string where it should have counted tool calls. The 154 survives only as a ceiling: after 154 of the 157 firings, the session made at least three more tool calls at some later point, whoever asked.
Three other things I counted
None of these three measures the five rules. They are adjacent parts of the same setup.
Agent notes. I checked 377 notes my agents had written about bugs against current code, with file and line citations. Notes saying something was still broken were false in 25 of 46 cases by the first reader, and 34 of 46 by a second reader who could not see the first labels. Both readers were agents reading code. Nobody ran it, and they disagreed on 14 of the 67 notes both read. I trust the direction, not the exact share. Full write-up: agent notes that lie.
Memory. A memory tool I ran until October 2026 delivered stored knowledge on 357 of 1,454 eligible agent edits (24.6%), from a hook ledger over two weeks. Two bugs in my own instrument had each produced a number that looked publishable. One of those wrong numbers I had already published. I corrected it and wrote up both bugs. Full write-up: does agent memory arrive.
Notes saying “still broken” that were false (second reader)
34 of 46. The first reader said 25 of 46.
Edits where stored knowledge was delivered
357 of 1,454 (24.6%), drawn as one square in four.
Two of the three. The two readers of the notes disagreed with each other, so read the left panel as a direction and not a rate.
Classifier denials. Claude Code’s auto mode denied 251 of 110,629 tool calls in my transcripts. This measures Anthropic’s classifier, not my guardrail. Full write-up: what auto mode denied.
What this does not show yet
- Whether the rules make “done” more accurate. I have not measured it. I have no count of false “done” claims the Stop hook refused, no count of defects a reviewer caught, and no before and after. The one check I measured counts tool calls after a block. It cannot say the block caused them, and it says nothing about quality.
- What the checks cost. I have no count of blocks that were wrong, no measure of how long a review holds work up, and no figure for the extra tool calls and tokens the checks themselves use. The usage my report could still read and price came to about $8,000 at list price over 30 days (22 Oct to 21 Nov 2026), about 47% of it delegated sub-agent work. I cannot say what share of that the checks account for.
- Whether agents satisfy the check and not the task. A verify command that checks too little gives a false green, and an agent that learns what the command looks at can pass it without finishing the job. I have not measured how often that happens.
- How often a check was running at all. The Stop hook, the guardrail, the role hooks and the measured check all let the action through if the check itself crashes. The measured check also stops firing after four blocks in one session and stands down during unattended runs. I have no count of hook errors, and no count of how often each rule applied.
- One operator, my own setup. No team has used this system. Every result above is one person’s agents on one person’s projects, and about a third of the agent notes came from a single project.
- I cannot quote a guardrail count. The guardrail’s log also holds its own test runs, and it cannot tell a test row from a real attempt, so any number of blocks would be inflated by an unknown amount. I describe what it refuses by category until the log tags its test runs.
- The usage figure is a floor. List price, not what I was billed, and some usage on unpriced models is not in the dollars.
- Delivery is not benefit. The memory study measures whether knowledge arrived, not whether outcomes improved.
What I would measure first: of the subtasks the Stop hook passed, how many were later found broken, and of the stops the measured check blocked, how many I would have wanted. After that, the test I cannot run alone: put the same rules in front of people who did not write them, and count where they stop trusting the rules.
Where each figure comes from
| Figure | Check it |
|---|---|
| 16 registered projects | jq 'keys|length' project-manager/projects.json |
| Five projects with the role kit installed | ls */scripts/roles-kit/VERSION across my projects |
| Three failed verifies, then parked | the header of ~/.claude/scripts/full-auto-stop.py |
| 26 of 212 sessions | a replay of my session transcripts, recorded in my decisions log on 11 Sep 2026 |
| 158 firings in 66 sessions; 21 and 137; 105, 80, 28 and 24 of 157; 11 of 21 and 94 of 136; 30, 25 and 10 of 65 sessions; ceiling 154 of 157 | sh ~/.claude/scripts/handback-report.sh 30, run 21 Nov 2026 after both script fixes |
| About $8,000, 47% sub-agent | claude-usage report --window all --since 2026-10-22 |
| Notes, memory, denials | the three write-ups linked above, each with its own data or script |
Related: What 17 Retrospectives Taught Me About Coding With AI is where the trading bot failure comes from. Claude Code Hooks has the Stop hook as a script you can copy.