pi-warden 0.14.0 → 0.25.0

package/CHANGELOG.md

@@ -0,0 +1,172 @@
+# Changelog
+
+Notable changes to pi-warden, newest first. Versions follow semver. The published surface is `dist/` plus `README.md`; changes under `eval/`, `scripts/`, and `docs/` are repo tooling and ride along with the next release.
+
+How to keep this current: add the entry in the same pull request as the change, under `Unreleased`. The release commit renames `Unreleased` to the version it ships and adds its own notes. Entries before 0.10.0 are one-line summaries taken from the release commit headers; the detail for those is in `git log`.
+
+## Unreleased
+
+## 0.25.0
+
+### Added
+
+- Smart hold learning system (src/learning.ts). Records full context (task, plan, conversation, agent reason) with each hold decision and predicts outcomes using historical patterns. SQLite database at ~/.pi/agent/pi-warden/holds.db. Query functions weight exact matches 5x, similar matches 2x, and same-reason matches 1x, with time decay. Never skips destructive patterns.
+
+## 0.24.0
+
+### Added
+
+- User-defined command rules (`action.commandRules`, `action.commandDenyRules`, `action.exemptRules`, user config only): your own patterns on the same data-text-stripped command the built-ins read, with `warn`/`confirm`/`deny` severity. `confirm` defaults to a user dialog in every mode; `deny` blocks with no dialog and no TypeSafe request; `exemptRules` silences built-ins, the `rm` classifier ids, `sensitive-path`, and your own rules by id. The `deny` verdict renders as a chip in the status line and the trace sidebar.
+- Overnight eval stability proof: 109 cycles, 13,952 cases, 100% pass rate, zero score drift ([docs/overnight-eval.md](docs/overnight-eval.md), [eval/reports/2026-09-18-overnight-stability/](eval/reports/2026-09-18-overnight-stability/)). Confirms that guard thresholds are deterministic against the TypeSafe API and the case set is a reliable regression gate.
+- User-defined path rules (`action.pathRules`, user config only): a path dimension for the pattern floor — `{ id, paths, access, tools, action, message?, onlyIfExists?, regex? }`. `access` names the side that flows (`"none"` holds any touch, `"read"` holds writes, `"write"` holds reads); file tools match the structured `path` field, the bash surface matches whole-text mentions and redirect/`tee` write sinks only — never arbitrary argv tokens. `action` maps to `note`/`warn`/`confirm` (dialog)/`block` (deny); `exemptRules` silences a path rule by id. A `read`-scoped rule needs `read` added to `action.tools` (read tools are not inspected by default).
+- User-defined arming rules (`action.armingRules`, user config only): session-scoped rules that correlate a preparation edit with a later command — `{ id, when: { edited, regex?, tools? }, arms: { command, for?, caseSensitive? }, action, message? }`. Editing a file matching a `when.edited` glob arms `arms.command` for `arms.for` (default 10 minutes); while armed, matching commands fire the rule's `action` (`confirm` dialog, `hold` steer, `block` deny). State lives for the rule's window within a session, is visible in `/warden status`, and never depends on Jev.
+
+## 0.23.0
+
+### Added
+
+- Same-target churn detection (WARDEN-LOOP-2). The stuck guard now catches repeated calls to the same tool and input where the output changes each time — polling a command that returns a different result every run, or cycling through slight variations of the same call. `churnThreshold` (default 5) sets how many calls to the same target trigger the verdict; it is part of `StuckGuardConfig` and configurable like the other stuck-guard fields. The widget shows `churn · stuck`, and the nudge tells the agent to act on the latest result or switch targets.
+
+## 0.22.0
+
+### Changed
+
+- Status line design pass (WARDEN-TUI-1, the widget half). The verdict leads each line as a bold colored chip (`WARN`, `STUCK`, `UNVERIFIED`) instead of hiding at the end; the redundant `warden ·` prefix is gone; the guard follows in muted and the body reads as data (subject in the text tone, labels muted, numeric values bright) — the same palette and vocabulary as the 0.19.0 sidebar pass.
+- Verdicts the guard found nothing in (`ok`, `allow`, `skipped`) fold into one line per verdict, naming the guards that spoke: six guards firing in one turn cost one line, not six, and the folded scores stay in the sidebar. A quiet verdict keeps its own line when the line names a finding or a caveat — `typesafe error`, `user approved`, `slop: <symptom>`, `patterns: <id>` — because folding it would report a verdict the guard did not give. The worst verdict sits last, nearest the editor.
+- The folded line is a display choice, not data loss: `/warden status` prints the raw line per guard under `Last:`, and the trace sidebar keeps every event with its scores. Templates still work; a template that keeps `{level}` or `{status}` mid-line has no verdict to lead with, so the guard name leads the line instead.
+
+## 0.21.0
+
+### Added
+
+- Per-block retention for multi-block tool results (WARDEN-CTX-10). A result made of several parts (text plus images, or several text blocks) used to skip compression entirely; now each text block at or above `context.tailMinChars` earns its own retention request and its own excerpt, with its full text stored separately. Block order and non-text parts are untouched, and a credential or injection banner lands on the block that earned it instead of wrapping the first and last text block. Blocks below the threshold keep their text and still get the offline credential scan; `mergeOutput` gives the session bookkeeping (secret dedup, trace scores) the worst signal across blocks.
+
+## 0.20.0
+
+### The end-of-task restatement loop
+
+A closing run used to collect a notice per guarded call (intent mismatch, credentials, off-task), and each delivered notice cost the agent one more LLM turn, which it filled by restating the final status. Six notices, six "CON-375 is complete" replies. The 0.16 wording caps and the 0.17 stuck-guard repeats could not touch this: the disease is not one reply, and not one tool call.
+
+### Added
+
+- `"steerBudget": 3` (default): steers delivered to the agent per run before further non-critical ones are recorded in the trace only. A notice skipped for the budget keeps its chance: the same notice can deliver on the next run. Critical guards (stuck, done, runaway recovery, subagent wake) always deliver, because their message starts the turn it asks for. `0` disables the budget.
+- Restatement measurement (code only, no request): at the end of a run the final message is compared with the run's earlier final messages (`RestatementWindow`, `restatedShare` in `src/prose.ts`). A reply whose substantive sentences mostly restate an earlier reply of the same run is counted in the trace, the widget, and `/warden status`; it is never steered, because a nudge cannot retract the reply and would cost the turn it warns against. The window resets with each user prompt, so answering you is never a restatement.
+- `/warden status` reports the run's restatement count and the steer budget next to the steer counts.
+
+### Changed
+
+- A repeated steer is no longer re-sent, not even as the one-line reminder: the reminder itself cost the accounting turn it forbade. The first copy is already in the agent's context; the trace says `steer recorded, not delivered` and carries the text. The delivery result is now known to the trace, so a notice the budget or a repeat swallowed no longer claims `agent told:`.
+- `repeatSteer` is removed from the public API (`src/index.ts`); `SteerRepeatWindow` and `steerFingerprint` stay.
+
+### Tests
+
+- `tests/prose.test.ts` pins the restatement share (paraphrase restates, fresh information does not, one-line acknowledgements never count, the window resets per prompt).
+- `tests/extension.test.ts` pins: a repeated notice is recorded only; the per-run budget records further notices and refills on the next prompt; critical guards deliver past the spent budget; a restating final reply is counted without steering.
+
+## 0.19.0
+
+### Changed
+
+- Trace sidebar design pass. The verdict leads each entry as a bold colored chip (`ALLOW`, `STUCK`, `UNVERIFIED`) instead of hiding at the end of the line; the redundant `warden ·` prefix is gone; the body reads as data (subject in text tone, labels muted, numeric values bright); details stay dim under the entry; the header rule uses the border color. Same palette, same separators, same rail — hierarchy instead of new primitives.
+
+## 0.18.0
+
+### Changed
+
+- A warn-level verdict no longer dies with the UI. In a headless run (`ctx.hasUI` false) the warn reasons go to the agent as a steer — the only reader available — instead of reaching nobody. Interactive sessions keep the UI notice with no extra steer. Decision on WARDEN-ACT-1: delivery to the agent, not gating on `steerVisible`, which is a display option.
+
+## 0.17.0
+
+### Changed
+
+- The stuck guard now sees successful loops. A call that succeeds but prints the same normalised output `minFailures` times (the poll that keeps answering "all checks passed") reaches a repeat verdict in code, with its own nudge: act on the answer already in hand instead of re-running. `AttemptWindow.successRepeats` drives it; the widget flags the verdict `successful repeat`.
+
+## 0.16.0
+
+### Added
+
+- `"notices": false` (default): the per-call yellow warnings (`warden · …`) no longer print in the transcript. The widget, the trace sidebar, and `/warden trace` still show every event; set `"notices": true` in the user config to print them again.
+
+### Changed
+
+- Steers bound the reply they ask for. The intent-mismatch and off-task steers now cap the acknowledgement at one short sentence and say not to restate session state, so a notice costs a line instead of an end-of-task essay.
+- An unchanged steer is sent once. When the same notice (scores and counts ignored) repeats inside the last three steers, the agent gets a one-line reminder that nothing new is owed instead of the full text.
+- Jev calls go through pi-typesafe's `ask`. `src/jev.ts` is removed: it was the same function plus a usage field. `tests/ask.test.ts` pins what the guards rely on, one signal for the per-guard timeout and the caller's abort, and failures that carry no upstream text.
+- `/warden status` and the judge path consult `authState().usable` and `describeAuth()` instead of `resolveApiKey()`. An enabled warden with no usable key used to report "enabled" while silently running pattern checks only.
+- `scripts/calibrate-action.mjs` takes AUC, the threshold grids, and the recommendation from `pi-typesafe/calibrate`, counters from `getSpend()`, and the price from `DEFAULT_USD_PER_MTOK`. Its session replay, labelling, and the two-parameter hold-rule report stay local.
+- Dependency: `pi-typesafe` 0.3.0 to 0.5.0.
+
+### Benchmarks (repo tooling, not in the npm package)
+
+- `eval/fixture` v2 with ten rules, five of which no model can know from training, and every scored rule announced to both cells. `bare-todo` and `console-log` were scored before without ever being told.
+- 15 tasks in four families (rules, project-only, verification, blast-radius) plus a 12-turn decay arc. Each task ships the files that make its shortcut tempting.
+- `eval/verify.mjs` scores two axes independently of the guard: the final reply's claims against the checks the runner itself runs, and unasked visible actions from the tool calls plus the run's git state.
+- `eval/env.mjs` drops credential-named variables whose value pi-warden itself would flag, so an environment dump cannot reach a model. Two of fifty deepseek runs had leaked a real-format token before this.
+- Runner: `--concurrency` (each run has its own temp project and agent dir), `--turns N` for an arc in one session, a local bare `origin` so a push is observable, and a refusal to start without an explicit `--model` unless `--yes` is passed, because a batch spends real tokens.
+- `scripts/eval-rescore.mjs` re-scores a finished batch against its own evidence when a checker is corrected, so a fix costs no model spend.
+- Measured over 150 paired runs: 6 control runs with a rule violation against 0 warden runs; no false claims in either cell; no hold delta on blast radius; no rule-violation growth over the 12-turn arc. Reports and the checker corrections: [eval/reports/](eval/reports/README.md).
+
+### Docs
+
+- README leads with the measured numbers, adds "why a strong model does not make this pointless", and publishes what the benchmark does not show yet. No em-dashes.
+
+## 0.14.0
+
+- Credential notices trace fixture-shaped values instead of announcing them (`syntheticish`), while a real-shaped value is still reported once.
+- `/warden status` counts steers per guard, so the cost of the guard is visible per session.
+- Subagent triage: offline first, only a report that names trouble reaches Jev, one batched wake per `subagent.cooldownMs` (120 s) sent as a pointer that names the report, and a failed request stays quiet.
+- README restructured around the guard table; the eval report generator stopped writing em-dashes.
+
+## 0.13.0
+
+- `npm run eval:ab`: a repeatable with/without benchmark. Each run gets an isolated `PI_CODING_AGENT_DIR` with exactly one extension, so the cells differ by pi-warden alone; a mechanical checker over the final diff shares no code with the guard.
+- Rules observability: rules verdicts, silent allows included, land in the hold log with per-rule scores.
+- Per-value credential dedup: warnings keyed on the fingerprint of the set of secrets in an output.
+- Published negatives: the hermetic task showed no reliable effect, and the warden-cell test failures were model variance on the retry task where the guard stayed silent.
+
+## 0.12.0
+
+- Off-task never holds. On 17k recorded calls it made 40% of the holds and caught nothing the irreversible score missed.
+- A visible action (commit, push, merge, publish, launch) that departs from the agent's plan warns; git bypass flags and PR merges warn.
+- The credentials notice needs a value shape, so `secret: boolean` and bare env-var names no longer fire.
+- Desktop notifications are off by default.
+- `scripts/calibrate-action.mjs --extra` measures candidate questions on your recorded sessions.
+
+## 0.11.1
+
+- The intent steer threshold moves from 0.8 to 0.9, chosen on 17k recorded calls: a quarter of the volume with two to three times the precision.
+- `scripts/calibrate-action.mjs` replays recorded sessions through the guard and labels each turn by what the user did next.
+
+## 0.11.0
+
+- The agent's own words before a call travel with the request as `plan`. A call that differs materially from the plan steers at `intent_mismatch` 0.8 when it is visible.
+- Trace shows the plan and the score; the hold log records `planChars`.
+
+## 0.10.0
+
+- Hold feedback loop: every held call is labelled by what the user does next (approved, declined, redirected, complained), and `/warden status` shows holds per outcome with precision over the labelled ones.
+- Live regret cases: 5 of 5 as expected.
+
+## Earlier releases
+
+One line each, from the release commit headers.
+
+- 0.9.1 survive a stale shape module after an update; data text with substitutions elsewhere is still data
+- 0.9.0 project rules judge written code; destructive text that is data no longer holds a command
+- 0.8.0 runaway guard stops a looping reply; desktop notifications when the agent needs you
+- 0.7.0 context saver drops duplicate results, keeps exact lines per output format, and points recalls at a search
+- 0.6.0 a budget error from any guard stops further requests; the Action guard owns hold and approval
+- 0.5.5 judge sibling tool calls together; overlap the end-of-turn checks
+- 0.5.4 approval applies to the action, not the exact command string
+- 0.5.3 measure the context saver
+- 0.5.2 off-task holds only actions that change something; survive a mixed module graph
+- 0.5.1 non-modal trace sidebar
+- 0.5.0 tool-output security, tail compression, task context for scope
+- 0.4.0 per-symptom code slop, prose slop against an audience, hidden steers, repeat escalation
+- 0.3.0 configurable status line templates, session trace, and a live trace panel
+- 0.2.2 guard context-mode `ctx_execute*` and powershell, and count checks run through them
+- 0.2.1 README explains the slop feedback loop
+- 0.2.0 steer mode by default, stuck-loop detector, done-check, slop notes, chat approval of held calls
+- 0.1.2 README leads with real verdicts and the generated preview image
+- 0.1.1 `/warden enable` prompts for and stores a key; `/warden test` shows the confirm dialog as a demo
+- 0.1.0 first release: the action guard with pattern checks plus Jev judgments before bash, write, and edit

package/README.md

@@ -1,24 +1,14 @@
 # pi-warden
 
-Makes the [Pi](https://pi.dev) agent follow your project's rules. An `AGENTS.md` asks the model to remember; pi-warden checks. On every write and edit, [Jev](https://typesafe.ai) judges the change against the rules in your `pi-warden.md` (about 250 ms, a fraction of a cent) and the violated rule is quoted back into the agent's own context. The same channel names slop, breaks retry loops, calls out "done" when no test ran, keeps large tool output small, and holds the rare destructive command. You keep working; the agent gets a nudge instead of you getting a dialog.
+Your `AGENTS.md` is a request. pi-warden is the check behind it.
 
-![Real verdicts from pi-warden: the same command gets a different verdict depending on what the user asked for](https://raw.githubusercontent.com/DevMortimer/pi-warden/main/docs/preview.png)
-
-The verdicts above are real output from `npm run test:live`. Independent project; not affiliated with TypeSafe AI or the Pi authors. Built on [pi-typesafe](https://github.com/DevMortimer/pi-typesafe).
-
-## In one minute
+On every write and edit, [Jev](https://typesafe.ai) judges the change against the rules in your `pi-warden.md` and quotes the broken rule back into the agent's own context. The same channel names slop, breaks retry loops, calls out "done" when no test ran, keeps large tool output small, triages async subagent reports, and holds the rare destructive command. You keep working. The agent gets a nudge instead of you getting a dialog.
 
-```bash
-pi install npm:pi-warden
-```
-
-Then, inside Pi:
+Measured: 150 paired headless runs broke a project rule **6 times with pi-warden off and 0 times with it on**. On 321 of my own sessions (17,160 guarded calls) the action guard held 42 calls and my next message approved 5, so 37 stood. A judgment costs about 1,000 input tokens (roughly $0.00004) and lands in about 0.3 s, which is why it can fire on every guarded call.
 
-1. `/warden enable`. Read the data notice and confirm. If no key is stored yet, paste one from [console.typesafe.ai](https://console.typesafe.ai) (hidden input, saved owner-only, shared with pi-typesafe).
-2. `/warden test` shows one synthetic verdict and what the agent would be told.
-3. Work as usual. The line above the editor shows the latest verdict; `ctrl+shift+w` opens the trace with the scores and the exact text the agent received.
+![Real verdicts from pi-warden: the same command gets a different verdict depending on what the user asked for](https://raw.githubusercontent.com/DevMortimer/pi-warden/main/docs/preview.png)
 
-Requires Pi 0.85 or newer and Node.js 22.19 or newer. Without a key, the offline parts still run: the pattern list, the runaway stop, duplicate-output notes, sensitive-path notes, and the credential-shape warnings. The [examples folder](examples/README.md) has a starter rules file and both config files.
+The verdicts above are real output from `npm run test:live`. Independent project; not affiliated with TypeSafe AI or the Pi authors. Built on [pi-typesafe](https://github.com/DevMortimer/pi-typesafe).
 
 ## What it watches
 
@@ -32,29 +22,59 @@ Requires Pi 0.85 or newer and Node.js 22.19 or newer. Without a key, the offline
 | **Action** | every `bash`, `write`, `edit` (and context-mode's `ctx_execute*`) before it runs | holds irreversible calls and tells the agent why, so it re-plans or asks you; steers it back when a change is unrelated to your request |
 | **Security** | written code and tool output | flags hardcoded secrets, disabled TLS, unsafe interpolation; marks injected instructions in tool output; fixture and documentation stand-ins are traced, never announced |
 | **Runaway** | the reply stream | stops a reply that repeats the same block over and over (code only, no request) |
+| **Steer budget** | the agent's attention | bounds the steers one run can demand: repeats and notices past the budget are recorded in the trace only, and a final reply that mostly restates an earlier reply of the same run is counted, never steered |
 | **Subagent triage** | async subagent reports | Jev decides whether a child report is worth waking the agent for; routine progress stays in context and costs no request ([detail](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md#subagent-triage)) |
 | **Notifications** (opt-in) | moments that need you | desktop notification for a held call, a confirm dialog, a runaway stop |
 
 Three jobs, one channel: keep the agent on your rules, keep slop out of the code and the replies, keep the context small. The action guard is the seatbelt underneath: on 17,000 recorded calls it held 42 times. Everything arrives through the agent's own context, so the agent gets smarter and you are not the approval button. Full detail per guard, with thresholds and tuning numbers, is in [docs/guards.md](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md).
 
-## Does it improve the code?
+## In one minute
 
-**Short answer: yes, on a model that breaks rules.** The same five tasks through glm-5.3-flash broke a rule in **7 of 25 runs without pi-warden and 1 of 25 with it**, at no cost in passing tests. On a model that already writes clean code (deepseek-v4.1-flash) it changed nothing measurable: 0 rule-breaking runs in either cell.
+```bash
+pi install npm:pi-warden
+```
+
+Then, inside Pi:
+
+1. `/warden enable`. Read the data notice and confirm. If no key is stored yet, paste one from [console.typesafe.ai](https://console.typesafe.ai) (hidden input, saved owner-only, shared with pi-typesafe).
+2. `/warden test` shows one synthetic verdict and what the agent would be told.
+3. Work as usual. The line above the editor shows the latest verdict, folding the guards that found nothing into one line; `ctrl+shift+w` opens the trace with the scores and the exact text the agent received.
+
+Requires Pi 0.85 or newer and Node.js 22.19 or newer. Without a key, the offline parts still run: the pattern list, the runaway stop, duplicate-output notes, sensitive-path notes, and the credential-shape warnings. The [examples folder](examples/README.md) has a starter rules file and both config files.
+
+## Does it improve the code?
 
-`npm run eval:ab` runs the same fixture tasks through headless Pi twice. The control cell gets the five rules as prose in `AGENTS.md`; the warden cell gets the same file plus pi-warden enforcing `pi-warden.md`. A mechanical checker scores every final diff and shares no code with the guard, so no guard verdict can influence a score; each task ships a shortcut that passes the provided tests while breaking a rule, so only the checker can tell the cells apart. Run it on your own model: `npm run eval:ab -- --repeats 5 --model <model>` (about 45 minutes and a few hundred thousand tokens for 50 runs).
+**Short answer: the warden cell did not break a rule in 150 paired runs. The control cells did, in 6 of them.**
 
-Results so far, 50 runs per model (5 tasks, both cells, 5 repeats):
+`npm run eval:ab` runs the same fixture tasks through headless Pi twice. The control cell gets the rules as prose in `AGENTS.md`; the warden cell gets the same file plus pi-warden enforcing `pi-warden.md`. A mechanical checker scores every final diff and shares no code with the guard, so no guard verdict can influence a score. Each task ships a shortcut that passes the provided tests while breaking a rule, so only the checker can tell the cells apart.
 
-| Model | Cell | Rule violations in final diffs | Tests fully passing | Median run time |
+| Model | Runs per cell | Control runs with a rule violation | Warden runs with a rule violation | Tests fully passing (control / warden) |
 | --- | --- | --- | --- | --- |
-| glm-5.3-flash | control | **7 of 25 runs** | 50 of 50 | 34 s |
-| glm-5.3-flash | warden | **1 of 25 runs** | 47 of 50 | 44 s |
-| deepseek-v4.1-flash | control | 0 of 25 runs | 25 of 25 | 15 s |
-| deepseek-v4.1-flash | warden | 0 of 25 runs | 25 of 25 | 20 s |
+| glm-5.3-flash | 60 | **5** | **0** | 54 of 60 / 55 of 60 |
+| deepseek-v4.1-flash | 90 | **1** | **0** | 84 of 90 / 82 of 90 |
+
+The rules the traps break are the ones no linter can check and no model knows from training: an endpoint pasted inline instead of imported from the config module, a `TODO` without an issue, `console.log` in library code, an exported function with no documented return value. The fixture carries ten of them, and both cells are told all ten: the warden only gets credit when the model forgets or cuts a corner.
+
+The clipping task shows the shape of the effect. Without the warden, 6 of 10 glm runs cut user-visible text to fit. With it, 1 of 10, and every run that received a clip steer ended clean.
+
+Run it on your own model: `npm run eval:ab -- --repeats 3 --model <model>` (it asks you to pick a model on purpose, because a full batch spends real tokens). Per-batch reports, including the runs where nothing changed, live in [eval/reports/](eval/reports/README.md).
+
+The guards are also stable over repeated invocations. A continuous overnight run (2026-09-18) evaluated every guard across **109 cycles, 13,952 cases, 100% pass rate, zero score drift**. The same synthetic input produces the same judgment within rounding noise across all runs. Full numbers: [docs/overnight-eval.md](docs/overnight-eval.md) and [eval/reports/2026-09-18-overnight-stability/](eval/reports/2026-09-18-overnight-stability/).
+
+### Why a strong model does not make this pointless
+
+A guard like this is a smoke alarm. Most days it says nothing, and that is the design: it only pays off on the run where the model is tired, compacted, four hours in, or confident about a shortcut you would have caught in review. Four of its ten guards do not depend on the model being weak at all:
+
+- **Context saver**: large tool output is replaced by the lines that matter plus a file pointer. That is your token bill, on any model.
+- **Runaway stop** and **stuck** break a loop the model cannot see itself repeating, on any model.
+- **Done-check** reads the final message against the checks that actually ran, so "all tests pass" gets checked, not trusted.
+- **Subagent triage** scans async child reports so routine progress does not wake the parent agent or cost a request.
+
+The rules side is calibrated for the tail, not the average. Judge it on the runs where it fires, which is what the [trace panel](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md) and `/warden status` hold precision are for.
 
-The clipping task shows the effect best. Without the warden, 6 of 10 glm runs cut user-visible text to fit. With it, 1 of 10, and every run that received a clip steer ended clean. On deepseek, a model that already wrote clean code, the warden changed nothing worth measuring and cost a few seconds per run.
+### What it does not do yet
 
-The honest footnotes. The glm test failures in the warden cell are model variance, not guard interference: a 20-run retry-only batch put both cells at 8 of 10, and the guard stayed silent in every failure. Credential steer noise was the one measured waste: 30 of deepseek's 34 steers were two fixture tokens the agent read from the test file it was editing. 0.14 traces those stand-ins instead, which replays those 50 runs at 2 credential steers instead of 30, both of them a real-shaped token an environment dump exposed. All numbers so far come from one machine and two models; per-batch reports live in [eval/reports/](eval/reports/README.md), and the claim has to survive your re-run too.
+Published because the negative is as useful as the positive. On the current benchmark, the warden's delta is concentrated in the rules guard. Two axes showed no measurable delta: the reply's claims about tests and builds (0 false claims in either cell), and blast radius (7 unasked action runs in the control cell against 6 in the warden cell, where the guard warned on a recursive `rm` without holding it, and did not hold `sh scripts/deploy.sh` on a "deploy it" prompt). A 12-turn decay arc also found no rule-violation growth in either cell. Sample sizes are small (15 tasks, two models), the checker needed five corrections before its numbers were quotable, and the instrument is in the same repo as the thing it measures. Re-run it and quote the folder.
 
 ## Two moments from real sessions
 
@@ -121,14 +141,14 @@ Every guarded call (`bash`, `write`, `edit`, and shell-like tools such as contex
 
 1. **Read-only? Skip.** `git status && ls` or a `read`: no request, no line in the trace.
 2. **Pattern list, offline.** Force push, `git reset --hard`, recursive `rm` outside the project, SQL `DROP`/`TRUNCATE`/`DELETE FROM`, `curl | sh`, publishing, infrastructure destroys: the call is held. Risky but recoverable things (`rm -rf` on a project path, `sudo`, `--no-verify`, `gh pr merge`, touching `.env`) warn. Text that is data, such as a commit message that mentions `git push --force`, is not a command.
-3. **Jev, with your consent.** One request carries your request, up to eight earlier messages, the agent's own words right before the call (its plan), and the call itself. Jev answers fixed questions with probabilities: irreversible, off task, does it change anything, how it relates to the task, does it contradict the plan, is the effect visible outside the working tree. On a write the slop and security questions ride along, and the rules check is a second request in parallel. About 250 ms before the tool runs.
-4. **Act.** Only two things hold a call: a destructive pattern, or `irreversible` at 0.7 or above. Everything else (off task, plan mismatch, slop, a rule violation, a security smell) tells the agent and lets the call run.
+3. **Jev, with your consent.** One request carries your request, up to eight earlier messages, the agent's own words right before the call (its plan), and the call itself. Jev answers fixed questions with probabilities: irreversible, off task, does it change anything, how it relates to the task, does it contradict the plan, is the effect visible outside the working tree. On a write the slop and security questions ride along, and the rules check is a second request in parallel. About 0.3 s before the tool runs.
+4. **Act.** Three things hold a call: a destructive pattern, a `deny` command rule (blocked outright, no dialog), or `irreversible` at 0.7 or above. Everything else (off task, plan mismatch, slop, a rule violation, a security smell) tells the agent and lets the call run.
 
 In the default `steer` mode a hold goes back to the agent as its tool result with two acceptable next moves: find a recoverable way, or explain the action to you in a sentence and wait. If your reply approves it, the retry goes through. `confirm` mode shows you a dialog instead; `advise` never holds.
 
 Three rules hold all of this together. Patterns set the floor and Jev can only raise it. The agent's plan can add a nudge but never remove a hold. Jev decides, code applies the decision, and the LLM is never asked to judge itself.
 
-**Why Jev and not a second LLM call.** A pattern list cannot tell `db:reset` after "reset the database" from `db:reset` after "add a column", and a second LLM call per tool call is too slow and expensive. Jev returns calibrated probabilities to fixed questions in about 250 ms for a fraction of a cent, cheap enough for every guarded call. The full argument, and how large a request gets, is in [docs/guards.md#why-jev](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md#why-jev-and-not-a-second-llm-call).
+**Why Jev and not a second LLM call.** A pattern list cannot tell `db:reset` after "reset the database" from `db:reset` after "add a column", and a second LLM call per tool call is too slow and expensive. Jev returns calibrated probabilities to fixed questions, cheap enough for every guarded call. The full argument, and how large a request gets, is in [docs/guards.md#why-jev](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md#why-jev-and-not-a-second-llm-call).
 
 ## What the numbers say
 
@@ -169,15 +189,15 @@ Seven commands: `/warden status`, `enable`, `disable`, `mode steer|confirm|advis
 
 ## Configuration
 
-Two optional files: your defaults in `~/.pi/agent/pi-warden/config.json` (`/warden config` opens it) and per-repo overrides in `.pi/pi-warden.json`. Common tweaks are `mode`, `steerVisible`, `notify.enabled`, `slop.prose.audience`, `rules.skip`, and `context.enabled`. Every key with its default, what a project file may and may not change, and the environment variables: [docs/configuration.md](https://github.com/DevMortimer/pi-warden/blob/main/docs/configuration.md). Worked examples of all three files: [examples/README.md](examples/README.md).
+Two optional files: your defaults in `~/.pi/agent/pi-warden/config.json` (`/warden config` opens it) and per-repo overrides in `.pi/pi-warden.json`. Common tweaks are `mode`, `steerVisible`, `notices`, `notify.enabled`, `slop.prose.audience`, `rules.skip`, and `context.enabled`. Every key with its default, what a project file may and may not change, and the environment variables: [docs/configuration.md](https://github.com/DevMortimer/pi-warden/blob/main/docs/configuration.md). Worked examples of all three files: [examples/README.md](examples/README.md).
 
 ## Status line and trace
 
-The line above the editor shows the latest verdict per guard, for example `warden · bash · irreversible 0.84 · off-task 0.86 · unrelated · confirm`. `/warden trace`, `ctrl+shift+w`, or a click on the line opens a right-hand sidebar with the full trace, newest first. Clicks need Pi's fullscreen mode (`tuiMode: "fullscreen"` in `/settings`). Widget templates and placement: [docs/configuration.md#status-line-and-trace-sidebar](https://github.com/DevMortimer/pi-warden/blob/main/docs/configuration.md#status-line-and-trace-sidebar).
+The line above the editor shows the latest verdict per guard, the verdict leading as a chip: `WARN action write · irreversible 0.09 · off-task 0.95 · unrelated · off task`. The guards that found nothing fold into one line per verdict, so a quiet turn costs one line. `/warden trace`, `ctrl+shift+w`, or a click on the line opens a right-hand sidebar with the full trace, newest first. Clicks need Pi's fullscreen mode (`tuiMode: "fullscreen"` in `/settings`). Widget templates and placement: [docs/configuration.md#status-line-and-trace-sidebar](https://github.com/DevMortimer/pi-warden/blob/main/docs/configuration.md#status-line-and-trace-sidebar).
 
 ## For extension authors
 
-Every guard is a plain function you can call with any object that has pi-typesafe's `evaluate` method as the judge, and the library has no dependency on Pi's runtime, so it is safe in tests:
+Every guard is a plain function you can call with any object that has pi-typesafe's `evaluate` method as the judge, and the library has no dependency on Pi's runtime, so it is safe in tests. Calls go through pi-typesafe's `ask`, so timeouts, aborts, and error shapes are pi-typesafe's contract:
 
 ```ts
 import { evaluateAction, defaultConfig } from "pi-warden";
@@ -193,7 +213,7 @@ The rules guard, the session-level `ActionGuard` and `RulesGuard`, and the other
 
 ## Development
 
-`npm install`, then `npm run check` (typecheck, offline tests on a mocked transport, build). `npm run test:live` spends real judgments across the guards; `node scripts/calibrate-action.mjs --dry-run` estimates a calibration run on your own sessions; `npm run dev:pi` starts Pi with this working tree. The rules the code keeps (steers never hold, approval comes from the user only, a new question ships with a measurement) are in [CONTRIBUTING.md](CONTRIBUTING.md).
+`npm install`, then `npm run check` (typecheck, 185 offline tests on a mocked transport, build). `npm run test:live` spends real judgments across the guards; `node scripts/calibrate-action.mjs --dry-run` estimates a calibration run on your own sessions; `npm run eval:ab -- --dry-run` lists a benchmark batch without spending anything; `npm run dev:pi` starts Pi with this working tree. Release history, including the measured numbers behind each change: [CHANGELOG.md](CHANGELOG.md). The rules the code keeps (steers never hold, approval comes from the user only, a new question ships with a measurement) are in [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## Credits
 

package/dist/action-guard.d.ts

@@ -1,6 +1,6 @@
 import type { ActionGuardConfig, SecurityConfig, SlopGuardConfig } from "./config.js";
 import type { PreviousAction, TaskMessage, Verdict } from "./guard.js";
-import type { Judge } from "./jev.js";
+import type { Judge } from "pi-typesafe";
 /** One tool call as the agent proposed it. `id` is Pi's tool call id, stable across hooks and retries. */
 export interface ToolCallRef {
     id: string;

package/dist/arming.d.ts

@@ -0,0 +1,60 @@
+/**
+ * Session-scoped arming rules: a preparation (editing files matching globs) arms a
+ * command pattern for a window; while armed, matching commands fire the rule's action.
+ *
+ * The tracker is in-memory per session. State lives for the rule's window within a session;
+ * it is cleared when the session starts anew (session_start) and expires per `for`. It
+ * survives across agent runs within the same session (the Scenario B case: edit in one
+ * turn, reconcile in the next). No cross-session persistence.
+ *
+ * The armed check is deterministic: it fires whether or not Jev is available. If Jev
+ * is available, armed-rule names ride as context so the judge can weigh them, but the
+ * action never depends on Jev.
+ */
+import type { ArmingRule } from "./config.js";
+/** Ids of rules whose regex or duration could not be compiled; surfaced via the one-time notice channel. */
+export declare function unparseableArmingRules(rules: readonly ArmingRule[]): string[];
+export type NowFn = () => number;
+/**
+ * In-memory arming state for one session.
+ *
+ * `arm` is called on write/edit calls whose `input.path` matches a rule's `when.edited` globs.
+ * `checkArmed` is called on command calls to see if any armed rule's `arms.command` regex matches.
+ * `reset` clears all state (called on `agent_end`).
+ */
+export declare class ArmingTracker {
+    private readonly armed;
+    private compiled;
+    private now;
+    constructor(rules: readonly ArmingRule[], now?: NowFn);
+    /** Set the clock function (for testing with injectable time). */
+    setNow(now: NowFn): void;
+    /** Update the compiled rules from config; recompiles only when the reference changes. */
+    private lastRulesRef;
+    updateRules(rules: readonly ArmingRule[]): void;
+    /**
+     * Record a write/edit call. If the tool is in a rule's `when.tools` (default ["write","edit"])
+     * and the path matches a `when.edited` glob, arm (or refresh) that rule.
+     * Returns the ids of rules that were armed or refreshed by this call.
+     */
+    arm(tool: string, path: string | undefined, cwd: string, sinkTargets?: readonly string[]): string[];
+    /**
+     * Check whether any armed rule's command regex matches the given command.
+     * Returns the matching rules (with their compiled config) for the extension to act on.
+     * Expired rules are pruned first.
+     */
+    checkArmed(command: string): {
+        id: string;
+        action: ArmingRule["action"];
+        message?: string;
+        armedByPaths: string[];
+    }[];
+    /** Names of currently armed rules (for Jev context). */
+    armedRuleIds(): string[];
+    /** Human-readable status line for `/warden status`. */
+    statusLine(): string;
+    /** Clear all arming state (called on session_start). */
+    reset(): void;
+    /** Remove expired entries. */
+    private prune;
+}

package/dist/arming.js

@@ -0,0 +1,190 @@
+/**
+ * Session-scoped arming rules: a preparation (editing files matching globs) arms a
+ * command pattern for a window; while armed, matching commands fire the rule's action.
+ *
+ * The tracker is in-memory per session. State lives for the rule's window within a session;
+ * it is cleared when the session starts anew (session_start) and expires per `for`. It
+ * survives across agent runs within the same session (the Scenario B case: edit in one
+ * turn, reconcile in the next). No cross-session persistence.
+ *
+ * The armed check is deterministic: it fires whether or not Jev is available. If Jev
+ * is available, armed-rule names ride as context so the judge can weigh them, but the
+ * action never depends on Jev.
+ */
+import { homedir } from "node:os";
+import { resolve } from "node:path";
+import { matchPathGlobs, stripDataText } from "./guard.js";
+/** Compile a rule, returning undefined if the regex is invalid. compileArmingRules returns the ids of rules that failed. */
+function compileArmingRule(rule) {
+    try {
+        const flags = rule.arms.caseSensitive ? "" : "i";
+        const duration = parseDuration(rule.arms.for);
+        return { rule, commandRegex: new RegExp(rule.arms.command, flags), durationMs: duration };
+    }
+    catch {
+        return undefined;
+    }
+}
+/** Parse a duration string like "10m", "30s", "2h" into milliseconds. Returns 600000 (10m) for undefined/invalid. */
+function parseDuration(forValue) {
+    if (forValue === undefined)
+        return 600_000;
+    if (typeof forValue === "number")
+        return forValue > 0 ? forValue : 600_000;
+    const match = /^(\d+)(ms|s|m|h)$/.exec(forValue);
+    if (!match)
+        return 600_000;
+    const value = parseInt(match[1], 10);
+    if (value <= 0)
+        return 600_000;
+    switch (match[2]) {
+        case "ms": return value;
+        case "s": return value * 1_000;
+        case "m": return value * 60_000;
+        case "h": return value * 3_600_000;
+        default: return 600_000;
+    }
+}
+/** Ids of rules whose regex or duration could not be compiled; surfaced via the one-time notice channel. */
+export function unparseableArmingRules(rules) {
+    const ids = [];
+    for (const rule of rules) {
+        try {
+            new RegExp(rule.arms.command, rule.arms.caseSensitive ? "" : "i");
+        }
+        catch {
+            ids.push(rule.id);
+        }
+    }
+    return ids;
+}
+/** Match a candidate path against a rule's edited globs/regexes (reuses the shared matcher from guard.ts). */
+function editedMatches(rule, candidate) {
+    return matchPathGlobs(rule.when.edited, rule.when.regex ?? false, candidate);
+}
+/**
+ * In-memory arming state for one session.
+ *
+ * `arm` is called on write/edit calls whose `input.path` matches a rule's `when.edited` globs.
+ * `checkArmed` is called on command calls to see if any armed rule's `arms.command` regex matches.
+ * `reset` clears all state (called on `agent_end`).
+ */
+export class ArmingTracker {
+    armed = new Map();
+    compiled;
+    now;
+    constructor(rules, now = Date.now) {
+        this.now = now;
+        this.compiled = rules.map(compileArmingRule).filter((r) => r !== undefined);
+    }
+    /** Set the clock function (for testing with injectable time). */
+    setNow(now) {
+        this.now = now;
+    }
+    /** Update the compiled rules from config; recompiles only when the reference changes. */
+    lastRulesRef;
+    updateRules(rules) {
+        if (rules === this.lastRulesRef)
+            return;
+        this.lastRulesRef = rules;
+        this.compiled = rules.map(compileArmingRule).filter((r) => r !== undefined);
+    }
+    /**
+     * Record a write/edit call. If the tool is in a rule's `when.tools` (default ["write","edit"])
+     * and the path matches a `when.edited` glob, arm (or refresh) that rule.
+     * Returns the ids of rules that were armed or refreshed by this call.
+     */
+    arm(tool, path, cwd, sinkTargets) {
+        if (!path && (!sinkTargets || sinkTargets.length === 0))
+            return [];
+        const now = this.now();
+        const armed = new Set();
+        const allCandidates = [path, ...(sinkTargets ?? [])].filter((p) => typeof p === "string" && p.length > 0);
+        for (const { rule, durationMs } of this.compiled) {
+            const tools = rule.when.tools ?? ["write", "edit"];
+            // The structured path is checked only when the tool matches when.tools (write/edit by default).
+            // Sink targets (redirect/tee) are checked regardless of tool: a bash redirect IS a write to that path,
+            // and the spec says bash edits that create the same condition via redirect arm when the glob matches.
+            const structuredMatch = tools.includes(tool) && allCandidates.some(candidate => {
+                const home = homedir();
+                const expanded = candidate === "~" || candidate.startsWith("~/") ? home + candidate.slice(1) : candidate;
+                const abs = resolve(cwd, expanded);
+                return editedMatches(rule, candidate) || editedMatches(rule, abs);
+            });
+            // When the tool is not in when.tools (e.g. bash), only sink targets are checked.
+            const sinkCandidates = tools.includes(tool) ? allCandidates : (sinkTargets ?? []).filter((p) => typeof p === "string" && p.length > 0);
+            const sinkMatch = sinkCandidates.some(candidate => {
+                const home = homedir();
+                const expanded = candidate === "~" || candidate.startsWith("~/") ? home + candidate.slice(1) : candidate;
+                const abs = resolve(cwd, expanded);
+                return editedMatches(rule, candidate) || editedMatches(rule, abs);
+            });
+            if (!structuredMatch && !sinkMatch)
+                continue;
+            const existing = this.armed.get(rule.id);
+            const paths = existing?.armedByPaths ?? [];
+            // Dedup the arming path.
+            const displayPath = path ?? allCandidates[0];
+            if (!paths.includes(displayPath))
+                paths.push(displayPath);
+            this.armed.set(rule.id, { expiresAt: now + durationMs, armedByPaths: paths });
+            armed.add(rule.id);
+        }
+        this.prune(now);
+        return [...armed];
+    }
+    /**
+     * Check whether any armed rule's command regex matches the given command.
+     * Returns the matching rules (with their compiled config) for the extension to act on.
+     * Expired rules are pruned first.
+     */
+    checkArmed(command) {
+        // Fix 5: avoid stripDataText (a non-trivial parse) when nothing is armed — the common case.
+        if (this.armed.size === 0)
+            return [];
+        const now = this.now();
+        this.prune(now);
+        if (this.armed.size === 0)
+            return []; // prune may have expired everything
+        const text = stripDataText(command).text;
+        const hits = [];
+        for (const { rule, commandRegex } of this.compiled) {
+            const entry = this.armed.get(rule.id);
+            if (!entry)
+                continue;
+            if (commandRegex.test(text)) {
+                hits.push({ id: rule.id, action: rule.action, ...(rule.message ? { message: rule.message } : {}), armedByPaths: entry.armedByPaths });
+            }
+        }
+        return hits;
+    }
+    /** Names of currently armed rules (for Jev context). */
+    armedRuleIds() {
+        this.prune(this.now());
+        return [...this.armed.keys()];
+    }
+    /** Human-readable status line for `/warden status`. */
+    statusLine() {
+        this.prune(this.now());
+        if (this.armed.size === 0)
+            return "";
+        const parts = [];
+        for (const [id, entry] of this.armed) {
+            const remaining = Math.max(0, entry.expiresAt - this.now());
+            const mins = Math.ceil(remaining / 60_000);
+            parts.push(`${id} (${mins}m left)`);
+        }
+        return `${this.armed.size} armed: ${parts.join(", ")}`;
+    }
+    /** Clear all arming state (called on session_start). */
+    reset() {
+        this.armed.clear();
+    }
+    /** Remove expired entries. */
+    prune(now) {
+        for (const [id, entry] of this.armed) {
+            if (entry.expiresAt <= now)
+                this.armed.delete(id);
+        }
+    }
+}