pi-warden 0.25.0 → 0.28.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +87 -0
- package/README.md +56 -176
- package/dist/backend.d.ts +23 -0
- package/dist/backend.js +36 -0
- package/dist/config.d.ts +21 -0
- package/dist/config.js +51 -0
- package/dist/extension.d.ts +1 -1
- package/dist/extension.js +170 -53
- package/dist/extension.js.bak +1369 -0
- package/dist/guard.d.ts +5 -0
- package/dist/guard.js +5 -0
- package/dist/index.d.ts +6 -4
- package/dist/index.js +4 -3
- package/dist/learning.d.ts +51 -5
- package/dist/learning.js +219 -29
- package/dist/output.d.ts +31 -1
- package/dist/output.js +80 -1
- package/dist/panel.d.ts +29 -0
- package/dist/panel.js +257 -10
- package/dist/saver.d.ts +2 -0
- package/dist/saver.js +4 -0
- package/dist/shape.js +4 -1
- package/dist/trace.d.ts +2 -0
- package/dist/widget.d.ts +96 -0
- package/dist/widget.js +259 -1
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,93 @@ How to keep this current: add the entry in the same pull request as the change,
|
|
|
6
6
|
|
|
7
7
|
## Unreleased
|
|
8
8
|
|
|
9
|
+
<!-- Empty. Next release starts here. -->
|
|
10
|
+
|
|
11
|
+
## 0.28.4
|
|
12
|
+
|
|
13
|
+
### Fixed
|
|
14
|
+
|
|
15
|
+
- Extension no longer crashes on load when node:sqlite is unavailable (e.g. some Node v25 builds). Learning features disabled gracefully; all guards still work.
|
|
16
|
+
|
|
17
|
+
## 0.28.3
|
|
18
|
+
|
|
19
|
+
### Fixed
|
|
20
|
+
|
|
21
|
+
- Temp output directories from saveOutput() are now cleaned up at session start (~2.5 MB/session leak).
|
|
22
|
+
|
|
23
|
+
## 0.28.2
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- One-time warning when confirm mode falls back to steer in headless sessions.
|
|
28
|
+
|
|
29
|
+
## 0.28.1
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
|
|
33
|
+
- TypeSafe consent disclosure now mentions the SQLite hold database, its contents, and the retention policy.
|
|
34
|
+
|
|
35
|
+
## 0.28.0
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
|
|
39
|
+
- \`learning.retentionDays\` config key (default 365) — prunes hold records older than this on startup. \`0\` disables pruning.
|
|
40
|
+
|
|
41
|
+
### Fixed
|
|
42
|
+
|
|
43
|
+
- SQLite hold database grew unboundedly; now pruned on startup with VACUUM.
|
|
44
|
+
- Empty catch blocks in learning.ts now log warnings instead of silently swallowing errors.
|
|
45
|
+
|
|
46
|
+
## 0.27.2
|
|
47
|
+
|
|
48
|
+
### Fixed
|
|
49
|
+
|
|
50
|
+
- Redact user prompts before storing in SQLite hold records (JSONL was already safe).
|
|
51
|
+
|
|
52
|
+
## 0.27.1
|
|
53
|
+
|
|
54
|
+
### Removed
|
|
55
|
+
|
|
56
|
+
- Dead `SessionCompressionTracker` class — `getMultiplier()` and `record()` were never called outside tests. ~50 lines removed.
|
|
57
|
+
|
|
58
|
+
## 0.27.0
|
|
59
|
+
|
|
60
|
+
### Added
|
|
61
|
+
|
|
62
|
+
- Sentence-style status bar templates (70+ across all 9 guards). `pickSentenceTemplate()` selects the best template from token context; zero LLM cost.
|
|
63
|
+
- `widget.barMode: "live"` (default): status bar shows one sentence like *"warden allowed bash action — write to src/config.ts"* instead of data-style tokens.
|
|
64
|
+
- `widget.barMode: "stack"`: the previous multi-guard data-style view.
|
|
65
|
+
- Compact sidebar mode: trace entries default to 1–3 lines; press `d` to toggle detailed view.
|
|
66
|
+
- Interactive config panel (`/warden config`): arrow-key navigation, Enter to toggle booleans or edit values, `s` to save.
|
|
67
|
+
- `/warden config set <key.path> <value>` and `/warden config get <key.path>` for quick CLI edits.
|
|
68
|
+
- `TraceEntry.tokens` field for live-mode re-rendering with sentence templates.
|
|
69
|
+
- `formatVerdictTokens()` returns both rendered line and raw tokens.
|
|
70
|
+
- `setNestedValue`, `getNestedValue`, `parseConfigValue` config helpers.
|
|
71
|
+
|
|
72
|
+
### Changed
|
|
73
|
+
|
|
74
|
+
- `widget.barMode` defaults to `"live"`.
|
|
75
|
+
- Removed 7 exported-but-never-called functions and 3 non-functional blocks (proactive guidance, adaptive thresholds, intent prediction).
|
|
76
|
+
|
|
77
|
+
### Fixed
|
|
78
|
+
|
|
79
|
+
- `/warden config` now works before any guard activity fires (`lastUi` set on session start).
|
|
80
|
+
|
|
81
|
+
## 0.26.0
|
|
82
|
+
|
|
83
|
+
### Added
|
|
84
|
+
|
|
85
|
+
- OpenRouter as a configurable judgment backend (`typesafeBackend` in user config). Set to `"openrouter"` to route Jev decisions through OpenRouter's decisions API instead of api.typesafe.ai. New `src/backend.ts` module; `OPENROUTER_API_KEY` env var; project overrides cannot redirect judgments.
|
|
86
|
+
|
|
87
|
+
### Changed
|
|
88
|
+
|
|
89
|
+
- Bumped pi-typesafe to ^0.6.0 (native `backend` option on `TypeSafeOptions`).
|
|
90
|
+
- Deduplicated `resolveBackend` call in `shape.ts`.
|
|
91
|
+
|
|
92
|
+
### Fixed
|
|
93
|
+
|
|
94
|
+
- Host-TUI capability guard: the extension no longer crashes when the host's bundled `pi-tui` lacks the `MouseRegion` component (e.g. omp 18.2.5). A missing named import was a link-time error that silently killed every guard, the `/warden` command, and the shortcut. The import is now a namespace property read (`tuiModule.MouseRegion`), which degrades to `undefined` instead of failing. The status widget renders the same guard text without the clickable wrapper.
|
|
95
|
+
|
|
9
96
|
## 0.25.0
|
|
10
97
|
|
|
11
98
|
### Added
|
package/README.md
CHANGED
|
@@ -1,174 +1,59 @@
|
|
|
1
1
|
# pi-warden
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Stop babysitting your coding agent.**
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
pi-warden supervises Pi while it works, catching risky actions, ignored rules, stuck loops, unverified "done" claims, security issues, runaway output, wasted context, and more.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Instead of interrupting you for every problem, Warden usually feeds the issue back to the agent so it can correct itself and keep going.
|
|
8
8
|
|
|
9
9
|

|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
## What it watches
|
|
14
|
-
|
|
15
|
-
| Guard | Watches | Does |
|
|
16
|
-
| --- | --- | --- |
|
|
17
|
-
| **Rules** | every `write` and `edit` | judges the written code against your project's Markdown rules (`pi-warden.md`) and names the violated rule to the agent |
|
|
18
|
-
| **Slop** | written code and final replies | names stubs, restating comments, dead code, hedging, padded replies; the agent fixes them in the next edit |
|
|
19
|
-
| **Stuck** | tool results | notices three failures with the same strategy and asks for a new hypothesis |
|
|
20
|
-
| **Done-check** | the final message | catches "done" claims after code changes when no test, build, or lint passed |
|
|
21
|
-
| **Context saver** | large or repeated tool output | keeps the exact lines that matter, stores the rest in a file, points recalls at a search |
|
|
22
|
-
| **Action** | every `bash`, `write`, `edit` (and context-mode's `ctx_execute*`) before it runs | holds irreversible calls and tells the agent why, so it re-plans or asks you; steers it back when a change is unrelated to your request |
|
|
23
|
-
| **Security** | written code and tool output | flags hardcoded secrets, disabled TLS, unsafe interpolation; marks injected instructions in tool output; fixture and documentation stand-ins are traced, never announced |
|
|
24
|
-
| **Runaway** | the reply stream | stops a reply that repeats the same block over and over (code only, no request) |
|
|
25
|
-
| **Steer budget** | the agent's attention | bounds the steers one run can demand: repeats and notices past the budget are recorded in the trace only, and a final reply that mostly restates an earlier reply of the same run is counted, never steered |
|
|
26
|
-
| **Subagent triage** | async subagent reports | Jev decides whether a child report is worth waking the agent for; routine progress stays in context and costs no request ([detail](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md#subagent-triage)) |
|
|
27
|
-
| **Notifications** (opt-in) | moments that need you | desktop notification for a held call, a confirm dialog, a runaway stop |
|
|
28
|
-
|
|
29
|
-
Three jobs, one channel: keep the agent on your rules, keep slop out of the code and the replies, keep the context small. The action guard is the seatbelt underneath: on 17,000 recorded calls it held 42 times. Everything arrives through the agent's own context, so the agent gets smarter and you are not the approval button. Full detail per guard, with thresholds and tuning numbers, is in [docs/guards.md](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md).
|
|
30
|
-
|
|
31
|
-
## In one minute
|
|
11
|
+
## Install
|
|
32
12
|
|
|
33
13
|
```bash
|
|
34
14
|
pi install npm:pi-warden
|
|
35
15
|
```
|
|
36
16
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
2. `/warden test` shows one synthetic verdict and what the agent would be told.
|
|
41
|
-
3. Work as usual. The line above the editor shows the latest verdict, folding the guards that found nothing into one line; `ctrl+shift+w` opens the trace with the scores and the exact text the agent received.
|
|
42
|
-
|
|
43
|
-
Requires Pi 0.85 or newer and Node.js 22.19 or newer. Without a key, the offline parts still run: the pattern list, the runaway stop, duplicate-output notes, sensitive-path notes, and the credential-shape warnings. The [examples folder](examples/README.md) has a starter rules file and both config files.
|
|
44
|
-
|
|
45
|
-
## Does it improve the code?
|
|
46
|
-
|
|
47
|
-
**Short answer: the warden cell did not break a rule in 150 paired runs. The control cells did, in 6 of them.**
|
|
48
|
-
|
|
49
|
-
`npm run eval:ab` runs the same fixture tasks through headless Pi twice. The control cell gets the rules as prose in `AGENTS.md`; the warden cell gets the same file plus pi-warden enforcing `pi-warden.md`. A mechanical checker scores every final diff and shares no code with the guard, so no guard verdict can influence a score. Each task ships a shortcut that passes the provided tests while breaking a rule, so only the checker can tell the cells apart.
|
|
50
|
-
|
|
51
|
-
| Model | Runs per cell | Control runs with a rule violation | Warden runs with a rule violation | Tests fully passing (control / warden) |
|
|
52
|
-
| --- | --- | --- | --- | --- |
|
|
53
|
-
| glm-5.3-flash | 60 | **5** | **0** | 54 of 60 / 55 of 60 |
|
|
54
|
-
| deepseek-v4.1-flash | 90 | **1** | **0** | 84 of 90 / 82 of 90 |
|
|
55
|
-
|
|
56
|
-
The rules the traps break are the ones no linter can check and no model knows from training: an endpoint pasted inline instead of imported from the config module, a `TODO` without an issue, `console.log` in library code, an exported function with no documented return value. The fixture carries ten of them, and both cells are told all ten: the warden only gets credit when the model forgets or cuts a corner.
|
|
57
|
-
|
|
58
|
-
The clipping task shows the shape of the effect. Without the warden, 6 of 10 glm runs cut user-visible text to fit. With it, 1 of 10, and every run that received a clip steer ended clean.
|
|
59
|
-
|
|
60
|
-
Run it on your own model: `npm run eval:ab -- --repeats 3 --model <model>` (it asks you to pick a model on purpose, because a full batch spends real tokens). Per-batch reports, including the runs where nothing changed, live in [eval/reports/](eval/reports/README.md).
|
|
61
|
-
|
|
62
|
-
The guards are also stable over repeated invocations. A continuous overnight run (2026-09-18) evaluated every guard across **109 cycles, 13,952 cases, 100% pass rate, zero score drift**. The same synthetic input produces the same judgment within rounding noise across all runs. Full numbers: [docs/overnight-eval.md](docs/overnight-eval.md) and [eval/reports/2026-09-18-overnight-stability/](eval/reports/2026-09-18-overnight-stability/).
|
|
63
|
-
|
|
64
|
-
### Why a strong model does not make this pointless
|
|
65
|
-
|
|
66
|
-
A guard like this is a smoke alarm. Most days it says nothing, and that is the design: it only pays off on the run where the model is tired, compacted, four hours in, or confident about a shortcut you would have caught in review. Four of its ten guards do not depend on the model being weak at all:
|
|
67
|
-
|
|
68
|
-
- **Context saver**: large tool output is replaced by the lines that matter plus a file pointer. That is your token bill, on any model.
|
|
69
|
-
- **Runaway stop** and **stuck** break a loop the model cannot see itself repeating, on any model.
|
|
70
|
-
- **Done-check** reads the final message against the checks that actually ran, so "all tests pass" gets checked, not trusted.
|
|
71
|
-
- **Subagent triage** scans async child reports so routine progress does not wake the parent agent or cost a request.
|
|
72
|
-
|
|
73
|
-
The rules side is calibrated for the tail, not the average. Judge it on the runs where it fires, which is what the [trace panel](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md) and `/warden status` hold precision are for.
|
|
74
|
-
|
|
75
|
-
### What it does not do yet
|
|
76
|
-
|
|
77
|
-
Published because the negative is as useful as the positive. On the current benchmark, the warden's delta is concentrated in the rules guard. Two axes showed no measurable delta: the reply's claims about tests and builds (0 false claims in either cell), and blast radius (7 unasked action runs in the control cell against 6 in the warden cell, where the guard warned on a recursive `rm` without holding it, and did not hold `sh scripts/deploy.sh` on a "deploy it" prompt). A 12-turn decay arc also found no rule-violation growth in either cell. Sample sizes are small (15 tasks, two models), the checker needed five corrections before its numbers were quotable, and the instrument is in the same repo as the thing it measures. Re-run it and quote the folder.
|
|
78
|
-
|
|
79
|
-
## Two moments from real sessions
|
|
80
|
-
|
|
81
|
-
Both are from my own work sessions, taken from the session logs. Project names are redacted, paths shortened, secrets replaced. The other two, a placeholder the agent talked itself out of removing and a retry loop broken mid-run, are in [docs/examples.md](https://github.com/DevMortimer/pi-warden/blob/main/docs/examples.md).
|
|
82
|
-
|
|
83
|
-
### It quoted a project rule back, and the agent fixed it in the next turn
|
|
84
|
-
|
|
85
|
-
The project's `pi-warden.md` rule "No hardcoded secrets" says connection URLs come from settings or the environment. The agent wrote a demo script with the local database DSN as a fallback default.
|
|
86
|
-
|
|
87
|
-
```text
|
|
88
|
-
assistant
|
|
89
|
-
[write] scripts/demo_rail_decisions.py (186 lines)
|
|
90
|
-
DSN = os.getenv("SUPABASE_DB_URL", "postgresql://[local default]")
|
|
91
|
-
|
|
92
|
-
pi-warden (steer message to the agent)
|
|
93
|
-
pi-warden: the content just written to scripts/demo_rail_decisions.py violates
|
|
94
|
-
project rule from pi-warden.md: "No hardcoded secrets" (0.88): Source and config
|
|
95
|
-
code must not contain passwords, API keys, tokens, or connection URLs (including
|
|
96
|
-
Supabase DSNs and `FLY_API_TOKEN` values). They come from settings
|
|
97
|
-
(`utils/config.py`) or the environ… [77 more chars]. Fix it in your next edit.
|
|
98
|
-
|
|
99
|
-
assistant
|
|
100
|
-
The warden flags the hardcoded local DSN. Right call. Making the script require
|
|
101
|
-
`SUPABASE_DB_URL` from the environment instead:
|
|
102
|
-
DSN = os.getenv("SUPABASE_DB_URL")
|
|
103
|
-
if not DSN:
|
|
104
|
-
sys.exit("Set SUPABASE_DB_URL to the local stack's DB_URL (supabase status)")
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
The rule is three lines of Markdown in the repo. No linter knows what a Supabase DSN looks like in a default argument; Jev does, and the agent got the rule text, not a red X.
|
|
108
|
-
|
|
109
|
-
### It held a database reset, and the agent finished everything else first
|
|
110
|
-
|
|
111
|
-
A feature branch with a migration. The agent decided to replay the local database. `supabase db reset` is legitimate in that repo, but it wipes local state and nobody had asked for it.
|
|
112
|
-
|
|
113
|
-
```text
|
|
114
|
-
assistant
|
|
115
|
-
[bash] cd .claude/worktrees/con-142 && supabase db reset 2>&1 | tail -6
|
|
116
|
-
|
|
117
|
-
tool result (written by pi-warden; the command did not run)
|
|
118
|
-
pi-warden held this bash call before it ran: irreversible 0.85. Do not retry it
|
|
119
|
-
unchanged. Either (1) reach the goal with a recoverable alternative that stays
|
|
120
|
-
inside the project (a targeted path, a dry run, a move instead of a delete, a
|
|
121
|
-
normal push), or (2) if this exact action is genuinely required, stop and tell
|
|
122
|
-
the user in one or two sentences what it does, what cannot be undone, and why
|
|
123
|
-
it is needed, then wait for their reply. [...the template continues]
|
|
124
|
-
|
|
125
|
-
assistant
|
|
126
|
-
The warden held `supabase db reset`. It's the sanctioned local-stack replay
|
|
127
|
-
(loopback docker only, no dev/prod reach), but I'll pause that step and ask at
|
|
128
|
-
the end of this update. Continuing with lanes that don't need it:
|
|
129
|
-
[bash] uv run pytest tests/canon tests/services/messaging ... 92 passed in 24.80s
|
|
130
|
-
|
|
131
|
-
assistant (final message, about 16 tool calls later)
|
|
132
|
-
Draft PR is up, work committed and pushed. One decision is pending for you.
|
|
133
|
-
PR: redacted#2640 (draft) ...
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
No dialog, no interruption. The agent moved the risky step to the end and asked in one sentence.
|
|
137
|
-
|
|
138
|
-
## How it decides
|
|
139
|
-
|
|
140
|
-
Every guarded call (`bash`, `write`, `edit`, and shell-like tools such as context-mode's `ctx_execute`) goes through four steps:
|
|
17
|
+
1. `/warden enable` to activate. Paste a key from [console.typesafe.ai](https://console.typesafe.ai) if you have one (hidden input, saved owner-only), or skip for offline-only mode.
|
|
18
|
+
2. `/warden test` to see one synthetic verdict.
|
|
19
|
+
3. Work as usual. Status line shows verdicts; `ctrl+shift+w` opens the trace sidebar.
|
|
141
20
|
|
|
142
|
-
|
|
143
|
-
2. **Pattern list, offline.** Force push, `git reset --hard`, recursive `rm` outside the project, SQL `DROP`/`TRUNCATE`/`DELETE FROM`, `curl | sh`, publishing, infrastructure destroys: the call is held. Risky but recoverable things (`rm -rf` on a project path, `sudo`, `--no-verify`, `gh pr merge`, touching `.env`) warn. Text that is data, such as a commit message that mentions `git push --force`, is not a command.
|
|
144
|
-
3. **Jev, with your consent.** One request carries your request, up to eight earlier messages, the agent's own words right before the call (its plan), and the call itself. Jev answers fixed questions with probabilities: irreversible, off task, does it change anything, how it relates to the task, does it contradict the plan, is the effect visible outside the working tree. On a write the slop and security questions ride along, and the rules check is a second request in parallel. About 0.3 s before the tool runs.
|
|
145
|
-
4. **Act.** Three things hold a call: a destructive pattern, a `deny` command rule (blocked outright, no dialog), or `irreversible` at 0.7 or above. Everything else (off task, plan mismatch, slop, a rule violation, a security smell) tells the agent and lets the call run.
|
|
21
|
+
Works without any key (offline guards: pattern list, runaway stop, sensitive-path notes, credential warnings). Requires Pi 0.85+, Node 22.19+.
|
|
146
22
|
|
|
147
|
-
|
|
23
|
+
## What Warden catches
|
|
148
24
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
**
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
25
|
+
| Guard | Watches | Does |
|
|
26
|
+
| --- | --- | --- |
|
|
27
|
+
| **Action** | `bash`, `write`, `edit` before run | Holds irreversible calls, steers off-task changes |
|
|
28
|
+
| **Rules** | every `write` and `edit` | Judges against project Markdown rules, quotes the broken rule |
|
|
29
|
+
| **Slop** | code and replies | Names stubs, restating comments, dead code, hedging |
|
|
30
|
+
| **Stuck** | tool results | Repeated failures using the same strategy |
|
|
31
|
+
| **Done-check** | final message | "Done" claims with no test/build/lint behind them |
|
|
32
|
+
| **Security** | code and output | Hardcoded secrets, injection risks |
|
|
33
|
+
| **Runaway** | reply stream | Stops replies that begin repeating themselves |
|
|
34
|
+
| **Subagent triage** | async child reports | Keeps noisy background work from waking the parent |
|
|
156
35
|
|
|
157
|
-
|
|
36
|
+
## How intervention works
|
|
158
37
|
|
|
159
|
-
|
|
38
|
+
1. **Read-only? Skip.** `git status`, `ls`, `read` means no request and no trace entry.
|
|
39
|
+
2. **Known-dangerous pattern? Catch it locally.** Force push, `git reset --hard`, recursive `rm`, SQL `DROP` held instantly.
|
|
40
|
+
3. **Needs judgment?** Warden evaluates the action in context.
|
|
41
|
+
4. **Steer or hold.** Most issues go back to the agent so it can correct itself. Irreversible actions can be stopped before they run.
|
|
160
42
|
|
|
161
|
-
|
|
162
|
-
An `AGENTS.md` is read once and asks the model to remember. Long sessions, compaction, and cheap models forget. pi-warden re-reads your rules file when it changes and judges every write against it at the moment it happens, then quotes the rule the agent broke. The rules can be things a linter cannot check: "exported functions declare return types", "a TODO must name a ticket", "comments must not restate the code". Keep the AGENTS.md; this is the check behind it.
|
|
43
|
+
In the default `steer` mode, Warden talks to the agent rather than interrupting you. `confirm` asks you directly; `advise` never blocks.
|
|
163
44
|
|
|
164
|
-
|
|
165
|
-
Only for Jev judgments. Without one you still get the pattern list, the runaway stop, duplicate-output notes, sensitive-path notes, and credential-shape warnings. Keys come through a waitlist at [console.typesafe.ai](https://console.typesafe.ai); install now and run `/warden enable` when yours arrives.
|
|
45
|
+
## More than guardrails
|
|
166
46
|
|
|
167
|
-
|
|
47
|
+
- **Learns from holds**, tracking outcomes and recommending policy changes via `/warden recommend`.
|
|
48
|
+
- **Custom policies**: command, path, and arming rules for your own workflow.
|
|
49
|
+
- **Context saver** trims oversized tool output while keeping the full result retrievable.
|
|
50
|
+
- **Multiple judgment backends**: TypeSafe by default, with OpenRouter support.
|
|
51
|
+
- **Full traceability**: inspect what Warden saw, decided, and told the agent.
|
|
52
|
+
- **Desktop alerts** for events that need you.
|
|
168
53
|
|
|
169
|
-
##
|
|
54
|
+
## Custom rules
|
|
170
55
|
|
|
171
|
-
Put `pi-warden.md` at the project root. Each heading is one rule
|
|
56
|
+
Put `pi-warden.md` at the project root. Each heading is one rule:
|
|
172
57
|
|
|
173
58
|
```markdown
|
|
174
59
|
# No console statements
|
|
@@ -179,45 +64,40 @@ paths: src/**/*.ts
|
|
|
179
64
|
Every exported function declares its return type.
|
|
180
65
|
```
|
|
181
66
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
Without a `pi-warden.md`, the first of `README.md`, `CLAUDE.md`, `AGENTS.md` is judged as one document instead, so existing context files work from day one. Starter file with a dozen rules: [`examples/pi-warden.md`](examples/pi-warden.md). Path globs, sensitive-path notes, and the tuning set: [docs/guards.md#rules](https://github.com/DevMortimer/pi-warden/blob/main/docs/guards.md#rules).
|
|
67
|
+
Jev judges every write and quotes violations back. Rules can be things no linter checks: "a TODO must name a ticket", "comments must not restate the code". Without `pi-warden.md`, the first of `README.md`/`CLAUDE.md`/`AGENTS.md` is judged instead. [Examples and config details.](docs/configuration.md)
|
|
185
68
|
|
|
186
|
-
##
|
|
69
|
+
## Does it actually help?
|
|
187
70
|
|
|
188
|
-
|
|
71
|
+
In 150 paired agent runs, the control setup violated the tested project rule **6 times**. With Warden: **0**.
|
|
189
72
|
|
|
190
|
-
|
|
73
|
+
An overnight stability run covered **13,952 guard cases across 109 cycles** with no score drift.
|
|
191
74
|
|
|
192
|
-
|
|
75
|
+
These are project-maintained benchmarks, not universal claims. Raw reports and reproducible eval tooling are in the repo.
|
|
193
76
|
|
|
194
|
-
|
|
77
|
+
[See the evals ->](eval/reports/)
|
|
195
78
|
|
|
196
|
-
|
|
79
|
+
## Privacy
|
|
197
80
|
|
|
198
|
-
|
|
81
|
+
Depending on the configured judgment backend, Warden sends to your provider: a redacted summary of each guarded call, the agent's plan, up to eight redacted prior messages, and redacted tool-output samples. Secrets are stripped before anything leaves the machine. [Full details.](docs/data-handling.md)
|
|
199
82
|
|
|
200
|
-
|
|
83
|
+
## Documentation
|
|
201
84
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
);
|
|
210
|
-
```
|
|
211
|
-
|
|
212
|
-
The rules guard, the session-level `ActionGuard` and `RulesGuard`, and the other exports are in [docs/extension-authors.md](https://github.com/DevMortimer/pi-warden/blob/main/docs/extension-authors.md).
|
|
85
|
+
- [docs/guards.md](docs/guards.md): every guard, thresholds, calibration, Jev questions
|
|
86
|
+
- [docs/configuration.md](docs/configuration.md): all config keys, defaults, status line templates
|
|
87
|
+
- [docs/commands.md](docs/commands.md): command reference
|
|
88
|
+
- [docs/faq.md](docs/faq.md): cost, permissions, headless CI, project overrides
|
|
89
|
+
- [docs/data-handling.md](docs/data-handling.md): what leaves the machine, what stays
|
|
90
|
+
- [docs/examples.md](docs/examples.md): real session examples
|
|
91
|
+
- [examples/](examples/): starter rules file, both config files
|
|
213
92
|
|
|
214
93
|
## Development
|
|
215
94
|
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
95
|
+
```bash
|
|
96
|
+
npm install
|
|
97
|
+
npm run check # typecheck + offline tests + build
|
|
98
|
+
npm run test:live # live tests (costs real judgments)
|
|
99
|
+
npm run eval:ab # A/B benchmark (costs real tokens)
|
|
100
|
+
```
|
|
221
101
|
|
|
222
102
|
## License
|
|
223
103
|
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
import type { TypeSafeOptions } from "pi-typesafe";
|
|
2
|
+
/** The judgment backend that receives pi-warden's Jev requests. */
|
|
3
|
+
export type JudgmentBackend = "typesafe" | "openrouter";
|
|
4
|
+
/** The host the consent disclosure names as the destination. */
|
|
5
|
+
export declare function backendHost(backend: JudgmentBackend): string;
|
|
6
|
+
/** Adapt the consent text to the active backend by substituting the destination host. */
|
|
7
|
+
export declare function disclosureFor(backend: JudgmentBackend, disclosure: string): string;
|
|
8
|
+
/**
|
|
9
|
+
* Whether the right credential exists for the chosen backend.
|
|
10
|
+
* TypeSafe uses the pi-typesafe keystore; other backends use an environment variable.
|
|
11
|
+
*/
|
|
12
|
+
export declare function keyAvailable(backend: JudgmentBackend, env: Record<string, string | undefined>, keystoreUsable: () => boolean): boolean;
|
|
13
|
+
/**
|
|
14
|
+
* Build the options forwarded to `createTypeSafe`.
|
|
15
|
+
* The caller must gate on `keyAvailable` before calling `createTypeSafe`.
|
|
16
|
+
*/
|
|
17
|
+
export declare function judgeOptions(config: {
|
|
18
|
+
maxRequests: number;
|
|
19
|
+
timeoutMs: number;
|
|
20
|
+
typesafeBackend: JudgmentBackend;
|
|
21
|
+
}): TypeSafeOptions;
|
|
22
|
+
/** Resolve the effective backend from a raw config value; invalid values fall back to "typesafe". */
|
|
23
|
+
export declare function resolveBackend(raw: unknown): JudgmentBackend;
|
package/dist/backend.js
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
import { authState } from "pi-typesafe";
|
|
2
|
+
const BACKENDS = {
|
|
3
|
+
typesafe: { host: "api.typesafe.ai" },
|
|
4
|
+
openrouter: { host: "openrouter.ai", keyEnv: "OPENROUTER_API_KEY" },
|
|
5
|
+
};
|
|
6
|
+
/** The host the consent disclosure names as the destination. */
|
|
7
|
+
export function backendHost(backend) {
|
|
8
|
+
return BACKENDS[backend].host;
|
|
9
|
+
}
|
|
10
|
+
/** Adapt the consent text to the active backend by substituting the destination host. */
|
|
11
|
+
export function disclosureFor(backend, disclosure) {
|
|
12
|
+
return backend === "typesafe" ? disclosure : disclosure.replace(BACKENDS.typesafe.host, BACKENDS[backend].host);
|
|
13
|
+
}
|
|
14
|
+
/**
|
|
15
|
+
* Whether the right credential exists for the chosen backend.
|
|
16
|
+
* TypeSafe uses the pi-typesafe keystore; other backends use an environment variable.
|
|
17
|
+
*/
|
|
18
|
+
export function keyAvailable(backend, env, keystoreUsable) {
|
|
19
|
+
const keyEnv = BACKENDS[backend].keyEnv;
|
|
20
|
+
return keyEnv ? (env[keyEnv]?.trim() ?? "") !== "" : keystoreUsable();
|
|
21
|
+
}
|
|
22
|
+
/**
|
|
23
|
+
* Build the options forwarded to `createTypeSafe`.
|
|
24
|
+
* The caller must gate on `keyAvailable` before calling `createTypeSafe`.
|
|
25
|
+
*/
|
|
26
|
+
export function judgeOptions(config) {
|
|
27
|
+
return {
|
|
28
|
+
maxRequests: config.maxRequests,
|
|
29
|
+
timeoutMs: config.timeoutMs,
|
|
30
|
+
...(config.typesafeBackend === "typesafe" ? {} : { backend: config.typesafeBackend }),
|
|
31
|
+
};
|
|
32
|
+
}
|
|
33
|
+
/** Resolve the effective backend from a raw config value; invalid values fall back to "typesafe". */
|
|
34
|
+
export function resolveBackend(raw) {
|
|
35
|
+
return raw === "typesafe" || raw === "openrouter" ? raw : "typesafe";
|
|
36
|
+
}
|
package/dist/config.d.ts
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import type { JudgmentBackend } from "./backend.js";
|
|
1
2
|
import type { WidgetConfig } from "./widget.js";
|
|
2
3
|
export interface Threshold {
|
|
3
4
|
/** P(yes) at or above this shows a warning and continues. */
|
|
@@ -212,11 +213,25 @@ export interface SubagentConfig {
|
|
|
212
213
|
export type RecallTool = "auto" | "rg" | "ag" | "ugrep" | "git-grep" | "grep" | "select-string" | "findstr" | "none";
|
|
213
214
|
export declare function isRecallTool(value: unknown): value is RecallTool;
|
|
214
215
|
export type WardenMode = "steer" | "confirm" | "advise";
|
|
216
|
+
export interface LearningConfig {
|
|
217
|
+
/** Enable adaptive thresholds based on learning data. */
|
|
218
|
+
adaptiveThresholds: boolean;
|
|
219
|
+
/** Enable pattern analysis and recommendations. */
|
|
220
|
+
patternAnalysis: boolean;
|
|
221
|
+
/** Minimum number of holds before adaptive thresholds kick in. */
|
|
222
|
+
minHoldsForAdaptive: number;
|
|
223
|
+
/** How aggressively to adjust thresholds (0-1). Higher values mean faster adaptation. */
|
|
224
|
+
adaptationRate: number;
|
|
225
|
+
/** Days to keep hold records in SQLite before pruning. 0 disables pruning. */
|
|
226
|
+
retentionDays: number;
|
|
227
|
+
}
|
|
215
228
|
export interface WardenConfig {
|
|
216
229
|
/** Master switch. false disables every guard, including offline pattern checks. */
|
|
217
230
|
enabled: boolean;
|
|
218
231
|
/** Consent to send task and action summaries to api.typesafe.ai. Set by /warden enable; never by a project file. */
|
|
219
232
|
typesafe: boolean;
|
|
233
|
+
/** The decisions service the judgments go to. User file only: a project must not redirect judgments to another vendor. */
|
|
234
|
+
typesafeBackend: JudgmentBackend;
|
|
220
235
|
/**
|
|
221
236
|
* steer (default): a confirm-level call is held and the agent receives the judgment as its tool result, so it re-plans or asks
|
|
222
237
|
* the user in chat. confirm: open a dialog and let the user decide (falls back to steer without a UI). advise: never hold; report only.
|
|
@@ -248,6 +263,8 @@ export interface WardenConfig {
|
|
|
248
263
|
* steer costs at least one LLM turn, and a closing run that collects six notices collects six restatements of the final
|
|
249
264
|
* status. 0 disables the budget. Critical guards (stuck, done, runaway, subagent wake) always deliver. */
|
|
250
265
|
steerBudget: number;
|
|
266
|
+
/** Learning and adaptation settings. */
|
|
267
|
+
learning: LearningConfig;
|
|
251
268
|
}
|
|
252
269
|
export declare const PACKAGE_NAME = "pi-warden";
|
|
253
270
|
/** Bumped when WardenConfig gains a section; extension.ts checks it so a half-updated module graph is reported, not crashed on. */
|
|
@@ -274,4 +291,8 @@ export declare function readUserConfig(): Json;
|
|
|
274
291
|
export declare function writeUserConfig(raw: Json): string;
|
|
275
292
|
/** Persists one top-level user setting without disturbing the rest of the file. */
|
|
276
293
|
export declare function setUserSetting(key: "typesafe" | "enabled" | "mode", value: boolean | WardenMode): string;
|
|
294
|
+
export declare function setNestedValue(obj: Record<string, unknown>, path: string, value: unknown): Record<string, unknown>;
|
|
295
|
+
export declare function getNestedValue(obj: Record<string, unknown>, path: string): unknown;
|
|
296
|
+
/** Coerce a CLI string into a JSON primitive so `/warden config set` stays ergonomic. */
|
|
297
|
+
export declare function parseConfigValue(value: string): unknown;
|
|
277
298
|
export {};
|
package/dist/config.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { chmodSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
|
|
2
2
|
import { homedir } from "node:os";
|
|
3
3
|
import { dirname, join } from "node:path";
|
|
4
|
+
import { resolveBackend } from "./backend.js";
|
|
4
5
|
import { COMMAND_TOOLS } from "./tools.js";
|
|
5
6
|
import { defaultWidgetConfig } from "./widget.js";
|
|
6
7
|
const RECALL_TOOLS = ["auto", "rg", "ag", "ugrep", "git-grep", "grep", "select-string", "findstr", "none"];
|
|
@@ -15,6 +16,7 @@ export function defaultConfig() {
|
|
|
15
16
|
return {
|
|
16
17
|
enabled: true,
|
|
17
18
|
typesafe: false,
|
|
19
|
+
typesafeBackend: "typesafe",
|
|
18
20
|
mode: "steer",
|
|
19
21
|
timeoutMs: 5000,
|
|
20
22
|
maxRequests: 500,
|
|
@@ -47,6 +49,7 @@ export function defaultConfig() {
|
|
|
47
49
|
steerVisible: false,
|
|
48
50
|
notices: false,
|
|
49
51
|
steerBudget: 3,
|
|
52
|
+
learning: { adaptiveThresholds: true, patternAnalysis: true, minHoldsForAdaptive: 20, adaptationRate: 0.1, retentionDays: 365 },
|
|
50
53
|
};
|
|
51
54
|
}
|
|
52
55
|
/** Mirrors Pi's agent directory rule so the file sits next to pi-typesafe's auth.json. */
|
|
@@ -349,6 +352,7 @@ function applyWidget(base, raw) {
|
|
|
349
352
|
return {
|
|
350
353
|
enabled: boolean(raw.enabled, base.enabled),
|
|
351
354
|
placement: raw.placement === "belowEditor" || raw.placement === "aboveEditor" ? raw.placement : base.placement,
|
|
355
|
+
barMode: raw.barMode === "live" || raw.barMode === "stack" ? raw.barMode : base.barMode,
|
|
352
356
|
shortcut: typeof raw.shortcut === "string" ? raw.shortcut.trim() : base.shortcut,
|
|
353
357
|
panelWidth: typeof raw.panelWidth === "number" && Number.isSafeInteger(raw.panelWidth) && raw.panelWidth >= 20 ? raw.panelWidth
|
|
354
358
|
: typeof raw.panelWidth === "string" && /^[1-9]\d?%$/.test(raw.panelWidth.trim()) ? raw.panelWidth.trim() : base.panelWidth,
|
|
@@ -404,6 +408,7 @@ export function applyUserOverrides(base, raw) {
|
|
|
404
408
|
return {
|
|
405
409
|
enabled: boolean(raw.enabled, base.enabled),
|
|
406
410
|
typesafe: boolean(raw.typesafe, base.typesafe),
|
|
411
|
+
typesafeBackend: resolveBackend(raw.typesafeBackend),
|
|
407
412
|
mode: isMode(raw.mode) ? raw.mode : base.mode,
|
|
408
413
|
...shared,
|
|
409
414
|
...applyGuards(base, raw, shared.timeoutMs, "user"),
|
|
@@ -411,6 +416,18 @@ export function applyUserOverrides(base, raw) {
|
|
|
411
416
|
steerVisible: boolean(raw.steerVisible, base.steerVisible),
|
|
412
417
|
notices: boolean(raw.notices, base.notices),
|
|
413
418
|
steerBudget: typeof raw.steerBudget === "number" && Number.isInteger(raw.steerBudget) && raw.steerBudget >= 0 ? raw.steerBudget : base.steerBudget,
|
|
419
|
+
learning: applyLearning(base.learning, raw.learning),
|
|
420
|
+
};
|
|
421
|
+
}
|
|
422
|
+
function applyLearning(base, raw) {
|
|
423
|
+
if (!isObject(raw))
|
|
424
|
+
return base;
|
|
425
|
+
return {
|
|
426
|
+
adaptiveThresholds: boolean(raw.adaptiveThresholds, base.adaptiveThresholds),
|
|
427
|
+
patternAnalysis: boolean(raw.patternAnalysis, base.patternAnalysis),
|
|
428
|
+
minHoldsForAdaptive: Math.max(5, positiveInteger(raw.minHoldsForAdaptive, base.minHoldsForAdaptive)),
|
|
429
|
+
adaptationRate: Math.max(0, Math.min(1, probability(raw.adaptationRate, base.adaptationRate))),
|
|
430
|
+
retentionDays: Math.max(0, positiveInteger(raw.retentionDays, base.retentionDays)),
|
|
414
431
|
};
|
|
415
432
|
}
|
|
416
433
|
/** Project files may tune the guards but cannot grant TypeSafe consent, change the mode, or raise budgets. */
|
|
@@ -442,3 +459,37 @@ export function writeUserConfig(raw) {
|
|
|
442
459
|
export function setUserSetting(key, value) {
|
|
443
460
|
return writeUserConfig({ ...readUserConfig(), [key]: value });
|
|
444
461
|
}
|
|
462
|
+
export function setNestedValue(obj, path, value) {
|
|
463
|
+
const keys = path.split(".");
|
|
464
|
+
const result = { ...obj };
|
|
465
|
+
let current = result;
|
|
466
|
+
for (let i = 0; i < keys.length - 1; i++) {
|
|
467
|
+
const key = keys[i];
|
|
468
|
+
current[key] = { ...(current[key] ?? {}) };
|
|
469
|
+
current = current[key];
|
|
470
|
+
}
|
|
471
|
+
current[keys.at(-1)] = value;
|
|
472
|
+
return result;
|
|
473
|
+
}
|
|
474
|
+
export function getNestedValue(obj, path) {
|
|
475
|
+
const keys = path.split(".");
|
|
476
|
+
let current = obj;
|
|
477
|
+
for (const key of keys) {
|
|
478
|
+
if (typeof current !== "object" || current === null)
|
|
479
|
+
return undefined;
|
|
480
|
+
current = current[key];
|
|
481
|
+
}
|
|
482
|
+
return current;
|
|
483
|
+
}
|
|
484
|
+
/** Coerce a CLI string into a JSON primitive so `/warden config set` stays ergonomic. */
|
|
485
|
+
export function parseConfigValue(value) {
|
|
486
|
+
if (value === "true")
|
|
487
|
+
return true;
|
|
488
|
+
if (value === "false")
|
|
489
|
+
return false;
|
|
490
|
+
if (value === "null")
|
|
491
|
+
return null;
|
|
492
|
+
if (/^-?\d+(\.\d+)?$/.test(value))
|
|
493
|
+
return Number(value);
|
|
494
|
+
return value;
|
|
495
|
+
}
|
package/dist/extension.d.ts
CHANGED
|
@@ -2,7 +2,7 @@ import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-a
|
|
|
2
2
|
import type { ToolCallRef } from "./action-guard.js";
|
|
3
3
|
import type { SlopSymptom, Verdict } from "./guard.js";
|
|
4
4
|
import type { ShapeResult } from "./shape.js";
|
|
5
|
-
export declare const disclosure = "With TypeSafe judgments enabled, pi-warden sends to api.typesafe.ai: your latest request and up to eight redacted prior user/assistant text messages for task context, plus a redacted, truncated summary of each guarded bash, write, or edit call before it runs, with the agent's own words from the message that makes the call (its stated plan); for a write or edit in a project with a rules file (pi-warden.md, the configured files, or README/CLAUDE/AGENTS as fallback), a larger redacted sample of the written content with the current file around each edit and the rule text; the last few tool calls and output tails when the agent keeps failing; the agent's final message when it reports completion without running checks; redacted tool-output samples for security and context saving (retention and output format); a redacted sample of an async subagent report that names a failure, a stop, or a question, with your latest request, when warden decides whether that report should wake the agent; and, on the first guarded call after your reply, the redacted summaries of the calls allowed in the previous turn, so Jev can say whether your reply regrets one of them. Compression and duplicate notes store an exact, owner-only copy in a temporary file on this machine; the hold feedback log stores tool names, pattern ids, scores, and outcomes (never commands) in an owner-only file under Pi's agent directory. Requests may incur charges. Secret redaction is best-effort. Results are model judgments, not proof or authorization; offline pattern checks stay active either way.";
|
|
5
|
+
export declare const disclosure = "With TypeSafe judgments enabled, pi-warden sends to api.typesafe.ai: your latest request and up to eight redacted prior user/assistant text messages for task context, plus a redacted, truncated summary of each guarded bash, write, or edit call before it runs, with the agent's own words from the message that makes the call (its stated plan); for a write or edit in a project with a rules file (pi-warden.md, the configured files, or README/CLAUDE/AGENTS as fallback), a larger redacted sample of the written content with the current file around each edit and the rule text; the last few tool calls and output tails when the agent keeps failing; the agent's final message when it reports completion without running checks; redacted tool-output samples for security and context saving (retention and output format); a redacted sample of an async subagent report that names a failure, a stop, or a question, with your latest request, when warden decides whether that report should wake the agent; and, on the first guarded call after your reply, the redacted summaries of the calls allowed in the previous turn, so Jev can say whether your reply regrets one of them. Compression and duplicate notes store an exact, owner-only copy in a temporary file on this machine; the hold feedback log stores tool names, pattern ids, scores, and outcomes (never commands) in an owner-only file under Pi's agent directory; an owner-only SQLite database under Pi's agent directory stores redacted hold context (plan, summary, reason) for learning and retention (configurable, default 365 days). Requests may incur charges. Secret redaction is best-effort. Results are model judgments, not proof or authorization; offline pattern checks stay active either way.";
|
|
6
6
|
/** Which guard spent the user's attention. The status line reports one count per guard. */
|
|
7
7
|
export type SteerGuard = "action" | "rules" | "security" | "stuck" | "done" | "prose" | "runaway" | "subagent";
|
|
8
8
|
/**
|