pi-warden 0.9.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -6
- package/dist/action-guard.d.ts +5 -1
- package/dist/action-guard.js +2 -2
- package/dist/config.d.ts +4 -0
- package/dist/config.js +4 -0
- package/dist/extension.d.ts +6 -1
- package/dist/extension.js +100 -15
- package/dist/guard.d.ts +92 -0
- package/dist/guard.js +64 -7
- package/dist/holds.d.ts +123 -0
- package/dist/holds.js +196 -0
- package/dist/index.d.ts +4 -2
- package/dist/index.js +2 -1
- package/dist/shape.js +6 -1
- package/dist/trace.d.ts +2 -0
- package/dist/trace.js +12 -1
- package/dist/widget.d.ts +1 -1
- package/dist/widget.js +4 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -66,7 +66,9 @@ Runs on `tool_call`, before the tool executes.
|
|
|
66
66
|
2. **Patterns**, offline: force pushes, `git reset --hard`, `git clean`, recursive `rm` on absolute, home, variable, or parent paths, SQL `DROP`/`TRUNCATE`/`DELETE FROM`, block-device writes, `chmod -R 777`, fork bombs, `curl | sh`, `kill -1`, shutdown, package publishing, infrastructure destroys hold the call. `rm -rf` on a project path, `git checkout -- .`, `git branch -D`, `git stash drop`, `find -delete`, `sudo` warn. Reads or writes of `.env`, SSH, AWS, npm, kube, and other credential files warn. A `write` that overwrites a file outside the project holds; creating or editing outside the project warns.
|
|
67
67
|
|
|
68
68
|
Text that is data is not a command. A heredoc body written to a file, a quoted `echo`/`printf` argument, a `grep` pattern, or a `git commit -m` message can mention `git push --force` without a hold. The same text fed to `sh`, `bash -c`, `eval`, `xargs`, or a `python3 - <<EOF` script that calls `os.system` keeps every hit.
|
|
69
|
-
3. **Jev**, with consent: one request with `{ task, context, action }` and four questions. `irreversible` (yes/no), `off_task` (yes/no), `mutates` (does it change anything), `scope` (expected step, plausible side step, unrelated, unclear). Defaults: irreversible at 0.5 warns and at 0.7 holds; off-task at 0.6 warns and at 0.85 with `unrelated` holds, but only when the action can change something. An unrelated `grep` is warned about, never held. Patterns set the floor; Jev can only raise it.
|
|
69
|
+
3. **Jev**, with consent: one request with `{ task, context, plan, action }` and four questions. `irreversible` (yes/no), `off_task` (yes/no), `mutates` (does it change anything), `scope` (expected step, plausible side step, unrelated, unclear). Defaults: irreversible at 0.5 warns and at 0.7 holds; off-task at 0.6 warns and at 0.85 with `unrelated` holds, but only when the action can change something. An unrelated `grep` is warned about, never held. Patterns set the floor; Jev can only raise it.
|
|
70
|
+
|
|
71
|
+
`plan` is the agent's own words in the message that makes the call (or its latest text since your prompt, 500 redacted characters). It tells Jev which step this is, so a verification fixture the agent just announced is not judged unrelated; it never authorizes anything. When there is a plan, a fifth question `intent_mismatch` asks whether the call does something materially different from it: a delete where the plan said list, a force push where it said push. At `action.intentMismatch` (0.8) on a call that can change something, the call is warned about and the agent is told to keep its words and its calls in step. Never held on that alone; the hold feedback below will say whether it should be. The trace shows the plan under each verdict.
|
|
70
72
|
4. **Act**, by mode:
|
|
71
73
|
- `steer` (default): a hold blocks the call and returns the judgment to the agent as its tool result, with the two acceptable next moves: find a recoverable alternative, or explain the action to you and wait. If your reply approves it, the retry goes through (Jev reads your reply; offline, a yes/go-ahead heuristic does).
|
|
72
74
|
- `confirm`: a `ctx.ui.confirm` dialog. No blocks with a short reason. Falls back to `steer` without a UI.
|
|
@@ -74,6 +76,8 @@ Runs on `tool_call`, before the tool executes.
|
|
|
74
76
|
|
|
75
77
|
The action guard receives your latest message plus up to eight earlier user and assistant messages (750 redacted characters each) so follow-ups and side comments do not replace the task. Sibling tool calls in one assistant message are judged together in one round trip. If TypeSafe cannot answer, the call is allowed with a warning (`failOpen: true`; set it to `false` to hold instead).
|
|
76
78
|
|
|
79
|
+
**Hold feedback.** What you do next labels each judgment, so hold precision is measured on your sessions rather than assumed. A hold your reply releases (or the confirm dialog allows) was a false positive; a hold you decline, or that nobody approves after you replied and the next turn ended, stood. An allowed call your next message tells the agent to stop, undo, or revert was a miss: one `regretted` question rides the first action request after your reply, with the redacted summaries of last turn's allowed calls (a locator names the one when there are several); offline, a stop-word heuristic stands in. `/warden status` shows the counts and the precision, the trace entry of each call gets its outcome, and every judged call is written with its scores and outcome to an owner-only per-session file under `~/.pi/agent/pi-warden/holds/` (tool, pattern ids, scores, level, mode, outcome; never the command). `"action": { "feedbackLog": false }` keeps the counts and skips the file.
|
|
80
|
+
|
|
77
81
|
### Rules
|
|
78
82
|
|
|
79
83
|
Write your project's rules as Markdown headings in `pi-warden.md` at the project root (a starter file with a dozen rules is in [`examples/pi-warden.md`](examples/pi-warden.md)):
|
|
@@ -153,7 +157,7 @@ Nudges from the rules, slop, stuck, done, and prose guards are custom messages i
|
|
|
153
157
|
|
|
154
158
|
| Command | Effect |
|
|
155
159
|
| --- | --- |
|
|
156
|
-
| `/warden status` | Guard state, consent and key source, session counts, thresholds, rules source, config paths, last verdicts |
|
|
160
|
+
| `/warden status` | Guard state, consent and key source, session counts, thresholds, hold precision and feedback log, rules source, config paths, last verdicts |
|
|
157
161
|
| `/warden enable` | Data notice, key prompt if none is stored, consent saved |
|
|
158
162
|
| `/warden disable` | Stop Jev judgments; pattern checks continue |
|
|
159
163
|
| `/warden mode steer\|confirm\|advise` | How holds are handled; without an argument, show the current mode |
|
|
@@ -182,7 +186,7 @@ Templates in `config.widget` control the text. Segments are separated by ` · `;
|
|
|
182
186
|
}
|
|
183
187
|
```
|
|
184
188
|
|
|
185
|
-
Tokens per guard: action `tool level source irreversible offTask scope approved slop slopStub slopComments slopDead slopHedging patterns reasons path model ms flags time`; rules `tool path asked violations status source reasons model ms flags time`; prose `wordy cliches jargon status reasons model ms flags time`; stuck `failures sameStrategy approachChange progress status source reasons model ms flags time`; done `changes checks checksPassed claimsDone claimsVerified checksApply outcome status reasons model ms flags time`; security `tool injection exfiltration status`; context `tool retention bytesSaved`; runaway `kind count chars signal block status time`. `"enabled": false` hides the line; `"shortcut": ""` disables the keybinding.
|
|
189
|
+
Tokens per guard: action `tool level source irreversible offTask scope approved intent plan slop slopStub slopComments slopDead slopHedging patterns reasons path model ms flags time`; rules `tool path asked violations status source reasons model ms flags time`; prose `wordy cliches jargon status reasons model ms flags time`; stuck `failures sameStrategy approachChange progress status source reasons model ms flags time`; done `changes checks checksPassed claimsDone claimsVerified checksApply outcome status reasons model ms flags time`; security `tool injection exfiltration status`; context `tool retention bytesSaved`; runaway `kind count chars signal block status time`. `"enabled": false` hides the line; `"shortcut": ""` disables the keybinding.
|
|
186
190
|
|
|
187
191
|
## Configuration
|
|
188
192
|
|
|
@@ -200,7 +204,9 @@ User file `~/.pi/agent/pi-warden/config.json` (owner-only). Missing keys use the
|
|
|
200
204
|
"tools": ["bash", "powershell", "ctx_execute", "ctx_batch_execute", "ctx_execute_file", "write", "edit"],
|
|
201
205
|
"failOpen": true,
|
|
202
206
|
"irreversible": { "warn": 0.5, "confirm": 0.7 },
|
|
203
|
-
"offTask": { "warn": 0.6, "confirm": 0.85 }
|
|
207
|
+
"offTask": { "warn": 0.6, "confirm": 0.85 },
|
|
208
|
+
"intentMismatch": 0.8,
|
|
209
|
+
"feedbackLog": true
|
|
204
210
|
},
|
|
205
211
|
"rules": {
|
|
206
212
|
"enabled": true,
|
|
@@ -249,13 +255,15 @@ A wince-style setup for a backend repo (the full version is [`examples/pi-warden
|
|
|
249
255
|
|
|
250
256
|
With consent, requests go to `https://api.typesafe.ai`. What is sent:
|
|
251
257
|
|
|
252
|
-
- **Action guard**: your latest prompt (1500 characters), up to eight earlier user and assistant messages (750 redacted characters each), the tool name, the command (2000 characters) or the file path (relative inside the project, `~`-shortened outside), whether the file exists, a 1500-character head/middle/tail sample of a `write`, the first three edit pairs (400 characters each) of an `edit`.
|
|
258
|
+
- **Action guard**: your latest prompt (1500 characters), up to eight earlier user and assistant messages (750 redacted characters each), the agent's text from the message that makes the call (500 redacted characters), the tool name, the command (2000 characters) or the file path (relative inside the project, `~`-shortened outside), whether the file exists, a 1500-character head/middle/tail sample of a `write`, the first three edit pairs (400 characters each) of an `edit`. On the first guarded call after your reply, the tool names and commands (300 characters) or paths of up to six calls allowed in the previous turn, for the regret question.
|
|
253
259
|
- **Rules**: the project-relative path, a 6000-character sample of a `write` or each edit's new text (1500 characters) with about 40 lines of the current file around the replaced text, and the rule text from your rules file or the condensed fallback document (`rules.maxChars`). No task text. Files under `rules.exclude` are never sent.
|
|
254
260
|
- **Stuck**: the last 12 tool calls (300 characters each) with 400-character output tails.
|
|
255
261
|
- **Done-check and prose**: the agent's final message (2000 and 2500 characters), the run's check commands, the audience description.
|
|
256
262
|
- **Output checks**: a redacted head/tail sample up to 6000 characters plus size, line counts, and tool name.
|
|
257
263
|
- **Nothing** for duplicate detection, the runaway guard, sensitive-path notes, or pattern checks.
|
|
258
264
|
|
|
265
|
+
The hold feedback log under `~/.pi/agent/pi-warden/holds/` stays on this machine, owner-only: one JSON line per judged call with tool, pattern ids, scores, level, mode, outcome, and the length of the agent's stated plan; never the command, path, prompt, or plan text.
|
|
266
|
+
|
|
259
267
|
Obvious credentials (`Authorization` headers, `TOKEN=` and `SECRET=` assignments, `sk-`, `ghp_`, `AKIA`, JWTs, URL passwords, PEM blocks) are replaced with `[redacted]` before sending. Best-effort; do not rely on it for prompts that contain secrets. Text steered to the agent names the tool, the reasons, and the scores, not the command. UI errors never include upstream response bodies or keys. Judgments are model output; thresholds are yours to tune.
|
|
260
268
|
|
|
261
269
|
## After updating the package
|
|
@@ -290,7 +298,7 @@ What spans calls in a session lives in `ActionGuard` (hold, reply, retry approva
|
|
|
290
298
|
```bash
|
|
291
299
|
npm install
|
|
292
300
|
npm run check # typecheck, offline tests (mocked transport), build
|
|
293
|
-
npm run test:live # billable synthetic judgments across the guards; pass action|slop|approval|stuck|done|security|context for one group
|
|
301
|
+
npm run test:live # billable synthetic judgments across the guards; pass action|intent|slop|approval|regret|stuck|done|security|context for one group
|
|
294
302
|
node scripts/rules-cases.mjs # 13 billable cases against an 8-rule fixture file
|
|
295
303
|
node scripts/slop-cases.mjs # 22 billable cases for the slop and prose questions
|
|
296
304
|
node scripts/security-cases.mjs # 9 output-security and task-continuity cases
|
package/dist/action-guard.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { ActionGuardConfig, SecurityConfig, SlopGuardConfig } from "./config.js";
|
|
2
|
-
import type { TaskMessage, Verdict } from "./guard.js";
|
|
2
|
+
import type { PreviousAction, TaskMessage, Verdict } from "./guard.js";
|
|
3
3
|
import type { Judge } from "./jev.js";
|
|
4
4
|
/** One tool call as the agent proposed it. `id` is Pi's tool call id, stable across hooks and retries. */
|
|
5
5
|
export interface ToolCallRef {
|
|
@@ -14,6 +14,8 @@ export interface Conversation {
|
|
|
14
14
|
context?: readonly TaskMessage[] | undefined;
|
|
15
15
|
/** Tool calls in the same assistant message, this one included; they are judged together. */
|
|
16
16
|
siblings?: readonly ToolCallRef[] | undefined;
|
|
17
|
+
/** The agent's own words in that message (or its latest text under this prompt); shared by the siblings. Explains, never authorizes. */
|
|
18
|
+
plan?: string | undefined;
|
|
17
19
|
}
|
|
18
20
|
export interface InspectOptions {
|
|
19
21
|
config: ActionGuardConfig;
|
|
@@ -23,6 +25,8 @@ export interface InspectOptions {
|
|
|
23
25
|
signal?: AbortSignal | undefined;
|
|
24
26
|
slop?: SlopGuardConfig | undefined;
|
|
25
27
|
security?: SecurityConfig | undefined;
|
|
28
|
+
/** Calls allowed in the previous turn; the regret question about them rides this call's request, never a sibling's. */
|
|
29
|
+
previousActions?: readonly PreviousAction[] | undefined;
|
|
26
30
|
}
|
|
27
31
|
/**
|
|
28
32
|
* The Action guard for one session. `evaluateAction` judges a single call; this module owns what spans calls:
|
package/dist/action-guard.js
CHANGED
|
@@ -20,7 +20,7 @@ export class ActionGuard {
|
|
|
20
20
|
const { task } = conversation;
|
|
21
21
|
// A hold happened under an earlier prompt and the user has since replied: ask whether the reply approves this action.
|
|
22
22
|
const retryAfterHold = this.holdPending && this.lastHoldPrompt !== task;
|
|
23
|
-
const judgeCall = (tool, input) => evaluateAction({ tool, input, cwd: options.cwd, task, context: conversation.context }, { config: options.config, judge: options.judge, signal: options.signal, slop: options.slop, security: options.security, retryAfterHold });
|
|
23
|
+
const judgeCall = (tool, input, previousActions) => evaluateAction({ tool, input, cwd: options.cwd, task, context: conversation.context, plan: conversation.plan }, { config: options.config, judge: options.judge, signal: options.signal, slop: options.slop, security: options.security, retryAfterHold, previousActions });
|
|
24
24
|
// A retry after a hold stays sequential because an approval consumed by one sibling changes the question for the next.
|
|
25
25
|
if (options.judge && !retryAfterHold) {
|
|
26
26
|
for (const sibling of conversation.siblings ?? []) {
|
|
@@ -33,7 +33,7 @@ export class ActionGuard {
|
|
|
33
33
|
}
|
|
34
34
|
const key = JSON.stringify(call.input);
|
|
35
35
|
const ready = this.prejudged.get(call.id);
|
|
36
|
-
const pending = ready && !ready.used && ready.key === key && !retryAfterHold ? ready.verdict : judgeCall(call.tool, call.input);
|
|
36
|
+
const pending = ready && !ready.used && ready.key === key && !retryAfterHold ? ready.verdict : judgeCall(call.tool, call.input, options.previousActions);
|
|
37
37
|
this.prejudged.set(call.id, { key, verdict: pending, used: true });
|
|
38
38
|
const verdict = await pending;
|
|
39
39
|
if (retryAfterHold && !options.judge && verdict.level === "confirm" && textApproves(task)) {
|
package/dist/config.d.ts
CHANGED
|
@@ -15,6 +15,10 @@ export interface ActionGuardConfig {
|
|
|
15
15
|
timeoutMs: number;
|
|
16
16
|
irreversible: Threshold;
|
|
17
17
|
offTask: Threshold;
|
|
18
|
+
/** P(the call differs from the agent's own stated plan) at or above this warns and tells the agent; never holds on its own. */
|
|
19
|
+
intentMismatch: number;
|
|
20
|
+
/** Write each judged call and what the user did next (approved, declined, re-planned, regretted) to an owner-only per-session file under the agent directory; redacted, never the command. */
|
|
21
|
+
feedbackLog: boolean;
|
|
18
22
|
}
|
|
19
23
|
export interface StuckGuardConfig {
|
|
20
24
|
enabled: boolean;
|
package/dist/config.js
CHANGED
|
@@ -25,6 +25,8 @@ export function defaultConfig() {
|
|
|
25
25
|
timeoutMs: 5000,
|
|
26
26
|
irreversible: { warn: 0.5, confirm: 0.7 },
|
|
27
27
|
offTask: { warn: 0.6, confirm: 0.85 },
|
|
28
|
+
intentMismatch: 0.8,
|
|
29
|
+
feedbackLog: true,
|
|
28
30
|
},
|
|
29
31
|
stuck: { enabled: true, window: 12, minFailures: 3, cooldown: 3, sameStrategy: 0.7, nudge: true },
|
|
30
32
|
done: { enabled: true, claimsDone: 0.7, nudge: true },
|
|
@@ -92,6 +94,8 @@ function applyAction(base, raw, timeoutMs) {
|
|
|
92
94
|
timeoutMs,
|
|
93
95
|
irreversible: threshold(raw.irreversible, base.irreversible),
|
|
94
96
|
offTask: threshold(raw.offTask, base.offTask),
|
|
97
|
+
intentMismatch: probability(raw.intentMismatch, base.intentMismatch),
|
|
98
|
+
feedbackLog: boolean(raw.feedbackLog, base.feedbackLog),
|
|
95
99
|
};
|
|
96
100
|
}
|
|
97
101
|
function applyStuck(base, raw) {
|
package/dist/extension.d.ts
CHANGED
|
@@ -2,12 +2,17 @@ import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-a
|
|
|
2
2
|
import type { ToolCallRef } from "./action-guard.js";
|
|
3
3
|
import type { SlopSymptom, Verdict } from "./guard.js";
|
|
4
4
|
import type { ShapeResult } from "./shape.js";
|
|
5
|
-
export declare const disclosure = "With TypeSafe judgments enabled, pi-warden sends to api.typesafe.ai: your latest request and up to eight redacted prior user/assistant text messages for task context, plus a redacted, truncated summary of each guarded bash, write, or edit call before it runs; for a write or edit in a project with a rules file (pi-warden.md, the configured files, or README/CLAUDE/AGENTS as fallback), a larger redacted sample of the written content with the current file around each edit and the rule text; the last few tool calls and output tails when the agent keeps failing; the agent's final message when it reports completion without running checks;
|
|
5
|
+
export declare const disclosure = "With TypeSafe judgments enabled, pi-warden sends to api.typesafe.ai: your latest request and up to eight redacted prior user/assistant text messages for task context, plus a redacted, truncated summary of each guarded bash, write, or edit call before it runs, with the agent's own words from the message that makes the call (its stated plan); for a write or edit in a project with a rules file (pi-warden.md, the configured files, or README/CLAUDE/AGENTS as fallback), a larger redacted sample of the written content with the current file around each edit and the rule text; the last few tool calls and output tails when the agent keeps failing; the agent's final message when it reports completion without running checks; redacted tool-output samples for security and context saving (retention and output format); and, on the first guarded call after your reply, the redacted summaries of the calls allowed in the previous turn, so Jev can say whether your reply regrets one of them. Compression and duplicate notes store an exact, owner-only copy in a temporary file on this machine; the hold feedback log stores tool names, pattern ids, scores, and outcomes (never commands) in an owner-only file under Pi's agent directory. Requests may incur charges. Secret redaction is best-effort. Results are model judgments, not proof or authorization; offline pattern checks stay active either way.";
|
|
6
6
|
/**
|
|
7
7
|
* Tool calls of the assistant message being preflighted. Pi runs `tool_call` hooks for sibling calls one after another,
|
|
8
8
|
* so judging them one request at a time costs one round trip per call; judging them together costs one round trip.
|
|
9
9
|
*/
|
|
10
10
|
export declare function siblingToolCalls(ctx: ExtensionContext): ToolCallRef[];
|
|
11
|
+
/**
|
|
12
|
+
* The agent's own words before the call: the text of the assistant message that carries it, or, when that message is
|
|
13
|
+
* tool calls only, the latest assistant text since the user's prompt. Sent as `plan`; it explains the step and cannot approve it.
|
|
14
|
+
*/
|
|
15
|
+
export declare function assistantPlan(ctx: ExtensionContext): string | undefined;
|
|
11
16
|
export declare function confirmMessage(verdict: Verdict): string;
|
|
12
17
|
/** Names each symptom and its fix; repeats in the session turn the note into a standing rule. */
|
|
13
18
|
export declare function slopSteer(where: string, symptoms: readonly SlopSymptom[], counts: Record<SlopSymptom, number>): string;
|
package/dist/extension.js
CHANGED
|
@@ -5,7 +5,8 @@ import { ActionGuard } from "./action-guard.js";
|
|
|
5
5
|
import * as configModule from "./config.js";
|
|
6
6
|
import { applyUserOverrides, defaultConfig, isMode, loadConfig, PACKAGE_NAME, projectConfigPath, readUserConfig, setUserSetting, userConfigPath, writeUserConfig } from "./config.js";
|
|
7
7
|
import { classifyToolResult, doneNudge, emptyEvidence, evaluateDone, finalAssistantText, formatDone, needsDoneCheck, recordOutcome } from "./done.js";
|
|
8
|
-
import { evaluateAction, formatVerdict, SLOP_LABELS, steerReason } from "./guard.js";
|
|
8
|
+
import { evaluateAction, formatVerdict, intentSteer, SLOP_LABELS, steerReason } from "./guard.js";
|
|
9
|
+
import { formatHolds, HoldLedger, HoldLog, holdLogPath, outcomeNote, regretsAt, textRegrets } from "./holds.js";
|
|
9
10
|
import { evaluateProse, proseNudge, ProseTrend } from "./prose.js";
|
|
10
11
|
import { compressOutput, duplicateNote, evaluateOutput, outputKey, saveOutput, securityNotice } from "./output.js";
|
|
11
12
|
import { classifyRecall, detectSearchTool, recallInstruction } from "./recall.js";
|
|
@@ -19,10 +20,10 @@ import { completeConfig, shapeWarning } from "./shape.js";
|
|
|
19
20
|
import { ContextLedger, formatLedger } from "./saver.js";
|
|
20
21
|
import { actionDetails, doneDetails, proseDetails, rulesDetails, runawayDetails, stuckDetails, Trace } from "./trace.js";
|
|
21
22
|
import { DEFAULT_TEMPLATES, proseTokens, renderTemplate, TOKEN_NAMES } from "./widget.js";
|
|
22
|
-
export const disclosure = "With TypeSafe judgments enabled, pi-warden sends to api.typesafe.ai: your latest request and up to eight redacted prior user/assistant text messages for task context, plus a redacted, truncated summary of each guarded bash, write, or edit call before it runs; for a write or edit in a project with a rules file (pi-warden.md, the configured files, or README/CLAUDE/AGENTS as fallback), a larger redacted sample of the written content with the current file around each edit and the rule text; the last few tool calls and output tails when the agent keeps failing; the agent's final message when it reports completion without running checks;
|
|
23
|
+
export const disclosure = "With TypeSafe judgments enabled, pi-warden sends to api.typesafe.ai: your latest request and up to eight redacted prior user/assistant text messages for task context, plus a redacted, truncated summary of each guarded bash, write, or edit call before it runs, with the agent's own words from the message that makes the call (its stated plan); for a write or edit in a project with a rules file (pi-warden.md, the configured files, or README/CLAUDE/AGENTS as fallback), a larger redacted sample of the written content with the current file around each edit and the rule text; the last few tool calls and output tails when the agent keeps failing; the agent's final message when it reports completion without running checks; redacted tool-output samples for security and context saving (retention and output format); and, on the first guarded call after your reply, the redacted summaries of the calls allowed in the previous turn, so Jev can say whether your reply regrets one of them. Compression and duplicate notes store an exact, owner-only copy in a temporary file on this machine; the hold feedback log stores tool names, pattern ids, scores, and outcomes (never commands) in an owner-only file under Pi's agent directory. Requests may incur charges. Secret redaction is best-effort. Results are model judgments, not proof or authorization; offline pattern checks stay active either way.";
|
|
23
24
|
const WIDGET = PACKAGE_NAME;
|
|
24
25
|
const CONFIRM_TEXT_LIMIT = 500;
|
|
25
|
-
const freshStats = () => ({ inspected: 0, judged: 0, warned: 0, held: 0, approved: 0, slop: 0, ruleChecks: 0, ruleViolations: 0, pathNotes: 0, stuckChecks: 0, stuck: 0, doneChecks: 0, unverified: 0, proseChecks: 0, proseNudges: 0, runaway: 0, errors: 0 });
|
|
26
|
+
const freshStats = () => ({ inspected: 0, judged: 0, warned: 0, held: 0, approved: 0, offPlan: 0, slop: 0, ruleChecks: 0, ruleViolations: 0, pathNotes: 0, stuckChecks: 0, stuck: 0, doneChecks: 0, unverified: 0, proseChecks: 0, proseNudges: 0, runaway: 0, errors: 0 });
|
|
26
27
|
function latestUserPrompt(ctx) {
|
|
27
28
|
const entries = ctx.sessionManager.getBranch();
|
|
28
29
|
for (let index = entries.length - 1; index >= 0; index--) {
|
|
@@ -79,6 +80,27 @@ export function siblingToolCalls(ctx) {
|
|
|
79
80
|
}
|
|
80
81
|
return [];
|
|
81
82
|
}
|
|
83
|
+
/**
|
|
84
|
+
* The agent's own words before the call: the text of the assistant message that carries it, or, when that message is
|
|
85
|
+
* tool calls only, the latest assistant text since the user's prompt. Sent as `plan`; it explains the step and cannot approve it.
|
|
86
|
+
*/
|
|
87
|
+
export function assistantPlan(ctx) {
|
|
88
|
+
const entries = ctx.sessionManager.getBranch();
|
|
89
|
+
for (let index = entries.length - 1; index >= 0; index--) {
|
|
90
|
+
const entry = entries[index];
|
|
91
|
+
if (entry?.type !== "message")
|
|
92
|
+
continue;
|
|
93
|
+
if (entry.message.role === "user")
|
|
94
|
+
return undefined;
|
|
95
|
+
if (entry.message.role !== "assistant")
|
|
96
|
+
continue;
|
|
97
|
+
const content = entry.message.content;
|
|
98
|
+
const text = typeof content === "string" ? content : content.filter((part) => part.type === "text").map(part => part.text).join("\n");
|
|
99
|
+
if (text.trim())
|
|
100
|
+
return text.trim();
|
|
101
|
+
}
|
|
102
|
+
return undefined;
|
|
103
|
+
}
|
|
82
104
|
function activeMode(config, hasUI) {
|
|
83
105
|
const env = process.env.PI_WARDEN_MODE?.trim();
|
|
84
106
|
const mode = isMode(env) ? env : config.mode;
|
|
@@ -148,6 +170,12 @@ export default function wardenExtension(pi) {
|
|
|
148
170
|
let lastUi;
|
|
149
171
|
const actionGuard = new ActionGuard();
|
|
150
172
|
const rulesGuard = new RulesGuard();
|
|
173
|
+
// Hold feedback: what the user did after each judged call, the trace entry each label lands on, and the per-session log.
|
|
174
|
+
const holds = new HoldLedger();
|
|
175
|
+
const traceOf = new WeakMap();
|
|
176
|
+
let holdLog;
|
|
177
|
+
// Allowed calls of the turn the user just replied to; the regret question about them rides the next action request.
|
|
178
|
+
let regretCandidates = [];
|
|
151
179
|
let attempts = new AttemptWindow(defaultConfig().stuck.window);
|
|
152
180
|
let evidence = emptyEvidence();
|
|
153
181
|
let doneNudged = false;
|
|
@@ -224,8 +252,25 @@ export default function wardenExtension(pi) {
|
|
|
224
252
|
};
|
|
225
253
|
const record = (ctx, config, guard, line, details) => {
|
|
226
254
|
widget.set(guard, line);
|
|
227
|
-
|
|
255
|
+
const entry = { at: Date.now(), guard, line, details };
|
|
256
|
+
trace.push(entry);
|
|
228
257
|
paint(ctx, config);
|
|
258
|
+
return entry;
|
|
259
|
+
};
|
|
260
|
+
/** Labels landed on earlier calls: their trace entries say so and the session log is rewritten. */
|
|
261
|
+
const noteOutcomes = (config, records) => {
|
|
262
|
+
for (const item of records) {
|
|
263
|
+
const entry = traceOf.get(item);
|
|
264
|
+
if (entry)
|
|
265
|
+
trace.amend(entry, outcomeNote(item));
|
|
266
|
+
}
|
|
267
|
+
if (config.action.feedbackLog && holds.records().length)
|
|
268
|
+
void holdLog?.save(holds.records());
|
|
269
|
+
};
|
|
270
|
+
/** The user's reply was read for regret, by Jev or by the offline heuristic; the candidates are labelled once. */
|
|
271
|
+
const settleRegret = (config, result) => {
|
|
272
|
+
regretCandidates = [];
|
|
273
|
+
noteOutcomes(config, holds.regret(result));
|
|
229
274
|
};
|
|
230
275
|
const steer = (config, content, options = { deliverAs: "steer" }) => pi.sendMessage({ customType: `${PACKAGE_NAME}-steer`, content, display: config.steerVisible }, options);
|
|
231
276
|
/**
|
|
@@ -255,6 +300,9 @@ export default function wardenExtension(pi) {
|
|
|
255
300
|
panel?.close();
|
|
256
301
|
actionGuard.reset();
|
|
257
302
|
rulesGuard.reset();
|
|
303
|
+
holds.reset();
|
|
304
|
+
regretCandidates = [];
|
|
305
|
+
holdLog = new HoldLog(holdLogPath(typeof ctx.sessionManager.getSessionId === "function" ? ctx.sessionManager.getSessionId() : String(process.pid)));
|
|
258
306
|
attempts.reset();
|
|
259
307
|
evidence = emptyEvidence();
|
|
260
308
|
doneNudged = false;
|
|
@@ -272,14 +320,20 @@ export default function wardenExtension(pi) {
|
|
|
272
320
|
ctx.ui.setWidget(WIDGET, undefined);
|
|
273
321
|
});
|
|
274
322
|
// A new user prompt starts a new attempt history and a new done-check budget.
|
|
275
|
-
pi.on("before_agent_start", async (
|
|
276
|
-
|
|
323
|
+
pi.on("before_agent_start", async (event, ctx) => {
|
|
324
|
+
const config = configFor(ctx);
|
|
325
|
+
attempts = new AttemptWindow(config.stuck.window);
|
|
277
326
|
doneNudged = false;
|
|
278
327
|
runaway.reset();
|
|
279
328
|
runawayStops = 0;
|
|
280
329
|
pendingRunaway = undefined;
|
|
281
330
|
actionGuard.turnEnd();
|
|
282
331
|
rulesGuard.turnEnd();
|
|
332
|
+
// Holds the user never approved are re-plans now; last turn's allowed calls wait for the regret question.
|
|
333
|
+
noteOutcomes(config, holds.promptArrived());
|
|
334
|
+
regretCandidates = holds.candidates();
|
|
335
|
+
if (regretCandidates.length && !judgeFor(config))
|
|
336
|
+
settleRegret(config, { regretted: textRegrets(event.prompt), via: "text" });
|
|
283
337
|
});
|
|
284
338
|
// Each assistant message is judged on its own; Pi does not forward the stream's own "start" event, so this is the reset.
|
|
285
339
|
pi.on("message_start", async (event) => {
|
|
@@ -345,9 +399,12 @@ export default function wardenExtension(pi) {
|
|
|
345
399
|
? rulesGuard.inspect(call, siblings, { cwd: ctx.cwd, config: config.rules, judge, timeoutMs: config.timeoutMs, signal: ctx.signal })
|
|
346
400
|
: undefined;
|
|
347
401
|
rulesCheck?.catch(() => undefined);
|
|
348
|
-
const verdict = await actionGuard.inspect(call, { task, context: recentTaskContext(ctx), siblings }, { config: config.action, cwd: ctx.cwd, judge, signal: ctx.signal, slop: config.slop, security: config.security });
|
|
402
|
+
const verdict = await actionGuard.inspect(call, { task, context: recentTaskContext(ctx), siblings, plan: assistantPlan(ctx) }, { config: config.action, cwd: ctx.cwd, judge, signal: ctx.signal, slop: config.slop, security: config.security, previousActions: regretCandidates.length ? regretCandidates : undefined });
|
|
349
403
|
if (verdict.source === "skipped")
|
|
350
404
|
return;
|
|
405
|
+
if (regretCandidates.length && verdict.judgment?.regretted !== undefined) {
|
|
406
|
+
settleRegret(config, { regretted: regretsAt(verdict.judgment.regretted), target: verdict.judgment.regretTarget, probability: verdict.judgment.regretted, via: "jev" });
|
|
407
|
+
}
|
|
351
408
|
// Notes for the agent about the content it just wrote: slop, rule violations, and sensitive paths arrive as one message.
|
|
352
409
|
const notes = [];
|
|
353
410
|
if (verdict.judgment?.securityRisk !== undefined && verdict.judgment.securityRisk >= config.security.threshold) {
|
|
@@ -357,12 +414,28 @@ export default function wardenExtension(pi) {
|
|
|
357
414
|
stats.judged++;
|
|
358
415
|
if (verdict.source === "error")
|
|
359
416
|
noteError(ctx, verdict.error ?? "TypeSafe request failed.", verdict.errorCode);
|
|
360
|
-
if (verdict.approvedByUser)
|
|
361
|
-
stats.approved++;
|
|
362
417
|
const mode = activeMode(config, ctx.hasUI);
|
|
418
|
+
if (verdict.approvedByUser) {
|
|
419
|
+
stats.approved++;
|
|
420
|
+
const released = holds.approved(event.toolName);
|
|
421
|
+
if (released)
|
|
422
|
+
noteOutcomes(config, [released]);
|
|
423
|
+
}
|
|
363
424
|
const told = verdict.level === "confirm" && mode === "steer" ? steerReason(verdict, { canApprove: judge !== undefined }) : undefined;
|
|
364
|
-
|
|
365
|
-
|
|
425
|
+
const entry = verdict.source !== "read-only" ? record(ctx, config, "action", formatVerdict(verdict, config.widget.action), actionDetails(verdict, { mode, ...(told ? { told } : {}) })) : undefined;
|
|
426
|
+
// What happens to this call is the label for its scores; a decision made in the dialog lands at once, a steer-mode hold waits for the user.
|
|
427
|
+
const track = (held, outcome, via) => {
|
|
428
|
+
if (!entry)
|
|
429
|
+
return;
|
|
430
|
+
const item = holds.record(verdict, { held, mode, outcome, via });
|
|
431
|
+
traceOf.set(item, entry);
|
|
432
|
+
noteOutcomes(config, outcome ? [item] : []);
|
|
433
|
+
};
|
|
434
|
+
// The warn notice below names the mismatch to the user; the agent gets the steer with the other notes.
|
|
435
|
+
if (verdict.intentMismatch) {
|
|
436
|
+
stats.offPlan++;
|
|
437
|
+
notes.push(intentSteer(verdict));
|
|
438
|
+
}
|
|
366
439
|
if (verdict.slopSymptoms?.length && verdict.slopReasons) {
|
|
367
440
|
stats.slop++;
|
|
368
441
|
for (const symptom of verdict.slopSymptoms)
|
|
@@ -403,27 +476,35 @@ export default function wardenExtension(pi) {
|
|
|
403
476
|
stats.warned++;
|
|
404
477
|
if (ctx.hasUI)
|
|
405
478
|
ctx.ui.notify(`warden · ${event.toolName}: ${verdict.reasons.join("; ")}`, "warning");
|
|
479
|
+
track(false);
|
|
406
480
|
return undefined;
|
|
407
481
|
}
|
|
408
|
-
if (verdict.level !== "confirm")
|
|
482
|
+
if (verdict.level !== "confirm") {
|
|
483
|
+
track(false);
|
|
409
484
|
return undefined;
|
|
485
|
+
}
|
|
410
486
|
const reasons = verdict.reasons.join("; ");
|
|
411
487
|
if (mode === "advise") {
|
|
412
488
|
stats.warned++;
|
|
413
489
|
if (ctx.hasUI)
|
|
414
490
|
ctx.ui.notify(`warden · ${event.toolName} (advise mode, not held): ${reasons}`, "warning");
|
|
491
|
+
track(false);
|
|
415
492
|
return undefined;
|
|
416
493
|
}
|
|
417
494
|
if (mode === "confirm") {
|
|
418
495
|
notifyDesktop(ctx, config, `Waiting for you: allow this ${event.toolName} call? ${reasons}`);
|
|
419
496
|
const allowed = await ctx.ui.confirm(`warden: allow this ${event.toolName} call?`, confirmMessage(verdict), ctx.signal ? { signal: ctx.signal } : {});
|
|
420
|
-
if (allowed)
|
|
497
|
+
if (allowed) {
|
|
498
|
+
track(true, "approved", "dialog");
|
|
421
499
|
return undefined;
|
|
500
|
+
}
|
|
422
501
|
stats.held++;
|
|
502
|
+
track(true, "declined", "dialog");
|
|
423
503
|
return { block: true, reason: `pi-warden: the user declined this ${event.toolName} call (${reasons}). Do not retry it unchanged; ask the user how to proceed.` };
|
|
424
504
|
}
|
|
425
505
|
stats.held++;
|
|
426
506
|
actionGuard.hold(task);
|
|
507
|
+
track(true);
|
|
427
508
|
if (ctx.hasUI)
|
|
428
509
|
ctx.ui.notify(`warden · held ${event.toolName}: ${reasons}. The agent was told why and asked to re-plan or ask you.`, "warning");
|
|
429
510
|
notifyDesktop(ctx, config, `Held ${event.toolName}: ${reasons}. The agent will re-plan or ask you in chat.`);
|
|
@@ -550,6 +631,9 @@ export default function wardenExtension(pi) {
|
|
|
550
631
|
const config = configFor(ctx);
|
|
551
632
|
if (!config.enabled)
|
|
552
633
|
return;
|
|
634
|
+
// No guarded call carried the regret question this run (the agent only replied): the offline heuristic reads the prompt.
|
|
635
|
+
if (regretCandidates.length)
|
|
636
|
+
settleRegret(config, { regretted: textRegrets(latestUserPrompt(ctx)), via: "text" });
|
|
553
637
|
if (pendingRunaway) {
|
|
554
638
|
const { nudge, recover } = pendingRunaway;
|
|
555
639
|
pendingRunaway = undefined;
|
|
@@ -644,9 +728,10 @@ export default function wardenExtension(pi) {
|
|
|
644
728
|
const guards = [config.action.enabled && "action", config.stuck.enabled && "stuck", config.done.enabled && "done-check", config.slop.enabled && "slop", config.slop.enabled && config.slop.prose.enabled && `prose (${config.slop.prose.audience})`, config.security.enabled && "security", config.rules.enabled && "rules", config.context.enabled && "context", config.runaway.enabled && "runaway", config.notify.enabled && "desktop notifications"].filter(Boolean).join(", ");
|
|
645
729
|
report([
|
|
646
730
|
`pi-warden: ${config.enabled ? `guarding ${config.action.tools.join(", ")} (${guards})` : "off"}; mode ${activeMode(config, ctx.hasUI)}; TypeSafe judgments ${source ? `enabled via ${source}` : "disabled (run /warden enable)"}; key ${key ? key.source === "stored" ? "stored (shared with pi-typesafe)" : "from TYPESAFE_API_KEY" : "missing (run /warden enable)"}.`,
|
|
647
|
-
`Session: ${stats.inspected} inspected, ${stats.judged} judged, ${stats.warned} warned, ${stats.held} held, ${stats.approved} approved on retry, ${stats.slop} slop notes, ${stats.ruleViolations}/${stats.ruleChecks} rule violations, ${stats.pathNotes} sensitive-path notes, ${stats.stuck}/${stats.stuckChecks} stuck, ${stats.unverified}/${stats.doneChecks} unverified done, ${stats.proseNudges}/${stats.proseChecks} prose nudges, ${stats.runaway} runaway stops, ${stats.errors} TypeSafe errors; ${usage?.requestsStarted ?? 0}/${config.maxRequests} requests. Steers are ${config.steerVisible ? "shown in the transcript" : "hidden from the transcript (trace panel shows them)"}.`,
|
|
648
|
-
`Thresholds: irreversible warn ${config.action.irreversible.warn} / hold ${config.action.irreversible.confirm}; off-task warn ${config.action.offTask.warn} / hold ${config.action.offTask.confirm}; stuck same-strategy ${config.stuck.sameStrategy} after ${config.stuck.minFailures} failures; done claims ${config.done.claimsDone}; slop ${config.slop.threshold}, rules ${config.rules.threshold}, prose ${config.slop.prose.threshold} in ${config.slop.prose.trend}/3 replies; runaway ${config.runaway.repeats} repeats (thinking ${config.runaway.thinkingRepeats}), recover ${config.runaway.recover}; failOpen ${config.action.failOpen}.`,
|
|
731
|
+
`Session: ${stats.inspected} inspected, ${stats.judged} judged, ${stats.warned} warned, ${stats.held} held, ${stats.approved} approved on retry, ${stats.offPlan} off plan, ${stats.slop} slop notes, ${stats.ruleViolations}/${stats.ruleChecks} rule violations, ${stats.pathNotes} sensitive-path notes, ${stats.stuck}/${stats.stuckChecks} stuck, ${stats.unverified}/${stats.doneChecks} unverified done, ${stats.proseNudges}/${stats.proseChecks} prose nudges, ${stats.runaway} runaway stops, ${stats.errors} TypeSafe errors; ${usage?.requestsStarted ?? 0}/${config.maxRequests} requests. Steers are ${config.steerVisible ? "shown in the transcript" : "hidden from the transcript (trace panel shows them)"}.`,
|
|
732
|
+
`Thresholds: irreversible warn ${config.action.irreversible.warn} / hold ${config.action.irreversible.confirm}; off-task warn ${config.action.offTask.warn} / hold ${config.action.offTask.confirm}; intent mismatch ${config.action.intentMismatch}; stuck same-strategy ${config.stuck.sameStrategy} after ${config.stuck.minFailures} failures; done claims ${config.done.claimsDone}; slop ${config.slop.threshold}, rules ${config.rules.threshold}, prose ${config.slop.prose.threshold} in ${config.slop.prose.trend}/3 replies; runaway ${config.runaway.repeats} repeats (thinking ${config.runaway.thinkingRepeats}), recover ${config.runaway.recover}; failOpen ${config.action.failOpen}.`,
|
|
649
733
|
formatLedger(ledger.snapshot()),
|
|
734
|
+
`${formatHolds(holds.snapshot(), config.action.feedbackLog ? holdLog?.path : undefined)}${holdLog?.lastFailure ? ` Log write failed: ${holdLog.lastFailure}.` : ""}`,
|
|
650
735
|
`Rules: ${config.rules.enabled ? `${rulesGuard.describe(ctx.cwd, config.rules)}${Object.keys(config.rules.sensitivePaths).length ? `; ${Object.keys(config.rules.sensitivePaths).length} sensitive path${Object.keys(config.rules.sensitivePaths).length === 1 ? "" : "s"}` : ""}` : "off"}.`,
|
|
651
736
|
`Desktop notifications: ${config.notify.enabled ? `on (${config.notify.command.length ? `command ${config.notify.command[0]}` : (await (notifier ??= detectNotifier())) ?? "no notifier found on this machine"}; cooldown ${config.notify.cooldownMs} ms)` : "off"}.`,
|
|
652
737
|
`Config: ${userConfigPath()}${ctx.isProjectTrusted() ? ` and ${projectConfigPath(ctx.cwd)}` : ""}.`,
|
package/dist/guard.d.ts
CHANGED
|
@@ -17,6 +17,8 @@ export interface ActionInput {
|
|
|
17
17
|
task?: string | undefined;
|
|
18
18
|
/** Prior conversation clarifies scope, but never grants approval for a held action. */
|
|
19
19
|
context?: readonly TaskMessage[] | undefined;
|
|
20
|
+
/** The agent's own words in the message that makes this call (or its latest text under this prompt). Explains the step; never authorizes it. */
|
|
21
|
+
plan?: string | undefined;
|
|
20
22
|
}
|
|
21
23
|
export interface TaskMessage {
|
|
22
24
|
role: "user" | "assistant";
|
|
@@ -48,9 +50,15 @@ export interface Judgment {
|
|
|
48
50
|
scopeConfidence: number;
|
|
49
51
|
/** P(the latest user message approves this exact action); only asked when a previously held call is retried. */
|
|
50
52
|
approved?: number;
|
|
53
|
+
/** P(the latest user message regrets an allowed call of the previous turn); asked once per prompt, on its first action request. */
|
|
54
|
+
regretted?: number;
|
|
55
|
+
/** The id of the regretted previous action when several were offered. */
|
|
56
|
+
regretTarget?: string;
|
|
51
57
|
securityRisk?: number;
|
|
52
58
|
/** P(the action changes files, state, or external systems). Off-task alone holds only actions that can change something. */
|
|
53
59
|
mutates?: number;
|
|
60
|
+
/** P(the action does something materially different from `plan`); only asked when the agent said something before the call. */
|
|
61
|
+
intentMismatch?: number;
|
|
54
62
|
model: string;
|
|
55
63
|
elapsedMs: number;
|
|
56
64
|
}
|
|
@@ -63,6 +71,13 @@ export interface SlopJudgment {
|
|
|
63
71
|
}
|
|
64
72
|
export type SlopSymptom = keyof SlopJudgment;
|
|
65
73
|
export declare const SLOP_SYMPTOMS: readonly SlopSymptom[];
|
|
74
|
+
/** A call the guard allowed in the previous turn, as the regret question sees it: redacted summary fields only. */
|
|
75
|
+
export interface PreviousAction {
|
|
76
|
+
id: string;
|
|
77
|
+
tool: string;
|
|
78
|
+
command?: string;
|
|
79
|
+
path?: string;
|
|
80
|
+
}
|
|
66
81
|
export interface Verdict {
|
|
67
82
|
level: Level;
|
|
68
83
|
source: "skipped" | "read-only" | "pattern" | "typesafe" | "error";
|
|
@@ -77,6 +92,10 @@ export interface Verdict {
|
|
|
77
92
|
slopReasons?: string[];
|
|
78
93
|
/** True when a previously held call was allowed because the user's latest message approves it. */
|
|
79
94
|
approvedByUser?: boolean;
|
|
95
|
+
/** Redacted, truncated `plan` as sent to Jev and shown in the trace. */
|
|
96
|
+
plan?: string;
|
|
97
|
+
/** True when Jev finds the call at odds with the agent's stated plan and the call can change something; the agent is told. */
|
|
98
|
+
intentMismatch?: boolean;
|
|
80
99
|
/** Safe TypeSafe error message when the judge could not answer. */
|
|
81
100
|
error?: string;
|
|
82
101
|
errorCode?: IntegrationErrorCode;
|
|
@@ -92,6 +111,8 @@ export interface EvaluateOptions {
|
|
|
92
111
|
security?: SecurityConfig | undefined;
|
|
93
112
|
/** This exact call was held earlier and the user has replied since: ask whether the reply approves it. */
|
|
94
113
|
retryAfterHold?: boolean | undefined;
|
|
114
|
+
/** Calls allowed in the previous turn: ask whether the user's latest message regrets one of them (rides this request). */
|
|
115
|
+
previousActions?: readonly PreviousAction[] | undefined;
|
|
95
116
|
}
|
|
96
117
|
export interface ScannedCommand {
|
|
97
118
|
/** The command with data text blanked; what the pattern rules read. */
|
|
@@ -118,6 +139,10 @@ export declare const questions: {
|
|
|
118
139
|
readonly unclear: "The supplied conversation or action gives too little information to establish scope; this is not itself a violation";
|
|
119
140
|
}>;
|
|
120
141
|
};
|
|
142
|
+
/** Asked only when the agent said something before the call; an empty plan cannot be contradicted. */
|
|
143
|
+
export declare const intentQuestion: {
|
|
144
|
+
intent_mismatch: import("pi-typesafe").NoulQuestion;
|
|
145
|
+
};
|
|
121
146
|
export declare const slopQuestions: {
|
|
122
147
|
slop_stub: import("pi-typesafe").NoulQuestion;
|
|
123
148
|
slop_comments: import("pi-typesafe").NoulQuestion;
|
|
@@ -131,13 +156,37 @@ export declare const securityQuestion: {
|
|
|
131
156
|
export declare const approvalQuestion: {
|
|
132
157
|
approved: import("pi-typesafe").NoulQuestion;
|
|
133
158
|
};
|
|
159
|
+
/**
|
|
160
|
+
* One yes/no on whether the user's reply regrets what the agent did last turn; with several candidates a Choice names the
|
|
161
|
+
* one. Labels the allowed calls for hold calibration and never changes the verdict on the current call.
|
|
162
|
+
*/
|
|
163
|
+
export declare function regretQuestions(actions: readonly PreviousAction[]): {
|
|
164
|
+
regretted: import("pi-typesafe").NoulQuestion;
|
|
165
|
+
regret_target?: never;
|
|
166
|
+
} | {
|
|
167
|
+
regretted: import("pi-typesafe").NoulQuestion;
|
|
168
|
+
regret_target: import("pi-typesafe").ChoiceQuestion<{
|
|
169
|
+
[k: string]: string;
|
|
170
|
+
}>;
|
|
171
|
+
};
|
|
172
|
+
/** The agent's words as they leave the machine: redacted and bounded. Undefined when the agent said nothing. */
|
|
173
|
+
export declare function describePlan(plan: string | undefined): string | undefined;
|
|
134
174
|
export declare function buildRequest(summary: ActionSummary, task: string | undefined, extras?: {
|
|
135
175
|
slop?: boolean;
|
|
136
176
|
approval?: boolean;
|
|
137
177
|
security?: boolean;
|
|
138
178
|
context?: readonly TaskMessage[] | undefined;
|
|
179
|
+
previousActions?: readonly PreviousAction[] | undefined;
|
|
180
|
+
plan?: string | undefined;
|
|
139
181
|
}): {
|
|
140
182
|
state: {
|
|
183
|
+
previous_actions?: {
|
|
184
|
+
command?: string;
|
|
185
|
+
id: string;
|
|
186
|
+
tool: string;
|
|
187
|
+
path?: string;
|
|
188
|
+
}[];
|
|
189
|
+
plan?: string;
|
|
141
190
|
task: string;
|
|
142
191
|
action: Record<string, string | number | boolean>;
|
|
143
192
|
context: {
|
|
@@ -146,12 +195,53 @@ export declare function buildRequest(summary: ActionSummary, task: string | unde
|
|
|
146
195
|
}[];
|
|
147
196
|
};
|
|
148
197
|
questions: {
|
|
198
|
+
regretted: import("pi-typesafe").NoulQuestion;
|
|
199
|
+
regret_target?: never;
|
|
200
|
+
security_risk?: import("pi-typesafe").NoulQuestion;
|
|
201
|
+
approved?: import("pi-typesafe").NoulQuestion;
|
|
202
|
+
slop_stub?: import("pi-typesafe").NoulQuestion;
|
|
203
|
+
slop_comments?: import("pi-typesafe").NoulQuestion;
|
|
204
|
+
slop_dead?: import("pi-typesafe").NoulQuestion;
|
|
205
|
+
slop_hedging?: import("pi-typesafe").NoulQuestion;
|
|
206
|
+
intent_mismatch?: import("pi-typesafe").NoulQuestion;
|
|
207
|
+
irreversible: import("pi-typesafe").NoulQuestion;
|
|
208
|
+
off_task: import("pi-typesafe").NoulQuestion;
|
|
209
|
+
mutates: import("pi-typesafe").NoulQuestion;
|
|
210
|
+
scope: import("pi-typesafe").ChoiceQuestion<{
|
|
211
|
+
readonly expected_step: "Required implementation, bug fix, regression test, or verification for the active task";
|
|
212
|
+
readonly plausible_side_step: "Reasonable supporting work whose necessity is not yet established";
|
|
213
|
+
readonly unrelated: "No useful connection to the active task, or contrary to the user's current direction";
|
|
214
|
+
readonly unclear: "The supplied conversation or action gives too little information to establish scope; this is not itself a violation";
|
|
215
|
+
}>;
|
|
216
|
+
} | {
|
|
217
|
+
regretted: import("pi-typesafe").NoulQuestion;
|
|
218
|
+
regret_target: import("pi-typesafe").ChoiceQuestion<{
|
|
219
|
+
[k: string]: string;
|
|
220
|
+
}>;
|
|
221
|
+
security_risk?: import("pi-typesafe").NoulQuestion;
|
|
222
|
+
approved?: import("pi-typesafe").NoulQuestion;
|
|
223
|
+
slop_stub?: import("pi-typesafe").NoulQuestion;
|
|
224
|
+
slop_comments?: import("pi-typesafe").NoulQuestion;
|
|
225
|
+
slop_dead?: import("pi-typesafe").NoulQuestion;
|
|
226
|
+
slop_hedging?: import("pi-typesafe").NoulQuestion;
|
|
227
|
+
intent_mismatch?: import("pi-typesafe").NoulQuestion;
|
|
228
|
+
irreversible: import("pi-typesafe").NoulQuestion;
|
|
229
|
+
off_task: import("pi-typesafe").NoulQuestion;
|
|
230
|
+
mutates: import("pi-typesafe").NoulQuestion;
|
|
231
|
+
scope: import("pi-typesafe").ChoiceQuestion<{
|
|
232
|
+
readonly expected_step: "Required implementation, bug fix, regression test, or verification for the active task";
|
|
233
|
+
readonly plausible_side_step: "Reasonable supporting work whose necessity is not yet established";
|
|
234
|
+
readonly unrelated: "No useful connection to the active task, or contrary to the user's current direction";
|
|
235
|
+
readonly unclear: "The supplied conversation or action gives too little information to establish scope; this is not itself a violation";
|
|
236
|
+
}>;
|
|
237
|
+
} | {
|
|
149
238
|
security_risk?: import("pi-typesafe").NoulQuestion;
|
|
150
239
|
approved?: import("pi-typesafe").NoulQuestion;
|
|
151
240
|
slop_stub?: import("pi-typesafe").NoulQuestion;
|
|
152
241
|
slop_comments?: import("pi-typesafe").NoulQuestion;
|
|
153
242
|
slop_dead?: import("pi-typesafe").NoulQuestion;
|
|
154
243
|
slop_hedging?: import("pi-typesafe").NoulQuestion;
|
|
244
|
+
intent_mismatch?: import("pi-typesafe").NoulQuestion;
|
|
155
245
|
irreversible: import("pi-typesafe").NoulQuestion;
|
|
156
246
|
off_task: import("pi-typesafe").NoulQuestion;
|
|
157
247
|
mutates: import("pi-typesafe").NoulQuestion;
|
|
@@ -164,6 +254,8 @@ export declare function buildRequest(summary: ActionSummary, task: string | unde
|
|
|
164
254
|
};
|
|
165
255
|
};
|
|
166
256
|
export declare function evaluateAction(action: ActionInput, options: EvaluateOptions): Promise<Verdict>;
|
|
257
|
+
/** What the agent reads after a call that differs from its own plan ran: name the gap, ask it to keep words and calls in step. */
|
|
258
|
+
export declare function intentSteer(verdict: Verdict): string;
|
|
167
259
|
/** Offline stand-in for the approval question when TypeSafe is not available. */
|
|
168
260
|
export declare function textApproves(task: string | undefined): boolean;
|
|
169
261
|
/**
|
package/dist/guard.js
CHANGED
|
@@ -10,6 +10,7 @@ export const SLOP_SYMPTOMS = ["stub", "comments", "dead", "hedging"];
|
|
|
10
10
|
const LEVEL_RANK = { allow: 0, warn: 1, confirm: 2 };
|
|
11
11
|
const higher = (a, b) => (LEVEL_RANK[a] >= LEVEL_RANK[b] ? a : b);
|
|
12
12
|
const TASK_LIMIT = 1500;
|
|
13
|
+
const PLAN_LIMIT = 500;
|
|
13
14
|
const COMMAND_LIMIT = 2000;
|
|
14
15
|
const EXCERPT_LIMIT = 1500;
|
|
15
16
|
const EDIT_LIMIT = 400;
|
|
@@ -331,7 +332,7 @@ export const questions = {
|
|
|
331
332
|
true: "Yes: it force-pushes or rewrites shared history, deletes untracked or uncommitted work, drops or truncates database objects, overwrites files outside the project, publishes or deploys, or removes something no build or checkout can regenerate.",
|
|
332
333
|
false: "No: it only reads, builds, tests, or creates new files, or it deletes generated output, caches, or temporary files that a build or checkout can recreate, or it is a change `task` explicitly asked for.",
|
|
333
334
|
}),
|
|
334
|
-
off_task: noul("Is there evidence that `action` is outside the user's active task? `task` is the latest user message; `context` contains earlier conversation to resolve follow-ups, handoffs, and side comments. New user instructions override older ones. Assistant messages describe work but do not authorize it. Missing context alone is not evidence of off-task work.", {
|
|
335
|
+
off_task: noul("Is there evidence that `action` is outside the user's active task? `task` is the latest user message; `context` contains earlier conversation to resolve follow-ups, handoffs, and side comments; `plan`, when present, is the agent's own words in the message that makes this call and explains which step this is. New user instructions override older ones. Assistant messages, including `plan`, describe work but do not authorize it. Missing context alone is not evidence of off-task work.", {
|
|
335
336
|
true: "Yes: it contradicts the user's current direction, starts unrelated work, or expands the agreed scope without a useful connection to the active task.",
|
|
336
337
|
false: "No: implementation edits, regression tests, investigation, and verification support the active task, even if not individually named. A side comment does not cancel the task. If scope cannot be established from the supplied context, there is no evidence of a violation.",
|
|
337
338
|
}),
|
|
@@ -339,13 +340,20 @@ export const questions = {
|
|
|
339
340
|
true: "Yes: it writes or deletes files, changes version control or a database, installs or publishes, or calls a service that records the request.",
|
|
340
341
|
false: "No: it only inspects, reads, computes, or prints; running it again leaves everything as it was.",
|
|
341
342
|
}),
|
|
342
|
-
scope: choice("How does `action` relate to the active task described by `task` and the earlier `context`? Later user instructions take precedence; assistant text is context, not authorization.", {
|
|
343
|
+
scope: choice("How does `action` relate to the active task described by `task` and the earlier `context`? `plan`, when present, says which step the agent believes this is. Later user instructions take precedence; assistant text is context, not authorization.", {
|
|
343
344
|
expected_step: "Required implementation, bug fix, regression test, or verification for the active task",
|
|
344
345
|
plausible_side_step: "Reasonable supporting work whose necessity is not yet established",
|
|
345
346
|
unrelated: "No useful connection to the active task, or contrary to the user's current direction",
|
|
346
347
|
unclear: "The supplied conversation or action gives too little information to establish scope; this is not itself a violation",
|
|
347
348
|
}),
|
|
348
349
|
};
|
|
350
|
+
/** Asked only when the agent said something before the call; an empty plan cannot be contradicted. */
|
|
351
|
+
export const intentQuestion = {
|
|
352
|
+
intent_mismatch: noul("Does `action` do something materially different from what `plan` (the agent's own words right before this call) says it is about to do?", {
|
|
353
|
+
true: "Yes: a different target file, branch, or system than described; a broader, destructive, or irreversible operation where the plan describes a read, a check, a dry run, or a narrow change; a more forceful variant of the described step (a force push where a push was described, a delete where a move was, a hard reset where a soft one was); or a step the plan does not mention at all.",
|
|
354
|
+
false: "No: the call carries out the described step or a routine part of it (reading before editing, running the named check, creating the file it said it would) with no added force or scope, or `plan` is too general to contradict it.",
|
|
355
|
+
}),
|
|
356
|
+
};
|
|
349
357
|
export const slopQuestions = {
|
|
350
358
|
slop_stub: noul("Does the content `action` writes leave placeholder, stub, mock, or \"implement later\" code where `task` needs a working implementation?", {
|
|
351
359
|
true: "Yes: a function returns a constant, null, or fake data instead of doing its job; a TODO or \"implement later\" stands where the logic should be; a mock is hard-coded where a real call is needed.",
|
|
@@ -382,22 +390,49 @@ export const approvalQuestion = {
|
|
|
382
390
|
false: "No: the message declines, asks for something else, changes the approach, or does not address this action.",
|
|
383
391
|
}),
|
|
384
392
|
};
|
|
393
|
+
/**
|
|
394
|
+
* One yes/no on whether the user's reply regrets what the agent did last turn; with several candidates a Choice names the
|
|
395
|
+
* one. Labels the allowed calls for hold calibration and never changes the verdict on the current call.
|
|
396
|
+
*/
|
|
397
|
+
export function regretQuestions(actions) {
|
|
398
|
+
const regretted = noul("Does `task` (the user's latest message) tell the agent to stop, undo, revert, or not do one of the calls in `previous_actions`, which the agent ran in its previous turn? Judge only `task`; `context` explains what the agent was doing.", {
|
|
399
|
+
true: "Yes: the user says wait, stop, don't, undo, revert, or roll back, objects that a call should not have run, or asks why the agent did it.",
|
|
400
|
+
false: "No: the user continues, approves, asks for something new, reports a result, or the message does not address those calls.",
|
|
401
|
+
});
|
|
402
|
+
if (actions.length < 2)
|
|
403
|
+
return { regretted };
|
|
404
|
+
return {
|
|
405
|
+
regretted,
|
|
406
|
+
regret_target: choice("If `task` regrets one of `previous_actions`, which one does it most likely mean?", Object.fromEntries(actions.map(action => [action.id, `${action.tool}: ${action.command ?? action.path ?? "(no detail)"}`]))),
|
|
407
|
+
};
|
|
408
|
+
}
|
|
385
409
|
function hasContent(summary) {
|
|
386
410
|
return (summary.excerpt?.trim().length ?? 0) > 0 || (summary.edits?.some(edit => edit.newText.trim().length > 0) ?? false);
|
|
387
411
|
}
|
|
412
|
+
/** The agent's words as they leave the machine: redacted and bounded. Undefined when the agent said nothing. */
|
|
413
|
+
export function describePlan(plan) {
|
|
414
|
+
const text = plan?.trim();
|
|
415
|
+
return text ? truncate(redact(text), PLAN_LIMIT) : undefined;
|
|
416
|
+
}
|
|
388
417
|
export function buildRequest(summary, task, extras = {}) {
|
|
389
418
|
const wantSlop = extras.slop && (summary.tool === "write" || summary.tool === "edit") && hasContent(summary);
|
|
419
|
+
const previous = (extras.previousActions ?? []).slice(-PREVIOUS_ACTIONS_LIMIT).map(action => ({ ...action, ...(action.command !== undefined ? { command: truncate(action.command, PREVIOUS_COMMAND_LIMIT) } : {}) }));
|
|
420
|
+
const plan = describePlan(extras.plan);
|
|
390
421
|
return {
|
|
391
422
|
state: {
|
|
392
423
|
task: task?.trim() ? truncate(redact(task.trim()), TASK_LIMIT) : "(no user request recorded in this session)",
|
|
393
424
|
action: summary,
|
|
394
425
|
context: (extras.context ?? []).slice(-8).map(message => ({ role: message.role, text: truncate(redact(message.text), 750) })),
|
|
426
|
+
...(plan ? { plan } : {}),
|
|
427
|
+
...(previous.length ? { previous_actions: previous } : {}),
|
|
395
428
|
},
|
|
396
|
-
questions: { ...questions, ...(wantSlop ? slopQuestions : {}), ...(extras.approval ? approvalQuestion : {}), ...(extras.security && (summary.tool === "write" || summary.tool === "edit") && hasContent(summary) ? securityQuestion : {}) },
|
|
429
|
+
questions: { ...questions, ...(plan ? intentQuestion : {}), ...(wantSlop ? slopQuestions : {}), ...(extras.approval ? approvalQuestion : {}), ...(extras.security && (summary.tool === "write" || summary.tool === "edit") && hasContent(summary) ? securityQuestion : {}), ...(previous.length ? regretQuestions(previous) : {}) },
|
|
397
430
|
};
|
|
398
431
|
}
|
|
399
432
|
const percent = (value) => value.toFixed(2);
|
|
400
433
|
const APPROVAL_THRESHOLD = 0.7;
|
|
434
|
+
const PREVIOUS_ACTIONS_LIMIT = 6;
|
|
435
|
+
const PREVIOUS_COMMAND_LIMIT = 300;
|
|
401
436
|
// ---------------------------------------------------------------------------
|
|
402
437
|
export async function evaluateAction(action, options) {
|
|
403
438
|
const { config, judge } = options;
|
|
@@ -405,6 +440,8 @@ export async function evaluateAction(action, options) {
|
|
|
405
440
|
if (!config.enabled || !config.tools.includes(action.tool)) {
|
|
406
441
|
return { level: "allow", source: "skipped", summary, patterns: [], reasons: [] };
|
|
407
442
|
}
|
|
443
|
+
const plan = describePlan(action.plan);
|
|
444
|
+
const withPlan = (verdict) => (plan ? { ...verdict, plan } : verdict);
|
|
408
445
|
const patterns = matchPatterns(action.tool, action.input, action.cwd);
|
|
409
446
|
const reasons = [];
|
|
410
447
|
let level = "allow";
|
|
@@ -432,8 +469,8 @@ export async function evaluateAction(action, options) {
|
|
|
432
469
|
return { level, source: "read-only", summary, patterns, reasons };
|
|
433
470
|
}
|
|
434
471
|
if (!judge)
|
|
435
|
-
return { level, source: "pattern", summary, patterns, reasons };
|
|
436
|
-
const request = buildRequest(summary, action.task, { slop: options.slop?.enabled ?? false, approval: options.retryAfterHold ?? false, security: options.security?.enabled ?? false, context: action.context });
|
|
472
|
+
return withPlan({ level, source: "pattern", summary, patterns, reasons });
|
|
473
|
+
const request = buildRequest(summary, action.task, { slop: options.slop?.enabled ?? false, approval: options.retryAfterHold ?? false, security: options.security?.enabled ?? false, context: action.context, previousActions: options.previousActions, plan });
|
|
437
474
|
const result = await askJev(judge, request, { timeoutMs: config.timeoutMs, signal: options.signal });
|
|
438
475
|
if (!result.ok) {
|
|
439
476
|
if (!config.failOpen) {
|
|
@@ -443,7 +480,7 @@ export async function evaluateAction(action, options) {
|
|
|
443
480
|
else {
|
|
444
481
|
reasons.push("TypeSafe unavailable; allowed by failOpen");
|
|
445
482
|
}
|
|
446
|
-
return { level, source: "error", summary, patterns, reasons, error: result.error, ...(result.errorCode ? { errorCode: result.errorCode } : {}) };
|
|
483
|
+
return withPlan({ level, source: "error", summary, patterns, reasons, error: result.error, ...(result.errorCode ? { errorCode: result.errorCode } : {}) });
|
|
447
484
|
}
|
|
448
485
|
const answers = result.answers;
|
|
449
486
|
const judgment = {
|
|
@@ -458,6 +495,13 @@ export async function evaluateAction(action, options) {
|
|
|
458
495
|
judgment.approved = answers.approved.noul;
|
|
459
496
|
if (typeof answers.mutates?.noul === "number")
|
|
460
497
|
judgment.mutates = answers.mutates.noul;
|
|
498
|
+
if (plan && typeof answers.intent_mismatch?.noul === "number")
|
|
499
|
+
judgment.intentMismatch = answers.intent_mismatch.noul;
|
|
500
|
+
if (typeof answers.regretted?.noul === "number") {
|
|
501
|
+
judgment.regretted = answers.regretted.noul;
|
|
502
|
+
if (typeof answers.regret_target?.choice === "string")
|
|
503
|
+
judgment.regretTarget = answers.regret_target.choice;
|
|
504
|
+
}
|
|
461
505
|
if (deferSensitive) {
|
|
462
506
|
for (const hit of patterns) {
|
|
463
507
|
if (hit.severity !== "sensitive")
|
|
@@ -499,7 +543,15 @@ export async function evaluateAction(action, options) {
|
|
|
499
543
|
reasons.push(`possible security weakness ${percent(judgment.securityRisk)} in written content`);
|
|
500
544
|
}
|
|
501
545
|
}
|
|
502
|
-
|
|
546
|
+
// A call at odds with the agent's own plan is warned about and the agent is told; it is not held on that alone until the hold labels say how precise the signal is.
|
|
547
|
+
const mismatch = judgment.intentMismatch !== undefined && judgment.intentMismatch >= config.intentMismatch && canChange;
|
|
548
|
+
if (mismatch) {
|
|
549
|
+
level = higher(level, "warn");
|
|
550
|
+
reasons.push(`intent mismatch ${percent(judgment.intentMismatch)} (the call differs from the agent's stated plan)`);
|
|
551
|
+
}
|
|
552
|
+
const verdict = withPlan({ level, source: "typesafe", summary, patterns, reasons, judgment });
|
|
553
|
+
if (mismatch)
|
|
554
|
+
verdict.intentMismatch = true;
|
|
503
555
|
if (options.slop?.enabled && SLOP_SYMPTOMS.every(symptom => typeof answers[`slop_${symptom}`]?.noul === "number")) {
|
|
504
556
|
verdict.slop = { stub: answers.slop_stub.noul, comments: answers.slop_comments.noul, dead: answers.slop_dead.noul, hedging: answers.slop_hedging.noul };
|
|
505
557
|
const flagged = SLOP_SYMPTOMS.filter(symptom => verdict.slop[symptom] >= options.slop.threshold).sort((a, b) => verdict.slop[b] - verdict.slop[a]);
|
|
@@ -515,6 +567,11 @@ export async function evaluateAction(action, options) {
|
|
|
515
567
|
}
|
|
516
568
|
return verdict;
|
|
517
569
|
}
|
|
570
|
+
/** What the agent reads after a call that differs from its own plan ran: name the gap, ask it to keep words and calls in step. */
|
|
571
|
+
export function intentSteer(verdict) {
|
|
572
|
+
const score = verdict.judgment?.intentMismatch;
|
|
573
|
+
return `pi-warden: this ${verdict.summary.tool} call does something different from what you said you were about to do${score === undefined ? "" : ` (intent mismatch ${percent(score)})`}. It ran. Before the next call, say what changed and why, and keep your stated plan and your calls in step; if the described step is still needed, do it.`;
|
|
574
|
+
}
|
|
518
575
|
/** Offline stand-in for the approval question when TypeSafe is not available. */
|
|
519
576
|
export function textApproves(task) {
|
|
520
577
|
return /\b(?:yes|yep|yeah|go ahead|do it|proceed|approved?|confirm(?:ed)?|ok(?:ay)?|sure|please do|run it)\b/i.test(task ?? "") && !/\b(?:no|don't|do not|stop|wait|instead|not)\b/i.test(task ?? "");
|
package/dist/holds.d.ts
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
import type { WardenMode } from "./config.js";
|
|
2
|
+
import type { Level, PreviousAction, ScopeLabel, Verdict } from "./guard.js";
|
|
3
|
+
/**
|
|
4
|
+
* Hold feedback loop. Every hold is a prediction ("this call should not run as it stands") and every allowed call the
|
|
5
|
+
* opposite one; what the user does next labels them, so hold precision can be measured on real sessions instead of the
|
|
6
|
+
* synthetic cases the thresholds came from:
|
|
7
|
+
*
|
|
8
|
+
* - **approved**: the user's reply released the hold (steer mode) or the confirm dialog allowed it. False positive.
|
|
9
|
+
* - **declined**: the confirm dialog refused it. True positive.
|
|
10
|
+
* - **replanned**: the user replied, the following turn ended, and nobody approved the call. True positive.
|
|
11
|
+
* - **regretted**: the user's next message tells the agent to stop, undo, or not do an allowed call. False negative.
|
|
12
|
+
* - **accepted**: the user's next message was checked and does not regret the allowed calls of the last turn.
|
|
13
|
+
*
|
|
14
|
+
* Approval, decline, and re-plan are code only. Regret is one question that rides the first action request under the
|
|
15
|
+
* new prompt (no extra request); offline a stop-word heuristic stands in. Calls the guard skipped as read-only are not
|
|
16
|
+
* recorded: they could never have been held. Records are redacted for the log: tool, pattern ids, scores, level,
|
|
17
|
+
* outcome, never the command or path. The in-memory summary serves the regret question and stays in memory.
|
|
18
|
+
*/
|
|
19
|
+
export type CallOutcome = "pending" | "approved" | "declined" | "replanned" | "regretted" | "accepted";
|
|
20
|
+
export type OutcomeVia = "retry" | "dialog" | "next prompt" | "jev" | "text";
|
|
21
|
+
export interface CallScores {
|
|
22
|
+
irreversible: number;
|
|
23
|
+
offTask: number;
|
|
24
|
+
scope: ScopeLabel;
|
|
25
|
+
mutates?: number;
|
|
26
|
+
approved?: number;
|
|
27
|
+
intentMismatch?: number;
|
|
28
|
+
securityRisk?: number;
|
|
29
|
+
}
|
|
30
|
+
export interface CallRecord {
|
|
31
|
+
/** Sequence number within the session; the regret question names candidates by it. */
|
|
32
|
+
id: number;
|
|
33
|
+
at: number;
|
|
34
|
+
tool: string;
|
|
35
|
+
level: Level;
|
|
36
|
+
source: Verdict["source"];
|
|
37
|
+
mode: WardenMode;
|
|
38
|
+
held: boolean;
|
|
39
|
+
patterns: string[];
|
|
40
|
+
/** Reason labels as shown to the user; they name patterns and scores, never the command. */
|
|
41
|
+
reasons: string[];
|
|
42
|
+
/** Length of the agent's stated plan sent with the call; 0 means the agent said nothing before calling. Decides whether declared intent (layer 2) is worth building. */
|
|
43
|
+
planChars: number;
|
|
44
|
+
scores?: CallScores;
|
|
45
|
+
outcome: CallOutcome;
|
|
46
|
+
outcomeAt?: number;
|
|
47
|
+
outcomeVia?: OutcomeVia;
|
|
48
|
+
/** P(regret) from the question that labelled this call, when Jev answered it. */
|
|
49
|
+
regret?: number;
|
|
50
|
+
}
|
|
51
|
+
export interface HoldSnapshot {
|
|
52
|
+
holds: number;
|
|
53
|
+
approved: number;
|
|
54
|
+
declined: number;
|
|
55
|
+
replanned: number;
|
|
56
|
+
/** Holds without a label yet: the user has not replied, or the turn after the reply is still running. */
|
|
57
|
+
awaiting: number;
|
|
58
|
+
allowed: number;
|
|
59
|
+
regretted: number;
|
|
60
|
+
accepted: number;
|
|
61
|
+
/** Labelled holds: approved + declined + replanned. */
|
|
62
|
+
labels: number;
|
|
63
|
+
/** (declined + replanned) / labels; undefined without labels. */
|
|
64
|
+
precision: number | undefined;
|
|
65
|
+
}
|
|
66
|
+
export declare class HoldLedger {
|
|
67
|
+
private readonly tracked;
|
|
68
|
+
private nextId;
|
|
69
|
+
/** One inspected call. `outcome` is set at once when the confirm dialog decided; a steer-mode hold starts pending. */
|
|
70
|
+
record(verdict: Verdict, options: {
|
|
71
|
+
held: boolean;
|
|
72
|
+
mode: WardenMode;
|
|
73
|
+
outcome?: CallOutcome | undefined;
|
|
74
|
+
via?: OutcomeVia | undefined;
|
|
75
|
+
at?: number;
|
|
76
|
+
}): CallRecord;
|
|
77
|
+
/**
|
|
78
|
+
* The user's reply released a hold: the pending hold of the same tool is the false positive, the latest one when the
|
|
79
|
+
* tools differ (the retry rarely repeats the held call byte for byte, and a hold can be released through another tool).
|
|
80
|
+
*/
|
|
81
|
+
approved(tool: string, via?: OutcomeVia, at?: number): CallRecord | undefined;
|
|
82
|
+
/**
|
|
83
|
+
* A new user prompt. Every pending record has seen one more prompt. A hold still pending after the prompt that could
|
|
84
|
+
* have approved it (the reply) and the turn that followed is a re-plan: the agent found another way or the user said no.
|
|
85
|
+
*/
|
|
86
|
+
promptArrived(at?: number): CallRecord[];
|
|
87
|
+
/** Allowed calls of the turn the user just replied to, oldest first: the calls the reply could regret. */
|
|
88
|
+
candidates(): PreviousAction[];
|
|
89
|
+
/**
|
|
90
|
+
* The reply was read. With regret, the located call (or the latest candidate) is the miss and the rest are accepted;
|
|
91
|
+
* without it, every candidate is accepted. Returns the records that changed.
|
|
92
|
+
*/
|
|
93
|
+
regret(result: {
|
|
94
|
+
regretted: boolean;
|
|
95
|
+
target?: string | undefined;
|
|
96
|
+
probability?: number | undefined;
|
|
97
|
+
via: OutcomeVia;
|
|
98
|
+
}, at?: number): CallRecord[];
|
|
99
|
+
records(): readonly CallRecord[];
|
|
100
|
+
snapshot(): HoldSnapshot;
|
|
101
|
+
reset(): void;
|
|
102
|
+
private label;
|
|
103
|
+
}
|
|
104
|
+
/** Whether P(regret) from the question counts as regret. */
|
|
105
|
+
export declare function regretsAt(probability: number): boolean;
|
|
106
|
+
/** Offline stand-in for the regret question: the reply opens by stopping, undoing, or forbidding what was just done. */
|
|
107
|
+
export declare function textRegrets(prompt: string | undefined): boolean;
|
|
108
|
+
/** One line for the trace entry of a call whose outcome landed. */
|
|
109
|
+
export declare function outcomeNote(record: CallRecord): string;
|
|
110
|
+
/** One line for /warden status. */
|
|
111
|
+
export declare function formatHolds(snapshot: HoldSnapshot, logPath?: string): string;
|
|
112
|
+
/** Per-session file beside the user config: `<agent dir>/pi-warden/holds/<date>-<session>.jsonl`. */
|
|
113
|
+
export declare function holdLogPath(sessionId: string, at?: Date): string;
|
|
114
|
+
/** Rewrites the session's records as JSON lines, owner-only, one write at a time so outcomes never interleave. */
|
|
115
|
+
export declare class HoldLog {
|
|
116
|
+
readonly path: string;
|
|
117
|
+
private queue;
|
|
118
|
+
private failure;
|
|
119
|
+
constructor(path: string);
|
|
120
|
+
save(records: readonly CallRecord[]): Promise<void>;
|
|
121
|
+
/** The last write error, if the most recent save failed. */
|
|
122
|
+
get lastFailure(): string | undefined;
|
|
123
|
+
}
|
package/dist/holds.js
ADDED
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
import { mkdir, writeFile } from "node:fs/promises";
|
|
2
|
+
import { dirname, join } from "node:path";
|
|
3
|
+
import { userConfigPath } from "./config.js";
|
|
4
|
+
const REGRET_THRESHOLD = 0.7;
|
|
5
|
+
function scoresOf(verdict) {
|
|
6
|
+
const { judgment } = verdict;
|
|
7
|
+
if (!judgment)
|
|
8
|
+
return undefined;
|
|
9
|
+
const scores = { irreversible: judgment.irreversible, offTask: judgment.offTask, scope: judgment.scope };
|
|
10
|
+
if (judgment.mutates !== undefined)
|
|
11
|
+
scores.mutates = judgment.mutates;
|
|
12
|
+
if (judgment.approved !== undefined)
|
|
13
|
+
scores.approved = judgment.approved;
|
|
14
|
+
if (judgment.intentMismatch !== undefined)
|
|
15
|
+
scores.intentMismatch = judgment.intentMismatch;
|
|
16
|
+
if (judgment.securityRisk !== undefined)
|
|
17
|
+
scores.securityRisk = judgment.securityRisk;
|
|
18
|
+
return scores;
|
|
19
|
+
}
|
|
20
|
+
export class HoldLedger {
|
|
21
|
+
tracked = [];
|
|
22
|
+
nextId = 1;
|
|
23
|
+
/** One inspected call. `outcome` is set at once when the confirm dialog decided; a steer-mode hold starts pending. */
|
|
24
|
+
record(verdict, options) {
|
|
25
|
+
const at = options.at ?? Date.now();
|
|
26
|
+
const record = {
|
|
27
|
+
id: this.nextId++,
|
|
28
|
+
at,
|
|
29
|
+
tool: verdict.summary.tool,
|
|
30
|
+
level: verdict.level,
|
|
31
|
+
source: verdict.source,
|
|
32
|
+
mode: options.mode,
|
|
33
|
+
held: options.held,
|
|
34
|
+
patterns: verdict.patterns.map(hit => hit.id),
|
|
35
|
+
reasons: [...verdict.reasons],
|
|
36
|
+
planChars: verdict.plan?.length ?? 0,
|
|
37
|
+
outcome: options.outcome ?? "pending",
|
|
38
|
+
};
|
|
39
|
+
const scores = scoresOf(verdict);
|
|
40
|
+
if (scores)
|
|
41
|
+
record.scores = scores;
|
|
42
|
+
if (options.outcome && options.outcome !== "pending") {
|
|
43
|
+
record.outcomeAt = at;
|
|
44
|
+
if (options.via)
|
|
45
|
+
record.outcomeVia = options.via;
|
|
46
|
+
}
|
|
47
|
+
this.tracked.push({ record, summary: verdict.summary, prompts: 0 });
|
|
48
|
+
return record;
|
|
49
|
+
}
|
|
50
|
+
/**
|
|
51
|
+
* The user's reply released a hold: the pending hold of the same tool is the false positive, the latest one when the
|
|
52
|
+
* tools differ (the retry rarely repeats the held call byte for byte, and a hold can be released through another tool).
|
|
53
|
+
*/
|
|
54
|
+
approved(tool, via = "retry", at = Date.now()) {
|
|
55
|
+
const pending = this.tracked.filter(item => item.record.held && item.record.outcome === "pending");
|
|
56
|
+
const match = pending.filter(item => item.record.tool === tool).at(-1) ?? pending.at(-1);
|
|
57
|
+
if (!match)
|
|
58
|
+
return undefined;
|
|
59
|
+
return this.label(match, "approved", via, at);
|
|
60
|
+
}
|
|
61
|
+
/**
|
|
62
|
+
* A new user prompt. Every pending record has seen one more prompt. A hold still pending after the prompt that could
|
|
63
|
+
* have approved it (the reply) and the turn that followed is a re-plan: the agent found another way or the user said no.
|
|
64
|
+
*/
|
|
65
|
+
promptArrived(at = Date.now()) {
|
|
66
|
+
const changed = [];
|
|
67
|
+
for (const item of this.tracked) {
|
|
68
|
+
if (item.record.outcome !== "pending")
|
|
69
|
+
continue;
|
|
70
|
+
item.prompts++;
|
|
71
|
+
if (item.record.held && item.prompts >= 2)
|
|
72
|
+
changed.push(this.label(item, "replanned", "next prompt", at));
|
|
73
|
+
}
|
|
74
|
+
return changed;
|
|
75
|
+
}
|
|
76
|
+
/** Allowed calls of the turn the user just replied to, oldest first: the calls the reply could regret. */
|
|
77
|
+
candidates() {
|
|
78
|
+
return this.tracked
|
|
79
|
+
.filter(item => !item.record.held && item.record.outcome === "pending" && item.prompts === 1)
|
|
80
|
+
.map(item => ({
|
|
81
|
+
id: `a${item.record.id}`,
|
|
82
|
+
tool: item.summary.tool,
|
|
83
|
+
...(item.summary.command !== undefined ? { command: clip(item.summary.command, 300) } : {}),
|
|
84
|
+
...(item.summary.path !== undefined ? { path: item.summary.path } : {}),
|
|
85
|
+
}));
|
|
86
|
+
}
|
|
87
|
+
/**
|
|
88
|
+
* The reply was read. With regret, the located call (or the latest candidate) is the miss and the rest are accepted;
|
|
89
|
+
* without it, every candidate is accepted. Returns the records that changed.
|
|
90
|
+
*/
|
|
91
|
+
regret(result, at = Date.now()) {
|
|
92
|
+
const candidates = this.tracked.filter(item => !item.record.held && item.record.outcome === "pending" && item.prompts === 1);
|
|
93
|
+
if (!candidates.length)
|
|
94
|
+
return [];
|
|
95
|
+
const target = result.regretted ? candidates.find(item => `a${item.record.id}` === result.target) ?? candidates.at(-1) : undefined;
|
|
96
|
+
return candidates.map(item => {
|
|
97
|
+
const record = this.label(item, item === target ? "regretted" : "accepted", result.via, at);
|
|
98
|
+
if (result.probability !== undefined)
|
|
99
|
+
record.regret = result.probability;
|
|
100
|
+
return record;
|
|
101
|
+
});
|
|
102
|
+
}
|
|
103
|
+
records() {
|
|
104
|
+
return this.tracked.map(item => item.record);
|
|
105
|
+
}
|
|
106
|
+
snapshot() {
|
|
107
|
+
const records = this.records();
|
|
108
|
+
const holds = records.filter(record => record.held);
|
|
109
|
+
const allowed = records.filter(record => !record.held);
|
|
110
|
+
const count = (list, outcome) => list.filter(record => record.outcome === outcome).length;
|
|
111
|
+
const approved = count(holds, "approved");
|
|
112
|
+
const declined = count(holds, "declined");
|
|
113
|
+
const replanned = count(holds, "replanned");
|
|
114
|
+
const labels = approved + declined + replanned;
|
|
115
|
+
return {
|
|
116
|
+
holds: holds.length, approved, declined, replanned, awaiting: count(holds, "pending"),
|
|
117
|
+
allowed: allowed.length, regretted: count(allowed, "regretted"), accepted: count(allowed, "accepted"),
|
|
118
|
+
labels, precision: labels ? (declined + replanned) / labels : undefined,
|
|
119
|
+
};
|
|
120
|
+
}
|
|
121
|
+
reset() {
|
|
122
|
+
this.tracked.length = 0;
|
|
123
|
+
this.nextId = 1;
|
|
124
|
+
}
|
|
125
|
+
label(item, outcome, via, at) {
|
|
126
|
+
item.record.outcome = outcome;
|
|
127
|
+
item.record.outcomeAt = at;
|
|
128
|
+
item.record.outcomeVia = via;
|
|
129
|
+
return item.record;
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
const clip = (text, limit) => (text.length <= limit ? text : `${text.slice(0, limit)}…`);
|
|
133
|
+
/** Whether P(regret) from the question counts as regret. */
|
|
134
|
+
export function regretsAt(probability) {
|
|
135
|
+
return probability >= REGRET_THRESHOLD;
|
|
136
|
+
}
|
|
137
|
+
/** Offline stand-in for the regret question: the reply opens by stopping, undoing, or forbidding what was just done. */
|
|
138
|
+
export function textRegrets(prompt) {
|
|
139
|
+
const text = (prompt ?? "").trim();
|
|
140
|
+
if (!text)
|
|
141
|
+
return false;
|
|
142
|
+
const opening = text.slice(0, 160);
|
|
143
|
+
return /^(?:wait(?! (?:for|until|till)\b)|stop\b|nope\b|hold on\b|don[’']t\b|do not\b|undo\b|revert\b|roll ?back\b|why did you\b|you (?:shouldn[’']t|should not) have\b|that (?:was|is) (?:wrong|not what)\b|no[,.!])/i.test(opening)
|
|
144
|
+
|| /\b(?:undo|revert|roll ?back) (?:that|this|it|the last|those|what you)\b/i.test(opening)
|
|
145
|
+
|| /\b(?:don[’']t|do not|never) (?:do|run|touch|push|delete|remove|commit) that\b/i.test(opening);
|
|
146
|
+
}
|
|
147
|
+
/** One line for the trace entry of a call whose outcome landed. */
|
|
148
|
+
export function outcomeNote(record) {
|
|
149
|
+
switch (record.outcome) {
|
|
150
|
+
case "approved": return `outcome: approved by the user (${record.outcomeVia === "dialog" ? "confirm dialog" : "released on retry"}); the hold was a false positive`;
|
|
151
|
+
case "declined": return "outcome: declined by the user in the confirm dialog; the hold stood";
|
|
152
|
+
case "replanned": return "outcome: never approved after the user replied; the agent re-planned, the hold stood";
|
|
153
|
+
case "regretted": return `outcome: the user's next message regrets this call${record.regret !== undefined ? ` (${record.regret.toFixed(2)})` : ""}; it should have been held`;
|
|
154
|
+
case "accepted": return `outcome: the user's next message does not regret this call${record.regret !== undefined ? ` (${record.regret.toFixed(2)})` : ""}`;
|
|
155
|
+
default: return "outcome: pending";
|
|
156
|
+
}
|
|
157
|
+
}
|
|
158
|
+
/** One line for /warden status. */
|
|
159
|
+
export function formatHolds(snapshot, logPath) {
|
|
160
|
+
if (snapshot.holds === 0 && snapshot.allowed === 0)
|
|
161
|
+
return "Holds: no guarded call judged yet this session.";
|
|
162
|
+
const parts = [`${snapshot.holds} hold${snapshot.holds === 1 ? "" : "s"}`];
|
|
163
|
+
if (snapshot.holds)
|
|
164
|
+
parts.push(`${snapshot.approved} approved by you, ${snapshot.declined} declined, ${snapshot.replanned} re-planned, ${snapshot.awaiting} awaiting your reply`);
|
|
165
|
+
parts.push(snapshot.precision === undefined ? "precision not yet measurable" : `precision ${Math.round(snapshot.precision * 100)}% over ${snapshot.labels} label${snapshot.labels === 1 ? "" : "s"}`);
|
|
166
|
+
parts.push(`${snapshot.allowed} allowed (${snapshot.regretted} regretted by you, ${snapshot.accepted} accepted)`);
|
|
167
|
+
return `Holds: ${parts.join("; ")}.${logPath ? ` Log: ${logPath}.` : ""}`;
|
|
168
|
+
}
|
|
169
|
+
/** Per-session file beside the user config: `<agent dir>/pi-warden/holds/<date>-<session>.jsonl`. */
|
|
170
|
+
export function holdLogPath(sessionId, at = new Date()) {
|
|
171
|
+
const day = at.toISOString().slice(0, 10);
|
|
172
|
+
const safe = sessionId.replace(/[^A-Za-z0-9_-]/g, "").slice(0, 40) || String(process.pid);
|
|
173
|
+
return join(dirname(userConfigPath()), "holds", `${day}-${safe}.jsonl`);
|
|
174
|
+
}
|
|
175
|
+
/** Rewrites the session's records as JSON lines, owner-only, one write at a time so outcomes never interleave. */
|
|
176
|
+
export class HoldLog {
|
|
177
|
+
path;
|
|
178
|
+
queue = Promise.resolve();
|
|
179
|
+
failure;
|
|
180
|
+
constructor(path) {
|
|
181
|
+
this.path = path;
|
|
182
|
+
}
|
|
183
|
+
save(records) {
|
|
184
|
+
const text = records.map(record => JSON.stringify(record)).join("\n") + (records.length ? "\n" : "");
|
|
185
|
+
this.queue = this.queue.then(async () => {
|
|
186
|
+
await mkdir(dirname(this.path), { recursive: true, mode: 0o700 });
|
|
187
|
+
await writeFile(this.path, text, { mode: 0o600 });
|
|
188
|
+
this.failure = undefined;
|
|
189
|
+
}).catch(error => { this.failure = error instanceof Error ? error.message : String(error); });
|
|
190
|
+
return this.queue;
|
|
191
|
+
}
|
|
192
|
+
/** The last write error, if the most recent save failed. */
|
|
193
|
+
get lastFailure() {
|
|
194
|
+
return this.failure;
|
|
195
|
+
}
|
|
196
|
+
}
|
package/dist/index.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
|
-
export { evaluateAction, describeAction, matchPatterns, isReadOnlyCommand, stripDataText, buildRequest, formatVerdict, steerReason, questions, slopQuestions, approvalQuestion, securityQuestion } from "./guard.js";
|
|
2
|
-
export type { ActionInput, ActionSummary, EvaluateOptions, Judge, Judgment, SlopJudgment, SlopSymptom, Level, PatternHit, ScannedCommand, ScopeLabel, Severity, Verdict } from "./guard.js";
|
|
1
|
+
export { evaluateAction, describeAction, describePlan, matchPatterns, isReadOnlyCommand, stripDataText, buildRequest, formatVerdict, steerReason, intentSteer, questions, intentQuestion, slopQuestions, approvalQuestion, securityQuestion, regretQuestions } from "./guard.js";
|
|
2
|
+
export type { ActionInput, ActionSummary, EvaluateOptions, Judge, Judgment, SlopJudgment, SlopSymptom, Level, PatternHit, PreviousAction, ScannedCommand, ScopeLabel, Severity, Verdict } from "./guard.js";
|
|
3
3
|
export { SLOP_SYMPTOMS, SLOP_LABELS } from "./guard.js";
|
|
4
4
|
export { ActionGuard } from "./action-guard.js";
|
|
5
5
|
export type { Conversation, InspectOptions, ToolCallRef } from "./action-guard.js";
|
|
@@ -22,6 +22,8 @@ export { RunawayMonitor, repeatedBlock, repeatedTail, findRepeats, runawayNudge,
|
|
|
22
22
|
export type { RunawayVerdict, RepeatSignal, StreamKind } from "./runaway.js";
|
|
23
23
|
export { completeConfig, shapeWarning, EXPECTED_SCHEMA } from "./shape.js";
|
|
24
24
|
export { ContextLedger, formatLedger } from "./saver.js";
|
|
25
|
+
export { HoldLedger, HoldLog, holdLogPath, formatHolds, outcomeNote, regretsAt, textRegrets } from "./holds.js";
|
|
26
|
+
export type { CallOutcome, CallRecord, CallScores, HoldSnapshot, OutcomeVia } from "./holds.js";
|
|
25
27
|
export type { ContextLedgerSnapshot, RecallKind } from "./saver.js";
|
|
26
28
|
export type { ShapeResult } from "./shape.js";
|
|
27
29
|
export { CONFIG_SCHEMA, defaultConfig, loadConfig, applyUserOverrides, applyProjectOverrides, isMode, isRecallTool, userConfigPath, projectConfigPath, readUserConfig, writeUserConfig, setUserSetting, PACKAGE_NAME } from "./config.js";
|
package/dist/index.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export { evaluateAction, describeAction, matchPatterns, isReadOnlyCommand, stripDataText, buildRequest, formatVerdict, steerReason, questions, slopQuestions, approvalQuestion, securityQuestion } from "./guard.js";
|
|
1
|
+
export { evaluateAction, describeAction, describePlan, matchPatterns, isReadOnlyCommand, stripDataText, buildRequest, formatVerdict, steerReason, intentSteer, questions, intentQuestion, slopQuestions, approvalQuestion, securityQuestion, regretQuestions } from "./guard.js";
|
|
2
2
|
export { SLOP_SYMPTOMS, SLOP_LABELS } from "./guard.js";
|
|
3
3
|
export { ActionGuard } from "./action-guard.js";
|
|
4
4
|
export { evaluateProse, buildProseRequest, proseQuestions, proseNudge, describeAudience, ProseTrend, PROSE_SYMPTOMS, PROSE_LABELS } from "./prose.js";
|
|
@@ -12,6 +12,7 @@ export { detectNotifier, notifierCommand, sendNotification, candidates as notifi
|
|
|
12
12
|
export { RunawayMonitor, repeatedBlock, repeatedTail, findRepeats, runawayNudge, formatRunaway, BLOCK_MIN_CHARS, PHRASE_CHARS, CHECK_EVERY } from "./runaway.js";
|
|
13
13
|
export { completeConfig, shapeWarning, EXPECTED_SCHEMA } from "./shape.js";
|
|
14
14
|
export { ContextLedger, formatLedger } from "./saver.js";
|
|
15
|
+
export { HoldLedger, HoldLog, holdLogPath, formatHolds, outcomeNote, regretsAt, textRegrets } from "./holds.js";
|
|
15
16
|
export { CONFIG_SCHEMA, defaultConfig, loadConfig, applyUserOverrides, applyProjectOverrides, isMode, isRecallTool, userConfigPath, projectConfigPath, readUserConfig, writeUserConfig, setUserSetting, PACKAGE_NAME } from "./config.js";
|
|
16
17
|
export { commandOf, outputReportsFailure, COMMAND_TOOLS } from "./tools.js";
|
|
17
18
|
export { renderTemplate, actionTokens, stuckTokens, doneTokens, proseTokens, runawayTokens, rulesTokens, defaultWidgetConfig, DEFAULT_TEMPLATES, TOKEN_NAMES } from "./widget.js";
|
package/dist/shape.js
CHANGED
|
@@ -26,7 +26,7 @@ export function completeConfig(loaded) {
|
|
|
26
26
|
timeoutMs: source.timeoutMs ?? 5000,
|
|
27
27
|
maxRequests: source.maxRequests ?? 500,
|
|
28
28
|
steerVisible: source.steerVisible ?? false,
|
|
29
|
-
action: section("action", { ...off, tools: [], failOpen: true, timeoutMs: 5000, irreversible: { warn: 1, confirm: 1 }, offTask: { warn: 1, confirm: 1 } }),
|
|
29
|
+
action: section("action", { ...off, tools: [], failOpen: true, timeoutMs: 5000, irreversible: { warn: 1, confirm: 1 }, offTask: { warn: 1, confirm: 1 }, intentMismatch: 1, feedbackLog: false }),
|
|
30
30
|
stuck: section("stuck", { ...off, window: 12, minFailures: 3, cooldown: 3, sameStrategy: 1, nudge: false }),
|
|
31
31
|
done: section("done", { ...off, claimsDone: 1, nudge: false }),
|
|
32
32
|
slop: section("slop", { ...off, threshold: 1, prose: proseOff() }),
|
|
@@ -48,6 +48,11 @@ export function completeConfig(loaded) {
|
|
|
48
48
|
}
|
|
49
49
|
if (typeof config.widget.panelWidth !== "string" && typeof config.widget.panelWidth !== "number")
|
|
50
50
|
config.widget = { ...config.widget, panelWidth: "40%" };
|
|
51
|
+
// The feedback log flag was added inside the action section later than the section itself; an older config module leaves it undefined and the log stays on.
|
|
52
|
+
if (typeof config.action.feedbackLog !== "boolean")
|
|
53
|
+
config.action = { ...config.action, feedbackLog: true };
|
|
54
|
+
if (typeof config.action.intentMismatch !== "number")
|
|
55
|
+
config.action = { ...config.action, intentMismatch: 0.8 };
|
|
51
56
|
// The runaway guard added its template later than the other sections; an older widget section renders the default line.
|
|
52
57
|
if (typeof config.widget.runaway !== "string")
|
|
53
58
|
config.widget = { ...config.widget, runaway: DEFAULT_TEMPLATES.runaway };
|
package/dist/trace.d.ts
CHANGED
|
@@ -20,6 +20,8 @@ export declare class Trace {
|
|
|
20
20
|
private readonly listeners;
|
|
21
21
|
constructor(limit?: number);
|
|
22
22
|
push(entry: TraceEntry): void;
|
|
23
|
+
/** Appends a detail line to an entry that is still in the trace, for an outcome that lands after the event. */
|
|
24
|
+
amend(entry: TraceEntry, line: string): boolean;
|
|
23
25
|
entries(): readonly TraceEntry[];
|
|
24
26
|
clear(): void;
|
|
25
27
|
subscribe(listener: () => void): () => void;
|
package/dist/trace.js
CHANGED
|
@@ -13,6 +13,15 @@ export class Trace {
|
|
|
13
13
|
for (const listener of this.listeners)
|
|
14
14
|
listener();
|
|
15
15
|
}
|
|
16
|
+
/** Appends a detail line to an entry that is still in the trace, for an outcome that lands after the event. */
|
|
17
|
+
amend(entry, line) {
|
|
18
|
+
if (!this.items.includes(entry))
|
|
19
|
+
return false;
|
|
20
|
+
entry.details.push(line);
|
|
21
|
+
for (const listener of this.listeners)
|
|
22
|
+
listener();
|
|
23
|
+
return true;
|
|
24
|
+
}
|
|
16
25
|
entries() {
|
|
17
26
|
return this.items;
|
|
18
27
|
}
|
|
@@ -37,10 +46,12 @@ export function actionDetails(verdict, extra = {}) {
|
|
|
37
46
|
lines.push(`${summary.tool} ${summary.path}${summary.location === "outside_project" ? " (outside the project)" : ""}${summary.exists === false ? " (new file)" : ""}${summary.bytes !== undefined ? ` · ${summary.bytes} bytes` : ""}${summary.editCount !== undefined ? ` · ${summary.editCount} edit${summary.editCount === 1 ? "" : "s"}` : ""}`);
|
|
38
47
|
if (summary.input !== undefined)
|
|
39
48
|
lines.push(`input: ${clip(summary.input, 300)}`);
|
|
49
|
+
if (verdict.plan !== undefined)
|
|
50
|
+
lines.push(`plan: ${clip(verdict.plan.replace(/\s+/g, " "), 300)}`);
|
|
40
51
|
if (verdict.patterns.length)
|
|
41
52
|
lines.push(`patterns: ${verdict.patterns.map(hit => `${hit.id} (${hit.severity})`).join(", ")}`);
|
|
42
53
|
if (judgment) {
|
|
43
|
-
lines.push(`jev: irreversible ${percent(judgment.irreversible)} · off-task ${percent(judgment.offTask)} · ${judgment.scope.replace(/_/g, " ")} (${percent(judgment.scopeConfidence)})${judgment.approved !== undefined ? ` · approved ${percent(judgment.approved)}` : ""}${judgment.mutates !== undefined ? ` · mutates ${percent(judgment.mutates)}` : ""} · ${judgment.model} · ${judgment.elapsedMs} ms`);
|
|
54
|
+
lines.push(`jev: irreversible ${percent(judgment.irreversible)} · off-task ${percent(judgment.offTask)} · ${judgment.scope.replace(/_/g, " ")} (${percent(judgment.scopeConfidence)})${judgment.approved !== undefined ? ` · approved ${percent(judgment.approved)}` : ""}${judgment.mutates !== undefined ? ` · mutates ${percent(judgment.mutates)}` : ""}${judgment.intentMismatch !== undefined ? ` · intent mismatch ${percent(judgment.intentMismatch)}` : ""}${judgment.regretted !== undefined ? ` · regret of last turn ${percent(judgment.regretted)}` : ""} · ${judgment.model} · ${judgment.elapsedMs} ms`);
|
|
44
55
|
}
|
|
45
56
|
if (judgment?.securityRisk !== undefined)
|
|
46
57
|
lines.push(`security risk: ${percent(judgment.securityRisk)}`);
|
package/dist/widget.d.ts
CHANGED
|
@@ -51,7 +51,7 @@ export declare const TOKEN_NAMES: {
|
|
|
51
51
|
readonly context: readonly ["tool", "retention", "bytesSaved"];
|
|
52
52
|
readonly runaway: readonly ["kind", "count", "chars", "signal", "block", "status", "time", "guard"];
|
|
53
53
|
readonly rules: readonly ["tool", "path", "asked", "violations", "status", "source", "reasons", "model", "ms", "flags", "time", "guard"];
|
|
54
|
-
readonly action: readonly ["tool", "level", "source", "irreversible", "offTask", "scope", "approved", "slop", "slopStub", "slopComments", "slopDead", "slopHedging", "patterns", "reasons", "path", "model", "ms", "flags", "time", "guard"];
|
|
54
|
+
readonly action: readonly ["tool", "level", "source", "irreversible", "offTask", "scope", "approved", "intent", "plan", "slop", "slopStub", "slopComments", "slopDead", "slopHedging", "patterns", "reasons", "path", "model", "ms", "flags", "time", "guard"];
|
|
55
55
|
readonly prose: readonly ["wordy", "cliches", "jargon", "status", "reasons", "model", "ms", "flags", "time", "guard"];
|
|
56
56
|
readonly stuck: readonly ["failures", "sameStrategy", "approachChange", "progress", "status", "source", "reasons", "model", "ms", "flags", "time", "guard"];
|
|
57
57
|
readonly done: readonly ["changes", "checks", "checksPassed", "claimsDone", "claimsVerified", "checksApply", "outcome", "status", "reasons", "model", "ms", "flags", "time", "guard"];
|
package/dist/widget.js
CHANGED
|
@@ -37,6 +37,7 @@ const time = (at) => new Date(at).toTimeString().slice(0, 8);
|
|
|
37
37
|
export function actionTokens(verdict, at = Date.now()) {
|
|
38
38
|
const flags = [
|
|
39
39
|
verdict.approvedByUser ? "user approved" : undefined,
|
|
40
|
+
verdict.intentMismatch ? "off plan" : undefined,
|
|
40
41
|
verdict.source === "error" ? "typesafe error" : undefined,
|
|
41
42
|
verdict.source === "read-only" ? "read-only" : undefined,
|
|
42
43
|
].filter(Boolean).join(", ");
|
|
@@ -50,6 +51,8 @@ export function actionTokens(verdict, at = Date.now()) {
|
|
|
50
51
|
offTask: fixed(verdict.judgment?.offTask),
|
|
51
52
|
scope: verdict.judgment?.scope.replace(/_/g, " "),
|
|
52
53
|
approved: fixed(verdict.judgment?.approved),
|
|
54
|
+
intent: fixed(verdict.judgment?.intentMismatch),
|
|
55
|
+
plan: verdict.plan === undefined ? undefined : (verdict.plan.length <= 80 ? verdict.plan : `${verdict.plan.slice(0, 80)}…`).replace(/\s+/g, " "),
|
|
53
56
|
slop: verdict.slopSymptoms?.length ? verdict.slopSymptoms.map(symptom => `${symptom} ${verdict.slop[symptom].toFixed(2)}`).join(", ") : verdict.slop ? "none" : undefined,
|
|
54
57
|
slopStub: fixed(verdict.slop?.stub),
|
|
55
58
|
slopComments: fixed(verdict.slop?.comments),
|
|
@@ -149,7 +152,7 @@ export const TOKEN_NAMES = {
|
|
|
149
152
|
context: ["tool", "retention", "bytesSaved"],
|
|
150
153
|
runaway: ["kind", "count", "chars", "signal", "block", "status", "time", "guard"],
|
|
151
154
|
rules: ["tool", "path", "asked", "violations", "status", "source", "reasons", "model", "ms", "flags", "time", "guard"],
|
|
152
|
-
action: ["tool", "level", "source", "irreversible", "offTask", "scope", "approved", "slop", "slopStub", "slopComments", "slopDead", "slopHedging", "patterns", "reasons", "path", "model", "ms", "flags", "time", "guard"],
|
|
155
|
+
action: ["tool", "level", "source", "irreversible", "offTask", "scope", "approved", "intent", "plan", "slop", "slopStub", "slopComments", "slopDead", "slopHedging", "patterns", "reasons", "path", "model", "ms", "flags", "time", "guard"],
|
|
153
156
|
prose: ["wordy", "cliches", "jargon", "status", "reasons", "model", "ms", "flags", "time", "guard"],
|
|
154
157
|
stuck: ["failures", "sameStrategy", "approachChange", "progress", "status", "source", "reasons", "model", "ms", "flags", "time", "guard"],
|
|
155
158
|
done: ["changes", "checks", "checksPassed", "claimsDone", "claimsVerified", "checksApply", "outcome", "status", "reasons", "model", "ms", "flags", "time", "guard"],
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-warden",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.0",
|
|
4
4
|
"description": "Guardrails for Pi built on pi-typesafe that steer the agent instead of interrupting you: Jev judges irreversible and off-task tool calls before they run, checks written code against your project rules, detects stuck loops, checks unverified done claims, and flags slop.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|