pi-jev-auto-mode 0.1.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -0
- package/README.md +13 -4
- package/docs/calibration.md +20 -0
- package/docs/design.md +27 -13
- package/docs/security.md +3 -1
- package/package.json +1 -1
- package/src/decide.ts +1 -1
- package/src/extension.ts +106 -11
- package/src/intent.ts +5 -2
- package/src/jev/engine.ts +1 -0
- package/src/jev/questions.ts +33 -3
- package/src/policy.ts +3 -1
- package/src/settings.ts +23 -0
- package/src/ui.ts +100 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,46 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.3.0
|
|
4
|
+
|
|
5
|
+
The 0.2.0 default resolved the middle band as a block, but two conditions were still
|
|
6
|
+
`required`, which made the gate strict for structurally wrong reasons rather than measured
|
|
7
|
+
ones. Corrected:
|
|
8
|
+
|
|
9
|
+
- `intent_coverage` 0.80 → **0.60**. Measured answers are 0.77–0.98 when the user asked and
|
|
10
|
+
0.06–0.15 when they did not, so 0.80 sat on top of the "asked" cluster instead of inside the
|
|
11
|
+
empty band between the two. The middle band is now (0.40, 0.60).
|
|
12
|
+
- `policy_compliance` required → **hazard**. It measured 0.66–0.85 on calls where nothing was
|
|
13
|
+
wrong, so as a requirement it blocked every gated call the moment a policy was configured.
|
|
14
|
+
Now only a clear violation stops a call.
|
|
15
|
+
- `path_not_protected` required → **hazard**. An unclear answer no longer blocks on its own; the
|
|
16
|
+
user's request decides. A target the model clearly identifies as a credential store still
|
|
17
|
+
blocks (`.env` measured p = 0.02, `~/.ssh` p = 0.03).
|
|
18
|
+
- New `no_fetched_code_execution`, required, asked only for commands the deterministic layer
|
|
19
|
+
already recognised as downloaded-script execution. `curl | bash` stays blocked (p = 0.02)
|
|
20
|
+
without making every other call strict.
|
|
21
|
+
- `.env.example`, `.env.sample`, `.env.template`, and `.env.dist` are no longer treated as
|
|
22
|
+
credential stores: templates belong in the repository.
|
|
23
|
+
- The intent window widened from 8 messages / 4000 characters to 12 / 6000, so an ongoing task
|
|
24
|
+
does not lose the request that justifies it and look unrequested as a result.
|
|
25
|
+
|
|
26
|
+
## 0.2.0
|
|
27
|
+
|
|
28
|
+
- **The middle band no longer asks the user by default.** An auto mode that stops to ask has
|
|
29
|
+
handed the decision back to a human, and the agent can always ask in conversation if it needs
|
|
30
|
+
guidance. A judgment that is neither satisfied nor rejected now blocks, so the gate never
|
|
31
|
+
takes over the screen.
|
|
32
|
+
- `uncertain` setting and `/jev-auto-mode uncertain deny|ask|allow` control it. `deny` is the
|
|
33
|
+
default; `ask` restores the confirmation dialog; `allow` trusts the band.
|
|
34
|
+
- `/jev-auto-mode threshold edit` picks a rule and prompts for a value, showing each rule's
|
|
35
|
+
current threshold next to the last probability the model returned for it.
|
|
36
|
+
|
|
37
|
+
## 0.1.2
|
|
38
|
+
|
|
39
|
+
- Fix the screen thrashing that happened whenever a judgment was delegated to the user: the
|
|
40
|
+
confirmation dialog was handed the whole command, and Pi's dialogs do not clip their content,
|
|
41
|
+
so a long command produced a dialog taller than the terminal. The dialog now shows a bounded
|
|
42
|
+
preview and says what was hidden.
|
|
43
|
+
|
|
3
44
|
## 0.1.1
|
|
4
45
|
|
|
5
46
|
- Correct the product name. It is **Jev** — TypeSafe's System One model, spelled with a
|
package/README.md
CHANGED
|
@@ -8,7 +8,7 @@ cannot be made.
|
|
|
8
8
|
|
|
9
9
|
> **Status: milestones 1–3 are complete.** The deterministic envelope, the Jev engine,
|
|
10
10
|
> real-API calibration, settings, policy notes, per-rule threshold tuning, and decision
|
|
11
|
-
> records are implemented and tested (
|
|
11
|
+
> records are implemented and tested (171 tests, no network). See [`docs/design.md`](./docs/design.md) for
|
|
12
12
|
> the roadmap and [`docs/calibration.md`](./docs/calibration.md) for the measured
|
|
13
13
|
> probabilities behind every threshold.
|
|
14
14
|
|
|
@@ -26,7 +26,7 @@ your deny pattern → block
|
|
|
26
26
|
your allow pattern → allow
|
|
27
27
|
safe read-only command → run, no record
|
|
28
28
|
in-project write/edit → run, no record
|
|
29
|
-
everything else → Jev: allow · block ·
|
|
29
|
+
everything else → Jev: allow · block · block-if-undecidable
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
`rm -rf build` inside the repository is recognized as a scoped local deletion. A write to
|
|
@@ -48,8 +48,13 @@ Conditions are phrased so the safe state is "yes", and each one is classified by
|
|
|
48
48
|
So `intent_coverage` ("is this what the user asked for?") is the permission question, and
|
|
49
49
|
questions like "is a secret being sent to a network endpoint" are hazard detectors that only
|
|
50
50
|
block when they are sure. Posting a private key is never cleared by intent; force-pushing a
|
|
51
|
-
feature branch the user asked for is.
|
|
52
|
-
|
|
51
|
+
feature branch the user asked for is.
|
|
52
|
+
|
|
53
|
+
**Nothing is delegated to the user by default.** The middle band — where Jev is neither
|
|
54
|
+
satisfied nor rejecting — resolves to a block, so Jev's probability is the whole answer and the
|
|
55
|
+
gate never takes over the screen. `/jev-auto-mode uncertain ask` restores the confirmation
|
|
56
|
+
dialog if you want it; `allow` trusts the band. Everything else that cannot be decided — no
|
|
57
|
+
engine, timeout, malformed response, cancellation — also blocks.
|
|
53
58
|
|
|
54
59
|
## Install
|
|
55
60
|
|
|
@@ -84,7 +89,10 @@ Packages are discovered in the [package gallery](https://pi.dev/packages) throug
|
|
|
84
89
|
/jev-auto-mode policy clear
|
|
85
90
|
/jev-auto-mode threshold show thresholds and the last observed probability per rule
|
|
86
91
|
/jev-auto-mode threshold <rule> <0.5-1> set one threshold
|
|
92
|
+
/jev-auto-mode threshold edit pick a rule and type a value
|
|
87
93
|
/jev-auto-mode threshold reset [rule] restore the calibrated default
|
|
94
|
+
/jev-auto-mode uncertain show what the middle band resolves to
|
|
95
|
+
/jev-auto-mode uncertain deny|ask|allow
|
|
88
96
|
```
|
|
89
97
|
|
|
90
98
|
```
|
|
@@ -164,6 +172,7 @@ Policy notes: `$PI_CODING_AGENT_DIR/jev-auto-mode-policy.md`.
|
|
|
164
172
|
"disallowedCommands": ["npm publish*"],
|
|
165
173
|
"extraProtectedPaths": [],
|
|
166
174
|
"maxStateCharacters": 120000,
|
|
175
|
+
"uncertain": "deny",
|
|
167
176
|
"thresholds": {}
|
|
168
177
|
}
|
|
169
178
|
```
|
package/docs/calibration.md
CHANGED
|
@@ -124,6 +124,26 @@ transcript. The fixture set gained `post a private key, asked for`, which verifi
|
|
|
124
124
|
the whole design rests on: with the user explicitly asking for it, `no_secret_egress` still
|
|
125
125
|
rejects (p = 0.01) and the call is blocked.
|
|
126
126
|
|
|
127
|
+
## Third revision: strictness for the wrong reason
|
|
128
|
+
|
|
129
|
+
With the middle band resolving to a block, the two `required` conditions left in the set turned
|
|
130
|
+
out to be doing the blocking, and not because the calls were risky:
|
|
131
|
+
|
|
132
|
+
- `intent_coverage` at 0.80 was above the entire "asked" cluster. The measured gap is
|
|
133
|
+
**0.15 → 0.77**: nothing the model answered landed between those values, so any threshold in
|
|
134
|
+
that range separates perfectly and 0.80 was simply the strict end of nothing. Now 0.60, with
|
|
135
|
+
the middle band at (0.40, 0.60).
|
|
136
|
+
- `policy_compliance` answered 0.66–0.85 even for calls that were plainly fine. As a requirement
|
|
137
|
+
that blocks, so **configuring a policy disabled the gate entirely**. It is a hazard detector
|
|
138
|
+
now: only a clear violation stops a call.
|
|
139
|
+
- `path_not_protected` had the same shape: the target was escalated *because* the deterministic
|
|
140
|
+
layer distrusts it, so asking the model to certify it as safe and blocking when it hedges put
|
|
141
|
+
the burden in the wrong place. The intent question decides; a clear credential-store answer
|
|
142
|
+
still blocks (`.env` p = 0.02, `~/.ssh` p = 0.03).
|
|
143
|
+
|
|
144
|
+
The lesson generalises: a threshold should be chosen so that no measured answer sits near it.
|
|
145
|
+
Both mistakes here came from picking the strict end of a band instead of the middle of a gap.
|
|
146
|
+
|
|
127
147
|
## Tuning without the script
|
|
128
148
|
|
|
129
149
|
The same numbers arrive in every session. Expand a decision record in the transcript to see
|
package/docs/design.md
CHANGED
|
@@ -45,7 +45,8 @@ tool_call(bash | write | edit)
|
|
|
45
45
|
├ every `required` condition satisfied, no hazard rejected → allow
|
|
46
46
|
├ any `hazard` condition rejected → block
|
|
47
47
|
├ any `soft` condition rejected → block, unless the user's own request covers it
|
|
48
|
-
├ any `required` condition unclear →
|
|
48
|
+
├ any `required` condition unclear → resolved by the `uncertain` setting
|
|
49
|
+
│ (default: block; `ask` prompts, `allow` passes)
|
|
49
50
|
└ unavailable → block (fail-closed)
|
|
50
51
|
└─ 7. record the decision via appendEntry (never enters LLM context)
|
|
51
52
|
```
|
|
@@ -63,20 +64,33 @@ participates:
|
|
|
63
64
|
- `severity`: `hazard` (a rejection always blocks) or `soft` (a rejection is cleared when the
|
|
64
65
|
user's own request covers the call)
|
|
65
66
|
|
|
66
|
-
| rule id | mode | severity | threshold |
|
|
67
|
-
|
|
68
|
-
| `intent_coverage` | required | hazard | 0.
|
|
69
|
-
| `
|
|
70
|
-
| `
|
|
71
|
-
| `
|
|
72
|
-
| `
|
|
73
|
-
| `
|
|
74
|
-
| `
|
|
75
|
-
| `
|
|
67
|
+
| rule id | mode | severity | threshold | asked when |
|
|
68
|
+
|---|---|---|---|---|
|
|
69
|
+
| `intent_coverage` | required | hazard | 0.60 | always |
|
|
70
|
+
| `no_fetched_code_execution` | required | hazard | 0.90 | the command downloads code and runs it |
|
|
71
|
+
| `policy_compliance` | hazard | hazard | 0.80 | a policy is configured |
|
|
72
|
+
| `path_not_protected` | hazard | hazard | 0.90 | the deterministic layer flagged the target |
|
|
73
|
+
| `local_scope` | hazard | soft | 0.90 | always |
|
|
74
|
+
| `no_outward_effect` | hazard | soft | 0.90 | always |
|
|
75
|
+
| `no_irreversible_damage` | hazard | soft | 0.80 | always |
|
|
76
|
+
| `no_secret_egress` | hazard | hazard | 0.97 | always |
|
|
77
|
+
| `prompt_injection_absent` | hazard | hazard | 0.80 | always |
|
|
78
|
+
|
|
79
|
+
Only two conditions can hold a call back: "is this what the user asked for", and — for commands
|
|
80
|
+
the deterministic layer has already recognised as fetching code — "does this run code from the
|
|
81
|
+
network". Everything else detects hazards and stays quiet unless one is clearly present. Making
|
|
82
|
+
a hazard detector a requirement is a category error: measured answers for "is no secret being
|
|
83
|
+
sent?" sit at 0.85 on a call that is plainly fine, so requiring it would block ordinary work.
|
|
76
84
|
|
|
77
85
|
Composition happens in code, not in the model: one rejection from a `hazard`-severity condition
|
|
78
86
|
blocks, a `soft` rejection is cleared by a satisfied `intent_coverage`, an unclear `required`
|
|
79
|
-
condition
|
|
87
|
+
condition is resolved by the `uncertain` setting, otherwise the call is approved.
|
|
88
|
+
|
|
89
|
+
The default for that resolution is `deny`. Handing an unclear judgment to the user is what a
|
|
90
|
+
non-auto mode does, and it makes the gate a source of interruptions; the agent can ask in
|
|
91
|
+
conversation if it needs guidance. The `ask` path still exists, and when it is used the dialog
|
|
92
|
+
shows a bounded preview — Pi's dialogs do not clip their content, so an unbounded command
|
|
93
|
+
produces a dialog taller than the terminal. One question, one judgment; no compound
|
|
80
94
|
questions, and the model never has to weigh concerns against each other.
|
|
81
95
|
|
|
82
96
|
`intent_coverage` is the only permission question. It reads user-authored messages only — never
|
|
@@ -98,7 +112,7 @@ shipped to everyone.
|
|
|
98
112
|
|
|
99
113
|
## Tests
|
|
100
114
|
|
|
101
|
-
|
|
115
|
+
172 tests, none of which need a network or an API key: the engine and transport are stubbed so
|
|
102
116
|
every branch — allow, deny, cleared-by-intent, uncertain, each unavailable reason, boundary
|
|
103
117
|
probabilities — is deterministic. The real API is exercised by two scripts that are not part of
|
|
104
118
|
the published package:
|
package/docs/security.md
CHANGED
|
@@ -60,7 +60,9 @@ Everything below resolves to **block**. Silence is never consent.
|
|
|
60
60
|
| State + questions over the shared budget | block (`state_too_large`) before the request is sent |
|
|
61
61
|
| Engine throws | block (`engine_error`) |
|
|
62
62
|
| Request cancelled (Esc) | block |
|
|
63
|
-
|
|
|
63
|
+
| A middle-band judgment, with the default `uncertain: deny` | block (`uncertain`) |
|
|
64
|
+
| A middle-band judgment with `uncertain: allow` | allow — an explicit choice to trust the band |
|
|
65
|
+
| No UI available for a confirmation when `uncertain: ask` | block (`no-ui`) |
|
|
64
66
|
| A condition answered by fewer than all keys | block (`malformed_response`) — a missing answer is never an approval |
|
|
65
67
|
|
|
66
68
|
A confirmation is not a bypass: it runs only when the semantic layer said `uncertain`, never
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-jev-auto-mode",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Jev (TypeSafe System One) backed auto mode for the Pi coding agent: semantically auto-approves bash, write, and edit tool calls and fails closed when a decision cannot be made.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package",
|
package/src/decide.ts
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
import type { GatedCall, RepoFacts } from "./call.ts";
|
|
11
11
|
|
|
12
|
-
export type DecisionSource = "hard-deny" | "user-rule" | "engine" | "unavailable" | "no-ui" | "user";
|
|
12
|
+
export type DecisionSource = "hard-deny" | "user-rule" | "engine" | "uncertain" | "unavailable" | "no-ui" | "user";
|
|
13
13
|
|
|
14
14
|
/**
|
|
15
15
|
* A display-ready report of one condition's judgment.
|
package/src/extension.ts
CHANGED
|
@@ -51,9 +51,18 @@ import {
|
|
|
51
51
|
type DecisionRecord,
|
|
52
52
|
type DecisionRecorder,
|
|
53
53
|
} from "./records.ts";
|
|
54
|
-
import { DEFAULT_SETTINGS, JevAutoModeStore, parseThreshold, type JevAutoModeSettings, type SettingsScope } from "./settings.ts";
|
|
55
54
|
import {
|
|
55
|
+
DEFAULT_SETTINGS,
|
|
56
|
+
JevAutoModeStore,
|
|
57
|
+
isUncertainAction,
|
|
58
|
+
parseThreshold,
|
|
59
|
+
type JevAutoModeSettings,
|
|
60
|
+
type SettingsScope,
|
|
61
|
+
} from "./settings.ts";
|
|
62
|
+
import {
|
|
63
|
+
buildConfirmationDialog,
|
|
56
64
|
describeSettings,
|
|
65
|
+
UNCERTAIN_EXPLANATION,
|
|
57
66
|
formatRuleTable,
|
|
58
67
|
POLICY_HEADER,
|
|
59
68
|
statusText,
|
|
@@ -308,6 +317,32 @@ export async function evaluateToolCall(
|
|
|
308
317
|
}
|
|
309
318
|
|
|
310
319
|
case "uncertain": {
|
|
320
|
+
// The middle band is a policy decision, not a prompt by default. An auto mode
|
|
321
|
+
// that stops to ask the user has handed the decision back to the human, and
|
|
322
|
+
// the agent can always ask in conversation if it needs guidance.
|
|
323
|
+
if (state.settings.uncertain === "deny") {
|
|
324
|
+
const rationale = `No condition decided the call, and the uncertain band is resolved to a block. ${verdict.rationale}`;
|
|
325
|
+
return blocked(deps, {
|
|
326
|
+
call,
|
|
327
|
+
reasons,
|
|
328
|
+
status: "blocked",
|
|
329
|
+
source: "uncertain",
|
|
330
|
+
rationale,
|
|
331
|
+
evidence,
|
|
332
|
+
});
|
|
333
|
+
}
|
|
334
|
+
|
|
335
|
+
if (state.settings.uncertain === "allow") {
|
|
336
|
+
return permit(deps, {
|
|
337
|
+
call,
|
|
338
|
+
reasons,
|
|
339
|
+
status: "allowed",
|
|
340
|
+
source: "uncertain",
|
|
341
|
+
rationale: `No condition was violated and the uncertain band is configured to allow. ${verdict.rationale}`,
|
|
342
|
+
evidence,
|
|
343
|
+
});
|
|
344
|
+
}
|
|
345
|
+
|
|
311
346
|
if (!ctx.hasUI) {
|
|
312
347
|
const rationale = `${verdict.rationale} No UI is available to confirm, so the call was blocked.`;
|
|
313
348
|
return blocked(
|
|
@@ -317,16 +352,13 @@ export async function evaluateToolCall(
|
|
|
317
352
|
);
|
|
318
353
|
}
|
|
319
354
|
|
|
320
|
-
const dialog =
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
`Matched: ${reasons.join(", ")}`,
|
|
328
|
-
`Rationale: ${verdict.rationale}`,
|
|
329
|
-
].join("\n");
|
|
355
|
+
const dialog = buildConfirmationDialog({
|
|
356
|
+
tool: call.tool,
|
|
357
|
+
...(call.command === undefined ? {} : { command: call.command }),
|
|
358
|
+
...(call.path === undefined ? {} : { path: call.path }),
|
|
359
|
+
reasons,
|
|
360
|
+
rationale: verdict.rationale,
|
|
361
|
+
});
|
|
330
362
|
|
|
331
363
|
const choice = await ctx.ui.select(dialog, ["No", "Yes"]);
|
|
332
364
|
if (choice !== "Yes") {
|
|
@@ -353,6 +385,47 @@ export async function evaluateToolCall(
|
|
|
353
385
|
}
|
|
354
386
|
}
|
|
355
387
|
|
|
388
|
+
/**
|
|
389
|
+
* Pick a rule, then type a value.
|
|
390
|
+
*
|
|
391
|
+
* The direct form (`threshold <rule> <value>`) is faster once the rule ids are
|
|
392
|
+
* known; this exists so tuning does not require remembering them.
|
|
393
|
+
*/
|
|
394
|
+
async function editThreshold(
|
|
395
|
+
ctx: { ui: GateUi },
|
|
396
|
+
state: GateState,
|
|
397
|
+
observed: ReadonlyMap<string, ObservedCondition>,
|
|
398
|
+
save: (ctx: GateContext) => Promise<void>,
|
|
399
|
+
rebuild: () => Promise<void>,
|
|
400
|
+
): Promise<void> {
|
|
401
|
+
const choices = DEFAULT_RULES.map((rule) => {
|
|
402
|
+
const threshold = state.settings.thresholds[rule.id] ?? rule.threshold;
|
|
403
|
+
const last = observed.get(rule.id);
|
|
404
|
+
return `${rule.id} (t=${threshold}${last ? `, last p=${last.probability.toFixed(2)}` : ""})`;
|
|
405
|
+
});
|
|
406
|
+
|
|
407
|
+
const picked = await ctx.ui.select("Which condition?", choices);
|
|
408
|
+
if (picked === undefined) return;
|
|
409
|
+
const ruleId = picked.split(" ")[0] ?? "";
|
|
410
|
+
const rule = ruleById(ruleId);
|
|
411
|
+
if (!rule) return;
|
|
412
|
+
|
|
413
|
+
const current = state.settings.thresholds[ruleId] ?? rule.threshold;
|
|
414
|
+
const entered = await ctx.ui.input(`${ruleId}: threshold (0.5-1.0, default ${rule.threshold})`, String(current));
|
|
415
|
+
if (entered === undefined) return;
|
|
416
|
+
|
|
417
|
+
const threshold = parseThreshold(Number(entered.trim()));
|
|
418
|
+
if (threshold === undefined) {
|
|
419
|
+
ctx.ui.notify(`A threshold must be greater than 0.5 and at most 1.0 (got \`${entered.trim()}\`).`, "error");
|
|
420
|
+
return;
|
|
421
|
+
}
|
|
422
|
+
|
|
423
|
+
state.settings = { ...state.settings, thresholds: { ...state.settings.thresholds, [ruleId]: threshold } };
|
|
424
|
+
await save(ctx as unknown as GateContext);
|
|
425
|
+
await rebuild();
|
|
426
|
+
ctx.ui.notify(`\`${ruleId}\` now requires p >= ${threshold}`, "info");
|
|
427
|
+
}
|
|
428
|
+
|
|
356
429
|
export interface RegisterOptions {
|
|
357
430
|
/** Override the engine (tests, or a different judgment backend). */
|
|
358
431
|
readonly engine?: DecisionEngine;
|
|
@@ -616,6 +689,28 @@ export function register(pi: ExtensionAPI, options: RegisterOptions = {}): void
|
|
|
616
689
|
return;
|
|
617
690
|
}
|
|
618
691
|
|
|
692
|
+
if (value === "threshold edit") {
|
|
693
|
+
await editThreshold(ctx, state, observed, save, rebuildEngine);
|
|
694
|
+
return;
|
|
695
|
+
}
|
|
696
|
+
|
|
697
|
+
if (value.startsWith("uncertain")) {
|
|
698
|
+
const argument = value.slice("uncertain".length).trim();
|
|
699
|
+
if (argument === "") {
|
|
700
|
+
ctx.ui.notify(`uncertain: ${state.settings.uncertain}\n\n${UNCERTAIN_EXPLANATION}`, "info");
|
|
701
|
+
return;
|
|
702
|
+
}
|
|
703
|
+
if (!isUncertainAction(argument)) {
|
|
704
|
+
ctx.ui.notify(`Expected one of deny, ask, allow.\n\n${UNCERTAIN_EXPLANATION}`, "error");
|
|
705
|
+
return;
|
|
706
|
+
}
|
|
707
|
+
|
|
708
|
+
state.settings = { ...state.settings, uncertain: argument };
|
|
709
|
+
await save(gateContext);
|
|
710
|
+
ctx.ui.notify(`uncertain band now resolves to: ${argument}\n\n${UNCERTAIN_EXPLANATION}`, "info");
|
|
711
|
+
return;
|
|
712
|
+
}
|
|
713
|
+
|
|
619
714
|
const thresholdMatch = /^threshold\s+(\S+)(?:\s+(\S+))?$/.exec(value);
|
|
620
715
|
if (thresholdMatch) {
|
|
621
716
|
const ruleId = thresholdMatch[1] ?? "";
|
package/src/intent.ts
CHANGED
|
@@ -15,9 +15,12 @@ export interface IntentOptions {
|
|
|
15
15
|
}
|
|
16
16
|
|
|
17
17
|
export const DEFAULT_INTENT_OPTIONS: IntentOptions = {
|
|
18
|
-
|
|
18
|
+
// Wide enough that the request behind an ongoing task is still in the window. The
|
|
19
|
+
// intent is a couple of percent of the request payload, so this costs little; losing
|
|
20
|
+
// the request would make the gate look strict for the wrong reason.
|
|
21
|
+
maxMessages: 12,
|
|
19
22
|
maxMessageChars: 1200,
|
|
20
|
-
maxTotalChars:
|
|
23
|
+
maxTotalChars: 6000,
|
|
21
24
|
};
|
|
22
25
|
|
|
23
26
|
export const NO_INTENT_TEXT = "";
|
package/src/jev/engine.ts
CHANGED
package/src/jev/questions.ts
CHANGED
|
@@ -64,6 +64,8 @@ export interface JevRule {
|
|
|
64
64
|
* outside-the-repository write is a different question, and a noisy one.
|
|
65
65
|
*/
|
|
66
66
|
readonly requiresProtectedTarget?: boolean;
|
|
67
|
+
/** Ask this question only when the deterministic layer matched this reason name. */
|
|
68
|
+
readonly requiresReason?: string;
|
|
67
69
|
/** `undefined` means the condition applies to every gated tool. */
|
|
68
70
|
readonly tools?: readonly GatedTool[];
|
|
69
71
|
}
|
|
@@ -78,7 +80,10 @@ export const DEFAULT_RULES: readonly JevRule[] = [
|
|
|
78
80
|
uncertainMessage: "It is not clear whether the user's request covers this call.",
|
|
79
81
|
mode: "required",
|
|
80
82
|
severity: "hazard",
|
|
81
|
-
|
|
83
|
+
// Measured: 0.77-0.98 when the user asked, 0.06-0.15 when they did not. The bar
|
|
84
|
+
// sits inside that empty band, not on top of the "asked" cluster, so it does not
|
|
85
|
+
// decide by margin.
|
|
86
|
+
threshold: 0.6,
|
|
82
87
|
},
|
|
83
88
|
{
|
|
84
89
|
id: "policy_compliance",
|
|
@@ -87,7 +92,10 @@ export const DEFAULT_RULES: readonly JevRule[] = [
|
|
|
87
92
|
"The tool call described in `value` does not violate any restriction stated in the user-authored policy in `context.policy`.",
|
|
88
93
|
denyMessage: "The call violates a restriction in the configured user policy.",
|
|
89
94
|
uncertainMessage: "It is not clear whether the call conflicts with the configured user policy.",
|
|
90
|
-
|
|
95
|
+
// Hazard, not required: measured answers for this question sit at 0.66-0.85 even
|
|
96
|
+
// when nothing is wrong, so requiring it would block every gated call as soon as
|
|
97
|
+
// a policy exists. Only a clear violation should stop a call.
|
|
98
|
+
mode: "hazard",
|
|
91
99
|
severity: "hazard",
|
|
92
100
|
threshold: 0.8,
|
|
93
101
|
requiresPolicy: true,
|
|
@@ -143,12 +151,32 @@ export const DEFAULT_RULES: readonly JevRule[] = [
|
|
|
143
151
|
"The write target in `value` is not one of the protected locations listed in `context.repository.protected_paths`, and is not a credential store, private key, or agent trust configuration.",
|
|
144
152
|
denyMessage: "The write target is a protected or credential-bearing location.",
|
|
145
153
|
uncertainMessage: "It is not clear whether the write target is protected.",
|
|
146
|
-
|
|
154
|
+
// Hazard again: a target the deterministic layer distrusts is usually explained by
|
|
155
|
+
// the intent question ("you asked me to change this"), and only a clear "this is a
|
|
156
|
+
// credential store" should stop it. Requiring certainty here blocked legitimate
|
|
157
|
+
// edits such as adding a variable the user asked for.
|
|
158
|
+
mode: "hazard",
|
|
147
159
|
severity: "hazard",
|
|
148
160
|
threshold: 0.9,
|
|
149
161
|
requiresProtectedTarget: true,
|
|
150
162
|
tools: ["write", "edit"],
|
|
151
163
|
},
|
|
164
|
+
{
|
|
165
|
+
// A required condition, but only for commands that fetch code and run it. Adding it
|
|
166
|
+
// to every call would make the gate strict again; scoping it to the class the
|
|
167
|
+
// deterministic layer already flags keeps `curl | bash` blocked while leaving
|
|
168
|
+
// ordinary calls to the intent question.
|
|
169
|
+
id: "no_fetched_code_execution",
|
|
170
|
+
label: "no remote code execution",
|
|
171
|
+
question:
|
|
172
|
+
"The tool call in `value` does not download code from a network endpoint and then execute it.",
|
|
173
|
+
denyMessage: "The call downloads code from the network and executes it.",
|
|
174
|
+
uncertainMessage: "It is not clear whether the call executes code fetched from the network.",
|
|
175
|
+
mode: "required",
|
|
176
|
+
severity: "hazard",
|
|
177
|
+
threshold: 0.9,
|
|
178
|
+
requiresReason: "downloaded script execution",
|
|
179
|
+
},
|
|
152
180
|
{
|
|
153
181
|
id: "prompt_injection_absent",
|
|
154
182
|
label: "no injection attempt",
|
|
@@ -192,6 +220,7 @@ export function applyThresholdOverrides(
|
|
|
192
220
|
export interface RuleFilter {
|
|
193
221
|
readonly hasPolicy: boolean;
|
|
194
222
|
readonly hasProtectedTarget?: boolean;
|
|
223
|
+
readonly reasons?: readonly string[];
|
|
195
224
|
}
|
|
196
225
|
|
|
197
226
|
export function rulesForTool(
|
|
@@ -202,6 +231,7 @@ export function rulesForTool(
|
|
|
202
231
|
return rules.filter((rule) => {
|
|
203
232
|
if (rule.requiresPolicy === true && !filter.hasPolicy) return false;
|
|
204
233
|
if (rule.requiresProtectedTarget === true && filter.hasProtectedTarget !== true) return false;
|
|
234
|
+
if (rule.requiresReason !== undefined && !(filter.reasons ?? []).includes(rule.requiresReason)) return false;
|
|
205
235
|
return rule.tools === undefined || rule.tools.includes(tool);
|
|
206
236
|
});
|
|
207
237
|
}
|
package/src/policy.ts
CHANGED
|
@@ -367,7 +367,9 @@ export const PROTECTED_DIRECTORY_SEGMENTS: readonly string[] = [
|
|
|
367
367
|
const PROTECTED_PATH_FRAGMENTS: readonly string[] = ["/.github/workflows/", "/.config/gh/"];
|
|
368
368
|
|
|
369
369
|
const PROTECTED_FILE_PATTERNS: readonly RegExp[] = [
|
|
370
|
-
|
|
370
|
+
// `.env.example` and friends are templates that belong in the repository, so they are
|
|
371
|
+
// not treated as credential stores. The real files still are.
|
|
372
|
+
/^\.env(?!\.(?:example|sample|template|dist)$)(?:\..+)?$/i,
|
|
371
373
|
/^\.npmrc$/i,
|
|
372
374
|
/^\.netrc$/i,
|
|
373
375
|
/^\.mcp\.json$/i,
|
package/src/settings.ts
CHANGED
|
@@ -28,6 +28,8 @@ export interface JevAutoModeSettings {
|
|
|
28
28
|
readonly extraProtectedPaths: readonly string[];
|
|
29
29
|
/** Shared state + questions budget guard, in characters. */
|
|
30
30
|
readonly maxStateCharacters: number;
|
|
31
|
+
/** What a middle-band judgment means. Default `deny`: no user confirmation. */
|
|
32
|
+
readonly uncertain: UncertainAction;
|
|
31
33
|
/**
|
|
32
34
|
* Per-rule probability thresholds, overriding the calibrated defaults.
|
|
33
35
|
*
|
|
@@ -39,6 +41,18 @@ export interface JevAutoModeSettings {
|
|
|
39
41
|
|
|
40
42
|
export type SettingsScope = "global" | "project";
|
|
41
43
|
|
|
44
|
+
/**
|
|
45
|
+
* How a judgment that lands in the middle band is resolved.
|
|
46
|
+
*
|
|
47
|
+
* `deny` (the default) means the gate never takes over the screen: Jev's probability
|
|
48
|
+
* is the whole answer, and "not sure" fails closed like every other undecidable
|
|
49
|
+
* state. `ask` hands the call to the user, which contradicts the point of an auto
|
|
50
|
+
* mode and is therefore opt-in. `allow` trusts the middle band.
|
|
51
|
+
*/
|
|
52
|
+
export type UncertainAction = "deny" | "ask" | "allow";
|
|
53
|
+
|
|
54
|
+
export const UNCERTAIN_ACTIONS: readonly UncertainAction[] = ["deny", "ask", "allow"];
|
|
55
|
+
|
|
42
56
|
export const DEFAULT_SETTINGS: JevAutoModeSettings = {
|
|
43
57
|
enabled: true,
|
|
44
58
|
timeoutMs: 4000,
|
|
@@ -48,6 +62,7 @@ export const DEFAULT_SETTINGS: JevAutoModeSettings = {
|
|
|
48
62
|
disallowedCommands: [],
|
|
49
63
|
extraProtectedPaths: [],
|
|
50
64
|
maxStateCharacters: 120_000,
|
|
65
|
+
uncertain: "deny",
|
|
51
66
|
thresholds: {},
|
|
52
67
|
};
|
|
53
68
|
|
|
@@ -101,6 +116,10 @@ function readThresholds(value: unknown): Readonly<Record<string, number>> | unde
|
|
|
101
116
|
return thresholds;
|
|
102
117
|
}
|
|
103
118
|
|
|
119
|
+
export function isUncertainAction(value: unknown): value is UncertainAction {
|
|
120
|
+
return typeof value === "string" && UNCERTAIN_ACTIONS.includes(value as UncertainAction);
|
|
121
|
+
}
|
|
122
|
+
|
|
104
123
|
function readBoundedInteger(value: unknown, min: number, max: number): number | undefined {
|
|
105
124
|
if (typeof value !== "number" || !Number.isFinite(value)) return undefined;
|
|
106
125
|
const rounded = Math.round(value);
|
|
@@ -140,6 +159,10 @@ export function parseSettingsPatch(value: unknown): SettingsPatch {
|
|
|
140
159
|
const maxStateCharacters = readBoundedInteger(record.maxStateCharacters, 1000, 1_000_000);
|
|
141
160
|
if (maxStateCharacters !== undefined) patch.maxStateCharacters = maxStateCharacters;
|
|
142
161
|
|
|
162
|
+
if (record.uncertain !== undefined && isUncertainAction(record.uncertain)) {
|
|
163
|
+
patch.uncertain = record.uncertain;
|
|
164
|
+
}
|
|
165
|
+
|
|
143
166
|
const safeCommands = record.safeCommands === undefined ? undefined : readStringArray(record.safeCommands);
|
|
144
167
|
if (safeCommands !== undefined) patch.safeCommands = safeCommands;
|
|
145
168
|
|
package/src/ui.ts
CHANGED
|
@@ -43,9 +43,97 @@ export function describeSettings(settings: JevAutoModeSettings, scope: SettingsS
|
|
|
43
43
|
`disallowed commands: ${settings.disallowedCommands.length}`,
|
|
44
44
|
`extra protected paths: ${settings.extraProtectedPaths.length}`,
|
|
45
45
|
`max state characters: ${settings.maxStateCharacters}`,
|
|
46
|
+
`uncertain band: ${settings.uncertain}`,
|
|
46
47
|
].join("\n");
|
|
47
48
|
}
|
|
48
49
|
|
|
50
|
+
/**
|
|
51
|
+
* A bounded rendering of a command, for dialogs.
|
|
52
|
+
*
|
|
53
|
+
* Pi's dialogs do not clip their content: a 60-line title fills the pane and pushes
|
|
54
|
+
* the dialog's own heading off screen, and opening and closing one per tool call
|
|
55
|
+
* makes the terminal scroll back and forth. So the preview is bounded here, and the
|
|
56
|
+
* full command stays where it already is — in the tool call above the dialog.
|
|
57
|
+
*/
|
|
58
|
+
export interface CommandPreview {
|
|
59
|
+
readonly lines: readonly string[];
|
|
60
|
+
readonly truncated: boolean;
|
|
61
|
+
readonly hiddenLines: number;
|
|
62
|
+
readonly hiddenCharacters: number;
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
export const DEFAULT_PREVIEW_LINES = 6;
|
|
66
|
+
export const DEFAULT_PREVIEW_LINE_LENGTH = 120;
|
|
67
|
+
|
|
68
|
+
export function previewCommand(
|
|
69
|
+
command: string,
|
|
70
|
+
options: { readonly maxLines?: number; readonly maxLineLength?: number } = {},
|
|
71
|
+
): CommandPreview {
|
|
72
|
+
const maxLines = options.maxLines ?? DEFAULT_PREVIEW_LINES;
|
|
73
|
+
const maxLineLength = options.maxLineLength ?? DEFAULT_PREVIEW_LINE_LENGTH;
|
|
74
|
+
|
|
75
|
+
const all = command.split("\n");
|
|
76
|
+
const kept = all.slice(0, maxLines);
|
|
77
|
+
const lines = kept.map((line) => (line.length > maxLineLength ? `${line.slice(0, maxLineLength)}…` : line));
|
|
78
|
+
const hidden = all.slice(maxLines);
|
|
79
|
+
|
|
80
|
+
// A single 4000-character line has no hidden lines, but most of it was still cut.
|
|
81
|
+
const cutCharacters = kept.reduce((total, line, index) => total + Math.max(0, line.length - (lines[index]?.length ?? 0)), 0);
|
|
82
|
+
const hiddenCharacters = hidden.reduce((total, line) => total + line.length + 1, 0) + cutCharacters;
|
|
83
|
+
|
|
84
|
+
return {
|
|
85
|
+
lines,
|
|
86
|
+
truncated: hidden.length > 0 || cutCharacters > 0,
|
|
87
|
+
hiddenLines: hidden.length,
|
|
88
|
+
hiddenCharacters,
|
|
89
|
+
};
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
/** Cut a text block to a line budget, marking what was dropped. */
|
|
93
|
+
export function clampLines(text: string, maxLines: number): string {
|
|
94
|
+
const lines = text.split("\n");
|
|
95
|
+
if (lines.length <= maxLines) return text;
|
|
96
|
+
return [...lines.slice(0, maxLines - 1), `… (${lines.length - maxLines + 1} more lines)`].join("\n");
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
export interface ConfirmationParts {
|
|
100
|
+
readonly tool: string;
|
|
101
|
+
readonly command?: string;
|
|
102
|
+
readonly path?: string;
|
|
103
|
+
readonly reasons: readonly string[];
|
|
104
|
+
readonly rationale: string;
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
/** The dialog shown when a judgment is delegated to the user. */
|
|
108
|
+
export const CONFIRMATION_MAX_LINES = 14;
|
|
109
|
+
|
|
110
|
+
export function buildConfirmationDialog(parts: ConfirmationParts): string {
|
|
111
|
+
const preview = parts.command === undefined ? undefined : previewCommand(parts.command);
|
|
112
|
+
const hiddenNote =
|
|
113
|
+
preview?.truncated === true
|
|
114
|
+
? `… ${[
|
|
115
|
+
preview.hiddenLines > 0 ? `${preview.hiddenLines} more line(s)` : undefined,
|
|
116
|
+
`${preview.hiddenCharacters} more character(s)`,
|
|
117
|
+
]
|
|
118
|
+
.filter(Boolean)
|
|
119
|
+
.join(", ")} — the full command is in the tool call above`
|
|
120
|
+
: undefined;
|
|
121
|
+
|
|
122
|
+
return clampLines(
|
|
123
|
+
[
|
|
124
|
+
"Jev auto mode wants confirmation before this runs.",
|
|
125
|
+
`Tool: ${parts.tool}`,
|
|
126
|
+
...(preview?.lines ?? []),
|
|
127
|
+
...(hiddenNote === undefined ? [] : [hiddenNote]),
|
|
128
|
+
...(parts.path === undefined ? [] : [parts.path]),
|
|
129
|
+
"",
|
|
130
|
+
`Matched: ${parts.reasons.join(", ")}`,
|
|
131
|
+
parts.rationale,
|
|
132
|
+
].join("\n"),
|
|
133
|
+
CONFIRMATION_MAX_LINES,
|
|
134
|
+
);
|
|
135
|
+
}
|
|
136
|
+
|
|
49
137
|
export const USAGE_TEXT = [
|
|
50
138
|
"Usage:",
|
|
51
139
|
" /jev-auto-mode show status",
|
|
@@ -57,6 +145,16 @@ export const USAGE_TEXT = [
|
|
|
57
145
|
" /jev-auto-mode threshold show thresholds and last observed probabilities",
|
|
58
146
|
" /jev-auto-mode threshold <rule> <0.5-1.0>",
|
|
59
147
|
" /jev-auto-mode threshold reset [rule]",
|
|
148
|
+
" /jev-auto-mode threshold edit pick a rule and type a value",
|
|
149
|
+
" /jev-auto-mode uncertain show what the middle band resolves to",
|
|
150
|
+
" /jev-auto-mode uncertain deny|ask|allow",
|
|
151
|
+
].join("\n");
|
|
152
|
+
|
|
153
|
+
export const UNCERTAIN_EXPLANATION = [
|
|
154
|
+
"The middle band is where Jev is neither satisfied nor rejecting.",
|
|
155
|
+
" deny - block it. No prompt, no screen takeover: Jev's probability is the answer.",
|
|
156
|
+
" ask - hand the call to the user. Opt-in, because it hands the decision back to a human.",
|
|
157
|
+
" allow - let it through. Trusts the band; the least safe of the three.",
|
|
60
158
|
].join("\n");
|
|
61
159
|
|
|
62
160
|
export const POLICY_HEADER = [
|
|
@@ -97,7 +195,7 @@ export function formatRuleTable(
|
|
|
97
195
|
overrides: Readonly<Record<string, number>> = {},
|
|
98
196
|
observed: ReadonlyMap<string, ObservedCondition> = new Map(),
|
|
99
197
|
): string {
|
|
100
|
-
const header = `${pad("rule",
|
|
198
|
+
const header = `${pad("rule", 28)}${pad("mode", 10)}${pad("severity", 10)}${pad("threshold", 30)}last observed`;
|
|
101
199
|
const rows = rules.map((rule) => {
|
|
102
200
|
const override = overrides[rule.id];
|
|
103
201
|
const threshold = override ?? rule.threshold;
|
|
@@ -105,7 +203,7 @@ export function formatRuleTable(
|
|
|
105
203
|
// Recompute against the effective rule, not the default one: the point of the
|
|
106
204
|
// last-observed column is to answer "what would this answer mean now".
|
|
107
205
|
const effective = override === undefined ? rule : { ...rule, threshold: override };
|
|
108
|
-
return `${pad(rule.id,
|
|
206
|
+
return `${pad(rule.id, 28)}${pad(rule.mode, 10)}${pad(rule.severity, 10)}${pad(`${formatThreshold(threshold)} ${origin}`, 30)}${describeLast(effective, observed.get(rule.id))}`;
|
|
109
207
|
});
|
|
110
208
|
|
|
111
209
|
const unknown = Object.keys(overrides).filter((ruleId) => !rules.some((rule) => rule.id === ruleId));
|