pi-jev-auto-mode 0.2.0 → 0.3.0

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.3.0
+
+The 0.2.0 default resolved the middle band as a block, but two conditions were still
+`required`, which made the gate strict for structurally wrong reasons rather than measured
+ones. Corrected:
+
+- `intent_coverage` 0.80 → **0.60**. Measured answers are 0.77–0.98 when the user asked and
+  0.06–0.15 when they did not, so 0.80 sat on top of the "asked" cluster instead of inside the
+  empty band between the two. The middle band is now (0.40, 0.60).
+- `policy_compliance` required → **hazard**. It measured 0.66–0.85 on calls where nothing was
+  wrong, so as a requirement it blocked every gated call the moment a policy was configured.
+  Now only a clear violation stops a call.
+- `path_not_protected` required → **hazard**. An unclear answer no longer blocks on its own; the
+  user's request decides. A target the model clearly identifies as a credential store still
+  blocks (`.env` measured p = 0.02, `~/.ssh` p = 0.03).
+- New `no_fetched_code_execution`, required, asked only for commands the deterministic layer
+  already recognised as downloaded-script execution. `curl | bash` stays blocked (p = 0.02)
+  without making every other call strict.
+- `.env.example`, `.env.sample`, `.env.template`, and `.env.dist` are no longer treated as
+  credential stores: templates belong in the repository.
+- The intent window widened from 8 messages / 4000 characters to 12 / 6000, so an ongoing task
+  does not lose the request that justifies it and look unrequested as a result.
+
 ## 0.2.0
 
 - **The middle band no longer asks the user by default.** An auto mode that stops to ask has

package/docs/calibration.md

@@ -124,6 +124,26 @@ transcript. The fixture set gained `post a private key, asked for`, which verifi
 the whole design rests on: with the user explicitly asking for it, `no_secret_egress` still
 rejects (p = 0.01) and the call is blocked.
 
+## Third revision: strictness for the wrong reason
+
+With the middle band resolving to a block, the two `required` conditions left in the set turned
+out to be doing the blocking, and not because the calls were risky:
+
+- `intent_coverage` at 0.80 was above the entire "asked" cluster. The measured gap is
+  **0.15 → 0.77**: nothing the model answered landed between those values, so any threshold in
+  that range separates perfectly and 0.80 was simply the strict end of nothing. Now 0.60, with
+  the middle band at (0.40, 0.60).
+- `policy_compliance` answered 0.66–0.85 even for calls that were plainly fine. As a requirement
+  that blocks, so **configuring a policy disabled the gate entirely**. It is a hazard detector
+  now: only a clear violation stops a call.
+- `path_not_protected` had the same shape: the target was escalated *because* the deterministic
+  layer distrusts it, so asking the model to certify it as safe and blocking when it hedges put
+  the burden in the wrong place. The intent question decides; a clear credential-store answer
+  still blocks (`.env` p = 0.02, `~/.ssh` p = 0.03).
+
+The lesson generalises: a threshold should be chosen so that no measured answer sits near it.
+Both mistakes here came from picking the strict end of a band instead of the middle of a gap.
+
 ## Tuning without the script
 
 The same numbers arrive in every session. Expand a decision record in the transcript to see

package/docs/design.md

@@ -64,16 +64,23 @@ participates:
 - `severity`: `hazard` (a rejection always blocks) or `soft` (a rejection is cleared when the
   user's own request covers the call)
 
-| rule id | mode | severity | threshold |
-|---|---|---|---|
-| `intent_coverage` | required | hazard | 0.80 |
-| `policy_compliance` (only when a policy exists) | required | hazard | 0.80 |
-| `path_not_protected` (only when the deterministic layer flagged the target) | required | hazard | 0.90 |
-| `local_scope` | hazard | soft | 0.90 |
-| `no_outward_effect` | hazard | soft | 0.90 |
-| `no_irreversible_damage` | hazard | soft | 0.80 |
-| `no_secret_egress` | hazard | hazard | 0.97 |
-| `prompt_injection_absent` | hazard | hazard | 0.80 |
+| rule id | mode | severity | threshold | asked when |
+|---|---|---|---|---|
+| `intent_coverage` | required | hazard | 0.60 | always |
+| `no_fetched_code_execution` | required | hazard | 0.90 | the command downloads code and runs it |
+| `policy_compliance` | hazard | hazard | 0.80 | a policy is configured |
+| `path_not_protected` | hazard | hazard | 0.90 | the deterministic layer flagged the target |
+| `local_scope` | hazard | soft | 0.90 | always |
+| `no_outward_effect` | hazard | soft | 0.90 | always |
+| `no_irreversible_damage` | hazard | soft | 0.80 | always |
+| `no_secret_egress` | hazard | hazard | 0.97 | always |
+| `prompt_injection_absent` | hazard | hazard | 0.80 | always |
+
+Only two conditions can hold a call back: "is this what the user asked for", and — for commands
+the deterministic layer has already recognised as fetching code — "does this run code from the
+network". Everything else detects hazards and stays quiet unless one is clearly present. Making
+a hazard detector a requirement is a category error: measured answers for "is no secret being
+sent?" sit at 0.85 on a call that is plainly fine, so requiring it would block ordinary work.
 
 Composition happens in code, not in the model: one rejection from a `hazard`-severity condition
 blocks, a `soft` rejection is cleared by a satisfied `intent_coverage`, an unclear `required`
@@ -105,7 +112,7 @@ shipped to everyone.
 
 ## Tests
 
-171 tests, none of which need a network or an API key: the engine and transport are stubbed so
+172 tests, none of which need a network or an API key: the engine and transport are stubbed so
 every branch — allow, deny, cleared-by-intent, uncertain, each unavailable reason, boundary
 probabilities — is deterministic. The real API is exercised by two scripts that are not part of
 the published package:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-jev-auto-mode",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Jev (TypeSafe System One) backed auto mode for the Pi coding agent: semantically auto-approves bash, write, and edit tool calls and fails closed when a decision cannot be made.",
   "keywords": [
     "pi-package",

package/src/intent.ts

@@ -15,9 +15,12 @@ export interface IntentOptions {
 }
 
 export const DEFAULT_INTENT_OPTIONS: IntentOptions = {
-  maxMessages: 8,
+  // Wide enough that the request behind an ongoing task is still in the window. The
+  // intent is a couple of percent of the request payload, so this costs little; losing
+  // the request would make the gate look strict for the wrong reason.
+  maxMessages: 12,
   maxMessageChars: 1200,
-  maxTotalChars: 4000,
+  maxTotalChars: 6000,
 };
 
 export const NO_INTENT_TEXT = "";

package/src/jev/engine.ts

@@ -68,6 +68,7 @@ export function createJevEngine(options: JevEngineOptions): DecisionEngine {
         rulesForTool(input.call.tool, rules, {
           hasPolicy,
           hasProtectedTarget: input.call.protectedReason !== undefined,
+          reasons: input.reasons,
         }),
         options.thresholds,
       );

package/src/jev/questions.ts

@@ -64,6 +64,8 @@ export interface JevRule {
    * outside-the-repository write is a different question, and a noisy one.
    */
   readonly requiresProtectedTarget?: boolean;
+  /** Ask this question only when the deterministic layer matched this reason name. */
+  readonly requiresReason?: string;
   /** `undefined` means the condition applies to every gated tool. */
   readonly tools?: readonly GatedTool[];
 }
@@ -78,7 +80,10 @@ export const DEFAULT_RULES: readonly JevRule[] = [
     uncertainMessage: "It is not clear whether the user's request covers this call.",
     mode: "required",
     severity: "hazard",
-    threshold: 0.8,
+    // Measured: 0.77-0.98 when the user asked, 0.06-0.15 when they did not. The bar
+    // sits inside that empty band, not on top of the "asked" cluster, so it does not
+    // decide by margin.
+    threshold: 0.6,
   },
   {
     id: "policy_compliance",
@@ -87,7 +92,10 @@ export const DEFAULT_RULES: readonly JevRule[] = [
       "The tool call described in `value` does not violate any restriction stated in the user-authored policy in `context.policy`.",
     denyMessage: "The call violates a restriction in the configured user policy.",
     uncertainMessage: "It is not clear whether the call conflicts with the configured user policy.",
-    mode: "required",
+    // Hazard, not required: measured answers for this question sit at 0.66-0.85 even
+    // when nothing is wrong, so requiring it would block every gated call as soon as
+    // a policy exists. Only a clear violation should stop a call.
+    mode: "hazard",
     severity: "hazard",
     threshold: 0.8,
     requiresPolicy: true,
@@ -143,12 +151,32 @@ export const DEFAULT_RULES: readonly JevRule[] = [
       "The write target in `value` is not one of the protected locations listed in `context.repository.protected_paths`, and is not a credential store, private key, or agent trust configuration.",
     denyMessage: "The write target is a protected or credential-bearing location.",
     uncertainMessage: "It is not clear whether the write target is protected.",
-    mode: "required",
+    // Hazard again: a target the deterministic layer distrusts is usually explained by
+    // the intent question ("you asked me to change this"), and only a clear "this is a
+    // credential store" should stop it. Requiring certainty here blocked legitimate
+    // edits such as adding a variable the user asked for.
+    mode: "hazard",
     severity: "hazard",
     threshold: 0.9,
     requiresProtectedTarget: true,
     tools: ["write", "edit"],
   },
+  {
+    // A required condition, but only for commands that fetch code and run it. Adding it
+    // to every call would make the gate strict again; scoping it to the class the
+    // deterministic layer already flags keeps `curl | bash` blocked while leaving
+    // ordinary calls to the intent question.
+    id: "no_fetched_code_execution",
+    label: "no remote code execution",
+    question:
+      "The tool call in `value` does not download code from a network endpoint and then execute it.",
+    denyMessage: "The call downloads code from the network and executes it.",
+    uncertainMessage: "It is not clear whether the call executes code fetched from the network.",
+    mode: "required",
+    severity: "hazard",
+    threshold: 0.9,
+    requiresReason: "downloaded script execution",
+  },
   {
     id: "prompt_injection_absent",
     label: "no injection attempt",
@@ -192,6 +220,7 @@ export function applyThresholdOverrides(
 export interface RuleFilter {
   readonly hasPolicy: boolean;
   readonly hasProtectedTarget?: boolean;
+  readonly reasons?: readonly string[];
 }
 
 export function rulesForTool(
@@ -202,6 +231,7 @@ export function rulesForTool(
   return rules.filter((rule) => {
     if (rule.requiresPolicy === true && !filter.hasPolicy) return false;
     if (rule.requiresProtectedTarget === true && filter.hasProtectedTarget !== true) return false;
+    if (rule.requiresReason !== undefined && !(filter.reasons ?? []).includes(rule.requiresReason)) return false;
     return rule.tools === undefined || rule.tools.includes(tool);
   });
 }

package/src/policy.ts

@@ -367,7 +367,9 @@ export const PROTECTED_DIRECTORY_SEGMENTS: readonly string[] = [
 const PROTECTED_PATH_FRAGMENTS: readonly string[] = ["/.github/workflows/", "/.config/gh/"];
 
 const PROTECTED_FILE_PATTERNS: readonly RegExp[] = [
-  /^\.env(?:\..+)?$/i,
+  // `.env.example` and friends are templates that belong in the repository, so they are
+  // not treated as credential stores. The real files still are.
+  /^\.env(?!\.(?:example|sample|template|dist)$)(?:\..+)?$/i,
   /^\.npmrc$/i,
   /^\.netrc$/i,
   /^\.mcp\.json$/i,

package/src/ui.ts

@@ -195,7 +195,7 @@ export function formatRuleTable(
   overrides: Readonly<Record<string, number>> = {},
   observed: ReadonlyMap<string, ObservedCondition> = new Map(),
 ): string {
-  const header = `${pad("rule", 24)}${pad("mode", 10)}${pad("severity", 10)}${pad("threshold", 30)}last observed`;
+  const header = `${pad("rule", 28)}${pad("mode", 10)}${pad("severity", 10)}${pad("threshold", 30)}last observed`;
   const rows = rules.map((rule) => {
     const override = overrides[rule.id];
     const threshold = override ?? rule.threshold;
@@ -203,7 +203,7 @@ export function formatRuleTable(
     // Recompute against the effective rule, not the default one: the point of the
     // last-observed column is to answer "what would this answer mean now".
     const effective = override === undefined ? rule : { ...rule, threshold: override };
-    return `${pad(rule.id, 24)}${pad(rule.mode, 10)}${pad(rule.severity, 10)}${pad(`${formatThreshold(threshold)} ${origin}`, 30)}${describeLast(effective, observed.get(rule.id))}`;
+    return `${pad(rule.id, 28)}${pad(rule.mode, 10)}${pad(rule.severity, 10)}${pad(`${formatThreshold(threshold)} ${origin}`, 30)}${describeLast(effective, observed.get(rule.id))}`;
   });
 
   const unknown = Object.keys(overrides).filter((ruleId) => !rules.some((rule) => rule.id === ruleId));