@gotgenes/pi-permission-model-judge 1.0.2 → 1.1.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## [1.1.0](https://github.com/gotgenes/pi-packages/compare/pi-permission-model-judge-v1.0.2...pi-permission-model-judge-v1.1.0) (2026-07-21)
+
+
+### Features
+
+* **pi-permission-model-judge:** distinguish model-call defer reasons in reviewPath ([9098f46](https://github.com/gotgenes/pi-packages/commit/9098f465d64dfee3994132bd87a71ad40152b259))
+* **pi-permission-model-judge:** record the decision trail to the permission review log ([e3964ae](https://github.com/gotgenes/pi-packages/commit/e3964ae6ca443d043db4e735ad69df90bac11c0d))
+* **pi-permission-model-judge:** return the matched typo pattern ([4db29fb](https://github.com/gotgenes/pi-packages/commit/4db29fbf098c8d3c853539b408316ee2a8fffe54))
+
+
+### Documentation
+
+* **pi-permission-model-judge:** document the decision trail ([335a7b5](https://github.com/gotgenes/pi-packages/commit/335a7b5b0f40650760d8597f511f32dc2116bf81))
+
 ## [1.0.2](https://github.com/gotgenes/pi-packages/compare/pi-permission-model-judge-v1.0.1...pi-permission-model-judge-v1.0.2) (2026-07-21)
 
 

package/README.md

@@ -23,6 +23,29 @@ The reviewer runs a short, cheap decision on each ask and defers at the first mi
 It is fail-safe by construction: a missing model, invalid config, model timeout, unparseable reply, or an unsure verdict all resolve to `defer`.
 Deferring means the ask falls through to the normal permission prompt — this extension only ever *removes* a hand-denial, never grants access (it emits no `allow`).
 
+## What it records
+
+Every review the link performs leaves a trail in pi-permission-system's shared review log (`~/.pi/agent/extensions/pi-permission-system/logs/pi-permission-system-permission-review.jsonl`), so you can answer "did the judge run, did it reach the model, and why did it defer?"
+without guesswork.
+
+Once an ask matches a `typoPattern` — the case that *should* reach the model — the link writes one `model_judge.decision` entry recording the outcome:
+
+| Field            | Meaning                                                                                                                             |
+| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| `requestId`      | Joins to the `permission_request.*` entries for the same ask.                                                                       |
+| `path`           | The candidate path reviewed.                                                                                                        |
+| `matchedPattern` | The `typoPatterns` entry (as you wrote it) that matched.                                                                            |
+| `modelCalled`    | `false` when the model or its auth did not resolve.                                                                                 |
+| `modelId`        | `<provider>/<model>`.                                                                                                               |
+| `latencyMs`      | Model-call wall-clock in ms, or `null` when no call was made.                                                                       |
+| `verdict`        | `"deny"` or `"defer"`.                                                                                                              |
+| `deferReason`    | `null` on a deny, else one of `model-unresolved` / `auth-failed` / `parse-failed` / `non-deny-verdict` / `timeout` / `call-failed`. |
+
+Cheaper events go to pi-permission-system's **debug** log, and only when its `debugLog` toggle is on: `model_judge.short_circuit` (a `no-path` or `pattern-miss` defer) and `model_judge.model_reply` (the model's raw reply text).
+A non-`external_directory` ask is not logged — it is not this link's concern.
+
+Because every pattern-matched ask leaves a positive record, a misconfiguration that silently defers every path (an auth failure, an unresolved model) shows up as a run of `deferReason` entries rather than an empty log.
+
 ## Install
 
 ```bash

package/docs/configuration.md

@@ -44,7 +44,7 @@ The link decides nothing until you name it in `authorizerChain` (opt-in activati
 ### `provider` and `model`
 
 The reviewer resolves `provider` / `model` against the session's model registry at review time.
-If the model does not resolve (wrong id, no credentials), the reviewer logs a warning and defers — it never blocks an ask because the judge is misconfigured.
+If the model does not resolve (wrong id, no credentials), the reviewer records a `model-unresolved` decision entry and defers — it never blocks an ask because the judge is misconfigured.
 
 ### `instructions`
 
@@ -56,7 +56,7 @@ The reviewer expects the model to answer with strict JSON — `{"verdict":"deny"
 
 Each string is compiled with `new RegExp(pattern)` (no flags) and tested against the candidate path.
 Only a path that matches at least one pattern is sent to the model — this is the cost gate that keeps the reviewer from calling the model on every ask.
-An invalid regular expression is skipped with a warning.
+An invalid regular expression is skipped and recorded in the debug log (`model_judge.invalid_patterns`).
 An empty list (the default) matches nothing, so the reviewer defers everything.
 
 Two typo shapes are worth featuring, each as its own pattern:
@@ -92,3 +92,27 @@ Every uncertain outcome defers, which sends the ask to the normal permission pro
 - an unresolved model, a timeout, an unparseable reply, or an unsure verdict.
 
 Deferring never grants access — this extension emits no `allow` verdict.
+
+## The decision trail
+
+The reviewer records what it did to pi-permission-system's shared logs, keyed by `requestId` so an entry joins to the `permission_request.*` lines for the same ask.
+This closes the observability gap that let an auth failure defer every path undetected: a silent defer is now a recorded defer with a reason.
+
+The **review** log (`~/.pi/agent/extensions/pi-permission-system/logs/pi-permission-system-permission-review.jsonl`, on by default via pi-permission-system's `permissionReviewLog`) carries one `model_judge.decision` entry per **pattern-matched** ask — the case that should reach the model:
+
+| Field            | Meaning                                                                                                                      |
+| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| `requestId`      | The ask id, shared with the `permission_request.*` entries.                                                                  |
+| `surface`        | Always `external_directory` (the only reviewed surface).                                                                     |
+| `path`           | The candidate path reviewed.                                                                                                 |
+| `matchedPattern` | The `typoPatterns` entry, as written, that matched.                                                                          |
+| `modelCalled`    | `false` for a `model-unresolved` / `auth-failed` defer.                                                                      |
+| `modelId`        | `<provider>/<model>`.                                                                                                        |
+| `latencyMs`      | Model-call wall-clock in milliseconds, or `null` when no call was made.                                                      |
+| `verdict`        | `"deny"` or `"defer"`.                                                                                                       |
+| `deferReason`    | `null` on a deny, else `model-unresolved` / `auth-failed` / `parse-failed` / `non-deny-verdict` / `timeout` / `call-failed`. |
+
+The **debug** log (same directory, `pi-permission-system-debug.jsonl`, only when pi-permission-system's `debugLog` is on) carries the verbose and cheap-path detail: `model_judge.short_circuit` (a `no-path` or `pattern-miss` defer before the model stage), `model_judge.model_reply` (the model's raw reply text), and `model_judge.invalid_patterns` (skipped `typoPatterns`).
+A non-`external_directory` ask is not logged at all.
+
+To diagnose "the judge defers everything," read the review log for `model_judge.decision` entries and inspect `deferReason`: an empty result means no ask ever matched a pattern, `auth-failed` / `model-unresolved` means it is misconfigured, and `non-deny-verdict` means the model saw the path and chose not to deny.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@gotgenes/pi-permission-model-judge",
-  "version": "1.0.2",
+  "version": "1.1.0",
   "description": "Deny-first typo-path model judge — a pi-permission-system Authorizer chain link",
   "type": "module",
   "imports": {
@@ -54,13 +54,13 @@
   "peerDependencies": {
     "@earendil-works/pi-ai": ">=0.79.0",
     "@earendil-works/pi-coding-agent": ">=0.79.0",
-    "@gotgenes/pi-permission-system": ">=20.9.0"
+    "@gotgenes/pi-permission-system": ">=20.10.0"
   },
   "devDependencies": {
     "@biomejs/biome": "^2.4.16",
     "@earendil-works/pi-ai": "0.79.1",
     "@earendil-works/pi-coding-agent": "0.79.1",
-    "@gotgenes/pi-permission-system": "20.9.0",
+    "@gotgenes/pi-permission-system": "20.10.0",
     "@types/node": "^22.15.3",
     "rumdl": "^0.2.10",
     "typescript": "^6.0.3",

package/src/extension.ts

@@ -67,7 +67,6 @@ export function createModelJudgeExtension(
       getConfig: () => config,
       getRegistry: () => registry,
       complete,
-      warn,
     });
     dispose = service.registerAuthorizer(LINK_NAME, authorize);
   }

package/src/model-review.ts

@@ -61,18 +61,46 @@ export interface ReviewPathInputs {
 }
 
 /**
- * Review one candidate path with the model and return its verdict.
+ * Why a model call defers, distinct enough to diagnose from the decision trail:
+ * the reply parsed but was not JSON (`parse-failed`), was valid JSON but not a
+ * `deny` (`non-deny-verdict`), the call was aborted at `timeoutMs` (`timeout`),
+ * or `complete` threw for any other reason (`call-failed` — the honest superset
+ * that catches, e.g., a 401 slipping past pre-call auth resolution).
+ */
+export type ModelCallDeferReason =
+  | "parse-failed"
+  | "non-deny-verdict"
+  | "timeout"
+  | "call-failed";
+
+/**
+ * The full result of a model review: the verdict plus the observability the
+ * decision trail records. `deferReason` is set iff the verdict is `defer` and a
+ * model call was made; `rawReply` carries the assistant text when a reply
+ * arrived (absent on a timeout/throw before any reply).
+ */
+export interface ReviewOutcome {
+  verdict: AuthorizerVerdict;
+  deferReason?: ModelCallDeferReason;
+  latencyMs: number;
+  rawReply?: string;
+}
+
+/**
+ * Review one candidate path with the model and return the structured outcome.
  *
  * The call is aborted after `config.timeoutMs`; an abort, a rejection, or any
- * non-`deny` reply yields `defer`.
+ * non-`deny` reply yields `defer` (more prompting, never less) with the reason
+ * annotated so the caller can record why.
  */
 export async function reviewPath(
   inputs: ReviewPathInputs,
-): Promise<AuthorizerVerdict> {
+): Promise<ReviewOutcome> {
   const controller = new AbortController();
   const timer = setTimeout(() => {
     controller.abort();
   }, inputs.config.timeoutMs);
+  const startedAt = Date.now();
   try {
     const context: Context = {
       systemPrompt: inputs.config.instructions,
@@ -89,9 +117,13 @@ export async function reviewPath(
       apiKey: inputs.apiKey,
       headers: inputs.headers,
     });
-    return parseVerdict(reply);
+    return parseOutcome(extractText(reply).trim(), Date.now() - startedAt);
   } catch {
-    return { kind: "defer" };
+    return {
+      verdict: { kind: "defer" },
+      deferReason: controller.signal.aborted ? "timeout" : "call-failed",
+      latencyMs: Date.now() - startedAt,
+    };
   } finally {
     clearTimeout(timer);
   }
@@ -109,23 +141,35 @@ function renderReviewPrompt(path: string): string {
   ].join("\n");
 }
 
-/** Map the assistant reply to a verdict; anything but a clean `deny` defers. */
-function parseVerdict(reply: AssistantMessage): AuthorizerVerdict {
-  const text = extractText(reply).trim();
+/**
+ * Map the assistant reply text to an outcome; anything but a clean `deny`
+ * defers with the reason that distinguishes it.
+ */
+function parseOutcome(rawReply: string, latencyMs: number): ReviewOutcome {
   let parsed: unknown;
   try {
-    parsed = JSON.parse(text);
+    parsed = JSON.parse(rawReply);
   } catch {
-    return { kind: "defer" };
+    return {
+      verdict: { kind: "defer" },
+      deferReason: "parse-failed",
+      latencyMs,
+      rawReply,
+    };
   }
   if (!isRecord(parsed) || parsed.verdict !== "deny") {
-    return { kind: "defer" };
+    return {
+      verdict: { kind: "defer" },
+      deferReason: "non-deny-verdict",
+      latencyMs,
+      rawReply,
+    };
   }
   const reason =
     typeof parsed.reason === "string" && parsed.reason.length > 0
       ? parsed.reason
       : GENERIC_TEACHING_REASON;
-  return { kind: "deny", reason };
+  return { verdict: { kind: "deny", reason }, latencyMs, rawReply };
 }
 
 /** Concatenate the text parts of an assistant reply. */

package/src/typo-patterns.ts

@@ -7,9 +7,16 @@
  * without a model call.
  */
 
-/** Result of compiling a pattern list: usable regexes plus any that failed. */
+/**
+ * Result of compiling a pattern list: usable regexes (with their original
+ * source strings, parallel to `regexes`) plus any that failed to compile.
+ * `patterns[i]` is the operator's literal config string for `regexes[i]`, kept
+ * so the decision trail records the pattern as written, not the slash-escaped
+ * `RegExp.source`.
+ */
 export interface CompiledTypoPatterns {
   regexes: RegExp[];
+  patterns: string[];
   invalidPatterns: string[];
 }
 
@@ -22,21 +29,28 @@ export function compileTypoPatterns(
   patterns: readonly string[],
 ): CompiledTypoPatterns {
   const regexes: RegExp[] = [];
+  const sources: string[] = [];
   const invalidPatterns: string[] = [];
   for (const pattern of patterns) {
     try {
       regexes.push(new RegExp(pattern));
+      sources.push(pattern);
     } catch {
       invalidPatterns.push(pattern);
     }
   }
-  return { regexes, invalidPatterns };
+  return { regexes, patterns: sources, invalidPatterns };
 }
 
-/** Whether any compiled pattern matches `path`. */
-export function matchesAnyTypoPattern(
+/**
+ * The source string of the first compiled pattern that matches `path`, or
+ * `undefined` if none do. The source (not the boolean) so the decision trail
+ * can record *which* pattern matched.
+ */
+export function matchTypoPattern(
   path: string,
   compiled: CompiledTypoPatterns,
-): boolean {
-  return compiled.regexes.some((regex) => regex.test(path));
+): string | undefined {
+  const index = compiled.regexes.findIndex((regex) => regex.test(path));
+  return index === -1 ? undefined : compiled.patterns[index];
 }

package/src/typo-reviewer.ts

@@ -4,17 +4,26 @@
  *
  * The decision runs top to bottom, deferring at the first miss so the cheap
  * gates short-circuit before any model call:
- *   1. surface is `external_directory` (else defer),
+ *   1. surface is `external_directory` (else defer, silently — not our surface),
  *   2. a candidate path is present (else defer),
  *   3. the path matches a configured typo pattern (else defer, no model call),
  *   4. the model confirms the typo (deny with a teaching reason) or defers.
  *
  * Every failure path defers — more prompting, never less (ADR 0007 invariant 2).
  * This slice never emits `allow`.
+ *
+ * Observability (issue #626): once an ask matches a typo pattern (the [#625]
+ * class — a candidate typo that *should* reach the model), the reviewer writes a
+ * positive `model_judge.decision` review entry recording the outcome and, on a
+ * defer, its reason — so a silent 100%-defer regression cannot hide again. The
+ * cheap short-circuits (`no-path`, `pattern-miss`) and the raw model reply go to
+ * the debug log; a non-`external_directory` surface is not logged at all.
  */
 
 import type {
   Authorizer,
+  AuthorizerLog,
+  AuthorizerVerdict,
   PromptPermissionDetails,
 } from "@gotgenes/pi-permission-system";
 
@@ -27,11 +36,31 @@ import {
 import {
   type CompiledTypoPatterns,
   compileTypoPatterns,
-  matchesAnyTypoPattern,
+  matchTypoPattern,
 } from "./typo-patterns";
 
 const REVIEWED_SURFACE = "external_directory";
 
+/** Review-log event: one positive decision record per pattern-matched ask. */
+const DECISION_EVENT = "model_judge.decision";
+/** Debug-log event: a cheap short-circuit before the model stage. */
+const SHORT_CIRCUIT_EVENT = "model_judge.short_circuit";
+/** Debug-log event: the raw model reply, gated behind `debugLog`. */
+const MODEL_REPLY_EVENT = "model_judge.model_reply";
+/** Debug-log event: `typoPatterns` entries that failed to compile. */
+const INVALID_PATTERNS_EVENT = "model_judge.invalid_patterns";
+
+/** A defer decided before the model call, still recorded positively. */
+type PreModelDeferReason = "model-unresolved" | "auth-failed";
+
+/** The shared fields of a `model_judge.decision` record, before the outcome. */
+interface DecisionBase {
+  requestId: string;
+  path: string;
+  matchedPattern: string;
+  modelId: string;
+}
+
 /** Collaborators for the reviewer, injected so the extension and tests wire them. */
 export interface TypoReviewerDeps {
   /** The loaded config, read live (absent until session config loads). */
@@ -40,21 +69,20 @@ export interface TypoReviewerDeps {
   getRegistry: () => ModelRegistryLike | undefined;
   /** The model-completion seam (production: `complete` from `@earendil-works/pi-ai`). */
   complete: CompleteFn;
-  /** Optional warn sink for operational notices (unresolved model, bad patterns). */
-  warn?: (message: string) => void;
 }
 
 /**
  * Build the `authorize` callback registered on the chain. The `query` argument
  * is unused in this slice — the deny-first reviewer decides from the path
- * pattern and the model, not from an engine re-query (slice 2 consumes it).
+ * pattern and the model, not from an engine re-query (slice 2 consumes it). The
+ * `log` argument is the injected review-log seam the decision trail records to.
  */
 export function createTypoReviewer(
   deps: TypoReviewerDeps,
 ): Authorizer["authorize"] {
-  const compiledFor = memoizeCompiledPatterns(deps.warn);
+  const compiledFor = memoizeCompiledPatterns();
 
-  return async (details) => {
+  return async (details, _query, log) => {
     const config = deps.getConfig();
     if (!config) {
       return { kind: "defer" };
@@ -62,29 +90,35 @@ export function createTypoReviewer(
     if (surfaceOf(details) !== REVIEWED_SURFACE) {
       return { kind: "defer" };
     }
+    const { requestId } = details;
     const path = pathOf(details);
     if (path === undefined) {
+      log.debug(SHORT_CIRCUIT_EVENT, { requestId, reason: "no-path" });
       return { kind: "defer" };
     }
-    if (!matchesAnyTypoPattern(path, compiledFor(config))) {
+    const matchedPattern = matchTypoPattern(path, compiledFor(config, log));
+    if (matchedPattern === undefined) {
+      log.debug(SHORT_CIRCUIT_EVENT, {
+        requestId,
+        path,
+        reason: "pattern-miss",
+      });
       return { kind: "defer" };
     }
+
+    const modelId = `${config.provider}/${config.model}`;
+    const base: DecisionBase = { requestId, path, matchedPattern, modelId };
     const registry = deps.getRegistry();
     const model = registry?.find(config.provider, config.model);
     if (!registry || !model) {
-      deps.warn?.(
-        `model "${config.provider}/${config.model}" did not resolve; deferring`,
-      );
-      return { kind: "defer" };
+      return deferWith(log, base, "model-unresolved");
     }
     const auth = await registry.getApiKeyAndHeaders(model);
     if (!auth.ok) {
-      deps.warn?.(
-        `auth for "${config.provider}/${config.model}" did not resolve (${auth.error}); deferring`,
-      );
-      return { kind: "defer" };
+      return deferWith(log, base, "auth-failed");
     }
-    return reviewPath({
+
+    const outcome = await reviewPath({
       path,
       config,
       model,
@@ -92,9 +126,52 @@ export function createTypoReviewer(
       apiKey: auth.apiKey,
       headers: auth.headers,
     });
+    if (outcome.rawReply !== undefined) {
+      log.debug(MODEL_REPLY_EVENT, {
+        requestId,
+        modelId,
+        rawReply: outcome.rawReply,
+      });
+    }
+    log.review(DECISION_EVENT, {
+      requestId,
+      surface: REVIEWED_SURFACE,
+      path,
+      matchedPattern,
+      modelCalled: true,
+      modelId,
+      latencyMs: outcome.latencyMs,
+      verdict: outcome.verdict.kind,
+      deferReason: outcome.deferReason ?? null,
+    });
+    return outcome.verdict;
   };
 }
 
+/**
+ * Record a pre-model defer (`model-unresolved` / `auth-failed`) as a positive
+ * `model_judge.decision` entry and return the defer verdict — so the two
+ * model-resolution failures leave evidence on record, not a silent absence.
+ */
+function deferWith(
+  log: AuthorizerLog,
+  base: DecisionBase,
+  deferReason: PreModelDeferReason,
+): AuthorizerVerdict {
+  log.review(DECISION_EVENT, {
+    requestId: base.requestId,
+    surface: REVIEWED_SURFACE,
+    path: base.path,
+    matchedPattern: base.matchedPattern,
+    modelCalled: false,
+    modelId: base.modelId,
+    latencyMs: null,
+    verdict: "defer",
+    deferReason,
+  });
+  return { kind: "defer" };
+}
+
 /** The gate-authoritative surface, falling back to the display surface. */
 function surfaceOf(details: PromptPermissionDetails): string | undefined {
   return details.accessIntent?.surface ?? details.surface ?? undefined;
@@ -110,21 +187,22 @@ function pathOf(details: PromptPermissionDetails): string | undefined {
  * object is in effect — the config is stable per session, so this avoids
  * recompiling the regexes on every ask.
  */
-function memoizeCompiledPatterns(
-  warn?: (message: string) => void,
-): (config: ModelJudgeConfig) => CompiledTypoPatterns {
+function memoizeCompiledPatterns(): (
+  config: ModelJudgeConfig,
+  log: AuthorizerLog,
+) => CompiledTypoPatterns {
   let cache:
     | { config: ModelJudgeConfig; compiled: CompiledTypoPatterns }
     | undefined;
-  return (config) => {
+  return (config, log) => {
     if (cache?.config === config) {
       return cache.compiled;
     }
     const compiled = compileTypoPatterns(config.typoPatterns);
     if (compiled.invalidPatterns.length > 0) {
-      warn?.(
-        `ignoring invalid typoPatterns: ${compiled.invalidPatterns.join(", ")}`,
-      );
+      log.debug(INVALID_PATTERNS_EVENT, {
+        invalidPatterns: compiled.invalidPatterns,
+      });
     }
     cache = { config, compiled };
     return compiled;