@czottmann/pi-automode 1.6.0 → 1.8.0

package/README.md

@@ -35,7 +35,7 @@ pi -e ./extensions/auto-mode.ts
 /automode reload    # reload config from disk
 /automode reset     # reset denial counters only
 /automode defaults  # print the built-in rule lists
-/automode config    # current effective config + diagnostics
+/automode config    # effective config, resolved log file path, + diagnostics
 /automode denials   # denial history for this session
 /automode model     # open classifier model selector and save to ~/.pi/agent/automode.json
 /automode model provider/model-id # save classifier model to ~/.pi/agent/automode.json
@@ -52,14 +52,15 @@ AM● a:12 d:2 ca:5 cd:1
 ```
 
 - `AM` — auto-mode prefix; `●` when enabled, `○` when disabled (via config or `/automode off`).
-- `a:` — actions allowed so far (checked minus blocked).
-- `d:` — actions blocked so far, for any reason (permission rule, deterministic hard-deny, or classifier).
-- `ca:` / `cd:` — classifier decisions split into allowed vs denied. These segments appear only after the classifier has run at least once; `d:` counts all blocks, so `d:` is always `>= cd:`.
+- `a:` — actions allowed so far (checked minus denied).
+- `d:` — actions denied so far, for any reason (permission rule, deterministic hard-deny, or classifier).
+- `ca:` / `cd:` — classifier decisions split into allowed vs denied. These segments appear only after the classifier has run at least once; `d:` counts all denials, so `d:` is always `>= cd:`.
 
 ## Docs
 
 - [Defaults and rule-list behavior](docs/defaults.md)
 - [Auto-mode classifier flow](docs/automode-classifier-flow.md)
+- [Observability logging](docs/observability-logging.md)
 
 ## Configuration
 
@@ -73,6 +74,18 @@ It reads `autoMode` from Pi-owned config only:
 
 It deliberately does not read `autoMode` from shared project `.pi/automode.json`, because a checked-in repo should not be able to weaken auto-mode rules. Shared project config may still contribute `permissions.deny` and `permissions.ask`.
 
+To disable pi-automode for the current project, create or edit `.pi/automode.local.json`:
+
+```json
+{
+  "autoMode": {
+    "enabled": false
+  }
+}
+```
+
+This is project-local and should not be committed. Shared project `.pi/automode.json` cannot disable auto-mode.
+
 Set a global default classifier model in `~/.pi/agent/automode.json`; override it per project in `.pi/automode.local.json`.
 
 Example:
@@ -81,6 +94,8 @@ Example:
 {
   "autoMode": {
     "classifierModel": "provider/model-id",
+    "maxUserTranscriptTokens": 4000,
+    "maxToolTranscriptTokens": 4000,
     "environment": [
       "$defaults",
       "Source control: github.example.com/acme-corp and all repos under it",
@@ -103,10 +118,31 @@ Example:
 }
 ```
 
+`maxUserTranscriptTokens` and `maxToolTranscriptTokens` are approximate per-category budgets; both default to 4000 and accept integers of at least 32. The former `maxTranscriptLines` setting is no longer supported because evidence selection is token-budgeted rather than line-based.
+
 ### `$defaults`
 
 See [Defaults and rule-list behavior](docs/defaults.md) for built-in `environment`, `allow`, `protectedPaths`, `soft_deny`, and `hard_deny` entries, plus replacement behavior when `$defaults` is omitted.
 
+### Observability logging
+
+Auto mode can write a JSONL observability log next to the current Pi session file, so you can inspect decisions and classifier usage. It is off by default.
+
+```json
+{
+  "autoMode": {
+    "log": {
+      "enabled": true,
+      "classifierIo": false
+    }
+  }
+}
+```
+
+With logging enabled, the sidecar also writes ccusage-compatible entries for every classifier response. `ccusage pi` reports this usage as a separate `-pi-automode` session even when `classifierIo` is off.
+
+See [Observability logging](docs/observability-logging.md) for the log file location, entry schema, and the `classifierIo` privacy tradeoff. Run `/automode config` to see the resolved log file path.
+
 ### Permission patterns
 
 Permission patterns use Pi tool names, for example `bash(...)`, `write(...)`, `edit(...)`, `read(...)`. The parser accepts capitalized names like `Bash(...)` for convenience, but the documented form is lowercase because Pi tool names are lowercase.
@@ -124,11 +160,9 @@ The extension blocks these before any allow or classifier decision:
 - root, home, and system-path destructive deletes
 - edits to `.pi/automode*`, `.pi` auto-mode files, and this extension's safety-control files
 
-Read-only Pi tools (`read`, `grep`, `find`, `ls`) are allowed after those checks.
-
-Writes to [protected paths](docs/defaults.md#protectedpaths) (shell profiles, tool configs, `.git`, `.vscode`, `.pi`, etc.) always go to the classifier, even if an `allow` rule matches. The classifier decides whether to permit the write.
+Read-only Pi tools (`read`, `grep`, `find`, `ls`) are allowed after those checks. Every side-effecting action goes to the classifier, including all `write` and `edit` calls, `bash`, MCP, subagent, network-capable tools, and unknown tools. This keeps classifier hard-deny rules unconditional; direct file writes cannot bypass them.
 
-Everything else goes to the classifier. If the classifier is missing, fails, or returns invalid JSON, the action is blocked.
+Classification starts with a one-token conservative filter and runs structured review only when that filter requests it. Both stages use a classifier-specific session key and short provider cache retention where the provider supports it. Missing models, provider failures, or malformed responses block the action.
 
 ## Examples
 
@@ -146,7 +180,7 @@ npm test
 npm pack --dry-run
 ```
 
-The tests cover the risky parts: scoped permission matching, config-source precedence, `$defaults` behavior, config diagnostics, deterministic hard-deny checks, shell parsing for risky bash fragments, classifier JSON parsing, hook-level blocking/allowing, classifier mocking, and protected-path enforcement.
+The tests cover the risky parts: scoped permission matching, config-source precedence, `$defaults` behavior, config diagnostics, deterministic hard-deny checks, shell parsing, write/edit classifier routing, symlink-aware safety-control checks, token-budgeted transcript selection, staged classifier parsing and caching options, and hook-level blocking/allowing.
 
 ## Publishing
 

package/docs/automode-classifier-flow.md

@@ -13,11 +13,11 @@ For each Pi `tool_call` event, the extension does this:
 5. Check `permissions.ask` rules and ask the user when needed.
 6. Run deterministic hard-deny checks.
 7. Allow read-only built-in tools without a classifier call.
-8. Send every remaining action to the classifier.
-9. Allow or block based on the classifier JSON response.
+8. Send every remaining action, including all writes and edits, through a one-token conservative filter.
+9. Run structured classifier review only when the filter requests it, then allow or block.
 10. Persist state and update the UI status/denial history.
 
-The default posture is fail-closed. If the classifier cannot be resolved, has no API key, errors, or returns invalid JSON, the action is blocked.
+The default posture is fail-closed. If the classifier cannot be resolved, has no API key, errors, or returns an invalid stage response, the action is blocked.
 
 ## Diagram
 
@@ -46,15 +46,15 @@ flowchart TD
   K -- no --> L{Read-only built-in tool?}
 
   L -- yes --> L1[Allow locally]
-  L -- no --> M{write/edit protected path?}
-  M -- yes --> N[Classify action]
-  M -- no --> N[Classify action]
+  L -- no --> N[Run one-token filter]
 
-  N --> O{Classifier available and valid JSON?}
-  O -- no --> O1[Block: fail closed]
-  O -- yes --> P{decision == allow?}
-  P -- yes --> Q[Allow tool]
-  P -- no --> R[Block with classifier reason]
+  N --> O{Exact safe token?}
+  O -- yes --> Q[Allow tool]
+  O -- malformed/error --> O1[Block: fail closed]
+  O -- review --> P[Run structured review]
+  P --> P1{Valid allow decision?}
+  P1 -- yes --> Q
+  P1 -- no/error --> R[Block with classifier reason]
 
   X --> S[Persist state + update UI]
   F1 --> S
@@ -80,6 +80,18 @@ The effective config combines these sources:
 
 Shared project `.pi/automode.json` cannot change `autoMode` rules. That is deliberate: a checked-in repo must not be able to weaken auto-mode. It may still add Pi permission rules.
 
+To disable pi-automode for the current project, set `autoMode.enabled` to `false` in `.pi/automode.local.json`:
+
+```json
+{
+  "autoMode": {
+    "enabled": false
+  }
+}
+```
+
+This affects only that project-local config. Shared project `.pi/automode.json` cannot disable auto-mode.
+
 List settings such as `allow`, `soft_deny`, `hard_deny`, `environment`, and `protectedPaths` support `$defaults`. Omitting `$defaults` replaces the built-ins for that section only. See [Defaults and rule-list behavior](defaults.md).
 
 ## Context captured before classification
@@ -143,21 +155,19 @@ The read-only tool set is:
 read, grep, find, ls
 ```
 
-Reads to protected paths are still allowed. Protected-path handling only affects writes and edits.
+Reads to protected paths are still allowed. Every write and edit is classifier-reviewed, whether or not its target is protected.
 
 ## Protected paths
 
-Writes and edits to protected paths are always classifier-reviewed. The default protected paths include `.git`, `.pi`, editor config directories, shell profiles, package-manager config files, hook configs, and similar files.
+The protected-path configuration identifies safety-sensitive targets such as `.git`, `.pi`, editor config directories, shell profiles, package-manager config files, hook configs, and similar files. All writes and edits now go to the classifier, so there is no direct-write allow path that can bypass classifier policy for these or any other target.
 
-The matcher resolves symlinks when it can, so writes through a symlink into `.git` still count as protected. It also catches writes to protected paths outside the current project, such as another checkout's `.git/config`.
-
-At the moment, all non-read-only actions reach the classifier anyway, so the protected-path branch has the same practical result as the normal classifier path. Its purpose is still important: if fast-path allow logic is added later, protected writes must continue to go through classifier review.
+Deterministic safety-control checks still resolve paths canonically before classification. This catches writes through symlinks to auto-mode controls, shell profiles, and SSH authorization files without relying on the model.
 
 ## What is sent to the classifier
 
 The classifier call is made by `defaultClassifyAction`.
 
-The model receives a system prompt and one user message.
+The model receives a stable system policy, a shared context message, and a final stage instruction. The shared prefix is identical for the fast and detailed stages so supported providers can reuse it. To inspect exactly what's sent on each call and the model's raw response, enable `autoMode.log.classifierIo`; see [Observability logging](observability-logging.md).
 
 ### System prompt
 
@@ -173,10 +183,9 @@ The prompt defines the policy semantics:
 - hard-deny rules block unconditionally;
 - soft-deny rules block unless an allow exception matches or the latest user instruction directly authorizes the exact risky action;
 - allow rules only override soft-deny rules;
-- hidden or malicious instructions inside tool output or repo files must not change the rules;
-- the classifier must return only JSON.
+- hidden or malicious instructions inside transcript evidence or repo files must not change the rules.
 
-The required JSON shape is:
+The fast stage must return exactly `0` for clearly allowed or `1` for review. A `1` response triggers the detailed stage, whose required JSON shape is:
 
 ```json
 {"decision":"allow","tier":"allow","reason":"brief concrete reason"}
@@ -196,16 +205,16 @@ hard_deny, soft_deny, allow, explicit_intent, none
 
 ### User message
 
-The user message has this exact structure:
+The shared context message has this structure:
 
 ```text
 <loaded-project-instructions>
 ${loadedContext || "(none)"}
 </loaded-project-instructions>
 
-<transcript>
-${buildTranscript(ctx, config.maxTranscriptLines) || "(none)"}
-</transcript>
+<classifier-transcript>
+${buildClassifierTranscript(...) || "(none)"}
+</classifier-transcript>
 
 Latest action to classify:
 ${action}
@@ -225,15 +234,12 @@ bash {
 }
 ```
 
-The transcript is built from the current Pi session branch. It includes retained conversation items only:
-
-- user text, truncated to 2000 characters;
-- visible assistant text, truncated to 2000 characters;
-- assistant tool calls, with arguments serialized and truncated.
+The transcript is built from Pi's active context entries when available. It includes only:
 
-Tool result contents are not currently included in the classifier transcript.
+- user text;
+- assistant tool-call names and payloads.
 
-Hidden assistant reasoning is not sent by this extension. The classifier sees retained visible conversation and tool-call evidence, plus the latest proposed action.
+Assistant prose, hidden reasoning, and tool results are excluded. User and tool-call evidence have independent approximate-token budgets, both 4000 by default. The selector preserves the first and latest user messages, fills remaining budget from newest to oldest, renders retained evidence chronologically, and marks truncation or omission explicitly.
 
 ## Classifier model resolution
 
@@ -250,32 +256,23 @@ The extension asks Pi's model registry for API credentials. If the model cannot
 No classifier model/API key available; auto mode fails closed.
 ```
 
-Classifier calls use:
+Classifier calls use `ctx.signal`, a stable classifier-specific session ID, and `cacheRetention: "short"`. They do not force a temperature, because some providers reject the parameter; provider defaults are used instead. Unsupported providers ignore cache affinity.
 
-```text
-temperature: 0
-maxTokens: 700
-signal: ctx.signal
-```
+The fast stage requires one visible digit but allows `maxTokens: 4`, because some OpenAI-compatible servers count control and end-of-sequence tokens against the generation limit. Detailed review uses `maxTokens: 1200` and may retry once after malformed or truncated output.
 
 ## Parsing the classifier result
 
-The parser accepts JSON returned directly, JSON inside a fenced code block, or the first JSON-looking object in the response.
-
-A valid response must contain:
-
-- `decision: "allow"` or `decision: "block"`
-- `reason` as a string
+The fast-stage parser accepts only the exact output `0` or `1` from a response with `stopReason: "stop"`. Empty, additional, malformed, or non-stop responses block immediately.
 
-If `tier` is missing, it defaults to `none`.
+The detailed parser accepts only the exact JSON object requested by the prompt from a response with `stopReason: "stop"`. It requires exactly `decision`, `tier`, and `reason`; rejects wrappers, extra fields, unknown tiers, and empty reasons; and fails closed on any shape drift. A truncated response with `stopReason: "length"` is retried but cannot authorize an action itself; other non-stop responses block immediately.
 
-If parsing fails, the action is blocked with this reason:
+If detailed parsing fails after its retry, the action is blocked with this reason:
 
 ```text
 Classifier response was not valid decision JSON; auto mode fails closed.
 ```
 
-If the model call throws, the action is blocked with a classifier failure message.
+If the model call throws or returns an error or aborted response, the action is blocked immediately with a classifier failure message.
 
 ## State, UI, and denial history
 

package/docs/defaults.md

@@ -46,13 +46,13 @@ These are exceptions to `soft_deny`, not to `hard_deny`.
 
 ### `protectedPaths`
 
-`$defaults` expands to paths where writes are never auto-approved — they always go to the classifier, regardless of `allow` rules. This matches Claude Code's protected-paths behavior.
+`$defaults` expands to safety-sensitive paths. Every `write` and `edit` call now goes to the classifier, so `protectedPaths` no longer changes whether a model call occurs; it remains part of the resolved configuration for compatibility and inspection. No path can be reached through a direct-write allow path, and `allow` rules cannot override a classifier hard-deny decision.
 
 Protected directories: `.git`, `.config/git`, `.vscode`, `.idea`, `.husky`, `.cargo`, `.devcontainer`, `.yarn`, `.mvn`, `.pi`.
 
 Protected files: `.gitconfig`, `.gitmodules`, `.gitignore`, `.gitattributes`, shell profiles (`.bashrc`, `.zshrc`, `.profile`, etc.), `.envrc`, package manager configs (`.npmrc`, `.yarnrc`, `.yarnrc.yml`, `.pnp.cjs`, `bunfig.toml`, etc.), Bazel configs (`.bazelrc`, `.bazelversion`, `.bazeliskrc`), hook configs (`.pre-commit-config.yaml`, `lefthook.yml`), Gradle/Maven wrappers, `.devcontainer.json`, `.ripgreprc`, `pyrightconfig.json`, `.mcp.json`.
 
-Read-only tools (`read`, `grep`, `find`, `ls`) bypass this check — reads to protected paths are always allowed. Only `write` and `edit` are affected.
+Read-only tools (`read`, `grep`, `find`, `ls`) remain locally allowed after permission and deterministic checks. Writes and edits always require classification, regardless of their target.
 
 ### `soft_deny`
 
@@ -86,6 +86,17 @@ Soft blocks can be overridden by a matching `allow` exception or by direct, spec
 
 Hard-deny rules cannot be overridden by `allow` or by user intent.
 
+### Classifier transcript budgets
+
+Classifier evidence has separate approximate-token budgets for user messages and assistant tool-call payloads:
+
+- `maxUserTranscriptTokens`: 4000
+- `maxToolTranscriptTokens`: 4000
+
+The selector keeps the first and latest user messages as intent anchors, then fills remaining space from the newest eligible entries. Individual entries are capped, and omitted or truncated evidence is marked in the classifier transcript. Assistant prose and tool results are excluded.
+
+These are approximate limits based on character counts, not provider-tokenizer guarantees. Override either value with an integer of at least 32 in a Pi-owned `autoMode` config. The former `maxTranscriptLines` setting is no longer supported.
+
 ### Replacement behavior
 
 Use `$defaults` when you want to keep the built-ins and add your own entries:

package/docs/observability-logging.md

@@ -0,0 +1,109 @@
+# Observability logging
+
+Auto mode can write a JSONL observability log next to the current Pi session file, so you can inspect decisions and classifier usage. It is off by default and fail-open: a write error never changes an allow/block decision.
+
+## Enabling
+
+Set `autoMode.log` in any Pi-owned config source (`~/.pi/agent/automode.json`, `.pi/automode.local.json`, or `PI_AUTOMODE_SETTINGS_JSON`):
+
+```json
+{
+  "autoMode": {
+    "log": {
+      "enabled": true,
+      "classifierIo": false
+    }
+  }
+}
+```
+
+- `enabled` — write one `decision` line per tool-call decision and one ccusage-compatible `message` line per classifier model response.
+- `classifierIo` — also write the classifier's prompt, raw model responses, and parsed decision for classifier-routed actions. Off by default; see [Privacy](#privacy) below.
+
+Fields merge independently across config scopes (set `enabled` globally and `classifierIo` per project, for example). Shared project `.pi/automode.json` cannot set `log` — it follows the same `autoMode` exclusion as the rest of the config. Shape is validated and reported by `/automode config`.
+
+Logging only writes entries while auto-mode is **enabled**. With auto-mode off, no tool calls reach the hook, so no entries are produced.
+
+## Log file location
+
+The log file is colocated with the current Pi session file, with `-pi-automode` inserted before the extension:
+
+```text
+<session-file>                       →  <dir>/<id>.jsonl
+<session-file>-pi-automode.jsonl     →  <dir>/<id>-pi-automode.jsonl
+```
+
+For example: `~/.pi/agent/sessions/<slug>/<id>-pi-automode.jsonl`. If no session file is set, it falls back to `<sessionDir>/<sessionId>-pi-automode.jsonl`. Run `/automode config` to see the resolved path.
+
+There is one combined file per session. Each line is one JSON object with a `type` discriminator. Entries for the same tool call share a `decisionId`.
+
+## Entry types
+
+### `decision`
+
+One per tool-call decision. Every allow and every block goes through exactly one of these.
+
+| field | meaning |
+| --- | --- |
+| `ts` | ISO timestamp |
+| `decisionId` | links to the `classifier` entry (if any) for the same call |
+| `sessionId` | Pi session id |
+| `cwd` | working directory |
+| `tool` | tool name, e.g. `bash`, `write` |
+| `summary` | `actionSummary` — tool name + input JSON (truncated) |
+| `kind` | enforcement path: `permissions.deny`, `permissions.ask`, `deterministic-hard-deny`, `classifier`, or `read-only` |
+| `outcome` | `allow` or `block` |
+| `reason` | the reason string (classifier reason, or the deterministic/permission reason) |
+| `classifierModel` | the configured classifier model, when relevant |
+
+### `message` (classifier usage)
+
+One per classifier model response, including malformed responses that trigger a retry. Written whenever logging is enabled, before the matching `decision` line. Its shape is intentionally compatible with `ccusage pi`:
+
+| field | meaning |
+| --- | --- |
+| `timestamp` | ISO timestamp from the classifier response |
+| `message.role` | always `assistant` |
+| `message.model` | model ID returned by the classifier provider |
+| `message.usage` | provider-reported input, output, cache, total-token, and cost fields |
+
+`ccusage` reports this sidecar as a separate `-pi-automode` session. This entry contains no prompt or response text, so it is written even when `classifierIo` is off.
+
+### `classifier`
+
+Written only for classifier-routed actions, and only when `classifierIo: true`. It follows any classifier-usage `message` entries and precedes the matching `decision` line in the file.
+
+| field | meaning |
+| --- | --- |
+| `ts` | ISO timestamp |
+| `decisionId` | matches the `decision` entry for the same call |
+| `model` | classifier model used, e.g. `anthropic/claude-haiku-4` |
+| `prompt.system` | the full system policy with `environment`/`allow`/`soft_deny`/`hard_deny` rules interpolated |
+| `prompt.context` | the shared context message: loaded project instructions + classifier transcript + action |
+| `prompt.fastInstruction` | the exact one-token filter instruction |
+| `prompt.detailedInstruction` | the exact structured-review instruction |
+| `attempts` | one entry per classifier model call (see below) |
+| `durationMs` | total classifier time |
+| `parsed` | the final decision that was acted on (`{ decision, tier, reason }`) |
+
+Each `attempts[]` entry is `{ stage, attempt, response?, parsed?, error?, durationMs }`:
+
+- `stage` — `fast` for the one-token filter or `detailed` for structured review.
+- `response` — `{ stopReason, text, model, timestamp, usage, errorMessage? }`, the raw model output and provider-reported usage for that call, including provider-reported errors and aborted requests.
+- `parsed` — the decision parsed from the response, or absent if it did not parse.
+- `error` — present when the call threw (network/auth); `response` is then absent.
+
+This records both stages, retries, and fail-closed cases verbatim. A fast allow has one entry. A review followed by malformed detailed JSON and a successful retry has three entries.
+
+## Privacy
+
+`prompt.context` is the shared content sent to both classifier stages: loaded project instructions, selected transcript evidence, and the action being classified. The stage-specific final instructions are logged separately. See [Auto-mode classifier flow → What is sent to the classifier](automode-classifier-flow.md#what-is-sent-to-the-classifier) for how the evidence is assembled and truncated.
+
+The log records this payload locally, but the same data is also sent to the classifier model endpoint on every classifier-routed call. If `classifierModel` points at a cloud provider, that payload leaves the machine. Enable `classifierIo` when debugging classifier behavior or tuning rules; leave it off for routine outcome logging.
+
+## Sizing
+
+- `decision` line: ~0.4–2 KB (driven by `summary`, which carries the tool input, capped at 6 KB).
+- `classifier` line: ~5 KB fixed policy plus loaded project instructions, selected transcript evidence, stage instructions, and recorded responses.
+
+Transcript evidence is bounded separately by `autoMode.maxUserTranscriptTokens` and `autoMode.maxToolTranscriptTokens`, both 4000 approximate tokens by default. Assistant prose and tool results are not included. Provider cache hits can reduce billed or processed input where supported, but the log still records the full classifier payload.