pi-warden 1.7.0 → 1.11.0

package/CHANGELOG.md

@@ -6,6 +6,68 @@ How to keep this current: add the entry in the same pull request as the change,
 
 ## Unreleased
 
+<!-- Empty. Next release starts here. -->
+
+## 1.11.0
+
+### Added
+
+- SubQ backend: `"typesafeBackend": "subq"` in the user config sends judgments to `https://api.subq.ai` with the model `subq-snap-preview`, using the key in `SUBQ_API_KEY`. ([Configuration](docs/configuration.md#user-config))
+
+### Changed
+
+- pi-typesafe 0.10.0.
+- Every judgment fits the backend's question limit: 16 questions per request on SubQ, 32 on the other backends. The rules guard, the end-of-run turn rules, the turn-start rules reminder, `/warden rules check`, the context filter, relevance compaction, and the conscience split a judgment with more questions into several requests and merge the answers by question id; the conscience repeats its disposition question in each request. On the other backends every guard sends the same requests as before, and a judge an extension author builds without `limits` keeps the 32-question limit. On SubQ a judgment with more than 16 questions uses more of `maxRequests`.
+- The action guard and `/warden rules calibrate` also fit the backend's question limit. On SubQ, an action guard call with more than 16 questions (one per `rm` target the request does not authorize, plus any `questions` a library caller adds) and a calibration sample with more than 16 rules split into requests of at most 16, sent together, with the answers merged by question id; the verdict and the scores are the same as from one request. No action guard question repeats in each request, because none of them reads another's answer. On the other backends, and for a judge without `limits`, both send the same requests as before.
+
+## 1.10.0
+
+### Changed
+
+- pi-warden now declares support for Pi 1.x: the `@earendil-works/pi-coding-agent` and `@earendil-works/pi-tui` peer ranges go from `>=0.85.1 <1` to `>=0.85.1 <2`, so installing pi-warden next to Pi 1.x no longer fails with a peer dependency conflict. npm still prints an `ERESOLVE overriding peer dependency` warning, because pi-typesafe declares Pi `<1`.
+
+### Fixed
+
+- A run that a custom message starts no longer crashes the turn-start handler. Pi versions that fire `before_agent_start` for such a run (for example, a background subagent's report) give it no prompt text; the turn-start rules reminder and the conscience assessment now skip that run and write a `custom_trigger` line to the trace.
+
+## 1.9.0
+
+### Changed
+
+- The action guard no longer lets text the agent writes into a shell comment sway Jev. The command in the request state and the `script_lines` are sent without shell comments (a `#` at the start of a word, outside quotes, including quotes that span lines, and heredoc bodies, and not `$#` or `${#…}`); a command the scan cannot read with certainty (an unclosed quote, backticks, arithmetic) is sent unchanged, while the confirm dialog and the trace keep the full command. Agent-bound reasons no longer carry Jev scores: the held-call message, the headless warn steer, the declined-call reason, and the intent, off-task, should-proceed, and large-output steers keep the words (`irreversible`, `possibly irreversible`) without the numbers, and the dialog, notices, trace, holds, and learning records keep them. ([Guards](docs/guards.md))
+
+## 1.8.1
+
+### Fixed
+
+- Repeated runs (`context.dedupeRuns`) no longer cut text the model never saw. On the `claude-bridge` provider, Claude Code saves a tool result over its MCP output limit to a file and gives the model only a "saved to file" message; pi-warden counted that result as seen and then cut later reads of the saved file as repeats. A tool result over 20,000 characters on `claude-bridge` (judged by the provider of the call that made it) is no longer counted as seen, and a `read` or `bash` call that names a file in a Claude Code `tool-results` directory is never cut. Other providers, and `claude-bridge` results under the limit, are cut as before. ([Guards](docs/guards.md))
+
+## 1.8.0
+
+### Added
+
+- `trial.silentHalf` (default `false`, user config only): a randomized comparison of the guards in real work. Each session is put in arm `live` or `silent`, 50/50 from its session id, so a resumed session keeps its arm. A `silent` session judges as usual — every guard runs and every request is made — but records each delivery it would have made instead of sending it: rules, slop, prose, stuck and repeat, done-check, cause-check, open loops, intent-mismatch, off-task, should-proceed, large-output, and action-warning steers, conscience tips, preference notes, turn-start rules reminders, compact evidence, subagent triage wakes, call-waste notes, and tool-output compression. Everything that protects data stays live in both arms: irreversible calls are held or blocked, credential values are masked, security, credential, and prompt-injection notices are delivered, and runaway stops and the warnings about pi-warden itself are unaffected. One `pi-warden-trial` custom session entry names the arm at session start, and one per withheld delivery carries its kind, guard, time, and the redacted text clipped to 300 characters; custom entries stay out of the model's context, so both arms can be read back from the session logs. With the setting off, no entry is written and no delivery changes. ([Configuration](docs/configuration.md#user-config))
+
+## 1.7.1
+
+### Added
+
+- A/B benchmark tooling, registration v4 (`eval/preregistration.md`): the run moves to the CheapestInference flat-rate subscription with no paid agent model (`cheapestinference/deepseek-v4.1-flash`, then `cheapestinference/mimo-v2.5`), the models run one after the other at concurrency 3, the repeats (6 and 7) are sized from the smoke timings to about 24 hours each, and the Jev cap is $3 per model. The stopped v3 batches are named as a stopped pilot and never analysed.
+- A/B batch Jev-error stop (`eval/jev-stop.mjs`): a run whose one failed Jev request is a cancelled trace-only sample (the guard's second request on one judged call in twenty, cancelled by its own 5-second deadline) is recorded as `cancelledTraceSamples: 1` and no longer stops the batch, found by time (the entry with a `jev:` line nearest to the failed request's time is within 1,000 ms of it and has no off-task answer) rather than by entry number; every other failure still does. Repo tooling; rides along with the next release.
+- A/B batch re-run pass (`scripts/eval-ab.mjs`, `eval/batch.mjs`, Corrections 4 of `eval/preregistration.md`): `--resume <batch folder> --rerun-excluded`, once a model's main pass is finished, runs every block that holds an `infraError` run once more (all three cells, registered cell order); the first runs move to `superseded` in `runs.json` with `supersededAt` and their evidence folders to `superseded/` in the batch folder, and leave every metric while their Jev dollars stay in the batch total. One pass per batch: the exclusions after it are final and the 10% limit applies to them; a stop during the pass resumes with `--resume`. Repo tooling; rides along with the next release.
+- A stall rule for batch runs (`eval/stall.mjs`, `--stall-min`, default 15): a pi process whose session log shows no new assistant message or tool result for that long is killed, and the run is an infrastructure failure, re-run and excluded like an API error. `--arc-timeout-min` sets the timeout of one turn of a multi-turn run.
+- Equivalent dollars for the subscription models (`eval/config.mjs`): `cheapestinference/deepseek-v4.1-flash` at DeepSeek's `deepseek-flash` rates by call timestamp, `cheapestinference/mimo-v2.5` at Xiaomi's official MiMo-V2.5 price.
+- `scripts/eval-timing.mjs`: per-run wall time, per-call queue wait and generation time, and runs per hour of a batch folder, from timings only; `node eval/power.mjs --registered R1,R2` prints the power at the registered sizes.
+- A/B thesis decision tool (`eval/decide.mjs`, `scripts/thesis-decide.mjs`): the decision rules of `eval/preregistration.md` ("Decision rules", Corrections 7) computed from one model's `runs.json` (the per-run values, the paired blocks, the exact McNemar p, the one-sided Clopper-Pearson lower bound, the violation rate ratio and its lower bound, the paired bootstrap, the Wald and delta-method sensitivity bounds, and the decision of each rule and of the model); `node scripts/thesis-decide.mjs <runs.json> --label A --blocks 120` prints the table (`--json` for JSON), and `--metrics FILE` writes the analysis set's per-run values. Tests on made-up data in `tests/eval-decide.test.ts`. Repo tooling; rides along with the next release.
+- A/B thesis decision tool metrics input (`scripts/thesis-decide.mjs`, `eval/decide.mjs`): the same script also reads a `--metrics` file (the analysis set's `task`, `family`, `repeat`, `cell`, `s`, `v`, `c`) and prints the same decision table, with a first line that says the input was a metrics file. Repo tooling; rides along with the next release.
+- A/B thesis pooled view (`eval/decide.mjs`, `scripts/thesis-pooled.mjs`): the two models' runs pooled after the run to describe the three thesis claims (the hallucination proxy `h`, the pooled paired success, and the token count `t` with medians, the 90th percentile, the count over 1,000,000, and the paired warden-vs-control ratio), labelled post hoc and deciding nothing; the `--metrics` rows of `scripts/thesis-decide.mjs` also carry `h` and `t`. Repo tooling; rides along with the next release.
+- A/B thesis registration, Corrections 5 to 7 (`eval/preregistration.md`): the stops and other events of both batches are recorded, and the analysis methods behind the decision rules' bounds and p-value are fixed before any of them is computed.
+- A/B thesis report (`eval/reports/2026-10-10-thesis-v4/README.md`): the registered decision rules applied to both models of the v4 batch, with the decision tables, per-cell tables, pair counts and bounds, sensitivity bounds, the secondary trap-capable analysis, and the infrastructure and power limits; both models show success not more than 5 points below control (one-sided 95% lower bounds +0.00 and -2.14 points), model B (`cheapestinference/mimo-v2.5`) also supports the rule-violation and Jev-increment rules, and the cost rule is inconclusive on both.
+
+### Fixed
+
+- A/B run key isolation (`scripts/eval-ab.mjs`): the harness copies the TypeSafe key into a run's agent folder only for a cell that asks Jev, so an agent in any other cell cannot turn Jev on by itself, as 6 `warden-offline` runs of the v4 batch did. Repo tooling; rides along with the next release.
+- Eval environment filtering (`eval/env.mjs`): `filterEnv` now drops an inherited `PI_WARDEN_TRACE_DIR`, so a `warden-offline` run cannot write its pi-warden trace into a folder inherited from the session that started the batch; only the cells that ask Jev get one, set by `scripts/eval-ab.mjs`.
 ## 1.7.0
 
 ### Added

package/dist/config.d.ts

@@ -352,6 +352,15 @@ export interface PrefsConfig {
     /** At session start, send the standing preferences that pass every injection rule to the agent as one context message. */
     inject: boolean;
 }
+/** The silent-half field trial (trial.ts): a random half of sessions records what it would have told the agent. */
+export interface TrialConfig {
+    /**
+     * Put each session in arm live or silent, 50/50 from its id. A silent session judges as usual but records every
+     * suppressed delivery instead of sending it; holds, secret masking, security notices, and runaway stops stay live.
+     * Off by default, user file only: turning it on weakens guarding in half the sessions, so a project file must not.
+     */
+    silentHalf: boolean;
+}
 /** Per-model steer adaptation: a steer kind a model rarely follows or often disputes becomes trace-only for it. */
 export interface SteersConfig {
     /** Make a steer kind trace-only for a model that rarely follows or often disputes it. */
@@ -469,6 +478,8 @@ export interface WardenConfig {
     conscience: ConscienceConfig;
     /** Standing preferences: corrections the user repeated in earlier sessions of this project. */
     prefs: PrefsConfig;
+    /** Silent-half field trial: off by default, user file only. */
+    trial: TrialConfig;
     /** Call-waste notes: advisory sentences attached to the tool result that triggers them. */
     waste: WasteConfig;
     /** Relevance compaction in place of Pi's compaction summary. Off by default. */
@@ -478,7 +489,7 @@ export interface WardenConfig {
 }
 export declare const PACKAGE_NAME = "pi-warden";
 /** Bumped when WardenConfig gains a section; extension.ts checks it so a half-updated module graph is reported, not crashed on. */
-export declare const CONFIG_SCHEMA = 13;
+export declare const CONFIG_SCHEMA = 14;
 export declare const PROJECT_CONFIG_FILE = "pi-warden.json";
 export declare function defaultConfig(): WardenConfig;
 /** The user file is in the host's agent directory: `~/.pi/agent` on Pi, `~/.omp/agent` on oh-my-pi. `PI_CODING_AGENT_DIR` overrides it. */

package/dist/config.js

@@ -32,7 +32,7 @@ export const WARDEN_MODES = ["steer", "confirm", "advise"];
 export const CONSCIENCE_SKILL_MODES = ["off", "recommend", "load"];
 export const PACKAGE_NAME = "pi-warden";
 /** Bumped when WardenConfig gains a section; extension.ts checks it so a half-updated module graph is reported, not crashed on. */
-export const CONFIG_SCHEMA = 13;
+export const CONFIG_SCHEMA = 14;
 export const PROJECT_CONFIG_FILE = `${PACKAGE_NAME}.json`;
 export function defaultConfig() {
     return {
@@ -102,6 +102,7 @@ export function defaultConfig() {
             localFloor: 0.5,
         },
         prefs: { enabled: true, inject: true },
+        trial: { silentHalf: false },
         waste: { enabled: true, tip: false, every: 20, sleep: true, paging: true, search: true, recheck: true },
         // pi-claude-bridge compacts its own models and cancels on failure; its summary must not be replaced.
         compaction: { enabled: false, keepThreshold: 0.5, maxSummaryTokens: 20000, timeoutMs: 20000, maxRequests: 12, skipProviders: ["claude-bridge"] },
@@ -646,9 +647,16 @@ export function applyUserOverrides(base, raw) {
         learning: applyLearning(base.learning, raw.learning),
         conscience: applyConscience(base.conscience, raw.conscience),
         prefs: applyPrefs(base.prefs, raw.prefs),
+        trial: applyTrial(base.trial, raw.trial),
         warnings: [...(base.warnings ?? []), ...warnings],
     };
 }
+/** A project cannot turn the trial on: applyProjectOverrides never copies this section, so the user file is the only source. */
+function applyTrial(base, raw) {
+    if (!isObject(raw))
+        return base;
+    return { silentHalf: boolean(raw.silentHalf, base.silentHalf) };
+}
 function applyWaste(base, raw) {
     if (!isObject(raw))
         return base;

package/dist/conscience.d.ts

@@ -5,7 +5,7 @@
  * Pre-measurement: all selections are trace-only. No steers, no auto-loading.
  */
 import type { Skill } from "@earendil-works/pi-coding-agent";
-import type { Questions } from "pi-typesafe";
+import type { BackendLimits, Questions } from "pi-typesafe";
 import type { JudgmentsOffReason } from "./backend.js";
 import type { ConscienceConfig } from "./config.js";
 import type { TaskSpine } from "./shape.js";
@@ -211,6 +211,7 @@ export interface Judge {
     }): Promise<{
         answers: Record<string, unknown>;
     }>;
+    readonly limits?: BackendLimits | undefined;
 }
 export interface ConscienceDeps {
     judge: Judge | undefined;

package/dist/conscience.js

@@ -1,4 +1,4 @@
-import { choice, noul, score } from "pi-typesafe";
+import { choice, DEFAULT_BACKEND_LIMITS, noul, score } from "pi-typesafe";
 import { createHash } from "node:crypto";
 import { redact } from "./redact.js";
 import { SPINE_GOAL_LIMIT, SPINE_HISTORY_LIMIT, SPINE_HISTORY_TURNS } from "./shape.js";
@@ -16,7 +16,7 @@ export const SCORE_LEVELS = [
 const MAX_ELIGIBLE = 256;
 /** Max UTF-8 bytes per candidate description. */
 const MAX_DESC_BYTES = 8192;
-/** Max questions per single judge request (including the shared disposition question). */
+/** Max questions per single judge request (including the shared disposition question); a backend with a lower limit gets its own. */
 const MAX_QUESTIONS_PER_REQUEST = 32;
 /* ─── Path/URL sanitizer (spec §6) ─────────────────────────────────── */
 /** Strip absolute paths, URLs, and credentials from a description. */
@@ -510,7 +510,7 @@ export async function assess(prompt, recentContext, skills, tools, activeSkills,
     const effectiveTimeout = Math.min(config.timeoutMs, sharedTimeoutMs);
     const deadline = start + effectiveTimeout;
     const MAX_IN_FLIGHT = 4;
-    const maxCandidatesPerRequest = MAX_QUESTIONS_PER_REQUEST - 1; // -1 for disposition
+    const maxCandidatesPerRequest = Math.min(MAX_QUESTIONS_PER_REQUEST, judge.limits?.maxQuestions ?? DEFAULT_BACKEND_LIMITS.maxQuestions) - 1; // -1 for disposition
     const cappedRanked = gate.ranked.map(r => ({ candidate: r.candidate, capped: capDescription(r.candidate) }));
     const workQueue = [];
     for (let i = 0; i < cappedRanked.length; i += maxCandidatesPerRequest) {

package/dist/dedupe.d.ts

@@ -8,6 +8,7 @@ export declare const RUN_MIN_LINES = 20;
 export declare const RUN_MIN_CHARS = 1500;
 /** Errors and verdicts usually sit at the end of a result, so the last characters are never replaced. */
 export declare const TAIL_KEEP_CHARS = 2000;
+export declare const SWAP_LIMIT_CHARS = 20000;
 export interface RepeatedRun {
     /** First line of the run in the new text, 0-based. */
     start: number;
@@ -37,5 +38,12 @@ export declare class SeenText {
 }
 /** Replaces each run with one pointer line; `path` holds the full original text. */
 export declare function collapseRuns(text: string, runs: RepeatedRun[], path: string): string;
+/** Whether `text` (a tool input) names a result file that `provider` saved instead of giving it to the model. */
+export declare function savedResultIn(text: string, provider: string | undefined): boolean;
+/**
+ * The seen items of a branch, without a tool result the model did not get: one over the output limit of the provider
+ * of the assistant message that called it. `provider` stands in when no assistant message names one.
+ */
+export declare function seenItems(entries: unknown[], provider: string | undefined): SeenItem[];
 /** The texts a session entry puts in context and where the agent saw them. Thinking, images, and summaries are not indexed. */
 export declare function seenItem(entry: unknown): SeenItem | undefined;

package/dist/dedupe.js

@@ -8,6 +8,15 @@ export const RUN_MIN_LINES = 20;
 export const RUN_MIN_CHARS = 1500;
 /** Errors and verdicts usually sit at the end of a result, so the last characters are never replaced. */
 export const TAIL_KEEP_CHARS = 2000;
+/**
+ * Providers whose host swaps a tool result over its output limit for a "saved to file" message: the model never gets
+ * that text, so it is not seen. claude-bridge runs Pi tools as Claude Code MCP tools, and nothing in the session marks the swap.
+ */
+const SWAPPING_PROVIDERS = new Set(["claude-bridge"]);
+// Claude Code saves an MCP result over 50,000 characters or MAX_MCP_OUTPUT_TOKENS (25,000 by default) to a file; 20,000 stays under both even at 1.25 characters per token.
+export const SWAP_LIMIT_CHARS = 20_000;
+/** A file in a Claude Code session's tool-results directory, where it saves a swapped result. */
+const SAVED_RESULT = /\/projects\/[^/\s"'\\]+\/[^/\s"'\\]+\/tool-results\/[^/\s"'\\]+/;
 const BASE = 1_000_003;
 /** BASE^(RUN_MIN_LINES - 1) mod 2^32, for the rolling window hash. */
 const TOP = (() => { let power = 1; for (let index = 1; index < RUN_MIN_LINES; index++)
@@ -142,6 +151,31 @@ function textsOf(content) {
         return [];
     return content.filter(part => part?.type === "text" && typeof part.text === "string").map(part => part.text);
 }
+/** Whether `text` (a tool input) names a result file that `provider` saved instead of giving it to the model. */
+export function savedResultIn(text, provider) {
+    return provider !== undefined && SWAPPING_PROVIDERS.has(provider) && SAVED_RESULT.test(text);
+}
+/**
+ * The seen items of a branch, without a tool result the model did not get: one over the output limit of the provider
+ * of the assistant message that called it. `provider` stands in when no assistant message names one.
+ */
+export function seenItems(entries, provider) {
+    const items = [];
+    let caller = provider;
+    for (const entry of entries) {
+        const message = entry?.message;
+        if (message?.role === "assistant")
+            caller = typeof message.provider === "string" ? message.provider : provider;
+        const item = seenItem(entry);
+        if (!item)
+            continue;
+        if (message?.role === "toolResult" && caller !== undefined && SWAPPING_PROVIDERS.has(caller)
+            && item.texts.reduce((sum, text) => sum + text.length, 0) > SWAP_LIMIT_CHARS)
+            continue;
+        items.push(item);
+    }
+    return items;
+}
 /** The texts a session entry puts in context and where the agent saw them. Thinking, images, and summaries are not indexed. */
 export function seenItem(entry) {
     if (!entry || typeof entry !== "object")