pi-t3-bridge 0.4.1 → 0.5.0

package/README.md

@@ -142,7 +142,13 @@ The bridge caps every shell command at 300 seconds, in the main thread and in ch
 
 ### Stop checks
 
-When a run is about to stop, the bridge asks a classifier how the run ended. The classifier is `typesafe/jev-latest` from your Pi model catalog. When the classifier says the agent gave up partway, the bridge adds a note and gives the agent one more turn, at most twice per prompt. Every 3 turns during a run, the bridge also asks the classifier whether the agent shows a known failure, and it adds a note when one is likely. The failures it checks for come from the MAST list of agent failures:
+When a run is about to stop, the bridge asks a classifier how the run ended. It picks the classifier in this order:
+
+1. The chat model named by `classifierModel` in the config, as `"provider/modelId"`.
+2. `typesafe/jev-latest`, when it is in your Pi model catalog and you have access to it.
+3. The model of the current session.
+
+A chat model gets the same questions through one tool call, and the bridge reads its answers from the tool arguments. A chat model's probabilities are its own estimates. They are not calibrated like Jev's, and the stop thresholds were fitted on Jev, so a chat model may nudge more or less often. If Jev is available but its call fails, the bridge does not fall back to a chat model. A `classifierModel` that is not in the catalog or has no auth is an error, and the bridge does not fall back from it either. When the classifier says the agent gave up partway, the bridge adds a note and gives the agent one more turn, at most twice per prompt. Every 3 turns during a run, the bridge also asks the classifier whether the agent shows a known failure, and it adds a note when one is likely. The failures it checks for come from the MAST list of agent failures:
 
 - Stuck in a loop
 - Doesn't know when to stop
@@ -161,6 +167,8 @@ on(run about to stop)
 
 The stop check and the notes come from [pi-mast-steer](https://github.com/dbreunig/pi-mast-steer). Set `steer` to `"shadow"` in the config to log verdicts without changing the session, or to `"off"` to turn the checks off.
 
+A chat-model judge costs tokens on your account, one call per stop check and one every 3 turns.
+
 ## The pstack extension
 
 The pstack extension adds the pstack skills to Pi's skill catalog, and it adds a note to each prompt that tells the agent when to enter `poteto-mode`. `poteto-mode` picks the workflow that fits your goal, for example a bug fix, a feature, a design review, or a PR watch. The skills come from [pstack](https://github.com/cursor/plugins/tree/main/pstack) by way of [t3-pstack](https://github.com/chhoumann/t3-pstack), and they live unchanged under `pstack/`.
@@ -223,6 +231,7 @@ The bridge reads one JSON file per machine, at `~/.config/pi-t3/config.json`. Ev
 {
   "bashTimeoutSeconds": 300,
   "childSandbox": "on",
+  "classifierModel": "anthropic/claude-sonnet-4-5",
   "mcpUrl": "http://127.0.0.1:3773/mcp",
   "steer": "on"
 }
@@ -232,6 +241,7 @@ The bridge reads one JSON file per machine, at `~/.config/pi-t3/config.json`. Ev
 | --- | --- | --- |
 | `bashTimeoutSeconds` | `300` | The longest a shell command can run, in seconds. |
 | `childSandbox` | `"on"` | Whether a delegated child's shell commands need a sandbox. With `"on"`, a child's `bash` is blocked when no sandbox can run. With `"off"`, commands run with no sandbox on every platform, and the child's report says `SANDBOX: off`. |
+| `classifierModel` | unset | A chat model that judges the stop checks, as `"provider/modelId"`. The model id may contain more slashes. When unset, the bridge uses `typesafe/jev-latest` if it is available, and the session model if not. A chat model's probabilities are its own estimates, not Jev-calibrated. |
 | `mcpUrl` | `http://127.0.0.1:3773/mcp` | The T3 MCP endpoint to use outside a T3 thread. |
 | `steer` | `"on"` | The stop checks, as `"on"`, `"shadow"`, or `"off"`. |
 
@@ -251,7 +261,7 @@ Earlier releases kept the state and the model sheet under `t3-pstack`. The first
 
 ## Data handling
 
-The package has no server and sends no telemetry. The stop checks send parts of the session transcript to the classifier provider in your Pi model catalog, and any file a skill asks the agent to read goes to your model provider. Scripts run on your machine, and the PR tools use your GitHub CLI login.
+The package has no server and sends no telemetry. The stop checks send parts of the session transcript to the classifier provider, which is Jev, the model you name in `classifierModel`, or your session model, and any file a skill asks the agent to read goes to your model provider. Scripts run on your machine, and the PR tools use your GitHub CLI login.
 
 ## Development
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-t3-bridge",
-  "version": "0.4.1",
+  "version": "0.5.0",
   "description": "Pi extension adapter for t3-pstack inside T3 Code.",
   "keywords": [
     "pi-extension",
@@ -85,7 +85,7 @@
     ]
   },
   "upstreams": {
-    "t3": "0.1.5-nightly.20261029.3877",
+    "t3": "0.1.7",
     "tstack": "1085050f3baed1ac724caa4a8166145fb2ac88af"
   }
 }

package/pstack/skills/create-verification-skill/SKILL.md

@@ -14,7 +14,7 @@ Answer these from the codebase and only ask the user what you cannot observe:
 
 - **Surface:** what does a user actually touch? A web UI, a CLI/TUI, a desktop app, an API, a mobile app, a library? A repo can have several; pick the primary one and note the rest.
 - **Run:** how does the app start locally? Prefer the repo's own documented dev command (package scripts, Makefile, README quickstart). Note ports, env vars, seed data, auth.
-- **Drive:** how can an agent interact with it programmatically? Existing harnesses first — Playwright/Cypress specs, expect scripts, PTY helpers, curl-able endpoints, a debug port. Only then pick a generic recipe: browser/CDP for web and Electron, a tmux/PTY harness for CLI/TUI, plain HTTP for services.
+- **Drive:** how can an agent interact with it programmatically? Existing harnesses first — browser test specs, expect scripts, PTY helpers, curl-able endpoints, a debug port. Only then pick a generic recipe: browser/CDP for web and Electron, a tmux/PTY harness for CLI/TUI, plain HTTP for services.
 - **Observe:** what evidence can be captured? Screenshots, terminal transcripts, response bodies, logs, exit codes, DB state.
 - **Isolate:** can two instances run side by side (ports, data dirs, profiles)? If not, say so in the generated skill: refusing to double-drive a shared instance beats corrupting the user's session.
 

package/pstack/skills/setup-pstack/scripts/t3-sheet.mjs

@@ -143,13 +143,22 @@ function checkTarget({ providerInstanceId, model, options }, catalog) {
   const problems = [];
   for (const [id, value] of Object.entries(options)) {
     const option = known.find((o) => o.id === id);
-    const allowed = option?.type === "boolean" ? ["true", "false"] : (option?.options ?? []).map((o) => o.id);
+    const allowed = allowedValues(option);
     if (!option) problems.push(`${at}: no option "${id}" (has ${known.map((o) => o.id).join(", ") || "none"})`);
     else if (!allowed.includes(value)) problems.push(`${at}: ${id}=${value} is not one of ${allowed.join(", ")}`);
   }
   return problems;
 }
 
+// orchestrator_capabilities lists an option's values as a plain array, with
+// booleans as JSON booleans. The `type` and `options: [{ id }]` shape is
+// still read for catalogs saved in that form.
+function allowedValues(option) {
+  if (Array.isArray(option?.values)) return option.values.map(String);
+  if (option?.type === "boolean") return ["true", "false"];
+  return (option?.options ?? []).map((o) => o.id);
+}
+
 const invoked = process.argv[1] && existsSync(process.argv[1]) ? realpathSync(process.argv[1]) : null;
 if (invoked === fileURLToPath(import.meta.url)) {
   const [sheetPath, catalogPath] = process.argv.slice(2);

package/src/bridge/config.ts

@@ -25,6 +25,7 @@ const ChildSandboxSchema = Type.Union([
 export interface Config {
   readonly bashTimeoutSeconds: number | undefined;
   readonly childSandbox: Static<typeof ChildSandboxSchema> | undefined;
+  readonly classifierModel: string | undefined;
   readonly mcpUrl: string | undefined;
   readonly steer: Static<typeof SteerSchema> | undefined;
 }
@@ -37,6 +38,7 @@ interface Loaded {
 const EMPTY: Config = {
   bashTimeoutSeconds: undefined,
   childSandbox: undefined,
+  classifierModel: undefined,
   mcpUrl: undefined,
   steer: undefined,
 };
@@ -88,6 +90,10 @@ const parse = (text: string): Loaded => {
         Type.Number({ exclusiveMinimum: 0 })
       ),
       childSandbox: field("childSandbox", ChildSandboxSchema),
+      classifierModel: field(
+        "classifierModel",
+        Type.String({ pattern: "^[^/\\s]+/.+$" })
+      ),
       mcpUrl: field("mcpUrl", Type.String({ minLength: 1 })),
       steer: field("steer", SteerSchema),
     },

package/src/bridge/steer-judge.ts

@@ -2,21 +2,25 @@ import { readFileSync } from "node:fs";
 import path from "node:path";
 
 import type {
+  Api,
   ClassifierBoolQuestion,
   ClassifierContext,
+  ClassifierQuestion,
   JsonObject,
+  Model,
 } from "@earendil-works/pi-ai";
 import type { ExtensionContext } from "@earendil-works/pi-coding-agent";
 import { Type } from "typebox";
-import type { Static } from "typebox";
+import type { Static, TSchema } from "typebox";
 import { Value } from "typebox/value";
 
+import { readConfig } from "./config.ts";
 import type { MastExcerpt, StopInput } from "./steer-transcript.ts";
 
-// Asks TypeSafe's Jev how a run ended. The classifier call follows
-// pi-mast-steer's src/judge.ts (MIT, github.com/dbreunig/pi-mast-steer): any
-// failure becomes an error value, so a judge that is down never blocks a
-// session.
+// Asks TypeSafe's Jev how a run ended, or a chat model when Jev is not
+// available. The classifier call follows pi-mast-steer's src/judge.ts (MIT,
+// github.com/dbreunig/pi-mast-steer): any failure becomes an error value, so a
+// judge that is down never blocks a session.
 
 const StopKindSchema = Type.Union([
   Type.Literal("done"),
@@ -92,27 +96,243 @@ const readQuestions = (): Static<typeof QuestionsFileSchema> => {
 // Fitted on the user's own labelled stops. See the stop-steer outline, phase 6.
 export const stopThresholds = () => readQuestions().thresholds;
 
-// Jev's failures are values. Nothing here throws on a classifier that is down.
-const askJev = async (
+type Asked = { readonly answers: unknown } | { readonly error: string };
+
+const ANSWER_TOOL = "classifier_answers";
+
+// One flat argument per asked question: a bool is a probability, a choice is
+// one of its criteria keys. No caller asks a score question.
+const answerSchema = (question: ClassifierQuestion): TSchema => {
+  switch (question.type) {
+    case "bool": {
+      return Type.Number({ maximum: 1, minimum: 0 });
+    }
+
+    case "choice": {
+      return Type.Union(
+        Object.keys(question.criteria).map((key) => Type.Literal(key))
+      );
+    }
+
+    default: {
+      return Type.Never();
+    }
+  }
+};
+
+const chatPrompt = (context: ClassifierContext): string =>
+  [
+    "Judge the state below by answering each question.",
+    `Call ${ANSWER_TOOL} exactly once, with one value for every question.`,
+    'For a bool question, give the probability from 0 to 1 that its "true" criteria holds.',
+    "For a choice question, give the key of the criteria that fits best.",
+    "",
+    "Questions, as JSON:",
+    JSON.stringify(context.questions),
+    "",
+    "State, as JSON:",
+    JSON.stringify(context.state),
+  ].join("\n");
+
+type FlatAnswer =
+  | { readonly probability: number; readonly type: "bool" }
+  | { readonly choice: string; readonly type: "choice" };
+
+// Parses one flat value per question, checked against that question's own
+// schema, and turns it into the answer shape a classifier returns.
+const parseAnswers = <S extends TSchema>(
+  schema: S,
+  questions: ClassifierContext["questions"],
+  candidate: Static<S>
+): Record<string, FlatAnswer> | undefined => {
+  if (!Value.Check(schema, candidate)) {
+    return undefined;
+  }
+
+  if (Object.keys(candidate).length !== Object.keys(questions).length) {
+    return undefined;
+  }
+
+  const answers: Record<string, FlatAnswer> = {};
+
+  for (const [name, question] of Object.entries(questions)) {
+    const value = candidate[name];
+
+    switch (question.type) {
+      case "bool": {
+        if (!Value.Check(Type.Number({ maximum: 1, minimum: 0 }), value)) {
+          return undefined;
+        }
+
+        answers[name] = { probability: value, type: "bool" };
+        break;
+      }
+
+      case "choice": {
+        if (
+          !Value.Check(Type.String(), value) ||
+          !Value.Check(answerSchema(question), value)
+        ) {
+          return undefined;
+        }
+
+        answers[name] = { choice: value, type: "choice" };
+        break;
+      }
+
+      default: {
+        return undefined;
+      }
+    }
+  }
+
+  return answers;
+};
+
+const askChat = async (
   ctx: ExtensionContext,
+  model: Model<Api>,
   context: ClassifierContext,
   signal: AbortSignal
-): Promise<{ readonly answers: unknown } | { readonly error: string }> => {
+): Promise<Asked> => {
+  const schema = Type.Object(
+    Object.fromEntries(
+      Object.entries(context.questions).map(([name, question]) => [
+        name,
+        answerSchema(question),
+      ])
+    ),
+    { additionalProperties: false }
+  );
+
+  try {
+    const reply = await ctx.modelRegistry
+      .streamSimple(
+        model,
+        {
+          messages: [
+            {
+              content: chatPrompt(context),
+              role: "user",
+              timestamp: Date.now(),
+            },
+          ],
+          tools: [
+            {
+              constrainedSampling: { strict: "prefer", type: "json_schema" },
+              description: "Record your answer to every question.",
+              name: ANSWER_TOOL,
+              parameters: schema,
+            },
+          ],
+        },
+        { signal }
+      )
+      .result();
+
+    if (reply.stopReason !== "stop" && reply.stopReason !== "toolUse") {
+      return { error: reply.errorMessage ?? reply.stopReason };
+    }
+
+    const call = reply.content.find(
+      (part) => part.type === "toolCall" && part.name === ANSWER_TOOL
+    );
+
+    const answers =
+      call?.type === "toolCall"
+        ? parseAnswers(schema, context.questions, call.arguments)
+        : undefined;
+
+    if (answers === undefined) {
+      return {
+        error: `${model.provider}/${model.id} did not answer through ${ANSWER_TOOL}`,
+      };
+    }
+
+    return { answers };
+  } catch (error) {
+    return {
+      error: error instanceof Error ? error.message : String(error),
+    };
+  }
+};
+
+// The judge is classifierModel when set, else Jev when it is available, else
+// the session model. Failures are values. Nothing here throws on a judge that
+// is down.
+const askClassifier = async (
+  ctx: ExtensionContext,
+  context: ClassifierContext,
+  signal: AbortSignal
+): Promise<Asked> => {
+  const { classifierModel } = readConfig();
+
+  if (classifierModel !== undefined) {
+    const slash = classifierModel.indexOf("/");
+
+    const chosen = ctx.modelRegistry.find(
+      classifierModel.slice(0, slash),
+      classifierModel.slice(slash + 1)
+    );
+
+    if (chosen === undefined) {
+      return {
+        error: `classifierModel "${classifierModel}" is not in the model catalog`,
+      };
+    }
+
+    if (!ctx.modelRegistry.hasConfiguredAuth(chosen)) {
+      return {
+        error: `classifierModel "${classifierModel}" has no configured auth`,
+      };
+    }
+
+    return askChat(ctx, chosen, context, signal);
+  }
+
   const jev = ctx.modelRegistry.findOfType(
     "classifier",
     "typesafe",
     "jev-latest"
   );
 
-  if (jev === undefined) {
-    return { error: "typesafe/jev-latest is not in the model catalog" };
+  let available: Awaited<
+    ReturnType<typeof ctx.modelRegistry.getAvailableOfType>
+  >;
+
+  try {
+    available = await ctx.modelRegistry.getAvailableOfType(
+      "classifier",
+      "typesafe",
+      { signal }
+    );
+  } catch (error) {
+    return {
+      error: error instanceof Error ? error.message : String(error),
+    };
   }
 
-  const result = await ctx.modelRegistry.classify(jev, context, { signal });
+  if (
+    jev !== undefined &&
+    available.some(
+      (model) => model.provider === jev.provider && model.id === jev.id
+    )
+  ) {
+    const result = await ctx.modelRegistry.classify(jev, context, { signal });
+
+    return result.stopReason === "stop"
+      ? { answers: result.answers }
+      : { error: result.errorMessage ?? result.stopReason };
+  }
 
-  return result.stopReason === "stop"
-    ? { answers: result.answers }
-    : { error: result.errorMessage ?? result.stopReason };
+  if (ctx.model !== undefined) {
+    return askChat(ctx, ctx.model, context, signal);
+  }
+
+  return {
+    error:
+      "typesafe/jev-latest is not available and the session has no model to judge with",
+  };
 };
 
 export const judgeStop = async (
@@ -144,7 +364,7 @@ export const judgeStop = async (
     asked.handoff_missing = questions.handoff_missing;
   }
 
-  const result = await askJev(ctx, { questions: asked, state }, signal);
+  const result = await askClassifier(ctx, { questions: asked, state }, signal);
 
   if ("error" in result) {
     return result;
@@ -230,7 +450,7 @@ export const judgeMast = async (
 ): Promise<MastVerdict> => {
   const mast = readMast();
 
-  const result = await askJev(
+  const result = await askClassifier(
     ctx,
     {
       // Pi sends `bool` questions to TypeSafe as `noul`, the type DSPy sends.