@spendgraph/workflows 0.3.2 → 0.4.0

package/README.md

@@ -24,7 +24,7 @@ result.route; // which of the four it earned
 purpose: the prompt client pulls the stored wording and records what the call
 cost, and the model client makes it.
 
-## Four routes, chosen in code
+## Three routes, chosen in code
 
 Stage 0 is a router prompt that returns five signals — difficulty, stakes,
 confidence, ambiguity, tool need — and never a route. `routeFor` in
@@ -34,7 +34,6 @@ an eval run without touching a prompt.
 | route | what it runs | when |
 | --- | --- | --- |
 | `direct` | 6C alone | easy, unambiguous, already known |
-| `retrieve` | one tool call, then 6C | easy and unambiguous, but the answer is elsewhere |
 | `standard` | frame, split, plan, execute, check, close | everything in between |
 | `full` | standard plus four audits | high stakes, expert difficulty, or real ambiguity |
 
@@ -115,6 +114,7 @@ result.usage.inputTokens;              // the model side
 result.usage.byModel["claude-sonnet-4-6"];
 
 result.usage.tools.calls;              // the tool side
+result.usage.tools.failed;             // and what did not come back
 result.usage.tools.costMicros;         // where a tool priced itself
 result.usage.tools.byTool.web_search;  // tokens, keyed by the tool that ran
 ```
@@ -123,6 +123,12 @@ Bill both or you undercharge every run that touched the web: `web_search`
 reports Perplexity's four token counts and `deep_recall` reports micro-USD,
 and stage 4 records zero tokens of its own.
 
+## The signals
+
+Stage 0 estimates five numbers and never a route; [SIGNALS.md](./SIGNALS.md) is the
+prompt block that produces them, kept here because the authoritative copy is a
+versioned prompt on the server that nothing in this repository can gate.
+
 ## The ledger
 
 Every stage files its result, its rollouts and its tokens in one place. That
@@ -200,7 +206,7 @@ exercised against the server:
    produced its evidence.
 
 `examples/` runs each route against the real prompts for a few pennies. Start
-with `04-retrieve.mjs`: it is the cheapest, and it exercises triage, the
+with `05-standard.mjs`: it is the cheapest that looks anything up, and it exercises triage, the
 dispatcher, the ledger and 6C. The routes that look something up need one of
 the two lookups configured — `LOCUSGRAPH_AGENT_SECRET` for `deep_recall`, or
 `PERPLEXITY_API_KEY` for `web_search` — and say so rather than half-running.

package/SIGNALS.md

@@ -0,0 +1,104 @@
+# SIGNALS
+
+Stage 0 estimates five numbers about a request. `routeFor` turns them into a
+route. This file is the copy of that prompt block that lives in the repository —
+the authoritative one is a stored, versioned prompt on the spendgraph server
+(`triage-router-v1-81c2cc`), which nothing here can gate. A revision promoted
+there takes effect on every consumer without a deploy, so a change made and not
+recorded here is a change nobody can find later.
+
+## What each signal is for
+
+| signal | what it measures | who reads it |
+| --- | --- | --- |
+| `difficulty` | how hard the task is for a strong model | `routeFor` — hard and unsure buys `full` |
+| `stakes` | blast radius if the answer is wrong | `routeFor` — high or critical buys `full` and the red team |
+| `confidence` | whether the model already knows the answer | `routeFor` — under 0.7 leaves `direct` |
+| `ambiguity` | how underspecified the **wording** is | `routeFor` — 0.6 and over buys `full` |
+| `tool_need` | whether an external tool is required | `routeFor` — 0.3 and over leaves `direct` |
+
+The prompt is told it does not choose the route, and it never sees the tool bus.
+`tool_need` says a tool is wanted and never which one: stage 2 decides that, per
+sub-question, and stage 4 dispatches what it named.
+
+## The block, as promoted
+
+```
+Estimate the following five signals for the request. Judge each independently.
+
+1. difficulty — how hard the task is for a strong LLM with standard tools. Choose ONE: trivial | easy | medium | hard | expert | research
+
+2. stakes — blast radius if the answer is wrong: how costly the error is and how reversible. Judge by consequence, not by topic (the same arithmetic is low stakes in homework, critical inside a medication dose). Choose ONE:
+- low: an error is easily noticed and costless to redo
+- medium: an error wastes real time or money but is recoverable
+- high: an error causes significant harm, cost, or exposure (legal, medical, security, production infrastructure)
+- critical: an error is irreversible or endangers safety, funds, or a live incident
+
+3. confidence — probability from 0.00 to 1.00 that you already know enough to answer correctly without external information or extended reasoning. This is a calibrated estimate and will be scored against outcomes; do not inflate it.
+
+4. ambiguity — how underspecified the request is, from 0.00 to 1.00. Anchors: 0.00 = fully specified; 0.10 = minor wording issues only; 0.30 = one material unknown; 0.60 = a key referent cannot be resolved (location, entity, requirement); 0.90 = the intent itself is unclear.
+
+This signal measures the WORDING, never your access to the answer. Two consequences, and both affect ambiguity alone — judge confidence and tool_need exactly as you otherwise would:
+
+- A request is not ambiguous because you lack the data to answer it. If the wording is clear and the missing piece is information you would look up — the user's own notes, a repository, a live price — score 0.10 or below and let tool_need carry the gap. "What is my current workout split?" is fully specified.
+- A request about your own capabilities or about this conversation is fully specified even though it names no external task; score 0.10 or below. This does not extend to content the user has stored: what is in their knowledge base is a lookup like any other.
+
+5. tool_need — probability from 0.00 to 1.00 that external tools (search, retrieval, code execution, APIs, live data) are required for a correct answer.
+```
+
+## Why item 4 says all that
+
+Both clauses are there because a live run cost real money without them.
+
+**"What can you do for me?"** scored 0.90 ambiguity and took `full`: sixteen
+stages and 96k tokens to describe the assistant. The anchors read "names no
+task" as "intent unclear", and nothing distinguished the two.
+
+The first fix said a request about *"you, this assistant, or the conversation"*
+is fully specified. That was too wide. **"What is inside my knowledge base?"**
+then scored `tool_need: 0` and took `direct` — the model reported it already
+knew what the user had stored, and answered from nothing. Worse than expensive:
+wrong.
+
+**"What is my current workout split?"** scored 1.00 ambiguity and took `full`.
+The model could not resolve *which* split, so it called the question unclear —
+but the question is perfectly clear and the gap is data. That is `tool_need`,
+which it scored 0.90 at the same time. Every question about the user's own
+stored knowledge — the whole point of a knowledge-base product — was buying the
+most expensive route.
+
+Hence the framing the block now leads with: ambiguity is about the **wording**,
+and the two clauses say so explicitly rather than by example alone.
+
+## What it does not fix
+
+**"Which version of vitest is this repo pinned to?"** still scores ~0.70 and
+takes `full`. The model reads "this repo" as a referent it cannot resolve, which
+is fair when the question arrives with no session context. Inside a real chat it
+has that context. Tuning the wording until this one passes standalone would fit
+the prompt to a test rig rather than to the product.
+
+## Testing a revision
+
+`evals/dataset/routing.json` gates `routeFor`, not this prompt — every row
+supplies hand-labelled estimates, so a prompt change passes it unchanged. The
+only way to check a revision is to run it:
+
+```sh
+node examples/triage/01-triage.mjs          # QUESTION=… for your own
+```
+
+The five that matter, and what they should produce:
+
+| question | expected |
+| --- | --- |
+| What can you do for me? | ambiguity ≤ 0.10 → `direct` |
+| What is inside my knowledge base? | tool_need high → `standard` |
+| What is my current workout split? | ambiguity ≤ 0.10, tool_need high → `standard` |
+| Make our agent better. | ambiguity ~0.85 → `full` |
+| Should we migrate every production workload onto one provider? | `full` |
+
+The fourth is the control. It also names no task, and it must keep taking `full`
+— it names a task and withholds the success criterion, which is what 3B and 3C
+exist for. A revision that makes chit-chat cheap by making that one cheap too
+has broken the thing the long route is for.

package/dist/noesis/execute/spend.d.ts

@@ -4,12 +4,16 @@ import type { SubQuestionTool } from "../producers/decompose/index.js";
 export interface ToolSpend {
     /** Tool calls that ran, whether or not the tool priced itself. */
     calls: number;
+    /** Calls that failed or were skipped. Counted, because a summary that
+     *  reported only what worked would read as a healthy run. */
+    failed: number;
     /** Summed where a tool priced itself. Undefined where none did. */
     costMicros?: number;
     byTool: Record<string, ToolUsage>;
 }
 export interface ToolUsage {
     calls: number;
+    failed: number;
     model?: string;
     inputTokens: number;
     outputTokens: number;
@@ -25,7 +29,7 @@ export declare const NO_TOOL_SPEND: ToolSpend;
  * reports Perplexity's four token counts, `deep_recall` reports micro-USD. Both
  * are read where present and nothing is invented where absent.
  */
-export declare function readToolUsage(result: unknown): Omit<ToolUsage, "calls"> | null;
+export declare function readToolUsage(result: unknown): Omit<ToolUsage, "calls" | "failed"> | null;
 /**
  * Every tool call in one execution, summed under the tool that actually ran.
  *

package/dist/noesis/execute/spend.js

@@ -1,4 +1,4 @@
-export const NO_TOOL_SPEND = { calls: 0, byTool: {} };
+export const NO_TOOL_SPEND = { calls: 0, failed: 0, byTool: {} };
 const numberAt = (source, field) => {
     const value = source[field];
     return typeof value === "number" && Number.isFinite(value) ? value : 0;
@@ -24,23 +24,32 @@ export function readToolUsage(result) {
 export function toolSpendOf(answers, wiring = {}) {
     const byTool = {};
     let calls = 0;
+    let failed = 0;
     let costMicros;
+    const blank = () => ({
+        calls: 0,
+        failed: 0,
+        inputTokens: 0,
+        outputTokens: 0,
+        citationTokens: 0,
+        reasoningTokens: 0,
+    });
     for (const answer of answers) {
+        const named = wiring[answer.tool]?.name ?? answer.tool;
         if (answer.status === "answered")
             calls++;
+        else {
+            failed++;
+            byTool[named] = { ...(byTool[named] ?? blank()), failed: (byTool[named]?.failed ?? 0) + 1 };
+            continue;
+        }
         const one = readToolUsage(answer.result);
         if (!one)
             continue;
-        const named = wiring[answer.tool]?.name ?? answer.tool;
-        const prior = byTool[named] ?? {
-            calls: 0,
-            inputTokens: 0,
-            outputTokens: 0,
-            citationTokens: 0,
-            reasoningTokens: 0,
-        };
+        const prior = byTool[named] ?? blank();
         const merged = {
             calls: prior.calls + 1,
+            failed: prior.failed,
             inputTokens: prior.inputTokens + one.inputTokens,
             outputTokens: prior.outputTokens + one.outputTokens,
             citationTokens: prior.citationTokens + one.citationTokens,
@@ -53,7 +62,7 @@ export function toolSpendOf(answers, wiring = {}) {
         }
         byTool[named] = merged;
     }
-    return { calls, byTool, ...(costMicros !== undefined ? { costMicros } : {}) };
+    return { calls, failed, byTool, ...(costMicros !== undefined ? { costMicros } : {}) };
 }
 export function addToolSpend(a, b) {
     const byTool = { ...a.byTool };
@@ -62,6 +71,7 @@ export function addToolSpend(a, b) {
         byTool[tool] = prior
             ? {
                 calls: prior.calls + one.calls,
+                failed: prior.failed + one.failed,
                 inputTokens: prior.inputTokens + one.inputTokens,
                 outputTokens: prior.outputTokens + one.outputTokens,
                 citationTokens: prior.citationTokens + one.citationTokens,
@@ -76,7 +86,12 @@ export function addToolSpend(a, b) {
     const costMicros = a.costMicros !== undefined || b.costMicros !== undefined
         ? (a.costMicros ?? 0) + (b.costMicros ?? 0)
         : undefined;
-    return { calls: a.calls + b.calls, byTool, ...(costMicros !== undefined ? { costMicros } : {}) };
+    return {
+        calls: a.calls + b.calls,
+        failed: a.failed + b.failed,
+        byTool,
+        ...(costMicros !== undefined ? { costMicros } : {}),
+    };
 }
 export function recorded(execution) {
     return {

package/dist/noesis/index.d.ts

@@ -45,7 +45,7 @@ export { dropPoint, parseStage, resumeAt } from "./run/redo.js";
 export { STAGE_OF_SLUG } from "./slugs.js";
 export type { StageOptions, StageOutcome } from "./stage.js";
 export { runStage, StageReplyError } from "./stage.js";
-export type { Ask, AskedOfYou, NoesisOptions, NoesisResult, RetrieveOptions, Route, Routing, RunUsage, Status, VerifyOptions, } from "./types.js";
+export type { Ask, AskedOfYou, NoesisOptions, NoesisResult, Route, Routing, RunUsage, Status, VerifyOptions, } from "./types.js";
 export type { PartialUsage, Usage } from "./usage.js";
 export { addUsage, billableTokens, NO_USAGE, spend, usageOf } from "./usage.js";
 export type { Checked, Verification } from "./verify/index.js";

package/dist/noesis/route.d.ts

@@ -21,9 +21,8 @@ import type { Routing, TriageEstimates } from "./index.js";
  * exist for, and it takes `full` below the ambiguity bar that would catch it on
  * its own.
  *
- * `retrieve` splits the two reasons a clear question fails `direct`: high tool
- * need, and a confidence that only just misses. Both mean the model should not
- * answer from memory, and neither makes it a hard question — looking it up is
- * cheaper than planning it, and likelier to be right.
+ * A clear question that still needs a tool takes `standard`: stage 2 is the only
+ * stage that decides which tool a sub-question wants, and a route that skipped
+ * it had to guess. Guessing sent every world-fact lookup at the knowledge base.
  */
 export declare function routeFor(estimates: TriageEstimates): Routing;

package/dist/noesis/route.js

@@ -4,17 +4,12 @@ const CONFIDENT = 0.7;
 const NEEDS_TOOLS = 0.3;
 const VAGUE = 0.5;
 const UNSURE = 0.5;
-const NEARLY = 0.6;
 const HARD = new Set(["expert", "research"]);
 const EASY = new Set(["trivial", "easy"]);
 const SERIOUS = new Set(["high", "critical"]);
 export function routeFor(estimates) {
     const redTeam = SERIOUS.has(estimates.stakes);
     const clear = EASY.has(estimates.difficulty) && estimates.ambiguity < CLEAR;
-    const lookup = clear && estimates.tool_need >= NEEDS_TOOLS;
-    if (lookup && estimates.stakes !== "critical") {
-        return { route: "retrieve", redTeam, reason: `a lookup — tool_need is ${estimates.tool_need}` };
-    }
     if (redTeam) {
         return { route: "full", redTeam, reason: `stakes are ${estimates.stakes}` };
     }
@@ -32,11 +27,8 @@ export function routeFor(estimates) {
             reason: `ambiguity ${estimates.ambiguity} with confidence ${estimates.confidence}`,
         };
     }
-    if (clear && estimates.confidence >= CONFIDENT) {
+    if (clear && estimates.confidence >= CONFIDENT && estimates.tool_need < NEEDS_TOOLS) {
         return { route: "direct", redTeam, reason: `${estimates.difficulty} and already known` };
     }
-    if (clear && estimates.confidence >= NEARLY) {
-        return { route: "retrieve", redTeam, reason: `${estimates.difficulty} but not certain` };
-    }
     return { route: "standard", redTeam, reason: "neither trivial nor serious" };
 }

package/dist/noesis/run/run.js

@@ -2,7 +2,7 @@ import { route } from "@spendgraph/harness";
 import { emitter } from "../events.js";
 import { notBuilt } from "./errors.js";
 import { finish } from "./result.js";
-import { classifyStage, directStage, fullStage, retrieveStage, standardStage, } from "./stages/index.js";
+import { classifyStage, directStage, fullStage, standardStage, } from "./stages/index.js";
 import { newRunState } from "./state.js";
 const DEFAULT_MAX_RETRIES = 5;
 const watched = (opts, emit) => ({
@@ -32,7 +32,6 @@ export async function run(client, llm, question, options = {}) {
         classify: classifyStage(client, llm, question, opts, state),
         routes: {
             direct: directStage(client, llm, question, opts, state),
-            retrieve: retrieveStage(client, llm, question, opts, state),
             standard: standardStage(client, llm, question, opts, state),
             full: fullStage(client, llm, question, opts, state),
         },

package/dist/noesis/run/stages/index.d.ts

@@ -1,5 +1,4 @@
 export { classifyStage } from "./classify.js";
 export { directStage } from "./direct.js";
 export { fullStage } from "./full.js";
-export { retrieveStage } from "./retrieve.js";
 export { standardStage } from "./standard.js";

package/dist/noesis/run/stages/index.js

@@ -1,5 +1,4 @@
 export { classifyStage } from "./classify.js";
 export { directStage } from "./direct.js";
 export { fullStage } from "./full.js";
-export { retrieveStage } from "./retrieve.js";
 export { standardStage } from "./standard.js";

package/dist/noesis/types.d.ts

@@ -2,7 +2,6 @@ import type { ToolBus } from "@spendgraph/tools";
 import type { ToolSpend } from "./execute/spend.js";
 import type { Asked } from "./context.js";
 import type { NoesisEvent } from "./events.js";
-import type { SubQuestionTool } from "./producers/decompose/index.js";
 import type { AnswerOptions, AnswerResult } from "./producers/toulmin/index.js";
 import type { TriageEstimates, TriageInput, TriageOptions } from "./router/triage/index.js";
 import type { Escalation, StageKey, StageUsage } from "./run/ledger.js";
@@ -14,7 +13,7 @@ import type { Usage } from "./usage.js";
  * Derived in code from the triage estimates, which is what the router prompt
  * itself says: it returns signals and never a route.
  */
-export type Route = "direct" | "retrieve" | "standard" | "full";
+export type Route = "direct" | "standard" | "full";
 /** What the estimates decide, before any of it is acted on. */
 export interface Routing {
     route: Route;
@@ -39,16 +38,6 @@ export interface AskedOfYou {
     /** Why it says so. */
     reason: string;
 }
-export interface RetrieveOptions {
-    /**
-     * What a lookup reaches for. Default `"deeprecall"`.
-     *
-     * Triage says how much a question needs a tool and never which one, so the
-     * route picks. The enum is ordered cheapest first and the knowledge base
-     * comes before the open web, which is the same preference.
-     */
-    tool?: SubQuestionTool;
-}
 export interface VerifyOptions {
     /**
      * The confidence a blind check has to reach before it settles a claim.
@@ -108,7 +97,6 @@ export interface NoesisOptions {
      * person. Default 5, counted across every gate together.
      */
     maxRetries?: number;
-    retrieve?: RetrieveOptions;
     /** How the blind checks in stage 5 are read. */
     verify?: VerifyOptions;
     answer?: AnswerOptions;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@spendgraph/workflows",
-  "version": "0.3.2",
+  "version": "0.4.0",
   "description": "Ready-made workflows assembled from the spendgraph packages.",
   "license": "MIT",
   "repository": {
@@ -30,13 +30,14 @@
   },
   "files": [
     "dist",
+    "SIGNALS.md",
     "README.md"
   ],
   "dependencies": {
-    "@spendgraph/harness": "^0.3.2",
-    "@spendgraph/llms": "^0.3.2",
-    "@spendgraph/prompt": "^0.3.2",
-    "@spendgraph/tools": "^0.3.2"
+    "@spendgraph/harness": "^0.4.0",
+    "@spendgraph/llms": "^0.4.0",
+    "@spendgraph/prompt": "^0.4.0",
+    "@spendgraph/tools": "^0.4.0"
   },
   "scripts": {
     "build": "tsc -p tsconfig.json --emitDeclarationOnly && tsc -p tsconfig.json --declaration false --removeComments",
@@ -47,7 +48,7 @@
   },
   "devDependencies": {
     "@locusgraph/client": "^0.8.1",
-    "@spendgraph/evals": "^0.3.2",
+    "@spendgraph/evals": "^0.4.0",
     "typescript": "^5"
   },
   "engines": {