@spendgraph/harness 0.1.0 → 0.2.0

package/docs/refine.md

@@ -0,0 +1,206 @@
+# refine
+
+Draft, judge, revise. Stop when it is good enough, or when the budget is out.
+
+```ts
+import { refine } from "@spendgraph/harness";
+
+const result = await refine({
+  rounds: 3,
+  maxTokens: 20_000,
+  attempt: (feedback) =>
+    llm.call([{ role: "user", content: feedback ? `${task}\n\nFix this: ${feedback}` : task }]),
+  judge: (attempt) =>
+    attempt.output.length <= 160
+      ? { accepted: true }
+      : { accepted: false, feedback: `That is ${attempt.output.length} characters. The limit is 160.` },
+});
+
+result.output;     // the winning draft
+result.accepted;   // false when the rounds or the tokens ran out
+result.stoppedBy;  // accepted · rounds · tokens · failed
+result.steps;      // every attempt and every judgement
+```
+
+One `attempt` callback rather than separate generate and revise, because in
+practice they are the same prompt with one extra paragraph — `feedback` is null
+on the first round and the judge's note after.
+
+---
+
+## Example: an invoice that has to add up
+
+The judge is plain code, which is the case worth reaching for. It costs nothing,
+it is deterministic, and it cannot be talked into agreeing.
+
+```ts
+const result = await refine({
+  rounds: 3,
+
+  attempt: (feedback) =>
+    llm.call([{ role: "user", content: `${email}${feedback ?? ""}` }], { schema: INVOICE }),
+
+  judge: (attempt) => {
+    const invoice = JSON.parse(attempt.output);
+    const summed = invoice.lines.reduce((n, l) => n + l.amount, 0);
+
+    return summed === invoice.total
+      ? { accepted: true }
+      : {
+          accepted: false,
+          feedback: `\n\nThe lines add up to ${summed} but the total says ${invoice.total}. A line is missing.`,
+        };
+  },
+});
+```
+
+What happens:
+
+```
+  round 0   → { lines: [1200, 50], total: 1550 }        ✗ lines sum to 1250
+              feedback: "The lines add up to 1250 but the total says 1550…"
+  round 1   → { lines: [1200, 50, 300], total: 1550 }   ✓ accepted
+
+  accepted: true   rounds: 2   steps: 4
+```
+
+The model dropped the shipping line. Told exactly what was inconsistent, it
+found it. A **retry** would have run the same prompt and often made the same
+mistake; the feedback is the difference.
+
+A model judge asked "is this extracted correctly?" would have said yes, because
+the JSON looks fine. Arithmetic does not have that problem.
+
+---
+
+## More judges that cost nothing
+
+**SQL that has to run.** `EXPLAIN` is free and the database's own error is the
+best feedback there is.
+
+```ts
+judge: async (attempt) => {
+  try {
+    await db.prepare(`EXPLAIN ${attempt.output}`).run();
+    return { accepted: true };
+  } catch (err) {
+    return { accepted: false, feedback: String(err.message) };
+  }
+}
+```
+
+Hand the error back verbatim. Rewriting it into something friendlier throws away
+the column name that tells the model what to fix.
+
+**Code that has to pass its tests.** The strongest judge there is, because it is
+the same bar a person would apply. Slow, so keep `rounds` low — three failed
+attempts at a minute each is three minutes before you learn the prompt is wrong.
+
+```ts
+judge: async (attempt) => {
+  const { failed, output } = await runTests(attempt.output);
+  return failed === 0 ? { accepted: true } : { accepted: false, feedback: output };
+}
+```
+
+**Placeholders that must survive a translation.** Models translate `{count}`
+about one time in fifty, and it fails at runtime for whoever speaks that
+language.
+
+```ts
+judge: (attempt) => {
+  const wanted = source.match(/\{[a-z_]+\}/g) ?? [];
+  const missing = wanted.filter((token) => !attempt.output.includes(token));
+  return missing.length === 0
+    ? { accepted: true }
+    : { accepted: false, feedback: `Keep these exactly as they are: ${missing.join(", ")}.` };
+}
+```
+
+---
+
+## When the judge has to be a model
+
+Only when no code can decide — tone, policy, "would a person find this rude".
+It is a second model call every round, so the loop is now at least twice the
+price of the answer.
+
+```ts
+judge: async (attempt) => {
+  const verdict = await judgeLlm.call([{ role: "user", content: `${RUBRIC}\n\n${attempt.output}` }], {
+    schema: {
+      type: "object",
+      properties: { score: { type: "number" }, fix: { type: "string" } },
+      required: ["score"],
+    },
+  });
+
+  const { score, fix } = verdict.data;
+  return score >= 8 ? { accepted: true, score } : { accepted: false, score, feedback: fix };
+}
+```
+
+Always return `score`, even on acceptance. Without it, a run that never passes
+hands back the **last** attempt rather than the best one.
+
+---
+
+## When not to use it
+
+**You cannot state the bar.** If the judge accepts everything you have tripled
+the cost to learn nothing. Write the check first; if you cannot, the pattern does
+not apply.
+
+**Round one already passes almost always.** At 99% you are paying a judge on
+every request to catch the hundredth. Retry-on-failure is cheaper than a loop by
+default.
+
+**The failure is not fixable by trying again.** A missing document, a permission
+the agent does not have, a question with no answer — feedback cannot help, and
+three rounds is three ways of saying so. Escalate instead.
+
+---
+
+## Reading the result
+
+`stoppedBy` is the field to watch in production.
+
+| | What it means |
+| --- | --- |
+| `accepted` | working as intended |
+| `rounds` | the judge is too strict, or the prompt cannot get there |
+| `tokens` | the ceiling is too low, or the attempts are too big |
+| `failed` | the provider failed; not a quality problem |
+
+A rising share of `rounds` is the signal to fix the prompt rather than raise the
+limit. `history` carries every attempt with its verdict and token count, so "does
+refining pay on this task?" is a query — round-one acceptance rate against the
+cost of the extra calls — rather than a hunch.
+
+**A failed attempt ends the run** rather than being judged. Asking a judge what
+it thinks of an error spends a call to be told it is bad.
+
+**The ceiling is in tokens, not money**, because this package cannot price a
+call — that happens on the server, after the fact. It is checked *before* each
+round, so it is never crossed rather than merely noticed once it has been.
+
+---
+
+## How the files fit
+
+```
+   types.ts       the contract
+        ▲
+   attempt.ts     what a draft cost, and how it is recorded
+   judge.ts       how a verdict is recorded, and which draft wins
+        ▲
+   refine.ts      the loop: draft · judge · stop
+```
+
+`attempt.ts` and `judge.ts` are siblings — neither knows about the other, and
+both import only the contract. The loop is the only file that knows the order
+they run in.
+
+`best` lives with the judge rather than the loop because the choice is the
+judge's: it is made on the verdicts, and a judge that never scores gets the
+fallback to the last attempt.

package/docs/route.md

@@ -0,0 +1,212 @@
+# route
+
+Classify once, then hand the work to the branch built for it.
+
+```ts
+import { route } from "@spendgraph/harness";
+
+const result = await route(
+  { subject, body },
+  {
+    classify: ({ subject }) => (/refund|invoice/i.test(subject) ? "billing" : "technical"),
+    routes: {
+      billing: (input) => haiku.bind(billingPrompt).invoke(input),
+      technical: (input) => opus.bind(docsPrompt).invoke(input),
+    },
+    fallback: (input) => opus.bind(generalPrompt).invoke(input),
+  }
+);
+
+result.route;        // "billing", or null when the fallback ran
+result.classified;   // what the classifier said, used or not
+result.inputTokens;  // the classifier's tokens plus the branch's
+```
+
+---
+
+## Example: support triage, routed for cost
+
+Most tickets are simple. Only a few need the expensive model.
+
+```ts
+const result = await route(
+  { subject, body },
+  {
+    classify: ({ subject, body }) => {
+      const text = `${subject} ${body}`.toLowerCase();
+      if (/refund|charged|invoice/.test(text)) return "billing";
+      if (/error|crash|not working/.test(text)) return "technical";
+      return "general";
+    },
+
+    routes: {
+      billing: (input) => haiku.bind(billingPrompt).invoke(input),
+      technical: (input) => opus.bind(docsPrompt).invoke(input),
+      general: () => ({ output: "Thanks — a human will reply within a day.", model: "none" }),
+    },
+
+    fallback: (input) => opus.bind(generalPrompt).invoke(input),
+  }
+);
+```
+
+```
+  result.route        "billing"
+  result.output       "Your 12 March charge was a duplicate; it's been refunded."
+  result.inputTokens  340        classifier (0) + branch
+  result.steps        [ classify, billing ]
+```
+
+Three things this is doing:
+
+The **classifier is a regex** — free, instant, and it cannot invent a category.
+That is the common case.
+
+The **`general` branch calls no model at all.** It returns a canned line at zero
+cost. If a third of your tickets land there, that third is now free, which no
+amount of prompt tuning gets you.
+
+**`billing` uses the small model, `technical` the big one.** That is the whole
+cost argument: you stop paying opus rates for "where is my invoice".
+
+---
+
+## Example: a classifier that is a model
+
+When no regex will ever be right — sentiment, intent, "is this actually a
+complaint" — use a model, and constrain it with an enum so it can only answer
+with a route you have.
+
+```ts
+classify: async ({ subject, body }) => {
+  const reply = await haiku.call(
+    [{ role: "user", content: `Classify this ticket.\n\n${subject}\n${body}` }],
+    {
+      schema: {
+        type: "object",
+        properties: {
+          route: { type: "string", enum: ["billing", "technical", "general"] },
+          confidence: { type: "number" },
+          reason: { type: "string" },
+        },
+        required: ["route", "confidence"],
+      },
+    }
+  );
+
+  return {
+    ...reply.data,
+    model: reply.model,
+    inputTokens: reply.inputTokens,
+    outputTokens: reply.outputTokens,
+  };
+},
+minConfidence: 0.7,
+```
+
+The `enum` is doing real work. A free-text classifier eventually answers
+`"Billing"` or `"billing question"` or a whole sentence, and every one of those
+falls through to the fallback silently. Constrained, it can only say one of
+three things — and on Anthropic that is a forced tool call, which is a guarantee
+rather than a request.
+
+Returning `model` and the token counts is what keeps the cost honest: a
+classifier running on every request is not free, and without them
+`result.inputTokens` reports only the branch.
+
+**Self-reported `confidence` is weak.** Models are overconfident and the number
+is not calibrated. `minConfidence` is a blunt safety net for the genuinely
+ambiguous cases, not a dial to tune to three decimal places.
+
+**Check the arithmetic before reaching for it.** A haiku classify might be 300
+in, 20 out. Routing 80% of traffic from opus to haiku pays for itself many times
+over. If every branch is a cheap model anyway, you have added a call to every
+request to save nothing.
+
+---
+
+## What else it is for
+
+**Capability.** A refund branch gets a bus with `escalate` pinned and a payments
+tool; a docs branch gets retrieval and no write access. You cannot express that
+in one call, because the tools are chosen before you know what was asked.
+
+**Quality.** One prompt covering billing, technical and legal has to carry
+instructions for all three, and every instruction is a chance to apply the wrong
+one. Three prompts each do one thing.
+
+**Safety.** Classify first; if it is out of scope or harmful, route to a branch
+that declines and hands off. More reliable than hoping one prompt's guardrails
+hold, because the decision is a separate call you can test.
+
+Beyond support, the same shape: document type → invoice, contract or receipt,
+each with its own extraction schema. Language detection → a localised prompt.
+Complexity → a template for the trivial, an agent loop for the rest.
+
+---
+
+## When not to use it
+
+**The branches would share 90% of their prompt.** You have split one thing into
+three that now drift apart.
+
+**The classification is genuinely hard.** You have moved the failure earlier
+without removing it — a confidently wrong route sends the question to a
+specialist with no idea what to do with it. `minConfidence` and a required
+`fallback` exist for exactly this, but they are a net, not a fix.
+
+---
+
+## Reading the result
+
+**Watch `route === null`.** A rising share of fallbacks means either the
+classifier is drifting or there is a kind of request you never built a branch
+for — and since `classified` records what the classifier *said* separately from
+what ran, you can tell which.
+
+```
+  classifier says "legal", no such branch    →  classified "legal"    route null
+  classifier says "billing" at 0.4, bar 0.7  →  classified "billing"  route null
+```
+
+Same `route`, different `classified`: the first is a **missing branch**, the
+second a **miscalibrated** one, and they want opposite fixes.
+
+**`fallback` is required.** A router that can fail to route fails in production
+at 3am, and "unknown" is a class every classifier eventually returns.
+
+`route` and `refine` compose: a branch can be a refine loop, and its rounds
+appear as `billing.attempt_0`, `billing.judge_0` rather than being flattened
+into one step.
+
+---
+
+## How the files fit
+
+```
+   types.ts        the contract
+        ▲
+   classify.ts     what the classifier said, and what runs because of it
+        ▲
+   dispatch.ts     where the work goes: a node per branch, and the edges to them
+        ▲
+   route.ts        compile · run · shape the result
+```
+
+Each file imports only downwards, which is why `decide` can be tested on its own
+with no graph in sight.
+
+It compiles to a graph rather than looping here. `@spendgraph/graph` already
+validates that every branch is reachable and that no default sits in front of
+one, already records a step per node with its tokens, and already returns the
+rollout shape. A second execution engine in this package would be a second thing
+to keep right.
+
+**The decision travels in the graph, not in a closure.** The classifier
+*returns* its decision, so it lands in `outputs.classify` like any node's return
+and the edges read it from the context they are already handed. A finished run
+can be read apart from `outputs.classify` alone.
+
+**The edge order matters.** Conditional edges first, the unconditional fallback
+last — `graph` refuses the other order, and is right to: an unconditional edge
+ahead of a conditional one means the branch behind it can never be reached.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@spendgraph/harness",
-  "version": "0.1.0",
-  "description": "Pull a stored prompt, render it, report what it cost. Zero dependencies.",
+  "version": "0.2.0",
+  "description": "Workflows: the shapes an LLM app takes, built on prompts, tools and graphs.",
   "license": "MIT",
   "repository": {
     "type": "git",
@@ -10,10 +10,11 @@
   },
   "keywords": [
     "llm",
-    "prompt",
-    "prompt-management",
-    "cost",
-    "evals"
+    "agent",
+    "workflow",
+    "orchestration",
+    "routing",
+    "evaluator"
   ],
   "type": "module",
   "main": "./dist/index.js",
@@ -23,22 +24,22 @@
       "types": "./dist/index.d.ts",
       "import": "./dist/index.js"
     },
-    "./core": {
-      "types": "./dist/core/index.d.ts",
-      "import": "./dist/core/index.js"
-    },
     "./package.json": "./package.json"
   },
   "files": [
-    "bin",
     "dist",
+    "docs",
     "README.md"
   ],
   "scripts": {
     "build": "tsc -p tsconfig.json",
-    "test": "npm run build && vitest run"
+    "prebuild": "npm run build --workspace @spendgraph/graph --workspace @spendgraph/tools --workspace @spendgraph/prompt",
+    "test": "vitest run",
+    "pretest": "npm run build --workspace @spendgraph/graph --workspace @spendgraph/prompt --workspace @spendgraph/tools --workspace @spendgraph/llms",
+    "examples": "npm run build && node examples/run-all.mjs"
   },
   "devDependencies": {
+    "@spendgraph/llms": "^0.2.0",
     "typescript": "^5"
   },
   "engines": {
@@ -47,7 +48,10 @@
   "publishConfig": {
     "access": "public"
   },
-  "bin": {
-    "harness": "bin/harness.mjs"
+  "dependencies": {
+    "@spendgraph/graph": "^0.2.0",
+    "@spendgraph/prompt": "^0.2.0",
+    "@spendgraph/sdk": "^0.2.0",
+    "@spendgraph/tools": "^0.2.0"
   }
 }

package/bin/harness.mjs

@@ -1,76 +0,0 @@
-#!/usr/bin/env node
-/**
- * harness codegen — writes a .d.ts describing every stored prompt's fields.
- *
- *   SPENDGRAPH_API_KEY=sg_… SPENDGRAPH_URL=https://costs.example.com \
- *     npx harness codegen --out src/harness.generated.d.ts
- *
- * Reads through the paginated list endpoint, so a project with hundreds of
- * prompts is many small requests rather than one enormous response.
- */
-import { writeFile, mkdir } from "node:fs/promises";
-import { dirname, resolve } from "node:path";
-import { emitPromptTypes } from "../dist/index.js";
-
-const args = process.argv.slice(2);
-const command = args[0];
-
-function flag(name, fallback) {
-  const i = args.indexOf(`--${name}`);
-  return i >= 0 && args[i + 1] ? args[i + 1] : fallback;
-}
-
-function fail(message) {
-  console.error(`harness: ${message}`);
-  process.exit(1);
-}
-
-if (command !== "codegen") {
-  console.log("usage: harness codegen [--out <file>] [--project <id>] [--module <specifier>]");
-  process.exit(command ? 1 : 0);
-}
-
-const apiKey = process.env.SPENDGRAPH_API_KEY;
-const baseUrl = (process.env.SPENDGRAPH_URL ?? flag("url", "")).replace(/\/+$/, "");
-if (!apiKey) fail("set SPENDGRAPH_API_KEY");
-if (!baseUrl) fail("set SPENDGRAPH_URL, or pass --url");
-
-const out = resolve(flag("out", "src/harness.generated.d.ts"));
-const project = flag("project", "");
-const moduleName = flag("module", "@spendgraph/harness");
-
-const prompts = [];
-let cursor = null;
-
-// Bounded: a server that kept handing back a cursor must not spin forever.
-for (let page = 0; page < 200; page++) {
-  // Archived prompts included on purpose. The dashboard hides them, but a
-  // generated type disappearing because somebody tidied a list would break the
-  // build of code that still pulls the prompt — archiving does not stop it
-  // serving, so it must not stop it type-checking either.
-  const params = new URLSearchParams({ limit: "200", archived: "all" });
-  if (project) params.set("project", project);
-  if (cursor) params.set("cursor", cursor);
-
-  const res = await fetch(`${baseUrl}/api/v1/prompts?${params}`, {
-    headers: { "x-api-key": apiKey },
-  });
-  if (!res.ok) {
-    const body = await res.text().catch(() => "");
-    fail(`${res.status} listing prompts — ${body.slice(0, 200)}`);
-  }
-
-  const body = await res.json();
-  prompts.push(...(body.prompts ?? []));
-  cursor = body.nextCursor;
-  if (!cursor) break;
-}
-
-await mkdir(dirname(out), { recursive: true });
-await writeFile(out, emitPromptTypes(prompts, { module: moduleName }));
-
-const named = prompts.filter((p) => (p.fieldSpec ?? []).some((f) => f.required)).length;
-console.log(
-  `harness: wrote ${prompts.length} prompt${prompts.length === 1 ? "" : "s"} to ${out}` +
-    (named ? ` (${named} with required fields)` : "")
-);

package/dist/core/budget.d.ts

@@ -1,46 +0,0 @@
-/**
- * Thrown when a client has spent its ceiling.
- *
- * Deliberately not a `HarnessError`: nothing was refused by the server, so a
- * caller inspecting `status` or `retryable` would be told a story about HTTP
- * that never happened.
- */
-export declare class BudgetExceededError extends Error {
-    readonly spentMicros: number;
-    readonly limitMicros: number;
-    constructor(spentMicros: number, limitMicros: number);
-}
-/**
- * What this client has spent, and whether it may spend more.
- *
- * A loop that samples k rollouts across a dataset is the shape of an expensive
- * accident: one bad `k`, one dataset larger than expected, and the bill arrives
- * before anyone notices. The guard is a counter and a comparison, and a cost
- * meter that shipped without one would be embarrassing.
- *
- * Governs spending this client *initiates* — `run`, `sample`, `runAll`. It does
- * not gate `report`, which records money the caller already spent elsewhere;
- * counting that would make the ceiling mean two different things at once.
- */
-export declare class Budget {
-    private used;
-    private readonly limit;
-    /**
-     * `undefined` means no ceiling and warns once — an unbounded client in a
-     * scheduled script is how a surprise bill happens. `null` means the same
-     * thing deliberately, and says nothing.
-     */
-    constructor(maxCostMicros?: number | null);
-    spent(): number;
-    remaining(): number;
-    add(costMicros: number): void;
-    /**
-     * Throws if the ceiling is already reached.
-     *
-     * Checked before a request rather than after: stopping once the bill has been
-     * incurred is a report, not a limit.
-     */
-    assertAffordable(): void;
-}
-/** Test seam: lets the once-only warning be exercised more than once. */
-export declare function resetBudgetWarning(): void;

package/dist/core/budget.js

@@ -1,72 +0,0 @@
-/**
- * Thrown when a client has spent its ceiling.
- *
- * Deliberately not a `HarnessError`: nothing was refused by the server, so a
- * caller inspecting `status` or `retryable` would be told a story about HTTP
- * that never happened.
- */
-export class BudgetExceededError extends Error {
-    spentMicros;
-    limitMicros;
-    constructor(spentMicros, limitMicros) {
-        super(`Cost ceiling reached: spent $${(spentMicros / 1e6).toFixed(4)} of ` +
-            `$${(limitMicros / 1e6).toFixed(4)}. Raise maxCostMicros or start a new client.`);
-        this.name = "BudgetExceededError";
-        this.spentMicros = spentMicros;
-        this.limitMicros = limitMicros;
-    }
-}
-let warned = false;
-/**
- * What this client has spent, and whether it may spend more.
- *
- * A loop that samples k rollouts across a dataset is the shape of an expensive
- * accident: one bad `k`, one dataset larger than expected, and the bill arrives
- * before anyone notices. The guard is a counter and a comparison, and a cost
- * meter that shipped without one would be embarrassing.
- *
- * Governs spending this client *initiates* — `run`, `sample`, `runAll`. It does
- * not gate `report`, which records money the caller already spent elsewhere;
- * counting that would make the ceiling mean two different things at once.
- */
-export class Budget {
-    used = 0;
-    limit;
-    /**
-     * `undefined` means no ceiling and warns once — an unbounded client in a
-     * scheduled script is how a surprise bill happens. `null` means the same
-     * thing deliberately, and says nothing.
-     */
-    constructor(maxCostMicros) {
-        this.limit = maxCostMicros ?? Infinity;
-        if (maxCostMicros === undefined && !warned) {
-            warned = true;
-            console.warn("[harness] No maxCostMicros set — this client can spend without limit. " +
-                "Pass maxCostMicros, or null to say you meant it.");
-        }
-    }
-    spent() {
-        return this.used;
-    }
-    remaining() {
-        return this.limit === Infinity ? Infinity : Math.max(0, this.limit - this.used);
-    }
-    add(costMicros) {
-        if (Number.isFinite(costMicros) && costMicros > 0)
-            this.used += costMicros;
-    }
-    /**
-     * Throws if the ceiling is already reached.
-     *
-     * Checked before a request rather than after: stopping once the bill has been
-     * incurred is a report, not a limit.
-     */
-    assertAffordable() {
-        if (this.used >= this.limit)
-            throw new BudgetExceededError(this.used, this.limit);
-    }
-}
-/** Test seam: lets the once-only warning be exercised more than once. */
-export function resetBudgetWarning() {
-    warned = false;
-}