@spendgraph/harness 0.1.0 → 0.2.1

package/docs/orchestrate.md

@@ -0,0 +1,128 @@
+# orchestrate
+
+A lead decomposes, workers do the pieces, the lead puts it back together.
+
+```ts
+import { orchestrate } from "@spendgraph/harness";
+
+const result = await orchestrate(
+  { brief },
+  {
+    plan: async (v) => {
+      const reply = await lead.call(planPrompt(v), { schema: SUBTASKS });
+      return { ...reply.data, model: reply.model, inputTokens: reply.inputTokens, outputTokens: reply.outputTokens };
+    },
+    work: (subtask, v) => worker.call(sectionPrompt(subtask, v)),
+    synthesise: (results, v) => lead.call(assemblePrompt(results, v)),
+    maxWorkers: 5,
+    concurrency: 4,
+    maxTokens: 80_000,
+  }
+);
+
+result.subtasks;   // what the lead asked for, after the cap
+result.dropped;    // what the cap refused
+result.stoppedBy;  // completed · planned-nothing · workers-failed · tokens · failed
+```
+
+The workflow for work whose **shape is not known until the input is read** —
+which is the whole difference from `chain`, where the stages are named in
+advance.
+
+---
+
+## The cap is the point
+
+This is the workflow that can spend the most, because the **lead** decides how
+many calls to make. Asked to decompose a large input it will cheerfully invent
+twelve subtasks, and twelve model calls leave before anyone has decided that was
+reasonable.
+
+`maxWorkers` defaults to 5. Anything past it is **dropped and said**, never
+dropped quietly:
+
+```
+  result.subtasks   ["a", "b", "c"]
+  result.dropped    ["d", "e", "f", "g"]
+  steps[0].error    "4 subtask(s) past the worker cap were dropped: d, e, f, g"
+```
+
+It keeps the head and drops the tail: a lead asked for the most important piece
+first usually gives it first, and truncating from the front would throw away
+exactly what it thought mattered.
+
+A duplicated name is made unique — `summary`, `summary_1` — rather than
+rejected. A lead naming two sections the same thing is a wording problem, not a
+reason to lose the run.
+
+---
+
+## When the lead finds nothing to do
+
+```ts
+{ stoppedBy: "planned-nothing", output: "There is nothing here to summarise." }
+```
+
+The workers never run and `synthesise` is never called. Synthesising over an
+empty list asks a model to write an answer from nothing, and it will — which is
+worse than saying there was nothing.
+
+---
+
+## When a worker falls over
+
+`minSuccess` decides, and it defaults to **all**.
+
+```ts
+minSuccess: 2   // three sections planned, two is enough
+```
+
+Sectioning a document usually wants all of them: a hole in the middle is worse
+than no answer. Research-style fan-out often does not — two sources out of three
+still supports a conclusion. Without a `minSuccess` the run fails and
+`synthesise` is not called, so the lead is never asked to write around a gap it
+does not know about.
+
+---
+
+## The ceiling is checked twice
+
+Once after planning and once before synthesising, because both are places where
+the run can be abandoned before spending more:
+
+- the plan alone reaching the ceiling means the workers never start
+- the workers reaching it means the lead is never asked to assemble
+
+Tokens rather than money, because this package cannot price a call — that
+happens on the server, after the fact.
+
+---
+
+## When not to use it
+
+**The stages are known in advance.** That is `chain`. Paying a lead to decide
+what you already know is a call for nothing.
+
+**The pieces are independent and uniform.** That is `parallel` — you do not need
+a model to tell you a document has three sections if you already know it does.
+
+**One call would do.** Three calls minimum here: plan, work, synthesise. On a
+small input the lead's overhead outweighs the parallelism.
+
+---
+
+## How the files fit
+
+```
+   types.ts          the contract
+        ▲
+   plan.ts           the lead's decomposition, and the cap
+   work.ts           the workers, delegated to `parallel`
+        ▲
+   orchestrate.ts    plan · work · synthesise
+```
+
+`work.ts` does not fan out again — `parallel` already caps the concurrency, turns
+a thrown worker into a result so the others land, and numbers the steps by
+declaration so two runs can be compared. This is the workflow that most benefits
+from the others existing first.

package/docs/parallel.md

@@ -0,0 +1,139 @@
+# parallel
+
+Every task at once, then one answer out of what came back. Two shapes use it:
+**sectioning** buys wall-clock, **voting** buys confidence.
+
+---
+
+## Sectioning — split the work
+
+```ts
+import { parallel } from "@spendgraph/harness";
+
+const result = await parallel(
+  { brief },
+  {
+    tasks: [
+      { name: "intro", run: (v) => llm.call(introPrompt(v)) },
+      { name: "body", run: (v) => llm.call(bodyPrompt(v)) },
+      { name: "close", run: (v) => llm.call(closePrompt(v)) },
+    ],
+    merge: (results) => results.map((r) => r.value.output).join("\n\n"),
+    concurrency: 4,
+  }
+);
+```
+
+Three calls take as long as the slowest, not the sum. The trade against `chain`
+is exact: chain buys accuracy with latency, parallel buys latency with tokens —
+you pay for the same context three times over.
+
+**A missing section fails the run by default.** A hole in the middle of an
+answer is worse than no answer, and `merge` is never called with one. The tasks
+that did land are still recorded, so the failure is legible.
+
+---
+
+## Voting — ask N times, take the consensus
+
+```ts
+import { majority, parallel, repeat } from "@spendgraph/harness";
+
+const result = await parallel(
+  { question },
+  {
+    tasks: repeat("guess", 5, (v) => llm.call(prompt(v))),
+    merge: (results) => majority(results)?.value,
+    minSuccess: 3,
+  }
+);
+
+result.merged;   // the answer most of them gave
+```
+
+`repeat` **rounds an even count up.** Four voters tie, and a tie has no answer
+that is not arbitrary.
+
+`minSuccess: 3` is the point of running five: if one call is overloaded the other
+four still land and still answer. Ties go to the first declared, so the same
+inputs give the same result on every run.
+
+**When voting is worth the tokens:** the model is right most of the time and
+wrong *differently* each time — the wrong answers scatter, the right one repeats.
+
+**When it buys nothing:** the model is confidently wrong the same way every
+time. Five identical wrong answers vote unanimously for the wrong one, and you
+have paid five times to be misled with more conviction. That is the failure it is
+easiest to mistake this pattern for a fix for.
+
+---
+
+## Why it is not a graph
+
+`route` and `chain` compile down to `@spendgraph/graph`. This one cannot:
+`graph` walks one node at a time and picks a single edge, so a fan-out is not a
+shape it can express. The concurrency comes from `mapLimit`, which already exists
+in `@spendgraph/prompt/internals`.
+
+Same reasoning as `refine`, which is a loop rather than a DAG walk.
+
+---
+
+## Steps are numbered by declaration, not by arrival
+
+A fan-out that numbered steps by completion order would record a different run
+every time for identical work, and two runs could not be compared. The slow task
+declared first is still step 0.
+
+```
+  steps:  slow  (index 0)   ← declared first, landed last
+          quick (index 1)
+```
+
+---
+
+## Failure is a result, not an exception
+
+A task that throws comes back as a failed `TaskResult`. The whole point of
+running five is that the other four still land, so one bad call cannot take the
+run down with it.
+
+```ts
+result.succeeded;  // ["a", "b"]
+result.failed;     // ["c"]
+result.results;    // every one, in declaration order
+```
+
+`minSuccess` decides whether that is fatal. Sectioning defaults to all —
+anything less is a hole. Voting should set a majority.
+
+---
+
+## When not to use it
+
+**The tasks need each other's output.** That is `chain`; a fan-out cannot pass
+anything between branches.
+
+**One call would do.** Voting on a task the model gets right every time is five
+times the cost for the same answer.
+
+**The context is enormous.** Sectioning resends it per task. Three sections of a
+50k-token brief is 150k tokens of input, and the latency you bought may cost more
+than it saved.
+
+---
+
+## How the files fit
+
+```
+   types.ts       the contract
+        ▲
+   task.ts        run one, record one
+   merge.ts       repeat · majority
+        ▲
+   parallel.ts    fan out · gather · merge
+```
+
+`task.ts` and `merge.ts` are siblings. `majority` and `repeat` are pure and
+tested on their own — the tie-break and the even-count rounding are the parts
+worth pinning.

package/docs/refine.md

@@ -0,0 +1,206 @@
+# refine
+
+Draft, judge, revise. Stop when it is good enough, or when the budget is out.
+
+```ts
+import { refine } from "@spendgraph/harness";
+
+const result = await refine({
+  rounds: 3,
+  maxTokens: 20_000,
+  attempt: (feedback) =>
+    llm.call([{ role: "user", content: feedback ? `${task}\n\nFix this: ${feedback}` : task }]),
+  judge: (attempt) =>
+    attempt.output.length <= 160
+      ? { accepted: true }
+      : { accepted: false, feedback: `That is ${attempt.output.length} characters. The limit is 160.` },
+});
+
+result.output;     // the winning draft
+result.accepted;   // false when the rounds or the tokens ran out
+result.stoppedBy;  // accepted · rounds · tokens · failed
+result.steps;      // every attempt and every judgement
+```
+
+One `attempt` callback rather than separate generate and revise, because in
+practice they are the same prompt with one extra paragraph — `feedback` is null
+on the first round and the judge's note after.
+
+---
+
+## Example: an invoice that has to add up
+
+The judge is plain code, which is the case worth reaching for. It costs nothing,
+it is deterministic, and it cannot be talked into agreeing.
+
+```ts
+const result = await refine({
+  rounds: 3,
+
+  attempt: (feedback) =>
+    llm.call([{ role: "user", content: `${email}${feedback ?? ""}` }], { schema: INVOICE }),
+
+  judge: (attempt) => {
+    const invoice = JSON.parse(attempt.output);
+    const summed = invoice.lines.reduce((n, l) => n + l.amount, 0);
+
+    return summed === invoice.total
+      ? { accepted: true }
+      : {
+          accepted: false,
+          feedback: `\n\nThe lines add up to ${summed} but the total says ${invoice.total}. A line is missing.`,
+        };
+  },
+});
+```
+
+What happens:
+
+```
+  round 0   → { lines: [1200, 50], total: 1550 }        ✗ lines sum to 1250
+              feedback: "The lines add up to 1250 but the total says 1550…"
+  round 1   → { lines: [1200, 50, 300], total: 1550 }   ✓ accepted
+
+  accepted: true   rounds: 2   steps: 4
+```
+
+The model dropped the shipping line. Told exactly what was inconsistent, it
+found it. A **retry** would have run the same prompt and often made the same
+mistake; the feedback is the difference.
+
+A model judge asked "is this extracted correctly?" would have said yes, because
+the JSON looks fine. Arithmetic does not have that problem.
+
+---
+
+## More judges that cost nothing
+
+**SQL that has to run.** `EXPLAIN` is free and the database's own error is the
+best feedback there is.
+
+```ts
+judge: async (attempt) => {
+  try {
+    await db.prepare(`EXPLAIN ${attempt.output}`).run();
+    return { accepted: true };
+  } catch (err) {
+    return { accepted: false, feedback: String(err.message) };
+  }
+}
+```
+
+Hand the error back verbatim. Rewriting it into something friendlier throws away
+the column name that tells the model what to fix.
+
+**Code that has to pass its tests.** The strongest judge there is, because it is
+the same bar a person would apply. Slow, so keep `rounds` low — three failed
+attempts at a minute each is three minutes before you learn the prompt is wrong.
+
+```ts
+judge: async (attempt) => {
+  const { failed, output } = await runTests(attempt.output);
+  return failed === 0 ? { accepted: true } : { accepted: false, feedback: output };
+}
+```
+
+**Placeholders that must survive a translation.** Models translate `{count}`
+about one time in fifty, and it fails at runtime for whoever speaks that
+language.
+
+```ts
+judge: (attempt) => {
+  const wanted = source.match(/\{[a-z_]+\}/g) ?? [];
+  const missing = wanted.filter((token) => !attempt.output.includes(token));
+  return missing.length === 0
+    ? { accepted: true }
+    : { accepted: false, feedback: `Keep these exactly as they are: ${missing.join(", ")}.` };
+}
+```
+
+---
+
+## When the judge has to be a model
+
+Only when no code can decide — tone, policy, "would a person find this rude".
+It is a second model call every round, so the loop is now at least twice the
+price of the answer.
+
+```ts
+judge: async (attempt) => {
+  const verdict = await judgeLlm.call([{ role: "user", content: `${RUBRIC}\n\n${attempt.output}` }], {
+    schema: {
+      type: "object",
+      properties: { score: { type: "number" }, fix: { type: "string" } },
+      required: ["score"],
+    },
+  });
+
+  const { score, fix } = verdict.data;
+  return score >= 8 ? { accepted: true, score } : { accepted: false, score, feedback: fix };
+}
+```
+
+Always return `score`, even on acceptance. Without it, a run that never passes
+hands back the **last** attempt rather than the best one.
+
+---
+
+## When not to use it
+
+**You cannot state the bar.** If the judge accepts everything you have tripled
+the cost to learn nothing. Write the check first; if you cannot, the pattern does
+not apply.
+
+**Round one already passes almost always.** At 99% you are paying a judge on
+every request to catch the hundredth. Retry-on-failure is cheaper than a loop by
+default.
+
+**The failure is not fixable by trying again.** A missing document, a permission
+the agent does not have, a question with no answer — feedback cannot help, and
+three rounds is three ways of saying so. Escalate instead.
+
+---
+
+## Reading the result
+
+`stoppedBy` is the field to watch in production.
+
+| | What it means |
+| --- | --- |
+| `accepted` | working as intended |
+| `rounds` | the judge is too strict, or the prompt cannot get there |
+| `tokens` | the ceiling is too low, or the attempts are too big |
+| `failed` | the provider failed; not a quality problem |
+
+A rising share of `rounds` is the signal to fix the prompt rather than raise the
+limit. `history` carries every attempt with its verdict and token count, so "does
+refining pay on this task?" is a query — round-one acceptance rate against the
+cost of the extra calls — rather than a hunch.
+
+**A failed attempt ends the run** rather than being judged. Asking a judge what
+it thinks of an error spends a call to be told it is bad.
+
+**The ceiling is in tokens, not money**, because this package cannot price a
+call — that happens on the server, after the fact. It is checked *before* each
+round, so it is never crossed rather than merely noticed once it has been.
+
+---
+
+## How the files fit
+
+```
+   types.ts       the contract
+        ▲
+   attempt.ts     what a draft cost, and how it is recorded
+   judge.ts       how a verdict is recorded, and which draft wins
+        ▲
+   refine.ts      the loop: draft · judge · stop
+```
+
+`attempt.ts` and `judge.ts` are siblings — neither knows about the other, and
+both import only the contract. The loop is the only file that knows the order
+they run in.
+
+`best` lives with the judge rather than the loop because the choice is the
+judge's: it is made on the verdicts, and a judge that never scores gets the
+fallback to the last attempt.

package/docs/route.md

@@ -0,0 +1,212 @@
+# route
+
+Classify once, then hand the work to the branch built for it.
+
+```ts
+import { route } from "@spendgraph/harness";
+
+const result = await route(
+  { subject, body },
+  {
+    classify: ({ subject }) => (/refund|invoice/i.test(subject) ? "billing" : "technical"),
+    routes: {
+      billing: (input) => haiku.bind(billingPrompt).invoke(input),
+      technical: (input) => opus.bind(docsPrompt).invoke(input),
+    },
+    fallback: (input) => opus.bind(generalPrompt).invoke(input),
+  }
+);
+
+result.route;        // "billing", or null when the fallback ran
+result.classified;   // what the classifier said, used or not
+result.inputTokens;  // the classifier's tokens plus the branch's
+```
+
+---
+
+## Example: support triage, routed for cost
+
+Most tickets are simple. Only a few need the expensive model.
+
+```ts
+const result = await route(
+  { subject, body },
+  {
+    classify: ({ subject, body }) => {
+      const text = `${subject} ${body}`.toLowerCase();
+      if (/refund|charged|invoice/.test(text)) return "billing";
+      if (/error|crash|not working/.test(text)) return "technical";
+      return "general";
+    },
+
+    routes: {
+      billing: (input) => haiku.bind(billingPrompt).invoke(input),
+      technical: (input) => opus.bind(docsPrompt).invoke(input),
+      general: () => ({ output: "Thanks — a human will reply within a day.", model: "none" }),
+    },
+
+    fallback: (input) => opus.bind(generalPrompt).invoke(input),
+  }
+);
+```
+
+```
+  result.route        "billing"
+  result.output       "Your 12 March charge was a duplicate; it's been refunded."
+  result.inputTokens  340        classifier (0) + branch
+  result.steps        [ classify, billing ]
+```
+
+Three things this is doing:
+
+The **classifier is a regex** — free, instant, and it cannot invent a category.
+That is the common case.
+
+The **`general` branch calls no model at all.** It returns a canned line at zero
+cost. If a third of your tickets land there, that third is now free, which no
+amount of prompt tuning gets you.
+
+**`billing` uses the small model, `technical` the big one.** That is the whole
+cost argument: you stop paying opus rates for "where is my invoice".
+
+---
+
+## Example: a classifier that is a model
+
+When no regex will ever be right — sentiment, intent, "is this actually a
+complaint" — use a model, and constrain it with an enum so it can only answer
+with a route you have.
+
+```ts
+classify: async ({ subject, body }) => {
+  const reply = await haiku.call(
+    [{ role: "user", content: `Classify this ticket.\n\n${subject}\n${body}` }],
+    {
+      schema: {
+        type: "object",
+        properties: {
+          route: { type: "string", enum: ["billing", "technical", "general"] },
+          confidence: { type: "number" },
+          reason: { type: "string" },
+        },
+        required: ["route", "confidence"],
+      },
+    }
+  );
+
+  return {
+    ...reply.data,
+    model: reply.model,
+    inputTokens: reply.inputTokens,
+    outputTokens: reply.outputTokens,
+  };
+},
+minConfidence: 0.7,
+```
+
+The `enum` is doing real work. A free-text classifier eventually answers
+`"Billing"` or `"billing question"` or a whole sentence, and every one of those
+falls through to the fallback silently. Constrained, it can only say one of
+three things — and on Anthropic that is a forced tool call, which is a guarantee
+rather than a request.
+
+Returning `model` and the token counts is what keeps the cost honest: a
+classifier running on every request is not free, and without them
+`result.inputTokens` reports only the branch.
+
+**Self-reported `confidence` is weak.** Models are overconfident and the number
+is not calibrated. `minConfidence` is a blunt safety net for the genuinely
+ambiguous cases, not a dial to tune to three decimal places.
+
+**Check the arithmetic before reaching for it.** A haiku classify might be 300
+in, 20 out. Routing 80% of traffic from opus to haiku pays for itself many times
+over. If every branch is a cheap model anyway, you have added a call to every
+request to save nothing.
+
+---
+
+## What else it is for
+
+**Capability.** A refund branch gets a bus with `escalate` pinned and a payments
+tool; a docs branch gets retrieval and no write access. You cannot express that
+in one call, because the tools are chosen before you know what was asked.
+
+**Quality.** One prompt covering billing, technical and legal has to carry
+instructions for all three, and every instruction is a chance to apply the wrong
+one. Three prompts each do one thing.
+
+**Safety.** Classify first; if it is out of scope or harmful, route to a branch
+that declines and hands off. More reliable than hoping one prompt's guardrails
+hold, because the decision is a separate call you can test.
+
+Beyond support, the same shape: document type → invoice, contract or receipt,
+each with its own extraction schema. Language detection → a localised prompt.
+Complexity → a template for the trivial, an agent loop for the rest.
+
+---
+
+## When not to use it
+
+**The branches would share 90% of their prompt.** You have split one thing into
+three that now drift apart.
+
+**The classification is genuinely hard.** You have moved the failure earlier
+without removing it — a confidently wrong route sends the question to a
+specialist with no idea what to do with it. `minConfidence` and a required
+`fallback` exist for exactly this, but they are a net, not a fix.
+
+---
+
+## Reading the result
+
+**Watch `route === null`.** A rising share of fallbacks means either the
+classifier is drifting or there is a kind of request you never built a branch
+for — and since `classified` records what the classifier *said* separately from
+what ran, you can tell which.
+
+```
+  classifier says "legal", no such branch    →  classified "legal"    route null
+  classifier says "billing" at 0.4, bar 0.7  →  classified "billing"  route null
+```
+
+Same `route`, different `classified`: the first is a **missing branch**, the
+second a **miscalibrated** one, and they want opposite fixes.
+
+**`fallback` is required.** A router that can fail to route fails in production
+at 3am, and "unknown" is a class every classifier eventually returns.
+
+`route` and `refine` compose: a branch can be a refine loop, and its rounds
+appear as `billing.attempt_0`, `billing.judge_0` rather than being flattened
+into one step.
+
+---
+
+## How the files fit
+
+```
+   types.ts        the contract
+        ▲
+   classify.ts     what the classifier said, and what runs because of it
+        ▲
+   dispatch.ts     where the work goes: a node per branch, and the edges to them
+        ▲
+   route.ts        compile · run · shape the result
+```
+
+Each file imports only downwards, which is why `decide` can be tested on its own
+with no graph in sight.
+
+It compiles to a graph rather than looping here. `@spendgraph/graph` already
+validates that every branch is reachable and that no default sits in front of
+one, already records a step per node with its tokens, and already returns the
+rollout shape. A second execution engine in this package would be a second thing
+to keep right.
+
+**The decision travels in the graph, not in a closure.** The classifier
+*returns* its decision, so it lands in `outputs.classify` like any node's return
+and the edges read it from the context they are already handed. A finished run
+can be read apart from `outputs.classify` alone.
+
+**The edge order matters.** Conditional edges first, the unconditional fallback
+last — `graph` refuses the other order, and is right to: an unconditional edge
+ahead of a conditional one means the branch behind it can never be reached.