@spendgraph/harness 0.1.0 → 0.2.1

package/docs/cascade.md

@@ -0,0 +1,165 @@
+# `cascade`
+
+Try the cheap model. Escalate only when its answer will not do.
+
+```ts
+const result = await cascade(values, {
+  tiers: [
+    { name: "haiku", run: (v) => haiku.bind(prompt).invoke(v) },
+    { name: "opus", run: (v) => opus.bind(prompt).invoke(v) },
+  ],
+  accept: (answer) => (answer.data as Reply).confidence >= 0.8,
+});
+
+result.tier;        // "haiku" — who answered
+result.escalated;   // [] — who did not
+result.accepted;    // true
+```
+
+---
+
+## The flow
+
+```
+                         values
+                           │
+                           ▼
+                  ┌─────────────────┐
+                  │  tier 1  haiku  │   cheap, tried first
+                  └────────┬────────┘
+                           │
+                    ┌──────┴──────┐
+                    │  accept()?  │
+                    └──────┬──────┘
+              yes ─────────┼───────── no, or it threw
+               │           │                │
+               │           │                ▼
+               │           │       ┌─────────────────┐
+               │           │       │  tier 2  opus   │   the fallback
+               │           │       └────────┬────────┘
+               │           │                │
+               │           │         ┌──────┴──────┐
+               │           │         │  accept()?  │
+               │           │         └──────┬──────┘
+               ▼           ▼                ▼        no → returned anyway,
+            ┌──────────────────────────────────┐        accepted: false
+            │  tier · escalated · accepted     │
+            └──────────────────────────────────┘
+```
+
+---
+
+## What it is not
+
+**Not `route`.** A router classifies *before* it has seen an answer, and commits.
+A cascade judges the answer itself, which is the only evidence that actually
+says whether the cheap model coped.
+
+**Not `refine`.** Refining revises with the same model and feeds back what was
+wrong. Cascading changes the model and asks again from scratch. Use `refine`
+when the draft is nearly right; use `cascade` when the model is out of its depth.
+
+---
+
+## `accept` is required
+
+There is no default and there cannot be one. Without `accept` this is a list of
+models with no reason to stop at any of them — the first would always answer, and
+the rest would be decoration.
+
+It runs on the answer, so it can read anything the tier returned:
+
+```ts
+accept: (answer) => (answer.data as Reply).confidence >= 0.8   // a self-report
+accept: (answer) => answer.output.length > 200                  // a floor
+accept: (answer) => schema.safeParse(answer.data).success       // a shape
+accept: async (answer) => (await judge(answer)).ok              // another model
+```
+
+The last one costs a call per tier and is usually worth it only when the tiers
+are far apart in price.
+
+---
+
+## A tier that throws escalates
+
+An overloaded cheap model is the case this exists for, so a throw hands on to the
+next rung rather than ending the run. So does a reply with `status: "failed"`.
+
+Both are recorded **distinguishably**:
+
+```ts
+result.history.map((r) => r.escalatedBy);
+// ["failed", "rejected", undefined]
+```
+
+`failed` is the call falling over. `rejected` is `accept` saying no. They want
+opposite fixes — a model that is down against a bar set too high — and a record
+that collapsed them into "escalated" could not tell you which you have.
+
+---
+
+## When nothing is accepted
+
+The **last tier that produced an answer** comes back, with `accepted: false`. The
+tiers are ordered by capability, so the most capable answer is the closest thing
+to a best effort, and throwing it away to return an error would waste the
+expensive call you just paid for.
+
+A tier that threw produced nothing, so it cannot be the fallback however late it
+ran:
+
+```
+  haiku  → "weak"      rejected
+  opus   → threw       failed
+  ────────────────────────────────
+  tier: "haiku", accepted: false
+```
+
+Only when **no** tier produced anything at all does the run fail, with
+`stoppedBy: "failed"` and the last error.
+
+---
+
+## The honest caveat
+
+**A cascade can cost more than not having one.** If the cheap tier is rejected
+most of the time, every run pays for it and then pays for the expensive model
+anyway.
+
+Which is why the totals count every tier tried:
+
+```ts
+result.inputTokens;   // haiku's and opus's, not the winner's
+```
+
+and why the thing to watch in production is the distribution of `result.tier`:
+
+```
+  tier=haiku  91%   ← paying off
+  tier=haiku  22%   ← paying for two calls to do one model's work
+```
+
+At the second number, delete the cascade and call the expensive model directly.
+There is no threshold that is right for everyone: it depends on the price gap
+between the tiers and on how much a rejected cheap answer costs relative to the
+expensive one.
+
+---
+
+## The ceiling
+
+`maxTokens` is checked before each tier, so it is never crossed rather than
+merely noticed. A cascade that has spent its budget stops with
+`stoppedBy: "tokens"` and returns the best answer it has — which is the fallback
+rule again, not a special case.
+
+---
+
+## How the files fit
+
+| | |
+| --- | --- |
+| `cascade.ts` | the walk down the tiers, and the result |
+| `tier.ts` | running one tier, turning a throw into an escalation, choosing the fallback |
+| `types.ts` | `Tier`, `Answer`, `TierRecord`, the options and the result |

package/docs/chain.md

@@ -0,0 +1,134 @@
+# chain
+
+Steps in sequence, each on the last one's output, with a gate between.
+
+```ts
+import { chain } from "@spendgraph/harness";
+
+const result = await chain(
+  { transcript },
+  {
+    steps: [
+      {
+        name: "extract",
+        run: ({ transcript }) => llm.call(extractPrompt(transcript), { schema: CHANGES }),
+        gate: (output) =>
+          JSON.parse(output.output).items.length > 0
+            ? { ok: true }
+            : { ok: false, reason: "found no changes to write about" },
+      },
+      {
+        name: "write",
+        run: (previous) => llm.call(writePrompt(previous)),
+        gate: (output) =>
+          output.output.length <= 1200
+            ? { ok: true }
+            : { ok: false, reason: `${output.output.length} characters, limit is 1200` },
+      },
+      { name: "translate", run: (previous) => llm.call(translatePrompt(previous)) },
+    ],
+  }
+);
+
+result.ran;        // ["extract", "write", "translate"]
+result.stoppedAt;  // null, or the step whose gate refused
+result.reason;     // what the gate said
+```
+
+`run` is handed whatever the step before it returned — the values the chain was
+called with, for the first one.
+
+---
+
+## The gate is the point
+
+Without one, a stage that produced nothing useful is passed to the next stage,
+which confidently works on it, and the stage after that works on *that*. You get
+a polished paragraph three calls later that is about nothing, and no indication
+which stage went wrong.
+
+```
+   extract ──► gate ──► write ──► gate ──► translate ──► output
+                 │                  │
+                 └── refuses ───────┴──► stops, and says which
+```
+
+A refused gate stops the chain and names the step. The tests pin the thing that
+matters: the next step **is not called**.
+
+**The output the gate refused is kept.** It is recorded as a completed step with
+whatever it produced, and the gate is recorded separately as the failure. The
+rejected output is the evidence for why the gate fired at all — losing it leaves
+you with "extract failed" and nothing to look at.
+
+```
+  steps:  extract (completed, "nothing found")
+          gate_extract (failed, "no items")
+```
+
+---
+
+## Why not one prompt
+
+Each call gets simpler instructions and does its one job better. Splitting trades
+latency for accuracy — and you find out *which* stage went wrong instead of
+receiving one bad paragraph with no explanation.
+
+The cost is real: three sequential calls is three times the wall-clock, and they
+cannot be parallelised because each needs the last. `chain` buys accuracy with
+latency, where `parallel` buys latency with tokens.
+
+---
+
+## When it wins
+
+The task decomposes into stages you can name **in advance**, and each stage's
+output is checkable.
+
+- extract → validate → transform
+- outline → check it covers the brief → draft
+- draft → translate → verify the placeholders survived
+
+## When it does not
+
+**The stages are only separable on paper.** If each one needs the full context
+anyway, you have paid for three calls to do one call's work.
+
+**You do not know the stages until you have read the input.** That is
+`orchestrate`, not `chain`.
+
+**Nothing between the stages is checkable.** Without gates this is a `graph` with
+extra ceremony — use `graph` directly.
+
+---
+
+## How it differs from its neighbours
+
+`refine` loops on the **same** step until it is good enough. `chain` moves
+**forward** through different steps. Refine asks "is this good yet?"; chain asks
+"is this good enough to continue?"
+
+`graph` can already express a chain — it is a line of nodes. `chain` is the named
+case with the gate built in, the way `route` is the named case for branching. The
+value is not the mechanism; it is that the gate is on by default rather than
+something you remember to add.
+
+---
+
+## How the files fit
+
+```
+   types.ts     the contract
+        ▲
+   step.ts      a stage as a node
+   gate.ts      a gate as its own node, and the refusal it throws
+        ▲
+   chain.ts     compile · run · shape
+```
+
+`step.ts` and `gate.ts` are siblings — neither knows about the other. `chain.ts`
+is the only file that knows they alternate.
+
+A stage reads its input from `ctx.outputs` rather than having it threaded in,
+because a gate may sit between two stages and the gate's return is not what the
+next stage wants — the stage before it is.

package/docs/loop.md

@@ -0,0 +1,318 @@
+# loop
+
+Model, tools, model again, until it stops asking.
+
+```ts
+import { loop } from "@spendgraph/harness";
+import { finish, toolbus } from "@spendgraph/tools";
+
+const bus = toolbus([lookupClause, issueRefund, finish()]);
+
+const result = await loop({
+  tools: bus,
+  maxTurns: 8,
+  maxTokens: 40_000,
+  turn: ({ results, last }) =>
+    llm.call(messagesFrom(results, last), { tools: bus }),
+});
+
+result.stoppedBy;  // finished · quiet · turns · tokens · failed
+result.answer;     // set when the model called finish()
+result.turns;
+```
+
+---
+
+## What this adds over a provider's own tool loop
+
+`@spendgraph/llms` already loops: it sends the declarations, reads the calls
+back, invokes them and goes round again. What is always hand-rolled around that,
+and always wrong the same way, is the rest:
+
+**A ceiling checked before the spend.** `maxTokens` is tested at the top of each
+turn, so it is never crossed rather than merely noticed once it has been.
+`maxTurns` defaults to 8.
+
+**Every turn and every tool call recorded.** Including the tool calls that
+failed — a run that dropped them cannot explain why the model changed course.
+
+```
+  steps:  turn_0 · lookup · lookup · turn_1
+```
+
+**`finished` told apart from `quiet`.**
+
+---
+
+## The distinction worth having
+
+Two things end a tool loop and they look identical from outside:
+
+| | |
+| --- | --- |
+| `finished` | the model called `finish()` and gave an answer |
+| `quiet` | the model said something and asked for nothing more |
+
+The second is what being *done* looks like. It is also exactly what being
+**lost** looks like — the model has run out of ideas and trailed off. Without
+`finish` in the bus you cannot tell them apart, and a run that quietly gave up
+is indistinguishable from one that succeeded.
+
+```ts
+toolbus([...yourTools, finish()]);
+```
+
+`result.answer` is set only on `finished`, and carries what the model said the
+answer was — not the surrounding prose.
+
+---
+
+## A failed tool does not end the run
+
+`invoke` returns a failed result rather than throwing, so a tool that fell over
+is something the model can read and work around. It is recorded as a failed step
+and the loop carries on.
+
+```
+  turn_0   "let me check"
+  nope     failed — No tool called "nope". Offered: lookup, finish
+  turn_1   "that tool is not available; answering from what I have"
+```
+
+A loop that crashed on a bad call would lose the whole turn, and with it the
+work already done.
+
+---
+
+## Hooks
+
+Deterministic code at fixed points. A guardrail written into the prompt is a
+request; this is enforcement — it runs whether or not the model read the
+instruction, and next turn's wording cannot argue it away.
+
+```ts
+await loop({
+  tools: bus,
+  turn,
+  hooks: {
+    beforeTurn: () => (overDailyBudget() ? { block: "daily budget spent" } : undefined),
+    beforeTool: (call) =>
+      call.name === "issue_refund" && Number(call.args.amount) > 50_000
+        ? { block: "over the approval limit" }
+        : undefined,
+    afterTool: (result) => audit.write(result),
+    afterTurn: (turn) => log.debug(turn),
+  },
+});
+```
+
+### Gate on what a tool does, not on what it is called
+
+`beforeTool` is handed the effect the tool declared, so a rule can be written
+against the thing that matters:
+
+```ts
+hooks: {
+  beforeTool: (call, effect) => {
+    if (effect === "destructive") return { block: "needs a person" };
+    if (effect === undefined) return { block: `${call.name} has not said what it does` };
+  },
+}
+```
+
+The version above it — `call.name === "issue_refund"` — is a rule that stops
+firing the day somebody renames the tool. Nothing errors, no test goes red, and
+the refund goes through. A rule written against `effect` survives the rename,
+because it never knew the name.
+
+`undefined` means the tool declared nothing, and it is deliberately not
+`readonly`. Blocking it is the safe direction; letting it through means every
+tool written before annotations existed is treated as harmless. Use
+`bus.unannotated()` in a test to keep that list empty.
+
+The effect comes from whatever you passed as `tools`: a `toolbus()` answers from
+its registry, a `bus.trace()` from the shortlist it offered that turn. Anything
+else with just an `invoke` is still a valid `tools`, and every effect reads
+`undefined` — which is what it is.
+
+### Blocking a tool is not an error
+
+A refused call comes back as a **failed tool result**, in the shape the model
+already knows how to read:
+
+```
+  turn_0         "refunding"
+  issue_refund   failed — blocked: over the approval limit
+  turn_1         "understood, escalating"
+```
+
+So the model reads the refusal and works around it. Throwing instead would lose
+the turn and the work already done in it. The tool itself never runs — a
+guardrail that refunds first and refuses afterwards is not a guardrail.
+
+### Blocking a turn ends the run
+
+There is no way around a blocked turn, so it stops with `stoppedBy: "blocked"`
+and a `failed` status. That is the shape for a budget ceiling or a kill switch.
+
+### A hook that throws is treated as permission
+
+A bug in the guardrail must not take down the thing it was guarding. The call it
+was checking has not happened yet, the next hook still gets its say, and the run
+continues. The alternative — an unreachable policy service ending every run —
+fails far more often than the case it was protecting against.
+
+The observing hooks, `afterTool` and `afterTurn`, change nothing when they
+throw. They were only watching.
+
+### Why hooks rather than an `if` inside the tool
+
+A blocked call is a **step**, so "what did this agent try to do and get refused"
+is a query rather than a grep. `afterTool` sees blocked results as well as
+successful ones, which is what an audit trail actually wants.
+
+## Telling the model what is left
+
+The ceilings stop a run. They do not stop the model planning as if there were no
+ceilings, which is how turn seven of eight opens a five-step approach and gets
+cut off after the first — paid for in full.
+
+`remaining` is on the state your `turn` already receives:
+
+```ts
+turn: ({ remaining }) =>
+  model.call([
+    { role: "user", content: question },
+    { role: "user", content: `${remaining.turns} turns and about ${remaining.tokens} tokens left.` },
+  ], { tools: bus.trace(question) });
+```
+
+| | |
+| --- | --- |
+| `remaining.turns` | turns left, **this one included** — the last turn reads `1`, not `0` |
+| `remaining.tokens` | what is left of `maxTokens`, floored at zero |
+
+`beforeTurn` gets the same state, so a run can be stopped rather than started
+with a budget that cannot finish it:
+
+```ts
+beforeTurn: ({ remaining }) =>
+  remaining.tokens < 30_000 ? { block: "not enough left to finish" } : undefined,
+```
+
+### Putting it in the prompt is your job
+
+The loop hands you the number and stops there. Which wording works is
+task-specific, and a library that quietly appends a sentence to your messages
+invalidates your cached prefix without asking.
+
+### No ceiling reads as `Infinity`, never zero
+
+This is the whole reason the field needs a test. The obvious check —
+
+```ts
+if (remaining.tokens < 1000) return "wrap up now";
+```
+
+— would fire on the **first turn of every run that never set `maxTokens`** if a
+missing budget read as zero. The agent would refuse to start work because it
+believes it is already out, and it would look like a model problem rather than a
+library one.
+
+---
+
+## Compaction
+
+A long run gets expensive because the transcript grows, not because the work got
+harder. Every turn resends what came before, so the tool results are the bulk of
+what you pay for on turn eight.
+
+```ts
+await loop({
+  tools: bus,
+  maxTokens: 40_000,
+  compact: {
+    at: 0.7,                                       // fraction of the ceiling
+    keep: 4,                                       // recent results, verbatim
+    summarise: (older) => lead.call(summarisePrompt(older)),
+  },
+  turn: ({ results, last }) => llm.call(messagesFrom(results, last), { tools: bus }),
+});
+
+result.compactions;   // { count: 2, replaced: 11 }
+```
+
+The older results are replaced by **one synthetic result named `compacted`**,
+which the next turn reads like any other. The most recent `keep` survive
+verbatim: summarising what just happened is how a loop forgets what it was in
+the middle of doing.
+
+### It measures the context, not the spend
+
+The threshold is on **the last turn's `inputTokens`** — the size of what is
+actually being resent — and not on what the run has spent so far.
+
+Spend only ever rises. A threshold on it fires once and then on every turn after,
+compacting an already-compacted history again and again. The input count falls
+the moment a compaction lands, so the next turn is under the line and it does not
+fire again until the history has genuinely grown back.
+
+Set `after: 12` instead to compact on a result count, which needs no ceiling.
+
+### What it costs
+
+The summary is a model call, recorded as a `compact` step with its tokens and
+counted against `maxTokens`. A compaction that costs more than it saves still has
+to be paid for, and the step is there so you can see whether it did.
+
+**A failed summariser is survivable.** The run carries on with the history it
+has rather than ending over a summary it could not write — losing the compaction
+costs tokens, losing the run costs the work.
+
+**The honest caveat:** compaction loses information, and the loss is silent. A
+run that compacted and then got the answer wrong looks identical to one that was
+simply wrong.
+
+## Reading `stoppedBy` in production
+
+| | What it means |
+| --- | --- |
+| `finished` | working as intended |
+| `quiet` | offer it `finish()`, or the prompt is not telling it what done looks like |
+| `turns` | the task needs more steps than you allowed, or it is going in circles |
+| `tokens` | the ceiling is too low, or the history is growing faster than the work |
+| `failed` | the provider failed; not an agent problem |
+
+A rising share of `turns` is the signal to look at the tool descriptions: a model
+that cannot tell two tools apart calls both and gets nowhere.
+
+---
+
+## When not to use it
+
+**The steps are known in advance.** That is `chain` — a loop pays for the model
+to decide what to do next, and if you already know, you are paying for nothing.
+
+**There are no tools.** A loop with an empty bus is one call with extra
+machinery.
+
+**The work is unbounded and the budget is not.** This is the workflow that can
+spend the most, which is why both ceilings are here and why `maxTurns` has a
+default when the others do not.
+
+---
+
+## How the files fit
+
+```
+   types.ts     the contract
+        ▲
+   turn.ts      one model call: what it cost, how it is recorded
+   act.ts       running what it asked for, and reading `finish`
+        ▲
+   loop.ts      the cycle and its ceilings
+```
+
+Not compiled to a graph, unlike `route` and `chain`. This is a cycle, and
+`@spendgraph/graph` is a DAG whose step ceiling exists to stop one — same
+reasoning as `refine`.