@spendgraph/harness 0.1.0 → 0.2.0

package/docs/loop.md

@@ -0,0 +1,318 @@
+# loop
+
+Model, tools, model again, until it stops asking.
+
+```ts
+import { loop } from "@spendgraph/harness";
+import { finish, toolbus } from "@spendgraph/tools";
+
+const bus = toolbus([lookupClause, issueRefund, finish()]);
+
+const result = await loop({
+  tools: bus,
+  maxTurns: 8,
+  maxTokens: 40_000,
+  turn: ({ results, last }) =>
+    llm.call(messagesFrom(results, last), { tools: bus }),
+});
+
+result.stoppedBy;  // finished · quiet · turns · tokens · failed
+result.answer;     // set when the model called finish()
+result.turns;
+```
+
+---
+
+## What this adds over a provider's own tool loop
+
+`@spendgraph/llms` already loops: it sends the declarations, reads the calls
+back, invokes them and goes round again. What is always hand-rolled around that,
+and always wrong the same way, is the rest:
+
+**A ceiling checked before the spend.** `maxTokens` is tested at the top of each
+turn, so it is never crossed rather than merely noticed once it has been.
+`maxTurns` defaults to 8.
+
+**Every turn and every tool call recorded.** Including the tool calls that
+failed — a run that dropped them cannot explain why the model changed course.
+
+```
+  steps:  turn_0 · lookup · lookup · turn_1
+```
+
+**`finished` told apart from `quiet`.**
+
+---
+
+## The distinction worth having
+
+Two things end a tool loop and they look identical from outside:
+
+| | |
+| --- | --- |
+| `finished` | the model called `finish()` and gave an answer |
+| `quiet` | the model said something and asked for nothing more |
+
+The second is what being *done* looks like. It is also exactly what being
+**lost** looks like — the model has run out of ideas and trailed off. Without
+`finish` in the bus you cannot tell them apart, and a run that quietly gave up
+is indistinguishable from one that succeeded.
+
+```ts
+toolbus([...yourTools, finish()]);
+```
+
+`result.answer` is set only on `finished`, and carries what the model said the
+answer was — not the surrounding prose.
+
+---
+
+## A failed tool does not end the run
+
+`invoke` returns a failed result rather than throwing, so a tool that fell over
+is something the model can read and work around. It is recorded as a failed step
+and the loop carries on.
+
+```
+  turn_0   "let me check"
+  nope     failed — No tool called "nope". Offered: lookup, finish
+  turn_1   "that tool is not available; answering from what I have"
+```
+
+A loop that crashed on a bad call would lose the whole turn, and with it the
+work already done.
+
+---
+
+## Hooks
+
+Deterministic code at fixed points. A guardrail written into the prompt is a
+request; this is enforcement — it runs whether or not the model read the
+instruction, and next turn's wording cannot argue it away.
+
+```ts
+await loop({
+  tools: bus,
+  turn,
+  hooks: {
+    beforeTurn: () => (overDailyBudget() ? { block: "daily budget spent" } : undefined),
+    beforeTool: (call) =>
+      call.name === "issue_refund" && Number(call.args.amount) > 50_000
+        ? { block: "over the approval limit" }
+        : undefined,
+    afterTool: (result) => audit.write(result),
+    afterTurn: (turn) => log.debug(turn),
+  },
+});
+```
+
+### Gate on what a tool does, not on what it is called
+
+`beforeTool` is handed the effect the tool declared, so a rule can be written
+against the thing that matters:
+
+```ts
+hooks: {
+  beforeTool: (call, effect) => {
+    if (effect === "destructive") return { block: "needs a person" };
+    if (effect === undefined) return { block: `${call.name} has not said what it does` };
+  },
+}
+```
+
+The version above it — `call.name === "issue_refund"` — is a rule that stops
+firing the day somebody renames the tool. Nothing errors, no test goes red, and
+the refund goes through. A rule written against `effect` survives the rename,
+because it never knew the name.
+
+`undefined` means the tool declared nothing, and it is deliberately not
+`readonly`. Blocking it is the safe direction; letting it through means every
+tool written before annotations existed is treated as harmless. Use
+`bus.unannotated()` in a test to keep that list empty.
+
+The effect comes from whatever you passed as `tools`: a `toolbus()` answers from
+its registry, a `bus.trace()` from the shortlist it offered that turn. Anything
+else with just an `invoke` is still a valid `tools`, and every effect reads
+`undefined` — which is what it is.
+
+### Blocking a tool is not an error
+
+A refused call comes back as a **failed tool result**, in the shape the model
+already knows how to read:
+
+```
+  turn_0         "refunding"
+  issue_refund   failed — blocked: over the approval limit
+  turn_1         "understood, escalating"
+```
+
+So the model reads the refusal and works around it. Throwing instead would lose
+the turn and the work already done in it. The tool itself never runs — a
+guardrail that refunds first and refuses afterwards is not a guardrail.
+
+### Blocking a turn ends the run
+
+There is no way around a blocked turn, so it stops with `stoppedBy: "blocked"`
+and a `failed` status. That is the shape for a budget ceiling or a kill switch.
+
+### A hook that throws is treated as permission
+
+A bug in the guardrail must not take down the thing it was guarding. The call it
+was checking has not happened yet, the next hook still gets its say, and the run
+continues. The alternative — an unreachable policy service ending every run —
+fails far more often than the case it was protecting against.
+
+The observing hooks, `afterTool` and `afterTurn`, change nothing when they
+throw. They were only watching.
+
+### Why hooks rather than an `if` inside the tool
+
+A blocked call is a **step**, so "what did this agent try to do and get refused"
+is a query rather than a grep. `afterTool` sees blocked results as well as
+successful ones, which is what an audit trail actually wants.
+
+## Telling the model what is left
+
+The ceilings stop a run. They do not stop the model planning as if there were no
+ceilings, which is how turn seven of eight opens a five-step approach and gets
+cut off after the first — paid for in full.
+
+`remaining` is on the state your `turn` already receives:
+
+```ts
+turn: ({ remaining }) =>
+  model.call([
+    { role: "user", content: question },
+    { role: "user", content: `${remaining.turns} turns and about ${remaining.tokens} tokens left.` },
+  ], { tools: bus.trace(question) });
+```
+
+| | |
+| --- | --- |
+| `remaining.turns` | turns left, **this one included** — the last turn reads `1`, not `0` |
+| `remaining.tokens` | what is left of `maxTokens`, floored at zero |
+
+`beforeTurn` gets the same state, so a run can be stopped rather than started
+with a budget that cannot finish it:
+
+```ts
+beforeTurn: ({ remaining }) =>
+  remaining.tokens < 30_000 ? { block: "not enough left to finish" } : undefined,
+```
+
+### Putting it in the prompt is your job
+
+The loop hands you the number and stops there. Which wording works is
+task-specific, and a library that quietly appends a sentence to your messages
+invalidates your cached prefix without asking.
+
+### No ceiling reads as `Infinity`, never zero
+
+This is the whole reason the field needs a test. The obvious check —
+
+```ts
+if (remaining.tokens < 1000) return "wrap up now";
+```
+
+— would fire on the **first turn of every run that never set `maxTokens`** if a
+missing budget read as zero. The agent would refuse to start work because it
+believes it is already out, and it would look like a model problem rather than a
+library one.
+
+---
+
+## Compaction
+
+A long run gets expensive because the transcript grows, not because the work got
+harder. Every turn resends what came before, so the tool results are the bulk of
+what you pay for on turn eight.
+
+```ts
+await loop({
+  tools: bus,
+  maxTokens: 40_000,
+  compact: {
+    at: 0.7,                                       // fraction of the ceiling
+    keep: 4,                                       // recent results, verbatim
+    summarise: (older) => lead.call(summarisePrompt(older)),
+  },
+  turn: ({ results, last }) => llm.call(messagesFrom(results, last), { tools: bus }),
+});
+
+result.compactions;   // { count: 2, replaced: 11 }
+```
+
+The older results are replaced by **one synthetic result named `compacted`**,
+which the next turn reads like any other. The most recent `keep` survive
+verbatim: summarising what just happened is how a loop forgets what it was in
+the middle of doing.
+
+### It measures the context, not the spend
+
+The threshold is on **the last turn's `inputTokens`** — the size of what is
+actually being resent — and not on what the run has spent so far.
+
+Spend only ever rises. A threshold on it fires once and then on every turn after,
+compacting an already-compacted history again and again. The input count falls
+the moment a compaction lands, so the next turn is under the line and it does not
+fire again until the history has genuinely grown back.
+
+Set `after: 12` instead to compact on a result count, which needs no ceiling.
+
+### What it costs
+
+The summary is a model call, recorded as a `compact` step with its tokens and
+counted against `maxTokens`. A compaction that costs more than it saves still has
+to be paid for, and the step is there so you can see whether it did.
+
+**A failed summariser is survivable.** The run carries on with the history it
+has rather than ending over a summary it could not write — losing the compaction
+costs tokens, losing the run costs the work.
+
+**The honest caveat:** compaction loses information, and the loss is silent. A
+run that compacted and then got the answer wrong looks identical to one that was
+simply wrong.
+
+## Reading `stoppedBy` in production
+
+| | What it means |
+| --- | --- |
+| `finished` | working as intended |
+| `quiet` | offer it `finish()`, or the prompt is not telling it what done looks like |
+| `turns` | the task needs more steps than you allowed, or it is going in circles |
+| `tokens` | the ceiling is too low, or the history is growing faster than the work |
+| `failed` | the provider failed; not an agent problem |
+
+A rising share of `turns` is the signal to look at the tool descriptions: a model
+that cannot tell two tools apart calls both and gets nowhere.
+
+---
+
+## When not to use it
+
+**The steps are known in advance.** That is `chain` — a loop pays for the model
+to decide what to do next, and if you already know, you are paying for nothing.
+
+**There are no tools.** A loop with an empty bus is one call with extra
+machinery.
+
+**The work is unbounded and the budget is not.** This is the workflow that can
+spend the most, which is why both ceilings are here and why `maxTurns` has a
+default when the others do not.
+
+---
+
+## How the files fit
+
+```
+   types.ts     the contract
+        ▲
+   turn.ts      one model call: what it cost, how it is recorded
+   act.ts       running what it asked for, and reading `finish`
+        ▲
+   loop.ts      the cycle and its ceilings
+```
+
+Not compiled to a graph, unlike `route` and `chain`. This is a cycle, and
+`@spendgraph/graph` is a DAG whose step ceiling exists to stop one — same
+reasoning as `refine`.

package/docs/orchestrate.md

@@ -0,0 +1,128 @@
+# orchestrate
+
+A lead decomposes, workers do the pieces, the lead puts it back together.
+
+```ts
+import { orchestrate } from "@spendgraph/harness";
+
+const result = await orchestrate(
+  { brief },
+  {
+    plan: async (v) => {
+      const reply = await lead.call(planPrompt(v), { schema: SUBTASKS });
+      return { ...reply.data, model: reply.model, inputTokens: reply.inputTokens, outputTokens: reply.outputTokens };
+    },
+    work: (subtask, v) => worker.call(sectionPrompt(subtask, v)),
+    synthesise: (results, v) => lead.call(assemblePrompt(results, v)),
+    maxWorkers: 5,
+    concurrency: 4,
+    maxTokens: 80_000,
+  }
+);
+
+result.subtasks;   // what the lead asked for, after the cap
+result.dropped;    // what the cap refused
+result.stoppedBy;  // completed · planned-nothing · workers-failed · tokens · failed
+```
+
+The workflow for work whose **shape is not known until the input is read** —
+which is the whole difference from `chain`, where the stages are named in
+advance.
+
+---
+
+## The cap is the point
+
+This is the workflow that can spend the most, because the **lead** decides how
+many calls to make. Asked to decompose a large input it will cheerfully invent
+twelve subtasks, and twelve model calls leave before anyone has decided that was
+reasonable.
+
+`maxWorkers` defaults to 5. Anything past it is **dropped and said**, never
+dropped quietly:
+
+```
+  result.subtasks   ["a", "b", "c"]
+  result.dropped    ["d", "e", "f", "g"]
+  steps[0].error    "4 subtask(s) past the worker cap were dropped: d, e, f, g"
+```
+
+It keeps the head and drops the tail: a lead asked for the most important piece
+first usually gives it first, and truncating from the front would throw away
+exactly what it thought mattered.
+
+A duplicated name is made unique — `summary`, `summary_1` — rather than
+rejected. A lead naming two sections the same thing is a wording problem, not a
+reason to lose the run.
+
+---
+
+## When the lead finds nothing to do
+
+```ts
+{ stoppedBy: "planned-nothing", output: "There is nothing here to summarise." }
+```
+
+The workers never run and `synthesise` is never called. Synthesising over an
+empty list asks a model to write an answer from nothing, and it will — which is
+worse than saying there was nothing.
+
+---
+
+## When a worker falls over
+
+`minSuccess` decides, and it defaults to **all**.
+
+```ts
+minSuccess: 2   // three sections planned, two is enough
+```
+
+Sectioning a document usually wants all of them: a hole in the middle is worse
+than no answer. Research-style fan-out often does not — two sources out of three
+still supports a conclusion. Without a `minSuccess` the run fails and
+`synthesise` is not called, so the lead is never asked to write around a gap it
+does not know about.
+
+---
+
+## The ceiling is checked twice
+
+Once after planning and once before synthesising, because both are places where
+the run can be abandoned before spending more:
+
+- the plan alone reaching the ceiling means the workers never start
+- the workers reaching it means the lead is never asked to assemble
+
+Tokens rather than money, because this package cannot price a call — that
+happens on the server, after the fact.
+
+---
+
+## When not to use it
+
+**The stages are known in advance.** That is `chain`. Paying a lead to decide
+what you already know is a call for nothing.
+
+**The pieces are independent and uniform.** That is `parallel` — you do not need
+a model to tell you a document has three sections if you already know it does.
+
+**One call would do.** Three calls minimum here: plan, work, synthesise. On a
+small input the lead's overhead outweighs the parallelism.
+
+---
+
+## How the files fit
+
+```
+   types.ts          the contract
+        ▲
+   plan.ts           the lead's decomposition, and the cap
+   work.ts           the workers, delegated to `parallel`
+        ▲
+   orchestrate.ts    plan · work · synthesise
+```
+
+`work.ts` does not fan out again — `parallel` already caps the concurrency, turns
+a thrown worker into a result so the others land, and numbers the steps by
+declaration so two runs can be compared. This is the workflow that most benefits
+from the others existing first.

package/docs/parallel.md

@@ -0,0 +1,139 @@
+# parallel
+
+Every task at once, then one answer out of what came back. Two shapes use it:
+**sectioning** buys wall-clock, **voting** buys confidence.
+
+---
+
+## Sectioning — split the work
+
+```ts
+import { parallel } from "@spendgraph/harness";
+
+const result = await parallel(
+  { brief },
+  {
+    tasks: [
+      { name: "intro", run: (v) => llm.call(introPrompt(v)) },
+      { name: "body", run: (v) => llm.call(bodyPrompt(v)) },
+      { name: "close", run: (v) => llm.call(closePrompt(v)) },
+    ],
+    merge: (results) => results.map((r) => r.value.output).join("\n\n"),
+    concurrency: 4,
+  }
+);
+```
+
+Three calls take as long as the slowest, not the sum. The trade against `chain`
+is exact: chain buys accuracy with latency, parallel buys latency with tokens —
+you pay for the same context three times over.
+
+**A missing section fails the run by default.** A hole in the middle of an
+answer is worse than no answer, and `merge` is never called with one. The tasks
+that did land are still recorded, so the failure is legible.
+
+---
+
+## Voting — ask N times, take the consensus
+
+```ts
+import { majority, parallel, repeat } from "@spendgraph/harness";
+
+const result = await parallel(
+  { question },
+  {
+    tasks: repeat("guess", 5, (v) => llm.call(prompt(v))),
+    merge: (results) => majority(results)?.value,
+    minSuccess: 3,
+  }
+);
+
+result.merged;   // the answer most of them gave
+```
+
+`repeat` **rounds an even count up.** Four voters tie, and a tie has no answer
+that is not arbitrary.
+
+`minSuccess: 3` is the point of running five: if one call is overloaded the other
+four still land and still answer. Ties go to the first declared, so the same
+inputs give the same result on every run.
+
+**When voting is worth the tokens:** the model is right most of the time and
+wrong *differently* each time — the wrong answers scatter, the right one repeats.
+
+**When it buys nothing:** the model is confidently wrong the same way every
+time. Five identical wrong answers vote unanimously for the wrong one, and you
+have paid five times to be misled with more conviction. That is the failure it is
+easiest to mistake this pattern for a fix for.
+
+---
+
+## Why it is not a graph
+
+`route` and `chain` compile down to `@spendgraph/graph`. This one cannot:
+`graph` walks one node at a time and picks a single edge, so a fan-out is not a
+shape it can express. The concurrency comes from `mapLimit`, which already exists
+in `@spendgraph/prompt/internals`.
+
+Same reasoning as `refine`, which is a loop rather than a DAG walk.
+
+---
+
+## Steps are numbered by declaration, not by arrival
+
+A fan-out that numbered steps by completion order would record a different run
+every time for identical work, and two runs could not be compared. The slow task
+declared first is still step 0.
+
+```
+  steps:  slow  (index 0)   ← declared first, landed last
+          quick (index 1)
+```
+
+---
+
+## Failure is a result, not an exception
+
+A task that throws comes back as a failed `TaskResult`. The whole point of
+running five is that the other four still land, so one bad call cannot take the
+run down with it.
+
+```ts
+result.succeeded;  // ["a", "b"]
+result.failed;     // ["c"]
+result.results;    // every one, in declaration order
+```
+
+`minSuccess` decides whether that is fatal. Sectioning defaults to all —
+anything less is a hole. Voting should set a majority.
+
+---
+
+## When not to use it
+
+**The tasks need each other's output.** That is `chain`; a fan-out cannot pass
+anything between branches.
+
+**One call would do.** Voting on a task the model gets right every time is five
+times the cost for the same answer.
+
+**The context is enormous.** Sectioning resends it per task. Three sections of a
+50k-token brief is 150k tokens of input, and the latency you bought may cost more
+than it saved.
+
+---
+
+## How the files fit
+
+```
+   types.ts       the contract
+        ▲
+   task.ts        run one, record one
+   merge.ts       repeat · majority
+        ▲
+   parallel.ts    fan out · gather · merge
+```
+
+`task.ts` and `merge.ts` are siblings. `majority` and `repeat` are pure and
+tested on their own — the tie-break and the even-count rounding are the parts
+worth pinning.