@noetaris/harness 0.11.0 → 0.13.0

package/README.md

@@ -29,13 +29,15 @@ interface Ctx {
 }
 
 // 2. Create the harness — fixes Ctx and infers State from the field schema
-const h = createHarness<Ctx>()({
+const base = createHarness<Ctx>()({
   messages:  field<string[]>({ default: () => [],  reduce: (a, b) => [...a, ...b] }),
   toolCalls: field<any[]>   ({ default: () => [] }),
+  toolResults: field<any[]>({ default: () => [] }),
 })
 
-// 3. Define the loop — validated immediately at call time
-h.loop(l =>
+// 3. Define the loop — validated immediately at call time.
+//    Each builder call returns a new harness: keep the result.
+const withLoop = base.loop(l =>
   l.start()
    .step('think', {
      run: async (state, ctx) => {
@@ -57,9 +59,10 @@ h.loop(l =>
 )
 
 // 4. Declare providers
-h.provide('tools',   { search: new MySearchTool() }) // hard-coded
-h.provide('prompts', required())                     // must be supplied at createAgent()
-h.provide('model',   runtime())                      // must be supplied at agent.run()
+const h = withLoop
+  .provide('tools',   { search: new MySearchTool() }) // hard-coded
+  .provide('prompts', required())                     // must be supplied at createAgent()
+  .provide('model',   runtime())                      // must be supplied at agent.run()
 
 // 5. Create an agent — assigns an ID and fills required() slots
 const agent = createAgent('my-agent', h, {
@@ -166,6 +169,7 @@ does depends on the session's latest run:
 | completed | a new turn | returns its outcome, runs nothing |
 | paused on an interrupt | resolves `$error` with `SessionPendingInterruptError` | returns its outcome (`signal: '$interrupt'`), runs nothing |
 | paused by a stop or an unhandled error | merges the input and continues from where it paused | continues from where it paused, **without** merging the input again |
+| a checkpoint (`'running'`: its instance died mid-run, see [Checkpoints](#checkpoints)) | merges the input and continues from the last finished step | continues after the last finished step, **without** merging the input again |
 
 A retry of an earlier request (not the latest) is found with the store's optional
 `findRequest` and its outcome is returned; it never runs again, since later turns have
@@ -187,6 +191,106 @@ Notes:
 - `sessionId` and `requestId` must be non-empty strings when given; otherwise `agent.run()`
   throws `InvalidRunResourceError`.
 
+### Checkpoints
+
+By default a session is saved only when a run settles. If the instance dies in the middle of a
+long turn, the next run starts the whole turn again, and the side effects of the steps that
+already finished happen again. With `checkpoint: 'step'`, the session is also saved after every
+finished top-level step:
+
+```ts
+const agent = createAgent('agent', h, {}, { checkpoint: 'step' })   // every run of this agent
+agent.run(input, { sessionId, requestId, checkpoint: 'off' })        // or per call: 'off' | 'step'
+```
+
+```
+ turn ── a ── save ── b ── save ── c ──✕ crash
+                         ↑
+       next run ─────────┘ continues from c; a and b do not run again
+```
+
+- Each checkpoint is a new stored run with `phase: 'running'`, `step` = the next step to run,
+  and the state after the finished step. `agent.status()` reports it as
+  `{ phase: 'running', step, instanceId?, checkpointedAt }`.
+- Running the session again continues from that step, like a stopped run (see the table above):
+  a retry with the same `requestId` does not merge its input again. With a store that supports
+  claims, another instance gets `SessionBusyError` while the owner's claim is alive, and can take
+  over once it expires.
+- Steps run **at least once**: only the step that was running when the instance died runs
+  again, so only that step needs to be idempotent.
+- A fork is saved as a whole: there is a checkpoint before it and after it, none inside. A crash
+  inside a fork runs the whole fork again.
+- After each checkpoint the claim is extended. A single step that runs longer than the claim TTL
+  still needs `ctx.keepAlive()`.
+- `onCheckpoint(ctx, { step, version, durationMs })` on an [observer](#observers) reports each
+  saved checkpoint. A checkpoint that is skipped (its state can't be copied with
+  `structuredClone`, or the store failed) is reported to `events.onStoreError` with phase
+  `'checkpoint'`, and the run goes on.
+- When the store reports that another writer saved the session first, or that this instance's
+  claim was taken over, the run stops at once with `LeaseExpiredError` (reported with phase
+  `'claim'`) and saves nothing more. Detecting a taken-over claim needs a store that checks
+  `SaveOptions.lease` on `save()` (see [Fencing](#fencing)).
+- Without a session store the option does nothing. Each checkpoint is one store write per step.
+
+Limits:
+- Nothing in core looks for runs whose instance died. The session continues only when something
+  runs it again: a retry with the same `requestId`, a queue that redelivers the request, or your
+  own sweep over sessions whose `agent.status()` is `'running'`.
+- Every checkpoint is a stored run, and none are removed: `loadHistory()` grows by one run per
+  finished step. Filter by `phase` when you read it.
+- `branch()` starts from the last stored run of a `runId`. It can't start from an earlier
+  checkpoint of that run.
+- It is all top-level steps or none; you can't pick which steps save.
+- A step has no fencing token to pass to other systems. A stale instance is stopped at its next
+  save to the session store, but what its step already wrote elsewhere stays written.
+- If a save reached the store but the call still failed (a timeout, a dropped connection), the
+  next checkpoint conflicts with it and the run stops with `LeaseExpiredError`. The next run
+  continues from that save; steps still run at least once.
+
+> **Upgrading.**
+> - `StoredRun.phase` and `SessionPhase` have a new value, `'running'`. A `switch` that handles
+>   every phase needs a case for it.
+> - `checkpoint` is now a reserved run resource. A `runtime()` slot named `checkpoint` no longer
+>   reaches `ctx`, and the value passed under that key must be `'off'` or `'step'`. Rename the slot.
+> - `SessionStore.save` takes an optional fourth argument, `SaveOptions`. Existing stores keep
+>   working, without [fencing](#fencing).
+> - With a store that fences, a run whose final save is rejected because it lost its claim now
+>   settles `$error` with `LeaseExpiredError`. Before, it returned its result although the store
+>   did not keep it.
+
+### Fencing
+
+With a store that supports claims, a run holds a lease on its session. A lease can be lost while
+the run still works: the claim expires (a step ran longer than the claim TTL without
+`ctx.keepAlive()`), and another instance claims the session and runs it. The first instance is
+now stale. Without fencing, its next save would overwrite the new owner's progress.
+
+Core already stops a run at the next step boundary once its lease has expired by the local clock.
+Fencing covers what that check can't see: a save made before that boundary, and clocks that differ
+between instances.
+
+So every save a run makes while it holds a claim (each checkpoint, the final save, and writing an
+interrupt answer on `resume()`) passes the lease as `SaveOptions.lease`. A store that fences
+checks, in the same atomic step as the version check, that the session's claim is still that
+lease. If not, it writes nothing and throws an error named `LeaseLostError`.
+
+```
+ A (stale): save {lease A} ──► store: claim is now B's → LeaseLostError → A stops, $error
+ B (owner): save {lease B} ──► store: claim ok, version ok             → saved
+```
+
+What the caller of the stale run sees:
+- The run settles `signal: '$error'` with `LeaseExpiredError` in `state.$error`. Its `cause` is the
+  store's `LeaseLostError`. `events.onStoreError` gets the same `LeaseExpiredError` with phase
+  `'claim'`.
+- Nothing more is saved for that run.
+- Any other error from the final save is still only reported (phase `'persist'`), and the run
+  keeps its signal.
+
+`PostgresSessionStore` and `RedisSessionStore` fence. `InMemorySessionStore` and
+`LocalFileSessionStore` have no claims, so there is nothing to fence. A custom store that ignores
+`SaveOptions` keeps only the version check.
+
 ### Interrupts
 
 A run can be stopped or resumed:
@@ -299,14 +403,99 @@ h.loop(l =>
 If a fork fails fast while a sub agent is paused, that sub agent's own session stays
 paused in its store — clean it up or reuse it the next time the step runs.
 
+### Debugging a run
+
+`agent.debug(initialState, resources)` starts a run held before its first step and returns a
+`DebugSession`. Each command runs part of the run and resolves with the next **stop**: where the
+run is held, the step that just settled, and a copy of the state. It takes the same arguments as
+`agent.run()` and drives the whole run chain — an interrupt is answered inside the session.
+
+```typescript
+const dbg = agent.debug({}, { sessionId: 's1' })
+
+let stop = await dbg.current          // held before the first step: kind 'step', reason 'start'
+stop = await dbg.next()               // run one step; stop.last is the step that settled
+if (stop.kind === 'step') console.log(stop.last?.name, stop.last?.update?.values)
+
+dbg.setBreakpoint('review', (state) => (state.score as number) < 0.5)
+stop = await dbg.continue()           // runs until a breakpoint, an interrupt or the end
+
+if (stop.kind === 'step') dbg.setState({ score: 0.9 })        // edit state before 'review' runs
+stop = await dbg.continue()
+if (stop.kind === 'interrupt') stop = await dbg.answer(true)  // resume; held before the re-run
+
+stop = await dbg.continue()           // { kind: 'end', outcome: { state, signal } }
+```
+
+The session is local to this process. While it lasts, `run()`, `resume()` and `debug()` on its
+session throw `SessionInDebugError` here; see [LIMITATIONS.md](./LIMITATIONS.md) for other processes.
+
+**Stops** (`DebugStop`):
+
+| `kind` | Fields | When |
+|---|---|---|
+| `step` | `reason`, `next` (the step about to run), `last` (the step that just settled: `name`, `outcome`, `update`, `signal`, `next`, `error`, `before`), `state`, `frame` | held at a step boundary. `reason` is `start`, `step`, `breakpoint`, `resume` (after `answer`), `fork-entry` or `branch-end` |
+| `interrupt` | `interruptId`, `prompt`, `pending` (every interrupt the run waits on), `last`, `state`, `frame` | a step called `ctx.interrupt()` |
+| `end` | `outcome` (`{ state, signal }`), `last` | the run completed, paused on an error, or was stopped |
+
+`state`, `last.before` and `last.update` are `Snapshot`s: `{ values, shared }`. Each top-level key
+is copied with `structuredClone`; a key that can't be cloned (a function, say) keeps its reference
+and is listed in `shared`.
+
+**Commands** (`DebugSession`):
+
+| Command | Does |
+|---|---|
+| `current` | promise of the latest stop (the start stop at first) |
+| `next()` | run exactly one step (a fork counts as one step) |
+| `continue()` | run until a breakpoint hits, an interrupt, or the end |
+| `setBreakpoint(step, when?)`, `clearBreakpoint(step)`, `breakpoints` | stop before `step` when `when(state)` is truthy (always without `when`); one breakpoint per step |
+| `setState(patch)` | while held at a `step` stop: overwrite keys before the next step (no reducers; values are copied; `$` keys are refused). Observers get event `debug.setState` with `{ patch }` |
+| `answer(response, interruptId?)` | at an `interrupt` stop: resume through the normal resume path and hold before the resumed step |
+| `stop()` | end the session at any time; resolves with the `end` stop. At an interrupt stop the interrupt stays pending as saved |
+| `done` | promise of the last run's outcome, as `await handle` gives it |
+
+One command at a time: a second `next()`, `continue()` or `answer()` while one runs rejects with
+`DebugCommandError` (`reason: 'busy'`). Other reasons: `not-held`, `not-a-fork`, `reserved-key`,
+`ended`.
+
+While the run is held, the session renews its claim on the session store (when the store supports
+claims), so the lease doesn't expire during a pause. Aborting the `signal` resource ends the
+session in any state; so does `stop()`. Step durations leave out the time held.
+
+#### Fork branches
+
+`next()` steps over a fork. To step inside it, call `stepInto()` when the next step is a fork:
+every branch starts and is held at its first step, and the first branch is focused. Branches are
+**frames**, named by their branch path (`[]` is the top level).
+
+```typescript
+stop = await dbg.stepInto()           // { frame: ['x'], reason: 'fork-entry', next: 'sx' }
+dbg.frames                            // [{ frame: [], status: 'in-fork' }, { frame: ['x'], status: 'held' }, …]
+stop = dbg.focus(['y'])               // switch to another held branch
+dbg.setState({ note: 'edited in y' }) // counts as written by branch y, so the join keeps it
+stop = await dbg.continueAll()        // release every branch; resolves with the first stop any frame makes
+```
+
+`next()` and `continue()` drive only the focused frame; other branches wait. When the driven frame
+ends, focus moves to the next held frame (`reason: 'branch-end'`, with `ended`). Breakpoints hit in
+any frame. If a branch fails and the fork fails fast, held siblings are released and end.
+
 ### Observers
 
 An `Observer` receives telemetry hooks for a run. Pass it as `observer` in the resources
-of `agent.run()` or `agent.resume()`. Combine several with `composeObservers([a, b])`;
+of `agent.run()` or `agent.resume()`, or once for every run of an agent in
+`createAgent(id, h, slots, { observer })`. Combine several with `composeObservers([a, b])`;
 a hook that throws is reported to `onObserverError` (or `console.error`) and never stops
 the run. The observer that LLM adapters call is isolated the same way, so an observer that
 throws on an adapter event never fails a step. Every hook is optional.
 
+An agent's observer serves `agent.run()`, `handle.resume()` and `agent.resume()`. A per-call
+`observer` is composed on top of it, never replacing it: the agent's hooks fire first, its `wrap` is
+outermost, and on a shared `carry` key the per-call value wins. Passing the agent's own observer per
+call uses it once. Hook errors go to the per-call `onObserverError`, else the agent's
+`onObserverError` option, else `console.error`.
+
 | Hook | Fires |
 |---|---|
 | `onRunStart(ctx)` | once when a run starts |
@@ -319,6 +508,7 @@ throws on an adapter event never fails a step. Every hook is optional.
 | `onStepSettled(ctx, event)` | once per step, when its outcome and destination are final |
 | `wrap(ctx, scope, fn)` | around a step's `run`, and around calls made with `ctx.within` |
 | `carry(ctx)` | once per top-level run, right after `onRunStart`; returns a string map to keep on the run's record (see [Linking runs](#linking-runs)) |
+| `onCheckpoint(ctx, { step, version, durationMs })` | after a checkpoint was saved, before `step` starts; top-level runs with `checkpoint: 'step'` only (see [Checkpoints](#checkpoints)) |
 
 Step hooks receive a `StepContext`: `agentId`, `sessionId`, `runId`, `stepName` and, inside a
 fork branch, `branchPath`. `runId` is the same value as `RunContext.runId`. Steps inside a branch
@@ -462,28 +652,41 @@ The harness never trusts a wrapper. `fn` always runs exactly once, and the calle
 | Export | Description |
 |---|---|
 | `createHarness<Ctx>()(schema)` | Creates a harness. Fixes `Ctx`, infers `State` from schema. |
-| `createAgent(id, h, slots, options?)` | Assigns the agent an ID and fills `required()` slots. `options` (`AgentOptions`) takes `instanceId`, `stores` (for `required()` store slots) and `claimOptions` (this agent's default claim TTL; per-call `claimOptions` wins, then `HARNESS_CLAIM_TTL_MS`, then 30 000 ms). Returns an `Agent`. |
+| `createAgent(id, h, slots, options?)` | Assigns the agent an ID and fills `required()` slots. `options` (`AgentOptions`) takes `instanceId`, `stores` (for `required()` store slots), `claimOptions` (this agent's default claim TTL; per-call `claimOptions` wins, then `HARNESS_CLAIM_TTL_MS`, then 30 000 ms), `observer` (for every run of the agent; see [Observers](#observers)), `onObserverError` (the default sink for hook errors) and `checkpoint` (`'off'` or `'step'`; see [Checkpoints](#checkpoints)). Returns an `Agent`. |
 | `field<T>(opts)` | Declares a state field with a default and optional reduce function. |
 | `required()` | Marks a provider slot as required at `createAgent()`. |
 | `runtime()` | Marks a provider slot as required at `agent.run()`. |
 | `composeObservers([a, b], onObserverError?)` | Merges multiple `Observer` instances into one fan-out observer; a throwing observer is isolated and reported to `onObserverError`. |
 | `SessionStore` | Interface for session persistence backends. Optional extensions: `loadHistory`, `branch`, `claim` / `release` / `extendClaim`, and `findRequest` (the newest run of a request, for retries of earlier requests). |
-| `StoredRun` | Type for a persisted run snapshot. Includes `agentId`, `runId`, `sessionId`, `phase`, state, and optionally `requestId` (the run's turn), `carrier` (the run's `carry` result) and `branchedFrom` (on a branch seed). |
+| `SaveOptions` | Last argument of `SessionStore.save`: `{ lease? }`, the lease the caller holds. A store that supports claims rejects a save whose lease is no longer the session's claim with an error named `LeaseLostError`. |
+| `CheckpointMode` | `'off' \| 'step'` — the `checkpoint` option and resource. |
+| `StoredRun` | Type for a persisted run snapshot. Includes `agentId`, `runId`, `sessionId`, `phase` (`'paused'`, `'completed'` or `'running'` for a checkpoint), state, and optionally `requestId` (the run's turn), `carrier` (the run's `carry` result) and `branchedFrom` (on a branch seed). |
 | `RunRef` | `{ runId, carrier? }` — points at an earlier run; used by `RunContext.resumedFrom`, `continuedFrom` and `branchedFrom` (which adds `sessionId`). |
 | `RunCarrier` | `Readonly<Record<string, string>>` — the opaque map an observer's `carry` returns. |
 | `Observer` | Interface for telemetry hooks on run and step lifecycle events (see [Observers](#observers)). |
 | `StepSettledEvent` | Payload of `Observer.onStepSettled` — outcome, duration, update, signal, next, error, interrupt, fork. |
+| `CheckpointEvent` | Payload of `Observer.onCheckpoint` — `step`, `version`, `durationMs`. |
 | `StepSettledNext` | Where the run goes after a step settles: `step`, `end`, `pause`, or `throw`. |
 | `ObserverAware` | Interface for provider objects that want per-step telemetry: the harness calls `withTelemetry(telemetry)` once per step and installs the returned view on `ctx`. |
 | `TelemetryContext` | What `withTelemetry` receives: `observer` (the run's observer, fault-isolated), `stepContext`, and `within(type, payload, fn)` bound to that step. |
 | `WrapScope` | What `Observer.wrap` is asked to enclose: `{ kind: 'step' }` or `{ kind: 'event', type, payload }`. |
 | `RunContext` | Context passed to run-level observer hooks — `agentId`, `sessionId`, `runId`, `parentRunId?`, `instanceId?`, `branchPath?`, `resumedFrom?`, `branchedFrom?`, `continuedFrom?`, `requestId?` (see [Linking runs](#linking-runs)). |
 | `StepContext` | Context passed to step-level observer hooks — `agentId`, `sessionId`, `runId`, `stepName`, `branchPath?`. |
-| `InvalidRunResourceError` | Thrown by `agent.run()` when `sessionId` or `requestId` is given but is not a non-empty string; `key` and `value` name it. |
+| `InvalidRunResourceError` | Thrown by `agent.run()` when `sessionId` or `requestId` is given but is not a non-empty string, and by `run()`, `debug()` and `resume()` when `checkpoint` is not `'off'` or `'step'`; `key` and `value` name it. |
 | `NoInterruptError` | Thrown when `resume()` is called but the session is not paused on a matching interrupt. |
 | `SessionInFlightError` | Thrown when a session is already running. |
 | `SessionPendingInterruptError` | `agent.run()` on a session paused on a pending interrupt — use `agent.resume()` instead. With a session store, the run resolves with `signal: '$error'` and this error in `state.$error`; without one, `agent.run()` throws it. |
 | `StoreLoadError` | Thrown when the session store fails to load state. |
+| `LeaseExpiredError` | In `state.$error` when a run lost its claim: the lease expired, or the store rejected a save because the session's claim is no longer this run's lease (see [Fencing](#fencing)). `sessionId`; `cause` holds the store's error when there is one. |
+| `InvalidAgentOptionError` | Thrown by `createAgent()` when `observer` is not an object, `onObserverError` is not a function, or `checkpoint` is not `'off'` or `'step'`; `key` and `value` name it. |
+| `InvalidClaimOptionsError` | Thrown by `createAgent()` when `claimOptions.ttlMs` is not a finite number greater than 0. |
+| `DebugSession` | Returned by `agent.debug()`: a run driven step by step (see [Debugging a run](#debugging-a-run)). |
+| `DebugStop` | Where a debug session stopped: `step`, `interrupt` or `end`. |
+| `SettledStep`, `Snapshot` | The step that just settled at a stop, and a per-key copy of a state or update. |
+| `DebugFrame`, `EndedFrame`, `PendingInterrupt` | A fork branch held by a debug session, a frame that ended, an interrupt the run waits on. |
+| `DebugBreakpoint` | `{ step, when? }`. |
+| `DebugCommandError` | A debug command that can't run now; `reason` is `busy`, `not-held`, `not-a-fork`, `reserved-key` or `ended`. |
+| `SessionInDebugError` | `run()`, `resume()` or `debug()` on a session that is being debugged in this process. |
 
 ## Design Principles
 
@@ -508,6 +711,8 @@ See [LIMITATIONS.md](LIMITATIONS.md) for known limitations.
 - [`@noetaris/harness-openai`](https://github.com/noetaris-lab/harness-openai) — OpenAI adapter
 - [`@noetaris/harness-google`](https://github.com/noetaris-lab/harness-google) — Google Gemini adapter
 - [`@noetaris/harness-otel`](https://github.com/noetaris-lab/harness-otel) — OpenTelemetry observer bridge
+- [`@noetaris/harness-recorder`](https://github.com/noetaris-lab/harness-recorder) — records runs step by step and plays a finished session back
+- [`@noetaris/harness-testing`](https://github.com/noetaris-lab/harness-testing) — `runStep`, `runRoute`, `MockObserver` for tests
 - [`@noetaris/harness-otel-genai`](https://github.com/noetaris-lab/harness-otel-genai) — GenAI semantic-conventions extension for the bridge
 
 ## License