nexus-ai-pro 2.8.0 → 2.9.0

package/API_STABILITY.md

@@ -41,6 +41,40 @@ Both are covered by this policy:
 Deep imports into `dist`, `dist-cjs`, or `src` are not supported. Import only from `nexus-ai-pro` or
 one of its explicit subpaths.
 
+## 2.9 stage (2.9.0)
+
+Everything 2.9 adds is stable from the start and additive, unless this section says otherwise.
+
+New and stable:
+- `OperationFamily` gains `video`;
+- `nexus-ai-pro/video`, `/video/google`, `/video/runway`, and `/video/mock` are new and experimental:
+  video generation graduates by the checklist once Veo and Runway pass its conformance suite live;
+- `RunQuery.after` with `RunCursor`, and `runCursor()` and `isAfterCursor()` on
+  `nexus-ai-pro/tracing`. A store of your own that does not honour `after` still works with the
+  online-evaluation worker, which falls back to offsets;
+- `OnlineSchedulePass.paging`;
+- on `nexus-ai-pro/lifecycle`: `BudgetReservation.id`; `BudgetReservationStatus`,
+  `BudgetReservationRecord`, `BudgetReservationOutcome`, `BudgetRecoveryReport`, and
+  `BudgetRecoverOptions`; `BudgetLedger.renew()`, `settle()`, and `recover()`, all optional;
+  `BudgetLedgerOptions.leaseMs` and `uncertain`; `BudgetLedgerWithReport.reservation()`; and the five
+  optional reservation methods of `BudgetUsageStore` with `BudgetReservationChange`. The statuses,
+  and which outcome moves a reservation from which status, are part of the promise;
+- `hold()`, `settle()`, `renew()`, `lapsed()`, and `reservation()` on `MemoryTenantUsage`,
+  `RedisTenantUsage`, and `PostgresTenantUsage`, and the tenancy migration version 2 with its
+  reservations table;
+- on `nexus-ai-pro/media/rag`, experimental with the media family: `MediaIndexOptions.manifest` and
+  `name`, `MediaIndexEntry`, `MediaIndexReader`, `MediaIndexReport.unchanged`, and
+  `MediaIndex.delete()`, `reindex()`, `describe()`, and `list()`.
+
+Changed, each to keep a promise the surface already made:
+- `TraceStore.query()` orders runs that share a start time by id, from last to first, as code units,
+  so a page means the same on every store. Before, that order was not defined.
+- `MemoryBudgetUsage`, the ledger's default store, records reservations, so `reserve()` returns one
+  with an `id`. An online-evaluation worker reserves every run of its evaluators, a run after a crash
+  included, where before it reserved only a first.
+- A deep agent with a checkpointer writes the helpers it creates to `<thread>/helpers-created`, so
+  `createHelpers.max` holds for a thread across processes, as it says.
+
 ## 2.8 stage (2.8.0)
 
 Everything 2.8 adds is stable from the start and additive, unless this section says otherwise.

package/CHANGELOG.md

@@ -4,6 +4,113 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [2.9.0] - 2026-10-15
+
+Video, and everything shipped, proven.
+
+### Added
+
+- **Video generation.** `nexus-ai-pro/video`: `VideoManager` makes videos from a prompt, a first or
+  last frame, or reference images, through a provider's asynchronous job. `generate()` is a durable
+  operation that records the job the moment it exists and each video as it is stored, so
+  `recover()` in another worker collects a job whose process died, never submitting it twice, and
+  streams each video into an asset store. Each generation runs in the lifecycle's new `video`
+  family, and is priced per second by `priceVideo()` from `VIDEO_PRICES`, the Veo 3.1 list prices,
+  or the account's own. `start()`, `resume()`, `status()`, `download()`, and `cancel()` work from a
+  JSON `VideoJobRef` in any process. A request is checked against the model's capabilities before
+  any job, and a failed job is submitted again only when the provider says that may help.
+  - `GoogleVideoProvider` (`/video/google`): Veo through its long-running predictions, on the
+    Gemini API, the video read from Google's URI with the key, or on Vertex AI, read from Cloud
+    Storage or inline.
+  - `RunwayVideoProvider` (`/video/runway`): Runway's task API, for its own models and the hosted
+    ones it serves, Veo, Gemini Omni Flash, and Seedance among them; each video read from its signed
+    URL without the secret.
+  - `MockVideoProvider` (`/video/mock`), and `runVideoConformance()`, which all three pass.
+
+  New guide `docs/video.md`. The family is experimental until Veo and Runway pass the conformance
+  suite live. OpenAI's Videos API was removed on 2026-09-24, so there is no OpenAI adapter.
+- **The experimental families, live.** `npm run test:families:live`, and a weekly *Family
+  conformance* job beside the vector stores: media understanding's conformance suite against an
+  OpenAI or Gemini vision model and Mistral's OCR, on a PDF and a PNG the script builds; a two-input
+  batch through the Gemini embedding batch tier, waited on for 20 minutes and cancelled past that;
+  and, when the repository variable `NEXUS_LIVE_VIDEO` is set, the video conformance suite on Veo 3.1
+  Lite and Runway. Each check runs where its key is set, and what it measured is kept. These are the
+  runs media, the Gemini batch tier, and video graduate on.
+- **Paging by cursor.** `RunQuery.after` takes a `RunCursor`, the start time and id of a page's last
+  run, and returns only the runs after it; `runCursor()` takes the cursor from a run, and
+  `isAfterCursor()` is the test on its own, for a trace store of your own. Every bundled store
+  honours it, and `applyQuery()` does for a store that filters in memory. A new Postgres trace store
+  migration, version 3, indexes runs in query order.
+- **Budget reservations that outlive their process.** Over a store that records reservations, a
+  `budgetLedger()` holds each one as a `BudgetReservationRecord` under a lease (`leaseMs`, 10
+  minutes), and moves it with its total in one step: `reserved`, then `settled`, `released`,
+  `expired`, or `uncertain`. `recover(resolve)` settles reservations whose lease lapsed by what
+  became of each operation, a `BudgetReservationOutcome`; `settle(id, outcome)` does it for one from
+  any process; `renew()` extends a lease; and `uncertain: 'charge' | 'release'` says what an unknown
+  outcome costs. `MemoryBudgetUsage`, `MemoryTenantUsage`, `RedisTenantUsage`, and
+  `PostgresTenantUsage` all record them: Redis keeps a reservation in its total's Cluster slot, so one
+  script moves both, and Postgres does it in one statement, in the reservations table of a new
+  tenancy migration, version 2. A store of your own records them through five optional methods on
+  `BudgetUsageStore`.
+- **A media index that remembers what it holds.** `MediaIndexOptions.manifest`, any `Store`, with a
+  `name`, keeps one `MediaIndexEntry` per medium: a hash of its content, source, metadata, and
+  indexing options, where it is kept, its pages, its chunk ids, the embedding model, and the
+  `MediaIndexReader` that read and described it. `add()` then skips a medium at the same version and
+  replaces exactly the chunks and asset of one that changed, and `MediaIndex` gains `delete()`,
+  `reindex()`, `describe()`, and `list()`. `MediaIndexReport.unchanged` counts what was skipped. A
+  change is recorded as pending first, so a crash leaves nothing the next call cannot finish.
+
+### Changed
+
+- Runs that started in the same millisecond come by id, from last to first, compared as code units,
+  on every trace store. The order of such runs was not defined before, and the memory and Postgres
+  stores disagreed on it.
+
+### Fixed
+
+- An online-evaluation worker reads a backlog of runs that started in the same millisecond through,
+  however many pages it spans. Before, it resumed each page at the last run's start time, so a page
+  that ended among such runs was read again on every pass, and `drain()` never finished. On a trace
+  store of your own that does not honour `RunQuery.after`, the worker pages by offset instead, and
+  its `OnlineSchedulePass` says `paging: 'offset'`.
+- A budget reservation held by an online-evaluation worker that dies no longer stays charged for the
+  period. The worker records the reservation just before it calls its evaluators, and whoever takes
+  over settles it: charged at what the scores cost if they were written, given back if the
+  evaluators never ran, and by the ledger's `uncertain` policy if they ran and wrote nothing.
+- An evaluation run again after a crash is reserved and charged. Its evaluators run again to find
+  the scores still missing, and before, only a first run drew on the budget.
+- A deep agent resumed after a crash counts the helpers it created before. With a checkpointer, each
+  created helper is recorded, with its instructions, on `<thread>/helpers-created` before the call
+  that created it returns, so the resumed agent cannot pass `createHelpers.max` and delegates to them
+  again by name. Before, the count was kept by the process, and a resumed agent could create `max`
+  more and no longer found the ones it had made.
+
+### Proof
+
+- `tests/video.test.ts`: the mock, Veo on the Gemini API and Vertex AI, and Runway pass the video
+  conformance suite against stand-ins for their APIs. A request a model cannot make is refused
+  before any job. A generation runs through the lifecycle, streams three megabytes into a
+  filesystem store, and is priced per second. A generation whose process dies after submitting, or
+  after storing its video, is collected by another worker with one submission and one stored video.
+  If the recorded job is ignored, the job is submitted twice.
+- `tests/online-evaluation-paging.test.ts`: the memory, JSONL, and Postgres trace stores order 5,000
+  runs, 4,000 of them in one millisecond, identically, and page through them by cursor without a
+  repeat or a gap. Ten workers reading 1,000 a page schedule each of them exactly once on every
+  store, and on a store that ignores the cursor. The previous worker never reads past the burst.
+- `tests/budget-reservations.test.ts`: on the memory, tenant memory, Redis, and Postgres stores a
+  reservation moves with its total, is settled from another process by its operation's outcome, and
+  of two processes settling it, one changes the total. An online-evaluation worker killed while
+  holding a reservation, before its evaluator, during it, or after its scores, leaves the budget at
+  what the judge billed once another worker takes over. With recovery switched off, the same test
+  finds the dead worker's estimate still charged.
+- `tests/media-rag.test.ts`: with a manifest, an unchanged medium is neither read nor written, a
+  changed one keeps exactly its chunks and one asset, and a deleted one leaves none, with a crash at
+  each write of a first index, a change, and a delete. Without the pending record, the crash proof
+  fails.
+- `tests/deep-agent-helpers.test.ts`: an agent that created its last allowed helper, killed and
+  continued in another process, is refused a second and delegates to the first again by name. Without
+  the record read back, the second is created.
+
 ## [2.8.0] - 2026-10-14
 
 Retrieval at production scale, and media that is understood.
@@ -2452,7 +2559,8 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.8.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.9.0...HEAD
+[2.9.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.8.0...v2.9.0
 [2.8.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.7.0...v2.8.0
 [2.7.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.6.0...v2.7.0
 [2.6.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.5.0...v2.6.0

package/README.md

@@ -156,6 +156,7 @@ from the doc comments. The guides live in the repository, so these links go to G
 | [Retrieval](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/retrieval.md) | Hybrid search, reranking, MMR, parent documents, and Redis, Pinecone, Weaviate, and Chroma stores |
 | [Embeddings and retrieval](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/embeddings.md) | Embeddings as a routed operation family, and RAG helpers |
 | [Media understanding](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/media.md) | Reading documents and images page by page, describing images, and answering questions about them, through any vision model or OCR |
+| [Video generation](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/video.md) | Videos from a prompt or a first frame, on Google Veo and Runway, as durable jobs streamed into an asset store and priced per second |
 | [Images (experimental)](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/images.md) | Generation and masked edits over three backends, input safety, moderation, asset stores |
 | [Voice](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/voice.md) | Transcription, speech, voice turns, and voice sessions |
 | [Realtime voice](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/realtime.md) | Browser and server realtime sessions with barge-in, tools, and exports |
@@ -217,11 +218,13 @@ interface, or enum, has no doc comment, and `npm run docs:guides` fails when a g
 every export of the entry points it covers, or its generated reference is stale. Both hold at 100%;
 `--list` names what is missing, and `npm run docs:update` regenerates the references.
 
-Real provider conformance, and the vector store contract against real servers, are opt-in:
+Real provider conformance, the vector store contract against real servers, and the experimental
+families against their live services are opt-in:
 
 ```bash
 npm run test:conformance:real
 npm run test:vectors:live
+npm run test:families:live
 ```
 
 ## Roadmap
@@ -285,12 +288,20 @@ Shipped so far:
   Chroma stores out of experimental after their checklist ran against the real servers; a shared
   embedding cache; images, audio, and documents as embedding input; a half-price embedding batch tier
   that ingestion survives a crash through; streaming asset stores; registry data you supply and
-  refresh; and reading, describing, and searching media, with answers given the media themselves.
+  refresh; and reading, describing, and searching media, with answers given the media themselves;
+- 2.9 video, and everything shipped, proven: videos from a prompt or a first frame on Google Veo and
+  Runway, as durable jobs another worker collects after a crash, streamed into an asset store and
+  priced per second; online evaluation that reads every run however many share a millisecond, and
+  budget reservations a dead worker leaves settled by what its call did; a deep agent's helper limit
+  across processes; a media index that remembers what it holds; and the experimental families run
+  against their live services every week.
 
 Upgrading from 1.x: `npx nexus migrate src --write` moves your imports, and
 [MIGRATING.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/MIGRATING.md) covers the rest.
-Next, Pinecone, images, the Gemini embedding batch tier, and media understanding graduate once the
-live workflow passes against each. The
+Next, 2.10 makes every conversation that runs on a worker durable, A2A tasks, ACP turns, and
+realtime sessions, and adds Gemini Live as a second realtime provider; Pinecone, images, the Gemini
+embedding batch tier, media understanding, and video graduate once the live workflow passes against
+each. The
 [roadmap](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md) lists what comes after, in the order
 applications ask for it.
 

package/dist/batch/poll.d.ts

@@ -1,6 +1,6 @@
 import { type BatchJobState } from '../types/batch.js';
-/** How a batch is waited on. */
-export interface BatchPollOptions {
+/** How a batch, or any job with a status, is waited on. */
+export interface BatchPollOptions<State = BatchJobState> {
     /** The first wait between polls, in milliseconds. */
     intervalMs: number;
     /** The longest wait, once backoff has grown it. */
@@ -8,11 +8,15 @@ export interface BatchPollOptions {
     /** Stops waiting. The provider-side batch keeps running. */
     signal: AbortSignal;
     /** Called with each state that is not yet final. */
-    onState?: (state: BatchJobState) => void;
+    onState?: (state: State) => void;
+    /** Whether a state is final. Defaults to a batch's terminal statuses. */
+    settled?: (state: State) => boolean;
 }
 /**
  * Polls a batch until it reaches a status it never leaves. A batch measured in hours does not deserve
  * a fixed short interval: the wait grows by half each time up to the maximum, which keeps the poll
  * count sane without delaying a fast batch, which the first few short polls still catch.
  */
-export declare function pollUntilSettled(poll: () => Promise<BatchJobState>, options: BatchPollOptions): Promise<BatchJobState>;
+export declare function pollUntilSettled<State extends {
+    status: string;
+} = BatchJobState>(poll: () => Promise<State>, options: BatchPollOptions<State>): Promise<State>;

package/dist/batch/poll.js

@@ -1,8 +1,9 @@
 import { isTerminalBatchStatus } from '../types/batch.js';
 export async function pollUntilSettled(poll, options) {
+    const settled = options.settled ?? ((state) => isTerminalBatchStatus(state.status));
     let interval = options.intervalMs;
     let state = await poll();
-    while (!isTerminalBatchStatus(state.status)) {
+    while (!settled(state)) {
         if (options.signal.aborted)
             throw options.signal.reason ?? new Error('aborted');
         options.onState?.(state);

package/dist/deep-agent/helpers.d.ts

@@ -6,6 +6,10 @@
  * its checkpoints to the deep agent's checkpointer. A crash leaves the helper's finished steps behind
  * it: collecting the task in a new process carries the helper on from its last checkpoint, and a call
  * the agent repeats after the crash finds the same thread instead of starting the helper again.
+ *
+ * The helpers an agent creates are recorded on a thread of their own beside its own, before the call
+ * that creates one returns, so an agent resumed in another process counts them against its `max` and
+ * delegates to them again by name.
  */
 import { type CreateAgentOptions } from '../agent/create-agent.js';
 import type { PermissionPolicyLike } from '../agent/permissions.js';

package/dist/deep-agent/helpers.js

@@ -1,11 +1,55 @@
 import { agentInput, createAgent, } from '../agent/create-agent.js';
 const MARK = 'deepAgentHelper';
+const CREATED = 'deepAgentCreatedHelpers';
 export function helperTools(setup) {
     const declared = new Map(setup.declared.map((helper) => [helper.name, helper]));
     const created = new Map();
     const graphs = new Map();
     const live = new Map();
-    const spec = (name, instructions, thread) => {
+    const recorded = new Map();
+    const createdThread = (thread) => `${thread || 'thread'}/helpers-created`;
+    const loadCreated = (thread) => {
+        const checkpointer = setup.checkpointer;
+        if (!checkpointer || !setup.create)
+            return Promise.resolve();
+        let loading = recorded.get(thread);
+        if (!loading) {
+            loading = (async () => {
+                const stored = await checkpointer.get(createdThread(thread));
+                const records = (stored?.metadata?.[CREATED] ?? []);
+                const mine = created.get(thread) ?? new Map();
+                for (const record of records) {
+                    if (!mine.has(record.name))
+                        mine.set(record.name, { ...record, created: true });
+                }
+                created.set(thread, mine);
+            })();
+            loading.catch(() => recorded.delete(thread));
+            recorded.set(thread, loading);
+        }
+        return loading;
+    };
+    const recordCreated = async (thread, mine) => {
+        const checkpointer = setup.checkpointer;
+        if (!checkpointer)
+            return;
+        const threadId = createdThread(thread);
+        const records = [...mine.values()].map(({ created: _created, ...record }) => record);
+        await checkpointer.put({
+            version: 2,
+            id: `${threadId}:${records.length}`,
+            threadId,
+            step: records.length,
+            state: {},
+            next: [],
+            tasks: [],
+            status: 'completed',
+            interrupts: [],
+            createdAt: new Date().toISOString(),
+            metadata: { [CREATED]: records },
+        });
+    };
+    const spec = async (name, instructions, thread) => {
         const helper = declared.get(name);
         if (helper) {
             return {
@@ -17,6 +61,7 @@ export function helperTools(setup) {
                 created: false,
             };
         }
+        await loadCreated(thread);
         const mine = created.get(thread) ?? new Map();
         const existing = mine.get(name);
         if (existing)
@@ -42,6 +87,13 @@ export function helperTools(setup) {
         };
         mine.set(name, made);
         created.set(thread, mine);
+        try {
+            await recordCreated(thread, mine);
+        }
+        catch (error) {
+            mine.delete(name);
+            throw error;
+        }
         return made;
     };
     const graph = (helper, durable) => {
@@ -165,7 +217,7 @@ export function helperTools(setup) {
         },
         capabilities: [],
         execute: async (args, context = {}) => {
-            const helper = spec(String(args.agent ?? ''), typeof args.instructions === 'string' ? args.instructions : undefined, context.threadId ?? '');
+            const helper = await spec(String(args.agent ?? ''), typeof args.instructions === 'string' ? args.instructions : undefined, context.threadId ?? '');
             const task = String(args.task ?? '');
             const call = context.toolCallId ?? Math.random().toString(36).slice(2, 10);
             if (args.background !== true) {
@@ -211,8 +263,8 @@ export function helperTools(setup) {
                 const mark = stored.metadata?.[MARK];
                 const name = mark?.name ?? task.slice(0, task.lastIndexOf('@'));
                 const helper = mark?.created
-                    ? spec(name, mark.instructions, context.threadId ?? '')
-                    : spec(name, undefined, context.threadId ?? '');
+                    ? await spec(name, mark.instructions, context.threadId ?? '')
+                    : await spec(name, undefined, context.threadId ?? '');
                 await start(helper, undefined, threadId, context);
             }
             const entry = live.get(threadId);

package/dist/evaluate/online.d.ts

@@ -62,6 +62,11 @@ export interface OnlineSchedulePass {
     throttled: boolean;
     /** True when the pass read back to the oldest matching run, so the next one starts at the newest. */
     exhausted: boolean;
+    /**
+     * How the pass paged: by cursor, or by offset on a trace store that does not honour
+     * `RunQuery.after`, where runs that start in the same millisecond may be read twice but never skipped.
+     */
+    paging: 'cursor' | 'offset';
 }
 /** What a worker has done since it was created. */
 export interface OnlineEvaluationWorkerReport {

package/dist/evaluate/online.js

@@ -1,6 +1,7 @@
 import { BudgetExceededError } from '../core/lifecycle.js';
 import { operationStats } from '../operations/store.js';
 import { OperationRunner } from '../operations/runner.js';
+import { isAfterCursor, runCursor } from '../tracing/query.js';
 class HandBack extends Error {
 }
 export function onlineEvaluationWorker(options) {
@@ -24,6 +25,25 @@ export function onlineEvaluationWorker(options) {
         requestId,
         ...(options.tenantId !== undefined ? { tenantId: options.tenantId } : {}),
     });
+    const outcomeOf = (evaluating, run, operationId) => {
+        const scores = run ? scoresWritten(run, options.source, operationId) : [];
+        if (scores.length > 0) {
+            return { kind: 'finished', cost: (run && options.costOf?.(scores, run)) ?? options.estimate ?? 0 };
+        }
+        return evaluating ? { kind: 'unknown' } : { kind: 'never-started' };
+    };
+    const resolveReservation = async (reservation) => {
+        const record = await options.operations.read(reservation.operationId);
+        if (record?.kind !== kind)
+            return undefined;
+        const checkpoint = (record.heartbeatDetails ?? {});
+        const mine = checkpoint.reservation === reservation.id;
+        const live = record.status === 'running' && record.lease !== undefined && Date.parse(record.lease.expiresAt) > now().getTime();
+        if (live && mine)
+            return { kind: 'running' };
+        const run = await options.traces.get(String(record.metadata?.runId ?? ''));
+        return mine ? outcomeOf(checkpoint.evaluating === true, run, reservation.operationId) : { kind: 'never-started' };
+    };
     const inSample = (run) => rate >= 1 || hash(`${options.source}:${run.id}`) < rate;
     const operationId = (runId) => `${kind}:${runId}`;
     const busy = async () => {
@@ -38,6 +58,10 @@ export function onlineEvaluationWorker(options) {
         }
         const runId = String(record.metadata?.runId ?? '');
         const run = await options.traces.get(runId);
+        const previous = context.previousHeartbeat;
+        if (previous?.reservation && options.budget?.settle) {
+            await options.budget.settle(previous.reservation, outcomeOf(previous.evaluating === true, run, context.operationId));
+        }
         if (!run) {
             tally.missing += 1;
             return { status: 'missing', runId };
@@ -47,7 +71,7 @@ export function onlineEvaluationWorker(options) {
             entry.value?.evaluation === context.operationId)
             .map((entry) => entry.key));
         let reservation;
-        if (options.budget && written.size === 0) {
+        if (options.budget) {
             try {
                 reservation = await options.budget.reserve(descriptor(context.operationId), options.estimate ?? 0);
             }
@@ -60,9 +84,16 @@ export function onlineEvaluationWorker(options) {
             }
         }
         try {
+            if (reservation?.id) {
+                await context.heartbeat({ reservation: reservation.id, evaluating: true });
+            }
             const scores = await scoreRun(run, options.evaluators);
             const missing = scores.filter((score) => !written.has(score.key));
             if (missing.length === 0 && written.size > 0) {
+                if (reservation) {
+                    await options.budget?.reconcile(reservation, options.costOf?.(scores, run) ?? options.estimate ?? 0);
+                    reservation = undefined;
+                }
                 tally.alreadyScored += 1;
                 return { status: 'already-scored', runId };
             }
@@ -98,17 +129,35 @@ export function onlineEvaluationWorker(options) {
             return runner.workerId;
         },
         async schedule() {
-            const pass = { scanned: 0, sampled: 0, queued: 0, throttled: false, exhausted: false };
+            const pass = {
+                scanned: 0,
+                sampled: 0,
+                queued: 0,
+                throttled: false,
+                exhausted: false,
+                paging: byOffset ? 'offset' : 'cursor',
+            };
             if (budgetSpent)
                 return pass;
             const limit = options.scanLimit ?? 1_000;
-            const runs = await options.traces.query({
+            const read = () => options.traces.query({
                 kind: 'model',
                 status: 'ok',
                 ...options.query,
-                ...(cursor ? { until: cursor } : {}),
+                ...(byOffset ? { offset } : cursor ? { after: cursor } : {}),
                 limit,
             });
+            let runs = await read();
+            if (!byOffset && cursor) {
+                const from = cursor;
+                if (runs.some((run) => !isAfterCursor(run, from))) {
+                    byOffset = true;
+                    pass.paging = 'offset';
+                    cursor = undefined;
+                    offset = 0;
+                    runs = await read();
+                }
+            }
             pass.scanned = runs.length;
             const maxQueued = options.maxQueued ?? 1_000;
             let open = await busy();
@@ -146,9 +195,12 @@ export function onlineEvaluationWorker(options) {
                 if (runs.length < limit) {
                     pass.exhausted = true;
                     cursor = undefined;
+                    offset = 0;
                 }
                 else if (pass.queued === 0) {
-                    cursor = runs.at(-1)?.startedAt;
+                    const last = runs.at(-1);
+                    cursor = last ? runCursor(last) : undefined;
+                    offset += runs.length;
                 }
             }
             return pass;
@@ -156,6 +208,12 @@ export function onlineEvaluationWorker(options) {
         async work() {
             if (budgetSpent)
                 return 0;
+            if (options.budget?.recover) {
+                await options.budget.recover(resolveReservation, {
+                    operation: `online-evaluation.${options.source}`,
+                    limit: concurrency * 4,
+                });
+            }
             const handles = [
                 ...(await runner.recover(evaluate, concurrency)),
                 ...(await runner.claimQueued(evaluate, concurrency, { kindPrefix: kind })),
@@ -218,8 +276,19 @@ export function onlineEvaluationWorker(options) {
     let running;
     let stopping;
     let cursor;
+    let offset = 0;
+    let byOffset = false;
     return worker;
 }
+function scoresWritten(run, source, operationId) {
+    return (run.feedback ?? [])
+        .filter((entry) => entry.source === source && entry.value?.evaluation === operationId)
+        .map((entry) => ({
+        key: entry.key,
+        score: entry.score ?? 0,
+        ...(entry.comment ? { comment: entry.comment } : {}),
+    }));
+}
 async function scoreRun(run, evaluators) {
     const context = {
         example: { id: run.id, inputs: run.inputs, expected: undefined },

package/dist/lifecycle/budget.d.ts

@@ -1,14 +1,34 @@
-import type { BudgetLedger, BudgetPeriod, OperationDescriptor } from '../types/lifecycle.js';
+import type { BudgetLedger, BudgetPeriod, BudgetReservation, BudgetReservationOutcome, BudgetReservationRecord, OperationDescriptor } from '../types/lifecycle.js';
+import { type BudgetReservationChange } from './reservations.js';
+export type { BudgetReservationChange } from './reservations.js';
 /**
  * Where a ledger keeps its totals: a sum per key that resets at a given time. Two operations, so any
  * shared store can implement it. `MemoryTenantUsage` and `RedisTenantUsage` from
  * `nexus-ai-pro/server/tenancy` already do, so one Redis counts both tenant runs and model spend.
+ *
+ * A store that also records reservations implements the five optional methods together. A ledger
+ * over it records each reservation under a lease, so one a dead process held is found and settled,
+ * and moves each reservation and its total in one step, so a crash never applies one without the
+ * other. Every bundled store does.
  */
 export interface BudgetUsageStore {
     /** Adds to a total that resets at `resetAt`, epoch milliseconds, and resolves to the new total. */
     add(key: string, amount: number, resetAt: number): number | Promise<number>;
     /** A total, or 0. */
     total(key: string): number | Promise<number>;
+    /** Records a reservation and adds what it charges to its total, together: resolves to the new total. */
+    hold?(record: BudgetReservationRecord): number | Promise<number>;
+    /**
+     * Moves a reservation and its total, by the difference in what it charges, together, if it stands
+     * where `change.from` allows. Resolves whether it moved.
+     */
+    settle?(id: string, change: BudgetReservationChange): boolean | Promise<boolean>;
+    /** Extends a reserved reservation's lease. Resolves whether it was still reserved. */
+    renew?(id: string, leasedUntil: number): boolean | Promise<boolean>;
+    /** Reserved reservations whose lease lapsed by `now`, soonest lapsed first, optionally for one operation. */
+    lapsed?(now: number, limit: number, operation?: string): BudgetReservationRecord[] | Promise<BudgetReservationRecord[]>;
+    /** One reservation, while the store keeps it. */
+    reservation?(id: string): BudgetReservationRecord | undefined | Promise<BudgetReservationRecord | undefined>;
 }
 /** Options for `budgetLedger()`. */
 export interface BudgetLedgerOptions {
@@ -26,13 +46,34 @@ export interface BudgetLedgerOptions {
     key?: (operation: OperationDescriptor) => string | undefined;
     /** Where totals are kept. Defaults to process memory, which is enough for one replica. */
     store?: BudgetUsageStore;
+    /**
+     * How long a recorded reservation is held without `renew()` before `recover()` may settle it.
+     * Defaults to 10 minutes: longer than the operations it covers take.
+     */
+    leaseMs?: number;
+    /**
+     * What a reservation whose operation's outcome is unknown costs until its operation says: `charge`
+     * keeps it charged, as a provider may have billed the call; `release` gives it back. Defaults to
+     * `charge`.
+     */
+    uncertain?: 'charge' | 'release';
     /** Replaces the clock, for tests. */
     now?: () => Date;
 }
-/** A ledger that can also say what a budget has spent this period. */
+/** A ledger that can also say what a budget has spent this period, and settle what a dead process held. */
 export interface BudgetLedgerWithReport extends BudgetLedger {
+    /** Holds `estimate` for the operation, or rejects with `BudgetExceededError` when it does not fit. */
+    reserve(operation: OperationDescriptor, estimate: number): Promise<BudgetReservation | undefined>;
     /** What a budget has spent so far this period, in US dollars, reservations included. */
     spent(key: string): Promise<number>;
+    /** A recorded reservation, while the store keeps it. */
+    reservation(id: string): Promise<BudgetReservationRecord | undefined>;
+    /** Extends a recorded reservation's lease. Does nothing for one the store does not record. */
+    renew(reservation: BudgetReservation): Promise<void>;
+    /** Settles a recorded reservation by its id. Resolves `false` on a store that does not record them. */
+    settle(id: string, outcome: BudgetReservationOutcome): Promise<boolean>;
+    /** Settles reservations whose lease lapsed. Settles nothing on a store that does not record them. */
+    recover: NonNullable<BudgetLedger['recover']>;
 }
 /**
  * A spend budget shared by every family of a client.
@@ -41,15 +82,31 @@ export interface BudgetLedgerWithReport extends BudgetLedger {
  * the limit, the addition is undone and `BudgetExceededError` stops the call. Afterwards the estimate
  * is replaced by what the call actually cost, or given back when it failed. Adding first and checking
  * the new total is what keeps concurrent calls from overshooting together, on one process or many.
+ *
+ * Over a store that records reservations, each is held under a lease. One whose process died is
+ * found by `recover()` when its lease lapses and settled by what became of its operation: charged at
+ * its cost if it finished, given back if it never started, and charged or given back by `uncertain`
+ * when no one knows.
  */
 export declare function budgetLedger(options: BudgetLedgerOptions): BudgetLedgerWithReport;
-/** Budget totals in process memory. The default store, and enough for one replica. */
+/** Budget totals, and the reservations against them, in process memory. The default store, and enough for one replica. */
 export declare class MemoryBudgetUsage implements BudgetUsageStore {
     private readonly now;
     private readonly totals;
+    private readonly book;
     constructor(now?: () => number);
     /** Adds to a total that resets at `resetAt`. */
     add(key: string, amount: number, resetAt: number): number;
     /** A total, or 0 once it has reset. */
     total(key: string): number;
+    /** Records a reservation and adds it to its total. */
+    hold(record: BudgetReservationRecord): number;
+    /** Moves a reservation and its total together. */
+    settle(id: string, change: BudgetReservationChange): boolean;
+    /** Extends a reserved reservation's lease. */
+    renew(id: string, leasedUntil: number): boolean;
+    /** Reserved reservations whose lease lapsed. */
+    lapsed(now: number, limit: number, operation?: string): BudgetReservationRecord[];
+    /** One reservation. */
+    reservation(id: string): BudgetReservationRecord | undefined;
 }