nexus-ai-pro 2.0.0 → 2.2.0

package/API_STABILITY.md

@@ -41,6 +41,71 @@ Both are covered by this policy:
 Deep imports into `dist`, `dist-cjs`, or `src` are not supported. Import only from `nexus-ai-pro` or
 one of its explicit subpaths.
 
+## 2.2 stage (2.2.0)
+
+Everything 2.2 adds is stable from the start and additive: no default changes meaning.
+
+- versioned migrations: `SchemaMigration`, `AppliedMigration`, `MigrationStatus`, `MigrationResult`,
+  `SchemaMigrationError`, `migrationChecksum()`, `applyPostgresMigrations()`,
+  `postgresMigrationStatus()`, `postgresMigrations()`, `PostgresMigrateOptions`, every adapter's
+  `…Migrations()`, and the SQLite twins (`nexus-ai-pro/postgres`, `nexus-ai-pro/postgres/migrations`,
+  `nexus-ai-pro/sqlite`, `nexus-ai-pro/sqlite/migrations`). A migration, once released, never
+  changes; later releases add versions. `migrate()` now resolves to a `MigrationResult` instead of
+  nothing;
+- `nexus db status` and `nexus db migrate`, and `nexus db sql --record`;
+- `RedisOperationStoreOptions.index` and `reindex()`;
+- `CircuitBreakerConfig.shareObservations`, `CircuitStateStore.observe()`, `CircuitObservation`,
+  `CircuitWindow`, and `CircuitSnapshot.shared`;
+- `PostgresRollupStore` (`nexus-ai-pro/postgres/rollups`);
+- the tenant model (`nexus-ai-pro/tenancy`): every `tenant…()` view, `tenantScope()`,
+  `TenantAssetStore`, `TenantStores`, `assertTenantId()`, `TenantIdError`, and `TENANT_NAMESPACE`.
+  The prefix a view stores names under (`tenant/<id>/`) is part of the contract and does not change
+  within 2.x; `GraphRunOptions.tenantId`, `NodeContext.tenantId`, the `tenantId` options of `state()`,
+  `history()`, `fork()`, and `updateState()`, `OperationRecord.tenantId`,
+  `OperationSubmitOptions.tenantId`, `OperationStoreFilter.tenantId`, `RateLimitedRequest.tenantId`,
+  and `rateLimit.key: 'tenantId'`;
+- `CacheConfig.namespace`, `CacheNamespace`, `CacheOutcome` and `ResponseMeta.cache`,
+  `CompletionRequest.responseCache`, `MemoryCache.lookup()`, and the `scope` of `SemanticCache`'s
+  `get()` and `set()`;
+- `diagnose()` and its types (`nexus-ai-pro/doctor`), and `nexus doctor`. Checks may be added in a
+  minor release; a check's id and what makes it fail do not change within 2.x;
+- workflow steps' `timeout`, `onError`, `StepTimeout`, `StepFailure`, `StepRecovery`,
+  `StepContext.heartbeat()`, `WorkflowOptions.stepDefaults`, `WorkflowRunOptions.control` and
+  `tenantId`, `WorkflowContext.tenantId`, and `WorkflowStepTimeoutError.kind`;
+- `RunHandOffError`, `AssistantRunContext.saveProgress()` and `progress`.
+
+Three races are closed rather than added: a busy thread is claimed atomically, and the in-memory and
+Redis operation stores refuse a second record under a held idempotency key, as the SQL stores
+already did. The semantic cache no longer matches across tenants, and a failing response cache no
+longer fails a request.
+
+## 2.1 stage (2.1.0)
+
+Everything 2.1 adds is stable from the start and additive: no default changes meaning.
+
+- graph durability modes, `flush()`, `maxPendingWrites`, `onError` with `NodeFailure` and
+  `RecoveredFailure`, `nodeDefaults`, `NodeTimeout`, `context.heartbeat()`, `SendOptions`,
+  `RunControl` and `RunControlLike`, and `GraphDrainedError` (`nexus-ai-pro/graph`);
+- `lintGraph()` and its finding types (`nexus-ai-pro/graph/lint`). New rules may be added in a minor
+  release; a rule's code and severity do not change within 2.x;
+- `OperationContext.previousHeartbeat` and `OperationRecord.heartbeatDetails`;
+- `AssistantRunContext.control`;
+- the graph event stream: `events()`, `resumeEvents()`, `continueEvents()`, `GraphEventStream`, its
+  event and option types, `GraphStreamOverflowError`, `context.message()` and `context.tool()`, and
+  `subgraphEvents`. New projections may be added in a minor release;
+- `TracerOptions.incremental` and `exporters`, `TraceExporter`, `traceparent` in and out, `flush()`,
+  `closeAbandonedRuns()`, `w3cTraceId()` and `w3cSpanId()` (`nexus-ai-pro/tracing`);
+- `OtlpTraceExporter` and `runToOtlpSpan()` (`nexus-ai-pro/tracing/otlp`). Attributes may be added as
+  the GenAI conventions settle; the ones listed in the tracing guide stay;
+- rollups (`nexus-ai-pro/tracing/rollups`), timed replay (`timing`, `pace`), evaluator provenance
+  (`withProvenance()`, `Experiment.evaluators` and `framework`, `LLMJudge.asEvaluator()`),
+  `createAgent({ streamTokens })`, and `graphAssistant(graph, { events })`;
+- `rateLimit.algorithm` and `burst`, `RateLimitStore.gcra()`, `RateLimitDecision`, and `gcraDecide()`;
+  `health.observationTtlMs`, `ProviderHealthStatus`, a snapshot's `status` and `stale`, and
+  `checkProviders({ staleOnly })`.
+
+The studio's `rollups` source follows the studio's experimental label.
+
 ## 2.0 stage (2.0.0)
 
 2.0 removes what 1.x deprecated and promotes the surfaces that have settled.

package/CHANGELOG.md

@@ -4,6 +4,270 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [2.2.0] - 2026-10-05
+
+Production scale and safe upgrades. Every database schema is versioned, so an upgrade applies only
+what changed, and a worker of the previous release keeps running beside the new one. A claim costs
+the same however long the queue is, routing costs the same however many models are configured, a
+circuit breaker counts every worker's failures, and work that several replicas race for is taken by
+exactly one. One tenant model reaches every store, and a deployment can check itself. Nothing
+changes until it is opted into, except races and leaks that are now closed.
+
+### Added
+
+- **Versioned migrations.** Every Postgres and SQLite adapter's schema is a list of numbered
+  `SchemaMigration`s, recorded in `nexus_schema_migrations` with a checksum. `migrate()` applies what
+  is pending, under a lock that holds through a connection pool, and refuses a migration whose
+  statements changed after it ran (`SchemaMigrationError`). Version 1 is the schema 2.0 created, so an
+  existing database is recognized as it is. `applyPostgresMigrations()`, `postgresMigrationStatus()`,
+  `postgresMigrations()`, and their SQLite twins work on any list, from `nexus-ai-pro/postgres`,
+  `nexus-ai-pro/sqlite`, and the new `nexus-ai-pro/postgres/migrations` and
+  `nexus-ai-pro/sqlite/migrations`. `PostgresMigrateOptions.transaction` runs each migration in one
+  transaction on one connection, and `postgresMigration({ record: true })` records what the script
+  creates.
+- **`nexus db status` and `nexus db migrate`**, through a module that exports your own client:
+  applied, pending, changed, and newer migrations, `--check` for a deploy step, and `--dry-run`.
+- **Indexed dispatch.** Version 2 of the Postgres and SQLite operation schemas indexes queued work by
+  age, so a claim reads the oldest few rows instead of sorting the queue, and version 3 by tenant.
+  `RedisOperationStore` takes `index: true`: queued work in a sorted set by age and leased work in one
+  by lease expiry, kept in the same Lua call as the record, with `reindex()` to index what is already
+  stored. A claim reads the same handful of records with 100,000 operations queued as with 1,000.
+- **A circuit breaker that counts across workers.** `shareObservations: true` adds each worker's
+  calls to a rolling window in the shared store, so nine failures over three workers count as nine
+  and the circuit opens on all of them. Checks stay synchronous; a worker learns the shared count on
+  its next sync. `CircuitStateStore.observe()` is implemented by the memory, Redis, and Postgres
+  stores, and `CircuitSnapshot.shared` reports the window.
+- **`PostgresRollupStore`**, on the new `nexus-ai-pro/postgres/rollups`: hourly rollups beside the
+  traces, summed in one statement so two workers adding to one hour both count.
+- **A routing index.** The auto-router plans once per configuration — candidates, allow and deny
+  lists, capability requirements as bitmasks, and strategy scores — and a request pays only for open
+  circuits and health. Over 1,000 models a warm route takes about 15 µs at p50 and 23 µs at p99.
+- **One tenant model**, on the new `nexus-ai-pro/tenancy`: a tenant's view of every store —
+  `tenantStore()`, `tenantTraceStore()`, `tenantOperationStore()`, `tenantPromptStore()` (and with it
+  the context hub), `tenantDatasetStore()`, `tenantExperimentStore()`, `tenantVectorStore()`,
+  `tenantCache()`, `tenantAssetStore()`, `tenantRollupStore()` — each implementing the store's own
+  contract over any backend, and `tenantScope()` for all of them at once. Graph runs take a
+  `tenantId`: recorded on every checkpoint, read as `context.tenantId`, scoping `context.store`, and
+  refusing another tenant's resume, continue, fork, or edit. `graphAssistant()` passes the
+  principal's tenant. Operations carry `tenantId`, and `OperationStoreFilter.tenantId` narrows queue
+  reads. Model calls are traced with their `tenantId`, so rollups count model spend per tenant, and
+  `rateLimit.key: 'tenantId'` gives each tenant its own share of a client's rate limit.
+- **Cache namespaces and outcomes.** `cache.namespace` keeps responses apart by tenant, environment,
+  model revision, and context version. Every response reports `meta.cache` — `hit`, `miss`, `stale`,
+  `bypass`, or `error`, with its layer and reason — and the request's trace records it.
+  `CompletionRequest.responseCache` skips the cache (`bypass`) or refreshes it (`refresh`).
+- **`nexus doctor`**, and `diagnose()` on the new `nexus-ai-pro/doctor`: the Node.js version,
+  optional packages, provider credentials (by name only), database connectivity and pending or
+  changed migrations, Redis, lapsed leases and a stalled queue, the registry's age, stale provider
+  health, and process-local stores behind several replicas. It exits 1 on a failure, or with
+  `--strict` on a warning.
+- **Runtime budgets.** `npm run bench:runtime`, part of the release gate, times fourteen paths — graph
+  creation, a 1,000-node compile, `Send`, checkpoints, routing at three registry sizes, a
+  20-middleware agent, traces, the event stream, server event fan-out, claims, and cache lookups —
+  as ratios to a calibration loop, so a budget holds across machines, and checks that 100,000
+  checkpoints and 10,000 runs leave the heap where it was.
+- **Workflow steps have what graph nodes have.** `timeout: { runMs, idleMs }` with
+  `context.heartbeat()`, an `onError` that decides once retries run out — recorded as `recovered`,
+  so a resume repeats neither the step nor its recovery — `stepDefaults`, a `control` that drains
+  between steps and ends in `GraphDrainedError`, and `tenantId`. A workflow the agent server serves
+  is handed off on a drain like a graph.
+- **A function assistant hands its run off on a drain**: it checks `context.control?.draining`,
+  saves how far it got with `context.saveProgress()`, and throws `RunHandOffError`; the next worker
+  reads `context.progress` and carries on.
+
+### Changed
+
+- `migrate()` on every Postgres and SQLite adapter resolves to a `MigrationResult` instead of
+  nothing, records what it applied, and loads the migration runner only when called.
+- `nexus db sql` and `postgresMigration()` include the rollups table and the new indexes by
+  default; every statement stays idempotent.
+- The root import grew by 11 KB, to 336 KB, with the routing plan, cache outcomes, and linked
+  signals; `nexus-ai-pro/graph` by 5 KB, to 78 KB, with tenant runs; and
+  `nexus-ai-pro/graph/functional` by 6 KB, to 32 KB, with step timeouts, recovery, drain, and
+  tenants.
+
+### Fixed
+
+- **Two replicas could both start a run on one busy thread.** The busy check read and wrote
+  separately. A thread is now claimed through the operation store under a unique key with
+  compare-and-set, so exactly one run proceeds and the rest meet the busy policy. An `enqueue` start
+  still busy when `queueTimeoutMs` runs out is refused instead of running beside the other.
+- **Two submissions under one idempotency key could both run** on the in-memory and Redis operation
+  stores, which overwrote the key's owner. Both now refuse a second record under a key another record
+  holds, as Postgres and SQLite do, and Redis checks and writes in one Lua call.
+- A circuit breaker's calls recorded before its shared-state module finished loading were not
+  shared.
+- **Security: the semantic response cache answered one tenant with another's response** when their
+  questions were alike, because it compared every entry regardless of tenant. A semantic match is
+  now kept to the request's tenant and cache namespace.
+- A response cache that threw failed the request, and one that failed to store an answer failed it
+  after the provider had answered. Both are now reported in `meta.cache`, and the request goes on.
+- The auto-router rebuilt the model registry for every candidate on every request: over 1,000 models
+  a request took most of a second.
+- **Memory grew with every run that shared a signal.** `AbortSignal.any()`, which composed each task
+  attempt's signal, keeps a reference per call for as long as any source lives in Node 22, so a
+  server's or a worker's long-lived signal held about 5 KB per graph run, and every client operation
+  with a deadline held one too. Signals are now linked and released when their work ends; 10,000
+  runs leave the heap within 0.3 MB of where it started, down from 52 MB.
+
+### Proof
+
+- A database created by the published 2.0.0 package upgrades with 2.2's migrations, on Postgres and
+  SQLite, and a published 2.1.0 worker and a 2.2 worker drain one queue together, nothing run twice.
+- Ten workers race one thread, one idempotency key, one queue, one tenant's slots and budget, and one
+  circuit, on every shared store, and every invariant holds.
+- A Redis claim reads the same number of records at 1,000 and 100,000 queued; Postgres and SQLite
+  plan a claim through the age index with no sort.
+- The Redis stores' Lua runs for real in the tests, in a Lua VM over an in-memory Redis.
+- The routing index decides exactly as the published 2.1.0 router on 300 random configurations, and
+  a warm route over 1,000 models stays under 100 µs at p50 and 1 ms at p99.
+- A tenant-isolation suite runs against every storage adapter — memory, Redis, Postgres, SQLite, and
+  files — and fails on any cross-tenant get, list, search, delete, fork, resume, trace query, vector
+  search, or cache hit.
+- `nexus doctor` reports each misconfiguration a fixture deployment is seeded with — an old Node.js,
+  no credentials, an old registry, pending migrations, an unreachable Redis, a lapsed lease and a
+  stalled queue, stale health, and memory stores behind three replicas — and passes it once fixed.
+- A workflow and a function assistant are each drained on one worker and finished on the next, with
+  every step and every item run exactly once.
+
+### Not in this release, and why
+
+- **The migration lock is a row with a lease, not a session advisory lock.** A session lock does not
+  hold through a connection pool or a serverless driver, where each statement may reach another
+  connection; a lease row does, everywhere. Without a `transaction` function, statements run one at a
+  time, which is safe because every bundled statement is idempotent.
+- **The Redis dispatch index is opt-in**, since a 2.1 worker writes records without index entries;
+  `stats()` on Redis still reads every record.
+- **Consecutive failures across workers are counted as failures since any worker's last success.**
+  Calls are reported in batches, so two workers' failures and successes in the same second may be
+  ordered differently than they happened.
+- **Tenant views scope by prefix and stamp, not by a column**, so they work over every backend
+  unchanged; data written before a view was used is not the tenant's until it is copied under its
+  prefix.
+- **Runtime budgets compare ratios to a calibration loop**, which catches a cliff, not a 10% drift;
+  the tolerance is three times the recorded ratio.
+
+## [2.1.0] - 2026-10-04
+
+Fault-tolerant graphs and crash-safe observability. A graph now chooses when its checkpoints are
+written, recovers from a node whose retries ran out, tells a hung task from a slow one, and stops
+cleanly when its worker has to go. Traces survive the process that wrote them and reach any
+OpenTelemetry backend, every consumer of a run reads one event stream, and dashboards read hourly
+totals instead of every trace. Nothing changes until it is opted into: the defaults keep their 2.0
+meaning.
+
+Two pieces planned for 2.3 come early: a rate limiter with no window edge to burst across, and
+provider health that expires.
+
+### Added
+
+- **Durability modes.** `compile({ durability })` takes `sync` (the default), `async`, or `exit`.
+  `async` writes checkpoints in the background, in order, while the next superstep runs, and waits
+  for them before the run returns, pauses, fails, or drains. `maxPendingWrites` (8) bounds the writes
+  in flight. `exit` writes only where the run stops. `flush()` waits for every run's writes.
+- **Recovery after retries.** A node's `onError` receives a `NodeFailure` once its retries run out
+  and returns an update or a `Command`, such as a route to a refund node. The decision is
+  checkpointed, as `recovered` on the next checkpoint, before anything it routes to runs, so a crash
+  after it resumes into the same path. `compile({ nodeDefaults })` sets retries, timeouts, and
+  `onError` for every node.
+- **Run and idle timeouts.** `timeout: { runMs, idleMs }` caps an attempt and the time between signs
+  of progress. `context.heartbeat()`, `emit()`, and `report()` refresh the idle timer.
+  `GraphNodeTimeoutError.kind` says which limit fired. A `Send` can override the timeout of the task
+  it creates, and the override is checkpointed with the task.
+- **Cooperative drain.** `RunControl`, passed as `control` to a run, stops it after its current
+  superstep with its checkpoint written, ending in `GraphDrainedError`; `continue()` resumes it
+  anywhere. The agent server drains its graph runs this way when a replica drains with a queue, so
+  they are handed off at once instead of after the drain timeout.
+- **`lintGraph()`** on the new `nexus-ai-pro/graph/lint`, and `nexus graph lint`: static cycles,
+  retried nodes that are not `idempotent` or have no timeout, deferred durability with side effects,
+  an in-process checkpointer under a deployment, unbounded concurrency, and unchecked routes. Nodes
+  can declare `idempotent: true`, and `describe()` reports durability, the checkpointer, and each
+  node's retries, timeouts, and recovery.
+- **Progress that survives a retry.** An operation's `context.heartbeat(details)` stores `details` as
+  `heartbeatDetails`, and the next attempt reads them as `context.previousHeartbeat`.
+- A `task_failed` graph event, and a semantics table in the graph guide that says what each
+  execution primitive repeats after a crash.
+- **Crash-safe traces.** `new Tracer({ incremental: true })` writes each run when it starts and when it
+  ends, so a crashed run stays in the store with every run beneath it that finished, and a live run
+  can be watched. Sampling is decided at the root's start, and an unsampled trace is still kept whole
+  when tail sampling wants it. `closeAbandonedRuns()` closes what a dead process left running, as a
+  `RunAbandoned` error. `tracer.flush()` waits for writes and exports.
+- **Traces across processes.** `handle.traceparent()` and `StartRunOptions.traceparent` carry a trace
+  through a W3C `traceparent` header: an operation's `traceContext`, a remote call, any OpenTelemetry
+  service. `w3cTraceId()` and `w3cSpanId()` give the ids every process derives alike.
+- **OpenTelemetry export**, on the new `nexus-ai-pro/tracing/otlp`. `TracerOptions.exporters` takes any
+  `TraceExporter`, and `OtlpTraceExporter` sends runs to a collector as OTLP/HTTP JSON with no SDK:
+  model calls as `chat <model>` client spans with the GenAI semantic conventions, tools, agents, and
+  graph nodes with theirs, and Nexus-only facts under `nexus.*`. `runToOtlpSpan()` is the conversion.
+- **One event stream.** `graph.events()`, `resumeEvents()`, and `continueEvents()` run a graph as
+  typed events with projections — `values`, `updates`, `messages`, `tools`, `tasks`, `checkpoints`,
+  `custom` — read together or one at a time through `messages()`, `tools()`, `values()`, and
+  `updates()`, with `result` for the outcome. Nodes stream model output with `context.message()`
+  and report tool calls with `context.tool()`. Subgraph events come through with a `namespace` when
+  asked, and `GraphRunOptions.subgraphEvents` does the same for `onEvent`.
+- **Bounded readers.** Every reader of a stream has its own buffer (`maxBuffered`, 1,000), and the run
+  waits for the slowest at each superstep. Within a step, `overflow` coalesces model chunks and state
+  snapshots (the default), drops the oldest or newest, or fails with `GraphStreamOverflowError`.
+  `streamStats()` reports the peak and what was merged or dropped.
+- **Agents stream.** `createAgent()` reports its model's output and every tool call on the stream,
+  token by token with `streamTokens` and a client with `stream()`. `traceGraph()` records the tool
+  calls as `tool` runs inside their node. `graphAssistant(graph, { events })` records chosen
+  projections on each server run.
+- **Rollups**, on the new `nexus-ai-pro/tracing/rollups`: `rollupTraceStore()` adds every finished run
+  to hourly rows per kind, model, provider, and tenant, in `MemoryRollupStore` or any `RollupStore`,
+  with latency buckets for `rollupPercentile()` and `sumRollups()` to total them. The studio's costs
+  view reads them when given `rollups`.
+- **Timed replay.** `recordingFetch({ timing: true })` keeps each chunk of a response and when it came,
+  and `replayFetch({ pace })` replays it at once, at the original pace, scaled, or released chunk by
+  chunk by a function.
+- **GCRA rate limiting.** `rateLimit.algorithm: 'gcra'` spreads calls at `maxRequests` per `windowMs`
+  with at most `burst` at once, so a burst across a window edge can no longer pass twice the limit.
+  Both stores decide it through `gcra()`; `RedisRateLimitStore` runs it as one Lua script on Redis's
+  clock. `gcraDecide()` is the arithmetic, for a store of your own.
+- **Health that expires.** `health.observationTtlMs` makes a provider with no recent calls `unknown`,
+  routed to as one never seen instead of kept down by an old failure or up by an old success.
+  Snapshots carry a `status` (`healthy`, `degraded`, `unhealthy`, `unknown`) and `stale`, and
+  `ai.checkProviders({ staleOnly: true })` refreshes the unknown ones.
+- **Evaluator provenance.** An experiment records each evaluator's position, name, the score keys it
+  produced, and what `withProvenance()` declared — version, judge model, prompt version, temperature,
+  and rubric — and the framework version that ran it. `LLMJudge.asEvaluator()` brings the judge to
+  `evaluate()`, declaring its own provenance.
+- A weekly `live-providers` workflow runs real conformance against every provider with a key in the
+  repository's secrets, DeepSeek included, and opens an issue when one breaks.
+
+### Fixed
+
+- **A manual operation heartbeat no longer strands the result.** `context.heartbeat()` wrote the
+  record behind the runner's back, so the runner's final write failed its sequence check and a
+  successful result was not saved. Heartbeats now go through the runner, one at a time.
+- `continue()` on a finished thread left it marked `running` with an empty result. It now leaves it
+  completed and returns its state.
+- A run started under a parent from another tracer, or another process, was never written: its trace
+  waited for a root that was not there. It is now written as its trace's root in this process.
+
+### Proof
+
+- A crash-injection suite kills a run before, during, and after every checkpoint write and inside
+  nodes, before and after their side effects, in `sync` and `async` mode, on the memory, operation
+  store, SQLite, Postgres, and Redis checkpointers. A fresh process always reaches the state of an
+  uninterrupted run.
+- A saga test: a charge exhausts its retries, `onError` routes to a refund, the process dies during
+  the refund, and the resumed run refunds without charging again.
+- A property test generates 40 graphs with branches, loops, `Send`, retries, and interrupts, and
+  replays each from every checkpoint of its history to the same state.
+- The graph benchmark runs 50 supersteps against an 8 ms store: `async` finishes in about half the
+  time of `sync`, in the same state, and CI fails if it stops beating it by a quarter.
+- A long agent's process dies at its fourteenth step: a second process reads every finished step from
+  the same JSONL store, and `closeAbandonedRuns()` closes the rest.
+- A server and a worker each export to a local OTLP collector, and the collector receives one trace,
+  the worker's run under the server's, with the GenAI attributes on the model call.
+- A reader a hundred times slower than a node that streams 100,000 chunks never holds more than its
+  buffer, under every overflow policy, and coalescing loses no text.
+- The studio's costs view answers over a month of a million runs, held as rollups, in under 200 ms,
+  reading the traces once.
+- In the two milliseconds around a window's edge, fixed windows pass 199 calls of a 100-call limit and
+  GCRA passes 100. A provider down on Monday and idle since is `unknown`, and routed to, on Wednesday.
+
 ## [2.0.0] - 2026-10-02
 
 Consolidation. One lifecycle runs every call of every family, the root import holds the core client
@@ -1373,7 +1637,9 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.0.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.2.0...HEAD
+[2.2.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.1.0...v2.2.0
+[2.1.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.0.0...v2.1.0
 [2.0.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.25.0...v2.0.0
 [1.25.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.24.0...v1.25.0
 [1.24.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.23.0...v1.24.0

package/MIGRATING.md

@@ -69,7 +69,7 @@ It cannot split a namespace import (`import * as nexus from 'nexus-ai-pro'`) or
 so it lists them. Import each name you use from its subpath instead. Every subpath, and what it costs,
 is in the [packaging guide](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/packaging.md).
 
-The root import now costs about 323 KB, down from 572 KB.
+The root import now costs about 325 KB, down from 572 KB.
 
 ## Validators and Node types are optional
 

package/README.md

@@ -8,12 +8,12 @@ team can share.
 Import only what you use. The root import is the client and nothing else, and every other capability
 has its own entry point with a size budget CI enforces; [the packaging guide](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/packaging.md)
 publishes what each one costs. The package has no required dependencies: a graph-only application
-loads 51 KB, and installs nothing but this package.
+loads 78 KB, and installs nothing but this package.
 
 - NPM: https://www.npmjs.com/package/nexus-ai-pro
 - GitHub: https://github.com/mkhitar-abrahamyan/nexus-ai
 - Contributing, testing, and release procedure: [CONTRIBUTING.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/CONTRIBUTING.md)
-- Delivered work and what is planned next: [ROADMAP.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md)
+- What is planned next: [ROADMAP.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md)
 - For AI assistants: [llms.txt](https://raw.githubusercontent.com/mkhitar-abrahamyan/nexus-ai/main/llms.txt) indexes every guide, and [llms-full.txt](https://raw.githubusercontent.com/mkhitar-abrahamyan/nexus-ai/main/llms-full.txt) holds them all in one file
 
 ## Install
@@ -158,6 +158,7 @@ from the doc comments. The guides live in the repository, so these links go to G
 | [Studio](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/studio.md) | A shared UI for traces, threads, approvals, experiments, prompts, deployments, costs, and health, as a separate package |
 | [Agent server](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/server.md) | Self-hosted HTTP server for assistants: threads, durable runs, resumable streams, cron, a worker queue, and metrics |
 | [Deployments](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/deployments.md) | Revisions and canaries with automatic rollback, autoscaling on Kubernetes and Helm, and per-tenant limits |
+| [Tenancy](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/tenancy.md) | One tenant model: a tenant's view of every store, tenant graph runs, and the isolation suite |
 | [Workflows](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/workflows.md) | Ready-made chains and domain workflows |
 | [Command line](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/cli.md) | The `nexus` command |
 | [Packaging and install weight](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/packaging.md) | Every entry point and what it costs |
@@ -200,7 +201,8 @@ npm run check:release
 
 `check` runs formatting and lint gates, source/test/example type checks, the build, unit and mock
 conformance tests, coverage thresholds, package import checks, and an external type-consumer test.
-`check:release` additionally verifies the dry-run tarball and a clean packed-package install.
+`check:release` additionally holds the scheduling and runtime budgets — speed against a calibration
+loop, and memory by retention — and verifies the dry-run tarball and a clean packed-package install.
 
 `npm run docs:check` fails when any public export, or any public member of an exported class,
 interface, or enum, has no doc comment, and `npm run docs:guides` fails when a guide does not name
@@ -216,8 +218,8 @@ npm run test:vectors:live
 
 ## Roadmap
 
-See [ROADMAP.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md) for what has
-shipped and what is planned. It is a design proposal, not a compatibility promise; the guarantees
+See [ROADMAP.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md) for what is
+planned, and [CHANGELOG.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/CHANGELOG.md) for what has shipped. It is a design proposal, not a compatibility promise; the guarantees
 live in [API_STABILITY.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/API_STABILITY.md).
 
 Shipped so far:
@@ -243,12 +245,19 @@ Shipped so far:
   a codemod that moves your imports;
 - 2.0 consolidation: one lifecycle for every call of every family, a shared spend budget, a root
   import that holds only the client, no required dependencies, and a model registry checked against
-  every provider.
+  every provider;
+- 2.1 fault tolerance and crash-safe observability: checkpoints written on your terms, recovery after
+  retries, idle timeouts and a clean drain, traces that survive a crash and reach any OpenTelemetry
+  backend, one event stream, and dashboard rollups;
+- 2.2 production scale and safe upgrades: versioned database migrations, queue claims and routing
+  that cost the same at any size, a circuit breaker that counts across workers, one tenant model
+  across every store, and `nexus doctor` to check a deployment before it serves traffic.
 
 Upgrading from 1.x: `npx nexus migrate src --write` moves your imports, and
 [MIGRATING.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/MIGRATING.md) covers the rest.
-Next, the image family leaves experimental once recorded live conformance passes on all three
-backends.
+Next, 2.3 gives agents a real security boundary — capabilities, permissions, and a sandbox contract —
+with a complete middleware catalog, and retrieval that works at production scale. The [roadmap](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md) lists the
+releases after it.
 
 ## Before Production
 

package/dist/agent/create-agent.d.ts

@@ -2,7 +2,7 @@ import type { OperationLifecycleLike } from '../types/lifecycle.js';
 import { type CompiledGraph } from '../graph/graph.js';
 import type { GraphCheckpointer, NodeContext } from '../types/graph.js';
 import type { CompletionRequest, Message, ToolDefinition } from '../types/messages.js';
-import type { NexusResponse } from '../types/response.js';
+import type { NexusResponse, StreamChunk } from '../types/response.js';
 import type { Store } from '../types/store.js';
 /**
  * An agent as a graph.
@@ -17,6 +17,8 @@ import type { Store } from '../types/store.js';
 export interface AgentModelClient {
     /** Runs one completion. */
     complete(request: CompletionRequest): Promise<NexusResponse>;
+    /** Streams one completion. Used when the agent is created with `streamTokens`. */
+    stream?(request: CompletionRequest): AsyncIterable<StreamChunk>;
 }
 /** A tool call the agent is about to make. */
 export interface AgentToolCall {
@@ -108,6 +110,12 @@ export interface CreateAgentOptions {
      * `ai.lifecycle`. Its model calls are operations of their own.
      */
     lifecycle?: OperationLifecycleLike;
+    /**
+     * Streams the model's output token by token onto the run's `messages` events, through the client's
+     * `stream()`. Without it, or with a client that cannot stream, each answer arrives as one message
+     * event once the model returns. Middleware sees the assembled response either way.
+     */
+    streamTokens?: boolean;
 }
 declare const agentChannels: () => {
     messages: import("../types/graph.js").Channel<Message[]>;

package/dist/agent/create-agent.js

@@ -37,7 +37,12 @@ export function createAgent(options) {
         for (const item of middleware) {
             request = (await item.beforeModel?.({ request, state, store: options.store })) ?? request;
         }
-        let response = await options.client.complete(request);
+        let response = options.streamTokens && options.client.stream
+            ? await streamed(options.client.stream(request), (chunk) => context.message(chunk))
+            : await options.client.complete(request);
+        if (!(options.streamTokens && options.client.stream) && response.content) {
+            context.message({ content: response.content });
+        }
         for (const item of middleware) {
             response = (await item.afterModel?.({ response, state, store: options.store })) ?? response;
         }
@@ -88,9 +93,13 @@ export function createAgent(options) {
                 args = verdict.args;
         }
         const run = async () => executor.execute(description.name, args);
+        context.tool({ phase: 'start', id: call.id, name: description.name, args });
         const result = await middleware.reduceRight((next, item) => item.wrapToolCall
             ? () => Promise.resolve(item.wrapToolCall?.({ ...description, args }, next))
             : next, run)();
+        context.tool(result.ok
+            ? { phase: 'result', id: call.id, name: description.name, result: result.result }
+            : { phase: 'error', id: call.id, name: description.name, error: result.error ?? 'The tool failed' });
         return { messages: [toolMessage(call, result)] };
     }, { ends: ['model'] })
         .setEntry('model')
@@ -156,3 +165,33 @@ function parseArguments(raw) {
         return { ok: false, error: `Tool arguments are not valid JSON (${reason}); the tool was not run` };
     }
 }
+async function streamed(chunks, onChunk) {
+    let content = '';
+    const toolCalls = [];
+    let meta = {};
+    for await (const chunk of chunks) {
+        if (chunk.type === 'text' && chunk.content) {
+            content += chunk.content;
+            onChunk({ kind: 'text', content: chunk.content });
+        }
+        else if (chunk.type === 'reasoning' && chunk.content) {
+            onChunk({ kind: 'reasoning', content: chunk.content });
+        }
+        else if (chunk.type === 'tool_call' && chunk.toolCall) {
+            toolCalls.push(chunk.toolCall);
+        }
+        else if (chunk.type === 'done') {
+            meta = chunk.meta ?? meta;
+        }
+        else if (chunk.type === 'error') {
+            throw new Error(chunk.error ?? 'The model stream failed');
+        }
+    }
+    return {
+        content,
+        role: 'assistant',
+        ...(toolCalls.length ? { toolCalls } : {}),
+        finishReason: toolCalls.length ? 'tool_calls' : 'stop',
+        meta: meta,
+    };
+}

package/dist/cache/memory-cache.d.ts

@@ -15,6 +15,14 @@ export declare class MemoryCache<T> {
     constructor(maxEntries?: number);
     /** Returns a value, or `undefined` for a miss or an expired entry. A hit counts as recent use. */
     get(key: string): T | undefined;
+    /**
+     * Like `get()`, also saying whether an entry was there but had expired, which a caller can report
+     * as stale rather than missing. The expired entry is removed either way.
+     */
+    lookup(key: string): {
+        value?: T;
+        stale: boolean;
+    };
     /** Stores a value. Defaults to 5 minutes. */
     set(key: string, value: T, ttlSeconds?: number): void;
     /** Removes every entry. */

package/dist/cache/memory-cache.js

@@ -16,6 +16,15 @@ export class MemoryCache {
         this.entries.set(key, entry);
         return entry.value;
     }
+    lookup(key) {
+        const entry = this.entries.get(key);
+        if (entry && entry.expiresAt < Date.now()) {
+            this.entries.delete(key);
+            return { stale: true };
+        }
+        const value = this.get(key);
+        return value === undefined ? { stale: false } : { value, stale: false };
+    }
     set(key, value, ttlSeconds = 300) {
         if (this.entries.size >= this.maxEntries) {
             const oldest = this.entries.keys().next().value;

package/dist/cache/semantic-cache.d.ts

@@ -25,10 +25,13 @@ export declare class SemanticCache {
     constructor(options?: SemanticCacheOptions);
     /**
      * The cached response for the most similar earlier request above the threshold, or `undefined`.
+     * Only entries stored under the same `scope` are compared: the client passes the request's tenant
+     * and cache namespace, so one tenant is never answered from another's entry however alike the
+     * questions are.
      */
-    get(request: CompletionRequest): Promise<NexusResponse | undefined>;
-    /** Caches a response under its request's embedding. */
-    set(key: string, request: CompletionRequest, response: NexusResponse): Promise<void>;
+    get(request: CompletionRequest, scope?: string): Promise<NexusResponse | undefined>;
+    /** Caches a response under its request's embedding, in a `scope` lookups must match. */
+    set(key: string, request: CompletionRequest, response: NexusResponse, scope?: string): Promise<void>;
     /** Removes every entry. */
     clear(): void;
 }

package/dist/cache/semantic-cache.js

@@ -7,7 +7,7 @@ export class SemanticCache {
         this.options = options;
         this.embed = options.embed || createHashEmbeddings;
     }
-    async get(request) {
+    async get(request, scope = '') {
         if (!this.options.enabled)
             return undefined;
         const text = requestToSearchText(request);
@@ -17,6 +17,8 @@ export class SemanticCache {
         let best;
         this.entries = this.entries.filter((entry) => entry.expiresAt > now);
         for (const entry of this.entries) {
+            if (entry.scope !== scope)
+                continue;
             const score = cosineSimilarity(embedding, entry.embedding);
             if (score >= threshold && (!best || score > best.score)) {
                 best = { entry, score };
@@ -37,7 +39,7 @@ export class SemanticCache {
             },
         };
     }
-    async set(key, request, response) {
+    async set(key, request, response, scope = '') {
         if (!this.options.enabled)
             return;
         const text = requestToSearchText(request);
@@ -47,6 +49,7 @@ export class SemanticCache {
             this.entries.shift();
         this.entries.push({
             key,
+            scope,
             text,
             embedding,
             response,

package/dist/cli/args.js

@@ -5,12 +5,15 @@ export const BOOLEAN_FLAGS = new Set([
     'allow-dataset-mismatch',
     'check',
     'densify',
+    'dry-run',
     'fail',
     'fail-on-regression',
     'help',
     'json',
     'print',
+    'record',
     'reveal-values',
+    'strict',
     'write',
 ]);
 export function parseArgs(args) {