nexus-ai-pro 1.14.0 → 1.16.0

package/API_STABILITY.md

@@ -210,10 +210,14 @@ What is guaranteed is the state machine — closed, open after a threshold is cr
 the cooldown, closed again only after the configured probe successes — and that routing still selects
 a provider when every circuit is open rather than failing the request untried.
 
-Breaker state is per process by design. Two workers can disagree about a provider, and neither the
-breaker nor its snapshot promises cluster-wide consensus. Rate-limit counters are the opposite: they
-are only shared when a store is configured, and the default in-memory counter gives each worker its
-own budget.
+Breaker state is per process unless `circuitBreaker.store` is set (since 1.16.0). With a store,
+workers converge on each other's decisions within about `syncIntervalMs`, and one worker at a time
+holds the probe; this is eventual agreement, not consensus, and two workers may briefly disagree.
+Failure counting stays per worker either way. The `CircuitStateStore` contract and the memory, Redis,
+and Postgres adapters follow the 1.x rules. Since 1.16.0 a caller's cancellation does not count as a
+failure unless `isFailure` says so, and probe limits apply to every attempt the failover executor
+makes. Rate-limit counters are only shared when a store is configured, and the default in-memory
+counter gives each worker its own budget.
 
 ## Batch API stage
 
@@ -328,6 +332,19 @@ Also since 1.12.0, all additive:
 - A subgraph node's thread id is `<parent thread>:<task id>`, which equals the previous
   `<parent thread>:<node>` for any node not reached through `Send`.
 
+Also since 1.16.0, all additive:
+
+- `createGraph({ channels, input, output })`. `StateGraph`, `CompiledGraph`, and `GraphResult` gained
+  defaulted type parameters for the input and output channels, so existing annotations such as
+  `CompiledGraph<S>` keep their meaning. A write to an undeclared input rejects with
+  `GraphValidationError`. Checkpoints and stream events still carry every channel.
+- `NodeOptions.cache` and `CompileOptions.cache`. Guaranteed: a failure, an interrupt, a node that
+  consumed an interrupt's answer, and a `Command.PARENT` result are never served from the cache, and a
+  cache that throws is treated as a miss. Not guaranteed: the default key's exact text, so entries
+  written by one minor release may miss after an upgrade.
+- `task_end` events may carry `cached: true`; `GraphDescription` may carry `input`, `output`, and a
+  node's `cache` flag.
+
 ## Agent, store, MCP, and tracing stages (1.14.0)
 
 The `nexus-ai-pro/agent`, `nexus-ai-pro/store`, `nexus-ai-pro/store/redis`, and `nexus-ai-pro/mcp`
@@ -355,6 +372,46 @@ subpaths are public and follow the 1.x rules. None is exported from the root imp
 - `AgentLoop`, `tool()`, and `ToolExecutor` keep their behaviour and are re-exported from
   `nexus-ai-pro/agent` as well as the root.
 
+## Evaluation stage (1.15.0)
+
+`nexus-ai-pro/evaluate` is public and follows the 1.x rules, and is not exported from the root.
+`Dataset`, `DatasetExample`, `Experiment`, `ExampleResult`, `MetricSummary`, and the comparison shapes
+are normalized; new optional fields and new bundled evaluators may be added in a minor release.
+
+- A dataset version is derived from example content. The hash itself is an implementation detail, but
+  the guarantee is not: identical examples produce identical versions, and any change produces a
+  different one.
+- `compareExperiments()` is statistical. Its intervals come from a seeded paired bootstrap, so the
+  same inputs give the same verdict; the exact interval bounds may change if the method improves, and
+  a verdict is a judgement about evidence rather than a compatibility guarantee.
+- `evaluate()` reports a target's failure as a failed example and an evaluator's failure as a failed
+  measurement; neither throws. Since 1.16.0 an aborted evaluation rejects with the abort reason and
+  stores nothing.
+- `EvalRunner` and `MediaEvalRunner` keep their APIs. Since 1.16.0 both run on `evaluate()` and carry
+  the resulting experiment in an optional `experiment` field; their reports are otherwise unchanged.
+- Since 1.16.0, additive: `underCost()`, `FileExperimentStore`, `readExperiment()`, the `cost`
+  option, per-example `repetitions`, and `signal` on the target's context.
+
+## Postgres, command-line, and record-and-replay stages (1.16.0)
+
+The `nexus-ai-pro/postgres` subpaths, `nexus-ai-pro/ops/circuit-store`, and
+`nexus-ai-pro/testing/record` are public and follow the 1.x rules. None is exported from the root.
+
+- Each Postgres adapter implements the same contract as the corresponding in-memory store, and is
+  tested against it. Ordering among rows that tie on a timestamp is not a guarantee.
+- `PostgresLikeClient` is the whole driver contract: `query(text, values)` resolving to `{ rows }`.
+- The schema is guaranteed additive for the 1.x line: a minor release may add a column or an index,
+  never drop or rename one, and `migrate()` stays idempotent. A change that needs a data migration
+  waits for 2.0.
+- Unique idempotency keys in `PostgresOperationStore` are a guarantee, and `OperationDuplicateError`
+  is the signal the runner relies on.
+- The `nexus` commands and flags listed by `nexus help` are public. Human-readable output is not a
+  guarantee and may improve in any release; `--json` output follows the 1.x rules; exit codes are
+  guaranteed — 0 for success, 1 for a failed check, 2 for a usage error.
+- The fixture file format and the default match key of `testing/record` are stable for the 1.x line,
+  so recordings made with one release keep replaying with the next. A change that would invalidate
+  recordings waits for 2.0.
+
 ## Deprecation process
 
 Deprecated APIs are marked with `@deprecated` in declarations and described in the changelog. Removals

package/CHANGELOG.md

@@ -4,6 +4,135 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [1.16.0] - 2026-09-22
+
+Shared state and the command line. Everything that has to outlive a process or be shared between
+workers now has a Postgres adapter, circuit decisions are shared across a fleet, the evaluation
+platform has a CI gate, and suites that needed credentials can run from recorded provider traffic.
+
+### Added
+
+- **A Postgres adapter family** (`nexus-ai-pro/postgres`, with one subpath per adapter):
+  `PostgresOperationStore`, `PostgresStore` (long-term memory, ranked by pgvector when
+  `vectorDimensions` is set), `PostgresTraceStore`, `PostgresDatasetStore`, `PostgresExperimentStore`,
+  and `PostgresCircuitStateStore`.
+  - Each takes any client with `query(text, values)` — `pg`, `@neondatabase/serverless`, PGlite — and
+    `fromPostgresJs()` adapts `postgres.js`. No driver becomes a dependency.
+  - Nothing creates a schema at import: `migrate()`, `postgresMigration()`, and `nexus db sql` apply
+    or print it.
+  - Idempotency keys are unique in the operations table, so two workers racing to submit one key run
+    the work once. Graph threads persist across workers through `OperationStoreCheckpointer`.
+- **Shared circuit state.** `circuitBreaker.store` takes a `CircuitStateStore`: memory and Redis
+  adapters in `nexus-ai-pro/ops/circuit-store`, Postgres in `/postgres/circuits`. A provider that
+  fails in one worker is taken out of routing in all of them, and one worker at a time probes it when
+  the cooldown ends. Checks stay synchronous; the coordination code loads only when a store is
+  configured. `ai.syncCircuitBreaker()` loads shared state at startup.
+- **Evaluation from the command line.** `nexus eval run`, `nexus eval compare`, and `nexus eval gate`
+  run an evaluation module to an experiment file and fail a build only on a regression beyond noise,
+  a new failure, or a dataset mismatch. `nexus traces list`, `show`, and `export` read a JSONL file or
+  any trace store a module exports. `nexus db sql` prints the Postgres schema. Every command takes
+  `--json`, and a usage error exits with 2.
+- **Record and replay** (`nexus-ai-pro/testing/record`). `recordingFetch()` writes provider traffic
+  to reviewable fixture files with credentials removed; `replayFetch()` serves it back with no network
+  and throws `FixtureMissingError` for anything unrecorded; `fixtureFetch()` picks the mode from
+  `NEXUS_FIXTURES`; `installFetch()` covers code that calls the global `fetch`. Multipart boundaries
+  and key order do not break matching.
+- **Image conformance from recordings.** `npm run conformance:images:record` records the image suite
+  against every backend with credentials present; `npm run test:conformance:images` replays it in CI
+  and reports any backend without recordings rather than counting it as a pass.
+- **Graph input and output channels.** `createGraph({ channels, input, output })` restricts what a
+  caller may pass in and what comes back, in the types and at run time. As a subgraph, a graph
+  receives only its inputs and merges back only its outputs.
+- **Per-node caching.** `addNode(name, fn, { cache: { key, ttlMs, store } })` reuses a node's result
+  for the same state and input, in a bounded in-process map or any cache adapter
+  (`compile({ cache })` sets a default). Failures, interrupts, human answers, and parent commands are
+  never cached; a failing cache is a miss.
+- **Evaluation additions.** `underCost()` per-example cost gate; `FileExperimentStore` and
+  `readExperiment()`; `repetitions` may be a function per example; the target receives the abort
+  signal.
+- `EvalRunner.run()` and `MediaEvalRunner.evaluate()` return the experiment underneath their report,
+  and accept options to name and store it.
+- `OperationDuplicateError`, raised by stores that enforce unique idempotency keys.
+- **Documentation coverage.** Every public export, and every public member of an exported class,
+  interface, or enum, now has a doc comment: 4,651 declarations across 88 entry points.
+  `npm run docs:check` enforces it and runs as part of `npm test`.
+
+### Changed
+
+- `EvalRunner` and `MediaEvalRunner` now run on `evaluate()`. Their results and errors are unchanged;
+  `evaluate()` is loaded the first time they run, so neither entry point grew.
+- A caller cancelling a request no longer counts against a provider's circuit.
+- Circuit probe limits are enforced on every attempt through the failover executor, so concurrent
+  requests go to a fallback instead of all probing a half-open provider.
+- Comments are stripped from the emitted JavaScript and kept in the type declarations, where editors
+  read them. Every entry point shrinks as a result: the root import is 572 KB, down from 612 KB, even
+  after the breaker fixes, the `EvalRunner` rebuild, and duplicate-key handling. The published size
+  table has the per-entry-point figures.
+- The packed tarball is 666 KB and 4.1 MB unpacked, up from 3.5 MB, almost all of it the doc comments
+  in the type declarations, which are shipped for both ESM and CommonJS.
+
+### Fixed
+
+- `evaluate()` ignored its `signal` option; aborting now stops new examples, reaches targets, and
+  never stores a partial experiment.
+- `evaluate()` never recorded cost, so `totalCost()` always reported 0. Cost is now read from
+  completion and image results or from a `cost` option.
+- A half-open probe slot was never released when an attempt's error did not count as a failure,
+  leaving the circuit refusing traffic until it was reset.
+- The breaker's documented `halfOpenMaxCalls` was not enforced on real requests: the client routed
+  every request to a half-open provider.
+- The context-window summary call could route to a provider whose circuit was open.
+- `routing.fallback` was accepted but never applied. `onError` models are now tried after every
+  route the router chose, `onTimeout` gives the first attempt its own timeout before its fallback,
+  and `onRateLimit` retries a rate-limited first attempt before falling back.
+- Privacy routing ignored a provider's own `isLocal` declaration, so an OpenAI-compatible server
+  configured as local was still scored as hosted.
+
+### Deprecated
+
+These options were accepted but never read. They still type-check, do nothing, and will be removed
+in 2.0.
+
+- `GoogleProviderConfig.projectId`.
+- `OllamaProviderConfig.timeout`.
+- `MetricsConfig.prometheus`: Prometheus text is always available from `getPrometheusMetrics()`.
+- `DensificationConfig.preserveMarkdown`: densification never rewrites Markdown structure.
+- `HealthConfig.latencyHalfLife`: latency is averaged with a fixed weight.
+
+## [1.15.0] - 2026-09-21
+
+Evaluation. One entry point for evaluating anything, datasets versioned by their content, experiments
+that can be compared with a verdict rather than a vibe, and the loop from a production failure back to
+a permanent regression test.
+
+### Added
+
+- **`nexus-ai-pro/evaluate`.** `evaluate(target, dataset, evaluators, options)` accepts any target: a
+  completion, an agent, a graph, an image operation, or plain code.
+  - Examples run concurrently, with repetitions for a target that is not deterministic, a timeout per
+    example, and per-metric statistics including a 95% interval.
+  - A target that throws is recorded as a failed example; an evaluator that throws is recorded as a
+    failed measurement. Neither voids the experiment.
+- **Datasets versioned by content.** Change an example and the version changes with it, so two
+  experiments are comparable only when they ran over the same data. `MemoryDatasetStore` and
+  `FileDatasetStore` (one reviewable JSON file per version), plus `splitOf()` for train and test
+  splits.
+- **`datasetFromTraces()`.** Turns recorded runs into examples, each keeping the run it came from, so
+  yesterday's production failure becomes tomorrow's regression test.
+- **Bundled evaluators.** `exactMatch`, `contains`, `mustNotMatch`, `completed`, `underLatency`,
+  `embeddingSimilarity` through any injected embedder, `pairwise` for side-by-side judgements, and
+  `trajectory`, which scores how an answer was reached rather than only what it said. `passRate` and
+  `totalCost` summarize a whole experiment.
+- **`compareExperiments()` with a real verdict.** A paired bootstrap over per-example differences
+  gives each metric a 95% interval; a metric whose interval spans zero is `unchanged` rather than an
+  improvement. The bootstrap is seeded, so a CI gate cannot flip between runs. The report names the
+  examples that moved most, the failures that are new, the ones that were fixed, and any dataset
+  version mismatch. `formatComparison()` prints it for a pull request.
+- **`evaluateOnline()`.** Scores sampled production runs, writes the scores back as trace feedback so
+  alert rules can watch them, and sends uncertain results to people.
+- **`AnnotationQueue`.** Review work with a rubric, claims that expire so nothing is stranded,
+  consensus across several reviewers, and `toExamples()` to turn reviewed items into dataset examples.
+
 ## [1.14.0] - 2026-09-20
 
 The memory, agent, and MCP work planned for 1.13.0 ships here together with tracing, so no 1.13.0 was
@@ -717,7 +846,9 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.14.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.16.0...HEAD
+[1.16.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.15.0...v1.16.0
+[1.15.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.14.0...v1.15.0
 [1.14.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.12.0...v1.14.0
 [1.12.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.11.0...v1.12.0
 [1.11.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.10.0...v1.11.0