nexus-ai-pro 1.23.0 → 1.24.0

package/API_STABILITY.md

@@ -437,8 +437,9 @@ from the root.
   `ThreadRecord`, and `RunEvent` follow the 1.x rules. A minor release may add routes and fields.
 - Event ids are per run and strictly increasing, which is what `Last-Event-ID` depends on. A client
   that reconnects with an id never receives it again.
-- The server is not a security boundary on its own: authentication, authorization beyond tenancy and
-  scopes, and rate limiting are the application's, through the hooks.
+- The server is not a security boundary on its own: authentication, and authorization beyond tenancy
+  and scopes, are the application's, through the hooks. Per-tenant limits are opt-in, through
+  `tenants`.
 - Recovery repeats the step a run died in for graphs and workflows served by `graphAssistant()`, and
   the whole run for any other assistant, so the repeated part should be idempotent or the run left at a
   single attempt.
@@ -456,6 +457,20 @@ with nexus-ai-pro and declares the core versions it works with as a peer depende
 - The access rules are a guarantee: loopback binding by default, a token on every request, the token
   in a header for every change, and loopback hosts only unless others are allowed.
 
+## Deployment stage (1.24.0)
+
+Deployments (`nexus-ai-pro/server/deployments`), tenant limits (`nexus-ai-pro/server/tenancy`), and the
+server's worker queue are experimental in production readiness. Their contracts follow the 1.x rules.
+
+- `bucket()` is stable across 1.x, so a thread keeps its place in a traffic split through an upgrade.
+- The stored `DeploymentRecord` and `ReplicaReport` shapes may gain fields but not lose them, so
+  replicas on different minor releases share one store.
+- The metric names on `/metrics` and the fields of `/scaling` follow the 1.x rules; a minor release may
+  add metrics and fields. Autoscaler configurations that read them keep working.
+- `drain()` hands off only runs it could not finish; a hand-off never counts against the retry budget.
+- The files under `deploy/` are templates in the repository, not part of the package, and may change in
+  any release.
+
 ## Team platform stage (1.23.0)
 
 The context hub (`nexus-ai-pro/context-hub`), insights (`nexus-ai-pro/insights`), and the studio's

package/CHANGELOG.md

@@ -4,6 +4,65 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [1.24.0] - 2026-09-30
+
+Deployment at scale, self-managed. Running assistants as a service without a hosted platform:
+revisions and canaries, a worker pool that scales on its queue, and limits per tenant.
+
+### Added
+
+- **Revisions and canaries** on `nexus-ai-pro/server/deployments`. `Deployments.assistant()` serves
+  several revisions of an assistant from one image; the traffic split is recorded in the server's
+  state store and changed with `canary()`, `split()`, `promote()`, and `rollback()`, each kept in a
+  history with who and why, and guarded by an expected version. New threads and stateless runs are
+  split by a stable hash, a thread keeps its revision while that revision takes traffic, and every
+  run records its revision, share, reason, and the deployment version.
+- **A canary guard**: `watchCanaries()` compares each canary with the live revision over the same
+  window, rolls it back when its error rate, latency, or cost regressed, and moves one that holds up
+  through its `steps` to promotion. A latency rise must be at least `minLatencyChangeMs`, 50 ms by
+  default, so noise on fast runs never rolls a canary back. `compareRuns()` on `nexus-ai-pro/insights`
+  is the comparison, lifted out of `detectRegressions()`.
+- **A worker queue** in the agent server: with `queue`, a replica runs at most `concurrency` runs and
+  leaves the rest queued for the next free worker; `claim: false` makes an API replica. Replicas take
+  over lapsed runs every `recoverEveryMs`. `drain()` answers `/health` with `503`, stops claiming, and
+  hands runs still going after its timeout to another worker without using a retry.
+- **Scaling signals**: `GET /scaling` as JSON and `GET /metrics` in Prometheus text — queued and running
+  runs across the deployment, their load, the oldest wait, lapsed leases, each replica's capacity, and
+  run counts and latency per revision. `metrics: { public: true }` serves them to in-cluster scrapers.
+- **Kubernetes and Helm**: `deploy/app` is one image run as an API, a worker, or both, configured from
+  the environment; `deploy/kubernetes` has the manifests with a KEDA `ScaledObject` and an HPA
+  alternative; `deploy/helm/agent-server` is the chart. The clean-install test runs the template against
+  the packed package.
+- **Tenant limits** on `nexus-ai-pro/server/tenancy`: `tenantLimits()` holds each tenant to its active
+  runs, its runs per window, and its spending per period, answered with `429` and `Retry-After`.
+  `MemoryTenantUsage`, and `RedisTenantUsage` with atomic scripts so replicas share one set of limits.
+  Assistants record spending with `context.recordCost()`; `GET /usage` reports it.
+- **Replica health**: with `deployments`, each replica heartbeats its runs, capacity, drain state, and
+  revisions, readable from `GET /replicas`.
+- **A deployments view** in the studio: traffic splits and history, each revision's runs, error rate,
+  and latency since the last change, replicas, the queue, and tenant usage, with canary, promote, and
+  rollback for admins.
+- **`nexus deploy`**: `status`, `canary`, `promote`, and `rollback` against a running server, for the
+  pipeline that rolls out an image.
+- Operation queues: `OperationRunner.enqueue()`, `claimQueued()`, and `release()`; `listQueued()` and
+  `stats()` on every operation store, and `operationStats()`.
+- Run records gain `revision`, `startedAt`, `finishedAt`, `durationMs`, `cost`, `worker`, and `attempt`.
+
+### Changed
+
+- `GET /health` is answered without authentication, so a load balancer's or Kubernetes' probe reaches
+  it.
+- A worker never takes over an operation whose lease another worker still holds.
+- `RedisStoreLikeClient` declares `set()` with its expiry and `del()` with one key, the shapes an
+  `ioredis` client satisfies as it is.
+- The last nine guides are rewritten for readability.
+- `nexus-ai-pro-studio` requires nexus-ai-pro 1.24 or newer.
+
+### Fixed
+
+- The deployment templates run. The Dockerfile started a file the build never produced, and the
+  example server ignored `REDIS_URL`, so its replicas shared nothing.
+
 ## [1.23.0] - 2026-09-30
 
 The team engineering platform. The studio becomes something a team shares, and it starts finding
@@ -1152,7 +1211,8 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.23.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.24.0...HEAD
+[1.24.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.23.0...v1.24.0
 [1.23.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.22.0...v1.23.0
 [1.22.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.21.0...v1.22.0
 [1.21.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.20.0...v1.21.0

package/README.md

@@ -99,6 +99,8 @@ You can still pass a plain `NexusAIConfig` to `new NexusAI(...)` when you want f
   environment
 - version prompts, instructions, tools, and skills together, and promote them through evaluation gates
 - find failing and slow runs, detect regressions, and review evaluated fix proposals in a shared studio
+- serve assistants as revisions, with canaries that roll back on a regression, workers that scale on
+  the queue under Kubernetes, and per-tenant quotas, rate limits, and budgets
 - trip routing away from a failing provider, and share one rate-limit budget across workers
 - reach the providers' half-price asynchronous batch tier behind one operation handle
 - persist generated media to disk or S3 with tenant isolation, retention, and checksums
@@ -139,8 +141,9 @@ from the doc comments. The guides live in the repository, so these links go to G
 | [Realtime voice](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/realtime.md) | Browser and server realtime sessions with barge-in, tools, and exports |
 | [Telephony](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/telephony.md) | Calls, webhooks, phone numbers, and phone agents on realtime sessions |
 | [Testing](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/testing.md) | Recording and replaying provider traffic, and conformance suites |
-| [Studio](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/studio.md) | A local UI for traces, threads, approvals, experiments, prompts, costs, and health, as a separate package |
-| [Agent server](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/server.md) | Self-hosted HTTP server for assistants: threads, durable runs, resumable streams, cron |
+| [Studio](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/studio.md) | A shared UI for traces, threads, approvals, experiments, prompts, deployments, costs, and health, as a separate package |
+| [Agent server](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/server.md) | Self-hosted HTTP server for assistants: threads, durable runs, resumable streams, cron, a worker queue, and metrics |
+| [Deployments](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/deployments.md) | Revisions and canaries with automatic rollback, autoscaling on Kubernetes and Helm, and per-tenant limits |
 | [Workflows](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/workflows.md) | Ready-made chains and domain workflows |
 | [Command line](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/cli.md) | The `nexus` command |
 | [Packaging and install weight](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/packaging.md) | Every entry point and what it costs |
@@ -219,9 +222,11 @@ Shipped so far:
   as images;
 - document loaders, five more vector stores, hybrid retrieval with reranking, and an MCP registry;
 - a shared studio with accounts, roles, an audit log, and comments; a context hub; insights with
-  evaluated fix proposals; and evaluation caching.
+  evaluated fix proposals; and evaluation caching;
+- self-managed deployment at scale: revisions and canaries with automatic rollback, a worker queue
+  that autoscales on Kubernetes, and per-tenant limits.
 
-Next: self-managed deployment at scale, before 2.0.0 consolidates. The image family leaves experimental once recorded live
+Next: 2.0.0 consolidates. The image family leaves experimental once recorded live
 conformance passes on all three backends.
 
 ## Before Production

package/dist/cli/deploy.d.ts

@@ -0,0 +1,15 @@
+import { type ParsedArgs } from './args.js';
+/**
+ * `nexus deploy` reads and changes an agent server's deployments over HTTP, for a pipeline that rolls
+ * out a new image and then moves traffic onto it:
+ *
+ *   nexus deploy status [assistant]
+ *   nexus deploy canary <assistant> <revision> <percent>
+ *   nexus deploy promote <assistant> <revision>
+ *   nexus deploy rollback <assistant> [--to <revision>]
+ *
+ * `--url` (or `NEXUS_SERVER_URL`) names the server, and `--token` (or `NEXUS_SERVER_TOKEN`) is sent
+ * as a bearer token. Changes take `--reason`, recorded in the deployment's history, and `--version`,
+ * which refuses the change if someone else moved the deployment first.
+ */
+export declare function runDeployCommand(subcommand: string | undefined, { positionals, flags }: ParsedArgs): Promise<void>;

package/dist/cli/deploy.js

@@ -0,0 +1,116 @@
+import { CliUsageError, numberFlag, printTable, stringFlag, writeJson } from './args.js';
+export async function runDeployCommand(subcommand, { positionals, flags }) {
+    const server = serverOf(flags);
+    const reason = stringFlag(flags, 'reason');
+    const expectedVersion = numberFlag(flags, 'version');
+    const change = async (assistant, body) => {
+        if (!assistant)
+            throw new CliUsageError(`nexus deploy ${subcommand} needs an assistant.`);
+        const deployment = await server.request('POST', `/deployments/${encodeURIComponent(assistant)}`, {
+            ...body,
+            ...(reason ? { reason } : {}),
+            ...(expectedVersion === undefined ? {} : { expectedVersion }),
+        });
+        if (flags.json === true)
+            writeJson(deployment);
+        else
+            console.log(`${assistant}: version ${deployment.version}, live ${deployment.live}, ${splitText(deployment.traffic)}`);
+    };
+    switch (subcommand) {
+        case 'status': {
+            const [assistant] = positionals;
+            if (assistant) {
+                const deployment = await server.request('GET', `/deployments/${encodeURIComponent(assistant)}`);
+                if (flags.json === true)
+                    return writeJson(deployment);
+                console.log(`${deployment.assistant}: version ${deployment.version}, live ${deployment.live}`);
+                console.log(`split: ${splitText(deployment.traffic)}`);
+                if (deployment.stats?.length) {
+                    console.log('');
+                    printTable(deployment.stats.map((item) => ({
+                        revision: item.revision || '(none)',
+                        runs: String(item.runs),
+                        failed: String(item.failed),
+                        'error rate': `${(item.errorRate * 100).toFixed(1)}%`,
+                        p95: item.p95Ms === undefined ? '' : `${Math.round(item.p95Ms)}ms`,
+                    })), ['revision', 'runs', 'failed', 'error rate', 'p95']);
+                }
+                if (deployment.history.length) {
+                    console.log('');
+                    printTable(deployment.history.slice(0, 10).map((entry) => ({
+                        version: String(entry.version),
+                        at: entry.at,
+                        action: entry.action,
+                        split: splitText(entry.traffic),
+                        by: entry.by ?? '',
+                        reason: entry.reason ?? '',
+                    })), ['version', 'at', 'action', 'split', 'by', 'reason']);
+                }
+                return;
+            }
+            const { deployments } = await server.request('GET', '/deployments');
+            if (flags.json === true)
+                return writeJson(deployments);
+            printTable(deployments.map((deployment) => ({
+                assistant: deployment.assistant,
+                live: deployment.live,
+                split: splitText(deployment.traffic),
+                version: String(deployment.version),
+                updated: deployment.version ? deployment.updatedAt : 'never',
+            })), ['assistant', 'live', 'split', 'version', 'updated']);
+            return;
+        }
+        case 'canary': {
+            const [assistant, revision, percent] = positionals;
+            const share = Number(percent);
+            if (!revision || !Number.isFinite(share) || share < 0 || share > 100) {
+                throw new CliUsageError('Usage: nexus deploy canary <assistant> <revision> <percent from 0 to 100>');
+            }
+            return change(assistant, { action: 'canary', revision, weight: share / 100 });
+        }
+        case 'promote': {
+            const [assistant, revision] = positionals;
+            if (!revision)
+                throw new CliUsageError('Usage: nexus deploy promote <assistant> <revision>');
+            return change(assistant, { action: 'promote', revision });
+        }
+        case 'rollback': {
+            const [assistant] = positionals;
+            const to = stringFlag(flags, 'to');
+            return change(assistant, { action: 'rollback', ...(to ? { to } : {}) });
+        }
+        default:
+            throw new CliUsageError(`Unknown deploy command "${subcommand ?? ''}". Use status, canary, promote, or rollback.`);
+    }
+}
+function serverOf(flags) {
+    const url = stringFlag(flags, 'url') ?? process.env.NEXUS_SERVER_URL;
+    if (!url)
+        throw new CliUsageError('nexus deploy needs the server: pass --url or set NEXUS_SERVER_URL.');
+    const token = stringFlag(flags, 'token') ?? process.env.NEXUS_SERVER_TOKEN;
+    const base = url.replace(/\/+$/, '');
+    return {
+        async request(method, route, body) {
+            const response = await fetch(`${base}${route}`, {
+                method,
+                headers: {
+                    ...(token ? { authorization: `Bearer ${token}` } : {}),
+                    ...(body === undefined ? {} : { 'content-type': 'application/json' }),
+                },
+                ...(body === undefined ? {} : { body: JSON.stringify(body) }),
+            });
+            const text = await response.text();
+            const parsed = text ? JSON.parse(text) : {};
+            if (!response.ok) {
+                const error = parsed.error;
+                throw new Error(`${response.status} ${error?.code ?? response.statusText}: ${error?.message ?? text}`);
+            }
+            return parsed;
+        },
+    };
+}
+function splitText(traffic) {
+    return Object.entries(traffic)
+        .map(([revision, weight]) => `${revision} ${Math.round(weight * 1000) / 10}%`)
+        .join(', ');
+}

package/dist/cli.js

@@ -49,6 +49,12 @@ async function main() {
             await runDbCommand(subcommand, { positionals: args, flags: parsed.flags });
             return;
         }
+        case 'deploy': {
+            const [subcommand, ...args] = parsed.positionals;
+            const { runDeployCommand } = await import('./cli/deploy.js');
+            await runDeployCommand(subcommand, { positionals: args, flags: parsed.flags });
+            return;
+        }
         case 'optimize':
             await runOptimize(parsed);
             return;
@@ -390,12 +396,17 @@ Usage:
   nexus traces show <traceId> --store <runs.jsonl|store.mjs> [--json]
   nexus traces export --store <runs.jsonl|store.mjs> [--out runs.jsonl] [same filters as list]
   nexus db sql [--adapters operations,store,traces,evaluation,circuits,prompts] [--vector-dimensions 1536]
+  nexus deploy status [assistant] --url <server> [--token t] [--json]
+  nexus deploy canary <assistant> <revision> <percent> --url <server> [--reason r] [--version n]
+  nexus deploy promote <assistant> <revision> --url <server> [--reason r] [--version n]
+  nexus deploy rollback <assistant> [--to revision] --url <server> [--reason r] [--version n]
   nexus optimize [request.json|prompt.txt] [--model gpt-5-mini] [--max-input-tokens 4000] [--densify] [--json]
 
 Eval files can export { cases, client } or { cases, config }. JSON cases may use "expected", "contains", and "match": "includes" | "exact" | "regex".
 Evaluation modules for "eval run" export { target, dataset, evaluators, summary?, options? }.
 "eval gate" exits 1 when a metric got worse beyond noise, a new failure appeared, or the datasets differ.
-Trace store modules export any TraceStore as { store }, such as a PostgresTraceStore over the application's own pool.`);
+Trace store modules export any TraceStore as { store }, such as a PostgresTraceStore over the application's own pool.
+"deploy" reads NEXUS_SERVER_URL and NEXUS_SERVER_TOKEN when --url and --token are not given.`);
 }
 main().catch((error) => {
     const message = error instanceof Error ? error.message : String(error);

package/dist/index.d.ts

@@ -19,7 +19,7 @@ export { BullMQOperationDispatcher, RedisOperationStore, type BullMQLikeOperatio
 export { OPERATION_WEBHOOK_SIGNATURE_HEADER, deliverOperationWebhook, signOperationWebhook, verifyOperationWebhook, } from './operations/webhooks.js';
 export { TERMINAL_OPERATION_STATUSES } from './types/operations.js';
 export { allowedTransitions, assertTransition, canTransition, isClaimable, isSettled, isTerminalOperationStatus, } from './operations/state-machine.js';
-export { OperationCancelledError, OperationConflictError, OperationDuplicateError, OperationError, OperationExpiredError, OperationLeaseLostError, OperationNotFoundError, OperationSerializationError, OperationTransitionError, } from './operations/errors.js';
+export { OperationCancelledError, OperationConflictError, OperationDuplicateError, OperationError, OperationExpiredError, OperationLeaseLostError, OperationNotFoundError, OperationReleasedError, OperationSerializationError, OperationTransitionError, } from './operations/errors.js';
 export { BatchManager, type BatchManagerRuntime } from './batch/manager.js';
 export { BatchCapabilityError, BatchError, BatchProviderError, BatchProviderNotFoundError, BatchProviderResponseError, BatchValidationError, } from './batch/errors.js';
 export { TERMINAL_BATCH_STATUSES, isTerminalBatchStatus } from './types/batch.js';
@@ -94,7 +94,7 @@ export type { SpeechRequest, SpeechResponse, TranscriptionRequest, Transcription
 export type { AssetBytesLocation, AssetChecksum, AssetDescriptor, AssetInput, AssetLocation, AssetLocationKind, AssetProvenance, AssetStoredLocation, AssetUrlLocation, ImageBackground, ImageConfig, ImageDelivery, ImageDimensions, ImageEditRequest, ImageGenerateRequest, ImageManagerConfig, ImageMaskInput, ImageOperation, ImageOperationSubmission, ImageOutputFormat, ImageProvider, ImageProviderCallContext, ImageProviderCapabilities, ImageProviderInfo, ImageQuality, ImageRequestBase, ImageResult, ImageSafetyContext, ImageSafetyPolicy, ImageWarning, MediaSafetyFinding, MediaUsage, OperationErrorDescriptor, OperationEvent, OperationEventBase, OperationHandle, OperationMeta, OperationStatus, } from './types/images.js';
 export type { AssetCapacityConstraint, AssetPutOptions, AssetSignOptions, AssetSigner, AssetSignerContext, AssetStat, AssetStore, AssetStoreResult, ByteAssetDescriptor, ByteAssetInput, MemoryAssetStoreOptions, MemoryAssetStoreSnapshot, } from './images/assets.js';
 export type { BatchConfig, BatchCounts, BatchInputItem, BatchJobRef, BatchJobResult, BatchJobState, BatchJobStatus, BatchOutputItem, BatchProvider, BatchProviderCallContext, BatchProviderCapabilities, BatchProviderInfo, BatchSubmitRequest, } from './types/batch.js';
-export type { DurableOperationHandle, OperationContext, OperationDispatcher, OperationExecutor, OperationLease, OperationRecord, OperationRetryConfig, OperationRunnerConfig, OperationStore, OperationSubmitOptions, OperationWebhookConfig, } from './types/operations.js';
+export type { DurableOperationHandle, OperationContext, OperationDispatcher, OperationExecutor, OperationLease, OperationRecord, OperationRetryConfig, OperationRunnerConfig, OperationStore, OperationStoreFilter, OperationStoreStats, OperationSubmitOptions, OperationWebhookConfig, } from './types/operations.js';
 export type { Embedding, EmbeddingConfig, EmbeddingCostBudgetConfig, EmbeddingEncodingFormat, EmbeddingInput, EmbeddingInputType, EmbeddingMeta, EmbeddingModelCapabilities, EmbeddingModelRegistryConfig, EmbeddingProviderCallContext, EmbeddingProviderCapabilities, EmbeddingProviderInfo, EmbeddingProviderRequest, EmbeddingProviderResult, EmbeddingProviderUsage, EmbeddingRequest, EmbeddingResponse, EmbeddingTruncateMode, EmbeddingsProvider, } from './types/embeddings.js';
 export type { CreateCallRequest, CreateCallResponse, TelephonyAudioEncoding, TelephonyCallDirection, TelephonyCallStatus, TelephonyConfig, TelephonyConnectedEvent, TelephonyDtmfEvent, TelephonyGatherConfig, TelephonyHttpMethod, TelephonyMarkEvent, TelephonyMediaEvent, TelephonyMediaStreamEvent, TelephonyOutboundAudioMessage, TelephonyProvider, TelephonyProviderInfo, TelephonyResponseRequest, TelephonyStartEvent, TelephonyStopEvent, TelephonyStreamConfig, TelephonyStreamMode, TelephonyStreamTrack, TelephonyWebhookResponse, TelephonyWebhookValidationRequest, } from './types/telephony.js';
 export type { AliasMetadata, AliasStage, CacheTtl, Modality, ModelCapabilities, ModelEndpoint, ModelStatus, PromptCachingCapability, ProviderCapabilities, ReasoningEffort, RoutingModelPreference, } from './types/providers.js';

package/dist/index.js

@@ -19,7 +19,7 @@ export { BullMQOperationDispatcher, RedisOperationStore, } from './operations/ad
 export { OPERATION_WEBHOOK_SIGNATURE_HEADER, deliverOperationWebhook, signOperationWebhook, verifyOperationWebhook, } from './operations/webhooks.js';
 export { TERMINAL_OPERATION_STATUSES } from './types/operations.js';
 export { allowedTransitions, assertTransition, canTransition, isClaimable, isSettled, isTerminalOperationStatus, } from './operations/state-machine.js';
-export { OperationCancelledError, OperationConflictError, OperationDuplicateError, OperationError, OperationExpiredError, OperationLeaseLostError, OperationNotFoundError, OperationSerializationError, OperationTransitionError, } from './operations/errors.js';
+export { OperationCancelledError, OperationConflictError, OperationDuplicateError, OperationError, OperationExpiredError, OperationLeaseLostError, OperationNotFoundError, OperationReleasedError, OperationSerializationError, OperationTransitionError, } from './operations/errors.js';
 export { BatchManager } from './batch/manager.js';
 export { BatchCapabilityError, BatchError, BatchProviderError, BatchProviderNotFoundError, BatchProviderResponseError, BatchValidationError, } from './batch/errors.js';
 export { TERMINAL_BATCH_STATUSES, isTerminalBatchStatus } from './types/batch.js';

package/dist/insights/index.d.ts

@@ -5,4 +5,4 @@
  */
 export { type ClusterBy, type ClusterOptions, clusterRuns, errorSignature, type FindIssuesOptions, findIssues, type Issue, type RunCluster, trajectoryOf, } from './clusters.js';
 export { FileProposalStore, type FixCandidate, type FixModelClient, type FixProposal, type FixProposer, type FixRequest, type FixSubject, MemoryProposalStore, type ModelFixProposerOptions, modelFixProposer, ProposalInbox, type ProposalInboxOptions, type ProposalStore, type ProposeFixOptions, proposeFix, type PullRequestClient, } from './proposals.js';
-export { type DetectRegressionsOptions, detectRegressions, type Regression, type RegressionMetric, type TimeWindow, } from './regressions.js';
+export { type CompareRunsOptions, compareRuns, type DetectRegressionsOptions, detectRegressions, type Regression, type RegressionMetric, type RunSample, type TimeWindow, } from './regressions.js';

package/dist/insights/index.js

@@ -1,3 +1,3 @@
 export { clusterRuns, errorSignature, findIssues, trajectoryOf, } from './clusters.js';
 export { FileProposalStore, MemoryProposalStore, modelFixProposer, ProposalInbox, proposeFix, } from './proposals.js';
-export { detectRegressions, } from './regressions.js';
+export { compareRuns, detectRegressions, } from './regressions.js';

package/dist/insights/regressions.d.ts

@@ -8,6 +8,43 @@ export interface TimeWindow {
 }
 /** A metric `detectRegressions()` can compare. */
 export type RegressionMetric = 'error-rate' | 'latency' | 'cost';
+/**
+ * The part of a run `compareRuns()` reads. A traced `Run` satisfies it, and so does a server run
+ * mapped to it, which is how a canary is judged against the live revision.
+ */
+export interface RunSample {
+    /** `error` counts against the error rate; anything else counts as a success. */
+    status: string;
+    /** How long it took, for p95 latency. */
+    latencyMs?: number;
+    /** What it cost, for mean cost. */
+    cost?: number;
+    /** Scores people or evaluators gave it, for the feedback comparison. */
+    feedback?: ReadonlyArray<{
+        key: string;
+        score?: number;
+    }>;
+}
+/** Options for `compareRuns()`: what is compared, and how large a change must be to count. */
+export interface CompareRunsOptions {
+    /** Names the group in each regression and its summary. Defaults to `runs`. */
+    group?: string;
+    /** Metrics compared. Defaults to all three. */
+    metrics?: readonly RegressionMetric[];
+    /** Feedback keys whose mean score is compared, such as `helpfulness`. */
+    feedback?: readonly string[];
+    /** Runs, or feedback scores, each side needs before it is judged. Defaults to 20. */
+    minRuns?: number;
+    /** A rise in p95 latency, relative, that counts. Defaults to 0.25, a quarter slower. */
+    latencyIncrease?: number;
+    /**
+     * The smallest rise in p95 latency, in milliseconds, that counts, however large it is relatively.
+     * Defaults to 0; raise it so a run that goes from 2 ms to 3 ms is not a regression.
+     */
+    minLatencyChangeMs?: number;
+    /** A rise in mean cost, relative, that counts. Defaults to 0.25. */
+    costIncrease?: number;
+}
 /** Options for `detectRegressions()`. */
 export interface DetectRegressionsOptions {
     /** Where the runs are. */
@@ -28,6 +65,11 @@ export interface DetectRegressionsOptions {
     minRuns?: number;
     /** A rise in p95 latency, relative, that counts. Defaults to 0.25, a quarter slower. */
     latencyIncrease?: number;
+    /**
+     * The smallest rise in p95 latency, in milliseconds, that counts, however large it is relatively.
+     * Defaults to 0; raise it so a run that goes from 2 ms to 3 ms is not a regression.
+     */
+    minLatencyChangeMs?: number;
     /** A rise in mean cost, relative, that counts. Defaults to 0.25. */
     costIncrease?: number;
     /** Only root runs. On by default. */
@@ -63,3 +105,13 @@ export interface Regression {
  * judged, so a quiet endpoint does not raise alarms on noise.
  */
 export declare function detectRegressions(options: DetectRegressionsOptions): Promise<Regression[]>;
+/**
+ * Compares two sets of runs and reports what got worse in the second, with the same tests
+ * `detectRegressions()` applies to time windows: a two-proportion z-test at 95% for the error rate, a
+ * relative margin for p95 latency and mean cost, and twice the standard error for a feedback score.
+ * Neither side is judged below `minRuns`, so a handful of runs never raises an alarm.
+ *
+ * The two sets can be anything comparable: last week and this week, or — as the canary guard in
+ * `nexus-ai-pro/server/deployments` uses it — the live revision and a canary over the same hour.
+ */
+export declare function compareRuns(baseline: readonly RunSample[], current: readonly RunSample[], options?: CompareRunsOptions): Regression[];

package/dist/insights/regressions.js

@@ -20,89 +20,98 @@ export async function detectRegressions(options) {
     };
     const baselineGroups = group(baselineRuns);
     const currentGroups = group(currentRuns);
+    const regressions = [];
+    for (const [name, current] of currentGroups) {
+        regressions.push(...compareRuns(baselineGroups.get(name) ?? [], current, { ...options, group: name }));
+    }
+    return regressions;
+}
+export function compareRuns(baseline, current, options = {}) {
+    const name = options.group ?? 'runs';
     const minRuns = options.minRuns ?? 20;
     const metrics = new Set(options.metrics ?? ['error-rate', 'latency', 'cost']);
     const regressions = [];
-    for (const [name, current] of currentGroups) {
-        const baseline = baselineGroups.get(name) ?? [];
-        const samples = { baseline: baseline.length, current: current.length };
-        if (samples.baseline >= minRuns && samples.current >= minRuns) {
-            if (metrics.has('error-rate')) {
-                const before = baseline.filter((run) => run.status === 'error').length;
-                const after = current.filter((run) => run.status === 'error').length;
-                const p1 = before / baseline.length;
-                const p2 = after / current.length;
-                const pooled = (before + after) / (baseline.length + current.length);
-                const standardError = Math.sqrt(pooled * (1 - pooled) * (1 / baseline.length + 1 / current.length));
-                if (p2 > p1 && standardError > 0 && (p2 - p1) / standardError >= 1.96) {
-                    regressions.push({
-                        group: name,
-                        metric: 'error-rate',
-                        baseline: p1,
-                        current: p2,
-                        change: p2 - p1,
-                        samples,
-                        summary: `${name}: error rate rose from ${percent(p1)} to ${percent(p2)}`,
-                    });
-                }
-            }
-            if (metrics.has('latency')) {
-                const before = percentile(baseline.map((run) => run.latencyMs).filter(isNumber), 0.95);
-                const after = percentile(current.map((run) => run.latencyMs).filter(isNumber), 0.95);
-                const margin = options.latencyIncrease ?? 0.25;
-                if (before !== undefined && after !== undefined && before > 0 && after >= before * (1 + margin)) {
-                    regressions.push({
-                        group: name,
-                        metric: 'latency-p95',
-                        baseline: before,
-                        current: after,
-                        change: after / before - 1,
-                        samples,
-                        summary: `${name}: p95 latency rose from ${Math.round(before)}ms to ${Math.round(after)}ms`,
-                    });
-                }
+    const samples = { baseline: baseline.length, current: current.length };
+    if (samples.baseline >= minRuns && samples.current >= minRuns) {
+        if (metrics.has('error-rate')) {
+            const before = baseline.filter((run) => run.status === 'error').length;
+            const after = current.filter((run) => run.status === 'error').length;
+            const p1 = before / baseline.length;
+            const p2 = after / current.length;
+            const pooled = (before + after) / (baseline.length + current.length);
+            const standardError = Math.sqrt(pooled * (1 - pooled) * (1 / baseline.length + 1 / current.length));
+            if (p2 > p1 && standardError > 0 && (p2 - p1) / standardError >= 1.96) {
+                regressions.push({
+                    group: name,
+                    metric: 'error-rate',
+                    baseline: p1,
+                    current: p2,
+                    change: p2 - p1,
+                    samples,
+                    summary: `${name}: error rate rose from ${percent(p1)} to ${percent(p2)}`,
+                });
             }
-            if (metrics.has('cost')) {
-                const before = mean(baseline.map((run) => run.cost).filter(isNumber));
-                const after = mean(current.map((run) => run.cost).filter(isNumber));
-                const margin = options.costIncrease ?? 0.25;
-                if (before !== undefined && after !== undefined && before > 0 && after >= before * (1 + margin)) {
-                    regressions.push({
-                        group: name,
-                        metric: 'cost',
-                        baseline: before,
-                        current: after,
-                        change: after / before - 1,
-                        samples,
-                        summary: `${name}: mean cost rose from $${before.toFixed(4)} to $${after.toFixed(4)}`,
-                    });
-                }
+        }
+        if (metrics.has('latency')) {
+            const before = percentile(baseline.map((run) => run.latencyMs).filter(isNumber), 0.95);
+            const after = percentile(current.map((run) => run.latencyMs).filter(isNumber), 0.95);
+            const margin = options.latencyIncrease ?? 0.25;
+            if (before !== undefined &&
+                after !== undefined &&
+                before > 0 &&
+                after >= before * (1 + margin) &&
+                after - before >= (options.minLatencyChangeMs ?? 0)) {
+                regressions.push({
+                    group: name,
+                    metric: 'latency-p95',
+                    baseline: before,
+                    current: after,
+                    change: after / before - 1,
+                    samples,
+                    summary: `${name}: p95 latency rose from ${Math.round(before)}ms to ${Math.round(after)}ms`,
+                });
             }
         }
-        for (const feedbackKey of options.feedback ?? []) {
-            const scores = (runs) => runs
-                .flatMap((run) => (run.feedback ?? []).filter((item) => item.key === feedbackKey).map((item) => item.score))
-                .filter(isNumber);
-            const before = scores(baseline);
-            const after = scores(current);
-            if (before.length < minRuns || after.length < minRuns)
-                continue;
-            const m1 = mean(before);
-            const m2 = mean(after);
-            const standardError = Math.sqrt(variance(before) / before.length + variance(after) / after.length);
-            if (m2 < m1 && (standardError === 0 || (m1 - m2) / standardError >= 1.96)) {
+        if (metrics.has('cost')) {
+            const before = mean(baseline.map((run) => run.cost).filter(isNumber));
+            const after = mean(current.map((run) => run.cost).filter(isNumber));
+            const margin = options.costIncrease ?? 0.25;
+            if (before !== undefined && after !== undefined && before > 0 && after >= before * (1 + margin)) {
                 regressions.push({
                     group: name,
-                    metric: `feedback:${feedbackKey}`,
-                    baseline: m1,
-                    current: m2,
-                    change: m2 - m1,
-                    samples: { baseline: before.length, current: after.length },
-                    summary: `${name}: ${feedbackKey} fell from ${m1.toFixed(2)} to ${m2.toFixed(2)}`,
+                    metric: 'cost',
+                    baseline: before,
+                    current: after,
+                    change: after / before - 1,
+                    samples,
+                    summary: `${name}: mean cost rose from $${before.toFixed(4)} to $${after.toFixed(4)}`,
                 });
             }
         }
     }
+    for (const feedbackKey of options.feedback ?? []) {
+        const scores = (runs) => runs
+            .flatMap((run) => (run.feedback ?? []).filter((item) => item.key === feedbackKey).map((item) => item.score))
+            .filter(isNumber);
+        const before = scores(baseline);
+        const after = scores(current);
+        if (before.length < minRuns || after.length < minRuns)
+            continue;
+        const m1 = mean(before);
+        const m2 = mean(after);
+        const standardError = Math.sqrt(variance(before) / before.length + variance(after) / after.length);
+        if (m2 < m1 && (standardError === 0 || (m1 - m2) / standardError >= 1.96)) {
+            regressions.push({
+                group: name,
+                metric: `feedback:${feedbackKey}`,
+                baseline: m1,
+                current: m2,
+                change: m2 - m1,
+                samples: { baseline: before.length, current: after.length },
+                summary: `${name}: ${feedbackKey} fell from ${m1.toFixed(2)} to ${m2.toFixed(2)}`,
+            });
+        }
+    }
     return regressions;
 }
 function isNumber(value) {

package/dist/operations/adapters.d.ts

@@ -1,4 +1,4 @@
-import type { OperationDispatcher, OperationRecord, OperationStore } from '../types/operations.js';
+import type { OperationDispatcher, OperationRecord, OperationStore, OperationStoreFilter, OperationStoreStats } from '../types/operations.js';
 /**
  * The Redis commands the operation store needs.
  *
@@ -57,6 +57,10 @@ export declare class RedisOperationStore<TResult = unknown> implements Operation
     findByIdempotencyKey(key: string): Promise<OperationRecord<TResult> | undefined>;
     /** Every record. */
     list(): Promise<Array<OperationRecord<TResult>>>;
+    /** Queued records no worker holds, oldest first. Reads every record, as `list()` does. */
+    listQueued(limit: number, filter?: OperationStoreFilter): Promise<Array<OperationRecord<TResult>>>;
+    /** Counts unfinished records by status. Reads every record, as `list()` does. */
+    stats(now: string, filter?: OperationStoreFilter): Promise<OperationStoreStats>;
     private recordsKey;
     private idempotencyKey;
 }