nexus-ai-pro 2.11.0 → 2.13.0

package/API_STABILITY.md

@@ -41,6 +41,75 @@ Both are covered by this policy:
 Deep imports into `dist`, `dist-cjs`, or `src` are not supported. Import only from `nexus-ai-pro` or
 one of its explicit subpaths.
 
+## 2.13 stage (2.13.0)
+
+Everything 2.13 adds is stable from the start and additive, unless this section says otherwise.
+
+New and stable:
+- `nexus-ai-pro/server/regions`: `regionalGateway()`, `RegionalGateway`, `RegionalGatewayOptions`,
+  `RegionTarget`, `RegionHealth`, and `RegionHealthOptions`;
+- `RunManagerOptions.region` and `residency`, as `AgentServerOptions` carries them; `ThreadRecord.region`,
+  `RunRecord.region`, and `ReplicaReport.region`; a regional server's ids start with its region;
+- `nexus-ai-pro/residency`: `residency()`, `Residency`, `ResidencyOptions`, `RegionRule`,
+  `RegionDefinition`, `ResidencyPolicy`, `ResidencyDecision`, `ResidencyTarget`, `ResidencyPlacement`,
+  `CallResidency`, `ResidencyError`, `regionalTraceStore()`, `RegionalTraceStore`,
+  `RegionalTraceStoreOptions`, `regionalAssetStore()`, and `RegionalAssetStoreOptions`;
+- `LifecycleConfig.residency`, `ProviderCallContext.residency`, and `ImageRequestBase.tenantId`;
+- `nexus-ai-pro/credentials`: `tenantCredentials()`, `TenantCredentials`, `TenantCredentialsOptions`,
+  `TenantCredentialStats`, `CredentialError`, `parseSecret()`, `credentialRequest()`, `SecretSource`,
+  `awsSecrets()`, `gcpSecrets()`, `azureSecrets()`, `vaultSecrets()`, `envSecrets()`,
+  `memorySecrets()`, `MemorySecrets`, `AwsSecretsClient`, `GcpSecretsClient`, `AzureSecretsClient`,
+  and `VaultSecretsOptions`;
+- `LifecycleConfig.credentials`, `CredentialResolver`, `CredentialRequest`, and `ProviderCredential`;
+  `ProviderCallContext.tenantId` and `credential`; `OperationTicket.refreshCredential()` and
+  `scrub()`;
+- `OpenAIProviderConfig.credentialHeader`, `EmbeddingRequest.tenantId`, and
+  `EmbeddingProviderError.attempts`.
+
+## 2.12 stage (2.12.0)
+
+Everything 2.12 adds is stable from the start and additive, unless this section says otherwise.
+
+New and stable:
+- `nexus-ai-pro/tracing/spool`: `spoolingExporter()`, `SpoolingExporter`, `SpoolingExporterOptions`,
+  `TraceBatchSender`, `SpoolStats`, and `SpoolDrainReport`;
+- `OtlpTraceExporter.send()`;
+- `RunManagerOptions.tracer` and `AgentServerOptions.tracer`, and `RunRecord.traceId`;
+- `Deployments.evaluationWorker()` and `DeploymentEvaluationWorkerOptions`; `RunEvaluationOptions`
+  gains `traces`, `operations`, `pollMs`, and the worker's options, and `RunEvaluationReport` gains
+  `failed`;
+- in `nexus-ai-pro/evaluate`: `traceEvidence()`, `mediaParts()`, `EvaluationTrace`, `MediaPart`,
+  `TraceToolCall`, `TraceRetrieval`, `TraceDocument`, `TraceModelCall`, `TraceMedia`, and
+  `EvaluationContext.trace`;
+- `OnlineEvaluationWorkerOptions.withTrace`, `onScored`, and `onError`, and `OnlineScored`;
+- `traceRetriever()`, `RetrieverLike`, and `Tracer.within()`;
+- `nexus-ai-pro/evaluate/automation`: `automationRules()`, `deploymentScores()`, `traceScores()`,
+  `memoryScores()`, `openReview()`, `sendWebhook()`, `markUnhealthy()`, `AutomationRule`,
+  `AutomationRules`, `AutomationRulesOptions`, `AutomationEvent`, `AutomationAction`,
+  `AutomationStatus`, `AutomationWebhookOptions`, `ScoreSample`, `ScoreSource`, `MemoryScoreSource`,
+  `ReviewOpener`, and `RuleStatistic`;
+- `Deployments.markUnhealthy()`, `markHealthy()`, and `scores()`, `RunScore`, the `health`
+  `DeploymentChange`, `DeploymentRecord.unhealthy`, `RevisionHealthMark`,
+  `DeploymentChangeRecord.revision` and `healthy`, and `CanaryDecision.unhealthy`;
+- `nexus-ai-pro/evaluate/media`: `mediaChecks()`, `mediaJudge()`, `mediaOf()`, `resolveMedia()`,
+  `MediaCheckOptions`, `MediaJudgeOptions`, `MediaResolveOptions`, `MediaSelection`, `MediaRange`,
+  `ResolvedMedia`, and `MediaSubjectKind`;
+- `nexus-ai-pro/media/catalog`: `mediaCatalog()`, `catalogedAssetStore()`, `MediaCatalog`,
+  `MediaCatalogOptions`, `MediaCatalogEntry`, `MediaCatalogQuery`, `MediaCatalogPage`,
+  `MediaCatalogStats`, `MediaCatalogRecordOptions`, `CatalogScore`, `CatalogMediaKind`, and
+  `CatalogedAssetStoreOptions`;
+- `nexus-ai-pro/tracing/otlp-grpc`: `OtlpGrpcTraceExporter`, `OtlpGrpcTraceExporterOptions`, and
+  `OtlpGrpcError`;
+- in `nexus-ai-pro/tracing/otlp`: `OtlpTraceExporterOptions.protocol`, `encodeOtlpTraceRequest()`,
+  `decodeOtlpTraceResponse()`, `OtlpExportRequest`, `OtlpKeyValue`, `OtlpAttributeValue`,
+  `OtlpPartialSuccess`, and `OtlpPartialSuccessError`.
+
+Changed: `DeploymentChange` gains a `health` action, so a `switch` over its actions that must be
+exhaustive needs a case for it.
+
+Changed: `Deployments.evaluate()` runs through the online-evaluation worker, scoring up to 4 runs at
+once rather than one, so evaluators that are not safe to call concurrently need `concurrency: 1`.
+
 ## 2.11 stage (2.11.0)
 
 Everything 2.11 adds is stable from the start and additive, unless this section says otherwise.

package/CHANGELOG.md

@@ -4,6 +4,188 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [2.13.0] - 2026-10-19
+
+Regions and residency, and credentials per tenant.
+
+### Added
+
+- **Agent servers across regions.** `regionalGateway()`, on the new `nexus-ai-pro/server/regions`, sits
+  in front of one agent server per region. A thread or run goes to the region its id names and waits
+  there while that region is down, since its data is there and nowhere else; new work goes to the
+  tenant's home region and fails over only to another region the tenant may reach; a tenant that may
+  reach no region served is refused with a 403 naming the rule. Regions are checked through their
+  `/health`, marked down by failed requests, and taken out by an operator with `markDown()`. An agent
+  server given `region` stamps it on its threads, runs, ids, replica report, and health, and, given
+  `residency`, refuses tenants that may not reach it.
+- **Residency: each tenant's requests and data kept in its regions.** `residency()`, on the new
+  `nexus-ai-pro/residency`, says which regions each tenant may reach (a map, or a function over the
+  tenant's record), which providers run in each region, and each provider's twin inside a region.
+  `LifecycleConfig.residency` applies it to every family: a completion or stream, its fallbacks and
+  its context summary included, is sent to the provider routing chose when that runs inside the
+  tenant's regions and to its twin there when not; an embedding never falls back outside them; and a
+  provider anything would reach outside them is refused with a `ResidencyError` naming the rule,
+  before anything is sent. `regionalTraceStore()` and `regionalAssetStore()` write each tenant's
+  traces, every run of a trace together, and assets in its home region, and read them only from its
+  regions.
+- **Credentials per tenant.** `LifecycleConfig.credentials` resolves each provider's key per tenant,
+  when a call is made, in every family that runs through the lifecycle: completions, streams,
+  embeddings, and images send it in place of the configured key. `tenantCredentials()`, on the new
+  `nexus-ai-pro/credentials`, reads it from a secret source, cached for `cacheMs`, shared by
+  concurrent calls, carried by the last key while the source is down for up to `staleMs`, and
+  refusing a tenant with no key unless told to fall back. Sources for AWS Secrets Manager, Google
+  Secret Manager, Azure Key Vault, HashiCorp Vault, the environment, and memory, each over the
+  vendor's own client with no dependency.
+- A completion or stream whose provider refuses its credential is resolved again and made once more
+  on the new key; in any other family, the next operation reads the key again. Every key and header
+  value an operation resolves is removed from the errors it raises, before an audit event, a log,
+  a trace, or the health report sees them.
+- `ProviderCallContext.residency`, the operation's `CallResidency`; `ImageRequestBase.tenantId`; and
+  an agent server's trace root records its run's tenant as `metadata.nexus.tenant`.
+- `ProviderCallContext.tenantId` and `credential()`; `OperationTicket.refreshCredential()` and
+  `scrub()`; `OpenAIProviderConfig.credentialHeader`; `EmbeddingRequest.tenantId`; and
+  `EmbeddingProviderError.attempts`, each attempt's error.
+
+### Proof
+
+- `tests/regions.test.ts`: the proof that closes 2.13. Three regions, each an agent server with its own
+  state, traces, and provider, behind a gateway, with every tenant's key from one secret manager. A run
+  of four model calls has its key rotated and revoked after the second; the third and fourth are
+  answered on the new key. Then the EU tenant's home region goes down: its new work fails over to the
+  other EU region, its thread waits for its own, a tenant held to that region waits too, and nothing
+  of either reaches the US: no thread, run, trace, or provider call. Without region ids, the server's
+  check, a thread's region kept, new work held to the tenant's regions, failover, checks, the
+  operator's mark, or the gateway's own refusal, a test fails.
+- `tests/residency.test.ts`: against a provider server per region, a tenant pinned to the EU is
+  answered by the EU twin of the provider routing chose, completions and streams alike, and nothing
+  of it reaches the US; a provider with no twin there is refused with the rule, nothing sent; an
+  embedding whose EU provider fails never falls back to the US; an image provider outside the regions
+  is refused; and each tenant's traces and assets land in its home region and are read only from its
+  regions. Without placement, twins, the decision, the routed check, the fallback filter, the image
+  check, traces kept together, the tenant view, the asset home, or the rule checks, a test fails.
+- `tests/tenant-credentials.test.ts`: against a provider of the test's own that takes only live keys,
+  each tenant's calls carry its own key through completions, streams, and embeddings; in a run of five
+  calls whose key is rotated and revoked after the second, the third is answered on the new key and
+  every call after uses it; a key the provider quotes in its refusal reaches no error, stack, cause,
+  audit event, log, trace, or health report. Without the call's credential, the refresh, its limit,
+  either scrub, an adapter's use of the key, the cache's expiry, its single read, its stale key, or
+  its refusal, a test fails.
+
+## [2.12.0] - 2026-10-18
+
+Observability that loses nothing, and media you can evaluate.
+
+### Added
+
+- **Spans that a collector being down cannot lose.** `spoolingExporter()`, on the new
+  `nexus-ai-pro/tracing/spool`, keeps each batch the collector refuses in an operation store (the
+  memory, Redis, Postgres, or SQLite one), retries it with backoff, and dead-letters it only once it is
+  too old or has failed too often, under a bound on what waits. Every replica on the store drains the
+  same spool, each batch once. `writeAhead` spools each batch before sending it. `stats()` reports
+  what was exported, retried, spooled, dropped, and dead-lettered, and the age of the oldest batch
+  waiting.
+- `OtlpTraceExporter.send()`: sends runs as one batch now, and rejects when the collector refuses
+  them.
+- **A deployment judged by what its runs did.** An agent server given a `tracer` makes each run the
+  root of a trace, with its input and output, and everything the assistant traces through the same
+  tracer beneath it; the run records the trace as `RunRecord.traceId`. `Deployments.evaluate()`
+  given those `traces` hands each evaluator the run's input and an `EvaluationTrace` of its tool
+  calls, retrievals, model calls, and media, read by the new `traceEvidence()`, which `mediaParts()`
+  backs. An assistant that traces itself is found by the run id it records.
+- `Deployments.evaluationWorker()`: the online-evaluation worker over a deployment's runs, to start
+  on every replica, scoring each run once between them as it finishes.
+- `onlineEvaluationWorker()` takes `withTrace`, which gives evaluators `context.trace`, and
+  `onScored`, which hears each run's scores once they are written, with `onError` for its failures.
+- `traceRetriever()`: every retrieval as a `retriever` run, with the query and the documents.
+- **Media you can evaluate.** On the new `nexus-ai-pro/evaluate/media`, `mediaChecks()` reads the
+  media an evaluation is about, from bytes, base64, a data URL, a URL it is allowed to fetch, a path,
+  or an asset store, and scores each check asked: the count, kind, MIME type, size, dimensions,
+  aspect ratio, and a WAV's duration. `mediaJudge()` shows them to a vision or audio model beside a
+  rubric and the brief, a stored asset sent as its bytes, and records the judge in an experiment.
+  `mediaOf()` finds the media in an output, or in what the run's trace says it produced, and
+  `resolveMedia()` reads one.
+- **Automation rules.** On the new `nexus-ai-pro/evaluate/automation`, `automationRules()` judges
+  a score over each revision's most recent runs, and when it falls below a threshold over enough
+  samples (optionally by the upper bound of a confidence interval), acts once, across every replica
+  sharing its `state`: `openReview()` sends the worst runs to people, `sendWebhook()` posts a signed
+  event, and `markUnhealthy()` marks the revision unhealthy, which rolls a canary back. Rules read
+  scores a deployment's run records hold (`deploymentScores()`), a trace store's
+  (`traceScores()`), or this process's (`memoryScores()`), and `observe()` judges them as an online
+  evaluation writes them. A rule fires once per fall, recovers, and can repeat.
+- **A revision's health.** `Deployments.markUnhealthy()` and `markHealthy()`: a `health` change in
+  the deployment's history, kept in `DeploymentRecord.unhealthy`. A canary marked unhealthy is rolled
+  back at once, and a canary guard rolls back any canary marked since it began, its `CanaryDecision`
+  carrying the mark. The studio shows each revision's health and marks one from the deployment view.
+  `Deployments.scores()` reads the scores of an assistant's runs.
+- **A catalog of every stored asset.** `mediaCatalog()`, on the new `nexus-ai-pro/media/catalog`,
+  keeps one record of each stored asset in any long-term store (Postgres, SQLite, Redis, or memory):
+  what it is, who made it, for which run, its tags, and its latest score per key, written by
+  compare-and-set. `catalogedAssetStore()` wraps any asset store so every asset stored through it is
+  recorded and every one deleted is forgotten, an entry expiring with its asset. `list()` finds
+  entries by kind, type, provider, model, run, store, tag, time, and score, a page at a time;
+  `attach()` scores the assets a run's output refers to; `stats()`, `purgeExpired()`, and
+  `reconcile()` keep account.
+- **OTLP over gRPC.** `OtlpGrpcTraceExporter`, on the new `nexus-ai-pro/tracing/otlp-grpc`, sends
+  spans to a collector's port 4317 with no gRPC library: the trace service's `Export` call framed by
+  hand over Node's HTTP/2 client, gzip-compressed, with metadata, a deadline, TLS options, one
+  connection kept and reopened, and a failed call rejecting with an `OtlpGrpcError` so a spool keeps
+  the batch. `OtlpTraceExporter` takes `protocol: 'http/protobuf'`, and both read a collector's
+  partial success into an `OtlpPartialSuccessError` for `onError`. `encodeOtlpTraceRequest()` and
+  `decodeOtlpTraceResponse()` are OTLP's protobuf encoding, written by hand, on any runtime.
+- `Tracer.within()`: runs a function with a run started by `startRun()` in scope.
+
+### Changed
+
+- `Deployments.evaluate()` runs through the online-evaluation worker, drained once, instead of beside
+  it. Replicas given one `operations` store score each run once between them, even calling at the
+  same moment, where before two calls could score a run twice; it takes the worker's budget, review
+  queue, and concurrency (4 at a time by default, one before), and `RunEvaluationReport` counts
+  `failed` evaluations. Which runs a `sampleRate` picks is unchanged.
+
+### Proof
+
+- `tests/trace-spool.test.ts`: a collector stopped for an hour while a span finishes every second,
+  and the process restarted halfway: every one of the 3,600 spans reaches it, once, on the memory,
+  SQLite, Postgres, and Redis operation stores. The bound, dead letters, two replicas on one spool,
+  and write-ahead through a process that dies mid-send. Without the spool, its bound, its claim, its
+  dead letters, write-ahead, or taking over a lapsed lease, a test fails.
+- `tests/media-evaluation.test.ts`: checks that read PNG, GIF, WAV, and PDF media from a store,
+  base64, a data URL, and a fetch only when allowed; a judge model shown a stored asset as its bytes
+  beside the rubric and the brief; media found through the trace; and an experiment over a target
+  that draws. Without the sniffing, the store, the wildcard, the tolerance, the duration, the size
+  limit, the count, the media shown, the brief, or the trace fallback, a test fails.
+- `tests/automation-rules.test.ts`: the proof that a canary is rolled back on a media evaluator's
+  score. A designer assistant's canary draws its banners square; two replicas score 80 runs with a
+  media judge shown each banner, and a rule on the judge's score marks the canary unhealthy, rolling
+  it back, opening review items for the five worst banners, and sending one signed webhook, once
+  between the replicas. The live revision is untouched, new runs go to it, and the media catalog
+  finds each of the canary's banners by its score. Then minimum samples, scopes, windows, the
+  confidence bound, the pass rate, recovery, repeats, filters, sources, failing actions, and a guard
+  rolling back a canary a rule only marked. Without the rollback, the guard's mark, the stale-mark
+  check, the claim, the minimum, the window, recovery, the bound, counting each run once, the
+  revision scope, the signature, or the repeat, a test fails.
+- `tests/media-catalog.test.ts`: on the memory, SQLite, Postgres, and Redis stores, forty assets of
+  five kinds, two providers, and two tenants, some deleted and some expiring: every page lists each
+  remaining asset once, every filter finds exactly its assets, expired ones are never read and are
+  purged with their assets, and ten evaluators scoring one asset at once keep all ten scores. An
+  asset deleted around the catalog is reconciled away, and one the catalog cannot record is not left
+  stored. Without the forgetting, the expiry, the purge, compare-and-set, the cursor, the filters,
+  the compensation, or the reconciliation, a test fails.
+- `tests/otlp-grpc.test.ts`: the OpenTelemetry Collector itself, in a container, takes spans over
+  gRPC and sends every one on as it came, ids, times, kinds, attributes, and status. Against a gRPC
+  server of the test's, with a protobuf reader written apart from the encoder: compression and
+  metadata, a refusal, a deadline, a partial success, a collector that is down, one that restarts
+  under a live connection, and a spool that keeps what is refused. Without the compressed flag, the
+  status, the deadline, the metadata, the parent span, the status message, typed values, the
+  reconnection, or the retry on a dropped connection, a test fails.
+- `tests/deployment-trace-evaluation.test.ts`: a traced server whose assistant retrieves a policy,
+  calls a tool, and asks a model about a photo; its evaluators see the input, the tool call, the
+  documents, the model call, and the photo once. Two replicas evaluating at once over one operation
+  store score each of 40 runs once, where queues of their own score runs twice. A worker scores runs
+  as they finish and hears each score once, and an assistant tracing itself is found. Without the
+  trace, the input, the run scope, the shared queue, the fallback, the media deduplication, or the
+  trace written before the run settles, a test fails.
+
 ## [2.11.0] - 2026-10-18
 
 Workspaces, and telephony in full.
@@ -2764,7 +2946,9 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.11.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.13.0...HEAD
+[2.13.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.12.0...v2.13.0
+[2.12.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.11.0...v2.12.0
 [2.11.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.10.0...v2.11.0
 [2.10.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.9.0...v2.10.0
 [2.9.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v2.8.0...v2.9.0

package/README.md

@@ -167,6 +167,7 @@ from the doc comments. The guides live in the repository, so these links go to G
 | [Agent server](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/server.md) | Self-hosted HTTP server for assistants: threads, durable runs, resumable streams, cron, a worker queue, and metrics |
 | [Deployments](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/deployments.md) | Revisions and canaries with automatic rollback, autoscaling on Kubernetes and Helm, and per-tenant limits |
 | [Tenancy](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/tenancy.md) | One tenant model: a tenant's view of every store, tenant graph runs, and the isolation suite |
+| [Regions and credentials](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/regions.md) | Agent servers across regions behind a gateway, residency rules that keep each tenant's requests, traces, and assets in its regions, and each tenant's provider keys from a secret manager |
 | [Workflows](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/workflows.md) | Ready-made chains and domain workflows |
 | [Command line](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/cli.md) | The `nexus` command |
 | [Packaging and install weight](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/docs/packaging.md) | Every entry point and what it costs |
@@ -303,14 +304,22 @@ Shipped so far:
 - 2.11 workspaces, and telephony in full: an agent's files apart from where its code runs, in memory,
   on disk, in any long-term store, read-only, or a composite of several; and a second phone provider,
   Telnyx, beside Twilio, both answering, transferring, conferencing, recording under a retention
-  policy, and sending and receiving messages.
+  policy, and sending and receiving messages;
+- 2.12 observability that loses nothing, and media you can evaluate: spans spooled through a collector
+  outage, OTLP over gRPC, deployment evaluation that reads each run's whole trace, evaluators that
+  read and judge images, documents, audio, and video, a catalog of every stored asset, and automation
+  rules that roll a canary back when a score falls;
+- 2.13 regions and residency, and credentials per tenant: agent servers across regions behind a
+  gateway that fails over only within a tenant's regions, residency rules that keep each tenant's
+  provider calls, traces, and assets in its regions, and each tenant's provider keys from a secret
+  manager, rotated without a restart and kept out of every error.
 
 Upgrading from 1.x: `npx nexus migrate src --write` moves your imports, and
 [MIGRATING.md](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/MIGRATING.md) covers the rest.
-Next, 2.12 makes observability lose nothing, with spans spooled when the collector is down and
-deployment evaluation that reads the whole trace, and makes media something you can evaluate.
-Pinecone, images, the Gemini embedding batch tier, media understanding, video, Gemini Live, SIP, and
-both phone providers graduate once the live workflow passes against each. The
+Next, 2.14 makes policy a reviewed file, checked by the command line before it ships, brings
+per-tenant keys to every family, and lets an evaluation run an agent over emulated tools with nothing
+sent. Pinecone, images, the Gemini embedding batch tier, media understanding, video, Gemini Live, SIP,
+and both phone providers graduate once the live workflow passes against each. The
 [roadmap](https://github.com/mkhitar-abrahamyan/nexus-ai/blob/main/ROADMAP.md) lists what comes after, in the order
 applications ask for it.
 

package/dist/core/lifecycle.d.ts

@@ -3,6 +3,7 @@ import type { ProviderHealthOutcome, LifecycleConfig, LifecycleStage, OperationD
 import type { AuditLogger } from '../ops/audit-logger.js';
 import type { MetricsCollector } from '../ops/metrics.js';
 import { type RateLimiter } from '../ops/rate-limiter.js';
+import type { CallResidency, ResidencyPolicy } from '../types/residency.js';
 /** Every lifecycle stage, in the order an operation passes through them. */
 export declare const LIFECYCLE_STAGES: readonly LifecycleStage[];
 /** Raised when `lifecycle.authorize` refuses an operation, before any provider is called. */
@@ -114,3 +115,5 @@ export declare class OperationLifecycle implements OperationLifecycleLike {
     private checkRateLimit;
     private authorize;
 }
+/** An operation's residency, as the families that call providers check it. */
+export declare function callResidency(policy: ResidencyPolicy, decision: CallResidency['decision']): CallResidency;

package/dist/core/lifecycle.js

@@ -1,5 +1,6 @@
 import { NexusRateLimitError } from '../ops/rate-limiter.js';
 import { generateRequestId } from '../utils/ids.js';
+import { scrubSecrets } from '../utils/scrub.js';
 import { linkSignals } from '../utils/signals.js';
 export const LIFECYCLE_STAGES = [
     'validate',
@@ -51,12 +52,24 @@ export class OperationLifecycle {
             ...(deadline !== undefined ? { deadline } : {}),
             ...(options.traceContext ? { traceContext: options.traceContext } : {}),
             ...(options.idempotencyKey ? { idempotencyKey: options.idempotencyKey } : {}),
+            ...(descriptor.tenantId !== undefined ? { tenantId: descriptor.tenantId } : {}),
         };
-        const ticket = new Ticket(this.runtime, descriptor, context, startedAt, call.dispose);
+        const credentials = this.runtime.config?.credentials
+            ? new CallCredentials(this.runtime.config.credentials, descriptor)
+            : undefined;
+        if (credentials)
+            context.credential = (provider) => credentials.resolve(provider);
+        const ticket = new Ticket(this.runtime, descriptor, context, startedAt, call.dispose, credentials);
         try {
             await this.runtime.metrics?.recordRequest(labelsOf(descriptor));
             await this.checkRateLimit(descriptor);
             await this.authorize(descriptor);
+            const policy = this.runtime.config?.residency;
+            if (policy) {
+                context.residency = callResidency(policy, await policy.regionsFor(descriptor.tenantId));
+                if (descriptor.provider)
+                    context.residency.check(descriptor.provider);
+            }
         }
         catch (error) {
             await ticket.fail(error);
@@ -73,8 +86,10 @@ export class OperationLifecycle {
             await plan.inputPolicy?.(ticket.context);
             await plan.resolveAssets?.(ticket.context);
             const routed = await plan.route?.();
-            if (routed?.provider)
+            if (routed?.provider) {
+                ticket.context.residency?.check(routed.provider);
                 ticket.descriptor.provider = routed.provider;
+            }
             if (routed?.model)
                 ticket.descriptor.model = routed.model;
             if (this.runtime.config?.budget)
@@ -107,7 +122,9 @@ export class OperationLifecycle {
             !this.runtime.rateLimit?.enabled &&
             !config?.authorize &&
             !config?.budget &&
-            !config?.hooks);
+            !config?.hooks &&
+            !config?.credentials &&
+            !config?.residency);
     }
     async checkRateLimit(descriptor) {
         const { rateLimiter, rateLimit } = this.runtime;
@@ -142,20 +159,98 @@ export class OperationLifecycle {
             throw new OperationDeniedError(descriptor);
     }
 }
+export function callResidency(policy, decision) {
+    return {
+        decision,
+        allows: (provider) => policy.allows(policy.regionOf(provider), decision),
+        place: (provider) => policy.place(provider, decision),
+        check: (provider) => {
+            const region = policy.regionOf(provider);
+            policy.check({ kind: 'provider', name: provider, ...(region ? { region } : {}) }, decision);
+        },
+    };
+}
+class CallCredentials {
+    resolver;
+    descriptor;
+    resolved = new Map();
+    secrets = new Set();
+    constructor(resolver, descriptor) {
+        this.resolver = resolver;
+        this.descriptor = descriptor;
+    }
+    resolve(provider) {
+        let found = this.resolved.get(provider);
+        if (found)
+            return found;
+        found = Promise.resolve(this.resolver.resolve({
+            provider,
+            ...(this.descriptor.tenantId !== undefined ? { tenantId: this.descriptor.tenantId } : {}),
+            family: this.descriptor.family,
+            operation: this.descriptor.operation,
+        })).then((credential) => {
+            if (credential?.apiKey)
+                this.secrets.add(credential.apiKey);
+            for (const value of Object.values(credential?.headers ?? {}))
+                this.secrets.add(value);
+            return credential;
+        });
+        this.resolved.set(provider, found);
+        found.catch(() => {
+            if (this.resolved.get(provider) === found)
+                this.resolved.delete(provider);
+        });
+        return found;
+    }
+    async refresh(provider) {
+        this.resolved.delete(provider);
+        await this.resolver.invalidate?.({
+            provider,
+            ...(this.descriptor.tenantId !== undefined ? { tenantId: this.descriptor.tenantId } : {}),
+        });
+    }
+    async refreshAll() {
+        for (const provider of [...this.resolved.keys()])
+            await this.refresh(provider);
+    }
+}
+function refusedCredential(error, depth = 0) {
+    if (!error || typeof error !== 'object' || depth > 4)
+        return false;
+    const failed = error;
+    if (failed.category === 'auth' || failed.status === 401 || failed.statusCode === 401)
+        return true;
+    for (const list of [failed.attempts, failed.errors]) {
+        if (Array.isArray(list) && list.some((item) => refusedCredential(item, depth + 1)))
+            return true;
+    }
+    return refusedCredential(failed.cause, depth + 1);
+}
 class Ticket {
     runtime;
     descriptor;
     context;
     startedAt;
     release;
+    credentials;
     reservation;
     finished = false;
-    constructor(runtime, descriptor, context, startedAt, release = () => undefined) {
+    constructor(runtime, descriptor, context, startedAt, release = () => undefined, credentials) {
         this.runtime = runtime;
         this.descriptor = descriptor;
         this.context = context;
         this.startedAt = startedAt;
         this.release = release;
+        this.credentials = credentials;
+    }
+    async refreshCredential(provider) {
+        if (!this.credentials)
+            return false;
+        await this.credentials.refresh(provider);
+        return true;
+    }
+    scrub(error) {
+        return this.credentials ? scrubSecrets(error, this.credentials.secrets) : error;
     }
     async reserve(estimate) {
         const budget = this.runtime.config?.budget;
@@ -193,9 +288,13 @@ class Ticket {
         });
     }
     async fail(error) {
+        this.scrub(error);
         if (this.finished)
             return;
         this.finished = true;
+        if (this.credentials && refusedCredential(error)) {
+            await this.credentials.refreshAll().catch(() => undefined);
+        }
         const budget = this.runtime.config?.budget;
         if (budget && this.reservation)
             await budget.release(this.reservation);
@@ -239,7 +338,11 @@ function outcomeOf(error, signal) {
         error instanceof BudgetExceededError ||
         error instanceof NexusRateLimitError ||
         (error instanceof Error &&
-            (error.name === 'CostBudgetError' || error.name === 'TenantLimitError' || error.name === 'NexusSecurityError'))) {
+            (error.name === 'CostBudgetError' ||
+                error.name === 'TenantLimitError' ||
+                error.name === 'NexusSecurityError' ||
+                error.name === 'ResidencyError' ||
+                error.name === 'CredentialError'))) {
         return 'denied';
     }
     if (signal.aborted ||

package/dist/core/nexus.d.ts

@@ -289,7 +289,17 @@ export declare class NexusAI {
     /** A completion as the lifecycle sees it. */
     private describe;
     private startOptions;
-    /** How failover runs the provider calls of one operation, with the context each attempt hands on. */
+    /**
+     * A route kept inside the regions the operation may reach: each attempt sent to its provider when
+     * that runs inside them, to the provider's twin inside them when there is one, and dropped when
+     * neither. With nothing left, the operation is refused with a `ResidencyError` naming the rule.
+     */
+    private placeRoute;
+    /**
+     * How failover runs the provider calls of one operation, with the context each attempt hands on,
+     * and, given the operation's ticket, how a refused credential is resolved again and every error
+     * cleaned of the credentials it resolved.
+     */
     private executionOptions;
     /**
      * Times a lifecycle stage into the trace when the lifecycle has work to do, so a client with

package/dist/core/nexus.js

@@ -1,4 +1,5 @@
 import { configuredProviders } from '../providers/lazy.js';
+import { ResidencyError } from '../residency/errors.js';
 import { Router, FailoverExecutor } from '../router/index.js';
 import { Logger } from '../utils/logger.js';
 import { SecurityPipeline } from '../security/index.js';
@@ -23,7 +24,7 @@ import { ProviderHealthMonitor } from '../ops/health.js';
 import { CircuitBreaker } from '../ops/circuit-breaker.js';
 import { runBatch } from '../jobs/batch.js';
 import { JobQueue } from '../jobs/queue.js';
-import { OperationLifecycle } from './lifecycle.js';
+import { callResidency, OperationLifecycle } from './lifecycle.js';
 export class NexusAI {
     images;
     lifecycle;
@@ -137,7 +138,7 @@ export class NexusAI {
             }
             context = await this.pipeline.runHook('afterSecurity', context);
             const decision = await this.pipeline.trace(context, 'routing', () => {
-                return this.router.route(context.request, this.config, this.providers, this.health.snapshot(), this.circuitBreaker.openProviders());
+                return this.placeRoute(this.router.route(context.request, this.config, this.providers, this.health.snapshot(), this.circuitBreaker.openProviders()), ticket.context.residency);
             });
             context.route = decision;
             this.logger.info('route decision', decision);
@@ -197,7 +198,7 @@ export class NexusAI {
             }
             context = await this.pipeline.runHook('beforeProvider', context);
             const response = await this.pipeline.trace(context, 'providerCall', () => {
-                return this.failover.complete(context.request, decision, this.providers, this.executionOptions(ticket.context));
+                return this.failover.complete(context.request, decision, this.providers, this.executionOptions(ticket.context, ticket));
             });
             context.response = response;
             context = await this.pipeline.runHook('afterProvider', context);
@@ -638,8 +639,43 @@ export class NexusAI {
             ...(this.config.auditLog?.includeInput ? { audit: { request } } : {}),
         };
     }
-    executionOptions(context) {
-        const { requestId, traceContext, idempotencyKey } = context;
+    placeRoute(decision, residency) {
+        if (!residency)
+            return decision;
+        const attempts = [
+            { providerName: decision.providerName, model: decision.model, ...decision.primary },
+            ...decision.fallbacks,
+        ];
+        const placed = [];
+        const seen = new Set();
+        for (const attempt of attempts) {
+            const where = residency.place(attempt.providerName);
+            if (!where || !this.providers.has(where.provider))
+                continue;
+            const key = `${where.provider}\u0000${attempt.model}`;
+            if (seen.has(key))
+                continue;
+            seen.add(key);
+            placed.push({ ...attempt, providerName: where.provider });
+        }
+        const [first, ...rest] = placed;
+        if (!first) {
+            residency.check(decision.providerName);
+            throw new ResidencyError({ kind: 'provider', name: decision.providerName }, residency.decision, 'its twin is not registered');
+        }
+        const { providerName, model, ...limits } = first;
+        return {
+            providerName,
+            model,
+            reason: providerName === decision.providerName
+                ? decision.reason
+                : `${decision.reason}; sent to ${providerName} inside ${residency.decision.rule}`,
+            fallbacks: rest,
+            ...(Object.keys(limits).length > 0 ? { primary: limits } : {}),
+        };
+    }
+    executionOptions(context, ticket) {
+        const { requestId, traceContext, idempotencyKey, tenantId, credential } = context;
         return {
             timeoutMs: this.config.timeout,
             retry: this.config.retry,
@@ -648,7 +684,13 @@ export class NexusAI {
                 requestId,
                 ...(traceContext ? { traceContext } : {}),
                 ...(idempotencyKey ? { idempotencyKey } : {}),
+                ...(tenantId !== undefined ? { tenantId } : {}),
+                ...(credential ? { credential } : {}),
             },
+            ...(ticket?.refreshCredential && credential
+                ? { refreshCredential: (provider) => ticket.refreshCredential?.(provider) ?? Promise.resolve(false) }
+                : {}),
+            ...(ticket?.scrub && credential ? { scrub: (error) => ticket.scrub?.(error) ?? error } : {}),
         };
     }
     lifecycleStep(context, name, fn) {
@@ -721,7 +763,7 @@ export class NexusAI {
             : { ok: true, value: optimizationResult.value, findings: [], guardrailsApplied: [] };
         if (this.isSecurityEnabled())
             this.security.assertSafe(securityResult);
-        const decision = this.router.route(securityResult.value, this.config, this.providers, this.health.snapshot(), this.circuitBreaker.openProviders());
+        const decision = this.placeRoute(this.router.route(securityResult.value, this.config, this.providers, this.health.snapshot(), this.circuitBreaker.openProviders()), ticket.context.residency);
         this.logger.info('stream route decision', {
             ...decision,
             tokensSaved: optimizationResult.usage.savedTokens,
@@ -740,7 +782,7 @@ export class NexusAI {
             if (shared)
                 await ticket.reserve(estimate.totalCost);
         }
-        const stream = this.failover.stream(routedRequest, decision, this.providers, this.executionOptions(ticket.context));
+        const stream = this.failover.stream(routedRequest, decision, this.providers, this.executionOptions(ticket.context, ticket));
         return {
             stream: this.isSecurityEnabled() ? protectStreamOutput(stream, this.security, routedRequest.signal) : stream,
             request: routedRequest,
@@ -785,7 +827,11 @@ export class NexusAI {
                 contextWindowSummary: true,
             },
         };
-        const decision = this.router.route(summaryRequest, this.config, this.providers, this.health.snapshot());
+        const policy = this.config.lifecycle?.residency;
+        const routed = this.router.route(summaryRequest, this.config, this.providers, this.health.snapshot());
+        const decision = policy
+            ? this.placeRoute(routed, callResidency(policy, await policy.regionsFor(summaryRequest.tenantId)))
+            : routed;
         const routedRequest = { ...summaryRequest, model: resolveModel(decision.model, this.config).model };
         const response = await this.lifecycle.run({ ...this.describe(routedRequest, 'summarize'), provider: decision.providerName }, {
             execute: (callContext) => this.failover.complete(routedRequest, decision, this.providers, this.executionOptions(callContext)),