@arnilo/prism 0.3.2 → 0.5.0

package/docs/providers/vertex.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-provider-vertex` registers a Vertex AI OpenAPI-compatible Chat Completions provider authenticated with host ADC / workload identity tokens. It is intentionally separate from `@arnilo/prism-provider-google` (consumer Gemini API keys).
+`@arnilo/prism-providers/vertex` registers a Vertex AI OpenAPI-compatible Chat Completions provider authenticated with host ADC / workload identity tokens. It is intentionally separate from `@arnilo/prism-providers/google` (consumer Gemini API keys).
 
 ## When to use it
 
@@ -11,7 +11,7 @@ Use it for GCP enterprise Vertex deployments with Application Default Credential
 ## Inputs / request
 
 ```ts
-import { createVertexProviderPackage } from "@arnilo/prism-provider-vertex";
+import { createVertexProviderPackage } from "@arnilo/prism-providers/vertex";
 
 createVertexProviderPackage({
   projectId: "my-gcp-project",
@@ -53,20 +53,36 @@ const provider = createVertexProvider({
 
 ## Extension and configuration notes
 
-`@arnilo/prism-provider-google` remains API-key Gemini (`generativelanguage.googleapis.com`) and must not register Vertex OAuth/ADC. Load this package explicitly for Vertex.
+`@arnilo/prism-providers/google` remains API-key Gemini (`generativelanguage.googleapis.com`) and must not register Vertex OAuth/ADC. Load this package explicitly for Vertex.
 
 ## Security and performance notes
 
 - No Google Cloud SDK dependency in the package.
 - Custom/private endpoint hosts are preserved.
 - Tokens redacted from errors; no import-time credential prefetch — the credential is resolved exactly once per request (a rotating `CredentialValueSource` is never consumed twice; the same resolved token drives the wrapper check and the inner auth header).
-- Conformance-proven (Task 6): package `setup()` performs zero fetch and zero credential resolution; an already-aborted signal fails fast; a truncated SSE stream (no `data: [DONE]`) ends in an `error` event; native Vertex cached-content lifecycle is intentionally unsupported on the OpenAI-compatible route — no cache wire fields are emitted even when the request carries Prism cache hints (use `@arnilo/prism-provider-google`'s `extra.cachedContent` on that package, or manage cache resources host-side).
+- Conformance-proven (Task 6): package `setup()` performs zero fetch and zero credential resolution; an already-aborted signal fails fast; a truncated SSE stream (no `data: [DONE]`) ends in an `error` event; native Vertex cached-content lifecycle is intentionally unsupported on the OpenAI-compatible route — no cache wire fields are emitted even when the request carries Prism cache hints (use `@arnilo/prism-providers/google`'s `extra.cachedContent` on that package, or manage cache resources host-side).
 - Pair with model-router residency allow-lists on `location`.
 
+## Live probe
+
+Opt-in smoke against Vertex's OpenAI-compatible endpoint. The suite takes a pre-minted bearer token (host ADC/workload identity stays host-owned):
+
+```bash
+PRISM_LIVE_PROVIDER_TESTS=1 GOOGLE_VERTEX_PROJECT=my-project \
+  PRISM_VERTEX_ACCESS_TOKEN=$(gcloud auth print-access-token) \
+  node --test packages/prism-providers/dist/vertex/__tests__/live.test.js
+```
+
+`GOOGLE_VERTEX_LOCATION` (default `us-central1`) and `PRISM_LIVE_VERTEX_MODEL` (default `gemini-2.5-flash`) are optional. Missing project or token → skip.
+
+## Thinking and reasoning
+
+Vertex AI OpenAI-compat chat uses the same sanitized thinking-compat forwarding as Azure: `reasoning_effort` (aliases `effort`/`reasoningEffort`) or a `reasoning` object, snapped to the model's declared levels (Gemini → `low/medium/high`). `reasoning_effort` and `extra_body.google.thinking_config` are mutually exclusive upstream — send one. See [Thinking and reasoning](../thinking-and-reasoning.md).
+
 ## Related APIs
 
 - [Google Gemini (consumer)](google.md)
 - [OpenAI-compatible provider](openai-compatible.md)
 - [Provider packages](../provider-packages.md)
 - [Model routing](../model-routing.md)
-- Package README: [`@arnilo/prism-provider-vertex`](../../packages/provider-vertex/README.md)
+- Package README: [`@arnilo/prism-providers` family README](../../packages/prism-providers/README.md)

package/docs/providers/xai.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-provider-xai` provides explicit, side-effect-free setup for the
+`@arnilo/prism-providers/xai` provides explicit, side-effect-free setup for the
 xAI Grok Chat Completions API (`POST https://api.x.ai/v1/chat/completions`)
 with implicit prefix caching via a sanitized `x-grok-conv-id` header, reasoning
 replay, and host-invoked SuperGrok / X Premium OAuth.
@@ -25,7 +25,7 @@ import {
   createXaiOAuthProvider,
   createXaiProviderPackage,
   listXaiModels,
-} from "@arnilo/prism-provider-xai";
+} from "@arnilo/prism-providers/xai";
 
 createXaiProviderPackage(options: XaiProviderPackageOptions): ProviderPackage
 createXaiOAuthProvider(options?: XaiOAuthOptions): OAuthProvider
@@ -84,7 +84,7 @@ Header: `x-grok-conv-id: sess-1`.
 
 ```ts
 import { createExtensionKernel, refreshOAuthCredential } from "@arnilo/prism";
-import { createXaiOAuthProvider, createXaiProviderPackage } from "@arnilo/prism-provider-xai";
+import { createXaiOAuthProvider, createXaiProviderPackage } from "@arnilo/prism-providers/xai";
 
 const kernel = createExtensionKernel();
 await kernel.load([createXaiProviderPackage({ apiKey: "fake-xai-key" })]);
@@ -123,6 +123,10 @@ await kernel.load([
 - Bounded OAuth and API error bodies. No retry loop. No refresh timer.
 - Live API-key smoke: `PRISM_LIVE_PROVIDER_TESTS=1` + `XAI_API_KEY`. SuperGrok login is operator-only (`PRISM_LIVE_XAI_OAUTH=1`).
 
+## Thinking and reasoning
+
+Reasoning models now send `reasoning_effort` (task-065 change — previously dropped): grok-4.6 declares `low/medium/high/xhigh` (default `high`), grok-4.5 `low/medium/high`, grok-4.3 `none/low/medium/high`. Effort snaps to the declared set (nearest, ties up); `grok-build` declares nothing and passes `reasoning_effort` through verbatim. Reasoning models must still replay `reasoning_content` — Featured Completions flatten nothing. See [Thinking and reasoning](../thinking-and-reasoning.md).
+
 ## Related APIs
 
 - [Provider packages](../provider-packages.md): OAuth support matrix.

package/docs/providers/zai.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-provider-zai` provides explicit, side-effect-free setup for the Z.AI
+`@arnilo/prism-providers/zai` provides explicit, side-effect-free setup for the Z.AI
 GLM Chat Completions API (`POST /chat/completions`) with official deep-thinking,
 reasoning-effort, and tool-stream request fields.
 
@@ -25,7 +25,7 @@ import {
   createZaiProviderPackage,
   defineZaiModel,
   listZaiModels,
-} from "@arnilo/prism-provider-zai";
+} from "@arnilo/prism-providers/zai";
 
 createZaiProviderPackage(options: ZaiProviderPackageOptions): ProviderPackage
 defineZaiModel(config: ZaiModelConfig): ModelConfig
@@ -87,7 +87,7 @@ Example request body (official Chat Completions shape):
 
 ```ts
 import { createExtensionKernel } from "@arnilo/prism";
-import { createZaiProviderPackage, listZaiModels } from "@arnilo/prism-provider-zai";
+import { createZaiProviderPackage, listZaiModels } from "@arnilo/prism-providers/zai";
 
 const kernel = createExtensionKernel();
 await kernel.load([createZaiProviderPackage({ apiKey: "fake-zai-key" })]);
@@ -153,6 +153,10 @@ await session.prompt("Plan the refactor", {
 - Live tests stay opt-in behind `PRISM_LIVE_PROVIDER_TESTS=1` plus `ZAI_API_KEY`;
   default tests are network-free.
 
+## Thinking and reasoning
+
+Z.AI models are family-stamped by id. GLM-5.3/5.3-FLASH: `reasoning_effort` restricted to `low/high/max` (declared; other levels snap, e.g. `medium`→`high`), and thinking can never be disabled — `zaiThinking` forces thinking on and never emits `thinking.type: "disabled"` (upstream rejects it; live-pinned). GLM-5.2: declared `low`–`max`; `none`/`minimal` stop thinking (no effort field), `low`/`medium` snap up to `high`. GLM-4.x and older: `thinking_type` toggle only (`clear_thinking` package-local). See [Thinking and reasoning](../thinking-and-reasoning.md).
+
 ## Related APIs
 
 - [Provider packages](../provider-packages.md): `defineProviderPackage`,

package/docs/rag.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-rag` is an optional package for deterministic text/Markdown chunking (with ATX heading-stack metadata), bounded embedding/vector indexing with embedder-identity drift guards, atomic scoped source replacement/deletion with content-hash skip and generation visibility, hybrid vector+lexical retrieval with reciprocal-rank fusion (one embed / one RRF / one rerank across one or many exact scopes), focused text/Markdown/HTML/PDF parsing, bounded reranking (host seam plus a TEI REST adapter), ingestion status, attributable citations, content-trust metadata, and explicit `ContextProvider` injection. It reuses `Embedder` and `VectorStore` from `@arnilo/prism-memory`; Prism core input assembly is unchanged.
+The `@arnilo/prism-memory/rag` subpath is an optional surface for deterministic text/Markdown chunking (with ATX heading-stack metadata), bounded embedding/vector indexing with embedder-identity drift guards, atomic scoped source replacement/deletion with content-hash skip and generation visibility, hybrid vector+lexical retrieval with reciprocal-rank fusion (one embed / one RRF / one rerank across one or many exact scopes), focused text/Markdown/HTML/PDF parsing, bounded reranking (host seam plus a TEI REST adapter), ingestion status, attributable citations, content-trust metadata, and explicit `ContextProvider` injection. It reuses `Embedder` and `VectorStore` from the memory root entry; Prism core input assembly is unchanged.
 
 ## When to use it
 
@@ -27,7 +27,7 @@ Document lifecycle:
 | `replaceSource({ sourceId, chunks, store, scope, ... })` | Atomically replaces one source after all bounded embedding succeeds; the store must implement scoped `getBySource()` and `transaction()`. |
 | `deleteSource({ sourceId, store, scope })` | Deletes only matching IDs under exact tenant/resource/corpus scope. |
 | `replaceDocument({ uri, loader, parser, store, scope, ... })` | Loads through a host seam, parses, chunks, and atomically replaces. `sourceId` is required unless loader supplies one. |
-| `DocumentLoader` / `Parser` | Small host-replaceable seams. Root and `@arnilo/prism-rag/loaders` / `@arnilo/prism-rag/parsers` export reference adapters. |
+| `DocumentLoader` / `Parser` | Small host-replaceable seams. `@arnilo/prism-memory/rag/loaders` and `/rag/parsers` export reference adapters. |
 | `textParser` / `markdownParser` / `htmlParser` / `pdfParser` | UTF-8 text, Markdown, script/style-stripping HTML, and uncompressed-text PDF parsers. |
 
 Index/retrieve:
@@ -46,7 +46,7 @@ Index/retrieve:
 | `statusStore` | no | `IngestionStatusStore` records per-source pending/indexed/failed/partial byte/chunk progress; use `listIngestionStatus()` for capped exact-scope pages. |
 | `contentHash` | no | Host-computed document digest; stamped on records and enables unchanged-source skip in `replaceSource` (`skipIfUnchanged`, default true when present) |
 | `reuseEmbeddings` | no | `ReadonlyMap<string, ReusableEmbedding>` — chunk id → `{ text, embedding }`; embeddings reused (no embed call) when texts match |
-| `telemetry` / `telemetryParent` | no | `RagTelemetry` seam (e.g. `createRagTelemetry()` from `@arnilo/prism-observability-opentelemetry`); spans nest under `telemetryParent` |
+| `telemetry` / `telemetryParent` | no | `RagTelemetry` seam (e.g. `createRagTelemetry()` from `@arnilo/prism-core/governance/observability`); spans nest under `telemetryParent` |
 | `redactor` / `secrets` | no | Redact before embedding, persistence, reranking, and injection |
 | `signal` | no | Abort embedding, vector operations, reranking, and batch progression |
 
@@ -84,7 +84,7 @@ Default/hard ceilings include 1,000/16,384 chunk characters, 100/4,096 overlap,
 ```ts
 import { createAgent, createMockProvider, providerDone, providerTextDelta } from "@arnilo/prism";
 import { createHashEmbedder, createMemoryVectorStore } from "@arnilo/prism-memory";
-import { chunkMarkdown, createMemoryIngestionStatusStore, createRagContextProvider, indexChunks, listIngestionStatus, retrieveContext } from "@arnilo/prism-rag";
+import { chunkMarkdown, createMemoryIngestionStatusStore, createRagContextProvider, indexChunks, listIngestionStatus, retrieveContext } from "@arnilo/prism-memory/rag";
 
 const embedder = createHashEmbedder(); // deterministic demo/test helper, not production semantic quality
 const store = createMemoryVectorStore();
@@ -118,7 +118,7 @@ console.log(found.text, await agent.createSession().run("How do approvals work?"
 Content-hash skip and hash validation:
 
 ```ts
-import { isValidContentHash } from "@arnilo/prism-rag";
+import { isValidContentHash } from "@arnilo/prism-memory/rag";
 
 const digest = "ab12..."; // host-computed SHA-256 hex of the document
 if (!isValidContentHash(digest)) throw new Error("invalid digest");
@@ -128,8 +128,8 @@ await replaceSource({ sourceId: "doc", chunks, embedder, store, scope, contentHa
 Hybrid retrieval, TEI reranking, and telemetry:
 
 ```ts
-import { createRagTelemetry } from "@arnilo/prism-observability-opentelemetry";
-import { createTeiReranker } from "@arnilo/prism-rag";
+import { createRagTelemetry } from "@arnilo/prism-core/governance/observability";
+import { createTeiReranker } from "@arnilo/prism-memory/rag";
 
 const telemetry = createRagTelemetry({ tracer, meter }); // @opentelemetry/api instruments
 const org = { tenantId: "t1", resourceId: "docs", corpusId: "org" };
@@ -152,6 +152,8 @@ const found = await retrieveContext("leave balance", {
 - Metadata filtering is package-local after a bounded candidate query so existing vector contracts/adapters remain unchanged. Increase `queryCandidates` only when selective filters measurably need it.
 - `Reranker` is a host seam, not a provider integration. Return each redacted candidate ID exactly once; Prism retains canonical hit/provenance/trust fields and exposes `retrievalRank` for diagnostics. Add a hosted reranker only when a host owns its credentials, quota, and retry policy.
 - `createTeiReranker({ baseUrl, model?, timeoutMs?, maxResponseBytes?, ssrf?, allowLoopback?, fetch? })` (`CreateTeiRerankerOptions`) adapts a Hugging Face TEI `POST <baseUrl>/rerank` endpoint (`{query, texts, raw_scores:false}` → `{results:[{index,score}]}`) into the `Reranker` seam. It returns a permutation-only reorder of the same hit objects, so provenance/trust move untouched. Response parsing is strict — short/duplicate/out-of-range indices, non-finite scores, HTTP errors, timeouts, and oversized bodies all fail closed; the `rerankHits` caps (`maxRerankBytes`, `maxRerankMs`, `rerankConcurrency`) still apply around it. The default transport is the core DNS-pinned `pinnedFetch` (redirect-free, byte-bounded to 65,536 by default); HTTPS is required unless `allowLoopback: true` (loopback dev/test) or the host supplies `ssrf`/`fetch` for cluster networking. The adapter validates URL shape only — SSRF policy enforcement stays host-side. No credentials are ever sent; there is no SaaS default URL.
+- Hosted rerank adapters over the same seam (plan 062): `createOpenAiCompatibleReranker({ baseUrl, model?, apiKey?, timeoutMs?, maxResponseBytes?, ssrf?, allowLoopback?, fetch? })` speaks the OpenAI-compatible `POST <baseUrl>/rerank` route (`{model, query, documents}` → `{results:[{index,relevance_score}]}`; pass the version segment in `baseUrl`, e.g. `https://api.jina.ai/v1`), and `createVoyageReranker({ baseUrl, model?, apiKey, … })` adapts Voyage AI (`…/v1/rerank` → `{data:[{index,relevance_score}]}`; `apiKey` required). Both send one request per rerank — no adapter-side batching — never send `top_k` (the retrieval seam owns top-K), return the same permutation-only reorder, and fail closed on the same malformed-response/HTTP/timeout/byte-bound cases. `apiKey` rides as `Authorization: Bearer …` and is never logged; errors carry status/host only. No SaaS default URL — hosts own credentials, quota, and retry policy.
+- `createFakeReranker()` is a network-free deterministic reranker (query-term-overlap scoring, stable ties) and `runRerankerConformance(createReranker)` is the shared network-free conformance for any `Reranker` implementation: empty input → `[]`, output is a permutation of the exact input references (provenance/trust untouched), repeated calls are deterministic.
 - Hybrid retrieval: pass `lexical: "fts"` (or `"bm25"` when the store supports it) to `retrieveContext()`; the two legs are fused with reciprocal-rank fusion (`fusion: "rrf"`, `rrfK` 60 default; the pure helper `fuseReciprocalRank()` returns `FusedCandidate[]` for custom orchestration). Stores advertise support via `lexicalModes?: readonly LexicalMode[]` and `tokenizeLexical()` is the shared tokenizer. Each hit's provenance `retrieval` field reports `vector`/`lexical`/`hybrid`; fusion internals expose `RetrievalLeg`.
 - Multi-scope retrieve: `scopes: RagScope[]` searches each exact scope against that scope's current generation, then runs **one** RRF over the union and **one** rerank. The query is embedded once. `queryCandidates` is per scope. Duplicate scopes are dropped. `HARD_RETRIEVE_SCOPE_CAP` is 8.
 - Embedder identity/drift guard: `Embedder.id` (memory contract) is stamped onto every vector record as `embedderId`. `retrieveContext()` fails closed with `ERR_PRISM_RAG_EMBEDDER_MISMATCH` when a stored record's `embedderId` or dimensions differ from the active embedder (for example after a model change) — re-index the source before retrieving. Legacy records without an `embedderId` also fail closed, naming the re-index path.
@@ -161,7 +163,7 @@ const found = await retrieveContext("leave balance", {
 - `createResourceDocumentLoader({ loader })` calls one host-owned `ResourceLoader`; it scans nothing and performs no filesystem or network I/O itself. Pass the host's permission/trust context to that loader.
 - `createWebFetchDocumentLoader({ fetcher })` accepts an already-configured `@arnilo/prism-web-tools` fetch adapter. It never opens a socket, rejects file/local/private/IP-literal URLs, and carries normalized citation/trust metadata forward. The fetch adapter still owns DNS/SSRF policy.
 - `pdfParser` is deliberately limited to bounded, uncompressed PDF text. Provide a host parser through `Parser` for compressed, scanned, or complex PDFs; do not silently index partial text.
-- Package is available directly or through `@arnilo/prism-all`; installation does not create an embedder, vector store, loader, parser, or context provider.
+- Package is available directly or via the `@arnilo/prism-memory` family tarball; installation does not create an embedder, vector store, loader, parser, or context provider.
 
 ## Security and performance notes
 
@@ -175,12 +177,32 @@ const found = await retrieveContext("leave balance", {
 - `replaceSource()` stages every bounded embedding before opening the store transaction. It requires a source-aware transactional store and fails closed rather than pretending generic upserts are atomic. `createMemoryVectorStore()` supplies the reference `getBySource()` / transaction capability; durable stores must implement equivalent exact-scope behavior.
 - `deleteSource()` rechecks every returned record's tenant/resource/corpus and source metadata before delete. Same source IDs in another corpus remain untouched.
 - Parsers enforce byte/page/time caps, abort before and after parsing, decode UTF-8 strictly, and strip HTML script/style content. Parsed and retrieved text remains untrusted inert context; it never gains tool authority.
-- Rerankers receive redacted input under byte/time/concurrency caps. Timeout, abort, unknown/duplicate/missing IDs, oversized input, and reranker failures fail closed; returned objects cannot overwrite Prism provenance/trust fields. The TEI adapter adds fail-closed response parsing (permutation completeness, finite scores) and honors the 65,536-byte response ceiling; SSRF/URL policy is host-side (see Extension notes).
+- Rerankers receive redacted input under byte/time/concurrency caps. Timeout, abort, unknown/duplicate/missing IDs, oversized input, and reranker failures fail closed; returned objects cannot overwrite Prism provenance/trust fields. The TEI adapter adds fail-closed response parsing (permutation completeness, finite scores) and honors the 65,536-byte response ceiling; SSRF/URL policy is host-side (see Extension notes). The hosted OpenAI-compatible and Voyage adapters carry the same guarantees and add Bearer credentials that are never logged and error messages that never contain document text or the API key.
 - Telemetry is a host-owned seam: `RagTelemetry` adapter (`createRagTelemetry()`) drops anything outside a fixed span-name set and `rag.*`-shaped attribute keys, so raw chunk text never reaches the tracer unless the host's own `attributeFilter` opts it in; when the seam is absent, instrumentation costs nothing.
 - Durable vector stores (PostgreSQL/pgvector path via `@arnilo/prism-memory`) run their DDL against the host's knowledge database — tables are created in a schema/table the host names (default `prism_memory.semantic_memory`), and hosts must own backup/retention of that database. See [Working and semantic memory](working-and-semantic-memory.md).
 - Ingestion failure errors are redacted before status storage. Status reads reject foreign scope entries and page-limit violations; status itself creates no permission or tool authority.
 - Filtering scans at most `queryCandidates` hits; rendering stops at top-K, UTF-8 result bytes, or estimated context-token ceiling.
 
+## Live probe (plans/064 Task 6)
+
+Reranker adapters are probed against operator-deployed endpoints — each leg skips (never fails) when its endpoint env var is unset:
+
+```bash
+PRISM_TEST_TEI_RERANKER_URL=http://tei.svc:8080 \
+  node --test packages/memory/dist/rag/__tests__/live.test.js
+```
+
+| Env var | Purpose |
+| --- | --- |
+| `PRISM_TEST_TEI_RERANKER_URL` | TEI service base URL (`/rerank` is appended) |
+| `PRISM_TEST_TEI_RERANKER_KEY` | optional gateway key for a TEI behind auth (rides the trusted-transport seam) |
+| `PRISM_LIVE_TEI_RERANKER_MODEL` | optional TEI model name |
+| `PRISM_TEST_HOSTED_RERANK_URL` | OpenAI-compatible rerank base URL (`/rerank` appended, include `/v1`) |
+| `PRISM_TEST_HOSTED_RERANK_KEY` | Bearer credential for the hosted endpoint |
+| `PRISM_LIVE_HOSTED_RERANK_MODEL` | optional hosted model name |
+
+Probes send one non-sensitive rerank request per configured endpoint and assert the live response conforms (permutation-only reorder, scores non-increasing, credential never in error transcripts). Registered in `scripts/live-matrix.json` as `memory/rag-rerankers-live`.
+
 ## Related APIs
 
 - [Working and semantic memory](working-and-semantic-memory.md): shared `Embedder`/`VectorStore` contracts and adapters.