@arnilo/prism 0.11.0 → 0.12.0

package/docs/providers/typesafe.md

@@ -0,0 +1,148 @@
+# TypeSafe Jev provider package
+
+## What it does
+
+`@arnilo/prism-providers/typesafe` registers the TypeSafe Jev decision model. Jev does not generate free text, stream, or call tools. A request is valid only with `options.structuredOutput`: the JSON schema compiles to System One questions, the messages become the state, and one `POST /v1/systemone` round trip returns typed answers rendered as schema-valid JSON.
+
+The package registers provider `typesafe`, models `jev-latest` and `jev-preview`, and an `api_key` auth method through `createExtensionKernel().load([...])`. Versioned pins (`jev-1.13.0`) go through `defineTypeSafeModel`.
+
+For raw probabilities, confidence, and the actual responding checkpoint instead of rendered JSON, use the shared [`decisions`](decisions.md) subpath (re-exported from this package); the adapter deliberately does not copy those values onto events.
+
+## When to use it
+
+Use it when a host wants a hosted yes/no, pick-one, or rubric decision and already has a JSON object schema. Do not use it as a general chat model, a tool-calling agent, or a streaming text model.
+
+## Inputs / request
+
+```ts
+import {
+  createTypeSafeProvider,
+  createTypeSafeProviderPackage,
+  defineTypeSafeModel,
+  TYPESAFE_API_KEY_ENV,
+  TYPESAFE_DEFAULT_BASE_URL,
+  typeSafeModels,
+} from "@arnilo/prism-providers/typesafe";
+```
+
+| Field | Type | Purpose |
+| --- | --- | --- |
+| `apiKey` | `CredentialValueSource` | Bearer token. Hosts wire `TYPESAFE_API_KEY` through the credential seam. The library does not read `process.env`. |
+| `fetch` | `typeof fetch` | Optional fetch for tests and hosts. |
+| `baseUrl` | `string` | Overrides `https://api.typesafe.ai`. Trailing slashes are trimmed. `/v1/systemone` is appended per request. |
+| `id` | `string` | Overrides the provider id (default `typesafe`). |
+| `models` | `readonly ModelConfig[]` | Overrides `typeSafeModels`. |
+| `maxRetries` | `number` | Retries after the first attempt. Default 2. Forwarded to the shared client. |
+
+`generate()` accepts a normal `ProviderRequest` with these constraints:
+
+| Field | Requirement |
+| --- | --- |
+| `options.structuredOutput` | Required. `{ name, schema }`. Missing schema fails before any fetch. |
+| `tools` | Rejected. Decision model, no tool support. Fails before any fetch. |
+| `options.compat.boolean_threshold` | Optional number in `0..1`. Default `0.5`. Out of range fails before any fetch. A boolean is true only when the noul probability is strictly greater than the threshold, so `0.5` renders `false` at the default. |
+| `messages` | State only. A single user text message is sent as a string; otherwise the client sends `[{ role, text }]` and drops non-text parts. Do not write the question into the state — the model judges that text instead of answering it. |
+| `model.model` | Sent verbatim. The responding version in the API body is not rewritten into the output. |
+| `signal` | A signal already aborted throws before fetch. Abort during the round trip rejects the request. |
+
+Schema mapping (unsupported shapes throw before any network call and name the field path):
+
+| JSON Schema | Question |
+| --- | --- |
+| `boolean` | `noul`. Instructions come from `description`, then `title`, then the field path. |
+| string `enum` of 2–255 values | `choice`. Criteria are the option labels unless `x-systemone.options` (or the `x-typesafe` alias) supplies descriptions. |
+| integer `enum` equal to `0..N-1` with 2–10 described levels | `score`. Levels come from `x-systemone.levels` (alias `x-typesafe`), aligned to the declared enum order and sent in numeric order. An undescribed whole-number enum stays a `choice`. |
+| nested object | One question per leaf, id `outer.inner`. |
+| string, unbounded number, array, union, empty object | Rejected. |
+
+Extension keys live on the property:
+
+```json
+{ "type": "boolean", "x-systemone": { "criteria": { "true": "destructive", "false": "safe" } } }
+{ "type": "string", "enum": ["run", "reject"], "x-systemone": { "options": { "run": "safe", "reject": "destructive" } } }
+{ "type": "integer", "enum": [0, 1, 2], "x-systemone": { "levels": ["opaque", "partial", "actionable"] } }
+```
+
+Limits rejected up front: more than 255 choice options, more than 10 score levels, more than 256 questions. Field names that contain a dot are rejected so dotted nesting stays unambiguous.
+
+Curated models declare `capabilities: { input: ["text"], output: ["text"], tools: false, streaming: false, structuredOutput: "json_schema" }`, `limits: { contextWindow: 32000, maxOutputTokens: 0 }`, and `cost: { input: 0.04, output: 0, currency: "USD", unit: "per_million_tokens" }`.
+
+## Outputs / response / events
+
+| Event | Behavior |
+| --- | --- |
+| `message_start` | One, before the answer. |
+| `content_delta` | One text block. The text is `JSON.stringify` of the schema-shaped object. |
+| `usage` | `inputTokens` from the API. `outputTokens` is `0`. `totalTokens` equals `inputTokens`. Omitted when the API sends no usage. |
+| `done` | `stopReason: "end_turn"`, same usage when present. |
+| `error` | Terminal event for gate failures, HTTP failures, and render failures. `401` is not retried. `422` is not retried and the message includes the API field detail. `429` and `5xx` (including `529`) retry, honoring `Retry-After`. |
+
+Booleans come from noul vs `boolean_threshold`. Enums come from the choice string (whole-number options stay numbers). Rubric integers are `Math.round` of the score, half rounds up, clamped to the rubric. Nested objects are reassembled from dotted ids. Confidence, probabilities, and score distributions are not copied onto events; the text must stay schema-valid. Success bodies are validated before output: malformed answer types, missing/non-finite numeric values, probabilities outside `0..1`, and invalid token counts produce an error rather than a coerced decision. Finite rubric scores still round and clamp as described above.
+
+## Request/response example
+
+```json
+{
+  "model": "jev-latest",
+  "state": "rm -rf ./build",
+  "questions": {
+    "verdict": {
+      "type": "choice",
+      "instructions": "How should this be handled?",
+      "criteria": { "run": "run", "reject": "reject", "ask": "ask" }
+    },
+    "irreversible": { "type": "noul", "instructions": "Would running this destroy data?" }
+  }
+}
+```
+
+```json
+{
+  "model": "jev-1.13.0",
+  "answers": {
+    "verdict": { "type": "choice", "choice": "ask" },
+    "irreversible": { "type": "noul", "noul": 0.91 }
+  },
+  "usage": { "input_tokens": 96, "output_tokens": 0 }
+}
+```
+
+Rendered text: `{"verdict":"ask","irreversible":true}`.
+
+## Implementation example
+
+```ts
+import { createEnvCredentialResolver, createExtensionKernel } from "@arnilo/prism";
+import { createTypeSafeProviderPackage, TYPESAFE_API_KEY_ENV } from "@arnilo/prism-providers/typesafe";
+
+const kernel = createExtensionKernel();
+await kernel.load([
+  createTypeSafeProviderPackage({
+    apiKey: createEnvCredentialResolver(process.env, { typesafe: TYPESAFE_API_KEY_ENV }),
+  }),
+]);
+```
+
+A request must carry `options.structuredOutput`. The answer arrives as one text delta.
+
+## Extension and configuration notes
+
+- Replace the catalog with `models` or register extra pins via `defineTypeSafeModel({ model: "jev-1.13.0" })`. Defaults (capabilities, 32k context, zero output tokens, $0.04/1M input) still apply.
+- `compat.boolean_threshold` is the only request knob. There is no temperature, tool choice, or streaming flag.
+- `baseUrl` points the same client at a proxy. It does not change the wire shape.
+- No model listing runs at setup or during `generate()`.
+
+## Security and performance notes
+
+- **Egress.** State text and compiled questions leave the process for TypeSafe's hosted API (`https://api.typesafe.ai` unless `baseUrl` overrides it). Do not put secrets in the state.
+- **Credentials.** The bearer token is resolved once per request under the provider id; that same value is used for transport and error redaction. It is never logged.
+- **Cost and latency.** One `POST /v1/systemone` round trip per structured-output request. Questions are free in parallel; state tokens are the cost. Output tokens are not billed (`output: 0`). There is no streaming and no second hop for model discovery.
+- **Fail closed.** Missing schema, tools, an out-of-range threshold, and unsupported fields fail before fetch.
+
+## Related APIs
+
+- [System One decisions](decisions.md): raw typed decision calls over the same wire, with probabilities, confidence, actual model, usage, and timing.
+- [Laya](laya.md): same wire and schema mapping against a self-hosted `laya-serve`.
+- [Structured output](../structured-output.md): `options.structuredOutput` contract this adapter requires.
+- [Provider packages](../provider-packages.md): registration and auth-method shape.
+- [Credentials and redaction](../credentials-and-redaction.md): `CredentialValueSource` and `createEnvCredentialResolver`.

package/docs/rag.md

@@ -34,7 +34,7 @@ Document lifecycle:
 | `resolveReranker(config)` | Declarative reranker config (`kind: "local" \| "tei" \| "openai-compatible" \| "voyage" \| "fake" \| "none"`) → `Reranker`. |
 | `createGoogleDriveConnector({ tokenProvider, resolveAccess })` | Drive `files.list` + `changes.list` connector. Host maps permissions; watch payloads are not authorization. |
 | `DocumentLoader` / `Parser` | Small host-replaceable seams. `@arnilo/prism-memory/rag/loaders` and `/rag/parsers` export reference adapters. |
-| `textParser` / `markdownParser` / `htmlParser` / `pdfParser` | UTF-8 text, Markdown, script/style-stripping HTML, and uncompressed-text PDF parsers. |
+| `textParser` / `markdownParser` / `htmlParser` / `pdfParser` | UTF-8 text, Markdown, script/style-stripping HTML, and uncompressed-text PDF parsers. A host parser from `createDocumentIngest` ([Document extraction](document-extraction.md)) can replace `pdfParser`; it must throw on failed OCR so `replaceDocument` does not index. Do not raise `HARD_MAX_PARSE_MS_CAP` (30 s). |
 
 Index/retrieve:
 
@@ -112,7 +112,7 @@ const result = await propagator.propagate("doc:erp-lead");
 - `propagate(sourceId)` expands the source through `_lineage.sourceIds` (`collectInvalidationIds`, depth 8) into a closed id set, tombstones **all** of it with the propagator's resolved reason (`forgotten` by default; `legal_hold` stamps `hold: true`) inside one store transaction, then runs every registered handler with `{ sourceId, ids, scope, reason, signal }`. Handlers return how many artifacts they removed (reported per `kind` in `layers`). The context `reason` is the single source of truth: a handler's own `reason` option is only the fallback for a hand-built context, so a `legal_hold` propagation cannot land in a handler's tombstones as `forgotten`.
 - Tombstones, not deletions, for derived rows: rows stay for explainability (`recall({ explain: true })` reports the invalidation), and lineage links never dangle. Handlers own physical removal (chunk rows, files, ledger entries).
 - Retrieval is belt-and-suspenders: `retrieveContext()` reads per-scope invalidations before assembly and drops any candidate whose record id, `_lineage.sourceIds`, or `_rag.sourceId` is tombstoned — so a delete that lands after the query legs read rows still returns zero hits. The split matters for direct store users: the store's own SQL predicate filters by record id and `_lineage` edge, while a source's *own* chunk rows are covered by the `_rag.sourceId` rule at the retrieval boundary (or removed physically by the `rag` handler) — a raw `store.query()` is not a recall path.
-- `HARD_PROPAGATION_EDGES` (4,096) is the one-pass privileged ceiling; over it the whole delete rejects (fail-closed), never a half-tombstoned document. Each store `invalidate` call carries at most `HARD_INVALIDATION_BATCH` (64) entries. On a durable store that shape holds: PostgreSQL/pgvector tombstones 1,001 rows (1,000 derived chunk rows + the source root) in **one transaction and 22 statements** (16 of them `HARD_INVALIDATION_BATCH`-sized `INSERT`s), measured at **29–155 ms** across runs on an AMD Ryzen 9 PRO 7940HS against `pgvector/pgvector:pg16` (more under parallel load) — the durable counterpart of the in-memory suite's 1k-under-2s check, and evidence rather than a gate. Re-run it with `PRISM_TEST_POSTGRES_URL=… npm run test:postgres` (`packages/memory/src/__tests__/postgres-propagation.integration.test.ts`); the leg also proves the store's own SQL predicate hides the tombstoned rows, not only the in-app guard, and that a denied propagation opens no transaction at all.
+- `HARD_PROPAGATION_EDGES` (4,096) is the one-pass privileged ceiling; over it the whole delete rejects (fail-closed), never a half-tombstoned document. Each store `invalidate` call carries at most `HARD_INVALIDATION_BATCH` (64) entries. On a durable store that shape holds: PostgreSQL/pgvector tombstones 1,001 rows (1,000 derived chunk rows + the source root) in **one transaction and 22 statements** (16 of them `HARD_INVALIDATION_BATCH`-sized `INSERT`s), measured at **29–155 ms** across runs on an AMD Ryzen 9 PRO 7940HS against `pgvector/pgvector:pg16` (more under parallel load) — the durable counterpart of the in-memory suite's 1k-under-2s check, and evidence rather than a gate. Re-run it with `PRISM_TEST_POSTGRES_URL=… bun run test:postgres` (`packages/memory/src/__tests__/postgres-propagation.integration.test.ts`); the leg also proves the store's own SQL predicate hides the tombstoned rows, not only the in-app guard, and that a denied propagation opens no transaction at all.
 - Deletion is privileged: `authorization` is required, tenant-checked, and enforced through the store's existing `checkSourceAccess` ACL when the store declares `authorization: "acl"` (missing grant → `MemoryScopeError` before anything is written). Retrieval paths never construct a propagator.
 - Observational memory registers its own leg: `createObservationalMemoryDropHandler({ session, appendEntry })` (from `@arnilo/prism-memory/compaction/observational-memory`) folds the session ledger once per propagation and writes one `om.observations.dropped` entry for the observations that rest on a tombstoned record id; see [observational memory](compaction-observational-memory.md).
 
@@ -152,6 +152,18 @@ const blocks = buildObservationalMemoryContextBlocks(entries, { invalidatedIds:
 - One `listInvalidatedIds` read per projection build (one scope read, `corrected` entries stay), and the recipe adds no work beyond the propagator: the same `layers` result already answers per-leg counts, so nothing is re-read to report it.
 - The fabric leg is the one that cannot be left out: a note names its document by `metadata.path`, so no `_lineage` edge exists to walk and a deleted path would otherwise keep being served. `createFabricRepointHandler()` tombstones the notes recorded against the deleted id in the same pass (plan 102 Task 11), and the same handler follows a `repointSource()` move — see the re-point section below.
 
+## Revocation through derived context
+
+Grant rechecks and tombstones cover retrieval; derived context (observational memory, wiki pages, fabric notes) needs the propagation pass above. `examples/revocation-propagation.ts` runs the whole chain offline on in-memory stores:
+
+1. **Access loss for one principal** (`setSourceAccess` with the grant narrowed) excludes the source for that principal, still serves it to another, keeps the rows, and writes no tombstone.
+2. **Deletion** (`createDeletionPropagator` + `createRagDeletionHandler`) tombstones the source, deletes its chunk rows, and excludes it for every principal — including one that still held a grant.
+3. **The observational-memory leg runs in two observable steps:** `listInvalidatedIds(store, scope)` feeds `invalidatedIds` into `buildObservationalMemoryContextBlocks()` so a revoked observation is withheld before the drop entry lands, and `createObservationalMemoryDropHandler()` later writes the `om.observations.dropped` entry; both paths render the same memory.
+4. **A revoke landing while a reranker runs** is withheld by the post-rerank gate and reported through `onAccessDenied` (`no_grant`), so a mid-flight grant change cannot leak.
+5. **A thrown grant check** fails closed as `check_failed` with the store error redacted and capped at 256 chars; the query still completes with the remaining hits.
+
+The example also keeps the disclosure boundary honest: a report generated before the revoke did contain the authorized text, and revocation prevents subsequent exposure — it does not claim to erase prior external disclosure.
+
 ## Grant recheck and re-pointing
 
 Retrieval never trusts a grant snapshot. `retrieveContext()` re-asks the store for **each distinct source** it is about to inject, on both sides of the reranker:
@@ -260,10 +272,10 @@ const result = await retrieveContext("How do approvals work?", { embedder, store
 ```
 
 - `resolveReranker({ kind: "local" })` is the zero-config path. The model runtime is a host seam exactly like `Embedder`: `createLocalReranker({ model?, runtime?, onLoad?, cacheDir?, dtype?, device?, allowRemoteModels? })`. Pass `runtime: { load(model) → { id, score({ query, documents, signal }) } }` to inject a runtime the host already owns (transformers.js, onnxruntime-node, llama.cpp). With no `runtime`, the built-in loader resolves `@huggingface/transformers` at first use — the package declares no inference dependency (no new dependency name in any manifest) and nothing resolves it at build/install time.
-- Sizing trade-off: the download is one-time and host-cached, and per-query latency is CPU-bound and grows with candidates × tokens, so keep `topK`/`queryCandidates` near what recall actually needs — the reranker reorders what retrieval returned, it cannot recover a chunk the candidate pool never returned. Measured on one corpus (24 queries / 96 chunks: one answering chunk + three mention-only chunks per query, k=5, `Xenova/bge-reranker-base` q8 on x86 CPU, vector-only) with two embedders: the deterministic lexical `createHashEmbedder` baseline gives recall@5 **0.21 → 0.79** and recall@20 **0.63** at the package default 20-candidate pool, while the semantic `Xenova/all-MiniLM-L6-v2` q8/cpu (384 dims) gives **0.79 → 0.79** and recall@20 **1.00** — the semantic baseline starts at the lexical reranked number, the reranker's lift is 0.000 on this corpus, and the pool is not the binding constraint. Read the row for your own embedder: with a lexical/deterministic embedder the pool bound is what to raise first; with a semantic one the reranker is ordering quality only and the residual misses need better retrieval. Top-50 median was 95–289 ms across runs. Corpus, misses, pool-bound numbers, latency, cache state, and the semantic side-by-side live in [`docs/_evidence/phase111-reranker-semantic-recall.md`](_evidence/phase111-reranker-semantic-recall.md), with the lexical control in [`docs/_evidence/phase102-local-rerank-latency.md`](_evidence/phase102-local-rerank-latency.md); both are regenerated by `PRISM_TEST_LOCAL_RERANK=1 npm run test:live`. Treat the numbers as one data point on one machine, not a ceiling: dtype, device, and the embedder move them. The non-CPU (fp16/GPU) leg is host-provisioned — re-measure before expecting the CPU numbers to hold. The package guarantees the plumbing (one lazy load, one batched score call per rerank), not the model's speed. The hosted/TEI adapters stay for scale (higher throughput, no local RAM, no download).
+- Sizing trade-off: the download is one-time and host-cached, and per-query latency is CPU-bound and grows with candidates × tokens, so keep `topK`/`queryCandidates` near what recall actually needs — the reranker reorders what retrieval returned, it cannot recover a chunk the candidate pool never returned. Measured on one corpus (24 queries / 96 chunks: one answering chunk + three mention-only chunks per query, k=5, `Xenova/bge-reranker-base` q8 on x86 CPU, vector-only) with two embedders: the deterministic lexical `createHashEmbedder` baseline gives recall@5 **0.21 → 0.79** and recall@20 **0.63** at the package default 20-candidate pool, while the semantic `Xenova/all-MiniLM-L6-v2` q8/cpu (384 dims) gives **0.79 → 0.79** and recall@20 **1.00** — the semantic baseline starts at the lexical reranked number, the reranker's lift is 0.000 on this corpus, and the pool is not the binding constraint. Read the row for your own embedder: with a lexical/deterministic embedder the pool bound is what to raise first; with a semantic one the reranker is ordering quality only and the residual misses need better retrieval. Top-50 median was 95–289 ms across runs. Corpus, misses, pool-bound numbers, latency, cache state, and the semantic side-by-side live in [`docs/_evidence/phase111-reranker-semantic-recall.md`](_evidence/phase111-reranker-semantic-recall.md), with the lexical control in [`docs/_evidence/phase102-local-rerank-latency.md`](_evidence/phase102-local-rerank-latency.md); both are regenerated by `PRISM_TEST_LOCAL_RERANK=1 bun run test:live`. Treat the numbers as one data point on one machine, not a ceiling: dtype, device, and the embedder move them. The non-CPU (fp16/GPU) leg is host-provisioned — re-measure before expecting the CPU numbers to hold. The package guarantees the plumbing (one lazy load, one batched score call per rerank), not the model's speed. The hosted/TEI adapters stay for scale (higher throughput, no local RAM, no download).
 - Host defaults: `dtype: "q8"` with `device: "cpu"` on x86 — fp32 weights are roughly 4× the download for no measurable ranking gain in this size class, and fp16/GPU is worth opting into only when the host already provisions it. Weights are cached per host: pass one `cacheDir` (e.g. `~/.cache/prism/models`) and the runtime lays out one subdirectory per model id, so a second model or a second process reuses the same files — point local embedders running through the same runtime at that directory too. With `cacheDir` omitted the runtime's own default cache applies (inside the installed package). On a cache miss the model is downloaded once into that directory and later runs stay on disk: add `allowRemoteModels: false` on an offline host to fail instead of reaching the model registry, which is exactly what the live leg's second pass proves.
 - Cheap by construction: the model loads lazily once per reranker instance, `score` is called once per rerank with every candidate (never one call per document), and `onLoad({ model, loadMs })` is the only opt-in observability — no document text is ever logged. Zero network after load; the built-in loader only touches the model registry at load time, and `allowRemoteModels: false` pins it to local files.
-- Failure is loud: a missing runtime, an unreachable model, or a runtime that returns no per-document scores throws a redacted `RagValidationError` naming the model and the install path (`npm i @huggingface/transformers` or pass `{ runtime }`). There is deliberately **no** silent lexical fallback.
+- Failure is loud: a missing runtime, an unreachable model, or a runtime that returns no per-document scores throws a redacted `RagValidationError` naming the model and the install path (`bun add @huggingface/transformers` or pass `{ runtime }`). There is deliberately **no** silent lexical fallback.
 - `rerankHits` is unchanged and still owns the caps and the trust boundary: local scores reorder the same `RagHit` references (provenance/trust untouched), byte/ms/concurrency limits apply, and abort/timeout/malformed-score cases fail closed.
 
 ## Implementation example
@@ -385,7 +397,7 @@ Reranker adapters are probed against operator-deployed endpoints — each leg sk
 
 ```bash
 PRISM_TEST_TEI_RERANKER_URL=http://tei.svc:8080 \
-  node --test packages/memory/dist/rag/__tests__/live.test.js
+  bun test packages/memory/dist/rag/__tests__/live.test.js
 ```
 
 | Env var | Purpose |