@arnilo/prism 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/CHANGELOG.md +27 -1
  2. package/README.md +18 -16
  3. package/dist/agent-run-lifecycle.d.ts +2 -1
  4. package/dist/agent-run-lifecycle.js +1 -1
  5. package/dist/agent-session/session/assemble.js +9 -7
  6. package/dist/agent-session/session/tool-round.js +30 -20
  7. package/dist/agent-session/session/types.d.ts +1 -0
  8. package/dist/agent-session/session.d.ts +1 -0
  9. package/dist/agent-session/session.js +3 -2
  10. package/dist/checkpoint-restore.d.ts +50 -14
  11. package/dist/checkpoint-restore.js +104 -28
  12. package/dist/contracts-core/session.d.ts +2 -1
  13. package/dist/contracts-run-state.d.ts +12 -4
  14. package/dist/index.d.ts +3 -3
  15. package/dist/index.js +1 -1
  16. package/dist/leases.js +32 -6
  17. package/dist/node/contribution-discovery.d.ts +16 -1
  18. package/dist/node/contribution-discovery.js +47 -0
  19. package/dist/node/session-store-jsonl.js +67 -17
  20. package/dist/run-limits.d.ts +11 -5
  21. package/dist/session-stores.js +61 -12
  22. package/dist/testing/prefix-stability-conformance.d.ts +44 -1
  23. package/dist/testing/prefix-stability-conformance.js +92 -29
  24. package/dist/usage-estimation.d.ts +7 -1
  25. package/dist/usage-estimation.js +16 -10
  26. package/docs/acp.md +2 -2
  27. package/docs/agent-events.md +7 -6
  28. package/docs/agent-session-runtime.md +1 -1
  29. package/docs/coding-agent-tools.md +1 -1
  30. package/docs/coding-tools.md +7 -11
  31. package/docs/context-and-skills.md +6 -7
  32. package/docs/contribution-discovery.md +13 -0
  33. package/docs/durable-runs.md +10 -3
  34. package/docs/embeddings.md +3 -1
  35. package/docs/execution-timeline.md +6 -0
  36. package/docs/extensions.md +1 -2
  37. package/docs/impeccable.md +1 -2
  38. package/docs/index.md +24 -20
  39. package/docs/live-testing.md +1 -2
  40. package/docs/memory-fabric.md +3 -2
  41. package/docs/migrate-to-0.11.md +65 -0
  42. package/docs/migration.md +12 -1
  43. package/docs/node-jsonl-session-store.md +4 -3
  44. package/docs/operations.md +1 -1
  45. package/docs/peer-dependencies.md +3 -5
  46. package/docs/policy-and-audit.md +1 -1
  47. package/docs/prefix-stability-conformance.md +30 -7
  48. package/docs/provider-packages.md +20 -20
  49. package/docs/public-contracts.md +1 -1
  50. package/docs/rag.md +2 -2
  51. package/docs/release-and-install.md +57 -57
  52. package/docs/runs-and-usage.md +6 -4
  53. package/docs/session-stores.md +2 -2
  54. package/docs/supervisors.md +14 -6
  55. package/docs/testing.md +17 -9
  56. package/docs/workflows.md +2 -2
  57. package/package.json +5 -4
  58. package/docs/caveman.md +0 -130
  59. package/docs/graft.md +0 -149
  60. package/docs/ponytail.md +0 -129
package/docs/index.md CHANGED
@@ -2,7 +2,16 @@
2
2
 
3
3
  Prism is a TypeScript/Node.js agent harness. Hosts own providers, tools, credentials, storage, and behavior; Prism supplies contracts, registries, events, and replaceable runtime primitives.
4
4
 
5
- ## Current line (0.10.0)
5
+ ## Current line (0.11.0)
6
+
7
+ - **Memory-store branch reads**: the built-in memory session store implements `readBranchPath`. A snapshot walks the branch once and clones each kept entry once.
8
+ - **JSONL parse cache**: a read after an in-process append reuses the parsed file when size and mtime match. A same-size write inside one filesystem timestamp tick can still look unchanged.
9
+ - **Idempotency window**: memory and JSONL stores remember the latest 4,096 dedup keys. Replaying an older key appends a new entry instead of rejecting the write.
10
+ - **In-memory lease sweep**: expired lease rows are deleted once the map reaches 1,024. A swept key starts its next fence at 1. A released key still in the map keeps `fencingToken + 1`. SQLite and Postgres adapters still keep the counter on the row.
11
+ - **Shared text token estimate**: plain-text estimates use one `ceil(length/4)` helper. Message and entry estimates are unchanged.
12
+ - **12 publishable packages** at current **0.11.0** lockstep, with the migration guide reachable from the release section below — inventory below.
13
+
14
+ ### Carried from the 0.10.0 line
6
15
 
7
16
  - **Attention budget axes**: `attentionCompiler.trigger` accepts one axis, a predicate, or an any-of array (`input_ratio`, `run_input_ratio`, `token_floor`), and `durable: true` keeps the fold ledger and sticky frontier in the checkpoint across a resume.
8
17
  - **Turn traces and exhaustion attribution**: `provider_turn_finished` carries a closed `stopReason`, a `budgets` snapshot, the effective tool menu (`count` / `idsHash`), and provider cache counts; `agent_finished` carries the run outcome, and the execution timeline adds per-turn stop reasons plus `exhaustion`.
@@ -17,8 +26,6 @@ Prism is a TypeScript/Node.js agent harness. Hosts own providers, tools, credent
17
26
  - **Shared work scopes**: explicitly granted observational-memory scopes shared across sessions, deny-by-default, audited, and revocable at the next read; session-private scopes stay the default.
18
27
  - **Retrieval revocation and local reranking**: deletion and revocation propagate through derived vector/wiki artifacts under bounded walks, and an in-process cross-encoder reranker ships with no declared inference dependency.
19
28
  - **Live-stream terminal semantics**: one `isTerminalAgentEventType` predicate (`agent_finished` / `agent_denied` / `error`) shared by every source, so a limit death delivers `run_limit_exceeded` → `budget_exhausted` → `error` before a stream or replay ends.
20
- - **12 publishable packages** at current **0.10.0** lockstep, with the migration guide reachable from the release section below — inventory below.
21
-
22
29
  ### Carried from the 0.8.0 line
23
30
 
24
31
  - **Messaging channels**: `@arnilo/prism-channels` transport-neutral runtime with deny-by-default authorization, owned bindings, one-use durable approvals, official Telegram (private DMs, opt-in granted groups/topics, drafts, bounded media/voice, opt-in notices) and experimental pinned signal-cli Signal.
@@ -50,7 +57,6 @@ Prism is a TypeScript/Node.js agent harness. Hosts own providers, tools, credent
50
57
  - **Peer and options truth**: the optional peer-dependency matrix and the configuration options index (both linked below) cover every third-party peer and public option surface (plan 070).
51
58
  - **Trusted extension activation**: `activateKernel(kernel)` returns ready-to-spread `AgentConfig` contributions; CLI loads allow-listed `--extension` packages (plan 069).
52
59
  - **Wiki ingest**: `/wiki-ingest` + `ingestWikiSource` stage text/file/image/PDF (and URLs via a host `fetchUrl` hook) into `raw/ingest/` with an OKF filing brief (plan 069).
53
- - **Graft graph commands**: `/graft-init`, `/graft-build`, `/graft-build-deep` (host-configured `deepModel`, key env-only) (plan 069).
54
60
  - **Run limits**: HARD caps are request/response bytes only; policy axes accept `null` (plan 067).
55
61
  - **Tool-result fold**: content-only `ToolResult`s fold into `tool_result.result` (0.5.3).
56
62
  - **Stream token coalesce**: adjacent text/thinking deltas merge on persist (0.5.2).
@@ -224,7 +230,7 @@ Prism is a TypeScript/Node.js agent harness. Hosts own providers, tools, credent
224
230
 
225
231
  ## Testing and examples
226
232
 
227
- - [Test layout and isolation](testing.md): the five `npm test` stages, scratch-root rule, and the tracked-fixture isolation gate.
233
+ - [Test layout and isolation](testing.md): the `npm test` stage table (build, performance budget, root, SQLite, gate, build-race, workspace, examples, and branch-coverage stages), scratch-root rule, and the tracked-fixture isolation gate.
228
234
  - [Contribution quality budgets](contributing.md): the non-null assertion allowance, export-surface ceilings, and the rule that keeps them shrinking.
229
235
  - [Live and end-to-end testing](live-testing.md): opt-in live matrix with skip-not-fail contract and credential scoping table.
230
236
  - Provider test doubles: `createMockProvider()` and provider event helpers are documented on the canonical Provider layer page above.
@@ -239,9 +245,6 @@ Prism is a TypeScript/Node.js agent harness. Hosts own providers, tools, credent
239
245
 
240
246
  ## Third-party integrations
241
247
 
242
- - [Caveman behavior integration](caveman.md): upstream Caveman skills with injector, persistence, and progressive catalog.
243
- - [Ponytail behavior integration](ponytail.md): upstream Ponytail skills with injector and peer resolution; opt-in.
244
- - [Graft context-graph integration](graft.md): graft CLI pull tools, retrieval-pack context provider, blast-radius middleware, and `/graft-init` / `/graft-build` / `/graft-build-deep` commands (host-configured `deepModel`).
245
248
  - [Impeccable behavior integration](impeccable.md): upstream Impeccable skill behind `load_skill`; host supplies the compiled `SKILL.md`.
246
249
  - [Messaging channels](messaging-channels.md): `@arnilo/prism-channels` transport-neutral runtime — deny-by-default sender authorization, owned session binding, serialized turns, current-run replies, one-use durable approvals, bounded attachment refs (images reach the model only when it declares image input), and opt-in host notices to one already-bound pair.
247
250
  - [Telegram channel](telegram-channel.md): official `@arnilo/prism-channels/telegram` long polling and mountable webhook ingress with durable offset/lease handling, approval callbacks, opt-in granted group/topic text, bounded media with optional voice transcription/synthesis, and opt-in streaming drafts.
@@ -251,6 +254,7 @@ Prism is a TypeScript/Node.js agent harness. Hosts own providers, tools, credent
251
254
  ## Release and install
252
255
 
253
256
  - [Release and install](release-and-install.md): install rules, package graph, and deterministic resumable publication.
257
+ - [Migrate 0.10 → 0.11](migrate-to-0.11.md): idempotency window, in-memory lease fence reset, and the persona/graft subpath removals.
254
258
  - [Migrate 0.8 → 0.9](migrate-to-0.9.md): the four behavior deltas inside existing surfaces (limit-death stream order, turn-trace metadata, cache-stable disclosure, labeled usage estimates), every new option with its sizing line, and 0.9.0 host migration steps.
255
259
  - [Migrate 0.7 → 0.8](migrate-to-0.8.md): work-family import map, messaging channels, connected apps, durable runs, and 0.8.0 host migration steps.
256
260
  - [Migrate 0.6 → 0.7](migrate-to-0.7.md): ACP MCP allow-list URL normalization, model router facade fail-closed governance, and 0.7.0 host migration steps.
@@ -268,16 +272,16 @@ The generated inventory below derives from [`scripts/package-truth.json`](../scr
268
272
 
269
273
  | package | version | notes |
270
274
  | --- | --- | --- |
271
- | `@arnilo/prism` | 0.10.0 | core — runtime, CLI/RPC, templates, docs |
272
- | `@arnilo/prism-channels` | 0.10.0 | family — transport-neutral messaging runtime, durable journal, pairing and one-use approvals; official /telegram (private DMs, opt-in granted groups/topics) and experimental pinned signal-cli /signal |
273
- | `@arnilo/prism-coding-tools` | 0.10.0 | family — /agent, /security, /openapi, /computer-use-linux, /dev, /caveman, /ponytail, /impeccable subpaths |
274
- | `@arnilo/prism-core` | 0.10.0 | family — /runtime, /sessions, /governance, /credentials, /enterprise, /validation subpaths |
275
- | `@arnilo/prism-providers` | 0.10.0 | family — all provider adapters as `/<adapter>` subpaths |
276
- | `@arnilo/prism-acp-agent` | 0.10.0 | capability — ACP adapter |
277
- | `@arnilo/prism-ag-ui` | 0.10.0 | capability — AG-UI/A2A/A2UI adapter |
278
- | `@arnilo/prism-hooks` | 0.10.0 | capability — Claude/Codex-compatible hooks.json adapter compiled onto middleware, guardrail, injector, and stop-hook seams |
279
- | `@arnilo/prism-mcp` | 0.10.0 | capability — MCP client/server/OAuth interop |
280
- | `@arnilo/prism-memory` | 0.10.0 | capability — memory plus /rag, /compaction/*, /fabric, /graft, /wiki subpaths |
281
- | `@arnilo/prism-web-tools` | 0.10.0 | capability — Brave/Exa/Firecrawl plus peer-gated /browser and /obscura subpaths |
282
- | `@arnilo/prism-work` | 0.10.0 | capability — /connectors, /documents, /sheets, /diagrams, /document-reader, /sandbox, /skills, /tools subpaths |
275
+ | `@arnilo/prism` | 0.11.0 | core — runtime, CLI/RPC, templates, docs |
276
+ | `@arnilo/prism-channels` | 0.11.0 | family — transport-neutral messaging runtime, durable journal, pairing and one-use approvals; official /telegram (private DMs, opt-in granted groups/topics) and experimental pinned signal-cli /signal |
277
+ | `@arnilo/prism-coding-tools` | 0.11.0 | family — /agent, /security, /openapi, /computer-use-linux, /dev, /impeccable subpaths |
278
+ | `@arnilo/prism-core` | 0.11.0 | family — /runtime, /sessions, /governance, /credentials, /enterprise, /validation subpaths |
279
+ | `@arnilo/prism-providers` | 0.11.0 | family — all provider adapters as `/<adapter>` subpaths |
280
+ | `@arnilo/prism-acp-agent` | 0.11.0 | capability — ACP adapter |
281
+ | `@arnilo/prism-ag-ui` | 0.11.0 | capability — AG-UI/A2A/A2UI adapter |
282
+ | `@arnilo/prism-hooks` | 0.11.0 | capability — Claude/Codex-compatible hooks.json adapter compiled onto middleware, guardrail, injector, and stop-hook seams |
283
+ | `@arnilo/prism-mcp` | 0.11.0 | capability — MCP client/server/OAuth interop |
284
+ | `@arnilo/prism-memory` | 0.11.0 | capability — memory plus /rag, /compaction/*, /fabric, /wiki subpaths |
285
+ | `@arnilo/prism-web-tools` | 0.11.0 | capability — Brave/Exa/Firecrawl plus peer-gated /browser and /obscura subpaths |
286
+ | `@arnilo/prism-work` | 0.11.0 | capability — /connectors, /documents, /sheets, /diagrams, /document-reader, /sandbox, /skills, /tools subpaths |
283
287
  <!-- generated:package-truth:inventory end -->
@@ -90,7 +90,7 @@ Set only the rows you want to run; everything else skips. Least-privilege scope
90
90
  | `calibration/vendor-count-tokens` | active | `PRISM_LIVE_PROVIDER_TESTS`; any of: `ANTHROPIC_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_API_KEY` | `PRISM_LIVE_ANTHROPIC_MODEL` (default `claude-haiku-4-5`), `PRISM_LIVE_GOOGLE_MODEL` (default `gemini-2.5-flash-lite`) | Count-only token requests (no generation) on non-sensitive fixture text, using the chat key each vendor already needs. | 3 token-count requests per credentialed vendor (prose, CJK, chat transcript); no generation, no streaming. |
91
91
  | `cli/live-journey` | planned | `PRISM_LIVE_PROVIDER_TESTS` + `OPENAI_API_KEY` | — | Packed CLI: init/provider-add/print/json/rpc over a real provider; transcript secret-scanned. | 1 pack + install, offline scaffold tests, <=3 wire prompts on the selected provider model (wire legs skip on 401/403). |
92
92
  | `memory/rag-rerankers-live` | active | any of: `PRISM_TEST_TEI_RERANKER_URL` / `PRISM_TEST_HOSTED_RERANK_URL`; optional: `PRISM_TEST_HOSTED_RERANK_URL` | `PRISM_LIVE_TEI_RERANKER_MODEL` (default `(endpoint default model)`), `PRISM_LIVE_HOSTED_RERANK_MODEL` (default `(endpoint default model)`) | Real TEI / OpenAI-compatible rerank endpoints; each leg self-skips when its endpoint env is unset. | 1 rerank request per configured endpoint (≤2 total). |
93
- | `memory/local-rerank-live` | active | `PRISM_TEST_LOCAL_RERANK`; optional: `PRISM_TEST_LOCAL_RERANK_CACHE_DIR` | `PRISM_LIVE_LOCAL_RERANK_MODEL` (default `Xenova/bge-reranker-base`) | In-process cross-encoder (transformers.js, q8/cpu) on this machine; no service and no credential. Weights come from the documented model id into a host cache dir; a cold run downloads a ≈280 MB int8 model. | One ≈280 MB model download on a cold cache; zero network after load, no API spend. ≈30 s wall clock after the cache is warm (the recall leg scores 24 × 96 query/document pairs). |
93
+ | `memory/local-rerank-live` | active | `PRISM_TEST_LOCAL_RERANK`; optional: `PRISM_TEST_LOCAL_RERANK_CACHE_DIR` | `PRISM_LIVE_LOCAL_RERANK_MODEL` (default `Xenova/bge-reranker-base`) | In-process cross-encoder (transformers.js, q8/cpu) plus a semantic embedder on this machine; no service and no credential. Weights come from the documented model ids into a host cache dir; a cold run downloads a ≈280 MB reranker and a ≈23 MB embedder. | Two model downloads on a cold cache (≈280 MB reranker + ≈23 MB embedder); zero network after load, no API spend. ≈50 s wall clock after the cache is warm (the recall legs score 24 × 96 query/document pairs three times). |
94
94
  | `memory/drive-sync-live` | active | `PRISM_TEST_DRIVE_ACCESS_TOKEN`; optional: `PRISM_TEST_DRIVE_FOLDER_ID` `PRISM_TEST_DRIVE_SHARED_DRIVE_ID` | — | Delegated Drive readonly token; least privilege: one throwaway folder. Folder/shared-drive ids optional. | <=2 Drive list/changes pages plus file media for that page; replay must not re-embed. |
95
95
  | `coding-tools/openapi-live` | active | `PRISM_LIVE_OPENAPI_TOOLS` | — | Real public OpenAPI 3.1 spec (warnely.com) + real GET tool calls; no credential. | 3 HTTP requests against example.com-class public hosts (plan budget ≤5). |
96
96
  | `coding-tools/computer-use-live` | active | `PRISM_TEST_COMPUTER_USE` + `PRISM_COMPUTER_USE_BIN` | — | Real host computer-use-linux MCP binary over stdio; real tool inventory + one bounded read-only screenshot. | Local desktop only; ≤30s ceiling. |
@@ -101,7 +101,6 @@ Set only the rows you want to run; everything else skips. Least-privilege scope
101
101
  | `core/artifact-bodies-s3-live` | active | `PRISM_TEST_S3_ENDPOINT` + `PRISM_TEST_S3_KEY` + `PRISM_TEST_S3_SECRET` + `PRISM_TEST_S3_BUCKET` | — | Dedicated throwaway S3-compatible bucket. | <=5 S3 requests (put/get/presign/delete x2). |
102
102
  | `cli/journey` | active | `PRISM_LIVE_PROVIDER_TESTS`; any of: `OPENAI_API_KEY` / `OPENROUTER_API_KEY` / `KIMI_API_KEY` / `ZAI_API_KEY` / `OPENCODE_API_KEY` / `NEURALWATT_API_KEY` / `DASHSCOPE_API_KEY` / `OLLAMA_API_KEY` | — | First init-catalog provider credential present in the environment. | 4-5 one-shot requests on the default catalog model. |
103
103
  | `memory/postgres` | active | `PRISM_TEST_POSTGRES_URL` | — | Postgres memory store + pgvector index round-trips on the operator database, including durable deletion propagation (1,000 derived rows tombstoned in one transaction) and grant-change re-pointing. | Bounded insert/query cycles against the configured Postgres. |
104
- | `memory/graft` | active | — (hermetic leg) | — | Graft upstream CLI child-process protocol (real binary spawns against the in-repo fixture graft bin). | Hermetic; no network. |
105
104
  | `memory/wiki` | active | — (hermetic leg) | — | Wiki lifecycle over real fs trees (init/refresh/lint/search fallback; qmd child-process client degrades without the binary). | Hermetic; no network. |
106
105
  | `coding-tools/lsp-forge` | active | — (hermetic leg) | — | LSP/language-intelligence + forge suites: real child-process spawns over the real LSP/forge wire protocols against fixture binaries. | Hermetic; no network. |
107
106
  | `ag-ui/conformance` | active | — (hermetic leg) | — | AG-UI + ACP conformance suites: real-event replay over the acp/a2a/ag-ui protocol surfaces (fixture agents, real event-source wire semantics). | Hermetic; no network. |
@@ -78,8 +78,9 @@ await createDeletionPropagator({ scope, vectorStore: store, authorization: princ
78
78
  and notes for other paths are untouched, and the write joins one store transaction when the store
79
79
  has one (a plain `upsert` otherwise).
80
80
  - Delete: the notes recorded against the deleted path are tombstoned through the store's own
81
- invalidation path (`invalidate`, batched at `HARD_INVALIDATION_BATCH`; reason `forgotten` by default,
82
- `legal_hold` when both the handler and the propagator are given it), so recall stops serving them
81
+ invalidation path (`invalidate`, batched at `HARD_INVALIDATION_BATCH`; the propagation's resolved
82
+ reason wins, `forgotten` by default, `legal_hold` stamps `hold: true` — the handler's own `reason`
83
+ option is only the fallback for a hand-built context), so recall stops serving them
83
84
  with no second revocation plane and no background cleanup to wait for.
84
85
  - One scope read per leg, selecting on `metadata.fabric.path` — never on content. A note in another
85
86
  scope is never visible, and a composition whose scope differs from the handler's is refused
@@ -0,0 +1,65 @@
1
+ # Migrate Prism 0.10 to 0.11
2
+
3
+ > **Status: 0.11.0** (store bounds, in-memory lease fence reset, persona and graft subpath removals).
4
+
5
+ This document is the host checklist for upgrading from Prism 0.10.0 to 0.11.0.
6
+
7
+ 0.11.0 is a lockstep minor for all **twelve** publishable packages. Node `>=22` stays the floor. No new public export was added by the review remediation. Two store contracts change for every host that uses the built-in memory or JSONL stores, or the in-memory lease store. Three subpaths that the published 0.10.0 tarball still shipped are removed.
8
+
9
+ ---
10
+
11
+ ## Behavior changes
12
+
13
+ ### 1. Idempotency dedup remembers 4,096 keys
14
+
15
+ The memory session store and the JSONL session store keep the latest 4,096 dedup keys (`sessionId`, `idempotencyKey`, `expectedParentId`). A replay inside that window is still rejected. A replay of a key that has fallen out of the window appends a new entry.
16
+
17
+ ```ts
18
+ // before — every dedup key was kept for the life of the store
19
+ await store.append(entry); // same idempotencyKey always rejected
20
+
21
+ // after — only the latest 4,096 keys are remembered
22
+ await store.append(older); // appends again once 4,096 newer keys have been seen
23
+ ```
24
+
25
+ **Migration actions**
26
+ - Do not treat idempotency rejection as permanent across a long-lived process. Persist the business result yourself if a replay after 4,096 later keys must stay a no-op.
27
+ - Durable SQLite and Postgres stores are unchanged by this window.
28
+
29
+ ### 2. An evicted in-memory lease starts at fencing 1
30
+
31
+ `docs/operations.md` used to say every expired row keeps its fencing counter. That remains true for the SQLite and Postgres lease adapters. The in-memory lease store now deletes expired rows once the map reaches 1,024, on write paths only.
32
+
33
+ ```ts
34
+ // before — a released key always inherited fencingToken + 1
35
+ const again = await leases.tryAcquireLease(sameKey); // fencingToken === previous + 1
36
+
37
+ // after — a key still in the map inherits + 1; a swept key starts at 1
38
+ const swept = await leases.tryAcquireLease(evictedKey); // fencingToken === 1
39
+ ```
40
+
41
+ **Migration actions**
42
+ - Do not assume an in-memory fencing token grows for the life of the process. A token of 1 can mean "first owner" or "owner after a sweep".
43
+ - A released key that has not been swept still inherits `fencingToken + 1`.
44
+ - Hosts on SQLite or Postgres need no change.
45
+
46
+ ### 3. Persona and graft subpaths are gone
47
+
48
+ `@arnilo/prism-coding-tools/caveman`, `@arnilo/prism-coding-tools/ponytail`, and `@arnilo/prism-memory/graft` are not in this line. The published 0.10.0 tarball still had them. Load persona skills with `loadSkillDirectory` from a host extension. Reach graft through a host MCP server or `registerTool` / `registerCommand`. `@arnilo/prism-coding-tools/impeccable` and the memory `/rag`, `/compaction/*`, `/fabric`, `/wiki`, and `/scoped` subpaths are unchanged. The name-level removal list is in [migration.md](migration.md).
49
+
50
+ ### Unchanged for hosts that only call the runtime
51
+
52
+ `readBranchPath` on the memory store and the JSONL parse cache change cost, not results, except for the same-size/same-mtime JSONL residual named above. Plain-text token estimates share one `ceil(length/4)` helper. Message and entry estimates are unchanged and are not billing numbers.
53
+
54
+ ---
55
+
56
+ ## Upgrade steps
57
+
58
+ 1. Move every `@arnilo/*` dependency and peer to `^0.11.0`.
59
+ 2. If you import `/caveman`, `/ponytail`, or `/graft`, switch to the host-owned seams before installing.
60
+ 3. If you persist idempotency by relying on an unbounded in-memory or JSONL dedup set, store the result yourself.
61
+ 4. If you compare in-memory lease fencing tokens for "never reused", treat a swept key's `1` as a new counter.
62
+
63
+ ## Rollback
64
+
65
+ Install the published `0.10.0` tarballs. That line still has the three subpaths, an unbounded idempotency set, and no in-memory lease sweep. Do not point `^0.11.0` peers at a `0.10.0` install.
package/docs/migration.md CHANGED
@@ -1,8 +1,19 @@
1
1
  # Migration guide
2
2
 
3
+ ## 0.10.0 → 0.11.0 (behavior persona and graft subpath removals)
4
+
5
+ **Prism 0.11.0 removes three public subpaths** that shipped through 0.10.0. Both capabilities stay supported on public seams — only the vendored convenience subpaths are gone. Compat baselines are regenerated for this line: 106 names removed (58 in `@arnilo/prism-memory`, 48 in `@arnilo/prism-coding-tools`) with no consumer-visible signature break; the only non-barrel signature changes are the widened `runCheckpointRestoreHooks` parameter and `describeBudgetExhaustion` return, plus two internal modules that no `exports` entry reaches. The 0.11.0 cut's own declaration change is `version` `"0.10.0"` → `"0.11.0"`. Plan 120 added no public name.
6
+
7
+ What a 0.10.0 host must check before upgrading:
8
+
9
+ - **`@arnilo/prism-coding-tools/caveman` and `/ponytail` are gone**, with their vendored upstream fixtures and the `@dietrichgebert/ponytail` optional peer. Load the upstream skill tree with the new `loadSkillDirectory(directory, { maxSkillBytes? })` on `@arnilo/prism/node/contribution-discovery`, then register the skills, a `/caveman`-style command, an every-turn injector, and session-entry mode persistence from a host extension — [examples/caveman-ponytail.ts](../examples/caveman-ponytail.ts) is the ported reference. `@arnilo/prism-coding-tools/impeccable` is unchanged.
10
+ - **`@arnilo/prism-memory/graft` is gone**, with its `@nanonets/graft` optional peer, the `prism-graft` fixture CLI, and the `/graft-init` / `/graft-build` / `/graft-build-deep` commands; `@arnilo/prism-memory` now declares only its required `@arnilo/prism` peer. Integrate graft as a host-exposed graft **MCP server**, or author the tools and commands against `registerTool` / `registerCommand` — the deleted extension was a subprocess CLI plus a retrieval-pack context provider plus blast-radius middleware, all host-composable. `/rag`, `/compaction/*`, `/fabric`, `/wiki`, and `/scoped` are unchanged.
11
+ - **Additive in the same line:** `CheckpointRestoreHandler` / `CheckpointRestoreCompensation` (a restore hook may now compensate) and the `BudgetExhaustionAttribution` on `describeBudgetExhaustion`'s return, `SubagentFailure` / `SubagentRecovery` / `SubagentRecoveryOutcome` lifecycle events, and `scorePrefixStability`. A host passing a plain restore hook to `runCheckpointRestoreHooks` needs no change — the new parameter type is that hook, widened.
12
+ - **Idempotency window and lease fence.** Memory and JSONL stores remember the latest 4,096 dedup keys; replaying an older key appends a new entry. The in-memory lease store deletes expired rows once the map reaches 1,024, and a swept key's next acquire starts at fencing 1. Durable SQLite and Postgres adapters still keep the counter. Full steps: [migrate-to-0.11.md](migrate-to-0.11.md).
13
+
3
14
  ## 0.9.0 → 0.10.0 (hook lifecycle completion, scoped agent memory)
4
15
 
5
- **Prism 0.10.0 is a lockstep minor for all twelve publishable packages** — `@arnilo/prism-hooks` is new. Node `>=22` stays the floor. Nothing was removed: no import path moved and no export was dropped (compat baseline: +47 names, zero removals, zero renames). Scoped memory is a new opt-in subpath that stays inert until a host constructs a policy.
16
+ **Prism 0.10.0 is a lockstep minor for all twelve publishable packages** — `@arnilo/prism-hooks` is new. Node `>=22` stays the floor. Nothing was removed in 0.10.0 itself: no import path moved and no export was dropped (compat baseline: +47 names, zero removals, zero renames); the persona and graft subpath removals recorded for this line land in **0.11.0** — see the section above. Scoped memory is a new opt-in subpath that stays inert until a host constructs a policy.
6
17
 
7
18
  What a 0.9.0 host must check before upgrading:
8
19
 
@@ -32,7 +32,7 @@ import { createJsonlSessionStore } from "@arnilo/prism/node/session-store-jsonl"
32
32
 
33
33
  `createJsonlSessionStore()` returns a `SessionStore`:
34
34
 
35
- - `append(entry, options?)` appends one JSON line, rejects duplicate entry ids, honors `expectedParentId` existence checks, and deduplicates exact idempotency retries within this store instance. Append **fails closed** when the file already contains any corrupt or shape-invalid line (`Invalid JSONL at line N: …`) so writers cannot extend a damaged log.
35
+ - `append(entry, options?)` appends one JSON line, rejects duplicate entry ids, honors `expectedParentId` existence checks, and deduplicates exact idempotency retries within this store instance (latest 4,096 keys; an older replay appends as a new entry). Append **fails closed** when the file already contains any corrupt or shape-invalid line (`Invalid JSONL at line N: …`) so writers cannot extend a damaged log.
36
36
  - `list(sessionId)` reads the file and returns valid entries for that session id. Corrupt or shape-invalid lines are skipped; they do not poison the whole file.
37
37
  - `get(id)` reads the file and returns the matching valid entry, if any.
38
38
  - `searchSessions(query)` reads the file and runs the shared linear session matcher. Corrupt or shape-invalid lines are quarantined exactly as in `list()`/`get()`, the contract linear caps bound sessions/entries/bytes scanned, and hits carry the same shape as the indexed adapters (`sessionId`, `leafId`, `entryId`, `runId`, `turn`, `snippet`) — without `score`, since a linear scan has no index relevance.
@@ -77,8 +77,9 @@ Use `createMemorySessionStore()` for tests or throwaway sessions; use the JSONL
77
77
  - Reads and writes use only the caller-provided path.
78
78
  - Errors include path/reason or line number, not file contents.
79
79
  - Do not put secrets in messages, metadata, summaries, labels, or custom entries.
80
- - Reads are linear in file size. Appends also re-read and re-parse the whole file for duplicate/parent/corruption checks before writing one line, and are serialized per store instance. A rejected append does not poison later appends; the rejected line is not written.
81
- - `searchSessions` is linear in file size too (there is no index): every query reads and parses the whole file before the capped scan, so latency and peak memory grow with the corpus. Use a SQLite/Postgres `SessionStore` when search latency matters, and treat search here as resume/filter tooling on small stores.
80
+ - Each store instance caches the parsed array keyed by `(size, mtimeMs)`. A stat match reuses it; a stat miss re-reads. Reads are linear in file size on a miss. `append()` refreshes the cache after a successful write so the next `list()` / `snapshot()` does not re-parse. A same-size rewrite inside one filesystem timestamp tick is not detected. `readJsonlSessionEntries()` does not use this cache.
81
+ - Appends still re-read and re-parse the whole file for duplicate/parent/corruption checks before writing one line, and are serialized per store instance. A rejected append does not poison later appends; the rejected line is not written.
82
+ - `searchSessions` has no index. A cache hit reuses the parsed array; a miss reads and parses the whole file before the capped scan, so latency and peak memory still grow with the corpus. Use a SQLite/Postgres `SessionStore` when search latency matters, and treat search here as resume/filter tooling on small stores.
82
83
  - There is no cross-process lock or durable idempotency table; two processes writing the same file can race. Add a database or external lock if multiple processes write the same file.
83
84
  - Treat this adapter as development/single-process storage. Production multi-writer hosts should use an indexed database `SessionStore` adapter.
84
85
 
@@ -20,7 +20,7 @@ This page is the operator runbook for the high-availability story proven by plan
20
20
 
21
21
  ## Outputs / response / events
22
22
 
23
- - A lease record: `{ namespace, key, ownerId, token, fencingToken, acquiredAt, expiresAt, updatedAt }`. Expired rows retain their fencing counter; the next owner inherits `fencingToken + 1`.
23
+ - A lease record: `{ namespace, key, ownerId, token, fencingToken, acquiredAt, expiresAt, updatedAt }`. Expired rows retain their fencing counter; the next owner inherits `fencingToken + 1`. The in-memory store deletes expired rows once it holds 1,024 of them; an evicted key's next owner starts at fencing 1. Durable adapters keep the counter.
24
24
  - A checkpoint record: `{ namespace, key, version, fencingToken?, value, createdAt, updatedAt }`. Cursor/value changes are CAS-committed; a peer can replay an unfinished step but can never skip ahead or move the cursor backward.
25
25
  - Failover timing: the drill reports `failoverMs` (wall time between the owner's death and the peer's acquisition) and asserts it against the frozen ceiling.
26
26
 
@@ -20,8 +20,6 @@ One row per declaration. `Unlocks` names the subpath whose import reaches the pe
20
20
  | Peer | Declared range | Optional | Declared by | Unlocks | Install | Network |
21
21
  | --- | --- | --- | --- | --- | --- | --- |
22
22
  | `zod` | `^3.25.0 \|\| ^4.0.0` | no | `@arnilo/prism-ag-ui` | `./acp` | `npm i zod` | no |
23
- | `@nanonets/graft` | `^0.16.0 \|\| ^0.18.0` | yes | `@arnilo/prism-memory` | `./graft` | `npm i @nanonets/graft` | no |
24
- | `@dietrichgebert/ponytail` | `^4.9.0` | yes | `@arnilo/prism-coding-tools` | `./ponytail` | `npm i @dietrichgebert/ponytail` | no |
25
23
  | `mammoth` | `^1.8.0` | yes | `@arnilo/prism-work` | `./document-reader` | `npm i mammoth` | no |
26
24
  | `pdf-parse` | `^2.4.5` | yes | `@arnilo/prism-work` | `./document-reader` | `npm i pdf-parse` | no |
27
25
  | `e2b` | `2.49.1` | yes | `@arnilo/prism-coding-tools` | `./security` | `npm i e2b@2.49.1` | yes |
@@ -47,7 +45,7 @@ Two peers are pinned to an exact version instead of a range, because the pin is
47
45
  `pg`, `@nats-io/jetstream`, `@nats-io/transport-node`, `playwright-core`, and `e2b` open sockets. For a supply-chain review of those five:
48
46
 
49
47
  - **Connection targets are host-owned.** Every one of them is passed a host-supplied connection string, endpoint list, browser instance, API key, or service URL. Prism holds no default endpoint, and no peer is reachable from the root import.
50
- - **Bytes stay local otherwise.** `better-sqlite3`, `mammoth`, `pdf-parse`, `@nanonets/graft`, and `@dietrichgebert/ponytail` are filesystem/process peers; the remaining two (`zod`, `@ai-sdk/provider`) are pure types/schemas.
48
+ - **Bytes stay local otherwise.** `better-sqlite3`, `mammoth`, and `pdf-parse` are filesystem/process peers; the remaining two (`zod`, `@ai-sdk/provider`) are pure types/schemas.
51
49
  - **No secrets are read by the peers.** Prism resolves credentials through host providers and redacts them at the boundary; peers only ever receive a resolved connection string or model object. See [Credentials and redaction](credentials-and-redaction.md) and [Host security guide](host-security.md).
52
50
  - **Nothing is installed implicitly.** Optional peers are never auto-installed by npm; a missing one fails closed at the call site with a typed error naming the peer and the subpath. Required peers (today only `zod`) are installed by npm with the package.
53
51
 
@@ -79,7 +77,7 @@ const tools = await createBrowserTools({ browser });
79
77
 
80
78
  ## Extension and configuration notes
81
79
 
82
- - A peer is an *implementation the host owns*. When a peer's default wiring is not what you want, pass your own implementation instead of installing theirs: the document reader accepts host parsers (`createReadTool({ documentReader })`), the memory `/graft` resolver accepts an explicit package root, and the browser surfaces accept a host `Browser`.
80
+ - A peer is an *implementation the host owns*. When a peer's default wiring is not what you want, pass your own implementation instead of installing theirs: the document reader accepts host parsers (`createReadTool({ documentReader })`), and the browser surfaces accept a host `Browser`.
83
81
  - Subpaths that need a peer isolate that import, so importing another subpath of the same package never evaluates it. The office family is the extreme case: zero peers, because it takes structural inputs. `@arnilo/prism-channels` also has no third-party peers: Telegram uses native `fetch`, and signal-cli is a host-operated binary rather than an npm peer.
84
82
  - Adding a peer to a Prism package is a release-gated change: the declaration must be optional unless a hard dependency's own peer forces it (the `zod` case), and exact pins must come with a version-gate or compatibility rationale.
85
83
 
@@ -93,4 +91,4 @@ const tools = await createBrowserTools({ browser });
93
91
 
94
92
  - [Release and install](release-and-install.md): install profiles that pair with each peer.
95
93
  - [Configuration options index](options-index.md): the option surfaces each peer unlocks.
96
- - Package-level detail: [Coding tools](coding-tools.md), [Core runtime](core.md), [Session stores](session-stores.md), [Browser automation](browser-automation.md), [Document reader](document-reader.md), [Graft](graft.md), [Ponytail](ponytail.md), [Provider packages](provider-packages.md), [Messaging channels](messaging-channels.md).
94
+ - Package-level detail: [Coding tools](coding-tools.md), [Core runtime](core.md), [Session stores](session-stores.md), [Browser automation](browser-automation.md), [Document reader](document-reader.md), [Provider packages](provider-packages.md), [Messaging channels](messaging-channels.md).
@@ -118,7 +118,7 @@ Memory retrieval keeps its own audit events next to policy decisions; hosts forw
118
118
  | `rag.acl_denied` | `{ sourceId, scope: { tenantId, resourceId, threadId }, reason: "no_grant" \| "check_failed", hits, error? }` via `retrieveContext({ onAccessDenied })` | A source was withheld: revoked/absent/version-mismatched grant, the grant lookup threw (`error` is redacted, capped at 256 chars), or the store's own predicate filtered it before ranking (`hits: 0`, reported only when the host wires `onDeniedSources` through `retrieveContext`, plan 102 Task 6) |
119
119
  | `Repointed` log line + result | `repointSource()` → `{ from, to, movedChunks, rewrittenEdges, layers, batched }` | A source's grant identity moved and derived artifacts followed |
120
120
  | `rag.repointed` (rename audit) | `applySourceRenames({ onRenamed })` → `{ from, to, outcome: "moved", movedChunks, rewrittenEdges, layers }` \| `{ from, to, outcome: "failed", error }` | A batch of identity moves ran: one event per rename that settled, successes and failures alike (`error` redacted and capped at 256 chars) — plan 102 Task 7 |
121
- | Invalidation rows | `store.invalidate()` rows (`{ id, reason: "corrected" \| "revoked" \| "forgotten" \| "legal_hold", at }`) read back by `listInvalidatedIds()` | A source was revoked/forgotten/held; tombstones stay for explainability |
121
+ | Invalidation rows | `store.invalidate()` rows (`{ id, reason: "corrected" \| "revoked" \| "forgotten" \| "legal_hold", at }`) read back by `listInvalidatedIds()` | A source was revoked/forgotten/held; tombstones stay for explainability. Handler tombstones (e.g. fabric notes) mirror the propagation's resolved reason — a `legal_hold` walk never lands as `forgotten` |
122
122
 
123
123
  Events are per *source*, not per hit, and are emitted once per query. They never contain document text, grant contents, or credentials; `check_failed` messages pass through the same redactor as retrieved content. Denials are fail-closed: a source is excluded whether the grant is absent, revoked, or the lookup failed, and the query returns the remaining hits. Aborts are not denials and are never recorded as such.
124
124
 
@@ -7,12 +7,16 @@ Prefix stability conformance drives a real agent session through two staggered s
7
7
  Exported from `@arnilo/prism/testing/prefix-stability-conformance`:
8
8
 
9
9
  - `runPrefixStabilityConformance(options)`
10
+ - `scorePrefixStability(requests, options?)` — score a capture the host already holds; no session, no provider call
10
11
  - `PrefixStabilityConformanceOptions`
11
12
  - `PrefixStabilityConformanceResult`
13
+ - `ScorePrefixStabilityOptions`
14
+ - `PrefixStabilitySample`
15
+ - `PrefixStabilityResetDetail`
12
16
 
13
17
  ## When to use it
14
18
 
15
- Use it when a host owns any part of prompt assembly — custom `inputBuilder`, `promptBuilder`, context providers, instruction injectors, input/prompt middleware, or an explicit `inputLayout` — and wants to prove that progressive disclosure still holds the cache prefix. The runner:
19
+ Use it when a host owns any part of prompt assembly — custom `inputBuilder`, `promptBuilder`, context providers, instruction injectors, input/prompt middleware, or an explicit `inputLayout` — and wants to prove that progressive disclosure still holds the cache prefix. A host that already holds captured `ProviderRequest`s scores them with `scorePrefixStability` instead of re-running the fixture; pass the session's `tailSegments` map if you have it. A wrong or empty map yields a wrong number, not an error, and never a fabricated `1`. The runner:
16
20
 
17
21
  - installs a fixture provider (no network) that loads `skills[0]` on the first turn and `skills[1]` on the second, two provider requests per turn — and, when the host runs an attention compiler, carries a deterministic reasoning block per skill-load round so the compiler's thinking stage has real content to fold;
18
22
  - keeps everything else in `host` exactly as production: system prompt, context providers, builders, middleware, disclosure settings;
@@ -41,21 +45,24 @@ const result = await runPrefixStabilityConformance({
41
45
  - `skills` — exactly two distinct `Skill` values with non-empty `instructions`, loaded in turn order
42
46
  - `minContinuity?` — minimum shared-prefix fraction between consecutive requests (default `0.95`); always measured against the provider-visible prefix
43
47
  - `assertOn?` — `"providerPrefix"` (default) gates the run on the provider-visible prefix; `"cacheablePrefix"` gates it on the tail-aware measurement instead, for an eager or body-heavy host that deliberately re-sends bodies after the stable prefix
44
- - `allowedResets?` — how many request pairs may break below `minContinuity` (default `0`, today's behavior). Set `1` for an assembly that folds, compacts, or evicts exactly one boundary; more resets than declared fail, and fewer fail too, because a fixture that was supposed to invalidate the prefix and never did cannot pass vacuously
48
+ - `allowedResets?` — how many request pairs may break below `minContinuity` (default `0`, today's behavior). Set `1` for an assembly that folds, compacts, or evicts exactly one boundary; more resets than declared fail, and fewer fail too, because a fixture that was supposed to invalidate the prefix and never did cannot pass vacuously. Each allowed gap is also a `resetDetails` row
49
+ - `foldableToolResultBytes?` — installs a runner-owned `prefix_stability_bulk` fixture tool that returns exactly this many bytes of generated text and adds a second tool call for it beside `load_skill` in the same round, so the attention compiler's tool-result stage has a foldable row. Unset, the capture is byte-identical to today (four requests, one `load_skill` call each)
45
50
  - `inputs?` — the two turn inputs (default fixed strings, so runs stay comparable across hosts)
46
51
 
47
52
  ## Outputs / response / events
48
53
 
49
- Returns `Promise<{ requests: number; minContinuity: number; cacheableContinuity: number; resets: readonly number[] }>`: the captured request count (four) and the two lowest shared-prefix fractions observed. Throws a plain `Error` naming the offending request pair and the measured percentage on the first violation. No events, no test runner, no network.
54
+ Returns `Promise<{ requests: number; minContinuity: number; cacheableContinuity: number; resets: readonly number[]; resetDetails: readonly PrefixStabilityResetDetail[] }>`: the captured request count (four), the two lowest shared-prefix fractions, and one detail row per reset. Throws a plain `Error` naming the offending request pair and the measured percentage on the first violation. No events, no test runner, no network. Detail rows are indexes and fractions only — no message text.
50
55
 
51
- A gap below `minContinuity` **on the metric `assertOn` selects** is collected as a reset instead of failing inside the loop: `resets` holds the 1-based index of the request that broke (the later request of the pair, ascending). With the default `allowedResets: 0` the first reset fails the run exactly as before, now adding the observed reset list to the message; `allowedResets: 1` lets a single documented boundary (an attention fold, a compaction, a budget eviction) pass while every other pair must stay byte-stable.
56
+ A gap below `minContinuity` **on the metric `assertOn` selects** is collected as a reset instead of failing inside the loop: `resets` holds the 1-based index of the request that broke (the later request of the pair, ascending). `resetDetails[i].request` equals `resets[i]`; `fraction` is that selected metric and `cacheableFraction` is the tail-excluded fraction for the same pair. A run with no gap returns `[]` for both. With the default `allowedResets: 0` the first reset fails the run exactly as before, now adding the observed reset list to the message; `allowedResets: 1` lets a single documented boundary (an attention fold, a compaction, a budget eviction) pass while every other pair must stay byte-stable, and the result carries exactly that one detail row.
52
57
 
53
58
  Both numbers measure the same consecutive request pairs, byte for byte:
54
59
 
55
60
  - **`minContinuity`** — the provider-visible prefix, messages **and** tool schemas as sent on the wire. This is what the prompt cache can keep paying for, and its meaning is frozen: `0.95` default, unchanged by `assertOn`.
56
61
  - **`cacheableContinuity`** — the same fraction recomputed after removing this session's tail segments (loaded skill bodies, resources moved to the tail) from **both** requests of each pair. A host whose provider-visible fraction dips only because of tail bodies reads `1` here.
57
62
 
58
- Tail segments are read from the session's own `tailSegments` map — the exact `Message` objects assembly appended — matched by object identity first and by serialized-value equality for a `promptBuilder` that clones messages. Nothing is added to the provider payload and no host content is pattern-matched. When no captured request carried a tail segment (a builder that renders bodies elsewhere), `cacheableContinuity` equals `minContinuity`; it is not reported as `1`.
63
+ Tail segments are read from the session's own `tailSegments` map — the exact `Message` objects assembly appended — matched by object identity first and by serialized-value equality for a `promptBuilder` that clones messages. Nothing is added to the provider payload and no host content is pattern-matched. When no captured request carried a tail segment (a builder that renders bodies elsewhere), `cacheableContinuity` equals `minContinuity`; it is not reported as `1`. `scorePrefixStability` uses the same rule: omit `tailSegments`, or pass a map that matches nothing, and the two fractions stay equal.
64
+
65
+ `scorePrefixStability(requests, options?)` returns `{ minContinuity, cacheableContinuity, resets, resetDetails }` for a list the host already captured. `resets` follows the provider-visible prefix unless `options.assertOn` is `"cacheablePrefix"` (the runner passes its own `assertOn` so the two surfaces share one measurement). `resetDetails[i].request` equals `resets[i]`; each row is `{ request, fraction, cacheableFraction }` — indexes and fractions only, never message text. Fewer than two requests throws a plain `Error` naming the count.
59
66
 
60
67
  ## Request/response example
61
68
 
@@ -75,7 +82,7 @@ const { minContinuity, cacheableContinuity, resets } = await runPrefixStabilityC
75
82
  ## Implementation example
76
83
 
77
84
  ```ts
78
- import { runPrefixStabilityConformance } from "@arnilo/prism/testing/prefix-stability-conformance";
85
+ import { runPrefixStabilityConformance, scorePrefixStability } from "@arnilo/prism/testing/prefix-stability-conformance";
79
86
 
80
87
  // The runner owns the provider and skills, so the same helper is the negative control too:
81
88
  // add a deliberately volatile context provider to prove the assertion can fail.
@@ -86,6 +93,12 @@ await runPrefixStabilityConformance({
86
93
  },
87
94
  skills: [alphaSkill, betaSkill],
88
95
  });
96
+
97
+ // A host that captures its own provider requests (middleware, proxy, or an adapter tap):
98
+ const sample = scorePrefixStability(captured, { tailSegments: sessionTails, minContinuity: 0.95 });
99
+ // → { minContinuity: 0.98, cacheableContinuity: 1, resets: [], resetDetails: [] }
100
+ // With no tail map a deliberate tail re-send counts against the score:
101
+ // → { minContinuity: 0.71, cacheableContinuity: 0.71, resets: [2], resetDetails: [{ request: 2, fraction: 0.71, cacheableFraction: 0.71 }] }
89
102
  ```
90
103
 
91
104
  ## Extension and configuration notes
@@ -113,8 +126,11 @@ await runPrefixStabilityConformance({
113
126
  allowedResets: 1, // add `assertOn: "cacheablePrefix"` when tail re-sends should not count either
114
127
  });
115
128
  result.resets; // [3] — the request after the fold; every other pair stayed append-only
129
+ result.resetDetails; // [{ request: 3, fraction, cacheableFraction }] — same gap, both fractions
116
130
  ```
117
131
 
132
+ - Both compiler stages are proven by the runner. The deterministic reasoning block proves the thinking stage; `foldableToolResultBytes` proves the **tool-result** stage — the bulk row becomes the stub at the reset while the sibling `load_skill` confirmation stays byte-identical (the shrink guard leaves the small row alone). The fixture tool is installed only when the option is set, and its payload is generated text, never host content.
133
+
118
134
  - Documented invalidation boundaries — what `resets` is expected to name. Each row is pinned by a fixture in [`src/__tests__/invalidation-inventory.test.ts`](../src/__tests__/invalidation-inventory.test.ts) that asserts the boundary *position* (message index, and tool index for schemas), so a reordering of the cache-aware layout fails that suite instead of silently relocating a boundary:
119
135
 
120
136
  | Segment | Boundary in the default `cache_aware` layout | Owner |
@@ -124,11 +140,18 @@ await runPrefixStabilityConformance({
124
140
  | Observational-memory blocks (`observational-memory`, `recent-messages`) | The same context slot; re-rendering identical blocks keeps the prefix byte-identical | [Context and skills](context-and-skills.md) |
125
141
  | Compaction summaries | Right after the leading system prompt while nothing user-role precedes it; a leading attachment moves the summary behind the context and skill slots | [Input and prompt assembly](input-and-prompt-assembly.md) |
126
142
  | Pending tool results / current input | The suffix: a result inserts immediately before the current input, so that input is the round's boundary | [Input and prompt assembly](input-and-prompt-assembly.md) |
127
- | `contextBudget` eviction | The first evicted group in the documented drop order (tool results → history → summaries → context → skills → attachments) | [Input and prompt assembly](input-and-prompt-assembly.md) |
143
+ | `contextBudget` eviction | The first evicted group in the documented drop order (tool results → history → summaries → context → skills → attachments). Tool results, history, the lowest-priority context block, skill-body demotion, and the newest attachment are pinned at their message index in [`src/__tests__/invalidation-inventory.test.ts`](../src/__tests__/invalidation-inventory.test.ts); the full drop order stays behavioral in [`src/__tests__/context-budget.test.ts`](../src/__tests__/context-budget.test.ts) | [Input and prompt assembly](input-and-prompt-assembly.md) |
128
144
  | Tool-schema selection | `request.tools` only: gaining or losing a schema leaves every message byte-identical, while a changed description is a boundary at that schema | [Provider caching](provider-caching.md) |
129
145
  | Attention-compiler / tool-result fold | In place at the fold frontier: the oldest stripped or stubbed row is the boundary — one reset, declared with `allowedResets` | [Attention compiler](attention-compiler.md) |
130
146
  | Skill bodies and URI resources (tail) | After the transcript, append-only: a newly loaded body never invalidates the prefix and `cacheableContinuity` stays `1` | [Input and prompt assembly](input-and-prompt-assembly.md) |
131
147
 
148
+ - Switching `inputLayout` is an explicit, documented invalidation, not a stability claim — the whole
149
+ order moves ([`legacy` includes the resolved context and skill slots ahead of the instruction
150
+ groups](input-and-prompt-assembly.md)). The same fixtures are pinned under both `cache_aware` and
151
+ `legacy` in [`src/__tests__/invalidation-inventory.test.ts`](../src/__tests__/invalidation-inventory.test.ts):
152
+ the injector-context boundary is message 2 under `cache_aware` and message 1 under `legacy`, and the
153
+ summary boundary is 1 and 3. A layout change that relocates either boundary fails that suite.
154
+
132
155
  ## Security and performance notes
133
156
 
134
157
  - No credentials, no network, no real skills required; the fixture provider is a local generator.
@@ -25,26 +25,26 @@ Do not use provider packages as a package manager, credential store, env loader,
25
25
 
26
26
  | adapter package | version |
27
27
  | --- | --- |
28
- | `@arnilo/prism-providers/ai-sdk` | 0.10.0 |
29
- | `@arnilo/prism-providers/alibaba` | 0.10.0 |
30
- | `@arnilo/prism-providers/anthropic` | 0.10.0 |
31
- | `@arnilo/prism-providers/azure` | 0.10.0 |
32
- | `@arnilo/prism-providers/bedrock` | 0.10.0 |
33
- | `@arnilo/prism-providers/clinepass` | 0.10.0 |
34
- | `@arnilo/prism-providers/commandcode` | 0.10.0 |
35
- | `@arnilo/prism-providers/deepseek` | 0.10.0 |
36
- | `@arnilo/prism-providers/google` | 0.10.0 |
37
- | `@arnilo/prism-providers/hyper` | 0.10.0 |
38
- | `@arnilo/prism-providers/kimi` | 0.10.0 |
39
- | `@arnilo/prism-providers/model-discovery` | 0.10.0 |
40
- | `@arnilo/prism-providers/neuralwatt` | 0.10.0 |
41
- | `@arnilo/prism-providers/ollama` | 0.10.0 |
42
- | `@arnilo/prism-providers/openai` | 0.10.0 |
43
- | `@arnilo/prism-providers/opencode-go` | 0.10.0 |
44
- | `@arnilo/prism-providers/openrouter` | 0.10.0 |
45
- | `@arnilo/prism-providers/vertex` | 0.10.0 |
46
- | `@arnilo/prism-providers/xai` | 0.10.0 |
47
- | `@arnilo/prism-providers/zai` | 0.10.0 |
28
+ | `@arnilo/prism-providers/ai-sdk` | 0.11.0 |
29
+ | `@arnilo/prism-providers/alibaba` | 0.11.0 |
30
+ | `@arnilo/prism-providers/anthropic` | 0.11.0 |
31
+ | `@arnilo/prism-providers/azure` | 0.11.0 |
32
+ | `@arnilo/prism-providers/bedrock` | 0.11.0 |
33
+ | `@arnilo/prism-providers/clinepass` | 0.11.0 |
34
+ | `@arnilo/prism-providers/commandcode` | 0.11.0 |
35
+ | `@arnilo/prism-providers/deepseek` | 0.11.0 |
36
+ | `@arnilo/prism-providers/google` | 0.11.0 |
37
+ | `@arnilo/prism-providers/hyper` | 0.11.0 |
38
+ | `@arnilo/prism-providers/kimi` | 0.11.0 |
39
+ | `@arnilo/prism-providers/model-discovery` | 0.11.0 |
40
+ | `@arnilo/prism-providers/neuralwatt` | 0.11.0 |
41
+ | `@arnilo/prism-providers/ollama` | 0.11.0 |
42
+ | `@arnilo/prism-providers/openai` | 0.11.0 |
43
+ | `@arnilo/prism-providers/opencode-go` | 0.11.0 |
44
+ | `@arnilo/prism-providers/openrouter` | 0.11.0 |
45
+ | `@arnilo/prism-providers/vertex` | 0.11.0 |
46
+ | `@arnilo/prism-providers/xai` | 0.11.0 |
47
+ | `@arnilo/prism-providers/zai` | 0.11.0 |
48
48
  <!-- generated:package-truth:providers end -->
49
49
 
50
50
 
@@ -147,7 +147,7 @@ Important request shapes:
147
147
  | `PrismManifest` | Data-only package manifest with config defaults, contribution declarations, and resource declarations. |
148
148
  | `ProductionPersistenceStore` | Adapter-facing interface for durable, paginated, multi-tenant storage plus optional `checkpoints?: CheckpointStore`, `leases?: LeaseStore`, and `feedback?: RunFeedbackStore`. No SQL/ORM/host file storage/network dependency. |
149
149
  | `CheckpointStore` | Generic versioned checkpoint capability: save/load/bounded-list/delete by namespace and key, with ownership, exact-version CAS, and lease fencing. `createMemoryCheckpointStore()` is the reference implementation; it is bounded — `maxRecords` (default 10,000, evicts least-recently-saved) and `maxValueBytes` (default 1 MiB per JSON value). |
150
- | `LeaseStore` | Atomic acquire/renew/release/get by namespace and key, with opaque claim tokens, expiry, ownership scope, and monotonically increasing takeover fences. `createMemoryLeaseStore()` is the reference implementation. |
150
+ | `LeaseStore` | Atomic acquire/renew/release/get by namespace and key, with opaque claim tokens, expiry, ownership scope, and takeover fences that increase while the row remains. `createMemoryLeaseStore()` sweeps expired rows at 1,024 and an evicted key restarts at fencing 1; durable adapters keep the counter. |
151
151
  | `RunFeedbackStore` | Immutable append, bounded owned query, and owned deletion for ratings/comments/tags linked to existing run/trace/evaluation IDs. `createMemoryRunFeedbackStore()` is the reference implementation. |
152
152
  | `EventMultiplexer<T>` | Generic bounded fan-in from async sources. `createEventMultiplexer()` owns queue limits, overflow policy, abort, source teardown, and close behavior. Graceful `close()` stops publishes/sources and drains already-queued events before the subscriber completes; overflow `close` still emits one notice and terminates. Single-consumer contract: a second concurrent `subscribe()` throws `EventMultiplexerError` (`ERR_PRISM_EVENT_MULTIPLEXER_SINGLE_CONSUMER`); the slot frees when the active consumer completes or is `return()`ed at a yield. `observe` fan-in is unchanged (broadcast happens at the source). |
153
153
  | `PersistencePage<T>` | Cursor-paginated result page: `items`, optional `nextCursor`, optional `total`. |
package/docs/rag.md CHANGED
@@ -109,7 +109,7 @@ const result = await propagator.propagate("doc:erp-lead");
109
109
  // { sourceId, ids, tombstoned, layers: { rag: 4, wiki: 1 }, batched: true }
110
110
  ```
111
111
 
112
- - `propagate(sourceId)` expands the source through `_lineage.sourceIds` (`collectInvalidationIds`, depth 8) into a closed id set, tombstones **all** of it with reason `forgotten` inside one store transaction, then runs every registered handler with `{ sourceId, ids, scope, signal }`. Handlers return how many artifacts they removed (reported per `kind` in `layers`).
112
+ - `propagate(sourceId)` expands the source through `_lineage.sourceIds` (`collectInvalidationIds`, depth 8) into a closed id set, tombstones **all** of it with the propagator's resolved reason (`forgotten` by default; `legal_hold` stamps `hold: true`) inside one store transaction, then runs every registered handler with `{ sourceId, ids, scope, reason, signal }`. Handlers return how many artifacts they removed (reported per `kind` in `layers`). The context `reason` is the single source of truth: a handler's own `reason` option is only the fallback for a hand-built context, so a `legal_hold` propagation cannot land in a handler's tombstones as `forgotten`.
113
113
  - Tombstones, not deletions, for derived rows: rows stay for explainability (`recall({ explain: true })` reports the invalidation), and lineage links never dangle. Handlers own physical removal (chunk rows, files, ledger entries).
114
114
  - Retrieval is belt-and-suspenders: `retrieveContext()` reads per-scope invalidations before assembly and drops any candidate whose record id, `_lineage.sourceIds`, or `_rag.sourceId` is tombstoned — so a delete that lands after the query legs read rows still returns zero hits. The split matters for direct store users: the store's own SQL predicate filters by record id and `_lineage` edge, while a source's *own* chunk rows are covered by the `_rag.sourceId` rule at the retrieval boundary (or removed physically by the `rag` handler) — a raw `store.query()` is not a recall path.
115
115
  - `HARD_PROPAGATION_EDGES` (4,096) is the one-pass privileged ceiling; over it the whole delete rejects (fail-closed), never a half-tombstoned document. Each store `invalidate` call carries at most `HARD_INVALIDATION_BATCH` (64) entries. On a durable store that shape holds: PostgreSQL/pgvector tombstones 1,001 rows (1,000 derived chunk rows + the source root) in **one transaction and 22 statements** (16 of them `HARD_INVALIDATION_BATCH`-sized `INSERT`s), measured at **29–155 ms** across runs on an AMD Ryzen 9 PRO 7940HS against `pgvector/pgvector:pg16` (more under parallel load) — the durable counterpart of the in-memory suite's 1k-under-2s check, and evidence rather than a gate. Re-run it with `PRISM_TEST_POSTGRES_URL=… npm run test:postgres` (`packages/memory/src/__tests__/postgres-propagation.integration.test.ts`); the leg also proves the store's own SQL predicate hides the tombstoned rows, not only the in-app guard, and that a denied propagation opens no transaction at all.
@@ -260,7 +260,7 @@ const result = await retrieveContext("How do approvals work?", { embedder, store
260
260
  ```
261
261
 
262
262
  - `resolveReranker({ kind: "local" })` is the zero-config path. The model runtime is a host seam exactly like `Embedder`: `createLocalReranker({ model?, runtime?, onLoad?, cacheDir?, dtype?, device?, allowRemoteModels? })`. Pass `runtime: { load(model) → { id, score({ query, documents, signal }) } }` to inject a runtime the host already owns (transformers.js, onnxruntime-node, llama.cpp). With no `runtime`, the built-in loader resolves `@huggingface/transformers` at first use — the package declares no inference dependency (no new dependency name in any manifest) and nothing resolves it at build/install time.
263
- - Sizing trade-off: the download is one-time and host-cached, and per-query latency is CPU-bound and grows with candidates × tokens, so keep `topK`/`queryCandidates` near what recall actually needs — the reranker reorders what retrieval returned, it cannot recover a chunk the candidate pool never returned. Measured on the phase 102 corpus (24 queries / 96 chunks: one answering chunk + three mention-only chunks per query, k=5, `Xenova/bge-reranker-base` q8 on x86 CPU, deterministic lexical `createHashEmbedder` baseline, vector-only): recall@5 **0.21 → 0.79**, top-50 median **119–289 ms**, and at the package default 20-candidate pool recall@20 was 0.63 before reranking — corpus, misses, pool-bound numbers, latency, and cache state live in [`docs/_evidence/phase102-local-rerank-latency.md`](_evidence/phase102-local-rerank-latency.md), regenerated by `PRISM_TEST_LOCAL_RERANK=1 npm run test:live`. Treat the numbers as one data point on one machine, not a ceiling: dtype, device, and the embedder move them (a semantic embedder starts higher and gains less). The package guarantees the plumbing (one lazy load, one batched score call per rerank), not the model's speed. The hosted/TEI adapters stay for scale (higher throughput, no local RAM, no download).
263
+ - Sizing trade-off: the download is one-time and host-cached, and per-query latency is CPU-bound and grows with candidates × tokens, so keep `topK`/`queryCandidates` near what recall actually needs — the reranker reorders what retrieval returned, it cannot recover a chunk the candidate pool never returned. Measured on one corpus (24 queries / 96 chunks: one answering chunk + three mention-only chunks per query, k=5, `Xenova/bge-reranker-base` q8 on x86 CPU, vector-only) with two embedders: the deterministic lexical `createHashEmbedder` baseline gives recall@5 **0.21 → 0.79** and recall@20 **0.63** at the package default 20-candidate pool, while the semantic `Xenova/all-MiniLM-L6-v2` q8/cpu (384 dims) gives **0.79 → 0.79** and recall@20 **1.00** — the semantic baseline starts at the lexical reranked number, the reranker's lift is 0.000 on this corpus, and the pool is not the binding constraint. Read the row for your own embedder: with a lexical/deterministic embedder the pool bound is what to raise first; with a semantic one the reranker is ordering quality only and the residual misses need better retrieval. Top-50 median was 95–289 ms across runs. Corpus, misses, pool-bound numbers, latency, cache state, and the semantic side-by-side live in [`docs/_evidence/phase111-reranker-semantic-recall.md`](_evidence/phase111-reranker-semantic-recall.md), with the lexical control in [`docs/_evidence/phase102-local-rerank-latency.md`](_evidence/phase102-local-rerank-latency.md); both are regenerated by `PRISM_TEST_LOCAL_RERANK=1 npm run test:live`. Treat the numbers as one data point on one machine, not a ceiling: dtype, device, and the embedder move them. The non-CPU (fp16/GPU) leg is host-provisioned — re-measure before expecting the CPU numbers to hold. The package guarantees the plumbing (one lazy load, one batched score call per rerank), not the model's speed. The hosted/TEI adapters stay for scale (higher throughput, no local RAM, no download).
264
264
  - Host defaults: `dtype: "q8"` with `device: "cpu"` on x86 — fp32 weights are roughly 4× the download for no measurable ranking gain in this size class, and fp16/GPU is worth opting into only when the host already provisions it. Weights are cached per host: pass one `cacheDir` (e.g. `~/.cache/prism/models`) and the runtime lays out one subdirectory per model id, so a second model or a second process reuses the same files — point local embedders running through the same runtime at that directory too. With `cacheDir` omitted the runtime's own default cache applies (inside the installed package). On a cache miss the model is downloaded once into that directory and later runs stay on disk: add `allowRemoteModels: false` on an offline host to fail instead of reaching the model registry, which is exactly what the live leg's second pass proves.
265
265
  - Cheap by construction: the model loads lazily once per reranker instance, `score` is called once per rerank with every candidate (never one call per document), and `onLoad({ model, loadMs })` is the only opt-in observability — no document text is ever logged. Zero network after load; the built-in loader only touches the model registry at load time, and `allowRemoteModels: false` pins it to local files.
266
266
  - Failure is loud: a missing runtime, an unreachable model, or a runtime that returns no per-document scores throws a redacted `RagValidationError` naming the model and the install path (`npm i @huggingface/transformers` or pass `{ runtime }`). There is deliberately **no** silent lexical fallback.