@arnilo/prism 0.8.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (141) hide show
  1. package/CHANGELOG.md +62 -1
  2. package/README.md +13 -12
  3. package/dist/agent-approval.d.ts +17 -2
  4. package/dist/agent-approval.js +15 -6
  5. package/dist/agent-event-source.d.ts +9 -1
  6. package/dist/agent-event-source.js +10 -3
  7. package/dist/agent-loops.js +7 -4
  8. package/dist/agent-run-lifecycle.d.ts +15 -1
  9. package/dist/agent-run-lifecycle.js +82 -11
  10. package/dist/agent-run-state.d.ts +47 -6
  11. package/dist/agent-run-state.js +154 -6
  12. package/dist/agent-session/event-subscriber.d.ts +2 -0
  13. package/dist/agent-session/event-subscriber.js +3 -0
  14. package/dist/agent-session/helpers.js +14 -0
  15. package/dist/agent-session/session/assemble.js +281 -32
  16. package/dist/agent-session/session/persist.d.ts +11 -0
  17. package/dist/agent-session/session/persist.js +48 -16
  18. package/dist/agent-session/session/provider-round.d.ts +14 -4
  19. package/dist/agent-session/session/provider-round.js +226 -19
  20. package/dist/agent-session/session/tool-round.d.ts +2 -2
  21. package/dist/agent-session/session/tool-round.js +78 -6
  22. package/dist/agent-session/session/types.d.ts +44 -3
  23. package/dist/agent-session/session.d.ts +100 -5
  24. package/dist/agent-session/session.js +224 -13
  25. package/dist/attention-compiler.d.ts +51 -2
  26. package/dist/attention-compiler.js +282 -21
  27. package/dist/cache-helpers.d.ts +4 -2
  28. package/dist/cache-helpers.js +8 -6
  29. package/dist/checkpoint-restore.d.ts +45 -0
  30. package/dist/checkpoint-restore.js +54 -0
  31. package/dist/context-budget.d.ts +13 -1
  32. package/dist/context-budget.js +57 -4
  33. package/dist/contracts-core/agent.d.ts +52 -1
  34. package/dist/contracts-core/attention.d.ts +95 -0
  35. package/dist/contracts-core/content.d.ts +10 -0
  36. package/dist/contracts-core/extensions.d.ts +3 -0
  37. package/dist/contracts-core/guardrail-packs.d.ts +46 -0
  38. package/dist/contracts-core/guardrail-packs.js +2 -0
  39. package/dist/contracts-core/loop.d.ts +36 -0
  40. package/dist/contracts-core/provider.d.ts +30 -0
  41. package/dist/contracts-core/run-limits.d.ts +29 -1
  42. package/dist/contracts-core/session.d.ts +23 -5
  43. package/dist/contracts-core/session.js +21 -2
  44. package/dist/contracts-core/usage.d.ts +40 -0
  45. package/dist/contracts-core/usage.js +8 -0
  46. package/dist/contracts-core.d.ts +2 -0
  47. package/dist/contracts-core.js +2 -0
  48. package/dist/contracts-protocol.d.ts +81 -5
  49. package/dist/contracts-run-state.d.ts +91 -2
  50. package/dist/contributions.d.ts +2 -1
  51. package/dist/contributions.js +1 -0
  52. package/dist/extensions.d.ts +15 -1
  53. package/dist/extensions.js +68 -0
  54. package/dist/guardrail-packs/coding-standard.d.ts +3 -0
  55. package/dist/guardrail-packs/coding-standard.js +63 -0
  56. package/dist/guardrail-packs/destructive-commands.d.ts +3 -0
  57. package/dist/guardrail-packs/destructive-commands.js +46 -0
  58. package/dist/guardrail-packs/errors.d.ts +7 -0
  59. package/dist/guardrail-packs/errors.js +9 -0
  60. package/dist/guardrail-packs/index.d.ts +4 -0
  61. package/dist/guardrail-packs/index.js +15 -0
  62. package/dist/guardrail-packs/secrets-hygiene.d.ts +3 -0
  63. package/dist/guardrail-packs/secrets-hygiene.js +23 -0
  64. package/dist/guardrail-packs/types.d.ts +26 -0
  65. package/dist/guardrail-packs/types.js +2 -0
  66. package/dist/guardrail-packs/validation-respect.d.ts +3 -0
  67. package/dist/guardrail-packs/validation-respect.js +69 -0
  68. package/dist/guardrails.d.ts +61 -1
  69. package/dist/guardrails.js +377 -0
  70. package/dist/index.d.ts +16 -11
  71. package/dist/index.js +10 -7
  72. package/dist/input.d.ts +8 -1
  73. package/dist/input.js +68 -6
  74. package/dist/middleware.d.ts +37 -2
  75. package/dist/middleware.js +41 -0
  76. package/dist/node/session-store-jsonl.js +18 -3
  77. package/dist/observability.js +6 -0
  78. package/dist/provider-events.d.ts +8 -2
  79. package/dist/provider-events.js +60 -2
  80. package/dist/providers/openai-compatible.js +6 -3
  81. package/dist/run-bundle.d.ts +6 -1
  82. package/dist/run-bundle.js +5 -1
  83. package/dist/run-limits.d.ts +11 -1
  84. package/dist/run-limits.js +59 -0
  85. package/dist/session-stores.d.ts +12 -1
  86. package/dist/session-stores.js +21 -4
  87. package/dist/testing/agent-event-source-conformance.js +41 -2
  88. package/dist/testing/prefix-stability-conformance.d.ts +59 -0
  89. package/dist/testing/prefix-stability-conformance.js +172 -0
  90. package/dist/testing/session-store-conformance.d.ts +3 -2
  91. package/dist/testing/session-store-conformance.js +48 -0
  92. package/dist/tools.d.ts +5 -0
  93. package/dist/tools.js +21 -6
  94. package/dist/usage-estimation.d.ts +29 -0
  95. package/dist/usage-estimation.js +79 -0
  96. package/docs/agent-events.md +75 -4
  97. package/docs/agent-session-runtime.md +10 -6
  98. package/docs/attention-compiler.md +89 -8
  99. package/docs/caveman.md +1 -1
  100. package/docs/coding-agent-tools.md +1 -1
  101. package/docs/compaction-and-retry.md +1 -1
  102. package/docs/compaction-llm.md +2 -0
  103. package/docs/compaction-observational-memory.md +54 -7
  104. package/docs/durable-runs.md +46 -3
  105. package/docs/embeddings.md +9 -0
  106. package/docs/evaluations.md +5 -0
  107. package/docs/execution-timeline.md +79 -1
  108. package/docs/extensions.md +20 -3
  109. package/docs/guardrails.md +50 -4
  110. package/docs/hooks.md +282 -0
  111. package/docs/index.md +37 -15
  112. package/docs/input-and-prompt-assembly.md +4 -4
  113. package/docs/instruction-injection.md +1 -0
  114. package/docs/knowledge-sync.md +4 -0
  115. package/docs/live-testing.md +3 -1
  116. package/docs/memory-fabric.md +28 -0
  117. package/docs/middleware-hooks.md +90 -4
  118. package/docs/migrate-to-0.9.md +210 -0
  119. package/docs/migration.md +26 -0
  120. package/docs/multi-agent-patterns.md +25 -2
  121. package/docs/node-jsonl-session-store.md +7 -1
  122. package/docs/observability.md +7 -3
  123. package/docs/options-index.md +4 -1
  124. package/docs/policy-and-audit.md +26 -1
  125. package/docs/prefix-stability-conformance.md +143 -0
  126. package/docs/provider-caching.md +4 -4
  127. package/docs/provider-conformance.md +16 -0
  128. package/docs/provider-packages.md +20 -20
  129. package/docs/public-contracts.md +3 -2
  130. package/docs/rag.md +188 -3
  131. package/docs/release-and-install.md +45 -40
  132. package/docs/runs-and-usage.md +56 -10
  133. package/docs/scoped-agent-memory.md +270 -0
  134. package/docs/scoped-memory.md +138 -0
  135. package/docs/session-store-conformance.md +1 -2
  136. package/docs/session-stores.md +17 -17
  137. package/docs/supervisors.md +32 -12
  138. package/docs/tools.md +18 -1
  139. package/docs/wiki.md +4 -2
  140. package/docs/workflows.md +5 -0
  141. package/package.json +8 -2
@@ -107,7 +107,32 @@ for await (const page of exportPolicyDecisions({
107
107
 
108
108
  ## Extension and configuration notes
109
109
 
110
- Policy is optional. Hosts wire `record*` helpers or `evaluateAndAppend` at permission/guardrail/tool-approval/router/connector boundaries. Model-router and work-connector packages (later Phase 8 tasks) may call the same store when configured. Replace file/memory adapters with host WORM/KMS without changing record shape.
110
+ Policy is optional. Hosts wire `record*` helpers or `evaluateAndAppend` at permission/guardrail/tool-approval/router/connector boundaries. Model-router and work-connector packages (later Phase 8 tasks) may call the same store when configured. Replace file/memory adapters with host WORM/KMS without changing record shape. Guardrail-pack denials record through `recordGuardrailDecision` like any other guardrail: the target id is the rule identity `pack:<pack>/<rule>`, `block`/`tripwire` map to outcome `deny`, and the evidence ref is `guardrail:pack:<pack>/<rule>:<stage>`. Pack config appears in run bundles as identity rows (`pack:<pack>/<rule>`, stage, `pack@version`) — never inline predicate code or tool arguments.
111
+
112
+ ## Memory retrieval ACL denials and re-pointing (plan 089)
113
+
114
+ Memory retrieval keeps its own audit events next to policy decisions; hosts forward them to the same append-only sink:
115
+
116
+ | Event | Shape | When |
117
+ | --- | --- | --- |
118
+ | `rag.acl_denied` | `{ sourceId, scope: { tenantId, resourceId, threadId }, reason: "no_grant" \| "check_failed", hits, error? }` via `retrieveContext({ onAccessDenied })` | A source was withheld: revoked/absent/version-mismatched grant, the grant lookup threw (`error` is redacted, capped at 256 chars), or the store's own predicate filtered it before ranking (`hits: 0`, reported only when the host wires `onDeniedSources` through `retrieveContext`, plan 102 Task 6) |
119
+ | `Repointed` log line + result | `repointSource()` → `{ from, to, movedChunks, rewrittenEdges, layers, batched }` | A source's grant identity moved and derived artifacts followed |
120
+ | `rag.repointed` (rename audit) | `applySourceRenames({ onRenamed })` → `{ from, to, outcome: "moved", movedChunks, rewrittenEdges, layers }` \| `{ from, to, outcome: "failed", error }` | A batch of identity moves ran: one event per rename that settled, successes and failures alike (`error` redacted and capped at 256 chars) — plan 102 Task 7 |
121
+ | Invalidation rows | `store.invalidate()` rows (`{ id, reason: "corrected" \| "revoked" \| "forgotten" \| "legal_hold", at }`) read back by `listInvalidatedIds()` | A source was revoked/forgotten/held; tombstones stay for explainability |
122
+
123
+ Events are per *source*, not per hit, and are emitted once per query. They never contain document text, grant contents, or credentials; `check_failed` messages pass through the same redactor as retrieved content. Denials are fail-closed: a source is excluded whether the grant is absent, revoked, or the lookup failed, and the query returns the remaining hits. Aborts are not denials and are never recorded as such.
124
+
125
+ The re-point rows are the same kind of evidence for identity moves: `repointSource()` returns its counts to the caller, and `applySourceRenames()` (plan 102 Task 7) writes a batch into a host sink through `onRenamed` — one event per rename that settled, carrying `from`/`to`, the outcome, and either the moved counts per layer or the redacted error. A failed rename is audited before the batch stops (fail-fast) or continues, and a rename that never started is not audited.
126
+
127
+ Since plan 102 Task 6 the table also covers what the **store's own predicate** withheld, which plan 089 recorded as unauditable. A store that declares `authorization: "acl"` reports the sources its `query`/`lexicalQuery` predicate filtered out when the caller opts in, and `retrieveContext()` forwards that report into the same `onAccessDenied` path (one event per source per query, `hits: 0` because no hit ever existed — the finer per-source rule stays on the query-level callback):
128
+
129
+ ```ts
130
+ // store-level: what this query's own predicate withheld, and why
131
+ await store.query({ ...scope, embedding, topK, authorization, onDeniedSources: (d) => audit.write(d) });
132
+ // → [{ sourceId: "doc:payroll", reason: "no_grant" }, { sourceId: "doc:hr", reason: "version_mismatch" }]
133
+ ```
134
+
135
+ Without the callback the store issues no extra statement and its SQL is unchanged; with it, PostgreSQL/pgvector adds one grouped anti-join (`GROUP BY source_id` over the rows the predicate refused) that measured **1.2–1.4ms** on the 23-row fixture on an AMD Ryzen 9 PRO 7940HS. Reports carry source ids and reasons only — never rows, text, grant contents, or principal ids — and never widen the predicate: a withheld source stays withheld with or without the callback.
111
136
 
112
137
  ## Security and performance notes
113
138
 
@@ -0,0 +1,143 @@
1
+ # Prefix stability conformance
2
+
3
+ ## What it does
4
+
5
+ Prefix stability conformance drives a real agent session through two staggered skill loads and asserts that each provider request keeps a byte-identical leading prefix with its predecessor — messages **and** tool schemas. It is the host-runnable form of the golden check behind [provider caching](provider-caching.md) and progressive skill disclosure: a late `load_skill` must append a body after the stable prefix instead of rewriting it.
6
+
7
+ Exported from `@arnilo/prism/testing/prefix-stability-conformance`:
8
+
9
+ - `runPrefixStabilityConformance(options)`
10
+ - `PrefixStabilityConformanceOptions`
11
+ - `PrefixStabilityConformanceResult`
12
+
13
+ ## When to use it
14
+
15
+ Use it when a host owns any part of prompt assembly — custom `inputBuilder`, `promptBuilder`, context providers, instruction injectors, input/prompt middleware, or an explicit `inputLayout` — and wants to prove that progressive disclosure still holds the cache prefix. The runner:
16
+
17
+ - installs a fixture provider (no network) that loads `skills[0]` on the first turn and `skills[1]` on the second, two provider requests per turn — and, when the host runs an attention compiler, carries a deterministic reasoning block per skill-load round so the compiler's thinking stage has real content to fold;
18
+ - keeps everything else in `host` exactly as production: system prompt, context providers, builders, middleware, disclosure settings;
19
+ - measures, for each consecutive captured request, the byte-shared prefix as a fraction of the previous request and fails below `minContinuity` (default `0.95`);
20
+ - reports that fraction twice: `minContinuity` (provider-visible prefix, what the prompt cache pays for) and `cacheableContinuity` (the same measurement with the session's tail segments removed), and asserts whichever `assertOn` selects (default `providerPrefix`);
21
+ - fails when a loaded body never reaches a provider request, so a builder that drops the tail cannot pass vacuously.
22
+
23
+ ## Inputs / request
24
+
25
+ ```ts
26
+ import { runPrefixStabilityConformance } from "@arnilo/prism/testing/prefix-stability-conformance";
27
+
28
+ const result = await runPrefixStabilityConformance({
29
+ host: {
30
+ model: { provider: "anthropic", model: "claude-sonnet-4-6" },
31
+ systemPrompt: { text: "..." },
32
+ context: [projectContextProvider],
33
+ },
34
+ skills: [skillA, skillB],
35
+ assertOn: "cacheablePrefix",
36
+ });
37
+ ```
38
+
39
+ `PrefixStabilityConformanceOptions`:
40
+ - `host` — the host's `AgentConfig` minus `provider`, `providerSource`, and `skills`; the runner supplies the fixture provider and fixture skill registry
41
+ - `skills` — exactly two distinct `Skill` values with non-empty `instructions`, loaded in turn order
42
+ - `minContinuity?` — minimum shared-prefix fraction between consecutive requests (default `0.95`); always measured against the provider-visible prefix
43
+ - `assertOn?` — `"providerPrefix"` (default) gates the run on the provider-visible prefix; `"cacheablePrefix"` gates it on the tail-aware measurement instead, for an eager or body-heavy host that deliberately re-sends bodies after the stable prefix
44
+ - `allowedResets?` — how many request pairs may break below `minContinuity` (default `0`, today's behavior). Set `1` for an assembly that folds, compacts, or evicts exactly one boundary; more resets than declared fail, and fewer fail too, because a fixture that was supposed to invalidate the prefix and never did cannot pass vacuously
45
+ - `inputs?` — the two turn inputs (default fixed strings, so runs stay comparable across hosts)
46
+
47
+ ## Outputs / response / events
48
+
49
+ Returns `Promise<{ requests: number; minContinuity: number; cacheableContinuity: number; resets: readonly number[] }>`: the captured request count (four) and the two lowest shared-prefix fractions observed. Throws a plain `Error` naming the offending request pair and the measured percentage on the first violation. No events, no test runner, no network.
50
+
51
+ A gap below `minContinuity` **on the metric `assertOn` selects** is collected as a reset instead of failing inside the loop: `resets` holds the 1-based index of the request that broke (the later request of the pair, ascending). With the default `allowedResets: 0` the first reset fails the run exactly as before, now adding the observed reset list to the message; `allowedResets: 1` lets a single documented boundary (an attention fold, a compaction, a budget eviction) pass while every other pair must stay byte-stable.
52
+
53
+ Both numbers measure the same consecutive request pairs, byte for byte:
54
+
55
+ - **`minContinuity`** — the provider-visible prefix, messages **and** tool schemas as sent on the wire. This is what the prompt cache can keep paying for, and its meaning is frozen: `0.95` default, unchanged by `assertOn`.
56
+ - **`cacheableContinuity`** — the same fraction recomputed after removing this session's tail segments (loaded skill bodies, resources moved to the tail) from **both** requests of each pair. A host whose provider-visible fraction dips only because of tail bodies reads `1` here.
57
+
58
+ Tail segments are read from the session's own `tailSegments` map — the exact `Message` objects assembly appended — matched by object identity first and by serialized-value equality for a `promptBuilder` that clones messages. Nothing is added to the provider payload and no host content is pattern-matched. When no captured request carried a tail segment (a builder that renders bodies elsewhere), `cacheableContinuity` equals `minContinuity`; it is not reported as `1`.
59
+
60
+ ## Request/response example
61
+
62
+ ```ts
63
+ import { runPrefixStabilityConformance } from "@arnilo/prism/testing/prefix-stability-conformance";
64
+
65
+ const { minContinuity, cacheableContinuity, resets } = await runPrefixStabilityConformance({
66
+ host: myAgentAssembly,
67
+ skills: [alphaSkill, betaSkill],
68
+ });
69
+ // throws: "request 2 → 3 kept 41.2% of the previous provider prefix (minimum 95.0%), and 1 pair(s)
70
+ // broke below it (resets [3] of 3 request pairs, allowedResets 0)"
71
+ // when a context block or the skill catalog is recomposed in place.
72
+ // minContinuity: 0.98 (provider-visible) · cacheableContinuity: 1 (tail bodies excluded) · resets: []
73
+ ```
74
+
75
+ ## Implementation example
76
+
77
+ ```ts
78
+ import { runPrefixStabilityConformance } from "@arnilo/prism/testing/prefix-stability-conformance";
79
+
80
+ // The runner owns the provider and skills, so the same helper is the negative control too:
81
+ // add a deliberately volatile context provider to prove the assertion can fail.
82
+ await runPrefixStabilityConformance({
83
+ host: {
84
+ model: myModel,
85
+ context: [{ name: "volatile", resolve: () => [{ title: "Now", content: `${Date.now()}` }] }],
86
+ },
87
+ skills: [alphaSkill, betaSkill],
88
+ });
89
+ ```
90
+
91
+ ## Extension and configuration notes
92
+
93
+ - Loaded skill bodies and URI resources are re-sent after new transcript content by design (the tail is append-only, not immutable); a body larger than `1 - minContinuity` of the whole prompt lowers the provider-visible fraction without indicating a prefix regression. For such a host assert `assertOn: "cacheablePrefix"` (the tail-aware number stays `1`), or raise the fixture's stable prefix or lower `minContinuity`.
94
+ - A volatile leading context provider lowers **both** fractions: context is not a tail segment, so `cacheablePrefix` cannot mask a real prefix regression.
95
+ - Prompt builders that render skill bodies outside the tail are welcome — the check measures the provider-visible prefix, not where the body sits.
96
+ - Attention/tool-result folding and context-budget eviction are explicit invalidation boundaries: with the default `allowedResets: 0` the run fails at that turn. Declare the boundary instead of loosening `minContinuity` — one fold is one reset, so a second reset, a post-fold rewrite, or a fold that never happened still fails:
97
+
98
+ ```ts
99
+ const result = await runPrefixStabilityConformance({
100
+ host: {
101
+ ...myAssembly,
102
+ // A predicate gate must settle under the stages it triggers: one that still fires after folding
103
+ // fails closed with `AttentionBudgetError` instead of reporting a reset. This one opens on a
104
+ // run's opening round while the carried request is over the floor, and the fold drops it back.
105
+ attentionCompiler: {
106
+ maxInputTokens: 4_000,
107
+ keepLast: 0,
108
+ thinkingKeepTurns: 0,
109
+ trigger: { kind: "predicate", shouldFold: (state) => state.turn === 1 && state.estimatedInputTokens >= 1_350 },
110
+ },
111
+ },
112
+ skills: [alphaSkill, betaSkill],
113
+ allowedResets: 1, // add `assertOn: "cacheablePrefix"` when tail re-sends should not count either
114
+ });
115
+ result.resets; // [3] — the request after the fold; every other pair stayed append-only
116
+ ```
117
+
118
+ - Documented invalidation boundaries — what `resets` is expected to name. Each row is pinned by a fixture in [`src/__tests__/invalidation-inventory.test.ts`](../src/__tests__/invalidation-inventory.test.ts) that asserts the boundary *position* (message index, and tool index for schemas), so a reordering of the cache-aware layout fails that suite instead of silently relocating a boundary:
119
+
120
+ | Segment | Boundary in the default `cache_aware` layout | Owner |
121
+ | --- | --- | --- |
122
+ | Per-turn instruction-injector text (`on_input`) | Message 0: merged into the leading system prompt — never moved behind the transcript | [Input and prompt assembly](input-and-prompt-assembly.md) |
123
+ | Host / injector context blocks | The context slot: after the hoisted leading system messages, before skills | [Context and skills](context-and-skills.md) |
124
+ | Observational-memory blocks (`observational-memory`, `recent-messages`) | The same context slot; re-rendering identical blocks keeps the prefix byte-identical | [Context and skills](context-and-skills.md) |
125
+ | Compaction summaries | Right after the leading system prompt while nothing user-role precedes it; a leading attachment moves the summary behind the context and skill slots | [Input and prompt assembly](input-and-prompt-assembly.md) |
126
+ | Pending tool results / current input | The suffix: a result inserts immediately before the current input, so that input is the round's boundary | [Input and prompt assembly](input-and-prompt-assembly.md) |
127
+ | `contextBudget` eviction | The first evicted group in the documented drop order (tool results → history → summaries → context → skills → attachments) | [Input and prompt assembly](input-and-prompt-assembly.md) |
128
+ | Tool-schema selection | `request.tools` only: gaining or losing a schema leaves every message byte-identical, while a changed description is a boundary at that schema | [Provider caching](provider-caching.md) |
129
+ | Attention-compiler / tool-result fold | In place at the fold frontier: the oldest stripped or stubbed row is the boundary — one reset, declared with `allowedResets` | [Attention compiler](attention-compiler.md) |
130
+ | Skill bodies and URI resources (tail) | After the transcript, append-only: a newly loaded body never invalidates the prefix and `cacheableContinuity` stays `1` | [Input and prompt assembly](input-and-prompt-assembly.md) |
131
+
132
+ ## Security and performance notes
133
+
134
+ - No credentials, no network, no real skills required; the fixture provider is a local generator.
135
+ - Four small provider requests per run, in-memory session store (unless `host.store` says otherwise); cheap enough for a conformance suite.
136
+
137
+ ## Related APIs
138
+
139
+ - [Provider caching](provider-caching.md)
140
+ - [Input and prompt assembly](input-and-prompt-assembly.md)
141
+ - [Context and skills](context-and-skills.md)
142
+ - [Provider conformance](provider-conformance.md)
143
+ - [Compaction conformance](compaction-conformance.md)
@@ -66,11 +66,11 @@ Cache helpers return plain data:
66
66
  | `canonicalizeJsonSchema(value)` | Clone with sorted object keys and `required` names; semantic arrays stay ordered. Used by first-party tool serializers. |
67
67
  | `cacheHitRate(usage)` | Cached input ratio or `undefined`. |
68
68
  | `cacheSavings(usage, model)` | Estimated read-token savings or `undefined` without pricing. |
69
- | `cacheUsageReport(usage, model?)` | Normalized read/write tokens, hit rate, estimated savings, and currency when available; `undefined` when no usage is supplied. |
69
+ | `cacheUsageReport(usage, model?)` | Normalized reported read/write tokens, hit rate, estimated savings, and currency when available; `undefined` when no cache token field is reported. Missing fields stay absent, never become `0`. |
70
70
 
71
- Provider events do not change. Cache accounting stays in normalized `Usage.cacheReadTokens` and `Usage.cacheWriteTokens`.
71
+ Cache accounting stays in normalized `Usage.cacheReadTokens` and `Usage.cacheWriteTokens`. Terminal `provider_turn_finished.metadata.cache` carries the same numeric report for a reporting provider; unavailable cache usage stays absent.
72
72
 
73
- For stable-prefix payloads, `inputLayout: "cache_aware"` is the default on the default input builder, `assembleProviderInput()`, `AgentConfig`, and `RunOptions`; set `inputLayout: "legacy"` to restore the prior order. The default prompt builder's cache-aware order is leading system instructions → resolved context blocks → selected/progressively disclosed skills → fallback text tool declarations → attachments/resources → summaries → prior history → pending tool results → current input. Declared tool schemas remain in `ProviderRequest.tools` and are never granted by prompt middleware. First-party tool serializers run `canonicalizeJsonSchema` so property insertion order cannot break that prefix. Changing only current input preserves the serialized message prefix before the final user suffix; changing dynamic context or loaded skills changes only from its own boundary onward, while tool schemas remain independently stable. The prefix is byte-stable only when those stable inputs are unchanged; Prism still does not guarantee provider cache hits.
73
+ For stable-prefix payloads, `inputLayout: "cache_aware"` is the default on the default input builder, `assembleProviderInput()`, `AgentConfig`, and `RunOptions`; set `inputLayout: "legacy"` to restore the prior order. The default prompt builder's cache-aware order is leading system instructions → resolved context blocks → selected/progressively disclosed skill catalogs → fallback text tool declarations → attachments/resources → summaries → prior history → pending tool results → current input → optional session tail. `RuntimeAgentSession` uses that tail for URI resources and loaded skill bodies: first insertion fixes `resource:<uri>` / `skill:<name>` order, so a later skill load appends instead of rewriting its catalog slot. Re-deriving the same id keeps its position; changed bytes explicitly invalidate from that tail segment. Declared tool schemas remain in `ProviderRequest.tools` and are never granted by prompt middleware. First-party tool serializers run `canonicalizeJsonSchema` so property insertion order cannot break that prefix. Context-budget eviction, custom builders/middleware, `toolResultFold`, and attention compilation are explicit invalidation boundaries; folding cannot move to an append-only tail without retaining the raw payload it exists to remove. The prefix is byte-stable only when those stable inputs are unchanged; Prism still does not guarantee provider cache hits.
74
74
 
75
75
  ## Request/response example
76
76
 
@@ -128,7 +128,7 @@ const retention = mapCacheRetention(hints.retention, model);
128
128
  const stamped = applyCacheControl(messages, hints.breakpoints ?? [], { maxBreakpoints: model.cache?.maxBreakpoints });
129
129
  const hitRate = cacheHitRate({ inputTokens: 1000, cacheReadTokens: 800 });
130
130
  const report = cacheUsageReport({ inputTokens: 1000, cacheReadTokens: 800 }, model);
131
- // { cacheReadTokens: 800, cacheWriteTokens: 0, hitRate: 0.8, ... }
131
+ // { cacheReadTokens: 800, hitRate: 0.8, ... }
132
132
 
133
133
  await session.run("Explain this", { inputLayout: "cache_aware" });
134
134
  ```
@@ -223,6 +223,22 @@ Canonical contract: [Thinking and reasoning](thinking-and-reasoning.md).
223
223
 
224
224
  Canonical contract: [AI SDK provider adapter](providers/ai-sdk.md).
225
225
 
226
+ ## Stop-reason checklist
227
+
228
+ Every adapter that parses a native completion reason must map it through the shared
229
+ `mapProviderStopReason` table and emit it on the normalized `done` event
230
+ (`providerDone(usage, mapped)`); `provider_turn_finished.metadata.stopReason` then carries it to
231
+ hosts (see [Agent events](agent-events.md#outputs--response--events)). Cover:
232
+
233
+ 1. **One mapped native reason per protocol** — a fake stream whose wire reason means truncation
234
+ (`finish_reason: "length"`, `stop_reason: "max_tokens"`, `finishReason: "MAX_TOKENS"`,
235
+ Converse `stopReason: "max_tokens"`) reaches `done.stopReason === "max_output_tokens"`.
236
+ 2. **Tool-call turns** — a native tool reason (`tool_calls` / `tool_use` / `tool-calls`) maps to
237
+ `tool_calls`; a generic completion reason on a turn that produced tool calls is normalized to
238
+ `tool_calls` by the session, not the adapter.
239
+ 3. **Unknown degrades** — a new or unmapped wire value yields `unknown` and never fails the stream.
240
+ 4. **No extra fields** — the adapter adds nothing else to `done`; redaction and bounds are unchanged.
241
+
226
242
  ## Extension and configuration notes
227
243
 
228
244
  The helpers are a testing subpath only. Provider packages can use them with their own mocked fetch/transport or `createMockProvider()`. Live provider tests should stay opt-in and env-gated outside Prism's default test suite.
@@ -25,26 +25,26 @@ Do not use provider packages as a package manager, credential store, env loader,
25
25
 
26
26
  | adapter package | version |
27
27
  | --- | --- |
28
- | `@arnilo/prism-providers/ai-sdk` | 0.8.0 |
29
- | `@arnilo/prism-providers/alibaba` | 0.8.0 |
30
- | `@arnilo/prism-providers/anthropic` | 0.8.0 |
31
- | `@arnilo/prism-providers/azure` | 0.8.0 |
32
- | `@arnilo/prism-providers/bedrock` | 0.8.0 |
33
- | `@arnilo/prism-providers/clinepass` | 0.8.0 |
34
- | `@arnilo/prism-providers/commandcode` | 0.8.0 |
35
- | `@arnilo/prism-providers/deepseek` | 0.8.0 |
36
- | `@arnilo/prism-providers/google` | 0.8.0 |
37
- | `@arnilo/prism-providers/hyper` | 0.8.0 |
38
- | `@arnilo/prism-providers/kimi` | 0.8.0 |
39
- | `@arnilo/prism-providers/model-discovery` | 0.8.0 |
40
- | `@arnilo/prism-providers/neuralwatt` | 0.8.0 |
41
- | `@arnilo/prism-providers/ollama` | 0.8.0 |
42
- | `@arnilo/prism-providers/openai` | 0.8.0 |
43
- | `@arnilo/prism-providers/opencode-go` | 0.8.0 |
44
- | `@arnilo/prism-providers/openrouter` | 0.8.0 |
45
- | `@arnilo/prism-providers/vertex` | 0.8.0 |
46
- | `@arnilo/prism-providers/xai` | 0.8.0 |
47
- | `@arnilo/prism-providers/zai` | 0.8.0 |
28
+ | `@arnilo/prism-providers/ai-sdk` | 0.10.0 |
29
+ | `@arnilo/prism-providers/alibaba` | 0.10.0 |
30
+ | `@arnilo/prism-providers/anthropic` | 0.10.0 |
31
+ | `@arnilo/prism-providers/azure` | 0.10.0 |
32
+ | `@arnilo/prism-providers/bedrock` | 0.10.0 |
33
+ | `@arnilo/prism-providers/clinepass` | 0.10.0 |
34
+ | `@arnilo/prism-providers/commandcode` | 0.10.0 |
35
+ | `@arnilo/prism-providers/deepseek` | 0.10.0 |
36
+ | `@arnilo/prism-providers/google` | 0.10.0 |
37
+ | `@arnilo/prism-providers/hyper` | 0.10.0 |
38
+ | `@arnilo/prism-providers/kimi` | 0.10.0 |
39
+ | `@arnilo/prism-providers/model-discovery` | 0.10.0 |
40
+ | `@arnilo/prism-providers/neuralwatt` | 0.10.0 |
41
+ | `@arnilo/prism-providers/ollama` | 0.10.0 |
42
+ | `@arnilo/prism-providers/openai` | 0.10.0 |
43
+ | `@arnilo/prism-providers/opencode-go` | 0.10.0 |
44
+ | `@arnilo/prism-providers/openrouter` | 0.10.0 |
45
+ | `@arnilo/prism-providers/vertex` | 0.10.0 |
46
+ | `@arnilo/prism-providers/xai` | 0.10.0 |
47
+ | `@arnilo/prism-providers/zai` | 0.10.0 |
48
48
  <!-- generated:package-truth:providers end -->
49
49
 
50
50
 
@@ -154,10 +154,11 @@ Important request shapes:
154
154
  | `PersistenceQuery` | Common pagination controls: `cursor?`, `limit?`, `order?: "asc" \| "desc"`. |
155
155
  | `OwnershipScope` | Multi-tenant scope: `tenantId?`, `accountId?`, `userId?`. Included in records and queries. |
156
156
  | `SessionRecord` / `SessionQuery` | Stored session and query filters (parent, agent definition, retention policy, timestamps, ownership). `SessionRecord.version` (with `appendSession` `expectedVersion`) is the optimistic metadata CAS: 0 = create-only, N = exact-version update; mismatch throws `SessionMetadataConflictError` (`metadata_conflict`). |
157
- | `SessionIndex` / `SessionSearchQuery` / `SessionSearchHit` | Bounded optional session search seam (`search` / `SessionStore.searchSessions?`). Filters: workspace (`metadata.workspaceRoot`), time, provider/model, label/summary, optional FTS `query`, ownership. Hits return `sessionId` + optional `leafId` for resume; never credentials. Caps via `resolveSessionSearchQuery` / `DEFAULT_*` / `HARD_MAX_*` session-search constants. |
157
+ | `SessionIndex` / `SessionSearchQuery` / `SessionSearchHit` | Bounded optional session search seam (`search` / `SessionStore.searchSessions?`). Filters: workspace (`metadata.workspaceRoot`), time, provider/model, label/summary, entry-kind (`kind`, e.g. annotation search), optional FTS `query`, ownership. Hits return `sessionId` + optional `leafId` for resume, and on a text match the matched entry pointer (`entryId`, `runId`, 1-based `turn`, store `score`) with a bounded matched-text `snippet`; never credentials. Caps via `resolveSessionSearchQuery` / `DEFAULT_*` / `HARD_MAX_*` session-search constants. SQLite/Postgres index (FTS5 / `tsvector`); memory and JSONL scan linearly. |
158
158
  | `contextBudget` / `getContextBudgetReport` / `ContextBudgetError` | Opt-in assembler budget on `AssembleProviderInputOptions`; deterministic eviction; omission report in `ProviderRequest.metadata` (kinds/ids/sizes only). |
159
159
  | `AgentSession.steer` / `SteerOptions` / pending-steer caps | Mid-run enqueue into active run; optional `softInterrupt`; default 8 msgs / 64 KiB UTF-8. |
160
- | `SessionSearchUnsupportedError` / `sessionSearchMode` | Memory opt-out + JSONL; typed throw (not empty success). Memory linear caps are host-overridable via `CreateMemorySessionStoreOptions.search`. |
160
+ | `AgentSession.close()` | Session teardown: dispatches `session_shutdown` middleware once (idempotent), then closes every subscriber (`acrossRuns` included). `session_start` is its mirror at the first run start; both are per-session, never per-turn. See [Middleware hooks](middleware-hooks.md). |
161
+ | `SessionSearchUnsupportedError` / `sessionSearchMode` | Memory opt-out (`sessionSearchMode: "unsupported"`); typed throw (not empty success). Memory linear caps are host-overridable via `CreateMemorySessionStoreOptions.search`; the JSONL store searches linearly with the contract default caps. |
161
162
  | `BranchRecord` / `BranchQuery` | Branch handle/leaf pointer and query filters (session, name, parent branch, leaf presence). |
162
163
  | `SessionEntryQuery` | Paginated entry filters: `sessionId`, `runId`, `parentId`, `leafId`, `kind`, timestamp range, ownership. |
163
164
  | `RunRecord` / `RunQuery` | Stored run and filters: session, branch, status, timestamps, ownership. |
package/docs/rag.md CHANGED
@@ -28,6 +28,10 @@ Document lifecycle:
28
28
  | `deleteSource({ sourceId, store, scope })` | Deletes only matching IDs under exact tenant/resource/corpus scope. |
29
29
  | `replaceDocument({ uri, loader, parser, store, scope, ... })` | Loads through a host seam, parses, chunks, and atomically replaces. `sourceId` is required unless loader supplies one. |
30
30
  | `syncKnowledge({ connector, checkpoints, checkpoint, store, embedder, scope })` | Paged connector import; cursor CAS only after each committed page. See [Knowledge synchronization](knowledge-sync.md). |
31
+ | `createDeletionPropagator({ scope, vectorStore, authorization })` | Privileged deletion orchestration: lineage-closed tombstone set + registered handlers. See [Deletion propagation](#deletion-propagation). |
32
+ | `createRagDeletionHandler({ store, scope, statusStore? })` | The RAG layer's propagation handler: removes a deleted source's chunk rows and ingestion status. |
33
+ | `createLocalReranker({ model?, runtime?, … })` | Zero-service default reranker: in-process cross-encoder behind the `LocalRerankRuntime` seam. See [Local reranker](#local-reranker). |
34
+ | `resolveReranker(config)` | Declarative reranker config (`kind: "local" \| "tei" \| "openai-compatible" \| "voyage" \| "fake" \| "none"`) → `Reranker`. |
31
35
  | `createGoogleDriveConnector({ tokenProvider, resolveAccess })` | Drive `files.list` + `changes.list` connector. Host maps permissions; watch payloads are not authorization. |
32
36
  | `DocumentLoader` / `Parser` | Small host-replaceable seams. `@arnilo/prism-memory/rag/loaders` and `/rag/parsers` export reference adapters. |
33
37
  | `textParser` / `markdownParser` / `htmlParser` / `pdfParser` | UTF-8 text, Markdown, script/style-stripping HTML, and uncompressed-text PDF parsers. |
@@ -83,6 +87,185 @@ Default/hard ceilings include 1,000/16,384 chunk characters, 100/4,096 overlap,
83
87
  }
84
88
  ```
85
89
 
90
+ ## Deletion propagation
91
+
92
+ `deleteSource()` removes one source's chunk rows. Derived artifacts (summaries, observational-memory entries, compiled wiki pages, host projections) are not chunk rows, so they need an explicit, privileged propagation pass:
93
+
94
+ ```ts
95
+ import { createDeletionPropagator, createMemoryVectorStore } from "@arnilo/prism-memory";
96
+ import { createRagDeletionHandler } from "@arnilo/prism-memory/rag";
97
+ import { createWikiDeletionHandler } from "@arnilo/prism-memory/wiki";
98
+
99
+ const store = createMemoryVectorStore();
100
+ const propagator = createDeletionPropagator({
101
+ scope: { tenantId: "t1", resourceId: "docs", threadId: "handbook" },
102
+ vectorStore: store,
103
+ authorization: { tenantId: "t1", principalId: "p1", groupIds: ["eng"] }, // host-verified; ACL-store grants are enforced here
104
+ });
105
+ propagator.register(createRagDeletionHandler({ store, scope: ragScope }));
106
+ propagator.register(createWikiDeletionHandler({ workspaceRoot }));
107
+
108
+ const result = await propagator.propagate("doc:erp-lead");
109
+ // { sourceId, ids, tombstoned, layers: { rag: 4, wiki: 1 }, batched: true }
110
+ ```
111
+
112
+ - `propagate(sourceId)` expands the source through `_lineage.sourceIds` (`collectInvalidationIds`, depth 8) into a closed id set, tombstones **all** of it with reason `forgotten` inside one store transaction, then runs every registered handler with `{ sourceId, ids, scope, signal }`. Handlers return how many artifacts they removed (reported per `kind` in `layers`).
113
+ - Tombstones, not deletions, for derived rows: rows stay for explainability (`recall({ explain: true })` reports the invalidation), and lineage links never dangle. Handlers own physical removal (chunk rows, files, ledger entries).
114
+ - Retrieval is belt-and-suspenders: `retrieveContext()` reads per-scope invalidations before assembly and drops any candidate whose record id, `_lineage.sourceIds`, or `_rag.sourceId` is tombstoned — so a delete that lands after the query legs read rows still returns zero hits. The split matters for direct store users: the store's own SQL predicate filters by record id and `_lineage` edge, while a source's *own* chunk rows are covered by the `_rag.sourceId` rule at the retrieval boundary (or removed physically by the `rag` handler) — a raw `store.query()` is not a recall path.
115
+ - `HARD_PROPAGATION_EDGES` (4,096) is the one-pass privileged ceiling; over it the whole delete rejects (fail-closed), never a half-tombstoned document. Each store `invalidate` call carries at most `HARD_INVALIDATION_BATCH` (64) entries. On a durable store that shape holds: PostgreSQL/pgvector tombstones 1,001 rows (1,000 derived chunk rows + the source root) in **one transaction and 22 statements** (16 of them `HARD_INVALIDATION_BATCH`-sized `INSERT`s), measured at **29–155 ms** across runs on an AMD Ryzen 9 PRO 7940HS against `pgvector/pgvector:pg16` (more under parallel load) — the durable counterpart of the in-memory suite's 1k-under-2s check, and evidence rather than a gate. Re-run it with `PRISM_TEST_POSTGRES_URL=… npm run test:postgres` (`packages/memory/src/__tests__/postgres-propagation.integration.test.ts`); the leg also proves the store's own SQL predicate hides the tombstoned rows, not only the in-app guard, and that a denied propagation opens no transaction at all.
116
+ - Deletion is privileged: `authorization` is required, tenant-checked, and enforced through the store's existing `checkSourceAccess` ACL when the store declares `authorization: "acl"` (missing grant → `MemoryScopeError` before anything is written). Retrieval paths never construct a propagator.
117
+ - Observational memory registers its own leg: `createObservationalMemoryDropHandler({ session, appendEntry })` (from `@arnilo/prism-memory/compaction/observational-memory`) folds the session ledger once per propagation and writes one `om.observations.dropped` entry for the observations that rest on a tombstoned record id; see [observational memory](compaction-observational-memory.md).
118
+
119
+ ### One wiring, every layer (host recipe)
120
+
121
+ A host composes the legs itself — no facade ships until a host asks for one, because the propagator already owns handler registration, privilege, and the lineage-closed id set:
122
+
123
+ ```ts
124
+ import { createDeletionPropagator, createMemoryVectorStore, listInvalidatedIds } from "@arnilo/prism-memory";
125
+ import { createRagDeletionHandler } from "@arnilo/prism-memory/rag";
126
+ import { createWikiDeletionHandler } from "@arnilo/prism-memory/wiki";
127
+ import { buildObservationalMemoryContextBlocks, createObservationalMemoryDropHandler } from "@arnilo/prism-memory/compaction/observational-memory";
128
+ import { createFabricRepointHandler } from "@arnilo/prism-memory/fabric";
129
+
130
+ const store = createMemoryVectorStore();
131
+ const propagator = createDeletionPropagator({
132
+ scope,
133
+ vectorStore: store,
134
+ authorization: hostVerifiedPrincipal, // required by the type; must match the scope's tenant
135
+ handlers: [
136
+ createRagDeletionHandler({ store, scope: ragScope }),
137
+ createWikiDeletionHandler({ workspaceRoot }),
138
+ createObservationalMemoryDropHandler({ session, appendEntry }),
139
+ createFabricRepointHandler({ scope, vectorStore: store }),
140
+ ],
141
+ });
142
+ const result = await propagator.propagate("docs/policy.md");
143
+ // { sourceId, ids: ["docs/policy.md", "summary:docs/policy.md"], tombstoned: 2, layers: { rag: 1, wiki: 1, observational: 1, fabric: 1 }, batched: true }
144
+
145
+ // Write path: the drop entry is the ledger's record of what the revocation retired (one append, ids only).
146
+ // Read path: a projection whose ledger was never written passes the same tombstones instead.
147
+ const blocked = await listInvalidatedIds(store, scope);
148
+ const blocks = buildObservationalMemoryContextBlocks(entries, { invalidatedIds: blocked });
149
+ ```
150
+
151
+ - Both paths keep a revoked observation out of memory, in the same rendered order: the drop entry retires every active observation whose id or `sourceEntryIds` intersect the tombstone set (`om.observations.dropped`, ids only — never observation text), and `invalidatedIds` withholds them at build time. So an id is withheld whether or not the physical drop ran, and a projection built from an older snapshot matches the post-drop one.
152
+ - One `listInvalidatedIds` read per projection build (one scope read, `corrected` entries stay), and the recipe adds no work beyond the propagator: the same `layers` result already answers per-leg counts, so nothing is re-read to report it.
153
+ - The fabric leg is the one that cannot be left out: a note names its document by `metadata.path`, so no `_lineage` edge exists to walk and a deleted path would otherwise keep being served. `createFabricRepointHandler()` tombstones the notes recorded against the deleted id in the same pass (plan 102 Task 11), and the same handler follows a `repointSource()` move — see the re-point section below.
154
+
155
+ ## Grant recheck and re-pointing
156
+
157
+ Retrieval never trusts a grant snapshot. `retrieveContext()` re-asks the store for **each distinct source** it is about to inject, on both sides of the reranker:
158
+
159
+ ```ts
160
+ const result = await retrieveContext("approval policy", {
161
+ embedder,
162
+ store,
163
+ scope,
164
+ authorization: hostVerifiedPrincipal, // every query re-reads the live grant
165
+ onAccessDenied: (denial) => audit.write({ kind: "rag.acl_denied", ...denial }),
166
+ });
167
+ // mid-turn revoke → the source is gone from this and every later result
168
+ await store.setSourceAccess(thread, [{ sourceId: "doc:payroll", principalIds: [], accessVersion: 2 }]);
169
+ ```
170
+
171
+ - Candidate pre-filter and, when a reranker ran, a fresh post-rerank gate (`createAccessRecheck()`, one instance per query). The post-rerank gate re-reads on purpose: a grant revoked *while the reranker was running* must not leak its text into the prompt.
172
+ - Cost is per source, not per hit: 200 candidates over 50 sources cost 50 lookups per gate, not 200. The in-memory hook does that inside the 5ms budget for 50 sources; a PostgreSQL store pays one indexed `checkSourceAccess` per source per gate.
173
+ - Fail closed, never silently: an absent/revoked/version-mismatched grant and a **thrown** store error both withhold the hits, and every withheld source is reported once through `onAccessDenied` as `{ sourceId, scope, reason: "no_grant" | "check_failed", hits, error? }` (`error` is redacted and capped at 256 chars). The query still completes with the remaining hits. Abort still aborts — it is not reclassified as a denial.
174
+ - There is no per-request off switch: passing `authorization` is what turns the gate on, and the only knob is the audit sink. A store that declares `authorization: "acl"` without `checkSourceAccess` fails closed before ranking.
175
+ - The store's own query/lexical predicate remains the first line of defense (unauthorized text never leaves the store); the boundary recheck also covers stores whose query leg ignores grants, and revokes that land after the query legs have read.
176
+ - Sources the store filtered *inside* its own predicate no longer go unaudited (plan 102 Task 6). An `authorization: "acl"` store reports what it withheld per query when the caller opts in — `onDeniedSources: (denials) => …` on `query`/`lexicalQuery`, carrying `{ sourceId, reason: "no_grant" | "version_mismatch" | "unknown" }` (ids and reasons only) — and `retrieveContext()` passes that report into the same `onAccessDenied` path, so a store-filtered source produces one event per query with `hits: 0`. The report never widens the predicate, and without the callback the store's SQL is unchanged: PostgreSQL/pgvector then issues no extra statement, or exactly one grouped anti-join with it (1.2–1.4ms on the 23-row protected fixture).
177
+
178
+ When a source's grant identity moves (`doc:a` → `doc:b`, a document re-filed under a new source id), `repointSource()` makes the derived artifacts follow **without re-embedding**:
179
+
180
+ ```ts
181
+ import { repointSource } from "@arnilo/prism-memory";
182
+ import { createWikiRepointHandler } from "@arnilo/prism-memory/wiki";
183
+
184
+ const moved = await repointSource({
185
+ scope: { tenantId: "t1", resourceId: "docs", threadId: "handbook" },
186
+ vectorStore: store,
187
+ from: "doc:a",
188
+ to: "doc:b",
189
+ authorization: hostVerifiedPrincipal, // must admit BOTH ids on an ACL store
190
+ handlers: [createWikiRepointHandler({ workspaceRoot })],
191
+ });
192
+ // { from, to, movedChunks, rewrittenEdges, layers: { wiki: 1 }, batched: true } (add `cursor` for the next page)
193
+ ```
194
+
195
+ - Chunk rows keep their text, embeddings, offsets, and generation: the row id (`doc:a#0001` → `doc:b#0001`), `_rag.sourceId`, and `_rag.citationId` are rewritten, old ids are deleted, and the page lands in one store transaction (`batched: true`) or not at all — never a half-written page, and never a re-embed. A durable store keeps that promise: PostgreSQL/pgvector re-keys the chunk rows and rewrites the lineage edges in **one transaction per page**, and the grant check for both ids runs before the first transaction opens. The protected leg measures the small fixture (1 chunk row + 3 lineage edges, 3–8 ms); the 1k-row durable cost above is the propagation number.
196
+ - Lineage edges (`_lineage.sourceIds`) on derived rows move from `from` to `to` in the same pass, so `createDeletionPropagator()` stays correct afterwards: deleting `doc:b` still tombstones the derived rows, deleting `doc:a` no longer touches them.
197
+ - Privileged like deletion propagation: on a store that declares `authorization: "acl"` the caller must pass an `authorization` that admits **both** the source and the destination, and re-point never creates or copies grants — grant the destination first or the move fails closed.
198
+ - **One call moves one page.** `HARD_REPOINT_RECORDS` (4,096) is the default `pageSize`, not a ceiling on the scope: a bigger scope returns a `cursor` and the host continues from it. Each page re-keys and rewrites inside its own transaction, so a page is atomic; the ACL check runs once per call (a resumed call re-validates rather than trusting a cached decision), and the whole scope is read per page because no store exposes a ranged read — the bound is on the write, which is where the cost was.
199
+
200
+ ```ts
201
+ let cursor: string | undefined;
202
+ let movedChunks = 0;
203
+ do {
204
+ const page = await repointSource({
205
+ scope,
206
+ vectorStore: store,
207
+ from: "doc:a",
208
+ to: "doc:b",
209
+ authorization: hostVerifiedPrincipal,
210
+ handlers: [createWikiRepointHandler({ workspaceRoot })],
211
+ pageSize: 1_000, // ≤ HARD_REPOINT_RECORDS if you want the documented ceiling
212
+ ...(cursor === undefined ? {} : { cursor }),
213
+ });
214
+ movedChunks += page.movedChunks;
215
+ cursor = page.cursor; // present while records past the page still need moving
216
+ } while (cursor !== undefined);
217
+ ```
218
+
219
+ - The cursor is an opaque token that names the last record **id** the page considered, ascending. Records an earlier page already moved no longer touch `from`, so a **stale cursor is a no-op** (no transaction opens, no handler runs, `movedChunks: 0`), and a cursor from another scope or source pair is rejected with a validation error before the store is read — a partial move is never silently completed as a different one. Re-running a completed move with the final page's cursor reports `movedChunks: 0`.
220
+ - Pages are id-ordered, not store-ordered, so a resume is deterministic even for a store that returns rows in an unstable order. A destination collision fails closed on whichever page it appears in (`MemoryValidationError`, before that page's write).
221
+ - Passing `maxRecords` keeps the plan 089 all-or-nothing posture: over that many records in the scope the call rejects instead of paging. Use it when a partial move is worse than no move; `pageSize` alone is the paged mode.
222
+ - Row ids that already exist at the destination (other than rows of the moved source) abort the move instead of overwriting (`MemoryValidationError`).
223
+ - `createWikiRepointHandler()` moves the wiki projection: manifest `rawSources`/`anchors`, the `sourceFileHashes` entry, every page that names the old path, the index pages, and a `Repointed` log line — no recompilation. `pathsFor(sourceId → paths)` maps ids to paths when they differ.
224
+ - Observational memory: `listInvalidatedIds(vectorStore, scope)` returns the ids a scope currently withholds (`corrected` sources stay) — pass them as `invalidatedIds` to `buildObservationalMemoryProjection()` / recall so already-emitted blocks that rest on a revoked source go stale on the next build instead of being re-injected.
225
+ - Fabric notes (`@arnilo/prism-memory/fabric`): `createFabricRepointHandler({ scope, vectorStore })` is the same handler on **both** seams — registered for a move it rewrites `metadata.fabric.path` on `kind: "file"` notes recorded against `from` (id, text, embedding, `sourceEntryIds`, and every other field reused verbatim, so nothing is re-embedded and no `_lineage` field is invented), and registered on `createDeletionPropagator()` it tombstones the notes of a deleted path through the store's own invalidation path. Notes are store-backed metadata, not derived chunk rows, so this handler is the only path that reaches them; it reads the scope once per leg, selects on `metadata.fabric.path` (never on content), only ever touches its own scope, and reports its count as the `fabric` layer.
226
+
227
+ ### Renaming in batches
228
+
229
+ ```ts
230
+ import { applySourceRenames } from "@arnilo/prism-memory";
231
+
232
+ const { results, failures } = await applySourceRenames({
233
+ scope,
234
+ vectorStore: store,
235
+ authorization: hostVerifiedPrincipal,
236
+ renames: [
237
+ { from: "doc:a", to: "doc:b" },
238
+ { from: "doc:c", to: "doc:d" },
239
+ ],
240
+ handlers: [createWikiRepointHandler({ workspaceRoot })],
241
+ onRenamed: (event) => audit.write({ kind: "rag.repointed", ...event }),
242
+ });
243
+ ```
244
+
245
+ - A thin, audited loop over `repointSource()`: no second re-key path and no new store method. Every pair is handed over as-is, so `repointSource` re-checks its own ACLs (both ids) — nothing is cached or pre-authorized across renames — and each rename walks its own page loop, so a pair above one page still moves completely while every page keeps its one ACL check, one transaction, and one handler pass. The counts folded into the audit event are the rename's totals across pages, and a rename's result carries no `cursor`: a batch is all pages or an error.
246
+ - The batch is validated as a **set before the first store read**: duplicate ids, a chained move (`a→b` then `b→c`), an overlapping move, `from === to`, or an empty id rejects the whole call with `MemoryValidationError` and writes nothing. An empty list is a no-op, not an error.
247
+ - Fail fast by default: the first failing pair writes nothing, later pairs never start, and its original error propagates. `continueOnError: true` records it in `failures` — `{ from, to, error }` — and keeps going.
248
+ - `onRenamed` is the audit sink, called once per rename when it settles: `{ from, to, outcome: "moved", movedChunks, rewrittenEdges, layers }`, or `{ from, to, outcome: "failed", error }` — ids and counts only, never rows or text, and `error` passes through the optional `redact` before it is capped at 256 chars. A pair a fail-fast run never started is not audited; an abort stops the batch and is never recorded as a rename failure.
249
+
250
+ ## Local reranker
251
+
252
+ The default reranker runs in-process — no service, no credential, no per-query egress:
253
+
254
+ ```ts
255
+ import { createHashEmbedder, createMemoryVectorStore } from "@arnilo/prism-memory";
256
+ import { resolveReranker, retrieveContext } from "@arnilo/prism-memory/rag";
257
+
258
+ const reranker = resolveReranker({ kind: "local" }); // Xenova/bge-reranker-base via transformers.js
259
+ const result = await retrieveContext("How do approvals work?", { embedder, store, scope, reranker });
260
+ ```
261
+
262
+ - `resolveReranker({ kind: "local" })` is the zero-config path. The model runtime is a host seam exactly like `Embedder`: `createLocalReranker({ model?, runtime?, onLoad?, cacheDir?, dtype?, device?, allowRemoteModels? })`. Pass `runtime: { load(model) → { id, score({ query, documents, signal }) } }` to inject a runtime the host already owns (transformers.js, onnxruntime-node, llama.cpp). With no `runtime`, the built-in loader resolves `@huggingface/transformers` at first use — the package declares no inference dependency (no new dependency name in any manifest) and nothing resolves it at build/install time.
263
+ - Sizing trade-off: the download is one-time and host-cached, and per-query latency is CPU-bound and grows with candidates × tokens, so keep `topK`/`queryCandidates` near what recall actually needs — the reranker reorders what retrieval returned, it cannot recover a chunk the candidate pool never returned. Measured on the phase 102 corpus (24 queries / 96 chunks: one answering chunk + three mention-only chunks per query, k=5, `Xenova/bge-reranker-base` q8 on x86 CPU, deterministic lexical `createHashEmbedder` baseline, vector-only): recall@5 **0.21 → 0.79**, top-50 median **119–289 ms**, and at the package default 20-candidate pool recall@20 was 0.63 before reranking — corpus, misses, pool-bound numbers, latency, and cache state live in [`docs/_evidence/phase102-local-rerank-latency.md`](_evidence/phase102-local-rerank-latency.md), regenerated by `PRISM_TEST_LOCAL_RERANK=1 npm run test:live`. Treat the numbers as one data point on one machine, not a ceiling: dtype, device, and the embedder move them (a semantic embedder starts higher and gains less). The package guarantees the plumbing (one lazy load, one batched score call per rerank), not the model's speed. The hosted/TEI adapters stay for scale (higher throughput, no local RAM, no download).
264
+ - Host defaults: `dtype: "q8"` with `device: "cpu"` on x86 — fp32 weights are roughly 4× the download for no measurable ranking gain in this size class, and fp16/GPU is worth opting into only when the host already provisions it. Weights are cached per host: pass one `cacheDir` (e.g. `~/.cache/prism/models`) and the runtime lays out one subdirectory per model id, so a second model or a second process reuses the same files — point local embedders running through the same runtime at that directory too. With `cacheDir` omitted the runtime's own default cache applies (inside the installed package). On a cache miss the model is downloaded once into that directory and later runs stay on disk: add `allowRemoteModels: false` on an offline host to fail instead of reaching the model registry, which is exactly what the live leg's second pass proves.
265
+ - Cheap by construction: the model loads lazily once per reranker instance, `score` is called once per rerank with every candidate (never one call per document), and `onLoad({ model, loadMs })` is the only opt-in observability — no document text is ever logged. Zero network after load; the built-in loader only touches the model registry at load time, and `allowRemoteModels: false` pins it to local files.
266
+ - Failure is loud: a missing runtime, an unreachable model, or a runtime that returns no per-document scores throws a redacted `RagValidationError` naming the model and the install path (`npm i @huggingface/transformers` or pass `{ runtime }`). There is deliberately **no** silent lexical fallback.
267
+ - `rerankHits` is unchanged and still owns the caps and the trust boundary: local scores reorder the same `RagHit` references (provenance/trust untouched), byte/ms/concurrency limits apply, and abort/timeout/malformed-score cases fail closed.
268
+
86
269
  ## Implementation example
87
270
 
88
271
  ```ts
@@ -159,11 +342,11 @@ const found = await retrieveContext("leave balance", {
159
342
 
160
343
  - Supply any Phase 7-conforming embedder/vector store, including the in-memory reference or PostgreSQL/pgvector adapter.
161
344
  - Metadata filtering is package-local after a bounded candidate query so existing vector contracts/adapters remain unchanged. Increase `queryCandidates` only when selective filters measurably need it. `filter` never grants document access.
162
- - Document ACL is opt-in via `authorization` on `retrieveContext` / `store.query` / `store.lexicalQuery`. Reference memory and PostgreSQL adapters declare `authorization: "acl"` and apply principal/group predicates **before** top-K. `setSourceAccess` replaces grants per source (empty principal+group lists revoke). Access version is independent of embedding generation; an unresolved `accessVersion` denies. Missing grants deny. Stores that omit the capability throw rather than claim protection. Group lists cap at 32.
345
+ - Document ACL is opt-in via `authorization` on `retrieveContext` / `store.query` / `store.lexicalQuery`. Reference memory and PostgreSQL adapters declare `authorization: "acl"` and apply principal/group predicates **before** top-K. `setSourceAccess` replaces grants per source (empty principal+group lists revoke). Access version is independent of embedding generation; an unresolved `accessVersion` denies. Missing grants deny. Stores that omit the capability throw rather than claim protection. Group lists cap at 32. `onAccessDenied` observes the boundary recheck; it never disables it.
163
346
  - `Reranker` is a host seam, not a provider integration. Return each redacted candidate ID exactly once; Prism retains canonical hit/provenance/trust fields and exposes `retrievalRank` for diagnostics. Add a hosted reranker only when a host owns its credentials, quota, and retry policy.
164
347
  - `createTeiReranker({ baseUrl, model?, timeoutMs?, maxResponseBytes?, ssrf?, allowLoopback?, fetch? })` (`CreateTeiRerankerOptions`) adapts a Hugging Face TEI `POST <baseUrl>/rerank` endpoint (`{query, texts, raw_scores:false}` → `{results:[{index,score}]}`) into the `Reranker` seam. It returns a permutation-only reorder of the same hit objects, so provenance/trust move untouched. Response parsing is strict — short/duplicate/out-of-range indices, non-finite scores, HTTP errors, timeouts, and oversized bodies all fail closed; the `rerankHits` caps (`maxRerankBytes`, `maxRerankMs`, `rerankConcurrency`) still apply around it. The default transport is the core DNS-pinned `pinnedFetch` (redirect-free, byte-bounded to 65,536 by default); HTTPS is required unless `allowLoopback: true` (loopback dev/test) or the host supplies `ssrf`/`fetch` for cluster networking. The adapter validates URL shape only — SSRF policy enforcement stays host-side. No credentials are ever sent; there is no SaaS default URL.
165
348
  - Hosted rerank adapters over the same seam (plan 062): `createOpenAiCompatibleReranker({ baseUrl, model?, apiKey?, timeoutMs?, maxResponseBytes?, ssrf?, allowLoopback?, fetch? })` speaks the OpenAI-compatible `POST <baseUrl>/rerank` route (`{model, query, documents}` → `{results:[{index,relevance_score}]}`; pass the version segment in `baseUrl`, e.g. `https://api.jina.ai/v1`), and `createVoyageReranker({ baseUrl, model?, apiKey, … })` adapts Voyage AI (`…/v1/rerank` → `{data:[{index,relevance_score}]}`; `apiKey` required). Both send one request per rerank — no adapter-side batching — never send `top_k` (the retrieval seam owns top-K), return the same permutation-only reorder, and fail closed on the same malformed-response/HTTP/timeout/byte-bound cases. `apiKey` rides as `Authorization: Bearer …` and is never logged; errors carry status/host only. No SaaS default URL — hosts own credentials, quota, and retry policy.
166
- - `createFakeReranker()` is a network-free deterministic reranker (query-term-overlap scoring, stable ties) and `runRerankerConformance(createReranker)` is the shared network-free conformance for any `Reranker` implementation: empty input → `[]`, output is a permutation of the exact input references (provenance/trust untouched), repeated calls are deterministic.
349
+ - `createFakeReranker()` is a network-free deterministic reranker (query-term-overlap scoring, stable ties) and `runRerankerConformance(createReranker)` is the shared network-free conformance for any `Reranker` implementation: empty input → `[]`, output is a permutation of the exact input references (provenance/trust untouched), repeated calls are deterministic. `createLocalReranker()` passes the same conformance; `resolveReranker({ kind: "local" })` is the zero-service default, and no reranker ever constructs itself from retrieval options (host config only).
167
350
  - Hybrid retrieval: pass `lexical: "fts"` (or `"bm25"` when the store supports it) to `retrieveContext()`; the two legs are fused with reciprocal-rank fusion (`fusion: "rrf"`, `rrfK` 60 default; the pure helper `fuseReciprocalRank()` returns `FusedCandidate[]` for custom orchestration). Stores advertise support via `lexicalModes?: readonly LexicalMode[]` and `tokenizeLexical()` is the shared tokenizer. Each hit's provenance `retrieval` field reports `vector`/`lexical`/`hybrid`; fusion internals expose `RetrievalLeg`.
168
351
  - Multi-scope retrieve: `scopes: RagScope[]` searches each exact scope against that scope's current generation, then runs **one** RRF over the union and **one** rerank. The query is embedded once. `queryCandidates` is per scope. Duplicate scopes are dropped. `HARD_RETRIEVE_SCOPE_CAP` is 8.
169
352
  - Embedder identity/drift guard: `Embedder.id` (memory contract) is stamped onto every vector record as `embedderId`. `retrieveContext()` fails closed with `ERR_PRISM_RAG_EMBEDDER_MISMATCH` when a stored record's `embedderId` or dimensions differ from the active embedder (for example after a model change) — re-index the source before retrieving. Legacy records without an `embedderId` also fail closed, naming the re-index path.
@@ -178,7 +361,8 @@ const found = await retrieveContext("leave balance", {
178
361
  ## Security and performance notes
179
362
 
180
363
  - Every index/query includes exact tenant/resource/corpus scope; returned records are rechecked and malformed/foreign records fail closed. `retrieveContext` accepts `scope` or `scopes` (never both, never neither). Empty `scopes` is the host “no allowed corpora” path — no embed, no search, no rerank. A hit whose stored scope is not in the requested list fails closed. Generation filters stay per scope.
181
- - When `authorization` is set, unauthorized text, titles, citations, counts, and reranker payloads never leave the store. Recheck runs after fusion (before rerank) and again after rerank before injection, so revocation between those steps drops the candidate. `authorization.tenantId` must match every retrieve scope.
364
+ - When `authorization` is set, unauthorized text, titles, citations, counts, and reranker payloads never leave the store. Recheck runs after fusion (before rerank) and again after rerank before injection, so a revoke that lands between those steps drops the candidate; the post-rerank gate deliberately re-reads the live grant instead of reusing the pre-filter decision, and both gates dedupe to one lookup per distinct source. A thrown grant lookup withholds the source and reports `reason: "check_failed"` rather than failing open or aborting the query. `authorization.tenantId` must match every retrieve scope.
365
+ - Re-pointing is the only path that can re-key a source's row ids; it is ACL-gated on both ends, transactional, capped, and refuses destination collisions. It changes identity metadata only — content, embeddings, provenance, and trust are copied verbatim, and no text is ever re-embedded or re-injected because of a move.
182
366
  - Embedding identity is a privacy/consistency boundary: records from a different embedder (or dimension) never silently mingle with new ones — retrieval fails closed and names the re-index path. Generation pointers are scope-scoped: a pointer row belongs to exactly one scope, and visibility is computed inside the store (SQL), never by post-filtering in JS.
183
367
  - Source IDs become citation/storage IDs and must be stable non-secret identifiers. Text and user metadata can be redacted before external embedding and persistence.
184
368
  - Heading metadata is document text only — it passes through the existing `maxMetadataBytes` cap as chunk metadata; no new content path is introduced.
@@ -187,6 +371,7 @@ const found = await retrieveContext("leave balance", {
187
371
  - Remote sources must pass existing resource/media trust, SSRF, MIME, and byte policies before their decoded text reaches this package.
188
372
  - `replaceSource()` stages every bounded embedding before opening the store transaction. It requires a source-aware transactional store and fails closed rather than pretending generic upserts are atomic. `createMemoryVectorStore()` supplies the reference `getBySource()` / transaction capability; durable stores must implement equivalent exact-scope behavior.
189
373
  - `deleteSource()` rechecks every returned record's tenant/resource/corpus and source metadata before delete. Same source IDs in another corpus remain untouched.
374
+ - Deletion propagation reuses the existing memory ACL (`checkSourceAccess` plus the caller's host-verified `authorization`) and rejects unprivileged callers before writing any tombstone; it is only reachable from the explicit `createDeletionPropagator()` seam, never from `retrieveContext()` or any `filter`/query option. The retrieval-side tombstone guard is an exclusion only — it grants nothing.
190
375
  - Parsers enforce byte/page/time caps, abort before and after parsing, decode UTF-8 strictly, and strip HTML script/style content. Parsed and retrieved text remains untrusted inert context; it never gains tool authority.
191
376
  - Rerankers receive redacted input under byte/time/concurrency caps. Timeout, abort, unknown/duplicate/missing IDs, oversized input, and reranker failures fail closed; returned objects cannot overwrite Prism provenance/trust fields. The TEI adapter adds fail-closed response parsing (permutation completeness, finite scores) and honors the 65,536-byte response ceiling; SSRF/URL policy is host-side (see Extension notes). The hosted OpenAI-compatible and Voyage adapters carry the same guarantees and add Bearer credentials that are never logged and error messages that never contain document text or the API key.
192
377
  - Telemetry is a host-owned seam: `RagTelemetry` adapter (`createRagTelemetry()`) drops anything outside a fixed span-name set and `rag.*`-shaped attribute keys, so raw chunk text never reaches the tracer unless the host's own `attributeFilter` opts it in; when the seam is absent, instrumentation costs nothing.