ds4-context-engine 0.1.0

package/docs/MEMORY_AND_PINS.md

@@ -0,0 +1,165 @@
+# Memory and Persistent Pins
+
+M9 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical.
+
+## Authority and scope
+
+Pins are explicit user-confirmed context with maximum planner priority:
+
+- `session`: available throughout the current Pi session tree;
+- `branch`: available only when its creation leaf is on the active branch;
+- `project`: available to sessions using the same canonical project path, but only when Pi reports the project trusted.
+
+Memory is quoted historical data rather than policy:
+
+- `session`: available only in the source Pi session;
+- `project`: shared across sessions for the same trusted canonical project path.
+
+Pins remain subordinate to system/developer instructions. Memory tells the model to validate a claim against current evidence. Neither is created automatically in M9.
+
+## Commands
+
+```text
+/context pins
+/context pin [--scope session|branch|project] [--classification LEVEL] [--source ENTRY] [--file PATH] <content>
+/context pin --scope session --supersedes PIN_ID [--classification LEVEL] <replacement>
+/context unpin PIN_ID [reason]
+
+/context memory
+/context memory add [--scope session|project] [--classification LEVEL] [--key KEY] [--source ID,ID] <claim>
+/context memory supersede MEMORY_ID [--classification LEVEL] [--source ID,ID] <new claim>
+/context memory invalidate MEMORY_ID [reason]
+/context memory expire MEMORY_ID [reason]
+```
+
+Arguments support single/double quotes and backslash escaping. `--` ends option parsing. Source entry IDs must be on Pi's active branch. Project scope requires Pi project trust.
+
+Mutations are manual-first. Repeating the same normalized pin/claim returns its existing ID without appending another entry.
+
+## Canonical append-only mutations
+
+Every accepted operation is appended through `pi.appendEntry()` as a Pi `CustomEntry`:
+
+```text
+ds4-context-pin-v1
+ds4-context-memory-v1
+```
+
+Custom entries do not participate directly in Pi's LLM context. Their payload is an immutable versioned mutation:
+
+```text
+add
+supersede previous ID with a new immutable record
+status -> deleted / invalid / expired
+```
+
+No row is silently overwritten. SQLite mutation and materialized tables are disposable projections. On startup or `/context rebuild-index`, DS4:
+
+1. indexes the custom entries with their canonical scoped entry keys;
+2. replaces the current session's mutation projection;
+3. replays all known mutations in timestamp + canonical entry order;
+4. rebuilds memory, pin, source, lifecycle, and FTS rows transactionally.
+
+Deleting `context.db` and reopening the canonical source session reconstructs its state. Project items from other sessions reappear when those canonical sessions are replayed.
+
+## Supersession and contradiction handling
+
+A memory may have an explicit normalized key:
+
+```text
+/context memory add --key package-export-mode \
+  "Package export mode defaults to PerEndpoint."
+```
+
+For claims shaped like `subject defaults to value`, `subject is value`, or `subject = value`, DS4 derives a conservative key automatically. An active item with the same scope/key and a different claim is a conflict. Opposite-polarity claims such as `Feature is enabled` and `Feature is disabled` are also detected when their normalized bases match.
+
+A conflicting `add` is rejected with the IDs involved. The user must issue explicit supersession:
+
+```text
+/context memory supersede MEMORY_ID \
+  "Package export mode defaults to SingleFile."
+```
+
+The old item remains stored as `superseded` and points to the new active item. During replay, concurrent active records with the same key are both preserved; the deterministic later record is marked `invalid` with a conflict reason rather than replacing the earlier one.
+
+Pins use the same immutable replacement pattern through `--supersedes`. `/context unpin` records a soft `deleted` lifecycle mutation.
+
+## Context selection
+
+The managed planner inserts selected synthetic messages immediately before the current request in this order:
+
+```text
+persistent pins      priority 950, mandatory within maxPinnedTokens
+memory               priority 90, maxMemoryTokens
+historical retrieval priority 85
+project snippets     priority 80
+current request      mandatory
+```
+
+All applicable pins are considered in deterministic creation order. Branch pins are hard-filtered against `SessionManager.getBranch()`.
+
+Memory ranking uses current request identifiers, file/symbol/keyword terms, optional keys, scope authority, and recency. If any memories match, unrelated memory is excluded. When nothing matches, at most three recent active items provide conservative continuity. `memory.maxResults` and `context.maxMemoryTokens` bound the final set.
+
+A pin budget overflow causes planner fallback rather than silently splitting mandatory context. Commands cap individual pin/claim characters, while final system/tools/messages validation still enforces the active model hard limit.
+
+## Prompt-injection boundary
+
+Pins are rendered as user-confirmed content:
+
+```text
+[DS4 PINNED CONTEXT — USER-CONFIRMED]
+...
+Pinned content JSON: "..."
+[END DS4 PINNED CONTEXT]
+```
+
+Memory is rendered as quoted data:
+
+```text
+[DS4 DURABLE MEMORY — QUOTED DATA]
+...
+Claim JSON: "..."
+[END DS4 DURABLE MEMORY]
+```
+
+User text is JSON-quoted. Context Manifests contain IDs, scope, classification, key, source session/entry IDs, score, token estimate, and selection reason—but never pin content or memory claim text.
+
+M10 stores an optional classification in the canonical mutation. Before a remote call, a prohibited pin/memory is omitted as a whole and recorded only by ID/classification/reason. Explicit classification cannot be downgraded by markers inside its content. See [`PRIVACY.md`](PRIVACY.md).
+
+## SQLite schema v9
+
+Schema v9 extends materialized `memory_items` and `pins` with:
+
+- normalized key, origin session, branch leaf;
+- update/status reason, optional classification in `metadata_json`, and immutable supersession links;
+- indexes for scope/lifecycle/branch selection.
+
+`memory_mutations` and `pin_mutations` reference canonical indexed Pi custom entries. `memory_sources` retains exact scoped entry provenance. `memory_fts` is rebuilt transactionally.
+
+Migration preserves legacy materialized rows for inspection. Because they have no canonical mutation entry, a later full replay may discard them; new operations are always event-sourced.
+
+## Diagnostics
+
+```text
+/context status
+/context tokens
+/context manifest
+/context included
+/context excluded
+/context pins
+/context memory
+/context health
+/context rebuild-index
+```
+
+Structured logs contain mutation/item IDs, scopes, counts, lifecycle, and warning counts—not pin or claim text.
+
+## Performance
+
+`tests/benchmarks/memory-selection.bench.ts` ranks 1,000 active session/project memories. On the development host:
+
+```text
+mean 5.85 ms, p99 7.22 ms, max 8.49 ms
+```
+
+This is below the initial 50 ms typical context-planning target; it is not a portable guarantee.

package/docs/MODEL_AWARENESS.md

@@ -0,0 +1,131 @@
+# Advanced Model Awareness
+
+M11 resolves an independent deterministic planning profile for every exact `provider/model`. Pi's model descriptor remains the default source for context window, output ceiling, reasoning, and image support; explicit DS4 overrides can repair provider metadata or tune category limits without changing canonical session state.
+
+## Profile resolution
+
+Overrides merge from least to most specific:
+
+```text
+*
+provider/*
+provider/model-id
+```
+
+Supported fields are:
+
+```text
+contextWindow
+maxOutputTokens
+safetyMarginTokens
+recentTailTokens
+maxRetrievedHistoryTokens
+maxProjectTokens
+```
+
+Unknown fields, malformed keys, non-integer token limits, impossible output/window combinations, and unsafe calibration settings reject that configuration source. Global and trusted-project override maps merge by key. Project configuration remains subject to Pi's project trust decision.
+
+Example:
+
+```json
+{
+  "modelAwareness": {
+    "enabled": true,
+    "calibrationWindow": 24,
+    "minimumCalibrationSamples": 3,
+    "calibrationRatioLowerBound": 0.5,
+    "calibrationRatioUpperBound": 2.0,
+    "overrides": {
+      "*": { "safetyMarginTokens": 2048 },
+      "ollama/*": { "recentTailTokens": 16000 },
+      "openrouter/vendor/model": {
+        "contextWindow": 200000,
+        "maxRetrievedHistoryTokens": 12000
+      }
+    }
+  }
+}
+```
+
+Setting `modelAwareness.enabled` to `false` disables calibration, profile overrides, and adaptive history/project retrieval. The pre-M11 model-adaptive recent-tail safety ceiling remains active.
+
+## Adaptive category budgets
+
+With no explicit override, category ceilings derive from the resolved context window and remain bounded by the corresponding `context.*` maximum:
+
+| Context window | Recent tail | Historical retrieval | Project retrieval |
+|---|---:|---:|---:|
+| up to 40k | 12k | 4k | 4k |
+| up to 128k | 24k | 8k | 12k |
+| up to 256k | 32k | 16k | 20k |
+| larger | 64k | 32k | 32k |
+
+The default `context.recentTailTokens` is 64k so these automatic tiers are effective by default. Category limits are independent: a small model does not spend its entire input on retrieval, while a large model can retain more verbatim work and source evidence.
+
+## Model-specific token calibration
+
+Each successful assistant response is correlated with one pending Context Manifest. Provider input is defined exactly as Pi defines request input usage:
+
+```text
+actualInputTokens = input + cacheRead + cacheWrite
+ratio = actualInputTokens / raw chars-v1 estimate
+```
+
+Calibration is isolated by exact provider and model ID. The latest configured window is validated and processed deterministically:
+
+1. reject invalid or zero samples;
+2. reject ratios outside the configured hard bounds;
+3. compute the bounded median and median absolute deviation (MAD);
+4. reject values outside the larger of 0.05, 10% of the median, or three scaled MADs;
+5. require `minimumCalibrationSamples` accepted values;
+6. use the accepted median as the next-call multiplier.
+
+Until enough samples exist, the multiplier remains `1.0`. The default window is 24 samples, minimum is 3, and hard ratio bounds are 0.5–2.0. Outliers remain counted in metadata but do not influence the applied ratio.
+
+Provider-token capacities are computed first from context window, output reserve, safety margin, and policy ratios. Planner limits are then converted into local-estimator units by dividing by the accepted multiplier. Raw manifest estimates remain uncalibrated so future samples do not feed a corrected estimate back into itself. Adaptive tail/history/project budgets use the same conversion.
+
+## Provider cache metrics
+
+Schema v10 stores separate per-call values for:
+
+```text
+input_tokens
+cache_read_tokens
+cache_write_tokens
+```
+
+The manifest records the finalized call's three values, total provider input, and read/write shares. The active model profile reports aggregate totals and shares over its calibration window. Error, aborted, missing, zero-usage, duplicate-response, or uncorrelated messages create no sample.
+
+Cache metrics are observational. Stable deterministic ordering and unchanged system/tool prefixes make cache reuse possible, but the provider decides whether a prefix is reusable. M12's separately configured native continuation may attach a real provider response handle only after exact hashed-prefix validation; it never fabricates a handle or treats cache/continuation state as canonical. See [`NATIVE_CONTINUATION.md`](NATIVE_CONTINUATION.md).
+
+## Model and provider switches
+
+`model_select` records source (`set`, `cycle`, or `restore`), previous profile, whether the exact profile was seen before, and cache disposition. A change to a different provider/model is treated as a cold cache boundary and invalidates volatile native-continuation state. Switching back reuses that exact model's calibration history and adaptive policy, not another model's ratio.
+
+Every subsequent `context` call rebuilds provider-facing context from canonical Pi JSONL and current derived indexes. It reruns privacy classification for the new destination, so a local-to-remote switch can remove `local-only` content while a later switch back to an allowed local model can recover it from canonical state. No switch mutates or truncates the session, memory/pin mutations, project source, artifact bytes, or summary provenance.
+
+## Manifest and diagnostics
+
+The metadata-only Context Manifest includes:
+
+- profile key, effective window/output/safety margin, and matched override keys;
+- calibration window, bounds, accepted/rejected/outlier counts, median, and applied ratio;
+- adaptive nominal and estimator-adjusted category limits;
+- cache totals/shares over the model window;
+- switch source, prior profile, reuse flag, and cache disposition;
+- finalized per-call uncached input, cache-read, cache-write, and total input usage.
+
+It never stores prompt text, provider payloads, cache keys, headers, classified spans, or credentials.
+
+Use:
+
+```text
+/context model
+/context tokens
+/context manifest
+/context status
+```
+
+## Performance and tests
+
+`tests/benchmarks/model-awareness.bench.ts` measures a bounded 200-sample calibration analysis and repeated 32k/128k/200k profile resolution. Unit and golden tests cover deterministic tiers, override precedence, robust outlier rejection, cache accounting, and calibrated budgets. Integration tests switch local/remote providers and 32k/128k/200k models while checking profile isolation, privacy re-enforcement, canonical JSONL preservation, SQLite cache metrics, and profile reuse.

package/docs/NATIVE_CONTINUATION.md

@@ -0,0 +1,127 @@
+# Optional Native Continuation
+
+M12 adds an opt-in OpenAI Responses optimization without making provider state canonical.
+
+## Decision
+
+Pi 0.84.3 exposes two relevant integration surfaces:
+
+- `before_provider_request` can replace the serialized payload, but it cannot transparently retry the same model call after a provider rejects stale state;
+- `registerProvider(..., { streamSimple })` can delegate to Pi's built-in serializer/transport while observing the terminal stream and retrying before any partial assistant event is exposed.
+
+DS4 therefore registers a narrow provider wrapper only for explicitly configured providers. The wrapper delegates to Pi's built-in `openai-responses` implementation. It does not implement authentication, HTTP, SSE parsing, tool serialization, usage accounting, or model discovery itself.
+
+OpenAI conversation IDs were evaluated but are not enabled in M12. `previous_response_id` is the smaller state primitive, requires no second canonical conversation tree, and can always fall back to the full managed replay. Pi's `openai-codex-responses` transport already has its own connection-scoped cached continuation and is not replaced by this wrapper.
+
+## Explicit provider-storage consent
+
+Native continuation is disabled by default. Eligible requests require both:
+
+```json
+{
+  "nativeContinuation": {
+    "enabled": true,
+    "allowProviderStorage": true,
+    "profiles": ["openai/*"],
+    "maxStateAgeMs": 1800000,
+    "retryManagedReplay": true
+  }
+}
+```
+
+`allowProviderStorage` acknowledges that DS4 changes eligible OpenAI Responses payloads from Pi's default `store: false` to `store: true`. Provider-side retention and deletion are governed by the selected provider. `maxStateAgeMs` limits only reuse of the volatile local handle; it does not delete provider-side data.
+
+Profiles must be exact `provider/model` values or explicit `provider/*` wildcards. A global wildcard is rejected. Model IDs may contain `/`, for example `proxy/vendor/model`.
+
+A trusted project configuration may enable this feature. Configuration from an untrusted project remains ignored by the existing trust gate.
+
+## Request algorithm
+
+For an eligible main-agent request, after all Pi `before_provider_request` handlers have produced the final privacy-sanitized payload, DS4:
+
+1. normalizes the full request to `store: true`;
+2. hashes every full `input` item independently with deterministic SHA-256;
+3. hashes all non-input request options, including model, tools, output settings, and cache settings;
+4. compares the current input prefix with:
+
+```text
+previous full request input
++ previous serialized response output items
+```
+
+5. only on an exact hash match sends:
+
+```json
+{
+  "previous_response_id": "<volatile handle>",
+  "input": ["only new suffix items"],
+  "store": true
+}
+```
+
+6. otherwise sends the complete DS4-managed payload and establishes a new chain from the successful response.
+
+The manager retains hashes plus the minimum volatile response handle. It does not retain prompt text, tool payloads, or response output text. The provider handle is never logged or copied into a Context Manifest. Pi may persist the provider's normal `AssistantMessage.responseId` in canonical JSONL; DS4 adds no custom canonical entry or SQLite continuation table.
+
+## Transparent managed-replay retry
+
+If a continuation request is rejected before streaming starts and the provider error identifies invalid, expired, unknown, or missing previous-response state, the wrapper:
+
+1. suppresses that unstarted error event;
+2. clears the volatile handle;
+3. rebuilds the request through Pi's normal serializer and all payload hooks;
+4. retries once with the complete managed replay and no `previous_response_id`.
+
+Unrelated failures such as authentication or rate limiting are not retried by this layer. Errors after a stream has started are never replayed because doing so could duplicate visible output or tool effects. Pi's ordinary retry policy remains available after DS4 returns a terminal failure.
+
+## Conservative invalidation
+
+State is discarded on:
+
+- provider or model switch;
+- session branch/tree navigation;
+- successful compaction;
+- session shutdown/reload or degraded runtime;
+- local state age above `maxStateAgeMs`;
+- changed model/tools/output/cache request options;
+- any managed-context prefix mismatch, including changed summaries, privacy filtering, project evidence, pins, memory, or active tools;
+- empty deltas;
+- missing provider response IDs or non-reconstructable response items;
+- provider request/stream failure.
+
+Nested DS4 summary calls use a fresh Pi `sessionId` and are never opted into provider storage or continuation. Observer-mode calls also remain unchanged.
+
+## Privacy and canonical state
+
+The continuation transform runs after Pi's payload callback chain, so it only removes an already checked prefix and adds state metadata. It never reintroduces classified content. If enabled privacy enforcement replaces a failed payload with `{}`, continuation is skipped and the provider call fails closed as before.
+
+The full Pi JSONL history, DS4 summaries, memory/pins, and Context Manifests remain sufficient for managed replay. Deleting or expiring provider state affects optimization only.
+
+## Diagnostics
+
+Use:
+
+```text
+/context continuation
+/context manifest
+/context status
+/context health
+```
+
+Diagnostics expose only:
+
+- enabled/consent/profile/wrapper state;
+- full versus continuation request counts;
+- full/sent/omitted input-item counts;
+- state age, generic fallback/invalidation reason, and retry outcome;
+- warning counts.
+
+They never expose provider response IDs or provider payload content.
+
+## Tests and performance
+
+Unit tests cover exact/wildcard eligibility, consent validation, cold establishment, exact suffix derivation, option/prefix/age invalidation, nested-call exclusion, and response-ID non-disclosure. Stream tests cover pre-stream stale-handle replay and refusal to retry unrelated failures. Integration coverage verifies provider registration, metadata-only manifests/SQLite, and branch invalidation.
+
+`tests/benchmarks/native-continuation.bench.ts` hashes and verifies a 1,000-item managed prefix. On the development host it measured about 2.88 ms mean and 5.39 ms p99, below the existing 50 ms context-operation target; this is not a portable guarantee.
+
+A real Pi 0.84.3 isolated RPC E2E used a local OpenAI Responses-compatible HTTP server for three turns. It observed full → one-item delta → rejected stale delta → automatic full replay, with `store: true`, successful retry diagnostics, canonical JSONL preservation, metadata-only manifests, SQLite integrity/FK checks, and zero `extension_error` events.

package/docs/PORTABLE_CORE.md

@@ -0,0 +1,92 @@
+# Portable Core
+
+M13 extracts the runtime-neutral implementation into the independently buildable `ds4-context-core` workspace package. The root `ds4-context-engine` package remains the Pi integration.
+
+## Dependency rule
+
+```text
+agent runtime
+    ↓
+runtime adapter
+    ↓
+ds4-context-core
+```
+
+Core never imports an adapter or runtime SDK. The Pi adapter imports core through its public ESM exports.
+
+The core may use Node.js standard-library facilities such as `node:sqlite`, filesystem APIs and cryptographic hashing. Portable means independent of Pi and reusable by another Node-based agent runtime; it does not mean browser-compatible.
+
+## Package contents
+
+`packages/core/src` owns:
+
+- canonical message and model projections;
+- token estimation, calibration and context budgets;
+- deterministic planning and atomic group validation;
+- manifest and provenance models;
+- summary contracts, validation and graph records;
+- historical and project retrieval;
+- project indexing and artifact policy;
+- memory/pin materialization;
+- privacy classification and provider policy;
+- native-continuation eligibility and hash state;
+- rebuildable SQLite repositories;
+- stable serialization, hashing and logging.
+
+The root adapter owns all Pi-specific behavior:
+
+- Pi message, model and session conversion;
+- JSONL branch and custom-entry projection;
+- extension lifecycle hooks and commands;
+- summary model completion through Pi's registry;
+- OpenAI Responses transport wrapping through Pi AI;
+- fail-open integration orchestration and privacy fail-closed enforcement.
+
+## Adapter responsibilities
+
+A future runtime adapter must:
+
+1. preserve its runtime's canonical history and expose stable source identifiers;
+2. convert native messages to DS4 canonical messages without losing tool-call/result atomicity;
+3. describe provider/model limits using the core model projection;
+4. supply the current branch, request, system prompt, tools and trusted project path;
+5. apply core plans without persisting provider-facing synthetic context as canonical history;
+6. append memory/pin mutations to canonical runtime history before materializing derived state;
+7. invoke model completion at the adapter boundary for generated summaries;
+8. enforce privacy immediately before provider transport;
+9. discard or rebuild SQLite and artifact projections safely;
+10. fall back to native runtime behavior when operational integration fails.
+
+## Build and exports
+
+```bash
+npm run build:core
+npm run typecheck
+npm test
+npm run pack:check
+npm pack --dry-run --workspace ds4-context-core
+```
+
+TypeScript sources compile to `packages/core/dist` as ESM JavaScript, source maps and declaration files. The npm package exports a top-level API and fine-grained subpaths such as:
+
+```ts
+import { calculateContextBudget } from "ds4-context-core";
+import { planManagedContext } from "ds4-context-core/planner/context-planner";
+```
+
+The Pi package declares an exact same-release dependency on `ds4-context-core`. Release order is therefore core first, adapter second. `npm run pack:check` verifies both tarball inventories, installs them together in a clean temporary consumer, probes core ESM exports and starts the packaged adapter with isolated Pi RPC state. See [Releasing DS4](RELEASING.md) for the publication checklist.
+
+## Enforcement
+
+`tests/unit/portable-core-boundary.test.ts` recursively rejects Pi SDK and adapter imports from core source, then imports the compiled package and exercises portable model/budget policy. Existing integration tests consume core through package exports, so the Pi adapter is tested across the actual package boundary.
+
+## State guarantees
+
+Extraction does not change DS4 state semantics:
+
+- Pi JSONL remains canonical for Pi sessions;
+- SQLite remains disposable and rebuildable;
+- compaction remains non-destructive and strictly validated;
+- provider continuation handles remain volatile;
+- planner, retrieval, compaction and persistence failures still fail open at the adapter boundary;
+- enabled privacy enforcement still fails closed before remote transport.

package/docs/PRIVACY.md

@@ -0,0 +1,157 @@
+# Privacy and Remote Provider Policy
+
+M10 adds an opt-in, provider-aware privacy boundary around the existing managed context. Pi JSONL remains canonical and may retain restricted local content; the policy controls only provider-facing copies, summary-generation input, artifact excerpts, and the final serialized provider payload.
+
+Privacy is disabled by default for backward compatibility.
+
+## Classifications
+
+DS4 recognizes four classifications, ordered from least to most restrictive:
+
+```text
+normal < internal < sensitive < local-only
+```
+
+`local-only` is a hard ceiling: configuration validation rejects every remote allow rule containing it. A marker cannot downgrade a stricter default or explicit classification.
+
+Content can be marked inline:
+
+```text
+\[ds4:internal]team-only context\[/ds4:internal]
+\[ds4:sensitive]restricted value\[/ds4:sensitive]
+\[ds4:local-only]must remain on this machine\[/ds4:local-only]
+```
+
+Remove the example backslashes to activate the markers. A leading marker without a closing tag classifies the remainder of that text. Unbalanced markers split across content blocks conservatively classify the complete message. Markers are control metadata: allowed local/remote copies contain the enclosed text without the tags; disallowed copies contain only a classification placeholder. Prefix a marker's opening bracket with `\` when the syntax itself must remain literal source/documentation.
+
+Persistent pin and memory commands also accept an explicit classification:
+
+```text
+/context pin --scope project --classification local-only -- "Private deployment rule"
+/context memory add --scope project --classification sensitive --key release-channel -- "Release channel defaults to private."
+/context memory supersede MEMORY_ID --classification internal -- "Replacement claim"
+```
+
+The classification is part of the canonical Pi custom-entry mutation and survives SQLite deletion, session resume, branch changes, and compaction. Existing unclassified mutations use `privacy.defaultClassification` at selection time.
+
+Automatic sensitivity classification remains disabled. M10 uses explicit markers, explicit pin/memory metadata, and the configured default.
+
+## Provider destination and allow rules
+
+All providers are treated as remote unless their exact, case-insensitive provider ID appears in `localProviders`.
+
+```json
+{
+  "privacy": {
+    "enabled": true,
+    "defaultClassification": "normal",
+    "localProviders": ["ollama", "llama-cpp", "lmstudio"],
+    "remoteDefaultAllowed": ["normal", "internal"],
+    "remoteProviders": {
+      "openrouter": ["normal"],
+      "anthropic": ["normal", "internal"],
+      "private-gateway": ["normal", "internal", "sensitive"]
+    },
+    "redactSecrets": true
+  }
+}
+```
+
+Rules are exact allow lists, not maximum-enum shorthand. `remoteDefaultAllowed` applies to unknown remote providers. Provider-specific rules override it. A provider cannot appear in both `localProviders` and `remoteProviders`. Only mark a provider local when its transport is guaranteed to remain local; a trusted project configuration has the same execution authority as other trusted Pi project resources.
+
+Defaults when privacy is enabled:
+
+```text
+known configured local provider -> normal, internal, sensitive, local-only
+remote provider override        -> exact configured allow list
+other provider                  -> normal, internal
+```
+
+The shipped `faux` test provider is in the default local list. Real unknown/custom provider IDs are remote by default.
+
+## Enforcement pipeline
+
+### Context hook
+
+Before artifact condensation, retrieval, or planning, DS4 sanitizes every native `AgentMessage` content field and records a classification without storing its text in diagnostics. Tool call arguments, text/thinking blocks, tool results, and image/binary content participate; protocol identifiers such as role, type, model, tool name, and call ID remain intact.
+
+DS4 separately sanitizes the effective system prompt and active tool descriptions/schema text for token accounting and manifest hashing. The final provider hook applies the same policy to their actual serialized forms.
+
+Historical retrieval, project snippets, pin, and memory supplements are checked independently. A supplement containing a prohibited block is omitted as a whole and receives a metadata-only `excluded due to privacy policy` record. Native turns remain structurally valid: prohibited spans become placeholders so current-user and complete tool-call/result atomic groups survive.
+
+Planner exceptions return the already-sanitized native array. If privacy preparation itself fails, every message content field is replaced with a fail-closed placeholder while protocol identity is preserved. Privacy enforcement therefore does not use the ordinary fail-open rule.
+
+### Compaction and summary graph
+
+Conversation text, custom instructions, and file lists are sanitized before summary generation. Validation runs against the sanitized source. Generated segment/aggregate nodes inherit the highest source classification and persist it as a DS4 marker around the summary. A local model may summarize local-only source, but switching to a remote provider strips that summary before it can be serialized.
+
+Pi's fallback compactor is still covered by the final provider-payload hook.
+
+### Artifact store and search
+
+The local content-addressed object may retain exact restricted bytes because Pi JSONL is canonical and object hashes require exact recovery. Artifact references persist the derived classification in `metadata_json`. Context selection hides prohibited tool results before offload/reference injection. `context_artifact_search` applies the stored artifact classification to every returned excerpt; a remote request receives no matches/content for a prohibited artifact.
+
+### Final provider payload
+
+`before_provider_request` runs after provider-specific serialization. DS4 recursively checks known provider content containers (`system`, `messages`, `input`, `contents`, `context`, tool descriptions/arguments, and related text fields), strips classification markers, removes prohibited blocks, and redacts credential-like values. Structural provider fields remain unchanged.
+
+The handler catches its own failures because Pi 0.84.3 reports extension-hook exceptions and would otherwise continue with the unchanged payload. On an unexpected sanitizer failure DS4 returns an empty object, intentionally causing the provider request to fail rather than leaking content.
+
+Extension handlers execute in load order. A later extension can replace the payload after DS4. For the strongest boundary, load DS4 after every extension that mutates context or provider payloads. DS4 cannot police network traffic created directly by another extension outside Pi's provider pipeline.
+
+### Optional native continuation
+
+When M12 native continuation is explicitly enabled, the registered OpenAI Responses wrapper invokes Pi's complete payload callback chain first. It then hashes and validates the final sanitized full prefix, removes only an exact already-sent prefix, and adds `store: true` plus a volatile `previous_response_id`. It never adds content or bypasses the final privacy sanitizer. A fail-closed `{}` payload is ineligible and remains `{}`.
+
+`allowProviderStorage: true` is mandatory because provider persistence changes the privacy/retention posture. Profiles are explicit and cannot use a global wildcard. Response IDs, payload text, and item hashes are absent from manifests and logs. See [`NATIVE_CONTINUATION.md`](NATIVE_CONTINUATION.md).
+
+## Secret redaction
+
+For remote destinations, `redactSecrets` recognizes common private-key blocks, Bearer tokens, OpenAI/Anthropic-style keys, GitHub tokens, AWS access IDs, and common key/token/password assignments. It is defense in depth, not an automatic data-classification oracle. Local providers retain allowed exact content.
+
+Redaction is performed on provider-facing copies only. Canonical JSONL, local artifact objects, and trusted local SQLite indexes may retain source bytes.
+
+## Manifest and logging contract
+
+Context Manifest items may contain:
+
+- classification;
+- source ID/kind and inclusion/exclusion reason;
+- counts by selected classification;
+- provider ID and local/remote destination;
+- allowed classification names;
+- blocked/excluded/redacted counts;
+- final provider-check/redaction counts;
+- enforcement stage.
+
+They never contain classified text, memory claims, pin content, project/retrieval excerpts, provider payloads, or secret values. A provider check updates the pending manifest in place after serialization.
+
+Structured privacy logs contain provider/destination and numeric counters only. On a fail-closed exception DS4 records only the exception type, never its message or payload.
+
+Inspect locally with:
+
+```text
+/context privacy
+/context manifest
+/context included
+/context excluded
+/context health
+```
+
+## Tests and performance
+
+Coverage includes:
+
+- span and prefix classification, no-downgrade, split-marker fail-closed behavior;
+- local/remote and provider-specific rules;
+- current messages, system prompt, tools, history, project, pins, memory, summaries, and artifacts;
+- secret redaction and metadata-only manifests/logs;
+- provider switch, planner/preparation failure, final provider-payload recheck, and real Pi faux-provider E2E.
+
+`tests/benchmarks/privacy-policy.bench.ts` sanitizes a 1,000-message provider payload on the development host:
+
+```text
+mean 1.75 ms, p99 3.03 ms, max 6.91 ms
+```
+
+This is below the initial 50 ms typical context-operation target, not a portable guarantee.