ds4-context-engine 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DS4 Context Engine contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,399 @@
+# DS4 Context Engine for Pi
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Pi 0.84.3](https://img.shields.io/badge/Pi-0.84.3-blue.svg)](https://github.com/earendil-works/pi)
+[![Node.js >=22.19](https://img.shields.io/badge/Node.js-%3E%3D22.19-339933.svg)](https://nodejs.org/)
+
+DS4 Context Engine is a non-destructive, provider-independent context management layer for [Pi](https://github.com/earendil-works/pi). It keeps Pi's native JSONL session as the canonical history and builds a smaller, inspectable, model-aware working context for each model call.
+
+```text
+complete Pi JSONL history
+          ↓
+DS4 planning, retrieval, summaries and policy
+          ↓
+bounded active context with provenance
+          ↓
+Pi provider
+```
+
+> **Project status:** M0–M13 are implemented. The Pi adapter and standalone `ds4-context-core` package are version `0.1.0`; the adapter targets Pi `0.84.3`.
+
+## Why DS4
+
+Long coding sessions accumulate old decisions, repeated context, large tool outputs and project state faster than any model context window can hold them. DS4 separates durable history from the model's current working set.
+
+It provides:
+
+- deterministic token budgeting with soft and hard input limits;
+- preservation of the current request, recent turns and atomic tool call/result groups;
+- exact and FTS5 historical retrieval with source provenance;
+- trust-gated project indexing, Git-aware invalidation and bounded source snippets;
+- hierarchical, validated, non-destructive compaction summaries;
+- persistent pins and append-only durable memory stored canonically in Pi JSONL;
+- content-addressed storage and bounded references for large tool results;
+- privacy classifications, secret redaction and provider-specific allow rules;
+- model-specific calibration and adaptive context allocation;
+- optional verified continuation for eligible OpenAI Responses profiles;
+- an inspectable Context Manifest explaining included and excluded material;
+- fail-open recovery to Pi's native context path for operational failures.
+
+## Architectural guarantees
+
+1. **Pi JSONL remains canonical.** DS4 never replaces Pi's session format.
+2. **SQLite is disposable.** It contains derived projections and can be rebuilt from canonical sources.
+3. **Compaction is non-destructive.** Raw history is not deleted or rewritten.
+4. **Provenance is preserved.** Retrieved and summarized context identifies its source.
+5. **Tool groups remain atomic.** Tool calls are not separated from their results.
+6. **Provider state is optional.** Continuation handles and cache state are never canonical.
+7. **Operational failures fail open.** Pi can continue with its native context behavior.
+8. **Enabled privacy enforcement fails closed.** Restricted content is replaced or the provider payload is rejected instead of being leaked.
+
+## Requirements
+
+- [Pi](https://github.com/earendil-works/pi) `0.84.3`
+- Node.js `22.19.0` or newer
+- SQLite support provided by Node's built-in `node:sqlite`
+
+Pi is intentionally pinned until extension contract tests validate a newer release.
+
+## Installation
+
+Pi packages execute with the user's full permissions. Review and trust the source before installing this or any other extension.
+
+### From GitHub
+
+```bash
+pi install git:github.com/Alucard24/ds4-context-engine
+```
+
+To try it for one run without adding it to settings:
+
+```bash
+pi -e git:github.com/Alucard24/ds4-context-engine
+```
+
+### From npm
+
+Install the public npm package with:
+
+```bash
+pi install npm:ds4-context-engine
+```
+
+### Local checkout
+
+```bash
+git clone https://github.com/Alucard24/ds4-context-engine.git
+cd ds4-context-engine
+npm ci
+npm run check
+pi install -l .
+```
+
+For extension development without installing the package:
+
+```bash
+pi -e ./src/extension/index.ts
+```
+
+Restart Pi or run `/reload` after installing or changing the extension.
+
+## Quick start
+
+DS4 starts in managed mode with conservative defaults. No configuration file is required.
+
+After loading the extension, inspect its state:
+
+```text
+/context status
+/context tokens
+/context health
+```
+
+Global configuration is loaded from:
+
+```text
+~/.pi/agent/ds4-context.json
+```
+
+A trusted project can override it with:
+
+```text
+.pi/ds4-context.json
+```
+
+Use observer mode as a pass-through rollback while retaining diagnostics:
+
+```json
+{
+  "context": {
+    "mode": "observer"
+  }
+}
+```
+
+Disable the extension's behavior without uninstalling it:
+
+```json
+{
+  "enabled": false
+}
+```
+
+Project configuration and project source indexing are disabled when Pi reports the project as untrusted.
+
+## Commands
+
+### Inspection
+
+| Command | Purpose |
+| --- | --- |
+| `/context` or `/context status` | Runtime, session, planner and subsystem status |
+| `/context tokens` | Token budget and active-context composition |
+| `/context manifest` | Latest Context Manifest |
+| `/context explain` | Human-readable planning explanation |
+| `/context included` | Items selected for the latest model call |
+| `/context excluded` | Items excluded from the latest model call |
+| `/context summaries` | Hierarchical summary graph diagnostics |
+| `/context retrieved` | Historical retrieval diagnostics |
+| `/context project` | Project index and retrieval status |
+| `/context privacy` | Classification and provider-policy status |
+| `/context model` | Active model profile and calibration |
+| `/context continuation` | Native continuation decisions and counters |
+| `/context artifacts` | Artifact storage and integrity status |
+| `/context compaction` | Last compaction status |
+| `/context compact-preview` | Preview compaction diagnostics |
+| `/context health` | SQLite and subsystem health checks |
+| `/context rebuild-index` | Rebuild derived state from canonical sources |
+
+### Pins and durable memory
+
+```text
+/context pins
+/context pin [--scope session|branch|project] [--classification LEVEL] <content>
+/context unpin PIN_ID [reason]
+
+/context memory
+/context memory list
+/context memory add [--scope session|project] [--key KEY] [--classification LEVEL] <claim>
+/context memory supersede MEMORY_ID [--source ID,ID] <new claim>
+/context memory invalidate MEMORY_ID [reason]
+/context memory expire MEMORY_ID [reason]
+```
+
+Valid privacy classifications are `normal`, `internal`, `sensitive` and `local-only`.
+
+## Configuration reference
+
+The following example shows the main configuration groups. Omitted values use the defaults in [`packages/core/src/config/config.ts`](packages/core/src/config/config.ts).
+
+```json
+{
+  "enabled": true,
+  "context": {
+    "mode": "managed",
+    "targetFillRatio": 0.7,
+    "softLimitRatio": 0.8,
+    "hardLimitRatio": 0.9,
+    "minimumOutputReserve": 8192,
+    "preferredOutputReserve": 32768,
+    "recentTailTokens": 64000,
+    "maxPinnedTokens": 16000,
+    "maxMemoryTokens": 8000,
+    "maxRetrievedHistoryTokens": 16000,
+    "maxProjectTokens": 20000,
+    "maxSummaryTokens": 12000
+  },
+  "retrieval": {
+    "exact": true,
+    "fts": true,
+    "semantic": false,
+    "maxResults": 12
+  },
+  "project": {
+    "enabled": true,
+    "maxFiles": 10000,
+    "maxFileBytes": 512000,
+    "maxTotalBytes": 50000000,
+    "snippetLines": 80,
+    "snippetOverlapLines": 12,
+    "maxResults": 8
+  },
+  "memory": {
+    "enabled": true,
+    "maxPinChars": 4000,
+    "maxClaimChars": 2000,
+    "maxResults": 12
+  },
+  "artifacts": {
+    "enabled": true,
+    "maxInlineToolResultChars": 12000,
+    "maxArtifactBytes": 100000000,
+    "maxSearchBytes": 50000000,
+    "excerptChars": 6000,
+    "maxSearchMatches": 12,
+    "storeLargeOutputs": true
+  },
+  "compaction": {
+    "enabled": true,
+    "mode": "hierarchical",
+    "validate": true,
+    "segmentTargetTokens": 30000,
+    "preserveRecentVerbatim": true
+  },
+  "privacy": {
+    "enabled": false,
+    "defaultClassification": "normal",
+    "localProviders": ["faux", "ollama", "llama-cpp", "lmstudio"],
+    "remoteDefaultAllowed": ["normal", "internal"],
+    "remoteProviders": {
+      "openrouter": ["normal"]
+    },
+    "redactSecrets": true
+  },
+  "modelAwareness": {
+    "enabled": true,
+    "calibrationWindow": 24,
+    "minimumCalibrationSamples": 3,
+    "calibrationRatioLowerBound": 0.5,
+    "calibrationRatioUpperBound": 2.0,
+    "overrides": {
+      "openrouter/vendor/model": {
+        "contextWindow": 200000,
+        "maxRetrievedHistoryTokens": 12000
+      }
+    }
+  },
+  "nativeContinuation": {
+    "enabled": false,
+    "allowProviderStorage": false,
+    "profiles": ["openai/*"],
+    "maxStateAgeMs": 1800000,
+    "retryManagedReplay": true
+  },
+  "diagnostics": {
+    "storeContextManifest": true,
+    "storeFullRenderedContext": false,
+    "logLevel": "info"
+  },
+  "storage": {
+    "databasePath": "ds4-context/context.db"
+  }
+}
+```
+
+Invalid or unknown values are ignored with a warning. Model overrides merge deterministically from `*` to `provider/*` to an exact `provider/model` profile.
+
+## Privacy and provider storage
+
+Privacy enforcement is disabled by default and must be configured for the providers you use. Unknown providers are treated as remote unless explicitly listed as local. `local-only` content is never permitted by a remote allow rule.
+
+Native continuation is also disabled by default. Enabling it requires both explicit storage consent and an exact or provider-scoped profile:
+
+```json
+{
+  "nativeContinuation": {
+    "enabled": true,
+    "allowProviderStorage": true,
+    "profiles": ["openai/*"]
+  }
+}
+```
+
+Eligible OpenAI Responses requests then set `store: true`. Review the provider's retention policy before enabling this option. DS4 keeps response handles only in volatile memory, verifies exact managed prefixes before reuse and retries once with a full managed replay when recognized continuation state is stale.
+
+See [`docs/PRIVACY.md`](docs/PRIVACY.md) and [`docs/NATIVE_CONTINUATION.md`](docs/NATIVE_CONTINUATION.md).
+
+## Storage and recovery
+
+By default, derived state is stored below Pi's agent directory:
+
+```text
+~/.pi/agent/ds4-context/
+├── context.db
+└── artifacts/
+```
+
+The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
+
+To validate or rebuild derived state:
+
+```text
+/context health
+/context rebuild-index
+```
+
+Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated.
+
+## Development
+
+```bash
+npm ci
+npm run build:core
+npm run typecheck
+npm test
+npm run check
+npm run pack:check
+npm pack --dry-run
+npm pack --dry-run --workspace ds4-context-core
+```
+
+The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, the portable-core dependency boundary and Pi extension lifecycle behavior. The package check also builds both tarballs, installs them in a clean temporary consumer and starts the packaged extension with isolated Pi RPC state.
+
+### Portable core
+
+`ds4-context-core` is a compiled ESM package with no Pi dependency. It owns runtime-neutral policy, storage and projections; agent adapters translate native sessions and lifecycle hooks at the boundary. The root `ds4-context-engine` package is the Pi adapter and depends one-way on the core workspace.
+
+### Repository layout
+
+```text
+packages/core/src   portable policy, planning, compaction, retrieval and storage
+src/pi-adapter      Pi JSONL projection, summary completion and provider integration
+src/extension       Pi hooks, commands and fail-open orchestration
+tests               core contract, unit, integration, golden and benchmark coverage
+scripts             package and release-readiness checks
+.github/workflows   continuous integration
+```
+
+## Documentation
+
+- [Architecture](docs/ARCHITECTURE.md)
+- [Context planner](docs/CONTEXT_PLANNER.md)
+- [Context Manifest](docs/CONTEXT_MANIFEST.md)
+- [Compaction](docs/COMPACTION.md)
+- [Summary graph](docs/SUMMARY_GRAPH.md)
+- [Historical retrieval](docs/RETRIEVAL.md)
+- [Project knowledge](docs/PROJECT_KNOWLEDGE.md)
+- [Artifacts](docs/ARTIFACTS.md)
+- [Memory and pins](docs/MEMORY_AND_PINS.md)
+- [Privacy](docs/PRIVACY.md)
+- [Model awareness](docs/MODEL_AWARENESS.md)
+- [Native continuation](docs/NATIVE_CONTINUATION.md)
+- [Portable core](docs/PORTABLE_CORE.md)
+- [Storage](docs/STORAGE.md)
+- [Release process](docs/RELEASING.md)
+- [Architecture decisions](docs/ADR/README.md)
+- [Original development plan](DS4_Context_Engine_Extension_Piano_Sviluppo.md)
+
+## Roadmap
+
+The original M0–M13 roadmap is complete. `ds4-context-core` now contains the compiled Pi-independent implementation, while runtime-specific behavior remains in the Pi adapter.
+
+Possible later work includes additional agent-runtime adapters, semantic retrieval, richer symbol indexing, cross-session project memory, context quality metrics, learned ranking and local KV integration. These are not required by the current MVP.
+
+## Contributing
+
+Issues and focused pull requests are welcome. Before submitting a change:
+
+1. preserve Pi JSONL as canonical history;
+2. keep SQLite and artifacts rebuildable;
+3. preserve provenance and atomic tool groups;
+4. retain strict compaction validation and safe fallback behavior;
+5. add or update tests;
+6. run `npm run check`, `npm run pack:check` and `git diff --check`.
+
+Please include reproduction steps for bugs and avoid attaching real session files, credentials or private provider payloads.
+
+## License
+
+[MIT](LICENSE)

package/docs/ADR/README.md

@@ -0,0 +1,60 @@
+# Architecture Decision Records
+
+The initial decisions from the development plan are accepted:
+
+| ADR | Decision | Status |
+|---|---|---|
+| 001 | Pi remains the agent runtime | Accepted |
+| 002 | Pi JSONL is the canonical source; SQLite is rebuildable | Accepted |
+| 003 | Compaction creates derived artifacts and never destroys raw history | Accepted |
+| 004 | The Pi `context` hook is the primary integration point | Accepted |
+| 005 | Lexical and exact retrieval precede semantic retrieval | Accepted |
+| 006 | Provider continuation/cache state is non-canonical | Accepted |
+| 007 | Core policy is published separately from the Pi adapter | Accepted |
+| 008 | Fail open to Pi's native behavior | Accepted |
+| 009 | Use built-in `node:sqlite` behind a storage adapter for M0 | Accepted |
+| 010 | Scope Pi's short entry IDs by session in the derived database | Accepted |
+| 011 | Validate append-only checkpoints by byte offset and physical-line hash | Accepted |
+| 012 | Persist Context Manifests without prompt or message content | Accepted |
+| 013 | Correlate each assistant response with the most recent pending manifest | Accepted |
+| 014 | Use deterministic whole-turn fitting and fail open on unsafe plans | Accepted |
+| 015 | Treat `ds4:pin` labels on active entries as mandatory atomic groups | Accepted |
+| 016 | Preserve Pi's compaction cut point and replace only summary generation | Accepted |
+| 017 | Reject invalid custom summaries and fall back to Pi's generator | Accepted |
+| 018 | Store summary provenance in canonical Pi details and derived SQLite rows | Accepted |
+| 019 | Keep summary nodes immutable and represent replacement only with parent-child edges | Accepted |
+| 020 | Resolve the aggregation predecessor from Pi's active branch, never session-global recency | Accepted |
+| 021 | Embed newly created non-active nodes in Pi details for complete graph reconstruction | Accepted |
+| 022 | Correlate compaction commits by pending summary ID, not duplicate summary text | Accepted |
+| 023 | Rank literal identifiers and phrases ahead of FTS; semantic retrieval stays optional | Accepted |
+| 024 | Inject automatic historical evidence only from Pi's active branch | Accepted |
+| 025 | Treat retrieved text as JSON-quoted data and discard it on planner fallback | Accepted |
+| 026 | Never enumerate or query project knowledge without Pi project trust | Accepted |
+| 027 | Validate live file hashes before injection and retain prior snippets only as stale derived rows | Accepted |
+| 028 | Treat project source as JSON-quoted untrusted data and discard it on planner fallback | Accepted |
+| 029 | Record Git revision metadata but keep live files, not Git or SQLite, canonical | Accepted |
+| 030 | Keep full tool results canonical in Pi JSONL; condense only provider-facing copies | Accepted |
+| 031 | Address artifact objects by verified SHA-256 and separate deduplicated bytes from source references | Accepted |
+| 032 | Permit artifact search only by explicit current-branch reference with bounded redacted literal excerpts | Accepted |
+| 033 | Disable artifact offload for observer and ephemeral sessions | Accepted |
+| 034 | Persist memory and pin mutations as versioned Pi custom entries; keep SQLite materialized | Accepted |
+| 035 | Require manual creation and explicit supersession; never silently overwrite durable claims | Accepted |
+| 036 | Make branch pins conditional on their creation leaf appearing in Pi's active branch | Accepted |
+| 037 | Treat pins as user-confirmed mandatory context and memory as quoted historical data | Accepted |
+| 038 | Share project-scoped state only for the same trusted canonical project path | Accepted |
+| 039 | Disable durable memory and pin mutation for observer and ephemeral sessions | Accepted |
+| 040 | Treat every provider as remote unless its exact ID is explicitly configured local | Accepted |
+| 041 | Reject `local-only` in all remote allow rules and prevent marker-based downgrades | Accepted |
+| 042 | Sanitize native context before planning and omit prohibited synthetic source groups whole | Accepted |
+| 043 | Recheck provider-specific serialization in `before_provider_request` without logging payloads | Accepted |
+| 044 | Fail closed with placeholders/empty payload when privacy enforcement fails | Accepted |
+| 045 | Persist pin/memory/artifact/summary classifications through their canonical or rebuildable metadata | Accepted |
+| 046 | Load DS4 last when later extensions could replace provider payloads | Accepted |
+| 047 | Keep native continuation disabled until provider storage is explicitly acknowledged | Accepted |
+| 048 | Wrap only explicitly profiled OpenAI Responses providers and delegate serialization/transport to Pi | Accepted |
+| 049 | Send a continuation delta only after exact request-option and request-plus-response prefix hashes match | Accepted |
+| 050 | Retry rejected stale continuation state once with the complete managed replay before exposing output | Accepted |
+| 051 | Keep continuation handles volatile and exclude them from manifests, logs, and DS4 persistence | Accepted |
+| 052 | Compile `ds4-context-core` as ESM and keep the Pi adapter dependency one-way | Accepted |
+
+Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.