ds4-context-engine 0.4.0 → 0.4.1

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. See the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
+> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. See the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
 
 **Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
@@ -515,6 +515,7 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.4.1 release notes](docs/releases/0.4.1.md)
 - [0.4.0 release notes](docs/releases/0.4.0.md)
 - [0.3.10 release notes](docs/releases/0.3.10.md)
 - [0.3.9 release notes](docs/releases/0.3.9.md)
@@ -544,7 +545,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt without changing deterministic validation or the repair bounds. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/COMPACTION.md

@@ -32,7 +32,7 @@ Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, an
 ## Critical Exact Values
 ```
 
-Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. A direct update is validated against both the sanitized previous summary and new source, plus cumulative sanitized file evidence; previous-summary-only exact values remain valid evidence. Custom focus instructions are not factual evidence. Any unrepaired failure prevents the whole graph batch from being installed.
+Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim, and to keep every backticked span one contiguous excerpt rather than joining separately supported values (a setting name with its value, a path with a line range, a command with its flags). Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. A direct update is validated against both the sanitized previous summary and new source, plus cumulative sanitized file evidence; previous-summary-only exact values remain valid evidence. Custom focus instructions are not factual evidence. Any unrepaired failure prevents the whole graph batch from being installed.
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 

package/docs/releases/0.4.0.md

@@ -81,6 +81,11 @@ On Node 26.5.1, the coordinated release passed `npm run check` (99 Vitest files,
 615 tests, TypeScript builds and root typecheck), deterministic
 `npm run quality:compare`, the `npm run schema:context-persistence` size bound,
 and `npm run pack:check` in a clean consumer after synchronizing both exported
-runtime version constants. `git diff --check` was clean and dry-run tarball
-review contained 247 core files, 7 reference-adapter files and 98 extension
-files, none including untracked local state.
+runtime version constants. `git diff --check` was clean and tarball review
+contained 247 core files, 7 reference-adapter files and 99 extension files,
+none including untracked local state. All three packages were then published to
+npm at 0.4.0 in dependency order (`ds4-context-core`, then
+`ds4-context-reference-adapter`, then `ds4-context-engine`). After registry
+propagation, `npm run registry:check -- 0.4.0` verified the exact-version
+artifacts, exact adapter/core dependencies and the packaged extension in a
+fresh consumer.

package/docs/releases/0.4.1.md

@@ -0,0 +1,87 @@
+# Release 0.4.1 — Contiguous backticked spans in compaction summaries
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.1.
+**Implementation commit:** `18d5e19`.
+
+## Summary
+
+Patch release that adds one explicit rule to the compaction summarizer prompt:
+each backticked span must be one contiguous excerpt copied as-is from the
+segment source, and a span must never join separately supported values. The
+deterministic validation is unchanged. No configuration, schema, contract or
+provider-facing default changes.
+
+## Changes
+
+- `buildSummaryPrompt` states that a backticked span must be one contiguous
+  excerpt and must not assemble a value from parts that occur separately in the
+  evidence (a setting name plus its value, a path plus a line range, a command
+  plus its flags), and that the bullet must be omitted when the literal span is
+  unavailable. The segment, aggregate and update prompts all inherit the rule
+  because `buildAggregateSummaryPrompt` delegates to `buildSummaryPrompt`.
+- `tests/unit/summary-contract.test.ts` asserts the rule in both the segment and
+  the aggregate prompt.
+- `docs/COMPACTION.md` records the rule in the summary contract description.
+
+## Why
+
+Two production sessions on separate hosts failed closed to Pi default
+compaction with `compaction.custom_fallback`, `unsupported-exact-value`,
+`repair=too-many-bullets`, and 30/20 then 23/19 unsupported spans and affected
+bullets. The bounded repair removes at most eight whole bullets, so nineteen
+affected bullets are refused without attempting any removal.
+
+`compaction.maxConcurrentSegments: 1` was tested before this change: the same
+failure class and the same magnitude appeared with a single segment in flight,
+so request concurrency was not the cause. The remaining local mechanism is span
+composition. Validation accepts a backticked span only when it is a literal
+substring of the segment's own sanitized source
+(`unsupportedExactMatches` in `packages/core/src/compaction/summary-contract.ts`),
+and the value the model sees is part of that source by construction
+(`sourceText` in `src/pi-adapter/compaction-adapter.ts` includes the serialized
+conversation). Composite forms therefore cannot validate, and the previous
+wording forbade them only implicitly through the verbatim requirement.
+
+## Compatibility and persistence
+
+- Validation strictness is unchanged: the same literal-substring rule, the same
+  eight-bullet and 25% repair bounds, the same fail-closed fallback to Pi
+  default compaction. Only the instruction given to the summarizer changed.
+- No SQLite migration, no schema change (still 16), no canonical Pi JSONL
+  change, no privacy consent change, no portable runtime-adapter change.
+- Provider-facing defaults are unchanged: `chars-v1` remains the default
+  estimator, `modelAwareness.autoTune` and DS4 native continuation stay
+  disabled, `storage.scope` keeps its `project` default.
+
+## Measured scope and limitations
+
+The effect on the observed fallback is **not verified in a real session** by
+this release. The change is a prompt instruction, so the outcome depends on the
+summarizer model's compliance. Expected observations for the next real
+compaction, with `compaction.summary.thinking` deliberately left unchanged so
+that the prompt is the only variable:
+
+| Observation | Reading |
+| --- | --- |
+| `compaction.custom_fallback` absent | span composition was the cause |
+| lower counts, still above eight | partial effect; the model reduces but does not eliminate |
+| unchanged counts | composition hypothesis rejected; move to the summarizer model or reasoning level |
+
+Lowering `compaction.segmentTargetTokens` is not expected to resolve this
+counter on its own: the affected-bullet count scales with segment size while the
+eight-bullet bound is absolute, and every segment in a batch must validate.
+
+## Validation and publication
+
+On Node 26.5.1 the release passed a clean `npm ci`, `npm run check` (99 Vitest
+files, 615 tests, TypeScript builds and root typecheck), deterministic
+`npm run quality:compare` (candidate `0.9875` against baseline `0.808156`),
+`npm run schema:context-persistence` (`passed: true`), and `npm run pack:check`
+in a clean consumer for all three packages at 0.4.1 (247 core files, 7
+reference-adapter files, 99 extension files). Tarball review found no session,
+`.pi`, database, JSONL or credential files, and `git diff --check` was clean.
+The coordinated publication then proceeds in dependency order
+(`ds4-context-core`, then `ds4-context-reference-adapter`, then
+`ds4-context-engine`) with `npm run registry:check -- 0.4.1` afterwards; the
+verified registry result is recorded in the follow-up documentation commit. No
+provider calls are involved in this release procedure.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -63,7 +63,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.4.0",
+    "ds4-context-core": "0.4.1",
     "js-tiktoken": "1.0.21"
   },
   "peerDependencies": {

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.4.0";
+export const EXTENSION_VERSION = "0.4.1";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";