ds4-context-engine 0.4.3 → 0.4.4

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch `0.4.2` adds class-only diagnostics for rejected exact-value spans: the existing `custom_fallback` warning carries `unsupportedSpanClasses` as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. Patch `0.4.3` makes those diagnostics interpretable: every distinct span receives the cheap transformed-form lookups before the bounded near-miss analysis starts, each near-miss candidate may spend only an equal share of what remains, and spans that were not analysed report `not-classified-length`, `not-classified-partial` or `not-classified-budget` instead of `no-near-miss`. See the [0.4.3 release record](docs/releases/0.4.3.md), the [0.4.2 release record](docs/releases/0.4.2.md), the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
+> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch `0.4.2` adds class-only diagnostics for rejected exact-value spans: the existing `custom_fallback` warning carries `unsupportedSpanClasses` as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. Patch `0.4.3` makes those diagnostics interpretable: every distinct span receives the cheap transformed-form lookups before the bounded near-miss analysis starts, each near-miss candidate may spend only an equal share of what remains, and spans that were not analysed report `not-classified-length`, `not-classified-partial` or `not-classified-budget` instead of `no-near-miss`. Patch `0.4.4` accepts a backticked exact value whose canonical JSON-escaped rendering is literally present in the evidence — the decoded rendering a summarizer produces when it quotes serialized JSON — while absent values, the reverse escape direction, span composition and single-character variants stay rejected; the diagnostics probe budget rises from 4000 to 24000 evidence-source scans. See the [0.4.4 release record](docs/releases/0.4.4.md), the [0.4.3 release record](docs/releases/0.4.3.md), the [0.4.2 release record](docs/releases/0.4.2.md), the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 065](docs/ADR/065-exact-value-escaped-equivalence.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
 
 **Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 

package/docs/ADR/065-exact-value-escaped-equivalence.md

@@ -0,0 +1,56 @@
+# 065 — Accept canonical JSON-escaped exact values in compaction summary validation
+
+**Date:** 2026-09-27
+**Status:** Accepted
+**Related:** [061](061-compaction-latency.md)
+
+## Context
+
+Every compaction summary is validated against the segment evidence: a backticked
+exact value is accepted only when it occurs literally in the sanitized segment
+source, in the ordered child-summary content, or in Pi's file-operation
+inventory. Production sessions on separate hosts failed closed with
+`compaction.custom_fallback`, `unsupported-exact-value`,
+`repair=too-many-bullets` and observed counts of 30/20, 23/19, 19/15 and 17/11
+unsupported spans and affected bullets, so each run fell back to Pi default
+compaction unmodified.
+
+The class-only report added in 0.4.2, made interpretable in 0.4.3, attributed
+the next observed run (16 spans across 12 bullets): `escaped-form-present` 11,
+`not-classified-partial` 4, `not-classified-length` 1, and no `no-near-miss`
+entry. Shapes (`quote` 11, `colon` 11, `json-punctuation` 9, no `backslash`) and
+the absence of escapes inside the rejected spans show decoded values, not
+identifiers: the summarizer quoted the readable value while the evidence holds
+its JSON-encoded rendering. The prompt rule added in 0.4.1 could not address
+that, and it should not have to: decoding JSON escapes while quoting a value is
+a reasonable rendering choice, not a fabrication.
+
+## Decision
+
+`unsupportedExactMatches` accepts a backticked value when it occurs literally in
+the evidence, either as written or in its canonical JSON-escaped rendering
+(`jsonEscaped`, the same transform the diagnostics classifier uses for
+`escaped-form-present`). The rule is unconditional and adds no configuration
+key. The reverse direction is deliberately not accepted: an escaped span whose
+raw form is present stays unsupported and is still reported as
+`unescaped-form-present`.
+
+Canonical JSON escaping is injective and reversible, so a decoded value can only
+pass when its encoded text is literally present in the evidence. The decision
+widens the accepted rendering, never the evidence: invented values, spans
+assembled from separately present parts, single-character variants, the
+eight-bullet and 25% repair bounds and the fail-closed fallback are unchanged.
+
+## Consequences
+
+- A span whose evidence is stored JSON-encoded now validates, and its bullet
+  survives instead of being pruned or forcing the bounded repair.
+- `escaped-form-present` becomes unreachable in practice: validation and the
+  classifier share the predicate, so such spans no longer reach the classifier.
+  The class stays in the taxonomy for compatibility, and new reports are not
+  expected to contain it.
+- The classifier's shared probe budget rises from 4000 to 24000 evidence-source
+  scans, so a report after this change can finish the near-miss analysis for the
+  spans that remain unsupported.
+- No SQLite migration, schema change, canonical JSONL change, privacy consent
+  change, provider-facing default change or new configuration key.

package/docs/ADR/README.md

@@ -68,5 +68,6 @@ The initial decisions from the development plan are accepted:
 | [062](062-cache-aware-context-planning.md) | Opt-in cache-aware tail planning using model pricing and observed cache shares | Accepted |
 | [063](063-fts-rowid-key-mappings.md) | Resolve FTS key deletes through rowid mapping tables | Accepted |
 | [064](064-per-project-databases-with-shared-calibration.md) | Split project state into per-project databases and keep token calibration shared | Accepted |
+| [065](065-exact-value-escaped-equivalence.md) | Accept canonical JSON-escaped exact values in compaction summary validation | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/COMPACTION.md

@@ -32,13 +32,13 @@ Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, an
 ## Critical Exact Values
 ```
 
-Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim, and to keep every backticked span one contiguous excerpt rather than joining separately supported values (a setting name with its value, a path with a line range, a command with its flags). Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. A direct update is validated against both the sanitized previous summary and new source, plus cumulative sanitized file evidence; previous-summary-only exact values remain valid evidence. Custom focus instructions are not factual evidence. Any unrepaired failure prevents the whole graph batch from being installed.
+Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur literally in the serialized segment source, ordered child-summary content, or those known file-operation paths, either as written or in the canonical JSON-escaped rendering of the value. A decoded value whose encoded text is present in the evidence is accepted ([ADR 065](ADR/065-exact-value-escaped-equivalence.md)); the reverse direction — an escaped span whose raw form is present — remains unsupported and is reported as `unescaped-form-present`. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim, and to keep every backticked span one contiguous excerpt rather than joining separately supported values (a setting name with its value, a path with a line range, a command with its flags). Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. A direct update is validated against both the sanitized previous summary and new source, plus cumulative sanitized file evidence; previous-summary-only exact values remain valid evidence. Custom focus instructions are not factual evidence. Any unrepaired failure prevents the whole graph batch from being installed.
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
 The same failure carries a class-only span report on the fallback warning (`unsupportedSpanClasses`): rejected-span count, affected bullets, length buckets, character shapes (spaces, backslashes, escape sequences, separators, quotes, typographic characters, JSON punctuation), and how each distinct span relates to the evidence — escaped or unescaped rendering, collapsed whitespace, case or typographic variant, one-character deletion, two separately present values joined into one span, or no near-miss (every applicable lookup ran and found nothing). The report contains class names and counters only, never span text, and the classifier is observation-only: it cannot change validation, repair bounds, or the fail-closed decision. It exists to separate a summarizer that invents values from one whose rendering or composition rules differ from the validation domain, which require opposite fixes.
 
-Three relations mark spans that were *not* analysed, and they must never be read as invention. `not-classified-length` means the span exceeds the near-miss length limit (96 characters); `not-classified-partial` means its share of the budget ran out; `not-classified-budget` means the shared budget was already exhausted. To keep the histogram interpretable, transformed-form lookups cover every distinct span before any near-miss analysis starts, and each near-miss candidate may spend only an equal share of what remains (at least 32 lookups, never more than the budget left). The report also carries `spansClassifiedCheap` (span occurrences attributed by a transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`, so a run can be read without guessing how much of the budget was consumed. `classificationComplete` is false when the shared budget or a per-span share stopped an analysis; a `not-classified-length` span does not clear it, because that limit is deterministic rather than a resource shortfall.
+Three relations mark spans that were *not* analysed, and they must never be read as invention. `not-classified-length` means the span exceeds the near-miss length limit (96 characters); `not-classified-partial` means its share of the budget ran out; `not-classified-budget` means the shared budget was already exhausted. To keep the histogram interpretable, transformed-form lookups cover every distinct span before any near-miss analysis starts, and each near-miss candidate may spend only an equal share of what remains (at least 32 lookups, never more than the budget left). The report also carries `spansClassifiedCheap` (span occurrences attributed by a transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`, so a run can be read without guessing how much of the budget was consumed. The shared budget defaults to 24000 evidence-source scans, where one lookup scans every corpus source once. `escaped-form-present` is not expected in reports from 0.4.4 on: validation accepts the canonical escaped rendering and the classifier consumes the same predicate, so such spans no longer reach it; the class stays in the taxonomy for compatibility. `classificationComplete` is false when the shared budget or a per-span share stopped an analysis; a `not-classified-length` span does not clear it, because that limit is deterministic rather than a resource shortfall.
 
 ## Provenance and recovery
 

package/docs/releases/0.4.3.md

@@ -152,8 +152,19 @@ Limitations that remain:
   requires a natural failure and is expected to appear on the next
   `compaction.custom_fallback` warning.
 
-## Publication
-
-Published manually in dependency order (core, reference adapter, engine) with
-`npm publish`, after `npm run pack:check` on the same commit. Registry
-verification is recorded in the follow-up documentation commit.
+## Publication and registry verification
+
+Published manually in dependency order from `928f1a4`: `ds4-context-core@0.4.3`,
+then `ds4-context-reference-adapter@0.4.3`, then `ds4-context-engine@0.4.3`, all
+with the default `latest` tag, after `npm run pack:check` on the same commit.
+
+A first `npm run registry:check -- 0.4.3` failed with `ETARGET` because the local
+npm cache still served the previous packument; `npm view --prefer-online`
+already reported 0.4.3 with `latest` resolving to it for all three packages, and
+the check passed when re-run against an empty cache directory: "Verified all DS4
+registry packages at exact version 0.4.3".
+
+Annotated tag `v0.4.3` and the GitHub Release were created from `928f1a4`. No
+provider calls are involved in this release procedure; the classifier runs only
+after a validation failure has already been decided, and no session data,
+credentials, or provider payloads were read or written while preparing it.

package/docs/releases/0.4.4.md

@@ -0,0 +1,87 @@
+# Release 0.4.4 — Accept canonical JSON-escaped exact values
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.4.
+**Implementation commit:** `ad6e4c8`.
+
+## Summary
+
+Behavior patch. Compaction summary validation now accepts a backticked exact
+value that occurs literally in the evidence in its canonical JSON-escaped
+rendering, not only as written. The accepted rendering widens; the evidence does
+not: canonical JSON escaping is injective, so a decoded value passes only when
+its encoded text is literally present. The diagnostics probe budget rises from
+4000 to 24000 evidence-source scans. The eight-bullet and 25% repair bounds, the
+fail-closed fallback to Pi default compaction, the summarizer prompt, the SQLite
+schema and every provider-facing default are unchanged.
+
+## Changes
+
+- `packages/core/src/compaction/summary-contract.ts`:
+  - `unsupportedExactMatches` also accepts a span whose `jsonEscaped` rendering
+    occurs literally in the segment source, the ordered child-summary content or
+    the file-operation inventory. `jsonEscaped` is the transform the diagnostics
+    classifier already used for `escaped-form-present`, so the accepted domain
+    and the reported class agree.
+  - `UNSUPPORTED_SPAN_PROBE_BUDGET` rises from 4000 to 24000 source scans.
+- `tests/unit/summary-contract.test.ts`: acceptance for a decoded value whose
+  escaped rendering is present; continued rejection when the escaped rendering
+  is absent and for an escaped span whose raw form is present; the default budget
+  is 24000 and an explicit 4000 budget still saturates without overspending.
+- `tests/integration/compaction.test.ts`: a segment whose evidence carries
+  literal JSON escapes and a summary that quotes the decoded value commits with
+  `validationStatus: "valid"` instead of falling back.
+- `docs/COMPACTION.md`: the summary contract and the diagnostics section.
+- `docs/ADR/065-exact-value-escaped-equivalence.md` records the decision.
+
+## Why
+
+Production sessions failed closed on `unsupported-exact-value` /
+`repair=too-many-bullets` with 30/20, 23/19, 19/15, 17/11 and 16/12 unsupported
+spans and affected bullets. The report added in 0.4.2 and made interpretable in
+0.4.3 attributed the last observed run: 11 of 16 spans were
+`escaped-form-present`, no span was established as absent (`no-near-miss` was
+zero), and the shape histogram (`quote` 11, `colon` 11, `json-punctuation` 9, no
+`backslash`) showed decoded values rather than identifiers. The values were
+present in the evidence; only the rendering differed. The remaining candidates
+for the five unanalysed spans were model-dependent and unverifiable offline,
+while escape equivalence is deterministic and testable: the new integration test
+fails on 0.4.3 and passes on 0.4.4.
+
+## Privacy and safety
+
+- Acceptance still requires a literal occurrence in the evidence, and canonical
+  JSON escaping is injective, so no invented value can pass; the validation path
+  reads the same sanitized sources as before.
+- Diagnostics stay class-only: the budget change alters how many lookups the
+  classifier may spend, never what it emits. The probe runs only after a
+  validation failure has already been decided.
+- No provider call, network access or configuration change is involved in this
+  release.
+
+## Compatibility and persistence
+
+- No migrations, no schema change (still 16), no new configuration key, and no
+  change to any provider-facing default.
+- The summarizer prompt is unchanged; only the accepted rendering widens.
+- `escaped-form-present` stays in the report taxonomy but is no longer expected
+  in reports, because validation accepts the same canonical rendering that the
+  classifier probes.
+
+## Measured scope and limitations
+
+The effect on a real session is not verified by this release. If escape
+equivalence was the binding cause, the next production compaction should not
+produce `compaction.custom_fallback`; if one appears, its report now covers only
+spans that are not canonical-escape-equivalent, with a budget large enough to
+finish the near-miss analysis for a handful of candidates. The 25% removal bound
+and the fail-closed fallback remain in force, so an unsupported set that exceeds
+the repair bounds still falls back to Pi default compaction.
+
+## Validation
+
+On Node 26.5.1 the release passed `npm run check` (99 Vitest files, 632 tests,
+TypeScript builds and root typecheck), deterministic `npm run quality:compare`
+(candidate `0.9875` against baseline `0.808156`), `npm run schema:context-persistence`
+(`passed: true`), and `npm run pack:check` in a clean consumer for all three
+packages at 0.4.4 (247 core files, 7 reference-adapter files, 103 extension
+files). No provider calls are involved in this release procedure.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.4.3",
+  "version": "0.4.4",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -63,7 +63,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.4.3",
+    "ds4-context-core": "0.4.4",
     "js-tiktoken": "1.0.21"
   },
   "peerDependencies": {

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.4.3";
+export const EXTENSION_VERSION = "0.4.4";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";