ds4-context-engine 0.4.1 → 0.4.2

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. See the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
+> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch `0.4.2` adds class-only diagnostics for rejected exact-value spans: the existing `custom_fallback` warning carries `unsupportedSpanClasses` as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. See the [0.4.2 release record](docs/releases/0.4.2.md), the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
 
 **Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
@@ -515,6 +515,7 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.4.2 release notes](docs/releases/0.4.2.md)
 - [0.4.1 release notes](docs/releases/0.4.1.md)
 - [0.4.0 release notes](docs/releases/0.4.0.md)
 - [0.3.10 release notes](docs/releases/0.3.10.md)
@@ -545,7 +546,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt without changing deterministic validation or the repair bounds. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt without changing deterministic validation or the repair bounds. Version 0.4.2 adds class-only diagnostics to the same fail-closed path: the `custom_fallback` warning reports `unsupportedSpanClasses` as class names and counters only, never span text, without changing validation strictness, repair bounds or defaults. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/COMPACTION.md

@@ -36,6 +36,8 @@ Every section must occur once, in order, and contain content or `- None`. DS4 re
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
+The same failure carries a class-only span report on the fallback warning (`unsupportedSpanClasses`): rejected-span count, affected bullets, length buckets, character shapes (spaces, backslashes, escape sequences, separators, quotes, typographic characters, JSON punctuation), and how each distinct span relates to the evidence — escaped or unescaped rendering, collapsed whitespace, case or typographic variant, one-character deletion, two separately present values joined into one span, no near-miss, or not classified because the bounded probe budget was exhausted. The report contains class names and counters only, never span text, and the classifier is observation-only: it cannot change validation, repair bounds, or the fail-closed decision. It exists to separate a summarizer that invents values from one whose rendering or composition rules differ from the validation domain, which require opposite fixes.
+
 ## Provenance and recovery
 
 `CompactionEntry.details` contains cumulative `readFiles` and `modifiedFiles` plus:
@@ -146,4 +148,4 @@ It requests compaction at most once per session leaf. Pi's native threshold and
 /context summaries
 ```
 
-The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the effective provider/model, chosen path (`direct-update` or `hierarchical`), budget mode, calibrated input budget, estimated new-source and full-update prompt sizes, logical summary calls (excluding retries), generated segment count, aggregate-call count, concurrency cap, and completed transport-retry count without source content. Monotonic wall timings cover preparation, generation (direct or parallel segments, including validation and retries), aggregation, graph preparation/persistence, and total DS4 hook time. Parallel generation time is elapsed wall time, not a sum of overlapping requests. Total time excludes Pi's subsequent canonical append or native fallback. Timings survive the in-process commit/fallback notification, but are not persisted in JSONL or reconstructed as fake durations after restart. Metadata-only `compaction.timings` is emitted at `debug` for successful and failed attempts. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.
+The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the effective provider/model, chosen path (`direct-update` or `hierarchical`), budget mode, calibrated input budget, estimated new-source and full-update prompt sizes, logical summary calls (excluding retries), generated segment count, aggregate-call count, concurrency cap, and completed transport-retry count without source content. Monotonic wall timings cover preparation, generation (direct or parallel segments, including validation and retries), aggregation, graph preparation/persistence, and total DS4 hook time. Parallel generation time is elapsed wall time, not a sum of overlapping requests. Total time excludes Pi's subsequent canonical append or native fallback. Timings survive the in-process commit/fallback notification, but are not persisted in JSONL or reconstructed as fake durations after restart. Metadata-only `compaction.timings` is emitted at `debug` for successful and failed attempts. Failed validation additionally embeds the class-only `unsupportedSpanClasses` report in the `compaction.custom_fallback` warning, so a failing compaction can be diagnosed from logs alone without reproducing the session or reading its content. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.

package/docs/releases/0.4.1.md

@@ -80,8 +80,14 @@ files, 615 tests, TypeScript builds and root typecheck), deterministic
 in a clean consumer for all three packages at 0.4.1 (247 core files, 7
 reference-adapter files, 99 extension files). Tarball review found no session,
 `.pi`, database, JSONL or credential files, and `git diff --check` was clean.
-The coordinated publication then proceeds in dependency order
+The coordinated publication then proceeded in dependency order
 (`ds4-context-core`, then `ds4-context-reference-adapter`, then
-`ds4-context-engine`) with `npm run registry:check -- 0.4.1` afterwards; the
-verified registry result is recorded in the follow-up documentation commit. No
-provider calls are involved in this release procedure.
+`ds4-context-engine`), each command reporting the exact 0.4.1 artifact.
+Registry propagation lagged the publishes: the first
+`npm run registry:check -- 0.4.1` failed with
+`ETARGET ... No matching version found for ds4-context-core@0.4.1`, and
+`ds4-context-engine` still resolved to 0.4.0 after core, reference adapter and
+core again were visible. Once propagation completed, the repeated
+`npm run registry:check -- 0.4.1` reported "Verified all DS4 registry packages
+at exact version 0.4.1", and the `latest` dist-tag resolved to 0.4.1 for all
+three packages. No provider calls are involved in this release procedure.

package/docs/releases/0.4.2.md

@@ -0,0 +1,113 @@
+# Release 0.4.2 — Class-only diagnostics for rejected exact-value spans
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.2.
+**Implementation commit:** `6ecd2af`.
+
+## Summary
+
+Diagnostics patch. Fail-closed exact-value validation now produces a class-only
+report of the rejected backticked spans, and the existing
+`compaction.custom_fallback` warning carries it as `unsupportedSpanClasses`.
+Validation strictness, the eight-bullet and 25% repair bounds, the fail-closed
+fallback to Pi default compaction, and every provider-facing default are
+unchanged. The report contains class names and counters only: it never contains
+span text, so it is safe to log and to quote in reports.
+
+## Changes
+
+- `packages/core/src/compaction/summary-contract.ts` adds
+  `classifyUnsupportedExactValueSpans()`, a pure observation-only classifier
+  returning an `UnsupportedSpanClassReport`:
+  - `relations` counts near-miss relations between the rejected span and the
+    segment source: `escaped-form-present`, `unescaped-form-present`,
+    `whitespace-collapsed-present`, `case-variant-present`,
+    `typographic-variant-present`, `single-deletion-present`,
+    `composed-two-present-parts`, `no-near-miss`, `not-classified-budget`.
+  - `shapes` counts structural shapes: `space`, `backslash`, `double-backslash`,
+    `colon`, `equals`, `double-dash`, `slash`, `quote`, `typographic-char`,
+    `digit-group-separator`, `json-punctuation`.
+  - `lengthBuckets` counts span lengths: `len-1-8`, `len-9-16`, `len-17-32`,
+    `len-33-64`, `len-65-120`, `len-121-plus`.
+  - `spans`, optional `bullets`, and `classificationComplete` report the totals
+    and whether the evidence-probe budget stopped the near-miss classification.
+  - The classifier is bounded: probe budget 4000 evidence lookups, near-miss
+    spans limited to 96 characters, composed parts required to be at least four
+    characters. It cannot alter validation or repair.
+- `src/pi-adapter/summary-generator.ts`: `SummaryValidationError` carries the
+  report and machine-readable issue codes. Error message and behavior are
+  unchanged.
+- `src/pi-adapter/compaction-coordinator.ts`: the existing
+  `custom_fallback` warning gains `unsupportedSpanClasses` when a report exists.
+- Tests: core classifier coverage, generator wiring, coordinator logging, and a
+  guard asserting that logged diagnostics contain no span text.
+- `docs/COMPACTION.md`: diagnostics and validation sections describe the report
+  and its limits.
+
+## Why
+
+Production sessions on separate hosts failed closed with
+`compaction.custom_fallback`, `unsupported-exact-value`,
+`repair=too-many-bullets`, and observed counts of 30/20, 23/19, 19/15, and
+17/11 unsupported spans and affected bullets. The bounded repair removes at most
+eight whole bullets and refuses without attempting any removal above that, so
+every one of those runs fell back to Pi default compaction unmodified.
+
+Counts alone cannot attribute the failure: a summarizer that invents values and
+a summarizer whose rendering or composition rules differ from the validation
+domain need opposite fixes, and the two are indistinguishable in a bare
+`unsupportedSpans` number. 0.4.1 changed the prompt for the composition
+hypothesis; the following observed runs kept the same failure class and the same
+order of magnitude, so the next step is attribution rather than another prompt
+edit. This release adds only the measurement.
+
+## Privacy and safety
+
+- No conversation text is added to any log, warning detail, or error message.
+  The report is a fixed taxonomy of class names plus integer counters.
+- The classifier performs bounded lookups against the already-sanitized segment
+  source used by validation; it does not read session files, SQLite, or any
+  other data, and it does not run on successful summaries.
+- Privacy enforcement, provider-facing payloads, and `compaction` phase timings
+  are untouched.
+
+## Compatibility and persistence
+
+- No SQLite migration or schema change (still 16), no canonical Pi JSONL change,
+  no privacy consent change, no portable runtime-adapter contract change.
+- Validation strictness, the eight-bullet bound, the 25% removal bound, and the
+  fail-open behavior of the extension remain exactly as in 0.4.1.
+- Provider-facing defaults are unchanged: `chars-v1` remains the default
+  estimator, `modelAwareness.autoTune` and DS4 native continuation stay
+  disabled, `storage.scope` keeps its `project` default.
+
+## Measured scope and limitations
+
+The classifier is diagnostic: it does not fix the observed fallback and no
+reduction of the fallback rate is claimed. Its output is only as good as the
+rejected spans it is given, and `classificationComplete: false` marks the runs
+where the probe budget stopped the near-miss classification.
+
+## Related operational finding (not a change in this release)
+
+On the same host an independent failure mode was identified outside DS4: Pi's
+own summarization request caps output at
+`Math.min(Math.floor(0.8 * reserveTokens), model.maxTokens)` with
+`reserveTokens` defaulting to 16384, i.e. 13107 output tokens for the history
+summary and 8192 for the turn-prefix summary, and Pi passes the session thinking
+level to that request. Both the dedicated-model attempt and the fallback attempt
+reported the same stop condition
+(`Summarization failed: generation hit the token cap and the summary is
+incomplete`, raised for `stopReason === "length"`), so compaction ended in an
+error even after DS4 and the dedicated compaction model had both declined.
+Raising `compaction.reserveTokens` globally, or per model through
+`compaction.modelOverrides`, is an operator setting in Pi and is not part of
+this release.
+
+## Validation
+
+On Node 26.5.1 the release passed `npm run check` (builds, `tsc --noEmit`, 99
+Vitest files and 623 tests), deterministic `npm run quality:compare` (candidate
+`0.9875` against baseline `0.808156`), `npm run schema:context-persistence`
+(`passed: true`), and `npm run pack:check` in a clean consumer for all three
+packages at 0.4.2 (247 core files, 7 reference-adapter files, 100 extension
+files). No provider calls are involved in this release procedure.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.4.1",
+  "version": "0.4.2",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -63,7 +63,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.4.1",
+    "ds4-context-core": "0.4.2",
     "js-tiktoken": "1.0.21"
   },
   "peerDependencies": {

package/src/pi-adapter/compaction-coordinator.ts

@@ -43,6 +43,7 @@ import {
 import {
   generateValidatedSummary,
   sumUsage,
+  SummaryValidationError,
   type GeneratedSummary,
 } from "./summary-generator.ts";
 import {
@@ -501,6 +502,9 @@ export class CompactionCoordinator {
       };
     } catch (error) {
       const message = error instanceof Error ? error.message : String(error);
+      const spanClasses = error instanceof SummaryValidationError
+        ? error.spanClassReport
+        : undefined;
       this.state = {
         ...this.state,
         phase: "failed",
@@ -508,7 +512,11 @@ export class CompactionCoordinator {
         completedAt: this.dependencies.now(),
         lastError: message,
       };
-      this.dependencies.logger.warn("compaction.custom_fallback", { trigger, error: message });
+      this.dependencies.logger.warn("compaction.custom_fallback", {
+        trigger,
+        error: message,
+        ...(spanClasses ? { unsupportedSpanClasses: spanClasses } : {}),
+      });
       if (!event.signal.aborted && ctx.hasUI) {
         ctx.ui.notify(`DS4 compaction unavailable; using Pi default. ${message}`, "warning");
       }

package/src/pi-adapter/summary-generator.ts

@@ -7,16 +7,38 @@ import type {
 import type { CompactionThinkingLevel } from "ds4-context-core/config/config";
 import {
   analyzeUnsupportedExactValueBullets,
+  classifyUnsupportedExactValueSpans,
   groundSummaryFileSections,
   validateSummary,
   type SummaryValidationInput,
   type SummaryValidationResult,
+  type UnsupportedSpanClassReport,
 } from "ds4-context-core/compaction/summary-contract";
 
 export const DEFAULT_COMPACTION_TRANSPORT_MAX_ATTEMPTS = 4;
 export const DEFAULT_COMPACTION_TRANSPORT_BASE_DELAY_MS = 2000;
 export const COMPACTION_TRANSPORT_MAX_DELAY_MS = 60_000;
 
+/**
+ * Fail-closed summary rejection. Carries the class-only span diagnostics so the
+ * coordinator can log why spans were rejected without logging span text.
+ */
+export class SummaryValidationError extends Error {
+  readonly codes: readonly string[];
+  readonly spanClassReport?: UnsupportedSpanClassReport;
+
+  constructor(
+    message: string,
+    codes: readonly string[],
+    spanClassReport?: UnsupportedSpanClassReport,
+  ) {
+    super(message);
+    this.name = "SummaryValidationError";
+    this.codes = codes;
+    this.spanClassReport = spanClassReport;
+  }
+}
+
 /**
  * Transport retry policy for compaction summary requests: four total attempts
  * (the initial call plus three retries), 2000 ms base delay, exponential backoff, abort-aware.
@@ -330,11 +352,20 @@ export async function generateValidatedSummary(
     }
   }
   if (validation.status === "invalid") {
-    const codes = unique(validation.issues.map((issue) => issue.code)).join(", ");
+    const codes = unique(validation.issues.map((issue) => issue.code));
     const repairDiagnostics = exactRepair
       ? `; repair=${exactRepairFailure ?? exactRepair.status}; unsupportedSpans=${exactRepair.unsupportedSpans}; affectedBullets=${exactRepair.affectedBullets}`
       : "";
-    throw new Error(`Compaction ${input.stage} summary validation failed: ${codes}${repairDiagnostics}`);
+    const spanClassReport = validation.issues.some((issue) => issue.code === "unsupported-exact-value")
+      ? classifyUnsupportedExactValueSpans(content, validationInput, {
+          ...(exactRepair ? { affectedBullets: exactRepair.affectedBullets } : {}),
+        })
+      : undefined;
+    throw new SummaryValidationError(
+      `Compaction ${input.stage} summary validation failed: ${codes.join(", ")}${repairDiagnostics}`,
+      codes,
+      spanClassReport,
+    );
   }
   return { content, validation, usage: sumUsage([...retryUsages, response.usage]) };
 }

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.4.1";
+export const EXTENSION_VERSION = "0.4.2";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";