ds4-context-engine 0.4.1 → 0.4.3

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. See the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
+> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch `0.4.2` adds class-only diagnostics for rejected exact-value spans: the existing `custom_fallback` warning carries `unsupportedSpanClasses` as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. Patch `0.4.3` makes those diagnostics interpretable: every distinct span receives the cheap transformed-form lookups before the bounded near-miss analysis starts, each near-miss candidate may spend only an equal share of what remains, and spans that were not analysed report `not-classified-length`, `not-classified-partial` or `not-classified-budget` instead of `no-near-miss`. See the [0.4.3 release record](docs/releases/0.4.3.md), the [0.4.2 release record](docs/releases/0.4.2.md), the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
 
 **Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
@@ -515,6 +515,8 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.4.3 release notes](docs/releases/0.4.3.md)
+- [0.4.2 release notes](docs/releases/0.4.2.md)
 - [0.4.1 release notes](docs/releases/0.4.1.md)
 - [0.4.0 release notes](docs/releases/0.4.0.md)
 - [0.3.10 release notes](docs/releases/0.3.10.md)
@@ -545,7 +547,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt without changing deterministic validation or the repair bounds. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.4.1 states the contiguous-span rule for compaction summaries in the summarizer prompt without changing deterministic validation or the repair bounds. Version 0.4.2 adds class-only diagnostics to the same fail-closed path: the `custom_fallback` warning reports `unsupportedSpanClasses` as class names and counters only, never span text, without changing validation strictness, repair bounds or defaults. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/COMPACTION.md

@@ -36,6 +36,10 @@ Every section must occur once, in order, and contain content or `- None`. DS4 re
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
+The same failure carries a class-only span report on the fallback warning (`unsupportedSpanClasses`): rejected-span count, affected bullets, length buckets, character shapes (spaces, backslashes, escape sequences, separators, quotes, typographic characters, JSON punctuation), and how each distinct span relates to the evidence — escaped or unescaped rendering, collapsed whitespace, case or typographic variant, one-character deletion, two separately present values joined into one span, or no near-miss (every applicable lookup ran and found nothing). The report contains class names and counters only, never span text, and the classifier is observation-only: it cannot change validation, repair bounds, or the fail-closed decision. It exists to separate a summarizer that invents values from one whose rendering or composition rules differ from the validation domain, which require opposite fixes.
+
+Three relations mark spans that were *not* analysed, and they must never be read as invention. `not-classified-length` means the span exceeds the near-miss length limit (96 characters); `not-classified-partial` means its share of the budget ran out; `not-classified-budget` means the shared budget was already exhausted. To keep the histogram interpretable, transformed-form lookups cover every distinct span before any near-miss analysis starts, and each near-miss candidate may spend only an equal share of what remains (at least 32 lookups, never more than the budget left). The report also carries `spansClassifiedCheap` (span occurrences attributed by a transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`, so a run can be read without guessing how much of the budget was consumed. `classificationComplete` is false when the shared budget or a per-span share stopped an analysis; a `not-classified-length` span does not clear it, because that limit is deterministic rather than a resource shortfall.
+
 ## Provenance and recovery
 
 `CompactionEntry.details` contains cumulative `readFiles` and `modifiedFiles` plus:
@@ -146,4 +150,4 @@ It requests compaction at most once per session leaf. Pi's native threshold and
 /context summaries
 ```
 
-The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the effective provider/model, chosen path (`direct-update` or `hierarchical`), budget mode, calibrated input budget, estimated new-source and full-update prompt sizes, logical summary calls (excluding retries), generated segment count, aggregate-call count, concurrency cap, and completed transport-retry count without source content. Monotonic wall timings cover preparation, generation (direct or parallel segments, including validation and retries), aggregation, graph preparation/persistence, and total DS4 hook time. Parallel generation time is elapsed wall time, not a sum of overlapping requests. Total time excludes Pi's subsequent canonical append or native fallback. Timings survive the in-process commit/fallback notification, but are not persisted in JSONL or reconstructed as fake durations after restart. Metadata-only `compaction.timings` is emitted at `debug` for successful and failed attempts. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.
+The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the effective provider/model, chosen path (`direct-update` or `hierarchical`), budget mode, calibrated input budget, estimated new-source and full-update prompt sizes, logical summary calls (excluding retries), generated segment count, aggregate-call count, concurrency cap, and completed transport-retry count without source content. Monotonic wall timings cover preparation, generation (direct or parallel segments, including validation and retries), aggregation, graph preparation/persistence, and total DS4 hook time. Parallel generation time is elapsed wall time, not a sum of overlapping requests. Total time excludes Pi's subsequent canonical append or native fallback. Timings survive the in-process commit/fallback notification, but are not persisted in JSONL or reconstructed as fake durations after restart. Metadata-only `compaction.timings` is emitted at `debug` for successful and failed attempts. Failed validation additionally embeds the class-only `unsupportedSpanClasses` report in the `compaction.custom_fallback` warning, so a failing compaction can be diagnosed from logs alone without reproducing the session or reading its content. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.

package/docs/releases/0.4.1.md

@@ -80,8 +80,14 @@ files, 615 tests, TypeScript builds and root typecheck), deterministic
 in a clean consumer for all three packages at 0.4.1 (247 core files, 7
 reference-adapter files, 99 extension files). Tarball review found no session,
 `.pi`, database, JSONL or credential files, and `git diff --check` was clean.
-The coordinated publication then proceeds in dependency order
+The coordinated publication then proceeded in dependency order
 (`ds4-context-core`, then `ds4-context-reference-adapter`, then
-`ds4-context-engine`) with `npm run registry:check -- 0.4.1` afterwards; the
-verified registry result is recorded in the follow-up documentation commit. No
-provider calls are involved in this release procedure.
+`ds4-context-engine`), each command reporting the exact 0.4.1 artifact.
+Registry propagation lagged the publishes: the first
+`npm run registry:check -- 0.4.1` failed with
+`ETARGET ... No matching version found for ds4-context-core@0.4.1`, and
+`ds4-context-engine` still resolved to 0.4.0 after core, reference adapter and
+core again were visible. Once propagation completed, the repeated
+`npm run registry:check -- 0.4.1` reported "Verified all DS4 registry packages
+at exact version 0.4.1", and the `latest` dist-tag resolved to 0.4.1 for all
+three packages. No provider calls are involved in this release procedure.

package/docs/releases/0.4.2.md

@@ -0,0 +1,123 @@
+# Release 0.4.2 — Class-only diagnostics for rejected exact-value spans
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.2.
+**Implementation commit:** `6ecd2af`.
+
+## Summary
+
+Diagnostics patch. Fail-closed exact-value validation now produces a class-only
+report of the rejected backticked spans, and the existing
+`compaction.custom_fallback` warning carries it as `unsupportedSpanClasses`.
+Validation strictness, the eight-bullet and 25% repair bounds, the fail-closed
+fallback to Pi default compaction, and every provider-facing default are
+unchanged. The report contains class names and counters only: it never contains
+span text, so it is safe to log and to quote in reports.
+
+## Changes
+
+- `packages/core/src/compaction/summary-contract.ts` adds
+  `classifyUnsupportedExactValueSpans()`, a pure observation-only classifier
+  returning an `UnsupportedSpanClassReport`:
+  - `relations` counts near-miss relations between the rejected span and the
+    segment source: `escaped-form-present`, `unescaped-form-present`,
+    `whitespace-collapsed-present`, `case-variant-present`,
+    `typographic-variant-present`, `single-deletion-present`,
+    `composed-two-present-parts`, `no-near-miss`, `not-classified-budget`.
+  - `shapes` counts structural shapes: `space`, `backslash`, `double-backslash`,
+    `colon`, `equals`, `double-dash`, `slash`, `quote`, `typographic-char`,
+    `digit-group-separator`, `json-punctuation`.
+  - `lengthBuckets` counts span lengths: `len-1-8`, `len-9-16`, `len-17-32`,
+    `len-33-64`, `len-65-120`, `len-121-plus`.
+  - `spans`, optional `bullets`, and `classificationComplete` report the totals
+    and whether the evidence-probe budget stopped the near-miss classification.
+  - The classifier is bounded: probe budget 4000 evidence lookups, near-miss
+    spans limited to 96 characters, composed parts required to be at least four
+    characters. It cannot alter validation or repair.
+- `src/pi-adapter/summary-generator.ts`: `SummaryValidationError` carries the
+  report and machine-readable issue codes. Error message and behavior are
+  unchanged.
+- `src/pi-adapter/compaction-coordinator.ts`: the existing
+  `custom_fallback` warning gains `unsupportedSpanClasses` when a report exists.
+- Tests: core classifier coverage, generator wiring, coordinator logging, and a
+  guard asserting that logged diagnostics contain no span text.
+- `docs/COMPACTION.md`: diagnostics and validation sections describe the report
+  and its limits.
+
+## Why
+
+Production sessions on separate hosts failed closed with
+`compaction.custom_fallback`, `unsupported-exact-value`,
+`repair=too-many-bullets`, and observed counts of 30/20, 23/19, 19/15, and
+17/11 unsupported spans and affected bullets. The bounded repair removes at most
+eight whole bullets and refuses without attempting any removal above that, so
+every one of those runs fell back to Pi default compaction unmodified.
+
+Counts alone cannot attribute the failure: a summarizer that invents values and
+a summarizer whose rendering or composition rules differ from the validation
+domain need opposite fixes, and the two are indistinguishable in a bare
+`unsupportedSpans` number. 0.4.1 changed the prompt for the composition
+hypothesis; the following observed runs kept the same failure class and the same
+order of magnitude, so the next step is attribution rather than another prompt
+edit. This release adds only the measurement.
+
+## Privacy and safety
+
+- No conversation text is added to any log, warning detail, or error message.
+  The report is a fixed taxonomy of class names plus integer counters.
+- The classifier performs bounded lookups against the already-sanitized segment
+  source used by validation; it does not read session files, SQLite, or any
+  other data, and it does not run on successful summaries.
+- Privacy enforcement, provider-facing payloads, and `compaction` phase timings
+  are untouched.
+
+## Compatibility and persistence
+
+- No SQLite migration or schema change (still 16), no canonical Pi JSONL change,
+  no privacy consent change, no portable runtime-adapter contract change.
+- Validation strictness, the eight-bullet bound, the 25% removal bound, and the
+  fail-open behavior of the extension remain exactly as in 0.4.1.
+- Provider-facing defaults are unchanged: `chars-v1` remains the default
+  estimator, `modelAwareness.autoTune` and DS4 native continuation stay
+  disabled, `storage.scope` keeps its `project` default.
+
+## Measured scope and limitations
+
+The classifier is diagnostic: it does not fix the observed fallback and no
+reduction of the fallback rate is claimed. Its output is only as good as the
+rejected spans it is given, and `classificationComplete: false` marks the runs
+where the probe budget stopped the near-miss classification.
+
+## Related operational finding (not a change in this release)
+
+On the same host an independent failure mode was identified outside DS4: Pi's
+own summarization request caps output at
+`Math.min(Math.floor(0.8 * reserveTokens), model.maxTokens)` with
+`reserveTokens` defaulting to 16384, i.e. 13107 output tokens for the history
+summary and 8192 for the turn-prefix summary, and Pi passes the session thinking
+level to that request. Both the dedicated-model attempt and the fallback attempt
+reported the same stop condition
+(`Summarization failed: generation hit the token cap and the summary is
+incomplete`, raised for `stopReason === "length"`), so compaction ended in an
+error even after DS4 and the dedicated compaction model had both declined.
+Raising `compaction.reserveTokens` globally, or per model through
+`compaction.modelOverrides`, is an operator setting in Pi and is not part of
+this release.
+
+## Validation
+
+On Node 26.5.1 the release passed `npm run check` (builds, `tsc --noEmit`, 99
+Vitest files and 623 tests), deterministic `npm run quality:compare` (candidate
+`0.9875` against baseline `0.808156`), `npm run schema:context-persistence`
+(`passed: true`), and `npm run pack:check` in a clean consumer for all three
+packages at 0.4.2 (247 core files, 7 reference-adapter files, 100 extension
+files). The coordinated publication then proceeded in dependency order
+(`ds4-context-core`, then `ds4-context-reference-adapter`, then
+`ds4-context-engine`), each command reporting the exact 0.4.2 artifact,
+followed by `npm run registry:check -- 0.4.2`. Registry propagation lagged the
+publishes: the first `npm run registry:check -- 0.4.2` failed with
+`ETARGET ... No matching version found for ds4-context-core@0.4.2`, and six
+retries over roughly four minutes reported the same error while the registry
+metadata already listed 0.4.2. Once propagation completed, the repeated
+`npm run registry:check -- 0.4.2` reported "Verified all DS4 registry packages
+at exact version 0.4.2", and the `latest` dist-tag resolved to 0.4.2 for all
+three packages. No provider calls are involved in this release procedure.

package/docs/releases/0.4.3.md

@@ -0,0 +1,159 @@
+# Release 0.4.3 — Fair span coverage and honest "unanalysed" classes
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.3.
+**Implementation commit:** `47100ad`.
+
+## Summary
+
+Diagnostics patch. The class-only report that `0.4.2` attached to a failed
+compaction could not answer the question it exists for: in its first production
+run, 13 of 23 rejected spans came back as `not-classified-budget`, and the two
+`no-near-miss` entries could not be distinguished from spans the classifier
+never analysed. The classifier now covers every distinct span with the cheap
+transformed-form lookups before any near-miss analysis starts, gives each
+near-miss candidate an equal share of what remains, and reports spans it did not
+analyse as unanalysed instead of guessing.
+
+Validation strictness, the eight-bullet and 25% repair bounds, the fail-closed
+fallback to Pi default compaction, the provider-facing defaults, the SQLite
+schema and every configuration key are unchanged. The report still contains
+class names and counters only, so it stays safe to log.
+
+## Changes
+
+- `packages/core/src/compaction/summary-contract.ts`:
+  - Transformed-form lookups (`escaped-form-present`, `unescaped-form-present`,
+    `whitespace-collapsed-present`, `case-variant-present`,
+    `typographic-variant-present`) run for every distinct span before the first
+    near-miss lookup, so span order can no longer decide which spans are
+    classified.
+  - Each remaining near-miss candidate may spend an equal share of the lookups
+    left, at least 32 and never more than the budget remaining, so one long span
+    cannot consume the budget of the others.
+  - New relations distinguish "not analysed" from "absent": `not-classified-length`
+    above the 96-character near-miss limit, `not-classified-partial` when the
+    span's share ran out, and the existing `not-classified-budget` when the
+    shared budget was already exhausted. `no-near-miss` now means every
+    applicable lookup ran and found nothing.
+  - The report adds `spansClassifiedCheap` (span occurrences attributed by a
+    transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`.
+    `classificationComplete` is false when the shared budget or a per-span share
+    stopped an analysis; a `not-classified-length` span does not clear it,
+    because that limit is deterministic rather than a resource shortfall.
+  - `UnsupportedSpanClassOptions.probeBudget` can override the 4000-lookup
+    budget; values below one or non-finite values are ignored. No configuration
+    key was added.
+- `tests/unit/summary-contract.test.ts`: the existing bound test now asserts
+  `not-classified-partial` and `probesUsed <= probeBudget`, plus new coverage for
+  the phase split, the length limit, budget exhaustion and unusable budget
+  values.
+- `docs/COMPACTION.md`: the diagnostics section documents the three
+  not-classified classes, the phase split and the new counters.
+
+No change was needed in `src/pi-adapter/summary-generator.ts` or
+`src/pi-adapter/compaction-coordinator.ts`: the existing error and the existing
+`custom_fallback` warning already carry the report, so the new fields appear in
+`unsupportedSpanClasses` automatically.
+
+## Why
+
+The first production run of the `0.4.2` report (a manual compaction, 23 rejected
+spans across 16 bullets) came back as:
+
+| Class | Count |
+| --- | --- |
+| `escaped-form-present` | 8 |
+| `no-near-miss` | 2 |
+| `not-classified-budget` | 13 |
+
+Length buckets were dominated by long spans (10 in `len-33-64`, 7 in
+`len-65-120`, 2 in `len-121-plus`) and shapes by prose (20 spans containing
+spaces, 10 quotes, 10 colons), so the entries looked like quoted excerpts rather
+than identifiers. Only the eight escaped-form spans were actionable: the value
+was present, only its rendering differed. Two defects in the instrument blocked
+the rest:
+
+- **Coverage.** Every lookup costs one scan of each corpus source
+  (`sourceText`, `readFiles`, `modifiedFiles`) and the budget was spent in span
+  order, while `single-deletion-present` costs up to one lookup per character
+  and `composed-two-present-parts` up to two more per length. A handful of long
+  spans, or their complete analysis, could therefore exhaust the budget before
+  the remaining spans received a single cheap lookup.
+- **Ambiguity.** `no-near-miss` was also returned for spans above
+  `MAX_NEAR_MISS_SPAN_LENGTH`, where the near-miss analysis is never attempted.
+
+Neither defect weakened validation: the repair bound and the fail-closed
+fallback were respected in every one of these runs. The problem was that counts
+alone could not separate a summarizer that invents values from one whose
+rendering or composition rules differ from the validation domain, and those two
+findings call for opposite fixes. This release makes the next report
+interpretable before any change to validation, repair, or the summarizer prompt
+is considered.
+
+## Privacy and safety
+
+- The classifier remains observation-only and cannot alter validation, repair
+  bounds, or the fail-closed decision.
+- The report still contains class names and counters only. The new fields are
+  integers (occurrence, source and lookup counts); no span text, no evidence
+  text, and no hashes of either are added to logs, warnings, or errors.
+- No provider call, network access, or configuration change is involved in this
+  release; the classifier runs only after a validation failure has already been
+  decided.
+
+## Compatibility and persistence
+
+- No migrations, no schema change (still 16), no new configuration key, and no
+  change to any default.
+- The report shape is additive: existing consumers keep working, but a reader of
+  `relations` must tolerate the new keys `not-classified-length` and
+  `not-classified-partial`, and can no longer assume that `no-near-miss` covers
+  every unanalysed span.
+- `classificationComplete: false` is now also possible with the shared budget
+  intact, when a per-span share stops an analysis.
+
+## Measured scope and limitations
+
+Measured locally on the development host with the unit fixtures, no provider
+calls:
+
+- A 96-character span with no transformed-form match costs 185 lookups when
+  analysed completely (96 single-deletion plus 89 prefix probes), confirming that
+  two or three such spans could exhaust the whole budget under the `0.4.2` order.
+- With 80 distinct spans of 91-92 characters and the default budget, the cheap
+  phase spends 80 lookups and each span then receives a 49-lookup share:
+  `probesUsed` equals `probeBudget` exactly and every span is reported as
+  `not-classified-partial`.
+- The knife edge is observable: in a combined fixture, a budget of 190 reports
+  `not-classified-partial` while 191 reports `no-near-miss`, because the share is
+  exhausted on the lookup that completes the analysis.
+
+Limitations that remain:
+
+- Distinct spans are deduplicated, so `spansClassifiedCheap` counts occurrences
+  while the budget is spent per distinct value.
+- Spans above 96 characters are still never near-miss analysed; they now say so
+  instead of claiming absence.
+- The default budget is still 4000 lookups per failure, unchanged from `0.4.2`.
+- A `not-classified-*` class still cannot say whether the span was invented;
+  it says only that the classifier did not look.
+
+## Validation
+
+- `npm run build:core`, `npm run build:adapters`, and `npx tsc --noEmit` clean.
+- `npm run check`: 99 test files, 627 tests, all passing.
+- `npm run quality:compare`, `npm run schema:context-persistence` (passed) and
+  `npm run pack:check`: core 0.4.3 (247 files), reference adapter 0.4.3
+  (7 files) and engine 0.4.3 (101 files) verified in a clean consumer.
+- Classifier behavior was additionally checked against the `0.4.2` semantics
+  with a synthetic reconstruction of the production histogram shape (long spans
+  first, cheap matches after), without touching session data.
+- Not verified: the improved histogram on a real failed compaction. That
+  requires a natural failure and is expected to appear on the next
+  `compaction.custom_fallback` warning.
+
+## Publication
+
+Published manually in dependency order (core, reference adapter, engine) with
+`npm publish`, after `npm run pack:check` on the same commit. Registry
+verification is recorded in the follow-up documentation commit.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.4.1",
+  "version": "0.4.3",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -63,7 +63,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.4.1",
+    "ds4-context-core": "0.4.3",
     "js-tiktoken": "1.0.21"
   },
   "peerDependencies": {

package/src/pi-adapter/compaction-coordinator.ts

@@ -43,6 +43,7 @@ import {
 import {
   generateValidatedSummary,
   sumUsage,
+  SummaryValidationError,
   type GeneratedSummary,
 } from "./summary-generator.ts";
 import {
@@ -501,6 +502,9 @@ export class CompactionCoordinator {
       };
     } catch (error) {
       const message = error instanceof Error ? error.message : String(error);
+      const spanClasses = error instanceof SummaryValidationError
+        ? error.spanClassReport
+        : undefined;
       this.state = {
         ...this.state,
         phase: "failed",
@@ -508,7 +512,11 @@ export class CompactionCoordinator {
         completedAt: this.dependencies.now(),
         lastError: message,
       };
-      this.dependencies.logger.warn("compaction.custom_fallback", { trigger, error: message });
+      this.dependencies.logger.warn("compaction.custom_fallback", {
+        trigger,
+        error: message,
+        ...(spanClasses ? { unsupportedSpanClasses: spanClasses } : {}),
+      });
       if (!event.signal.aborted && ctx.hasUI) {
         ctx.ui.notify(`DS4 compaction unavailable; using Pi default. ${message}`, "warning");
       }

package/src/pi-adapter/summary-generator.ts

@@ -7,16 +7,38 @@ import type {
 import type { CompactionThinkingLevel } from "ds4-context-core/config/config";
 import {
   analyzeUnsupportedExactValueBullets,
+  classifyUnsupportedExactValueSpans,
   groundSummaryFileSections,
   validateSummary,
   type SummaryValidationInput,
   type SummaryValidationResult,
+  type UnsupportedSpanClassReport,
 } from "ds4-context-core/compaction/summary-contract";
 
 export const DEFAULT_COMPACTION_TRANSPORT_MAX_ATTEMPTS = 4;
 export const DEFAULT_COMPACTION_TRANSPORT_BASE_DELAY_MS = 2000;
 export const COMPACTION_TRANSPORT_MAX_DELAY_MS = 60_000;
 
+/**
+ * Fail-closed summary rejection. Carries the class-only span diagnostics so the
+ * coordinator can log why spans were rejected without logging span text.
+ */
+export class SummaryValidationError extends Error {
+  readonly codes: readonly string[];
+  readonly spanClassReport?: UnsupportedSpanClassReport;
+
+  constructor(
+    message: string,
+    codes: readonly string[],
+    spanClassReport?: UnsupportedSpanClassReport,
+  ) {
+    super(message);
+    this.name = "SummaryValidationError";
+    this.codes = codes;
+    this.spanClassReport = spanClassReport;
+  }
+}
+
 /**
  * Transport retry policy for compaction summary requests: four total attempts
  * (the initial call plus three retries), 2000 ms base delay, exponential backoff, abort-aware.
@@ -330,11 +352,20 @@ export async function generateValidatedSummary(
     }
   }
   if (validation.status === "invalid") {
-    const codes = unique(validation.issues.map((issue) => issue.code)).join(", ");
+    const codes = unique(validation.issues.map((issue) => issue.code));
     const repairDiagnostics = exactRepair
       ? `; repair=${exactRepairFailure ?? exactRepair.status}; unsupportedSpans=${exactRepair.unsupportedSpans}; affectedBullets=${exactRepair.affectedBullets}`
       : "";
-    throw new Error(`Compaction ${input.stage} summary validation failed: ${codes}${repairDiagnostics}`);
+    const spanClassReport = validation.issues.some((issue) => issue.code === "unsupported-exact-value")
+      ? classifyUnsupportedExactValueSpans(content, validationInput, {
+          ...(exactRepair ? { affectedBullets: exactRepair.affectedBullets } : {}),
+        })
+      : undefined;
+    throw new SummaryValidationError(
+      `Compaction ${input.stage} summary validation failed: ${codes.join(", ")}${repairDiagnostics}`,
+      codes,
+      spanClassReport,
+    );
   }
   return { content, validation, usage: sumUsage([...retryUsages, response.usage]) };
 }

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.4.1";
+export const EXTENSION_VERSION = "0.4.3";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";