ds4-context-engine 0.4.2 → 0.4.3

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch `0.4.2` adds class-only diagnostics for rejected exact-value spans: the existing `custom_fallback` warning carries `unsupportedSpanClasses` as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. See the [0.4.2 release record](docs/releases/0.4.2.md), the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
+> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. Patch `0.4.1` states the contiguous-span rule for compaction summaries in the summarizer prompt while leaving validation strictness, the eight-bullet repair bound and every provider-facing default unchanged. Patch `0.4.2` adds class-only diagnostics for rejected exact-value spans: the existing `custom_fallback` warning carries `unsupportedSpanClasses` as class names and counters, never span text, with validation, repair bounds and provider-facing defaults unchanged. Patch `0.4.3` makes those diagnostics interpretable: every distinct span receives the cheap transformed-form lookups before the bounded near-miss analysis starts, each near-miss candidate may spend only an equal share of what remains, and spans that were not analysed report `not-classified-length`, `not-classified-partial` or `not-classified-budget` instead of `no-near-miss`. See the [0.4.3 release record](docs/releases/0.4.3.md), the [0.4.2 release record](docs/releases/0.4.2.md), the [0.4.1 release record](docs/releases/0.4.1.md) and the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
 
 **Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
@@ -515,6 +515,7 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.4.3 release notes](docs/releases/0.4.3.md)
 - [0.4.2 release notes](docs/releases/0.4.2.md)
 - [0.4.1 release notes](docs/releases/0.4.1.md)
 - [0.4.0 release notes](docs/releases/0.4.0.md)

package/docs/COMPACTION.md

@@ -36,7 +36,9 @@ Every section must occur once, in order, and contain content or `- None`. DS4 re
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
-The same failure carries a class-only span report on the fallback warning (`unsupportedSpanClasses`): rejected-span count, affected bullets, length buckets, character shapes (spaces, backslashes, escape sequences, separators, quotes, typographic characters, JSON punctuation), and how each distinct span relates to the evidence — escaped or unescaped rendering, collapsed whitespace, case or typographic variant, one-character deletion, two separately present values joined into one span, no near-miss, or not classified because the bounded probe budget was exhausted. The report contains class names and counters only, never span text, and the classifier is observation-only: it cannot change validation, repair bounds, or the fail-closed decision. It exists to separate a summarizer that invents values from one whose rendering or composition rules differ from the validation domain, which require opposite fixes.
+The same failure carries a class-only span report on the fallback warning (`unsupportedSpanClasses`): rejected-span count, affected bullets, length buckets, character shapes (spaces, backslashes, escape sequences, separators, quotes, typographic characters, JSON punctuation), and how each distinct span relates to the evidence — escaped or unescaped rendering, collapsed whitespace, case or typographic variant, one-character deletion, two separately present values joined into one span, or no near-miss (every applicable lookup ran and found nothing). The report contains class names and counters only, never span text, and the classifier is observation-only: it cannot change validation, repair bounds, or the fail-closed decision. It exists to separate a summarizer that invents values from one whose rendering or composition rules differ from the validation domain, which require opposite fixes.
+
+Three relations mark spans that were *not* analysed, and they must never be read as invention. `not-classified-length` means the span exceeds the near-miss length limit (96 characters); `not-classified-partial` means its share of the budget ran out; `not-classified-budget` means the shared budget was already exhausted. To keep the histogram interpretable, transformed-form lookups cover every distinct span before any near-miss analysis starts, and each near-miss candidate may spend only an equal share of what remains (at least 32 lookups, never more than the budget left). The report also carries `spansClassifiedCheap` (span occurrences attributed by a transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`, so a run can be read without guessing how much of the budget was consumed. `classificationComplete` is false when the shared budget or a per-span share stopped an analysis; a `not-classified-length` span does not clear it, because that limit is deterministic rather than a resource shortfall.
 
 ## Provenance and recovery
 

package/docs/releases/0.4.2.md

@@ -110,4 +110,14 @@ Vitest files and 623 tests), deterministic `npm run quality:compare` (candidate
 `0.9875` against baseline `0.808156`), `npm run schema:context-persistence`
 (`passed: true`), and `npm run pack:check` in a clean consumer for all three
 packages at 0.4.2 (247 core files, 7 reference-adapter files, 100 extension
-files). No provider calls are involved in this release procedure.
+files). The coordinated publication then proceeded in dependency order
+(`ds4-context-core`, then `ds4-context-reference-adapter`, then
+`ds4-context-engine`), each command reporting the exact 0.4.2 artifact,
+followed by `npm run registry:check -- 0.4.2`. Registry propagation lagged the
+publishes: the first `npm run registry:check -- 0.4.2` failed with
+`ETARGET ... No matching version found for ds4-context-core@0.4.2`, and six
+retries over roughly four minutes reported the same error while the registry
+metadata already listed 0.4.2. Once propagation completed, the repeated
+`npm run registry:check -- 0.4.2` reported "Verified all DS4 registry packages
+at exact version 0.4.2", and the `latest` dist-tag resolved to 0.4.2 for all
+three packages. No provider calls are involved in this release procedure.

package/docs/releases/0.4.3.md

@@ -0,0 +1,159 @@
+# Release 0.4.3 — Fair span coverage and honest "unanalysed" classes
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.3.
+**Implementation commit:** `47100ad`.
+
+## Summary
+
+Diagnostics patch. The class-only report that `0.4.2` attached to a failed
+compaction could not answer the question it exists for: in its first production
+run, 13 of 23 rejected spans came back as `not-classified-budget`, and the two
+`no-near-miss` entries could not be distinguished from spans the classifier
+never analysed. The classifier now covers every distinct span with the cheap
+transformed-form lookups before any near-miss analysis starts, gives each
+near-miss candidate an equal share of what remains, and reports spans it did not
+analyse as unanalysed instead of guessing.
+
+Validation strictness, the eight-bullet and 25% repair bounds, the fail-closed
+fallback to Pi default compaction, the provider-facing defaults, the SQLite
+schema and every configuration key are unchanged. The report still contains
+class names and counters only, so it stays safe to log.
+
+## Changes
+
+- `packages/core/src/compaction/summary-contract.ts`:
+  - Transformed-form lookups (`escaped-form-present`, `unescaped-form-present`,
+    `whitespace-collapsed-present`, `case-variant-present`,
+    `typographic-variant-present`) run for every distinct span before the first
+    near-miss lookup, so span order can no longer decide which spans are
+    classified.
+  - Each remaining near-miss candidate may spend an equal share of the lookups
+    left, at least 32 and never more than the budget remaining, so one long span
+    cannot consume the budget of the others.
+  - New relations distinguish "not analysed" from "absent": `not-classified-length`
+    above the 96-character near-miss limit, `not-classified-partial` when the
+    span's share ran out, and the existing `not-classified-budget` when the
+    shared budget was already exhausted. `no-near-miss` now means every
+    applicable lookup ran and found nothing.
+  - The report adds `spansClassifiedCheap` (span occurrences attributed by a
+    transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`.
+    `classificationComplete` is false when the shared budget or a per-span share
+    stopped an analysis; a `not-classified-length` span does not clear it,
+    because that limit is deterministic rather than a resource shortfall.
+  - `UnsupportedSpanClassOptions.probeBudget` can override the 4000-lookup
+    budget; values below one or non-finite values are ignored. No configuration
+    key was added.
+- `tests/unit/summary-contract.test.ts`: the existing bound test now asserts
+  `not-classified-partial` and `probesUsed <= probeBudget`, plus new coverage for
+  the phase split, the length limit, budget exhaustion and unusable budget
+  values.
+- `docs/COMPACTION.md`: the diagnostics section documents the three
+  not-classified classes, the phase split and the new counters.
+
+No change was needed in `src/pi-adapter/summary-generator.ts` or
+`src/pi-adapter/compaction-coordinator.ts`: the existing error and the existing
+`custom_fallback` warning already carry the report, so the new fields appear in
+`unsupportedSpanClasses` automatically.
+
+## Why
+
+The first production run of the `0.4.2` report (a manual compaction, 23 rejected
+spans across 16 bullets) came back as:
+
+| Class | Count |
+| --- | --- |
+| `escaped-form-present` | 8 |
+| `no-near-miss` | 2 |
+| `not-classified-budget` | 13 |
+
+Length buckets were dominated by long spans (10 in `len-33-64`, 7 in
+`len-65-120`, 2 in `len-121-plus`) and shapes by prose (20 spans containing
+spaces, 10 quotes, 10 colons), so the entries looked like quoted excerpts rather
+than identifiers. Only the eight escaped-form spans were actionable: the value
+was present, only its rendering differed. Two defects in the instrument blocked
+the rest:
+
+- **Coverage.** Every lookup costs one scan of each corpus source
+  (`sourceText`, `readFiles`, `modifiedFiles`) and the budget was spent in span
+  order, while `single-deletion-present` costs up to one lookup per character
+  and `composed-two-present-parts` up to two more per length. A handful of long
+  spans, or their complete analysis, could therefore exhaust the budget before
+  the remaining spans received a single cheap lookup.
+- **Ambiguity.** `no-near-miss` was also returned for spans above
+  `MAX_NEAR_MISS_SPAN_LENGTH`, where the near-miss analysis is never attempted.
+
+Neither defect weakened validation: the repair bound and the fail-closed
+fallback were respected in every one of these runs. The problem was that counts
+alone could not separate a summarizer that invents values from one whose
+rendering or composition rules differ from the validation domain, and those two
+findings call for opposite fixes. This release makes the next report
+interpretable before any change to validation, repair, or the summarizer prompt
+is considered.
+
+## Privacy and safety
+
+- The classifier remains observation-only and cannot alter validation, repair
+  bounds, or the fail-closed decision.
+- The report still contains class names and counters only. The new fields are
+  integers (occurrence, source and lookup counts); no span text, no evidence
+  text, and no hashes of either are added to logs, warnings, or errors.
+- No provider call, network access, or configuration change is involved in this
+  release; the classifier runs only after a validation failure has already been
+  decided.
+
+## Compatibility and persistence
+
+- No migrations, no schema change (still 16), no new configuration key, and no
+  change to any default.
+- The report shape is additive: existing consumers keep working, but a reader of
+  `relations` must tolerate the new keys `not-classified-length` and
+  `not-classified-partial`, and can no longer assume that `no-near-miss` covers
+  every unanalysed span.
+- `classificationComplete: false` is now also possible with the shared budget
+  intact, when a per-span share stops an analysis.
+
+## Measured scope and limitations
+
+Measured locally on the development host with the unit fixtures, no provider
+calls:
+
+- A 96-character span with no transformed-form match costs 185 lookups when
+  analysed completely (96 single-deletion plus 89 prefix probes), confirming that
+  two or three such spans could exhaust the whole budget under the `0.4.2` order.
+- With 80 distinct spans of 91-92 characters and the default budget, the cheap
+  phase spends 80 lookups and each span then receives a 49-lookup share:
+  `probesUsed` equals `probeBudget` exactly and every span is reported as
+  `not-classified-partial`.
+- The knife edge is observable: in a combined fixture, a budget of 190 reports
+  `not-classified-partial` while 191 reports `no-near-miss`, because the share is
+  exhausted on the lookup that completes the analysis.
+
+Limitations that remain:
+
+- Distinct spans are deduplicated, so `spansClassifiedCheap` counts occurrences
+  while the budget is spent per distinct value.
+- Spans above 96 characters are still never near-miss analysed; they now say so
+  instead of claiming absence.
+- The default budget is still 4000 lookups per failure, unchanged from `0.4.2`.
+- A `not-classified-*` class still cannot say whether the span was invented;
+  it says only that the classifier did not look.
+
+## Validation
+
+- `npm run build:core`, `npm run build:adapters`, and `npx tsc --noEmit` clean.
+- `npm run check`: 99 test files, 627 tests, all passing.
+- `npm run quality:compare`, `npm run schema:context-persistence` (passed) and
+  `npm run pack:check`: core 0.4.3 (247 files), reference adapter 0.4.3
+  (7 files) and engine 0.4.3 (101 files) verified in a clean consumer.
+- Classifier behavior was additionally checked against the `0.4.2` semantics
+  with a synthetic reconstruction of the production histogram shape (long spans
+  first, cheap matches after), without touching session data.
+- Not verified: the improved histogram on a real failed compaction. That
+  requires a natural failure and is expected to appear on the next
+  `compaction.custom_fallback` warning.
+
+## Publication
+
+Published manually in dependency order (core, reference adapter, engine) with
+`npm publish`, after `npm run pack:check` on the same commit. Registry
+verification is recorded in the follow-up documentation commit.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.4.2",
+  "version": "0.4.3",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -63,7 +63,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.4.2",
+    "ds4-context-core": "0.4.3",
     "js-tiktoken": "1.0.21"
   },
   "peerDependencies": {

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.4.2";
+export const EXTENSION_VERSION = "0.4.3";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";