ds4-context-engine 0.3.5 → 0.3.6

package/README.md

@@ -264,6 +264,7 @@ The following example shows the main configuration groups. Omitted values use th
     "minimumOutputReserve": 8192,
     "preferredOutputReserve": 32768,
     "recentTailTokens": 64000,
+    "rescueImmediatePredecessor": true,
     "maxPinnedTokens": 16000,
     "maxMemoryTokens": 8000,
     "maxRetrievedHistoryTokens": 16000,

package/docs/CONTEXT_PLANNER.md

@@ -12,7 +12,7 @@ The managed planner is synchronous, deterministic, provider-independent, and doe
 6. Merge groups linked by assistant tool calls and every matching tool result.
 7. Select the current request, labelled pin groups, and applicable allowed persistent pins as mandatory.
 8. Enforce `maxPinnedTokens`, then fit relevant allowed durable memory under `maxMemoryTokens`.
-9. Walk older turns newest-first, stopping at the first group that would break the contiguous recent tail, target, or hard limit.
+9. Walk older turns newest-first, stopping at the first group that would break the contiguous recent tail, target, or hard limit. When the immediate-predecessor turn itself exceeds the recent-tail cap but still fits the target and hard budgets, it is kept verbatim (`context.rescueImmediatePredecessor`, default true) and the tail then closes.
 10. Privacy-filter and fit source-labelled historical retrieval groups under `maxRetrievedHistoryTokens`.
 11. Privacy-filter and fit hash-current project snippets under `maxProjectTokens`.
 12. Fit active allowed Pi compaction/branch summaries in the remaining summary and input budgets.
@@ -41,6 +41,8 @@ Artifact condensation occurs before atomic grouping. It preserves `toolCallId`,
 
 The retrieval engine produces independent synthetic user-role evidence groups. They are never mandatory: recent turns have priority 100, durable memory 90, retrieved history 85, project snippets 80, and active summaries 75. Each group is selected or excluded whole, carries its original Pi entry ID, and is represented as `retrieval` in the Context Manifest. If planner validation falls back, every synthetic evidence message is discarded and Pi receives its original `AgentMessage[]` unchanged.
 
+Retrieval deduplicates against the entries the managed plan actually commits, not against Pi's native context: before retrieving, the runtime plans the context with mandatory supplements only, maps the committed messages back to Pi session entry IDs, and passes those IDs as the retrieval exclusion set. An entry that Pi still exposes but the planner excludes (for example a turn larger than the recent-tail cap) therefore remains retrievable and reappears as bounded `retrieval` evidence instead of being silently lost. The Context Manifest planning block records `rescuedImmediatePredecessor` and `oversizedTurnExclusions`; an oversized exclusion also emits a `context.excluded_oversized_turn` warning, and `/context explain` surfaces both counters.
+
 Evidence text is a JSON-quoted historical excerpt with an explicit data-only boundary. It is inserted immediately before the latest real user request, so the current task remains the final message and provider conversation order stays deterministic.
 
 ## Project snippets

package/docs/releases/0.3.5.md

@@ -1,6 +1,6 @@
 # DS4 Context Engine 0.3.5
 
-Status: local release gates passed; commit, push and coordinated publication authorized. Validation-only CI, publication and registry verification are pending.
+Status: published as stable on npm under `latest`; annotated tag `v0.3.5` points to validated release commit `8b1d5f4`.
 
 ## Added since 0.3.4
 
@@ -65,6 +65,22 @@ Candidate validation on Node.js `26.5.1`, from a sanitized release source with a
 
 Pre-existing local `allowScripts` additions and `.serena/` are excluded from release commits and public packages.
 
+The final clean committed worktree repeated fresh `npm ci`, all **508 tests**, clean-consumer package verification and tarball inspection successfully. Validation-only CI run [`33970061602`](https://github.com/Alucard24/ds4-context-engine/actions/runs/33970061602) on `8b1d5f4` passed both Node `22.19.0` and `24.x`, including the full suite and package checks.
+
 ## Registry verification and release
 
-Pending successful local gates, validation-only CI, manual publication and exact registry verification. No release tag has been created yet.
+Published manually as `alucard_24`, in dependency order, from reviewed tarballs built in the clean committed worktree:
+
+- `ds4-context-core@0.3.5`: shasum `71105c666a8c88f1abfe5ba3efefe63fc32bba74`;
+- `ds4-context-reference-adapter@0.3.5`: shasum `8eca29472ea4721606d947818eba0aa53a6c6f63`;
+- `ds4-context-engine@0.3.5`: shasum `304b61c21e28f89e82edbee39880210c2c9dc868`.
+
+`npm run registry:check -- 0.3.5` passed: fresh exact-version installation, matching adapter/core dependencies, public core exports, compiled reference conformance, packaged quality corpus, storage CLI usage probe and isolated offline Pi extension startup. Registry SHA-1 and SHA-512 integrity values match the local tarballs for all three packages.
+
+The immediate first post-publication install returned `ETARGET` for the engine. Once exact-version registry metadata was available, verification was repeated with npm `prefer-online` and passed; no package was republished or tag moved.
+
+Verified dist-tags for all three: `latest=0.3.5`, `beta=0.3.0-beta.3`, `alpha=0.3.0-alpha.5`, `rc=0.2.0-rc.1`.
+
+Annotated tag `v0.3.5` targets `8b1d5f4`, the tested/published source. This post-publication evidence update is documentation-only.
+
+GitHub Release: https://github.com/Alucard24/ds4-context-engine/releases/tag/v0.3.5

package/docs/releases/0.3.6.md

@@ -0,0 +1,53 @@
+# DS4 Context Engine 0.3.6
+
+Status: published release on 2026-09-05; tag `v0.3.6`.
+
+This coordinated release fixes a managed-context selection defect: a single oversized atomic conversation turn could cause the planner to discard turns that are still essential to the current request, and its own retrieval second pass could not recover them.
+
+## Fixed
+
+- **Immediate-predecessor rescue**: when the turn immediately preceding the current request exceeds the recent-tail cap but still fits the active target and hard budgets, the planner keeps it verbatim instead of closing the entire recent tail at that group. The tail closes after the rescue, so older turns keep their prior bounded behavior. Controlled by `context.rescueImmediatePredecessor` (default `true`).
+- **Retrieval second chance**: retrieval now deduplicates against the entries the managed plan actually commits, not against Pi's native context. Before the retrieval pass, the runtime plans the context with mandatory supplements only, maps the committed messages back to Pi session entry IDs, and passes those IDs as the retrieval exclusion set. Entries the planner excludes (for example an oversized turn) can therefore reappear as bounded, source-labelled `retrieval` evidence instead of being silently lost.
+- **Diagnostics**: a `context.excluded_oversized_turn` warning reports how many turn groups at or above the recent-tail cap were excluded, whether the immediate predecessor was rescued, and the active limits. The Context Manifest planning block records optional `rescuedImmediatePredecessor` and `oversizedTurnExclusions` counters, and `/context explain` surfaces both.
+
+No validation, privacy, provenance, cancellation, fallback, canonical-history, or bounded-retention behavior was weakened. The runtime retry policy, compaction pipeline, and storage formats are unchanged.
+
+## Compatibility and persistence
+
+The context manifest adds only optional metadata fields. `ds4-context-config-v1`, `runtime-adapter-v1`, `ds4-context-persistence-tool-v1`, `ds4-context-persistence-result-v1`, SQLite schema 15, and migration checksums are unchanged.
+
+## Package/version policy
+
+The coordinated version is `0.3.6` for:
+
+```text
+ds4-context-core
+ds4-context-reference-adapter
+ds4-context-engine
+```
+
+Both adapters depend exactly on `ds4-context-core@0.3.6`. The packages were published manually under npm `latest`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+
+## Validation evidence
+
+Local candidate verification on Node.js `26.5.1`:
+
+- `npm run check`: 81 files and 518 tests passed (including planner rescue cases, second-chance retrieval integration, and entry-id mapping).
+- `npm run quality:compare`: candidate quality `0.9875` versus baseline `0.808156`.
+- `npm run schema:context-persistence`: 1,266 bytes and 317 estimated tokens; below the 1,500 absolute and 320 relative limits.
+- `npm run latency:check -- <exact ds4-context-core@0.1.2>`: passed with ratio `1.082064`, at or below `1.10`.
+- `npm run pack:check`: verified core (235 files), reference adapter (7 files), and Pi adapter (87 files) in a clean consumer.
+- `npm pack --dry-run --json` for all three packages: passed with the same bounded inventories.
+- `git diff --check`: passed.
+- Protected CI, compatibility golden, Pi fixture, migration, canonical Pin/Memory, and persistence-tool contract files: unchanged except the intentional `compatibility-0.2.0.json` default addition.
+
+- `npm run registry:check -- 0.3.6`: passed against all three exact published versions; `latest` resolves to `0.3.6` for every package.
+
+Exact registry verification passed before the annotated tag and GitHub release were created.
+
+## Documentation
+
+- [`../CONTEXT_PLANNER.md`](../CONTEXT_PLANNER.md)
+- [`../COMPACTION.md`](../COMPACTION.md)
+- [`../RELEASING.md`](../RELEASING.md)
+- [`0.3.5.md`](0.3.5.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.3.5",
+  "version": "0.3.6",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -62,7 +62,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.3.5"
+    "ds4-context-core": "0.3.6"
   },
   "peerDependencies": {
     "@earendil-works/pi-ai": "0.84.3",

package/src/extension/commands.ts

@@ -403,6 +403,12 @@ function formatExplain(diagnostics: RuntimeDiagnostics): string {
     `Message target:       ${count(planning.messageTargetTokens)}`,
     `Message hard limit:   ${count(planning.messageHardLimitTokens)}`,
     `Recent-tail limit:    ${count(planning.recentTailTokenLimit)}`,
+    ...(planning.rescuedImmediatePredecessor
+      ? ["Rescued predecessor:  yes (immediate previous turn kept beyond the recent-tail cap within the input budget)"]
+      : []),
+    ...(planning.oversizedTurnExclusions
+      ? [`Oversized turn excl:  ${count(planning.oversizedTurnExclusions)} (turn group(s) at/above the recent-tail cap; recovered by retrieval only if it fits)`]
+      : []),
     `Selected groups:      ${count(planning.selectedGroupCount)}`,
     `Excluded groups:      ${count(planning.excludedGroupCount)}`,
     `Duration:             ${planning.durationMs === undefined ? "n/a" : `${planning.durationMs.toFixed(1)} ms`}`,

package/src/extension/runtime.ts

@@ -97,7 +97,7 @@ import {
   type PinScope,
   type ProjectMemorySource,
 } from "ds4-context-core/memory/memory-types";
-import { planManagedContext, type SupplementalContextMessage } from "ds4-context-core/planner/context-planner";
+import { planManagedContext, type ManagedContextPlan, type SupplementalContextMessage } from "ds4-context-core/planner/context-planner";
 import {
   disabledContextQualityDiagnostics,
   evaluateManifestQuality,
@@ -151,6 +151,7 @@ import {
   buildPiObserverManifest,
   findExactPiMessageSourceIds,
   findPiPinnedMessageIndices,
+  findPiSourceEntryIds,
 } from "../pi-adapter/context-observer.ts";
 import { projectSessionFileMutations } from "../pi-adapter/memory-adapter.ts";
 import { ProjectMemorySynchronizer } from "../pi-adapter/project-memory-sync.ts";
@@ -973,10 +974,43 @@ export class Ds4ContextRuntime {
           memoryTokens: 0,
         };
         if (this.memoryManager) this.lastMemory = this.memoryManager.diagnostics();
+        const fixedTokens = baseline.composition.systemTokens + baseline.composition.toolTokens;
+        const pinnedMessageIndices = findPiPinnedMessageIndices(effectiveEvent, ctx);
+        const retrievalEnabled = this.config.retrieval.exact
+          || this.config.retrieval.fts
+          || this.config.retrieval.semantic;
+        const retrievalActiveContextEntryIds = retrievalEnabled
+          ? this.plannedContextEntryIds(ctx, planManagedContext({
+              messages: effectiveEvent.messages,
+              fixedTokens,
+              budget,
+              config: effectiveContextConfig,
+              pinnedMessageIndices,
+              supplementalMessages: [
+                ...memorySelection.pins.map((evidence) => ({
+                  id: `pin:${evidence.item.id}`,
+                  message: evidence.message,
+                  kind: "pin" as const,
+                  sourceIds: [evidence.item.id],
+                  score: 950,
+                  reason: evidence.reason,
+                })),
+                ...memorySelection.memories.map((evidence) => ({
+                  id: `memory:${evidence.item.id}`,
+                  message: evidence.message,
+                  kind: "memory" as const,
+                  sourceIds: [evidence.item.id],
+                  score: 90 + Math.min(0.999999, Math.max(0, evidence.score) / 1_000),
+                  reason: evidence.reason,
+                })),
+              ],
+            }))
+          : undefined;
         const retrieval = this.retrieveHistory(
           effectiveEvent,
           ctx,
           effectiveContextConfig.maxRetrievedHistoryTokens,
+          retrievalActiveContextEntryIds,
         );
         const project = this.retrieveProjectKnowledge(
           requestText,
@@ -1110,10 +1144,10 @@ export class Ds4ContextRuntime {
         });
         const plan = planManagedContext({
           messages: effectiveEvent.messages,
-          fixedTokens: baseline.composition.systemTokens + baseline.composition.toolTokens,
+          fixedTokens,
           budget,
           config: effectiveContextConfig,
-          pinnedMessageIndices: findPiPinnedMessageIndices(effectiveEvent, ctx),
+          pinnedMessageIndices,
           supplementalMessages: rankedSupplementalMessages,
           ...(ranking.diagnostics.status === "active"
             ? { supplementalSelectionOrder: ranking.ranked.map((candidate) => candidate.id) }
@@ -1175,6 +1209,17 @@ export class Ds4ContextRuntime {
           selected: plannerSelectedProject,
         };
         plan.planning.durationMs = Math.max(0, this.now() - planningStartedAt);
+        const oversizedTurnExclusions = plan.planning.oversizedTurnExclusions ?? 0;
+        if (plan.mode === "managed" && oversizedTurnExclusions > 0) {
+          this.logger.warn("context.excluded_oversized_turn", {
+            oversizedTurnCount: oversizedTurnExclusions,
+            rescuedImmediatePredecessor: plan.planning.rescuedImmediatePredecessor ?? false,
+            recentTailTokenLimit: plan.planning.recentTailTokenLimit,
+            messageTargetTokens: plan.planning.messageTargetTokens,
+            selectedGroupCount: plan.planning.selectedGroupCount,
+            excludedGroupCount: plan.planning.excludedGroupCount,
+          });
+        }
         const plannedEvent: ContextEvent = { type: "context", messages: plan.messages };
         const selectedArtifactReferences = plan.mode === "managed"
           ? plan.selected.flatMap((metadata) => {
@@ -2758,10 +2803,36 @@ export class Ds4ContextRuntime {
     }
   }
 
+  private plannedContextEntryIds(
+    ctx: ExtensionContext,
+    plan: ManagedContextPlan<ContextEvent["messages"][number]>,
+  ): Set<string> {
+    if (plan.mode === "fallback") {
+      return new Set(ctx.sessionManager.buildContextEntries().map((entry) => entry.id));
+    }
+    const syntheticIndices = new Set(
+      [...plan.selected, ...plan.excluded]
+        .filter((metadata) =>
+          metadata.kind === "memory"
+          || (metadata.kind === "pin" && metadata.sourceId !== undefined)
+        )
+        .map((metadata) => metadata.originalIndex),
+    );
+    const selectedIndices = new Set(plan.selected.map((metadata) => metadata.originalIndex));
+    const sources = findPiSourceEntryIds(plan.originalMessages, ctx, syntheticIndices);
+    const entryIds = new Set<string>();
+    for (const index of selectedIndices) {
+      const sourceId = sources[index];
+      if (sourceId) entryIds.add(sourceId);
+    }
+    return entryIds;
+  }
+
   private retrieveHistory(
     event: ContextEvent,
     ctx: ExtensionContext,
     maxTokens = this.config.context.maxRetrievedHistoryTokens,
+    activeContextEntryIds?: ReadonlySet<string>,
   ): RetrievalDiagnostics {
     const requestText = currentRequestText(event.messages);
     if (!this.retrievalEngine || !this.session?.sessionFile || !requestText) {
@@ -2775,7 +2846,8 @@ export class Ds4ContextRuntime {
         sessionId: this.session.sessionId,
         requestText,
         activeBranchEntryIds: new Set(ctx.sessionManager.getBranch().map((entry) => entry.id)),
-        activeContextEntryIds: new Set(ctx.sessionManager.buildContextEntries().map((entry) => entry.id)),
+        activeContextEntryIds: activeContextEntryIds
+          ?? new Set(ctx.sessionManager.buildContextEntries().map((entry) => entry.id)),
         exact: this.config.retrieval.exact,
         fts: this.config.retrieval.fts,
         semantic: this.config.retrieval.semantic,

package/src/pi-adapter/context-observer.ts

@@ -244,6 +244,22 @@ export function findExactPiMessageSourceIds(
   });
 }
 
+/**
+ * Maps managed-plan messages back to Pi session entry ids. Synthetic evidence
+ * indices are skipped so role/order fallbacks stay aligned with real messages.
+ */
+export function findPiSourceEntryIds(
+  messages: readonly unknown[],
+  ctx: Pick<ExtensionContext, "sessionManager">,
+  syntheticIndices: ReadonlySet<number> = new Set(),
+): Array<string | undefined> {
+  return mapMessageSources(
+    messages,
+    sourceCandidates(ctx.sessionManager.buildContextEntries()),
+    syntheticIndices,
+  ).map((source) => source.sourceId);
+}
+
 export function findPiPinnedMessageIndices(event: ContextEvent, ctx: ExtensionContext): number[] {
   const candidates = sourceCandidates(ctx.sessionManager.buildContextEntries());
   const sources = mapMessageSources(event.messages, candidates);

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.3.5";
+export const EXTENSION_VERSION = "0.3.6";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";