ds4-context-engine 0.3.4 → 0.3.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -16,7 +16,9 @@ bounded active context with provenance
16
16
  Pi provider
17
17
  ```
18
18
 
19
- > **Project status:** The coordinated `0.3.4` release adds optional anchored editing, post-edit reports, adaptive reads, adaptive artifact budgets and managed local bash jobs. All five switches default to off. It carries forward the stable 0.3 line's persistence, offline storage maintenance, dedicated compaction model and configurable transport retries. Canonical history, SQLite schema 15 and runtime contracts remain unchanged; Pi remains pinned to `0.84.3`. See the [0.3.4 release record](docs/releases/0.3.4.md) for validation and publication status.
19
+ > **Project status:** The coordinated `0.3.5` release adds bounded compaction optimizations: one validated previous-summary/source update when the complete prompt fits, a dedicated calibrated input budget, up to two concurrent segments, and metadata-only phase timings in `/context compaction`. Canonical history, SQLite schema 15 and runtime contracts remain unchanged; Pi remains pinned to `0.84.3`. See the [0.3.5 release record](docs/releases/0.3.5.md) for validation and publication status.
20
+
21
+ **New compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="summary"`, `compaction.maxConcurrentSegments=2`. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls) for the legacy-path settings. No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
20
22
 
21
23
  ## Why DS4
22
24
 
@@ -87,7 +89,7 @@ Install the latest stable public npm package with:
87
89
  pi install npm:ds4-context-engine
88
90
  ```
89
91
 
90
- The three packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) are released together with the same exact version. Both adapters require the matching core version. See [0.3.4](docs/releases/0.3.4.md) for this release's changes and compatibility.
92
+ The three packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) are released together with the same exact version. Both adapters require the matching core version. See [0.3.5](docs/releases/0.3.5.md) for this release's changes and compatibility.
91
93
 
92
94
  To dogfood the beta without replacing a global stable installation, pin the exact version in a disposable project:
93
95
 
@@ -262,6 +264,7 @@ The following example shows the main configuration groups. Omitted values use th
262
264
  "minimumOutputReserve": 8192,
263
265
  "preferredOutputReserve": 32768,
264
266
  "recentTailTokens": 64000,
267
+ "rescueImmediatePredecessor": true,
265
268
  "maxPinnedTokens": 16000,
266
269
  "maxMemoryTokens": 8000,
267
270
  "maxRetrievedHistoryTokens": 16000,
@@ -328,7 +331,10 @@ The following example shows the main configuration groups. Omitted values use th
328
331
  "mode": "hierarchical",
329
332
  "validate": true,
330
333
  "segmentTargetTokens": 30000,
331
- "preserveRecentVerbatim": true
334
+ "preserveRecentVerbatim": true,
335
+ "directUpdate": true,
336
+ "inputBudget": "summary",
337
+ "maxConcurrentSegments": 2
332
338
  },
333
339
  "privacy": {
334
340
  "enabled": false,
@@ -506,6 +512,7 @@ scripts package and release-readiness checks
506
512
  - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
507
513
  - [Release process](docs/RELEASING.md)
508
514
  - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
515
+ - [0.3.5 release notes](docs/releases/0.3.5.md)
509
516
  - [0.3.4 release notes](docs/releases/0.3.4.md)
510
517
  - [0.3.3 release notes](docs/releases/0.3.3.md)
511
518
  - [0.3.2 release notes](docs/releases/0.3.2.md)
@@ -528,7 +535,7 @@ scripts package and release-readiness checks
528
535
 
529
536
  The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
530
537
 
531
- The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.3.4 adds opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md), without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
538
+ The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.3.5 adds bounded compaction updates, summary input headroom, concurrent segments and phase timings. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
532
539
 
533
540
  ## Contributing
534
541
 
@@ -0,0 +1,21 @@
1
+ # ADR-061 — Bounded compaction latency optimizations
2
+
3
+ Status: accepted and implemented; coordinated 0.3.5 publication explicitly authorized by the user on 2026-09-05. See the [release record](../releases/0.3.5.md) for validation and publication evidence.
4
+
5
+ ## Context
6
+
7
+ Pi normally updates previous summary plus discarded messages in one request. DS4 previously always summarized new segments first and then generated an aggregate when a predecessor existed. Independent segments ran sequentially and summary inputs reused the ordinary context fill target. This can add avoidable calls; provider latency and summary quality still require controlled real-provider comparisons.
8
+
9
+ ## Decision
10
+
11
+ - Add `compaction.directUpdate` (default true): estimate the complete sanitized update prompt, including previous summary, cumulative file evidence, custom instructions, split-turn instructions and output contract. If it fits, generate and validate one immutable `task-state` node with the predecessor as a child, canonical source IDs from both spans, and a hash bound to both the new source and predecessor identity/hash/content. With no predecessor, the existing one-segment path remains sufficient. Oversized updates use bounded hierarchical segmentation/aggregation, not truncation.
12
+ - Add `compaction.inputBudget` (`summary` default, or `context`). Summary mode uses the calibrated hard input limit instead of the ordinary active/fill target. Additionally bound input by model context minus safety margin and the actual summary output cap, converted to estimator units. Do not raise model/configured hard limits, remove calibration, reduce safety margins, or reuse ordinary context space reserved for output. Context mode preserves the old active target (also respecting actual summary output headroom).
13
+ - Add `compaction.maxConcurrentSegments` (default 2, range 1–2). Only independent segment requests overlap. Allocate identities and assemble graph/usage in source order, not completion order. On failure stop scheduling, abort siblings, await their settlement before fallback, and persist no partial graph. Cancellation remains cooperative; provider work already accepted may still be billed. Aggregation remains ordered and bounded.
14
+ - Expose process-local metadata-only timings for preparation, segment/direct generation, aggregation, graph preparation/persistence and total DS4 hook duration, plus chosen path, effective provider/model, direct prompt size and concurrency. Timings use a monotonic clock, include retries and local validation within generation, and are wall times (not sums of overlapping calls). They are not canonical evidence, are not persisted in JSONL, and do not measure a subsequent native Pi fallback.
15
+ - Preserve schema-v2 compaction metadata, the existing `task-state` kind, source/classification/validation contracts, Pi cut points, atomic tool exchanges, fresh routing IDs, retry policy, canonical JSONL and all-or-nothing graph preparation. `segmentSummaryId` refers to the update node itself on a direct update; do not invent a segment that was never generated.
16
+
17
+ ## Compatibility and validation
18
+
19
+ Configuration is additive; absent fields use the new defaults. The old scheduling path can be compared using `directUpdate=false`, `inputBudget=context`, and `maxConcurrentSegments=1`. The compaction master switch still delegates to Pi when disabled. The original local implementation excluded versioning and publication; the user subsequently authorized the coordinated 0.3.5 release. No dependency upgrade, live database maintenance or Pi upgrade is part of this change.
20
+
21
+ Cover exact budget boundaries and calibration/output reserve, predecessor-only exact evidence and privacy, native predecessor warnings, source ordering/provenance/hash/rebuild, bounded concurrency and out-of-order completion, cancellation/failure/transport retry drainage, and no graph persistence before full success. Retain hierarchical regression coverage explicitly even where default routing now chooses a direct update. Automated call counts and synchronization tests establish scheduling behavior, not a promise of 40-second compactions or unchanged semantic quality on real provider outputs.
@@ -64,5 +64,6 @@ The initial decisions from the development plan are accepted:
64
64
  | [058](058-bounded-manifest-storage.md) | Bound persisted manifests and defer compression to a versioned migration | Accepted |
65
65
  | [059](059-optional-anchored-editing.md) | Opt in to anchored edit expansion inside Pi's native mutation queue | Accepted |
66
66
  | [060](060-optional-portable-agent-tools.md) | Opt in to edit reports, adaptive reads/results and session-owned local jobs | Accepted |
67
+ | [061](061-compaction-latency.md) | Bound compaction update calls, input budgets, concurrent segments and phase timings | Accepted |
67
68
 
68
69
  Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.
@@ -6,14 +6,14 @@ DS4 intercepts Pi's `session_before_compact` event but preserves Pi's cut-point
6
6
 
7
7
  1. Pi determines `messagesToSummarize`, optional split-turn prefix, and `firstKeptEntryId`.
8
8
  2. DS4 maps every source message by exact fingerprint to a canonical branch entry ID.
9
- 3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, custom instructions, and file paths for the active provider.
10
- 4. DS4 estimates the complete sanitized summary request against the calibrated active-model input budget. If the whole request does not fit, it partitions the source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results; an exceptionally large turn may split only between such groups.
11
- 5. The active model generates and validates each immutable segment summary only against that segment's sanitized evidence, with cache retention disabled and a fresh routing session ID per call. Only failures categorized as `transport` are retried: at most two retries follow bounded 200 ms and 500 ms delays, each with another fresh routing session ID. The delay and the next call both honor Pi's abort signal; all other failure categories fall back immediately.
12
- 6. DS4 recursively aggregates ordered child summaries in bounded fan-in calls until one root remains. The ordered roots include the previous branch summary, when present, followed by every new segment. DS4-generated IDs, hashes, kinds, and graph levels remain outside model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
13
- 7. The highest input classification is wrapped around each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage is summed across every segment and aggregate request; Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
9
+ 3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, previous summary, custom instructions, and file paths for the effective compaction provider (dedicated model when configured and eligible).
10
+ 4. DS4 estimates the **complete** sanitized request against a calibrated summary-specific input budget, including framing, instructions, file inventories and output contract. With `compaction.directUpdate` enabled, a previous summary plus new source that fits is updated and validated in **one call**, producing an immutable `task-state` node. No predecessor needs only the existing one-segment call. Oversized updates fall through to hierarchical planning; no additional source is truncated to force a fit.
11
+ 5. The hierarchical path partitions oversized source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results. Up to `compaction.maxConcurrentSegments` independent segment requests run concurrently (default 2). Each summary is validated against only its own sanitized evidence. Identities, source/child order and usage accumulation follow source order, not completion order. Cache retention stays disabled and every attempt has a fresh routing session ID.
12
+ 6. Hierarchical requests recursively aggregate ordered children until one root remains: previous branch summary first, then new segments. Aggregation stays sequential and budget-checked. Direct updates instead link their single node to the predecessor and new canonical source IDs, without generating a synthetic segment or a separate aggregate. DS4-generated IDs, hashes, kinds, and graph levels never become model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
13
+ 7. The highest input classification wraps each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage includes every returned direct/segment/aggregate request and transport replay. Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
14
14
  8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the complete prepared batch `failed`.
15
15
 
16
- Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. Transport replay mirrors Pi's assistant retry policy: `compaction.transport.maxAttempts` (default 3) total attempts and `compaction.transport.baseDelayMs` (default 2000 ms, capped at 60 s) backoff, doubling per attempt, abort-aware. Replay never applies to input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run.
16
+ Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. The DS4 transport replay policy is: `compaction.transport.maxAttempts` (default 3) total attempts and `compaction.transport.baseDelayMs` (default 2000 ms, capped at 60 s) backoff, doubling per attempt, abort-aware. Replay never applies to input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run. On segment failure or cancellation DS4 stops scheduling, aborts siblings and awaits all started workers before fallback; no partial graph is installed. Cancellation is cooperative: accepted provider work may still cost tokens, and a provider that ignores abort can delay settlement.
17
17
 
18
18
  ## Required summary contract
19
19
 
@@ -32,7 +32,7 @@ Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, an
32
32
  ## Critical Exact Values
33
33
  ```
34
34
 
35
- Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. Any unrepaired failure prevents the whole graph batch from being installed.
35
+ Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. A direct update is validated against both the sanitized previous summary and new source, plus cumulative sanitized file evidence; previous-summary-only exact values remain valid evidence. Custom focus instructions are not factual evidence. Any unrepaired failure prevents the whole graph batch from being installed.
36
36
 
37
37
  An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
38
38
 
@@ -49,7 +49,7 @@ An unrepaired exact-value failure reports only the stage, issue code, categorica
49
49
  - trigger, split-turn flag, source message count;
50
50
  - generation time, provider, and model.
51
51
 
52
- For aggregate compactions, details also embed every non-active node created by that operation. The active node content remains `CompactionEntry.summary`; prior ancestors remain in earlier canonical entries. SQLite stores content, ordered edges, direct/transitive sources, graph level, and lifecycle as a disposable projection. On resume or after deleting the database, DS4 replays Pi entries in append order and recreates the complete graph.
52
+ For aggregate and direct-update compactions, details also embed every non-active node created by that operation. Direct updates reuse the schema-v2 `task-state` kind, set `segmentSummaryId` to their own node ID, increment the predecessor's level, and include the union of previous/new canonical source IDs. Their hash binds the new source hash to predecessor ID, hash, content and level. Prior nodes are never rewritten; no unused segment is fabricated. The active node content remains `CompactionEntry.summary`; prior ancestors remain in earlier canonical entries. SQLite stores content, ordered edges, direct/transitive sources, graph level, and lifecycle as a disposable projection. On resume or after deleting the database, DS4 replays Pi entries in append order and recreates the complete graph.
53
53
 
54
54
  Memory and pin custom entries do not participate directly in Pi's LLM context and are not replaced by summary text. Their append-only mutations remain in the session tree, so durable decisions and explicit classifications replay after any number of compactions without depending exclusively on a summary.
55
55
 
@@ -78,9 +78,31 @@ Semantics:
78
78
  - `compaction.summary.thinking` defaults to `off` and applies only to summary requests: `off` keeps the pre-existing request shape (no thinking fields), while other levels map per API (`thinkingEnabled`/`effort` for `anthropic-messages`, `samplingParams.reasoning_effort` for OpenAI-compatible APIs) and are ignored for unsupported providers;
79
79
  - `context.maxSummaryTokens` remains a session-level limit and does not rise for the dedicated model; the minimum with the model's `maxTokens` still applies.
80
80
 
81
+ ## Latency controls
82
+
83
+ The coordinated `0.3.5` release introduces these additive defaults (absent from `0.3.4`):
84
+
85
+ ```json
86
+ {
87
+ "compaction": {
88
+ "directUpdate": true,
89
+ "inputBudget": "summary",
90
+ "maxConcurrentSegments": 2
91
+ }
92
+ }
93
+ ```
94
+
95
+ - `directUpdate`: one validated previous-summary plus new-source request when the entire prompt fits. Set false to always retain the segment-then-aggregate route. Validation or provider failures still fall back to Pi, not an unvalidated update.
96
+ - `inputBudget`: `summary` uses calibrated `hardInputLimit` rather than the ordinary context fill target (`activeInputBudget`). `context` selects that legacy fill target. Both are additionally capped by `(context window - safety margin - actual summary output cap) / calibration ratio`, rounded down, and never exceed the configured/model hard limit. Existing output reservations remain conservative; ordinary session planning and proactive thresholds are unchanged. This reduces avoidable fragmentation, not a guarantee of provider fit or faster processing for a larger prompt.
97
+ - `maxConcurrentSegments`: integer **1–2**, default 2. Only independent segments overlap, including their retries. Aggregates do not run until their children have completed. Use 1 for sequential execution or providers with restrictive concurrent-request limits. Rate-limit failures are not transport-retried.
98
+
99
+ For an old-path comparison set `directUpdate=false`, `inputBudget=context`, `maxConcurrentSegments=1`. All features remain behind the existing compaction/master switches. Settings are applied on session load; after upgrading the package or rebuilding a development checkout, fully restart Pi to avoid stale compiled-core modules. No schema migration is required.
100
+
101
+ Mock-provider regression tests verify fewer calls, bounded overlap, exact budget boundaries, validation, privacy, immutable provenance and JSONL rebuild. They do **not** establish real-provider wall-time gains, semantic equivalence of generated summaries, or a guaranteed completion time. See [ADR-061](ADR/061-compaction-latency.md).
102
+
81
103
  ## Transport retry policy
82
104
 
83
- Summary requests are replayed only for transport-classified failures (thrown transport errors or `stopReason: "error"` responses whose message matches network/timeout patterns). The replay policy defaults to Pi's assistant retry policy and can be tuned per deployment:
105
+ Summary requests are replayed only for transport-classified failures (thrown transport errors or `stopReason: "error"` responses whose message matches network/timeout patterns). The DS4 replay policy uses **three total attempts**, not three retries after the initial call, and can be tuned per deployment:
84
106
 
85
107
  ```json
86
108
  {
@@ -93,7 +115,7 @@ Summary requests are replayed only for transport-classified failures (thrown tra
93
115
  }
94
116
  ```
95
117
 
96
- - `compaction.transport.maxAttempts`: total attempts per segment or aggregate call, integer 1–10, default 3. With 1, no transport failure is retried.
118
+ - `compaction.transport.maxAttempts`: total attempts per direct update, segment or aggregate call, integer 1–10, default 3. With 1, no transport failure is retried.
97
119
  - `compaction.transport.baseDelayMs`: base backoff before the first replay, integer 0–60000, default 2000. The delay doubles per attempt (2000, 4000, 8000, …) and is capped at 60 s.
98
120
  - Replays use a fresh routing session per attempt; diagnostics expose only stage, failed/next attempt, max attempts, and delay.
99
121
  - Aborts (including during backoff) never trigger replay; non-transport failures are never retried; usage is summed across replayed responses.
@@ -117,4 +139,4 @@ It requests compaction at most once per session leaf. Pi's native threshold and
117
139
  /context summaries
118
140
  ```
119
141
 
120
- The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the calibrated input budget, estimated whole-source prompt size, generated segment count, aggregate-call count, and completed transport-retry count without source content. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.
142
+ The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the effective provider/model, chosen path (`direct-update` or `hierarchical`), budget mode, calibrated input budget, estimated new-source and full-update prompt sizes, logical summary calls (excluding retries), generated segment count, aggregate-call count, concurrency cap, and completed transport-retry count without source content. Monotonic wall timings cover preparation, generation (direct or parallel segments, including validation and retries), aggregation, graph preparation/persistence, and total DS4 hook time. Parallel generation time is elapsed wall time, not a sum of overlapping requests. Total time excludes Pi's subsequent canonical append or native fallback. Timings survive the in-process commit/fallback notification, but are not persisted in JSONL or reconstructed as fake durations after restart. Metadata-only `compaction.timings` is emitted at `debug` for successful and failed attempts. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.
@@ -12,7 +12,7 @@ The managed planner is synchronous, deterministic, provider-independent, and doe
12
12
  6. Merge groups linked by assistant tool calls and every matching tool result.
13
13
  7. Select the current request, labelled pin groups, and applicable allowed persistent pins as mandatory.
14
14
  8. Enforce `maxPinnedTokens`, then fit relevant allowed durable memory under `maxMemoryTokens`.
15
- 9. Walk older turns newest-first, stopping at the first group that would break the contiguous recent tail, target, or hard limit.
15
+ 9. Walk older turns newest-first, stopping at the first group that would break the contiguous recent tail, target, or hard limit. When the immediate-predecessor turn itself exceeds the recent-tail cap but still fits the target and hard budgets, it is kept verbatim (`context.rescueImmediatePredecessor`, default true) and the tail then closes.
16
16
  10. Privacy-filter and fit source-labelled historical retrieval groups under `maxRetrievedHistoryTokens`.
17
17
  11. Privacy-filter and fit hash-current project snippets under `maxProjectTokens`.
18
18
  12. Fit active allowed Pi compaction/branch summaries in the remaining summary and input budgets.
@@ -41,6 +41,8 @@ Artifact condensation occurs before atomic grouping. It preserves `toolCallId`,
41
41
 
42
42
  The retrieval engine produces independent synthetic user-role evidence groups. They are never mandatory: recent turns have priority 100, durable memory 90, retrieved history 85, project snippets 80, and active summaries 75. Each group is selected or excluded whole, carries its original Pi entry ID, and is represented as `retrieval` in the Context Manifest. If planner validation falls back, every synthetic evidence message is discarded and Pi receives its original `AgentMessage[]` unchanged.
43
43
 
44
+ Retrieval deduplicates against the entries the managed plan actually commits, not against Pi's native context: before retrieving, the runtime plans the context with mandatory supplements only, maps the committed messages back to Pi session entry IDs, and passes those IDs as the retrieval exclusion set. An entry that Pi still exposes but the planner excludes (for example a turn larger than the recent-tail cap) therefore remains retrievable and reappears as bounded `retrieval` evidence instead of being silently lost. The Context Manifest planning block records `rescuedImmediatePredecessor` and `oversizedTurnExclusions`; an oversized exclusion also emits a `context.excluded_oversized_turn` warning, and `/context explain` surfaces both counters.
45
+
44
46
  Evidence text is a JSON-quoted historical excerpt with an explicit data-only boundary. It is inserted immediately before the latest real user request, so the current task remains the final message and provider conversation order stays deterministic.
45
47
 
46
48
  ## Project snippets
@@ -1,6 +1,6 @@
1
1
  # DS4 Context Engine 0.3.4
2
2
 
3
- Status: release candidate; publication pending final validation and CI.
3
+ Status: published as stable on npm under `latest`; annotated tag `v0.3.4` points to validated release commit `5845a64`.
4
4
 
5
5
  ## Added since 0.3.3
6
6
 
@@ -24,7 +24,7 @@ Local tool wrappers must not be combined with remote/sandbox replacements. `bash
24
24
 
25
25
  ## Update and enable
26
26
 
27
- After publication, install the exact package and fully restart Pi to load the matching compiled core:
27
+ Install the exact package and fully restart Pi to load the matching compiled core:
28
28
 
29
29
  ```bash
30
30
  pi install npm:ds4-context-engine@0.3.4
@@ -60,4 +60,22 @@ Candidate validation on Node.js `26.5.1`, from a sanitized release source with a
60
60
 
61
61
  Pre-existing local `allowScripts` additions and `.serena/` are excluded from release commits and public packages.
62
62
 
63
- The initial CI run `33965143704` on candidate `4e665e3` passed Node `22.19.0`. Node `24.x` exceeded the default 5-second timeout in the existing disk-backed schema-v10 upgrade fixture (484 tests passed, one timeout). That individual correctness test now has a bounded 15-second timeout, with all assertions unchanged; no runtime/storage behavior changed. Final CI and exact registry evidence will be recorded after execution.
63
+ The initial CI run `33965143704` on candidate `4e665e3` passed Node `22.19.0`. Node `24.x` exceeded the default 5-second timeout in the existing disk-backed schema-v10 upgrade fixture (484 tests passed, one timeout). That individual correctness test now has a bounded 15-second timeout, with all assertions unchanged; no runtime/storage behavior changed.
64
+
65
+ Final validation-only CI run [`33965385188`](https://github.com/Alucard24/ds4-context-engine/actions/runs/33965385188) on `5845a64` passed both Node `22.19.0` and `24.x`, including the full test suite and clean-consumer package checks.
66
+
67
+ ## Registry verification and release
68
+
69
+ Published manually as `alucard_24`, in dependency order, from reviewed tarballs built in a clean committed worktree:
70
+
71
+ - `ds4-context-core@0.3.4`: shasum `c1f1f6c5eec392e43f8b6a0917581e5d02141c84`;
72
+ - `ds4-context-reference-adapter@0.3.4`: shasum `61402dc240af5963f4cbbf5f69752fe9a451170d`;
73
+ - `ds4-context-engine@0.3.4`: shasum `99b9ba52c2bc628f5fb197ea5356459963994b89`.
74
+
75
+ `npm run registry:check -- 0.3.4` passed: fresh exact-version installation, matching adapter/core dependencies, public core exports, compiled reference conformance, packaged quality corpus, storage CLI usage probe and isolated offline Pi extension startup. Registry SHA-1 and integrity values match the local tarballs for all three packages.
76
+
77
+ Verified dist-tags for all three: `latest=0.3.4`, `beta=0.3.0-beta.3`, `alpha=0.3.0-alpha.5`, `rc=0.2.0-rc.1`.
78
+
79
+ Annotated tag `v0.3.4` targets `5845a64`, the tested/published source. This post-publication evidence update is documentation-only.
80
+
81
+ GitHub Release: https://github.com/Alucard24/ds4-context-engine/releases/tag/v0.3.4
@@ -0,0 +1,86 @@
1
+ # DS4 Context Engine 0.3.5
2
+
3
+ Status: published as stable on npm under `latest`; annotated tag `v0.3.5` points to validated release commit `8b1d5f4`.
4
+
5
+ ## Added since 0.3.4
6
+
7
+ Bounded compaction optimizations, enabled by default when DS4 custom compaction is enabled:
8
+
9
+ - **`compaction.directUpdate=true`**: one validated previous-summary plus new-source request when the complete sanitized prompt fits. The immutable `task-state` node retains predecessor edges, transitive source IDs and a source hash bound to both inputs. Oversized updates retain hierarchical segmentation/aggregation; no synthetic segment or source truncation is introduced.
10
+ - **`compaction.inputBudget="summary"`**: use the calibrated hard input limit rather than the ordinary context fill target, additionally capped by model context minus safety margin and actual summary output headroom. Configured/model hard limits and calibration remain authoritative; ordinary context planning is unchanged.
11
+ - **`compaction.maxConcurrentSegments=2`** (integer 1–2): overlap only independent segment requests. Node identities, sources, graph edges and usage follow source order, never completion order. Failure or cancellation stops scheduling, aborts siblings and drains every started worker before Pi fallback; no partial graph is installed. Aggregation stays ordered and bounded.
12
+ - **Metadata-only diagnostics**: `/context compaction` reports path, effective provider/model, full-update prompt size, logical calls, concurrency, retries and monotonic wall timings for preparation, generation, aggregation, persistence and total DS4 hook duration. Timings are process-local, exclude native fallback, and are not persisted in canonical JSONL.
13
+
14
+ Transport documentation now explicitly distinguishes **three total attempts** from three retries; runtime transport policy is unchanged.
15
+
16
+ See [compaction controls](../COMPACTION.md#latency-controls) and [ADR-061](../ADR/061-compaction-latency.md).
17
+
18
+ ## Compatibility and limitations
19
+
20
+ Pi cut points, atomic tool exchanges, summary validation and bounded exact-value repair, privacy classification, immutable provenance and JSONL rebuild remain in place. Schema-v2 compaction metadata reuses the existing `task-state` kind. Pi JSONL remains canonical and append-only; SQLite remains disposable/rebuildable at schema `15`.
21
+
22
+ Unchanged contracts: `ds4-context-config-v1`, `runtime-adapter-v1`, `ds4-context-persistence-tool-v1`, `ds4-context-persistence-result-v1`. The new configuration keys are additive; absent keys use the new defaults. The five optional portable tools/features introduced in 0.3.4 remain default-off. Pi stays pinned to `0.84.3`; Node.js remains `>=22.19.0`. No dependency upgrade, live database maintenance or Pi upgrade is part of this release.
23
+
24
+ Mock-provider tests establish call counts, bounded overlap and deterministic safety properties, not real-provider speedups, semantic equivalence or a guaranteed compaction duration. A larger prompt may take longer to process. Rate-limit errors are not transport-retried; providers with restrictive concurrency should use `maxConcurrentSegments=1`. Cancellation is cooperative: accepted work may still be billed and providers ignoring abort may delay settlement. Linux validation does not establish Windows execution correctness.
25
+
26
+ ## Update and configure
27
+
28
+ Install the exact coordinated package and **fully restart Pi** to load its matching compiled core:
29
+
30
+ ```bash
31
+ pi install npm:ds4-context-engine@0.3.5
32
+ ```
33
+
34
+ No extra opt-in is needed for the three new defaults when DS4 and custom compaction are enabled. Inspect configuration and subsequent compaction diagnostics with:
35
+
36
+ ```text
37
+ /context config show
38
+ /context compaction
39
+ ```
40
+
41
+ For a legacy-path comparison in a trusted project:
42
+
43
+ ```text
44
+ /context config set compaction.directUpdate false
45
+ /context config set compaction.inputBudget context
46
+ /context config set compaction.maxConcurrentSegments 1
47
+ /reload
48
+ ```
49
+
50
+ Use `true`, `summary`, and `2` to restore the new defaults. Add `--global` to each `set` for agent-directory configuration; trusted project settings take precedence. Disabling DS4 custom compaction still delegates to Pi.
51
+
52
+ ## Package policy and validation
53
+
54
+ All three packages use `0.3.5`: `ds4-context-core`, `ds4-context-reference-adapter`, `ds4-context-engine`. Both adapters depend exactly on `ds4-context-core@0.3.5`. Publication is manual in that order under npm's stable `latest` tag. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
55
+
56
+ Candidate validation on Node.js `26.5.1`, from a sanitized release source with a fresh `npm ci`:
57
+
58
+ - `npm run check`: **80 files / 508 tests passed**, including exact full-prompt budget boundaries, direct-update evidence/privacy, immutable graph rebuild, out-of-order segments, abort/failure drainage and transport retry cancellation.
59
+ - `npm run quality:compare`: passed; frozen-corpus candidate score `0.9875` versus baseline `0.808156`. This is a planner corpus, not a real-provider summary-quality comparison.
60
+ - `npm run schema:context-persistence`: passed, `1266` bytes / `317` estimated tokens (limits `1500` / `320`).
61
+ - `npm run latency:check` against freshly installed exact `ds4-context-core@0.1.2`: passed; disabled-planning p95 ratio `0.903863`, maximum `1.1`. This is not a compaction/provider latency measurement.
62
+ - `npm run pack:check`: passed in a clean consumer, core **235 files**, reference adapter **7**, engine **87**, including isolated offline Pi extension startup and registry scenarios.
63
+ - All three `npm pack --dry-run --json` inventories and `git diff --check` passed. Tarball inventories exclude sessions, databases, credentials, test state and `.serena/`.
64
+ - Manifests, exact core dependencies, lockfile and runtime version constants are synchronized. The lockfile contains only coordinated version changes; no dependency upgrade was performed.
65
+
66
+ Pre-existing local `allowScripts` additions and `.serena/` are excluded from release commits and public packages.
67
+
68
+ The final clean committed worktree repeated fresh `npm ci`, all **508 tests**, clean-consumer package verification and tarball inspection successfully. Validation-only CI run [`33970061602`](https://github.com/Alucard24/ds4-context-engine/actions/runs/33970061602) on `8b1d5f4` passed both Node `22.19.0` and `24.x`, including the full suite and package checks.
69
+
70
+ ## Registry verification and release
71
+
72
+ Published manually as `alucard_24`, in dependency order, from reviewed tarballs built in the clean committed worktree:
73
+
74
+ - `ds4-context-core@0.3.5`: shasum `71105c666a8c88f1abfe5ba3efefe63fc32bba74`;
75
+ - `ds4-context-reference-adapter@0.3.5`: shasum `8eca29472ea4721606d947818eba0aa53a6c6f63`;
76
+ - `ds4-context-engine@0.3.5`: shasum `304b61c21e28f89e82edbee39880210c2c9dc868`.
77
+
78
+ `npm run registry:check -- 0.3.5` passed: fresh exact-version installation, matching adapter/core dependencies, public core exports, compiled reference conformance, packaged quality corpus, storage CLI usage probe and isolated offline Pi extension startup. Registry SHA-1 and SHA-512 integrity values match the local tarballs for all three packages.
79
+
80
+ The immediate first post-publication install returned `ETARGET` for the engine. Once exact-version registry metadata was available, verification was repeated with npm `prefer-online` and passed; no package was republished or tag moved.
81
+
82
+ Verified dist-tags for all three: `latest=0.3.5`, `beta=0.3.0-beta.3`, `alpha=0.3.0-alpha.5`, `rc=0.2.0-rc.1`.
83
+
84
+ Annotated tag `v0.3.5` targets `8b1d5f4`, the tested/published source. This post-publication evidence update is documentation-only.
85
+
86
+ GitHub Release: https://github.com/Alucard24/ds4-context-engine/releases/tag/v0.3.5
@@ -0,0 +1,53 @@
1
+ # DS4 Context Engine 0.3.6
2
+
3
+ Status: published release on 2026-09-05; tag `v0.3.6`.
4
+
5
+ This coordinated release fixes a managed-context selection defect: a single oversized atomic conversation turn could cause the planner to discard turns that are still essential to the current request, and its own retrieval second pass could not recover them.
6
+
7
+ ## Fixed
8
+
9
+ - **Immediate-predecessor rescue**: when the turn immediately preceding the current request exceeds the recent-tail cap but still fits the active target and hard budgets, the planner keeps it verbatim instead of closing the entire recent tail at that group. The tail closes after the rescue, so older turns keep their prior bounded behavior. Controlled by `context.rescueImmediatePredecessor` (default `true`).
10
+ - **Retrieval second chance**: retrieval now deduplicates against the entries the managed plan actually commits, not against Pi's native context. Before the retrieval pass, the runtime plans the context with mandatory supplements only, maps the committed messages back to Pi session entry IDs, and passes those IDs as the retrieval exclusion set. Entries the planner excludes (for example an oversized turn) can therefore reappear as bounded, source-labelled `retrieval` evidence instead of being silently lost.
11
+ - **Diagnostics**: a `context.excluded_oversized_turn` warning reports how many turn groups at or above the recent-tail cap were excluded, whether the immediate predecessor was rescued, and the active limits. The Context Manifest planning block records optional `rescuedImmediatePredecessor` and `oversizedTurnExclusions` counters, and `/context explain` surfaces both.
12
+
13
+ No validation, privacy, provenance, cancellation, fallback, canonical-history, or bounded-retention behavior was weakened. The runtime retry policy, compaction pipeline, and storage formats are unchanged.
14
+
15
+ ## Compatibility and persistence
16
+
17
+ The context manifest adds only optional metadata fields. `ds4-context-config-v1`, `runtime-adapter-v1`, `ds4-context-persistence-tool-v1`, `ds4-context-persistence-result-v1`, SQLite schema 15, and migration checksums are unchanged.
18
+
19
+ ## Package/version policy
20
+
21
+ The coordinated version is `0.3.6` for:
22
+
23
+ ```text
24
+ ds4-context-core
25
+ ds4-context-reference-adapter
26
+ ds4-context-engine
27
+ ```
28
+
29
+ Both adapters depend exactly on `ds4-context-core@0.3.6`. The packages were published manually under npm `latest`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
30
+
31
+ ## Validation evidence
32
+
33
+ Local candidate verification on Node.js `26.5.1`:
34
+
35
+ - `npm run check`: 81 files and 518 tests passed (including planner rescue cases, second-chance retrieval integration, and entry-id mapping).
36
+ - `npm run quality:compare`: candidate quality `0.9875` versus baseline `0.808156`.
37
+ - `npm run schema:context-persistence`: 1,266 bytes and 317 estimated tokens; below the 1,500 absolute and 320 relative limits.
38
+ - `npm run latency:check -- <exact ds4-context-core@0.1.2>`: passed with ratio `1.082064`, at or below `1.10`.
39
+ - `npm run pack:check`: verified core (235 files), reference adapter (7 files), and Pi adapter (87 files) in a clean consumer.
40
+ - `npm pack --dry-run --json` for all three packages: passed with the same bounded inventories.
41
+ - `git diff --check`: passed.
42
+ - Protected CI, compatibility golden, Pi fixture, migration, canonical Pin/Memory, and persistence-tool contract files: unchanged except the intentional `compatibility-0.2.0.json` default addition.
43
+
44
+ - `npm run registry:check -- 0.3.6`: passed against all three exact published versions; `latest` resolves to `0.3.6` for every package.
45
+
46
+ Exact registry verification passed before the annotated tag and GitHub release were created.
47
+
48
+ ## Documentation
49
+
50
+ - [`../CONTEXT_PLANNER.md`](../CONTEXT_PLANNER.md)
51
+ - [`../COMPACTION.md`](../COMPACTION.md)
52
+ - [`../RELEASING.md`](../RELEASING.md)
53
+ - [`0.3.5.md`](0.3.5.md)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ds4-context-engine",
3
- "version": "0.3.4",
3
+ "version": "0.3.6",
4
4
  "description": "Non-destructive, provider-independent context management for Pi.",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -62,7 +62,7 @@
62
62
  ]
63
63
  },
64
64
  "dependencies": {
65
- "ds4-context-core": "0.3.4"
65
+ "ds4-context-core": "0.3.6"
66
66
  },
67
67
  "peerDependencies": {
68
68
  "@earendil-works/pi-ai": "0.84.3",
@@ -403,6 +403,12 @@ function formatExplain(diagnostics: RuntimeDiagnostics): string {
403
403
  `Message target: ${count(planning.messageTargetTokens)}`,
404
404
  `Message hard limit: ${count(planning.messageHardLimitTokens)}`,
405
405
  `Recent-tail limit: ${count(planning.recentTailTokenLimit)}`,
406
+ ...(planning.rescuedImmediatePredecessor
407
+ ? ["Rescued predecessor: yes (immediate previous turn kept beyond the recent-tail cap within the input budget)"]
408
+ : []),
409
+ ...(planning.oversizedTurnExclusions
410
+ ? [`Oversized turn excl: ${count(planning.oversizedTurnExclusions)} (turn group(s) at/above the recent-tail cap; recovered by retrieval only if it fits)`]
411
+ : []),
406
412
  `Selected groups: ${count(planning.selectedGroupCount)}`,
407
413
  `Excluded groups: ${count(planning.excludedGroupCount)}`,
408
414
  `Duration: ${planning.durationMs === undefined ? "n/a" : `${planning.durationMs.toFixed(1)} ms`}`,
@@ -421,6 +427,7 @@ function formatSummaryGraph(graph: SummaryGraphDiagnostics): string {
421
427
  `Nodes: ${count(graph.totalNodes)}`,
422
428
  `Committed/prepared/failed: ${count(graph.committedNodes)} / ${count(graph.preparedNodes)} / ${count(graph.failedNodes)}`,
423
429
  `Segment/aggregate/branch: ${count(graph.segmentNodes)} / ${count(graph.aggregateNodes)} / ${count(graph.branchNodes)}`,
430
+ `Direct update nodes: ${count(graph.taskStateNodes)}`,
424
431
  `Maximum graph level: ${count(graph.maxGraphLevel)}`,
425
432
  `Active summary: ${graph.activeSummaryId ?? "none on current branch"}`,
426
433
  `Active path nodes: ${count(graph.activePathIds.length)}`,
@@ -845,11 +852,22 @@ function formatCompaction(diagnostics: RuntimeDiagnostics, preview: boolean): st
845
852
  `Last trigger: ${compaction.trigger ?? "n/a"}`,
846
853
  `Summary ID: ${compaction.summaryId ?? "n/a"}`,
847
854
  `Source entries: ${count(compaction.sourceEntries)}`,
855
+ `Compaction model: ${compaction.provider && compaction.model ? `${compaction.provider}/${compaction.model}` : "n/a"}`,
856
+ `Chosen path: ${compaction.path ?? "n/a"}`,
857
+ `Input budget mode: ${compaction.inputBudgetMode ?? "n/a"}`,
848
858
  `Input budget: ${count(compaction.inputBudgetTokens)}`,
849
859
  `Whole-source prompt: ${count(compaction.sourcePromptTokens)}`,
860
+ `Direct-update prompt: ${count(compaction.directPromptTokens)}`,
861
+ `Segment concurrency cap: ${count(compaction.maxConcurrentSegments)}`,
862
+ `Summary calls: ${count(compaction.summaryCalls)}`,
850
863
  `Generated segments: ${count(compaction.segmentCount)}`,
851
864
  `Aggregate calls: ${count(compaction.aggregateCalls)}`,
852
865
  `Transport retries: ${count(compaction.transportRetries)}`,
866
+ ...(compaction.timings ? [
867
+ `DS4 hook time (ms): ${compaction.timings.totalMs.toFixed(1)}`,
868
+ `Prepare/generate (ms): ${compaction.timings.preparationMs.toFixed(1)} / ${compaction.timings.generationMs.toFixed(1)}`,
869
+ `Aggregate/persist (ms): ${compaction.timings.aggregationMs.toFixed(1)} / ${compaction.timings.persistenceMs.toFixed(1)}`,
870
+ ] : []),
853
871
  `Validation: ${compaction.validationStatus ?? "n/a"}`,
854
872
  `First kept entry: ${compaction.firstKeptEntryId ?? "n/a"}`,
855
873
  `Tokens before: ${count(compaction.tokensBefore)}`,
@@ -97,7 +97,7 @@ import {
97
97
  type PinScope,
98
98
  type ProjectMemorySource,
99
99
  } from "ds4-context-core/memory/memory-types";
100
- import { planManagedContext, type SupplementalContextMessage } from "ds4-context-core/planner/context-planner";
100
+ import { planManagedContext, type ManagedContextPlan, type SupplementalContextMessage } from "ds4-context-core/planner/context-planner";
101
101
  import {
102
102
  disabledContextQualityDiagnostics,
103
103
  evaluateManifestQuality,
@@ -151,6 +151,7 @@ import {
151
151
  buildPiObserverManifest,
152
152
  findExactPiMessageSourceIds,
153
153
  findPiPinnedMessageIndices,
154
+ findPiSourceEntryIds,
154
155
  } from "../pi-adapter/context-observer.ts";
155
156
  import { projectSessionFileMutations } from "../pi-adapter/memory-adapter.ts";
156
157
  import { ProjectMemorySynchronizer } from "../pi-adapter/project-memory-sync.ts";
@@ -973,10 +974,43 @@ export class Ds4ContextRuntime {
973
974
  memoryTokens: 0,
974
975
  };
975
976
  if (this.memoryManager) this.lastMemory = this.memoryManager.diagnostics();
977
+ const fixedTokens = baseline.composition.systemTokens + baseline.composition.toolTokens;
978
+ const pinnedMessageIndices = findPiPinnedMessageIndices(effectiveEvent, ctx);
979
+ const retrievalEnabled = this.config.retrieval.exact
980
+ || this.config.retrieval.fts
981
+ || this.config.retrieval.semantic;
982
+ const retrievalActiveContextEntryIds = retrievalEnabled
983
+ ? this.plannedContextEntryIds(ctx, planManagedContext({
984
+ messages: effectiveEvent.messages,
985
+ fixedTokens,
986
+ budget,
987
+ config: effectiveContextConfig,
988
+ pinnedMessageIndices,
989
+ supplementalMessages: [
990
+ ...memorySelection.pins.map((evidence) => ({
991
+ id: `pin:${evidence.item.id}`,
992
+ message: evidence.message,
993
+ kind: "pin" as const,
994
+ sourceIds: [evidence.item.id],
995
+ score: 950,
996
+ reason: evidence.reason,
997
+ })),
998
+ ...memorySelection.memories.map((evidence) => ({
999
+ id: `memory:${evidence.item.id}`,
1000
+ message: evidence.message,
1001
+ kind: "memory" as const,
1002
+ sourceIds: [evidence.item.id],
1003
+ score: 90 + Math.min(0.999999, Math.max(0, evidence.score) / 1_000),
1004
+ reason: evidence.reason,
1005
+ })),
1006
+ ],
1007
+ }))
1008
+ : undefined;
976
1009
  const retrieval = this.retrieveHistory(
977
1010
  effectiveEvent,
978
1011
  ctx,
979
1012
  effectiveContextConfig.maxRetrievedHistoryTokens,
1013
+ retrievalActiveContextEntryIds,
980
1014
  );
981
1015
  const project = this.retrieveProjectKnowledge(
982
1016
  requestText,
@@ -1110,10 +1144,10 @@ export class Ds4ContextRuntime {
1110
1144
  });
1111
1145
  const plan = planManagedContext({
1112
1146
  messages: effectiveEvent.messages,
1113
- fixedTokens: baseline.composition.systemTokens + baseline.composition.toolTokens,
1147
+ fixedTokens,
1114
1148
  budget,
1115
1149
  config: effectiveContextConfig,
1116
- pinnedMessageIndices: findPiPinnedMessageIndices(effectiveEvent, ctx),
1150
+ pinnedMessageIndices,
1117
1151
  supplementalMessages: rankedSupplementalMessages,
1118
1152
  ...(ranking.diagnostics.status === "active"
1119
1153
  ? { supplementalSelectionOrder: ranking.ranked.map((candidate) => candidate.id) }
@@ -1175,6 +1209,17 @@ export class Ds4ContextRuntime {
1175
1209
  selected: plannerSelectedProject,
1176
1210
  };
1177
1211
  plan.planning.durationMs = Math.max(0, this.now() - planningStartedAt);
1212
+ const oversizedTurnExclusions = plan.planning.oversizedTurnExclusions ?? 0;
1213
+ if (plan.mode === "managed" && oversizedTurnExclusions > 0) {
1214
+ this.logger.warn("context.excluded_oversized_turn", {
1215
+ oversizedTurnCount: oversizedTurnExclusions,
1216
+ rescuedImmediatePredecessor: plan.planning.rescuedImmediatePredecessor ?? false,
1217
+ recentTailTokenLimit: plan.planning.recentTailTokenLimit,
1218
+ messageTargetTokens: plan.planning.messageTargetTokens,
1219
+ selectedGroupCount: plan.planning.selectedGroupCount,
1220
+ excludedGroupCount: plan.planning.excludedGroupCount,
1221
+ });
1222
+ }
1178
1223
  const plannedEvent: ContextEvent = { type: "context", messages: plan.messages };
1179
1224
  const selectedArtifactReferences = plan.mode === "managed"
1180
1225
  ? plan.selected.flatMap((metadata) => {
@@ -2758,10 +2803,36 @@ export class Ds4ContextRuntime {
2758
2803
  }
2759
2804
  }
2760
2805
 
2806
+ private plannedContextEntryIds(
2807
+ ctx: ExtensionContext,
2808
+ plan: ManagedContextPlan<ContextEvent["messages"][number]>,
2809
+ ): Set<string> {
2810
+ if (plan.mode === "fallback") {
2811
+ return new Set(ctx.sessionManager.buildContextEntries().map((entry) => entry.id));
2812
+ }
2813
+ const syntheticIndices = new Set(
2814
+ [...plan.selected, ...plan.excluded]
2815
+ .filter((metadata) =>
2816
+ metadata.kind === "memory"
2817
+ || (metadata.kind === "pin" && metadata.sourceId !== undefined)
2818
+ )
2819
+ .map((metadata) => metadata.originalIndex),
2820
+ );
2821
+ const selectedIndices = new Set(plan.selected.map((metadata) => metadata.originalIndex));
2822
+ const sources = findPiSourceEntryIds(plan.originalMessages, ctx, syntheticIndices);
2823
+ const entryIds = new Set<string>();
2824
+ for (const index of selectedIndices) {
2825
+ const sourceId = sources[index];
2826
+ if (sourceId) entryIds.add(sourceId);
2827
+ }
2828
+ return entryIds;
2829
+ }
2830
+
2761
2831
  private retrieveHistory(
2762
2832
  event: ContextEvent,
2763
2833
  ctx: ExtensionContext,
2764
2834
  maxTokens = this.config.context.maxRetrievedHistoryTokens,
2835
+ activeContextEntryIds?: ReadonlySet<string>,
2765
2836
  ): RetrievalDiagnostics {
2766
2837
  const requestText = currentRequestText(event.messages);
2767
2838
  if (!this.retrievalEngine || !this.session?.sessionFile || !requestText) {
@@ -2775,7 +2846,8 @@ export class Ds4ContextRuntime {
2775
2846
  sessionId: this.session.sessionId,
2776
2847
  requestText,
2777
2848
  activeBranchEntryIds: new Set(ctx.sessionManager.getBranch().map((entry) => entry.id)),
2778
- activeContextEntryIds: new Set(ctx.sessionManager.buildContextEntries().map((entry) => entry.id)),
2849
+ activeContextEntryIds: activeContextEntryIds
2850
+ ?? new Set(ctx.sessionManager.buildContextEntries().map((entry) => entry.id)),
2779
2851
  exact: this.config.retrieval.exact,
2780
2852
  fts: this.config.retrieval.fts,
2781
2853
  semantic: this.config.retrieval.semantic,
@@ -1,3 +1,6 @@
1
+ import { performance } from "node:perf_hooks";
2
+ import { compactionInputBudget } from "ds4-context-core/compaction/input-budget";
3
+ import { mapCompactionSegments } from "./compaction-workers.ts";
1
4
  import type {
2
5
  CompactionResult,
3
6
  ExtensionContext,
@@ -47,6 +50,7 @@ import {
47
50
  buildAggregateSummaryPrompt,
48
51
  buildSummaryPrompt,
49
52
  computeAggregateSourceHash,
53
+ computeUpdateSourceHash,
50
54
  type SummaryValidationStatus,
51
55
  } from "ds4-context-core/compaction/summary-contract";
52
56
 
@@ -63,6 +67,14 @@ export type CompactionPhase =
63
67
  | "pi-default"
64
68
  | "failed";
65
69
 
70
+ export interface CompactionTimings {
71
+ preparationMs: number;
72
+ generationMs: number;
73
+ aggregationMs: number;
74
+ persistenceMs: number;
75
+ totalMs: number;
76
+ }
77
+
66
78
  export interface CompactionDiagnostics {
67
79
  enabled: boolean;
68
80
  validate: boolean;
@@ -83,6 +95,14 @@ export interface CompactionDiagnostics {
83
95
  segmentCount?: number;
84
96
  aggregateCalls?: number;
85
97
  transportRetries?: number;
98
+ path?: "direct-update" | "hierarchical";
99
+ provider?: string;
100
+ model?: string;
101
+ inputBudgetMode?: "summary" | "context";
102
+ directPromptTokens?: number;
103
+ maxConcurrentSegments?: number;
104
+ summaryCalls?: number;
105
+ timings?: CompactionTimings;
86
106
  contextTokens?: number;
87
107
  softLimitTokens?: number;
88
108
  proactiveThresholdTokens?: number;
@@ -110,6 +130,7 @@ export interface SummaryGraphDiagnostics {
110
130
  segmentNodes: number;
111
131
  aggregateNodes: number;
112
132
  branchNodes: number;
133
+ taskStateNodes: number;
113
134
  maxGraphLevel: number;
114
135
  rootSummaryIds: string[];
115
136
  activeSummaryId?: string;
@@ -212,6 +233,7 @@ export class CompactionCoordinator {
212
233
  event: SessionBeforeCompactEvent,
213
234
  ctx: ExtensionContext,
214
235
  ): Promise<{ compaction?: CompactionResult } | undefined> {
236
+ const startedAt = performance.now();
215
237
  const config = this.dependencies.config;
216
238
  const model = this.resolveCompactionModel(ctx) ?? ctx.model;
217
239
  if (!config.compaction.enabled || !config.enabled || !model || config.context.maxSummaryTokens <= 0) {
@@ -219,6 +241,25 @@ export class CompactionCoordinator {
219
241
  }
220
242
 
221
243
  const requestedAt = this.dependencies.now();
244
+ const timings: CompactionTimings = {
245
+ preparationMs: performance.now() - startedAt,
246
+ generationMs: 0,
247
+ aggregationMs: 0,
248
+ persistenceMs: 0,
249
+ totalMs: 0,
250
+ };
251
+ const measure = async <T>(
252
+ phase: Exclude<keyof CompactionTimings, "totalMs">,
253
+ run: () => T | Promise<T>,
254
+ ): Promise<T> => {
255
+ const start = performance.now();
256
+ try {
257
+ return await run();
258
+ } finally {
259
+ timings[phase] += performance.now() - start;
260
+ }
261
+ };
262
+ const maxConcurrentSegments = config.compaction.maxConcurrentSegments ?? 2;
222
263
  const trigger: CompactionTrigger = this.proactiveRequested ? "proactive" : event.reason;
223
264
  this.state = {
224
265
  phase: "generating",
@@ -227,28 +268,43 @@ export class CompactionCoordinator {
227
268
  firstKeptEntryId: event.preparation.firstKeptEntryId,
228
269
  tokensBefore: event.preparation.tokensBefore,
229
270
  transportRetries: 0,
271
+ aggregateCalls: 0,
272
+ summaryCalls: 0,
273
+ provider: model.provider,
274
+ model: model.id,
275
+ inputBudgetMode: config.compaction.inputBudget ?? "summary",
276
+ maxConcurrentSegments,
277
+ timings,
230
278
  };
231
279
 
232
280
  try {
233
- this.dependencies.syncSessionIndex(ctx);
234
- const source = prepareCompactionSource(event);
235
- const inputBudgetTokens = this.inputBudgetTokens(model);
236
- if (inputBudgetTokens <= 0) throw new Error("Active model has no safe compaction input budget");
237
- const wholePlan = this.buildSegmentPlan(source, event, model.provider);
238
- this.state = {
239
- ...this.state,
240
- sourceEntries: source.sourceEntryIds.length,
241
- inputBudgetTokens,
242
- sourcePromptTokens: wholePlan.promptTokens,
243
- };
244
- const segmentPlans = this.partitionSegmentPlans(
245
- source,
246
- wholePlan,
247
- event,
248
- model.provider,
249
- inputBudgetTokens,
250
- );
251
- this.state.segmentCount = segmentPlans.length;
281
+ const { source, inputBudgetTokens, wholePlan, directPlan, segmentPlans } = await measure("preparationMs", () => {
282
+ if (event.signal.aborted) throw new Error("Compaction summary generation aborted");
283
+ this.dependencies.syncSessionIndex(ctx);
284
+ const source = prepareCompactionSource(event);
285
+ const inputBudgetTokens = this.inputBudgetTokens(model);
286
+ if (inputBudgetTokens <= 0) throw new Error("Active model has no safe compaction input budget");
287
+ const wholePlan = this.buildSegmentPlan(source, event, model.provider);
288
+ const update = (config.compaction.directUpdate ?? true) && source.previousSummary
289
+ ? this.buildSegmentPlan({
290
+ ...source,
291
+ segmentReadFiles: source.readFiles,
292
+ segmentModifiedFiles: source.modifiedFiles,
293
+ }, event, model.provider, source.previousSummary)
294
+ : undefined;
295
+ const directPlan = update && update.promptTokens <= inputBudgetTokens ? update : undefined;
296
+ this.state = {
297
+ ...this.state,
298
+ path: directPlan ? "direct-update" : "hierarchical",
299
+ sourceEntries: source.sourceEntryIds.length,
300
+ inputBudgetTokens,
301
+ sourcePromptTokens: wholePlan.promptTokens,
302
+ ...(update ? { directPromptTokens: update.promptTokens } : {}),
303
+ };
304
+ const segmentPlans = directPlan ? [] : this.partitionSegmentPlans(source, wholePlan, event, model.provider, inputBudgetTokens);
305
+ this.state.segmentCount = segmentPlans.length;
306
+ return { source, inputBudgetTokens, wholePlan, directPlan, segmentPlans };
307
+ });
252
308
 
253
309
  const usedIds = new Set(this.graphRecords.keys());
254
310
  if (source.previousNode) usedIds.add(source.previousNode.id);
@@ -269,53 +325,59 @@ export class CompactionCoordinator {
269
325
  };
270
326
 
271
327
  const usages: GeneratedSummary["usage"][] = [];
272
- const segmentNodes: EmbeddedSummaryNode[] = [];
273
- for (const plan of segmentPlans) {
274
- const generated = await this.generateSummary({
275
- stage: "segment",
328
+ // Allocate in source order before concurrent requests; completion order must
329
+ // never determine IDs, graph edges, source order or usage accumulation.
330
+ const plans = directPlan ? [directPlan] : segmentPlans;
331
+ const ids = plans.map(() => nextId());
332
+ const createdNodes: EmbeddedSummaryNode[] = [];
333
+ let previousNode = source.previousNode;
334
+ if (!previousNode && source.previousSummary) {
335
+ previousNode = importUntrackedPreviousSummary({
336
+ id: nextId(),
337
+ content: source.previousSummary,
338
+ createdAt: this.dependencies.now(),
339
+ provider: model.provider,
340
+ model: model.id,
341
+ });
342
+ createdNodes.push(previousNode);
343
+ }
344
+ const results = await measure("generationMs", () => mapCompactionSegments(
345
+ plans, maxConcurrentSegments, event.signal, (plan, _index, signal) => this.generateSummary({
346
+ stage: directPlan ? "update" : "segment",
276
347
  prompt: plan.prompt,
277
348
  validationSource: plan.validationSource,
278
349
  readFiles: plan.readFiles,
279
350
  modifiedFiles: plan.modifiedFiles,
280
- event,
351
+ event: { ...event, signal },
281
352
  ctx,
282
353
  model,
283
- thinking: this.dependencies.config.compaction.summary?.thinking,
284
- });
354
+ thinking: config.compaction.summary?.thinking,
355
+ }),
356
+ ));
357
+ const generatedNodes: EmbeddedSummaryNode[] = results.map((generated, index) => {
358
+ const plan = plans[index] as SegmentGenerationPlan;
359
+ const predecessor = directPlan ? previousNode : undefined;
285
360
  usages.push(generated.usage);
286
- segmentNodes.push({
287
- id: nextId(),
288
- kind: "segment",
361
+ return {
362
+ id: ids[index] as string,
363
+ kind: predecessor ? "task-state" : "segment",
289
364
  content: classifiedSummary(generated.content, plan.classification),
290
- sourceHash: plan.source.sourceHash,
291
- sourceEntryIds: [...plan.source.sourceEntryIds],
292
- childSummaryIds: [],
293
- graphLevel: 0,
365
+ sourceHash: predecessor ? computeUpdateSourceHash(source.sourceHash, predecessor) : plan.source.sourceHash,
366
+ sourceEntryIds: unique([...(predecessor?.sourceEntryIds ?? []), ...plan.source.sourceEntryIds]),
367
+ childSummaryIds: predecessor ? [predecessor.id] : [],
368
+ graphLevel: predecessor ? predecessor.graphLevel + 1 : 0,
294
369
  createdAt: this.dependencies.now(),
295
370
  validationStatus: generated.validation.status,
296
371
  validationIssueCodes: unique(generated.validation.issues.map((issue) => issue.code)),
297
372
  provider: model.provider,
298
373
  model: model.id,
299
- });
300
- }
301
- const segmentId = segmentNodes[0]?.id;
302
- if (!segmentId) throw new Error("Compaction fan-out produced no segment summary node");
303
-
304
- const createdNodes: EmbeddedSummaryNode[] = [];
305
- let previousNode = source.previousNode;
306
- if (!previousNode && source.previousSummary) {
307
- previousNode = importUntrackedPreviousSummary({
308
- id: nextId(),
309
- content: source.previousSummary,
310
- createdAt: this.dependencies.now(),
311
- provider: model.provider,
312
- model: model.id,
313
- });
314
- createdNodes.push(previousNode);
315
- }
316
- createdNodes.push(...segmentNodes);
317
- const roots = [...(previousNode ? [previousNode] : []), ...segmentNodes];
318
- const aggregated = await this.aggregateSummaryNodes({
374
+ };
375
+ });
376
+ const segmentId = generatedNodes[0]?.id;
377
+ if (!segmentId) throw new Error("Compaction produced no summary node");
378
+ createdNodes.push(...generatedNodes);
379
+ const roots = [...(!directPlan && previousNode ? [previousNode] : []), ...generatedNodes];
380
+ const aggregated = await measure("aggregationMs", () => this.aggregateSummaryNodes({
319
381
  roots,
320
382
  event,
321
383
  ctx,
@@ -328,29 +390,34 @@ export class CompactionCoordinator {
328
390
  createdNodes,
329
391
  usages,
330
392
  modelObject: model,
331
- });
393
+ }));
332
394
  const activeNode = aggregated.activeNode;
333
395
  const embeddedNodes = createdNodes.filter((node) => node.id !== activeNode.id);
334
396
 
335
- const records = embeddedNodes.map((node) => createSummaryRecord({
336
- sessionId: this.dependencies.sessionId,
337
- node,
338
- boundary,
339
- segmentSummaryId: node.kind === "segment" ? node.id : segmentId,
340
- lifecycleStatus: "prepared",
341
- }));
342
- records.push(createSummaryRecord({
343
- sessionId: this.dependencies.sessionId,
344
- node: activeNode,
345
- boundary,
346
- segmentSummaryId: segmentId,
347
- lifecycleStatus: "prepared",
348
- embeddedNodes,
349
- }));
350
- if (this.dependencies.persisted) this.dependencies.database?.summaries.saveGraph(records);
351
- for (const record of records) this.graphRecords.set(record.id, record);
352
- this.pendingSummaryIds = records.map((record) => record.id);
397
+ const records = await measure("persistenceMs", () => {
398
+ if (event.signal.aborted) throw new Error("Compaction summary generation aborted");
399
+ const records = embeddedNodes.map((node) => createSummaryRecord({
400
+ sessionId: this.dependencies.sessionId,
401
+ node,
402
+ boundary,
403
+ segmentSummaryId: node.kind === "segment" ? node.id : segmentId,
404
+ lifecycleStatus: "prepared",
405
+ }));
406
+ records.push(createSummaryRecord({
407
+ sessionId: this.dependencies.sessionId,
408
+ node: activeNode,
409
+ boundary,
410
+ segmentSummaryId: segmentId,
411
+ lifecycleStatus: "prepared",
412
+ embeddedNodes,
413
+ }));
414
+ if (this.dependencies.persisted) this.dependencies.database?.summaries.saveGraph(records);
415
+ for (const record of records) this.graphRecords.set(record.id, record);
416
+ this.pendingSummaryIds = records.map((record) => record.id);
417
+ return records;
418
+ });
353
419
  this.state = {
420
+ ...this.state,
354
421
  phase: "prepared",
355
422
  trigger,
356
423
  summaryId: activeNode.id,
@@ -409,6 +476,16 @@ export class CompactionCoordinator {
409
476
  ctx.ui.notify(`DS4 compaction unavailable; using Pi default. ${message}`, "warning");
410
477
  }
411
478
  return undefined;
479
+ } finally {
480
+ timings.totalMs = performance.now() - startedAt;
481
+ this.dependencies.logger.debug("compaction.timings", {
482
+ path: this.state.path,
483
+ phase: this.state.phase,
484
+ provider: model.provider,
485
+ model: model.id,
486
+ summaryCalls: this.state.summaryCalls,
487
+ ...timings,
488
+ });
412
489
  }
413
490
  }
414
491
 
@@ -433,7 +510,10 @@ export class CompactionCoordinator {
433
510
  const attempt = this.state;
434
511
  const customError = attempt.lastError;
435
512
  this.state = {
513
+ ...attempt,
436
514
  phase: "pi-default",
515
+ summaryId: undefined,
516
+ validationStatus: undefined,
437
517
  trigger: event.reason,
438
518
  firstKeptEntryId: event.compactionEntry.firstKeptEntryId,
439
519
  tokensBefore: event.compactionEntry.tokensBefore,
@@ -467,6 +547,7 @@ export class CompactionCoordinator {
467
547
  this.pendingSummaryIds = [];
468
548
  const metadata = details.ds4ContextEngine;
469
549
  this.state = {
550
+ ...this.state,
470
551
  phase: "committed",
471
552
  trigger: metadata.reason,
472
553
  summaryId: metadata.summaryId,
@@ -474,7 +555,7 @@ export class CompactionCoordinator {
474
555
  validationStatus: metadata.validationStatus,
475
556
  firstKeptEntryId: effectiveEntry.firstKeptEntryId,
476
557
  tokensBefore: effectiveEntry.tokensBefore,
477
- requestedAt: metadata.generatedAt,
558
+ requestedAt: this.state.requestedAt ?? metadata.generatedAt,
478
559
  completedAt: this.dependencies.now(),
479
560
  ...(this.state.inputBudgetTokens !== undefined
480
561
  ? { inputBudgetTokens: this.state.inputBudgetTokens }
@@ -660,6 +741,7 @@ export class CompactionCoordinator {
660
741
  segmentNodes: records.filter((record) => record.kind === "segment").length,
661
742
  aggregateNodes: records.filter((record) => record.kind === "aggregate").length,
662
743
  branchNodes: records.filter((record) => record.kind === "branch").length,
744
+ taskStateNodes: records.filter((record) => record.kind === "task-state").length,
663
745
  maxGraphLevel: records.reduce((maximum, record) => Math.max(maximum, record.graphLevel), 0),
664
746
  rootSummaryIds: roots,
665
747
  ...(activeId ? { activeSummaryId: activeId } : {}),
@@ -717,7 +799,8 @@ export class CompactionCoordinator {
717
799
  config.context.recentTailTokens,
718
800
  ),
719
801
  };
720
- return Math.floor(resolved.budget.activeInputBudget);
802
+ const maxOutputTokens = Math.max(1, Math.min(config.context.maxSummaryTokens, model.maxTokens ?? config.context.maxSummaryTokens));
803
+ return compactionInputBudget(resolved.budget, maxOutputTokens, config.compaction.inputBudget ?? "summary");
721
804
  }
722
805
 
723
806
  private classify(text: string, provider: string): {
@@ -743,7 +826,9 @@ export class CompactionCoordinator {
743
826
  source: PreparedCompactionSourceSlice,
744
827
  event: SessionBeforeCompactEvent,
745
828
  provider: string,
829
+ previousSummary?: string,
746
830
  ): SegmentGenerationPlan {
831
+ const previousPrivacy = previousSummary === undefined ? undefined : this.classify(previousSummary, provider);
747
832
  const sourcePrivacy = this.classify(source.conversationText, provider);
748
833
  const validationPrivacy = this.classify(source.sourceText, provider);
749
834
  const instructionPrivacy = event.customInstructions
@@ -754,6 +839,7 @@ export class CompactionCoordinator {
754
839
  const classification = [
755
840
  sourcePrivacy.classification,
756
841
  validationPrivacy.classification,
842
+ ...(previousPrivacy ? [previousPrivacy.classification] : []),
757
843
  ...(instructionPrivacy ? [instructionPrivacy.classification] : []),
758
844
  ...readFilePrivacy.map((item) => item.classification),
759
845
  ...modifiedFilePrivacy.map((item) => item.classification),
@@ -762,6 +848,7 @@ export class CompactionCoordinator {
762
848
  const modifiedFiles = modifiedFilePrivacy.map((item) => item.value);
763
849
  const prompt = buildSummaryPrompt({
764
850
  conversationText: sourcePrivacy.value,
851
+ ...(previousPrivacy ? { previousSummary: previousPrivacy.value, purpose: "update" as const } : {}),
765
852
  ...(instructionPrivacy ? { customInstructions: instructionPrivacy.value } : {}),
766
853
  readFiles,
767
854
  modifiedFiles,
@@ -770,7 +857,7 @@ export class CompactionCoordinator {
770
857
  return {
771
858
  source,
772
859
  prompt,
773
- validationSource: validationPrivacy.value,
860
+ validationSource: [previousPrivacy?.value, validationPrivacy.value].filter(Boolean).join("\n\n"),
774
861
  readFiles,
775
862
  modifiedFiles,
776
863
  classification,
@@ -950,6 +1037,7 @@ export class CompactionCoordinator {
950
1037
  if (plan.promptTokens > input.inputBudgetTokens) {
951
1038
  throw new Error("Compaction aggregate prompt exceeded its preflight input budget");
952
1039
  }
1040
+ this.state.aggregateCalls = aggregateCalls;
953
1041
  const generated = await this.generateSummary({
954
1042
  stage: "aggregate",
955
1043
  prompt: plan.prompt,
@@ -991,7 +1079,7 @@ export class CompactionCoordinator {
991
1079
  }
992
1080
 
993
1081
  private generateSummary(input: {
994
- stage: "segment" | "aggregate";
1082
+ stage: "segment" | "aggregate" | "update";
995
1083
  prompt: string;
996
1084
  validationSource: string;
997
1085
  readFiles: readonly string[];
@@ -1001,6 +1089,7 @@ export class CompactionCoordinator {
1001
1089
  model: Model<Api>;
1002
1090
  thinking?: CompactionThinkingLevel;
1003
1091
  }): Promise<GeneratedSummary> {
1092
+ this.state.summaryCalls = (this.state.summaryCalls ?? 0) + 1;
1004
1093
  return generateValidatedSummary({
1005
1094
  ...input,
1006
1095
  validate: this.dependencies.config.compaction.validate,
@@ -1101,6 +1190,8 @@ export class CompactionCoordinator {
1101
1190
  validationStatus: summary.validationStatus,
1102
1191
  firstKeptEntryId: summary.firstKeptEntryId,
1103
1192
  tokensBefore: summary.tokensBefore,
1193
+ provider: summary.provider,
1194
+ model: summary.model,
1104
1195
  requestedAt: summary.createdAt,
1105
1196
  ...(summary.lifecycleStatus !== "prepared" ? { completedAt: summary.createdAt } : {}),
1106
1197
  ...(summary.lifecycleStatus === "failed" ? { lastError: "Prepared summary was not committed by Pi" } : {}),
@@ -1128,6 +1219,7 @@ export function defaultSummaryGraphDiagnostics(): SummaryGraphDiagnostics {
1128
1219
  segmentNodes: 0,
1129
1220
  aggregateNodes: 0,
1130
1221
  branchNodes: 0,
1222
+ taskStateNodes: 0,
1131
1223
  maxGraphLevel: 0,
1132
1224
  rootSummaryIds: [],
1133
1225
  activePathIds: [],
@@ -0,0 +1,43 @@
1
+ /**
2
+ * Ordered results from bounded, abort-cooperative work. A failure cancels peers
3
+ * and drains every started worker before returning, so fallback cannot overlap
4
+ * with unfinished custom requests. No new item starts after failure/abort.
5
+ */
6
+ export async function mapCompactionSegments<T, R>(
7
+ items: readonly T[],
8
+ concurrency: number,
9
+ signal: AbortSignal,
10
+ run: (item: T, index: number, signal: AbortSignal) => Promise<R>,
11
+ ): Promise<R[]> {
12
+ if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > 2) {
13
+ throw new Error("Compaction segment concurrency must be an integer between 1 and 2");
14
+ }
15
+ const controller = new AbortController();
16
+ const abort = (): void => controller.abort();
17
+ signal.addEventListener("abort", abort, { once: true });
18
+ if (signal.aborted) abort();
19
+ const results: R[] = new Array(items.length);
20
+ let next = 0;
21
+ let failure: unknown;
22
+ let failed = false;
23
+ const worker = async (): Promise<void> => {
24
+ while (!controller.signal.aborted) {
25
+ const index = next++;
26
+ if (index >= items.length) return;
27
+ try {
28
+ results[index] = await run(items[index] as T, index, controller.signal);
29
+ } catch (error) {
30
+ if (!failed) { failure = error; failed = true; }
31
+ controller.abort();
32
+ }
33
+ }
34
+ };
35
+ try {
36
+ await Promise.all(Array.from({ length: Math.min(concurrency, items.length) }, worker));
37
+ if (failed) throw failure;
38
+ if (controller.signal.aborted) throw new Error("Compaction summary generation aborted");
39
+ return results;
40
+ } finally {
41
+ signal.removeEventListener("abort", abort);
42
+ }
43
+ }
@@ -244,6 +244,22 @@ export function findExactPiMessageSourceIds(
244
244
  });
245
245
  }
246
246
 
247
+ /**
248
+ * Maps managed-plan messages back to Pi session entry ids. Synthetic evidence
249
+ * indices are skipped so role/order fallbacks stay aligned with real messages.
250
+ */
251
+ export function findPiSourceEntryIds(
252
+ messages: readonly unknown[],
253
+ ctx: Pick<ExtensionContext, "sessionManager">,
254
+ syntheticIndices: ReadonlySet<number> = new Set(),
255
+ ): Array<string | undefined> {
256
+ return mapMessageSources(
257
+ messages,
258
+ sourceCandidates(ctx.sessionManager.buildContextEntries()),
259
+ syntheticIndices,
260
+ ).map((source) => source.sourceId);
261
+ }
262
+
247
263
  export function findPiPinnedMessageIndices(event: ContextEvent, ctx: ExtensionContext): number[] {
248
264
  const candidates = sourceCandidates(ctx.sessionManager.buildContextEntries());
249
265
  const sources = mapMessageSources(event.messages, candidates);
@@ -18,9 +18,8 @@ export const DEFAULT_COMPACTION_TRANSPORT_BASE_DELAY_MS = 2000;
18
18
  export const COMPACTION_TRANSPORT_MAX_DELAY_MS = 60_000;
19
19
 
20
20
  /**
21
- * Transport retry policy for compaction summary requests. Defaults mirror Pi's
22
- * assistant retry settings (`retry.maxRetries` 3, `retry.baseDelayMs` 2000,
23
- * exponential backoff, abort-aware).
21
+ * Transport retry policy for compaction summary requests: three total attempts
22
+ * (not three retries), 2000 ms base delay, exponential backoff, abort-aware.
24
23
  */
25
24
  export interface CompactionTransportPolicy {
26
25
  /** Total attempts for transport-classified failures. Default: 3. */
@@ -48,7 +47,7 @@ export function transportRetryDelayMs(baseDelayMs: number, failedAttempt: number
48
47
  }
49
48
 
50
49
  export interface CompactionTransportRetryDiagnostic {
51
- stage: "segment" | "aggregate";
50
+ stage: "segment" | "aggregate" | "update";
52
51
  failedAttempt: number;
53
52
  nextAttempt: number;
54
53
  maxAttempts: number;
@@ -56,7 +55,7 @@ export interface CompactionTransportRetryDiagnostic {
56
55
  }
57
56
 
58
57
  export interface GenerateValidatedSummaryInput {
59
- stage: "segment" | "aggregate";
58
+ stage: "segment" | "aggregate" | "update";
60
59
  prompt: string;
61
60
  validationSource: string;
62
61
  readFiles: readonly string[];
@@ -69,7 +68,7 @@ export interface GenerateValidatedSummaryInput {
69
68
  model?: Model<Api>;
70
69
  /** Reasoning level for the summary request; `off` (default) keeps the pre-existing request shape. */
71
70
  thinking?: CompactionThinkingLevel;
72
- /** Transport retry policy; defaults mirror Pi's assistant retry settings. */
71
+ /** Transport-only retry policy; three total attempts by default. */
73
72
  transport?: CompactionTransportPolicy;
74
73
  now: () => number;
75
74
  onTransportRetry?: (diagnostic: CompactionTransportRetryDiagnostic) => void;
@@ -1,4 +1,4 @@
1
- export const EXTENSION_VERSION = "0.3.4";
1
+ export const EXTENSION_VERSION = "0.3.6";
2
2
  export const SUPPORTED_PI_VERSION = "0.84.3";
3
3
  export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
4
4
  export const PLANNER_VERSION = "managed-learned-ranking-v1";