ds4-context-engine 0.3.7 → 0.3.9

package/README.md

@@ -16,9 +16,9 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.3.5` release adds bounded compaction optimizations: one validated previous-summary/source update when the complete prompt fits, a dedicated calibrated input budget, up to two concurrent segments, and metadata-only phase timings in `/context compaction`. Canonical history, SQLite schema 15 and runtime contracts remain unchanged; Pi remains pinned to `0.84.3`. See the [0.3.5 release record](docs/releases/0.3.5.md) for validation and publication status.
+> **Project status:** The coordinated `0.3.9` release adds hard estimated-input limits for every compaction request and the complete compaction operation, including retries and concurrent segments. The conservative context-fill input budget is now the default, while the prior hard-limit budget remains opt-in. Canonical history, SQLite schema 16 and runtime contracts remain unchanged from `0.3.8`; Pi remains pinned to `0.84.3`. See the [0.3.9 release record](docs/releases/0.3.9.md) for validation and publication status.
 
-**New compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="summary"`, `compaction.maxConcurrentSegments=2`. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls) for the legacy-path settings. No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
+**Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
 ## Why DS4
 
@@ -331,9 +331,11 @@ The following example shows the main configuration groups. Omitted values use th
     "mode": "hierarchical",
     "validate": true,
     "segmentTargetTokens": 30000,
+    "maxRequestInputTokens": 64000,
+    "maxOperationInputTokens": 2000000,
     "preserveRecentVerbatim": true,
     "directUpdate": true,
-    "inputBudget": "summary",
+    "inputBudget": "context",
     "maxConcurrentSegments": 2
   },
   "privacy": {
@@ -512,6 +514,10 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.3.9 release notes](docs/releases/0.3.9.md)
+- [0.3.8 release notes](docs/releases/0.3.8.md)
+- [0.3.7 release notes](docs/releases/0.3.7.md)
+- [0.3.6 release notes](docs/releases/0.3.6.md)
 - [0.3.5 release notes](docs/releases/0.3.5.md)
 - [0.3.4 release notes](docs/releases/0.3.4.md)
 - [0.3.3 release notes](docs/releases/0.3.3.md)
@@ -535,7 +541,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.3.5 adds bounded compaction updates, summary input headroom, concurrent segments and phase timings. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ADR/061-compaction-latency.md

@@ -14,6 +14,14 @@ Pi normally updates previous summary plus discarded messages in one request. DS4
 - Expose process-local metadata-only timings for preparation, segment/direct generation, aggregation, graph preparation/persistence and total DS4 hook duration, plus chosen path, effective provider/model, direct prompt size and concurrency. Timings use a monotonic clock, include retries and local validation within generation, and are wall times (not sums of overlapping calls). They are not canonical evidence, are not persisted in JSONL, and do not measure a subsequent native Pi fallback.
 - Preserve schema-v2 compaction metadata, the existing `task-state` kind, source/classification/validation contracts, Pi cut points, atomic tool exchanges, fresh routing IDs, retry policy, canonical JSONL and all-or-nothing graph preparation. `segmentSummaryId` refers to the update node itself on a direct update; do not invent a segment that was never generated.
 
+## Subsequent default revision
+
+After operational testing, the default for `compaction.inputBudget` was changed from `summary` to `context`. The original `summary` mode remains available as an explicit throughput-oriented opt-in. The conservative default limits direct updates and indivisible atomic groups to the ordinary active input target, reducing request-size peaks at the cost of potentially more segment and aggregate calls. This revision changes only the default: calibrated hard limits, output headroom, atomicity, validation, and fail-closed behavior remain intact.
+
+The hierarchical partitioner applies `min(compaction.segmentTargetTokens, effective request input limit)` to ordinary segments. A single indivisible message or complete tool exchange may exceed the target, but it is isolated and must still fit the effective request input limit.
+
+Post-release hardening adds two additive safeguards. `maxRequestInputTokens` caps the estimated prompt size of every direct-update, segment, aggregate, and retry attempt after intersecting it with the selected model input budget. `maxOperationInputTokens` caps the cumulative estimated prompt input reserved across the whole DS4 operation, including retries and concurrent segment attempts. Crossing either limit fails closed to Pi without committing a partial summary graph.
+
 ## Compatibility and validation
 
 Configuration is additive; absent fields use the new defaults. The old scheduling path can be compared using `directUpdate=false`, `inputBudget=context`, and `maxConcurrentSegments=1`. The compaction master switch still delegates to Pi when disabled. The original local implementation excluded versioning and publication; the user subsequently authorized the coordinated 0.3.5 release. No dependency upgrade, live database maintenance or Pi upgrade is part of this change.

package/docs/ADR/063-fts-rowid-key-mappings.md

@@ -0,0 +1,53 @@
+# 063 — Resolve FTS key deletes through rowid mapping tables
+
+**Date:** 2026-09-06
+**Status:** Accepted
+**Related:** [002](002-pi-jsonl-canonical-sqlite-rebuildable.md), [058](058-bounded-manifest-storage.md)
+
+## Context
+
+The FTS5 shadow tables used by the session index and the project knowledge
+index declare `entry_key` / `snippet_id` as `UNINDEXED` columns. FTS5 cannot
+use the token vocabulary to locate a row by an `UNINDEXED` column, so every
+`DELETE FROM …_fts WHERE entry_key = ?` is a full scan of the whole virtual
+table, whose cost is proportional to the size of the entire FTS index.
+
+For incremental appends the cost is small, but a session-index **rebuild**
+(fork, path change, truncation, missing checkpoint) re-writes every entry:
+a large session (11k entries, ~600MB database) paid thousands of full scans
+and stalled for hours, surfacing as an indefinite hang of the agent session.
+
+A first candidate — making `entry_key` an indexed FTS5 column and recreating
+the tables in a migration — was benchmarked and rejected: FTS5 then treats the
+key as searchable vocabulary, changing the `MATCH` surface (a term present
+only in an entry key becomes a false-positive hit) and the `bm25` column
+weight mapping of existing queries.
+
+## Decision
+
+Keep the FTS table definitions byte-identical and add dedicated mapping
+tables that derive the SQLite rowid of each FTS row:
+
+- `entries_fts_keys(entry_key PRIMARY KEY, fts_rowid)`
+- `project_snippets_fts_keys(snippet_id, project_path, fts_rowid)`
+
+Deletes become `DELETE FROM …_fts WHERE rowid = <mapped>` (O(log n)) and
+rebuild cleanups resolve stale rows through the same mapping joined on the
+base tables. Insert paths upsert the mapping from the FTS `last_insert_rowid`
+inside the same write transaction as the FTS insert, so mapping and index
+cannot diverge (a crash rolls back both).
+
+The mapping tables are purely derived, rebuildable state: migration 16
+backfills them from the live FTS rows with one scan, in a single
+transaction — a failure rolls back and the previous schema keeps working.
+
+## Consequences
+
+- Per-row FTS deletes: ~0.23 ms instead of ~40 ms–2 s (cold) per row;
+  a 11k-row rebuild drops from hours to ~3 s.
+- The FTS schema, the `MATCH` surface and the `bm25` weight mapping are
+  unchanged (verified by the migration and repository tests).
+- Existing databases upgrade non-destructively on first open after the
+  extension update; no user data is stored in the mapping tables.
+- Future migrations that drop/recreate FTS tables must rebuild the mapping
+  tables in the same migration.

package/docs/ADR/README.md

@@ -66,5 +66,6 @@ The initial decisions from the development plan are accepted:
 | [060](060-optional-portable-agent-tools.md) | Opt in to edit reports, adaptive reads/results and session-owned local jobs | Accepted |
 | [061](061-compaction-latency.md) | Bound compaction update calls, input budgets, concurrent segments and phase timings | Accepted |
 | [062](062-cache-aware-context-planning.md) | Opt-in cache-aware tail planning using model pricing and observed cache shares | Accepted |
+| [063](063-fts-rowid-key-mappings.md) | Resolve FTS key deletes through rowid mapping tables | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/COMPACTION.md

@@ -8,12 +8,12 @@ DS4 intercepts Pi's `session_before_compact` event but preserves Pi's cut-point
 2. DS4 maps every source message by exact fingerprint to a canonical branch entry ID.
 3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, previous summary, custom instructions, and file paths for the effective compaction provider (dedicated model when configured and eligible).
 4. DS4 estimates the **complete** sanitized request against a calibrated summary-specific input budget, including framing, instructions, file inventories and output contract. With `compaction.directUpdate` enabled, a previous summary plus new source that fits is updated and validated in **one call**, producing an immutable `task-state` node. No predecessor needs only the existing one-segment call. Oversized updates fall through to hierarchical planning; no additional source is truncated to force a fit.
-5. The hierarchical path partitions oversized source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results. Up to `compaction.maxConcurrentSegments` independent segment requests run concurrently (default 2). Each summary is validated against only its own sanitized evidence. Identities, source/child order and usage accumulation follow source order, not completion order. Cache retention stays disabled and every attempt has a fresh routing session ID.
+5. The hierarchical path partitions source into ordered contiguous segments capped at `min(compaction.segmentTargetTokens, requestInputLimitTokens)`, where the effective request limit is also bounded by the selected model input budget. The target is soft only for one indivisible atomic group: an individual message or complete tool call/result exchange may exceed it, but must still fit the effective request input limit and is isolated in its own segment. Up to `compaction.maxConcurrentSegments` independent segment requests run concurrently (default 2). Each summary is validated against only its own sanitized evidence. Identities, source/child order and usage accumulation follow source order, not completion order. Cache retention stays disabled and every attempt has a fresh routing session ID.
 6. Hierarchical requests recursively aggregate ordered children until one root remains: previous branch summary first, then new segments. Aggregation stays sequential and budget-checked. Direct updates instead link their single node to the predecessor and new canonical source IDs, without generating a synthetic segment or a separate aggregate. DS4-generated IDs, hashes, kinds, and graph levels never become model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
 7. The highest input classification wraps each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage includes every returned direct/segment/aggregate request and transport replay. Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
 8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the complete prepared batch `failed`.
 
-Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. The DS4 transport replay policy is: `compaction.transport.maxAttempts` (default 3) total attempts and `compaction.transport.baseDelayMs` (default 2000 ms, capped at 60 s) backoff, doubling per attempt, abort-aware. Replay never applies to input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run. On segment failure or cancellation DS4 stops scheduling, aborts siblings and awaits all started workers before fallback; no partial graph is installed. Cancellation is cooperative: accepted provider work may still cost tokens, and a provider that ignores abort can delay settlement.
+Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. The DS4 transport replay policy is: `compaction.transport.maxAttempts` (default 4, matching Pi's initial call plus three retries) total attempts and `compaction.transport.baseDelayMs` (default 2000 ms, capped at 60 s) backoff, doubling per attempt, abort-aware. Replay never applies to input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run. On segment failure or cancellation DS4 stops scheduling, aborts siblings and awaits all started workers before fallback; no partial graph is installed. Cancellation is cooperative: accepted provider work may still cost tokens, and a provider that ignores abort can delay settlement.
 
 ## Required summary contract
 
@@ -80,42 +80,49 @@ Semantics:
 
 ## Latency controls
 
-The coordinated `0.3.5` release introduces these additive defaults (absent from `0.3.4`):
+The current defaults favor bounded request size while retaining direct updates and bounded parallelism:
 
 ```json
 {
   "compaction": {
     "directUpdate": true,
-    "inputBudget": "summary",
+    "inputBudget": "context",
+    "segmentTargetTokens": 30000,
+    "maxRequestInputTokens": 64000,
+    "maxOperationInputTokens": 2000000,
     "maxConcurrentSegments": 2
   }
 }
 ```
 
-- `directUpdate`: one validated previous-summary plus new-source request when the entire prompt fits. Set false to always retain the segment-then-aggregate route. Validation or provider failures still fall back to Pi, not an unvalidated update.
-- `inputBudget`: `summary` uses calibrated `hardInputLimit` rather than the ordinary context fill target (`activeInputBudget`). `context` selects that legacy fill target. Both are additionally capped by `(context window - safety margin - actual summary output cap) / calibration ratio`, rounded down, and never exceed the configured/model hard limit. Existing output reservations remain conservative; ordinary session planning and proactive thresholds are unchanged. This reduces avoidable fragmentation, not a guarantee of provider fit or faster processing for a larger prompt.
+`segmentTargetTokens` is the soft partition target for source segments. `maxRequestInputTokens` is the hard estimated-input cap applied to every direct-update, segment, aggregate, and retry attempt; the effective request limit is the minimum of this value and the safe model input budget. An indivisible message or complete tool exchange above that limit fails closed to Pi's native compaction rather than creating an oversized DS4 request.
+
+`maxOperationInputTokens` bounds the sum of estimated prompt tokens reserved immediately before all provider attempts in one DS4 compaction, including transport retries and concurrently scheduled segments. Exceeding it aborts remaining DS4 work and falls back without committing a partial summary graph. Defaults are 64,000 per request and 2,000,000 per operation; both must be positive integers, and the operation limit must be at least the configured request limit.
+
+- `directUpdate`: one validated previous-summary plus new-source request when the entire prompt fits the effective request input limit. Set false to always retain the segment-then-aggregate route. Validation or provider failures still fall back to Pi, not an unvalidated update.
+- `inputBudget`: `context` (default) uses the ordinary context fill target (`activeInputBudget`). `summary` is an explicit throughput-oriented opt-in that permits the calibrated `hardInputLimit`. Both are additionally capped by `(context window - safety margin - actual summary output cap) / calibration ratio`, rounded down, and never exceed the configured/model hard limit. Existing output reservations remain conservative; ordinary session planning and proactive thresholds are unchanged. The conservative default reduces request-size peaks but can produce more segments and calls; it is not a guarantee of provider fit or lower total token use.
 - `maxConcurrentSegments`: integer **1–2**, default 2. Only independent segments overlap, including their retries. Aggregates do not run until their children have completed. Use 1 for sequential execution or providers with restrictive concurrent-request limits. Rate-limit failures are not transport-retried.
 
-For an old-path comparison set `directUpdate=false`, `inputBudget=context`, `maxConcurrentSegments=1`. All features remain behind the existing compaction/master switches. Settings are applied on session load; after upgrading the package or rebuilding a development checkout, fully restart Pi to avoid stale compiled-core modules. No schema migration is required.
+For an old sequential-path comparison set `directUpdate=false`, `inputBudget=context`, `maxConcurrentSegments=1`. To reproduce the `0.3.5` throughput-oriented budget, set `inputBudget=summary`. All features remain behind the existing compaction/master switches. Settings are applied on session load; after upgrading the package or rebuilding a development checkout, fully restart Pi to avoid stale compiled-core modules. No schema migration is required.
 
 Mock-provider regression tests verify fewer calls, bounded overlap, exact budget boundaries, validation, privacy, immutable provenance and JSONL rebuild. They do **not** establish real-provider wall-time gains, semantic equivalence of generated summaries, or a guaranteed completion time. See [ADR-061](ADR/061-compaction-latency.md).
 
 ## Transport retry policy
 
-Summary requests are replayed only for transport-classified failures (thrown transport errors or `stopReason: "error"` responses whose message matches network/timeout patterns). The DS4 replay policy uses **three total attempts**, not three retries after the initial call, and can be tuned per deployment:
+Summary requests are replayed only for transport-classified failures (thrown transport errors or `stopReason: "error"` responses whose message matches network/timeout patterns). The DS4 replay policy uses **four total attempts** (the initial call plus up to three retries), matching Pi's standard retry count, and can be tuned per deployment:
 
 ```json
 {
   "compaction": {
     "transport": {
-      "maxAttempts": 3,
+      "maxAttempts": 4,
       "baseDelayMs": 2000
     }
   }
 }
 ```
 
-- `compaction.transport.maxAttempts`: total attempts per direct update, segment or aggregate call, integer 1–10, default 3. With 1, no transport failure is retried.
+- `compaction.transport.maxAttempts`: total attempts per direct update, segment or aggregate call, integer 1–10, default 4. With 1, no transport failure is retried.
 - `compaction.transport.baseDelayMs`: base backoff before the first replay, integer 0–60000, default 2000. The delay doubles per attempt (2000, 4000, 8000, …) and is capped at 60 s.
 - Replays use a fresh routing session per attempt; diagnostics expose only stage, failed/next attempt, max attempts, and delay.
 - Aborts (including during backoff) never trigger replay; non-transport failures are never retried; usage is summed across replayed responses.

package/docs/CONTEXT_PLANNER.md

@@ -76,7 +76,7 @@ With privacy disabled, DS4 returns Pi's original `AgentMessage[]` when:
 - final estimated input exceeds the hard limit;
 - an unexpected adapter or planner exception occurs.
 
-Expected fallbacks are recorded in the Context Manifest. With privacy enabled, the fallback baseline is the sanitized native array—not raw Pi messages—and an unexpected privacy failure replaces content/payload fields instead of sending unchecked data. Observer mode disables planning but still enforces enabled privacy policy and records manifests/usage calibration.
+Expected fallbacks are recorded in the Context Manifest. Because fail-open preserves the complete native array, DS4's hard input limit is not guaranteed in fallback when the current request, fixed overhead, or another mandatory atomic group is itself oversized. With privacy enabled, the fallback baseline is the sanitized native array—not raw Pi messages—and an unexpected privacy failure replaces content/payload fields instead of sending unchecked data. Observer mode disables planning but still enforces enabled privacy policy and records manifests/usage calibration.
 
 ## Quality measurement
 

package/docs/releases/0.3.7.md

@@ -70,6 +70,15 @@ same tail caps as 0.3.6.
 - Prefix-cache simulator: 7 passed.
 - Cache-aware runtime integration: 4 passed.
 - Config catalog/loader validation: passed.
+- `pack:check` from a clean HEAD snapshot: core 239 files,
+  reference-adapter 7 files, engine 91 files; consumer install clean.
+- Published and registry-verified: `ds4-context-core` 0.3.7,
+  `ds4-context-reference-adapter` 0.3.7, `ds4-context-engine` 0.3.7
+  (`registry:check` PASS at exact version; one transient registry replica
+  miss on first attempt, confirmed present via `npm view`).
+- Release fix `7404a9a`: root `package.json` must depend exactly on
+  `ds4-context-core@0.3.7` (caught by the `pack:check` gate on the
+  coordinated-version requirement).
 
 ## Known limits
 

package/docs/releases/0.3.8.md

@@ -0,0 +1,64 @@
+# Release 0.3.8 — Indexed FTS key deletes via rowid mappings (schema 16)
+
+**Version analyzed:** DS4 Context Engine `0.3.8`
+**Commit:** `18f36e1`
+**Coordinated packages:** `ds4-context-core` 0.3.8, `ds4-context-reference-adapter` 0.3.8, `ds4-context-engine` 0.3.8
+
+## Summary
+
+Bugfix release: FTS deletes were full virtual-table scans (FTS5 `UNINDEXED`
+key columns), making session-index rebuilds on large sessions stall for
+hours. The FTS tables are now rebuilt over rowid mapping tables, turning
+every keyed delete into an indexed lookup. No behavior, configuration or
+search-surface change: the fix is transparent and addresses the
+fork/resume stall on large sessions.
+
+## Root cause
+
+`DELETE FROM entries_fts WHERE entry_key = ?` cannot use the FTS5
+vocabulary because `entry_key` (and `project_snippets_fts.snippet_id`) are
+declared `UNINDEXED`. Every row removal scanned the entire FTS index.
+A rebuild of a session with ~11k entries performed thousands of those
+scans: measured 65k `pread64`/s against a 615MB database and ~100 full
+index passes per ~140s, with an estimated **hours** to completion —
+surfacing as an indefinite hang of the agent session after a session
+fork/resume. The behavior class pre-dates 0.3.7; the 0.3.7 work sessions
+made it visible by pushing the database into the hundreds of MB.
+
+Indexing the keys directly inside FTS5 was benchmarked and **rejected**:
+it would make key text part of the `MATCH` vocabulary (false positives)
+and shift the `bm25` column weights. See ADR 063.
+
+## Changes
+
+- Migration **16** `fts-rowid-key-mappings`: two derived mapping tables
+  (`entries_fts_keys`, `project_snippets_fts_keys`) backfilled from the
+  live FTS rows in one scan, inside a single transaction.
+- `SessionIndexRepository`: per-row deletes go through the mapping
+  (`DELETE … WHERE rowid = ?`); the rebuild stale-row cleanup joins the
+  mapping with `entries` and uses the session index; the mapping is
+  upserted from the FTS `last_insert_rowid` in the same transaction.
+- `ProjectKnowledgeRepository`: same pattern for per-snippet deletes
+  (`replaceFile`) and per-project cleanup (`clearProject`).
+- The FTS schemas, the `MATCH` queries and the `bm25` weights are
+  byte-identical to 0.3.7; search results do not change.
+- Tests: `tests/integration/fts-rowid-keys.test.ts` covering the
+  v15 → v16 upgrade (data preservation, key ↔ rowid correctness, search
+  surface unchanged), rebuild/append/stale-cleanup consistency and the
+  snippet `replaceFile`/`clearProject` paths; version-pinned tests moved
+  to `CURRENT_SCHEMA_VERSION`.
+
+## Compatibility and migration
+
+- **Non-destructive by design:** the FTS indexes are derived state; the
+  migration is transactional, so on any failure the previous schema
+  remains intact and the extension keeps working with the old behavior.
+- **No data stored in the mapping tables:** the base tables remain the
+  single source of truth.
+- Upgrade runs automatically on first open after the extension update
+  (measured 467 ms on a 615MB database with 31k entries / 62k snippets;
+  no user-visible step required).
+- The mapping backfill is one scan; a very large database adds only
+  seconds to the first open after update.
+- Existing guarantees (fail-open, atomicity, privacy, pins, compaction,
+  hard limits) are unchanged.

package/docs/releases/0.3.9.md

@@ -0,0 +1,80 @@
+# Release 0.3.9 — Bounded compaction request and operation input
+
+**Version analyzed:** DS4 Context Engine `0.3.9`
+**Commit:** (recorded after pack and registry verification)
+**Coordinated packages:** `ds4-context-core` 0.3.9, `ds4-context-reference-adapter` 0.3.9, `ds4-context-engine` 0.3.9
+
+## Summary
+
+Hardens custom compaction against request-size and total-operation token spikes.
+Every direct-update, segment, aggregate and retry attempt is now bounded by an
+effective estimated-input limit. A second cumulative budget covers the complete
+compaction operation, including concurrent work and transport retries.
+
+The default compaction input budget changes from the summary hard limit to the
+ordinary context-fill budget. The former behavior remains available explicitly
+for users who accept larger provider requests in exchange for fewer hierarchy
+levels.
+
+## Changes
+
+- Added `compaction.maxRequestInputTokens`, default 64,000, as a hard ceiling on
+  estimated input for each compaction provider request. The effective ceiling is
+  the minimum of this setting and the selected model-aware compaction budget.
+- Added `compaction.maxOperationInputTokens`, default 2,000,000, for cumulative
+  estimated input across the complete operation. Every accepted attempt reserves
+  its input before provider dispatch, so retries and concurrent segments count.
+- Configuration validation requires both limits to be positive integers and the
+  operation limit to be at least the configured request limit.
+- Changed the default `compaction.inputBudget` from `summary` to `context`.
+  Explicit `summary` configurations retain the previous calibrated-hard-limit
+  behavior.
+- Changed the default `compaction.transport.maxAttempts` from 3 to 4. Attempts
+  remain covered by the cumulative operation budget.
+- Segment partitioning now uses the minimum of `segmentTargetTokens` and the
+  effective request limit. An indivisible message or complete tool exchange that
+  cannot fit fails closed to Pi's native compaction.
+- `/context compaction` reports the effective request limit and cumulative
+  operation input consumed.
+- Direct update, segment and aggregate paths continue to use strict validation,
+  privacy filtering, immutable provenance and all-or-nothing graph installation.
+
+## Validation
+
+Before publication, the coordinated release passed:
+
+- TypeScript builds and typecheck for the root, core and reference adapter;
+- 85 Vitest files with 561 tests;
+- deterministic context-quality comparison;
+- context-persistence schema measurement;
+- clean-consumer package verification for all three tarballs;
+- whitespace/error-marker checks and coordinated manifest inspection.
+
+The package verifier also checks the new defaults and both input-budget modes
+from the packed `ds4-context-core` artifact.
+
+## Compatibility and migration
+
+- No SQLite migration: schema 16 remains current.
+- No Pi, runtime-adapter, history, persistence-tool or summary-contract version
+  changes.
+- Existing configurations that explicitly set `inputBudget` or transport retry
+  attempts retain those values.
+- New limit fields are additive. Configurations that omit them receive the
+  documented defaults.
+- Pi JSONL remains canonical and custom-compaction failure still delegates to
+  Pi's native fallback without installing a partial Summary Graph.
+- A full Pi restart is required after upgrading so compiled core modules and
+  session configuration are reloaded.
+
+## Known limits
+
+- If the current request, fixed overhead or another mandatory atomic group is
+  itself above the planner hard limit, fail-open context planning preserves the
+  native context; DS4 cannot transparently split an in-flight Pi tool loop.
+- Provider work accepted before a concurrent sibling fails may still be billed,
+  even though DS4 aborts and awaits all started workers before fallback.
+- Limits use DS4's calibrated token estimator. Provider-side tokenization and
+  billing remain authoritative.
+- Mock-provider tests establish bounded scheduling and request accounting, not
+  real-provider latency or semantic-equivalence guarantees.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.3.7",
+  "version": "0.3.9",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -62,7 +62,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.3.7"
+    "ds4-context-core": "0.3.9"
   },
   "peerDependencies": {
     "@earendil-works/pi-ai": "0.84.3",
@@ -77,5 +77,9 @@
   },
   "engines": {
     "node": ">=22.19.0"
+  },
+  "allowScripts": {
+    "@google/genai@1.52.0": true,
+    "protobufjs@7.6.5": true
   }
 }

package/src/extension/commands.ts

@@ -881,6 +881,8 @@ function formatCompaction(diagnostics: RuntimeDiagnostics, preview: boolean): st
     `Chosen path:             ${compaction.path ?? "n/a"}`,
     `Input budget mode:       ${compaction.inputBudgetMode ?? "n/a"}`,
     `Input budget:            ${count(compaction.inputBudgetTokens)}`,
+    `Request input limit:     ${count(compaction.requestInputLimitTokens)} (configured max ${count(compaction.maxRequestInputTokens)})`,
+    `Operation input:         ${count(compaction.operationInputTokens)} / ${count(compaction.maxOperationInputTokens)}`,
     `Whole-source prompt:     ${count(compaction.sourcePromptTokens)}`,
     `Direct-update prompt:    ${count(compaction.directPromptTokens)}`,
     `Segment concurrency cap: ${count(compaction.maxConcurrentSegments)}`,

package/src/pi-adapter/compaction-coordinator.ts

@@ -9,7 +9,12 @@ import type {
   SessionEntry,
 } from "@earendil-works/pi-coding-agent";
 import type { Api, Model } from "@earendil-works/pi-ai";
-import type { CompactionThinkingLevel, Ds4ContextConfig } from "ds4-context-core/config/config";
+import {
+  DEFAULT_COMPACTION_MAX_OPERATION_INPUT_TOKENS,
+  DEFAULT_COMPACTION_MAX_REQUEST_INPUT_TOKENS,
+  type CompactionThinkingLevel,
+  type Ds4ContextConfig,
+} from "ds4-context-core/config/config";
 import { calculateContextBudget, type ContextBudget } from "ds4-context-core/core/budget-manager";
 import { createModelProfile, type ModelDescriptor } from "ds4-context-core/core/model-profile";
 import type { ContextManifest } from "ds4-context-core/manifest/context-manifest";
@@ -80,6 +85,8 @@ export interface CompactionDiagnostics {
   validate: boolean;
   preserveRecentVerbatim: boolean;
   segmentTargetTokens: number;
+  maxRequestInputTokens: number;
+  maxOperationInputTokens: number;
   phase: CompactionPhase;
   trigger?: CompactionTrigger;
   summaryId?: string;
@@ -91,6 +98,8 @@ export interface CompactionDiagnostics {
   completedAt?: number;
   lastError?: string;
   inputBudgetTokens?: number;
+  requestInputLimitTokens?: number;
+  operationInputTokens?: number;
   sourcePromptTokens?: number;
   segmentCount?: number;
   aggregateCalls?: number;
@@ -189,9 +198,20 @@ interface CompactionCoordinatorDependencies {
 
 type MutableCompactionState = Omit<
   CompactionDiagnostics,
-  "enabled" | "validate" | "preserveRecentVerbatim" | "segmentTargetTokens" | "proactiveEligible"
+  | "enabled"
+  | "validate"
+  | "preserveRecentVerbatim"
+  | "segmentTargetTokens"
+  | "maxRequestInputTokens"
+  | "maxOperationInputTokens"
+  | "proactiveEligible"
 >;
 
+interface CompactionOperationInputBudget {
+  limitTokens: number;
+  usedTokens: number;
+}
+
 function classifiedSummary(content: string, classification: PrivacyClassification): string {
   return classification === "normal"
     ? content
@@ -260,6 +280,14 @@ export class CompactionCoordinator {
       }
     };
     const maxConcurrentSegments = config.compaction.maxConcurrentSegments ?? 2;
+    const maxRequestInputTokens = config.compaction.maxRequestInputTokens
+      ?? DEFAULT_COMPACTION_MAX_REQUEST_INPUT_TOKENS;
+    const maxOperationInputTokens = config.compaction.maxOperationInputTokens
+      ?? DEFAULT_COMPACTION_MAX_OPERATION_INPUT_TOKENS;
+    const operationInputBudget: CompactionOperationInputBudget = {
+      limitTokens: maxOperationInputTokens,
+      usedTokens: 0,
+    };
     const trigger: CompactionTrigger = this.proactiveRequested ? "proactive" : event.reason;
     this.state = {
       phase: "generating",
@@ -272,18 +300,20 @@ export class CompactionCoordinator {
       summaryCalls: 0,
       provider: model.provider,
       model: model.id,
-      inputBudgetMode: config.compaction.inputBudget ?? "summary",
+      inputBudgetMode: config.compaction.inputBudget ?? "context",
       maxConcurrentSegments,
+      operationInputTokens: 0,
       timings,
     };
 
     try {
-      const { source, inputBudgetTokens, wholePlan, directPlan, segmentPlans } = await measure("preparationMs", () => {
+      const { source, inputBudgetTokens, requestInputLimitTokens, wholePlan, directPlan, segmentPlans } = await measure("preparationMs", () => {
         if (event.signal.aborted) throw new Error("Compaction summary generation aborted");
         this.dependencies.syncSessionIndex(ctx);
         const source = prepareCompactionSource(event);
         const inputBudgetTokens = this.inputBudgetTokens(model);
         if (inputBudgetTokens <= 0) throw new Error("Active model has no safe compaction input budget");
+        const requestInputLimitTokens = Math.min(inputBudgetTokens, maxRequestInputTokens);
         const wholePlan = this.buildSegmentPlan(source, event, model.provider);
         const update = (config.compaction.directUpdate ?? true) && source.previousSummary
           ? this.buildSegmentPlan({
@@ -292,18 +322,21 @@ export class CompactionCoordinator {
             segmentModifiedFiles: source.modifiedFiles,
           }, event, model.provider, source.previousSummary)
           : undefined;
-        const directPlan = update && update.promptTokens <= inputBudgetTokens ? update : undefined;
+        const directPlan = update && update.promptTokens <= requestInputLimitTokens ? update : undefined;
         this.state = {
           ...this.state,
           path: directPlan ? "direct-update" : "hierarchical",
           sourceEntries: source.sourceEntryIds.length,
           inputBudgetTokens,
+          requestInputLimitTokens,
           sourcePromptTokens: wholePlan.promptTokens,
           ...(update ? { directPromptTokens: update.promptTokens } : {}),
         };
-        const segmentPlans = directPlan ? [] : this.partitionSegmentPlans(source, wholePlan, event, model.provider, inputBudgetTokens);
+        const segmentPlans = directPlan
+          ? []
+          : this.partitionSegmentPlans(source, wholePlan, event, model.provider, requestInputLimitTokens);
         this.state.segmentCount = segmentPlans.length;
-        return { source, inputBudgetTokens, wholePlan, directPlan, segmentPlans };
+        return { source, inputBudgetTokens, requestInputLimitTokens, wholePlan, directPlan, segmentPlans };
       });
 
       const usedIds = new Set(this.graphRecords.keys());
@@ -352,6 +385,9 @@ export class CompactionCoordinator {
           ctx,
           model,
           thinking: config.compaction.summary?.thinking,
+          promptTokens: plan.promptTokens,
+          requestInputLimitTokens,
+          operationInputBudget,
         }),
       ));
       const generatedNodes: EmbeddedSummaryNode[] = results.map((generated, index) => {
@@ -385,7 +421,8 @@ export class CompactionCoordinator {
         model: model.id,
         readFiles: source.readFiles,
         modifiedFiles: source.modifiedFiles,
-        inputBudgetTokens,
+        requestInputLimitTokens,
+        operationInputBudget,
         nextId,
         createdNodes,
         usages,
@@ -698,6 +735,10 @@ export class CompactionCoordinator {
       validate: config.compaction.validate,
       preserveRecentVerbatim: config.compaction.preserveRecentVerbatim,
       segmentTargetTokens: config.compaction.segmentTargetTokens,
+      maxRequestInputTokens: config.compaction.maxRequestInputTokens
+        ?? DEFAULT_COMPACTION_MAX_REQUEST_INPUT_TOKENS,
+      maxOperationInputTokens: config.compaction.maxOperationInputTokens
+        ?? DEFAULT_COMPACTION_MAX_OPERATION_INPUT_TOKENS,
       ...this.state,
       ...(contextTokens !== undefined ? { contextTokens } : {}),
       ...(budget ? { softLimitTokens: this.providerSoftLimit(budget) } : {}),
@@ -800,7 +841,7 @@ export class CompactionCoordinator {
         ),
       };
     const maxOutputTokens = Math.max(1, Math.min(config.context.maxSummaryTokens, model.maxTokens ?? config.context.maxSummaryTokens));
-    return compactionInputBudget(resolved.budget, maxOutputTokens, config.compaction.inputBudget ?? "summary");
+    return compactionInputBudget(resolved.budget, maxOutputTokens, config.compaction.inputBudget ?? "context");
   }
 
   private classify(text: string, provider: string): {
@@ -870,9 +911,13 @@ export class CompactionCoordinator {
     wholePlan: SegmentGenerationPlan,
     event: SessionBeforeCompactEvent,
     provider: string,
-    inputBudgetTokens: number,
+    requestInputLimitTokens: number,
   ): SegmentGenerationPlan[] {
-    if (wholePlan.promptTokens <= inputBudgetTokens) return [wholePlan];
+    const segmentTargetTokens = Math.min(
+      requestInputLimitTokens,
+      this.dependencies.config.compaction.segmentTargetTokens,
+    );
+    if (wholePlan.promptTokens <= segmentTargetTokens) return [wholePlan];
 
     const groups = buildCompactionAtomicGroups(source.messages);
     const plans: SegmentGenerationPlan[] = [];
@@ -886,7 +931,7 @@ export class CompactionCoordinator {
         event,
         provider,
       );
-      if (candidatePlan.promptTokens <= inputBudgetTokens) {
+      if (candidatePlan.promptTokens <= segmentTargetTokens) {
         currentIndices = candidateIndices;
         currentPlan = candidatePlan;
         continue;
@@ -903,9 +948,9 @@ export class CompactionCoordinator {
         event,
         provider,
       );
-      if (atomicPlan.promptTokens > inputBudgetTokens) {
+      if (atomicPlan.promptTokens > requestInputLimitTokens) {
         throw new Error(
-          `Compaction source contains an indivisible atomic group above the model input budget (promptTokens=${atomicPlan.promptTokens}; inputBudgetTokens=${inputBudgetTokens})`,
+          `Compaction source contains an indivisible atomic group above the request input limit (promptTokens=${atomicPlan.promptTokens}; requestInputLimitTokens=${requestInputLimitTokens})`,
         );
       }
       currentIndices = [...group.messageIndices];
@@ -974,7 +1019,8 @@ export class CompactionCoordinator {
     modelObject: Model<Api>;
     readFiles: readonly string[];
     modifiedFiles: readonly string[];
-    inputBudgetTokens: number;
+    requestInputLimitTokens: number;
+    operationInputBudget: CompactionOperationInputBudget;
     nextId: () => string;
     createdNodes: EmbeddedSummaryNode[];
     usages: GeneratedSummary["usage"][];
@@ -1003,13 +1049,13 @@ export class CompactionCoordinator {
           input.readFiles,
           input.modifiedFiles,
         );
-        if (candidatePlan.promptTokens <= input.inputBudgetTokens) {
+        if (candidatePlan.promptTokens <= input.requestInputLimitTokens) {
           current = candidate;
           continue;
         }
         if (current.length === 1) {
           throw new Error(
-            `Compaction child summaries cannot be aggregated within the model input budget (inputBudgetTokens=${input.inputBudgetTokens})`,
+            `Compaction child summaries cannot be aggregated within the request input limit (requestInputLimitTokens=${input.requestInputLimitTokens})`,
           );
         }
         batches.push(current);
@@ -1034,7 +1080,7 @@ export class CompactionCoordinator {
           input.readFiles,
           input.modifiedFiles,
         );
-        if (plan.promptTokens > input.inputBudgetTokens) {
+        if (plan.promptTokens > input.requestInputLimitTokens) {
           throw new Error("Compaction aggregate prompt exceeded its preflight input budget");
         }
         this.state.aggregateCalls = aggregateCalls;
@@ -1048,6 +1094,9 @@ export class CompactionCoordinator {
           ctx: input.ctx,
           model: input.modelObject,
           thinking: this.dependencies.config.compaction.summary?.thinking,
+          promptTokens: plan.promptTokens,
+          requestInputLimitTokens: input.requestInputLimitTokens,
+          operationInputBudget: input.operationInputBudget,
         });
         input.usages.push(generated.usage);
         const aggregateNode: EmbeddedSummaryNode = {
@@ -1088,7 +1137,15 @@ export class CompactionCoordinator {
     ctx: ExtensionContext;
     model: Model<Api>;
     thinking?: CompactionThinkingLevel;
+    promptTokens: number;
+    requestInputLimitTokens: number;
+    operationInputBudget: CompactionOperationInputBudget;
   }): Promise<GeneratedSummary> {
+    if (input.promptTokens > input.requestInputLimitTokens) {
+      throw new Error(
+        `Compaction ${input.stage} prompt exceeded the request input limit (promptTokens=${input.promptTokens}; requestInputLimitTokens=${input.requestInputLimitTokens})`,
+      );
+    }
     this.state.summaryCalls = (this.state.summaryCalls ?? 0) + 1;
     return generateValidatedSummary({
       ...input,
@@ -1096,6 +1153,16 @@ export class CompactionCoordinator {
       maxSummaryTokens: this.dependencies.config.context.maxSummaryTokens,
       transport: this.dependencies.config.compaction.transport,
       now: this.dependencies.now,
+      onAttempt: (diagnostic) => {
+        const nextInputTokens = input.operationInputBudget.usedTokens + input.promptTokens;
+        if (nextInputTokens > input.operationInputBudget.limitTokens) {
+          throw new Error(
+            `Compaction operation input limit exceeded before ${diagnostic.stage} attempt ${diagnostic.attempt} (nextInputTokens=${nextInputTokens}; maxOperationInputTokens=${input.operationInputBudget.limitTokens})`,
+          );
+        }
+        input.operationInputBudget.usedTokens = nextInputTokens;
+        this.state.operationInputTokens = nextInputTokens;
+      },
       onTransportRetry: (diagnostic) => {
         this.state.transportRetries = (this.state.transportRetries ?? 0) + 1;
         this.dependencies.logger.debug("compaction.transport_retry", { ...diagnostic });
@@ -1205,6 +1272,10 @@ export function defaultCompactionDiagnostics(config: Ds4ContextConfig): Compacti
     validate: config.compaction.validate,
     preserveRecentVerbatim: config.compaction.preserveRecentVerbatim,
     segmentTargetTokens: config.compaction.segmentTargetTokens,
+    maxRequestInputTokens: config.compaction.maxRequestInputTokens
+      ?? DEFAULT_COMPACTION_MAX_REQUEST_INPUT_TOKENS,
+    maxOperationInputTokens: config.compaction.maxOperationInputTokens
+      ?? DEFAULT_COMPACTION_MAX_OPERATION_INPUT_TOKENS,
     phase: "idle",
     proactiveEligible: false,
   };

package/src/pi-adapter/summary-generator.ts

@@ -13,16 +13,16 @@ import {
   type SummaryValidationResult,
 } from "ds4-context-core/compaction/summary-contract";
 
-export const DEFAULT_COMPACTION_TRANSPORT_MAX_ATTEMPTS = 3;
+export const DEFAULT_COMPACTION_TRANSPORT_MAX_ATTEMPTS = 4;
 export const DEFAULT_COMPACTION_TRANSPORT_BASE_DELAY_MS = 2000;
 export const COMPACTION_TRANSPORT_MAX_DELAY_MS = 60_000;
 
 /**
- * Transport retry policy for compaction summary requests: three total attempts
- * (not three retries), 2000 ms base delay, exponential backoff, abort-aware.
+ * Transport retry policy for compaction summary requests: four total attempts
+ * (the initial call plus three retries), 2000 ms base delay, exponential backoff, abort-aware.
  */
 export interface CompactionTransportPolicy {
-  /** Total attempts for transport-classified failures. Default: 3. */
+  /** Total attempts for transport-classified failures. Default: 4. */
   maxAttempts?: number;
   /** Base backoff delay in ms, doubled per attempt. Default: 2000. */
   baseDelayMs?: number;
@@ -54,6 +54,12 @@ export interface CompactionTransportRetryDiagnostic {
   delayMs: number;
 }
 
+export interface CompactionAttemptDiagnostic {
+  stage: "segment" | "aggregate" | "update";
+  attempt: number;
+  maxAttempts: number;
+}
+
 export interface GenerateValidatedSummaryInput {
   stage: "segment" | "aggregate" | "update";
   prompt: string;
@@ -68,9 +74,11 @@ export interface GenerateValidatedSummaryInput {
   model?: Model<Api>;
   /** Reasoning level for the summary request; `off` (default) keeps the pre-existing request shape. */
   thinking?: CompactionThinkingLevel;
-  /** Transport-only retry policy; three total attempts by default. */
+  /** Transport-only retry policy; four total attempts by default. */
   transport?: CompactionTransportPolicy;
   now: () => number;
+  /** Called synchronously before every provider attempt, including retries. */
+  onAttempt?: (diagnostic: CompactionAttemptDiagnostic) => void;
   onTransportRetry?: (diagnostic: CompactionTransportRetryDiagnostic) => void;
 }
 
@@ -201,6 +209,7 @@ export async function generateValidatedSummary(
   for (;;) {
     if (input.event.signal.aborted) throw abortedError();
     attempt++;
+    input.onAttempt?.({ stage: input.stage, attempt, maxAttempts });
     try {
       response = await input.ctx.modelRegistry.complete(
         model,

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.3.7";
+export const EXTENSION_VERSION = "0.3.9";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";