ds4-context-engine 0.3.3 → 0.3.5

package/README.md

@@ -16,7 +16,9 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** Stable `0.2.0` includes M0–M20 and frozen 0.2 contracts. Published prerelease `0.3.0-beta.2` carries forward beta.1's bounded Context Manifest storage, cooperative database leases, metadata-only storage diagnostics, and explicit offline inspect/compact/recover maintenance after successful live maintenance validation. Canonical records, SQLite schema 15, and runtime contracts remain unchanged. npm `beta` points to `0.3.0-beta.2`, `alpha` remains `0.3.0-alpha.5`, `latest` remains `0.2.0`, and the maintenance line targets Pi `0.84.3`.
+> **Project status:** The coordinated `0.3.5` release adds bounded compaction optimizations: one validated previous-summary/source update when the complete prompt fits, a dedicated calibrated input budget, up to two concurrent segments, and metadata-only phase timings in `/context compaction`. Canonical history, SQLite schema 15 and runtime contracts remain unchanged; Pi remains pinned to `0.84.3`. See the [0.3.5 release record](docs/releases/0.3.5.md) for validation and publication status.
+
+**New compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="summary"`, `compaction.maxConcurrentSegments=2`. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls) for the legacy-path settings. No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
 ## Why DS4
 
@@ -87,12 +89,12 @@ Install the latest stable public npm package with:
 pi install npm:ds4-context-engine
 ```
 
-The stable `0.2.0` packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) use the same exact version. Both adapters require the matching core version.
+The three packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) are released together with the same exact version. Both adapters require the matching core version. See [0.3.5](docs/releases/0.3.5.md) for this release's changes and compatibility.
 
 To dogfood the beta without replacing a global stable installation, pin the exact version in a disposable project:
 
 ```bash
-pi install -l npm:ds4-context-engine@0.3.0-beta.2
+pi install -l npm:ds4-context-engine@0.3.0-beta.3
 ```
 
 Follow the [0.3 beta dogfooding runbook](docs/DOGFOODING_0.3.0_BETA.md); use synthetic data and a dedicated session directory.
@@ -209,7 +211,7 @@ Valid privacy classifications are `normal`, `internal`, `sensitive` and `local-o
 
 ### LLM-callable tools
 
-DS4 registers two model-callable tools:
+DS4 registers two model-callable tools by default:
 
 | Tool | Purpose |
 | --- | --- |
@@ -220,6 +222,24 @@ DS4 registers two model-callable tools:
 
 Canonical Pin and Memory changes append Pi custom entries and reconcile disposable SQLite projections. Project-memory source include/exclude is derived local SQLite policy and never appends a fake canonical entry. See [`docs/CONTEXT_PERSISTENCE_TOOL.md`](docs/CONTEXT_PERSISTENCE_TOOL.md).
 
+**Optional anchored editing:** `/context config set editing.anchored true`, then
+`/reload`, enables a same-name `edit` wrapper. Use `head[upto]tail` in `oldText` to
+replace an inclusive range without repeating its intermediate old content. Exact
+anchors are validated inside Pi's native mutation queue; ordinary calls and diff
+feedback remain native. All edits in a batch containing anchors must match exactly.
+Use per-edit `literal: true` for real marker text.
+Disabled by default; no inference-time forcer or provider change is involved.
+See [`docs/ANCHORED_EDITING.md`](docs/ANCHORED_EDITING.md) for semantics and limits.
+
+**Other optional agent tools:** `editing.postEditReport` adds bounded old/new line
+ranges and updated context; `reading.adaptive` chooses model-window-aware default
+read limits; `artifacts.adaptiveBudget` lowers inline/excerpt caps under context
+pressure; `jobs.enabled` exposes confirmed, session-owned local `bash_job`
+start/status/stop/list operations. All default off and require session reload after
+configuration changes. Jobs are a separate module, not a replacement for `bash`.
+See [`docs/PORTABLE_AGENT_TOOLS.md`](docs/PORTABLE_AGENT_TOOLS.md) for activation,
+limits, privacy and lifecycle behavior.
+
 Learned-ranking feedback and local training are explicit:
 
 ```text
@@ -250,6 +270,16 @@ The following example shows the main configuration groups. Omitted values use th
     "maxProjectTokens": 20000,
     "maxSummaryTokens": 12000
   },
+  "editing": {
+    "anchored": false,
+    "postEditReport": false
+  },
+  "reading": {
+    "adaptive": false
+  },
+  "jobs": {
+    "enabled": false
+  },
   "retrieval": {
     "exact": true,
     "fts": true,
@@ -287,6 +317,7 @@ The following example shows the main configuration groups. Omitted values use th
   },
   "artifacts": {
     "enabled": true,
+    "adaptiveBudget": false,
     "maxInlineToolResultChars": 12000,
     "maxArtifactBytes": 100000000,
     "maxSearchBytes": 50000000,
@@ -299,7 +330,10 @@ The following example shows the main configuration groups. Omitted values use th
     "mode": "hierarchical",
     "validate": true,
     "segmentTargetTokens": 30000,
-    "preserveRecentVerbatim": true
+    "preserveRecentVerbatim": true,
+    "directUpdate": true,
+    "inputBudget": "summary",
+    "maxConcurrentSegments": 2
   },
   "privacy": {
     "enabled": false,
@@ -477,8 +511,15 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.3.5 release notes](docs/releases/0.3.5.md)
+- [0.3.4 release notes](docs/releases/0.3.4.md)
+- [0.3.3 release notes](docs/releases/0.3.3.md)
+- [0.3.2 release notes](docs/releases/0.3.2.md)
+- [0.3.1 release notes](docs/releases/0.3.1.md)
+- [0.3.0 release notes](docs/releases/0.3.0.md)
 - [0.2.0 release notes](docs/releases/0.2.0.md)
 - [0.2.0-rc.1 release notes](docs/releases/0.2.0-rc.1.md)
+- [0.3.0-beta.3 prerelease notes](docs/releases/0.3.0-beta.3.md)
 - [0.3.0-beta.2 prerelease notes](docs/releases/0.3.0-beta.2.md)
 - [0.3.0-beta.1 prerelease notes](docs/releases/0.3.0-beta.1.md)
 - [0.3.0-alpha.5 prerelease notes](docs/releases/0.3.0-alpha.5.md)
@@ -493,7 +534,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. Published prerelease `0.3.0-beta.2` carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md) hardening, bounded persisted manifests, per-profile calibration retention, cooperative client leases, storage diagnostics, and recoverable offline maintenance, with successful live maintenance evidence from beta.1. Confirmation, exact targeting, strict summary grounding, Pi fallback, and stable canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.3.5 adds bounded compaction updates, summary input headroom, concurrent segments and phase timings. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ADR/059-optional-anchored-editing.md

@@ -0,0 +1,51 @@
+# ADR 059 — Optional execution-time anchored editing
+
+Status: Accepted
+
+## Context
+
+DS4's standalone agent supports `head[upto]tail` replacement and an optional
+inference-time marker forcer. The former is portable tool semantics; the latter
+requires backend control over generation state. Pi already provides batch edits,
+mutation serialization, normalization, cancellation and diff feedback.
+
+## Decision
+
+- Add the default-off `editing.anchored` configuration key, gated by the DS4 master
+  switch and loaded through existing trusted configuration handling.
+- Register a same-name `edit` override only when enabled and native edit is not
+  already replaced. Do not introduce an additional tool or force tool activation.
+- Support one marker between nonempty, non-whitespace exact anchors, inclusive
+  replacement and suffix-only tail uniqueness. Add per-edit `literal: true` to
+  escape real marker text. Preserve native edit behavior without marker syntax.
+- Expand anchors using private copies inside native `EditOperations.readFile`,
+  under Pi's own mutation queue, then delegate batch validation and writing to
+  `createEditToolDefinition`. Do not copy Pi's editing implementation or expand in
+  an unqueued `tool_call` hook.
+- Require exact matching for all edits in a batch containing anchors: native
+  fuzzy fallback changes the whole batch's coordinate space and can relocate an
+  exact anchored span. Marker-free calls retain native fuzzy behavior.
+- Keep native normalized ambiguity checks, even when stricter than exact anchor
+  validation. Invalid anchored edits fail closed, without a fallback to a broader
+  match. ADR 008's context-planning fallback is not permission to guess file edits.
+- Preserve native diff results and append compact original-line range metadata.
+  Skip marker-unaware native previews until the actual result diff is available.
+- Restore the native definition if the same extension instance later loads
+  disabled configuration; a fresh disabled load registers no editing override.
+
+## Consequences
+
+No provider change, Pi fork or dependency upgrade is required for execution-time
+anchors. Only the configuration shape enters the portable core; filesystem/tool
+integration stays in the Pi extension. Existing configuration defaults otherwise
+remain unchanged, and the additive compatibility golden is updated.
+
+The adapter relies on native edit reading through the supplied operations before
+matching the provided edit array, while holding its shared queue. Tests against
+Pi 0.84.3 pin that ordering and cancellation behavior. Dependency upgrades must
+re-run these contracts. Other edit overrides, external editors/processes, atomic
+writes and backend generation-state control remain outside the guarantee.
+
+See [usage and limitations](../ANCHORED_EDITING.md). No automatic marker forcer,
+constrained grammar, live configuration activation or measured provider savings
+are part of this decision.

package/docs/ADR/060-optional-portable-agent-tools.md

@@ -0,0 +1,38 @@
+# ADR 060 — Optional portable DS4 agent improvements
+
+Status: accepted and implemented; included in the coordinated 0.3.4 release. Backend integration remains excluded.
+
+## Boundary
+
+Pi remains the agent runtime and JSONL remains canonical history. These changes add optional tool behavior, not another agent loop. No inference rewind, forced sampling, native KV handles, database migration or dependency upgrade is included.
+
+Four independently default-off switches, also gated by `enabled`:
+
+- `editing.postEditReport`: bounded report from the actual native unified patch, including changed old/new ranges, line delta and post-edit context. Works independently of `editing.anchored`.
+- `reading.adaptive`: choose the default `read` limit from the execution-time model window (120 / 240 / 500 lines at <=8192 / <=16384 / larger windows). Explicit limits, images, cancellation and native byte caps remain native. No shared `more` cursor.
+- `artifacts.adaptiveBudget`: lower inline/excerpt limits according to the calibrated context budget after output/safety reserves and estimated fixed overhead. Never increase configured limits. Preserve minimum reference metadata, canonical results, provenance, privacy, atomic tool groups and normal planner fallback. This is an estimate, not a guaranteed fit or reentrant compaction trigger.
+- `jobs.enabled`: opt-in `bash_job` module in this repository, separate from the portable core and existing `bash`. A single start/status/stop/list tool manages local jobs through Pi's public local BashOperations.
+
+## Editing and reading
+
+Do not take over missing or visibly overridden native tools. Restore native definitions when an already-enabled instance loads disabled configuration. Fresh disabled loads register no override. Render the real native diff; report failures must not turn a successful write into an apparent failure. Limit report regions, context and escaped text; never re-read a potentially changed file for reporting. Retain ADR 059's strict mixed-anchor batch behavior.
+
+## Adaptive artifacts
+
+Use provider-facing privacy-sanitized messages only, and the same calibrated model budget as the planner. Keep all configured storage/search caps. Share the available text budget among candidate results, with a metadata floor bounded by the configured maximum. Missing/invalid budget data keeps static behavior. Rebuild uses the conservative adaptive floor so artifacts from earlier small-budget contexts remain reconstructible. No policy values or canonical messages are mutated. Do not replace a result with a larger estimated result.
+
+## Local jobs
+
+- Require an active built-in local `bash`, no conflicting `bash_job`, explicit configuration, project trust, and local UI confirmation for every start. No UI means no start. This separate tool does not inherit other extensions' bash-only permission policies or SDK shell options.
+- Use opaque job IDs, not arbitrary PIDs. Owner is the current session plus the originating branch entry. Status/stop/list never cross that boundary.
+- At most 4 concurrent jobs and 16 retained records. Default timeout 300 seconds, hard maximum 3600 seconds. Cap each log at 8 MiB and stop on overflow or log-write failure.
+- Logs are raw local files in a private temporary directory, not SQLite state. Responses contain bounded quoted head/tail excerpts and a local output path. Keep completed jobs until bounded eviction or session cleanup; explicitly report output truncation and stop reason.
+- Cancellation while start is pending stops the new job. Once start returns, the job has an independent lifetime; cancelling status does not stop it. Stop and shutdown use native process-tree cancellation. No promise of rollback, daemon ownership or restart survival.
+- Stop jobs and remove temporary logs on session replacement, reload and shutdown. Stop jobs made invisible by branch navigation. Compaction preserves processes and appends a bounded canonical metadata snapshot without launching another model turn; status must be refreshed before acting on old snapshots.
+- Completion marks project indexing dirty. Never launch automatic follow-up turns.
+
+## Verification
+
+Cover independent opt-ins/default-off paths, legacy missing config, argument mutation, native matching and line endings, explicit read limits/images/model changes, estimated budgets/provenance/privacy/rebuild, process caps/timeouts/aborts/ownership/cleanup/compaction, and clean-consumer packaging. Linux process tests do not establish successful Windows execution; the native Pi backend owns platform-specific process handling.
+
+Implementation validation: `npm run check` passed 78 files / 485 tests, including real local bash execution, stop and timeout. Quality comparison and persistence schema gates passed. Clean-consumer packaging passed with Pi 0.84.3 and eight real registry scenarios (core 231 files, reference adapter 7, engine 83). Packaging used a staging copy with the canonical root package manifest; pre-existing user `package.json` changes and `.serena/` were excluded. No provider-token/latency benchmark or Windows execution claim is made. That implementation validation preceded release preparation. See the [0.3.4 release record](../releases/0.3.4.md) for final release gates and publication evidence.

package/docs/ADR/061-compaction-latency.md

@@ -0,0 +1,21 @@
+# ADR-061 — Bounded compaction latency optimizations
+
+Status: accepted and implemented; coordinated 0.3.5 publication explicitly authorized by the user on 2026-09-05. See the [release record](../releases/0.3.5.md) for validation and publication evidence.
+
+## Context
+
+Pi normally updates previous summary plus discarded messages in one request. DS4 previously always summarized new segments first and then generated an aggregate when a predecessor existed. Independent segments ran sequentially and summary inputs reused the ordinary context fill target. This can add avoidable calls; provider latency and summary quality still require controlled real-provider comparisons.
+
+## Decision
+
+- Add `compaction.directUpdate` (default true): estimate the complete sanitized update prompt, including previous summary, cumulative file evidence, custom instructions, split-turn instructions and output contract. If it fits, generate and validate one immutable `task-state` node with the predecessor as a child, canonical source IDs from both spans, and a hash bound to both the new source and predecessor identity/hash/content. With no predecessor, the existing one-segment path remains sufficient. Oversized updates use bounded hierarchical segmentation/aggregation, not truncation.
+- Add `compaction.inputBudget` (`summary` default, or `context`). Summary mode uses the calibrated hard input limit instead of the ordinary active/fill target. Additionally bound input by model context minus safety margin and the actual summary output cap, converted to estimator units. Do not raise model/configured hard limits, remove calibration, reduce safety margins, or reuse ordinary context space reserved for output. Context mode preserves the old active target (also respecting actual summary output headroom).
+- Add `compaction.maxConcurrentSegments` (default 2, range 1–2). Only independent segment requests overlap. Allocate identities and assemble graph/usage in source order, not completion order. On failure stop scheduling, abort siblings, await their settlement before fallback, and persist no partial graph. Cancellation remains cooperative; provider work already accepted may still be billed. Aggregation remains ordered and bounded.
+- Expose process-local metadata-only timings for preparation, segment/direct generation, aggregation, graph preparation/persistence and total DS4 hook duration, plus chosen path, effective provider/model, direct prompt size and concurrency. Timings use a monotonic clock, include retries and local validation within generation, and are wall times (not sums of overlapping calls). They are not canonical evidence, are not persisted in JSONL, and do not measure a subsequent native Pi fallback.
+- Preserve schema-v2 compaction metadata, the existing `task-state` kind, source/classification/validation contracts, Pi cut points, atomic tool exchanges, fresh routing IDs, retry policy, canonical JSONL and all-or-nothing graph preparation. `segmentSummaryId` refers to the update node itself on a direct update; do not invent a segment that was never generated.
+
+## Compatibility and validation
+
+Configuration is additive; absent fields use the new defaults. The old scheduling path can be compared using `directUpdate=false`, `inputBudget=context`, and `maxConcurrentSegments=1`. The compaction master switch still delegates to Pi when disabled. The original local implementation excluded versioning and publication; the user subsequently authorized the coordinated 0.3.5 release. No dependency upgrade, live database maintenance or Pi upgrade is part of this change.
+
+Cover exact budget boundaries and calibration/output reserve, predecessor-only exact evidence and privacy, native predecessor warnings, source ordering/provenance/hash/rebuild, bounded concurrency and out-of-order completion, cancellation/failure/transport retry drainage, and no graph persistence before full success. Retain hierarchical regression coverage explicitly even where default routing now chooses a direct update. Automated call counts and synchronization tests establish scheduling behavior, not a promise of 40-second compactions or unchanged semantic quality on real provider outputs.

package/docs/ADR/README.md

@@ -62,5 +62,8 @@ The initial decisions from the development plan are accepted:
 | 056 | Keep project-memory source exclusion as disposable derived SQLite policy | Accepted |
 | 057 | Derive mutation provenance from the active branch and exclude model-supplied source IDs from V1 | Accepted |
 | [058](058-bounded-manifest-storage.md) | Bound persisted manifests and defer compression to a versioned migration | Accepted |
+| [059](059-optional-anchored-editing.md) | Opt in to anchored edit expansion inside Pi's native mutation queue | Accepted |
+| [060](060-optional-portable-agent-tools.md) | Opt in to edit reports, adaptive reads/results and session-owned local jobs | Accepted |
+| [061](061-compaction-latency.md) | Bound compaction update calls, input budgets, concurrent segments and phase timings | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/ANCHORED_EDITING.md

@@ -0,0 +1,144 @@
+# Optional anchored editing
+
+DS4 can replace Pi's native `edit` definition with a compatible, opt-in wrapper.
+This is execution-time editing, not a generation-time forcer. It works with any
+provider that can call Pi tools; it does not require a custom inference backend.
+
+## Enable or disable
+
+Default: `editing.anchored: false`. In a trusted project:
+
+```text
+/context config set editing.anchored true
+/reload
+```
+
+Use `--global` on the config command to target the agent-directory configuration.
+As with other DS4 config writes, the active session configuration is not changed
+by `set`/`unset`; it is loaded on session startup/replacement or reload.
+
+When updating a development checkout, rebuild the core (`npm run build:core`)
+and fully restart Pi. Reloading extension TypeScript can retain previously loaded
+compiled core modules. A core configuration without `editing` safely leaves this
+feature disabled, but the updated core must be loaded to recognize the option.
+
+```json
+{
+  "editing": { "anchored": true }
+}
+```
+
+`enabled: false` also disables this override. `context.mode` still controls context
+planning, not explicitly opted-in editing. Project configuration is ignored when
+Pi does not trust the project; global configuration continues to apply.
+
+On a fresh disabled load DS4 does not register `edit` at all. If an enabled
+extension instance later loads disabled configuration, it restores Pi's native
+edit definition, schema and guidelines. Pi has no public `unregisterTool`; a
+reload while disabled removes DS4's registration entirely.
+
+Registration is skipped if native `edit` is unavailable or already owned by
+another extension/SDK tool, with a UI warning when available. DS4 does not enable
+an inactive native edit tool. Only ownership reported by Pi can be checked: SDK
+custom base tools labeled as built-ins cannot be distinguished through this API.
+Do not enable this feature with other edit replacements or remote/sandbox edit
+implementations; the registered wrapper uses local filesystem operations.
+
+## Tool input
+
+The tool name remains `edit`, with `path` and `edits[]`:
+
+```json
+{
+  "path": "src/example.ts",
+  "edits": [{
+    "oldText": "function oldImplementation() {\n[upto]\n}\n// end oldImplementation\n",
+    "newText": "function oldImplementation() {\n  return improved();\n}\n// end oldImplementation\n"
+  }]
+}
+```
+
+Each anchored `oldText` must contain exactly one `[upto]` marker:
+
+- The **head**, before the marker, must occur exactly once in the original file.
+- The **tail**, after the marker, must occur exactly once after the end of the
+  head. The same tail occurring before the head is allowed.
+- Both anchors must contain non-whitespace text. Spaces are significant.
+- Newlines immediately after the marker are separators and are removed from the
+  tail needle, matching DS4's marker-line convention. Other whitespace is not
+  trimmed. Anchors may be inline; full distinctive lines are usually clearer.
+- Replacement is **inclusive**: head, intermediate content and tail are all
+  replaced. Put anchors in `newText` if they should be retained.
+- All edits refer to the original file, not previous replacements in the batch.
+  Overlaps (including mixed ordinary/anchored overlaps), missing or ambiguous
+  anchors and no-op batches are rejected without a write.
+- No 64-byte/two-line threshold applies to manually supplied anchors; those DS4
+  thresholds belong to automatic marker forcing, which is not implemented here.
+
+A marker is special only in `oldText`, never in `newText`. To match literal marker
+text, set `literal: true` **on that edit**:
+
+```json
+{
+  "path": "notes.md",
+  "edits": [{ "oldText": "[upto]", "newText": "range marker", "literal": true }]
+}
+```
+
+Calls without anchored edits use native Pi matching, including its normalized
+fuzzy fallback (also for `literal: true` edits). **Every edit in a batch containing
+anchors must match exactly**, ordinary edits included. Pi's fuzzy fallback changes
+the coordinate space for the entire batch and can relocate an otherwise exact
+anchored span; mixed fuzzy batches therefore fail closed. Re-read and correct the
+ordinary oldText, or perform the ordinary edit separately and re-read before
+anchored edits.
+
+Exact matching uses BOM-free, LF-normalized original content. Expanded spans then
+pass through native batch checks, including Pi's stricter normalized-ambiguity
+check. Thus a range can be rejected even if its literal anchors pass, when the
+expanded text is ambiguous under Pi's normalization. Re-read and disambiguate;
+do not silently broaden the range.
+
+## Execution and feedback
+
+`src/extension/anchored-edit-tool.ts` uses public Pi `createEditToolDefinition` and
+`EditOperations`. Its per-call read operation expands anchors only after native
+edit has entered `withFileMutationQueue`. Native edit then performs batch matching,
+overlap checks, cancellation checks, BOM/line-ending restoration, writing and
+diff generation. It is deliberately **not** a `tool_call` preflight expansion:
+there is no unqueued read followed by a later queued write.
+
+Expansion uses private argument copies: the full old span is not inserted into
+canonical tool-call arguments or sent back as an expanded input. Native results
+and diff/patch details remain available, plus a compact text report of anchored
+ranges in original line numbers. Canonical result details can still contain old
+text through native diffs; this is not an output privacy/filtering feature.
+
+The native TUI matcher cannot preview marker syntax. Anchored calls therefore
+skip its pre-execution preview and display the **actual native result diff** after
+execution. Ordinary edits retain their native preview behavior.
+
+The shared Pi queue serializes cooperating edits/writes in the same runtime,
+including symlink aliases. It is not a cross-process file lock, transaction with
+external editors, atomic filesystem write or rollback guarantee. Cancellation
+retains native behavior; once a write has begun it may still complete. Existing
+`tool_call`/`tool_result` hooks continue to see the `edit` tool, not a new tool name.
+Hooks that interpret `oldText` themselves must understand the opt-in syntax.
+
+## Scope and verification
+
+The implementation targets the project's Pi **0.84.3** dependency. No dependency
+upgrade, provider grammar, streaming interception, KV rewind or forced continuation
+is included. The reference adapter does not implement this Pi editing capability.
+
+Tests cover exact matching, suffix-only tail uniqueness, overlapping matches,
+malformed markers, literal escaping, mixed batches, fuzzy-fallback compatibility,
+BOM/CRLF/Unicode, unchanged arguments, result diffs, cancellation, queue ordering,
+symlinks, permissions, working-directory selection and session-bound registration.
+The package smoke check also inspects the real Pi tool registry offline with the
+flag absent, enabled, master-disabled and enabled while native edit is inactive
+or unavailable.
+A deterministic large-block fixture compares serialized request bytes and verifies
+the same output as native editing. Real-provider reliability, token usage and
+latency improvements have **not** been measured; shorter oldText can save generated
+text, while the opt-in tool schema/instructions also add prompt overhead.

package/docs/ARCHITECTURE.md

@@ -165,6 +165,44 @@ Pin and Memory mutations resolve provenance from the active Pi branch, revalidat
 
 Historical tool arguments and results are a provider-egress surface even when general privacy is disabled. A dedicated guard removes content, query, key, reason, paths and raw errors while preserving only action linkage and safe IDs/revisions. Provenance IDs and source paths are runtime-derived; the model cannot supply them in the V1 schema. Post-append failures distinguish an indeterminate append from a known canonical commit with pending projection, so callers are never encouraged to retry blindly.
 
+## Optional anchored editing
+
+After trusted configuration loads at `session_start`, `editing.anchored` can
+register a same-name native `edit` wrapper (default off, also gated by `enabled`).
+The Pi extension resolves exact `head[upto]tail` ranges inside the native edit read
+operation, under Pi's shared file mutation queue, then delegates batch checks,
+cancellation, BOM/EOL handling, writing and diffs to native edit. Expansion never
+rewrites canonical tool-call arguments. Calls without anchors retain native fuzzy
+matching; every edit in an anchored batch must match exactly to prevent native
+fuzzy rematching from moving its ranges. Invalid anchored edits fail closed.
+No inference-state control is involved.
+See [anchored editing](ANCHORED_EDITING.md) and [ADR 059](ADR/059-optional-anchored-editing.md).
+
+## Other optional portable agent tools
+
+`editing.postEditReport` derives bounded old/new coordinates, line deltas and
+updated context from the actual native patch, independently of anchored editing.
+`reading.adaptive` wraps native read with execution-time model-aware default line
+limits; explicit limits and image/byte handling stay native. Both wrappers restore
+native definitions when disabled and do not initially claim other tool overrides.
+
+`artifacts.adaptiveBudget` is a pure portable-core policy applied per context to
+privacy-prepared messages. It uses the calibrated planner budget and estimated
+fixed overhead to lower inline/excerpt caps, never raises configured limits, and
+preserves source identities and canonical history. Rebuild uses a conservative
+floor to reconstruct earlier adaptive references. Normal planner fallback remains
+responsible for oversized mandatory context.
+
+`src/tools/bash-job-manager.ts` and `src/extension/bash-job-tool.ts` form a separate
+optional local job module. It delegates process execution/cancellation to Pi's
+public BashOperations, limits concurrency/log storage, enforces session/branch
+ownership and requires local confirmation for starts. Jobs survive compaction but
+not session replacement/reload/shutdown; compaction appends bounded metadata only.
+There is no new agent loop, SQLite state, automatic model turn or backend KV control.
+
+All new switches are default-off and master-gated. See [portable agent tools](PORTABLE_AGENT_TOOLS.md)
+and [ADR 060](ADR/060-optional-portable-agent-tools.md).
+
 ## Boundaries
 
 Dependency direction is one-way:

package/docs/COMPACTION.md

@@ -6,14 +6,14 @@ DS4 intercepts Pi's `session_before_compact` event but preserves Pi's cut-point
 
 1. Pi determines `messagesToSummarize`, optional split-turn prefix, and `firstKeptEntryId`.
 2. DS4 maps every source message by exact fingerprint to a canonical branch entry ID.
-3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, custom instructions, and file paths for the active provider.
-4. DS4 estimates the complete sanitized summary request against the calibrated active-model input budget. If the whole request does not fit, it partitions the source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results; an exceptionally large turn may split only between such groups.
-5. The active model generates and validates each immutable segment summary only against that segment's sanitized evidence, with cache retention disabled and a fresh routing session ID per call. Only failures categorized as `transport` are retried: at most two retries follow bounded 200 ms and 500 ms delays, each with another fresh routing session ID. The delay and the next call both honor Pi's abort signal; all other failure categories fall back immediately.
-6. DS4 recursively aggregates ordered child summaries in bounded fan-in calls until one root remains. The ordered roots include the previous branch summary, when present, followed by every new segment. DS4-generated IDs, hashes, kinds, and graph levels remain outside model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
-7. The highest input classification is wrapped around each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage is summed across every segment and aggregate request; Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
+3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, previous summary, custom instructions, and file paths for the effective compaction provider (dedicated model when configured and eligible).
+4. DS4 estimates the **complete** sanitized request against a calibrated summary-specific input budget, including framing, instructions, file inventories and output contract. With `compaction.directUpdate` enabled, a previous summary plus new source that fits is updated and validated in **one call**, producing an immutable `task-state` node. No predecessor needs only the existing one-segment call. Oversized updates fall through to hierarchical planning; no additional source is truncated to force a fit.
+5. The hierarchical path partitions oversized source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results. Up to `compaction.maxConcurrentSegments` independent segment requests run concurrently (default 2). Each summary is validated against only its own sanitized evidence. Identities, source/child order and usage accumulation follow source order, not completion order. Cache retention stays disabled and every attempt has a fresh routing session ID.
+6. Hierarchical requests recursively aggregate ordered children until one root remains: previous branch summary first, then new segments. Aggregation stays sequential and budget-checked. Direct updates instead link their single node to the predecessor and new canonical source IDs, without generating a synthetic segment or a separate aggregate. DS4-generated IDs, hashes, kinds, and graph levels never become model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
+7. The highest input classification wraps each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage includes every returned direct/segment/aggregate request and transport replay. Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
 8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the complete prepared batch `failed`.
 
-Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. Transport replay mirrors Pi's assistant retry policy: `compaction.transport.maxAttempts` (default 3) total attempts and `compaction.transport.baseDelayMs` (default 2000 ms, capped at 60 s) backoff, doubling per attempt, abort-aware. Replay never applies to input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run.
+Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. The DS4 transport replay policy is: `compaction.transport.maxAttempts` (default 3) total attempts and `compaction.transport.baseDelayMs` (default 2000 ms, capped at 60 s) backoff, doubling per attempt, abort-aware. Replay never applies to input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run. On segment failure or cancellation DS4 stops scheduling, aborts siblings and awaits all started workers before fallback; no partial graph is installed. Cancellation is cooperative: accepted provider work may still cost tokens, and a provider that ignores abort can delay settlement.
 
 ## Required summary contract
 
@@ -32,7 +32,7 @@ Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, an
 ## Critical Exact Values
 ```
 
-Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. Any unrepaired failure prevents the whole graph batch from being installed.
+Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. A direct update is validated against both the sanitized previous summary and new source, plus cumulative sanitized file evidence; previous-summary-only exact values remain valid evidence. Custom focus instructions are not factual evidence. Any unrepaired failure prevents the whole graph batch from being installed.
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
@@ -49,7 +49,7 @@ An unrepaired exact-value failure reports only the stage, issue code, categorica
 - trigger, split-turn flag, source message count;
 - generation time, provider, and model.
 
-For aggregate compactions, details also embed every non-active node created by that operation. The active node content remains `CompactionEntry.summary`; prior ancestors remain in earlier canonical entries. SQLite stores content, ordered edges, direct/transitive sources, graph level, and lifecycle as a disposable projection. On resume or after deleting the database, DS4 replays Pi entries in append order and recreates the complete graph.
+For aggregate and direct-update compactions, details also embed every non-active node created by that operation. Direct updates reuse the schema-v2 `task-state` kind, set `segmentSummaryId` to their own node ID, increment the predecessor's level, and include the union of previous/new canonical source IDs. Their hash binds the new source hash to predecessor ID, hash, content and level. Prior nodes are never rewritten; no unused segment is fabricated. The active node content remains `CompactionEntry.summary`; prior ancestors remain in earlier canonical entries. SQLite stores content, ordered edges, direct/transitive sources, graph level, and lifecycle as a disposable projection. On resume or after deleting the database, DS4 replays Pi entries in append order and recreates the complete graph.
 
 Memory and pin custom entries do not participate directly in Pi's LLM context and are not replaced by summary text. Their append-only mutations remain in the session tree, so durable decisions and explicit classifications replay after any number of compactions without depending exclusively on a summary.
 
@@ -78,9 +78,31 @@ Semantics:
 - `compaction.summary.thinking` defaults to `off` and applies only to summary requests: `off` keeps the pre-existing request shape (no thinking fields), while other levels map per API (`thinkingEnabled`/`effort` for `anthropic-messages`, `samplingParams.reasoning_effort` for OpenAI-compatible APIs) and are ignored for unsupported providers;
 - `context.maxSummaryTokens` remains a session-level limit and does not rise for the dedicated model; the minimum with the model's `maxTokens` still applies.
 
+## Latency controls
+
+The coordinated `0.3.5` release introduces these additive defaults (absent from `0.3.4`):
+
+```json
+{
+  "compaction": {
+    "directUpdate": true,
+    "inputBudget": "summary",
+    "maxConcurrentSegments": 2
+  }
+}
+```
+
+- `directUpdate`: one validated previous-summary plus new-source request when the entire prompt fits. Set false to always retain the segment-then-aggregate route. Validation or provider failures still fall back to Pi, not an unvalidated update.
+- `inputBudget`: `summary` uses calibrated `hardInputLimit` rather than the ordinary context fill target (`activeInputBudget`). `context` selects that legacy fill target. Both are additionally capped by `(context window - safety margin - actual summary output cap) / calibration ratio`, rounded down, and never exceed the configured/model hard limit. Existing output reservations remain conservative; ordinary session planning and proactive thresholds are unchanged. This reduces avoidable fragmentation, not a guarantee of provider fit or faster processing for a larger prompt.
+- `maxConcurrentSegments`: integer **1–2**, default 2. Only independent segments overlap, including their retries. Aggregates do not run until their children have completed. Use 1 for sequential execution or providers with restrictive concurrent-request limits. Rate-limit failures are not transport-retried.
+
+For an old-path comparison set `directUpdate=false`, `inputBudget=context`, `maxConcurrentSegments=1`. All features remain behind the existing compaction/master switches. Settings are applied on session load; after upgrading the package or rebuilding a development checkout, fully restart Pi to avoid stale compiled-core modules. No schema migration is required.
+
+Mock-provider regression tests verify fewer calls, bounded overlap, exact budget boundaries, validation, privacy, immutable provenance and JSONL rebuild. They do **not** establish real-provider wall-time gains, semantic equivalence of generated summaries, or a guaranteed completion time. See [ADR-061](ADR/061-compaction-latency.md).
+
 ## Transport retry policy
 
-Summary requests are replayed only for transport-classified failures (thrown transport errors or `stopReason: "error"` responses whose message matches network/timeout patterns). The replay policy defaults to Pi's assistant retry policy and can be tuned per deployment:
+Summary requests are replayed only for transport-classified failures (thrown transport errors or `stopReason: "error"` responses whose message matches network/timeout patterns). The DS4 replay policy uses **three total attempts**, not three retries after the initial call, and can be tuned per deployment:
 
 ```json
 {
@@ -93,7 +115,7 @@ Summary requests are replayed only for transport-classified failures (thrown tra
 }
 ```
 
-- `compaction.transport.maxAttempts`: total attempts per segment or aggregate call, integer 1–10, default 3. With 1, no transport failure is retried.
+- `compaction.transport.maxAttempts`: total attempts per direct update, segment or aggregate call, integer 1–10, default 3. With 1, no transport failure is retried.
 - `compaction.transport.baseDelayMs`: base backoff before the first replay, integer 0–60000, default 2000. The delay doubles per attempt (2000, 4000, 8000, …) and is capped at 60 s.
 - Replays use a fresh routing session per attempt; diagnostics expose only stage, failed/next attempt, max attempts, and delay.
 - Aborts (including during backoff) never trigger replay; non-transport failures are never retried; usage is summed across replayed responses.
@@ -117,4 +139,4 @@ It requests compaction at most once per session leaf. Pi's native threshold and
 /context summaries
 ```
 
-The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the calibrated input budget, estimated whole-source prompt size, generated segment count, aggregate-call count, and completed transport-retry count without source content. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.
+The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the effective provider/model, chosen path (`direct-update` or `hierarchical`), budget mode, calibrated input budget, estimated new-source and full-update prompt sizes, logical summary calls (excluding retries), generated segment count, aggregate-call count, concurrency cap, and completed transport-retry count without source content. Monotonic wall timings cover preparation, generation (direct or parallel segments, including validation and retries), aggregation, graph preparation/persistence, and total DS4 hook time. Parallel generation time is elapsed wall time, not a sum of overlapping requests. Total time excludes Pi's subsequent canonical append or native fallback. Timings survive the in-process commit/fallback notification, but are not persisted in JSONL or reconstructed as fake durations after restart. Metadata-only `compaction.timings` is emitted at `debug` for successful and failed attempts. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.