@iowarp/clio-coder 0.3.3 → 0.3.6

package/docs/context-working-set.md

@@ -0,0 +1,194 @@
+# Working Set
+
+The working set is the part of the session ledger the model actually receives on the next request. When context pressure crosses `compaction.threshold`, Clio narrows that view before it considers summarizing anything: selected tool-result bodies and closed-turn thinking blocks stop being replayed, and a one-line marker takes each body's place. Nothing is deleted. The ledger keeps every byte the tools produced, the transcript keeps showing them, and the model can ask for any evicted body back by ref.
+
+Source of truth is `src/domains/context/working-set/` (`contract.ts`, `fold.ts`, `project.ts`, `marker.ts`, `protect.ts`, `engine.ts`, `recall.ts`, `policies/`), the ledger records in `src/domains/session/entries.ts`, and the compaction stage in `src/interactive/turn-context.ts` (`runAutoCompact`).
+
+> [!WARNING]
+> This is an experimental community alpha surface. The default policy is `structural-v1`; `age-horizon` reproduces the selection Clio made before this layer existed and stays available.
+
+## Vocabulary
+
+| Term | Definition |
+| --- | --- |
+| Working set | What the model sees on the next request: the ledger with the current projection applied. It is never a file. |
+| Ledger | The durable append-only session record (`current.jsonl`). The working-set layer appends to it and never rewrites it. |
+| Evicted | A unit whose body the projection replaces with a marker. The ledger entry that holds the original body is untouched. |
+| Offloaded | A result the observation envelope already wrote to a file because it exceeded the per-call cap. Its marker carries the pointer instead of a preview, and recall returns the pointer rather than inlining the file. |
+| Recall | Readmitting an evicted body by ref, through `context(scope="recall", ref=...)` for the model or `/context recall <ref>` for the operator. |
+| Marker | The byte-stable one-line stub the projection renders in place of an evicted body. It names the ref, the reason, the size, and the exact call that brings the body back. |
+| Projection | A pure, in-memory transform from ledger entries to the entries the replay builder hands the model. `projectWorkingSet(entries, view)` is that function. |
+
+## Eviction is a projection, not a rewrite
+
+The stage this layer replaces rewrote history. `maskStaleObservations` walked the entries, replaced observation bodies with a masked-out string, and called `session.replaceEntries`. That destroyed the only copy: after a mask, `/resume`, `/tree`, `/fork`, and the HTML export all showed the placeholder, and the content was gone for the operator as well as the model.
+
+The working-set layer separates the two audiences. What leaves is recorded as a `contextEviction` entry, appended like any other. `refreshAgentMessagesFromSession` folds those entries into a `WorkingSetView` and applies `projectWorkingSet` before `buildReplayAgentMessagesFromTurns` runs, so only the messages bound for the provider carry markers. Every reader that shows the session to a human reads the raw ledger and sees the full bodies.
+
+Three properties follow from that shape:
+
+- **Idempotence.** Projecting an already-projected slice reproduces it byte for byte, because the marker comes from the ledger entry rather than from the body being replaced.
+- **Branch safety.** The fold runs through `filterEntriesToActivePath` (issue #94), so an eviction recorded on a branch `/tree` later abandoned cannot project onto the live one, and a fork inherits the view of its shared prefix.
+- **Determinism.** A policy is a pure function of `PolicyInput`. The same ledger and the same settings select the same units in a live session and in an offline replay of that session.
+
+Usage anchors recorded before an eviction described a longer prompt than the model will now receive, so the projection stamps `contextUsageInvalidated` on assistant entries that precede the newest eviction event. Without that, `calculateContextTokens` would keep reporting the pre-eviction size and the pressure estimator would never see the space the event freed. The stamp is replay bookkeeping, not provider-visible message content, and the prompt estimator deliberately excludes it. That keeps plan-time `tokensAfter` and the post-event cold-prefix metric on the same byte definition.
+
+## Ledger records and format v4
+
+Two entry kinds carry the layer, both defined in `src/domains/session/entries.ts`:
+
+| Kind | Fields | Meaning |
+| --- | --- | --- |
+| `contextEviction` | `policyId`, `trigger` (`pressure` or `operator`), `evicted[]`, `tokensBefore`, `tokensAfter`, `pressureBefore`, `snapshotIdBefore` | One applied event. Each `evicted[]` item is `{ ref, reason, tokensFreed, marker, by? }`. |
+| `contextRecall` | `ref`, `trigger` (`tool` or `operator`), `tokensReadmitted`, `toolCallId?` | One readmission of one ref. It is a churn record, not an un-eviction. |
+
+`reason` is one of `superseded_read`, `stale_after_mutation`, `listing_consumed`, `failure_resolved`, `thinking_turn_closed`, `age_horizon`, `operator`. A `ref` is the `turnId` of the ledger entry that holds the unit: for a `tool_result` message the unit is the result body, and for an `assistant` message it is every thinking block the message carries. Per-block eviction is deliberately not modelled.
+
+Adding those kinds bumps the session format to version 4 (`CURRENT_SESSION_FORMAT_VERSION = 4` in `src/engine/session.ts`). The bump is additive: no existing entry kind changes shape, so a version 3 session migrates to 4 in place when Clio opens it and no entry is rewritten. `runMigrations` refuses only what it cannot read, a session written by a newer build, with "upgrade clio-coder to resume this session". The bump is still one-way for the operator: Clio 0.3.3 does not know these kinds and cannot open a session this release wrote.
+
+## The marker contract
+
+A marker is one line, its fields are in fixed order, and it carries no timestamp and no counter. That is not cosmetic. The marker is persisted inside the `contextEviction` entry and replayed on every subsequent request, so a marker whose bytes drifted between renders would cold-start the provider prefix cache on a turn that evicted nothing new. It would also make two replays of the same recorded ledger disagree.
+
+Field order is `ref`, `reason`, `by`, `tool`, `path`, `size`, `offload`, `recall`, then the body tail. Undefined fields are omitted rather than rendered empty. `path` is the one file the result was about: `details.paths` when the tool recorded exactly one (`edit`, `write`, `artifact`), otherwise the `path` argument of the call as the model wrote it, which is how a `read` marker names its file. Real output from `renderMarker` in `src/domains/context/working-set/marker.ts`:
+
+```text
+[evicted ref=0198f3c2-7a10-7c31-9d44-2b0c5f1e88a3 reason=stale_after_mutation by=0198f3c2-9b02-7f55-8e10-6d21ac9e4471 tool=read path=src/domains/context/working-set/engine.ts size=41 lines/3.8KB recall=context(scope="recall", ref="0198f3c2-7a10-7c31-9d44-2b0c5f1e88a3") preview="export function planEviction(policy: WorkingSetPolicy, input: PolicyInput): EvictionPlan | null { export function planEv"]
+```
+
+```text
+[evicted ref=0198f3c2-1d44-7a90-b201-77c0e1a2f5de reason=failure_resolved by=0198f3c3-0002-7ab1-9c33-14ff90bb2c07 tool=bash size=4 lines/152B recall=context(scope="recall", ref="0198f3c2-1d44-7a90-b201-77c0e1a2f5de") first_line="src/interactive/turn-context.ts(466,15): error TS2345: Argument of type 'PolicyInput' is not assignable to parameter of "]
+```
+
+```text
+[evicted ref=0198f3c4-55aa-7be2-8f01-9a3d6c2b1e77 reason=listing_consumed tool=grep size=1 lines/234.4KB offload=/home/dev/.local/state/clio-coder/offload/0198f3c4-grep.txt recall=context(scope="recall", ref="0198f3c4-55aa-7be2-8f01-9a3d6c2b1e77")]
+```
+
+Three rules govern the tail. Most reasons render `preview`: the first 120 characters of the body, whitespace collapsed and double quotes escaped, so the preview cannot break the quoted field or spill onto a second line. A `failure_resolved` eviction renders `first_line` instead, because the line that says what failed is worth the marker's tokens where a preview of a stack trace is not. An offloaded body renders neither, because the `offload=` pointer already promises the full artifact at a stable path and a preview would spend tokens repeating it.
+
+Thinking eviction renders no marker at all. The reasoning simply stops being replayed. A marker there would spend tokens announcing that something the model cannot act on is gone.
+
+## Policies
+
+A policy answers one question: which units should leave. It never writes, never reads a clock, and never calls a model. `planEviction` then materializes the selection into `EvictedItem`s with markers rendered and tokens measured, and prices the result against the projection the model will actually receive.
+
+### Protection predicates
+
+`protect.ts` runs before every rule in `structural-v1` and is absolute. A policy is allowed to be wrong about relevance; it is not allowed to drop these:
+
+1. Anything that is not a `tool_result` or `assistant` message. Operator words, compaction and branch summaries, skill activations, task ledgers, worker runs, and bash executions are the session's record of itself.
+2. Anything inside the recent window, which starts at `protectionCutoffIndex(entries, protectLastTurns)`. A turn starts at a user message, a `bashExecution`, or a `branchSummary`.
+3. A result whose estimated body is below `minEvictableTokens`. This protects low-yield bodies from churn; the engine independently rejects a marker that would free no tokens.
+4. A body the legacy destructive stage already replaced, which has nothing left to evict.
+5. A call the safety rails blocked. A refused call is a decision the session made, not an observation it can re-fetch.
+6. A write or edit the turn in flight is still standing on.
+7. A failure nothing later resolved, and any unindexed failure, because without an observation there is no way to ask whether it was resolved.
+
+### `age-horizon`
+
+The rule `maskStaleObservations` applied, recorded instead of destroyed. Every `tool_result` body older than the protection horizon leaves the working set, and every `assistant` message older than the horizon loses its thinking blocks. Same turn-start definition, same cutoff, and a body carrying a legacy compaction marker is skipped the same way.
+
+One skip condition is new, so this is today's selection minus small results rather than a byte-identical reproduction of it: a result whose estimated body is below `minEvictableTokens` (200 tokens by default) stays, whatever its age. The engine already rejects markers that save no tokens; the higher default is a measured low-yield churn guard. The old mask had no such floor and masked those results too. Thinking has no size floor either way, because dropping it renders no marker.
+
+`age-horizon` has no target stop. It evicts everything beyond the horizon in one event, exactly as the mask did, and ignores `context.workingSet.target`; the replay tables show this as `saturated events = 1.000` on every row. That is deliberate: the policy exists to reproduce the old selection through the ledger, and an operator who wants batching to a target wants `structural-v1`. Candidates arrive newest-safe-first, so a caller that stops early has evicted the newest safe unit rather than the oldest one.
+
+Age is not a quality signal. A file read twenty turns ago and never touched since is more useful than a directory listing from two turns ago, which is the whole reason `structural-v1` exists and is the default.
+
+### `structural-v1` (default)
+
+Rule order is the policy. Each rung emits candidates newest-first, every candidate passes `isProtected`, and no unit is claimed twice, so a read that is both stale and superseded is evicted for the reason that came first and carries the `by` ref that explains it. The rungs, in order:
+
+| # | Reason | Fires when |
+| --- | --- | --- |
+| 1 | `stale_after_mutation` | A read-class observation is followed by a write or edit of the same file. Whatever the body said is now a claim about a file that no longer exists in that form. |
+| 2 | `superseded_read` | A later successful read of the same file covers this one's lines. A full read covers everything; any other read covers only an identical or containing range, and an unknown range covers nothing. |
+| 3 | `failure_resolved` | A later call succeeded with byte-identical arguments, or, for `read`, `grep`, and `find`, reached the same file by any route. |
+| 4 | `listing_consumed` | Every path the listing surfaced went on to be read. One surfaced path still unread and the listing stays, because that is the path the agent comes back to. |
+| 5 | `thinking_turn_closed` | An assistant message beyond the protection horizon carries thinking blocks. |
+| 6 | `age_horizon` | Only under pressure, and only until the projection reaches `target`. |
+
+Rungs 1 through 5 are unconditional: redundant content is free to drop, whatever the pressure. Rung 6 is the only one that looks at token counts, and it stops the moment the projected size reaches `context.workingSet.target × contextWindow`. Newest-first within a rung is a cost decision: evicting the youngest safe unit keeps the cold region after the eviction point small, so the turn that pays for the event pays least.
+
+The long-trace sweep found that targets 0.4 and an exhaustive rung 6 produced identical results because the usable candidate pool ran out first. Relative to the 0.6 default, 0.4 reduced cold-prefix tokens by 2.8% at 64k and 7.3% at 128k, did not reduce summaries, and lowered retention covered by 0.00072 at 128k. The default therefore remains 0.6. The replay README records the full grid and the numeric reopening rule.
+
+The facts the rungs read come from `path-index.ts`, one deterministic pass over the active-path entries producing one observation per tool result that names a path: which file, which line range, which paths a listing surfaced, whether the call failed, and where in the turn sequence it sits. Tools that observe no path (dispatch, web fetch, tasks, ask user, context) produce no observation. There are no content fingerprints.
+
+## Recall
+
+Recall is explicit and by ref. There is no auto-readmission: the marker tells the model exactly which call brings the body back, and the model decides.
+
+`resolveRecall(entries, view, ref, activeLeafTurnId)` resolves a ref against the fold at the live leaf and returns the original body byte-exact, read with the same field precedence the projection would have used. It fails in three typed ways, and each message names the nearest valid ref when one exists:
+
+- `invalid_ref` when the ref is empty or carries whitespace.
+- `not_on_active_path` when the session has no such turn on this branch, which includes a ref from a branch `/tree` abandoned.
+- `not_evicted` when the unit is still in context. An assistant turn reports separately that thinking is not recallable.
+
+Both messages end with the refs that can be recalled on the active path (tool results only, up to eight, then a count), because a failed recall is usually a mistyped ref and the listing is what the next call needs.
+
+An LLM summary also preserves recall discovery across its cut. When an evicted tool result falls before `firstKeptTurnId`, the generated checkpoint carries a `<recallable-refs>` block with the same `ref (tool path)` rows used by recall failures, bounded to eight rows plus a remaining count. Results that stay after the cut keep their ordinary markers and are not repeated in the block.
+
+**A recall does not un-evict.** The key stays in `view.evicted`, the marker stays byte-identical at its original position, and the recalled body arrives at the tail of the working set inside the recall result. Readmitting it in place would duplicate the bytes and invalidate the provider prefix cache for everything after that point, which costs more than the recall saved.
+
+That also makes recall the churn signal. `churn = recalls / itemsEvicted` over the active path. A high churn number means the policy keeps evicting content the session still needs, which is a reason to change the policy rather than to raise the threshold.
+
+The procedural replay does not synthesize churn from path reuse. Its reference graph maps each earlier observation to every later reread or discovery of the same path, while a real `contextRecall` is an explicit model choice of one ref. A later reread already returns current content at the tail, so also injecting the old body would duplicate data and misread stale or superseded observations as recall demand. Replay reports `recallTokens` as a one-time demand bound per evicted item and waits for explicit `contextRecall` records before reporting recall count, churn, or tail growth. The graph-density measurements and reopening condition are in the replay README.
+
+An offloaded result returns its pointer, never the file. The model gets the same `full: <path>` promise the original tool result ended with and reads it with `read` when it wants it.
+
+The two entry points differ in where the body lands:
+
+| Caller | Entry point | Where the body goes | Ledger record |
+| --- | --- | --- | --- |
+| Model | `context(scope="recall", ref=...)` | Back into the working set through the normal observation envelope, so the per-turn pool and the self cap still apply | `contextRecall` with `trigger: "tool"` and the `toolCallId` |
+| Operator | `/context recall <ref>` | The transcript only. It is never submitted as a turn and never counted against the context window | `contextRecall` with `trigger: "operator"` |
+
+Both publish `BusChannels.ContextRecalled`, and both route through the middleware `on_compaction` hook as stage `working_set_recall`.
+
+## Settings
+
+```yaml
+context:
+  workingSet:
+    enabled: true
+    policy: structural-v1
+    target: 0.6
+    protectLastTurns: 6
+    minEvictableTokens: 200
+```
+
+| Key | Default | Accepted | Meaning |
+| --- | --- | --- | --- |
+| `context.workingSet.enabled` | `true` | boolean | Master switch. `false` skips eviction and goes straight to summary compaction. It does not restore the destructive mask. |
+| `context.workingSet.policy` | `structural-v1` | `age-horizon`, `structural-v1` | Candidate selection rule set. |
+| `context.workingSet.target` | `0.6` | number greater than 0 and less than 1 | Used-over-window ratio an applied `structural-v1` event batches down to. `age-horizon` ignores it. |
+| `context.workingSet.protectLastTurns` | `6` | integer ≥ 1 | Recent turns whose observations and thinking are never evicted. |
+| `context.workingSet.minEvictableTokens` | `200` | integer ≥ 0 | Results below this body estimate are never evicted. The default protects low-yield bodies; marker break-even is enforced separately. |
+
+`compaction.excludeLastTurns` governs only the temporary legacy mask path; working-set protection uses `protectLastTurns`. Settings validation is strict, so an unknown key under this block fails startup with its exact path.
+
+`CLIO_CODER_LEGACY_MASK=1` restores the destructive stale-observation stage for one release as a compatibility escape hatch. It rewrites the ledger, and it is removed in the next release.
+
+## What the operator sees
+
+- **`/context` overlay.** A working-set section under the category legend: the configured policy with its state (`policy structural-v1 · no events yet` until the first event, `disabled` when `context.workingSet.enabled` is off, and `(last event by <policy>)` when the setting changed after an event), evicted item count, evicted tokens, event count, recall count, and churn. Evicted tokens render as one line after the legend rather than as a meter category, because they are outside the window rather than a slice of it.
+- **Transcript.** An evicted tool row keeps its full body and gains a dim `evicted · <reason>` tag. The transcript shows the ledger, never the projection, so `/resume`, `/tree`, `/fork`, and the HTML export are unaffected by eviction.
+- **`/context recall <ref>`.** Prints the ref, why it was evicted, the token count, and the offload pointer when there is one, followed by the original body. Transcript only.
+- **Prompt cache line.** Every applied event stamps `working_set_evict` on the next assistant entry's `promptCache.expectedColdReasons`. When the last settled run came back cold for that reason, the overlay adds `last cold turn: working-set eviction (expected)` and drops the shell-reused-but-backend-cold warning, because the cold turn is explained rather than surprising.
+- **Notice.** One line per applied event: `[context engine] working set: N items evicted by <policy>; ~X -> ~Y tokens, recall by ref with context(scope="recall")`. The numbers are the plan's, priced over the visible ledger slice, and they are the same numbers the `contextEviction` entry, the `[Compaction] Reclaimed context` toast, and the overlay's `last compaction` line carry. The footer meter is a separate live estimate over the agent message list and can differ from them by the tool schemas and replay text it includes.
+
+## Not in this release
+
+These are tracked follow-ups, not available behavior:
+
+- **Auto-readmission.** Nothing brings an evicted body back on its own. There are no path fingerprints and no registry of what the model is likely to need next.
+- **Cost model and deferred scheduling.** Pressure is the only trigger, and it is `compaction.threshold`, not `target`. The replay tables price every applied event by the cold prefix it re-prefills (about 29k tokens per event at a 64k budget), and batching from the threshold down to the target is what keeps one event per cycle; a trigger at the target would make every turn above 60% with one newly redundant read an event of its own, and no row in the sweep shows fewer summaries in return. There is no break-even horizon, no deferred eviction plan, and no piggybacking beyond the fact that the working-set stage already runs first inside `runAutoCompact`.
+- **Intra-turn eviction.** Eviction runs before a request is sent. A single turn whose tool results overflow the window is handled by the observation envelope's caps and by summary compaction, not by this layer.
+- **Worker runtimes.** Dispatched workers replay their own ledgers without the working-set stage.
+- **Digests.** A marker carries tool, size, and a first-line preview. The generated summaries from #165 are not embedded in it.
+
+## See also
+
+- `clio-coder context replay --sessions <path>...` replays Clio ledgers, and `--synthetic <ids>` replays the seeded procedural corpora, through the same fold, projection, and policy code with `none`, `random`, and `oracle` controls; `clio-coder context working-set --session <id|path>` prints one session's fold and path index. Both are described under [Working-set replay](commands-and-modes.md#working-set-replay). Generated replay tables are local artifacts rather than versioned benchmark results.
+- [context-engine.md](context-engine.md) for context window resolution, token accounting, and how this stage sits ahead of summary compaction.
+- [session-lifecycle.md](session-lifecycle.md) for the ledger format, active-path lineage, and branching.
+- [glossary.md](glossary.md) for the one-line definitions of these terms.

package/docs/development-pipeline.md

@@ -80,7 +80,7 @@ New `area:*` labels are proposed in an issue, not created ad hoc.
 
 ## Milestones are releases
 
-Each open milestone is the next version (`v0.3.3`, `v0.4.0`). Triage means
+Each open milestone is the next version (`v0.3.6`, `v0.4.0`). Triage means
 assigning an issue to a milestone or explicitly leaving it in the backlog.
 A release cut requires every issue in its milestone to be closed
 or bumped; the milestone closes when the tag is published.

package/docs/documentation-coverage.md

@@ -1,6 +1,6 @@
 # Clio Coder Documentation Coverage Matrix
 
-This matrix maps every top-level directory in `src/` and every domain directory under `src/domains/` to its authoritative documentation page. It records coverage status (`documented`, `partial`, `undocumented`), missing concepts, and key source contracts for `v0.3.3`.
+This matrix maps every top-level directory in `src/` and every domain directory under `src/domains/` to its authoritative documentation page. It records coverage status (`documented`, `partial`, `undocumented`), missing concepts, and key source contracts for `v0.3.6`.
 
 ## Coverage Matrix
 
@@ -18,9 +18,9 @@ This matrix maps every top-level directory in `src/` and every domain directory
 | `src/domains/agents/` | 12 built-in recipes, agent catalog, recipe schema, fleet commands, fleet contract v4 | [built-in-agents.md](built-in-agents.md), [fleet-dispatch.md](fleet-dispatch.md) | `documented` | Documented in built-in agent recipes guide and fleet dispatch architecture. |
 | `src/domains/components/` | Component scanning, snapshots, hashing, diffing | [middleware-and-components.md](middleware-and-components.md) | `documented` | Documented in active component snapshot and middleware guide. |
 | `src/domains/config/` | Configuration contracts, file watcher, keybinding definitions, setting classifiers | [configuration-and-targets.md](configuration-and-targets.md), [commands-and-modes.md](commands-and-modes.md) | `documented` | Documented in configuration targets and command/keybinding reference. |
-| `src/domains/context/` | `CLIO-CODER.md` bootstrap, codewiki generation, prompt context assembly, project rules | [context-engine.md](context-engine.md) | `documented` | Documented in context window, token accounting, and compaction reference. |
+| `src/domains/context/` | `CLIO-CODER.md` bootstrap, codewiki generation, prompt context assembly, project rules, non-destructive working-set eviction (`age-horizon` and `structural-v1` policies, protection predicates, path index, byte-stable markers, recall by ref) | [context-engine.md](context-engine.md), [context-working-set.md](context-working-set.md) | `documented` | Context window, token accounting, and the three compaction mechanisms in the engine reference; the working-set layer has its own guide covering the vocabulary, both ledger record kinds and format v4, the marker contract, both policies with their rule order, recall semantics, and the operator surfaces. |
 | `src/domains/dispatch/` | Fleet orchestration, assignment store, batch tracker, admission, route planner, receipt integrity v15 | [fleet-dispatch.md](fleet-dispatch.md), [dispatch-architecture-rationale.md](dispatch-architecture-rationale.md), [worker-dispatch-mechanics.md](worker-dispatch-mechanics.md) | `documented` | Multi-node fleet dispatch, admission invariants, and receipt verification fully documented. |
-| `src/domains/eval/` | Suite v2 YAML schema, eval runner, metrics, reporters, workspace sandboxing | [eval-runner.md](eval-runner.md), [evals-internal.md](evals-internal.md) | `documented` | Documented in eval runner and soak benchmark guides. |
+| `src/domains/eval/` | Suite v2 YAML schema, eval runner, metrics, reporters, workspace sandboxing | [eval-runner.md](eval-runner.md), [evals-internal.md](evals-internal.md) | `documented` | Product evals are documented independently from external benchmarks. |
 | `src/domains/evidence/` | Evidence bundles, findings taxonomy, provenance store, failure attribution | [evidence-and-memory.md](evidence-and-memory.md) | `documented` | Documented in evidence directory structures and memory retrieval guide. |
 | `src/domains/evolution/` | Falsifiable Change Manifest JSON templates and `clio-coder evolve` self-edit gates | [evolution.md](evolution.md) | `documented` | Documented in evolution manifest reference and mutation validation rules. |
 | `src/domains/extensions/` | Extension manifest schemas, resource roots, portable share archives | [extensions-and-sharing.md](extensions-and-sharing.md) | `documented` | Documented in extensions and sharing guide. |
@@ -33,14 +33,14 @@ This matrix maps every top-level directory in `src/` and every domain directory
 | `src/domains/resources/` | Skill package discovery, marketplace index resolution, prompt resources | [skills-marketplace.md](skills-marketplace.md), [extensions-and-sharing.md](extensions-and-sharing.md) | `documented` | Skills marketplace, publishing flows, and resource managers documented. |
 | `src/domains/safety/` | Policy engine, action classifiers, damage-control rules, path policies, finish contract, audit log | [safety-model.md](safety-model.md), [scientific-validation.md](scientific-validation.md) | `documented` | Policy evaluation order, 10-step sequence, write containment, and finish contract documented. |
 | `src/domains/scheduling/` | Capacity lease acquisition, heartbeats, expiry, cross-process locks, cluster scheduling | [capacity-and-scheduling.md](capacity-and-scheduling.md), [fleet-dispatch.md](fleet-dispatch.md) | `documented` | Dedicated capacity leasing, heartbeat TTL, and cross-process lock reference. |
-| `src/domains/session/` | Context ledger v3, tree branching (`/tree`), `/fork`, `/resume`, checkpoints, protected-artifact journal | [session-lifecycle.md](session-lifecycle.md) | `documented` | Dedicated session lifecycle guide covering ledger format v3, branching, journal, and recovery. |
+| `src/domains/session/` | Session ledger format v4, tree branching (`/tree`), `/fork`, `/resume`, checkpoints, protected-artifact journal | [session-lifecycle.md](session-lifecycle.md), [context-working-set.md](context-working-set.md) | `documented` | Dedicated session lifecycle guide covering branching, journal, and recovery; the `contextEviction` and `contextRecall` records added at format v4 are specified in the working-set guide. |
 | `src/domains/share/` | Portable share archive bundles, manifest verification, import/export flows | [extensions-and-sharing.md](extensions-and-sharing.md) | `documented` | Share archives and portable bundle formats documented in extensions guide. |
-| `src/domains/webhook/` | Empty directory | None (Inert) | `inert` | Directory contains no active modules or exports in v0.3.3. |
+| `src/domains/webhook/` | Empty directory | None (Inert) | `inert` | Directory contains no active modules or exports in v0.3.6. |
 
 ## Cross-Cutting Reference Guides
 
 In addition to source subsystem mappings, the documentation set includes cross-cutting contracts:
 
 1. [artifact-versions.md](artifact-versions.md): Canonical version registry and migration contract for all 9 serialized artifact schemas across Clio Coder.
-2. [glossary.md](glossary.md): Formal definitions of 40 core architectural concepts mapped to their TypeScript types in `src/`.
+2. [glossary.md](glossary.md): Formal definitions of 45 core architectural concepts mapped to their TypeScript types in `src/`.
 3. [troubleshooting.md](troubleshooting.md): Comprehensive diagnostic and remediation guide keyed by exact user-facing error strings.

package/docs/documentation-guide.md

@@ -1,7 +1,7 @@
 # Documentation Standards and Codebase Alignment
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive documentation link linter, phrasing/claim evaluator, and alignment portal is located at [docs/html/documentation_blueprint.html](html/documentation_blueprint.html) (Version: 0.3.3).
+> **Interactive Spec Available:** An interactive documentation link linter, phrasing/claim evaluator, and alignment portal is located at [docs/html/documentation_blueprint.html](html/documentation_blueprint.html) (Version: 0.3.6).
 
 Clio Coder is an experimental community alpha. Documentation should help contributors and early users work from the source of truth without overstating maturity. When docs drift, prefer the current source and tests over older prose or aspirational roadmap notes.
 
@@ -34,7 +34,8 @@ Classify claims clearly:
 | [README.md](../README.md) | `CHANGELOG.md`, package metadata, release receipts | Product overview, install, first run, alpha framing, and release status. |
 | [docs/README.md](README.md) | This docs directory | Documentation hub. |
 | [commands-and-modes.md](commands-and-modes.md) | `src/cli/index.ts`, `src/cli/args.ts`, `src/interactive/slash-commands.ts`, `src/domains/dispatch/**` | CLI commands, headless run flags (`--session`, `--continue`, `--json-events`), session continuity, `--json` wire projection promise, slash commands, keybindings, live steering. |
-| [context-engine.md](context-engine.md) | `src/domains/context/**`, `src/domains/session/context-accounting.ts`, `src/domains/session/context-ledger.ts`, `src/domains/session/compaction/` | Context window resolution, per-model probe capabilities, token accounting, snapshots, progressive compaction, model-driven `clio-coder context init`, format v3 session enforcement. |
+| [context-engine.md](context-engine.md) | `src/domains/context/**`, `src/domains/session/context-accounting.ts`, `src/domains/session/context-ledger.ts`, `src/domains/session/compaction/` | Context window resolution, per-model probe capabilities, token accounting, snapshots, the three compaction mechanisms, model-driven `clio-coder context init`, format v4 session enforcement. |
+| [context-working-set.md](context-working-set.md) | `src/domains/context/working-set/**`, `src/domains/session/entries.ts`, `src/interactive/turn-context.ts` | Working-set vocabulary, eviction as a projection, the `contextEviction` / `contextRecall` records, the marker contract, the `age-horizon` and `structural-v1` policies, recall semantics, and the operator surfaces. |
 | [architecture.md](architecture.md) | `tests/boundaries/check-boundaries.ts`, `src/core/domain-loader.ts`, `src/engine/**`, `src/worker/**` | Source layout, 5 enforced boundary rules (dependency direction vs import form), runtime flow mermaid diagram, event/audit model, detect-and-rollback write boundaries. |
 | [dispatch-architecture-rationale.md](dispatch-architecture-rationale.md) | `src/domains/dispatch/**`, `tests/boundaries/check-boundaries.ts` | Design rationale, not behavior: invariants that cross the seams a dispatch split would use, what any future split must preserve, the one dispatch→eval import, and the closed barrel-import decision. |
 | [configuration-and-targets.md](configuration-and-targets.md) | `src/core/defaults.ts`, `src/core/config.ts`, `src/domains/providers/**`, `src/cli/configure.ts`, `src/cli/targets.ts`, `src/cli/models.ts`, `src/cli/auth.ts` | TargetDescriptor, contextWindowProvenance (`configured`, `discovered`, `catalog`, `runtime-default`), settings.yaml, strict validation, saved defaults vs live routing. |
@@ -49,22 +50,22 @@ Classify claims clearly:
 | [capacity-and-scheduling.md](capacity-and-scheduling.md) | `src/domains/scheduling/**`, `src/domains/dispatch/capacity-lease.ts`, `src/domains/dispatch/reservation-store.ts` | Multi-process capacity leases (`dispatch-admission.json`), heartbeat TTLs, cross-process transaction locks (`dispatch-admission.json.lock`), and cluster drain controls. |
 | [worker-dispatch-mechanics.md](worker-dispatch-mechanics.md) | `src/worker/**` | NDJSON parent-child socket protocols, control/bulk lane demuxing, watchdog timers, worker attestation (13 protocol fields), permission parking, exit codes. |
 | [fleet-demo-runbook.md](fleet-demo-runbook.md) | `src/domains/dispatch/**` | Multi-node fleet demo: SSH setup, C++ build/repair workflow, reviewer gates, receipt verification v15. |
-| [session-lifecycle.md](session-lifecycle.md) | `src/engine/session.ts`, `src/domains/session/**` | Session lifecycle, on-disk ledger format v3 (`current.jsonl`), tree branching (`tree.json`), active-path lineage selection, `/fork`, `/resume`, checkpoints, and write-ahead protected-artifact journal. |
+| [session-lifecycle.md](session-lifecycle.md) | `src/engine/session.ts`, `src/domains/session/**` | Session lifecycle, on-disk ledger format v4 (`current.jsonl`), tree branching (`tree.json`), active-path lineage selection, `/fork`, `/resume`, checkpoints, and write-ahead protected-artifact journal. |
 | [acp.md](acp.md) | `src/engine/acp/**`, `src/cli/acp.ts` | Agent Client Protocol (ACP) server over stdio, tool mediation, non-stall permission handling, timeout bounds, and error taxonomy. |
 | [artifact-versions.md](artifact-versions.md) | `src/domains/dispatch/receipt-integrity.ts`, `src/engine/session.ts`, `src/worker/spec-contract.ts`, `src/domains/agents/fleet-contract.ts`, `src/domains/eval/schema/`, `src/domains/observability/trace-store.ts` | Version registry and migration policies for all 9 serialized artifact schemas across Clio Coder. |
 | [exit-codes-and-output.md](exit-codes-and-output.md) | `src/cli/**`, `src/entry/**` | Global process exit codes (0, 1, 2, 3), `--help` standard on stdout, machine-readable JSON streaming (`--json`, `--json-events`), and headless stdout deliverable contracts. |
 | [troubleshooting.md](troubleshooting.md) | `src/core/**`, `src/cli/**`, `src/domains/**` | Actionable error remediation and diagnostics keyed by exact user-facing messages. |
-| [glossary.md](glossary.md) | `src/domains/dispatch/types.ts`, `src/tools/**`, `src/domains/agents/**`, `src/core/**` | Canonical definitions of 40 core architectural concepts mapped to `src/` types. |
+| [glossary.md](glossary.md) | `src/domains/dispatch/types.ts`, `src/tools/**`, `src/domains/agents/**`, `src/core/**` | Canonical definitions of 45 core architectural concepts mapped to `src/` types. |
 | [documentation-coverage.md](documentation-coverage.md) | `src/**` | Complete source-to-documentation mapping matrix and subsystem coverage status. |
 | [tui-design.md](tui-design.md) | `src/interactive/theme/tokens.ts`, `src/interactive/theme/glyphs.ts` | TUI color system, glyph vocabulary (`contextReserve`), structural layouts, state choreography, code ink. |
 | [installation-and-lifecycle.md](installation-and-lifecycle.md) | `src/cli/paths.ts`, `src/cli/doctor.ts`, `src/cli/uninstall.ts`, `src/cli/removal.ts` | Installation, upgrade, reset, uninstallation, launcher ownership and what `--remove-binary` will and will not remove, partial-failure behavior, configuration folders (`credentials.yaml` `0o600`), and permissions. |
-| [release-cut-checklist.md](release-cut-checklist.md) | `scripts/check-release.mjs`, `scripts/lifecycle-matrix.mjs`, `package.json` | Ordered release-cut steps with an explicit authorization boundary: everything external or irreversible is marked not run and needs an operator decision. |
+| [release-cut-checklist.md](release-cut-checklist.md) | `scripts/check-release.mjs`, `tests/smoke/pack-install.test.ts`, `benchmarks/internal/`, `package.json` | Ordered release-cut steps with an explicit authorization boundary: everything external or irreversible is marked not run and needs an operator decision. |
 | [observability.md](observability.md) | `src/domains/observability/**`, `src/interactive/view/**`, `src/domains/dispatch/**`, `src/core/bus-events.ts` | `/view` artifact browsing, receipt verification, worker diagnostics, event routing, and cost snapshots. |
 | [evidence-and-memory.md](evidence-and-memory.md) | `src/domains/evidence/**`, `src/domains/memory/**`, `src/cli/evidence.ts`, `src/cli/memory.ts` | Evidence corpus layout, findings, memory lifecycle and prompt injection. |
 | [proactive-memory.md](proactive-memory.md) | `src/domains/memory/**` | Proactive task memory architecture, session task bank, intervention rules, and handoff carrying. |
 | [trace-store.md](trace-store.md) | `src/cli/trace.ts`, `src/domains/observability/trace-store.ts` | WAL SQLite trace mirror database schema, rowid cursor queries, rebuildability, 6 `clio-coder trace` subcommands (`runs`, `phases`, `tail`, `procs`, read-only `sql` SELECT, `ui`). |
 | [eval-runner.md](eval-runner.md) | `src/domains/eval/**`, `src/cli/eval.ts` | Local YAML eval tasks, dual token accountings (`tokens.*` wire vs `receiptUsage.*` journal), fail-closed null totals, EvalArtifactV4 format, `verify.measure` task outcome recording. |
-| [evals-internal.md](evals-internal.md) | `src/domains/eval/**`, `benchmarks/soak/**` | Private context index determinism, target smoke matrices, soak machinery benchmark suite (4 suites: `clio-soak`, `clio-soak-boundary`, `clio-soak-chaos`, `clio-soak-loop`). |
+| [evals-internal.md](evals-internal.md) | `src/domains/eval/**` | Private context index determinism and target smoke matrices. External model benchmarks are documented under `benchmarks/`. |
 | [extensions-and-sharing.md](extensions-and-sharing.md) | `src/domains/extensions/**`, `src/domains/resources/**`, `src/domains/share/**`, `src/cli/extensions.ts`, `src/cli/share.ts` | Prompt and skill resources, extension manifests, portable share archives. |
 | [skills-marketplace.md](skills-marketplace.md) | `src/interactive/overlays/skills-hub.ts`, `src/domains/resources/skills/marketplace.ts` | Skills Hub marketplace discovery through the install resolver, empty state, install actions, publishing flow. |
 | [model-catalog.md](model-catalog.md) | `src/domains/providers/catalog.ts`, `src/domains/providers/models/**`, `src/domains/providers/probe/**`, `src/domains/providers/model-capabilities.ts` | Model catalog, live probes (`--offline` toggle), exact-id selector `probeCapabilitiesForModel`, field-note promotion. |

package/docs/environment-variables.md

@@ -30,6 +30,7 @@ Durable values live in the `guardrails:` section of settings.yaml (see [configur
 | `CLIO_CODER_TRUST_PROJECT_SKILLS` | off | `1` trusts project-local skills for execution (`src/domains/resources/skills/loader.ts`). |
 | `CLIO_CODER_ALLOW_EXTERNAL_FULL_ACCESS` | off | `1` lets full-auto pass through to external CLI runtimes with their own full access (`src/engine/claude/subprocess-runtime.ts`, `src/engine/antigravity/subprocess-runtime.ts`). |
 | `CLIO_CODER_FORCE_COMPACT` | off | `1` forces compaction on the next interactive turn (`src/interactive/chat-loop.ts`). |
+| `CLIO_CODER_LEGACY_MASK` | off | `1` temporarily restores the destructive stale-observation mask before summary compaction; remove it after compatibility diagnosis. |
 | `CLIO_CODER_STATUS_STUCK_MS` | 180000 | Stuck-turn watchdog threshold (`src/interactive/status/watchdog.ts`). |
 | `CLIO_CODER_SHUTDOWN_HOOK_MS` | 500 | Wall-clock budget per shutdown hook (`src/core/termination.ts`). |
 | `CLIO_CODER_HOOK_BUDGET_MS` | per-phase built-ins | Global middleware hook wall-clock budget (`src/domains/middleware/budget.ts`). |
@@ -111,4 +112,4 @@ Set by Clio for its own processes; not operator knobs.
 | `CLIO_CODER_TEST_STAGE1_DELAY_MS`, `CLIO_CODER_TEST_STAGE1_FAIL` | `NODE_ENV=test`-only, bounded instant-shell interleaving and injected hydration failure seams for the built PTY acceptance suite (`src/cli/clio.ts`). |
 | `CLIO_CODER_REQUIRE_HOME_PREFIX` | Test guardrail: abort if resolved directories escape `CLIO_CODER_HOME` (`src/core/init.ts`). |
 
-Variables used only by `scripts/` and `benchmarks/` harnesses (the `CLIO_CODER_LIVE_*` smoke-test family, benchmark fleet configuration, install-script inputs) are not part of the shipped runtime and are documented inline where they are consumed.
+Variables used only by the Terminal-Bench agent under `benchmarks/community/` and by the install script are not part of the shipped runtime and are documented inline where they are consumed. The live drivers under `benchmarks/internal/` take no environment of their own: the target comes from `--target <id>`.

package/docs/eval-runner.md

@@ -1,7 +1,7 @@
 # Clio Coder Local Evaluation Runner
 
 > [!TIP]
-> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.3).
+> **Interactive Spec Available:** An interactive task suite validator, subprocess execution simulator, and compare calculator is located at [docs/html/eval_blueprint.html](html/eval_blueprint.html) (Version: 0.3.6).
 
 The local evaluation runner executes repository-local YAML task suites as deterministic subprocess checks. It is useful for comparing harness changes, prompts, tools, or local workflows.
 

package/docs/evals-internal.md

@@ -1,7 +1,7 @@
 # Internal Eval Suites
 
 > [!TIP]
-> **Interactive Spec Available:** Interactive blueprints are available for internal evaluation suites at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) and soak benchmark suites at [docs/html/soak_blueprint.html](html/soak_blueprint.html) (Version: 0.3.3).
+> **Interactive Spec Available:** An interactive blueprint is available at [docs/html/evals_internal_blueprint.html](html/evals_internal_blueprint.html) (Version: 0.3.6).
 
 Private suites should live outside this repository. Keep datasets, prompts,
 live fleet coordinates, calibration outputs, and raw run artifacts in a private
@@ -15,9 +15,9 @@ clio-coder eval run --suite <external-path> --clio-coder-entry dist/cli/index.js
 ```
 
 Use `--out <dir>` when the artifact should be written outside the default Clio
-data directory. Public summaries can be copied into
-`benchmarks/results/<suite>/<run-id>/` only after they have been sanitized down
-to `manifest.json` and `summary.json`.
+data directory. Product eval artifacts and external benchmark campaigns are
+separate: public benchmark adapters live under `benchmarks/community/` and do
+not use the eval runner.
 
 ## Context Regression Seed
 
@@ -268,31 +268,3 @@ thresholds:
       value: 0
 ```
 
----
-
-## Soak Benchmark Suite
-
-The soak benchmark suite located under [`benchmarks/soak/`](../benchmarks/soak/) measures Clio's own machinery performance, integrity, and structural invariant promises under load. Unlike standard evaluation suites, the soak suite evaluates the reliability of Clio rather than model capability. A weak model that fails to solve the workload still passes the suite if Clio's machinery behaves correctly; a strong model fails the suite if Clio fails to seal a receipt, cannot authenticate a receipt, or violates a system invariant.
-
-The soak suite comprises four specialized suite files:
-
-### 1. Machinery Under Load (`clio-soak.yaml`)
-Evaluates the same task workload across two execution surfaces: the headless main-agent surface (`clio-run`) and a dispatched worker surface (`agent: coder`). It tests single-file bugs, multi-file bugs, and compaction continuity across restarts.
-- **Surface Differences**: Main-agent tasks verify session ledger continuity (`ledger.formatVersion`, `ledger.toolPairsUnmatched`, `ledger.assistantBetweenCallAndResult`), while dispatch worker tasks verify process group cleanup (`process.orphanedChildren == 0`).
-- **Compaction Continuity**: Verifies that compaction summaries are present (`continuity.compactionSummaryPresent`) and that pre-compaction facts are preserved (`continuity.answeredFromPreCompaction`).
-- **Suite-Wide Gates**: Gates on `receipt.sealed`, `receipt.integrityValid`, `receipt.outcomeMatchesExit`, `tokens.measured`, `stream.cumulativeSnapshots == 0`, `stream.usageDoubleCounted == false`, and `stream.segmentUsageMatchesMessages == true`.
-
-### 2. Per-Step Write Boundaries (`clio-soak-boundary.yaml`)
-Validates write boundary enforcement across steps without model participation. Enforcement is strictly detect-and-rollback and is never sandboxing.
-- `write-boundary.rolled-back`: Verifies clean detection of allowlist violations (`writes_boundary_violation`), git-level file restoration, and sealed verdict generation (`boundary.violationsRolledBack == 1`, `boundary.rollbackIncomplete == 0`).
-- `write-boundary.rollback-incomplete`: Tests honest failure reporting when a path was dirty prior to snapshot taking. prior bytes exist only in the overwritten tree, so rollback leaves the tree unchanged and records incomplete rollback (`boundary.rollbackIncomplete == 1`, `boundary.violationsRolledBack == 0`).
-
-### 3. Fault Injection Chaos (`clio-soak-chaos.yaml`)
-Evaluates system resilience against process signals.
-- `chaos.sigint-mid-tool`: Prompts Clio for a long-running bash tool call and injects `SIGINT` once the subprocess initializes. Asserts exit code `130`, confirms no orphaned children remain (`process.orphanedChildren == 0`), and verifies receipt sealing, receipt integrity, and provider token reporting.
-
-### 4. Bounded Loops (`clio-soak-loop.yaml`)
-Validates iteration bounds and receipt accounting for fleet loops (`bounded-loop.fleet`).
-- **Loop Bounds**: Asserts that verification attempts do not exceed declared limits (`loop.attemptsSpent <= 3`), recovery attempts seal individual receipts (`loop.receiptsMatchRepairs == true`), and unneeded nodes report as `unneeded` rather than skipped or failed (`loop.skippedNodes == 0`).
-- **Two Token Accountings**: Distinguishes `tokens.*` (folded live off wire stdout by `createStreamInvariantFold`) from `receiptUsage.*` (journal receipts sealed and authenticated against ledger envelopes). On fleet runs, wire streaming is absent (`tokens.measured == false`), while journal receipts provide authenticated usage (`receiptUsage.measured == true`).
-