dsh-compound-engineering 0.0.0-stage → 0.1.0

package/skills/ce-optimize/references/spec.md

@@ -0,0 +1,107 @@
+# Phase 0: input type and the optimization spec
+
+Read this at the start of Phase 0. It carries how to tell a spec path from a goal description, and how to load, build, or validate the spec before CP-0. The schemas it validates against are `references/optimize-spec-schema.yaml` and `references/experiment-log-schema.yaml`.
+
+**Horizon:** continue while another measurement or experiment is worthwhile within the stopping rules. Start serial until measurement is trusted. Stop as soon as a stopping criterion holds.
+
+### 0.1 Determine Input Type
+
+Check whether the input is:
+- **A spec file path** (ends in `.yaml` or `.yml`): read and validate it
+- **A description of the optimization goal**: help the user create a spec interactively
+
+### 0.2 Load or Create Spec
+
+**If spec file provided:**
+1. Read the YAML spec file. The orchestrating agent parses YAML natively -- no shell script parsing.
+2. Validate the spec against **every** rule in the `validation_rules` section of `references/optimize-spec-schema.yaml`. That section is the single source of truth for what a valid spec requires. Do not rely on a remembered subset; conditional rules such as the singleton-rubric and exclusive-resources requirements live only there.
+3. If any rule fails, report the specific failures and ask the user to fix them before proceeding
+
+**If description provided:**
+1. Analyze the project to understand what can be measured. `references/usage-guide.md` has longer kickoff prompt shapes if the interview needs them.
+2. **Detect whether the optimization target is qualitative or quantitative.** This determines `type: hard` vs `type: judge` and is the single most important spec decision:
+
+   **Use `type: hard`** when:
+   - The metric is a scalar number with a clear "better" direction
+   - The metric is objectively measurable (build time, test pass rate, latency, memory usage)
+   - No human judgment is needed to evaluate "is this result actually good?"
+   - Examples: reduce build time, increase test coverage, reduce API latency, decrease bundle size
+
+   If the user names more than one hard success condition that must all hold (local wall time and CI critical path and runner-minutes, for example), put them in `metric.objectives` as `role: required` and keep `metric.primary` as the ranking key. A spec without `metric.objectives` keeps single-primary acceptance. If each evaluation costs minutes, set `measurement.stability.mode: ladder` with a relative or paired comparison and a futility bound; do not spend the full confirmation protocol on every exploratory experiment. Start from `references/example-expensive-benchmark-spec.yaml` for that shape.
+
+   **Use `type: judge`** when:
+   - The quality of the output requires semantic understanding to evaluate
+   - A human reviewer would need to look at the results to say "this is better"
+   - Proxy metrics exist but can mislead (e.g., "more clusters" does not mean "better clusters")
+   - The optimization could produce degenerate solutions that look good on paper
+   - Examples: clustering quality, search relevance, summarization quality, code readability, UX copy, recommendation relevance
+
+   **IMPORTANT**: If the target is qualitative, **strongly recommend `type: judge`**. Explain that hard metrics alone will optimize proxy numbers without checking actual quality. Show the user the three-tier approach:
+   - **Degenerate gates** (hard, cheap, fast checks that reject a candidate outright): catch obviously broken solutions such as "all items in 1 cluster" or "0% coverage". Run first. If any gate fails, skip the expensive judge step.
+   - **LLM-as-judge** (the actual optimization target): sample outputs, score them against a rubric, aggregate. This is what the loop optimizes.
+   - **Diagnostics** (logged, never used to reject): distribution stats, counts, timing. These help explain WHY a judge score changed.
+
+   If the user insists on `type: hard` for a qualitative target, proceed but warn that the results may optimize a misleading proxy.
+
+   **Exemplars for the validity gate.** A person's input is needed only where the score is itself a judgment. When the primary is measured objectively (a duration, a size, a count, a pass rate over fixed cases), ask the user for nothing here: the measurement is its own ground truth, and Phase 1 builds the shortcut probe itself. When the primary is a judgment (`type: judge`, or a hard metric standing in for a quality the user would otherwise rate), use the known-good and known-bad outputs the project already has (recorded failures, fixtures, the cases of a text target) and ask the user only for what is missing: a few outputs they would score at the top and a few at the bottom. A user who has none to give is not blocked; Phase 1 runs the probes it can and states the rest as a limitation at approval. When the target is instruction text (a skill, an agent-instructions file, a persona, a tool description), read `references/text-targets.md` before going further: the eval set and the hypothesis moves differ from a code target.
+
+3. **Design the sampling strategy** (for `type: judge`):
+
+   Guide the user through defining stratified sampling. The key question is: "What parts of the output space do you need to check quality on?"
+
+   Walk through these questions:
+   - **What does one "item" look like?** (a cluster, a search result page, a summary, etc.)
+   - **What are the natural size/quality strata?** (e.g., large clusters vs small clusters vs singletons)
+   - **Where are quality failures most likely?** (e.g., very large clusters may be degenerate merges; singletons may be missed groupings)
+   - **What total sample size balances cost vs signal?** (default: 30 items, adjust based on output volume)
+
+   Example stratified sampling for clustering:
+   ```yaml
+   stratification:
+     - bucket: "top_by_size"     # largest clusters: check for degenerate mega-clusters
+       count: 10
+     - bucket: "mid_range"       # middle of non-solo cluster size range: representative quality
+       count: 10
+     - bucket: "small_clusters"  # clusters with 2-3 items: check if connections are real
+       count: 10
+   singleton_sample: 15          # singletons: check for false negatives (items that should cluster)
+   ```
+
+   The sampling strategy is domain-specific. For search relevance, strata might be "top-3 results", "results 4-10", "tail results". For summarization, strata might be "short documents", "long documents", "multi-topic documents".
+
+   **Singleton evaluation is critical when the goal involves coverage.** Sampling singletons with the singleton rubric checks whether the system is missing obvious groupings.
+
+4. **Design the rubric** (for `type: judge`):
+
+   Help the user define the scoring rubric. A good rubric:
+   - Has a 1-5 scale (or similar) with concrete descriptions for each level
+   - Includes supplementary fields that help diagnose issues (e.g., `distinct_topics`, `outlier_count`)
+   - Is specific enough that two judges would give similar scores
+   - Does NOT assume bigger/more is better. "3 items per cluster average" is not inherently good or bad
+   - Asks for a per-item `feedback` string (what is wrong, what would fix it); the judge template requires it and Phase 3 generates hypotheses from it
+
+   Example for clustering:
+   ```yaml
+   rubric: |
+     Rate this cluster 1-5:
+     - 5: All items clearly about the same issue/feature
+     - 4: Strong theme, minor outliers
+     - 3: Related but covers 2-3 sub-topics that could reasonably be split
+     - 2: Weak connection: items share superficial similarity only
+     - 1: Unrelated items grouped together
+     Also report: distinct_topics (integer), outlier_count (integer)
+   ```
+
+5. Guide the user through the remaining spec fields:
+   - What degenerate cases should be rejected? (degenerate gates, for example "solo_pct <= 0.95" catches all-singletons and "max_cluster_size <= 500" catches mega-clusters)
+   - What command runs the measurement?
+   - What files can be modified? What is immutable?
+   - Any constraints or dependencies?
+   - If this is the first run: recommend `execution.mode: serial`, `execution.max_concurrent: 1`, `stopping.max_iterations: 4`, and `stopping.max_hours: 1`
+   - If the user named multiple required hard targets or an expensive harness: recommend `metric.objectives` plus `stability.mode: ladder` as above, and show `references/example-expensive-benchmark-spec.yaml`
+   - If `type: judge`: recommend `sample_size: 10`, `batch_size: 5`, and `max_total_cost_usd: 5` until the rubric and harness are trusted
+   - Holdout: a held-out set the loop confirms against but never selects from. `measurement.holdout.command` (same JSON shape as the measurement command), or for a judge primary `metric.judge.confirmation_seed` distinct from `sample_seed`, whose sample shares no item with the selection sample. Required for `type: judge` and when the run waits between ticks through a wake after turn end (`references/persistence.md`, The Wait Record and Ticks); otherwise recommended, and its absence is stated at the approval gate
+   - If `type: judge`: do not ask for hand-scored examples here. When the user already has a labeled sample, name it in `metric.judge.calibration.labels`; otherwise leave it unset, and Phase 1 offers a short guided check once the baseline has been judged (`references/measurement.md` 1.3)
+   - Per-case results: when the harness can emit a `cases` map (case id to pass/fail), set `measurement.per_case: true` so keeps and the wrap-up report which cases the baseline passed and the candidate fails
+6. Write the spec to `<state-root>/spec.yaml`, with `<state-root>` resolved by `references/persistence.md` (The State Root)
+7. Present the proposed scope, behavior constraints, measurement approach, and limits for approval before proceeding, with a link to the saved spec. Apply the SKILL.md body's user-facing reporting rule. Put whatever the user is being asked to judge in front of them: for a judge primary that is the rubric's scale, level by level, because it is their definition of better and they cannot approve it from a link to a file. Send all of that as an ordinary message, laid out so it is easy to read (the scale as a list, scope and limits on their own lines), and only then ask. The question itself is one short sentence and its options: a question picker shows long text as a single cramped block.

package/skills/ce-optimize/references/text-targets.md

@@ -0,0 +1,51 @@
+# Targets that are instruction text
+
+Read this when the files in `scope.mutable` are instructions a model runs under: a skill, an agent-instructions file, a persona or system prompt, a tool description. The loop is the same as for code. What differs is how the eval set is built, which hypotheses are moves rather than guesses, which model runs what, and how to read a perfect score. `references/example-text-target-spec.yaml` is a complete spec of this shape.
+
+## The eval set
+
+An instruction-text run is only as good as the cases it is scored on, and the cases are the artifact the run keeps; the text is regenerated by every experiment.
+
+- **Build the set from failures that happened.** Each case records four fields: the input, what you wanted, what went wrong, and the output that was produced. Five to ten recorded occurrences is the floor before the text is worth optimizing at all; below that, the fix is to collect more failures, not to run the loop.
+- **Coverage rule: every rule you want preserved needs a case that fails without it.** The optimizer keeps text that moves the score and drops text that does not. A rule with no failing case has no measured reason to exist, so expect the run to remove or weaken it; if that would be a defect, add the case before Phase 1, not after the rule is gone.
+- **Cover each failure mode several times.** A mode seen once can land in either the selection sample or the holdout but not both, so it can never be both learned from and confirmed. Synthetic variants of a real failure are acceptable for this; a synthetic mode nobody has observed is not.
+- **Fix the set for the run.** The cases live in `scope.immutable` alongside the harness. Growing or editing the set is a scope change: it ends this run and starts another with a new baseline.
+
+## The harness
+
+The measurement command runs every case against the candidate text under the task model and prints one JSON object: a pass rate or judge score for the primary, a `cases` map (case id to pass/fail) so `measurement.per_case` can report regressions, and `prompt_tokens` or an equivalent length diagnostic. A binary PASS/FAIL judge given the case's "what went wrong" as the tell is usually enough; a graded rubric earns its cost only when partial credit changes which candidate wins. The holdout is a second case file the selection command never reads.
+
+## Model roles
+
+Two models are involved and the spec pins one of them. The **task model** is the one the text will run under in production; the harness runs every case under it, and it does not change during the run. The **proposer** is whatever writes each candidate: the experiment worker, or an external optimizer. The proposer may be stronger than the task model; a candidate written by a strong model and measured under a cheap one is the normal shape, and improvements found this way carry to stronger task models more often than the reverse. Never let the harness silently use the proposer's model to score; that measures the wrong thing.
+
+## Hypotheses are optimizer moves
+
+Each hypothesis is one of a small number of moves applied to the current text. Name the move in the hypothesis `category` so the digest can see which moves are exhausted.
+
+| Move | What the experiment does | When it is the cheapest next step |
+|---|---|---|
+| `add-examples` | Insert labeled input/output pairs drawn from cases the current text fails | Failure themes name a pattern the prose already describes but the model does not follow |
+| `bootstrap-examples` | Run the task model on passing cases and keep its own good outputs as examples | Hand-written examples read differently from what the model produces |
+| `rewrite-from-feedback` | Rewrite the failing section from the digest's failure themes, as a condition rather than a case list | Several cases fail for one stated reason |
+| `shorten` | Cut text while `prompt_tokens` is a required or secondary objective | The primary is flat and the text is long |
+| `restructure` | Move, merge, or split sections; change the order rules fire in | A rule fires in the wrong place, or two rules compete |
+| `decompose` | Split one instruction into a caller and a reference it loads at the point of use | The always-loaded text carries steps most runs never reach |
+| `vote` | Have the task model produce several outputs and pick by a stated rule | Single outputs are inconsistent on cases the model can solve |
+| `external-optimizer` | See below | An optimizer library is available and a metric-call budget is cheaper than a batch of hand-written candidates |
+
+A rule added for one failing case is not a move. It is the shape the failure themes exist to prevent: when the same section keeps absorbing a case per experiment, the next hypothesis is `rewrite-from-feedback` on that section, restating the condition the cases share.
+
+### An external optimizer as an experiment
+
+When the project has a prompt-optimizer library installed and the harness can be called as its metric, one hypothesis may be: run the optimizer against the seed text for a stated number of metric calls, and take its returned candidate as the experiment's diff. The candidate is then measured by the immutable harness exactly like any other diff; the optimizer's own reported score is not a measurement. Run it with the harness's selection cases only; the holdout stays unseen. The library counts as available when it can be imported or invoked from the project right now; a missing config entry or environment variable proves nothing either way. Name the library and its metric-call budget in the hypothesis so the digest can compare its cost to the hand-written moves. When no optimizer is available, this move is absent from the backlog.
+
+## Reading the score
+
+- **A ceiling is a smell, not a finish.** When the current best scores at the top of the scale on the selection set, the eval is too easy for the text as it now stands. Stop on `target_reached`, report that the set no longer separates candidates, and recommend adding harder or adversarial cases for the next run. Do not tighten the rubric or grow the set inside the run.
+- **Real gains lost to noise** on a small case set are a reason to raise the sample or repeat count, not to lower the threshold.
+- **Length is a stated objective or it is not measured.** If shorter text matters, put `prompt_tokens` in `metric.objectives` (minimize, `role: secondary` to log it, `required` to make it a keep condition). Otherwise do not penalize length by hand.
+
+## Where things live
+
+The case files, the calibration labels, and the harness are repository files in `scope.immutable`; they are reused across runs. The judge cache, per-item feedback, failure themes, and the harness-validation record are run state in the experiment log and its scratch directory. The winning text is the diff on `optimize/<spec-name>`, reviewed by a person at wrap-up like any other kept change.

package/skills/ce-optimize/references/usage-guide.md

@@ -0,0 +1,147 @@
+# `ce-optimize` Usage Guide
+
+## What This Skill Is For
+
+The `ce-optimize` skill is for hard engineering problems where:
+
+1. You can measure the same target twice.
+2. You can either attribute a named-workload cost or try multiple scored variants.
+3. You want the skill to keep confirmed improvements and reject the rest.
+
+On a cost target, the first useful action is often a locating measurement, not a batch of implementation experiments. On a scored variant space, the skill searches and keeps. It is not one-shot implementation of a change you already know.
+
+## When To Use It
+
+Reach for `ce-optimize` when the problem looks like:
+
+- "Find the smallest memory limit that stops OOM crashes without wasting RAM."
+- "Tune clustering parameters without collapsing everything into one garbage cluster."
+- "Find a prompt that is cheaper but still produces summaries good enough for downstream clustering."
+- "Compare several ranking, retrieval, batching, or threshold strategies against the same harness."
+
+Choose `type: hard` when success is objective and cheap to measure:
+
+- Memory usage
+- Latency
+- Throughput
+- Test pass rate
+- Build time
+
+Choose `type: judge` when a numeric metric can be gamed or when human usefulness matters:
+
+- Cluster coherence
+- Search relevance
+- Summary quality
+- Prompt quality
+- Classification quality with semantic edge cases
+
+## When Not To Use It
+
+`ce-optimize` is usually the wrong tool when:
+
+- The change is already known. Make it, or use `ce-work`
+- The job is diagnosing failing or slow behavior. That is `ce-debug`
+- There is no repeatable measurement harness
+- The search space is fake and only has one plausible answer
+- The cost of evaluating variants is too high to justify multiple runs
+
+## How To Think About It
+
+The pattern is:
+
+1. Define the target.
+2. Build or validate the measurement harness first.
+3. Take the cheapest next action that would change what gets implemented: a locating measurement on a cost target, or a scored variant on a search target.
+4. Keep confirmed improvements and reject the rest.
+
+The core rule is simple:
+
+- If a hard metric captures "better," optimize the hard metric.
+- If a hard metric can be gamed, add LLM-as-judge.
+
+Example: lowering a clustering threshold may increase cluster coverage. That sounds good until everything ends up in one giant cluster. Hard metrics may say "improved"; an LLM judge sampling real clusters can say "this is trash."
+
+## First-Run Advice
+
+For the first run:
+
+- Prefer `execution.mode: serial`
+- Set `execution.max_concurrent: 1`
+- Keep `stopping.max_iterations` small
+- Keep `stopping.max_hours` small
+- Avoid new dependencies until the baseline is trustworthy
+- In judge mode, use a small sample and a low cost cap
+
+The goal of the first run is to validate the harness, not to win the optimization immediately.
+
+## Example Prompts
+
+### 1. Memory Tuning
+
+```text
+Run the `ce-optimize` skill to find the smallest memory setting that keeps this service stable under our load test.
+
+The current container limit is 512 MB and the app sometimes OOM-crashes. Do not just jump to 8 GB. Try a small set of realistic memory limits, run the same load test for each one, and score the results using:
+- did the process OOM
+- did tail latency spike badly
+- did GC pauses become excessive
+
+Prefer the smallest memory limit that passes the guard rails.
+```
+
+### 2. Clustering Quality
+
+```text
+Run the `ce-optimize` skill to improve issue and PR clustering quality.
+
+We have about 18k open issues and PRs. We want to test changes that improve clustering quality, reduce singleton clusters, and improve match quality within each cluster.
+
+Do not mutate the shared default database. Copy it for the run, then use per-experiment copies when needed.
+
+Do not optimize only for coverage. Use LLM-as-judge to sample clusters and confirm they still preserve real semantic similarity instead of collapsing into giant low-quality clusters.
+```
+
+### 3. Expensive Test Suite
+
+```text
+Run the `ce-optimize` skill to reduce this repository's full test-suite wall time without making CI slower or spending more runner-minutes.
+
+Local warm median is currently about six minutes and the range is wide. Treat local wall time, CI critical path, and aggregate runner-minutes as required targets: a change that helps only CI may be kept if it does not regress the others, and the run is not done until every declared target is met.
+
+Do not spend a five-run cold/warm protocol on every exploratory experiment. Smoke for correctness, take one paired sample, abort anything already far worse than the current best, and reserve the full protocol for a candidate you are about to keep and for final confirmation.
+```
+
+### 4. Prompt Optimization
+
+```text
+Run the `ce-optimize` skill to create a summarization prompt for issues and PRs that minimizes token spend while still producing summaries that are good enough for downstream clustering.
+
+I want the loop to compare prompt variants, measure token cost, and judge whether the summaries preserve the distinctions needed to cluster related issues together without merging unrelated ones.
+```
+
+## Choosing Between Hard Metrics And Judge Mode
+
+Use hard metrics alone when:
+
+- "Better" is obvious from the numbers.
+
+Add judge mode when:
+
+- The numbers can improve while the real output gets worse.
+
+Common pattern:
+
+- Hard gates reject broken outputs.
+- Judge mode scores the surviving candidates for actual usefulness.
+
+That hybrid setup is often the best default for ranking, clustering, and prompt work.
+
+## First-run defaults
+
+A first run optimizes for signal and safety, not throughput:
+
+- Start from `references/example-hard-spec.yaml` when the metric is objective and cheap to measure; use `references/example-judge-spec.yaml` only when quality genuinely requires semantic judgment; use `references/example-expensive-benchmark-spec.yaml` when each run costs minutes or several hard targets must all hold.
+- Prefer `execution.mode: serial` with `execution.max_concurrent: 1`.
+- Cap the run with `stopping.max_iterations: 4` and `stopping.max_hours: 1`.
+- Add no new dependencies until the baseline and measurement harness are trusted.
+- For judge mode, start at `sample_size: 10`, `batch_size: 5`, and `max_total_cost_usd: 5`.

package/skills/ce-optimize/references/wrap-up.md

@@ -0,0 +1,54 @@
+# Phase 4: wrap-up
+
+Read this at wrap-up. The SKILL.md body states the post-completion options the user chooses from. This file carries what each one needs: the deferred-hypothesis presentation, the results summary, what is preserved and what is not, the mechanical-apply bar for review findings, and the cleanup rules.
+
+## Phase 4: Wrap-Up
+
+### 4.1 Present Deferred Hypotheses
+
+If any hypotheses were deferred due to unapproved dependencies:
+1. List them with their dependency requirements
+2. Ask the user whether to approve, skip, or save for a future run
+3. If approved: add to backlog and offer to re-enter Phase 3 for one more round
+
+### 4.2 Summarize Results
+
+Report from the persisted forecasts and measurements, with the final state confirmed using the configured measurement protocol. The confirmation measures the original baseline revision (the commit the optimization branch started from) and the final state as one pairing, and repeats that pairing on the holdout when one is configured; the per-keep holdout pairs compare successive states and do not stand in for it. Persist both pairings in `comparisons` (the held-out one as `kind: holdout`). If confirmation is unavailable, label the final values unconfirmed and state why. A legacy log remains reportable: missing forecasts, comparison baselines, or uncertainty stay unrecorded rather than being reconstructed from the final result.
+
+The summary must contain:
+
+- **Overall result:** original baseline -> final for every required objective (the primary when no objectives are declared), with units, absolute change, target status, and percentage change where meaningful. A zero baseline has no defined percentage change; an ordinal judge score is reported in score points, not as a percentage improvement.
+- **Opportunity -> result:** each retained change's original expected benefit beside its measured before/after result, the comparison identity and workload, and whether the evidence supports the estimate. Identify standalone versus integrated results. If the forecast and result use different baselines or workloads, label them non-comparable instead of declaring that the forecast was met or missed.
+- **Evidence quality:** measurement uncertainty and confirmation status, correctness checks and their results, and any unverified constraints, including each one a revert found the harness does not check (the change the harness needs before the next run). Report measured incremental contributions only when the corresponding reference measurements exist. Do not add percentages from successive changes or count a standalone runner-up gain as its integrated contribution; the overall gain comes from original-to-final measurement.
+- **Held-out result and regressions:** the final state's holdout comparison against the original baseline (or that no holdout was configured, so the reported gain was selected and reported on the same sample), and when `measurement.per_case` is set the list of cases the baseline passed and the final state fails. An empty regression list is reported as empty; an absent one as unrecorded.
+- **Remaining opportunity:** what still costs time or resources, with current evidence and whether further work appears worthwhile. Without fresh evidence, label remaining estimates stale or unknown; do not claim a new bottleneck from the old profile alone.
+- **Run accounting:** stopping reason, duration, outcome counts, judge cost when applicable, estimated whole-run cost with which parts are estimates, and the log path. Preserve the existing outcome distinctions, including `Not selected: <count>`, inconclusive, censored, deferred, errors, and timeouts. Short reports may omit individual rejected experiments, but retain required-objective results and evidence limitations.
+
+### 4.3 Preserve and Offer Next Steps
+
+The optimization branch (`optimize/<spec-name>`) is preserved with all commits from kept experiments.
+The experiment log and strategy digest remain under `<state-root>` for resume and audit; they do not travel with the branch from either root (`.context/` is gitignored, and a durable store is outside the checkout).
+
+**Export the report.** Write the 4.2 summary to `<root>/optimize/<spec-name>-report.md` (`<root>` from the body's Artifact Root rule) and commit it on the optimization branch, so a PR from the branch carries its evidence and the summary outlives the ledger. Report the path to the user together with the summary; the user may name a different tracked path instead.
+
+Present these options after the summary. When the stop was reached on an unattended wake, presenting them is the point where this turn ends: `run_state.status` stays `waiting` on the user's choice, and CP-5 is written only after that choice, as the body states.
+
+1. **Run `ce-code-review`** on the cumulative diff (baseline to final), on the optimization branch. Do not commit or push from this step.
+2. **Run `ce-compound`** to document the winning strategy as an institutional learning.
+3. **Create PR** from the optimization branch to the default branch.
+4. **Continue**: re-enter Phase 3, state re-read first.
+5. **Done**: leave the branch for manual review.
+
+For option 1, load `ce-code-review` on the optimization branch, interactive or `mode:agent`, and land eligible fixes under the bar below before moving to the next option.
+
+**Mechanical-apply bar:** apply any finding with a concrete `suggested_fix` that is a clear, reversible improvement. Push back (keep, don't apply) when the reviewer is wrong, noting why. Defer anything whose right fix needs a design or product decision (architecture direction, contract shape, behavior change needing sign-off) and any finding with no concrete fix to act on. Tell the user what was deferred. Confirm evidence still matches at `file:line` before editing. After applying, run tests (at least targeted tests for what changed; broader suite for multi-file edits). Do not commit or push from this step. Leave the diff on the optimization branch for the Create PR option.
+Option 4 (continue) re-enters Phase 3 with the current state, state re-read from disk first.
+
+### 4.4 Cleanup
+
+Once CP-5 is written the run is over, so leave behind only what someone will still read. Remove the working files this run created under `<state-root>`: the strategy digest, the judge cache, result markers, and any scratch payloads or decision files you wrote there. Keep `experiment-log.yaml` and `spec.yaml`, tell the user where they are, and remove them too only if the user asks. A run that is not final (`waiting` or `blocked`) keeps everything, because a resume reads it.
+
+Remove files the way this host and the user's own rules prefer, such as moving them to the trash rather than deleting them permanently when that is available. Remove only files you can name that this run created inside `<state-root>`; never compose a path to delete from a pattern or an unresolved placeholder, and leave anything you are unsure about in place and name it in the final message. Remove experiment worktrees only through `scripts/experiment-worktree.sh cleanup`, which checks what it removes.
+
+Do NOT delete the experiment log if the user may resume or wants an audit trail. The exported report in 4.3 is the durable shared artifact; the ledger is not. Unregister any wake still registered for this run.
+Do NOT delete experiment worktrees that are still being referenced.