dsh-compound-engineering 0.0.0-stage → 0.1.1

package/skills/ce-optimize/references/experiment-prompt-template.md

@@ -0,0 +1,143 @@
+# Experiment Worker Prompt Template
+
+This template is used by the orchestrator to dispatch each experiment to a subagent or Codex. Variable substitution slots are filled at spawn time.
+
+---
+
+## Template
+
+```
+You are an optimization experiment worker.
+
+Your job is to implement a single hypothesis to improve a measurable outcome. You will modify code within a defined scope, then stop. You do NOT run the measurement harness, commit changes, or evaluate results -- the orchestrator handles all of that.
+
+<experiment-context>
+Experiment: #{iteration} for optimization target: {spec_name}
+Hypothesis: {hypothesis_description}
+Category: {hypothesis_category}
+
+Current best metrics:
+{current_best_metrics}
+
+Baseline metrics (before any optimization):
+{baseline_metrics}
+</experiment-context>
+
+<scope-rules>
+You MAY modify files in these paths:
+{scope_mutable}
+
+You MUST NOT modify files in these paths:
+{scope_immutable}
+
+CRITICAL: Do not modify any file outside the mutable scope. The measurement harness and evaluation data are immutable, which means the metric cannot be changed, not that it cannot be gamed: a gain from anything other than doing the work the metric stands for, or one that breaks a constraint below, is reverted by the orchestrator however well it scores.
+</scope-rules>
+
+<constraints>
+{constraints}
+</constraints>
+
+<approved-dependencies>
+You may add or use these dependencies without further approval:
+{approved_dependencies}
+
+If your implementation requires a dependency NOT in this list, STOP and note it in your output. Do not install unapproved dependencies.
+</approved-dependencies>
+
+<source-digest>
+What you need to know about the material this target works on and the current approach:
+
+{source_digest}
+</source-digest>
+
+<failure-evidence>
+Worst cases from the current best that bear on this hypothesis, with what went wrong. Use them to check your change against real failures. The hypothesis still decides what you implement, and it may replace the approach rather than fix these cases:
+
+{failure_cases}
+</failure-evidence>
+
+<previous-experiments>
+Recent experiments and their outcomes (for context -- avoid re-trying approaches that already failed):
+
+{recent_experiment_summaries}
+</previous-experiments>
+
+<instructions>
+1. Read the mutable files your change touches. Rely on the source digest for the rest of the material, and open other files only when the hypothesis depends on a detail the digest lacks
+2. Implement the hypothesis described above
+3. Make your changes focused and minimal -- change only what is needed for this hypothesis
+4. Do NOT run the measurement harness (the orchestrator handles this)
+5. Do NOT commit (the orchestrator will commit the winning diff before merge if this experiment succeeds)
+6. Do NOT modify files outside the mutable scope
+7. When done, run `git diff --stat` so the orchestrator can see your changes
+8. If you discover you need an unapproved dependency, note it and stop
+
+Focus on implementing the hypothesis well. The orchestrator will measure and evaluate the results.
+</instructions>
+```
+
+## Delta for `execution.backend: remote`: measure and report
+
+A remote worker has its own checkout on its own machine, so the orchestrator cannot measure its tree. Append this block to the template above (inside the same prompt) and replace the template's instruction 4 ("Do NOT run the measurement harness") with it. Nothing else in the template changes: scope, constraints, and dependency rules are identical.
+
+```
+<remote-worker>
+You are running detached, on your own checkout. The orchestrator will not see your tree; it sees only the branch you push and the result you report.
+
+Base: commit {base_sha} on branch {optimization_branch}. Before changing anything, verify your HEAD is exactly {base_sha}. If it is not, check it out; if you cannot, stop and report that.
+
+After implementing the hypothesis, measure PAIRED on this machine, using the immutable harness as-is:
+1. Baseline: check out {base_sha} (stash or branch your change), run `{measurement_command}` {sample_count} time(s), record every sample.
+2. Candidate: restore your change, run the same command the same number of times, record every sample.
+Alternate baseline and candidate runs when {sample_count} > 1. Never edit the measurement command, its working directory, or any file in the immutable scope; a changed harness makes your result unusable.
+
+Commit your mutable-scope changes, and nothing else, with the message `optimize({spec_name}): exp-{iteration} <hypothesis, short>`. That commit is the candidate; its SHA is `head_sha`.
+
+Then write `result.yaml` at the repo root of your checkout with exactly this shape:
+  experiment: {iteration}
+  base_sha: {base_sha}
+  head_sha: <the candidate commit's SHA>
+  machine: <a stable identifier for this machine or worker>
+  measured_at: <ISO 8601>
+  baseline:  { gates: {...}, metrics: { <name>: { aggregate: <n>, samples: [...] } }, diagnostics: {...} }
+  candidate: { gates: {...}, metrics: { <name>: { aggregate: <n>, samples: [...] } }, diagnostics: {...} }
+  correctness: <checks you ran and their results, or "none">
+Include the harness's per-case `cases` object and any cost, tokens, or latency fields in each snapshot only when the harness emits them; do not compute them yourself.
+Commit `result.yaml` alone on top of the candidate commit, so the pushed tip's parent is `head_sha`, and push to `{result_ref}`. Do not push to {optimization_branch} or any other branch.
+
+Your final message is the structured result: the pushed ref, head_sha, machine, and the two aggregates. Report an unapproved dependency, a base mismatch, or a harness you could not run as a blocker instead of a result.
+</remote-worker>
+```
+
+The worker's numbers are its own claim. The orchestrator runs `decide.mjs` on the paired snapshots and, before any keep, obtains a confirmation measurement the worker did not produce (`references/loop.md` 3.4).
+
+## Variable Reference
+
+| Variable | Source | Description |
+|----------|--------|-------------|
+| `{iteration}` | Experiment counter | Sequential experiment number |
+| `{spec_name}` | Spec file `name` field | Optimization target identifier |
+| `{hypothesis_description}` | Hypothesis backlog | What this experiment should try |
+| `{hypothesis_category}` | Hypothesis backlog | Category (signal-extraction, algorithm, etc.) |
+| `{current_best_metrics}` | Experiment log `best` section | Current best metric values (compact YAML or key: value pairs) |
+| `{baseline_metrics}` | Experiment log `baseline` section | Original baseline before any optimization |
+| `{scope_mutable}` | Spec `scope.mutable` | List of files/dirs the worker may modify |
+| `{scope_immutable}` | Spec `scope.immutable` | List of files/dirs the worker must not touch |
+| `{constraints}` | Spec `constraints` | Free-text constraints to follow |
+| `{approved_dependencies}` | Spec `dependencies.approved` | Dependencies approved for use |
+| `{source_digest}` | `source-digest.md` in the run's scratch directory | What workers need to know about the target's material and current approach |
+| `{failure_cases}` | `worst_cases` on the current best (kept experiment entry, or baseline) | The cases relevant to this hypothesis, with reasons; "none recorded" when absent |
+| `{recent_experiment_summaries}` | Rolling window (last 10) from experiment log | Compact summaries: hypothesis, outcome, learnings |
+| `{base_sha}` (remote only) | `git rev-parse optimize/<spec-name>` at dispatch | The commit the worker must start from and measure as baseline |
+| `{optimization_branch}` (remote only) | `optimize/<spec-name>` | Named so the worker knows what not to push to |
+| `{result_ref}` (remote only) | `optimize-exp/<spec-name>/exp-<NNN>` | The branch the worker pushes; the orchestrator fetches it to collect |
+| `{measurement_command}` (remote only) | Spec `measurement.command` (+ `working_directory`) | Run verbatim; the harness is immutable |
+| `{sample_count}` (remote only) | `1` for the exploratory pair in `ladder` mode; `repeat_count` for `repeat`; `1` for `stable` | Paired samples per side; `decide.mjs` may ask for more via a follow-up dispatch |
+
+## Notes
+
+- This template works for subagent, Codex, and remote dispatch. No platform-specific assumptions. Only `remote` appends the measure-and-report delta.
+- For Codex dispatch: write the filled template to a temp file and pipe via stdin (`cat /tmp/optimize-exp-XXXXX.txt | codex exec --skip-git-repo-check - 2>&1`).
+- For subagent dispatch: pass the filled template as the subagent prompt.
+- Keep `{recent_experiment_summaries}` concise -- 2-3 lines per experiment, last 10 only. Do not include the full experiment log.
+- The worker should NOT read the full experiment log or strategy digest. It receives only what the orchestrator provides.

package/skills/ce-optimize/references/judge-prompt-template.md

@@ -0,0 +1,117 @@
+# Judge Evaluation Prompt Template
+
+This template is used by the orchestrator to dispatch batched LLM-as-judge evaluation calls. Each judge sub-agent evaluates a batch of sampled output items and returns structured JSON scores.
+
+The orchestrator:
+1. Reads the experiment's output
+2. Selects samples per the stratification config (using fixed seed)
+3. Groups samples into batches of `judge.batch_size`
+4. Dispatches `ceil(sample_size / batch_size)` parallel sub-agents using this template
+5. Aggregates returned JSON scores
+
+---
+
+## Item Evaluation Template
+
+```
+You are a quality judge evaluating output items for an optimization experiment.
+
+Your job is to score each item using the rubric below and return structured JSON. Be consistent and calibrated -- the same quality level should get the same score across items.
+
+<rubric>
+{rubric}
+</rubric>
+
+<items>
+{items_json}
+</items>
+
+<output-contract>
+Return ONLY a valid JSON array. No prose, no markdown, no explanation outside the JSON.
+
+Each element must have:
+- "item_id": the identifier of the item being evaluated (string or number, matching the input)
+- All fields requested by the rubric (scores, counts, etc.)
+- "reason": one short sentence naming what most held this item's score down, or what earned it the score when nothing did
+- "feedback": one or two sentences naming what is wrong with this item and what change would fix it. For an item at the top of the scale, name what keeps it there. This text drives the next round of hypotheses, so name the defect, not the score.
+- "ambiguous": true if you cannot confidently score this item (e.g., insufficient context, borderline case). When ambiguous, still provide your best-guess score but flag it.
+
+Example output format (adapt field names to match the rubric):
+[
+  {"item_id": "cluster-42", "score": 4, "distinct_topics": 1, "outlier_count": 0, "reason": "One coherent topic; the title undersells it.", "ambiguous": false, "feedback": "The topic is coherent but the title is vague; name the shared login issue in the title."},
+  {"item_id": "cluster-17", "score": 2, "distinct_topics": 3, "outlier_count": 2, "reason": "Three unrelated topics merged because all mention 'timeout'.", "ambiguous": false, "feedback": "Three unrelated topics grouped on the shared word 'timeout'; splitting on the affected component would fix it."},
+  {"item_id": "cluster-99", "score": 3, "distinct_topics": 2, "outlier_count": 1, "reason": "Two related topics; unclear whether the rubric wants them split.", "ambiguous": true, "feedback": "Two sub-topics (import and export failures) that could reasonably be one or two clusters."}
+]
+
+Rules:
+- Evaluate each item independently
+- Score the same content the same way every time: the same item must receive the same score and the same fields on a repeat call
+- Score based on the rubric, not on how other items in this batch scored
+- If an item is empty or has only 1 element when it should have more, score it based on what is present
+- For very large items (many elements), focus on a representative subset and note if quality varies across the item
+- Every item in the batch MUST appear in your output
+</output-contract>
+```
+
+## Singleton Evaluation Template
+
+```
+You are a quality judge evaluating singleton items -- items that are currently NOT in any group/cluster.
+
+Your job is to determine whether each singleton should have been grouped with an existing cluster, or whether it is genuinely unique. Return structured JSON.
+
+<rubric>
+{singleton_rubric}
+</rubric>
+
+<singletons>
+{singletons_json}
+</singletons>
+
+<existing-clusters>
+A summary of existing clusters for reference (titles/themes only, not full contents):
+{cluster_summaries}
+</existing-clusters>
+
+<output-contract>
+Return ONLY a valid JSON array. No prose, no markdown, no explanation outside the JSON.
+
+Each element must have:
+- "item_id": the identifier of the singleton
+- All fields requested by the singleton rubric (should_cluster, best_cluster_id, confidence, etc.)
+
+Example output format (adapt field names to match the rubric):
+[
+  {"item_id": "issue-1234", "should_cluster": true, "best_cluster_id": "cluster-42", "confidence": 4},
+  {"item_id": "issue-5678", "should_cluster": false, "best_cluster_id": null, "confidence": 5}
+]
+
+Rules:
+- A singleton that genuinely has no match in existing clusters should get should_cluster: false
+- A singleton that clearly belongs in an existing cluster should get should_cluster: true with the cluster ID
+- High confidence (4-5) means you are very sure. Low confidence (1-2) means the item is borderline.
+- Every singleton in the batch MUST appear in your output
+</output-contract>
+```
+
+## Variable Reference
+
+| Variable | Source | Description |
+|----------|--------|-------------|
+| `{rubric}` | Spec `metric.judge.rubric` | User-defined scoring rubric |
+| `{items_json}` | Sampled output items | JSON array of items to evaluate (one batch worth) |
+| `{singleton_rubric}` | Spec `metric.judge.singleton_rubric` | User-defined rubric for singleton evaluation |
+| `{singletons_json}` | Sampled singleton items | JSON array of singleton items to evaluate |
+| `{cluster_summaries}` | Experiment output | Summary of existing clusters (titles/themes) for singleton reference |
+
+## Notes
+
+- Written for the `cheap` judge tier by default (`metric.judge.model`) -- prompts are concise and well-structured for smaller models; the harness resolves the tier to a concrete model
+- Where the dispatch surface exposes sampling controls, use the most deterministic setting available; the same item must score the same on a repeat call, which is what makes the run's judge cache and the calibration check valid
+- The rubric is part of the immutable measurement harness -- the experiment agent cannot modify it
+- The `reason` field does not change the score. The orchestrator keeps the reasons for the worst-scoring items so later hypotheses and workers can see why items failed, not just how much
+- The `feedback` field is required on every item. The orchestrator persists it under `judge.items[]`, groups it into the digest's failure themes, and generates hypotheses from those themes; a batch without feedback gives the loop nothing to act on
+- The `ambiguous` flag on items helps the orchestrator identify noisy evaluations without forcing bad scores
+- For singleton evaluation, the orchestrator provides cluster summaries (not full contents) to keep judge context lean
+- Each sub-agent evaluates one batch independently -- sub-agents do not see each other's results
+- **That independence is required, not merely preferred.** These scores decide accept or revert, so a judge must be a separate context from the one that authored the hypothesis and ran the experiment. Where no dispatch is available, block the judge pass rather than scoring inline. An orchestrator grading its own experiment is not a measurement.

package/skills/ce-optimize/references/loop.md

@@ -0,0 +1,286 @@
+# Phases 2-3: hypotheses and the optimization loop
+
+Read this before generating hypotheses and follow it for the whole loop. The SKILL.md body states the dependency pre-approval gate (the check that stops the run until the user has approved new dependencies) and the stopping criteria. This file carries hypothesis generation, batch selection, experiment dispatch, result collection and persistence, batch evaluation, the state update, the tick boundary, and the cross-cutting concerns. `<state-root>` is the run's ledger directory resolved in Phase 0 (`references/persistence.md`).
+
+## Phase 2: Hypothesis Generation
+
+### 2.1 Analyze Current Approach
+
+Read the code within `scope.mutable` to understand:
+- The current implementation approach
+- Obvious improvement opportunities
+- Constraints and dependencies between components
+
+When the baseline recorded `worst_cases`, read them. They show where the current approach fails and are one source of hypotheses, beside the code, attribution, and prior learnings. They do not limit the search: a hypothesis may replace the approach instead of fixing those cases.
+
+Write `source-digest.md` in the run's scratch directory once, now. It holds what an experiment worker needs to know about the material the target works on (evaluation data, corpora, large inputs, and how the harness uses them) and about the current approach, short enough to include in every worker prompt. Each worker starts with an empty context, so without the digest every experiment pays to read that material again. Rewrite the digest only when a keep changes something it describes.
+
+The next action is the cheapest executable step that would change what gets implemented. A locating measurement is one that finds where the cost actually sits (a profile, per-stage timing, a query count) rather than testing a change. It belongs in this phase when it is cheaper than an implementation experiment, would change whether a hypothesis is worth keeping or skipping, and can be taken. For a named workload's cost, the locating measurement is attribution: shares by stage, query, or call. Take it before an implementation experiment when the Phase 1 baseline total cannot say which hypothesis is worth keeping and those same conditions hold. The baseline total stays the scoring reference; the shares only decide what to try. A scored variant space does not require a performance profile: rubric evidence decides what to try, and numerical benefit may stay unknown.
+
+Do not treat the implementation backlog as empty, and do not proceed to wrap-up, while a cheaper locating measurement can still be taken and would change whether a hypothesis is kept or skipped. If such a measurement would change that decision but cannot be obtained, wrap up and say what blocked it. Do not implement without that measurement.
+
+Optionally read `references/agents/repo-research-analyst.md` and dispatch a generic subagent seeded with that local prompt for deeper codebase analysis if the scope is large or unfamiliar. Do not dispatch a standalone agent by type/name. Pass the active project and optimization context, request only question-specific scopes such as `patterns`, and go directly to current owning code. If the optimization cannot be scoped, allow one targeted root or workspace probe.
+
+A generic subagent is one whose whole brief is the prompt you send it. When the host's subagent tool makes you name an agent, use only a name you have seen the host list, and list its agents before the first dispatch if you have not. Pick the plain one that adds no role of its own, which still counts as a generic subagent. Never pick a specialist, such as a reviewer or worker agent, whose own role prompt, output format, or tool limits would replace or narrow your prompt.
+
+### 2.2 Generate Hypothesis List
+
+Generate an initial set of hypotheses. Each hypothesis should have:
+- **Description**: what to try
+- **Category**: one of the standard categories (signal-extraction, graph-signals, embedding, algorithm, preprocessing, parameter-tuning, architecture, data-handling) or a domain-specific category
+- **Priority**: high, medium, or low as a summary label
+- **Required dependencies**: any new packages or tools needed
+- **Opportunity**: the log schema's `opportunity` record (estimate or explicit unknown)
+
+Include user-provided hypotheses if any were given as input.
+
+Record an `opportunity` on every hypothesis using the log schema before implementation. Connect whatever observed cost or rubric evidence exists to the expected change in the target metric, with units, a comparison baseline, and the assumptions behind the estimate. Prefer a supported range or upper bound over a point estimate. If the benefit or the cost share cannot be estimated, record it as unknown and name the cheapest measurement that would resolve the uncertainty. A subjective priority score is not a measured benefit. The `priority` field does not rank the backlog.
+
+An unknown opportunity may sit on the backlog. It is not a runnable implementation experiment while a cheaper locating measurement would change whether it is kept or skipped.
+
+The backlog contains the credible opportunities supported by current evidence, not a required number of ideas. Rank by expected target benefit, confidence, implementation and measurement cost, and behavioral risk. Persist and verify the ranked opportunities and estimates at CP-2. Follow the SKILL.md body's reporting rule when you tell the user the findings that explain the chosen direction. The full backlog remains available on disk.
+
+### 2.3 Dependency Pre-Approval
+
+The SKILL.md body states this gate. A dependency the user does not approve stays in the backlog, is skipped in batch selection, and comes back at wrap-up. Record its outcome on each hypothesis as `dep_status: approved` or `needs_approval`, which is what batch selection reads.
+
+### 2.4 Record Hypothesis Backlog (CP-2)
+
+**MANDATORY CHECKPOINT.** Write the initial backlog to the experiment log file and verify. CP-2 is incomplete until each hypothesis in that write carries its opportunity record.
+```yaml
+hypothesis_backlog:
+  - description: "Remove template boilerplate before embedding"
+    category: "signal-extraction"
+    priority: high
+    dep_status: approved
+    required_deps: []
+    opportunity:
+      workload: "notification-clustering fixture"
+      baseline: "CP-1 baseline"
+      evidence: "judge rubric: boilerplate dilutes embeddings; no profile"
+      expected_benefit: "unknown; cheapest resolve: one judged stripped-vs-current sample"
+      confidence: "low: unmeasured"
+      cost_and_risk: "small edit; judge sample; no new deps"
+  - description: "Try HDBSCAN clustering algorithm"
+    category: "algorithm"
+    priority: medium
+    dep_status: needs_approval
+    required_deps: ["scikit-learn"]
+    opportunity:
+      workload: "notification-clustering fixture"
+      baseline: "CP-1 baseline"
+      evidence: "algorithm family untried on this fixture"
+      expected_benefit: "unknown; cheapest resolve: one exploratory run after dep approval"
+      confidence: "low: unmeasured"
+      cost_and_risk: "new dependency scikit-learn; judge sample"
+```
+
+---
+
+## Phase 3: Optimization Loop
+
+This phase runs in ticks until a stopping criterion is met. One tick is one batch: select (3.1), dispatch (3.2), collect and persist each result as it lands (3.3), evaluate (3.4), update state (3.5), check the stopping criteria (3.6). The tick boundary is the only place a turn may end with work outstanding; what that requires is stated there. A tick reaches the tick boundary after 3.6, or earlier at any step that cannot go further until results arrive for receipts recorded in `run_state.pending_waits` and has nothing else it can do first. An early crossing leaves the rest of the tick undone. It writes `run_state` with the in-tick time not yet counted added to `active_seconds`, and it leaves `tick` unchanged. When the awaited results land, the resumed turn continues the same batch at the step that was waiting. It does not select a new batch at 3.1. In a session with no wake capability, ticks follow one another in this session exactly as batches did before, and nothing in this file asks you to do anything else.
+
+### 3.1 Batch Selection
+
+Select hypotheses for this batch:
+- Build a runnable backlog by excluding hypotheses with `dep_status: needs_approval`
+- A hypothesis is not runnable while a cheaper locating measurement would still change whether it is kept or skipped
+- If `execution.mode` is `serial`, or the current decision needs to attribute a cost change to one lever, force `batch_size = 1`
+- Otherwise, `batch_size = min(runnable_backlog_size, execution.max_concurrent)`, further capped by `execution.max_experiments_per_tick` when the spec sets it
+- Shrink the batch until its estimated worker and judge cost fits every configured spend cap (3.6). If not even one experiment fits, stop with that criterion
+- Select by the ranked expected benefit, confidence, cost, and risk above; the priority label does not decide order. Category diversity breaks remaining ties.
+
+When a cheaper locating measurement can be taken and would still change whether a hypothesis is kept or skipped, take that measurement and update the backlog before selecting a batch. Do not treat that state as an empty backlog.
+
+When no executable next action remains, proceed to Phase 4 (wrap-up). An action is executable only if it can be taken now. A locating measurement that would change a keep-or-skip decision but cannot be obtained is a blocker, not a reason to keep the loop open. Wrap up and say what blocked it. Deferred dependencies are presented at wrap-up instead of the loop spinning forever.
+
+### 3.2 Dispatch Experiments
+
+The experiment's forecast is the backlog `opportunity` as of dispatch. Do not reconstruct it from later results. Copy that backlog value into the experiment entry at its first CP-3 write, including when that write recovers a `result.yaml` marker that has no forecast. Revised estimates for later experiments must not overwrite an earlier experiment's forecast. Missing forecasts in resumed legacy runs stay unrecorded.
+
+For each hypothesis in the batch, dispatch according to `execution.mode`. In `serial` mode, run exactly one experiment to completion before selecting the next hypothesis. In `parallel` mode, dispatch the batch concurrently.
+
+**Bounded dispatch.** Do not assume the host will accept all concurrent subagents at once; the active-subagent cap varies by host and profile and is independent of `execution.max_concurrent` (which caps worktrees, a separate budget). Queue the selected experiments, dispatch only as many as the host accepts, and when a capacity or active-agent-limit error appears, treat it as backpressure: retry the queued experiment after a slot frees rather than marking it failed. Mark an experiment failed only when dispatch fails for a non-capacity reason that survives correcting the invocation, or a successfully dispatched experiment errors/times out.
+
+The Phase 3 blocks below each set `SKILL_DIR` inline as well (the loaded `ce-optimize` skill directory; see the Bundled scripts note in Phase 1). Shell state does not persist from Phase 1, so each block carries its own assignment.
+
+**One dispatch recipe, per experiment.** The backend changes where the experiment runs and how its result comes back, not what the worker is told.
+
+1. **Isolate.** For `worktree` and `codex`, create the experiment worktree on the optimization branch:
+   ```bash
+   SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
+   WORKTREE_PATH=$(bash "$SKILL_DIR/scripts/experiment-worktree.sh" create "<spec_name>" <exp_index> "optimize/<spec_name>" <shared_files...>)  # creates optimize-exp/<spec_name>/exp-<NNN>
+   ```
+   Apply port parameterization if configured (set env vars for the measurement script). For `remote`, isolation is the worker's own checkout: record `base_sha` (`git rev-parse optimize/<spec_name>`) and make sure that commit is pushed to the remote the worker will fetch from.
+2. **Brief.** Fill the experiment prompt template (`references/experiment-prompt-template.md`): iteration number and spec name; hypothesis description and category; current best and baseline metrics; mutable and immutable scope; constraints and approved dependencies; the rolling window of the last 10 experiments as concise summaries; the source digest; and the current best's relevant `worst_cases`, from its kept entry or the baseline before any keep. For `remote`, append the template's measure-and-report delta with `base_sha`, the result ref `optimize-exp/<spec_name>/exp-<NNN>`, the measurement command, and the paired sample count the current ladder step needs.
+3. **Dispatch** by `execution.backend`, then record the return shape.
+
+| Backend | Dispatch | What returns |
+|---|---|---|
+| `worktree` | A subagent with the filled prompt, working in the experiment worktree | The subagent's completion, with its edits in the worktree |
+| `codex` | Write the filled prompt to a temp file and run `cat /tmp/optimize-exp-XXXXX.txt \| codex exec --skip-git-repo-check - 2>&1`, with the security posture from `execution.codex_security` (ask once per session when unset) | The process exit, with its edits in the worktree |
+| `remote` | The detached-worker dispatch the body's Execution Surface names, with the filled prompt; when no such capability is in the tool list, use the `worktree` row for the whole run and log the fallback | A receipt naming the launch. The result is the pushed result ref (`result.yaml` at its root) or, when the host delivers results as a store file or terminal message, that file or message |
+
+**Codex delegation condition.** Delegate to `codex exec` only when this session is not itself running inside a Codex sandbox and `.git` is writable; otherwise use the `worktree` row for that experiment and log the fallback. A set sandbox marker in your environment is proof you are inside one; an unset marker proves nothing on its own, which is why the writable-`.git` check stays in the test:
+```bash
+# any true branch means: do not delegate this experiment to codex
+test -n "${CODEX_SANDBOX:-}" || test -n "${CODEX_SESSION_ID:-}" || test ! -w .git
+```
+
+**Return shape.** A dispatch returns either a **result** (the work is done and its edits are where the table says) or a **receipt** (a handle for work that will finish after this call returns). A result goes straight to 3.3. A receipt is a pending wait: record it in `run_state.pending_waits` as `references/persistence.md` specifies, including the observable that will carry the result and the action to take (collect it through 3.3), and verify the write before dispatching the next experiment. Collect a receipt's result when that observable arrives, never by polling in a foreground loop.
+
+### 3.3 Collect and Persist Results
+
+Persist a `comparisons` record for each distinct reference, candidate, and workload pairing used in a decision. Each side's identity must uniquely identify the bytes that were measured; a shared HEAD is not enough when the candidate is uncommitted. Record the workload, both snapshots, and the decision's uncertainty and correctness evidence. Standalone and integrated pairings stay distinct in this array; a later in-place update must not replace a previously persisted distinct pairing. A runner-up's contribution is its confirmed change against the branch it was added to, not its standalone gain. These records explain results. `decide.mjs` still makes the accept or revert decision, using the existing snapshot fields.
+
+Process experiments as they complete: do NOT wait for the entire batch to finish before writing results. An experiment dispatched as a receipt completes when its recorded observable arrives; collect it through the same steps, then clear its `pending_waits` entry in the CP-3 write.
+
+**Collecting a `remote` result.** Fetch the result ref and read `result.yaml` from it. Accept it only when `base_sha` equals the `base_sha` you dispatched, the result ref's tip adds only `result.yaml` and its parent is `head_sha`, the commit range `base_sha..head_sha` touches only `scope.mutable`, and both `baseline` and `candidate` snapshots are present with the required objectives. A result failing any of these is `error` with the reason, logged and closed like a failed measurement; never repair a worker's numbers. An accepted result replaces steps 1 and 2 below: the worker's paired snapshots are the measurement and its `result.yaml` is the marker (copy it into `<state-root>/exp-<NNN>-result.yaml` so recovery does not depend on the remote ref). Continue from step 3 with the worker's `baseline` as the reference snapshot for `decide.mjs`, and record `machine` and `measured_by: worker` on the `comparisons` entry. A worker's `result.yaml` carries only the selection pairing, so a configured holdout answers a would-be keep with `next_measurement: holdout` exactly as for any other backend; that holdout is collected by the independent confirmation in 3.4, on the confirming machine, never by the worker that wrote the candidate. When `decide.mjs` asks for more selection samples (`add_sample`, `confirm`), re-dispatch the same worker (or a new one from the same `head_sha`) for that count; the entry stays one experiment.
+
+For each completed experiment, **immediately**:
+
+1. **Run measurement** in the experiment's worktree. Spend only the measurement the current decision needs (see Phase 1). When `stability.mode` is `ladder` and a smoke command is set, run that smoke check first. A smoke failure is terminally `degenerate`, and success proceeds to the first exploratory sample of `measurement.command` before comparison. Otherwise start with one exploratory sample. Pass `CE_OPTIMIZE_CENSOR_AFTER` to `measure.sh` only when elapsed wall time itself proves the candidate cannot become eligible, meaning every required objective is already hopeless, not merely the primary. Otherwise let measurement finish so other required objectives can still win, and let `decide.mjs` assess futility after the payload is complete.
+   ```bash
+   SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
+   bash "$SKILL_DIR/scripts/measure.sh" "<measurement.command>" <timeout_seconds> "<worktree_path>/<measurement.working_directory or .>" <env_vars...>
+   ```
+   When mode is `repeat`, keep running `repeat_count` times and aggregating as in Phase 1. When mode is `stable`, run once.
+
+2. **Write crash-recovery marker.** Immediately after measurement, write `result.yaml` in the experiment worktree containing the raw metrics. This keeps the measurement recoverable even if the agent crashes before updating the main log.
+
+3. **Read raw JSON output** from the measurement script. When it reports per-case results, keep the worst few, with what went wrong, as `worst_cases` for step 7.
+
+4. **Evaluate degenerate gates** (the cheap hard checks in `metric.degenerate_gates` that reject obviously broken output):
+   - For each gate in `metric.degenerate_gates`, parse the operator and threshold
+   - Compare the metric value against the threshold
+   - If ANY gate fails, mark the outcome `degenerate` and skip judge evaluation. This saves money.
+
+5. **If the degenerate gates pass AND primary type is `judge`**:
+   - **Check judge independence before dispatching.** A judge must not have authored the hypothesis or run the experiment it is scoring, and must not see other judges' results. That independence is what makes these scores usable for the accept or revert decision. If the host exposes no way to dispatch judges as separate agents, do **not** score inline. Mark the experiment's outcome `error` with the reason (judges undispatchable), skip judge evaluation exactly as a failed degenerate gate does, and continue to the log-and-append step so the entry is still written to disk. An experiment stopped here never carries judge metrics, so it is not eligible to become `best` and does not enter the accept/revert comparison. It is unmeasured, not poor-scoring. Report the blocker and its effect on the run to the user.
+   - Read the experiment's output (cluster assignments, search results, etc.)
+   - Apply stratified sampling per `metric.judge.stratification` config (using `sample_seed`)
+   - Group samples into batches of `metric.judge.batch_size`
+   - **Judge cache.** Before filling a batch, hash each sampled item's bytes and look the hash up in `judge-cache.yaml` beside the experiment log. An item already scored in this run under this rubric reuses its cached result (scores, `reason`, `feedback`, `ambiguous`); only uncached items are dispatched, and their results are written to the cache as they land. Identical output is scored once per run.
+   - Fill the judge prompt template (`references/judge-prompt-template.md`) for each batch
+   - Dispatch the `ceil(sample_size / batch_size)` judge sub-agents using the same bounded dispatch as Phase 3.2: queue them, dispatch to whatever concurrency the host accepts, and treat a capacity error as backpressure (retry the queued batch after a slot frees) rather than a scoring failure. These judge sub-agents are a separate budget from the experiment worktrees.
+   - Each sub-agent returns structured JSON scores with a `feedback` string per item. A batch that comes back without `feedback` is re-dispatched once; persist what the second call returns. Store every item under the entry's `judge.items[]` with its `content_hash`; the digest reads that feedback in step 3.5.
+   - Aggregate scores: compute the configured primary judge field from `metric.judge.scoring.primary` (which should match `metric.primary.name`) plus any `scoring.secondary` values
+   - Keep the items that scored worst by `metric.primary.direction`, about five, with their `reason` as `worst_cases` for step 7
+   - If `singleton_sample > 0`: also dispatch singleton evaluation sub-agents
+
+6. **Compare with `decide.mjs`.** Invoke it only after the degenerate gates pass and the payload holds every required objective value, meaning the hard metrics from measurement and the judge scores when those were collected. The payload is the spec as loaded plus the baseline and candidate snapshots. The script reads the nested spec (`metric`, `measurement.stability`) and decides eligibility, noise, and the ladder next step. Do not reconstruct a flattened payload, and do not re-derive the threshold in prose.
+   ```bash
+   SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
+   NODE="$(for c in node nodejs; do command -v "$c" >/dev/null 2>&1 && "$c" -e '' >/dev/null 2>&1 && { echo "$c"; break; }; done)";
+   [ -n "$NODE" ] || { echo "no working Node runtime on PATH (tried node, nodejs)" >&2; exit 1; };
+   "$NODE" "$SKILL_DIR/scripts/decide.mjs" "<payload.json>"
+   ```
+   If that probe finds no runtime, do not invoke an empty command. Mark the experiment `error` with that reason and continue the batch. Use `decision` and `next_measurement`. Collect the requested measurement and repeat this sequence whenever `next_measurement` is not `none`. Do not keep a candidate until `next_measurement` is `none`. Record `inconclusive` and `censored` as those outcomes, not as `reverted`. Each extra sample belongs to this same experiment: write it onto the existing entry at CP-3, then decide again.
+
+   **Held-out confirmation.** When the spec configures a holdout, `decide.mjs` answers a would-be keep with `next_measurement: holdout` until the payload carries a `holdout` snapshot pair. Collect it then and not before: the holdout depends on the selection decision, so it is never dispatched alongside the selection measurement, however independent the two calls look. Scoring it early spends on candidates that would have been rejected anyway and puts held-out feedback in front of you before the next hypothesis. Run `measurement.holdout.command` (or, for a judge primary, the judge sample under `metric.judge.confirmation_seed`, drawn only from items of each output that the selection sample did not draw) on the reference and on the candidate, persist the pairing as `kind: holdout` in `comparisons`, and decide again. The holdout result decides keep or not; it does not enter the digest, the rolling window summaries, or hypothesis generation. A holdout that does not confirm the gain withholds the keep; record the outcome the script returned.
+
+   **Per-experiment diagnostics.** When the harness or the judge dispatch reports cost, tokens, or latency, record them under the entry's `cost`. When `measurement.per_case` is true, record the script's `regressions` on the entry. Neither changes the decision.
+
+7. **IMMEDIATELY persist this experiment on disk (CP-3).** Do not defer this to batch evaluation. The durable unit is one log entry per experiment at `<state-root>/experiment-log.yaml`. After the first measurement, append that entry, including any `worst_cases`. After every later ladder sample for the same experiment, write the accumulated metrics and current outcome onto that same entry. Do not append a second entry for the same hypothesis, and do not rewrite a different experiment's samples. Write a decide terminal only when `next_measurement` is `none`. Until then the entry stays nonterminal, `promising` while the keep path still needs samples and `measured` otherwise (including an inconclusive result that still wants samples). When `next_measurement` is `none`, an eligible result stays `measured` until its diff is on the optimization branch; a non-eligible result gets the decide terminal (`reverted`, `inconclusive`, `censored`, `degenerate`). `kept` and `runner_up_kept` wait until that integration. The raw metrics are on disk and safe from context compaction.
+
+8. **VERIFY the write (CP-3 verification).** Read the experiment log back from disk and confirm the entry just written is present. If verification fails, retry the write. Do NOT proceed to the next experiment until this entry is confirmed on disk.
+
+**Why immediately + verify?** The agent's context window is NOT a durable store. Context compaction, session crashes, and restarts are expected during long runs, so results that exist only in the agent's memory are lost. The verification step catches silent write failures that would otherwise lose data.
+
+### 3.4 Evaluate Batch
+
+Every retained change must have earned its gain by doing the work the metric stands for. Before retaining any integration, including a runner-up, read the combined diff against the spec's `description` and `constraints`; a gain beyond the hypothesis's recorded opportunity upper bound calls for a slower read. A gain that comes from what the harness does not observe, or from breaking a `constraints` entry the harness does not check, is not a keep even though the gates, the ladder, and the holdout passed: revert it, record in the entry's `learnings` which observation or constraint the harness lacks, and carry that gap into the report. Closing the gap is a harness or spec change for a new run, not something this run patches.
+
+After all experiments in the batch have been measured:
+
+1. **Decide eligibility from `decide.mjs`, not from the primary metric alone.** An experiment is eligible when it improves at least one required objective beyond the configured comparison threshold and does not violate any other required objective. When `metric.objectives` is absent, the primary is the only required objective. `inconclusive` is not a keep.
+
+2. **Rank** the eligible experiments in the batch by the script's `rank_score` (primary relative gain when the primary moved; otherwise the strongest required-objective relative gain). Identify that winner as the experiment to keep. An eligible experiment may be kept even if the ranking primary did not move.
+
+3. **Confirm a `remote` winner independently before the keep test.** A `remote` winner enters this step when the decision on the worker's pair is eligible and asks for nothing more than the holdout: `keep` with `next_measurement: none`, or `promising` with `next_measurement: holdout`. For that winner, one independent measurement owns both the confirmation and the holdout: obtain a pairing the candidate's author did not produce by fetching `head_sha`, inspecting the diff against `base_sha`, and measuring it paired on this checkout or through a fresh confirmation worker that did not implement it, at the full configured protocol. When the worker's decision asked for `holdout`, that same measurement also runs the holdout on the reference and the candidate. Persist each pairing in `comparisons` (`kind: standalone` for the confirmation pair, `kind: holdout` for the holdout pair), every one with the `machine` and `measured_by` that produced it. Run `decide.mjs` on the payload built from those independent snapshots, never from the worker's. When that decision is not `keep` with `next_measurement: none`, the experiment takes that decide terminal and the worker's own claim is recorded as unconfirmed. The same independence that makes judge scores usable makes a self-measured keep usable.
+
+4. **If the winner's deciding result is `keep` with `next_measurement: none`: KEEP**
+   - For a `remote` winner, the deciding result is the independent decision from step 3. For any other winner, it is the last `decide.mjs` result from 3.3.
+   - A `keep` has already passed held-out confirmation when the spec configures one; an entry whose last decision is `promising` with `next_measurement: holdout` is not a keep until the holdout pair is collected and the script decides again
+   - Commit the experiment branch first so the winning diff exists as a real commit before any merge or cherry-pick (a `remote` winner is already a commit at `head_sha`; merge `head_sha`, not the result ref's tip, which only adds `result.yaml`)
+   - Include only mutable-scope changes in that commit; if no eligible diff remains, treat the experiment as non-improving and revert it
+   - Merge the committed experiment branch into the optimization branch
+   - Use the message `optimize(<spec-name>): <hypothesis description>` for the experiment commit
+   - After the merge succeeds, clean up the winner's experiment worktree and branch; the integrated commit on the optimization branch is the durable artifact
+   - This is now the new baseline for subsequent batches
+
+5. **Check file-disjoint runners-up** (up to `max_runner_up_merges_per_batch`):
+   - For each runner-up that also improved, check file-level disjointness with the kept experiment
+   - **File-level disjointness**: two experiments are disjoint if they modified completely different files. Same file = overlapping, even if different lines.
+   - If disjoint, cherry-pick the runner-up onto the new baseline and run the same decide loop as step 3.3 against a fresh sample set for that combined snapshot. Do not reuse the standalone experiment's accumulated samples; they were measured against the previous baseline. Collect further measurement whenever `next_measurement` is not `none`. Persist the combined pairing as `kind: integrated` on that same log entry without replacing the standalone comparison. Keep the original standalone log entry for audit.
+   - Keep the cherry-pick only when that result is eligible and `next_measurement` is `none` (outcome: `runner_up_kept`); then clean up that runner-up's experiment worktree and branch
+   - Otherwise revert the cherry-pick, log it as "promising alone but neutral/harmful in combination" (outcome: `runner_up_reverted`), then clean up the runner-up's experiment worktree and branch
+   - Stop after first failed combination
+
+6. **Handle deferred dependencies.** Experiments that need unapproved dependencies get outcome `deferred_needs_approval`
+
+7. **Close the rest.** Cleanup worktrees; a `remote` result ref may be deleted once its entry is verified in the log and its `result.yaml` copy is in `<state-root>`. `kept` and `runner_up_kept` are only for diffs on the optimization branch. Eligible candidates that were not integrated become `not_selected`. Leave `inconclusive`, `censored`, and `degenerate` as `decide.mjs` returned them.
+
+### 3.5 Update State (CP-4)
+
+**MANDATORY CHECKPOINT.** By this point, individual experiment results are already on disk (written in step 3.3). This step updates aggregate state and verifies.
+
+1. **Re-read the experiment log from disk.** Do not trust in-memory state. The log is the source of truth.
+
+2. **Finalize outcomes.** Update experiment entries from the step 3.4 evaluation (mark `kept`, `reverted`, `runner_up_kept`, etc.). Write these outcome updates to disk immediately.
+
+3. **Update the `best` section** in the experiment log if a new best was found. Update the running spend totals there too: judge spend, and the estimated spend of the whole run, which counts experiment workers, judges, and research subagents. Use the usage the host reports where it reports any, estimate the rest from the work dispatched, and record which parts are estimates. Write to disk.
+
+4. **Write strategy digest** to `<state-root>/strategy-digest.md`:
+   - Categories tried so far (with success/failure counts)
+   - Key learnings from this batch and overall
+   - Failure patterns that recur across `worst_cases`, and which ones the current best still shows
+   - Remaining opportunities, their supporting evidence, and whether current measurements still support their estimates; mark stale estimates for reassessment before selecting them
+   - Current best metrics and improvement from baseline
+   - **Failure themes:** the recurring `feedback` strings from the current best's `judge.items[]` (or its failing `cases` when `measurement.per_case` is set), grouped by what is wrong, each with a count and one representative item. Selection-sample items only; holdout items never enter the digest
+
+5. **Generate new hypotheses** based on learnings:
+   - Re-read the strategy digest from disk (not from memory)
+   - Read the rolling window (last 10 experiments from the log on disk)
+   - Do NOT read the full experiment log -- use the digest for broad context
+   - Start from the failure themes: a theme with a count worth acting on becomes at most one hypothesis that addresses the theme, with the representative item as its evidence. A rule per failing item is not a hypothesis; it is the pattern that overfits the selection sample
+   - After a keep on a cost target, re-measure how the cost divides among the parts before adding implementation hypotheses only when the keep leaves the current shares unable to say whether the next hypothesis is worth keeping
+   - Add new hypotheses to the backlog and write the updated backlog to disk
+
+6. **Write the updated hypothesis backlog to disk.** The backlog section of the experiment log must reflect newly added hypotheses and removed (tested) ones.
+
+7. **Write `run_state`.** Add the in-tick time not yet counted to `active_seconds`, increment `tick`, set `status`, and preserve all outstanding waits in `pending_waits`, including the deadline timer (`references/persistence.md`).
+
+**CP-4 Verification:** Read the experiment log back from disk. Confirm: (a) all experiment outcomes from this batch are finalized, (b) the `best` section reflects the current best, (c) the hypothesis backlog is updated, (d) `run_state` reflects this tick. Read `strategy-digest.md` back and confirm it exists. Only THEN proceed to the stopping criteria check.
+
+**Checkpoint: at this point, all state for this tick is on disk. If the agent crashes and restarts, it can resume from the experiment log without loss.**
+
+### 3.6 Check Stopping Criteria
+
+Stop the loop as soon as any one of these holds. Two clocks are involved: `run_state.active_seconds` counts time spent inside ticks since Phase 3 started, summed across ticks and excluding time spent waiting; `run_state.phase3_started_at` anchors calendar time. In a single uninterrupted session the two clocks read the same.
+
+- **Target reached**: `stopping.target_reached` is true and the current best meets every declared required target (`decide.mjs` `target_reached` on the current-best snapshot). When `metric.objectives` is absent, that is the single `metric.primary.target` if set. Do not stop for a primary-only hit while another required target is still unmet.
+- **Max iterations**: total experiments run >= `stopping.max_iterations`
+- **Max active hours**: `run_state.active_seconds` (not time since the invocation, not time spent waiting) >= `stopping.max_hours`
+- **Wall-clock backstop**: calendar time since `run_state.phase3_started_at` >= `stopping.max_wall_hours`. No wake extends it. A wake that fires after it has passed collects any arrived results, then goes to Phase 4.
+- **Spend cap reached**: the spend already recorded plus the estimated cost of the next dispatch would pass a configured cap: `metric.judge.max_total_cost_usd` for judge spend, `stopping.max_total_cost_usd` for whole-run spend. A cap limits spend; it does not trigger after spend passes it. Check it before every dispatch that costs money, including Phase 2 research and each batch with its judge passes, not only at this step
+- **Plateau**: no improvement for `stopping.plateau_iterations` **consecutive** experiments
+- **Manual stop**: the user interrupts. Save state, then go to Phase 4.
+- **No runnable hypothesis left**: no executable next action remains
+
+If none is met, cross the tick boundary.
+
+**Tick boundary.** Check the wall deadline before crossing, including an early crossing while results are missing. At or after the deadline, persist already-arrived results and go to Phase 4 without waiting or integrating candidates. Otherwise, with experiment receipts pending, apply `references/persistence.md` (The Wait Record and Ticks) to continue an attached wait, park with bounded automatic resumption, or hand back a manual-resume checkpoint. Resume the same batch at its waiting step. With no experiment receipts pending, start the next tick at 3.1; a deadline timer alone is not an experiment to wait for.
+
+### 3.7 Cross-Cutting Concerns
+
+**Codex failure cascade**: Track consecutive Codex delegation failures. After 3 consecutive failures, auto-disable Codex for remaining experiments and fall back to subagent dispatch. Log the switch.
+
+**Error handling**: Classify a failed measurement from what `measure.sh` actually signaled. The censored stderr marker (with exit 125) is `censored`. Exit 124 is `timeout`. Any other non-zero exit (including 125 without that marker) is `error`. Log that outcome with the error message, revert the experiment, and continue the batch.
+
+**Progress reporting:** follow the SKILL.md body's reporting rule. Base any reported improvement on persisted, verified measurements, distinguish preliminary from confirmed results, and state what was measured and any limits on the claim. Batch counts and cumulative scoring cost remain in the log for wrap-up. Report them to the user during the run when they affect a decision.
+
+**Crash recovery**: See the Persistence Discipline section. Per-experiment `result.yaml` markers are written in step 3.3. Individual experiment results are appended to the log immediately in step 3.3. Tick-level state (outcomes, best, digest, `run_state`) is written in step 3.5. On resume (Phase 0.4), the log at `<state-root>` is the ground truth. Scan for any `result.yaml` markers not yet reflected in the log, then collect any pending wait whose result has arrived.
+
+---