@dzhechkov/skills-bto 1.3.3 → 1.3.4

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dzhechkov/skills-bto",
-  "version": "1.3.3",
+  "version": "1.3.4",
   "description": "Build-Benchmark-Test-Optimize skill pack for Claude Code — deterministic benchmarking, quality gates, witness chain, judge attestation, and optimization",
   "main": "src/cli.js",
   "bin": {

package/templates/.claude/agents/bto-optimizer-worker.md

@@ -179,3 +179,10 @@ Each worker writes a log regardless of success:
 ## Reusability Note
 This template is artifact-type agnostic. Replace BASE_ARTIFACT_PATH and RUBRIC_PATH
 to optimize any text artifact: prompts, skills, presentations, research sections, code docstrings.
+
+## Hold-out validation (delegate to `dz bto-optimize`)
+When `dz` is available, do NOT accept a variant on its tuning-set score alone. Split scenarios
+(`dz bto-optimize --split`), score candidates on the tune set, validate the top candidate on the held-out set,
+and call `dz bto-optimize --select` — it accepts a winner only if the weakest dimension improves on the HOLDOUT
+with no regression elsewhere. This is the anti-Goodhart guard the heuristic loop lacks. See
+`skills/bto/modules/optimize.md` → "Rigorous validation".

package/templates/.claude/commands/bto-optimize.md

@@ -206,3 +206,11 @@ This command is also invoked internally by `/bto` as the final step (OPTIMIZE ph
 - Always save a `.pre-optimize.bak` backup before overwriting original
 - Round 3 evaluation must use Layer 2 (sonnet judges), not Layer 1 (haiku)
 - Preserve original artifact intent — optimization changes HOW, not WHAT
+
+## Rigorous validation (recommended when `dz` ≥ 0.3.119)
+
+For a reproducible, budget-capped run that resists judge-gaming, follow the **"Rigorous validation (hold-out +
+no-regress)"** protocol in `skills/bto/modules/optimize.md`: it splits scenarios into a tuning set and a held-out
+set, tunes candidates on the tune set, and accepts a winner ONLY if it lifts the weakest dimension on the UNSEEN
+holdout without regressing the others (`dz bto-optimize --split/--plan/--select`). Prose-only, diff-confirmed,
+never auto-written. Absent `dz`, the heuristic evolutionary loop runs unchanged.

package/templates/.claude/skills/bto/modules/optimize.md

@@ -199,3 +199,46 @@ Stop optimization immediately if:
 2. Critical structural checks (Layer 0) fail on any variant
 3. Artifact semantics change fundamentally
 4. User requests stop
+
+---
+
+## Rigorous validation (hold-out + no-regress) — `dz bto-optimize` (when `dz` ≥ 0.3.119)
+
+The evolutionary loop above SELECTS the highest score on the **same** eval it tuned on — which lets a variant
+that flatters the judge panel (verbosity, buzzwords) win even if it doesn't help real users (**Goodhart**). When
+the `dz` CLI is available, delegate the deterministic validation steps to it so acceptance is gated on **unseen**
+scenarios with a **no-regress** guard and a **hard budget cap**. This strengthens the loop; it does not replace it.
+
+**Scope (Phase-1):** only the **directive prose** of `SKILL.md` is mutated — the "when to activate" block + the
+core instruction. Frontmatter, section headings, and examples are OFF-LIMITS (`--scope-check` rejects a candidate
+that touches them). Never auto-write; the human confirms the diff.
+
+Protocol (folds into the steps above):
+
+1. **Split** the BTO scenario ids into a tuning set and a held-out set (deterministic):
+   ```bash
+   dz bto-optimize --split --scenarios @scenarios.json --holdout 0.34   # → { tune:[...], holdout:[...] }
+   ```
+2. **Plan the budget** and respect the hard cap (the engine trims candidates/rounds to fit and reports it):
+   ```bash
+   dz bto-optimize --plan --candidates 5 --rounds 1 --tune <#tune> --holdout <#holdout> --max 24
+   ```
+   Never run more judge passes than the printed plan.
+3. **Tune** — generate the K prose candidates (existing strategies) and score each via the judge panel on the
+   **tune** scenarios ONLY. Record per-candidate per-dimension scores.
+4. **Validate** — score the top tune candidate(s) on the **holdout** scenarios (unseen).
+5. **Select** — the engine accepts a candidate ONLY if, on the holdout, the weakest dimension improves AND no
+   other dimension / the aggregate regresses (beyond `--tolerance`, default 0):
+   ```bash
+   dz bto-optimize --select --baseline @baseline.json --candidates @candidates.json   # → { winner|null, reason }
+   ```
+   `baseline.json` = `{ "holdout": {<DimScores>} }`; each candidate = `{ id, prose, tune:{DimScores}, holdout:{DimScores} }`.
+   A tune-winner that regresses on the holdout is **rejected** — this is the anti-gaming guarantee.
+6. **Confirm gate** — show the prose diff + the tune/holdout deltas and let the human accept before any write:
+   ```bash
+   dz bto-optimize --scope-check --original SKILL.md --candidate candidate.md   # prose-only guard
+   dz bto-optimize --diff --original SKILL.md --candidate candidate.md          # the diff to confirm
+   ```
+
+Grounded in dspy.ts MIPROv2 (propose → minibatch-tune → **validate on held-out** → best). Absent `dz` ⇒ fall
+back to the heuristic loop above (unchanged).