code-lens-cli 0.7.1__tar.gz → 0.8.0__tar.gz

code_lens_cli-0.8.0/.claude/skills/eval/SKILL.md

@@ -0,0 +1,399 @@
+---
+name: eval
+description: >
+  Run one scripts-eval set — one `(target, question)` row from
+  `experiments/scripts_eval/corpus.yaml` × 3 trials × one arm — including
+  tester subagent dispatches + captures, plus (for arm C) judge subagent
+  dispatches + records, then `summarize` + commit to the round's
+  accumulator file. Use when the user says "run eval set", "eval",
+  "scripts-eval", "round-NN set", or asks to execute a row of the corpus.
+  Three arms: A (banned — rider forbids the seer skills), B (directed
+  — rider instructs use of seer skills), C (organic — rider permits
+  but doesn't direct). Two judge pairs: A-vs-B ("do the skills help
+  when used") and A-vs-C ("do the skills get adopted organically").
+  `judge prepare --pair AB|AC` selects the pair.
+---
+
+# scripts-eval — running a set
+
+This skill drives one **set** of the scripts-eval harness:
+one `(target, question)` row × 3 trials × one arm.
+
+The harness pipeline (`trial` / `validate` / `judge` / `summarize`) and
+the corpus (`corpus.yaml`) are repo state — this skill is just the
+operator procedure that sequences them per session.
+
+## When to push back
+
+Before doing anything, verify the user's intent matches the session
+state. Stop and ask if any of these hold:
+
+- `env | grep SEER_EVAL_RUN_ID` is empty → the harness hooks no-op, no
+  metrics get captured. Operator needs to re-launch with the env vars
+  exported.
+- `SEER_EVAL_ARM` is set to anything other than `A`, `B`, or `C` → bad config.
+- User says "do arm C" but the matching arm-A cells don't exist on
+  disk under `experiments/scripts_eval/results/$SEER_EVAL_RUN_ID/arm-A/`
+  → arm A must complete first; there's nothing to pair against.
+
+All three arms run with `repo-map` and `code-lookup` enabled on disk.
+Arm-A's constraint is **verbal** — the rider in the dispatched prompt
+is the sole guard against the subagent using the seer skills. Do not
+edit the rider; copy it verbatim. (Earlier versions of this skill
+physically moved `.claude/skills/repo-map/` aside for arm A as
+defense-in-depth; that step was dropped because the rider proved
+sufficient and the move-aside dance made operator setup brittle.)
+
+Three arms, three questions they answer:
+
+- **A (banned)** — verbal rider forbids both seer skills. Establishes
+  the "without the new skills" baseline.
+- **B (directed)** — verbal rider instructs the subagent to use the
+  seer skills where applicable. Establishes the "with the new skills,
+  when actually used" upper bound.
+- **C (organic)** — verbal rider permits but does not direct use of
+  the seer skills. Measures organic adoption rate.
+
+A-vs-B is the primary "do the skills help?" comparison; A-vs-C is the
+adoption canary. The judge supports both pairs via the `--pair` flag.
+
+## Preflight (every session)
+
+```bash
+env | grep -E "^SEER_EVAL_(RUN_ID|ARM)="
+# expect both set to the intended round / arm
+```
+
+If unset, export them in your shell before launching `claude`:
+
+```bash
+# arm-A session (banned):
+export SEER_EVAL_RUN_ID=2026-05-NN-round-XX SEER_EVAL_ARM=A
+# arm-B session (directed):
+export SEER_EVAL_RUN_ID=2026-05-NN-round-XX SEER_EVAL_ARM=B
+# arm-C session (organic):
+export SEER_EVAL_RUN_ID=2026-05-NN-round-XX SEER_EVAL_ARM=C
+```
+
+`experiments/scripts_eval/switch-arm.sh A|B|C <run_id>` does the same
+thing.
+
+If this is the first set of the run (idempotent, safe to re-run):
+
+```bash
+uv run --group experiments python -m experiments.scripts_eval.manifest \
+    init --run $SEER_EVAL_RUN_ID
+```
+
+## Arm-A procedure
+
+**For each trial in {1, 2, 3}:**
+
+1. Read the question template for the target's `question_id` from
+   `experiments/scripts_eval/corpus.yaml`. Look up the target's path
+   from the same file's `targets:` list.
+
+2. Substitute `{repo_path}` (or `{workspace_root}` for the workspace
+   question) in the template, then append **verbatim**:
+
+   ```text
+
+   Constraints (verbatim):
+   - You may NOT use the `repo-map` skill, `python -m seer.repo`,
+     the `seer.repo` Python module, or any `scripts/*.sh` paths under
+     `.claude/skills/repo-map/`.
+   - You may NOT use the `code-lookup` skill, the `seer.lookup`
+     Python module, the `seer grep` / `seer recent` / `seer classify`
+     CLI verbs, or any `scripts/*.sh` paths under
+     `.claude/skills/code-lookup/`.
+     If you cannot answer without them, say so explicitly and stop.
+   - Use only Read, Grep, Glob, and Bash.
+   - After answering, append two sections and stop:
+     ### tools_used
+     - <ToolName>: <count>  (one line per distinct tool)
+     ### evidence
+     - <one path per line>
+   ```
+
+3. **Before dispatch** — start the trial. The script reads
+   `CLAUDE_CODE_SESSION_ID` from env, stamps an in-flight record, and
+   prints the `trial_id` to stdout:
+
+   ```bash
+   TRIAL_ID=$(uv run --group experiments python -m experiments.scripts_eval.trial \
+       start --run $SEER_EVAL_RUN_ID --arm $SEER_EVAL_ARM \
+       --target <target> --question <question_id> --trial <n>)
+   ```
+
+   (For the workspace-scope question, omit `--target`.)
+
+4. Dispatch **one** `Explore` subagent with the full prompt.
+
+5. After the subagent finishes, end the trial. The script reads the
+   subagent's sidechain transcript from
+   `$HOME/.claude/projects/<encoded_cwd>/<session>/subagents/agent-*.jsonl`
+   and writes the cell JSON:
+
+   ```bash
+   uv run --group experiments python -m experiments.scripts_eval.trial \
+       end --trial-id "$TRIAL_ID"
+   ```
+
+6. Confirm the cell JSON appeared under
+   `experiments/scripts_eval/results/$SEER_EVAL_RUN_ID/arm-A/`.
+
+**After all 3 trials**, summarize + commit:
+
+```bash
+uv run --group experiments python -m experiments.scripts_eval.summarize \
+    --run $SEER_EVAL_RUN_ID \
+    --out docs/eval-rounds/$SEER_EVAL_RUN_ID.md
+
+git add docs/eval-rounds/$SEER_EVAL_RUN_ID.md
+git commit -m "$SEER_EVAL_RUN_ID: arm-A captured for <target>/<question_id> (3 trials)"
+```
+
+Report back: cell count under arm-A/, what's pending for arm-B and
+arm-C on this set, the next pending set per the run-state table in the
+accumulator file.
+
+## Arm-B procedure
+
+Arm-B captures the **directed** trials so the A-vs-B judge run can
+assess "do the skills help when actually used?". Capture happens in
+its own session (`SEER_EVAL_ARM=B`); the A-vs-B judges then run in
+the arm-C session's Judge phase, alongside the A-vs-C judges
+(`judge prepare --pair AB`).
+
+**For each trial in {1, 2, 3}:**
+
+1. Substitute the corpus question template (same target / question
+   resolution as arm A), then append **verbatim** the arm-B rider:
+
+   ```text
+
+   Constraints (verbatim):
+   - For this question, you MUST use the seer skills where they
+     apply:
+       * `repo-map` (`scripts/profile.sh`, `scripts/connections.sh`,
+         `scripts/graph.sh` under `.claude/skills/repo-map/`) for
+         repo overview, dependencies, and workspace shape.
+       * `code-lookup` (`seer grep`, `seer recent`, `seer classify`,
+         or the equivalent scripts under
+         `.claude/skills/code-lookup/`) for symbol references,
+         recent commit-symbol diffs, and project-kind classification.
+     Only fall back to Read / Grep / Glob / Bash for facts the
+     scripts do not cover.
+   - After answering, append two sections and stop:
+     ### tools_used
+     - <ToolName>: <count>  (one line per distinct tool)
+     ### evidence
+     - <one path per line>
+   ```
+
+2. Bookend the dispatch with `trial start` and `trial end` exactly
+   as in arm A, just with `--arm B`:
+
+   ```bash
+   TRIAL_ID=$(uv run --group experiments python -m experiments.scripts_eval.trial \
+       start --run $SEER_EVAL_RUN_ID --arm $SEER_EVAL_ARM \
+       --target <target> --question <question_id> --trial <n>)
+   # dispatch one Explore subagent with the rendered prompt above
+   uv run --group experiments python -m experiments.scripts_eval.trial \
+       end --trial-id "$TRIAL_ID"
+   ```
+
+3. Confirm the cell JSON appeared under
+   `experiments/scripts_eval/results/$SEER_EVAL_RUN_ID/arm-B/`.
+
+**After all 3 trials**, summarize + commit:
+
+```bash
+uv run --group experiments python -m experiments.scripts_eval.summarize \
+    --run $SEER_EVAL_RUN_ID \
+    --out docs/eval-rounds/$SEER_EVAL_RUN_ID.md
+
+git add docs/eval-rounds/$SEER_EVAL_RUN_ID.md
+git commit -m "$SEER_EVAL_RUN_ID: arm-B captured for <target>/<question_id> (3 trials)"
+```
+
+Report back: cell count under arm-B/, whether the subagent actually
+followed the directive (look at the `### tools_used` of each arm-B
+cell — `B_did_not_use_scripts` is a finding, not a bug), the next
+pending set per the run-state table.
+
+## Arm-C procedure
+
+**Precondition check (mandatory):**
+
+```bash
+ls experiments/scripts_eval/results/$SEER_EVAL_RUN_ID/arm-A/<target>-<question_id>-t*.json
+# expect: 3 files (t1, t2, t3)
+```
+
+If fewer than 3, stop — arm A must complete first.
+
+### Tester phase
+
+**For each trial in {1, 2, 3}:**
+
+1. Substitute the corpus question template (same as arm A) but with the
+   arm-C rider:
+
+   ```text
+
+   Constraints (verbatim):
+   - You may use the `repo-map` skill (and its scripts under
+     `.claude/skills/repo-map/`) and the `code-lookup` skill (and its
+     scripts under `.claude/skills/code-lookup/`) at your discretion.
+     This includes `seer grep` / `seer recent` / `seer classify`.
+   - After answering, append two sections and stop:
+     ### tools_used
+     - <ToolName>: <count>
+     ### evidence
+     - <one path per line>
+   ```
+
+2. Bookend the dispatch with `trial start` and `trial end`:
+
+   ```bash
+   TRIAL_ID=$(uv run --group experiments python -m experiments.scripts_eval.trial \
+       start --run $SEER_EVAL_RUN_ID --arm $SEER_EVAL_ARM \
+       --target <target> --question <question_id> --trial <n>)
+   # dispatch one Explore subagent with the rendered prompt above
+   uv run --group experiments python -m experiments.scripts_eval.trial \
+       end --trial-id "$TRIAL_ID"
+   ```
+
+   (For the workspace-scope question, omit `--target`.)
+
+### Judge phase
+
+Two pairs are judged independently:
+
+- **A-vs-C** — the original "with vs without (organic)" comparison.
+- **A-vs-B** — the new "with (directed) vs without" comparison; needs
+  arm-B cells captured first.
+
+Both pairs use the same `prepare` / `record` flow; only `--pair`
+(`AC` or `AB`) and the matching `--blind-label-for-<arm>` flags differ.
+
+**For each trial in {1, 2, 3}**, run the A-vs-C judge first (if arm-C
+cells exist) and then the A-vs-B judge (if arm-B cells exist):
+
+1. Prepare the blinded job. `--pair` defaults to `AC`; pass `--pair AB`
+   for the A-vs-B run.
+
+   ```bash
+   uv run --group experiments python -m experiments.scripts_eval.judge \
+       prepare --run $SEER_EVAL_RUN_ID \
+       --pair AC \
+       --pair-key <target>/<question_id>/<n> \
+       --seed 0 > /tmp/judge-AC-<n>.json
+   ```
+
+2. Materialise the prompt to a text file for dispatch (`jq -j`
+   joins without adding a trailing newline, so the bytes match what
+   `prepare` emitted):
+
+   ```bash
+   jq -j '.prompt_text' /tmp/judge-AC-<n>.json > /tmp/judge-AC-<n>.txt
+   ```
+
+3. Dispatch the judge subagent. **The description prefix is
+   load-bearing** — the `pre_tool` hook recognises `scripts_eval judge:`
+   and skips logging, so the judge dispatch does not pollute the
+   harness's `raw/` directory:
+
+   - `subagent_type`: `general-purpose`
+   - `description`: `scripts_eval judge: AC <target>/<question_id>/<n>`
+   - `prompt`: the verbatim contents of `/tmp/judge-AC-<n>.txt`
+
+4. Capture the subagent's final-text response and record. The blind
+   labels for an AC pair come back as `blind_label_for_A` /
+   `blind_label_for_C`:
+
+   ```bash
+   A_LABEL=$(jq -r .blind_label_for_A /tmp/judge-AC-<n>.json)
+   C_LABEL=$(jq -r .blind_label_for_C /tmp/judge-AC-<n>.json)
+   uv run --group experiments python -m experiments.scripts_eval.judge \
+       record --run $SEER_EVAL_RUN_ID \
+       --pair AC \
+       --pair-key <target>/<question_id>/<n> \
+       --blind-label-for-a "$A_LABEL" \
+       --blind-label-for-c "$C_LABEL" \
+       --verdict-file -
+   ```
+
+5. **Repeat the four steps with `--pair AB`** to judge the directed
+   arm. The job JSON for an AB pair carries `blind_label_for_A` and
+   `blind_label_for_B` (no `_C`); use `--blind-label-for-b` instead of
+   `--blind-label-for-c`:
+
+   ```bash
+   uv run --group experiments python -m experiments.scripts_eval.judge \
+       prepare --run $SEER_EVAL_RUN_ID \
+       --pair AB \
+       --pair-key <target>/<question_id>/<n> \
+       --seed 0 > /tmp/judge-AB-<n>.json
+   jq -j '.prompt_text' /tmp/judge-AB-<n>.json > /tmp/judge-AB-<n>.txt
+   # …dispatch general-purpose subagent with description
+   #   "scripts_eval judge: AB <target>/<question_id>/<n>" and the txt prompt…
+   A_LABEL=$(jq -r .blind_label_for_A /tmp/judge-AB-<n>.json)
+   B_LABEL=$(jq -r .blind_label_for_B /tmp/judge-AB-<n>.json)
+   uv run --group experiments python -m experiments.scripts_eval.judge \
+       record --run $SEER_EVAL_RUN_ID \
+       --pair AB \
+       --pair-key <target>/<question_id>/<n> \
+       --blind-label-for-a "$A_LABEL" \
+       --blind-label-for-b "$B_LABEL" \
+       --verdict-file -
+   ```
+
+   If `record` exits non-zero with `non-JSON` / `winner` / `margin` /
+   `blind_label` in the error, re-dispatch the judge subagent for that
+   trial and re-record. `record` is idempotent on replay — the operator's
+   recovery path is "re-dispatch + re-record"; no manual cell editing.
+
+Storage: AC verdicts land under `cell["judges"]["AC"]` (and are mirrored
+to `cell["judge"]` for back-compat with pre-phase-2 readers); AB verdicts
+land under `cell["judges"]["AB"]` only.
+
+### Wrap-up
+
+```bash
+uv run --group experiments python -m experiments.scripts_eval.validate \
+    --run $SEER_EVAL_RUN_ID
+
+uv run --group experiments python -m experiments.scripts_eval.summarize \
+    --run $SEER_EVAL_RUN_ID \
+    --out docs/eval-rounds/$SEER_EVAL_RUN_ID.md
+
+git add docs/eval-rounds/$SEER_EVAL_RUN_ID.md
+git commit -m "$SEER_EVAL_RUN_ID: completed <target>/<question_id> (both arms + judge)"
+```
+
+Report back:
+- A-vs-B winners (A / B / tie) and A-vs-C winners (A / C / tie).
+- Whether arm B and arm C actually used the seer scripts (look at the
+  `### tools_used` of each cell — `B_did_not_use_scripts` and
+  `C_did_not_use_scripts` are findings, not bugs).
+- The next pending set per the run-state table.
+
+## Reading the run state
+
+The committed run-state table and per-set verdicts live in
+`docs/eval-rounds/$SEER_EVAL_RUN_ID.md`, between the
+`<!-- runstate:start -->` / `<!-- runstate:end -->` and
+`<!-- evidence:start -->` / `<!-- evidence:end -->` markers. `summarize.py`
+rewrites those regions idempotently — do not hand-edit them.
+
+The accumulator file is also the operator's source of truth for what's
+pending: a row's `arm-A` or `arm-C` count below `3/3` means more trials
+are needed; `judged` below the arm counts means judges still owe verdicts.
+
+## Cite-don't-import
+
+This skill is original to seer-cli (the harness only exists here). When
+promoted upstream, it would re-vendor into steward's skill suppliers —
+update `docs/skill-sources.md` accordingly at that point.

{code_lens_cli-0.7.1 → code_lens_cli-0.8.0}/CHANGELOG.md

@@ -5,6 +5,21 @@ All notable changes to this project will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/). This project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0] - 2026-05-16
+
+### Added
+
+- scripts-eval: arm-B (directed-use) — rider explicitly instructs the subagent to use repo-map + code-lookup scripts. Cells captured under `results/<run>/arm-B/`. `corpus.yaml` arms field becomes `[A, B, C]`.
+- scripts-eval: `judge.py` is now pair-aware. `judge prepare` / `judge record` take `--pair AC|AB|BC`; new label flag `--blind-label-for-b`. Verdicts land under `cell["judges"][pair]`; the AC pair still mirrors to legacy `cell["judge"]` for back-compat. New `iter_jobs_pair` / `record_verdict_pair` public APIs; old `iter_jobs` / `record_verdict` stay as AC-pair wrappers.
+- scripts-eval: summarize.py renders both A-vs-B and A-vs-C winner tallies in the run-state table, with per-pair verdict tables in the evidence section.
+- eval skill: arm-A rider tightened to also forbid the code-lookup skill / seer.lookup / seer grep/recent/classify so "without" means without both new skills. Added arm-B procedure section and pair-aware judge procedure (--pair AB | AC).
+
+### Changed
+
+- eval skill: switch-arm.sh no longer moves .claude/skills/repo-map/ on disk; arm-A relies on the verbal rider alone (rider proved sufficient; move-aside dance made operator setup brittle).
+- scripts-eval: report.py violation patterns now also flag code-lookup script use; report aggregates median validation across all three arms; per-cell view shows every captured arm.
+- scripts-eval: validate.py / backfill.py iterate over _io.ARMS instead of hardcoding (A, C).
+
 ## [0.7.1] - 2026-05-16
 
 ### Changed

{code_lens_cli-0.7.1 → code_lens_cli-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: code-lens-cli
-Version: 0.7.1
+Version: 0.8.0
 Summary: seer — codebase lookup and indexing for agent skills (greenfield AgentCulture sibling).
 Project-URL: Homepage, https://github.com/agentculture/seer-cli
 Project-URL: Issues, https://github.com/agentculture/seer-cli/issues

code_lens_cli-0.8.0/docs/eval-rounds/2026-05-16-round-02.md

@@ -0,0 +1,74 @@
+# scripts-eval Round 02 — 2026-05-16
+
+Run id: `2026-05-16-round-02`
+Corpus: `experiments/scripts_eval/corpus.yaml` (corpus_version: 1)
+Trials per cell: 3
+Arms: A (verbal rider bans both seer skills), B (verbal rider
+directs use of seer skills), C (verbal rider permits but doesn't
+direct). Judge pairs: A-vs-B + A-vs-C.
+
+This file is the round's **evidence accumulator**. The two
+`<!-- runstate:... -->` / `<!-- evidence:... -->` regions below are
+rewritten by `experiments/scripts_eval/summarize.py` at the end of every
+session — operator never edits them by hand.
+
+Raw per-cell JSONs stay gitignored under
+`experiments/scripts_eval/results/2026-05-16-round-02/`. This file is
+the committed evidence.
+
+## Procedure
+
+The operator procedure is locked in the **`eval` skill** —
+`.claude/skills/eval/SKILL.md`. Read or invoke that skill from a fresh
+session; this file does not duplicate it.
+
+Round 02 is the first run under the post-move-aside skill: all three
+arms run with `.claude/skills/repo-map/` and
+`.claude/skills/code-lookup/` present on disk; arm A's verbal rider in
+the dispatched prompt is the sole guard against the subagent using
+either skill. It is also the first run under the pair-aware judge
+(`judge prepare --pair AB|AC`), so the run-state table carries A-vs-B
+winners alongside A-vs-C.
+
+## Per-session preflight (your shell, *before* `claude`)
+
+```bash
+# arm-A session (banned):
+export SEER_EVAL_RUN_ID=2026-05-16-round-02 SEER_EVAL_ARM=A
+
+# arm-B session (directed):
+export SEER_EVAL_RUN_ID=2026-05-16-round-02 SEER_EVAL_ARM=B
+
+# arm-C session (organic; same run id, different arm):
+export SEER_EVAL_RUN_ID=2026-05-16-round-02 SEER_EVAL_ARM=C
+```
+
+Then launch `claude` from the seer-cli repo root.
+
+## Run state
+
+<!-- runstate:start -->
+
+| target | question | arm-A | arm-B | arm-C | A-vs-B judged | A-vs-B (A/B/tie) | A-vs-C judged | A-vs-C (A/C/tie) |
+|---|---|---|---|---|---|---|---|---|
+| culture | q-profile-overview | 3/3 | 3/3 | 0/3 | 3/3 | 2/1/0 | 0/3 | 0/0/0 |
+
+<!-- runstate:end -->
+
+## Evidence per set
+
+<!-- evidence:start -->
+
+### culture / q-profile-overview
+
+**AB** winners: A=2, B=1, tie=0 (of 3 judged).
+
+| trial | winner | margin | A duration | B duration | A tools | B tools | judge reasoning |
+|---|---|---|---|---|---|---|---|
+| 1 | A | slight | 136.7s | 47.2s | Bash:26, Read:8 | Bash:4, Read:2, Skill:2 | Y is more accurate about the agentirc shim relationship and gives a concrete runnable test command with coverage details, while X invents some specifics (e.g. publish.yml trigger) and lists skills not matching Y's set; both are solid but Y is tighter and more actionable. |
+| 2 | A | slight | 78.7s | 66.8s | Bash:27, Read:8 | Bash:4, Read:2, Skill:1 | Y is more accurate and actionable with concrete test commands, CI workflow names, and architectural choices, while X includes some questionable specifics (e.g., exact 90% coverage floor, v12.1.7 release details) that read as potentially fabricated. |
+| 3 | B | slight | 74.6s | 42.5s | Bash:20, Read:7 | Bash:5, Skill:2 | Y adds concrete dependency list, release state, and vendored skills inventory that aid actionability, while X has a slightly cleaner structure but less verifiable detail; Y's minor duplication (afi-cli listed twice) is a small blemish. |
+
+*No AC verdicts yet.*
+
+<!-- evidence:end -->

{code_lens_cli-0.7.1 → code_lens_cli-0.8.0}/docs/skill-sources.md

@@ -17,7 +17,7 @@ explicitly — these copies do not auto-update.
 | `version-bump` | `steward` (`../steward/.claude/skills/version-bump/`) | 2026-05-15 | None — portable verbatim. Pure Python, no per-repo customization. seer-cli's `CHANGELOG.md` keeps a `# Changelog` + intro-prose header, so the first `## [` entry is a valid insertion point for the upstream script. |
 | `repo-map` | _internal implementation_ — seer-cli origin | 2026-05-15 | **Runtime:** thin shell wrappers under `.claude/skills/repo-map/scripts/{profile,connections,graph}.sh` that invoke `uv run --directory <repo-root> python -m seer.repo <verb>`. Engine lives in `seer/repo/` in this repo. **Divergence:** N/A — this skill is original to seer-cli, not vendored from steward. If/when promoted upstream, this row flips to a `Re-vendor from steward` pointer. |
 | `code-lookup` | _internal implementation_ — seer-cli origin | 2026-05-16 | **Runtime:** thin shell wrapper under `.claude/skills/code-lookup/scripts/classify.sh` that invokes `uv run --directory <repo-root> python -m seer classify`. Engine lives in `seer/lookup/` in this repo. **Divergence:** N/A — original to seer-cli, sibling of `repo-map`. If/when promoted upstream, this row flips to a `Re-vendor from steward` pointer. |
-| `eval` | _internal implementation_ — seer-cli origin | 2026-05-15 | **Runtime:** locked operator procedure for one scripts-eval set (one `(target, question)` row × 3 trials × one arm). Backing CLIs live in `experiments/scripts_eval/` (`capture`, `validate`, `judge`, `summarize`, `manifest`); hooks live in `experiments/scripts_eval/hooks/`. Round-agnostic — reads `$SEER_EVAL_RUN_ID` and writes the round's accumulator at `docs/eval-rounds/$SEER_EVAL_RUN_ID.md`. **Divergence:** N/A — original to seer-cli (the harness only exists here). If/when promoted upstream, this row flips to a `Re-vendor from steward` pointer. |
+| `eval` | _internal implementation_ — seer-cli origin | 2026-05-16 | **Runtime:** locked operator procedure for one scripts-eval set (one `(target, question)` row × 3 trials × one arm). Three arms — **A** (banned), **B** (directed), **C** (organic) — and two judge pairs — **A-vs-B** ("do the skills help when used?") and **A-vs-C** ("do the skills get adopted organically?"). Backing CLIs live in `experiments/scripts_eval/` (`trial`, `validate`, `judge`, `summarize`, `manifest`); hooks live in `experiments/scripts_eval/hooks/`. `judge prepare --pair AB\|AC` selects the pair; verdicts land under `cell["judges"][pair]` (the AC pair also mirrors to legacy `cell["judge"]`). Round-agnostic — reads `$SEER_EVAL_RUN_ID` and writes the round's accumulator at `docs/eval-rounds/$SEER_EVAL_RUN_ID.md`. **Divergence:** N/A — original to seer-cli (the harness only exists here). If/when promoted upstream, this row flips to a `Re-vendor from steward` pointer. |
 
 ## Vendoring policy
 

{code_lens_cli-0.7.1 → code_lens_cli-0.8.0}/experiments/scripts_eval/_io.py

@@ -25,13 +25,16 @@ def eval_run_id() -> str | None:
     return val if val else None
 
 
+ARMS = ("A", "B", "C")
+
+
 def eval_arm() -> str | None:
-    """Return SEER_EVAL_ARM (must be 'A' or 'C'), or None if unset."""
+    """Return SEER_EVAL_ARM (must be 'A', 'B', or 'C'), or None if unset."""
     val = os.environ.get("SEER_EVAL_ARM")
     if val is None or val == "":
         return None
-    if val not in ("A", "C"):
-        raise ValueError(f"SEER_EVAL_ARM must be 'A' or 'C' (got {val!r})")
+    if val not in ARMS:
+        raise ValueError(f"SEER_EVAL_ARM must be one of {ARMS} (got {val!r})")
     return val
 
 

{code_lens_cli-0.7.1 → code_lens_cli-0.8.0}/experiments/scripts_eval/backfill.py

@@ -327,7 +327,7 @@ def main(argv: list[str] | None = None) -> int:
         return 1
 
     total = 0
-    for arm in ("A", "C"):
+    for arm in _io.ARMS:
         arm_dir = rd / f"arm-{arm}"
         if not arm_dir.exists():
             continue

{code_lens_cli-0.7.1 → code_lens_cli-0.8.0}/experiments/scripts_eval/corpus.yaml

@@ -2,7 +2,7 @@ corpus_version: 1
 
 config:
   trials_per_cell: 3
-  arms: [A, C]
+  arms: [A, B, C]
   workspace_root: /home/spark/git
 
 targets: