rote-cli 0.7.0__tar.gz → 0.8.0__tar.gz

{rote_cli-0.7.0 → rote_cli-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rote-cli
-Version: 0.7.0
+Version: 0.8.0
 Summary: Graduate fuzzy AI skills into deterministic, reliable workflows
 Project-URL: Homepage, https://github.com/trevhud/rote
 Project-URL: Repository, https://github.com/trevhud/rote

{rote_cli-0.7.0 → rote_cli-0.8.0}/skills/rote-graduate/references/eval-estimates.md

@@ -25,6 +25,13 @@ Ground rules:
   (agents batch parallel calls). If it says "repeat until you have 25
   qualified contacts", estimate the realistic iteration count from any
   hints (quota sizes, batch sizes, typical yields), not the worst case.
+- **Steps that repeat per item MUST declare `iterations`.** A "process
+  each row" step costs its per-row turns *times the row count* — this
+  is almost always the dominant term, and flattening it understates the
+  whole run by 5–20×. Put the per-iteration turns in `estimated_turns`
+  and the realistic item-count range in `iterations` (page sizes,
+  pagination labels, batch quotas, and filter windows in the skill text
+  are your evidence). The loader multiplies them; never pre-multiply.
 - **Include the overhead the prose hides.** Reading reference files,
   re-checking rubric sections, correcting malformed tool calls — a
   realistic agent spends turns on these. A step whose happy path is 2
@@ -54,9 +61,14 @@ steps:
     phase: "1"
     estimated_turns: {low: 1, high: 2}
     estimated_tool_calls: {low: 4, high: 4}
-totals:                             # optional — omit if the sum is right
-  low: 28
-  high: 45
+  - description: "Push each eligible invoice and capture its sent date"
+    node_id: process_invoices_loop
+    phase: "Processing"
+    estimated_turns: {low: 3, high: 6}      # PER ROW —
+    iterations: {low: 20, high: 90}         # — × realistic row count
+totals:                             # optional — omit if the sum is right;
+  low: 88                           # must be WHOLE-RUN (post-iteration)
+  high: 602                         # when any step declares iterations
 notes: >-
   Lead generation dominates: the loop re-queries ZoomInfo with adjusted
   filters until the quota is met; typical campaigns converge in 3-5
@@ -72,30 +84,39 @@ Field reference:
 | `steps[].description` | yes | The step, in the skill's own vocabulary |
 | `steps[].node_id` | no | IR node id, only when the mapping is clean 1:1 |
 | `steps[].phase` | no | Source skill phase, same convention as the IR |
-| `steps[].estimated_turns` | yes | `{low, high}` agent turns |
+| `steps[].estimated_turns` | yes | `{low, high}` agent turns (per iteration when `iterations` is set) |
 | `steps[].estimated_tool_calls` | no | `{low, high}` tool calls, when distinct from turns |
-| `totals` | no | Whole-run `{low, high}`; omit to mean "sum the steps" |
+| `steps[].iterations` | when the step repeats | `{low, high}` repeats per run (rows, pages, items); whole-run cost = turns × iterations |
+| `totals` | no | Whole-run `{low, high}`; omit to mean "sum the steps (× iterations)" |
 | `notes` | no | The one paragraph a reader needs to trust the numbers |
 
 Validated by `rote.eval.sidecar.EvalEstimates` — if you can run Python,
 `load_eval_estimates("eval.yaml")` is the dry check, same pattern as
 `load_pipeline`.
 
-## Calibration anchors (BDR reference points)
-
-From real runs of BDR-scale skills (7 phases, ~20 distinct steps,
-heavy tool use):
-
-- Whole-run turn counts landed in the **30–57 turn** range.
-- Wall clock was **~13 seconds per turn** including tool execution.
-- A single fixed API call step: 1–2 turns. A batch-and-review step:
-  2–4. A bounded search loop: 6–15. A genuinely open research step:
-  4–10.
-
-If your step estimates sum to something wildly outside the anchor range
-for a comparably sized skill, re-examine before writing — either the
-skill really is bigger/smaller (fine, say so in `notes`) or a step
-estimate is off.
+## Calibration anchors (measured production runs)
+
+Whole-run turn counts are regime-dependent — anchor to the skill's
+*shape*, not to a universal range:
+
+- **Sequential tool-heavy skills** (BDR-scale: 7 phases, ~20 distinct
+  steps, no per-item loop): **30–57 turns** measured. ~13 seconds per
+  turn including tool execution.
+- **Per-item loop skills** (browser automation over a table, per-record
+  processing): measured production runs landed at **184 and 730 turns**
+  — the loop multiplier dominates everything else in the file. If a
+  skill says "for each row…", expect hundreds of turns, not tens, and
+  express it via `iterations`, never by inflating per-step turns.
+- Per-step anchors: a single fixed API call step: 1–2 turns. A
+  batch-and-review step: 2–4. A bounded search loop: 6–15. A genuinely
+  open research step: 4–10. One iteration of a per-row browser cycle
+  (locate, act, confirm, refresh): 2–6.
+
+If your estimate lands wildly outside the anchor for the skill's shape,
+re-examine before writing — either the skill really is bigger/smaller
+(fine, say so in `notes`) or a step estimate is off. A per-item skill
+whose whole-run total sums to under ~100 turns almost certainly has a
+missing `iterations` block.
 
 ## What NOT to do
 

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/__init__.py

@@ -1,3 +1,3 @@
 """rote — graduate fuzzy AI skills into deterministic workflows."""
 
-__version__ = "0.7.0"
+__version__ = "0.8.0"

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/cli.py

@@ -784,7 +784,9 @@ def _run_empirical(
     from rote.eval.empirical import (
         EmpiricalResult,
         append_corpus,
+        mcp_servers_for_pipeline,
         measured_to_dict,
+        min_expected_turns_for,
         render_measured_markdown,
         run_pipeline_trial,
         run_skill_trial,
@@ -846,6 +848,25 @@ def _run_empirical(
 
     skill_runs = []
     if skill_dir is not None:
+        # Wire the pipeline's MCP bindings into the skill trial so the
+        # agent runs over the same live tools the pipeline binds to —
+        # the measurement is only representative when the skill can
+        # actually pull its data.
+        mcp_servers, missing_servers = mcp_servers_for_pipeline(pipeline)
+        if mcp_servers:
+            print(
+                "rote eval: wiring MCP servers into the skill trial: "
+                + ", ".join(sorted(mcp_servers)),
+                file=sys.stderr,
+            )
+        if missing_servers:
+            print(
+                "rote eval: WARNING — pipeline binds MCP servers with no "
+                "resolvable endpoint (set ROTE_MCP_<SERVER>_URL): "
+                + ", ".join(missing_servers)
+                + " — skill trials will be flagged unrepresentative",
+                file=sys.stderr,
+            )
         for i in range(args.trials):
             print(
                 f"rote eval: skill trial {i + 1}/{args.trials} "
@@ -859,6 +880,9 @@ def _run_empirical(
                     model=skill_model,
                     max_turns=args.max_turns,
                     output_fields=sorted(output_fields) or None,
+                    mcp_servers=mcp_servers or None,
+                    missing_mcp_servers=missing_servers or None,
+                    min_expected_turns=min_expected_turns_for(pipeline),
                 )
             )
     else:

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/eval/empirical.py

@@ -43,15 +43,69 @@ from rote.graduator.drivers.claude import (
     DEFAULT_ALLOWED_TOOLS,
     build_subscription_env,
 )
+from rote.ir import NodeKind, Pipeline
 
 RESULT_FILENAME = "result.json"
+MCP_CONFIG_FILENAME = "mcp-config.json"
 DEFAULT_CORPUS_PATH = Path.home() / ".local" / "share" / "rote" / "eval-corpus.jsonl"
 
+# IR transport values → Claude Code mcp-config `type` values. The CLI
+# spells Streamable HTTP as plain "http" (verified against the current
+# docs at code.claude.com/docs/en/mcp — July 2026, CLI v2.1.205).
+_TRANSPORT_TO_MCP_TYPE = {"streamable-http": "http", "sse": "sse"}
+
 
 class EmpiricalError(RuntimeError):
     """A trial could not be started (misconfiguration, missing deps)."""
 
 
+# ───────── MCP auto-wiring ─────────
+
+
+def mcp_servers_for_pipeline(pipeline: Pipeline) -> tuple[dict[str, dict[str, Any]], list[str]]:
+    """Build the ``claude -p`` MCP server config from the pipeline's bindings.
+
+    The graduated pipeline's ``mcp:`` bindings name exactly the servers the
+    source skill uses, so wiring them into the skill trial makes the
+    "before" measurement representative: the agent pulls real data over the
+    same tools instead of erroring out or improvising. URL resolution
+    follows the adapters' rule — an explicit ``mcp.url`` wins, else the
+    ``ROTE_MCP_<SERVER>_URL`` env var.
+
+    Returns ``(mcp_servers, missing)``: the ``mcpServers`` mapping for the
+    config file, and the servers the pipeline binds that could not be
+    resolved to an endpoint (a trial run without them is flagged
+    unrepresentative rather than silently measured).
+    """
+    servers: dict[str, dict[str, Any]] = {}
+    missing: list[str] = []
+    for node in pipeline.nodes:
+        if node.mcp is None or node.mcp.server in servers or node.mcp.server in missing:
+            continue
+        binding = node.mcp
+        url = binding.url or os.environ.get(f"ROTE_MCP_{binding.server.upper()}_URL")
+        if not url:
+            missing.append(binding.server)
+            continue
+        servers[binding.server] = {
+            "type": _TRANSPORT_TO_MCP_TYPE[binding.transport],
+            "url": url,
+        }
+    return servers, missing
+
+
+def min_expected_turns_for(pipeline: Pipeline) -> int:
+    """The floor below which a skill run cannot have done the work.
+
+    Each ``external_call`` node is a data pull the source skill also
+    performs; an agent run with fewer turns than data pulls necessarily
+    skipped sources (auth failure, refusal, early error) and would poison
+    the calibration corpus if fitted as a real measurement.
+    """
+    pulls = sum(1 for n in pipeline.nodes if n.kind is NodeKind.EXTERNAL_CALL)
+    return max(3, pulls)
+
+
 @dataclass(frozen=True)
 class MeasuredRun:
     """One executed trial, either side."""
@@ -69,11 +123,22 @@ class MeasuredRun:
     model: str | None = None
     # Pipeline side (from the emitted $ROTE_USAGE_LOG hook):
     judge_usage: tuple[dict[str, Any], ...] = field(default_factory=tuple)
+    flags: tuple[str, ...] = ()
+    """Reliability flags: why this run should not calibrate priors.
+    ``errored`` / ``hit_max_turns`` (truncated — its cost is a floor, not
+    a measurement) / ``suspiciously_few_turns`` (fewer turns than the
+    pipeline has data pulls — the agent never did the work) /
+    ``missing_mcp_servers`` (the skill's tools weren't all available)."""
 
     @property
     def succeeded(self) -> bool:
         return self.error is None
 
+    @property
+    def representative(self) -> bool:
+        """Safe to fit priors from: succeeded and nothing flagged."""
+        return self.succeeded and not self.flags
+
 
 # ───────── Skill (before) trials ─────────
 
@@ -116,12 +181,24 @@ def run_skill_trial(
     executable: str = "claude",
     timeout_seconds: float = 1800.0,
     output_fields: list[str] | None = None,
+    mcp_servers: dict[str, dict[str, Any]] | None = None,
+    missing_mcp_servers: list[str] | None = None,
+    min_expected_turns: int | None = None,
 ) -> MeasuredRun:
     """Run the raw skill once under ``claude -p`` and measure it.
 
     Billing follows the ClaudeDriver rules: API-key env vars are
     scrubbed so the run uses the subscription OAuth session, and the
     reported ``total_cost_usd`` is Claude Code's list-price notional.
+
+    ``mcp_servers`` (from :func:`mcp_servers_for_pipeline`) is written to
+    a config file and passed via ``--mcp-config --strict-mcp-config`` —
+    strict, so the trial exercises exactly the pipeline-declared servers,
+    never whatever happens to be configured on the host. Each wired
+    server's tools are allowlisted wholesale (``mcp__<server>__*``): the
+    skill may legitimately use more of a server's tools than the pipeline
+    binds. ``missing_mcp_servers`` and ``min_expected_turns`` feed the
+    reliability flags on the returned run.
     """
     skill_path = Path(skill_dir).resolve()
     if not (skill_path / "SKILL.md").is_file():
@@ -134,6 +211,10 @@ def run_skill_trial(
     prompt = _skill_prompt(skill_path, input_payload, output_fields)
     with tempfile.TemporaryDirectory(prefix="rote-eval-skill-") as workdir_str:
         workdir = Path(workdir_str)
+        if mcp_servers:
+            allowed_tools = ",".join(
+                [allowed_tools, *(f"mcp__{name}__*" for name in sorted(mcp_servers))]
+            )
         args = [
             executable,
             "-p",
@@ -149,6 +230,12 @@ def run_skill_trial(
             "--max-turns",
             str(max_turns),
         ]
+        if mcp_servers:
+            mcp_config_path = workdir / MCP_CONFIG_FILENAME
+            mcp_config_path.write_text(
+                json.dumps({"mcpServers": mcp_servers}, indent=2), encoding="utf-8"
+            )
+            args += ["--mcp-config", str(mcp_config_path), "--strict-mcp-config"]
         started = time.monotonic()
         try:
             proc = subprocess.run(
@@ -165,6 +252,13 @@ def run_skill_trial(
                 output=None,
                 error=f"timed out after {timeout_seconds:g}s",
                 model=model,
+                flags=_reliability_flags(
+                    error="timeout",
+                    subtype=None,
+                    turns=None,
+                    min_expected_turns=min_expected_turns,
+                    missing_mcp_servers=missing_mcp_servers,
+                ),
             )
         wall = time.monotonic() - started
 
@@ -191,20 +285,60 @@ def run_skill_trial(
         error = f"claude exited {proc.returncode}: {(proc.stderr or proc.stdout)[:500]}"
     elif output is None:
         error = parse_error
+    turns = meta.get("num_turns")
     return MeasuredRun(
         wall_seconds=wall,
         output=output,
         error=error,
-        turns=meta.get("num_turns"),
+        turns=turns,
         cost_usd=meta.get("total_cost_usd"),
         input_tokens=usage.get("input_tokens"),
         cache_read_tokens=usage.get("cache_read_input_tokens"),
         cache_creation_tokens=usage.get("cache_creation_input_tokens"),
         output_tokens=usage.get("output_tokens"),
         model=model,
+        flags=_reliability_flags(
+            error=error,
+            subtype=meta.get("subtype"),
+            turns=turns,
+            min_expected_turns=min_expected_turns,
+            missing_mcp_servers=missing_mcp_servers,
+        ),
     )
 
 
+def _reliability_flags(
+    *,
+    error: str | None,
+    subtype: str | None,
+    turns: int | None,
+    min_expected_turns: int | None,
+    missing_mcp_servers: list[str] | None,
+) -> tuple[str, ...]:
+    """Why a run must not calibrate priors (empty = representative).
+
+    Flags are additive and deliberately structural — no LLM judges the
+    run. A truncated run's cost is a floor, not a measurement; a run
+    with fewer turns than the pipeline has data pulls never did the
+    work; a run whose skill lacked its tools measured a different task.
+    """
+    flags: list[str] = []
+    if error is not None:
+        flags.append("errored")
+    if subtype == "error_max_turns":
+        flags.append("hit_max_turns")
+    if (
+        min_expected_turns is not None
+        and turns is not None
+        and turns < min_expected_turns
+        and subtype != "error_max_turns"
+    ):
+        flags.append("suspiciously_few_turns")
+    if missing_mcp_servers:
+        flags.append("missing_mcp_servers")
+    return tuple(flags)
+
+
 # ───────── Pipeline (after) trials ─────────
 
 
@@ -452,8 +586,15 @@ def suggested_priors(
     skill_runs: tuple[MeasuredRun, ...], priors: Priors | None = None
 ) -> dict[str, float]:
     """Prior values re-fitted from measured agent runs (reported, never
-    silently applied — the static model's constants stay explicit)."""
+    silently applied — the static model's constants stay explicit).
+
+    Only representative runs participate: a flagged run (errored,
+    truncated at max-turns, too few turns to have done the work, or
+    missing its MCP servers) measured something other than the skill
+    executing normally, and fitting it would poison the calibration.
+    """
     priors = priors or Priors()
+    skill_runs = tuple(r for r in skill_runs if r.representative)
     fitted: dict[str, float] = {}
     timed = [r for r in skill_runs if r.turns and r.wall_seconds]
     if timed:
@@ -500,6 +641,7 @@ def append_corpus(result: EmpiricalResult, *, generated_at: str, path: Path | No
                 "cache_creation_tokens": r.cache_creation_tokens,
                 "output_tokens": r.output_tokens,
                 "succeeded": r.succeeded,
+                "flags": list(r.flags),
             }
             for r in result.skill_runs
         ],
@@ -577,6 +719,17 @@ def render_measured_markdown(result: EmpiricalResult, catalog: PricingCatalog) -
         )
         lines.append("")
 
+    flagged = [r for r in result.skill_runs if r.flags]
+    if flagged:
+        lines.append(
+            "Unrepresentative agent runs (excluded from prior fits and "
+            "not safe to read as the skill's real cost):"
+        )
+        for i, r in enumerate(result.skill_runs, 1):
+            if r.flags:
+                lines.append(f"- trial {i}: {', '.join(r.flags)}")
+        lines.append("")
+
     fitted = suggested_priors(result.skill_runs)
     if fitted:
         lines.append("Suggested prior re-fits from these runs (not auto-applied):")
@@ -610,6 +763,7 @@ def measured_to_dict(result: EmpiricalResult, catalog: PricingCatalog) -> dict[s
                     "cost_usd": r.cost_usd,
                     "output_tokens": r.output_tokens,
                     "error": r.error,
+                    "flags": list(r.flags),
                 }
                 for r in result.skill_runs
             ],

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/eval/estimate.py

@@ -369,9 +369,13 @@ def estimate_skill(
 ) -> AgentRunEstimate:
     """Predict running the skill as raw agent instructions.
 
-    Cache-aware transcript model: turn *i* re-reads C₀ + (i−1)·Δ tokens
-    of context, of which only ~Δ is first-sight (billed at cache-write
-    price); the rest is a cache read. Totals over N turns:
+    Cache-aware transcript model: turn *i* re-reads
+    min(C₀ + (i−1)·Δ, cap) tokens of context, of which only ~Δ is
+    first-sight (billed at cache-write price); the rest is a cache read.
+    The cap (``priors.transcript_cap_tokens``) models harness compaction:
+    real transcripts saturate near the context window instead of growing
+    without bound, so on long runs the cached-read total transitions from
+    quadratic to linear. Uncapped totals over N turns:
 
     * fresh  ≈ C₀ + (N−1)·Δ
     * cached ≈ (N−1)·C₀ + Δ·(N−2)(N−1)/2
@@ -404,14 +408,24 @@ def estimate_skill(
         turn_method = f"structural heuristic ({step_count} steps in SKILL.md)"
 
     def _fresh(n: float) -> float:
+        # Fresh (first-sight) content is unaffected by the cap: each
+        # turn's new tool results are written to cache once even when
+        # compaction evicts older content to make room.
         return context_tokens + max(0.0, n - 1) * priors.transcript_growth_per_turn
 
     def _cached(n: float) -> float:
+        # Turn i (i >= 2) re-reads min(C₀ + (i−2)·Δ, cap): quadratic
+        # growth until the transcript saturates at the compaction cap,
+        # linear after. m counts the pre-saturation re-reads.
         if n <= 1:
             return 0.0
-        return (n - 1) * context_tokens + priors.transcript_growth_per_turn * (
-            (n - 2) * (n - 1) / 2
-        )
+        delta = priors.transcript_growth_per_turn
+        cap = priors.transcript_cap_tokens
+        reads = n - 1
+        if context_tokens >= cap or delta <= 0:
+            return reads * min(float(context_tokens), cap)
+        m = min(reads, (cap - context_tokens) / delta + 1)
+        return m * context_tokens + delta * (m - 1) * m / 2 + (reads - m) * cap
 
     output_tokens = turns.scale(priors.output_tokens_per_turn)
 

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/eval/priors.py

@@ -46,6 +46,19 @@ class Priors:
     turn *i* re-reads roughly C₀ + (i−1)·Δ tokens of context.
     """
 
+    transcript_cap_tokens: float = 165_000.0
+    """Ceiling on the transcript the agent re-reads per turn. Real agent
+    harnesses compact or truncate the conversation as it approaches the
+    model's context window, so per-turn input saturates instead of
+    growing without bound. Without the cap, the quadratic model
+    overshoots long runs severely (a measured 730-turn browser-automation
+    run read 78M cache tokens; uncapped Δ·N²/2 alone predicts 240M+).
+
+    Measured 2026-07-09 on two production browser-automation runs
+    (184 and 730 turns, Sonnet 4.6, 200k window): per-turn cache-read
+    plateaued at ~165k in both.
+    """
+
     system_overhead_tokens: int = 16_000
     """Context the agent carries beyond the skill itself (system prompt,
     tool definitions, environment preamble) — part of C₀.

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/eval/scorecard.py

@@ -253,8 +253,9 @@ class Scorecard:
             lines.append(f"- Token counting: {self.skill.token_count_method}.")
             lines.append(
                 "- Agent cost model: cache-aware transcript growth — turn *i* "
-                "re-reads C₀ + (i−1)·Δ context; fresh tokens bill at cache-write "
-                "price, re-reads at cache-read price."
+                "re-reads min(C₀ + (i−1)·Δ, cap) context, where the cap models "
+                "harness compaction near the context window; fresh tokens bill "
+                "at cache-write price, re-reads at cache-read price."
             )
         lines.append("- HITL gate waits are human time and excluded from wall-clock on both sides.")
         lines.append(

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/eval/sidecar.py

@@ -53,6 +53,27 @@ class StepEstimate(BaseModel):
     phase: str | None = None
     estimated_turns: TurnRange
     estimated_tool_calls: TurnRange | None = None
+    iterations: TurnRange | None = Field(
+        default=None,
+        description=(
+            "How many times this step repeats per run (per row, per page, "
+            "per item). When present, estimated_turns/estimated_tool_calls "
+            "are PER-ITERATION and the whole-run contribution is their "
+            "product. Absent means the step runs once. This is where loop "
+            "cost lives: a 3-turn step over 200 rows is 600 turns, and a "
+            "sidecar that cannot say so underestimates loop-dominated "
+            "skills by an order of magnitude."
+        ),
+    )
+
+    def turn_contribution(self) -> TurnRange:
+        """This step's whole-run turns: per-iteration turns × iterations."""
+        if self.iterations is None:
+            return self.estimated_turns
+        return TurnRange(
+            low=self.estimated_turns.low * self.iterations.low,
+            high=self.estimated_turns.high * self.iterations.high,
+        )
 
     @model_validator(mode="before")
     @classmethod
@@ -81,11 +102,17 @@ class EvalEstimates(BaseModel):
     notes: str | None = None
 
     def turn_range(self) -> TurnRange:
-        """The whole-run turn estimate: explicit totals, else the step sum."""
+        """The whole-run turn estimate: explicit totals, else the step sum.
+
+        The step sum multiplies per-iteration estimates by their
+        ``iterations`` range, so a per-row loop step contributes its
+        whole-run cost. An explicit ``totals`` must therefore also be a
+        whole-run (post-multiplication) figure.
+        """
         if self.totals is not None:
             return self.totals
-        low = sum(s.estimated_turns.low for s in self.steps)
-        high = sum(s.estimated_turns.high for s in self.steps)
+        low = sum(s.turn_contribution().low for s in self.steps)
+        high = sum(s.turn_contribution().high for s in self.steps)
         return TurnRange(low=low, high=high)
 
 

{rote_cli-0.7.0 → rote_cli-0.8.0}/src/rote/graduator/__init__.py

@@ -29,6 +29,7 @@ from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any
 
+from rote.eval.sidecar import EVAL_SIDECAR_FILENAME, load_eval_estimates
 from rote.graduator.drivers import (
     DriverError,
     GraduatorDriver,
@@ -299,6 +300,9 @@ class Graduator:
             pipeline = self._repoint_source_skill(
                 output_dir / "pipeline.yaml", pipeline, skill_dir, output_dir
             )
+            self._repoint_sidecar_source_skill(
+                output_dir / EVAL_SIDECAR_FILENAME, skill_dir, output_dir
+            )
 
             # Re-point the result's path to the moved location for the
             # caller's benefit.
@@ -395,6 +399,40 @@ class Graduator:
             pipeline_yaml.write_text(original, encoding="utf-8")
             return pipeline
 
+    @staticmethod
+    def _repoint_sidecar_source_skill(
+        sidecar_path: Path, skill_dir: Path, output_dir: Path
+    ) -> None:
+        """Rewrite the eval sidecar's ``source_skill`` the same way.
+
+        The sidecar suffers the identical dead-pointer failure as
+        pipeline.yaml (the agent records the path relative to its temp
+        work dir), just with a quieter blast radius: its field is
+        documentary today, but a stale path checked into an example is a
+        bug report waiting to happen. Same surgical substitution, same
+        restore-on-breakage guarantee, validated with the sidecar's own
+        loader.
+        """
+        if not sidecar_path.is_file():
+            return
+        try:
+            source_ref = os.path.relpath(skill_dir, output_dir)
+        except ValueError:
+            source_ref = str(skill_dir)
+
+        original = sidecar_path.read_text(encoding="utf-8")
+        replacement = f"source_skill: {source_ref}"
+        text, n = re.subn(r"(?m)^source_skill:[^\n]*$", replacement, original, count=1)
+        if n == 0:
+            text, n = re.subn(r"(?m)^(version:[^\n]*)$", rf"\1\n{replacement}", original, count=1)
+        if n == 0:
+            return
+        sidecar_path.write_text(text, encoding="utf-8")
+        try:
+            load_eval_estimates(sidecar_path)
+        except Exception:
+            sidecar_path.write_text(original, encoding="utf-8")
+
     @staticmethod
     def _merge_work_dir_to_output(work_dir: Path, output_dir: Path) -> None:
         """File-level merge of ``work_dir`` into ``output_dir``.