evalrx 0.1.2__py3-none-any.whl

evalrx/eval_agent/prompts/diagnosis.py

@@ -0,0 +1,125 @@
+"""Prompt templates for M3 diagnosis."""
+
+_DIAGNOSE_PROMPT = """\
+You are an expert ML diagnostician. Based on the analysis report below, propose
+specific, falsifiable hypotheses about the root cause of the model's failures.
+
+Each hypothesis is written twice: once for a specialist (HYPOTHESIS, TEST) and
+once for the person who will read the report. That reader builds and runs
+evaluations but does not do statistics — they know what a benchmark, a prompt
+and a failure case are, and they have never met an e-value, a McNemar test or a
+multiple-comparisons correction. The PLAIN_STATEMENT line is written for them.
+{prior_section}
+Model: {model_name}
+Overall severity (threshold rules): {severity}
+
+Experiment protocol (authoritative task and response contract):
+{protocol_section}
+
+Analysis conclusion (the analyst's interpretation):
+{conclusion}
+{evidence_section}{stats_section}{explore_section}{failure_modes_section}
+Raw findings (JSON):
+{findings_json}
+
+{available_signals_section}Propose 1-3 hypotheses. For each write:
+HYPOTHESIS: <one-sentence falsifiable technical claim about the failure mode>
+PLAIN_STATEMENT: <the SAME claim as HYPOTHESIS, restated in ONE everyday
+  sentence, written for the reader described above. State what the model is
+  doing wrong, not what statistic was computed. Numbers are welcome and make
+  it better — but a number must come with what it amounts to ("gets 45 of the
+  125 wrong, nearly all of them lists longer than eight words"), never bare
+  ("fail rate 0.36"). No acronyms, no statistics jargon, no symbols (→, ρ, σ),
+  no bare metric identifiers as the subject of the sentence. Do not copy the
+  HYPOTHESIS line verbatim — this is checked, and a jargon-y or copy-pasted
+  line is sent back for a rewrite.>
+FAILURE_MODE: <short snake_case tag naming the MECHANISM, not the symptom.
+  Vision/agent: attention_sink / hallucination / loop / ignored_obs / language_prior_bias
+  Text reasoning: computation_slip / chain_break / knowledge_gap / selection_failure /
+    overthinking / brittleness / memorization / self_correction_failure
+  Harness (suspect these before any mechanism): answer_extraction / truncation /
+    degenerate_repetition
+  Use a tag from these lists when one fits; invent one only when none does.>
+TEST: <which evidence verifies this claim — name a signal/analyzer from the
+available evidence list when one fits (e.g. "relative_attention.max_relative_weight
+HIGHER on failing cases" or "prompt_contrast describe_first contrast"), and for a
+per-case signal SAY whether it should be HIGHER or LOWER on failing cases — that
+is the prediction the test checks. On a yes/no task a directional claim ("answers
+Yes regardless of the evidence") is tested on the DIRECTION marginals
+answer_extraction_audit.answered_yes (the answer's direction, e.g. HIGHER on
+failing cases) and answer_extraction_audit.gold_yes (the question's), never on
+extracted_answer, labelled_fail or a correctness flag — those are text or the
+label itself and cannot be evidence; otherwise describe the analyzer or
+intervention that should be run next cycle>
+EXPECTED_ASSOCIATION: <higher_on_failures if larger/present values of the named
+signal support the hypothesis, or lower_on_failures if smaller/absent values
+support it. Pre-register this direction from the claim; do not infer it from an
+observed effect. For an intervention expected to repair failures, use
+higher_on_failures for a fixed_by_* signal.>
+
+Base your hypotheses on the analysis conclusion and evidence above — an analyzer
+can surface a real failure mode even when no numeric threshold fired, so do NOT
+rely on the threshold severity alone.
+Do NOT repeat hypotheses already listed in the prior cycles above.
+If the conclusion and evidence genuinely show no problem, respond with: NO_ISSUE"""
+
+_VALIDATE_PROMPT = """\
+You are an adversarial ML reviewer. Your job is to find reasons to REJECT each
+hypothesis below. Only approve a hypothesis if you cannot find a significant flaw.
+
+Check each for:
+1. Unsupported claim — does the cited evidence actually imply this failure mode?
+2. Circular reasoning — does the hypothesis merely restate the symptom?
+3. Overgeneralisation — does it make a claim far broader than the evidence supports?
+4. Confounded alternative — is there a simpler explanation the hypothesis ignores?
+{context_section}
+Findings summary (the evidence the hypotheses were drawn from):
+{findings_json}
+
+Hypotheses to review:
+{hypotheses_text}
+
+For each hypothesis output two lines:
+KEEP: <hypothesis statement>  or  REJECT: <hypothesis statement>
+REASON: <one or two sentences naming the flaw you found, or why it survives>
+REASON: <specific flaw, or "evidence directly supports this claim" if keeping>"""
+
+
+_FORMAT_REPAIR_PROMPT = """\
+Your previous answer below could not be read: it contains no line that starts
+with the label HYPOTHESIS:. The hypotheses are only usable in the exact
+line-labelled format.
+
+Previous answer:
+{raw}
+
+Rewrite the SAME hypotheses (do not invent new ones) so that each one is a block
+of these lines, in this order, each label at the start of its own line:
+HYPOTHESIS: <one-sentence falsifiable technical claim>
+PLAIN_STATEMENT: <the same claim in one everyday sentence, no jargon>
+FAILURE_MODE: <short snake_case tag naming the mechanism>
+TEST: <which signal/analyzer verifies the claim>
+EXPECTED_ASSOCIATION: <higher_on_failures or lower_on_failures>
+
+If your previous answer meant that there is no issue to report, reply with the
+single word NO_ISSUE."""
+
+_PLAIN_REPAIR_PROMPT = """\
+Your previous answer below proposed hypotheses correctly, but some
+PLAIN_STATEMENT lines fail a plain-language check.
+
+PLAIN_STATEMENT must be a one-sentence everyday paraphrase of the matching
+HYPOTHESIS line, written for an engineer who runs evaluations but does no
+statistics: no statistics terms, no acronyms, no symbols, and not a verbatim
+copy of the HYPOTHESIS line. Numbers are welcome, but each one must come with
+what it amounts to rather than standing bare.
+
+Previous answer:
+{raw}
+
+Problems found:
+{violations}
+
+Rewrite the FULL set of hypotheses in the exact same
+HYPOTHESIS/PLAIN_STATEMENT/FAILURE_MODE/TEST/EXPECTED_ASSOCIATION format,
+fixing only the flagged PLAIN_STATEMENT lines. Do not change any other line."""

evalrx/eval_agent/prompts/experiment_writer.py

@@ -0,0 +1,265 @@
+"""Prompt templates for the M5 experiment writer."""
+
+from typing import Any
+
+_BLUEPRINT_SYSTEM = """\
+You are an expert ML debugging engineer. Plan the structure of a multi-file
+Python diagnostic project that will test a specific hypothesis about a model's
+behaviour.
+
+Output a YAML specification with this structure:
+```yaml
+description: one-line summary
+files:
+  - name: main.py
+    generation_order: 3
+    description: entry point
+    pseudocode: |
+      1. Load model
+      2. Run cases
+      3. Compute metrics
+      4. Print metrics in name: value format
+      5. Print verdict: 1.0 or verdict: 0.0
+    dependencies: [model_probe.py, utils.py]
+  - name: utils.py
+    generation_order: 1
+    description: shared helpers
+    pseudocode: |
+      - helper functions
+    dependencies: []
+output_contract:
+  metrics: [mean_consistency, n_cases]
+  verdict: "1.0 = hypothesis SUPPORTED, 0.0 = REFUTED"
+```
+
+Rules:
+- Generate 2–4 files maximum
+- main.py must be last in generation_order
+- main.py MUST have `if __name__ == "__main__":` block
+- The last printed line of the entire run MUST be `verdict: 1.0` or `verdict: 0.0`
+- Stay under {timeout_sec} seconds total runtime\
+"""
+
+_BLUEPRINT_USER = """\
+## Hypothesis to test
+Statement   : {statement}
+Failure mode: {failure_mode}
+
+## Model setup
+```python
+{import_expr}
+model = {load_expr}
+```
+
+Available capabilities: {capabilities}
+
+## Model access policy
+{model_access_note}
+
+## Failure cases (JSON)
+```json
+{cases_json_snippet}
+```
+
+Write the YAML blueprint now:\
+"""
+
+_GENERATE_FILE_SYSTEM = """\
+You are an expert ML debugging engineer. Generate a single Python file
+as part of a multi-file diagnostic experiment project.
+
+Rules:
+- Implement EXACTLY the pseudocode in the file spec
+- Use only the evalrx APIs shown
+- Output ONLY the Python code — no markdown prose before or after
+- Code must be complete and immediately executable\
+"""
+
+_GENERATE_FILE_USER = """\
+## File to generate: {file_name}
+
+### File spec
+```json
+{file_spec}
+```
+
+### Blueprint (full project)
+```yaml
+{blueprint}
+```
+
+### Dependencies already generated
+{dep_summaries}
+
+{dep_code}
+
+## Model access policy
+{model_access_note}
+
+## EvalRX API reference
+```python
+from evalrx.core.case import CaseBatch, FailureCase, Inputs
+output: str = model.generate(case.inputs)
+# logprobs (only if 'LOGPROBS' in capabilities)
+lps = model.logprobs(case.inputs)
+# internals (only if 'ATTENTION' or 'HIDDEN_STATES' in capabilities)
+trace = model.forward(case.inputs, capture={{Capability.ATTENTION}})
+```
+
+Write the complete `{file_name}` now (Python code only):\
+"""
+
+_WRITE_SYSTEM = """\
+You are an expert ML debugging engineer. Write a concise, self-contained Python
+diagnostic script that gathers evidence for or against a specific hypothesis
+about a model's behaviour.
+
+RULES:
+- Use ONLY the evalrx APIs shown in the user message.
+- The script must be completely self-contained — no external files.
+- Print ALL metrics in EXACTLY this format (one per line):
+    metric_name: float_value
+- Print a final verdict line as the LAST output:
+    verdict: 1.0    # hypothesis is SUPPORTED by the evidence
+    verdict: 0.0    # hypothesis is REFUTED
+- Never print anything after the verdict line.
+- Catch exceptions and print error metrics rather than crashing silently.
+- Stay under {timeout_sec} seconds total runtime.
+- The script MUST have `if __name__ == "__main__":` block.\
+"""
+
+_WRITE_USER = """\
+## Hypothesis to test
+Statement  : {statement}
+Failure mode: {failure_mode}
+
+## Model setup (copy this verbatim)
+```python
+{import_expr}
+model = {load_expr}
+```
+
+Available capabilities: {capabilities}
+
+## Model access policy
+{model_access_note}
+
+{blueprint_context}
+
+## EvalRX API reference
+```python
+from evalrx.core.case import CaseBatch, FailureCase, Inputs
+from evalrx.core.capability import Capability
+
+# Text generation (always available)
+output: str = model.generate(case.inputs)
+
+# Token logprobs  (only if 'LOGPROBS' in capabilities)
+lps = model.logprobs(case.inputs)   # list[TokenLogprob]; each has .token, .logprob
+
+# Internals capture (only if 'ATTENTION' or 'HIDDEN_STATES' in capabilities)
+trace = model.forward(case.inputs, capture={{Capability.ATTENTION}})
+# trace.attentions     list[Tensor]  one per layer, shape [heads, seq, seq]
+# trace.hidden_states  list[Tensor]  one per layer, shape [seq, hidden]
+# trace.tokens         list[str]
+```
+
+## Failure cases (JSON — deserialize inside your script)
+```json
+{cases_json}
+```
+
+## Required output (EXACTLY this format)
+```
+metric_a: 0.72
+metric_b: 3.0
+verdict: 1.0
+```
+
+Write the complete Python script now:\
+"""
+
+_REPAIR_SYSTEM = """\
+You are a Python debugging expert. The script below crashed.
+Return the COMPLETE corrected script — no explanations, just the fixed code.
+Preserve all logic; fix only the error.\
+"""
+
+_REPAIR_USER = """\
+## Error
+```
+{error}
+```
+
+## Script to fix
+```python
+{code}
+```\
+"""
+
+_REVIEW_SYSTEM = """\
+You are a senior code reviewer. Review this diagnostic experiment code and
+return a JSON object with:
+  - "verdict": "APPROVE" or "REQUEST_CHANGES"
+  - "score": integer 1-10
+  - "critical_issues": list of strings (empty if APPROVE)
+
+Focus on: correctness of hypothesis testing logic, metric computation,
+verdict output format, and error handling.  Do NOT flag style issues.\
+"""
+
+
+def build_cli_prompt(
+    hypothesis: Any,
+    model_context: dict[str, Any],
+    timeout_sec: int,
+    n_images: int = 0,
+) -> str:
+    """Build the prompt used by external CLI coding agents."""
+    caps = ", ".join(model_context.get("capabilities", [])) or "GENERATE"
+    access_note = model_context.get("access_note", "")
+    image_note = (
+        f", and image_path (JPEG path for {n_images} case(s)).\n"
+        "Load images and build inputs like this:\n"
+        "```python\n"
+        "from PIL import Image\n"
+        "from evalrx.core.case import Inputs\n"
+        "img = Image.open(case['image_path'])\n"
+        "inputs = Inputs(prompt=case['prompt'], image=img)\n"
+        "# Then: model.generate(inputs) or model.forward(inputs, capture={Capability.ATTENTION})\n"
+        "```\n"
+        "Do NOT pass raw dicts or lists as inputs — always use Inputs().\n\n"
+        if n_images else ".\n\n"
+    )
+    return (
+        "You are an ML debugging engineer. Write a self-contained Python "
+        "diagnostic script that tests a specific hypothesis about a model.\n\n"
+        "## Hypothesis\n"
+        f"Statement   : {hypothesis.statement}\n"
+        f"Failure mode: {hypothesis.predicted_failure_mode}\n\n"
+        "## Model setup (copy verbatim)\n"
+        "```python\n"
+        f"{model_context.get('import_expr', 'import evalrx')}\n"
+        f"model = {model_context.get('load_expr', '# model')}\n"
+        "```\n\n"
+        f"Available capabilities: {caps}\n\n"
+        "## Model access policy\n"
+        f"{access_note}\n\n"
+        "## Input data\n"
+        "Read `cases.json` from the current directory. "
+        "Each record has: prompt, label (PASS/FAIL), id, metadata"
+        + image_note
+        + "## Required output (print to stdout, EXACTLY this format)\n"
+        "```\n"
+        "metric_a: 0.72\n"
+        "metric_b: 3.0\n"
+        "verdict: 1.0    # 1.0=SUPPORTED  0.0=REFUTED\n"
+        "```\n\n"
+        "## Rules\n"
+        f"- Stay under {timeout_sec} seconds total runtime.\n"
+        "- Save the script as `experiment.py` in the current directory.\n"
+        "- Print all metrics as `name: float_value` lines.\n"
+        "- The last printed line must be `verdict: 1.0` or `verdict: 0.0`.\n"
+        "- Use only the evalrx APIs shown above.\n"
+        "- Do NOT make external network calls or read files other than `cases.json`.\n"
+    )

evalrx/eval_agent/prompts/explore_step.py

@@ -0,0 +1,37 @@
+"""Question template for the loop's optional in-cycle explore step.
+
+The explorer (:class:`~evalrx.analysis.explorer.ExploratoryAnalysisAgent`)
+is a free-form EDA coder: it only knows what the question tells it. This
+template turns the run's :class:`~evalrx.eval_agent.stages.protocol.ExperimentProtocol`
+into a question that (a) names the outcome column, (b) says what the columns
+are (M1 analyzer per-case signals), and (c) asks for FROZEN, threshold-explicit
+recipes — the only form a candidate signal can later be re-evaluated in.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+_EXPLORE_QUESTION = (
+    "Which per-case signals distinguish the FAIL cases (label=fail) from the "
+    "PASS cases (label=pass)?{context} Every other column is a per-case signal "
+    "an M1 analyzer computed on the model under diagnosis (column name = "
+    "analyzer.metric, sanitized). Compare each signal's distribution between "
+    "FAIL and PASS, look for interactions and thresholds, and make every "
+    "candidate signal a FROZEN, threshold-explicit recipe over the available "
+    "numeric columns (explicit numeric cut-offs, no re-fitting) so it can be "
+    "re-evaluated verbatim on held-out cases. Charts are leads for hypotheses, "
+    "not proof: this is the exploration half of a diagnosis loop, and a "
+    "separate confirmatory stage will test whatever you surface."
+)
+
+
+def default_explore_question(protocol: Any | None = None) -> str:
+    """Build the explore question from the run's protocol (or a generic one)."""
+    parts: list[str] = []
+    for attr in ("description", "task_domain", "failure_patterns"):
+        value = getattr(protocol, attr, None) if protocol is not None else None
+        if value:
+            parts.append(str(value).strip())
+    context = (" Context: " + " ".join(parts)) if parts else ""
+    return _EXPLORE_QUESTION.format(context=context)

evalrx/eval_agent/prompts/fix_agent.py

@@ -0,0 +1,257 @@
+"""Prompt templates for fix candidate generation and repair."""
+
+_L1_PROMPT = """\
+You are designing PROMPT-LEVEL fixes (tier L1: the input space only) for a \
+model failure. The model may be text-only or vision-language.
+
+VERIFIED FAILURE HYPOTHESES:
+{hypotheses}
+{context}
+EXAMPLE CASES — read the model's own outputs: what it does when it fails, what
+it does when it succeeds, and the answer format the scorer expects:
+{examples}
+
+Propose up to {k} prompt rewrite strategies that could repair these failures
+WITHOUT changing the model or adding pipeline steps.  Each strategy is a
+template applied to every case prompt; it MUST contain the literal placeholder
+{{prompt}}.  Keep the final-answer format the scorer expects recoverable; do
+not ask the model to suppress its reasoning if the task needs it.  If your
+template asks the model to write intermediate work (a transcription, a list, a
+check) that may not fit the baseline decode budget, add "generation_kwargs":
+{{"max_tokens": <int>}} to that strategy — it can only RAISE the budget (a
+lower value is raised to the baseline), and no other decoding control is
+allowed at this tier.
+
+
+"what_it_does" is shown to a reader who has never heard of this model, this
+benchmark, or this field -- write it for a bright high-school student.  One
+sentence, present tense, plain words, saying what CHANGES for the model.  Do
+not name the tier, do not use metric or method jargon, and do not restate the
+snake_case name in English.
+Good:  "Asks the model to describe what it hears before it answers."
+Bad:   "L1 audio-evidence-first prompt scaffold with deferred answering."
+
+Reply with ONLY a JSON array (generation_kwargs is optional; omit it unless
+the rewrite needs the extra room):
+[{{"name": "<short_snake_case>", "what_it_does": "<one plain sentence>",
+   "prompt_template": "<template with {{prompt}}>",
+   "generation_kwargs": {{"max_tokens": <int>}}}}]"""
+
+_L2_PROMPT = """\
+You are designing SCAFFOLD-LEVEL fixes (tier L2: a pipeline around the \
+unchanged model) for a model failure. The model may be text-only or
+vision-language.
+
+VERIFIED FAILURE HYPOTHESES:
+{hypotheses}
+{context}
+EXAMPLE CASES — read the model's own outputs: what it does when it fails, what
+it does when it succeeds, and the answer format the scorer expects:
+{examples}
+
+IMAGE TOOLS (only meaningful when the case has an image):
+{catalog}
+
+Propose up to {k} pipelines. Each may chain image tools, rewrite the prompt
+(template MUST contain {{prompt}}), sample the model n_samples times (1..5;
+applies to EVERY strategy — a multi-call strategy is repeated end-to-end and
+its final answers vote on the extracted final answer), select one reviewed
+multi-call strategy (direct, least_to_most, self_refine,
+chain_of_verification), and set bounded decoding controls in
+generation_kwargs (temperature, top_p, stop, max_tokens). max_tokens can only
+RAISE the baseline budget — a value below it is raised to the baseline, so
+never try to shorten the model's chain of thought; temperature 0 makes
+n_samples>1 pointless. For structured answer tasks, output_key_pattern may be
+a regex with one capture group used for answer-only voting; it must not
+contain gold answers.
+
+"what_it_does" is shown to a reader who has never heard of this model, this
+benchmark, or this field -- write it for a bright high-school student.  One
+sentence, present tense, plain words, saying what CHANGES for the model.  Do
+not name the tier, do not use metric or method jargon, and do not restate the
+snake_case name in English.
+Good:  "Asks the model to describe what it hears before it answers."
+Bad:   "L1 audio-evidence-first prompt scaffold with deferred answering."
+
+Reply with ONLY a JSON array:
+[{{"name": "<short_snake_case>", "what_it_does": "<one plain sentence>",
+   "image_ops": [{{"tool": "<catalog name>", "params": {{}}}}],
+   "prompt_template": "{{prompt}}", "n_samples": 1,
+   "generation_kwargs": {{}}, "strategy": "direct",
+   "output_key_pattern": ""}}]"""
+
+_L2_CODE_PROMPT = """\
+You are writing a PYTHON PIPELINE (tier L2: a scaffold around the unchanged \
+model) that repairs the failures described below.  Design any
+pipeline you want — the only constraint is that the model itself is unchanged.
+
+VERIFIED FAILURE HYPOTHESES:
+{hypotheses}
+{context}
+EXAMPLE CASES — read the model's own outputs: what it does when it fails, what
+it does when it succeeds, and the answer format the scorer expects:
+{examples}
+
+EXECUTION CONTRACT:
+- "{cases_file}" in the current directory: {{"cases": [{{"id": str, "prompt": str,
+  "baseline_output": str|null}}]}} — baseline_output is the model's ORIGINAL
+  recorded answer to that prompt: the DIRECT BASELINE.  It may be wrong; it is
+  NOT a label and carries no correctness information.  Use it directly as the
+  baseline answer (compare against it, vote with it, ask the model to
+  double-check it) — you do NOT need to call model_generate(case_id) to
+  obtain it; a plain model_generate(case_id) is answered from this record.
+- A function  model_generate(case_id, prompt=None, image_ops=None,
+  generation_kwargs=None) -> str  is ALREADY DEFINED in your namespace (do NOT
+  import or redefine it).  It runs the ORIGINAL model on that case: optional
+  prompt override, optional image transforms applied to the case's image
+  first, optional bounded decoding controls (temperature, top_p, stop,
+  max_tokens — max_tokens can only RAISE the baseline budget; a lower value is
+  raised to it).  IMPORTANT: ``prompt`` REPLACES the original prompt; it is
+  not prepended automatically.  Every answer-producing override must include
+  ``case["prompt"]`` itself (for example, ``instruction + "\\n\\n" +
+  case["prompt"]``), or the model will never see the question.  image_ops MUST
+  be a list of
+  {{"tool": "<name>", "params": {{...}}}} dicts using ONLY these tools
+  (anything else is rejected with an error):
+{catalog}{attend_hint}
+- You may call the model SEVERAL times per case, but keep it to at most
+  4 calls per case that hit the model (genuinely independent enhanced/reasoned
+  passes: a different prompt and/or image_ops each); the host enforces this
+  per-case cap.  A plain model_generate(case_id) (no prompt override, no
+  image_ops, no generation_kwargs) is answered from the recorded
+  baseline_output, costs nothing and does not count.  Branch on outputs
+  within that budget — e.g. describe first, then decide; vote over
+  independent variants.
+- model_generate is thread-safe and concurrent calls are serviced in
+  parallel: fan out over CASES with concurrent.futures.ThreadPoolExecutor
+  (max_workers=8) — a serial loop over every case x several calls is slow and
+  risks the wall-clock limit.
+- SELECTION RULE (enforced by the host, without labels): for each case the
+  host anchors on baseline_output (or on your plain direct call when a case
+  has no record).  Your final "output" may differ from that anchor only when
+  at least {min_support} of your enhanced calls with DISTINCT prompt/image_ops
+  returned the same answer; otherwise the host reverts that case to the
+  anchor.  Return the model's own reply text (or the tagged answer you asked
+  it to emit, e.g. "FINAL: <answer>" — the host strips such tags when
+  matching); never rewrite, reformat or compute the answer yourself.
+  Pattern the host accepts:
+      base = case["baseline_output"] or ""
+      question = case["prompt"]
+      votes = [model_generate(cid, prompt=p1 + "\\n\\n" + question),
+               model_generate(cid, prompt=p2 + "\\n\\n" + question, image_ops=ops)]
+      final = the answer that >= {min_support} votes agree on if it differs from base, else base
+{selection_guidance}
+- The LAST line of stdout MUST be exactly:
+  {marker}{{"per_case": [{{"sample_id": "<case id>", "output": "<final answer text>"}}]}}
+- Emit an entry for EVERY case.  The "output" is scored externally against the
+  original question, so it must answer that question faithfully (e.g. contain
+  a clear yes/no for yes/no questions).
+- Standard library + numpy only.  No network, no file writes.  Keep it under
+  ~80 lines.
+- REPAIR THE MODEL, NOT THE TASK.  Your code is re-run a second time with every
+  model_generate() call answered by the model's ORIGINAL recorded answer.  Any
+  failing case your pipeline gets right in THAT run was solved by your code,
+  not the model, and is excluded from the fix score — a pipeline that computes
+  the answer itself (sorting, arithmetic, lookup, a hard-coded default) scores
+  zero.  Verify, vote, re-ask, re-prompt, zoom: make the model get it right.
+
+The FIRST line of your code must be a WHAT_IT_DOES comment, on ONE line, for a
+reader who has never heard of this model, this benchmark, or this field: plain
+words, present tense, saying what your pipeline makes the model do differently.
+No jargon, no tier names, no restating the code.  For example:
+  # WHAT_IT_DOES: Asks the model twice with different wording and keeps the answer both tries agree on.
+
+Return ONLY the Python code{fences_hint}."""
+
+_REPAIR_PROMPT_BODY = """\
+Your previously written repair pipeline FAILED TO EXECUTE.
+
+ERROR:
+{error}
+
+YOUR PREVIOUS CODE:
+```python
+{code}
+```
+
+Fix the code.  Follow the execution contract EXACTLY:
+- the ONLY model access is the predefined model_generate(case_id, prompt=None, \
+image_ops=None, generation_kwargs=None){attend_clause} — do not import or redefine it;
+- image_ops must be a list of {{"tool": "<name>", "params": {{...}}}} dicts \
+using ONLY these tools:
+{catalog}
+- baseline_output in "{cases_file}" IS the direct baseline answer — use it as \
+the baseline; a plain model_generate(case_id) is answered from that record and \
+is free; the TOTAL number of enhanced model_generate(...) PLUS model_attend(...) \
+calls must be at most 4 per case (model_attend is not free);
+- the host's selection guard reverts a case to its baseline unless at least \
+{min_support} DISTINCT enhanced calls returned your final answer (answer tags \
+such as "FINAL: <answer>" are stripped when matching);
+{selection_guidance}
+- read "{cases_file}", emit an entry for EVERY case, and end stdout with \
+exactly:
+  {marker}{{"per_case": [{{"sample_id": "<case id>", "output": "<final answer text>"}}]}}
+- standard library + numpy only; no network, no file writes; under ~80 lines.
+- overwrite pipeline.py with ONLY the corrected user pipeline; do not copy or \
+modify fix_pipeline_exec.py and do not append the old program after the repair.
+- repair the MODEL, not the task: the code is re-run with every model_generate() \
+call answered by the model's original recorded answer, and failing cases it still \
+gets right then are excluded from the score.  If it timed out, make FEWER model \
+calls per case — do not replace them with code that computes the answer itself.
+- keep (or add) the one-line `# WHAT_IT_DOES: ...` first line describing, in \
+plain words for a reader with no background, what your pipeline makes the model \
+do differently.
+"""
+
+_L3_PROMPT = """\
+You are configuring WHITE-BOX intervention primitives (tier L3: the model's \
+internals) against the failures below.  The primitives are pre-audited host \
+code — you choose which to run and with what parameters.
+
+VERIFIED FAILURE HYPOTHESES:
+{hypotheses}
+
+AVAILABLE PRIMITIVES:
+{catalog}
+
+Propose up to {k} configurations.  Reply with ONLY a JSON array:
+[{{"primitive": "<name from the list>", "params": {{...}}}}]"""
+
+_REPAIR_CATALOG_PROMPT = """\
+You are selecting which REGISTERED repair capability, if any, applies to the \
+failure(s) below.  Each candidate is a specific, pre-audited \
+intervention that targets ONE named failure mechanism — read what mechanism \
+each one actually targets, then select it ONLY when the verified hypotheses \
+describe that same mechanism, not merely because it is technically able to \
+run on this model and task (that eligibility has already been checked for \
+you; your only job is judging whether the mechanism matches).
+
+VERIFIED FAILURE HYPOTHESES:
+{hypotheses}
+
+ELIGIBLE CANDIDATES (already filtered to what this model/task can run):
+{catalog}
+
+Propose up to {k} candidates whose targeted mechanism is genuinely \
+consistent with the hypotheses above.  A hypothesis naming a DIFFERENT \
+mechanism (e.g. a flat knowledge gap, a positional/answer-choice bias, or a \
+hallucination cause when the candidate targets something else entirely) is \
+NOT a match, even if the candidate would run without error.  If none of the \
+candidates address the diagnosed mechanism, reply with an empty array — do \
+not select one just because it is available.  Reply with ONLY a JSON array:
+[{{"name": "<name from the list>", "rationale": "<one sentence: how the hypothesis's mechanism matches this candidate's>"}}]"""
+
+_L4_PROMPT = """\
+You are writing a PARAMETER-SPACE repair recipe (tier L4: fine-tuning) for \
+the failures below.  The recipe is RECORDED for a human decision — it will \
+not be executed automatically.
+
+VERIFIED FAILURE HYPOTHESES:
+{hypotheses}
+
+Reply with ONLY a JSON object:
+{{"dataset_recipe": "<how to build training data that generalises the failure
+   mechanism — never just the observed failing cases>",
+  "method": "lora|sft", "target": "vision_encoder|llm|projector|full",
+  "eval_protocol": "<held-out repair effect + regression battery>",
+  "rationale": "<why parameter-space change is the minimum effective tier>"}}"""