evalkit-py 0.2.0.dev12__py3-none-any.whl

evalkit/bundled/skills/eval-driven-development/references/diagnose.md

@@ -0,0 +1,130 @@
+# Diagnose one eval failure
+
+Instructions for diagnosing **one** eval failure (one failing case) from its
+pulled run directories and returning one structured JSON finding (root cause +
+surface + evidence). On agents with subagent support, dispatch these
+instructions as a read-only subagent, one per failing case; otherwise follow
+them inline, one failure at a time. Input: the absolute paths of the case's
+`runs/<run_id>/` directories produced by `evalkit runs pull --experiment`
+(one per repetition), plus the repo checkout(s) of the agent under test.
+
+## What you may touch
+
+- **Read access to the entire repo.** Use read-only inspection (`cat`,
+  `grep`, `jq`, `ls`, `git log`/`git show` — never anything that writes).
+- **Hard rule: never modify the code repo.** No writes, no edits. You may
+  write throwaway scratch scripts under a temp dir to reproduce or probe a
+  root cause — but never create, edit, move, or delete anything inside the
+  repository. Your finding IS your final output; you have no repo-write step.
+
+## How to diagnose
+
+1. Read every file in each run directory. Weigh every section:
+   - `case.json` (`input`, `expected`, `evaluators`) and `evaluations.json` —
+     what went wrong on the surface and which evaluator(s) scored low and why (read
+     each `explanation`; judge-backed evaluators pack per-item verdicts into
+     it as JSON).
+   - each run's `trajectory.json` — its ATIF trajectory. Steps run in
+     `step_id` order; each step's `tool_calls` carry structured `arguments`,
+     their results are in `observation.results` (matched by `source_call_id`;
+     `extra.is_error: true` marks a failed call, `extra.error_type` its kind
+     when the harness sets it), and `timestamp` shows where time went.
+     Delegated work is in `subagent_trajectories`, one trajectory per
+     subagent. When a result was shortened by the harness, the full output is
+     at its `extra.full_output_path`; fetch it with `evalkit runs files <pull
+     dir> <run_id> --path <that path>` (see the run's files below).
+     **This is the strongest signal for *why* it failed.** Look for: retry
+     loops (same call re-issued after a non-fatal error), discovery overhead
+     (probing for a schema or identifier that should be derivable first-try),
+     wrong-then-right argument values, dead-end pivots. Treat these as
+     diagnosable in their own right: a tool the model can't call correctly the
+     first time is a real defect even when the run eventually recovers. Record
+     each shape in `trajectory_patterns` and localize the surface that would
+     make the correct call derivable first-try.
+   - `run.json` `status`, `answer`, `metrics` and `error` — timeouts, token blowups,
+     run crashes. `metrics.by_trajectory` splits tokens, cost and tool calls
+     per trajectory: `main`, then one entry per subagent keyed like the
+     evidence refs below, each with its `agent` name.
+   - the run's files — its deliverables and other workspace files;
+     read them (including images, which enter your context) to judge the
+     deliverable directly. `evalkit runs pull` does not fetch them. When you
+     need them, `evalkit runs files <pull dir> <run_id>` lists the tree, and
+     `--path <file or folder>` fetches only those into
+     `runs/<run_id>/files/<path>`.
+2. Trace the surface (skill / tool / prompt / data) responsible. Open the
+   actual files in the repo to confirm — cite concrete paths.
+3. The run is history: it shows the agent as of `run.json` `started_at`, and
+   the repo may have moved on. Run `git log --since=<started_at> -- <surface
+   file>` (and `git log -S '<text from the trajectory>'` for prompt or tool
+   text) and read any commit that touches the surface. Commit dates reset on
+   rebase and squash merge, so also read the last few commits on the surface
+   file (`git log -n 10 -- <file>`) whatever their date. If a later commit
+   already fixes the failure, fill `already_addressed` with its sha and one
+   line on what it changed, and keep `suggested_direction` to "re-run to
+   confirm".
+4. If the capability the case needs does not exist at all, that is
+   `root_cause: absent` / `surface.kind: absent` — the build-new-capability
+   case.
+5. When the surface is a skill or another authored prompt, judge it against
+   the authoring standards too — a bloated, coupled, or weakly-described skill
+   is itself a candidate root cause (`skill-gap` / `prompt-ambiguity`).
+
+## Output — return ONLY this JSON object
+
+No prose before or after. A single JSON object matching this schema exactly:
+
+```json
+{
+  "case_id": "<from case.json>",
+  "verdict": "failed | borderline",
+  "root_cause": "skill-gap | tool-wrong-shape | tool-wrong-data | tool-call-inefficiency | prompt-ambiguity | hallucination | data-missing | absent | unknown",
+  "surface": {"kind": "skill | tool | prompt | mcp | data | absent", "name": "<surface name>", "file": "<repo path or null>"},
+  "evidence": [
+    {"type": "trajectory-span | evaluator | artifact | prose", "ref": "<main:<step_id> or <trajectory_id>:<step_id> / evaluator name / artifact path>", "detail": "<one specific line>"}
+  ],
+  "trajectory_patterns": ["<short label of a recurring shape, e.g. 'lookup -> latest_lookup retry, +1 turn'>"],
+  "confidence": 0.0,
+  "already_addressed": null,
+  "coverage_gap": null,
+  "suggested_direction": "<one or two sentences: where the fix likely belongs and why — direction, not a diff>"
+}
+```
+
+Rules for the finding:
+
+- Every `evidence` entry must cite something concrete (a step as
+  `main:<step_id>` or `<trajectory_id>:<step_id>` — `subagent_trajectories[<n>]:<step_id>`
+  when the subagent has no `trajectory_id` — an evaluator name, an artifact
+  path, a quoted line). No vague evidence.
+- Use `root_cause: tool-call-inefficiency` when the defect is *how* the agent
+  reached a right-or-close answer — retried/errored/redundant calls or
+  discovery overhead — rather than a wrong final call shape
+  (`tool-wrong-shape`) or wrong data (`tool-wrong-data`). Point `surface` at
+  the tool description or skill that would remove the wasted turns, and
+  populate `trajectory_patterns`.
+- `surface.file` must be a path that exists in the repo, or `null` if you
+  genuinely cannot localize it (then set `root_cause: unknown` or `absent`).
+- If a run's `run.json` has a non-null `error` (the agent run's own error), set
+  `root_cause: unknown`, `verdict: failed`, and put the error in `evidence`.
+- If a run's `run.json` has a non-null `pull_error`, evalkit could not fetch
+  it in full and there is no `trajectory.json`; diagnose from the other runs
+  of the case, and if none is complete, set `root_cause: unknown` and cite the
+  `pull_error`.
+- `confidence` is your own 0–1 estimate.
+- `already_addressed` is `null`, or `{"commit": "<sha>", "detail": "<what it
+  changed>"}` when a commit made after the run already changes the failing
+  surface in a way that fixes it.
+- `coverage_gap` is `null`, or `{"kind": "uncovered_scenario | wrong_case",
+  "note": "<one line>"}`. Use `uncovered_scenario` when the failure shows a
+  nearby scenario no case in the suite asserts (search the suite's case YAML
+  before saying so). Use `wrong_case` when the expected answer is not
+  derivable from the data or the domain, so the case itself needs fixing. A
+  coverage gap never replaces the diagnosis; fill the rest of the finding as
+  usual.
+
+## EDD-infra note (optional, after the JSON)
+
+If — and only if — you found a gap in the **eval setup itself** (data you
+needed that the run directory did not contain, a missing evaluator signal, etc.), add
+one short line after the JSON beginning `EDD-infra note:`. A nudge, not a
+report. Otherwise omit it entirely.

evalkit/bundled/skills/eval-driven-development/references/synthesize-cases.md

@@ -0,0 +1,87 @@
+# Synthesize a case plan
+
+Instructions for turning a batch of app-run observations into one ranked case
+plan. Run once per batch — as a subagent where supported, else inline.
+Inputs: the `observations/` directory (one JSON per run, from
+`analyze-run.md`), the batch's `index.json`, the target suite directory (the
+case YAML the new cases will join), and the path to write
+(`<dir>/case-plan.md`).
+
+## What you may touch
+
+- **Read access to the entire repo and the suite directory.**
+- **Writing is allowed for exactly one file: the case plan path you were
+  given.** Don't write case YAML; M5 does that with the developer.
+
+## Your job
+
+1. Read every observation and `index.json`.
+2. Cluster the `failed` and `degraded` observations into patterns. Two
+   observations share a pattern when they have the same `task_type`, the same
+   failure `kind`, and the evidence points at the same root cause (the same
+   surface misbehaving the same way). Name each pattern in plain words.
+3. Score each pattern:
+
+   ```
+   frequency = runs in the pattern / all observations
+   impact    = highest weight among its failures:
+               wrong_answer 4, refusal 3, missing_step 3, wrong_tool 2,
+               inefficiency 1, presentation 1
+   score     = frequency × impact
+   fixed     = every failing run in the pattern has fixed_since set
+   ```
+
+4. Check coverage: read the suite's case YAML and find cases that already
+   assert the pattern's ideal behaviour. Drop a pattern that is fully covered;
+   keep a partly covered one and say which part no case asserts.
+5. Rank by score, highest first. Within the same score, unfixed patterns come
+   before fixed ones.
+6. Propose 2–3 varied cases per pattern from the observations' case seeds.
+7. Write the plan.
+
+## Case plan format
+
+```
+# Case plan — <agent>, <since> to <until>
+
+<N> runs analyzed: <f> failed, <d> degraded, <o> ok, <u> unclear. <P> patterns.
+
+## Pattern 1 — <plain name>   [<k> of <N> runs · impact <i>]
+Runs: <run_id>, <run_id>, …
+Surface: <kind> <file or "not located">
+Already covered by: none | <case id> (partly: <what is missing>)
+Fixed since the runs: no | yes, <sha> "<what changed>" (expected green)
+Proposed cases:
+  - id: <kebab-id>   type: <task_type>   ground truth: known | NEEDS DEVELOPER
+    input: <time-anchored input>
+    asserts: <the ideal behaviour, one line>
+Level: blackbox rubrics; overlay <check> if the check names harness tools
+
+## Healthy
+- <task_type>: no failures in <n> runs
+
+## Unclear or low confidence
+- <run_id>: <why>
+```
+
+## Rules for proposed cases
+
+- Generalise: each case states the ideal behaviour for a new, time-anchored
+  input. Never copy a user's run, words or data into a case; that overfits to
+  one run and puts user data into a suite.
+- Assert what correct output looks like for this input and every sibling like
+  it, not only that today's failure is absent.
+- `ground truth: NEEDS DEVELOPER` when the correct answer can't be checked
+  from the runs' tool results or the agent's data sources. Never guess.
+- Blackbox first. A check that names tools or skills belongs in an overlay
+  (see the authoring skill's `references/overlays.md`).
+- A fixed pattern still gets cases: they guard the fix. Mark it "expected
+  green" with the commit.
+- Unclear and low-confidence observations are listed, never clustered or
+  turned into cases.
+
+## EDD-infra note (optional, at the end of the plan)
+
+If the batch showed a gap in the eval setup itself (runs missing data you
+needed, a check evalkit should offer), add one line starting `EDD-infra
+note:`. Otherwise omit it.

evalkit/bundled/skills/eval-driven-development/references/synthesize.md

@@ -0,0 +1,165 @@
+# Synthesize a fix proposal
+
+Instructions for synthesizing one consolidated fix proposal from the per-case
+findings produced by the diagnose step. Run once per experiment — as a
+subagent where supported, else inline. Inputs: the `findings/` directory (one
+`<case_id>.json` per failure), the `index.json` path, and the path to write
+the proposal (`<review-dir>/proposals.md`).
+
+## What you may touch
+
+- **Read access to the entire repo** (read-only inspection).
+- **Writing is allowed for exactly one file: the proposal path you were
+  given.** Do not write or edit anything else.
+
+## Your job
+
+1. Read every finding in `findings/` and `index.json`.
+2. Group findings two ways:
+   - by `(surface, root_cause)` — the direct fixes;
+   - by cross-cutting `trajectory_patterns` that recur across cases even when
+     surfaces differ (these are the highest-leverage skill/prompt improvements
+     — treat them as first-class). Tool-call inefficiency belongs here: the
+     fix is usually a clearer, self-contained tool description — better arg
+     docs, an example call, sharper error messages — so the correct, minimal
+     call is derivable the first time. Make the tool stand on its own rather
+     than leaning on a companion skill, and never a per-case patch.
+3. For each group, open the cited files and pin the exact change: path +
+   section + concrete content or diff. Verify the path exists.
+4. Fix at the right surface. Propose **creating** a new skill / tool / handler
+   only for `surface.kind == absent` / `root_cause: absent` where no existing
+   surface can host the capability, or where the developer explicitly asked
+   for a new one; otherwise change the existing tool interface, skill, or
+   prompt. When you do propose a new surface, name it, place it, sketch its
+   contract, and call it out as a new-surface proposal so the human can
+   confirm.
+5. Write the proposal to the given path.
+
+## Interpreting the findings
+
+Findings are not uniform inputs — read their flags before grouping:
+
+- **`confidence`** weights a finding inside its group: a group built only on
+  low-confidence findings is itself weak — say so in the group's diagnosis.
+- **`verdict: borderline`** findings support a group but don't justify one on
+  their own; list a borderline-only pattern under the summary, not as a fix
+  group.
+- **`already_addressed`** findings name a commit made after the run that
+  already fixes the failure. Keep them out of fix groups; list them under
+  "Already fixed since the run — re-run to confirm" with the commit and case
+  id. Before writing any group, also check `git log` since the oldest run in
+  the group for changes to its surface, so the proposal never repeats a fix
+  that has already landed.
+- **`coverage_gap`** findings say a scenario has no case, or the case itself
+  is wrong. Collect them into the "Case proposals" section, one entry each,
+  apart from the fix groups; they are never applied as agent changes. A
+  finding can carry both a fix and a coverage gap; then it appears in both.
+- **`root_cause: unknown`** findings (including run-crash cases) don't belong
+  in any fix group. Collect them into an escalation list at the end of the
+  proposal with their evidence.
+- **`EDD-infra note:` lines** attached to findings are about the eval setup,
+  not the agent. Carry them through verbatim into your own EDD-infra section
+  at the end — never convert one into a proposed agent fix.
+
+## Constraints on proposed prompt text — read carefully
+
+The repo is full context for *your reasoning*, but the **content you propose
+to write into any application prompt surface must stay general-purpose**. This
+governs proposed text for skills, tool descriptions, system-prompt parts, and
+any agent prompt strings:
+
+- **No eval / golden-dataset awareness.** Never write "the eval expects…",
+  "the reference answer uses…", "to match the scorer…", or mention evals,
+  regression tests, or expected outputs. These surfaces must not know they are
+  evaluated. Justify every recipe on intrinsic data/workflow correctness.
+- **No calling-application concepts.** These surfaces do not know who invokes
+  them. Refer to intent abstractly: "the query", "the request".
+- **Keep surfaces decoupled.** A skill, a tool description, and a prompt part
+  must each stand on its own. Don't hard-code one surface into another —
+  cross-references couple the surfaces, so changing one silently breaks the
+  other. Describe a capability by its intent, not by naming the other surface
+  that provides it.
+- **Minimal, generic, declarative.** Prefer extending an existing section over
+  adding a new top-level one. No examples that smuggle in eval-only nuance.
+
+**Fix from first principles, not case by case.** A failure shows you the
+case's expected answer. Justify every proposed instruction by how the
+data/domain actually works — never by the fact that you've seen that answer.
+Test each one: *would it still be correct to someone who had never seen this
+eval's expected answer?*
+
+- **Yes** → it's a real, general improvement; propose it. (e.g. "the current
+  rating is in `latest_rating`; `rating` is historical" — true about the data,
+  regardless of any eval.)
+- **No, but a general rule does explain it** → you've written a case-specific
+  patch. Don't propose that — generalize it to the underlying principle that
+  produces the right answer for this case *and* every sibling.
+- **No, and no general rule explains it** → the expected answer isn't
+  derivable from the data/domain at all, which means the eval case itself is
+  likely the bug. Don't bake the specific answer into any prompt; surface it
+  to the developer as a possibly-wrong case (a dataset fix), and mark the
+  proposal `unverified`.
+
+## Prompt-change guidelines (apply to every proposed prompt edit)
+
+1. **Describe, don't legislate.** The first fix states facts about the
+   mechanism — what a tool does, what state the world is in afterwards — not
+   imperative rules. Strong wording (NEVER / ALWAYS / MUST) is an escalation,
+   not a default register.
+2. **Escalate only on observed persistence.** Reach for rule-style wording
+   only after a re-run shows the failure survived the descriptive fix; cite
+   the experiment that justified the escalation.
+3. **Prefer deterministic guards over prompt text.** If a tool can reject the
+   bad call, add the guard — prompt wording then only has to explain, not
+   enforce.
+4. **Error messages are prompts.** When rejecting a call, echo the inputs the
+   caller actually sent, name the offending value, and state the shape a
+   correct call has.
+
+## Output — `proposals.md`
+
+```markdown
+# Fix proposals — <experiment>
+
+## Summary
+- N failures total
+- By root cause: skill-gap (X), tool-wrong-shape (Y), absent (Z), ...
+- Recurring trajectory patterns: <one line each>
+
+## Cross-cutting trajectory fixes (highest leverage)
+
+### <pattern label>
+**Cases:** `id-1`, `id-2`, ...
+**Pattern:** <1–2 lines: the recurring shape + its cost in turns/latency>
+**Proposed change** — **File:** `<path>` — under `<section>`:
+> <concrete content / diff>
+
+## <root-cause> · <surface>
+
+**Cases:** `id-a`, `id-b`
+**Diagnosis:** <1 paragraph>
+**Proposed change** — **File:** `<path>` — under `<section>`:
+> <concrete content / diff>
+**Verified against repo:** yes | unverified (<why>)
+
+(repeat per group)
+
+## Case proposals
+
+- `<case_id>` · uncovered_scenario | wrong_case — <the note>; suggested case:
+  <one line, or "fix the case: <what is wrong>">
+
+## Already fixed since the run — re-run to confirm
+
+- `<case_id>` · `<commit>` — <what it changed>
+```
+
+Rules: be code-shaped, not advisory — "add this rule under section X" beats
+"improve the skill". Every change cites a real path. Mark anything you could
+not check against actual code as `unverified` with the reason.
+
+## EDD-infra note (optional, at the end)
+
+If you hit a gap in the **eval setup itself** while synthesizing, add one
+short `EDD-infra note:` line at the very end. A nudge, not a report. Otherwise
+omit.