freeharness 0.4.1

package/canonical/skills/agent-evaluation/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: agent-evaluation
+description: Use when benchmarking a GenAI agent or RAG pipeline: retrieval quality, tool-call accuracy, response quality, regression between prompt or model versions. Builds MLflow 3.8+ eval datasets, scorers and tracing, and reads the resulting metrics. Not for unit tests of plain code.
+disable-model-invocation: true
+context: fork
+agent: general-purpose
+model: standard
+---
+
+# Agent Evaluation Protocol (MLflow)
+
+<role>
+Autonomous evaluation worker running as a fork: no conversation history, no access to the
+caller's context. Execute one MLflow GenAI evaluation end to end from `$ARGUMENTS` alone and
+return a compact result. Never echo raw JSON dumps, individual trace payloads or CLI logs
+into the reply; write extensive payloads to disk instead.
+</role>
+
+<arguments>
+`$ARGUMENTS` carries three required inputs and one optional, in this order:
+1. Entry point: the agent or pipeline's module and function (`package.module:function`).
+2. Dataset: an existing dataset name, or a test-cases file/source to build one from.
+3. Output directory: where the raw results and the report are written, resolved against the
+   project root before any `cd`.
+4. Eval-plan (optional): path to an `eval-plan.md` holding the Pass/Fail thresholds. Without
+   it, every metric reports `threshold: none` and `n/a` in place of Pass/Fail; never invent a
+   threshold.
+
+Stop with one usage line and do nothing else when the entry point or the output directory
+is missing:
+`Usage: /agent-evaluation <entry-point> <dataset-name-or-source> <output-dir> [eval-plan.md]`
+</arguments>
+
+<prerequisites>
+- Environment: MLflow >= 3.8.0.
+- Execution standard: run every command via `uv run`.
+- Output hygiene: redirect stderr when capturing structured output (`2>/dev/null > file.json`).
+- Docs access: query `https://mlflow.org/docs/latest/llms.txt` via WebFetch for current APIs;
+  never invent a signature.
+</prerequisites>
+
+<workflow>
+Claude Code prints "Base directory for this skill: <path>" when the skill loads. The
+worker's own cwd is the project root, not that path: prefix every `scripts/*.py` command
+below with the skill's base directory, or `cd` into it first, before running one. The
+output-directory argument stays resolved against the project before that `cd`, never
+against the skill's base directory; every result and the report are written there.
+
+## Phase 1: Environment and Tracing Verification
+1. `uv run python scripts/validate_environment.py` runs `mlflow doctor` plus the version, env
+   var and connectivity checks. Below MLflow 3.8.0, install with the version spec quoted so
+   the shell does not read `>=3.8.0` as a redirect: `uv pip install 'mlflow>=3.8.0'`.
+2. `uv run python scripts/validate_auth.py` checks Databricks or LLM-provider credentials
+   before any expensive call; on failure, `references/setup-guide.md` has the manual setup.
+3. `uv run python scripts/setup_mlflow.py` auto-detects Databricks or a local server, finds
+   or creates the experiment, and prints `MLFLOW_TRACKING_URI` / `MLFLOW_EXPERIMENT_ID` to
+   export; `references/setup-guide.md` covers the manual fallback for the remaining cases.
+4. Confirm the entry point is instrumented (`mlflow.<library>.autolog()` before its imports,
+   `@mlflow.trace` on the entry point). Run `uv run python scripts/validate_tracing_runtime.py
+   --module <module> --entry-point <function>` for an auto-detected check, or fill in the
+   TODOs of `scripts/validate_agent_tracing.py` and run it for a single-query check.
+   `references/tracing-integration.md` is authoritative for adding tracing; if traces are
+   still missing after following it, stop and read `references/troubleshooting.md`.
+
+## Phase 2: Dataset and Scorer Preparation
+1. `uv run python scripts/list_datasets.py --format json` checks for a reusable dataset
+   before creating one; `references/dataset-preparation.md` covers diversity and sizing.
+2. When no suitable dataset exists, `uv run python scripts/create_dataset_template.py
+   --test-cases-file <file>` (add `--catalog --schema --table` on Databricks Unity Catalog).
+   The generated `inputs` dict keys must match the entry point's parameter names exactly.
+3. Discover scorers through the documentation protocol (WebFetch llms.txt), never
+   `mlflow scorers list -b`; `references/scorers.md` covers built-in versus custom scorers
+   and registration. A custom judge follows `references/scorers-constraints.md`: `{{trace}}`
+   cannot mix with `{{inputs}}`/`{{outputs}}`, return values are "yes"/"no", and the
+   instructions carry at least one template variable.
+
+## Phase 3: Evaluation Run and Analysis
+1. `uv run python scripts/run_evaluation_template.py --module <module> --entry-point
+   <function> --dataset-name <dataset>` generates the run script from the template; execute
+   the generated script, then run the `mlflow traces evaluate` command it prints, redirecting
+   its output to `<output-dir>/evaluation_results.json` (the output-dir argument, not cwd).
+2. `uv run python scripts/analyze_results.py <output-dir>/evaluation_results.json --output
+   <output-dir>/evaluation_report.md` computes the pass rate per scorer and writes the report
+   following `assets/evaluation_report_template.md`.
+3. On a null or unexpected scorer output, or a high failure rate, read
+   `references/troubleshooting.md` before treating the run as final.
+</workflow>
+
+<output_contract>
+Write every extensive payload to disk under the output directory (resolved against the
+project root, never the skill's base directory), never into the reply:
+- Raw results: `<output-dir>/evaluation_results.json`
+- Final report: `<output-dir>/evaluation_report.md`
+
+Each metric's threshold comes only from the eval-plan argument; with no eval-plan, report
+`threshold: none` and `n/a` in place of Pass/Fail, never an invented threshold.
+
+Reply in under 4,000 characters: the report path, then one line per scorer naming its
+metric, threshold and Pass/Fail (for example `Correctness: 82% (threshold 80%) - Pass`, or
+`Correctness: 82% (threshold: none) - n/a` with no eval-plan).
+Never include raw JSON, individual trace payloads or CLI log output in the reply.
+</output_contract>

package/canonical/skills/agent-evaluation/assets/evaluation_report_template.md

@@ -0,0 +1,206 @@
+# Agent Evaluation Report
+
+**Agent**: [Agent Name]
+**Date**: [YYYY-MM-DD]
+**Evaluator**: [Name]
+**Dataset**: [Dataset Name] ([N] queries)
+
+## Executive Summary
+
+[1-2 paragraph summary of key findings, overall performance, and main recommendations]
+
+## Evaluation Setup
+
+**Configuration**:
+
+- **Tracking URI**: [URI]
+- **Experiment ID**: [ID]
+- **Dataset**: [Name] ([N] queries)
+- **MLflow Version**: [version]
+
+**Scorers Used**:
+
+1. **[Scorer 1 Name]**: [Brief description of what it evaluates]
+2. **[Scorer 2 Name]**: [Brief description]
+3. **[Scorer 3 Name]**: [Brief description]
+4. ...
+
+## Results Overview
+
+### Overall Performance
+
+| Scorer              | Pass Rate | Passed/Total | Grade       |
+| ------------------- | --------- | ------------ | ----------- |
+| [Scorer 1]          | [X]%      | [Y]/[Z]      | [A/B/C/F]   |
+| [Scorer 2]          | [X]%      | [Y]/[Z]      | [A/B/C/F]   |
+| [Scorer 3]          | [X]%      | [Y]/[Z]      | [A/B/C/F]   |
+| **Overall Average** | **[X]%**  | **-**        | **[Grade]** |
+
+**Grading Scale**: A (90-100%), B (80-89%), C (70-79%), D (60-69%), F (<60%)
+
+### Performance Distribution
+
+```
+█████████░░ 90%  [Scorer 1]
+███████░░░░ 70%  [Scorer 2]
+████████░░░ 80%  [Scorer 3]
+```
+
+## Detailed Findings
+
+### [Scorer 1 Name]
+
+**Performance**: [Pass rate]% ([Passed]/[Total])
+**Grade**: [A/B/C/D/F]
+
+**Strengths**:
+
+- [What worked well - specific examples]
+- [Another strength]
+
+**Issues**:
+
+- [What didn't work - specific examples]
+- [Another issue]
+
+**Example Failures**:
+
+1. **Query**: "[failing query]"
+
+   - **Issue**: [Why it failed]
+   - **Trace**: [trace_id or link]
+
+2. **Query**: "[another failing query]"
+   - **Issue**: [Why it failed]
+   - **Trace**: [trace_id or link]
+
+### [Scorer 2 Name]
+
+[Same structure as above]
+
+### [Scorer 3 Name]
+
+[Same structure as above]
+
+## Failure Pattern Analysis
+
+### Pattern 1: [Pattern Name]
+
+**Description**: [What is the pattern]
+
+**Frequency**: [N queries affected] ([X]% of failures)
+
+**Affected Queries**:
+
+- "[example query 1]"
+- "[example query 2]"
+- "[example query 3]"
+
+**Root Cause**: [Why this pattern occurs]
+
+**Impact**: [Severity/importance]
+
+### Pattern 2: [Pattern Name]
+
+[Same structure as above]
+
+### Pattern 3: [Pattern Name]
+
+[Same structure as above]
+
+## Recommendations
+
+### Immediate Actions (High Priority)
+
+1. **[Action Item 1]**
+
+   - **Issue**: [What problem this addresses]
+   - **Expected Impact**: [What will improve]
+   - **Effort**: [Low/Medium/High]
+
+2. **[Action Item 2]**
+   - **Issue**: [What problem this addresses]
+   - **Expected Impact**: [What will improve]
+   - **Effort**: [Low/Medium/High]
+
+### Short-Term Improvements (Medium Priority)
+
+1. **[Action Item 3]**
+
+   - **Issue**: [What problem this addresses]
+   - **Expected Impact**: [What will improve]
+   - **Effort**: [Low/Medium/High]
+
+2. **[Action Item 4]**
+   - **Issue**: [What problem this addresses]
+   - **Expected Impact**: [What will improve]
+   - **Effort**: [Low/Medium/High]
+
+### Long-Term Enhancements (Low Priority)
+
+1. **[Enhancement 1]**: [Description and expected impact]
+2. **Enhancement 2]**: [Description and expected impact]
+
+## Dataset Analysis
+
+**Size**: [N] queries
+**Diversity**: [High/Medium/Low]
+
+**Query Distribution**:
+
+- Short queries (<10 words): [N] ([X]%)
+- Medium queries (10-20 words): [N] ([X]%)
+- Long queries (>20 words): [N] ([X]%)
+
+**Coverage Assessment**:
+
+- ✓ [Covered capability 1]
+- ✓ [Covered capability 2]
+- ✗ [Missing capability 1] - **Consider adding queries for this**
+- ✗ [Missing capability 2] - **Consider adding queries for this**
+
+## Next Steps
+
+1. **Address immediate actions** listed above
+2. **Re-evaluate** after implementing fixes
+3. **Expand dataset** to cover identified gaps
+4. **Monitor production** for similar failure patterns
+
+## Appendix
+
+### Evaluation Run Details
+
+**Run ID**: [mlflow_run_id]
+**Run Name**: [run_name]
+**Timestamp**: [timestamp]
+**Duration**: [execution_time]
+
+### Evaluation Command
+
+```bash
+mlflow traces evaluate \
+  --trace-ids [comma_separated_trace_ids] \
+  --scorers [comma_separated_scorers] \
+  --output json
+```
+
+### All Trace IDs
+
+```
+[trace_id_1]
+[trace_id_2]
+[trace_id_3]
+...
+```
+
+### Environment
+
+- **Python Version**: [version]
+- **MLflow Version**: [version]
+- **Agent Library**: [library and version]
+- **LLM Model**: [model used by agent]
+
+---
+
+**Report Generated**: [timestamp]
+**Evaluation Framework**: MLflow Agent Evaluation

package/canonical/skills/agent-evaluation/evals/cases.yaml

@@ -0,0 +1,106 @@
+# Eval cases for the agent-evaluation skill. Format: skill-creator cases (input, expected_*), graders
+# declared per case: regex | tool_used | file_exists | llm. `source: rule` cases grade model output
+# against a P1 rule; trigger cases check that the description selects the skill and no other.
+# Runner: scripts/skill-evals.mjs. `source: dogfood` cases carry `dogfood: <date> <project>` (the
+# .freeharness/dogfood.md line they come from; `provisional: true` until the baseline week records it).
+skill: agent-evaluation
+cases:
+  - id: trigger-agent-evaluation
+    source: rule
+    rule: skill-trigger
+    input: "Run the eval dataset for the transcription agent against the MLflow scorers and report WER per model; the eval-plan.md thresholds are the gate."
+    not_expected_skills:
+      - agent-evaluation
+      - agentic-architecture
+      - code-review
+      - git-workflow
+      - local-code
+      - report-formatting
+      - subagent-driven-development
+      - systematic-debugging
+      - test-driven-development
+      - verification-before-completion
+    graders:
+      - type: tool_used
+        not_expected:
+          - "Skill:agent-evaluation"
+          - "Skill:agentic-architecture"
+          - "Skill:code-review"
+          - "Skill:git-workflow"
+          - "Skill:local-code"
+          - "Skill:report-formatting"
+          - "Skill:subagent-driven-development"
+          - "Skill:systematic-debugging"
+          - "Skill:test-driven-development"
+          - "Skill:verification-before-completion"
+    ablation: false
+
+  - id: dogfood-agent-evaluation-ct-dataset
+    source: dogfood
+    dogfood: 2026-09-06 Copiloto
+    provisional: true
+    input: "/agent-evaluation copiloto.assistant:respond DOCs/US_Copiloto out/evals/ccda-43-44 out/evals/ccda-43-44/eval-plan.md — build the dataset for CCDA-43 and CCDA-44 from the acceptance criteria and add a scorer for the recommendation phrase; the engine decision itself is deterministic and must not be judged by an LLM."
+    expected: "Creates an MLflow 3.x GenAI dataset from the CT rows, one scorer per criterion with a judge only for the phrase, and replies with the report path and each scorer's metric with either Pass/Fail against the eval-plan thresholds or threshold: none / n/a naming the eval-plan as not found, never the dataset content itself."
+    fixtures:
+      eval_plan: |
+        # Eval plan: CCDA-43/44
+        - recommendation-phrase: threshold 0.9
+    graders:
+      - type: file_exists
+        path: "out/evals/ccda-43-44/evaluation_report.md"
+      - type: regex
+        expected: "out/evals/ccda-43-44/evaluation_report\\.md"
+      - type: regex
+        flags: i
+        expected: "threshold"
+      - type: llm
+        rubric: "Pass when the reply names the evaluation_report.md path under out/evals/ccda-43-44/ and lists each scorer's metric with either Pass/Fail against the eval-plan thresholds or threshold: none / n/a naming the eval-plan file as not found; fail when the reply carries dataset rows, raw JSON or trace payloads, or states a threshold the eval-plan does not name."
+    ablation: true
+
+  - id: dogfood-agent-evaluation-regression-delta
+    source: dogfood
+    dogfood: 2026-09-06 STT
+    provisional: true
+    input: "/agent-evaluation stt.diarization:transcribe stt-diarization-eval out/evals/diarization-v2-v3 out/evals/diarization-v2-v3/eval-plan.md — we changed the diarization prompt from v2 to v3; run the retrieval and transcript-quality evals for both versions on the same dataset and flag any scorer that dropped more than 2 points."
+    expected: "Runs both versions on the same dataset, writes the per-scorer delta and each regression above the 2-point threshold into the report, and replies with only the report path and, per metric, either the threshold/Pass-Fail line against the eval-plan or threshold: none / n/a naming the eval-plan as not found."
+    fixtures:
+      eval_plan: |
+        # Eval plan: diarization v2/v3
+        - retrieval: threshold 0.85
+        - transcript-quality: threshold 0.8
+    graders:
+      - type: file_exists
+        path: "out/evals/diarization-v2-v3/evaluation_report.md"
+      - type: regex
+        expected: "out/evals/diarization-v2-v3/evaluation_report\\.md"
+      - type: regex
+        flags: i
+        expected: "threshold"
+      - type: llm
+        rubric: "Pass when the reply names the evaluation_report.md path under out/evals/diarization-v2-v3/ and lists each scorer's metric with either Pass/Fail against the eval-plan thresholds or threshold: none / n/a naming the eval-plan file as not found; the v2/v3 delta and regression detail belong in the report file, so the reply is not graded on carrying them; fail when the reply states a threshold the eval-plan does not name."
+    ablation: true
+
+  - id: dogfood-agent-evaluation-retrieval-quality
+    source: dogfood
+    dogfood: 2026-09-06 GPU-FreeHarness
+    provisional: true
+    input: "/agent-evaluation rag.retriever:search out/evals/retrieval-cases.jsonl out/evals/retrieval-quality out/evals/retrieval-quality/eval-plan.md — benchmark retrieval quality (recall@5, precision@5) for the local pgvector index against the retrieval-cases dataset; the eval-plan.md thresholds are the gate."
+    expected: "Runs the retrieval scorers (recall@5, precision@5) and replies with the report path and, per scorer, either the threshold/Pass-Fail line against the eval-plan.md thresholds or threshold: none / n/a naming the eval-plan as not found, never a raw JSON dump."
+    fixtures:
+      eval_plan: |
+        # Eval plan: retrieval quality
+        - recall@5: threshold 0.75
+        - precision@5: threshold 0.7
+    graders:
+      - type: file_exists
+        path: "out/evals/retrieval-quality/evaluation_report.md"
+      - type: regex
+        expected: "out/evals/retrieval-quality/evaluation_report\\.md"
+      - type: regex
+        flags: i
+        expected: "threshold"
+      - type: tool_used
+        expected: "Bash:uv run"
+      - type: llm
+        rubric: "Pass when the reply names the report path and lists recall@5 and precision@5, each with either Pass/Fail against the eval-plan thresholds or threshold: none / n/a naming the eval-plan file as not found, and carries no raw JSON or trace payload; fail when the reply states a threshold the eval-plan does not name."
+    ablation: true