agentpack-cli 0.3.9__tar.gz → 0.3.10__tar.gz

{agentpack_cli-0.3.9 → agentpack_cli-0.3.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentpack-cli
-Version: 0.3.9
+Version: 0.3.10
 Summary: Local context engine for AI coding agents that ranks relevant files and builds task-focused context packs.
 License: MIT
 License-File: LICENSE
@@ -40,13 +40,14 @@ Description-Content-Type: text/markdown
 # AgentPack
 
 [![PyPI version](https://img.shields.io/pypi/v/agentpack-cli.svg)](https://pypi.org/project/agentpack-cli/)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/agentpack-cli?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/agentpack-cli)
 [![npm version](https://img.shields.io/npm/v/@vishal2612200/agentpack.svg)](https://www.npmjs.com/package/@vishal2612200/agentpack)
 [![npm downloads](https://img.shields.io/npm/dm/@vishal2612200/agentpack.svg)](https://www.npmjs.com/package/@vishal2612200/agentpack)
 [![Python versions](https://img.shields.io/pypi/pyversions/agentpack-cli.svg)](https://pypi.org/project/agentpack-cli/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/vishal2612200/agentpack/actions/workflows/ci.yml/badge.svg)](https://github.com/vishal2612200/agentpack/actions/workflows/ci.yml)
 
-> **Status: alpha (v0.3.9).** Works, tested, used in real sessions. Python and JavaScript/TypeScript are the best-supported languages. Public benchmark proof exists for the current suite, but broader repo coverage is still growing. API may change before 1.0.
+> **Status: alpha (v0.3.10).** Works, tested, used in real sessions. Python and JavaScript/TypeScript are the best-supported languages. Public benchmark proof exists for the current suite, but broader repo coverage is still growing. API may change before 1.0.
 >
 > **Platform note:** macOS, Linux, and Windows are supported. Windows support targets PowerShell plus Git for Windows. `cmd.exe` and bare Git setups are not a supported path yet.
 
@@ -601,6 +602,7 @@ Command map:
 | `agentpack doctor` | Audit hooks, agent files, CLI path, and repo health |
 | `agentpack explain` | Understand why a file was selected or omitted |
 | `agentpack benchmark` | Measure recall, precision, and misses against real tasks |
+| `agentpack eval` | Run deterministic failure evals with tests, diff limits, and taxonomy labels |
 | `agentpack tune` | Suggest fixes from recent pack metrics and benchmark misses |
 | `agentpack status` | Inspect current pack freshness and metadata |
 | `agentpack diff` | Show what changed between context snapshots |
@@ -1211,6 +1213,82 @@ This command does not pretend a pack is correct. It gives the next thing to insp
 
 ---
 
+### `agentpack eval`
+
+Run deterministic failure evals. AgentPack does not run the coding agent and
+does not use an LLM judge; it verifies the current or replayed worktree with
+commands and diff policies.
+
+```bash
+agentpack eval --init
+# edit .agentpack/evals.toml with real failures and checks
+agentpack eval
+agentpack eval --case auth-timeout --prove-targets
+agentpack eval --capture auth-timeout --failure-class context --check "pytest tests/test_auth.py -q"
+agentpack eval --watch --until-pass
+agentpack eval --replay --prove-targets
+agentpack eval --variant baseline
+agentpack eval --variant agentpack
+agentpack eval --compare-variants baseline:agentpack
+agentpack eval --ci-template
+agentpack eval --report
+```
+
+Example case:
+
+```toml
+[[cases]]
+id = "auth-timeout"
+task = "fix auth token timeout"
+failure_class = "context"
+failure_source = "agent_failed"
+base_ref = "HEAD"
+patch_file = ".agentpack/evals/auth-timeout.patch"
+required_changed_files = ["src/auth/token.py"]
+forbidden_changed_files = ["src/db/**"]
+max_changed_files = 5
+max_changed_lines = 250
+agent = "codex"
+context_file = ".agentpack/context.md"
+context_hash = "..."
+selected_files = ["src/auth/token.py", "tests/test_auth.py"]
+
+[[cases.checks]]
+name = "tests"
+command = "pytest tests/test_auth.py -q"
+timeout_s = 120
+retries = 1 # optional, marks pass-after-fail checks as flaky
+```
+
+Use `eval` after an agent run: capture the real failure, add deterministic
+checks such as tests, typecheck, lint, schema validation, API contract tests,
+diff size, forbidden files, or golden outputs, then rerun until the harness
+passes. The model can propose; the harness must verify.
+
+For hands-free local iteration, keep `agentpack eval --watch --until-pass`
+running in a terminal while the agent or developer edits. It reruns when the
+case file, patch artifacts, golden files, or git diff content changes and stops
+when all deterministic checks pass. `--capture` stores the current patch under
+`.agentpack/evals/<case-id>.patch` plus context metadata; `--replay` checks out
+`base_ref` into an isolated git worktree, applies that patch, and runs the same
+deterministic checks there. To measure AgentPack's contribution, run the same
+case with `--variant baseline` and then with `--variant agentpack`;
+`--compare-variants baseline:agentpack` reports which cases improved, regressed,
+stayed unchanged, or still need both sides. Use `--ci-template` to scaffold a
+GitHub Actions workflow for `benchmarks/evals.toml`.
+
+Eval files are executable trust boundaries: commands in `checks.command` run
+locally and in CI. Review eval TOML from contributors with the same care as
+shell scripts or workflow files.
+
+Captured patch artifacts are secret-scanned with the same local redactor used
+for context packs before they are written. If a patch line contains a real
+secret, the artifact stores `[REDACTED:<type>]` and the case records
+`patch_redaction_warnings`. Secret-bearing patches may replay with redacted
+values; replace secrets with safe fixture values when exact replay matters.
+
+---
+
 ### `agentpack status`
 
 Check whether the context pack is stale.

{agentpack_cli-0.3.9 → agentpack_cli-0.3.10}/README.md

@@ -1,13 +1,14 @@
 # AgentPack
 
 [![PyPI version](https://img.shields.io/pypi/v/agentpack-cli.svg)](https://pypi.org/project/agentpack-cli/)
+[![PyPI Downloads](https://static.pepy.tech/personalized-badge/agentpack-cli?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/agentpack-cli)
 [![npm version](https://img.shields.io/npm/v/@vishal2612200/agentpack.svg)](https://www.npmjs.com/package/@vishal2612200/agentpack)
 [![npm downloads](https://img.shields.io/npm/dm/@vishal2612200/agentpack.svg)](https://www.npmjs.com/package/@vishal2612200/agentpack)
 [![Python versions](https://img.shields.io/pypi/pyversions/agentpack-cli.svg)](https://pypi.org/project/agentpack-cli/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/vishal2612200/agentpack/actions/workflows/ci.yml/badge.svg)](https://github.com/vishal2612200/agentpack/actions/workflows/ci.yml)
 
-> **Status: alpha (v0.3.9).** Works, tested, used in real sessions. Python and JavaScript/TypeScript are the best-supported languages. Public benchmark proof exists for the current suite, but broader repo coverage is still growing. API may change before 1.0.
+> **Status: alpha (v0.3.10).** Works, tested, used in real sessions. Python and JavaScript/TypeScript are the best-supported languages. Public benchmark proof exists for the current suite, but broader repo coverage is still growing. API may change before 1.0.
 >
 > **Platform note:** macOS, Linux, and Windows are supported. Windows support targets PowerShell plus Git for Windows. `cmd.exe` and bare Git setups are not a supported path yet.
 
@@ -562,6 +563,7 @@ Command map:
 | `agentpack doctor` | Audit hooks, agent files, CLI path, and repo health |
 | `agentpack explain` | Understand why a file was selected or omitted |
 | `agentpack benchmark` | Measure recall, precision, and misses against real tasks |
+| `agentpack eval` | Run deterministic failure evals with tests, diff limits, and taxonomy labels |
 | `agentpack tune` | Suggest fixes from recent pack metrics and benchmark misses |
 | `agentpack status` | Inspect current pack freshness and metadata |
 | `agentpack diff` | Show what changed between context snapshots |
@@ -1172,6 +1174,82 @@ This command does not pretend a pack is correct. It gives the next thing to insp
 
 ---
 
+### `agentpack eval`
+
+Run deterministic failure evals. AgentPack does not run the coding agent and
+does not use an LLM judge; it verifies the current or replayed worktree with
+commands and diff policies.
+
+```bash
+agentpack eval --init
+# edit .agentpack/evals.toml with real failures and checks
+agentpack eval
+agentpack eval --case auth-timeout --prove-targets
+agentpack eval --capture auth-timeout --failure-class context --check "pytest tests/test_auth.py -q"
+agentpack eval --watch --until-pass
+agentpack eval --replay --prove-targets
+agentpack eval --variant baseline
+agentpack eval --variant agentpack
+agentpack eval --compare-variants baseline:agentpack
+agentpack eval --ci-template
+agentpack eval --report
+```
+
+Example case:
+
+```toml
+[[cases]]
+id = "auth-timeout"
+task = "fix auth token timeout"
+failure_class = "context"
+failure_source = "agent_failed"
+base_ref = "HEAD"
+patch_file = ".agentpack/evals/auth-timeout.patch"
+required_changed_files = ["src/auth/token.py"]
+forbidden_changed_files = ["src/db/**"]
+max_changed_files = 5
+max_changed_lines = 250
+agent = "codex"
+context_file = ".agentpack/context.md"
+context_hash = "..."
+selected_files = ["src/auth/token.py", "tests/test_auth.py"]
+
+[[cases.checks]]
+name = "tests"
+command = "pytest tests/test_auth.py -q"
+timeout_s = 120
+retries = 1 # optional, marks pass-after-fail checks as flaky
+```
+
+Use `eval` after an agent run: capture the real failure, add deterministic
+checks such as tests, typecheck, lint, schema validation, API contract tests,
+diff size, forbidden files, or golden outputs, then rerun until the harness
+passes. The model can propose; the harness must verify.
+
+For hands-free local iteration, keep `agentpack eval --watch --until-pass`
+running in a terminal while the agent or developer edits. It reruns when the
+case file, patch artifacts, golden files, or git diff content changes and stops
+when all deterministic checks pass. `--capture` stores the current patch under
+`.agentpack/evals/<case-id>.patch` plus context metadata; `--replay` checks out
+`base_ref` into an isolated git worktree, applies that patch, and runs the same
+deterministic checks there. To measure AgentPack's contribution, run the same
+case with `--variant baseline` and then with `--variant agentpack`;
+`--compare-variants baseline:agentpack` reports which cases improved, regressed,
+stayed unchanged, or still need both sides. Use `--ci-template` to scaffold a
+GitHub Actions workflow for `benchmarks/evals.toml`.
+
+Eval files are executable trust boundaries: commands in `checks.command` run
+locally and in CI. Review eval TOML from contributors with the same care as
+shell scripts or workflow files.
+
+Captured patch artifacts are secret-scanned with the same local redactor used
+for context packs before they are written. If a patch line contains a real
+secret, the artifact stores `[REDACTED:<type>]` and the case records
+`patch_redaction_warnings`. Secret-bearing patches may replay with redacted
+values; replace secrets with safe fixture values when exact replay matters.
+
+---
+
 ### `agentpack status`
 
 Check whether the context pack is stale.

{agentpack_cli-0.3.9 → agentpack_cli-0.3.10}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agentpack-cli"
-version = "0.3.9"
+version = "0.3.10"
 description = "Local context engine for AI coding agents that ranks relevant files and builds task-focused context packs."
 readme = "README.md"
 requires-python = ">=3.10"

{agentpack_cli-0.3.9 → agentpack_cli-0.3.10}/src/agentpack/__init__.py

@@ -1,3 +1,3 @@
 """AgentPack — task-aware context packing for AI coding agents."""
 
-__version__ = "0.3.9"
+__version__ = "0.3.10"

{agentpack_cli-0.3.9 → agentpack_cli-0.3.10}/src/agentpack/cli.py

@@ -6,6 +6,7 @@ from agentpack.commands import (
     claude_cmd,
     diff,
     doctor,
+    eval_cmd,
     explain,
     guard,
     hook_cmd,
@@ -60,6 +61,7 @@ for mod in [
     explain,
     guard,
     doctor,
+    eval_cmd,
     tune,
     watch,
     claude_cmd,

agentpack_cli-0.3.10/src/agentpack/commands/eval_cmd.py

@@ -0,0 +1,264 @@
+from __future__ import annotations
+
+from pathlib import Path
+import time
+
+import typer
+from rich.table import Table
+from rich import box
+
+from agentpack.commands._shared import console, _root
+from agentpack.core.evals import (
+    FAILURE_CLASSES,
+    append_captured_eval_case,
+    compare_eval_variants,
+    default_eval_cases_path,
+    eval_results_path,
+    eval_watch_fingerprint,
+    load_eval_cases,
+    load_eval_result_records,
+    persist_eval_results,
+    run_eval_suite,
+    scaffold_eval_cases,
+    write_eval_ci_template,
+    write_eval_report,
+)
+
+
+def register(app: typer.Typer) -> None:
+    @app.command(name="eval")
+    def eval_command(
+        init: bool = typer.Option(False, "--init", is_flag=True, help="Scaffold .agentpack/evals.toml and exit."),
+        cases: str = typer.Option("", "--cases", help="Path to eval TOML file (default: .agentpack/evals.toml)."),
+        case: str = typer.Option("", "--case", help="Run one eval case by id."),
+        prove_targets: bool = typer.Option(False, "--prove-targets", is_flag=True, help="Exit non-zero when any eval case fails."),
+        capture: str = typer.Option("", "--capture", help="Append a case from current git diff using this id."),
+        failure_class: str = typer.Option("context", "--failure-class", help=f"Failure class ({' | '.join(FAILURE_CLASSES)})."),
+        failure_source: str = typer.Option("agent_failed", "--failure-source", help="Failure source for captured cases."),
+        check: list[str] | None = typer.Option(None, "--check", help="Deterministic command check for --capture. Repeatable."),
+        task: str = typer.Option("", "--task", help="Task text for --capture."),
+        base_ref: str = typer.Option("HEAD", "--base-ref", help="Git base ref for diff checks."),
+        report: bool = typer.Option(False, "--report", is_flag=True, help="Write benchmarks/results/YYYY-MM-DD-eval.md."),
+        ci_template: bool = typer.Option(False, "--ci-template", is_flag=True, help="Scaffold .github/workflows/agentpack-eval.yml and exit."),
+        variant: str = typer.Option("agentpack", "--variant", help="Result variant label, e.g. baseline or agentpack."),
+        compare_variants: str = typer.Option("", "--compare-variants", help="Compare latest results as BASELINE:VARIANT."),
+        replay: bool = typer.Option(False, "--replay", is_flag=True, help="Run cases in isolated git worktrees using captured patch_file artifacts."),
+        watch: bool = typer.Option(False, "--watch", is_flag=True, help="Rerun evals when git diff state changes."),
+        interval: float = typer.Option(2.0, "--interval", help="Watch polling interval in seconds."),
+        max_runs: int = typer.Option(0, "--max-runs", help="Maximum watch runs (0 = unlimited)."),
+        until_pass: bool = typer.Option(False, "--until-pass", is_flag=True, help="Stop watch mode after all cases pass."),
+        agent: str = typer.Option("", "--agent", help="Agent label to store with --capture metadata."),
+        prompt_file: str = typer.Option("", "--prompt-file", help="Prompt artifact path to store with --capture."),
+        context_file: str = typer.Option(".agentpack/context.md", "--context-file", help="Context artifact path to store with --capture."),
+    ) -> None:
+        """Run deterministic eval cases without using an LLM judge."""
+        root = _root()
+        cases_path = Path(cases) if cases else default_eval_cases_path(root)
+
+        if compare_variants:
+            _print_variant_comparison(root, compare_variants)
+            return
+
+        if ci_template:
+            out = write_eval_ci_template(root)
+            console.print(f"[green]✓[/] Created [bold]{out}[/]")
+            return
+
+        if init:
+            out = scaffold_eval_cases(root)
+            console.print(f"[green]✓[/] Created [bold]{out}[/]")
+            console.print("  Edit it with real failures, then run [bold]agentpack eval[/].")
+            return
+
+        if capture:
+            try:
+                captured = append_captured_eval_case(
+                    cases_path,
+                    root=root,
+                    case_id=capture,
+                    failure_class=failure_class,
+                    checks=check or [],
+                    task=task,
+                    failure_source=failure_source,
+                    base_ref=base_ref,
+                    agent=agent,
+                    prompt_file=prompt_file,
+                    context_file=context_file,
+                )
+            except ValueError as exc:
+                console.print(f"[red]{exc}[/]")
+                raise typer.Exit(1) from exc
+            console.print(f"[green]✓[/] Captured eval case [bold]{captured.id}[/] in [bold]{cases_path}[/]")
+            console.print(f"  Required changed files: {len(captured.required_changed_files)}")
+            if captured.patch_redaction_warnings:
+                console.print(f"  [yellow]Redacted {len(captured.patch_redaction_warnings)} secret(s) from patch artifact.[/]")
+            return
+
+        if report and not cases_path.exists():
+            records = load_eval_result_records(eval_results_path(root))
+            out = write_eval_report(root, records)
+            console.print(f"[green]✓[/] Wrote eval report: [bold]{out}[/]")
+            return
+
+        if not cases_path.exists():
+            console.print(f"[yellow]No eval cases file found at {cases_path}[/]")
+            console.print("  Run [bold]agentpack eval --init[/] to scaffold one.")
+            raise typer.Exit(1)
+
+        try:
+            eval_cases = load_eval_cases(cases_path)
+        except ValueError as exc:
+            console.print(f"[red]{exc}[/]")
+            raise typer.Exit(1) from exc
+
+        if case:
+            eval_cases = [item for item in eval_cases if item.id == case]
+            if not eval_cases:
+                console.print(f"[yellow]No eval case found with id: {case}[/]")
+                raise typer.Exit(1)
+
+        if not eval_cases:
+            console.print("[yellow]No eval cases defined.[/]")
+            raise typer.Exit(1)
+
+        if watch:
+            results = _watch_eval_cases(
+                root,
+                eval_cases,
+                variant=variant,
+                replay=replay,
+                interval=interval,
+                max_runs=max_runs,
+                until_pass=until_pass,
+                extra_paths=[cases_path],
+            )
+        else:
+            results = _run_once(root, eval_cases, variant=variant, replay=replay)
+
+        if report:
+            records = load_eval_result_records(eval_results_path(root))
+            out = write_eval_report(root, records)
+            console.print(f"[green]✓[/] Wrote eval report: [bold]{out}[/]")
+
+        if prove_targets and not all(result.passed for result in results):
+            raise typer.Exit(2)
+
+
+def _run_once(root: Path, eval_cases, *, variant: str, replay: bool):
+    console.print(f"\n[bold]Running {len(eval_cases)} deterministic eval case(s)...[/]\n")
+    results = run_eval_suite(root, eval_cases, variant=variant, replay=replay)
+    persist_eval_results(root, results)
+    _print_results(results)
+    return results
+
+
+def _watch_eval_cases(
+    root: Path,
+    eval_cases,
+    *,
+    variant: str,
+    replay: bool,
+    interval: float,
+    max_runs: int,
+    until_pass: bool,
+    extra_paths: list[Path],
+):
+    if interval <= 0:
+        console.print("[red]--interval must be greater than 0[/]")
+        raise typer.Exit(1)
+    if max_runs < 0:
+        console.print("[red]--max-runs must be 0 or greater[/]")
+        raise typer.Exit(1)
+
+    console.print("[bold]Watching deterministic evals.[/] Press Ctrl-C to stop.")
+    last_fingerprint = ""
+    last_results = []
+    patch_paths = [root / case.patch_file for case in eval_cases if case.patch_file]
+    golden_paths = [root / golden.expected for case in eval_cases for golden in case.golden_files]
+    watched_paths = extra_paths + patch_paths + golden_paths
+    runs = 0
+    try:
+        while True:
+            fingerprint = eval_watch_fingerprint(root, eval_cases, extra_paths=watched_paths)
+            if fingerprint != last_fingerprint:
+                runs += 1
+                last_fingerprint = fingerprint
+                last_results = _run_once(root, eval_cases, variant=variant, replay=replay)
+                if until_pass and all(result.passed for result in last_results):
+                    console.print("[green]✓[/] All eval cases pass; watch stopped.")
+                    break
+                if max_runs and runs >= max_runs:
+                    break
+            time.sleep(interval)
+    except KeyboardInterrupt:
+        console.print("\n[yellow]Eval watch stopped.[/]")
+    return last_results
+
+
+def _print_results(results) -> None:
+    tbl = Table(box=box.SIMPLE, show_header=True, padding=(0, 1))
+    tbl.add_column("case", max_width=32)
+    tbl.add_column("status", width=8)
+    tbl.add_column("class", max_width=18)
+    tbl.add_column("checks", justify="right")
+    tbl.add_column("changed", justify="right")
+    tbl.add_column("lines", justify="right")
+    tbl.add_column("time", justify="right")
+
+    for result in results:
+        status = "[green]pass[/]" if result.passed else "[red]fail[/]"
+        failed = len(result.failed_checks)
+        checks = f"{len(result.checks) - failed}/{len(result.checks)}"
+        tbl.add_row(
+            result.case.id,
+            status,
+            result.case.failure_class,
+            checks,
+            str(len(result.changed_files)),
+            str(result.changed_lines),
+            f"{result.duration_s:.2f}s",
+        )
+
+    console.print(tbl)
+    for result in results:
+        for check in result.failed_checks:
+            detail = f": {check.detail}" if check.detail else ""
+            console.print(f"  [red]![/] {result.case.id} / {check.name}{detail}", soft_wrap=True)
+
+
+def _print_variant_comparison(root: Path, compare_variants: str) -> None:
+    try:
+        baseline, variant = compare_variants.split(":", 1)
+    except ValueError as exc:
+        console.print("[red]--compare-variants must use BASELINE:VARIANT, e.g. baseline:agentpack[/]")
+        raise typer.Exit(1) from exc
+    records = load_eval_result_records(eval_results_path(root))
+    comparison = compare_eval_variants(records, baseline, variant)
+
+    tbl = Table(title=f"Eval Variant Comparison: {baseline} → {variant}", box=box.SIMPLE, show_header=True, padding=(0, 1))
+    tbl.add_column("case", max_width=36)
+    tbl.add_column(baseline, justify="center")
+    tbl.add_column(variant, justify="center")
+    tbl.add_column("status", max_width=12)
+    for row in comparison["rows"]:
+        tbl.add_row(
+            row["case_id"],
+            _pass_label(row["baseline_passed"]),
+            _pass_label(row["variant_passed"]),
+            row["status"],
+        )
+    console.print(tbl)
+    console.print(
+        f"  improved [bold green]{comparison['improved']}[/]  "
+        f"regressed [bold red]{comparison['regressed']}[/]  "
+        f"unchanged [bold]{comparison['unchanged']}[/]  "
+        f"incomplete [bold yellow]{comparison['incomplete']}[/]"
+    )
+
+
+def _pass_label(value) -> str:
+    if value is True:
+        return "[green]pass[/]"
+    if value is False:
+        return "[red]fail[/]"
+    return "[yellow]-[/]"

{agentpack_cli-0.3.9 → agentpack_cli-0.3.10}/src/agentpack/commands/tune.py

@@ -49,6 +49,7 @@ def _build_tuning_suggestions(root: Path, *, include_benchmark: bool = True) ->
     suggestions: list[TuningSuggestion] = []
     metrics = _load_jsonl(root / ".agentpack" / "metrics.jsonl")
     benchmark = _load_jsonl(root / ".agentpack" / "benchmark_results.jsonl") if include_benchmark else []
+    eval_results = _load_jsonl(root / ".agentpack" / "eval_results.jsonl")
 
     accuracy_rows = [row for row in metrics if "selection_recall" in row][-10:]
     if accuracy_rows:
@@ -107,6 +108,29 @@ def _build_tuning_suggestions(root: Path, *, include_benchmark: bool = True) ->
                     suggestion = "Use `agentpack explain --omitted --task <task>` to inspect the miss."
                 suggestions.append(TuningSuggestion("benchmark misses", f"{count} miss(es): {status}", suggestion))
 
+    if eval_results:
+        class_counts: dict[str, int] = {}
+        check_counts: dict[str, int] = {}
+        for row in eval_results[-20:]:
+            if row.get("passed") is False:
+                failure_class = str(row.get("failure_class") or "unknown")
+                class_counts[failure_class] = class_counts.get(failure_class, 0) + 1
+                for check in row.get("failed_checks", []) or []:
+                    if isinstance(check, str):
+                        check_counts[check] = check_counts.get(check, 0) + 1
+        for failure_class, count in sorted(class_counts.items(), key=lambda item: (-item[1], item[0]))[:4]:
+            suggestions.append(TuningSuggestion(
+                "eval failures",
+                f"{count} failure(s): {failure_class}",
+                "Use `agentpack eval --report` and inspect the failing deterministic checks before changing prompts or scoring.",
+            ))
+        for check, count in sorted(check_counts.items(), key=lambda item: (-item[1], item[0]))[:3]:
+            suggestions.append(TuningSuggestion(
+                "eval checks",
+                f"{count} failure(s): {check}",
+                "Strengthen or narrow this harness check if it is flaky; fix the agent workflow if it is deterministic.",
+            ))
+
     return suggestions