runledger 0.2.1__tar.gz → 0.2.2__tar.gz

{runledger-0.2.1 → runledger-0.2.2}/PKG-INFO

@@ -1,9 +1,13 @@
 Metadata-Version: 2.4
 Name: runledger
-Version: 0.2.1
+Version: 0.2.2
 Summary: CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates.
 Author: RunLedger
 License-Expression: MIT
+Project-URL: Homepage, https://runledger.io
+Project-URL: Repository, https://github.com/runledger/Runledger
+Project-URL: Documentation, https://github.com/runledger/Runledger/tree/main/docs
+Project-URL: Changelog, https://github.com/runledger/Runledger/blob/main/CHANGELOG.md
 Keywords: llm,agents,evaluation,ci,github-actions,replay,testing,junit,jsonschema
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
@@ -30,25 +34,57 @@ Dynamic: license-file
 
 Website: https://runledger.io
 
-**Deterministic CI regression checks for tool-using agents.** Record tool calls once, replay in CI, and fail PRs on mismatches, contract violations, or budget regressions.
+**Deterministic CI regression checks for tool-using agents.** Record tool calls once, replay them in CI, and fail the run when a schema, tool-order, or tool-call budget gate fails.
 
-- Demo repo: https://github.com/runledger/runledger-demo (main is green; regression branches fail for real reasons)
+- Demo: https://github.com/ZackMitchell910/runledger-demo/pull/1 (intentionally left open to show RunLedger failing a bad agent change)
 - If this is useful, please star the repo (it helps a lot).
 
-RunLedger is a **CI harness**, not an "eval metrics framework":
-- **DeepEval-style tools** help you *score* behavior.
-- **RunLedger** helps you *ship safely* by making agent tests deterministic and merge-gated.
+## Five-minute quickstart
 
----
-
-## Try it now (60 seconds)
+From an empty directory, with Python 3.9+:
 
 ```bash
-pipx install runledger
+python3 -m pip install runledger
 runledger init
 runledger run ./evals/demo --mode replay --baseline baselines/demo.json
 ```
 
+`runledger init` writes:
+
+- `evals/demo/` — replay suite, cassette, JSON Schema, and a small agent that calls `search_docs`
+- `baselines/demo.json` — baseline from that passing run
+- `.github/workflows/runledger.yml` — pull-request workflow that runs the same command
+
+A second `runledger init` leaves those files unchanged. `--force` regenerates them.
+
+The demo exits non-zero unless all three of these hold:
+
+- **Schema.** The final output matches `evals/demo/schema.json` (`category` and `reply`).
+- **Tool order.** The agent calls `search_docs` (`call_order` in `evals/demo/cases/t1.yaml`).
+- **Tool-call budget.** The run stays within `max_tool_calls: 1`.
+
+A passing run prints `PASS` and writes `runledger_out/demo/<run_id>/` (`summary.json`, `junit.xml`, `report.html`, `run.jsonl`). Commit the init files to run that gate on pull requests.
+
+The same check as a GitHub Action. `suite.yaml` already points at `baselines/demo.json`, so the action compares against that baseline:
+
+```yaml
+- id: evals
+  uses: runledger/Runledger@v0.2
+  with:
+    path: ./evals/demo
+    mode: replay
+```
+
+This repo's `action.yml` also accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. Those fields ship when a release containing this file is tagged. The in-repo self-test calls the action with `uses: ./`.
+
+Page copy for runledger.io lives in `docs/site-quickstart.md`.
+
+---
+
+RunLedger is a **CI harness**, not an "eval metrics framework":
+- **DeepEval-style tools** help you *score* behavior.
+- **RunLedger** helps you *ship safely* by making agent tests deterministic and merge-gated.
+
 ## Integrations (merged)
 
 - hoangsonww/Agentic-AI-Pipeline - https://github.com/hoangsonww/Agentic-AI-Pipeline/pull/24
@@ -73,7 +109,7 @@ RunLedger stops regressions by shifting from "vibes-based" evaluation to determi
 
 - **Record & replay:** record tool outputs once; replay them in CI for instant, stable tests.
 - **Strict contracts:** enforce tool schemas, calling order, and allowlists.
-- **Budget gating:** fail PRs automatically if wall time, tool-call count, or tool errors exceed defined limits.
+- **Budget gating:** fail the run when tool-call count exceeds `max_tool_calls`. Wall time and tool-error caps are separate gates, configured with `max_wall_ms` and `max_tool_errors`.
 
 ### RunLedger vs evaluation frameworks
 
@@ -352,18 +388,18 @@ jobs:
       - uses: actions/checkout@v4
 
       - name: Run deterministic evals (replay)
+        id: evals
         uses: runledger/Runledger@v0.2
         with:
           path: ./evals/demo
           mode: replay
 
-      - name: Upload eval artifacts
-        uses: actions/upload-artifact@v4
-        with:
-          name: agent-eval-artifacts
-          path: runledger_out/**
+      - name: Show the action result
+        run: echo "result=${{ steps.evals.outputs.result }} exit=${{ steps.evals.outputs.exit-code }}"
 ```
 
+`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml`. It uploads `runledger_out/**` itself. On a release that contains this `action.yml`, the step also sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`.
+
 ---
 
 ## Determinism guide (rules of the road)
@@ -386,6 +422,7 @@ jobs:
 
 ## Docs
 
+- `docs/site-quickstart.md` -- five-minute copy for runledger.io
 - `docs/quickstart.md` -- install + first run
 - `docs/assertions.md` -- assertions, tool contracts, budgets
 - `docs/baselines.md` -- baselines, diffing, promotion
@@ -398,7 +435,6 @@ jobs:
 ## Roadmap
 
 * `init` templates for Python (OpenAI SDK), LangGraph/LangChain, and Node/TS
-* richer budgets (tokens/cost) via optional `task_metrics`
 * PR comments bot for regression summaries
 * plugin system for custom assertions
 * HTML report trace viewer improvements

runledger-0.2.1/src/runledger.egg-info/PKG-INFO → runledger-0.2.2/README.md

@@ -1,27 +1,3 @@
-Metadata-Version: 2.4
-Name: runledger
-Version: 0.2.1
-Summary: CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates.
-Author: RunLedger
-License-Expression: MIT
-Keywords: llm,agents,evaluation,ci,github-actions,replay,testing,junit,jsonschema
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: typer>=0.9.0
-Requires-Dist: rich>=13.0.0
-Requires-Dist: pydantic>=2.0.0
-Requires-Dist: pyyaml>=6.0.0
-Requires-Dist: jsonschema>=4.0.0
-Requires-Dist: jinja2>=3.1.0
-Requires-Dist: eval_type_backport>=0.2.0; python_version < "3.10"
-Provides-Extra: dev
-Requires-Dist: pytest>=7.0.0; extra == "dev"
-Requires-Dist: pytest-timeout>=2.1.0; extra == "dev"
-Requires-Dist: ruff>=0.1.0; extra == "dev"
-Requires-Dist: mypy>=1.0.0; extra == "dev"
-Dynamic: license-file
-
 # RunLedger
 
 [![CI](https://github.com/runledger/Runledger/actions/workflows/ci.yml/badge.svg)](https://github.com/runledger/Runledger/actions/workflows/ci.yml)
@@ -30,25 +6,57 @@ Dynamic: license-file
 
 Website: https://runledger.io
 
-**Deterministic CI regression checks for tool-using agents.** Record tool calls once, replay in CI, and fail PRs on mismatches, contract violations, or budget regressions.
+**Deterministic CI regression checks for tool-using agents.** Record tool calls once, replay them in CI, and fail the run when a schema, tool-order, or tool-call budget gate fails.
 
-- Demo repo: https://github.com/runledger/runledger-demo (main is green; regression branches fail for real reasons)
+- Demo: https://github.com/ZackMitchell910/runledger-demo/pull/1 (intentionally left open to show RunLedger failing a bad agent change)
 - If this is useful, please star the repo (it helps a lot).
 
-RunLedger is a **CI harness**, not an "eval metrics framework":
-- **DeepEval-style tools** help you *score* behavior.
-- **RunLedger** helps you *ship safely* by making agent tests deterministic and merge-gated.
+## Five-minute quickstart
 
----
-
-## Try it now (60 seconds)
+From an empty directory, with Python 3.9+:
 
 ```bash
-pipx install runledger
+python3 -m pip install runledger
 runledger init
 runledger run ./evals/demo --mode replay --baseline baselines/demo.json
 ```
 
+`runledger init` writes:
+
+- `evals/demo/` — replay suite, cassette, JSON Schema, and a small agent that calls `search_docs`
+- `baselines/demo.json` — baseline from that passing run
+- `.github/workflows/runledger.yml` — pull-request workflow that runs the same command
+
+A second `runledger init` leaves those files unchanged. `--force` regenerates them.
+
+The demo exits non-zero unless all three of these hold:
+
+- **Schema.** The final output matches `evals/demo/schema.json` (`category` and `reply`).
+- **Tool order.** The agent calls `search_docs` (`call_order` in `evals/demo/cases/t1.yaml`).
+- **Tool-call budget.** The run stays within `max_tool_calls: 1`.
+
+A passing run prints `PASS` and writes `runledger_out/demo/<run_id>/` (`summary.json`, `junit.xml`, `report.html`, `run.jsonl`). Commit the init files to run that gate on pull requests.
+
+The same check as a GitHub Action. `suite.yaml` already points at `baselines/demo.json`, so the action compares against that baseline:
+
+```yaml
+- id: evals
+  uses: runledger/Runledger@v0.2
+  with:
+    path: ./evals/demo
+    mode: replay
+```
+
+This repo's `action.yml` also accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. Those fields ship when a release containing this file is tagged. The in-repo self-test calls the action with `uses: ./`.
+
+Page copy for runledger.io lives in `docs/site-quickstart.md`.
+
+---
+
+RunLedger is a **CI harness**, not an "eval metrics framework":
+- **DeepEval-style tools** help you *score* behavior.
+- **RunLedger** helps you *ship safely* by making agent tests deterministic and merge-gated.
+
 ## Integrations (merged)
 
 - hoangsonww/Agentic-AI-Pipeline - https://github.com/hoangsonww/Agentic-AI-Pipeline/pull/24
@@ -73,7 +81,7 @@ RunLedger stops regressions by shifting from "vibes-based" evaluation to determi
 
 - **Record & replay:** record tool outputs once; replay them in CI for instant, stable tests.
 - **Strict contracts:** enforce tool schemas, calling order, and allowlists.
-- **Budget gating:** fail PRs automatically if wall time, tool-call count, or tool errors exceed defined limits.
+- **Budget gating:** fail the run when tool-call count exceeds `max_tool_calls`. Wall time and tool-error caps are separate gates, configured with `max_wall_ms` and `max_tool_errors`.
 
 ### RunLedger vs evaluation frameworks
 
@@ -352,18 +360,18 @@ jobs:
       - uses: actions/checkout@v4
 
       - name: Run deterministic evals (replay)
+        id: evals
         uses: runledger/Runledger@v0.2
         with:
           path: ./evals/demo
           mode: replay
 
-      - name: Upload eval artifacts
-        uses: actions/upload-artifact@v4
-        with:
-          name: agent-eval-artifacts
-          path: runledger_out/**
+      - name: Show the action result
+        run: echo "result=${{ steps.evals.outputs.result }} exit=${{ steps.evals.outputs.exit-code }}"
 ```
 
+`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml`. It uploads `runledger_out/**` itself. On a release that contains this `action.yml`, the step also sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`.
+
 ---
 
 ## Determinism guide (rules of the road)
@@ -386,6 +394,7 @@ jobs:
 
 ## Docs
 
+- `docs/site-quickstart.md` -- five-minute copy for runledger.io
 - `docs/quickstart.md` -- install + first run
 - `docs/assertions.md` -- assertions, tool contracts, budgets
 - `docs/baselines.md` -- baselines, diffing, promotion
@@ -398,7 +407,6 @@ jobs:
 ## Roadmap
 
 * `init` templates for Python (OpenAI SDK), LangGraph/LangChain, and Node/TS
-* richer budgets (tokens/cost) via optional `task_metrics`
 * PR comments bot for regression summaries
 * plugin system for custom assertions
 * HTML report trace viewer improvements

{runledger-0.2.1 → runledger-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "runledger"
-version = "0.2.1"
+version = "0.2.2"
 description = "CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -34,6 +34,12 @@ dependencies = [
   "eval_type_backport>=0.2.0; python_version < \"3.10\"",
 ]
 
+[project.urls]
+Homepage = "https://runledger.io"
+Repository = "https://github.com/runledger/Runledger"
+Documentation = "https://github.com/runledger/Runledger/tree/main/docs"
+Changelog = "https://github.com/runledger/Runledger/blob/main/CHANGELOG.md"
+
 [project.optional-dependencies]
 dev = [
   "pytest>=7.0.0",

{runledger-0.2.1 → runledger-0.2.2}/src/runledger/__init__.py

@@ -1,3 +1,3 @@
 __all__ = ["__version__"]
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"

{runledger-0.2.1 → runledger-0.2.2}/src/runledger/cli.py

@@ -4,6 +4,7 @@ from datetime import datetime, timezone
 import json
 import os
 import shutil
+import sys
 from pathlib import Path
 from typing import Optional
 
@@ -35,6 +36,22 @@ baseline_app = typer.Typer(add_completion=False, no_args_is_help=True)
 app.add_typer(baseline_app, name="baseline")
 
 
+def select_python_launcher() -> str:
+    """Pick a Python command for suites generated by ``runledger init``.
+
+    Prefer a portable name on PATH so the suite still runs on other machines.
+    ``python3`` comes first because Ubuntu, macOS, and pipx installs often have
+    no ``python`` executable. When neither name exists, use the interpreter
+    that is running RunLedger (``sys.executable``).
+    """
+    for name in ("python3", "python"):
+        if shutil.which(name):
+            return name
+    if sys.executable:
+        return sys.executable
+    return "python3"
+
+
 def _display_path(path: Path, *, base_dir: Path) -> str:
     if not path.is_absolute():
         return path.as_posix()
@@ -47,6 +64,57 @@ def _display_path(path: Path, *, base_dir: Path) -> str:
             return path.as_posix()
 
 
+def _posix_rel(path: Path, root: Path) -> str:
+    text = _display_path(path, base_dir=root)
+    if text.startswith(("/", ".")):
+        return text
+    return f"./{text}"
+
+
+def render_init_workflow(*, suite_rel: str, baseline_rel: str) -> str:
+    """GitHub Actions workflow that replays the suite scaffolded by ``runledger init``."""
+    for label, value in (("suite path", suite_rel), ("baseline path", baseline_rel)):
+        if any(char in value for char in '\n\r"'):
+            raise ValueError(f"Unsafe {label} for workflow: {value!r}")
+    lines = [
+        "name: runledger",
+        "on:",
+        "  pull_request:",
+        "",
+        "jobs:",
+        "  evals:",
+        "    runs-on: ubuntu-latest",
+        "    steps:",
+        "      - uses: actions/checkout@v4",
+        "      - uses: actions/setup-python@v5",
+        "        with:",
+        '          python-version: "3.11"',
+        "      - name: Install RunLedger",
+        "        run: |",
+        "          python -m pip install --upgrade pip",
+        "          python -m pip install runledger",
+        "      - name: Run deterministic evals",
+        f'        run: runledger run "{suite_rel}" --mode replay --baseline "{baseline_rel}"',
+        "      - name: Upload artifacts",
+        "        if: always()",
+        "        uses: actions/upload-artifact@v4",
+        "        with:",
+        "          name: runledger-artifacts",
+        "          path: runledger_out/**",
+        "",
+    ]
+    return "\n".join(lines)
+
+
+def _write_unless_present(path: Path, content: str) -> bool:
+    """Write ``content`` when ``path`` is missing. Return True if the file was created."""
+    if path.exists():
+        return False
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(content, encoding="utf-8")
+    return True
+
+
 def _resolve_summary_path(run_path: Path) -> Path:
     if run_path.is_dir():
         summary_path = run_path / "summary.json"
@@ -141,7 +209,11 @@ def init(
     ),
     suite: str = typer.Option("demo", help="Suite name to generate"),
     template: str = typer.Option("support-triage", help="Template name"),
-    force: bool = typer.Option(False, "--force", help="Overwrite existing suite files"),
+    force: bool = typer.Option(
+        False,
+        "--force",
+        help="Overwrite existing init files, including the workflow and baseline",
+    ),
     language: str = typer.Option("python", help="Agent language (python or node)"),
 ) -> None:
     """Initialize an example eval suite."""
@@ -167,18 +239,22 @@ def init(
         agent_command = ["node", "agent/agent.js"]
     else:
         agent_filename = "agent.py"
-        agent_command = ["python", "agent/agent.py"]
+        agent_command = [select_python_launcher(), "agent/agent.py"]
     agent_path = agent_dir / agent_filename
 
-    if evals_dir.exists():
-        if not force:
-            console.print(f"[red]Target already exists:[/red] {evals_dir}")
-            raise typer.Exit(code=1)
-        shutil.rmtree(evals_dir)
-    if baselines_dir.exists():
-        baseline_file = baselines_dir / f"{suite}.json"
-        if baseline_file.exists() and force:
+    project_root = base_dir.parent
+    baseline_file = baselines_dir / f"{suite}.json"
+    workflow_path = project_root / ".github" / "workflows" / "runledger.yml"
+    if evals_dir.exists() and not evals_dir.is_dir():
+        console.print(f"[red]Target exists and is not a directory:[/red] {evals_dir}")
+        raise typer.Exit(code=1)
+    if force:
+        if evals_dir.exists():
+            shutil.rmtree(evals_dir)
+        if baseline_file.exists():
             baseline_file.unlink()
+        if workflow_path.exists():
+            workflow_path.unlink()
 
     agent_dir.mkdir(parents=True, exist_ok=True)
     cases_dir.mkdir(parents=True, exist_ok=True)
@@ -186,7 +262,7 @@ def init(
     baselines_dir.mkdir(parents=True, exist_ok=True)
 
     if language == "node":
-        agent_path.write_text(
+        agent_text = (
             "\n".join(
                 [
                     "const readline = require(\"readline\");",
@@ -237,11 +313,10 @@ def init(
                     ");",
                     "",
                 ]
-            ),
-            encoding="utf-8",
+            )
         )
     else:
-        agent_path.write_text(
+        agent_text = (
             "\n".join(
                 [
                     "import json",
@@ -269,12 +344,10 @@ def init(
                     "    raise SystemExit(main())",
                     "",
                 ]
-            ),
-            encoding="utf-8",
+            )
         )
 
-    schema_path = evals_dir / "schema.json"
-    schema_path.write_text(
+    schema_text = (
         json.dumps(
             {
                 "type": "object",
@@ -286,12 +359,9 @@ def init(
             },
             indent=2,
         )
-        + "\n",
-        encoding="utf-8",
+        + "\n"
     )
-
-    cassette_path = cassettes_dir / "t1.jsonl"
-    cassette_path.write_text(
+    cassette_text = (
         json.dumps(
             {
                 "tool": "search_docs",
@@ -302,115 +372,134 @@ def init(
             ensure_ascii=False,
             sort_keys=True,
         )
-        + "\n",
-        encoding="utf-8",
+        + "\n"
     )
-
-    case_path = cases_dir / "t1.yaml"
-    case_path.write_text(
-        yaml.safe_dump(
-            {
-                "id": "t1",
-                "description": "triage a login ticket",
-                "input": {"ticket": "reset password"},
-                "cassette": "cassettes/t1.jsonl",
-                "assertions": [
-                    {"type": "required_fields", "fields": ["category", "reply"]},
-                ],
+    case_text = yaml.safe_dump(
+        {
+            "id": "t1",
+            "description": "triage a login ticket",
+            "input": {"ticket": "reset password"},
+            "cassette": "cassettes/t1.jsonl",
+            "assertions": [
+                {"type": "required_fields", "fields": ["category", "reply"]},
+                {"type": "call_order", "order": ["search_docs"]},
+            ],
+        },
+        sort_keys=False,
+    )
+    suite_text = yaml.safe_dump(
+        {
+            "suite_name": suite,
+            "agent_command": agent_command,
+            "mode": "replay",
+            "cases_path": "cases",
+            "tool_registry": ["search_docs"],
+            "assertions": [
+                {"type": "json_schema", "schema_path": "schema.json"},
+            ],
+            "budgets": {
+                "max_wall_ms": 20000,
+                "max_tool_calls": 1,
+                "max_tool_errors": 0,
             },
-            sort_keys=False,
-        ),
-        encoding="utf-8",
+            "regression": {
+                "min_pass_rate": 1.0,
+            },
+            "baseline_path": f"../../baselines/{suite}.json",
+        },
+        sort_keys=False,
     )
 
+    schema_path = evals_dir / "schema.json"
+    cassette_path = cassettes_dir / "t1.jsonl"
+    case_path = cases_dir / "t1.yaml"
     suite_path = evals_dir / "suite.yaml"
-    suite_path.write_text(
-        yaml.safe_dump(
-            {
-                "suite_name": suite,
-                "agent_command": agent_command,
-                "mode": "replay",
-                "cases_path": "cases",
-                "tool_registry": ["search_docs"],
-                "assertions": [
-                    {"type": "json_schema", "schema_path": "schema.json"},
-                ],
-                "budgets": {
-                    "max_wall_ms": 20000,
-                    "max_tool_calls": 1,
-                    "max_tool_errors": 0,
-                },
-                "regression": {
-                    "min_pass_rate": 1.0,
-                },
-                "baseline_path": f"../../baselines/{suite}.json",
-            },
-            sort_keys=False,
-        ),
-        encoding="utf-8",
-    )
+    created: list[Path] = []
+    unchanged: list[Path] = []
+    for target, content in (
+        (agent_path, agent_text),
+        (schema_path, schema_text),
+        (cassette_path, cassette_text),
+        (case_path, case_text),
+        (suite_path, suite_text),
+    ):
+        if _write_unless_present(target, content):
+            created.append(target)
+        else:
+            unchanged.append(target)
 
-    try:
-        suite_config = load_suite(suite_path)
-        cases = load_cases(evals_dir, suite_config.cases_path)
-        suite_run = suite_config.model_copy(
-            update={"agent_command": [agent_command[0], str(agent_path)]}
-        )
-        suite_result = run_suite(suite_run, cases)
-        run_id = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
-        summary = build_summary(
-            suite=suite_config,
-            suite_path=suite_path,
-            suite_result=suite_result,
-            run_id=run_id,
-            generated_at=datetime.now(timezone.utc),
-        )
-        baseline = BaselineSummary.model_validate(summary)
-        write_baseline(baselines_dir / f"{suite}.json", baseline)
-    except Exception as exc:
-        console.print(f"[red]Failed to generate baseline:[/red] {exc}")
-        raise typer.Exit(code=1)
+    if baseline_file.exists():
+        unchanged.append(baseline_file)
+    else:
+        try:
+            suite_config = load_suite(suite_path)
+            cases = load_cases(evals_dir, suite_config.cases_path)
+            if suite_path in created:
+                suite_run = suite_config.model_copy(
+                    update={"agent_command": [agent_command[0], str(agent_path)]}
+                )
+            else:
+                suite_run = suite_config
+            suite_result = run_suite(suite_run, cases)
+        except Exception as exc:
+            console.print(f"[red]Failed to generate baseline:[/red] {exc}")
+            raise typer.Exit(code=1)
+
+        if not suite_result.passed:
+            console.print("[red]Failed to generate baseline.[/red] The example agent did not pass.")
+            for line in format_failure_lines(collect_case_failures(suite_result.cases), []):
+                console.print(line, markup=False, highlight=False, soft_wrap=True)
+            raise typer.Exit(code=1)
+
+        try:
+            run_id = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+            summary = build_summary(
+                suite=suite_config,
+                suite_path=suite_path,
+                suite_result=suite_result,
+                run_id=run_id,
+                generated_at=datetime.now(timezone.utc),
+            )
+            baseline = BaselineSummary.model_validate(summary)
+            write_baseline(baseline_file, baseline)
+        except Exception as exc:
+            console.print(f"[red]Failed to generate baseline:[/red] {exc}")
+            raise typer.Exit(code=1)
+        created.append(baseline_file)
+
+    if workflow_path.exists():
+        unchanged.append(workflow_path)
+    else:
+        suite_rel = _posix_rel(evals_dir, project_root)
+        baseline_rel = _posix_rel(baseline_file, project_root)
+        try:
+            workflow_text = render_init_workflow(suite_rel=suite_rel, baseline_rel=baseline_rel)
+        except ValueError as exc:
+            console.print(f"[red]{exc}[/red]")
+            raise typer.Exit(code=1)
+        _write_unless_present(workflow_path, workflow_text)
+        created.append(workflow_path)
 
     cwd = Path.cwd()
-    evals_dir_display = _display_path(evals_dir, base_dir=cwd)
-    baseline_display = _display_path(baselines_dir / f"{suite}.json", base_dir=cwd)
 
-    console.print(f"Created eval suite at: {evals_dir_display}")
-    console.print(f"Created baseline at: {baseline_display}")
+    def _show(paths: list[Path]) -> None:
+        for item in paths:
+            console.print(f"  {_display_path(item, base_dir=cwd)}")
+
+    if created:
+        console.print("Created:")
+        _show(created)
+    if unchanged:
+        console.print("Unchanged:")
+        _show(unchanged)
     console.print("")
     console.print("Next steps:")
     console.print(
-        f'  runledger run "{evals_dir_display}" --mode replay --baseline "{baseline_display}"'
+        "  runledger run "
+        f'"{_posix_rel(evals_dir, cwd)}" --mode replay --baseline "{_posix_rel(baseline_file, cwd)}"'
     )
     console.print(f"  open runledger_out/{suite}/<run_id>/report.html")
-    console.print("")
-    console.print("GitHub Actions snippet:")
-    console.print(
-        "\n".join(
-            [
-                "name: runledger-evals",
-                "on:",
-                "  pull_request:",
-                "",
-                "jobs:",
-                "  evals:",
-                "    runs-on: ubuntu-latest",
-                "    steps:",
-                "      - uses: actions/checkout@v4",
-                "      - name: Install RunLedger",
-                "        run: |",
-                "          python -m pip install --upgrade pip",
-                "          python -m pip install runledger",
-                "      - name: Run deterministic evals",
-                f'        run: runledger run "{evals_dir_display}" --mode replay --baseline "{baseline_display}"',
-                "      - name: Upload artifacts",
-                "        uses: actions/upload-artifact@v4",
-                "        with:",
-                "          name: runledger-artifacts",
-                "          path: runledger_out/**",
-            ]
-        )
-    )
+    console.print(f"  commit {_display_path(workflow_path, base_dir=cwd)}")
 
 
 @app.command()

{runledger-0.2.1 → runledger-0.2.2}/src/runledger/runner/subprocess.py

@@ -25,6 +25,57 @@ class AgentProcessError(Exception):
         return f"{self.message}\nAgent stderr (tail):\n{tail}"
 
 
+def _command_basename(program: str) -> str:
+    return program.replace("\\", "/").rsplit("/", 1)[-1].lower()
+
+
+def _agent_start_error(command: Sequence[str], exc: BaseException) -> AgentProcessError:
+    """Turn a failed exec into an actionable error for suite.yaml authors."""
+    program = command[0] if command else ""
+    rendered = " ".join(command) if command else "<empty>"
+    example = '["python3", "agent/agent.py"]'
+    if isinstance(exc, FileNotFoundError):
+        if _command_basename(program) in {"python", "python.exe"}:
+            message = (
+                f"Agent command not found: {program!r}. "
+                "There is no 'python' executable on PATH. "
+                "Ubuntu, macOS, and many pipx installs only provide python3. "
+                f"Set agent_command in suite.yaml to a Python that exists, for example {example}, "
+                "or re-run `runledger init --force` to regenerate the example suite."
+            )
+        elif program:
+            message = (
+                f"Agent command not found: {program!r}. "
+                "RunLedger could not start the agent because that program is not on PATH "
+                "and is not an existing file. "
+                "Install it, or set agent_command in suite.yaml to the command that starts "
+                f"your agent (for example {example})."
+            )
+        else:
+            message = (
+                "Agent command is empty. "
+                "Set agent_command in suite.yaml to the program that starts your agent, "
+                f"for example {example}."
+            )
+    elif isinstance(exc, PermissionError):
+        message = (
+            f"Agent command is not executable: {program!r}. {exc}. "
+            "Grant execute permission, or set agent_command in suite.yaml to a program you can run."
+        )
+    elif isinstance(exc, ValueError):
+        message = (
+            "Agent command is empty or invalid. "
+            "Set agent_command in suite.yaml to the program that starts your agent, "
+            f"for example {example}."
+        )
+    else:
+        message = (
+            f"Failed to start agent command {rendered!r}: {exc}. "
+            "Check that agent_command in suite.yaml points at an installed, runnable program."
+        )
+    return AgentProcessError(message, [])
+
+
 class AgentProcess:
     def __init__(self, command: Sequence[str], timeout_s: float = 30.0, stderr_tail: int = 200):
         self._command = list(command)
@@ -47,14 +98,19 @@ class AgentProcess:
     def start(self) -> None:
         if self._process is not None:
             return
-        self._process = subprocess.Popen(
-            self._command,
-            stdin=subprocess.PIPE,
-            stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE,
-            text=True,
-            bufsize=1,
-        )
+        if not self._command or not self._command[0].strip():
+            raise _agent_start_error(self._command, ValueError("empty command"))
+        try:
+            self._process = subprocess.Popen(
+                self._command,
+                stdin=subprocess.PIPE,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                text=True,
+                bufsize=1,
+            )
+        except OSError as exc:
+            raise _agent_start_error(self._command, exc) from exc
         if self._process.stdout is None or self._process.stdin is None or self._process.stderr is None:
             raise AgentProcessError("Failed to open subprocess pipes", [])
         # Background threads prevent blocking reads from stalling the main loop.

runledger-0.2.1/README.md → runledger-0.2.2/src/runledger.egg-info/PKG-INFO

@@ -1,3 +1,31 @@
+Metadata-Version: 2.4
+Name: runledger
+Version: 0.2.2
+Summary: CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates.
+Author: RunLedger
+License-Expression: MIT
+Project-URL: Homepage, https://runledger.io
+Project-URL: Repository, https://github.com/runledger/Runledger
+Project-URL: Documentation, https://github.com/runledger/Runledger/tree/main/docs
+Project-URL: Changelog, https://github.com/runledger/Runledger/blob/main/CHANGELOG.md
+Keywords: llm,agents,evaluation,ci,github-actions,replay,testing,junit,jsonschema
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer>=0.9.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0.0
+Requires-Dist: jsonschema>=4.0.0
+Requires-Dist: jinja2>=3.1.0
+Requires-Dist: eval_type_backport>=0.2.0; python_version < "3.10"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-timeout>=2.1.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Dynamic: license-file
+
 # RunLedger
 
 [![CI](https://github.com/runledger/Runledger/actions/workflows/ci.yml/badge.svg)](https://github.com/runledger/Runledger/actions/workflows/ci.yml)
@@ -6,25 +34,57 @@
 
 Website: https://runledger.io
 
-**Deterministic CI regression checks for tool-using agents.** Record tool calls once, replay in CI, and fail PRs on mismatches, contract violations, or budget regressions.
+**Deterministic CI regression checks for tool-using agents.** Record tool calls once, replay them in CI, and fail the run when a schema, tool-order, or tool-call budget gate fails.
 
-- Demo repo: https://github.com/runledger/runledger-demo (main is green; regression branches fail for real reasons)
+- Demo: https://github.com/ZackMitchell910/runledger-demo/pull/1 (intentionally left open to show RunLedger failing a bad agent change)
 - If this is useful, please star the repo (it helps a lot).
 
-RunLedger is a **CI harness**, not an "eval metrics framework":
-- **DeepEval-style tools** help you *score* behavior.
-- **RunLedger** helps you *ship safely* by making agent tests deterministic and merge-gated.
+## Five-minute quickstart
 
----
-
-## Try it now (60 seconds)
+From an empty directory, with Python 3.9+:
 
 ```bash
-pipx install runledger
+python3 -m pip install runledger
 runledger init
 runledger run ./evals/demo --mode replay --baseline baselines/demo.json
 ```
 
+`runledger init` writes:
+
+- `evals/demo/` — replay suite, cassette, JSON Schema, and a small agent that calls `search_docs`
+- `baselines/demo.json` — baseline from that passing run
+- `.github/workflows/runledger.yml` — pull-request workflow that runs the same command
+
+A second `runledger init` leaves those files unchanged. `--force` regenerates them.
+
+The demo exits non-zero unless all three of these hold:
+
+- **Schema.** The final output matches `evals/demo/schema.json` (`category` and `reply`).
+- **Tool order.** The agent calls `search_docs` (`call_order` in `evals/demo/cases/t1.yaml`).
+- **Tool-call budget.** The run stays within `max_tool_calls: 1`.
+
+A passing run prints `PASS` and writes `runledger_out/demo/<run_id>/` (`summary.json`, `junit.xml`, `report.html`, `run.jsonl`). Commit the init files to run that gate on pull requests.
+
+The same check as a GitHub Action. `suite.yaml` already points at `baselines/demo.json`, so the action compares against that baseline:
+
+```yaml
+- id: evals
+  uses: runledger/Runledger@v0.2
+  with:
+    path: ./evals/demo
+    mode: replay
+```
+
+This repo's `action.yml` also accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. Those fields ship when a release containing this file is tagged. The in-repo self-test calls the action with `uses: ./`.
+
+Page copy for runledger.io lives in `docs/site-quickstart.md`.
+
+---
+
+RunLedger is a **CI harness**, not an "eval metrics framework":
+- **DeepEval-style tools** help you *score* behavior.
+- **RunLedger** helps you *ship safely* by making agent tests deterministic and merge-gated.
+
 ## Integrations (merged)
 
 - hoangsonww/Agentic-AI-Pipeline - https://github.com/hoangsonww/Agentic-AI-Pipeline/pull/24
@@ -49,7 +109,7 @@ RunLedger stops regressions by shifting from "vibes-based" evaluation to determi
 
 - **Record & replay:** record tool outputs once; replay them in CI for instant, stable tests.
 - **Strict contracts:** enforce tool schemas, calling order, and allowlists.
-- **Budget gating:** fail PRs automatically if wall time, tool-call count, or tool errors exceed defined limits.
+- **Budget gating:** fail the run when tool-call count exceeds `max_tool_calls`. Wall time and tool-error caps are separate gates, configured with `max_wall_ms` and `max_tool_errors`.
 
 ### RunLedger vs evaluation frameworks
 
@@ -328,18 +388,18 @@ jobs:
       - uses: actions/checkout@v4
 
       - name: Run deterministic evals (replay)
+        id: evals
         uses: runledger/Runledger@v0.2
         with:
           path: ./evals/demo
           mode: replay
 
-      - name: Upload eval artifacts
-        uses: actions/upload-artifact@v4
-        with:
-          name: agent-eval-artifacts
-          path: runledger_out/**
+      - name: Show the action result
+        run: echo "result=${{ steps.evals.outputs.result }} exit=${{ steps.evals.outputs.exit-code }}"
 ```
 
+`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml`. It uploads `runledger_out/**` itself. On a release that contains this `action.yml`, the step also sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`.
+
 ---
 
 ## Determinism guide (rules of the road)
@@ -362,6 +422,7 @@ jobs:
 
 ## Docs
 
+- `docs/site-quickstart.md` -- five-minute copy for runledger.io
 - `docs/quickstart.md` -- install + first run
 - `docs/assertions.md` -- assertions, tool contracts, budgets
 - `docs/baselines.md` -- baselines, diffing, promotion
@@ -374,7 +435,6 @@ jobs:
 ## Roadmap
 
 * `init` templates for Python (OpenAI SDK), LangGraph/LangChain, and Node/TS
-* richer budgets (tokens/cost) via optional `task_metrics`
 * PR comments bot for regression summaries
 * plugin system for custom assertions
 * HTML report trace viewer improvements