runledger 0.2.2__tar.gz → 0.2.3__tar.gz

{runledger-0.2.2/src/runledger.egg-info → runledger-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: runledger
-Version: 0.2.2
+Version: 0.2.3
 Summary: CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates.
 Author: RunLedger
 License-Expression: MIT
@@ -69,13 +69,13 @@ The same check as a GitHub Action. `suite.yaml` already points at `baselines/dem
 
 ```yaml
 - id: evals
-  uses: runledger/Runledger@v0.2
+  uses: runledger/Runledger@v0.2.3
   with:
     path: ./evals/demo
     mode: replay
 ```
 
-This repo's `action.yml` also accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. Those fields ship when a release containing this file is tagged. The in-repo self-test calls the action with `uses: ./`.
+`runledger/Runledger@v0.2.3` accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. The in-repo self-test calls the action with `uses: ./`.
 
 Page copy for runledger.io lives in `docs/site-quickstart.md`.
 
@@ -229,7 +229,7 @@ Optional:
 
 ```yaml
 suite_name: support-triage
-agent_command: ["python", "agent.py"]
+agent_command: ["python3", "agent.py"]
 mode: replay            # replay | record | live
 cases_path: cases
 tool_registry:
@@ -306,13 +306,10 @@ Each line is one tool invocation:
 MVP assertions:
 
 * `json_schema` (validate final output with JSON Schema)
-* `required_fields` (keys exist / basic typing)
-* `regex` / `contains` (for specific fields)
-* `tool_contract`:
-
-  * must call tool X
-  * must not call tool Y
-  * X before Y (ordering)
+* `required_fields` (keys exist)
+* `must_call` (must call tool X)
+* `must_not_call` (must not call tool Y)
+* `call_order` (X before Y; tools do not have to be adjacent)
 
 Not default-gating (optional later):
 
@@ -323,18 +320,12 @@ Not default-gating (optional later):
 
 ## Budgets (merge gates)
 
-MVP budgets:
+Budgets that fail a run:
 
 * `max_wall_ms`
 * `max_tool_calls`
 * `max_tool_errors`
 
-Not yet enforced (accepted in config, reported in artifacts when the agent sends metrics, but
-they do not fail a run today):
-
-* `max_tokens_out`
-* `max_cost_usd`
-
 ---
 
 ## Baselines + regression checks
@@ -389,7 +380,7 @@ jobs:
 
       - name: Run deterministic evals (replay)
         id: evals
-        uses: runledger/Runledger@v0.2
+        uses: runledger/Runledger@v0.2.3
         with:
           path: ./evals/demo
           mode: replay
@@ -398,7 +389,7 @@ jobs:
         run: echo "result=${{ steps.evals.outputs.result }} exit=${{ steps.evals.outputs.exit-code }}"
 ```
 
-`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml`. It uploads `runledger_out/**` itself. On a release that contains this `action.yml`, the step also sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`.
+`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml` at `runledger/Runledger@v0.2.3`. It uploads `runledger_out/**` on success and on failure. The step sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`. A failing run fails the job.
 
 ---
 

{runledger-0.2.2 → runledger-0.2.3}/README.md

@@ -41,13 +41,13 @@ The same check as a GitHub Action. `suite.yaml` already points at `baselines/dem
 
 ```yaml
 - id: evals
-  uses: runledger/Runledger@v0.2
+  uses: runledger/Runledger@v0.2.3
   with:
     path: ./evals/demo
     mode: replay
 ```
 
-This repo's `action.yml` also accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. Those fields ship when a release containing this file is tagged. The in-repo self-test calls the action with `uses: ./`.
+`runledger/Runledger@v0.2.3` accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. The in-repo self-test calls the action with `uses: ./`.
 
 Page copy for runledger.io lives in `docs/site-quickstart.md`.
 
@@ -201,7 +201,7 @@ Optional:
 
 ```yaml
 suite_name: support-triage
-agent_command: ["python", "agent.py"]
+agent_command: ["python3", "agent.py"]
 mode: replay            # replay | record | live
 cases_path: cases
 tool_registry:
@@ -278,13 +278,10 @@ Each line is one tool invocation:
 MVP assertions:
 
 * `json_schema` (validate final output with JSON Schema)
-* `required_fields` (keys exist / basic typing)
-* `regex` / `contains` (for specific fields)
-* `tool_contract`:
-
-  * must call tool X
-  * must not call tool Y
-  * X before Y (ordering)
+* `required_fields` (keys exist)
+* `must_call` (must call tool X)
+* `must_not_call` (must not call tool Y)
+* `call_order` (X before Y; tools do not have to be adjacent)
 
 Not default-gating (optional later):
 
@@ -295,18 +292,12 @@ Not default-gating (optional later):
 
 ## Budgets (merge gates)
 
-MVP budgets:
+Budgets that fail a run:
 
 * `max_wall_ms`
 * `max_tool_calls`
 * `max_tool_errors`
 
-Not yet enforced (accepted in config, reported in artifacts when the agent sends metrics, but
-they do not fail a run today):
-
-* `max_tokens_out`
-* `max_cost_usd`
-
 ---
 
 ## Baselines + regression checks
@@ -361,7 +352,7 @@ jobs:
 
       - name: Run deterministic evals (replay)
         id: evals
-        uses: runledger/Runledger@v0.2
+        uses: runledger/Runledger@v0.2.3
         with:
           path: ./evals/demo
           mode: replay
@@ -370,7 +361,7 @@ jobs:
         run: echo "result=${{ steps.evals.outputs.result }} exit=${{ steps.evals.outputs.exit-code }}"
 ```
 
-`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml`. It uploads `runledger_out/**` itself. On a release that contains this `action.yml`, the step also sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`.
+`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml` at `runledger/Runledger@v0.2.3`. It uploads `runledger_out/**` on success and on failure. The step sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`. A failing run fails the job.
 
 ---
 

{runledger-0.2.2 → runledger-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "runledger"
-version = "0.2.2"
+version = "0.2.3"
 description = "CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates."
 readme = "README.md"
 requires-python = ">=3.9"

{runledger-0.2.2 → runledger-0.2.3}/src/runledger/__init__.py

@@ -1,3 +1,3 @@
 __all__ = ["__version__"]
 
-__version__ = "0.2.2"
+__version__ = "0.2.3"

{runledger-0.2.2 → runledger-0.2.3}/src/runledger/cli.py

@@ -498,7 +498,7 @@ def init(
         "  runledger run "
         f'"{_posix_rel(evals_dir, cwd)}" --mode replay --baseline "{_posix_rel(baseline_file, cwd)}"'
     )
-    console.print(f"  open runledger_out/{suite}/<run_id>/report.html")
+    console.print(f"  report: runledger_out/{suite}/<run_id>/report.html")
     console.print(f"  commit {_display_path(workflow_path, base_dir=cwd)}")
 
 

{runledger-0.2.2 → runledger-0.2.3}/src/runledger/config/loader.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import re
 from pathlib import Path
 from typing import Any
 
@@ -7,12 +8,33 @@ import yaml
 
 from .models import CaseConfig, SuiteConfig
 
+_WINDOWS_DRIVE = re.compile(r"^[A-Za-z]:[\\/]")
+
+
+def normalize_config_path(value: str) -> str:
+    """Turn Windows separators into POSIX separators.
+
+    Suites authored on Windows often store ``agent\\agent.py`` and
+    ``cassettes\\t1.jsonl``. Those strings do not resolve on the Linux runners
+    used by GitHub Actions. Forward slashes work on both.
+    """
+    if "\\" not in value:
+        return value
+    return value.replace("\\", "/")
+
+
+def _is_absolute_config_path(value: str) -> bool:
+    if Path(value).is_absolute():
+        return True
+    return bool(_WINDOWS_DRIVE.match(value)) or value.startswith("//")
+
 
 def _load_yaml(path: Path) -> dict[str, Any]:
     try:
         data = yaml.safe_load(path.read_text(encoding="utf-8"))
     except yaml.YAMLError as exc:
-        raise ValueError(f"Invalid YAML in {path}") from exc
+        detail = str(exc).strip() or "could not parse the file"
+        raise ValueError(f"Invalid YAML in {path}: {detail}") from exc
     except OSError as exc:
         raise FileNotFoundError(f"Unable to read {path}") from exc
 
@@ -28,8 +50,11 @@ def _resolve_schema_paths(assertions: list[object], base_dir: Path) -> None:
         if entry.get("type") != "json_schema":
             continue
         schema_path = entry.get("schema_path")
-        if isinstance(schema_path, str) and not Path(schema_path).is_absolute():
-            entry["schema_path"] = str((base_dir / schema_path).resolve())
+        if isinstance(schema_path, str):
+            schema_path = normalize_config_path(schema_path)
+            if not _is_absolute_config_path(schema_path):
+                schema_path = str((base_dir / schema_path).resolve())
+            entry["schema_path"] = schema_path
 
 
 def load_suite(path: Path) -> SuiteConfig:
@@ -39,31 +64,43 @@ def load_suite(path: Path) -> SuiteConfig:
     if not suite_path.is_file():
         raise FileNotFoundError(f"Suite file not found: {suite_path}")
     data = _load_yaml(suite_path)
+    cases_path = data.get("cases_path")
+    if isinstance(cases_path, str):
+        data["cases_path"] = normalize_config_path(cases_path)
     agent_command = data.get("agent_command")
     if isinstance(agent_command, list):
         resolved_command: list[object] = []
         for part in agent_command:
-            if isinstance(part, str) and not Path(part).is_absolute():
-                candidate = (suite_path.parent / part).resolve()
-                if candidate.exists():
-                    resolved_command.append(str(candidate))
-                    continue
+            if isinstance(part, str):
+                part = normalize_config_path(part)
+                if not _is_absolute_config_path(part):
+                    candidate = (suite_path.parent / part).resolve()
+                    if candidate.exists():
+                        resolved_command.append(str(candidate))
+                        continue
             resolved_command.append(part)
         data["agent_command"] = resolved_command
     assertions = data.get("assertions", [])
     if isinstance(assertions, list):
         _resolve_schema_paths(assertions, suite_path.parent)
     baseline_path = data.get("baseline_path")
-    if isinstance(baseline_path, str) and not Path(baseline_path).is_absolute():
-        data["baseline_path"] = str((suite_path.parent / baseline_path).resolve())
+    if isinstance(baseline_path, str):
+        baseline_path = normalize_config_path(baseline_path)
+        if not _is_absolute_config_path(baseline_path):
+            baseline_path = str((suite_path.parent / baseline_path).resolve())
+        data["baseline_path"] = baseline_path
     output_dir = data.get("output_dir")
-    if isinstance(output_dir, str) and not Path(output_dir).is_absolute():
-        data["output_dir"] = str((suite_path.parent / output_dir).resolve())
+    if isinstance(output_dir, str):
+        output_dir = normalize_config_path(output_dir)
+        if not _is_absolute_config_path(output_dir):
+            output_dir = str((suite_path.parent / output_dir).resolve())
+        data["output_dir"] = output_dir
     return SuiteConfig.model_validate(data)
 
 
 def load_cases(suite_dir: Path, cases_path: str) -> list[CaseConfig]:
     """Load cases with cassette paths resolved relative to the suite directory."""
+    cases_path = normalize_config_path(cases_path)
     cases_dir = (suite_dir / cases_path).resolve()
     if not cases_dir.is_dir():
         raise FileNotFoundError(f"Cases directory not found: {cases_dir}")
@@ -76,8 +113,11 @@ def load_cases(suite_dir: Path, cases_path: str) -> list[CaseConfig]:
     for case_file in case_files:
         data = _load_yaml(case_file)
         cassette_value = data.get("cassette")
-        if isinstance(cassette_value, str) and not Path(cassette_value).is_absolute():
-            data["cassette"] = str((suite_dir / cassette_value).resolve())
+        if isinstance(cassette_value, str):
+            cassette_value = normalize_config_path(cassette_value)
+            if not _is_absolute_config_path(cassette_value):
+                cassette_value = str((suite_dir / cassette_value).resolve())
+            data["cassette"] = cassette_value
         assertions = data.get("assertions")
         if isinstance(assertions, list):
             _resolve_schema_paths(assertions, suite_dir)

{runledger-0.2.2 → runledger-0.2.3}/src/runledger/reporting/failures.py

@@ -63,7 +63,7 @@ def _expected_observed(assertion_type: str, details: Mapping[str, Any] | None) -
     if assertion_type == "required_fields":
         missing = details.get("missing")
         return (
-            f"fields present: {_join_names(missing)}",
+            f"required fields: {_join_names(missing)}",
             f"missing: {_join_names(missing)}",
         )
     if assertion_type == "json_schema":

{runledger-0.2.2 → runledger-0.2.3}/src/runledger/runner/subprocess.py

@@ -4,6 +4,7 @@ from collections import deque
 from dataclasses import dataclass
 import json
 import queue
+import shutil
 import subprocess
 import threading
 import time
@@ -29,6 +30,29 @@ def _command_basename(program: str) -> str:
     return program.replace("\\", "/").rsplit("/", 1)[-1].lower()
 
 
+def _prefer_available_python(command: Sequence[str]) -> list[str]:
+    """Run ``python3`` when a suite asks for bare ``python`` and only python3 exists.
+
+    Ubuntu, macOS, and pipx installs often have no ``python`` executable. Suites
+    generated before that was the default, and the checked-in examples, still say
+    ``python``. Use ``python3`` instead of failing those runs. An explicit path
+    such as ``/usr/bin/python`` is left unchanged.
+    """
+    if not command:
+        return list(command)
+    program = command[0]
+    normalized = program.replace("\\", "/")
+    if "/" in normalized:
+        return list(command)
+    if _command_basename(program) not in {"python", "python.exe"}:
+        return list(command)
+    if shutil.which(program):
+        return list(command)
+    if shutil.which("python3"):
+        return ["python3", *list(command[1:])]
+    return list(command)
+
+
 def _agent_start_error(command: Sequence[str], exc: BaseException) -> AgentProcessError:
     """Turn a failed exec into an actionable error for suite.yaml authors."""
     program = command[0] if command else ""
@@ -100,6 +124,7 @@ class AgentProcess:
             return
         if not self._command or not self._command[0].strip():
             raise _agent_start_error(self._command, ValueError("empty command"))
+        self._command = _prefer_available_python(self._command)
         try:
             self._process = subprocess.Popen(
                 self._command,

{runledger-0.2.2 → runledger-0.2.3}/src/runledger/util/redaction.py

@@ -21,11 +21,11 @@ _SENSITIVE_PARTS = {
 
 _PATTERNS: list[tuple[re.Pattern[str], str]] = [
     (re.compile(r"(?i)bearer\s+[a-z0-9._~+/=-]+"), "Bearer [REDACTED]"),
-    (re.compile(r"\\bsk-[A-Za-z0-9]{20,}\\b"), "[REDACTED]"),
-    (re.compile(r"\\bghp_[A-Za-z0-9]{36}\\b"), "[REDACTED]"),
-    (re.compile(r"\\bAKIA[0-9A-Z]{16}\\b"), "[REDACTED]"),
+    (re.compile(r"\bsk-[A-Za-z0-9]{20,}\b"), "[REDACTED]"),
+    (re.compile(r"\bghp_[A-Za-z0-9]{36}\b"), "[REDACTED]"),
+    (re.compile(r"\bAKIA[0-9A-Z]{16}\b"), "[REDACTED]"),
     (
-        re.compile(r"\\beyJ[a-zA-Z0-9_-]+\\.[a-zA-Z0-9_-]+\\.[a-zA-Z0-9_-]+\\b"),
+        re.compile(r"\beyJ[a-zA-Z0-9_-]{8,}\.[a-zA-Z0-9_-]{8,}\.[a-zA-Z0-9_-]{8,}\b"),
         "[REDACTED]",
     ),
 ]

{runledger-0.2.2 → runledger-0.2.3/src/runledger.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: runledger
-Version: 0.2.2
+Version: 0.2.3
 Summary: CI for tool-using agents: deterministic eval suites with record/replay, hard assertions, budgets, and PR regression gates.
 Author: RunLedger
 License-Expression: MIT
@@ -69,13 +69,13 @@ The same check as a GitHub Action. `suite.yaml` already points at `baselines/dem
 
 ```yaml
 - id: evals
-  uses: runledger/Runledger@v0.2
+  uses: runledger/Runledger@v0.2.3
   with:
     path: ./evals/demo
     mode: replay
 ```
 
-This repo's `action.yml` also accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. Those fields ship when a release containing this file is tagged. The in-repo self-test calls the action with `uses: ./`.
+`runledger/Runledger@v0.2.3` accepts a `baseline` input and sets `result` (`pass` or `fail`), `exit-code`, and `artifacts-path`. The in-repo self-test calls the action with `uses: ./`.
 
 Page copy for runledger.io lives in `docs/site-quickstart.md`.
 
@@ -229,7 +229,7 @@ Optional:
 
 ```yaml
 suite_name: support-triage
-agent_command: ["python", "agent.py"]
+agent_command: ["python3", "agent.py"]
 mode: replay            # replay | record | live
 cases_path: cases
 tool_registry:
@@ -306,13 +306,10 @@ Each line is one tool invocation:
 MVP assertions:
 
 * `json_schema` (validate final output with JSON Schema)
-* `required_fields` (keys exist / basic typing)
-* `regex` / `contains` (for specific fields)
-* `tool_contract`:
-
-  * must call tool X
-  * must not call tool Y
-  * X before Y (ordering)
+* `required_fields` (keys exist)
+* `must_call` (must call tool X)
+* `must_not_call` (must not call tool Y)
+* `call_order` (X before Y; tools do not have to be adjacent)
 
 Not default-gating (optional later):
 
@@ -323,18 +320,12 @@ Not default-gating (optional later):
 
 ## Budgets (merge gates)
 
-MVP budgets:
+Budgets that fail a run:
 
 * `max_wall_ms`
 * `max_tool_calls`
 * `max_tool_errors`
 
-Not yet enforced (accepted in config, reported in artifacts when the agent sends metrics, but
-they do not fail a run today):
-
-* `max_tokens_out`
-* `max_cost_usd`
-
 ---
 
 ## Baselines + regression checks
@@ -389,7 +380,7 @@ jobs:
 
       - name: Run deterministic evals (replay)
         id: evals
-        uses: runledger/Runledger@v0.2
+        uses: runledger/Runledger@v0.2.3
         with:
           path: ./evals/demo
           mode: replay
@@ -398,7 +389,7 @@ jobs:
         run: echo "result=${{ steps.evals.outputs.result }} exit=${{ steps.evals.outputs.exit-code }}"
 ```
 
-`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml`. It uploads `runledger_out/**` itself. On a release that contains this `action.yml`, the step also sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`.
+`runledger init` writes `.github/workflows/runledger.yml`, which installs the CLI and runs the suite with `--baseline`. The action above is the Marketplace entry in `action.yml` at `runledger/Runledger@v0.2.3`. It uploads `runledger_out/**` on success and on failure. The step sets `result`, `exit-code`, and `artifacts-path`, and you can pass `baseline:` to override `suite.yaml`. A failing run fails the job.
 
 ---