agent-eval-contract 0.2.0__tar.gz

agent_eval_contract-0.2.0/CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+
+## 0.2.0 - 2026-07-04
+
+- Reworked the package into a public Pydantic contract library for agent evaluation records.
+- Added typed models, runtime validators, JSON Schema export, CLI subcommands, public examples, and fixture bundles.
+- Removed private workflow vocabulary from the public core.
+
+## 0.1.0 - 2026-06-27
+
+- Initial internal package extraction.

agent_eval_contract-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jakye Amos
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_eval_contract-0.2.0/MANIFEST.in

@@ -0,0 +1,7 @@
+include LICENSE
+include README.md
+include RELEASE.md
+include CHANGELOG.md
+recursive-include docs *.md
+recursive-include examples *.json
+recursive-include tests *.py

agent_eval_contract-0.2.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: agent-eval-contract
+Version: 0.2.0
+Summary: Pydantic contracts and JSON Schemas for portable agent evaluation records.
+Author: Jakye Amos
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/jakyeamos/agent-eval-contract
+Project-URL: Repository, https://github.com/jakyeamos/agent-eval-contract
+Project-URL: Issues, https://github.com/jakyeamos/agent-eval-contract/issues
+Keywords: agents,ai-evaluation,evals,harness,json-schema,pydantic
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic<3,>=2
+Dynamic: license-file
+
+# Agent Eval Contract
+
+Pydantic contracts and JSON Schemas for portable agent evaluation records.
+
+Use this package when you are experimenting with agents, harnesses, CI checks, or benchmark runners and need a stable record shape for tasks, runs, scores, failures, and normalized external results. It does not run evaluations, call model providers, store dashboards, or orchestrate agents. It gives those tools a shared contract.
+
+## Install
+
+```bash
+pip install agent-eval-contract
+```
+
+For local development from this repo:
+
+```bash
+uv sync --dev
+```
+
+## Validate A Record
+
+```python
+from agent_eval_contract import validate_eval_run
+
+run = validate_eval_run(
+    {
+        "run_id": "run-login-flow-001",
+        "task_id": "task-login-flow-001",
+        "harness": "pytest",
+        "model": "gpt-5",
+        "mode": "autonomous",
+        "context_profile": "repo_only",
+        "final_status": "success",
+        "checks": ["pytest tests/test_auth_redirect.py -q"],
+    }
+)
+
+print(run.model_dump(mode="json"))
+```
+
+Validation returns typed Pydantic model instances. Invalid records raise `pydantic.ValidationError` with structured field errors.
+
+## CLI
+
+```bash
+agent-eval-contract validate --kind run --file examples/eval_run.json
+agent-eval-contract schemas --output-dir /tmp/agent-eval-contract-schemas
+agent-eval-contract fixtures --output-dir /tmp/agent-eval-contract-fixtures
+agent-eval-contract normalize --harness terminal-bench --file examples/terminal_bench_result.json --task-id task-login-flow-001 --model gpt-5
+agent-eval-contract normalize --harness swe-bench --file examples/swe_bench_result.json
+```
+
+The legacy `agent-eval-contract-fixtures` command still writes fixture bundles for one release.
+
+## What It Provides
+
+- Pydantic models for eval tasks, runs, scores, failures, external results, normalized runs, and fixture manifests
+- runtime validators that return typed model instances
+- JSON Schema export for all public models
+- bundled sample records and markdown templates
+- Terminal-Bench and SWE-bench oriented normalization helpers
+- a small CLI for validation, schema export, fixture generation, and normalization
+
+## Contract Vocabulary
+
+The public core uses generic vocabulary only. Project-specific concepts should live in `metadata` or a separate adapter package.
+
+- `context_profile`: `repo_only`, `provided_context`, `clean_room`, `tool_augmented`, `full_workspace`
+- `source`: `manual`, `ci`, `benchmark`, `production_trace`, `synthetic`
+- `mode`: `interactive`, `autonomous`, `shadow`, `replay`, `benchmark`
+- `final_status`: `success`, `partial`, `failed`, `abandoned`, `error`
+
+See [docs/contract.md](docs/contract.md), [docs/field-reference.md](docs/field-reference.md), and [docs/adapters.md](docs/adapters.md) for the model contract and adapter guidance.
+
+## Development
+
+```bash
+uv run ruff check agent_eval_contract tests
+uv run ruff format --check agent_eval_contract tests
+uv run basedpyright agent_eval_contract tests
+uv run pytest -q
+uv build --out-dir /tmp/agent-eval-contract-dist
+```

agent_eval_contract-0.2.0/README.md

@@ -0,0 +1,82 @@
+# Agent Eval Contract
+
+Pydantic contracts and JSON Schemas for portable agent evaluation records.
+
+Use this package when you are experimenting with agents, harnesses, CI checks, or benchmark runners and need a stable record shape for tasks, runs, scores, failures, and normalized external results. It does not run evaluations, call model providers, store dashboards, or orchestrate agents. It gives those tools a shared contract.
+
+## Install
+
+```bash
+pip install agent-eval-contract
+```
+
+For local development from this repo:
+
+```bash
+uv sync --dev
+```
+
+## Validate A Record
+
+```python
+from agent_eval_contract import validate_eval_run
+
+run = validate_eval_run(
+    {
+        "run_id": "run-login-flow-001",
+        "task_id": "task-login-flow-001",
+        "harness": "pytest",
+        "model": "gpt-5",
+        "mode": "autonomous",
+        "context_profile": "repo_only",
+        "final_status": "success",
+        "checks": ["pytest tests/test_auth_redirect.py -q"],
+    }
+)
+
+print(run.model_dump(mode="json"))
+```
+
+Validation returns typed Pydantic model instances. Invalid records raise `pydantic.ValidationError` with structured field errors.
+
+## CLI
+
+```bash
+agent-eval-contract validate --kind run --file examples/eval_run.json
+agent-eval-contract schemas --output-dir /tmp/agent-eval-contract-schemas
+agent-eval-contract fixtures --output-dir /tmp/agent-eval-contract-fixtures
+agent-eval-contract normalize --harness terminal-bench --file examples/terminal_bench_result.json --task-id task-login-flow-001 --model gpt-5
+agent-eval-contract normalize --harness swe-bench --file examples/swe_bench_result.json
+```
+
+The legacy `agent-eval-contract-fixtures` command still writes fixture bundles for one release.
+
+## What It Provides
+
+- Pydantic models for eval tasks, runs, scores, failures, external results, normalized runs, and fixture manifests
+- runtime validators that return typed model instances
+- JSON Schema export for all public models
+- bundled sample records and markdown templates
+- Terminal-Bench and SWE-bench oriented normalization helpers
+- a small CLI for validation, schema export, fixture generation, and normalization
+
+## Contract Vocabulary
+
+The public core uses generic vocabulary only. Project-specific concepts should live in `metadata` or a separate adapter package.
+
+- `context_profile`: `repo_only`, `provided_context`, `clean_room`, `tool_augmented`, `full_workspace`
+- `source`: `manual`, `ci`, `benchmark`, `production_trace`, `synthetic`
+- `mode`: `interactive`, `autonomous`, `shadow`, `replay`, `benchmark`
+- `final_status`: `success`, `partial`, `failed`, `abandoned`, `error`
+
+See [docs/contract.md](docs/contract.md), [docs/field-reference.md](docs/field-reference.md), and [docs/adapters.md](docs/adapters.md) for the model contract and adapter guidance.
+
+## Development
+
+```bash
+uv run ruff check agent_eval_contract tests
+uv run ruff format --check agent_eval_contract tests
+uv run basedpyright agent_eval_contract tests
+uv run pytest -q
+uv build --out-dir /tmp/agent-eval-contract-dist
+```

agent_eval_contract-0.2.0/RELEASE.md

@@ -0,0 +1,53 @@
+# Release
+
+## Current Version
+
+- Version: `0.2.0`
+- Contract version: `0.1`
+- Status: public package release
+- Source package: `agent_eval_contract`
+
+## Public Promise
+
+`agent-eval-contract` defines, validates, serializes, exports JSON Schema for, and normalizes portable agent evaluation records.
+
+## Release Checks
+
+CI runs the same quality ladder on pull requests and pushes. Run this local block before tagging or publishing:
+
+```bash
+uv run ruff check agent_eval_contract tests
+uv run ruff format --check agent_eval_contract tests
+uv run basedpyright agent_eval_contract tests
+uv run pytest -q
+uv build --out-dir /tmp/agent-eval-contract-dist
+```
+
+Then install the wheel in a temp virtualenv and smoke test:
+
+```bash
+python -m venv /tmp/agent-eval-contract-venv
+/tmp/agent-eval-contract-venv/bin/pip install /tmp/agent-eval-contract-dist/agent_eval_contract-0.2.0-py3-none-any.whl
+/tmp/agent-eval-contract-venv/bin/agent-eval-contract validate --kind run --file examples/eval_run.json
+/tmp/agent-eval-contract-venv/bin/agent-eval-contract schemas --output-dir /tmp/agent-eval-contract-schemas
+/tmp/agent-eval-contract-venv/bin/agent-eval-contract normalize --harness swe-bench --file examples/swe_bench_result.json
+```
+
+## Boundaries
+
+Public core:
+
+- Pydantic record models
+- runtime validation helpers
+- JSON Schema export
+- external harness normalization
+- fixture bundle generation
+- CLI validation and schema export
+
+Out of scope:
+
+- evaluation execution
+- model provider calls
+- dashboard storage
+- private workflow vocabulary
+- agent orchestration

agent_eval_contract-0.2.0/agent_eval_contract/__init__.py

@@ -0,0 +1,104 @@
+from __future__ import annotations
+
+from .clean_room import run_clean_room_contract_check
+from .external import (
+    normalize_external_result,
+    normalize_swe_bench_result,
+    normalize_terminal_bench_result,
+    to_swe_bench_format,
+    to_terminal_bench_format,
+)
+from .models import (
+    CONTEXT_PROFILES,
+    EVAL_RUN_MODES,
+    EVAL_TASK_SOURCES,
+    EXTERNAL_HARNESSES,
+    FAILURE_PRIORITIES,
+    FINAL_STATUSES,
+    ContextProfile,
+    EvalFailure,
+    EvalRun,
+    EvalRunMode,
+    EvalScore,
+    EvalTask,
+    EvalTaskSource,
+    ExternalHarness,
+    ExternalResult,
+    FailurePriority,
+    FinalStatus,
+    FixtureBundleManifest,
+    JsonValue,
+    NormalizedRun,
+)
+from .release import load_release_metadata, validate_release_metadata
+from .samples import load_sample, validate_all_samples, validate_sample
+from .schema_export import export_json_schemas
+from .templates import (
+    render_eval_template,
+    supported_template_ids,
+    validate_eval_template,
+    validate_eval_template_file,
+    validate_template_directory,
+)
+from .validators import (
+    HARNESS_DIMENSION_NAMES,
+    validate_context_profile,
+    validate_eval_failure,
+    validate_eval_run,
+    validate_eval_score,
+    validate_eval_task,
+    validate_external_result,
+    validate_final_status,
+    validate_harness_fixture_components,
+    validate_priority,
+)
+
+__all__ = [
+    "CONTEXT_PROFILES",
+    "EVAL_RUN_MODES",
+    "EVAL_TASK_SOURCES",
+    "EXTERNAL_HARNESSES",
+    "FAILURE_PRIORITIES",
+    "FINAL_STATUSES",
+    "HARNESS_DIMENSION_NAMES",
+    "ContextProfile",
+    "EvalFailure",
+    "EvalRun",
+    "EvalRunMode",
+    "EvalScore",
+    "EvalTask",
+    "EvalTaskSource",
+    "ExternalHarness",
+    "ExternalResult",
+    "FailurePriority",
+    "FinalStatus",
+    "FixtureBundleManifest",
+    "JsonValue",
+    "NormalizedRun",
+    "export_json_schemas",
+    "load_release_metadata",
+    "load_sample",
+    "normalize_external_result",
+    "normalize_swe_bench_result",
+    "normalize_terminal_bench_result",
+    "render_eval_template",
+    "run_clean_room_contract_check",
+    "supported_template_ids",
+    "to_swe_bench_format",
+    "to_terminal_bench_format",
+    "validate_all_samples",
+    "validate_context_profile",
+    "validate_eval_failure",
+    "validate_eval_run",
+    "validate_eval_score",
+    "validate_eval_task",
+    "validate_eval_template",
+    "validate_eval_template_file",
+    "validate_external_result",
+    "validate_final_status",
+    "validate_harness_fixture_components",
+    "validate_priority",
+    "validate_release_metadata",
+    "validate_sample",
+    "validate_template_directory",
+]

agent_eval_contract-0.2.0/agent_eval_contract/clean_room.py

@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from .samples import validate_all_samples
+from .templates import validate_template_directory
+
+
+def run_clean_room_contract_check(
+    *,
+    template_root: Path,
+    sample_root: Path | None = None,
+) -> dict[str, Any]:
+    templates = validate_template_directory(template_root)
+    samples = (
+        validate_all_samples()
+        if sample_root is None
+        else validate_all_samples(sample_root=sample_root)
+    )
+    return {
+        "ok": True,
+        "template_count": len(templates),
+        "sample_count": len(samples),
+        "templates": templates,
+        "samples": samples,
+    }

agent_eval_contract-0.2.0/agent_eval_contract/cli.py

@@ -0,0 +1,114 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from pydantic import BaseModel
+
+from .external import normalize_external_result
+from .fixture_runner import write_contract_fixture_bundle
+from .schema_export import export_json_schemas
+from .validators import (
+    validate_eval_failure,
+    validate_eval_run,
+    validate_eval_score,
+    validate_eval_task,
+)
+
+
+def _load_json(path: Path) -> dict[str, Any]:
+    loaded = json.loads(path.expanduser().resolve().read_text(encoding="utf-8"))
+    if not isinstance(loaded, dict):
+        raise ValueError(f"{path} must contain a JSON object")
+    return loaded
+
+
+def _print_json(value: BaseModel | dict[str, Any] | list[str]) -> None:
+    payload = value.model_dump(mode="json") if isinstance(value, BaseModel) else value
+    print(json.dumps(payload, indent=2, sort_keys=True))
+
+
+def _run_fixtures(args: argparse.Namespace) -> int:
+    _print_json(write_contract_fixture_bundle(Path(args.output_dir)))
+    return 0
+
+
+def _run_schemas(args: argparse.Namespace) -> int:
+    _print_json({"schemas": export_json_schemas(Path(args.output_dir))})
+    return 0
+
+
+def _run_validate(args: argparse.Namespace) -> int:
+    data = _load_json(Path(args.file))
+    validators = {
+        "task": validate_eval_task,
+        "run": validate_eval_run,
+        "score": validate_eval_score,
+        "failure": validate_eval_failure,
+    }
+    _print_json(validators[args.kind](data))
+    return 0
+
+
+def _run_normalize(args: argparse.Namespace) -> int:
+    data = _load_json(Path(args.file))
+    normalized = normalize_external_result(
+        data,
+        eval_task_id=args.task_id,
+        harness=args.harness,
+        model=args.model,
+    )
+    _print_json(normalized)
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="agent-eval-contract",
+        description="Validate, normalize, and export portable agent evaluation contracts.",
+    )
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    fixtures = subparsers.add_parser(
+        "fixtures", help="Write sample records, templates, and schemas."
+    )
+    fixtures.add_argument(
+        "--output-dir", required=True, help="Directory to write fixture artifacts into."
+    )
+    fixtures.set_defaults(func=_run_fixtures)
+
+    schemas = subparsers.add_parser(
+        "schemas", help="Export JSON Schemas for public contract models."
+    )
+    schemas.add_argument(
+        "--output-dir", required=True, help="Directory to write schema files into."
+    )
+    schemas.set_defaults(func=_run_schemas)
+
+    validate = subparsers.add_parser(
+        "validate", help="Validate a JSON record against a contract model."
+    )
+    validate.add_argument("--kind", choices=("task", "run", "score", "failure"), required=True)
+    validate.add_argument("--file", required=True, help="JSON file to validate.")
+    validate.set_defaults(func=_run_validate)
+
+    normalize = subparsers.add_parser("normalize", help="Normalize external harness output.")
+    normalize.add_argument("--harness", choices=("terminal-bench", "swe-bench"), required=True)
+    normalize.add_argument("--file", required=True, help="External result JSON file to normalize.")
+    normalize.add_argument("--task-id", help="Task id to use when the external result omits one.")
+    normalize.add_argument("--model", help="Model name to use when the external result omits one.")
+    normalize.set_defaults(func=_run_normalize)
+
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    return int(args.func(args))
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())