agent-eval-contract 0.2.0__py3-none-any.whl

agent_eval_contract-0.2.0.dist-info/METADATA

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: agent-eval-contract
+Version: 0.2.0
+Summary: Pydantic contracts and JSON Schemas for portable agent evaluation records.
+Author: Jakye Amos
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/jakyeamos/agent-eval-contract
+Project-URL: Repository, https://github.com/jakyeamos/agent-eval-contract
+Project-URL: Issues, https://github.com/jakyeamos/agent-eval-contract/issues
+Keywords: agents,ai-evaluation,evals,harness,json-schema,pydantic
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic<3,>=2
+Dynamic: license-file
+
+# Agent Eval Contract
+
+Pydantic contracts and JSON Schemas for portable agent evaluation records.
+
+Use this package when you are experimenting with agents, harnesses, CI checks, or benchmark runners and need a stable record shape for tasks, runs, scores, failures, and normalized external results. It does not run evaluations, call model providers, store dashboards, or orchestrate agents. It gives those tools a shared contract.
+
+## Install
+
+```bash
+pip install agent-eval-contract
+```
+
+For local development from this repo:
+
+```bash
+uv sync --dev
+```
+
+## Validate A Record
+
+```python
+from agent_eval_contract import validate_eval_run
+
+run = validate_eval_run(
+    {
+        "run_id": "run-login-flow-001",
+        "task_id": "task-login-flow-001",
+        "harness": "pytest",
+        "model": "gpt-5",
+        "mode": "autonomous",
+        "context_profile": "repo_only",
+        "final_status": "success",
+        "checks": ["pytest tests/test_auth_redirect.py -q"],
+    }
+)
+
+print(run.model_dump(mode="json"))
+```
+
+Validation returns typed Pydantic model instances. Invalid records raise `pydantic.ValidationError` with structured field errors.
+
+## CLI
+
+```bash
+agent-eval-contract validate --kind run --file examples/eval_run.json
+agent-eval-contract schemas --output-dir /tmp/agent-eval-contract-schemas
+agent-eval-contract fixtures --output-dir /tmp/agent-eval-contract-fixtures
+agent-eval-contract normalize --harness terminal-bench --file examples/terminal_bench_result.json --task-id task-login-flow-001 --model gpt-5
+agent-eval-contract normalize --harness swe-bench --file examples/swe_bench_result.json
+```
+
+The legacy `agent-eval-contract-fixtures` command still writes fixture bundles for one release.
+
+## What It Provides
+
+- Pydantic models for eval tasks, runs, scores, failures, external results, normalized runs, and fixture manifests
+- runtime validators that return typed model instances
+- JSON Schema export for all public models
+- bundled sample records and markdown templates
+- Terminal-Bench and SWE-bench oriented normalization helpers
+- a small CLI for validation, schema export, fixture generation, and normalization
+
+## Contract Vocabulary
+
+The public core uses generic vocabulary only. Project-specific concepts should live in `metadata` or a separate adapter package.
+
+- `context_profile`: `repo_only`, `provided_context`, `clean_room`, `tool_augmented`, `full_workspace`
+- `source`: `manual`, `ci`, `benchmark`, `production_trace`, `synthetic`
+- `mode`: `interactive`, `autonomous`, `shadow`, `replay`, `benchmark`
+- `final_status`: `success`, `partial`, `failed`, `abandoned`, `error`
+
+See [docs/contract.md](docs/contract.md), [docs/field-reference.md](docs/field-reference.md), and [docs/adapters.md](docs/adapters.md) for the model contract and adapter guidance.
+
+## Development
+
+```bash
+uv run ruff check agent_eval_contract tests
+uv run ruff format --check agent_eval_contract tests
+uv run basedpyright agent_eval_contract tests
+uv run pytest -q
+uv build --out-dir /tmp/agent-eval-contract-dist
+```

agent_eval_contract-0.2.0.dist-info/RECORD

@@ -0,0 +1,25 @@
+agent_eval_contract/__init__.py,sha256=T7N3nI4rS-QopECHv5ETC15udtykMAELbwgAo2_3Kho,2617
+agent_eval_contract/clean_room.py,sha256=_YJethwAQcw-kxNxPR96YaWKonOZ7nHh_dNvpWRCtC0,686
+agent_eval_contract/cli.py,sha256=AHWNaK1dq4rTt6II00_vFsnfySAM-G4RdtusSPy6SG4,3742
+agent_eval_contract/external.py,sha256=SRK8EDJvOWLaG5SSkEc8EGb5htITU4ez0Fhq54ugxxg,7033
+agent_eval_contract/fixture_runner.py,sha256=UvfWd7g2uCI8AdRbeCS2ypDzvDbjypaG48Jdxx4eD0k,2838
+agent_eval_contract/models.py,sha256=wsxu9E0o-41kxzMdRZvUrR12MucWtR2QHo_OMW8KLF8,5441
+agent_eval_contract/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
+agent_eval_contract/release.py,sha256=ma4nASgM9dp4uwIyRYr0rVGSyd9O4ZcMu3nCM8EL2FQ,2118
+agent_eval_contract/release_metadata.json,sha256=GT7mAmwxRsQw8DBXYN7KSPZcfsdgOV_NjBTUuCCAa4Y,1189
+agent_eval_contract/samples.py,sha256=O-cDq6y34BsO5Kc8xYuPiaS4DO24OugHUj2HzUljjaw,2861
+agent_eval_contract/schema_export.py,sha256=d--Eb4zHWFK5LopsqqJcEKdOU5EO9TgC5k2yd_-0AF0,1076
+agent_eval_contract/schemas.py,sha256=lWGcMALDzWv8Byd4H_tZgOlsCwRi5sds1W0kkOWRK5k,893
+agent_eval_contract/templates.py,sha256=r37AwsLX2HfFPljx0yagZswy3kRK5I-C4mqEL03U_i8,3210
+agent_eval_contract/validators.py,sha256=tgQ4IS39T7h1-kvGy1l7VcLH_nYnocCs9I5WIJ2o16M,4015
+agent_eval_contract/samples/eval_failure.json,sha256=Bimd0LwT14zaJQ1NXq24oaAN6jzJvCgX70zz_lq5oTg,691
+agent_eval_contract/samples/eval_run.json,sha256=7yX8ZOyKFSPtVaKtscmXfndZb2GzBJDBhyEO3Lqpj_E,664
+agent_eval_contract/samples/eval_score.json,sha256=eQOxXLugT5nG9Xip3atPcdRcOgmJ5UjjsWAkzVz_IF8,371
+agent_eval_contract/samples/eval_task.json,sha256=BKsBwGOufAXvDVTjuCF8h2i2CEhTSCL3at9aSC7jQXU,693
+agent_eval_contract/samples/external_result_normalization.json,sha256=CcA9-ewXpaVs1qPhY_N6k5yn03DukgkPogs3l0GuLBs,860
+agent_eval_contract-0.2.0.dist-info/licenses/LICENSE,sha256=RodEmwx_oPFIr3EfIiApJ3-87ze5JZnSjzM4BGHhnfQ,1067
+agent_eval_contract-0.2.0.dist-info/METADATA,sha256=2xQ6vCsrqwA51y15dkSJbppN0TQdiXta_kCoC38XZn4,4089
+agent_eval_contract-0.2.0.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+agent_eval_contract-0.2.0.dist-info/entry_points.txt,sha256=YAG3pntEb5Nxq9vCD3OkEzOYitjNjLK9ibCJ8lJ9aLA,140
+agent_eval_contract-0.2.0.dist-info/top_level.txt,sha256=RjxE1iMAN1BgJY98BExHskv0wx_FhdIiyhGZhatTNRE,20
+agent_eval_contract-0.2.0.dist-info/RECORD,,

agent_eval_contract-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

agent_eval_contract-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+agent-eval-contract = agent_eval_contract.cli:main
+agent-eval-contract-fixtures = agent_eval_contract.fixture_runner:main

agent_eval_contract-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jakye Amos
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_eval_contract-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+agent_eval_contract