halflife-ai 0.3.0__tar.gz

halflife_ai-0.3.0/LICENSE

@@ -0,0 +1,17 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Copyright 2026 Evan Prawda
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

halflife_ai-0.3.0/PKG-INFO

@@ -0,0 +1,239 @@
+Metadata-Version: 2.4
+Name: halflife-ai
+Version: 0.3.0
+Summary: Measure how long a task you can trust an AI agent with.
+Author-email: Evan Prawda <evanprawda92@gmail.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/ev-prawda92/halflife
+Keywords: ai,agents,evaluation,benchmark,reliability,llm
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai>=1.40; extra == "openai"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# halflife
+
+**Your agent works on 5-step tasks. Does it work on 50?**
+
+[![tests](https://github.com/ev-prawda92/halflife/actions/workflows/tests.yml/badge.svg)](https://github.com/ev-prawda92/halflife/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/halflife-ai)](https://pypi.org/project/halflife-ai/)
+[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](pyproject.toml)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
+
+halflife measures your agent's **reliability half-life**: the task length at which it succeeds half the time. Then it tells you where runs broke, what to change, and re-runs exactly the cases that failed once you've changed it.
+
+Agents fail quietly as tasks get longer: they lose a running total from step 4 by step 30, skip the last item, pay a duplicate they saw twenty steps ago. Unit tests and single-turn evals don't catch it. halflife does, in your terminal, your tests, your CI, and your coding agent.
+
+<p align="center"><img src="docs/terminal.png" width="760" alt="halflife terminal output: success by length, half-life 32.7 steps, failure modes, and fixes"></p>
+
+> The numbers above come from `halflife demo`, a simulated agent built to show the tool. Run it on your own agent for real numbers.
+
+## Start in one minute
+
+```
+pip install halflife-ai
+halflife demo
+```
+
+Then pick the way in that fits what you have.
+
+### You build with Claude Code, Cursor or Codex → let your coding agent use it
+
+Most agents today are built with a coding agent. Give it halflife once and it checks every agent it builds for you:
+
+```
+claude mcp add halflife -- halflife mcp
+```
+
+Then ask it in plain words: *"check how reliable my agent is on long tasks and fix what breaks."* It runs the benchmark or audits your logs, reads where runs broke and why, proposes a change for you to approve, and replays exactly the failing cases to prove the fix worked. A Claude skill and setup for other editors are in [`integrations/`](integrations/).
+
+### You have agent logs → audit them
+
+No setup, any framework, any model. Point it at your logs (JSON Lines of tool calls) and let halflife find the rules your agent follows:
+
+```
+halflife audit agent_logs.jsonl --suggest-rules
+```
+
+It mines patterns from your own runs (a lookup before every action, one notification per claim, every listed item handled), writes them to `rules_suggested.py` with how often each held, and audits against them. Rules that held in every run guard future runs. Rules that broke in a few runs point straight at the runs that went wrong. The file describes what your agent *usually* does, so read it, delete anything that's coincidence, and keep what's policy.
+
+Or write the rules yourself:
+
+```python
+from halflife.rules import at_most_once, covers, eventually, requires
+
+RULES = [
+    covers("list_claims", field="claims", actions=["finalize_claim", "flag_for_review"], key="claim_id"),
+    requires("finalize_claim", before="get_claim", match=["claim_id"]),
+    eventually("finalize_claim", then="notify_member", match=["claim_id"]),
+    at_most_once("notify_member", key=["claim_id"]),
+]
+```
+
+```
+halflife audit agent_logs.jsonl --rules rules.py
+```
+
+You get the half-life measured in tool calls, which rules break and how often, and where. No answer key needed. Try it on the sample: `halflife audit examples/traces/claims.jsonl --suggest-rules`.
+
+### You have agent code → benchmark it
+
+```python
+from halflife import benchmark
+
+@benchmark(suite="claims-intake")
+def my_agent(task):
+    return agent.run(task)
+```
+
+```
+halflife run my_agent.py --quick
+```
+
+Your agent runs against a realistic simulated system with a hidden answer key, on tasks from 5 to 40 steps, several times each. Or scaffold everything with `halflife init`.
+
+### You want to compare models → bench them
+
+```
+halflife bench --model ollama:qwen2.5:7b --model anthropic:claude-sonnet-5-5 --quick
+```
+
+Built-in agents for Ollama, LM Studio, vLLM, OpenAI, OpenRouter, Together, Groq, Anthropic, and any OpenAI-compatible server. No code.
+
+## Every run tells you what to do next
+
+<p align="center"><img src="docs/curve.png" width="760" alt="Success rate falls with task length; fitted curve crosses 50% at 32.7 steps"></p>
+
+- **Half-life with a 95% interval.** Only lengths you actually tested count: if you tested up to 20 steps, it says "≥ 20", never a made-up number.
+- **Decay shape.** Do errors compound as tasks grow, or happen early? The fix differs.
+- **Where runs broke.** The first failure in each run, grouped: lost track of state, skipped a step, acted without a check, looped, broke a rule.
+- **What to fix first.** Ranked changes, structural fixes before prompt tweaks, each with a failing run as evidence.
+- **Should it act on its own?** `--workflow-length 26` predicts success at your real workflow length and recommends act autonomously, act with approval, suggest only, or shadow mode.
+- **Change since last run.** Every run compares itself with the previous one: `half-life 32.7 → 28.1 steps (−14%, worse)`.
+- **An eval pack.** The exact failing tasks, saved. After a fix: `halflife run my_agent.py --replay .halflife/<run>.evals.json`.
+- **An HTML report** to share, and JSON for your pipeline.
+
+## Make it a habit
+
+In your tests:
+
+```python
+import halflife
+
+def test_agent_is_reliable():
+    halflife.assert_reliable(my_agent, min_half_life=15)
+```
+
+In CI (`halflife init --ci` writes the workflow):
+
+```
+halflife run my_agent.py --min-half-life 20
+```
+
+In your README: `halflife badge` writes ![agent half-life](docs/badge-example.svg)
+
+## Plug in any agent
+
+Your function gets a `task`:
+
+| | |
+|---|---|
+| `task.instructions` | the goal and the policy |
+| `task.call(name, args)` | run a tool; errors come back as `{"error": ...}` so your agent can recover |
+| `task.openai_tools()` | tool definitions for OpenAI-compatible APIs (Ollama, vLLM, OpenRouter…) |
+| `task.anthropic_tools()` | tool definitions for the Anthropic API |
+| `task.max_tool_calls` | the budget; going over ends the run, which catches loops |
+
+Full examples: [`examples/claude_agent.py`](examples/claude_agent.py), [`examples/openai_agent.py`](examples/openai_agent.py).
+
+<details>
+<summary><strong>Providers for <code>halflife bench</code></strong></summary>
+
+| Provider | Key | Notes |
+|---|---|---|
+| `ollama` | none | local; set `OLLAMA_HOST` if not localhost |
+| `lmstudio`, `vllm` | none | local; `--base-url` if not the default port |
+| `compatible` | `HALFLIFE_API_KEY` (optional) | any OpenAI-compatible server, with `--base-url` |
+| `openai` | `OPENAI_API_KEY` | |
+| `openrouter` | `OPENROUTER_API_KEY` | one key for most open and closed models |
+| `together`, `groq` | `TOGETHER_API_KEY`, `GROQ_API_KEY` | |
+| `anthropic` | `ANTHROPIC_API_KEY` | prompt caching on by default |
+
+**Ollama:** start it with a large context window, or long tasks fail for the wrong reason, because the default window silently drops older messages:
+
+```
+OLLAMA_CONTEXT_LENGTH=32768 ollama serve
+```
+
+</details>
+
+## Rules
+
+Invariants checked on every run's tool calls. They need no answer key, so they work on any agent and on production logs.
+
+| Rule | Fails when |
+|---|---|
+| `requires(A, before=B, match=[...])` | A happens without an earlier matching B |
+| `at_most_once(A, key=[...])` | A happens twice for the same key |
+| `eventually(A, then=B, match=[...])` | A is never followed by a matching B |
+| `covers(S, field, actions, key)` | an item listed by S never reaches one of the actions |
+| `never(A, when=...)` | A happens with forbidden arguments |
+| `no_loops(n)` | the same call repeats more than n times in a row |
+| `recovers_from_errors()` | the agent resends a call that just failed |
+| `@rule` | anything you can write as a function of the steps |
+
+Use them in audits, or add them to benchmarks: `halflife run my_agent.py --rules rules.py`.
+
+## Built-in suites
+
+| Suite | Domain | What gets harder with length |
+|---|---|---|
+| `claims-intake` | Health plan claims: eligibility, network, coverage, prior auth, deductibles | Members recur across claims; the deductible shown is from the start of the batch, so state must be carried |
+| `invoice-matching` | Accounts payable: blocked vendors, duplicates, price and quantity checks | Duplicates repeat earlier invoices; receipts are consumed by earlier approvals |
+
+Every suite ships with a test proving that an agent following the written policy exactly scores 100%, so a failure is the agent's, not the answer key's. Write your own suite in about 40 lines: [`examples/custom_suite.py`](examples/custom_suite.py). `halflife show --suite claims-intake --length 10 --answers` prints exactly what an agent sees, plus the answer key.
+
+## How the number works
+
+Success at length L is modeled as `2^(−(L/h)^k)`. h (half-life) and k (decay shape) are fit by maximum likelihood over every run, with a bootstrap 95% interval. The observed 50% crossing is reported alongside as a model-free check. Full definitions, file formats, and reporting rules are in [SPEC.md](SPEC.md), so any tool can produce comparable numbers.
+
+**Related work.** Measuring capability by the length of task completed half the time builds on METR's time-horizon research on frontier models. halflife applies the idea to your own agent on your own workflows, explains why runs fail, and makes it a daily development check.
+
+## Commands
+
+```
+halflife run FILE[:function] [--quick] [--suite S] [--lengths 5,10,20,40] [--tasks N] [--runs N] [--rules FILE] [--workflow-length W] [--min-half-life H] [--replay PACK] [--save-traces FILE] [--workers N] [--out DIR]
+halflife bench --model PROVIDER:MODEL [--model ...] [same options]
+halflife audit TRACES.jsonl [--rules FILE] [--workflow-length W] [--min-half-life H]
+halflife init [--suite S] [--ci]
+halflife badge [RESULT.json]
+halflife compare RESULT.json [RESULT.json ...]
+halflife show [--suite S] [--length N] [--answers] [--json]
+halflife mcp
+halflife demo [--suite S] [--quick]
+halflife suites
+```
+
+## Roadmap
+
+- Generate test environments automatically from an MCP server or OpenAPI spec, so any agent gets a benchmark with no hand-written suite
+- Read OpenTelemetry GenAI traces directly in `halflife audit`
+- Adapters for popular agent frameworks
+- TypeScript SDK
+- More suites: support, IT ops, procurement, coding
+
+Ideas and suites welcome: see [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+Apache 2.0

halflife_ai-0.3.0/README.md

@@ -0,0 +1,216 @@
+# halflife
+
+**Your agent works on 5-step tasks. Does it work on 50?**
+
+[![tests](https://github.com/ev-prawda92/halflife/actions/workflows/tests.yml/badge.svg)](https://github.com/ev-prawda92/halflife/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/halflife-ai)](https://pypi.org/project/halflife-ai/)
+[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](pyproject.toml)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
+
+halflife measures your agent's **reliability half-life**: the task length at which it succeeds half the time. Then it tells you where runs broke, what to change, and re-runs exactly the cases that failed once you've changed it.
+
+Agents fail quietly as tasks get longer: they lose a running total from step 4 by step 30, skip the last item, pay a duplicate they saw twenty steps ago. Unit tests and single-turn evals don't catch it. halflife does, in your terminal, your tests, your CI, and your coding agent.
+
+<p align="center"><img src="docs/terminal.png" width="760" alt="halflife terminal output: success by length, half-life 32.7 steps, failure modes, and fixes"></p>
+
+> The numbers above come from `halflife demo`, a simulated agent built to show the tool. Run it on your own agent for real numbers.
+
+## Start in one minute
+
+```
+pip install halflife-ai
+halflife demo
+```
+
+Then pick the way in that fits what you have.
+
+### You build with Claude Code, Cursor or Codex → let your coding agent use it
+
+Most agents today are built with a coding agent. Give it halflife once and it checks every agent it builds for you:
+
+```
+claude mcp add halflife -- halflife mcp
+```
+
+Then ask it in plain words: *"check how reliable my agent is on long tasks and fix what breaks."* It runs the benchmark or audits your logs, reads where runs broke and why, proposes a change for you to approve, and replays exactly the failing cases to prove the fix worked. A Claude skill and setup for other editors are in [`integrations/`](integrations/).
+
+### You have agent logs → audit them
+
+No setup, any framework, any model. Point it at your logs (JSON Lines of tool calls) and let halflife find the rules your agent follows:
+
+```
+halflife audit agent_logs.jsonl --suggest-rules
+```
+
+It mines patterns from your own runs (a lookup before every action, one notification per claim, every listed item handled), writes them to `rules_suggested.py` with how often each held, and audits against them. Rules that held in every run guard future runs. Rules that broke in a few runs point straight at the runs that went wrong. The file describes what your agent *usually* does, so read it, delete anything that's coincidence, and keep what's policy.
+
+Or write the rules yourself:
+
+```python
+from halflife.rules import at_most_once, covers, eventually, requires
+
+RULES = [
+    covers("list_claims", field="claims", actions=["finalize_claim", "flag_for_review"], key="claim_id"),
+    requires("finalize_claim", before="get_claim", match=["claim_id"]),
+    eventually("finalize_claim", then="notify_member", match=["claim_id"]),
+    at_most_once("notify_member", key=["claim_id"]),
+]
+```
+
+```
+halflife audit agent_logs.jsonl --rules rules.py
+```
+
+You get the half-life measured in tool calls, which rules break and how often, and where. No answer key needed. Try it on the sample: `halflife audit examples/traces/claims.jsonl --suggest-rules`.
+
+### You have agent code → benchmark it
+
+```python
+from halflife import benchmark
+
+@benchmark(suite="claims-intake")
+def my_agent(task):
+    return agent.run(task)
+```
+
+```
+halflife run my_agent.py --quick
+```
+
+Your agent runs against a realistic simulated system with a hidden answer key, on tasks from 5 to 40 steps, several times each. Or scaffold everything with `halflife init`.
+
+### You want to compare models → bench them
+
+```
+halflife bench --model ollama:qwen2.5:7b --model anthropic:claude-sonnet-5-5 --quick
+```
+
+Built-in agents for Ollama, LM Studio, vLLM, OpenAI, OpenRouter, Together, Groq, Anthropic, and any OpenAI-compatible server. No code.
+
+## Every run tells you what to do next
+
+<p align="center"><img src="docs/curve.png" width="760" alt="Success rate falls with task length; fitted curve crosses 50% at 32.7 steps"></p>
+
+- **Half-life with a 95% interval.** Only lengths you actually tested count: if you tested up to 20 steps, it says "≥ 20", never a made-up number.
+- **Decay shape.** Do errors compound as tasks grow, or happen early? The fix differs.
+- **Where runs broke.** The first failure in each run, grouped: lost track of state, skipped a step, acted without a check, looped, broke a rule.
+- **What to fix first.** Ranked changes, structural fixes before prompt tweaks, each with a failing run as evidence.
+- **Should it act on its own?** `--workflow-length 26` predicts success at your real workflow length and recommends act autonomously, act with approval, suggest only, or shadow mode.
+- **Change since last run.** Every run compares itself with the previous one: `half-life 32.7 → 28.1 steps (−14%, worse)`.
+- **An eval pack.** The exact failing tasks, saved. After a fix: `halflife run my_agent.py --replay .halflife/<run>.evals.json`.
+- **An HTML report** to share, and JSON for your pipeline.
+
+## Make it a habit
+
+In your tests:
+
+```python
+import halflife
+
+def test_agent_is_reliable():
+    halflife.assert_reliable(my_agent, min_half_life=15)
+```
+
+In CI (`halflife init --ci` writes the workflow):
+
+```
+halflife run my_agent.py --min-half-life 20
+```
+
+In your README: `halflife badge` writes ![agent half-life](docs/badge-example.svg)
+
+## Plug in any agent
+
+Your function gets a `task`:
+
+| | |
+|---|---|
+| `task.instructions` | the goal and the policy |
+| `task.call(name, args)` | run a tool; errors come back as `{"error": ...}` so your agent can recover |
+| `task.openai_tools()` | tool definitions for OpenAI-compatible APIs (Ollama, vLLM, OpenRouter…) |
+| `task.anthropic_tools()` | tool definitions for the Anthropic API |
+| `task.max_tool_calls` | the budget; going over ends the run, which catches loops |
+
+Full examples: [`examples/claude_agent.py`](examples/claude_agent.py), [`examples/openai_agent.py`](examples/openai_agent.py).
+
+<details>
+<summary><strong>Providers for <code>halflife bench</code></strong></summary>
+
+| Provider | Key | Notes |
+|---|---|---|
+| `ollama` | none | local; set `OLLAMA_HOST` if not localhost |
+| `lmstudio`, `vllm` | none | local; `--base-url` if not the default port |
+| `compatible` | `HALFLIFE_API_KEY` (optional) | any OpenAI-compatible server, with `--base-url` |
+| `openai` | `OPENAI_API_KEY` | |
+| `openrouter` | `OPENROUTER_API_KEY` | one key for most open and closed models |
+| `together`, `groq` | `TOGETHER_API_KEY`, `GROQ_API_KEY` | |
+| `anthropic` | `ANTHROPIC_API_KEY` | prompt caching on by default |
+
+**Ollama:** start it with a large context window, or long tasks fail for the wrong reason, because the default window silently drops older messages:
+
+```
+OLLAMA_CONTEXT_LENGTH=32768 ollama serve
+```
+
+</details>
+
+## Rules
+
+Invariants checked on every run's tool calls. They need no answer key, so they work on any agent and on production logs.
+
+| Rule | Fails when |
+|---|---|
+| `requires(A, before=B, match=[...])` | A happens without an earlier matching B |
+| `at_most_once(A, key=[...])` | A happens twice for the same key |
+| `eventually(A, then=B, match=[...])` | A is never followed by a matching B |
+| `covers(S, field, actions, key)` | an item listed by S never reaches one of the actions |
+| `never(A, when=...)` | A happens with forbidden arguments |
+| `no_loops(n)` | the same call repeats more than n times in a row |
+| `recovers_from_errors()` | the agent resends a call that just failed |
+| `@rule` | anything you can write as a function of the steps |
+
+Use them in audits, or add them to benchmarks: `halflife run my_agent.py --rules rules.py`.
+
+## Built-in suites
+
+| Suite | Domain | What gets harder with length |
+|---|---|---|
+| `claims-intake` | Health plan claims: eligibility, network, coverage, prior auth, deductibles | Members recur across claims; the deductible shown is from the start of the batch, so state must be carried |
+| `invoice-matching` | Accounts payable: blocked vendors, duplicates, price and quantity checks | Duplicates repeat earlier invoices; receipts are consumed by earlier approvals |
+
+Every suite ships with a test proving that an agent following the written policy exactly scores 100%, so a failure is the agent's, not the answer key's. Write your own suite in about 40 lines: [`examples/custom_suite.py`](examples/custom_suite.py). `halflife show --suite claims-intake --length 10 --answers` prints exactly what an agent sees, plus the answer key.
+
+## How the number works
+
+Success at length L is modeled as `2^(−(L/h)^k)`. h (half-life) and k (decay shape) are fit by maximum likelihood over every run, with a bootstrap 95% interval. The observed 50% crossing is reported alongside as a model-free check. Full definitions, file formats, and reporting rules are in [SPEC.md](SPEC.md), so any tool can produce comparable numbers.
+
+**Related work.** Measuring capability by the length of task completed half the time builds on METR's time-horizon research on frontier models. halflife applies the idea to your own agent on your own workflows, explains why runs fail, and makes it a daily development check.
+
+## Commands
+
+```
+halflife run FILE[:function] [--quick] [--suite S] [--lengths 5,10,20,40] [--tasks N] [--runs N] [--rules FILE] [--workflow-length W] [--min-half-life H] [--replay PACK] [--save-traces FILE] [--workers N] [--out DIR]
+halflife bench --model PROVIDER:MODEL [--model ...] [same options]
+halflife audit TRACES.jsonl [--rules FILE] [--workflow-length W] [--min-half-life H]
+halflife init [--suite S] [--ci]
+halflife badge [RESULT.json]
+halflife compare RESULT.json [RESULT.json ...]
+halflife show [--suite S] [--length N] [--answers] [--json]
+halflife mcp
+halflife demo [--suite S] [--quick]
+halflife suites
+```
+
+## Roadmap
+
+- Generate test environments automatically from an MCP server or OpenAPI spec, so any agent gets a benchmark with no hand-written suite
+- Read OpenTelemetry GenAI traces directly in `halflife audit`
+- Adapters for popular agent frameworks
+- TypeScript SDK
+- More suites: support, IT ops, procurement, coding
+
+Ideas and suites welcome: see [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+Apache 2.0

halflife_ai-0.3.0/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "halflife-ai"
+version = "0.3.0"
+description = "Measure how long a task you can trust an AI agent with."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "Apache-2.0" }
+authors = [{ name = "Evan Prawda", email = "evanprawda92@gmail.com" }]
+keywords = ["ai", "agents", "evaluation", "benchmark", "reliability", "llm"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: Apache Software License",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Testing",
+]
+dependencies = []
+
+[project.optional-dependencies]
+anthropic = ["anthropic>=0.40"]
+openai = ["openai>=1.40"]
+dev = ["pytest>=7"]
+
+[project.scripts]
+halflife = "halflife.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/ev-prawda92/halflife"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["tests"]

halflife_ai-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

halflife_ai-0.3.0/src/halflife/__init__.py

@@ -0,0 +1,45 @@
+"""halflife: measure how long a task you can trust an AI agent with.
+
+    from halflife import benchmark
+
+    @benchmark(suite="claims-intake")
+    def my_agent(task):
+        return agent.run(task)
+
+Then run `halflife run my_agent.py`, or `my_agent.evaluate()` from Python.
+"""
+
+__version__ = "0.3.0"
+
+from .decorator import assert_reliable, benchmark
+from .report import Report
+from .runner import build_task, evaluate, grade
+from .sandbox import Sandbox, ToolBudgetExceeded, ToolError, param, tool
+from .suite import get_suite, list_suites, suite
+from .task import Checkpoint, Constraint, Task, TaskSpec
+from .traces import TraceWriter, audit
+from . import rules
+
+__all__ = [
+    "benchmark",
+    "assert_reliable",
+    "evaluate",
+    "Report",
+    "Task",
+    "TaskSpec",
+    "Checkpoint",
+    "Constraint",
+    "Sandbox",
+    "tool",
+    "param",
+    "ToolError",
+    "ToolBudgetExceeded",
+    "suite",
+    "get_suite",
+    "list_suites",
+    "build_task",
+    "grade",
+    "rules",
+    "audit",
+    "TraceWriter",
+]

halflife_ai-0.3.0/src/halflife/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

halflife_ai-0.3.0/src/halflife/agents/__init__.py

@@ -0,0 +1,93 @@
+"""Built-in agents for real models, so you can benchmark a model with no code.
+
+    halflife bench --model ollama:qwen2.5:7b
+    halflife bench --model anthropic:claude-sonnet-5-5
+    halflife bench --model openai:gpt-4.1
+    halflife bench --model openrouter:meta-llama/llama-3.3-70b-instruct
+    halflife bench --model vllm:my-model --base-url http://gpu-box:8000/v1
+
+Providers and where they read their key:
+    ollama       local, no key           OLLAMA_HOST (default http://localhost:11434)
+    lmstudio     local, no key           http://localhost:1234/v1
+    vllm         local, no key           http://localhost:8000/v1
+    openai       OPENAI_API_KEY
+    openrouter   OPENROUTER_API_KEY
+    together     TOGETHER_API_KEY
+    groq         GROQ_API_KEY
+    anthropic    ANTHROPIC_API_KEY
+    compatible   any OpenAI-compatible server; pass --base-url (key from HALFLIFE_API_KEY)
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Optional
+
+from ._http import ModelAPIError
+from .anthropic_api import AnthropicAgent
+from .openai_compat import OpenAICompatibleAgent
+
+PROVIDERS = {
+    "ollama": (None, None),
+    "lmstudio": ("http://localhost:1234/v1", None),
+    "vllm": ("http://localhost:8000/v1", None),
+    "openai": ("https://api.openai.com/v1", "OPENAI_API_KEY"),
+    "openrouter": ("https://openrouter.ai/api/v1", "OPENROUTER_API_KEY"),
+    "together": ("https://api.together.xyz/v1", "TOGETHER_API_KEY"),
+    "groq": ("https://api.groq.com/openai/v1", "GROQ_API_KEY"),
+    "compatible": (None, "HALFLIFE_API_KEY"),
+}
+
+
+def from_spec(spec: str, base_url: Optional[str] = None, temperature: float = 0.0):
+    """Build an agent from 'provider:model', e.g. 'ollama:qwen2.5:7b'."""
+    provider, sep, model = spec.partition(":")
+    if not sep or not model:
+        raise ValueError(f"model must look like provider:model, e.g. ollama:qwen2.5:7b (got '{spec}')")
+    name = spec
+    if provider == "anthropic":
+        return AnthropicAgent(model, base_url=base_url or "https://api.anthropic.com", name=name)
+    if provider not in PROVIDERS:
+        raise ValueError(f"unknown provider '{provider}'. Known: anthropic, {', '.join(PROVIDERS)}")
+    default_url, key_var = PROVIDERS[provider]
+    if provider == "ollama":
+        default_url = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/") + "/v1"
+        if not default_url.startswith("http"):
+            default_url = "http://" + default_url
+    url = base_url or default_url
+    if not url:
+        raise ValueError(f"provider '{provider}' needs --base-url")
+    key = os.environ.get(key_var) if key_var else None
+    if key_var and not key and provider not in ("compatible",):
+        raise ModelAPIError(f"{key_var} is not set")
+    return OpenAICompatibleAgent(model, url, api_key=key, temperature=temperature, name=name)
+
+
+def preflight(agent, timeout: float = 4.0) -> Optional[str]:
+    """Return a plain-English problem if the model server can't be reached, else None."""
+    import urllib.error
+    import urllib.request
+
+    if isinstance(agent, AnthropicAgent):
+        return None if agent.api_key else "ANTHROPIC_API_KEY is not set"
+    url = agent.base_url.rstrip("/") + "/models"
+    headers = {"authorization": f"Bearer {agent.api_key}"} if agent.api_key else {}
+    try:
+        urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=timeout)
+        return None
+    except urllib.error.HTTPError as exc:
+        if exc.code in (401, 403):
+            return f"the server at {agent.base_url} rejected the API key ({exc.code})"
+        return None
+    except (urllib.error.URLError, OSError):
+        host = agent.base_url.split("/v1")[0]
+        if "11434" in host:
+            return (
+                f"can't reach Ollama at {host}. Is it installed and running? Start it with:\n"
+                "    OLLAMA_CONTEXT_LENGTH=32768 ollama serve\n"
+                "  and pull the model first, e.g.: ollama pull qwen2.5:7b"
+            )
+        return f"can't reach the model server at {host}. Is it running?"
+
+
+__all__ = ["preflight", "AnthropicAgent", "OpenAICompatibleAgent", "ModelAPIError", "from_spec", "PROVIDERS"]