agent-attrib 0.1.0__tar.gz

agent_attrib-0.1.0/PKG-INFO

@@ -0,0 +1,181 @@
+Metadata-Version: 2.3
+Name: agent-attrib
+Version: 0.1.0
+Summary: Multi-factor attribution for agent release changes using factorial design of experiments
+Keywords: agents,llm,attribution,factorial,design-of-experiments,statistics
+Author: agent-attrib contributors
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Dist: numpy>=1.26
+Requires-Dist: pandas>=2.1
+Requires-Dist: scipy>=1.11
+Requires-Dist: statsmodels>=0.14
+Requires-Dist: pydantic>=2.5
+Requires-Dist: typer>=0.9
+Requires-Dist: rich>=13.7
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/agent-attrib/agent-attrib
+Project-URL: Issues, https://github.com/agent-attrib/agent-attrib/issues
+Description-Content-Type: text/markdown
+
+# agent-attrib
+
+A Python package for attributing metric changes across simultaneous agent release changes using factorial design of experiments.
+
+**The problem:** You shipped a release that changed the prompt, the model, and the retriever at once. Your eval pass rate moved. Which change caused it, and by how much?
+
+**The solution:** Design a factorial experiment, run your agent on the configurations it specifies, and let `agent-attrib` decompose the effects with confidence intervals.
+
+```bash
+pip install agent-attrib
+```
+
+## 60-second demo
+
+```bash
+agent-attrib demo
+```
+
+No LLM calls, no cost. Generates synthetic runs with planted effects and reports which factors are responsible.
+
+## Usage
+
+### 1. Design the experiment
+
+```bash
+agent-attrib plan --factors prompt,model,retriever --tasks 30 --trials 2 -o design.csv
+```
+
+```
+Design type  : 2^3 full factorial
+Resolution   : full
+Factors      : prompt, model, retriever
+Tasks        : 30
+Trials/config: 2
+Configurations: 8
+Total runs   : 480
+Seed         : 42
+MDE (success, α=0.05, power=0.8): 0.1283
+```
+
+For 5-7 factors, a fractional factorial is used automatically (resolution IV by default).
+
+### 2. Run your agent
+
+For each row in `design.csv`, set each factor to the specified level (`baseline` or `candidate`) and run your agent on the task. Write one line per run:
+
+```json
+{"run_id": "r001", "task_id": "t0001", "trial": 1,
+ "factors": {"prompt": "candidate", "model": "baseline", "retriever": "candidate"},
+ "metrics": {"success": 1, "cost_usd": 0.042, "latency_s": 11.3}}
+```
+
+### 3. Analyze
+
+```bash
+agent-attrib analyze runs.jsonl --metric success --metric cost_usd --margin 0.05
+```
+
+## Python API
+
+```python
+from agent_attrib import plan, analyze, load_runs
+
+design = plan(factors=["prompt", "model", "retriever"], tasks=30, trials=2, seed=7)
+print(design.summary())
+
+runset = load_runs("runs.jsonl")
+result = analyze(runset, metrics=["success", "cost_usd"], margin=0.05)
+print(result.to_markdown())
+```
+
+## Input format
+
+**JSONL** (one object per run):
+
+```json
+{"run_id": "r001", "task_id": "t07", "trial": 1,
+ "factors": {"prompt": "candidate", "model": "baseline", "retriever": "candidate"},
+ "metrics": {"success": 1, "cost_usd": 0.042, "latency_s": 11.3}}
+```
+
+**CSV** with columns `run_id, task_id, trial, f_<factor>..., m_<metric>...`:
+
+```csv
+run_id,task_id,trial,f_prompt,f_model,m_success,m_cost_usd
+r001,t07,1,candidate,baseline,1,0.042
+```
+
+Factor levels must be exactly `baseline` or `candidate`. Metrics must be numeric.
+
+## Method
+
+For each metric, `analyze` fits OLS with task fixed effects:
+
+```
+metric ~ task fixed effects + main effects + estimable two-factor interactions
+```
+
+- **Binary `success`:** linear probability model. Effects are in percentage points.
+  Not logistic regression: effects stay directly interpretable as probability differences.
+- **Cost and latency:** `log(metric)` model. Effects reported as percent change.
+- **Standard errors:** cluster-robust, clustered by task.
+- **Multiple testing:** Holm step-down correction across all effects within a metric.
+- **Verdicts** (given an equivalence margin `m`):
+  - `IMPROVEMENT` -- 95% CI sits entirely above +m
+  - `REGRESSION` -- 95% CI sits entirely below -m
+  - `EQUIVALENT` -- 95% CI lies within +/-m
+  - `INCONCLUSIVE` -- CI straddles a boundary; includes approximate tasks needed to resolve
+- Effect convention: switching a factor from baseline to candidate (= 2*beta under +/-1 coding).
+
+## Designs
+
+| Factors | Design | Runs (30 tasks, 2 trials) |
+|---------|--------|--------------------------|
+| 1-4 | 2^k full factorial | 60-480 |
+| 5 | 2^(5-1) res IV | 960 |
+| 6 | 2^(6-2) res IV | 960 |
+| 7 | 2^(7-3) res IV | 960 |
+
+Fractional designs print their alias structure. Use `--resolution V` to request resolution V (more runs, no 2FI aliasing).
+
+## Limitations
+
+- Results apply to this task suite and model versions only.
+- Attribution requires designed runs -- not observational production logs.
+- Maximum 7 factors in v0.1.
+- Two levels per factor only (baseline vs candidate).
+- No LLM-as-judge; `agent-attrib` never calls any LLM.
+
+## Prior art
+
+- **agentstats, AgentAssay, deltagate:** proper A/B statistics for a single pair. Do not decompose changes across multiple simultaneous factors.
+- **ablate-cli:** per-section ablation of `CLAUDE.md` files, for coding agents, runs Claude Code itself. Borrowed two ideas: four-way verdicts and printing the minimum detectable effect before any spend.
+- **Research papers:** factorial ablations over agent components exist as one-off scripts.
+
+`agent-attrib`'s niche: framework-agnostic, multi-factor, interaction-aware attribution for arbitrary release changes, decoupled from execution.
+
+## Development
+
+```bash
+git clone https://github.com/agent-attrib/agent-attrib
+cd agent-attrib
+uv sync --all-groups
+uv run pytest -m "not slow"          # fast tests (~seconds)
+uv run pytest                        # all tests including calibration (~minutes)
+uv run ruff check src tests
+uv run mypy src
+```
+
+## License
+
+MIT

agent_attrib-0.1.0/README.md

@@ -0,0 +1,152 @@
+# agent-attrib
+
+A Python package for attributing metric changes across simultaneous agent release changes using factorial design of experiments.
+
+**The problem:** You shipped a release that changed the prompt, the model, and the retriever at once. Your eval pass rate moved. Which change caused it, and by how much?
+
+**The solution:** Design a factorial experiment, run your agent on the configurations it specifies, and let `agent-attrib` decompose the effects with confidence intervals.
+
+```bash
+pip install agent-attrib
+```
+
+## 60-second demo
+
+```bash
+agent-attrib demo
+```
+
+No LLM calls, no cost. Generates synthetic runs with planted effects and reports which factors are responsible.
+
+## Usage
+
+### 1. Design the experiment
+
+```bash
+agent-attrib plan --factors prompt,model,retriever --tasks 30 --trials 2 -o design.csv
+```
+
+```
+Design type  : 2^3 full factorial
+Resolution   : full
+Factors      : prompt, model, retriever
+Tasks        : 30
+Trials/config: 2
+Configurations: 8
+Total runs   : 480
+Seed         : 42
+MDE (success, α=0.05, power=0.8): 0.1283
+```
+
+For 5-7 factors, a fractional factorial is used automatically (resolution IV by default).
+
+### 2. Run your agent
+
+For each row in `design.csv`, set each factor to the specified level (`baseline` or `candidate`) and run your agent on the task. Write one line per run:
+
+```json
+{"run_id": "r001", "task_id": "t0001", "trial": 1,
+ "factors": {"prompt": "candidate", "model": "baseline", "retriever": "candidate"},
+ "metrics": {"success": 1, "cost_usd": 0.042, "latency_s": 11.3}}
+```
+
+### 3. Analyze
+
+```bash
+agent-attrib analyze runs.jsonl --metric success --metric cost_usd --margin 0.05
+```
+
+## Python API
+
+```python
+from agent_attrib import plan, analyze, load_runs
+
+design = plan(factors=["prompt", "model", "retriever"], tasks=30, trials=2, seed=7)
+print(design.summary())
+
+runset = load_runs("runs.jsonl")
+result = analyze(runset, metrics=["success", "cost_usd"], margin=0.05)
+print(result.to_markdown())
+```
+
+## Input format
+
+**JSONL** (one object per run):
+
+```json
+{"run_id": "r001", "task_id": "t07", "trial": 1,
+ "factors": {"prompt": "candidate", "model": "baseline", "retriever": "candidate"},
+ "metrics": {"success": 1, "cost_usd": 0.042, "latency_s": 11.3}}
+```
+
+**CSV** with columns `run_id, task_id, trial, f_<factor>..., m_<metric>...`:
+
+```csv
+run_id,task_id,trial,f_prompt,f_model,m_success,m_cost_usd
+r001,t07,1,candidate,baseline,1,0.042
+```
+
+Factor levels must be exactly `baseline` or `candidate`. Metrics must be numeric.
+
+## Method
+
+For each metric, `analyze` fits OLS with task fixed effects:
+
+```
+metric ~ task fixed effects + main effects + estimable two-factor interactions
+```
+
+- **Binary `success`:** linear probability model. Effects are in percentage points.
+  Not logistic regression: effects stay directly interpretable as probability differences.
+- **Cost and latency:** `log(metric)` model. Effects reported as percent change.
+- **Standard errors:** cluster-robust, clustered by task.
+- **Multiple testing:** Holm step-down correction across all effects within a metric.
+- **Verdicts** (given an equivalence margin `m`):
+  - `IMPROVEMENT` -- 95% CI sits entirely above +m
+  - `REGRESSION` -- 95% CI sits entirely below -m
+  - `EQUIVALENT` -- 95% CI lies within +/-m
+  - `INCONCLUSIVE` -- CI straddles a boundary; includes approximate tasks needed to resolve
+- Effect convention: switching a factor from baseline to candidate (= 2*beta under +/-1 coding).
+
+## Designs
+
+| Factors | Design | Runs (30 tasks, 2 trials) |
+|---------|--------|--------------------------|
+| 1-4 | 2^k full factorial | 60-480 |
+| 5 | 2^(5-1) res IV | 960 |
+| 6 | 2^(6-2) res IV | 960 |
+| 7 | 2^(7-3) res IV | 960 |
+
+Fractional designs print their alias structure. Use `--resolution V` to request resolution V (more runs, no 2FI aliasing).
+
+## Limitations
+
+- Results apply to this task suite and model versions only.
+- Attribution requires designed runs -- not observational production logs.
+- Maximum 7 factors in v0.1.
+- Two levels per factor only (baseline vs candidate).
+- No LLM-as-judge; `agent-attrib` never calls any LLM.
+
+## Prior art
+
+- **agentstats, AgentAssay, deltagate:** proper A/B statistics for a single pair. Do not decompose changes across multiple simultaneous factors.
+- **ablate-cli:** per-section ablation of `CLAUDE.md` files, for coding agents, runs Claude Code itself. Borrowed two ideas: four-way verdicts and printing the minimum detectable effect before any spend.
+- **Research papers:** factorial ablations over agent components exist as one-off scripts.
+
+`agent-attrib`'s niche: framework-agnostic, multi-factor, interaction-aware attribution for arbitrary release changes, decoupled from execution.
+
+## Development
+
+```bash
+git clone https://github.com/agent-attrib/agent-attrib
+cd agent-attrib
+uv sync --all-groups
+uv run pytest -m "not slow"          # fast tests (~seconds)
+uv run pytest                        # all tests including calibration (~minutes)
+uv run ruff check src tests
+uv run mypy src
+```
+
+## License
+
+MIT

agent_attrib-0.1.0/pyproject.toml

@@ -0,0 +1,97 @@
+[project]
+name = "agent-attrib"
+version = "0.1.0"
+description = "Multi-factor attribution for agent release changes using factorial design of experiments"
+readme = "README.md"
+requires-python = ">=3.11"
+keywords = [
+    "agents",
+    "llm",
+    "attribution",
+    "factorial",
+    "design-of-experiments",
+    "statistics",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = [
+    "numpy>=1.26",
+    "pandas>=2.1",
+    "scipy>=1.11",
+    "statsmodels>=0.14",
+    "pydantic>=2.5",
+    "typer>=0.9",
+    "rich>=13.7",
+]
+
+[project.license]
+text = "MIT"
+
+[[project.authors]]
+name = "agent-attrib contributors"
+
+[project.scripts]
+agent-attrib = "agent_attrib.cli:app"
+
+[project.urls]
+Homepage = "https://github.com/agent-attrib/agent-attrib"
+Issues = "https://github.com/agent-attrib/agent-attrib/issues"
+
+[build-system]
+requires = ["uv_build>=0.13.7,<0.14.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=5.0",
+    "hypothesis>=6.100",
+    "ruff>=0.4",
+    "mypy>=1.10",
+    "pandas-stubs>=2.1",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+    "B",
+    "SIM",
+]
+ignore = ["E501"]
+
+[tool.mypy]
+python_version = "3.11"
+strict = true
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = ["slow: marks tests as slow (deselect with -m 'not slow')"]
+addopts = "--tb=short"
+
+[tool.coverage.run]
+source = ["src/agent_attrib"]
+omit = ["*/cli.py"]
+
+[tool.coverage.report]
+exclude_lines = [
+    "pragma: no cover",
+    "if TYPE_CHECKING:",
+]

agent_attrib-0.1.0/pyproject.toml.orig

@@ -0,0 +1,76 @@
+[project]
+name = "agent-attrib"
+version = "0.1.0"
+description = "Multi-factor attribution for agent release changes using factorial design of experiments"
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "agent-attrib contributors" }]
+requires-python = ">=3.11"
+keywords = ["agents", "llm", "attribution", "factorial", "design-of-experiments", "statistics"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = [
+    "numpy>=1.26",
+    "pandas>=2.1",
+    "scipy>=1.11",
+    "statsmodels>=0.14",
+    "pydantic>=2.5",
+    "typer>=0.9",
+    "rich>=13.7",
+]
+
+[project.scripts]
+agent-attrib = "agent_attrib.cli:app"
+
+[project.urls]
+Homepage = "https://github.com/agent-attrib/agent-attrib"
+Issues = "https://github.com/agent-attrib/agent-attrib/issues"
+
+[build-system]
+requires = ["uv_build>=0.13.7,<0.14.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=5.0",
+    "hypothesis>=6.100",
+    "ruff>=0.4",
+    "mypy>=1.10",
+    "pandas-stubs>=2.1",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+ignore = ["E501"]
+
+[tool.mypy]
+python_version = "3.11"
+strict = true
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = ["slow: marks tests as slow (deselect with -m 'not slow')"]
+addopts = "--tb=short"
+
+[tool.coverage.run]
+source = ["src/agent_attrib"]
+omit = ["*/cli.py"]
+
+[tool.coverage.report]
+exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]

agent_attrib-0.1.0/src/agent_attrib/__init__.py

@@ -0,0 +1,124 @@
+"""agent-attrib: multi-factor attribution for agent release changes."""
+
+from .model import MetricResult, analyze_runset
+from .schema import Design, Run, RunSet, load_runs
+
+__version__ = "0.1.0"
+__all__ = [
+    "plan",
+    "analyze",
+    "load_runs",
+    "RunSet",
+    "Run",
+    "Design",
+    "MetricResult",
+    "__version__",
+]
+
+
+def plan(
+    factors: list[str],
+    tasks: int = 30,
+    trials: int = 2,
+    seed: int = 42,
+    resolution: str = "IV",
+    sd: float | None = None,
+) -> "Design":
+    """Generate a factorial experiment design.
+
+    Args:
+        factors: list of factor names.
+        tasks: number of distinct tasks.
+        trials: trials per configuration per task.
+        seed: random seed for run-order randomization.
+        resolution: minimum design resolution for fractional designs ("IV" or "V").
+        sd: assumed SD for MDE calculation on continuous metrics.
+
+    Returns:
+        Design object with run order and summary statistics.
+    """
+    from .design import build_design, generate_run_order
+    from .power import mde_binary, mde_continuous
+    from .schema import Design, DesignRow
+
+    dm_matrix, dm = build_design(factors, tasks=tasks, trials=trials, seed=seed, resolution=resolution)
+    task_ids = [f"t{i:04d}" for i in range(tasks)]
+    df = generate_run_order(dm_matrix, factors, task_ids, trials, seed)
+
+    n_configs = dm_matrix.shape[0]
+    n_total = n_configs * tasks * trials
+
+    mde_val = mde_continuous(n_configs, tasks, trials, sd=sd) if sd else mde_binary(n_configs, tasks, trials)
+
+    rows: list[DesignRow] = []
+    for _, row in df.iterrows():
+        factor_levels = {f: str(row[f"f_{f}"]) for f in factors}
+        rows.append(
+            DesignRow(
+                config_id=str(row["config_id"]),
+                task_id=str(row["task_id"]),
+                trial=int(row["trial"]),
+                run_order=int(row["run_order"]),
+                factors=factor_levels,  # type: ignore[arg-type]
+            )
+        )
+
+    return Design(
+        factors=factors,
+        tasks=tasks,
+        trials=trials,
+        seed=seed,
+        design_type=dm.design_type,
+        resolution=dm.resolution,
+        alias_structure=dm.alias_structure,
+        runs=rows,
+        n_configs=n_configs,
+        n_total=n_total,
+        mde=mde_val,
+        mde_metric="success (binary, p=0.5)" if sd is None else "continuous",
+    )
+
+
+def analyze(
+    runset: "RunSet",
+    metrics: list[str] | None = None,
+    margin: float = 0.05,
+    binary_metrics: list[str] | None = None,
+) -> "AnalysisResult":
+    """Analyze a RunSet and return an AnalysisResult.
+
+    Args:
+        runset: loaded runs from load_runs().
+        metrics: metrics to analyze (default: all in runset).
+        margin: equivalence margin for EQUIVALENT/INCONCLUSIVE verdicts.
+        binary_metrics: metric names to treat as binary (default: ["success"]).
+
+    Returns:
+        AnalysisResult with per-metric results and reporting methods.
+    """
+    metrics = metrics or runset.metrics
+    results = analyze_runset(runset, metrics=metrics, margin=margin, binary_metrics=binary_metrics)
+    return AnalysisResult(results)
+
+
+class AnalysisResult:
+    """Container for per-metric analysis results with reporting methods."""
+
+    def __init__(self, results: list["MetricResult"]) -> None:
+        self.results = results
+
+    def to_markdown(self) -> str:
+        from .report import to_markdown
+        return to_markdown(self.results)
+
+    def to_json(self) -> str:
+        from .report import to_json
+        return to_json(self.results)
+
+    def print(self) -> None:
+        from .report import print_rich
+        print_rich(self.results)
+
+    def __repr__(self) -> str:
+        metrics = [r.metric for r in self.results]
+        return f"AnalysisResult(metrics={metrics})"