repobench 0.1.0__tar.gz

repobench-0.1.0/.gitignore

@@ -0,0 +1 @@
+.agentfit/

repobench-0.1.0/Makefile

@@ -0,0 +1,53 @@
+.PHONY: install install-dev build clean publish test lint
+
+# Install for users
+install:
+	pip install .
+
+# Install for development
+install-dev:
+	pip install -e ".[dev]"
+
+# Install with pipx (recommended for CLI tools)
+install-pipx:
+	pipx install .
+
+# Build package
+build: clean
+	python -m build
+
+# Clean build artifacts
+clean:
+	rm -rf dist/ build/ *.egg-info agentfit/*.pyc
+
+# Publish to PyPI (test)
+publish-test: build
+	twine upload --repository testpypi dist/*
+
+# Publish to PyPI (production)
+publish: build
+	twine upload dist/*
+
+# Run tests
+test:
+	pytest tests/ -v
+
+# Run linter
+lint:
+	ruff check agentfit/
+	ruff format agentfit/ --check
+
+# Format code
+format:
+	ruff check agentfit/ --fix
+	ruff format agentfit/
+
+# Type check
+typecheck:
+	mypy agentfit/
+
+# Show package info
+info:
+	@echo "Package: agentfit"
+	@echo "Version: $$(python -c 'from agentfit import __version__; print(__version__)')"
+	@echo "Entry point: agentfit = agentfit.cli.app:app"

repobench-0.1.0/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.5
+Name: repobench
+Version: 0.1.0
+Summary: Living repository-native evals for coding agents
+Project-URL: Homepage, https://github.com/agentfit/agentfit
+Project-URL: Repository, https://github.com/agentfit/agentfit
+Project-URL: Issues, https://github.com/agentfit/agentfit/issues
+Author: AgentFit Contributors
+License-Expression: Apache-2.0
+Keywords: benchmark,coding-agent,eval,harbor,swe-bench
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Natural Language :: English
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.12
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: typer>=0.12.0
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# AgentFit
+
+**Living repository-native evals for coding agents.**
+
+Find which coding agent actually works for your codebase.
+
+## Quick Start
+
+```bash
+# Install
+pipx install agentfit
+
+# Initialize in your repo
+cd my-project
+agentfit init
+
+# Analyze workload
+agentfit analyze
+
+# Build benchmark
+agentfit benchmark build
+
+# Run agents
+agentfit run codex-local claude-local
+
+# Get results
+agentfit report
+```
+
+## What is AgentFit?
+
+AgentFit transforms your repository's real engineering history into a private, reproducible, representative benchmark for coding agents.
+
+Instead of relying on public benchmarks that don't match your work, AgentFit mines your actual merged PRs to create evals that reflect what your team actually ships.
+
+### The Pipeline
+
+```
+Repository history
+        │
+        ▼
+   DISCOVER    What happened here?
+        │
+        ▼
+     FILTER    Can it become an eval?
+        │
+        ▼
+    VALIDATE   Can we prove correctness?
+        │
+        ▼
+  REPRESENT    Does the benchmark match
+               the actual workload?
+        │
+        ▼
+      RUN      Execute agent configurations
+        │
+        ▼
+    MEASURE    Quality / cost / latency
+        │
+        ▼
+    DECIDE     What should we use?
+```
+
+## Commands
+
+| Command | Description |
+|---|---|
+| `agentfit doctor` | Check prerequisites |
+| `agentfit init` | Initialize in current repo |
+| `agentfit analyze` | Analyze repository workload |
+| `agentfit candidates` | View mined candidate tasks |
+| `agentfit task inspect <id>` | Inspect a candidate task |
+| `agentfit benchmark build` | Build representative benchmark |
+| `agentfit benchmark list` | List existing benchmarks |
+| `agentfit run <configs...>` | Run agents via Harbor |
+| `agentfit report` | Generate comparison report |
+| `agentfit config show` | Show current configuration |
+
+## Configuration
+
+AgentFit creates `agentfit.yml` in your repo:
+
+```yaml
+version: 1
+
+repository:
+  provider: github
+  lookback_days: 180
+
+project:
+  languages:
+    - python
+    - go
+    - java
+  install_command: pip install -e ".[dev]"
+  test_command: pytest
+
+benchmark:
+  size: 24
+  dimensions:
+    task_type: 0.30
+    subsystem: 0.40
+    complexity: 0.30
+
+execution:
+  environment: docker
+  concurrency: 4
+
+agents:
+  codex-default:
+    agent: codex
+    model: openai/gpt-4o
+
+  claude-default:
+    agent: claude-code
+    model: anthropic/claude-opus-4
+```
+
+## Supported Languages
+
+- Python
+- JavaScript / TypeScript
+- Go
+- Java (Maven / Gradle)
+
+## How it Works
+
+### 1. Workload Analysis
+
+AgentFit fetches your merged PRs via GitHub API and classifies each by:
+- **Task type**: bugfix, feature, refactor
+- **Subsystem**: payments, auth, frontend, etc.
+- **Complexity**: small, medium, large
+
+### 2. Candidate Mining
+
+From the workload, AgentFit identifies PRs that can become reliable evals:
+- Have linked issues or clear PR descriptions
+- Include test changes (verifier evidence)
+- Are within configurable size bounds
+- Don't require unsupported environments
+
+### 3. Validation Pipeline
+
+Each candidate passes rigorous validation:
+- **Base health**: tests pass before the change
+- **No-op validation**: new tests fail without the fix
+- **Oracle validation**: tests pass with the fix
+- **Determinism**: tests produce consistent results
+- **Leakage protection**: agent can't access gold solution
+
+### 4. Representative Sampling
+
+AgentFit selects benchmark tasks using stratified optimization to match your actual workload distribution across task type, subsystem, and complexity.
+
+### 5. Execution via Harbor
+
+Benchmarks are exported in Harbor format and executed via `harbor run`:
+
+```bash
+agentfit run codex-default claude-default
+```
+
+### 6. Decision Report
+
+```bash
+agentfit report
+```
+
+```
+AGENTFIT REPORT
+────────────────────────────────────────
+
+Repository:    my-org/payments
+Benchmark:     af_b_20260825_a84f
+Tasks:         24
+Health:        87/100
+
+                      Solve    $/Solve
+Codex                  82%       $0.71
+Claude                 86%       $1.52
+
+Quality
+Claude +4pp vs Codex
+95% CI: -7pp → +14pp
+
+No conclusive quality difference.
+
+Recommendation: Codex
+Reason: lowest cost among statistically
+indistinguishable configurations.
+```
+
+## Development
+
+```bash
+# Clone
+git clone https://github.com/agentfit/agentfit.git
+cd agentfit
+
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Build package
+python -m build
+
+# Publish to PyPI
+twine upload dist/*
+```
+
+## License
+
+Apache 2.0

repobench-0.1.0/README.md

@@ -0,0 +1,216 @@
+# AgentFit
+
+**Living repository-native evals for coding agents.**
+
+Find which coding agent actually works for your codebase.
+
+## Quick Start
+
+```bash
+# Install
+pipx install agentfit
+
+# Initialize in your repo
+cd my-project
+agentfit init
+
+# Analyze workload
+agentfit analyze
+
+# Build benchmark
+agentfit benchmark build
+
+# Run agents
+agentfit run codex-local claude-local
+
+# Get results
+agentfit report
+```
+
+## What is AgentFit?
+
+AgentFit transforms your repository's real engineering history into a private, reproducible, representative benchmark for coding agents.
+
+Instead of relying on public benchmarks that don't match your work, AgentFit mines your actual merged PRs to create evals that reflect what your team actually ships.
+
+### The Pipeline
+
+```
+Repository history
+        │
+        ▼
+   DISCOVER    What happened here?
+        │
+        ▼
+     FILTER    Can it become an eval?
+        │
+        ▼
+    VALIDATE   Can we prove correctness?
+        │
+        ▼
+  REPRESENT    Does the benchmark match
+               the actual workload?
+        │
+        ▼
+      RUN      Execute agent configurations
+        │
+        ▼
+    MEASURE    Quality / cost / latency
+        │
+        ▼
+    DECIDE     What should we use?
+```
+
+## Commands
+
+| Command | Description |
+|---|---|
+| `agentfit doctor` | Check prerequisites |
+| `agentfit init` | Initialize in current repo |
+| `agentfit analyze` | Analyze repository workload |
+| `agentfit candidates` | View mined candidate tasks |
+| `agentfit task inspect <id>` | Inspect a candidate task |
+| `agentfit benchmark build` | Build representative benchmark |
+| `agentfit benchmark list` | List existing benchmarks |
+| `agentfit run <configs...>` | Run agents via Harbor |
+| `agentfit report` | Generate comparison report |
+| `agentfit config show` | Show current configuration |
+
+## Configuration
+
+AgentFit creates `agentfit.yml` in your repo:
+
+```yaml
+version: 1
+
+repository:
+  provider: github
+  lookback_days: 180
+
+project:
+  languages:
+    - python
+    - go
+    - java
+  install_command: pip install -e ".[dev]"
+  test_command: pytest
+
+benchmark:
+  size: 24
+  dimensions:
+    task_type: 0.30
+    subsystem: 0.40
+    complexity: 0.30
+
+execution:
+  environment: docker
+  concurrency: 4
+
+agents:
+  codex-default:
+    agent: codex
+    model: openai/gpt-4o
+
+  claude-default:
+    agent: claude-code
+    model: anthropic/claude-opus-4
+```
+
+## Supported Languages
+
+- Python
+- JavaScript / TypeScript
+- Go
+- Java (Maven / Gradle)
+
+## How it Works
+
+### 1. Workload Analysis
+
+AgentFit fetches your merged PRs via GitHub API and classifies each by:
+- **Task type**: bugfix, feature, refactor
+- **Subsystem**: payments, auth, frontend, etc.
+- **Complexity**: small, medium, large
+
+### 2. Candidate Mining
+
+From the workload, AgentFit identifies PRs that can become reliable evals:
+- Have linked issues or clear PR descriptions
+- Include test changes (verifier evidence)
+- Are within configurable size bounds
+- Don't require unsupported environments
+
+### 3. Validation Pipeline
+
+Each candidate passes rigorous validation:
+- **Base health**: tests pass before the change
+- **No-op validation**: new tests fail without the fix
+- **Oracle validation**: tests pass with the fix
+- **Determinism**: tests produce consistent results
+- **Leakage protection**: agent can't access gold solution
+
+### 4. Representative Sampling
+
+AgentFit selects benchmark tasks using stratified optimization to match your actual workload distribution across task type, subsystem, and complexity.
+
+### 5. Execution via Harbor
+
+Benchmarks are exported in Harbor format and executed via `harbor run`:
+
+```bash
+agentfit run codex-default claude-default
+```
+
+### 6. Decision Report
+
+```bash
+agentfit report
+```
+
+```
+AGENTFIT REPORT
+────────────────────────────────────────
+
+Repository:    my-org/payments
+Benchmark:     af_b_20260825_a84f
+Tasks:         24
+Health:        87/100
+
+                      Solve    $/Solve
+Codex                  82%       $0.71
+Claude                 86%       $1.52
+
+Quality
+Claude +4pp vs Codex
+95% CI: -7pp → +14pp
+
+No conclusive quality difference.
+
+Recommendation: Codex
+Reason: lowest cost among statistically
+indistinguishable configurations.
+```
+
+## Development
+
+```bash
+# Clone
+git clone https://github.com/agentfit/agentfit.git
+cd agentfit
+
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Build package
+python -m build
+
+# Publish to PyPI
+twine upload dist/*
+```
+
+## License
+
+Apache 2.0

repobench-0.1.0/agentfit/__init__.py

@@ -0,0 +1,3 @@
+"""AgentFit — Living repository-native evals for coding agents."""
+
+__version__ = "0.1.0"

repobench-0.1.0/agentfit/analysis/metrics.py

@@ -0,0 +1,58 @@
+"""Configuration metrics aggregation from trials."""
+
+from __future__ import annotations
+
+import statistics
+from typing import Iterable
+
+from agentfit.logging import get_logger
+from agentfit.models import ConfigMetrics, Trial
+from agentfit.analysis.statistics import wilson_ci
+
+log = get_logger("analysis.metrics")
+
+
+def compute_config_metrics(trials: Iterable[Trial]) -> ConfigMetrics:
+    """Compute aggregated metrics for one agent configuration."""
+    trials = list(trials)
+    metrics = ConfigMetrics()
+
+    if not trials:
+        return metrics
+
+    solved = [t for t in trials if t.solved]
+    metrics.solved = len(solved)
+    metrics.total = len(trials)
+    metrics.pass_rate = len(solved) / len(trials)
+
+    # Wilson 95% CI
+    ci_lower, ci_upper = wilson_ci(len(solved), len(trials))
+    metrics.ci_lower = ci_lower
+    metrics.ci_upper = ci_upper
+
+    # Economics
+    costs = [t.cost_usd for t in trials if t.cost_usd is not None]
+    if costs:
+        metrics.total_cost = sum(costs)
+        metrics.mean_cost_task = metrics.total_cost / len(costs)
+        if metrics.solved > 0:
+            metrics.cost_per_solve = metrics.total_cost / metrics.solved
+
+    # Efficiency
+    prompt_tokens = [t.prompt_tokens for t in trials if t.prompt_tokens is not None]
+    completion_tokens = [t.completion_tokens for t in trials if t.completion_tokens is not None]
+    metrics.total_prompt_tokens = sum(prompt_tokens)
+    metrics.total_completion_tokens = sum(completion_tokens)
+    if metrics.solved > 0:
+        total_tokens = metrics.total_prompt_tokens + metrics.total_completion_tokens
+        metrics.tokens_per_solve = round(total_tokens / metrics.solved)
+
+    # Performance
+    durations = [t.duration_ms for t in trials if t.duration_ms is not None]
+    if durations:
+        metrics.p50_duration_ms = round(statistics.median(durations))
+        sorted_d = sorted(durations)
+        p90_idx = min(len(sorted_d) - 1, round(0.9 * (len(sorted_d) - 1)))
+        metrics.p90_duration_ms = sorted_d[p90_idx]
+
+    return metrics

repobench-0.1.0/agentfit/analysis/recommendation.py

@@ -0,0 +1,104 @@
+"""Cost-aware recommendation logic."""
+
+from __future__ import annotations
+
+from typing import Iterable
+
+from agentfit.logging import get_logger
+from agentfit.models import ConfigMetrics
+
+log = get_logger("analysis.recommendation")
+
+
+def recommend(
+    metrics_dict: dict[str, ConfigMetrics],
+) -> tuple[str | None, str]:
+    """Recommend a default agent configuration.
+
+    Policy (``cost_effective``):
+    1. Find the configuration with the highest observed pass rate.
+    2. Identify configurations whose quality difference is not
+       statistically conclusive (overlapping Wilson CIs).
+    3. Within that set, choose the lowest cost per verified solve.
+
+    Returns (config_name, reason).
+    """
+    if not metrics_dict:
+        return None, "No configurations with results available."
+
+    # Consider only configs with trials
+    with_results = {name: m for name, m in metrics_dict.items() if m.total > 0}
+    if not with_results:
+        return None, "No configurations have completed trials."
+
+    # 1. Best observed pass rate
+    best_name = max(with_results, key=lambda n: with_results[n].pass_rate)
+    best = with_results[best_name]
+
+    # 2. Statistically indistinguishable set (overlapping Wilson CIs)
+    indistinguishable = []
+    for name, m in with_results.items():
+        if _cis_overlap(best, m):
+            indistinguishable.append(name)
+
+    # 3. Lowest cost per verified solve among indistinguishable
+    with_cost = [
+        (name, with_results[name])
+        for name in indistinguishable
+        if with_results[name].cost_per_solve is not None
+    ]
+
+    if with_cost:
+        recommended = min(with_cost, key=lambda nm: nm[1].cost_per_solve)[0]
+        reason = (
+            f"lowest cost per verified solve among configurations "
+            f"statistically indistinguishable from observed best quality."
+        )
+    else:
+        # No cost data: pick best observed pass rate
+        recommended = best_name
+        reason = "best observed pass rate (no cost data available)."
+
+    log.info("Recommendation: %s (%s)", recommended, reason)
+    return recommended, reason
+
+
+def pareto_frontier(
+    metrics_dict: dict[str, ConfigMetrics],
+) -> list[str]:
+    """Identify configurations on the Pareto frontier.
+
+    A configuration is dominated if another config has >= pass rate AND
+    <= cost per solve (with at least one strict inequality).
+
+    Returns config names on the frontier (highest quality first).
+    """
+    with_cost = {
+        name: m for name, m in metrics_dict.items()
+        if m.total > 0 and m.cost_per_solve is not None
+    }
+    if not with_cost:
+        return list(metrics_dict.keys())
+
+    frontier: list[str] = []
+    for name, m in with_cost.items():
+        dominated = False
+        for other_name, other in with_cost.items():
+            if other_name == name:
+                continue
+            if (other.pass_rate >= m.pass_rate
+                    and other.cost_per_solve <= m.cost_per_solve
+                    and (other.pass_rate > m.pass_rate
+                         or other.cost_per_solve < m.cost_per_solve)):
+                dominated = True
+                break
+        if not dominated:
+            frontier.append(name)
+
+    frontier.sort(key=lambda n: with_cost[n].pass_rate, reverse=True)
+    return frontier
+
+
+def _cis_overlap(a: ConfigMetrics, b: ConfigMetrics) -> bool:
+    """Check if two Wilson CIs overlap (statistically indistinguishable)."""
+    return not (a.ci_upper < b.ci_lower or b.ci_upper < a.ci_lower)