codeprobe 0.2.0__tar.gz → 0.2.1__tar.gz

codeprobe-0.2.1/PKG-INFO

@@ -0,0 +1,224 @@
+Metadata-Version: 2.4
+Name: codeprobe
+Version: 0.2.1
+Summary: Benchmark AI coding agents against your own codebase. Mine real tasks from repo history, run agents, interpret results.
+Author: codeprobe contributors
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/sjarmak/codeprobe
+Project-URL: Repository, https://github.com/sjarmak/codeprobe
+Project-URL: Issues, https://github.com/sjarmak/codeprobe/issues
+Keywords: ai,benchmark,eval,coding-agent,mcp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click<9,>=8.0
+Requires-Dist: pyyaml<7,>=6.0
+Requires-Dist: anthropic>=0.39
+Requires-Dist: openai>=1.66
+Requires-Dist: tiktoken<1,>=0.7
+Requires-Dist: scipy<2,>=1.11
+Provides-Extra: dev
+Requires-Dist: pytest<9,>=8.0; extra == "dev"
+Requires-Dist: pytest-cov<6,>=5.0; extra == "dev"
+Requires-Dist: ruff<1,>=0.4; extra == "dev"
+Requires-Dist: mypy<2,>=1.10; extra == "dev"
+Requires-Dist: types-PyYAML<7,>=6.0; extra == "dev"
+Requires-Dist: scipy<2,>=1.11; extra == "dev"
+Dynamic: license-file
+
+# codeprobe
+
+Benchmark AI coding agents against **your own codebase**.
+
+Mine real tasks from your repo history, run agents against them, and find out which setup actually works best for YOUR code — not someone else's benchmark suite.
+
+## Why codeprobe?
+
+Existing benchmarks (SWE-bench, HumanEval) use fixed task sets that AI models may have memorized from training data. codeprobe mines tasks from **your private repo history**, producing benchmarks that are impossible to contaminate.
+
+## Prerequisites
+
+codeprobe orchestrates external AI coding agents — you need at least one installed:
+
+| Agent              | Install                                          | Required env var                |
+| ------------------ | ------------------------------------------------ | ------------------------------- |
+| **Claude Code**    | [claude.ai/download](https://claude.ai/download) | `ANTHROPIC_API_KEY`             |
+| **GitHub Copilot** | `npm install -g @github/copilot-cli` (>= 1.0.4)  | GitHub auth via `gh auth login` |
+| **Codex**          | Included via `pip install codeprobe[codex]`      | `OPENAI_API_KEY`                |
+
+You also need:
+
+- **Python 3.11+**
+- **Git** (for task mining and worktree isolation)
+- **GitHub CLI** (`gh`) — optional, for mining tasks from GitHub PRs with linked issues
+
+The `assess` and `mine --enrich` commands need an LLM for scoring/enrichment. codeprobe auto-detects the best available backend:
+
+| Priority | Backend       | Install                                          | Env var             |
+| -------- | ------------- | ------------------------------------------------ | ------------------- |
+| 1        | Anthropic SDK | `pip install codeprobe[anthropic]`               | `ANTHROPIC_API_KEY` |
+| 2        | OpenAI SDK    | `pip install codeprobe[codex]`                   | `OPENAI_API_KEY`    |
+| 3        | Claude CLI    | [claude.ai/download](https://claude.ai/download) | `ANTHROPIC_API_KEY` |
+
+Override with `CODEPROBE_LLM_BACKEND=anthropic|openai|claude-cli`. Without any backend, `assess` falls back to heuristic scoring.
+
+## Quick Start
+
+```bash
+pip install codeprobe
+
+cd /path/to/your/repo
+
+codeprobe assess .      # Score benchmarking potential (optional)
+codeprobe mine .        # Extract tasks from repo history
+codeprobe run .         # Run agents against tasks
+codeprobe interpret .   # Get recommendations
+```
+
+## Commands
+
+| Command                  | Purpose                                          |
+| ------------------------ | ------------------------------------------------ |
+| `codeprobe assess`       | Score a codebase's benchmarking potential        |
+| `codeprobe init`         | Interactive wizard — choose what to compare      |
+| `codeprobe mine`         | Mine eval tasks from merged PRs/MRs              |
+| `codeprobe probe`        | Generate fast micro-benchmark probes (30s each)  |
+| `codeprobe experiment`   | Manage comparison experiments (init, add-config) |
+| `codeprobe run`          | Execute tasks against AI agents                  |
+| `codeprobe interpret`    | Analyze results, rank configurations             |
+| `codeprobe oracle-check` | Compare agent answer against oracle ground truth |
+| `codeprobe scaffold`     | Create/validate eval task directories            |
+| `codeprobe ratings`      | Record and analyze agent session quality ratings |
+
+## Two Ways to Generate Tasks
+
+### 1. SDLC Tasks (from merged PRs)
+
+Mine real code-change tasks from your git history. Agents must reproduce known fixes and features.
+
+```bash
+codeprobe mine . --count 10 --source github
+codeprobe mine . --count 5 --min-files 4    # Harder tasks (more files changed)
+codeprobe mine . --enrich                    # LLM-enriched instructions
+```
+
+### 2. Micro-Benchmark Probes
+
+Fast exact-match tasks (30s each) that test code navigation and comprehension — no agent sandbox needed.
+
+```bash
+codeprobe probe . -n 10 -l python -s 42 -o ./probes
+```
+
+Generates four probe types: find-function, count-callers, return-type, module-dependency.
+
+## MCP Comparison Experiments
+
+Compare agent performance with and without MCP tools (Sourcegraph, GitHub, etc.).
+
+### Mine org-scale comprehension tasks
+
+```bash
+# Set up Sourcegraph credentials
+export SOURCEGRAPH_TOKEN="your-token"
+
+# Mine MCP-optimized tasks with Sourcegraph ground truth enrichment
+codeprobe mine /path/to/repo \
+  --org-scale --mcp-families --count 5 \
+  --no-interactive --no-llm \
+  --sg-repo github.com/sg-evals/your-repo
+```
+
+MCP task families: `symbol-reference-trace`, `type-hierarchy-consumers`, `change-scope-audit`.
+
+### Set up the experiment
+
+```bash
+# Create experiment
+codeprobe experiment init /path/to/repo --name mcp-comparison
+
+# Copy mined tasks into the experiment
+cp -r /path/to/repo/.codeprobe/tasks/* /path/to/repo/mcp-comparison/tasks/
+
+# Baseline config (no MCP, no preamble)
+codeprobe experiment add-config /path/to/repo/mcp-comparison \
+  --label baseline --agent claude --model claude-haiku-4-5-20251001
+
+# Sourcegraph MCP config (preamble + MCP server)
+codeprobe experiment add-config /path/to/repo/mcp-comparison \
+  --label with-sourcegraph --agent claude --model claude-haiku-4-5-20251001 \
+  --preamble sourcegraph \
+  --mcp-config '{"mcpServers":{"sourcegraph":{"type":"http","url":"https://sourcegraph.com/.api/mcp/v1","headers":{"Authorization":"token $SOURCEGRAPH_TOKEN"}}}}'
+
+# Run and interpret
+codeprobe run /path/to/repo/mcp-comparison --agent claude --max-cost-usd 5.00
+codeprobe interpret /path/to/repo/mcp-comparison
+```
+
+### Preambles
+
+Preambles are composable instruction templates prepended to the agent's prompt for MCP-enabled configs. Built-in preambles: `sourcegraph`, `github`.
+
+Override built-ins by placing a `.md` file in:
+
+- `<task_dir>/preambles/` (per-task)
+- `.codeprobe/preambles/` (project-level)
+- `~/.codeprobe/preambles/` (user-level)
+
+Template variables: `{{sg_repo}}`, `{{repo_name}}`, `{{repo_path}}`, `{{task_id}}`
+
+## Key Flags
+
+```bash
+# Running
+codeprobe run . --parallel 5          # Run 5 tasks concurrently (worktree-isolated)
+codeprobe run . --max-cost-usd 2.00   # Stop when cost budget is reached
+codeprobe run . --dry-run             # Estimate resource usage without running
+
+# Mining
+codeprobe mine . --enrich             # Use LLM to improve weak task instructions
+codeprobe mine . --org-scale          # Mine comprehension tasks (not SDLC)
+codeprobe mine . --mcp-families       # Include MCP-optimized task families
+codeprobe mine . --sg-repo REPO       # Sourcegraph repo for ground truth enrichment
+
+# Experiment configs
+codeprobe experiment add-config . --preamble sourcegraph  # Attach MCP preamble
+codeprobe experiment add-config . --mcp-config config.json  # Attach MCP server
+
+# Output
+codeprobe interpret . --format csv    # Export for pivot tables
+codeprobe interpret . --format html   # Self-contained HTML report
+```
+
+## Supported Agents
+
+- **Claude Code** (`--agent claude`) — headless via `claude -p`
+- **GitHub Copilot** (`--agent copilot`) — via Copilot CLI
+- **Codex** (`--agent codex`) — via OpenAI API
+- Custom agents via the `AgentAdapter` protocol
+
+## Supported Git Hosts
+
+GitHub, GitLab, Bitbucket, Azure DevOps, Gitea/Forgejo, and local repos.
+
+## Configuration
+
+Create a `.evalrc.yaml` in your repo root:
+
+```yaml
+name: my-experiment
+agents: [claude, copilot]
+models: [claude-sonnet-4-6, claude-opus-4-6]
+tasks_dir: .codeprobe/tasks
+```
+
+## License
+
+Apache-2.0

codeprobe-0.2.1/README.md

@@ -0,0 +1,189 @@
+# codeprobe
+
+Benchmark AI coding agents against **your own codebase**.
+
+Mine real tasks from your repo history, run agents against them, and find out which setup actually works best for YOUR code — not someone else's benchmark suite.
+
+## Why codeprobe?
+
+Existing benchmarks (SWE-bench, HumanEval) use fixed task sets that AI models may have memorized from training data. codeprobe mines tasks from **your private repo history**, producing benchmarks that are impossible to contaminate.
+
+## Prerequisites
+
+codeprobe orchestrates external AI coding agents — you need at least one installed:
+
+| Agent              | Install                                          | Required env var                |
+| ------------------ | ------------------------------------------------ | ------------------------------- |
+| **Claude Code**    | [claude.ai/download](https://claude.ai/download) | `ANTHROPIC_API_KEY`             |
+| **GitHub Copilot** | `npm install -g @github/copilot-cli` (>= 1.0.4)  | GitHub auth via `gh auth login` |
+| **Codex**          | Included via `pip install codeprobe[codex]`      | `OPENAI_API_KEY`                |
+
+You also need:
+
+- **Python 3.11+**
+- **Git** (for task mining and worktree isolation)
+- **GitHub CLI** (`gh`) — optional, for mining tasks from GitHub PRs with linked issues
+
+The `assess` and `mine --enrich` commands need an LLM for scoring/enrichment. codeprobe auto-detects the best available backend:
+
+| Priority | Backend       | Install                                          | Env var             |
+| -------- | ------------- | ------------------------------------------------ | ------------------- |
+| 1        | Anthropic SDK | `pip install codeprobe[anthropic]`               | `ANTHROPIC_API_KEY` |
+| 2        | OpenAI SDK    | `pip install codeprobe[codex]`                   | `OPENAI_API_KEY`    |
+| 3        | Claude CLI    | [claude.ai/download](https://claude.ai/download) | `ANTHROPIC_API_KEY` |
+
+Override with `CODEPROBE_LLM_BACKEND=anthropic|openai|claude-cli`. Without any backend, `assess` falls back to heuristic scoring.
+
+## Quick Start
+
+```bash
+pip install codeprobe
+
+cd /path/to/your/repo
+
+codeprobe assess .      # Score benchmarking potential (optional)
+codeprobe mine .        # Extract tasks from repo history
+codeprobe run .         # Run agents against tasks
+codeprobe interpret .   # Get recommendations
+```
+
+## Commands
+
+| Command                  | Purpose                                          |
+| ------------------------ | ------------------------------------------------ |
+| `codeprobe assess`       | Score a codebase's benchmarking potential        |
+| `codeprobe init`         | Interactive wizard — choose what to compare      |
+| `codeprobe mine`         | Mine eval tasks from merged PRs/MRs              |
+| `codeprobe probe`        | Generate fast micro-benchmark probes (30s each)  |
+| `codeprobe experiment`   | Manage comparison experiments (init, add-config) |
+| `codeprobe run`          | Execute tasks against AI agents                  |
+| `codeprobe interpret`    | Analyze results, rank configurations             |
+| `codeprobe oracle-check` | Compare agent answer against oracle ground truth |
+| `codeprobe scaffold`     | Create/validate eval task directories            |
+| `codeprobe ratings`      | Record and analyze agent session quality ratings |
+
+## Two Ways to Generate Tasks
+
+### 1. SDLC Tasks (from merged PRs)
+
+Mine real code-change tasks from your git history. Agents must reproduce known fixes and features.
+
+```bash
+codeprobe mine . --count 10 --source github
+codeprobe mine . --count 5 --min-files 4    # Harder tasks (more files changed)
+codeprobe mine . --enrich                    # LLM-enriched instructions
+```
+
+### 2. Micro-Benchmark Probes
+
+Fast exact-match tasks (30s each) that test code navigation and comprehension — no agent sandbox needed.
+
+```bash
+codeprobe probe . -n 10 -l python -s 42 -o ./probes
+```
+
+Generates four probe types: find-function, count-callers, return-type, module-dependency.
+
+## MCP Comparison Experiments
+
+Compare agent performance with and without MCP tools (Sourcegraph, GitHub, etc.).
+
+### Mine org-scale comprehension tasks
+
+```bash
+# Set up Sourcegraph credentials
+export SOURCEGRAPH_TOKEN="your-token"
+
+# Mine MCP-optimized tasks with Sourcegraph ground truth enrichment
+codeprobe mine /path/to/repo \
+  --org-scale --mcp-families --count 5 \
+  --no-interactive --no-llm \
+  --sg-repo github.com/sg-evals/your-repo
+```
+
+MCP task families: `symbol-reference-trace`, `type-hierarchy-consumers`, `change-scope-audit`.
+
+### Set up the experiment
+
+```bash
+# Create experiment
+codeprobe experiment init /path/to/repo --name mcp-comparison
+
+# Copy mined tasks into the experiment
+cp -r /path/to/repo/.codeprobe/tasks/* /path/to/repo/mcp-comparison/tasks/
+
+# Baseline config (no MCP, no preamble)
+codeprobe experiment add-config /path/to/repo/mcp-comparison \
+  --label baseline --agent claude --model claude-haiku-4-5-20251001
+
+# Sourcegraph MCP config (preamble + MCP server)
+codeprobe experiment add-config /path/to/repo/mcp-comparison \
+  --label with-sourcegraph --agent claude --model claude-haiku-4-5-20251001 \
+  --preamble sourcegraph \
+  --mcp-config '{"mcpServers":{"sourcegraph":{"type":"http","url":"https://sourcegraph.com/.api/mcp/v1","headers":{"Authorization":"token $SOURCEGRAPH_TOKEN"}}}}'
+
+# Run and interpret
+codeprobe run /path/to/repo/mcp-comparison --agent claude --max-cost-usd 5.00
+codeprobe interpret /path/to/repo/mcp-comparison
+```
+
+### Preambles
+
+Preambles are composable instruction templates prepended to the agent's prompt for MCP-enabled configs. Built-in preambles: `sourcegraph`, `github`.
+
+Override built-ins by placing a `.md` file in:
+
+- `<task_dir>/preambles/` (per-task)
+- `.codeprobe/preambles/` (project-level)
+- `~/.codeprobe/preambles/` (user-level)
+
+Template variables: `{{sg_repo}}`, `{{repo_name}}`, `{{repo_path}}`, `{{task_id}}`
+
+## Key Flags
+
+```bash
+# Running
+codeprobe run . --parallel 5          # Run 5 tasks concurrently (worktree-isolated)
+codeprobe run . --max-cost-usd 2.00   # Stop when cost budget is reached
+codeprobe run . --dry-run             # Estimate resource usage without running
+
+# Mining
+codeprobe mine . --enrich             # Use LLM to improve weak task instructions
+codeprobe mine . --org-scale          # Mine comprehension tasks (not SDLC)
+codeprobe mine . --mcp-families       # Include MCP-optimized task families
+codeprobe mine . --sg-repo REPO       # Sourcegraph repo for ground truth enrichment
+
+# Experiment configs
+codeprobe experiment add-config . --preamble sourcegraph  # Attach MCP preamble
+codeprobe experiment add-config . --mcp-config config.json  # Attach MCP server
+
+# Output
+codeprobe interpret . --format csv    # Export for pivot tables
+codeprobe interpret . --format html   # Self-contained HTML report
+```
+
+## Supported Agents
+
+- **Claude Code** (`--agent claude`) — headless via `claude -p`
+- **GitHub Copilot** (`--agent copilot`) — via Copilot CLI
+- **Codex** (`--agent codex`) — via OpenAI API
+- Custom agents via the `AgentAdapter` protocol
+
+## Supported Git Hosts
+
+GitHub, GitLab, Bitbucket, Azure DevOps, Gitea/Forgejo, and local repos.
+
+## Configuration
+
+Create a `.evalrc.yaml` in your repo root:
+
+```yaml
+name: my-experiment
+agents: [claude, copilot]
+models: [claude-sonnet-4-6, claude-opus-4-6]
+tasks_dir: .codeprobe/tasks
+```
+
+## License
+
+Apache-2.0

{codeprobe-0.2.0 → codeprobe-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "codeprobe"
-version = "0.2.0"
+version = "0.2.1"
 description = "Benchmark AI coding agents against your own codebase. Mine real tasks from repo history, run agents, interpret results."
 readme = "README.md"
 license = "Apache-2.0"

{codeprobe-0.2.0 → codeprobe-0.2.1}/src/codeprobe/core/executor.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import json as _json
 import logging
+import shutil
 import subprocess
 import threading
 from collections.abc import Callable
@@ -13,7 +14,11 @@ from pathlib import Path
 from typing import TYPE_CHECKING
 
 from codeprobe.core.checkpoint import CheckpointStore
-from codeprobe.core.isolation import IsolationStrategy, WorktreeIsolation
+from codeprobe.core.isolation import (
+    IsolationStrategy,
+    WorktreeIsolation,
+    git_restore_clean,
+)
 from codeprobe.core.preamble import PreambleResolver, _base_prompt, compose_instruction
 from codeprobe.core.scoring import get_scorer, sanitize_secrets
 from codeprobe.models.experiment import CompletedTask, ExperimentConfig
@@ -255,18 +260,25 @@ def execute_task(
     # Done before error checks so partial results from timeouts are scored.
     effective_repo = worktree_path or repo_path
     answer_src = effective_repo / "answer.txt"
+    # Also check the original repo root — agent may have followed
+    # TASK_REPO_ROOT from the instruction (which points to the real repo
+    # when worktree_path wasn't rewritten in older instructions).
+    answer_fallback = repo_path / "answer.txt" if worktree_path else None
+    found_answer = None
     if answer_src.is_file():
+        found_answer = answer_src
+    elif answer_fallback is not None and answer_fallback.is_file():
+        found_answer = answer_fallback
+    if found_answer is not None:
         try:
-            import shutil
-
-            shutil.copy2(answer_src, task_dir / "answer.txt")
+            shutil.copy2(found_answer, task_dir / "answer.txt")
         except OSError:
             pass  # Non-fatal; scorer will report missing answer
 
     # If the agent failed with no output AND no answer.txt was produced,
     # return an error. But if answer.txt exists (e.g. agent timed out
     # after writing it), fall through to scoring.
-    has_answer = (task_dir / "answer.txt").is_file()
+    has_answer = found_answer is not None
     if output.exit_code != 0 and not output.stdout.strip() and not has_answer:
         error_msg = output.stderr or f"Agent exited with code {output.exit_code}"
         return TaskResult(
@@ -325,22 +337,11 @@ _BILLABLE_COST_MODELS = frozenset({"per_token"})
 def _git_reset_workdir(repo_path: Path) -> None:
     """Reset the working directory to a clean state between sequential tasks.
 
-    Runs ``git checkout -- .`` and ``git clean -fd`` to discard modifications
+    Runs ``git restore .`` and ``git clean -fd`` to discard modifications
     and remove untracked files so task N's leftovers don't corrupt task N+1.
     """
     try:
-        subprocess.run(
-            ["git", "checkout", "--", "."],
-            cwd=repo_path,
-            check=True,
-            capture_output=True,
-        )
-        subprocess.run(
-            ["git", "clean", "-fd", "-e", ".codeprobe", "-e", ".codeprobe-worktrees"],
-            cwd=repo_path,
-            check=True,
-            capture_output=True,
-        )
+        git_restore_clean(repo_path)
     except subprocess.CalledProcessError as exc:
         logger.warning(
             "Git reset failed (exit %d): %s",

{codeprobe-0.2.0 → codeprobe-0.2.1}/src/codeprobe/core/isolation.py

@@ -12,6 +12,38 @@ from typing import Protocol, runtime_checkable
 logger = logging.getLogger(__name__)
 
 
+def git_restore_clean(workdir: Path, *, extra_excludes: tuple[str, ...] = ()) -> None:
+    """Restore tracked files and remove untracked files in *workdir*.
+
+    Uses ``git restore .`` (tolerant of empty diffs) followed by
+    ``git clean -fd``.  Always excludes ``.codeprobe`` and
+    ``.codeprobe-worktrees``; pass *extra_excludes* for more.
+    """
+    result = subprocess.run(
+        ["git", "restore", "."],
+        cwd=workdir,
+        capture_output=True,
+    )
+    if result.returncode != 0:
+        stderr = result.stderr.decode(errors="replace")
+        # "could not resolve HEAD" is expected in a truly empty/detached
+        # worktree — not worth warning about.
+        if "could not resolve" not in stderr:
+            logger.debug("git restore in %s: %s", workdir, stderr)
+    clean_cmd = [
+        "git",
+        "clean",
+        "-fd",
+        "-e",
+        ".codeprobe",
+        "-e",
+        ".codeprobe-worktrees",
+    ]
+    for exc in extra_excludes:
+        clean_cmd += ["-e", exc]
+    subprocess.run(clean_cmd, cwd=workdir, check=True, capture_output=True)
+
+
 @runtime_checkable
 class IsolationStrategy(Protocol):
     """Protocol for workspace isolation strategies."""
@@ -79,26 +111,7 @@ class WorktreeIsolation:
     def reset(self, workspace: Path) -> None:
         """Reset a worktree to clean state."""
         try:
-            subprocess.run(
-                ["git", "checkout", "--", "."],
-                cwd=workspace,
-                check=True,
-                capture_output=True,
-            )
-            subprocess.run(
-                [
-                    "git",
-                    "clean",
-                    "-fd",
-                    "-e",
-                    ".codeprobe",
-                    "-e",
-                    ".codeprobe-worktrees",
-                ],
-                cwd=workspace,
-                check=True,
-                capture_output=True,
-            )
+            git_restore_clean(workspace)
         except subprocess.CalledProcessError as exc:
             logger.warning(
                 "Worktree reset failed for %s (exit %d): %s",

{codeprobe-0.2.0 → codeprobe-0.2.1}/src/codeprobe/core/preamble.py

@@ -88,6 +88,14 @@ def _base_prompt(
     references the worktree instead of the original repo path.
     """
     effective_path = worktree_path if worktree_path is not None else repo_path
+    # Rewrite TASK_REPO_ROOT in the instruction so agents write to the
+    # worktree, not the original repo (avoids cross-task collisions and
+    # ensures answer.txt lands where the executor expects it).
+    if worktree_path is not None:
+        instruction = instruction.replace(
+            f"TASK_REPO_ROOT={repo_path}",
+            f"TASK_REPO_ROOT={worktree_path}",
+        )
     return (
         f"You are working on the repository at {effective_path}. "
         "Follow the instruction below.\n\n"