mcpbr 0.1.0__tar.gz

mcpbr-0.1.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,84 @@
+name: Bug Report
+description: Report a bug or unexpected behavior
+title: "[Bug]: "
+labels: ["bug", "triage"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to report a bug! Please fill out the form below.
+
+  - type: textarea
+    id: description
+    attributes:
+      label: Bug Description
+      description: A clear and concise description of what the bug is.
+      placeholder: Describe the bug...
+    validations:
+      required: true
+
+  - type: textarea
+    id: reproduction
+    attributes:
+      label: Steps to Reproduce
+      description: Steps to reproduce the behavior.
+      placeholder: |
+        1. Run `mcpbr ...`
+        2. See error...
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected Behavior
+      description: What you expected to happen.
+      placeholder: What should have happened?
+    validations:
+      required: true
+
+  - type: textarea
+    id: actual
+    attributes:
+      label: Actual Behavior
+      description: What actually happened.
+      placeholder: What actually happened?
+    validations:
+      required: true
+
+  - type: textarea
+    id: environment
+    attributes:
+      label: Environment
+      description: Please provide your environment details.
+      value: |
+        - OS: [e.g., macOS 14.0, Ubuntu 22.04]
+        - Python version: [e.g., 3.11.5]
+        - mcpbr version: [e.g., 0.1.0]
+        - Docker version: [e.g., 24.0.0]
+    validations:
+      required: true
+
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant Logs
+      description: Please copy and paste any relevant log output.
+      render: shell
+
+  - type: textarea
+    id: config
+    attributes:
+      label: Configuration
+      description: If relevant, share your config.yaml (redact API keys!).
+      render: yaml
+
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      options:
+        - label: I have searched existing issues to ensure this is not a duplicate
+          required: true
+        - label: I have redacted any sensitive information (API keys, etc.)
+          required: true

mcpbr-0.1.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Documentation
+    url: https://github.com/greynewell/mcpbr#readme
+    about: Check out the documentation before opening an issue
+  - name: Discussions
+    url: https://github.com/greynewell/mcpbr/discussions
+    about: Ask questions and discuss ideas in GitHub Discussions

mcpbr-0.1.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,68 @@
+name: Feature Request
+description: Suggest a new feature or enhancement
+title: "[Feature]: "
+labels: ["enhancement", "triage"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for suggesting a feature! Please fill out the form below.
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem Statement
+      description: What problem would this feature solve? Is this related to a frustration?
+      placeholder: I'm always frustrated when...
+    validations:
+      required: true
+
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed Solution
+      description: Describe the solution you'd like.
+      placeholder: I would like to be able to...
+    validations:
+      required: true
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives Considered
+      description: Describe any alternative solutions or features you've considered.
+      placeholder: I've also considered...
+
+  - type: dropdown
+    id: area
+    attributes:
+      label: Feature Area
+      description: What area of the project does this feature relate to?
+      options:
+        - CLI
+        - Configuration
+        - Providers (LLM APIs)
+        - Agent Harnesses
+        - Docker/Environment
+        - Evaluation/Testing
+        - Reporting
+        - Documentation
+        - Other
+    validations:
+      required: true
+
+  - type: textarea
+    id: additional
+    attributes:
+      label: Additional Context
+      description: Add any other context, mockups, or screenshots about the feature request.
+
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      options:
+        - label: I have searched existing issues to ensure this is not a duplicate
+          required: true
+        - label: I am willing to help implement this feature
+          required: false

mcpbr-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,42 @@
+## Description
+
+<!-- Describe your changes in detail -->
+
+## Related Issue
+
+<!-- If this PR addresses an issue, link it here -->
+Fixes #
+
+## Type of Change
+
+<!-- Mark the relevant option with an "x" -->
+
+- [ ] Bug fix (non-breaking change that fixes an issue)
+- [ ] New feature (non-breaking change that adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
+- [ ] Documentation update
+- [ ] Refactoring (no functional changes)
+- [ ] CI/CD or tooling changes
+
+## Checklist
+
+<!-- Mark completed items with an "x" -->
+
+- [ ] I have read the [CONTRIBUTING](../CONTRIBUTING.md) guidelines
+- [ ] My code follows the project's code style
+- [ ] I have added tests that prove my fix/feature works
+- [ ] New and existing unit tests pass locally
+- [ ] I have updated the documentation (if applicable)
+- [ ] I have added an entry to CHANGELOG.md (if applicable)
+
+## Testing
+
+<!-- Describe how you tested your changes -->
+
+## Screenshots (if applicable)
+
+<!-- Add screenshots to help explain your changes -->
+
+## Additional Notes
+
+<!-- Any additional information that reviewers should know -->

mcpbr-0.1.0/.github/release-drafter.yml

@@ -0,0 +1,50 @@
+name-template: 'v$RESOLVED_VERSION'
+tag-template: 'v$RESOLVED_VERSION'
+categories:
+  - title: 'New Features'
+    labels:
+      - 'feature'
+      - 'enhancement'
+  - title: 'Bug Fixes'
+    labels:
+      - 'fix'
+      - 'bugfix'
+      - 'bug'
+  - title: 'Documentation'
+    labels:
+      - 'documentation'
+      - 'docs'
+  - title: 'Maintenance'
+    labels:
+      - 'chore'
+      - 'maintenance'
+      - 'dependencies'
+  - title: 'Other Changes'
+    labels:
+      - '*'
+change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
+change-title-escapes: '\<*_&'
+version-resolver:
+  major:
+    labels:
+      - 'major'
+      - 'breaking'
+  minor:
+    labels:
+      - 'minor'
+      - 'feature'
+      - 'enhancement'
+  patch:
+    labels:
+      - 'patch'
+      - 'fix'
+      - 'bugfix'
+  default: patch
+template: |
+  ## Changes
+
+  $CHANGES
+
+  ## Contributors
+
+  $CONTRIBUTORS

mcpbr-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,74 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install ruff
+
+      - name: Run ruff check
+        run: ruff check src/ tests/
+
+      - name: Run ruff format check
+        run: ruff format --check src/ tests/
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run unit tests
+        run: pytest -m "not integration" -v
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install build dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Check distribution
+        run: |
+          pip install twine
+          twine check dist/*

mcpbr-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,51 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install build dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Store distribution packages
+        uses: actions/upload-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+
+  publish-pypi:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/mcpbr
+
+    steps:
+      - name: Download distribution packages
+        uses: actions/download-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

mcpbr-0.1.0/.github/workflows/release-drafter.yml

@@ -0,0 +1,22 @@
+name: Release Drafter
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    types: [opened, reopened, synchronize]
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  update_release_draft:
+    permissions:
+      contents: write
+      pull-requests: write
+    runs-on: ubuntu-latest
+    steps:
+      - uses: release-drafter/release-drafter@v6
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

mcpbr-0.1.0/.gitignore

@@ -0,0 +1,87 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Environments
+.env
+.env.*
+.venv/
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# OS
+.DS_Store
+
+# IDE
+.idea/
+.vscode/
+.cursor/
+*.swp
+*.swo
+*~
+
+# Project specific
+results/
+*.json
+!config/*.json
+config.yaml
+reports/
+.swebench-mcp/
+.logs/
+logs/
+
+# Local secrets for act (GitHub Actions local runner)
+.secrets
+
+# Docker
+docker-build/
+
+# Datasets cache
+.cache/

mcpbr-0.1.0/CHANGELOG.md

@@ -0,0 +1,40 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-01-17
+
+### Added
+
+- Initial release of mcpbr (Model Context Protocol Benchmark Runner)
+- Support for multiple LLM providers:
+  - OpenRouter (multi-provider gateway)
+  - OpenAI (direct API)
+  - Anthropic (direct API)
+  - Google AI / Gemini (direct API)
+- Agent harness implementations:
+  - Claude Code CLI with MCP server support
+  - OpenAI Codex CLI
+  - OpenCode CLI
+  - Gemini CLI
+- Docker-based task isolation for SWE-bench evaluation
+- Baseline agent for comparison (no tools)
+- JSON and Markdown report generation
+- Configurable evaluation parameters via YAML
+- CLI commands: `run`, `init`, `models`, `harnesses`, `providers`
+- Real-time streaming output when using `--verbose` flag with Claude Code harness
+- Tool usage tracking: counts tool calls by name and includes breakdown in results JSON
+- Each streamed output line is prefixed with task instance ID for parallel worker disambiguation
+- **In-container agent execution**: The Claude Code CLI now runs inside the Docker container where all dependencies are installed. This ensures Python imports work correctly (e.g., `from astropy import ...`) and the agent can verify fixes by running tests.
+- Pre-built SWE-bench Docker images from Epoch AI's registry (`ghcr.io/epoch-research/swe-bench.eval`) are now used when available, providing:
+  - Repository at the correct commit with all dependencies pre-installed
+  - Consistent, reproducible evaluation environments
+  - x86_64 images with automatic emulation on ARM64 (Apple Silicon) Macs
+- Timestamped log filenames to prevent overwrites: `{instance_id}_{run_type}_{YYYYMMDD_HHMMSS}.json`
+- `--no-prebuilt` CLI flag to disable pre-built images and build from scratch
+- Network access for containers to enable API calls from within Docker
+
+[0.1.0]: https://github.com/greynewell/mcpbr/releases/tag/v0.1.0

mcpbr-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,135 @@
+# Contributing to mcpbr
+
+Thank you for your interest in contributing to mcpbr! This document provides guidelines and information for contributors.
+
+## How to Contribute
+
+### Reporting Bugs
+
+Before creating a bug report, please check the existing issues to avoid duplicates. When creating a bug report, include:
+
+- A clear, descriptive title
+- Steps to reproduce the behavior
+- Expected behavior
+- Actual behavior
+- Your environment (OS, Python version, Docker version)
+- Any relevant logs or error messages
+
+### Suggesting Features
+
+Feature requests are welcome! Please:
+
+- Check existing issues and discussions first
+- Clearly describe the feature and its use case
+- Explain why this feature would be useful to most users
+
+### Pull Requests
+
+1. Fork the repository
+2. Create a feature branch from `main`
+3. Make your changes
+4. Add or update tests as needed
+5. Ensure all tests pass
+6. Update documentation if needed
+7. Submit a pull request
+
+## Development Setup
+
+### Prerequisites
+
+- Python 3.11+
+- Docker
+- An API key for at least one supported provider (OpenRouter, OpenAI, Anthropic, or Google)
+
+### Installation
+
+```bash
+# Clone your fork
+git clone https://github.com/greynewell/mcpbr.git
+cd mcpbr
+
+# Create a virtual environment
+python -m venv venv
+source venv/bin/activate  # or `venv\Scripts\activate` on Windows
+
+# Install in development mode with dev dependencies
+pip install -e ".[dev]"
+```
+
+### Running Tests
+
+```bash
+# Run unit tests only (no Docker or API keys required)
+pytest -m "not integration"
+
+# Run all tests (requires Docker and API keys)
+pytest
+
+# Run with coverage
+pytest --cov=mcpbr
+```
+
+### Code Style
+
+This project uses:
+
+- [ruff](https://github.com/astral-sh/ruff) for linting and formatting
+- Type hints throughout the codebase
+
+Before submitting a PR:
+
+```bash
+# Run linter
+ruff check src/ tests/
+
+# Auto-fix issues
+ruff check --fix src/ tests/
+
+# Format code
+ruff format src/ tests/
+```
+
+### Commit Messages
+
+- Use clear, descriptive commit messages
+- Start with a verb in the imperative mood (e.g., "Add", "Fix", "Update")
+- Keep the first line under 72 characters
+- Reference issues when applicable (e.g., "Fix #123")
+
+## Project Structure
+
+```
+mcpbr/
+├── src/mcpbr/          # Main package
+│   ├── cli.py          # CLI commands
+│   ├── config.py       # Configuration models
+│   ├── harness.py      # Main orchestrator
+│   ├── harnesses.py    # Agent implementations
+│   ├── providers.py    # LLM provider abstractions
+│   └── ...
+├── tests/              # Test suite
+├── config/             # Example configurations
+└── docs/               # Documentation
+```
+
+## Adding New Features
+
+### Adding a New Provider
+
+1. Create a new class in `src/mcpbr/providers.py` implementing the `ModelProvider` protocol
+2. Add it to `PROVIDER_REGISTRY`
+3. Update `VALID_PROVIDERS` in `config.py`
+4. Add tests
+5. Update documentation
+
+### Adding a New Agent Harness
+
+1. Create a new class in `src/mcpbr/harnesses.py` implementing the `AgentHarness` protocol
+2. Add it to `HARNESS_REGISTRY`
+3. Update `VALID_HARNESSES` in `config.py`
+4. Add tests
+5. Update documentation
+
+## Questions?
+
+Feel free to open an issue for any questions about contributing.

mcpbr-0.1.0/Dockerfile

@@ -0,0 +1,21 @@
+# Base image for SWE-bench task environments
+FROM python:3.11-slim
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    git \
+    curl \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set up workspace
+WORKDIR /workspace
+
+# Install common Python testing tools
+RUN pip install --no-cache-dir \
+    pytest \
+    pytest-xdist \
+    coverage
+
+# Default command
+CMD ["/bin/bash"]