mcpbr-cli 0.3.28 → 0.4.1

package/README.md

@@ -1,11 +1,11 @@
 # mcpbr
 
 ```bash
-# Install via pip
-pip install mcpbr && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
+# One-liner install (installs + runs quick test)
+curl -sSL https://raw.githubusercontent.com/greynewell/mcpbr/main/install.sh | bash
 
-# Or via npm
-npm install -g mcpbr-cli && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
+# Or install and run manually
+pip install mcpbr && mcpbr run -n 1
 ```
 
 Benchmark your MCP server against real GitHub issues. One command, hard numbers.
@@ -19,6 +19,7 @@ Benchmark your MCP server against real GitHub issues. One command, hard numbers.
 **Model Context Protocol Benchmark Runner**
 
 [![PyPI version](https://badge.fury.io/py/mcpbr.svg)](https://pypi.org/project/mcpbr/)
+[![npm version](https://badge.fury.io/js/mcpbr-cli.svg)](https://www.npmjs.com/package/mcpbr-cli)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
 [![CI](https://github.com/greynewell/mcpbr/actions/workflows/ci.yml/badge.svg)](https://github.com/greynewell/mcpbr/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -60,11 +61,15 @@ mcpbr supports multiple software engineering benchmarks through a flexible abstr
 ### SWE-bench (Default)
 Real GitHub issues requiring bug fixes and patches. The agent generates unified diffs evaluated by running pytest test suites.
 
-- **Dataset**: [SWE-bench/SWE-bench_Lite](https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite)
 - **Task**: Generate patches to fix bugs
 - **Evaluation**: Test suite pass/fail
 - **Pre-built images**: Available for most tasks
 
+**Variants:**
+- **swe-bench-verified** (default) - Manually validated test cases for higher quality evaluation ([SWE-bench/SWE-bench_Verified](https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified))
+- **swe-bench-lite** - 300 tasks, quick testing ([SWE-bench/SWE-bench_Lite](https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite))
+- **swe-bench-full** - 2,294 tasks, complete benchmark ([SWE-bench/SWE-bench](https://huggingface.co/datasets/SWE-bench/SWE-bench))
+
 ### CyberGym
 Security vulnerabilities requiring Proof-of-Concept (PoC) exploits. The agent generates exploits that trigger crashes in vulnerable code.
 
@@ -84,16 +89,16 @@ Large-scale MCP tool use evaluation across 45+ categories. Tests agent capabilit
 - **Learn more**: [MCPToolBench++ Paper](https://arxiv.org/pdf/2508.07575) | [GitHub](https://github.com/mcp-tool-bench/MCPToolBenchPP)
 
 ```bash
-# Run SWE-bench (default)
+# Run SWE-bench Verified (default - manually validated tests)
 mcpbr run -c config.yaml
 
-# Run CyberGym at level 2
-mcpbr run -c config.yaml --benchmark cybergym --level 2
+# Run SWE-bench Lite (300 tasks, quick testing)
+mcpbr run -c config.yaml -b swe-bench-lite
 
-# Run MCPToolBench++
-mcpbr run -c config.yaml --benchmark mcptoolbench
+# Run SWE-bench Full (2,294 tasks, complete benchmark)
+mcpbr run -c config.yaml -b swe-bench-full
 
-# List available benchmarks
+# List all available benchmarks
 mcpbr benchmarks
 ```
 
@@ -211,6 +216,8 @@ Run `mcpbr models` to see the full list.
 
 ### via npm
 
+[![npm package](https://img.shields.io/npm/v/mcpbr-cli.svg)](https://www.npmjs.com/package/mcpbr-cli)
+
 ```bash
 # Run with npx (no installation)
 npx mcpbr-cli run -c config.yaml
@@ -220,6 +227,8 @@ npm install -g mcpbr-cli
 mcpbr run -c config.yaml
 ```
 
+> **Package**: [`mcpbr-cli`](https://www.npmjs.com/package/mcpbr-cli) on npm
+>
 > **Note**: The npm package requires Python 3.11+ and the mcpbr Python package (`pip install mcpbr`)
 
 ### via pip
@@ -271,9 +280,13 @@ See the **[Examples README](examples/README.md)** for the complete guide.
 export ANTHROPIC_API_KEY="your-api-key"
 ```
 
-2. **Generate a configuration file:**
+2. **Run mcpbr (config auto-created if missing):**
 
 ```bash
+# Config is auto-created on first run
+mcpbr run -n 1
+
+# Or explicitly generate a config file first
 mcpbr init
 ```
 
@@ -311,55 +324,135 @@ mcpbr run --config config.yaml
 
 [![Claude Code Ready](https://img.shields.io/badge/Claude_Code-Ready-5865F2?style=flat&logo=anthropic)](https://claude.ai/download)
 
-mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. When you clone this repository, Claude Code automatically detects the plugin and gains specialized knowledge about mcpbr.
+mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. The plugin provides specialized skills and knowledge about mcpbr configuration, execution, and troubleshooting.
 
-### What This Means for You
+### Installation Options
 
-When using Claude Code in this repository, you can simply say:
+You have three ways to enable the mcpbr plugin in Claude Code:
 
-- "Run the SWE-bench Lite benchmark"
-- "Generate a config for my MCP server"
-- "Run a quick test with 1 task"
+#### Option 1: Clone Repository (Automatic Detection)
 
-Claude will automatically:
-- Verify Docker is running before starting
-- Check for required API keys
-- Generate valid configurations with proper `{workdir}` placeholders
-- Use correct CLI flags and options
-- Provide helpful troubleshooting when issues occur
+When you clone this repository, Claude Code automatically detects and loads the plugin:
 
-### Available Skills
+```bash
+git clone https://github.com/greynewell/mcpbr.git
+cd mcpbr
 
-The plugin includes three specialized skills:
+# Plugin is now active - try asking Claude:
+# "Run the SWE-bench Lite eval with 5 tasks"
+```
 
-1. **run-benchmark**: Expert at running evaluations with proper validation
-   - Checks prerequisites (Docker, API keys, config files)
-   - Constructs valid `mcpbr run` commands
-   - Handles errors gracefully with actionable feedback
+**Best for**: Contributors, developers testing changes, or users who want the latest unreleased features.
 
-2. **generate-config**: Generates valid mcpbr configuration files
-   - Ensures `{workdir}` placeholder is included
-   - Validates MCP server commands
-   - Provides benchmark-specific templates
+#### Option 2: npm Global Install (Planned for v0.4.0)
 
-3. **swe-bench-lite**: Quick-start command for SWE-bench Lite
-   - Pre-configured for 5-task evaluation
-   - Includes sensible defaults for output files
-   - Perfect for testing and demonstrations
+Install the plugin globally via npm for use across any project:
 
-### Getting Started with Claude Code
+```bash
+# Planned for v0.4.0 (not yet released)
+npm install -g @mcpbr/claude-code-plugin
+```
 
-Just clone the repository and start asking Claude to run benchmarks:
+> **Note**: The npm package is not yet published. This installation method will be available in a future release. Track progress in [issue #265](https://github.com/greynewell/mcpbr/issues/265).
 
-```bash
-git clone https://github.com/greynewell/mcpbr.git
-cd mcpbr
+**Best for**: Users who want plugin features available in any directory.
 
-# In Claude Code, simply say:
-# "Run the SWE-bench Lite eval with 5 tasks"
-```
+#### Option 3: Claude Code Plugin Manager (Planned for v0.4.0)
+
+Install via Claude Code's built-in plugin manager:
+
+1. Open Claude Code settings
+2. Navigate to Plugins > Browse
+3. Search for "mcpbr"
+4. Click Install
+
+> **Note**: Plugin manager installation is not yet available. This installation method will be available after plugin marketplace submission. Track progress in [issue #267](https://github.com/greynewell/mcpbr/issues/267).
+
+**Best for**: Users who prefer a GUI and want automatic updates.
+
+### Installation Comparison
+
+| Method | Availability | Auto-updates | Works Anywhere | Latest Features |
+|--------|-------------|--------------|----------------|-----------------|
+| Clone Repository | Available now | Manual (git pull) | No (repo only) | Yes (unreleased) |
+| npm Global Install | Planned (not yet released) | Via npm | Yes | Yes (published) |
+| Plugin Manager | Planned (not yet released) | Automatic | Yes | Yes (published) |
+
+### What You Get
+
+The plugin includes three specialized skills that enhance Claude's ability to work with mcpbr:
+
+#### 1. run-benchmark
+Expert at running evaluations with proper validation and error handling.
+
+**Capabilities**:
+- Validates prerequisites (Docker running, API keys set, config files exist)
+- Constructs correct `mcpbr run` commands with appropriate flags
+- Handles errors gracefully with actionable troubleshooting steps
+- Monitors progress and provides meaningful status updates
+
+**Example interactions**:
+- "Run the SWE-bench Lite benchmark with 10 tasks"
+- "Evaluate my MCP server using CyberGym level 2"
+- "Test my config with a single task"
+
+#### 2. generate-config
+Generates valid mcpbr configuration files with benchmark-specific templates.
+
+**Capabilities**:
+- Ensures required `{workdir}` placeholder is included in MCP server args
+- Validates MCP server command syntax
+- Provides templates for different benchmarks (SWE-bench, CyberGym, MCPToolBench++)
+- Suggests appropriate timeouts and concurrency settings
+
+**Example interactions**:
+- "Generate a config for the filesystem MCP server"
+- "Create a config for testing my custom MCP server"
+- "Set up a CyberGym evaluation config"
+
+#### 3. swe-bench-lite
+Quick-start command for running SWE-bench Lite evaluations.
+
+**Capabilities**:
+- Pre-configured for 5-task evaluation (fast testing)
+- Includes sensible defaults for output files and logging
+- Perfect for demonstrations and initial testing
+- Automatically sets up verbose output for debugging
 
-The bundled plugin ensures Claude makes no silly mistakes and follows best practices automatically.
+**Example interactions**:
+- "Run a quick SWE-bench Lite test"
+- "Show me how mcpbr works"
+- "Test the filesystem server"
+
+### Benefits
+
+When using Claude Code with the mcpbr plugin active, Claude will automatically:
+
+- Verify Docker is running before starting evaluations
+- Check for required API keys (`ANTHROPIC_API_KEY`)
+- Generate valid configurations with proper `{workdir}` placeholders
+- Use correct CLI flags and avoid deprecated options
+- Provide contextual troubleshooting when issues occur
+- Follow mcpbr best practices for optimal results
+
+### Troubleshooting
+
+**Plugin not detected in cloned repository**:
+- Ensure you're in the repository root directory
+- Verify the `claude-code.json` file exists in the repo
+- Try restarting Claude Code
+
+**Skills not appearing**:
+- Check Claude Code version (requires v2.0+)
+- Verify plugin is listed in Settings > Plugins
+- Try running `/reload-plugins` in Claude Code
+
+**Commands failing**:
+- Ensure mcpbr is installed: `pip install mcpbr`
+- Verify Docker is running: `docker info`
+- Check API key is set: `echo $ANTHROPIC_API_KEY`
+
+For more help, see the [troubleshooting guide](https://greynewell.github.io/mcpbr/troubleshooting/) or [open an issue](https://github.com/greynewell/mcpbr/issues).
 
 ## Configuration
 
@@ -509,7 +602,7 @@ Run SWE-bench evaluation with the configured MCP server.
 
 | Option | Short | Description |
 |--------|-------|-------------|
-| `--config PATH` | `-c` | Path to YAML configuration file (required) |
+| `--config PATH` | `-c` | Path to YAML configuration file (default: `mcpbr.yaml`, auto-created if missing) |
 | `--model TEXT` | `-m` | Override model from config |
 | `--benchmark TEXT` | `-b` | Override benchmark from config (`swe-bench`, `cybergym`, or `mcptoolbench`) |
 | `--level INTEGER` | | Override CyberGym difficulty level (0-3) |
@@ -536,6 +629,7 @@ Run SWE-bench evaluation with the configured MCP server.
 | `--smtp-port PORT` | | SMTP server port (default: 587) |
 | `--smtp-user USER` | | SMTP username for authentication |
 | `--smtp-password PASS` | | SMTP password for authentication |
+| `--profile` | | Enable comprehensive performance profiling (tool latency, memory, overhead) |
 | `--help` | `-h` | Show help message |
 
 </details>
@@ -650,6 +744,58 @@ mcpbr cleanup -f
 
 </details>
 
+## Performance Profiling
+
+mcpbr includes comprehensive performance profiling to understand MCP server overhead and identify optimization opportunities.
+
+### Enable Profiling
+
+```bash
+# Via CLI flag
+mcpbr run -c config.yaml --profile
+
+# Or in config.yaml
+enable_profiling: true
+```
+
+### What Gets Measured
+
+- **Tool call latencies** with percentiles (p50, p95, p99)
+- **Memory usage** (peak and average RSS/VMS)
+- **Infrastructure overhead** (Docker and MCP server startup times)
+- **Tool discovery speed** (time to first tool use)
+- **Tool switching overhead** (time between tool calls)
+- **Automated insights** from profiling data
+
+### Example Profiling Output
+
+```json
+{
+  "profiling": {
+    "task_duration_seconds": 140.5,
+    "tool_call_latencies": {
+      "Read": {"count": 15, "avg_seconds": 0.8, "p95_seconds": 1.5},
+      "Bash": {"avg_seconds": 2.3, "p95_seconds": 5.1}
+    },
+    "memory_profile": {"peak_rss_mb": 512.3, "avg_rss_mb": 387.5},
+    "docker_startup_seconds": 2.1,
+    "mcp_server_startup_seconds": 1.8
+  }
+}
+```
+
+### Automated Insights
+
+The profiler automatically identifies performance issues:
+
+```text
+- Bash is the slowest tool (avg: 2.3s, p95: 5.1s)
+- Docker startup adds 2.1s overhead per task
+- Fast tool discovery: first tool use in 8.3s
+```
+
+See [docs/profiling.md](docs/profiling.md) for complete profiling documentation.
+
 ## Example Run
 
 Here's what a typical evaluation looks like:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcpbr-cli",
-  "version": "0.3.28",
+  "version": "0.4.1",
   "description": "Model Context Protocol Benchmark Runner - CLI tool for evaluating MCP servers",
   "keywords": [
     "mcpbr",