mcpbr-cli 0.3.26 → 0.4.0

package/README.md

@@ -1,11 +1,11 @@
 # mcpbr
 
 ```bash
-# Install via pip
-pip install mcpbr && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
+# One-liner install (installs + runs quick test)
+curl -sSL https://raw.githubusercontent.com/greynewell/mcpbr/main/install.sh | bash
 
-# Or via npm
-npm install -g mcpbr-cli && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
+# Or install and run manually
+pip install mcpbr && mcpbr run -n 1
 ```
 
 Benchmark your MCP server against real GitHub issues. One command, hard numbers.
@@ -60,11 +60,15 @@ mcpbr supports multiple software engineering benchmarks through a flexible abstr
 ### SWE-bench (Default)
 Real GitHub issues requiring bug fixes and patches. The agent generates unified diffs evaluated by running pytest test suites.
 
-- **Dataset**: [SWE-bench/SWE-bench_Lite](https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite)
 - **Task**: Generate patches to fix bugs
 - **Evaluation**: Test suite pass/fail
 - **Pre-built images**: Available for most tasks
 
+**Variants:**
+- **swe-bench-verified** (default) - Manually validated test cases for higher quality evaluation ([SWE-bench/SWE-bench_Verified](https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified))
+- **swe-bench-lite** - 300 tasks, quick testing ([SWE-bench/SWE-bench_Lite](https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite))
+- **swe-bench-full** - 2,294 tasks, complete benchmark ([SWE-bench/SWE-bench](https://huggingface.co/datasets/SWE-bench/SWE-bench))
+
 ### CyberGym
 Security vulnerabilities requiring Proof-of-Concept (PoC) exploits. The agent generates exploits that trigger crashes in vulnerable code.
 
@@ -84,16 +88,16 @@ Large-scale MCP tool use evaluation across 45+ categories. Tests agent capabilit
 - **Learn more**: [MCPToolBench++ Paper](https://arxiv.org/pdf/2508.07575) | [GitHub](https://github.com/mcp-tool-bench/MCPToolBenchPP)
 
 ```bash
-# Run SWE-bench (default)
+# Run SWE-bench Verified (default - manually validated tests)
 mcpbr run -c config.yaml
 
-# Run CyberGym at level 2
-mcpbr run -c config.yaml --benchmark cybergym --level 2
+# Run SWE-bench Lite (300 tasks, quick testing)
+mcpbr run -c config.yaml -b swe-bench-lite
 
-# Run MCPToolBench++
-mcpbr run -c config.yaml --benchmark mcptoolbench
+# Run SWE-bench Full (2,294 tasks, complete benchmark)
+mcpbr run -c config.yaml -b swe-bench-full
 
-# List available benchmarks
+# List all available benchmarks
 mcpbr benchmarks
 ```
 
@@ -271,9 +275,13 @@ See the **[Examples README](examples/README.md)** for the complete guide.
 export ANTHROPIC_API_KEY="your-api-key"
 ```
 
-2. **Generate a configuration file:**
+2. **Run mcpbr (config auto-created if missing):**
 
 ```bash
+# Config is auto-created on first run
+mcpbr run -n 1
+
+# Or explicitly generate a config file first
 mcpbr init
 ```
 
@@ -296,6 +304,9 @@ dataset: "SWE-bench/SWE-bench_Lite"
 sample_size: 10
 timeout_seconds: 300
 max_concurrent: 4
+
+# Optional: disable default logging (logs are saved to output_dir/logs/ by default)
+# disable_logs: true
 ```
 
 4. **Run the evaluation:**
@@ -308,55 +319,135 @@ mcpbr run --config config.yaml
 
 [![Claude Code Ready](https://img.shields.io/badge/Claude_Code-Ready-5865F2?style=flat&logo=anthropic)](https://claude.ai/download)
 
-mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. When you clone this repository, Claude Code automatically detects the plugin and gains specialized knowledge about mcpbr.
+mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. The plugin provides specialized skills and knowledge about mcpbr configuration, execution, and troubleshooting.
 
-### What This Means for You
+### Installation Options
 
-When using Claude Code in this repository, you can simply say:
+You have three ways to enable the mcpbr plugin in Claude Code:
 
-- "Run the SWE-bench Lite benchmark"
-- "Generate a config for my MCP server"
-- "Run a quick test with 1 task"
+#### Option 1: Clone Repository (Automatic Detection)
 
-Claude will automatically:
-- Verify Docker is running before starting
-- Check for required API keys
-- Generate valid configurations with proper `{workdir}` placeholders
-- Use correct CLI flags and options
-- Provide helpful troubleshooting when issues occur
+When you clone this repository, Claude Code automatically detects and loads the plugin:
 
-### Available Skills
+```bash
+git clone https://github.com/greynewell/mcpbr.git
+cd mcpbr
 
-The plugin includes three specialized skills:
+# Plugin is now active - try asking Claude:
+# "Run the SWE-bench Lite eval with 5 tasks"
+```
 
-1. **run-benchmark**: Expert at running evaluations with proper validation
-   - Checks prerequisites (Docker, API keys, config files)
-   - Constructs valid `mcpbr run` commands
-   - Handles errors gracefully with actionable feedback
+**Best for**: Contributors, developers testing changes, or users who want the latest unreleased features.
 
-2. **generate-config**: Generates valid mcpbr configuration files
-   - Ensures `{workdir}` placeholder is included
-   - Validates MCP server commands
-   - Provides benchmark-specific templates
+#### Option 2: npm Global Install (Planned for v0.4.0)
 
-3. **swe-bench-lite**: Quick-start command for SWE-bench Lite
-   - Pre-configured for 5-task evaluation
-   - Includes sensible defaults for output files
-   - Perfect for testing and demonstrations
+Install the plugin globally via npm for use across any project:
 
-### Getting Started with Claude Code
+```bash
+# Planned for v0.4.0 (not yet released)
+npm install -g @mcpbr/claude-code-plugin
+```
 
-Just clone the repository and start asking Claude to run benchmarks:
+> **Note**: The npm package is not yet published. This installation method will be available in a future release. Track progress in [issue #265](https://github.com/greynewell/mcpbr/issues/265).
 
-```bash
-git clone https://github.com/greynewell/mcpbr.git
-cd mcpbr
+**Best for**: Users who want plugin features available in any directory.
 
-# In Claude Code, simply say:
-# "Run the SWE-bench Lite eval with 5 tasks"
-```
+#### Option 3: Claude Code Plugin Manager (Planned for v0.4.0)
+
+Install via Claude Code's built-in plugin manager:
+
+1. Open Claude Code settings
+2. Navigate to Plugins > Browse
+3. Search for "mcpbr"
+4. Click Install
+
+> **Note**: Plugin manager installation is not yet available. This installation method will be available after plugin marketplace submission. Track progress in [issue #267](https://github.com/greynewell/mcpbr/issues/267).
+
+**Best for**: Users who prefer a GUI and want automatic updates.
+
+### Installation Comparison
+
+| Method | Availability | Auto-updates | Works Anywhere | Latest Features |
+|--------|-------------|--------------|----------------|-----------------|
+| Clone Repository | Available now | Manual (git pull) | No (repo only) | Yes (unreleased) |
+| npm Global Install | Planned (not yet released) | Via npm | Yes | Yes (published) |
+| Plugin Manager | Planned (not yet released) | Automatic | Yes | Yes (published) |
+
+### What You Get
+
+The plugin includes three specialized skills that enhance Claude's ability to work with mcpbr:
+
+#### 1. run-benchmark
+Expert at running evaluations with proper validation and error handling.
+
+**Capabilities**:
+- Validates prerequisites (Docker running, API keys set, config files exist)
+- Constructs correct `mcpbr run` commands with appropriate flags
+- Handles errors gracefully with actionable troubleshooting steps
+- Monitors progress and provides meaningful status updates
+
+**Example interactions**:
+- "Run the SWE-bench Lite benchmark with 10 tasks"
+- "Evaluate my MCP server using CyberGym level 2"
+- "Test my config with a single task"
+
+#### 2. generate-config
+Generates valid mcpbr configuration files with benchmark-specific templates.
+
+**Capabilities**:
+- Ensures required `{workdir}` placeholder is included in MCP server args
+- Validates MCP server command syntax
+- Provides templates for different benchmarks (SWE-bench, CyberGym, MCPToolBench++)
+- Suggests appropriate timeouts and concurrency settings
 
-The bundled plugin ensures Claude makes no silly mistakes and follows best practices automatically.
+**Example interactions**:
+- "Generate a config for the filesystem MCP server"
+- "Create a config for testing my custom MCP server"
+- "Set up a CyberGym evaluation config"
+
+#### 3. swe-bench-lite
+Quick-start command for running SWE-bench Lite evaluations.
+
+**Capabilities**:
+- Pre-configured for 5-task evaluation (fast testing)
+- Includes sensible defaults for output files and logging
+- Perfect for demonstrations and initial testing
+- Automatically sets up verbose output for debugging
+
+**Example interactions**:
+- "Run a quick SWE-bench Lite test"
+- "Show me how mcpbr works"
+- "Test the filesystem server"
+
+### Benefits
+
+When using Claude Code with the mcpbr plugin active, Claude will automatically:
+
+- Verify Docker is running before starting evaluations
+- Check for required API keys (`ANTHROPIC_API_KEY`)
+- Generate valid configurations with proper `{workdir}` placeholders
+- Use correct CLI flags and avoid deprecated options
+- Provide contextual troubleshooting when issues occur
+- Follow mcpbr best practices for optimal results
+
+### Troubleshooting
+
+**Plugin not detected in cloned repository**:
+- Ensure you're in the repository root directory
+- Verify the `claude-code.json` file exists in the repo
+- Try restarting Claude Code
+
+**Skills not appearing**:
+- Check Claude Code version (requires v2.0+)
+- Verify plugin is listed in Settings > Plugins
+- Try running `/reload-plugins` in Claude Code
+
+**Commands failing**:
+- Ensure mcpbr is installed: `pip install mcpbr`
+- Verify Docker is running: `docker info`
+- Check API key is set: `echo $ANTHROPIC_API_KEY`
+
+For more help, see the [troubleshooting guide](https://greynewell.github.io/mcpbr/troubleshooting/) or [open an issue](https://github.com/greynewell/mcpbr/issues).
 
 ## Configuration
 
@@ -506,7 +597,7 @@ Run SWE-bench evaluation with the configured MCP server.
 
 | Option | Short | Description |
 |--------|-------|-------------|
-| `--config PATH` | `-c` | Path to YAML configuration file (required) |
+| `--config PATH` | `-c` | Path to YAML configuration file (default: `mcpbr.yaml`, auto-created if missing) |
 | `--model TEXT` | `-m` | Override model from config |
 | `--benchmark TEXT` | `-b` | Override benchmark from config (`swe-bench`, `cybergym`, or `mcptoolbench`) |
 | `--level INTEGER` | | Override CyberGym difficulty level (0-3) |
@@ -519,7 +610,8 @@ Run SWE-bench evaluation with the configured MCP server.
 | `--output-junit PATH` | | Path to save JUnit XML report (for CI/CD integration) |
 | `--verbose` | `-v` | Verbose output (`-v` summary, `-vv` detailed) |
 | `--log-file PATH` | `-l` | Path to write raw JSON log output (single file) |
-| `--log-dir PATH` | | Directory to write per-instance JSON log files |
+| `--log-dir PATH` | | Directory to write per-instance JSON log files (default: `output_dir/logs/`) |
+| `--disable-logs` | | Disable detailed execution logs (overrides default and config) |
 | `--task TEXT` | `-t` | Run specific task(s) by instance_id (repeatable) |
 | `--prompt TEXT` | | Override agent prompt (use `{problem_statement}` placeholder) |
 | `--baseline-results PATH` | | Path to baseline results JSON for regression detection |
@@ -773,6 +865,38 @@ Results saved to results.json
 }
 ```
 
+### Output Directory Structure
+
+By default, mcpbr consolidates all outputs into a single timestamped directory:
+
+```text
+.mcpbr_run_20260126_133000/
+├── config.yaml                # Copy of configuration used
+├── evaluation_state.json      # Task results and state
+├── logs/                      # Detailed MCP server logs
+│   ├── task_1_mcp.log
+│   ├── task_2_mcp.log
+│   └── ...
+└── README.txt                 # Auto-generated explanation
+```
+
+This makes it easy to:
+- **Archive results**: `tar -czf results.tar.gz .mcpbr_run_*`
+- **Clean up**: `rm -rf .mcpbr_run_*`
+- **Share**: Just zip one directory
+
+You can customize the output directory:
+
+```bash
+# Custom output directory
+mcpbr run -c config.yaml --output-dir ./my-results
+
+# Or in config.yaml
+output_dir: "./my-results"
+```
+
+**Note:** The `--output-dir` CLI flag takes precedence over the `output_dir` config setting. This ensures that the README.txt file in the output directory reflects the final effective configuration values after all CLI overrides are applied.
+
 ### Markdown Report (`--report`)
 
 Generates a human-readable report with:
@@ -782,6 +906,17 @@ Generates a human-readable report with:
 
 ### Per-Instance Logs (`--log-dir`)
 
+**Logging is enabled by default** to prevent data loss. Detailed execution traces are automatically saved to `output_dir/logs/` unless disabled.
+
+To disable logging:
+```bash
+# Via CLI flag
+mcpbr run -c config.yaml --disable-logs
+
+# Or in config file
+disable_logs: true
+```
+
 Creates a directory with detailed JSON log files for each task run. Filenames include timestamps to prevent overwrites:
 
 ```text

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcpbr-cli",
-  "version": "0.3.26",
+  "version": "0.4.0",
   "description": "Model Context Protocol Benchmark Runner - CLI tool for evaluating MCP servers",
   "keywords": [
     "mcpbr",