mcpbr-cli 0.4.3 → 0.4.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -332,6 +332,56 @@ max_concurrent: 4
|
|
|
332
332
|
mcpbr run --config config.yaml
|
|
333
333
|
```
|
|
334
334
|
|
|
335
|
+
## Infrastructure Modes
|
|
336
|
+
|
|
337
|
+
mcpbr supports running evaluations on different infrastructure platforms, allowing you to scale evaluations or offload compute-intensive tasks to cloud VMs.
|
|
338
|
+
|
|
339
|
+
### Local (Default)
|
|
340
|
+
|
|
341
|
+
Run evaluations on your local machine:
|
|
342
|
+
|
|
343
|
+
```yaml
|
|
344
|
+
infrastructure:
|
|
345
|
+
mode: local # default
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
This is the default mode - evaluations run directly on your machine using local Docker containers.
|
|
349
|
+
|
|
350
|
+
### Azure VM
|
|
351
|
+
|
|
352
|
+
Run evaluations on Azure Virtual Machines with automatic provisioning and cleanup:
|
|
353
|
+
|
|
354
|
+
```yaml
|
|
355
|
+
infrastructure:
|
|
356
|
+
mode: azure
|
|
357
|
+
azure:
|
|
358
|
+
resource_group: mcpbr-benchmarks
|
|
359
|
+
location: eastus
|
|
360
|
+
cpu_cores: 10
|
|
361
|
+
memory_gb: 40
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
**Key features:**
|
|
365
|
+
- Zero manual VM setup - provisioned automatically from config
|
|
366
|
+
- Automatic Docker, Python, and mcpbr installation
|
|
367
|
+
- Test task validation before full evaluation
|
|
368
|
+
- Auto-cleanup after completion (configurable)
|
|
369
|
+
- Cost-optimized with automatic VM deletion
|
|
370
|
+
|
|
371
|
+
**Example usage:**
|
|
372
|
+
```bash
|
|
373
|
+
# Run evaluation on Azure VM
|
|
374
|
+
mcpbr run -c azure-config.yaml
|
|
375
|
+
|
|
376
|
+
# VM is automatically created, evaluation runs, results are downloaded, VM is deleted
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
See [docs/infrastructure/azure.md](docs/infrastructure/azure.md) for full documentation including:
|
|
380
|
+
- Prerequisites and authentication
|
|
381
|
+
- VM sizing and cost estimation
|
|
382
|
+
- Debugging with `preserve_on_error`
|
|
383
|
+
- Troubleshooting guide
|
|
384
|
+
|
|
335
385
|
## Side-by-Side Server Comparison
|
|
336
386
|
|
|
337
387
|
Compare two MCP servers head-to-head in a single evaluation run to see which implementation performs better.
|