@fre4x/benchmark 1.1.0-beta.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +73 -0
- package/dist/index.js +31342 -0
- package/package.json +44 -0
package/README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# benchmark — Unified Agent Evaluation
|
|
2
|
+
|
|
3
|
+
This package exposes a consistent MCP workflow for benchmark-driven agent evaluation.
|
|
4
|
+
|
|
5
|
+
GAIA is the first built-in adapter, but the tool surface is generic so other benchmarks can plug in later without changing client behavior.
|
|
6
|
+
|
|
7
|
+
## Tools
|
|
8
|
+
|
|
9
|
+
| Tool | Purpose |
|
|
10
|
+
|------|---------|
|
|
11
|
+
| `benchmark_list_challenges` | List available benchmark suites with version and asset metadata |
|
|
12
|
+
| `benchmark_start_challenge` | Start an attempt and return the first question |
|
|
13
|
+
| `benchmark_submit_solution` | Grade one answer and return the next question or final score |
|
|
14
|
+
| `benchmark_get_asset` | Read an attached benchmark asset by `asset_id` |
|
|
15
|
+
| `benchmark_get_attempt` | Inspect attempt status and the current question |
|
|
16
|
+
| `benchmark_cancel_attempt` | Cancel an active attempt |
|
|
17
|
+
|
|
18
|
+
## Workflow
|
|
19
|
+
|
|
20
|
+
1. Call `benchmark_list_challenges`
|
|
21
|
+
2. Pick a `challenge_id`
|
|
22
|
+
3. Call `benchmark_start_challenge`
|
|
23
|
+
4. If the question has assets, call `benchmark_get_asset`
|
|
24
|
+
5. Call `benchmark_submit_solution`
|
|
25
|
+
6. Repeat until `done: true`
|
|
26
|
+
|
|
27
|
+
Each response includes machine-readable guidance for the most likely next tool call.
|
|
28
|
+
|
|
29
|
+
## Mock Mode
|
|
30
|
+
|
|
31
|
+
Run without any external dataset file:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
MOCK=true npx @fre4x/benchmark
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Optional Environment
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
BENCHMARK_GAIA_DATA_FILE=/absolute/path/to/gaia-challenges.json
|
|
41
|
+
BENCHMARK_STATE_DIR=/absolute/path/to/store-attempt-json
|
|
42
|
+
BENCHMARK_MOCK=true
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
- `BENCHMARK_GAIA_DATA_FILE`: Optional JSON file with GAIA-compatible normalized challenge definitions
|
|
46
|
+
- `BENCHMARK_STATE_DIR`: Where attempt state is persisted
|
|
47
|
+
- `BENCHMARK_MOCK`: Alternate mock-mode flag
|
|
48
|
+
|
|
49
|
+
## Claude Desktop
|
|
50
|
+
|
|
51
|
+
```json
|
|
52
|
+
{
|
|
53
|
+
"mcpServers": {
|
|
54
|
+
"benchmark": {
|
|
55
|
+
"command": "npx",
|
|
56
|
+
"args": ["-y", "@fre4x/benchmark"],
|
|
57
|
+
"env": {
|
|
58
|
+
"BENCHMARK_GAIA_DATA_FILE": "/absolute/path/to/gaia-challenges.json"
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Development
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
npm install
|
|
69
|
+
npm run build -w @fre4x/benchmark
|
|
70
|
+
npm run typecheck -w @fre4x/benchmark
|
|
71
|
+
npm test -w @fre4x/benchmark
|
|
72
|
+
MOCK=true npm run inspector -w @fre4x/benchmark
|
|
73
|
+
```
|