@fre4x/benchmark 1.1.0-beta.0 → 1.1.0-beta.3

package/README.md

@@ -1,18 +1,20 @@
-# benchmark — Unified Agent Evaluation
+# benchmark — Deterministic Agent Evaluation
 
-This package exposes a consistent MCP workflow for benchmark-driven agent evaluation.
+This package exposes a consistent MCP workflow for deterministic benchmark-driven agent evaluation.
 
-GAIA is the first built-in adapter, but the tool surface is generic so other benchmarks can plug in later without changing client behavior.
+The rebuilt core is organized around challenge catalogs, typed task assets, and explicit checker kinds so coding, web, and OS-style tasks can share one MCP surface without relying on LLM judges.
 
 ## Tools
 
 | Tool | Purpose |
 |------|---------|
-| `benchmark_list_challenges` | List available benchmark suites with version and asset metadata |
-| `benchmark_start_challenge` | Start an attempt and return the first question |
-| `benchmark_submit_solution` | Grade one answer and return the next question or final score |
+| `benchmark_list_challenges` | List deterministic benchmark suites with family, runner, and checker metadata |
+| `benchmark_get_catalog_status` | Inspect catalog source configuration, cache state, and availability |
+| `benchmark_sync_catalog` | Fetch and cache the remote benchmark catalog when a URL source is configured |
+| `benchmark_start_challenge` | Start an attempt and return the first task |
+| `benchmark_submit_solution` | Grade one task and return checker evidence plus the next task or final score |
 | `benchmark_get_asset` | Read an attached benchmark asset by `asset_id` |
-| `benchmark_get_attempt` | Inspect attempt status and the current question |
+| `benchmark_get_attempt` | Inspect attempt status, current task, and paginated evaluation history |
 | `benchmark_cancel_attempt` | Cancel an active attempt |
 
 ## Workflow
@@ -20,45 +22,147 @@ GAIA is the first built-in adapter, but the tool surface is generic so other ben
 1. Call `benchmark_list_challenges`
 2. Pick a `challenge_id`
 3. Call `benchmark_start_challenge`
-4. If the question has assets, call `benchmark_get_asset`
+4. If the task has assets, call `benchmark_get_asset`
 5. Call `benchmark_submit_solution`
 6. Repeat until `done: true`
 
 Each response includes machine-readable guidance for the most likely next tool call.
 
+## Fallback benchmark families
+
+- **Code** — deterministic JSON/text answers backed by explicit checkers
+- **Web** — DOM snapshot extraction tasks with JSON field assertions
+- **OS** — filesystem/log review tasks with deterministic text grading
+
+## Zero-config run
+
+Run with the bundled fallback catalog and no extra configuration:
+
+```bash
+npx @fre4x/benchmark
+```
+
+Or from this repo:
+
+```bash
+cd /home/fritzprix/my_works/b1te
+npm run inspector -w @fre4x/benchmark
+```
+
 ## Mock Mode
 
-Run without any external dataset file:
+Run with the same bundled fallback catalog in mock mode:
 
 ```bash
 MOCK=true npx @fre4x/benchmark
 ```
 
-## Optional Environment
+## Optional environment
 
 ```bash
-BENCHMARK_GAIA_DATA_FILE=/absolute/path/to/gaia-challenges.json
+BENCHMARK_CATALOG_FILE=/absolute/path/to/benchmark-catalog.json
+BENCHMARK_CATALOG_URL=https://example.com/benchmark-catalog.json
+BENCHMARK_CACHE_DIR=/absolute/path/to/catalog-cache
+BENCHMARK_CACHE_TTL_SECONDS=3600
 BENCHMARK_STATE_DIR=/absolute/path/to/store-attempt-json
 BENCHMARK_MOCK=true
 ```
 
-- `BENCHMARK_GAIA_DATA_FILE`: Optional JSON file with GAIA-compatible normalized challenge definitions
-- `BENCHMARK_STATE_DIR`: Where attempt state is persisted
+- `BENCHMARK_CATALOG_FILE`: Optional JSON file with challenge definitions in the rebuilt deterministic catalog format
+- `BENCHMARK_CATALOG_URL`: Optional remote JSON catalog URL for fetch/cache based ingestion
+- `BENCHMARK_CACHE_DIR`: Optional cache directory for remote catalog snapshots
+- `BENCHMARK_CACHE_TTL_SECONDS`: Freshness window for remote catalog cache reuse
+- `BENCHMARK_STATE_DIR`: Where attempt files and lock directories are persisted
 - `BENCHMARK_MOCK`: Alternate mock-mode flag
 
+`BENCHMARK_GAIA_DATA_FILE` is still accepted as a backward-compatible alias, but the rebuilt package is no longer GAIA-first.
+
+When `BENCHMARK_CATALOG_URL` is set, the package will reuse a fresh cached copy when available and can be explicitly refreshed with `benchmark_sync_catalog`.
+
+## Catalog shape
+
+External catalogs must be a JSON array of challenge definitions shaped like:
+
+```json
+[
+  {
+    "challenge_id": "custom_suite",
+    "benchmark_id": "custom",
+    "family": "code",
+    "runner_kind": "code_runner",
+    "title": "Custom Challenge",
+    "description": "Deterministic single-task suite",
+    "version": "v1",
+    "source": "external",
+    "tasks": [
+      {
+        "task_id": "custom-1",
+        "title": "Return yes",
+        "prompt": "Return only yes.",
+        "response_format": "text",
+        "difficulty": 1,
+        "assets": [],
+        "checkers": [
+          {
+            "checker_id": "custom-yes",
+            "kind": "exact_text",
+            "expected": "yes"
+          }
+        ]
+      }
+    ]
+  }
+]
+```
+
+Supported checker kinds today:
+
+- `exact_text`
+- `normalized_text`
+- `regex_match`
+- `contains_all_text`
+- `json_field_equals`
+- `runner_fact_equals`
+- `runner_log_contains_text`
+
+Each submission now also records runner execution metadata:
+
+- workspace directory
+- materialized asset and submission artifacts
+- runner facts
+- runner logs
+
+## Example external catalog
+
+The repo includes the bundled fallback catalog source at:
+
+```bash
+benchmark/catalogs/expanded-catalog.json
+```
+
+Use it like this:
+
+```bash
+cd /home/fritzprix/my_works/b1te
+BENCHMARK_CATALOG_FILE=/home/fritzprix/my_works/b1te/benchmark/catalogs/expanded-catalog.json npm run inspector -w @fre4x/benchmark
+```
+
+If no catalog env is provided at runtime, the published package falls back to the bundled copy of this catalog automatically.
+
 ## Claude Desktop
 
 ```json
 {
   "mcpServers": {
     "benchmark": {
-      "command": "npx",
-      "args": ["-y", "@fre4x/benchmark"],
-      "env": {
-        "BENCHMARK_GAIA_DATA_FILE": "/absolute/path/to/gaia-challenges.json"
+        "command": "npx",
+        "args": ["-y", "@fre4x/benchmark"],
+        "env": {
+          "BENCHMARK_CATALOG_URL": "https://example.com/benchmark-catalog.json",
+          "BENCHMARK_CACHE_DIR": "/absolute/path/to/benchmark-cache"
+        }
       }
     }
-  }
 }
 ```