@athenaflow/plugin-web-bench 1.0.5

package/skills/load-dataset/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: load-dataset
+description: >
+  Download and prepare the Halluminate/WebBench dataset from HuggingFace for benchmarking.
+  Triggers: "load dataset", "download WebBench", "prepare benchmark data", "fetch tasks".
+  Downloads the CSV dataset via curl, converts to JSONL with Node.js, applies optional filters
+  (category, sample size, website allowlist/blocklist), and writes web-bench-tasks.jsonl to the
+  working directory. Zero Python dependencies — uses only curl and Node.js.
+  Does NOT execute tasks — use execute-task for that.
+allowed-tools: Bash Read Write Edit Glob
+---
+
+# Load WebBench Dataset
+
+Download the Halluminate/WebBench dataset from HuggingFace and prepare it for benchmark execution.
+
+## Dataset Source
+
+- **HuggingFace:** `Halluminate/WebBench`
+- **Source file:** `webbenchfinal.csv` (CSV format)
+- **Size:** ~2,454 tasks across 452 websites
+- **Fields per row:** `ID` (int), `Starting_URL` (string), `Category` (enum), `Task` (string)
+
+## Pre-check: Skip Download if Dataset Exists
+
+Before downloading, check if `web-bench-tasks.jsonl` already exists in the working directory:
+
+```bash
+if [ -f web-bench-tasks.jsonl ]; then
+  echo "Dataset already exists: $(wc -l < web-bench-tasks.jsonl) tasks"
+  head -1 web-bench-tasks.jsonl
+fi
+```
+
+**If `web-bench-tasks.jsonl` exists and is non-empty, skip the download and conversion entirely.** Jump straight to [Applying Filters](#applying-filters) if filters need to be applied, or report the existing dataset to the tracker.
+
+Only proceed with download if the file does not exist or is empty.
+
+## Download Method
+
+Download the CSV directly with `curl`, then convert to JSONL with Node.js. No Python dependencies required.
+
+### Step 1: Download the CSV
+
+```bash
+curl -fSL -o web-bench-dataset.csv \
+  "https://huggingface.co/datasets/Halluminate/WebBench/resolve/main/webbenchfinal.csv"
+```
+
+If the above URL fails (HuggingFace sometimes changes paths), try:
+
+```bash
+curl -fSL -o web-bench-dataset.csv \
+  "https://huggingface.co/datasets/Halluminate/WebBench/raw/main/webbenchfinal.csv"
+```
+
+### Step 2: Convert CSV to JSONL
+
+```bash
+node -e "
+const fs = require('fs');
+const csv = fs.readFileSync('web-bench-dataset.csv', 'utf-8');
+const lines = csv.split('\n');
+const header = lines[0].split(',').map(h => h.trim().replace(/^\"|\"$/g, ''));
+
+// Find column indices
+const idIdx = header.findIndex(h => h === 'ID');
+const urlIdx = header.findIndex(h => h === 'Starting_URL');
+const catIdx = header.findIndex(h => h === 'Category');
+const taskIdx = header.findIndex(h => h === 'Task');
+
+const out = fs.createWriteStream('web-bench-tasks.jsonl');
+let count = 0;
+
+for (let i = 1; i < lines.length; i++) {
+  const line = lines[i].trim();
+  if (!line) continue;
+
+  // Parse CSV line respecting quoted fields
+  const fields = [];
+  let field = '';
+  let inQuotes = false;
+  for (let j = 0; j < line.length; j++) {
+    const ch = line[j];
+    if (ch === '\"') {
+      inQuotes = !inQuotes;
+    } else if (ch === ',' && !inQuotes) {
+      fields.push(field.trim());
+      field = '';
+    } else {
+      field += ch;
+    }
+  }
+  fields.push(field.trim());
+
+  if (fields.length > taskIdx) {
+    out.write(JSON.stringify({
+      id: parseInt(fields[idIdx], 10),
+      url: fields[urlIdx],
+      category: fields[catIdx],
+      task: fields[taskIdx]
+    }) + '\n');
+    count++;
+  }
+}
+
+out.end();
+console.log('Wrote ' + count + ' tasks to web-bench-tasks.jsonl');
+"
+```
+
+### Step 3: Verify the output
+
+```bash
+wc -l web-bench-tasks.jsonl
+head -1 web-bench-tasks.jsonl
+node -e "
+const fs = require('fs');
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+const cats = {};
+const sites = new Set();
+for (const t of tasks) {
+  cats[t.category] = (cats[t.category] || 0) + 1;
+  sites.add(t.url);
+}
+for (const [c, n] of Object.entries(cats).sort()) console.log('  ' + c + ': ' + n);
+console.log('Total: ' + tasks.length + ' tasks across ' + sites.size + ' websites');
+"
+```
+
+## Applying Filters
+
+After downloading, apply filters based on tracker configuration. All filters use Node.js.
+
+### Category Filter
+
+If the tracker specifies a category filter (e.g., `READ`, `CREATE`):
+
+```bash
+node -e "
+const fs = require('fs');
+const category = process.argv[1];
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+const filtered = tasks.filter(t => t.category === category);
+fs.writeFileSync('web-bench-tasks.jsonl', filtered.map(JSON.stringify).join('\n') + '\n');
+console.log('Filtered to ' + filtered.length + ' ' + category + ' tasks');
+" "READ"
+```
+
+### Sample Size
+
+If the tracker specifies a sample size (e.g., `--sample 50`):
+
+```bash
+node -e "
+const fs = require('fs');
+const n = parseInt(process.argv[1], 10);
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+
+// Deterministic shuffle (seed-based) for reproducibility
+function seededShuffle(arr, seed) {
+  const a = [...arr];
+  let s = seed;
+  for (let i = a.length - 1; i > 0; i--) {
+    s = (s * 1664525 + 1013904223) & 0xffffffff;
+    const j = ((s >>> 0) % (i + 1));
+    [a[i], a[j]] = [a[j], a[i]];
+  }
+  return a;
+}
+
+const sample = seededShuffle(tasks, 42).slice(0, Math.min(n, tasks.length));
+fs.writeFileSync('web-bench-tasks.jsonl', sample.map(JSON.stringify).join('\n') + '\n');
+console.log('Sampled ' + sample.length + ' tasks');
+" "50"
+```
+
+### Website Blocklist
+
+```bash
+node -e "
+const fs = require('fs');
+const blocklist = new Set(process.argv[1] ? process.argv[1].split(',') : []);
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+const filtered = tasks.filter(t => !blocklist.has(t.url));
+fs.writeFileSync('web-bench-tasks.jsonl', filtered.map(JSON.stringify).join('\n') + '\n');
+console.log(filtered.length + ' tasks after blocklist filter');
+" ""
+```
+
+## Output
+
+- **File:** `web-bench-tasks.jsonl` in working directory
+- **Intermediate file:** `web-bench-dataset.csv` (can be deleted after conversion)
+- **Format:** One JSON object per line
+- **Schema:**
+  ```json
+  {"id": 42, "url": "https://acehardware.com", "category": "READ", "task": "Navigate to..."}
+  ```
+
+## Cleanup
+
+After successful conversion, remove the intermediate CSV:
+
+```bash
+rm -f web-bench-dataset.csv
+```
+
+Report the total count and category breakdown to the tracker.

package/skills/load-dataset/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false

package/skills/run-benchmark/SKILL.md

@@ -0,0 +1,92 @@
+---
+name: run-benchmark
+description: >
+  Run the WebBench browser agent benchmark — main entry point and orchestrator.
+  Triggers: "run benchmark", "run WebBench", "start benchmark", "benchmark browser agent",
+  "web bench", "execute WebBench", "run web-bench".
+  Parses user configuration (category filter, sample size, resume), delegates to
+  load-dataset, execute-task, evaluate-task, and generate-report skills.
+  This is the user-invocable orchestrator that ties the full benchmark pipeline together.
+allowed-tools: Read Write Edit Glob Grep Bash Task
+---
+
+# Run WebBench Benchmark
+
+Main entry point for running the WebBench browser agent benchmark. This skill is used in interactive (single-session) mode. For multi-session workflow execution, see the system prompt.
+
+## Input
+
+Parse configuration from: `$ARGUMENTS`
+
+Supported flags:
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--category <CAT>` | Filter tasks by category (READ, CREATE, UPDATE, DELETE, FILE_MANIPULATION) | All categories |
+| `--sample <N>` | Random sample of N tasks (deterministic seed=42) | Full dataset |
+| `--resume` | Resume from existing web-bench-results.jsonl, skip completed task IDs | Fresh run |
+| `--report-only` | Skip execution, just generate report from existing results | Full run |
+
+Examples:
+- `run-benchmark --category READ --sample 50` — 50 random READ tasks
+- `run-benchmark --resume` — continue from where last run stopped
+- `run-benchmark --report-only` — just aggregate existing results
+
+## Interactive Execution Protocol
+
+When run interactively (not via the workflow loop), this skill executes the full pipeline in a single session:
+
+### 1. Setup
+
+1. Parse arguments
+2. Check for existing state (`web-bench-tasks.jsonl`, `web-bench-results.jsonl`)
+3. If `--resume` and results exist: determine completed task IDs, skip them
+4. If not resuming: load the `load-dataset` skill to download and prepare the dataset
+5. Report configuration and task count
+
+### 2. Execute Tasks
+
+For each task in `web-bench-tasks.jsonl` (skipping completed if resuming):
+
+1. Read the task line
+2. Record start time: `date +%s%3N`
+3. Load `execute-task` methodology and have the browser-capable calling context perform the browser automation
+4. Load `evaluate-task` methodology and score the result
+5. Record end time: `date +%s%3N`, compute duration
+6. Append result to `web-bench-results.jsonl`:
+   ```json
+   {"id": 42, "url": "...", "category": "READ", "task": "...", "score": 1.0, "verdict": "PASS", "reasoning": "...", "error": null, "duration_ms": 34200, "tokens_used": {"input": 12450, "output": 3200}, "timestamp": "2026-03-19T14:30:00Z"}
+   ```
+7. Print progress: `[42/2454] PASS (1.0) — acehardware.com — READ — 34.2s`
+
+### 3. Generate Report
+
+After all tasks are processed (or if `--report-only`):
+
+1. Load `generate-report` methodology
+2. Aggregate `web-bench-results.jsonl` into `web-bench-report.md`
+3. Print summary statistics to console
+
+## Token Tracking
+
+Token usage should be tracked per task. The agent should estimate tokens consumed during task execution by recording:
+
+- **Input tokens:** Approximate from the size of prompts, page snapshots, and tool responses received during execution
+- **Output tokens:** Approximate from the size of responses and tool calls generated
+
+If exact token counts are available from the session metadata, prefer those over estimates.
+
+## Progress Display
+
+After each task, print a status line:
+
+```
+[1/50] PASS  (1.0)  acehardware.com          READ    34.2s   15,650 tokens
+[2/50] FAIL  (0.0)  airbnb.com               CREATE  12.1s    8,200 tokens  [auth_required]
+[3/50] PARTIAL(0.5) amazon.com               READ    45.8s   22,100 tokens
+```
+
+## Guardrails
+
+- **Always append, never overwrite** results. The JSONL file is append-only.
+- **Respect the dataset.** Do not modify task descriptions or skip tasks without recording a FAIL.

package/skills/run-benchmark/agents/claude.yaml

@@ -0,0 +1,3 @@
+frontmatter:
+  argument-hint: "[--category READ|CREATE|UPDATE|DELETE|FILE_MANIPULATION] [--sample N] [--resume]"
+  user-invocable: true