@athenaflow/plugin-web-bench 1.0.5

package/skills/evaluate-task/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: evaluate-task
+description: >
+  Evaluate whether a WebBench task was successfully completed using LLM-as-judge scoring.
+  Triggers: "evaluate task", "score task", "judge result", "grade benchmark task".
+  Examines the execution trace, final page state, and extracted data against the original
+  task description. Produces a structured verdict (PASS/PARTIAL/FAIL) with reasoning.
+  Does NOT execute browser actions — use execute-task for that.
+allowed-tools: Read Write Edit
+---
+
+# Evaluate WebBench Task
+
+Judge whether a completed task execution meets the success criteria defined in the original task description. This is a post-hoc evaluation — no browser interaction, only analysis of the execution trace and captured state.
+
+## Input
+
+You receive:
+
+1. **Original task:** `{"id": 42, "url": "...", "category": "READ", "task": "Navigate to the news section and summarize..."}`
+2. **Execution trace:** Actions taken, final URL, extracted data, blockers encountered, screenshots
+
+## Scoring Rubric
+
+### Verdict Scale
+
+| Verdict | Score | Criteria |
+|---------|-------|----------|
+| **PASS** | 1.0 | Task fully completed. All requested information extracted or all requested actions performed. |
+| **PARTIAL** | 0.5 | Task partially completed. Some but not all requirements met. Meaningful progress was made. |
+| **FAIL** | 0.0 | Task not completed. No meaningful progress, wrong information, or blocked before starting. |
+
+### Category-Specific Evaluation
+
+#### READ Tasks
+- **PASS:** All requested data was extracted accurately and completely
+- **PARTIAL:** Some data extracted but incomplete (e.g., found the page but missed some fields)
+- **FAIL:** Wrong data, wrong page, or no data extracted
+
+#### CREATE Tasks
+- **PASS:** Item was created as specified, confirmation visible
+- **PARTIAL:** Creation started but not confirmed (e.g., form filled but not submitted)
+- **FAIL:** Could not reach the creation form, or creation failed
+
+#### UPDATE Tasks
+- **PASS:** Data was modified as specified, change confirmed
+- **PARTIAL:** Found the item but could not complete the modification
+- **FAIL:** Could not find the item or reach the edit interface
+
+#### DELETE Tasks
+- **PASS:** Item was deleted and removal confirmed
+- **PARTIAL:** Found the item and initiated deletion but could not confirm
+- **FAIL:** Could not find the item or reach the delete action
+
+#### FILE_MANIPULATION Tasks
+- **PASS:** File downloaded with correct name/content
+- **PARTIAL:** Download initiated but not verified
+- **FAIL:** Could not locate or download the file
+
+### Blocker Handling
+
+If the execution trace contains blockers, evaluate based on the blocker type:
+
+| Blocker | Verdict | Reasoning |
+|---------|---------|-----------|
+| Login required (no credentials) | FAIL | Infrastructure limitation — task requires auth |
+| CAPTCHA | FAIL | Infrastructure limitation — cannot solve programmatically |
+| Site down / 404 | FAIL | External dependency — site unavailable |
+| Geo-restricted | FAIL | Infrastructure limitation — content not accessible |
+| Paywall | FAIL | Infrastructure limitation — paid content |
+| Pop-up could not be dismissed | PARTIAL or FAIL | Depends on whether task could proceed |
+
+### Evaluation Dimensions
+
+Score each dimension and use them to determine the overall verdict:
+
+1. **Navigation (required):** Did the agent reach the correct page/section?
+   - Correct site? Correct section? Correct page?
+
+2. **Comprehension (required):** Did the agent understand what was being asked?
+   - Did it attempt the right type of action? Did it target the right elements?
+
+3. **Completeness (required):** Did the agent fulfill ALL parts of the task?
+   - Multi-part tasks: each part must be addressed
+   - Quantitative tasks: all requested data points must be present
+
+4. **Accuracy (for READ tasks):** Is the extracted information correct?
+   - Does it match what's visible on the page?
+   - Are numbers, names, and details accurate?
+
+5. **Confirmation (for WRITE tasks):** Is there evidence the action was performed?
+   - Success message visible? Item appears in list? State changed?
+
+## Evaluation Process
+
+### Step 1: Parse the Task Requirements
+
+Break the task description into discrete, verifiable requirements:
+
+```
+Task: "Navigate to the news section and summarize the headline and key points from the latest science policy update."
+
+Requirements:
+1. Navigate to the news section
+2. Find the latest science policy update
+3. Extract the headline
+4. Extract key points
+```
+
+### Step 2: Check Each Requirement Against the Trace
+
+For each requirement, determine if the execution trace shows it was fulfilled:
+
+```
+1. Navigate to news section → DONE (action 2: clicked "News", URL changed to /news)
+2. Find latest science policy update → DONE (action 3: found article "New Science Policy...")
+3. Extract headline → DONE (extracted: "New Science Policy Framework Announced")
+4. Extract key points → NOT DONE (only headline extracted, no key points)
+```
+
+### Step 3: Determine Verdict
+
+- All requirements met → **PASS**
+- Some requirements met → **PARTIAL**
+- No requirements met or fundamentally wrong approach → **FAIL**
+
+### Step 4: Write Reasoning
+
+Provide clear, structured reasoning:
+
+```
+Verdict: PARTIAL (0.5)
+Reasoning: Agent successfully navigated to the news section and identified the correct article.
+The headline was extracted accurately. However, the task also requested "key points" from the
+article, which were not extracted. 3 of 4 requirements met.
+```
+
+## Output Format
+
+Return a structured evaluation result:
+
+```json
+{
+  "task_id": 42,
+  "verdict": "PARTIAL",
+  "score": 0.5,
+  "reasoning": "Agent navigated correctly and extracted the headline, but missed the key points requirement. 3/4 requirements fulfilled.",
+  "requirements_met": 3,
+  "requirements_total": 4,
+  "blocker": null
+}
+```
+
+If a blocker prevented execution:
+
+```json
+{
+  "task_id": 43,
+  "verdict": "FAIL",
+  "score": 0.0,
+  "reasoning": "Task requires account login. No credentials available — infrastructure limitation.",
+  "requirements_met": 0,
+  "requirements_total": 3,
+  "blocker": "auth_required"
+}
+```
+
+## Guardrails
+
+- **Be strict but fair.** A task that asks for 5 data points and delivers 4 is PARTIAL, not PASS.
+- **Do not hallucinate success.** If the trace doesn't show evidence of completion, it didn't happen.
+- **Separate agent failure from infrastructure failure.** Auth requirements, CAPTCHAs, and site outages are not agent failures — but they are still FAIL verdicts for scoring purposes. Note the distinction in reasoning.
+- **Evaluate what was asked, not what was attempted.** A well-executed wrong approach is still a FAIL.

package/skills/evaluate-task/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false

package/skills/execute-task/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: execute-task
+description: >
+  Methodology for executing a single WebBench benchmark task via browser automation.
+  Triggers: "execute task", "run task", "perform benchmark task", "browser task".
+  Interprets the natural-language task description, defines the required browser actions,
+  and specifies what final evidence to capture (for example screenshot + snapshot).
+  Records an execution trace with actions taken and errors encountered.
+  Does NOT evaluate success — use evaluate-task for that.
+allowed-tools: Read Write Edit Bash
+---
+
+# Execute WebBench Task
+
+Execute a single WebBench task using a browser-capable calling context. This skill does not own browser MCP tools; it defines the execution protocol the caller should follow while navigating, clicking, typing, and extracting data.
+
+## Input
+
+You receive a single task object:
+
+```json
+{"id": 42, "url": "https://acehardware.com", "category": "READ", "task": "Navigate to the news section and summarize..."}
+```
+
+## Execution Protocol
+
+### 1. Record Start Time
+
+Before any browser interaction by the calling context:
+
+```bash
+date +%s%3N
+```
+
+Save this as `start_time_ms`. You will need it for the result record.
+
+### 2. Navigate to Starting URL
+
+```
+navigate → task.url
+```
+
+The calling context should wait for the page to load and take an initial snapshot to understand the page structure.
+
+### 3. Interpret the Task
+
+Read the task description carefully. Classify it:
+
+| Category | What to Do |
+|----------|------------|
+| **READ** | Navigate to the right page/section, extract the requested information. Your "result" is the extracted data. |
+| **CREATE** | Fill forms, create accounts/entries/posts as described. Your "result" is confirmation the item was created. |
+| **UPDATE** | Find existing data and modify it as described. Your "result" is confirmation the update was applied. |
+| **DELETE** | Find and remove the specified item. Your "result" is confirmation of deletion. |
+| **FILE_MANIPULATION** | Download the specified file. Your "result" is the filename and confirmation of download. |
+
+### 4. Execute Browser Actions
+
+Work through the task step by step:
+
+1. **Observe** — Use `snapshot` or `find` to understand the current page state
+2. **Plan** — Decide the next action based on what you see
+3. **Act** — Use the appropriate browser tool available in the calling context (`click`, `type`, `select`, etc.)
+4. **Verify** — Check that the action had the expected effect
+
+**Key principles:**
+
+- **Use `find` with `kind` filters** to locate interactive elements (buttons, links, textboxes)
+- **Use `snapshot`** to get a full page state when you need orientation
+- **Use `screenshot`** to visually verify state when snapshots are ambiguous
+- **Handle pop-ups and modals** — cookie banners, newsletter pop-ups, chat widgets. Dismiss them before proceeding
+- **Stay on the specified site** — Tasks often say "Only use [site] to achieve the task." Respect this constraint
+- **Handle pagination** — If data spans multiple pages, navigate through them
+- **Be patient with slow sites** — Some sites load content dynamically. If elements aren't found immediately, try scrolling or waiting
+
+### 5. Handle Common Obstacles
+
+| Obstacle | Strategy |
+|----------|----------|
+| **Cookie consent banner** | Find and click "Accept" or "Close" |
+| **Login required** | Record as blocker — do not attempt to create accounts or guess credentials |
+| **CAPTCHA** | Record as blocker — cannot solve programmatically |
+| **Paywall** | Record as blocker |
+| **Geo-restricted content** | Record as blocker |
+| **Site down / 404** | Record as error |
+| **Pop-up / overlay blocking interaction** | Dismiss it, then continue |
+
+### 6. Capture Final State
+
+After completing the task (or hitting a blocker):
+
+1. Have the calling context take a **screenshot** of the final page state
+2. Have the calling context take a **snapshot** of the final DOM state
+3. Record the **current URL**
+
+### 7. Record End Time
+
+```bash
+date +%s%3N
+```
+
+Save as `end_time_ms`. Compute `duration_ms = end_time_ms - start_time_ms`.
+
+### 8. Build Execution Trace
+
+Construct a structured trace of what happened:
+
+```json
+{
+  "task_id": 42,
+  "actions": [
+    {"step": 1, "action": "navigate", "target": "https://acehardware.com", "result": "loaded"},
+    {"step": 2, "action": "click", "target": "News section link", "result": "navigated to /news"},
+    {"step": 3, "action": "extract", "target": "headline text", "result": "Black Friday Deals..."}
+  ],
+  "final_url": "https://acehardware.com/news",
+  "blockers": [],
+  "extracted_data": "The headline is 'Black Friday Deals'...",
+  "duration_ms": 34200
+}
+```
+
+## Output
+
+Return the execution trace to the calling context (system prompt / run-benchmark skill). Do NOT write results to disk — that is the system prompt's responsibility after evaluation.
+
+## Guardrails
+
+- **One task only.** Execute exactly the task given. Do not browse further or attempt other tasks.
+- **No account creation.** If a task requires login, record it as a blocker. Do not create accounts.
+- **No credential guessing.** Never attempt to guess passwords or bypass authentication.
+- **Time limit awareness.** If a task is taking an unreasonable number of steps (>20 actions), consider it likely stuck and record what you have.
+- **No destructive actions on real sites.** For WRITE/UPDATE/DELETE tasks on production sites, be aware these are real websites. If the task would create real accounts or modify real data, record this concern but still attempt the task as specified (this is the nature of the benchmark).

package/skills/execute-task/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false

package/skills/generate-report/SKILL.md

@@ -0,0 +1,204 @@
+---
+name: generate-report
+description: >
+  Aggregate WebBench benchmark results into a comprehensive evaluation report.
+  Triggers: "generate report", "create benchmark report", "summarize results",
+  "aggregate scores", "produce evaluation report".
+  Reads web-bench-results.jsonl, computes statistics by category/website/failure mode,
+  and writes web-bench-report.md with pass rates, timing, token usage, and analysis.
+  Does NOT execute or evaluate tasks — only aggregates existing results.
+allowed-tools: Read Write Edit Bash Glob Grep
+---
+
+# Generate WebBench Benchmark Report
+
+Aggregate all results from `web-bench-results.jsonl` into a comprehensive markdown report.
+
+## Input
+
+- **Results file:** `web-bench-results.jsonl` — one JSON line per completed task
+- **Result line schema:**
+  ```json
+  {
+    "id": 42,
+    "url": "https://acehardware.com",
+    "category": "READ",
+    "task": "Navigate to...",
+    "score": 1.0,
+    "verdict": "PASS",
+    "reasoning": "Successfully extracted all specs",
+    "error": null,
+    "duration_ms": 34200,
+    "tokens_used": {"input": 12450, "output": 3200},
+    "timestamp": "2026-03-19T14:30:00Z"
+  }
+  ```
+
+## Report Generation
+
+Use Node.js to compute all statistics:
+
+```bash
+node -e "
+const fs = require('fs');
+const results = fs.readFileSync('web-bench-results.jsonl','utf-8').trim().split('\n')
+  .filter(l => l.trim()).map(JSON.parse);
+
+const total = results.length;
+const passed = results.filter(r => r.verdict === 'PASS').length;
+const partial = results.filter(r => r.verdict === 'PARTIAL').length;
+const failed = results.filter(r => r.verdict === 'FAIL').length;
+const totalScore = results.reduce((s, r) => s + r.score, 0);
+
+// Timing
+const totalDurationMs = results.reduce((s, r) => s + (r.duration_ms || 0), 0);
+const avgDurationMs = total ? totalDurationMs / total : 0;
+
+// Tokens
+const totalInputTokens = results.reduce((s, r) => s + ((r.tokens_used || {}).input || 0), 0);
+const totalOutputTokens = results.reduce((s, r) => s + ((r.tokens_used || {}).output || 0), 0);
+const totalTokens = totalInputTokens + totalOutputTokens;
+
+// By category
+const byCat = {};
+for (const r of results) {
+  if (!byCat[r.category]) byCat[r.category] = [];
+  byCat[r.category].push(r);
+}
+
+// By website (top failures)
+const bySite = {};
+for (const r of results) {
+  if (!bySite[r.url]) bySite[r.url] = [];
+  bySite[r.url].push(r);
+}
+
+// Failure modes
+const blockers = {};
+for (const r of results) {
+  if (r.error) blockers[r.error] = (blockers[r.error] || 0) + 1;
+}
+
+const catStats = {};
+for (const [cat, rs] of Object.entries(byCat)) {
+  catStats[cat] = {
+    total: rs.length,
+    passed: rs.filter(r => r.verdict === 'PASS').length,
+    score: rs.reduce((s, r) => s + r.score, 0) / rs.length
+  };
+}
+
+const siteFailures = Object.entries(bySite)
+  .map(([site, rs]) => [site, rs.filter(r => r.verdict === 'FAIL').length])
+  .sort((a, b) => b[1] - a[1]).slice(0, 20);
+
+console.log(JSON.stringify({
+  total, passed, partial, failed,
+  totalScore, avgScore: total ? totalScore / total : 0,
+  totalDurationMs, avgDurationMs,
+  totalInputTokens, totalOutputTokens, totalTokens,
+  byCategory: catStats,
+  bySiteFailures: Object.fromEntries(siteFailures),
+  blockers
+}, null, 2));
+"
+```
+
+## Report Template
+
+Write `web-bench-report.md` with this structure:
+
+```markdown
+# WebBench Benchmark Report
+
+**Date:** {timestamp}
+**Agent:** {agent identifier}
+**Dataset:** Halluminate/WebBench
+**Tasks evaluated:** {total} / 2454
+
+---
+
+## Overall Results
+
+| Metric | Value |
+|--------|-------|
+| **Pass Rate** | {passed}/{total} ({pass_pct}%) |
+| **Partial Rate** | {partial}/{total} ({partial_pct}%) |
+| **Fail Rate** | {failed}/{total} ({fail_pct}%) |
+| **Average Score** | {avg_score:.2f} / 1.0 |
+| **Total Duration** | {total_duration_formatted} |
+| **Avg Duration/Task** | {avg_duration_formatted} |
+| **Total Tokens** | {total_tokens:,} ({input_tokens:,} input + {output_tokens:,} output) |
+| **Avg Tokens/Task** | {avg_tokens:,} |
+
+## Results by Category
+
+| Category | Total | Pass | Partial | Fail | Pass Rate | Avg Score |
+|----------|-------|------|---------|------|-----------|-----------|
+| READ | ... | ... | ... | ... | ...% | ... |
+| CREATE | ... | ... | ... | ... | ...% | ... |
+| UPDATE | ... | ... | ... | ... | ...% | ... |
+| DELETE | ... | ... | ... | ... | ...% | ... |
+| FILE_MANIPULATION | ... | ... | ... | ... | ...% | ... |
+
+## Timing Breakdown
+
+| Category | Avg Duration | Min | Max |
+|----------|-------------|-----|-----|
+| READ | ... | ... | ... |
+| CREATE | ... | ... | ... |
+| ... | | | |
+
+## Token Usage Breakdown
+
+| Category | Avg Input Tokens | Avg Output Tokens | Avg Total |
+|----------|-----------------|-------------------|-----------|
+| READ | ... | ... | ... |
+| CREATE | ... | ... | ... |
+| ... | | | |
+
+## Top Failure Modes
+
+| Failure Mode | Count | % of Failures |
+|-------------|-------|---------------|
+| Auth required | ... | ... |
+| CAPTCHA | ... | ... |
+| Site unavailable | ... | ... |
+| Navigation failure | ... | ... |
+
+## Worst Performing Websites (by failure count)
+
+| Website | Tasks | Failures | Failure Rate |
+|---------|-------|----------|-------------|
+| ... | ... | ... | ... |
+
+## Best Performing Websites (by pass rate, min 3 tasks)
+
+| Website | Tasks | Pass Rate | Avg Score |
+|---------|-------|-----------|-----------|
+| ... | ... | ... | ... |
+
+## Sample Failures
+
+{Show 5-10 representative failures with task description, what went wrong, and verdict reasoning}
+
+## Methodology
+
+- **Execution:** One task per session via agent-web-interface browser automation
+- **Evaluation:** LLM-as-judge with structured rubric (PASS=1.0, PARTIAL=0.5, FAIL=0.0)
+- **Scoring dimensions:** Navigation, Comprehension, Completeness, Accuracy, Confirmation
+- **Infrastructure blockers** (auth, CAPTCHA, site down) scored as FAIL but flagged separately
+```
+
+## Output
+
+- **File:** `web-bench-report.md` in working directory
+- Report should be self-contained and readable without the raw JSONL data
+
+## Guardrails
+
+- Use the JSONL file as the sole source of truth — do not fabricate statistics
+- Format all durations as human-readable (e.g., "2h 34m 12s" not "9252000ms")
+- Format token counts with thousands separators
+- Round percentages to one decimal place
+- If results file has fewer than the expected total tasks, note this prominently in the report header

package/skills/generate-report/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false