@athenaflow/plugin-web-bench 1.0.5

package/dist/1.0.3/claude/plugin/skills/load-dataset/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: load-dataset
+description: >
+  Download and prepare the Halluminate/WebBench dataset from HuggingFace for benchmarking.
+  Triggers: "load dataset", "download WebBench", "prepare benchmark data", "fetch tasks".
+  Downloads the CSV dataset via curl, converts to JSONL with Node.js, applies optional filters
+  (category, sample size, website allowlist/blocklist), and writes web-bench-tasks.jsonl to the
+  working directory. Zero Python dependencies — uses only curl and Node.js.
+  Does NOT execute tasks — use execute-task for that.
+allowed-tools: Bash Read Write Edit Glob
+---
+
+# Load WebBench Dataset
+
+Download the Halluminate/WebBench dataset from HuggingFace and prepare it for benchmark execution.
+
+## Dataset Source
+
+- **HuggingFace:** `Halluminate/WebBench`
+- **Source file:** `webbenchfinal.csv` (CSV format)
+- **Size:** ~2,454 tasks across 452 websites
+- **Fields per row:** `ID` (int), `Starting_URL` (string), `Category` (enum), `Task` (string)
+
+## Pre-check: Skip Download if Dataset Exists
+
+Before downloading, check if `web-bench-tasks.jsonl` already exists in the working directory:
+
+```bash
+if [ -f web-bench-tasks.jsonl ]; then
+  echo "Dataset already exists: $(wc -l < web-bench-tasks.jsonl) tasks"
+  head -1 web-bench-tasks.jsonl
+fi
+```
+
+**If `web-bench-tasks.jsonl` exists and is non-empty, skip the download and conversion entirely.** Jump straight to [Applying Filters](#applying-filters) if filters need to be applied, or report the existing dataset to the tracker.
+
+Only proceed with download if the file does not exist or is empty.
+
+## Download Method
+
+Download the CSV directly with `curl`, then convert to JSONL with Node.js. No Python dependencies required.
+
+### Step 1: Download the CSV
+
+```bash
+curl -fSL -o web-bench-dataset.csv \
+  "https://huggingface.co/datasets/Halluminate/WebBench/resolve/main/webbenchfinal.csv"
+```
+
+If the above URL fails (HuggingFace sometimes changes paths), try:
+
+```bash
+curl -fSL -o web-bench-dataset.csv \
+  "https://huggingface.co/datasets/Halluminate/WebBench/raw/main/webbenchfinal.csv"
+```
+
+### Step 2: Convert CSV to JSONL
+
+```bash
+node -e "
+const fs = require('fs');
+const csv = fs.readFileSync('web-bench-dataset.csv', 'utf-8');
+const lines = csv.split('\n');
+const header = lines[0].split(',').map(h => h.trim().replace(/^\"|\"$/g, ''));
+
+// Find column indices
+const idIdx = header.findIndex(h => h === 'ID');
+const urlIdx = header.findIndex(h => h === 'Starting_URL');
+const catIdx = header.findIndex(h => h === 'Category');
+const taskIdx = header.findIndex(h => h === 'Task');
+
+const out = fs.createWriteStream('web-bench-tasks.jsonl');
+let count = 0;
+
+for (let i = 1; i < lines.length; i++) {
+  const line = lines[i].trim();
+  if (!line) continue;
+
+  // Parse CSV line respecting quoted fields
+  const fields = [];
+  let field = '';
+  let inQuotes = false;
+  for (let j = 0; j < line.length; j++) {
+    const ch = line[j];
+    if (ch === '\"') {
+      inQuotes = !inQuotes;
+    } else if (ch === ',' && !inQuotes) {
+      fields.push(field.trim());
+      field = '';
+    } else {
+      field += ch;
+    }
+  }
+  fields.push(field.trim());
+
+  if (fields.length > taskIdx) {
+    out.write(JSON.stringify({
+      id: parseInt(fields[idIdx], 10),
+      url: fields[urlIdx],
+      category: fields[catIdx],
+      task: fields[taskIdx]
+    }) + '\n');
+    count++;
+  }
+}
+
+out.end();
+console.log('Wrote ' + count + ' tasks to web-bench-tasks.jsonl');
+"
+```
+
+### Step 3: Verify the output
+
+```bash
+wc -l web-bench-tasks.jsonl
+head -1 web-bench-tasks.jsonl
+node -e "
+const fs = require('fs');
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+const cats = {};
+const sites = new Set();
+for (const t of tasks) {
+  cats[t.category] = (cats[t.category] || 0) + 1;
+  sites.add(t.url);
+}
+for (const [c, n] of Object.entries(cats).sort()) console.log('  ' + c + ': ' + n);
+console.log('Total: ' + tasks.length + ' tasks across ' + sites.size + ' websites');
+"
+```
+
+## Applying Filters
+
+After downloading, apply filters based on tracker configuration. All filters use Node.js.
+
+### Category Filter
+
+If the tracker specifies a category filter (e.g., `READ`, `CREATE`):
+
+```bash
+node -e "
+const fs = require('fs');
+const category = process.argv[1];
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+const filtered = tasks.filter(t => t.category === category);
+fs.writeFileSync('web-bench-tasks.jsonl', filtered.map(JSON.stringify).join('\n') + '\n');
+console.log('Filtered to ' + filtered.length + ' ' + category + ' tasks');
+" "READ"
+```
+
+### Sample Size
+
+If the tracker specifies a sample size (e.g., `--sample 50`):
+
+```bash
+node -e "
+const fs = require('fs');
+const n = parseInt(process.argv[1], 10);
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+
+// Deterministic shuffle (seed-based) for reproducibility
+function seededShuffle(arr, seed) {
+  const a = [...arr];
+  let s = seed;
+  for (let i = a.length - 1; i > 0; i--) {
+    s = (s * 1664525 + 1013904223) & 0xffffffff;
+    const j = ((s >>> 0) % (i + 1));
+    [a[i], a[j]] = [a[j], a[i]];
+  }
+  return a;
+}
+
+const sample = seededShuffle(tasks, 42).slice(0, Math.min(n, tasks.length));
+fs.writeFileSync('web-bench-tasks.jsonl', sample.map(JSON.stringify).join('\n') + '\n');
+console.log('Sampled ' + sample.length + ' tasks');
+" "50"
+```
+
+### Website Blocklist
+
+```bash
+node -e "
+const fs = require('fs');
+const blocklist = new Set(process.argv[1] ? process.argv[1].split(',') : []);
+const tasks = fs.readFileSync('web-bench-tasks.jsonl','utf-8').trim().split('\n').map(JSON.parse);
+const filtered = tasks.filter(t => !blocklist.has(t.url));
+fs.writeFileSync('web-bench-tasks.jsonl', filtered.map(JSON.stringify).join('\n') + '\n');
+console.log(filtered.length + ' tasks after blocklist filter');
+" ""
+```
+
+## Output
+
+- **File:** `web-bench-tasks.jsonl` in working directory
+- **Intermediate file:** `web-bench-dataset.csv` (can be deleted after conversion)
+- **Format:** One JSON object per line
+- **Schema:**
+  ```json
+  {"id": 42, "url": "https://acehardware.com", "category": "READ", "task": "Navigate to..."}
+  ```
+
+## Cleanup
+
+After successful conversion, remove the intermediate CSV:
+
+```bash
+rm -f web-bench-dataset.csv
+```
+
+Report the total count and category breakdown to the tracker.

package/dist/1.0.3/claude/plugin/skills/load-dataset/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false

package/dist/1.0.3/claude/plugin/skills/run-benchmark/SKILL.md

@@ -0,0 +1,92 @@
+---
+name: run-benchmark
+description: >
+  Run the WebBench browser agent benchmark — main entry point and orchestrator.
+  Triggers: "run benchmark", "run WebBench", "start benchmark", "benchmark browser agent",
+  "web bench", "execute WebBench", "run web-bench".
+  Parses user configuration (category filter, sample size, resume), delegates to
+  load-dataset, execute-task, evaluate-task, and generate-report skills.
+  This is the user-invocable orchestrator that ties the full benchmark pipeline together.
+allowed-tools: Read Write Edit Glob Grep Bash Task mcp__browser__ping mcp__browser__navigate mcp__browser__find mcp__browser__get_element mcp__browser__get_form mcp__browser__get_field mcp__browser__click mcp__browser__type mcp__browser__press mcp__browser__select mcp__browser__hover mcp__browser__drag mcp__browser__scroll mcp__browser__scroll_to mcp__browser__wheel mcp__browser__snapshot mcp__browser__screenshot mcp__browser__go_back mcp__browser__go_forward mcp__browser__reload mcp__browser__list_pages mcp__browser__close_page
+---
+
+# Run WebBench Benchmark
+
+Main entry point for running the WebBench browser agent benchmark. This skill is used in interactive (single-session) mode. For multi-session workflow execution, see the system prompt.
+
+## Input
+
+Parse configuration from: `$ARGUMENTS`
+
+Supported flags:
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--category <CAT>` | Filter tasks by category (READ, CREATE, UPDATE, DELETE, FILE_MANIPULATION) | All categories |
+| `--sample <N>` | Random sample of N tasks (deterministic seed=42) | Full dataset |
+| `--resume` | Resume from existing web-bench-results.jsonl, skip completed task IDs | Fresh run |
+| `--report-only` | Skip execution, just generate report from existing results | Full run |
+
+Examples:
+- `run-benchmark --category READ --sample 50` — 50 random READ tasks
+- `run-benchmark --resume` — continue from where last run stopped
+- `run-benchmark --report-only` — just aggregate existing results
+
+## Interactive Execution Protocol
+
+When run interactively (not via the workflow loop), this skill executes the full pipeline in a single session:
+
+### 1. Setup
+
+1. Parse arguments
+2. Check for existing state (`web-bench-tasks.jsonl`, `web-bench-results.jsonl`)
+3. If `--resume` and results exist: determine completed task IDs, skip them
+4. If not resuming: load the `load-dataset` skill to download and prepare the dataset
+5. Report configuration and task count
+
+### 2. Execute Tasks
+
+For each task in `web-bench-tasks.jsonl` (skipping completed if resuming):
+
+1. Read the task line
+2. Record start time: `date +%s%3N`
+3. Load `execute-task` methodology and perform browser automation
+4. Load `evaluate-task` methodology and score the result
+5. Record end time: `date +%s%3N`, compute duration
+6. Append result to `web-bench-results.jsonl`:
+   ```json
+   {"id": 42, "url": "...", "category": "READ", "task": "...", "score": 1.0, "verdict": "PASS", "reasoning": "...", "error": null, "duration_ms": 34200, "tokens_used": {"input": 12450, "output": 3200}, "timestamp": "2026-03-19T14:30:00Z"}
+   ```
+7. Print progress: `[42/2454] PASS (1.0) — acehardware.com — READ — 34.2s`
+
+### 3. Generate Report
+
+After all tasks are processed (or if `--report-only`):
+
+1. Load `generate-report` methodology
+2. Aggregate `web-bench-results.jsonl` into `web-bench-report.md`
+3. Print summary statistics to console
+
+## Token Tracking
+
+Token usage should be tracked per task. The agent should estimate tokens consumed during task execution by recording:
+
+- **Input tokens:** Approximate from the size of prompts, page snapshots, and tool responses received during execution
+- **Output tokens:** Approximate from the size of responses and tool calls generated
+
+If exact token counts are available from the session metadata, prefer those over estimates.
+
+## Progress Display
+
+After each task, print a status line:
+
+```
+[1/50] PASS  (1.0)  acehardware.com          READ    34.2s   15,650 tokens
+[2/50] FAIL  (0.0)  airbnb.com               CREATE  12.1s    8,200 tokens  [auth_required]
+[3/50] PARTIAL(0.5) amazon.com               READ    45.8s   22,100 tokens
+```
+
+## Guardrails
+
+- **Always append, never overwrite** results. The JSONL file is append-only.
+- **Respect the dataset.** Do not modify task descriptions or skip tasks without recording a FAIL.

package/dist/1.0.3/claude/plugin/skills/run-benchmark/agents/claude.yaml

@@ -0,0 +1,3 @@
+frontmatter:
+  argument-hint: "[--category READ|CREATE|UPDATE|DELETE|FILE_MANIPULATION] [--sample N] [--resume]"
+  user-invocable: true

package/dist/1.0.3/codex/plugin/.codex-plugin/plugin.json

@@ -0,0 +1,16 @@
+{
+  "name": "web-bench",
+  "version": "1.0.3",
+  "description": "WebBench benchmark runner \u2014 executes real-world browser tasks from the Halluminate/WebBench dataset, scores via LLM-as-judge, and produces evaluation reports",
+  "author": {
+    "name": "Athenaflow"
+  },
+  "skills": "./skills/",
+  "mcpServers": "./.mcp.json",
+  "interface": {
+    "displayName": "WebBench",
+    "shortDescription": "Browser task benchmarks scored via LLM-as-judge",
+    "developerName": "Athenaflow",
+    "category": "Evaluation"
+  }
+}

package/dist/1.0.3/codex/plugin/.mcp.json

@@ -0,0 +1,19 @@
+{
+  "mcpServers": {
+    "agent-web-interface": {
+      "command": "npx",
+      "args": ["-y", "agent-web-interface@latest"],
+      "env": {
+        "NODE_ENV": "production"
+      },
+      "options": [
+        { "label": "Auto (user → persistent → isolated)", "env": {} },
+        { "label": "User's Chrome", "env": { "AWI_BROWSER_MODE": "user" } },
+        { "label": "Persistent profile", "env": { "AWI_BROWSER_MODE": "persistent" } },
+        { "label": "Isolated (temp profile)", "env": { "AWI_BROWSER_MODE": "isolated" } },
+        { "label": "Headless", "env": { "AWI_HEADLESS": "true" } },
+        { "label": "Connect to CDP endpoint", "env": { "AWI_CDP_URL": "http://localhost:9222" } }
+      ]
+    }
+  }
+}

package/dist/1.0.3/codex/plugin/package.json

@@ -0,0 +1,9 @@
+{
+  "name": "@athenaflow/plugin-web-bench",
+  "version": "1.0.3",
+  "description": "WebBench benchmark runner — executes real-world browser tasks from the Halluminate/WebBench dataset, scores via LLM-as-judge, and produces evaluation reports",
+  "license": "MIT",
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/dist/1.0.3/codex/plugin/skills/evaluate-task/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: evaluate-task
+description: >
+  Evaluate whether a WebBench task was successfully completed using LLM-as-judge scoring.
+  Triggers: "evaluate task", "score task", "judge result", "grade benchmark task".
+  Examines the execution trace, final page state, and extracted data against the original
+  task description. Produces a structured verdict (PASS/PARTIAL/FAIL) with reasoning.
+  Does NOT execute browser actions — use execute-task for that.
+allowed-tools: Read Write Edit
+---
+
+# Evaluate WebBench Task
+
+Judge whether a completed task execution meets the success criteria defined in the original task description. This is a post-hoc evaluation — no browser interaction, only analysis of the execution trace and captured state.
+
+## Input
+
+You receive:
+
+1. **Original task:** `{"id": 42, "url": "...", "category": "READ", "task": "Navigate to the news section and summarize..."}`
+2. **Execution trace:** Actions taken, final URL, extracted data, blockers encountered, screenshots
+
+## Scoring Rubric
+
+### Verdict Scale
+
+| Verdict | Score | Criteria |
+|---------|-------|----------|
+| **PASS** | 1.0 | Task fully completed. All requested information extracted or all requested actions performed. |
+| **PARTIAL** | 0.5 | Task partially completed. Some but not all requirements met. Meaningful progress was made. |
+| **FAIL** | 0.0 | Task not completed. No meaningful progress, wrong information, or blocked before starting. |
+
+### Category-Specific Evaluation
+
+#### READ Tasks
+- **PASS:** All requested data was extracted accurately and completely
+- **PARTIAL:** Some data extracted but incomplete (e.g., found the page but missed some fields)
+- **FAIL:** Wrong data, wrong page, or no data extracted
+
+#### CREATE Tasks
+- **PASS:** Item was created as specified, confirmation visible
+- **PARTIAL:** Creation started but not confirmed (e.g., form filled but not submitted)
+- **FAIL:** Could not reach the creation form, or creation failed
+
+#### UPDATE Tasks
+- **PASS:** Data was modified as specified, change confirmed
+- **PARTIAL:** Found the item but could not complete the modification
+- **FAIL:** Could not find the item or reach the edit interface
+
+#### DELETE Tasks
+- **PASS:** Item was deleted and removal confirmed
+- **PARTIAL:** Found the item and initiated deletion but could not confirm
+- **FAIL:** Could not find the item or reach the delete action
+
+#### FILE_MANIPULATION Tasks
+- **PASS:** File downloaded with correct name/content
+- **PARTIAL:** Download initiated but not verified
+- **FAIL:** Could not locate or download the file
+
+### Blocker Handling
+
+If the execution trace contains blockers, evaluate based on the blocker type:
+
+| Blocker | Verdict | Reasoning |
+|---------|---------|-----------|
+| Login required (no credentials) | FAIL | Infrastructure limitation — task requires auth |
+| CAPTCHA | FAIL | Infrastructure limitation — cannot solve programmatically |
+| Site down / 404 | FAIL | External dependency — site unavailable |
+| Geo-restricted | FAIL | Infrastructure limitation — content not accessible |
+| Paywall | FAIL | Infrastructure limitation — paid content |
+| Pop-up could not be dismissed | PARTIAL or FAIL | Depends on whether task could proceed |
+
+### Evaluation Dimensions
+
+Score each dimension and use them to determine the overall verdict:
+
+1. **Navigation (required):** Did the agent reach the correct page/section?
+   - Correct site? Correct section? Correct page?
+
+2. **Comprehension (required):** Did the agent understand what was being asked?
+   - Did it attempt the right type of action? Did it target the right elements?
+
+3. **Completeness (required):** Did the agent fulfill ALL parts of the task?
+   - Multi-part tasks: each part must be addressed
+   - Quantitative tasks: all requested data points must be present
+
+4. **Accuracy (for READ tasks):** Is the extracted information correct?
+   - Does it match what's visible on the page?
+   - Are numbers, names, and details accurate?
+
+5. **Confirmation (for WRITE tasks):** Is there evidence the action was performed?
+   - Success message visible? Item appears in list? State changed?
+
+## Evaluation Process
+
+### Step 1: Parse the Task Requirements
+
+Break the task description into discrete, verifiable requirements:
+
+```
+Task: "Navigate to the news section and summarize the headline and key points from the latest science policy update."
+
+Requirements:
+1. Navigate to the news section
+2. Find the latest science policy update
+3. Extract the headline
+4. Extract key points
+```
+
+### Step 2: Check Each Requirement Against the Trace
+
+For each requirement, determine if the execution trace shows it was fulfilled:
+
+```
+1. Navigate to news section → DONE (action 2: clicked "News", URL changed to /news)
+2. Find latest science policy update → DONE (action 3: found article "New Science Policy...")
+3. Extract headline → DONE (extracted: "New Science Policy Framework Announced")
+4. Extract key points → NOT DONE (only headline extracted, no key points)
+```
+
+### Step 3: Determine Verdict
+
+- All requirements met → **PASS**
+- Some requirements met → **PARTIAL**
+- No requirements met or fundamentally wrong approach → **FAIL**
+
+### Step 4: Write Reasoning
+
+Provide clear, structured reasoning:
+
+```
+Verdict: PARTIAL (0.5)
+Reasoning: Agent successfully navigated to the news section and identified the correct article.
+The headline was extracted accurately. However, the task also requested "key points" from the
+article, which were not extracted. 3 of 4 requirements met.
+```
+
+## Output Format
+
+Return a structured evaluation result:
+
+```json
+{
+  "task_id": 42,
+  "verdict": "PARTIAL",
+  "score": 0.5,
+  "reasoning": "Agent navigated correctly and extracted the headline, but missed the key points requirement. 3/4 requirements fulfilled.",
+  "requirements_met": 3,
+  "requirements_total": 4,
+  "blocker": null
+}
+```
+
+If a blocker prevented execution:
+
+```json
+{
+  "task_id": 43,
+  "verdict": "FAIL",
+  "score": 0.0,
+  "reasoning": "Task requires account login. No credentials available — infrastructure limitation.",
+  "requirements_met": 0,
+  "requirements_total": 3,
+  "blocker": "auth_required"
+}
+```
+
+## Guardrails
+
+- **Be strict but fair.** A task that asks for 5 data points and delivers 4 is PARTIAL, not PASS.
+- **Do not hallucinate success.** If the trace doesn't show evidence of completion, it didn't happen.
+- **Separate agent failure from infrastructure failure.** Auth requirements, CAPTCHAs, and site outages are not agent failures — but they are still FAIL verdicts for scoring purposes. Note the distinction in reasoning.
+- **Evaluate what was asked, not what was attempted.** A well-executed wrong approach is still a FAIL.

package/dist/1.0.3/codex/plugin/skills/evaluate-task/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false

package/dist/1.0.3/codex/plugin/skills/execute-task/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: execute-task
+description: >
+  Execute a single WebBench benchmark task via browser automation.
+  Triggers: "execute task", "run task", "perform benchmark task", "browser task".
+  Navigates to the Starting_URL, interprets the natural-language task description,
+  performs the required browser actions, and captures the final state (screenshot + snapshot).
+  Records an execution trace with actions taken and errors encountered.
+  Does NOT evaluate success — use evaluate-task for that.
+allowed-tools: Read Write Edit Bash mcp__browser__ping mcp__browser__navigate mcp__browser__find mcp__browser__get_element mcp__browser__get_form mcp__browser__get_field mcp__browser__click mcp__browser__type mcp__browser__press mcp__browser__select mcp__browser__hover mcp__browser__drag mcp__browser__scroll mcp__browser__scroll_to mcp__browser__wheel mcp__browser__snapshot mcp__browser__screenshot mcp__browser__go_back mcp__browser__go_forward mcp__browser__reload mcp__browser__list_pages mcp__browser__close_page
+---
+
+# Execute WebBench Task
+
+Execute a single WebBench task via browser automation. This skill handles the browser interaction — navigating, clicking, typing, extracting data — required to complete the task.
+
+## Input
+
+You receive a single task object:
+
+```json
+{"id": 42, "url": "https://acehardware.com", "category": "READ", "task": "Navigate to the news section and summarize..."}
+```
+
+## Execution Protocol
+
+### 1. Record Start Time
+
+Before any browser interaction:
+
+```bash
+date +%s%3N
+```
+
+Save this as `start_time_ms`. You will need it for the result record.
+
+### 2. Navigate to Starting URL
+
+```
+navigate → task.url
+```
+
+Wait for the page to load. Take an initial snapshot to understand the page structure.
+
+### 3. Interpret the Task
+
+Read the task description carefully. Classify it:
+
+| Category | What to Do |
+|----------|------------|
+| **READ** | Navigate to the right page/section, extract the requested information. Your "result" is the extracted data. |
+| **CREATE** | Fill forms, create accounts/entries/posts as described. Your "result" is confirmation the item was created. |
+| **UPDATE** | Find existing data and modify it as described. Your "result" is confirmation the update was applied. |
+| **DELETE** | Find and remove the specified item. Your "result" is confirmation of deletion. |
+| **FILE_MANIPULATION** | Download the specified file. Your "result" is the filename and confirmation of download. |
+
+### 4. Execute Browser Actions
+
+Work through the task step by step:
+
+1. **Observe** — Use `snapshot` or `find` to understand the current page state
+2. **Plan** — Decide the next action based on what you see
+3. **Act** — Use the appropriate MCP tool (click, type, select, etc.)
+4. **Verify** — Check that the action had the expected effect
+
+**Key principles:**
+
+- **Use `find` with `kind` filters** to locate interactive elements (buttons, links, textboxes)
+- **Use `snapshot`** to get a full page state when you need orientation
+- **Use `screenshot`** to visually verify state when snapshots are ambiguous
+- **Handle pop-ups and modals** — cookie banners, newsletter pop-ups, chat widgets. Dismiss them before proceeding
+- **Stay on the specified site** — Tasks often say "Only use [site] to achieve the task." Respect this constraint
+- **Handle pagination** — If data spans multiple pages, navigate through them
+- **Be patient with slow sites** — Some sites load content dynamically. If elements aren't found immediately, try scrolling or waiting
+
+### 5. Handle Common Obstacles
+
+| Obstacle | Strategy |
+|----------|----------|
+| **Cookie consent banner** | Find and click "Accept" or "Close" |
+| **Login required** | Record as blocker — do not attempt to create accounts or guess credentials |
+| **CAPTCHA** | Record as blocker — cannot solve programmatically |
+| **Paywall** | Record as blocker |
+| **Geo-restricted content** | Record as blocker |
+| **Site down / 404** | Record as error |
+| **Pop-up / overlay blocking interaction** | Dismiss it, then continue |
+
+### 6. Capture Final State
+
+After completing the task (or hitting a blocker):
+
+1. Take a **screenshot** of the final page state
+2. Take a **snapshot** of the final DOM state
+3. Record the **current URL**
+
+### 7. Record End Time
+
+```bash
+date +%s%3N
+```
+
+Save as `end_time_ms`. Compute `duration_ms = end_time_ms - start_time_ms`.
+
+### 8. Build Execution Trace
+
+Construct a structured trace of what happened:
+
+```json
+{
+  "task_id": 42,
+  "actions": [
+    {"step": 1, "action": "navigate", "target": "https://acehardware.com", "result": "loaded"},
+    {"step": 2, "action": "click", "target": "News section link", "result": "navigated to /news"},
+    {"step": 3, "action": "extract", "target": "headline text", "result": "Black Friday Deals..."}
+  ],
+  "final_url": "https://acehardware.com/news",
+  "blockers": [],
+  "extracted_data": "The headline is 'Black Friday Deals'...",
+  "duration_ms": 34200
+}
+```
+
+## Output
+
+Return the execution trace to the calling context (system prompt / run-benchmark skill). Do NOT write results to disk — that is the system prompt's responsibility after evaluation.
+
+## Guardrails
+
+- **One task only.** Execute exactly the task given. Do not browse further or attempt other tasks.
+- **No account creation.** If a task requires login, record it as a blocker. Do not create accounts.
+- **No credential guessing.** Never attempt to guess passwords or bypass authentication.
+- **Time limit awareness.** If a task is taking an unreasonable number of steps (>20 actions), consider it likely stuck and record what you have.
+- **No destructive actions on real sites.** For WRITE/UPDATE/DELETE tasks on production sites, be aware these are real websites. If the task would create real accounts or modify real data, record this concern but still attempt the task as specified (this is the nature of the benchmark).

package/dist/1.0.3/codex/plugin/skills/execute-task/agents/claude.yaml

@@ -0,0 +1,2 @@
+frontmatter:
+  user-invocable: false