opendevbrowser 0.0.12 → 0.0.16

package/skills/opendevbrowser-data-extraction/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: opendevbrowser-data-extraction
+description: This skill should be used when the user asks to "extract data from a page", "scrape tables", "collect paginated results", "parse list/card content", or "export structured web data" with OpenDevBrowser.
+version: 2.0.0
+---
+
+# Data Extraction Skill
+
+Use this skill to extract structured, auditable datasets from dynamic pages with compliance-aware workflows.
+
+## Pack Contents
+
+- `artifacts/extraction-workflows.md`
+- `assets/templates/extraction-schema.json`
+- `assets/templates/pagination-state.json`
+- `assets/templates/quality-gates.json`
+- `assets/templates/compliance-checklist.md`
+- `scripts/run-extraction-workflow.sh`
+- `scripts/validate-skill-assets.sh`
+- Shared robustness matrix: `../opendevbrowser-best-practices/artifacts/browser-agent-known-issues-matrix.md`
+
+## Fast Start
+
+```bash
+./skills/opendevbrowser-data-extraction/scripts/validate-skill-assets.sh
+./skills/opendevbrowser-data-extraction/scripts/run-extraction-workflow.sh list
+./skills/opendevbrowser-data-extraction/scripts/run-extraction-workflow.sh pagination
+./skills/opendevbrowser-data-extraction/scripts/run-extraction-workflow.sh infinite-scroll
+```
+
+## Core Rules
+
+- Define schema before extraction.
+- Track provenance for each record (`source_url`, `provider`, `captured_at`, `page`).
+- Prefer embedded structured data (JSON-LD/microdata) where available.
+- Stop on sustained anti-bot pressure (repeated 403/429/challenge loops).
+- Honor `Retry-After` and preserve checkpoint state before retrying pagination.
+
+## Parallel Multitab Alignment
+
+- Apply shared concurrency policy from `../opendevbrowser-best-practices/SKILL.md` ("Parallel Operations").
+- Run extraction acceptance on `managed`, `extension`, and `cdpConnect` before claiming mode parity.
+- Keep one session per worker; avoid interleaving `target-use` streams inside a single session.
+
+## Robustness Coverage (Known-Issue Matrix)
+
+Matrix source: `../opendevbrowser-best-practices/artifacts/browser-agent-known-issues-matrix.md`
+
+- `ISSUE-01`: stale refs after dynamic content updates
+- `ISSUE-06`: 429/backoff and retry budgeting
+- `ISSUE-08`: blocked/restricted origins and policy checks
+- `ISSUE-09`: pagination drift, duplicate accumulation, terminal detection
+- `ISSUE-10`: locale/currency parsing consistency
+
+## Extraction Planning
+
+1. Define required fields and null policy.
+2. Snapshot and map refs to schema.
+3. Choose pagination strategy.
+4. Apply quality gates each page.
+
+```text
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+```
+
+## Structured Data First
+
+Attempt extraction in this order:
+1. JSON-LD product/article blocks
+2. semantic table/list/card DOM
+3. fallback text parsing
+
+```text
+opendevbrowser_dom_get_text sessionId="<session-id>" ref="<json-ld-ref>"
+opendevbrowser_dom_get_html sessionId="<session-id>" ref="<table-ref>"
+```
+
+## Pagination Patterns
+
+### Numbered/Next pagination
+
+```text
+opendevbrowser_click sessionId="<session-id>" ref="<next-ref>"
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+```
+
+### Infinite scroll
+
+```text
+opendevbrowser_scroll sessionId="<session-id>" dy=1000
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+```
+
+### Load more
+
+```text
+opendevbrowser_click sessionId="<session-id>" ref="<load-more-ref>"
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+```
+
+## Quality Gates
+
+Apply per page:
+- dedupe by stable key (URL or canonical ID)
+- null-rate check for required fields
+- count delta check (new records must increase)
+- consistency check for currency/units
+- max consecutive challenge/429 loops before stop
+
+Use `assets/templates/quality-gates.json`.
+
+## Compliance and Safety
+
+- Respect robots and site terms.
+- Use pacing; do not flood endpoints.
+- Treat robots as policy guidance, not auth.
+- Stop or back off on repeated 429/403 and challenges.
+
+## References
+
+- RFC 9309 (robots protocol): https://www.rfc-editor.org/rfc/rfc9309
+- Google robots docs: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
+- Schema.org Offer: https://schema.org/Offer
+- Google Product structured data: https://developers.google.com/search/docs/appearance/structured-data/product-snippet
+- Playwright best practices: https://playwright.dev/docs/best-practices

package/skills/opendevbrowser-data-extraction/artifacts/extraction-workflows.md

@@ -0,0 +1,31 @@
+# Extraction Workflows
+
+## List/Card workflow
+
+1. Snapshot actionables
+2. Map repeating nodes
+3. Extract title/price/link fields
+4. Normalize and dedupe records
+5. Record parse confidence and currency normalization status
+
+## Table workflow
+
+1. Wait for table visibility
+2. Extract table HTML
+3. Parse rows/cells in host process
+4. Validate schema completeness
+
+## Pagination workflow
+
+1. Extract current page
+2. Advance page
+3. Wait + snapshot
+4. Repeat until terminal condition
+5. Persist checkpoint state every page
+
+## Anti-bot pressure workflow
+
+1. Detect repeated 403/429/challenge
+2. Parse and honor `Retry-After` if present
+3. Slow down / cooldown and resume from last checkpoint
+4. Stop and report if pressure persists after bounded retries

package/skills/opendevbrowser-data-extraction/assets/templates/compliance-checklist.md

@@ -0,0 +1,7 @@
+# Compliance Checklist
+
+- robots policy reviewed
+- site terms reviewed for data-use constraints
+- extraction rate limited and paced
+- anti-bot pressure monitored (429/403/challenge)
+- sensitive data excluded from artifacts

package/skills/opendevbrowser-data-extraction/assets/templates/extraction-schema.json

@@ -0,0 +1,17 @@
+{
+  "dataset": "",
+  "required_fields": [
+    "title",
+    "url"
+  ],
+  "optional_fields": [
+    "price",
+    "rating",
+    "availability"
+  ],
+  "field_types": {
+    "title": "string",
+    "url": "string",
+    "price": "number"
+  }
+}

package/skills/opendevbrowser-data-extraction/assets/templates/pagination-state.json

@@ -0,0 +1,11 @@
+{
+  "strategy": "next",
+  "current_page": 1,
+  "pages_processed": 0,
+  "records_collected": 0,
+  "last_page_delta": 0,
+  "checkpoint_token": "",
+  "retry_after_seconds": 0,
+  "challenge_loops": 0,
+  "stopping_reason": ""
+}

package/skills/opendevbrowser-data-extraction/assets/templates/quality-gates.json

@@ -0,0 +1,10 @@
+{
+  "max_required_null_rate": 0.05,
+  "max_duplicate_rate": 0.02,
+  "require_positive_page_delta": true,
+  "max_consecutive_empty_pages": 2,
+  "max_consecutive_challenge_loops": 2,
+  "max_retries_per_page": 2,
+  "respect_retry_after": true,
+  "stop_on_repeated_anti_bot_signals": true
+}

package/skills/opendevbrowser-data-extraction/examples/sample-schema.json

@@ -0,0 +1,19 @@
+{
+  "dataset": "shopping_offers",
+  "required_fields": [
+    "title",
+    "url",
+    "price"
+  ],
+  "optional_fields": [
+    "rating",
+    "reviews_count",
+    "availability"
+  ],
+  "field_types": {
+    "title": "string",
+    "url": "string",
+    "price": "number",
+    "rating": "number"
+  }
+}

package/skills/opendevbrowser-data-extraction/scripts/run-extraction-workflow.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+print_help() {
+  cat <<'HELP'
+Data extraction workflow router
+
+Usage:
+  ./skills/opendevbrowser-data-extraction/scripts/run-extraction-workflow.sh <workflow>
+
+Workflows:
+  list
+  table
+  pagination
+  infinite-scroll
+  load-more
+  anti-bot-pressure
+  checkpoint-resume
+  list-workflows
+HELP
+}
+
+workflow="${1:-list-workflows}"
+
+case "$workflow" in
+  list-workflows)
+    print_help
+    ;;
+  list)
+    cat <<'FLOW'
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+opendevbrowser_dom_get_text sessionId="<session-id>" ref="<item-title-ref>"
+opendevbrowser_get_attr sessionId="<session-id>" ref="<item-link-ref>" name="href"
+FLOW
+    ;;
+  table)
+    cat <<'FLOW'
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+opendevbrowser_dom_get_html sessionId="<session-id>" ref="<table-ref>"
+FLOW
+    ;;
+  pagination)
+    cat <<'FLOW'
+opendevbrowser_click sessionId="<session-id>" ref="<next-ref>"
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+FLOW
+    ;;
+  infinite-scroll)
+    cat <<'FLOW'
+opendevbrowser_scroll sessionId="<session-id>" dy=1000
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+FLOW
+    ;;
+  load-more)
+    cat <<'FLOW'
+opendevbrowser_click sessionId="<session-id>" ref="<load-more-ref>"
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+FLOW
+    ;;
+  anti-bot-pressure)
+    cat <<'FLOW'
+opendevbrowser_network_poll sessionId="<session-id>" max=100
+# Stop or back off on repeated 403/429/challenge signals; honor Retry-After.
+FLOW
+    ;;
+  checkpoint-resume)
+    cat <<'FLOW'
+# Resume from stored pagination checkpoint token/page index.
+opendevbrowser_snapshot sessionId="<session-id>" format="outline"
+opendevbrowser_wait sessionId="<session-id>" until="networkidle"
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+FLOW
+    ;;
+  *)
+    echo "Unknown workflow: $workflow" >&2
+    print_help
+    exit 2
+    ;;
+esac

package/skills/opendevbrowser-data-extraction/scripts/validate-skill-assets.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+skill_file="$root/SKILL.md"
+
+required=(
+  "artifacts/extraction-workflows.md"
+  "assets/templates/extraction-schema.json"
+  "assets/templates/pagination-state.json"
+  "assets/templates/quality-gates.json"
+  "assets/templates/compliance-checklist.md"
+  "scripts/run-extraction-workflow.sh"
+  "scripts/validate-skill-assets.sh"
+)
+
+status=0
+for rel in "${required[@]}"; do
+  if [[ ! -f "$root/$rel" ]]; then
+    echo "Missing required asset: $rel" >&2
+    status=1
+  fi
+  if ! grep -Fq "$rel" "$skill_file"; then
+    echo "SKILL.md missing reference: $rel" >&2
+    status=1
+  fi
+done
+
+for rel in scripts/run-extraction-workflow.sh scripts/validate-skill-assets.sh; do
+  if [[ -f "$root/$rel" && ! -x "$root/$rel" ]]; then
+    echo "Script is not executable: $rel" >&2
+    status=1
+  fi
+done
+
+for rel in assets/templates/extraction-schema.json assets/templates/pagination-state.json assets/templates/quality-gates.json; do
+  if [[ -f "$root/$rel" ]]; then
+    if ! node -e 'const fs=require("fs"); JSON.parse(fs.readFileSync(process.argv[1], "utf8"));' "$root/$rel" >/dev/null 2>&1; then
+      echo "Invalid JSON template: $rel" >&2
+      status=1
+    fi
+  fi
+done
+
+if [[ $status -ne 0 ]]; then
+  exit $status
+fi
+
+echo "Data extraction skill assets validated."

package/skills/opendevbrowser-form-testing/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: opendevbrowser-form-testing
+description: This skill should be used when the user asks to "test a form", "verify validation", "check submission behavior", "test multi-step forms", or "debug form errors" with OpenDevBrowser.
+version: 2.0.0
+---
+
+# Form Testing Skill
+
+Use this skill for robust form automation across dynamic, multi-step, challenge-protected, and accessibility-critical flows.
+
+## Pack Contents
+
+- `artifacts/form-workflows.md`
+- `assets/templates/validation-matrix.json`
+- `assets/templates/challenge-decision-tree.json`
+- `assets/templates/a11y-assertions.md`
+- `assets/templates/multi-step-state.json`
+- `scripts/run-form-workflow.sh`
+- `scripts/validate-skill-assets.sh`
+- Shared robustness matrix: `../opendevbrowser-best-practices/artifacts/browser-agent-known-issues-matrix.md`
+
+## Fast Start
+
+```bash
+./skills/opendevbrowser-form-testing/scripts/validate-skill-assets.sh
+./skills/opendevbrowser-form-testing/scripts/run-form-workflow.sh validation
+./skills/opendevbrowser-form-testing/scripts/run-form-workflow.sh multi-step
+./skills/opendevbrowser-form-testing/scripts/run-form-workflow.sh challenge-checkpoint
+```
+
+## Core Rules
+
+- Snapshot first, then act.
+- Keep one field mutation per decision loop in failure analysis.
+- Re-snapshot after each invalid submit to avoid stale refs.
+- Verify both UX and network behavior for each submission branch.
+- Keep challenge retries bounded and honor `Retry-After` when 429 pressure appears.
+
+## Parallel Multitab Alignment
+
+- Apply shared concurrency policy from `../opendevbrowser-best-practices/SKILL.md` ("Parallel Operations").
+- Validate form workflows in `managed`, `extension`, and `cdpConnect` paths before parity sign-off.
+- Keep one session per worker and avoid alternating `target-use` inside one command stream.
+
+## Robustness Coverage (Known-Issue Matrix)
+
+Matrix source: `../opendevbrowser-best-practices/artifacts/browser-agent-known-issues-matrix.md`
+
+- `ISSUE-01`: stale refs after dynamic validation rerenders
+- `ISSUE-02`: iframe challenge or embedded payment/auth forms
+- `ISSUE-05`: anti-bot checkpoint loops on submit
+- `ISSUE-06`: rate-limit/backoff behavior
+- `ISSUE-08`: restricted origin or unsupported test surface
+
+## Form Coverage Matrix
+
+Cover at least:
+- required/optional fields
+- format/pattern constraints
+- numeric and length boundaries
+- dependent and conditional sections
+- disabled/enabled transitions
+- server-side rejection paths
+
+Use the matrix template in `assets/templates/validation-matrix.json`.
+
+| Validation area | Minimal assertion | Evidence capture |
+| --- | --- | --- |
+| Required vs optional | Empty required field blocks submit; optional field allows submit | `snapshot`, inline error text, submit request status |
+| Format and pattern | Invalid format shows field error and keeps form invalid | `dom_get_text`, `get_attr aria-invalid`, request payload |
+| Numeric and length bounds | Min/max boundaries are enforced consistently client+server | field value checks, server response body |
+| Conditional sections | Dependent fields appear/disappear with correct state reset | before/after snapshots, visibility checks |
+| Disabled/enabled transitions | Submit only enables when constraints are satisfied | `is_enabled`, transition snapshots |
+| Server-side rejection | Backend validation errors map to correct field/global errors | `network_poll`, rendered error copy |
+
+## Canonical Validation Flow
+
+```text
+opendevbrowser_snapshot sessionId="<session-id>" format="actionables"
+opendevbrowser_type sessionId="<session-id>" ref="<field-ref>" text=""
+opendevbrowser_click sessionId="<session-id>" ref="<submit-ref>"
+opendevbrowser_snapshot sessionId="<session-id>" format="outline"
+opendevbrowser_dom_get_text sessionId="<session-id>" ref="<error-ref>"
+```
+
+## Accessibility Assertions
+
+Validate error handling semantics:
+- error text bound to invalid field
+- `aria-invalid="true"` when invalid
+- keyboard focus on first invalid field when applicable
+- labels/instructions remain programmatically associated
+
+```text
+opendevbrowser_get_attr sessionId="<session-id>" ref="<field-ref>" name="aria-invalid"
+opendevbrowser_get_attr sessionId="<session-id>" ref="<field-ref>" name="aria-describedby"
+```
+
+## Anti-Bot and Challenge Branches
+
+For forms guarded by challenge providers:
+
+1. Detect challenge widget/frame state.
+2. Mark manual checkpoint in run log.
+3. Complete challenge manually or by approved test key in non-production.
+4. Resume from fresh snapshot and continue validation.
+
+Do not implement challenge bypass behavior.
+
+Loop guardrail:
+- Stop after 2 repeated challenge checkpoints without forward progress.
+- Capture network evidence and escalate as blocker.
+
+## Multi-Step and Conditional Forms
+
+For wizard flows:
+1. validate current step
+2. submit and wait for next-step container ref
+3. assert state carryover from prior step
+4. finalize and validate confirmation payload
+
+Use `assets/templates/multi-step-state.json` to track progression.
+
+## Network Correlation
+
+Always poll and inspect submit requests:
+
+```text
+opendevbrowser_network_poll sessionId="<session-id>" max=50
+```
+
+Flag mismatch cases:
+- UI shows success but network fails
+- UI hides server validation errors
+- repeated 429/403 challenge pressure on submit endpoint
+
+## References
+
+- Playwright best practices: https://playwright.dev/docs/best-practices
+- WCAG 2.2 (Input assistance + accessible authentication): https://www.w3.org/TR/WCAG22/
+- Cloudflare Turnstile testing: https://developers.cloudflare.com/turnstile/tutorials/testing/
+- reCAPTCHA v2 test keys: https://developers.google.com/recaptcha/docs/faq
+- hCaptcha test keys: https://docs.hcaptcha.com/

package/skills/opendevbrowser-form-testing/artifacts/form-workflows.md

@@ -0,0 +1,37 @@
+# Form Workflows
+
+## Validation workflow
+
+1. Discover refs
+2. Run negative matrix cases
+3. Assert field-level and summary errors
+4. Run positive submission path
+5. Re-snapshot after each submit branch
+
+## Multi-step workflow
+
+1. Validate each step in isolation
+2. Preserve and verify carried values
+3. Validate final confirmation state and IDs
+
+## Dynamic requirement workflow
+
+1. Trigger conditional sections (toggles/selectors)
+2. Re-snapshot and discover newly required fields
+3. Validate requiredness transitions and submit gating
+4. Verify hidden fields are not submitted unexpectedly
+
+## File upload workflow
+
+1. Validate accepted type/size constraints
+2. Validate invalid file rejection path
+3. Validate success path and server response correlation
+4. Confirm upload state survives step transitions
+
+## Challenge checkpoint workflow
+
+1. Detect challenge state
+2. Pause and annotate checkpoint
+3. Resume after manual completion
+4. Continue assertions from refreshed snapshot
+5. Stop and escalate after repeated challenge loops

package/skills/opendevbrowser-form-testing/assets/templates/a11y-assertions.md

@@ -0,0 +1,7 @@
+# Accessibility Assertions
+
+- Field has associated label
+- Error message is specific and linked (`aria-describedby`)
+- Invalid state uses `aria-invalid="true"`
+- Focus moves to first invalid field after submit
+- Keyboard-only path can complete correction and submit

package/skills/opendevbrowser-form-testing/assets/templates/challenge-decision-tree.json

@@ -0,0 +1,16 @@
+{
+  "challenge_detected": false,
+  "provider": "unknown",
+  "issue_ids": [
+    "ISSUE-02",
+    "ISSUE-05",
+    "ISSUE-06"
+  ],
+  "action": "manual_checkpoint",
+  "max_challenge_retries": 2,
+  "retry_after_seconds": 0,
+  "resume_conditions": [
+    "challenge_cleared",
+    "new_snapshot_taken"
+  ]
+}

package/skills/opendevbrowser-form-testing/assets/templates/multi-step-state.json

@@ -0,0 +1,11 @@
+{
+  "flow": "",
+  "step_index": 0,
+  "step_name": "",
+  "completed_steps": [],
+  "carried_state": {},
+  "final_confirmation": {
+    "seen": false,
+    "reference_id": ""
+  }
+}

package/skills/opendevbrowser-form-testing/assets/templates/validation-matrix.json

@@ -0,0 +1,24 @@
+{
+  "form": "",
+  "fields": [
+    {
+      "name": "",
+      "type": "text",
+      "required": true,
+      "cases": [
+        "empty",
+        "whitespace",
+        "out_of_range",
+        "valid"
+      ]
+    }
+  ],
+  "conditional_cases": [
+    "becomes_required_after_toggle",
+    "hidden_field_not_submitted"
+  ],
+  "submit_cases": [
+    "invalid_blocked",
+    "valid_success"
+  ]
+}

package/skills/opendevbrowser-form-testing/examples/sample-validation-matrix.json

@@ -0,0 +1,29 @@
+{
+  "form": "checkout",
+  "fields": [
+    {
+      "name": "email",
+      "type": "email",
+      "required": true,
+      "cases": [
+        "empty",
+        "invalid_format",
+        "valid"
+      ]
+    },
+    {
+      "name": "postal_code",
+      "type": "text",
+      "required": true,
+      "cases": [
+        "too_short",
+        "invalid_charset",
+        "valid"
+      ]
+    }
+  ],
+  "submit_cases": [
+    "invalid_blocked",
+    "valid_success"
+  ]
+}