opencode-skills-collection 4.0.53 → 4.0.54

package/bundled-skills/web-scraper/references/detailed-guide.md

@@ -0,0 +1,792 @@
+# Web Scraper - Detailed Guide
+
+> This file contains the detailed procedure and reference material extracted from `SKILL.md` for focused loading. The root skill defines activation, examples, safety constraints, and limitations.
+
+## Overview
+
+Web scraping inteligente multi-estrategia. Extrai dados estruturados de paginas web (tabelas, listas, precos). Paginacao, monitoramento e export CSV/JSON.
+
+
+## How It Works
+
+Execute phases in strict order. Each phase feeds the next.
+
+```
+1. CLARIFY  ->  2. RECON  ->  3. STRATEGY  ->  4. EXTRACT  ->  5. TRANSFORM  ->  6. VALIDATE  ->  7. FORMAT
+```
+
+Never skip Phase 1 or Phase 2. They prevent wasted effort and failed extractions.
+
+**Fast path**: If user provides URL + clear data target + the request is simple
+(single page, one data type), compress Phases 1-3 into a single action:
+fetch, classify, and extract in one WebFetch call. Still validate and format.
+
+---
+
+
+## Escalation Consent
+
+- Auto-escalation from WebFetch to Browser automation is allowed only for URLs
+  the user explicitly provided.
+- For URLs found via Discovery Mode (WebSearch) or links discovered inside
+  scraped pages, ask the user before driving a browser on them.
+
+---
+
+
+## Capabilities
+
+- **Multi-strategy**: WebFetch (static), Browser automation (JS-rendered), Bash/curl (APIs), WebSearch (discovery)
+- **Extraction modes**: table, list, article, product, contact, FAQ, pricing, events, jobs, custom
+- **Output formats**: Markdown tables (default), JSON, CSV
+- **Pagination**: auto-detect and follow (page numbers, infinite scroll, load-more)
+- **Multi-URL**: extract same structure across sources with comparison and diff
+- **Validation**: confidence ratings (HIGH/MEDIUM/LOW) on every extraction
+- **Auto-escalation**: WebFetch fails silently -> automatic Browser fallback
+  (user-provided URLs only; see Escalation Consent)
+- **Data transforms**: cleaning, normalization, deduplication, enrichment
+- **Differential mode**: detect changes between scraping runs
+
+
+## Web Scraper
+
+Multi-strategy web data extraction with intelligent approach selection,
+automatic fallback escalation, data transformation, and structured output.
+
+
+## Phase 1: Clarify
+
+Establish extraction parameters before touching any URL.
+
+
+## Required Parameters
+
+| Parameter     | Resolve                              | Default        |
+|:--------------|:-------------------------------------|:---------------|
+| Target URL(s) | Which page(s) to scrape?             | *(required)*   |
+| Data Target   | What specific data to extract?       | *(required)*   |
+| Output Format | Markdown table, JSON, CSV, or text?  | Markdown table |
+| Scope         | Single page, paginated, or multi-URL?| Single page    |
+
+
+## Optional Parameters
+
+| Parameter     | Resolve                                | Default      |
+|:--------------|:---------------------------------------|:-------------|
+| Pagination    | Follow pagination? Max pages?          | No, 1 page   |
+| Max Items     | Maximum number of items to collect?    | Unlimited    |
+| Filters       | Data to exclude or include?            | None         |
+| Sort Order    | How to sort results?                   | Source order  |
+| Save Path     | Save to file? Which path?              | Display only |
+| Language      | Respond in which language?             | User's lang  |
+| Diff Mode     | Compare with previous run?             | No           |
+
+
+## Clarification Rules
+
+- If user provides a URL and clear data target, proceed directly to Phase 2.
+  Do NOT ask unnecessary questions.
+- If request is ambiguous (e.g. "scrape this site"), ask ONLY:
+  "What specific data do you want me to extract from this page?"
+- Default to Markdown table output. Mention alternatives only if relevant.
+- Accept requests in any language. Always respond in the user's language.
+- If user says "everything" or "all data", perform recon first, then present
+  what's available and let user choose.
+
+
+## Discovery Mode
+
+When user has a topic but no specific URL:
+1. Use WebSearch to find the most relevant pages
+2. Present top 3-5 URLs with descriptions
+3. Let user choose which to scrape, or scrape all
+4. Proceed to Phase 2 with selected URL(s)
+
+Example: "find and extract pricing data for CRM tools"
+-> WebSearch("CRM tools pricing comparison 2026")
+-> Present top results -> User selects -> Extract
+
+---
+
+
+## Phase 2: Reconnaissance
+
+Analyze the target page before extraction.
+
+
+## Step 2.1: Initial Fetch
+
+Use WebFetch to retrieve and analyze the page structure:
+
+```
+WebFetch(
+  url = TARGET_URL,
+  prompt = "Analyze this page structure and report:
+    1. Page type: article, product listing, search results, data table,
+       directory, dashboard, API docs, FAQ, pricing page, job board, events, or other
+    2. Main content structure: tables, ordered/unordered lists, card grid, free-form text,
+       accordion/collapsible sections, tabs
+    3. Approximate number of distinct data items visible
+    4. JavaScript rendering indicators: empty containers, loading spinners,
+       SPA framework markers (React root, Vue app, Angular), minimal HTML with heavy JS
+    5. Pagination: next/prev links, page numbers, load-more buttons,
+       infinite scroll indicators, total results count
+    6. Data density: how much structured, extractable data exists
+    7. List the main data fields/columns available for extraction
+    8. Embedded structured data: JSON-LD, microdata, OpenGraph tags
+    9. Available download links: CSV, Excel, PDF, API endpoints"
+)
+```
+
+
+## Step 2.2: Evaluate Fetch Quality
+
+| Signal                                      | Interpretation                    | Action                    |
+|:--------------------------------------------|:----------------------------------|:--------------------------|
+| Rich content with data clearly visible      | Static page                       | Strategy A (WebFetch)     |
+| Empty containers, "loading...", minimal text | JS-rendered                       | Strategy B (Browser)      |
+| Login wall, CAPTCHA, 403/401 response       | Blocked                           | Report to user            |
+| Content present but poorly structured       | Needs precision                   | Strategy B (Browser)      |
+| JSON or XML response body                   | API endpoint                      | Strategy C (Bash/curl)    |
+| Download links for CSV/Excel available      | Direct data file                  | Strategy C (download)     |
+
+
+## Step 2.3: Content Classification
+
+Classify into an extraction mode:
+
+| Mode       | Indicators                                 | Examples                          |
+|:-----------|:-------------------------------------------|:----------------------------------|
+| `table`    | HTML `<table>`, grid layout with headers   | Price comparison, statistics, specs|
+| `list`     | Repeated similar elements, card grids      | Search results, product listings  |
+| `article`  | Long-form text with headings/paragraphs    | Blog post, news article, docs     |
+| `product`  | Product name, price, specs, images, rating | E-commerce product page           |
+| `contact`  | Names, emails, phones, addresses, roles    | Team page, staff directory        |
+| `faq`      | Question-answer pairs, accordions          | FAQ page, help center             |
+| `pricing`  | Plan names, prices, features, tiers        | SaaS pricing page                 |
+| `events`   | Dates, locations, titles, descriptions     | Event listings, conferences       |
+| `jobs`     | Titles, companies, locations, salaries     | Job boards, career pages          |
+| `custom`   | User specified CSS selectors or fields     | Anything not matching above       |
+
+Record: **page type**, **extraction mode**, **JS rendering needed (yes/no)**,
+**available fields**, **structured data present (JSON-LD etc.)**.
+
+If user asked for "everything", present the available fields and let them choose.
+
+---
+
+
+## Phase 3: Strategy Selection
+
+Choose the extraction approach based on recon results.
+
+
+## Decision Tree
+
+```
+Structured data (JSON-LD, microdata) has what we need?
+ |
+ +-- YES --> STRATEGY E: Extract structured data directly
+ |
+ +-- NO: Content fully visible in WebFetch?
+      |
+      +-- YES: Need precise element targeting?
+      |    |
+      |    +-- NO  --> STRATEGY A: WebFetch + AI extraction
+      |    +-- YES --> STRATEGY B: Browser automation
+      |
+      +-- NO: JavaScript rendering detected?
+           |
+           +-- YES --> STRATEGY B: Browser automation
+           +-- NO:  API/JSON/XML endpoint or download link?
+                |
+                +-- YES --> STRATEGY C: Bash (curl + jq)
+                +-- NO  --> Report access issue to user
+```
+
+
+## Strategy A: Webfetch With Ai Extraction
+
+**Best for**: Static pages, articles, simple tables, well-structured HTML.
+
+Use WebFetch with a targeted extraction prompt tailored to the mode:
+
+```
+WebFetch(
+  url = URL,
+  prompt = "Extract [DATA_TARGET] from this page.
+    Return ONLY the extracted data as [FORMAT] with these columns/fields: [FIELDS].
+    Rules:
+    - If a value is missing or unclear, use 'N/A'
+    - Do not include navigation, ads, footers, or unrelated content
+    - Preserve original values exactly (numbers, currencies, dates)
+    - Include ALL matching items, not just the first few
+    - For each item, also extract the URL/link if available"
+)
+```
+
+**Auto-escalation**: If WebFetch returns suspiciously few items (less than
+50% of expected from recon), or mostly empty fields, escalate to Strategy B.
+Escalate automatically only for URLs the user explicitly provided; for
+discovered URLs, ask first (see Escalation Consent). Log the escalation in notes.
+
+
+## Strategy B: Browser Automation
+
+**Best for**: JS-rendered pages, SPAs, interactive content, lazy-loaded data.
+
+Sequence:
+1. Get tab context: `tabs_context_mcp(createIfEmpty=true)` -> get tabId
+2. Navigate to URL: `navigate(url=TARGET_URL, tabId=TAB)`
+3. Wait for content to load: `computer(action="wait", duration=3, tabId=TAB)`
+4. Check for cookie/consent banners: `find(query="cookie consent or accept button", tabId=TAB)`
+   - If found, dismiss it (prefer privacy-preserving option)
+5. Read page structure: `read_page(tabId=TAB)` or `get_page_text(tabId=TAB)`
+6. Locate target elements: `find(query="[DESCRIPTION]", tabId=TAB)`
+7. Extract with JavaScript for precise data via `javascript_tool`
+
+```javascript
+// Table extraction
+const rows = document.querySelectorAll('TABLE_SELECTOR tr');
+const data = Array.from(rows).map(row => {
+  const cells = row.querySelectorAll('td, th');
+  return Array.from(cells).map(c => c.textContent.trim());
+});
+JSON.stringify(data);
+```
+
+```javascript
+// List/card extraction
+const items = document.querySelectorAll('ITEM_SELECTOR');
+const data = Array.from(items).map(item => ({
+  field1: item.querySelector('FIELD1_SELECTOR')?.textContent?.trim() || null,
+  field2: item.querySelector('FIELD2_SELECTOR')?.textContent?.trim() || null,
+  link: item.querySelector('a')?.href || null,
+}));
+JSON.stringify(data);
+```
+
+8. For lazy-loaded content, scroll and re-extract:
+   `computer(action="scroll", scroll_direction="down", tabId=TAB)`
+   then `computer(action="wait", duration=2, tabId=TAB)`
+
+
+## Strategy C: Bash (Curl + Jq)
+
+**Best for**: REST APIs, JSON endpoints, XML feeds, CSV/Excel downloads.
+
+```bash
+
+## Json Api
+
+curl -s "API_URL" | jq '[.items[] | {field1: .key1, field2: .key2}]'
+
+## Csv Download
+
+## Confirm The Exact Output Path With The User Before Downloading
+
+curl --fail --silent --show-error --location \
+  --output "<CONFIRMED_OUTPUT_PATH>/scraped_data.csv" -- "CSV_URL"
+
+## Xml Parsing
+
+curl -s "XML_URL" | python3 -c "
+import xml.etree.ElementTree as ET, json, sys
+tree = ET.parse(sys.stdin)
+
+## ... Parse And Output Json
+
+"
+```
+
+
+## Strategy D: Hybrid
+
+When a single strategy is insufficient, combine:
+1. WebSearch to discover relevant URLs
+2. WebFetch for initial content assessment
+3. Browser automation for JS-heavy sections
+4. Bash for post-processing (jq, python for data cleaning)
+
+
+## Strategy E: Structured Data Extraction
+
+When JSON-LD, microdata, or OpenGraph is present:
+1. Use Browser `javascript_tool` to extract structured data:
+```javascript
+const scripts = document.querySelectorAll('script[type="application/ld+json"]');
+const data = Array.from(scripts).map(s => {
+  try { return JSON.parse(s.textContent); } catch { return null; }
+}).filter(Boolean);
+JSON.stringify(data);
+```
+2. This often provides cleaner, more reliable data than DOM scraping
+3. Fall back to DOM extraction only for fields not in structured data
+
+
+## Pagination Handling
+
+When pagination is detected and user wants multiple pages:
+
+**Page-number pagination (any strategy):**
+1. Extract data from current page
+2. Identify URL pattern (e.g. `?page=N`, `/page/N`, `&offset=N`)
+3. Iterate through pages up to user's max (default: 5 pages)
+4. Show progress: "Extracting page 2/5..."
+5. Concatenate all results, deduplicate if needed
+
+**Infinite scroll (Browser only):**
+1. Extract currently visible data
+2. Record item count
+3. Scroll down: `computer(action="scroll", scroll_direction="down", tabId=TAB)`
+4. Wait: `computer(action="wait", duration=2, tabId=TAB)`
+5. Extract newly loaded data
+6. Compare count - if no new items after 2 scrolls, stop
+7. Repeat until no new content or max iterations (default: 5)
+
+**"Load More" button (Browser only):**
+1. Extract currently visible data
+2. Find button: `find(query="load more button", tabId=TAB)`
+3. Click it: `computer(action="left_click", ref=REF, tabId=TAB)`
+4. Wait and extract new content
+5. Repeat until button disappears or max iterations reached
+
+---
+
+
+## Phase 4: Extract
+
+Execute the selected strategy using mode-specific patterns.
+See [references/extraction-patterns.md](../references/extraction-patterns.md)
+for CSS selectors and JavaScript snippets.
+
+
+## Table Mode
+
+WebFetch prompt:
+```
+"Extract ALL rows from the table(s) on this page.
+Return as a markdown table with exact column headers.
+Include every row - do not truncate or summarize.
+Preserve numeric precision, currencies, and units."
+```
+
+
+## List Mode
+
+WebFetch prompt:
+```
+"Extract each [ITEM_TYPE] from this page.
+For each item, extract: [FIELD_LIST].
+Return as a JSON array of objects with these keys: [KEY_LIST].
+Include ALL items, not just the first few. Include link/URL for each item if available."
+```
+
+
+## Article Mode
+
+WebFetch prompt:
+```
+"Extract article metadata:
+- title, author, date, tags/categories, word count estimate
+- Key factual data points, statistics, and named entities
+Return as structured markdown. Summarize the content; do not reproduce full text."
+```
+
+
+## Product Mode
+
+WebFetch prompt:
+```
+"Extract product data with these exact fields:
+- name, brand, price, currency, originalPrice (if discounted),
+  availability, description (first 200 chars), rating, reviewCount,
+  specifications (as key-value pairs), productUrl, imageUrl
+Return as JSON. Use null for missing fields."
+```
+
+Also check for JSON-LD `Product` schema (Strategy E) first.
+
+
+## Contact Mode
+
+WebFetch prompt:
+```
+"Extract contact information for each person/entity:
+- name, title, role, email, phone, address, organization, website, linkedinUrl
+Return as a markdown table. Only extract real contacts visible on the page."
+```
+
+
+## Faq Mode
+
+WebFetch prompt:
+```
+"Extract all question-answer pairs from this page.
+For each FAQ item extract:
+- question: the exact question text
+- answer: the answer text (first 300 chars if long)
+- category: the section/category if grouped
+Return as a JSON array of objects."
+```
+
+
+## Pricing Mode
+
+WebFetch prompt:
+```
+"Extract all pricing plans/tiers from this page.
+For each plan extract:
+- planName, monthlyPrice, annualPrice, currency
+- features (array of included features)
+- limitations (array of limits or excluded features)
+- ctaText (call-to-action button text)
+- highlighted (true if marked as recommended/popular)
+Return as JSON. Use null for missing fields."
+```
+
+
+## Events Mode
+
+WebFetch prompt:
+```
+"Extract all events/sessions from this page.
+For each event extract:
+- title, date, time, endTime, location, description (first 200 chars)
+- speakers (array of names), category, registrationUrl
+Return as JSON. Use null for missing fields."
+```
+
+
+## Jobs Mode
+
+WebFetch prompt:
+```
+"Extract all job listings from this page.
+For each job extract:
+- title, company, location, salary, salaryRange, type (full-time/part-time/contract)
+- postedDate, description (first 200 chars), applyUrl, tags
+Return as JSON. Use null for missing fields."
+```
+
+
+## Custom Mode
+
+When user provides specific selectors or field descriptions:
+- Use Browser automation with `javascript_tool` and user's CSS selectors
+- Or use WebFetch with a prompt built from user's field descriptions
+- Always confirm extracted schema with user before proceeding to multi-URL
+
+
+## Multi-Url Extraction
+
+When extracting from multiple URLs:
+1. Extract from the **first URL** to establish the data schema
+2. Show user the first results and confirm the schema is correct
+3. Extract from remaining URLs using the same schema
+4. Add a `source` column/field to every record with the origin URL
+5. Combine all results into a single output
+6. Show progress: "Extracting 3/7 URLs..."
+
+---
+
+
+## Phase 5: Transform
+
+Clean, normalize, and enrich extracted data before validation.
+See [references/data-transforms.md](../references/data-transforms.md) for patterns.
+
+
+## Automatic Transforms (Always Apply)
+
+| Transform              | Action                                               |
+|:-----------------------|:-----------------------------------------------------|
+| Whitespace cleanup     | Trim, collapse multiple spaces, remove `\n` in cells |
+| HTML entity decode     | `&amp;` -> `&`, `&lt;` -> `<`, `&#39;` -> `'`       |
+| Unicode normalization  | NFKC normalization for consistent characters          |
+| Empty string to null   | `""` -> `null` (for JSON), `""` -> `N/A` (for tables)|
+
+
+## Conditional Transforms (Apply When Relevant)
+
+| Transform             | When                         | Action                                  |
+|:----------------------|:-----------------------------|:----------------------------------------|
+| Price normalization   | Product/pricing modes        | Extract numeric value + currency symbol |
+| Date normalization    | Any dates found              | Normalize to ISO-8601 (YYYY-MM-DD)      |
+| URL resolution        | Relative URLs extracted      | Convert to absolute URLs                |
+| Phone normalization   | Contact mode                 | Standardize to E.164 format if possible |
+| Deduplication         | Multi-page or multi-URL      | Remove exact duplicate rows             |
+| Sorting               | User requested or natural    | Sort by user-specified field            |
+
+
+## Data Enrichment (Only When Useful)
+
+| Enrichment             | When                         | Action                                |
+|:-----------------------|:-----------------------------|:--------------------------------------|
+| Currency conversion    | User asks for single currency| Note original + convert (approximate) |
+| Domain extraction      | URLs in data                 | Add domain column from full URLs      |
+| Word count             | Article mode                 | Count words in extracted text         |
+| Relative dates         | Dates present                | Add "X days ago" column if useful     |
+
+
+## Deduplication Strategy
+
+When combining data from multiple pages or URLs:
+1. Exact match: rows with identical values in all fields -> keep first
+2. Near match: rows with same key fields (name+source) but different details
+   -> keep most complete (fewer nulls), flag in notes
+3. Report: "Removed N duplicate rows" in delivery notes
+
+---
+
+
+## Phase 6: Validate
+
+Verify extraction quality before delivering results.
+
+
+## Validation Checks
+
+| Check                | Action                                              |
+|:---------------------|:----------------------------------------------------|
+| Item count           | Compare extracted count to expected count from recon |
+| Empty fields         | Count N/A or null values per field                   |
+| Data type consistency| Numbers should be numeric, dates parseable           |
+| Duplicates           | Flag exact duplicate rows (post-dedup)               |
+| Encoding             | Check for HTML entities, garbled characters           |
+| Completeness         | All user-requested fields present in output          |
+| Truncation           | Verify data wasn't cut off (check last items)        |
+| Outliers             | Flag values that seem anomalous (e.g. $0.00 price)  |
+
+
+## Confidence Rating
+
+Assign to every extraction:
+
+| Rating     | Criteria                                                        |
+|:-----------|:----------------------------------------------------------------|
+| **HIGH**   | All fields populated, count matches expected, no anomalies      |
+| **MEDIUM** | Minor gaps (<10% empty fields) or count slightly differs        |
+| **LOW**    | Significant gaps (>10% empty), structural issues, partial data  |
+
+Always report confidence with specifics:
+> Confidence: **HIGH** - 47 items extracted, all 6 fields populated,
+> matches expected count from page analysis.
+
+
+## Auto-Recovery (Try Before Reporting Issues)
+
+| Issue              | Auto-Recovery Action                                  |
+|:-------------------|:------------------------------------------------------|
+| Missing data       | Re-attempt with Browser if WebFetch was used          |
+| Encoding problems  | Apply HTML entity decode + unicode normalization      |
+| Incomplete results | Check for pagination or lazy-loading, fetch more      |
+| Count mismatch     | Scroll/paginate to find remaining items               |
+| All fields empty   | Page likely JS-rendered, switch to Browser strategy   |
+| Partial fields     | Try JSON-LD extraction as supplement                  |
+
+Log all recovery attempts in delivery notes.
+Inform user of any irrecoverable gaps with specific details.
+
+---
+
+
+## Phase 7: Format And Deliver
+
+Structure results according to user preference.
+See [references/output-templates.md](../references/output-templates.md)
+for complete formatting templates.
+
+
+## Delivery Envelope
+
+ALWAYS wrap results with this metadata header:
+
+```markdown
+
+## Extraction Results
+
+**Source:** [Page Title](http://example.com)
+**Date:** YYYY-MM-DD HH:MM UTC
+**Items:** N records (M fields each)
+**Confidence:** HIGH | MEDIUM | LOW
+**Strategy:** A (WebFetch) | B (Browser) | C (API) | E (Structured Data)
+**Format:** Markdown Table | JSON | CSV
+
+---
+
+[DATA HERE]
+
+---
+
+**Notes:**
+- [Any gaps, issues, or observations]
+- [Transforms applied: deduplication, normalization, etc.]
+- [Pages scraped if paginated: "Pages 1-5 of 12"]
+- [Auto-escalation if it occurred: "Escalated from WebFetch to Browser"]
+```
+
+
+## Markdown Table Rules
+
+- Left-align text columns (`:---`), right-align numbers (`---:`)
+- Consistent column widths for readability
+- Include summary row for numeric data when useful (totals, averages)
+- Maximum 10 columns per table; split wider data into multiple tables
+  or suggest JSON format
+- Truncate long cell values to 60 chars with `...` indicator
+- Use `N/A` for missing values, never leave cells empty
+- For multi-page results, show combined table (not per-page)
+
+
+## Json Rules
+
+- Use camelCase for keys (e.g. `productName`, `unitPrice`)
+- Wrap in metadata envelope:
+  ```json
+  {
+    "metadata": {
+      "source": "URL",
+      "title": "Page Title",
+      "extractedAt": "ISO-8601",
+      "itemCount": 47,
+      "fieldCount": 6,
+      "confidence": "HIGH",
+      "strategy": "A",
+      "transforms": ["deduplication", "priceNormalization"],
+      "notes": []
+    },
+    "data": [ ... ]
+  }
+  ```
+- Pretty-print with 2-space indentation
+- Numbers as numbers (not strings), booleans as booleans
+- null for missing values (not empty strings)
+
+
+## Csv Rules
+
+- First row is always headers
+- Quote any field containing commas, quotes, or newlines
+- UTF-8 encoding with BOM for Excel compatibility
+- Use `,` as delimiter (standard)
+- Include metadata as comments: `# Source: URL`
+
+
+## File Output
+
+When user requests file save:
+- Markdown: `.md` extension
+- JSON: `.json` extension
+- CSV: `.csv` extension
+- Confirm path before writing
+- Report full file path and item count after saving
+
+
+## Multi-Url Comparison Format
+
+When comparing data across multiple sources:
+- Add `Source` as the first column/field
+- Use short identifiers for sources (domain name or user label)
+- Group by source or interleave based on user preference
+- Highlight differences if user asks for comparison
+- Include summary: "Best price: $X at store-b.com"
+
+
+## Differential Output
+
+When user requests change detection (diff mode):
+- Compare current extraction with previous run
+- Mark new items with `[NEW]`
+- Mark removed items with `[REMOVED]`
+- Mark changed values with `[WAS: old_value]`
+- Include summary: "Changes since last run: +5 new, -2 removed, 3 modified"
+
+---
+
+
+## Rate Limiting
+
+- Maximum 1 request per 2 seconds for sequential page fetches
+- For multi-URL jobs, process sequentially with pauses
+- If a site returns 429 (Too Many Requests), stop and report to user
+
+
+## Access Respect
+
+- If a page blocks access (403, CAPTCHA, login wall), report to user
+- Do NOT attempt to bypass bot detection, CAPTCHAs, or access controls
+- Do NOT scrape behind authentication unless user explicitly provides access
+- Respect robots.txt directives when known
+
+
+## Copyright
+
+- Do NOT reproduce large blocks of copyrighted article text
+- For articles: extract factual data, statistics, and structured info;
+  summarize narrative content
+- Always include source attribution (http://example.com) in output
+
+
+## Data Scope
+
+- Extract ONLY what the user explicitly requested
+- Warn user before collecting potentially sensitive data at scale
+  (emails, phone numbers, personal information)
+- Do not store or transmit extracted data beyond what the user sees
+
+
+## Failure Protocol
+
+When extraction fails or is blocked:
+1. Explain the specific reason (JS rendering, bot detection, login, etc.)
+2. Suggest alternatives (different URL, API if available, manual approach)
+3. Never retry aggressively or escalate access attempts
+
+---
+
+
+## Quick Reference: Mode Cheat Sheet
+
+| User Says...                         | Mode      | Strategy  | Output Default   |
+|:-------------------------------------|:----------|:----------|:-----------------|
+| "extract the table"                  | table     | A or B    | Markdown table   |
+| "get all products/prices"            | product   | E then A  | Markdown table   |
+| "scrape the listings"                | list      | A or B    | Markdown table   |
+| "extract contact info / team page"   | contact   | A         | Markdown table   |
+| "get the article data"               | article   | A         | Markdown text    |
+| "extract the FAQ"                    | faq       | A or B    | JSON             |
+| "get pricing plans"                  | pricing   | A or B    | Markdown table   |
+| "scrape job listings"                | jobs      | A or B    | Markdown table   |
+| "get event schedule"                 | events    | A or B    | Markdown table   |
+| "find and extract [topic]"           | discovery | WebSearch | Markdown table   |
+| "compare prices across sites"        | multi-URL | A or B    | Comparison table |
+| "what changed since last time"       | diff      | any       | Diff format      |
+
+---
+
+
+## References
+
+- **Extraction patterns**: [references/extraction-patterns.md](../references/extraction-patterns.md)
+  CSS selectors, JavaScript snippets, JSON-LD parsing, domain tips.
+
+- **Output templates**: [references/output-templates.md](../references/output-templates.md)
+  Markdown, JSON, CSV templates with complete examples.
+
+- **Data transforms**: [references/data-transforms.md](../references/data-transforms.md)
+  Cleaning, normalization, deduplication, enrichment patterns.
+
+
+## Best Practices
+
+- Provide clear, specific context about your project and requirements
+- Review all suggestions before applying them to production code
+- Combine with other complementary skills for comprehensive analysis
+
+
+## Common Pitfalls
+
+- Using this skill for tasks outside its domain expertise
+- Applying recommendations without understanding your specific context
+- Not providing enough project context for accurate analysis