@diottodev/runic 1.0.0 → 1.1.0

package/engineering/universal-scraping-architect/references/firecrawl-technical-guide.md

@@ -0,0 +1,44 @@
+# Firecrawl Technical Guide
+
+Technical integration patterns for the Firecrawl API within the Universal Scraping Architect (Mode 1: API-Driven). Use this when the source is a public URL, a JS-heavy SPA, requires search-first discovery, or involves bulk crawling across a domain.
+
+## Authentication (BYOK)
+Firecrawl uses a Bring-Your-Own-Key model. The key is **always** read from the
+`FIRECRAWL_API_KEY` environment variable — never hardcoded, never committed.
+The free tier is sufficient for evaluation and small jobs, which keeps this skill
+compliant with the repo's "no paid commercial dependency" rule.
+
+```python
+import os
+from firecrawl import FirecrawlApp
+app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))
+```
+
+## Choosing the right operation
+- **`scrape`** — single URL → clean markdown/HTML/structured JSON. Cheapest; one credit per page. Default for "give me this page."
+- **`crawl`** — follow links across a domain up to a depth/limit. Credit cost scales with pages; always set `limit` and `maxDepth` to bound spend.
+- **`map`** — fast URL discovery (sitemap-style) without full extraction. Use it *first* to scope a crawl before paying for full extraction.
+- **`search`** — query-first discovery when you don't yet have URLs.
+
+## Quota & rate-limit awareness
+Estimate credits **before** launching bulk jobs (`map` → count URLs → multiply).
+Treat HTTP 429 / quota errors as expected: back off exponentially, log the quota
+state, and checkpoint progress so a re-run resumes rather than restarts. Firecrawl
+returns structured error payloads — surface the message rather than swallowing it.
+
+## Token-budget discipline
+Extracted markdown frequently exceeds an LLM context window. Estimate tokens
+(`chars / 4` as a cheap proxy) and chunk before passing downstream, reserving
+output tokens for the model's response. See `scripts/firecrawl_example.py`.
+
+## Output formats
+Request only the formats you need (`["markdown"]` is smallest). Markdown for prose,
+structured-JSON extraction for tabular/typed data, HTML only when you need the DOM.
+
+### Authoritative Sources
+1. [Firecrawl API Documentation](https://docs.firecrawl.dev/api-reference/introduction)
+2. [Firecrawl SDK for Python](https://github.com/mendableai/firecrawl-py)
+3. [REST API Design Best Practices (Microsoft)](https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design)
+4. [Handling API Rate Limits (Cloudflare)](https://developers.cloudflare.com/fundamentals/api/reference/rate-limits/)
+5. [JSON Schema Standard](https://json-schema.org/specification.html)
+6. [Exponential Backoff and Jitter (AWS Architecture Blog)](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)

package/engineering/universal-scraping-architect/references/local-extraction-patterns.md

@@ -0,0 +1,46 @@
+# Local Extraction Patterns (BeautifulSoup + pandas)
+
+Patterns for Mode 2 (local Python) and the local half of Mode 3 (hybrid). Use these
+when extracting from static HTML, local files (PDF/Excel/CSV), private/sensitive
+data, or simple pages where Firecrawl is overkill. See `scripts/local_bs4_example.py`.
+
+## Fetch safely
+Use `requests` with an identifying `User-Agent`, a timeout, and bounded retries with
+backoff. Always call `raise_for_status()` and handle the specific exception types
+(`HTTPError`, `ConnectionError`, `Timeout`) rather than a bare `except`.
+
+## Select robustly, not brittly
+Anchor on stable signals — `id`, `data-*` attributes, semantic tags — not deep
+positional chains. **Anti-pattern:** `div > span > ul > li:nth-child(3)` breaks the
+moment the layout shifts. Prefer `soup.find("table", {"id": "..."})` or attribute
+selectors, and fail loudly (with a message telling the user to update the selector)
+when the target isn't found.
+
+## Parse tables with pandas
+`pd.read_html(str(table))[0]` turns an HTML `<table>` into a DataFrame. Scope it to
+the specific table element you located with BeautifulSoup rather than letting pandas
+guess across the whole page.
+
+## Normalize then validate
+1. **Column names** → snake_case (`strip().lower()`, collapse whitespace, drop
+   non-word chars) so downstream code is stable.
+2. **Clean** → strip string cells, drop fully-empty rows, coerce dtypes.
+3. **Validate before saving** → assert the frame is non-empty *and* every required
+   column is present. Never blindly write an empty or malformed result to disk.
+
+## Choose the output format
+- **CSV** — flat/tabular data.
+- **JSON** — nested or typed structures (validate it with `scripts/validate_extraction.py`).
+- **Markdown** — clean prose destined for an LLM.
+
+## Privacy
+Local processing keeps sensitive data on-machine — this is the safe default whenever
+the source is private. Don't ship local files to an external API just for convenience.
+
+### Authoritative Sources
+1. [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
+2. [pandas.read_html — API reference](https://pandas.pydata.org/docs/reference/api/pandas.read_html.html)
+3. [Requests: HTTP for Humans](https://requests.readthedocs.io/en/latest/)
+4. [WHATWG HTML Living Standard — Tabular data](https://html.spec.whatwg.org/multipage/tables.html)
+5. [MDN: CSS selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_selectors)
+6. [pandas — Working with missing data](https://pandas.pydata.org/docs/user_guide/missing_data.html)

package/engineering/universal-scraping-architect/references/scraping-ethics-security.md

@@ -0,0 +1,48 @@
+# Scraping Ethics, Robots.txt, and Security
+
+Guidelines for ethical, legal, and secure data collection. These rules apply across
+all three extraction modes and are the basis for the skill's proactive triggers.
+
+## Respect the Robots Exclusion Protocol
+`robots.txt` (now standardized as RFC 9309) is the first thing to check. Parse it
+with the stdlib — no extra dependency required:
+
+```python
+import urllib.robotparser
+rp = urllib.robotparser.RobotFileParser()
+rp.set_url("https://example.com/robots.txt")
+rp.read()
+allowed = rp.can_fetch("UniversalScrapingArchitect/1.0", target_url)
+```
+
+If a path is disallowed, stop and surface it to the user rather than scraping anyway.
+
+## Be a polite client
+- Send an honest, identifying `User-Agent` with contact info.
+- Set timeouts and a sane retry budget (exponential backoff, capped attempts).
+- Rate-limit: add delay between requests; never hammer an origin in a tight loop.
+- Prefer official APIs / sitemaps / data dumps over HTML scraping when they exist.
+
+## Legal & privacy posture
+Scraping public data is generally permissible, but **terms of service, copyright,
+and data-protection law (GDPR/CCPA) still apply**. Personal data carries extra
+obligations. This skill's **Private Data Leakage** trigger fires when a user asks
+to send local/sensitive files to an external API — in that case prefer Mode 2
+(local Python) so data never leaves the machine. None of this is legal advice;
+flag uncertainty and route the user to counsel for high-stakes collection.
+
+## Security of the pipeline itself
+- **Secrets**: API keys via environment variables only (the **Hardcoded API Keys**
+  trigger). Never log the key value.
+- **Untrusted input**: scraped HTML/markdown is untrusted. Never `eval`/`exec` it,
+  and validate before writing or passing downstream.
+- **SSRF / scope**: validate target URLs; don't let a crawl wander into internal
+  network ranges.
+
+### Authoritative Sources
+1. [The Robots Exclusion Protocol (RFC 9309)](https://datatracker.ietf.org/doc/rfc9309/)
+2. [OWASP Automated Threats to Web Applications](https://owasp.org/www-project-automated-threats-to-web-applications/)
+3. [Scraping Ethics Best Practices (Ethical Web Scraping)](https://www.ethicalwebscraping.org/)
+4. [Legal Aspects of Web Scraping (Lexology)](https://www.lexology.com/library/detail.aspx?g=e6e0287a-62ad-4d6d-852a-9e535e69e6b4)
+5. [Cloudflare Bot Management Overview](https://www.cloudflare.com/en-gb/pg-lp/bot-management-for-everyone/)
+6. [Python urllib.robotparser (stdlib)](https://docs.python.org/3/library/urllib.robotparser.html)

package/engineering/universal-scraping-architect/requirements.txt

@@ -0,0 +1,7 @@
+# Universal Scraping Architect — runtime dependencies (BYOK, free-tier compatible).
+# Dependencies are declared here for pre-install review rather than installed ad hoc.
+# Firecrawl is only needed for Mode 1 (API-driven); the local/BS4 path needs the rest.
+firecrawl
+pandas
+requests
+beautifulsoup4

package/engineering/universal-scraping-architect/scripts/firecrawl_example.py

@@ -0,0 +1,217 @@
+"""
+Universal Scraping Architect: Firecrawl Example (Path C — Repeatable Deliverable)
+===================================================================================
+Extracts clean markdown content from a target URL using the Firecrawl SDK.
+
+Demonstrates:
+  - Safe API key loading from environment
+  - robots.txt courtesy check (stdlib, no extra dependency)
+  - Token budget tracking before LLM processing
+  - Firecrawl quota awareness logging
+  - Structured error handling
+  - Clean output saving with validation
+
+This is an editable runner template: tweak the CONFIG block, then run it.
+It also supports `--help` and `--sample` so it passes the repo smoke tests
+without Firecrawl installed (the SDK is imported lazily inside main()).
+
+Usage:
+  export FIRECRAWL_API_KEY="fc-YOUR_KEY_HERE"     # Linux/macOS
+  $env:FIRECRAWL_API_KEY = "fc-YOUR_KEY_HERE"     # Windows PowerShell
+  python scripts/firecrawl_example.py
+"""
+
+import argparse
+import os
+import sys
+import urllib.robotparser
+from datetime import datetime
+
+# =============================================================================
+# CONFIG — Edit these for your task
+# =============================================================================
+TARGET_URL = "https://example.com/research-data"
+OUTPUT_FILE = "clean_extraction.md"
+LOG_FILE = "firecrawl_run.log"
+
+# Firecrawl / Token settings
+FIRECRAWL_API_KEY_ENV = "FIRECRAWL_API_KEY"
+USER_AGENT = "UniversalScrapingArchitect/1.0 (contact: your@email.com)"
+TOKEN_CONTEXT_LIMIT = 100_000     # Adjust to your model's context window
+RESERVED_OUTPUT_TOKENS = 4_000    # Tokens held back for the model's response
+RESPECT_ROBOTS = True             # Set False only for targets you know are safe to fetch
+
+
+# =============================================================================
+# HELPERS
+# =============================================================================
+
+def log(message: str, level: str = "INFO"):
+    """Prints a timestamped log line and appends it to the log file."""
+    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+    line = f"[{timestamp}] [{level}] {message}"
+    print(line)
+    with open(LOG_FILE, "a", encoding="utf-8") as f:
+        f.write(line + "\n")
+
+
+def check_environment() -> str:
+    """Loads the Firecrawl API key from the environment. Fails fast if missing."""
+    api_key = os.getenv(FIRECRAWL_API_KEY_ENV)
+    if not api_key:
+        log(
+            f"Missing environment variable: {FIRECRAWL_API_KEY_ENV}. "
+            "Set it before running this script.",
+            level="ERROR"
+        )
+        sys.exit(1)
+    log("API key loaded successfully from environment.")
+    return api_key
+
+
+def check_robots(url: str) -> bool:
+    """
+    Courtesy robots.txt check using the stdlib (RFC 9309). Returns True if the
+    fetch is allowed (or robots.txt is unreachable, which we treat as allowed
+    but log). The skill mandates this — model the correct behaviour by default.
+    """
+    if not RESPECT_ROBOTS:
+        log("robots.txt check skipped (RESPECT_ROBOTS=False).", level="WARNING")
+        return True
+    try:
+        parts = url.split("/")
+        robots_url = f"{parts[0]}//{parts[2]}/robots.txt"
+        rp = urllib.robotparser.RobotFileParser()
+        rp.set_url(robots_url)
+        rp.read()
+        allowed = rp.can_fetch(USER_AGENT, url)
+        if not allowed:
+            log(f"robots.txt DISALLOWS fetching {url}. Aborting.", level="ERROR")
+        else:
+            log("robots.txt allows this fetch.")
+        return allowed
+    except Exception as e:
+        log(f"Could not read robots.txt ({e}); proceeding with caution.", level="WARNING")
+        return True
+
+
+def estimate_tokens(text: str) -> int:
+    """Rough token estimate based on character count (characters / 4)."""
+    return len(text) // 4
+
+
+def check_token_budget(text: str) -> bool:
+    """
+    Estimates token usage and warns if over budget.
+    Returns True if within budget, False if over.
+    """
+    estimated = estimate_tokens(text)
+    available = TOKEN_CONTEXT_LIMIT - RESERVED_OUTPUT_TOKENS
+    log(f"Token estimate: {estimated:,} / {available:,} available tokens.")
+    if estimated > available:
+        log(
+            f"OVER TOKEN BUDGET by {estimated - available:,} tokens. "
+            "Consider chunking the output before passing to an LLM.",
+            level="WARNING"
+        )
+        return False
+    log("Within token budget.")
+    return True
+
+
+def validate_content(content: str) -> bool:
+    """Basic validation — checks the response is non-empty."""
+    if not content or not content.strip():
+        log("Validation failed: empty content received from Firecrawl.", level="ERROR")
+        return False
+    log(f"Validation passed. Content length: {len(content):,} characters.")
+    return True
+
+
+def save_output(content: str, path: str):
+    """Saves validated content to the output path."""
+    with open(path, "w", encoding="utf-8") as f:
+        f.write(content)
+    log(f"Output saved to: {path}")
+
+
+# =============================================================================
+# MAIN
+# =============================================================================
+
+def main():
+    log("=" * 60)
+    log("Universal Scraping Architect — Firecrawl Path C")
+    log("=" * 60)
+
+    # Step 1: Environment check
+    api_key = check_environment()
+
+    # Step 2: robots.txt courtesy check (the skill mandates this)
+    if not check_robots(TARGET_URL):
+        sys.exit(1)
+
+    # Step 3: Initialise Firecrawl client (imported lazily so --help works without the SDK)
+    from firecrawl import FirecrawlApp
+    app = FirecrawlApp(api_key=api_key)
+    log(f"Firecrawl client initialised. Target: {TARGET_URL}")
+
+    # Step 4: Scrape
+    #   Firecrawl Python SDK v1+ passes options directly: scrape_url(url, formats=[...]).
+    #   Older SDKs used the params={"formats": [...]} dict wrapper.
+    try:
+        log("Starting scrape...")
+        result = app.scrape_url(TARGET_URL, formats=["markdown"])
+        # Result may be a dict (older SDK) or an object with a .markdown attribute (v1+).
+        markdown_content = (
+            result.get("markdown", "") if isinstance(result, dict)
+            else getattr(result, "markdown", "")
+        )
+    except Exception as e:
+        log(f"Firecrawl scrape failed: {e}", level="ERROR")
+        log(
+            "Tip: if this is a quota or auth error, check your FIRECRAWL_API_KEY "
+            "and run `firecrawl --status` to inspect account state.",
+            level="WARNING"
+        )
+        sys.exit(1)
+
+    # Step 5: Validate
+    if not validate_content(markdown_content):
+        sys.exit(1)
+
+    # Step 6: Token budget check
+    check_token_budget(markdown_content)
+
+    # Step 7: Save clean output
+    save_output(markdown_content, OUTPUT_FILE)
+
+    # Step 8: Final summary
+    log("=" * 60)
+    log("EXTRACTION COMPLETE")
+    log(f"  Source URL  : {TARGET_URL}")
+    log(f"  Output file : {OUTPUT_FILE}")
+    log(f"  Characters  : {len(markdown_content):,}")
+    log(f"  Est. tokens : {estimate_tokens(markdown_content):,}")
+    log(f"  Log file    : {LOG_FILE}")
+    log("=" * 60)
+    log("Customisable: TARGET_URL, OUTPUT_FILE, TOKEN_CONTEXT_LIMIT, formats.")
+
+
+def _print_sample():
+    """Prints a non-network sample of what this template does (for smoke tests)."""
+    print("Firecrawl example (template). Edit the CONFIG block, set FIRECRAWL_API_KEY, then run.")
+    print(f"  TARGET_URL          = {TARGET_URL}")
+    print(f"  OUTPUT_FILE         = {OUTPUT_FILE}")
+    print(f"  TOKEN_CONTEXT_LIMIT = {TOKEN_CONTEXT_LIMIT}")
+    print("  Pipeline: env-check -> robots.txt -> scrape -> validate -> token-budget -> save")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Firecrawl extraction example (editable runner template).")
+    parser.add_argument("--sample", action="store_true", help="Print a sample summary and exit (no network).")
+    args = parser.parse_args()
+    if args.sample:
+        _print_sample()
+        sys.exit(0)
+    main()

package/engineering/universal-scraping-architect/scripts/local_bs4_example.py

@@ -0,0 +1,293 @@
+"""
+Universal Scraping Architect: Traditional / Local Scraping Example
+==================================================================
+Extracts a data table from a static HTML page using Requests and BeautifulSoup,
+then validates, cleans, and saves to CSV.
+
+Demonstrates:
+  - Safe HTTP fetching with headers, timeouts, and retry logic
+  - HTML table parsing with pandas
+  - Column name normalisation to snake_case
+  - Required field validation before saving
+  - Structured error handling at every stage
+  - Clean, logged output
+
+Usage:
+  pip install -r requirements.txt
+  python scripts/local_bs4_example.py
+"""
+
+from __future__ import annotations  # defer annotation eval so --help works without deps
+
+import argparse
+import sys
+import time
+import urllib.robotparser
+from datetime import datetime
+
+# Third-party deps are imported lazily/guarded so `--help` and `--sample` work
+# even when pandas/requests/bs4 aren't installed (repo smoke-test convention).
+try:
+    import pandas as pd
+    import requests
+    from bs4 import BeautifulSoup
+    _DEPS_OK = True
+except ImportError:
+    pd = requests = BeautifulSoup = None
+    _DEPS_OK = False
+
+# =============================================================================
+# CONFIG — Edit these for your task
+# =============================================================================
+TARGET_URL = "https://example.com/macroeconomic-indicators/energy-prices"
+OUTPUT_FILE = "energy_price_data.csv"
+LOG_FILE = "local_scrape_run.log"
+
+# HTTP settings
+USER_AGENT = "UniversalScrapingArchitect/1.0 (contact: your@email.com)"
+TIMEOUT_SECONDS = 15
+MAX_RETRIES = 3
+RETRY_BASE_DELAY_SECONDS = 1     # exponential backoff: 1s, 2s, 4s, ...
+RESPECT_ROBOTS = True            # Set False only for targets you know are safe to fetch
+
+# Validation
+REQUIRED_COLUMNS = ["date", "price_index", "yoy_change"]
+TABLE_SELECTOR = {"id": "indicator-data"}     # Edit to match your target table's HTML attributes
+
+
+# =============================================================================
+# HELPERS
+# =============================================================================
+
+def log(message: str, level: str = "INFO"):
+    """Prints a timestamped log line and appends it to the log file."""
+    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+    line = f"[{timestamp}] [{level}] {message}"
+    print(line)
+    with open(LOG_FILE, "a", encoding="utf-8") as f:
+        f.write(line + "\n")
+
+
+def check_robots(url: str) -> bool:
+    """
+    Courtesy robots.txt check using the stdlib (RFC 9309). Returns True if the
+    fetch is allowed (or robots.txt is unreachable, which we treat as allowed
+    but log). The skill mandates this — model the correct behaviour by default.
+    """
+    if not RESPECT_ROBOTS:
+        log("robots.txt check skipped (RESPECT_ROBOTS=False).", level="WARNING")
+        return True
+    try:
+        parts = url.split("/")
+        robots_url = f"{parts[0]}//{parts[2]}/robots.txt"
+        rp = urllib.robotparser.RobotFileParser()
+        rp.set_url(robots_url)
+        rp.read()
+        allowed = rp.can_fetch(USER_AGENT, url)
+        log("robots.txt allows this fetch." if allowed
+            else f"robots.txt DISALLOWS fetching {url}. Aborting.",
+            level="INFO" if allowed else "ERROR")
+        return allowed
+    except Exception as e:
+        log(f"Could not read robots.txt ({e}); proceeding with caution.", level="WARNING")
+        return True
+
+
+def safe_get(url: str) -> str:
+    """
+    Fetches HTML from a URL with polite headers, a timeout, and retry logic
+    using exponential backoff (RETRY_BASE_DELAY_SECONDS * 2**n).
+    Raises on non-2xx status.
+    """
+    headers = {"User-Agent": USER_AGENT}
+
+    for attempt in range(1, MAX_RETRIES + 1):
+        try:
+            log(f"Attempt {attempt}/{MAX_RETRIES}: GET {url}")
+            response = requests.get(url, headers=headers, timeout=TIMEOUT_SECONDS)
+            response.raise_for_status()
+            log(f"HTTP {response.status_code} — content length: {len(response.text):,} chars.")
+            return response.text
+        except requests.exceptions.HTTPError as e:
+            log(f"HTTP error on attempt {attempt}: {e}", level="ERROR")
+        except requests.exceptions.ConnectionError as e:
+            log(f"Connection error on attempt {attempt}: {e}", level="ERROR")
+        except requests.exceptions.Timeout:
+            log(f"Timeout on attempt {attempt} after {TIMEOUT_SECONDS}s.", level="WARNING")
+        except requests.exceptions.RequestException as e:
+            log(f"Request failed on attempt {attempt}: {e}", level="ERROR")
+
+        if attempt < MAX_RETRIES:
+            delay = RETRY_BASE_DELAY_SECONDS * (2 ** (attempt - 1))
+            log(f"Waiting {delay}s before retry (exponential backoff)...")
+            time.sleep(delay)
+
+    raise RuntimeError(f"All {MAX_RETRIES} attempts failed for URL: {url}")
+
+
+def find_table(html: str) -> BeautifulSoup:
+    """
+    Parses the HTML and returns the target table element.
+    Adjust TABLE_SELECTOR to match the table you need.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+    table = soup.find("table", TABLE_SELECTOR)
+    if not table:
+        raise ValueError(
+            f"Target table not found. Selector used: {TABLE_SELECTOR}. "
+            "Inspect the page source and update TABLE_SELECTOR in CONFIG."
+        )
+    log("Target table found in HTML.")
+    return table
+
+
+def parse_table(table) -> pd.DataFrame:
+    """Converts a BeautifulSoup table element into a pandas DataFrame."""
+    df = pd.read_html(str(table))[0]
+    log(f"Parsed table: {len(df)} rows, {len(df.columns)} columns.")
+    return df
+
+
+def clean_column_names(df: pd.DataFrame) -> pd.DataFrame:
+    """Normalises all column names to snake_case."""
+    df.columns = (
+        df.columns
+        .str.strip()
+        .str.lower()
+        .str.replace(r"\s+", "_", regex=True)
+        .str.replace(r"[^\w]", "", regex=True)
+    )
+    log(f"Columns after normalisation: {list(df.columns)}")
+    return df
+
+
+def clean_data(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Applies general cleaning rules.
+    Extend this function for your task-specific cleaning needs.
+    """
+    # Strip whitespace from string columns
+    for col in df.select_dtypes(include="object").columns:
+        df[col] = df[col].str.strip()
+
+    # Drop fully empty rows
+    before = len(df)
+    df = df.dropna(how="all")
+    dropped = before - len(df)
+    if dropped:
+        log(f"Dropped {dropped} fully empty rows.", level="WARNING")
+
+    return df
+
+
+def validate(df: pd.DataFrame) -> bool:
+    """
+    Checks that the DataFrame is non-empty and contains all required columns.
+    Returns True if valid, False otherwise.
+    """
+    if df.empty:
+        log("Validation failed: DataFrame is empty.", level="ERROR")
+        return False
+
+    missing = [col for col in REQUIRED_COLUMNS if col not in df.columns]
+    if missing:
+        log(
+            f"Validation failed: missing required columns: {missing}. "
+            f"Available columns: {list(df.columns)}",
+            level="ERROR"
+        )
+        return False
+
+    log(f"Validation passed. {len(df)} rows, {len(df.columns)} columns.")
+    return True
+
+
+def save_output(df: pd.DataFrame, path: str):
+    """Saves the validated DataFrame to CSV."""
+    df.to_csv(path, index=False, encoding="utf-8")
+    log(f"Output saved to: {path}")
+
+
+# =============================================================================
+# MAIN
+# =============================================================================
+
+def main():
+    log("=" * 60)
+    log("Universal Scraping Architect — Traditional / Local Scraping")
+    log("=" * 60)
+
+    if not _DEPS_OK:
+        log(
+            "Missing dependencies (requests, beautifulsoup4, pandas). "
+            "Install them from the skill's requirements.txt before running.",
+            level="ERROR"
+        )
+        sys.exit(1)
+
+    # Step 1: robots.txt courtesy check (the skill mandates this), then fetch
+    if not check_robots(TARGET_URL):
+        sys.exit(1)
+    try:
+        html = safe_get(TARGET_URL)
+    except RuntimeError as e:
+        log(str(e), level="ERROR")
+        sys.exit(1)
+
+    # Step 2: Find table
+    try:
+        table = find_table(html)
+    except ValueError as e:
+        log(str(e), level="ERROR")
+        sys.exit(1)
+
+    # Step 3: Parse
+    try:
+        df = parse_table(table)
+    except Exception as e:
+        log(f"Failed to parse table into DataFrame: {e}", level="ERROR")
+        sys.exit(1)
+
+    # Step 4: Clean
+    df = clean_column_names(df)
+    df = clean_data(df)
+
+    # Step 5: Validate
+    if not validate(df):
+        sys.exit(1)
+
+    # Step 6: Save
+    save_output(df, OUTPUT_FILE)
+
+    # Step 7: Final summary
+    log("=" * 60)
+    log("EXTRACTION COMPLETE")
+    log(f"  Source URL   : {TARGET_URL}")
+    log(f"  Output file  : {OUTPUT_FILE}")
+    log(f"  Rows saved   : {len(df)}")
+    log(f"  Columns      : {list(df.columns)}")
+    log(f"  Log file     : {LOG_FILE}")
+    log("=" * 60)
+    log("Customisable: TARGET_URL, OUTPUT_FILE, TABLE_SELECTOR, REQUIRED_COLUMNS.")
+
+
+def _print_sample():
+    """Prints a non-network sample of what this template does (for smoke tests)."""
+    print("Local BS4 example (template). Edit the CONFIG block, install deps, then run.")
+    print(f"  TARGET_URL       = {TARGET_URL}")
+    print(f"  OUTPUT_FILE      = {OUTPUT_FILE}")
+    print(f"  TABLE_SELECTOR   = {TABLE_SELECTOR}")
+    print(f"  REQUIRED_COLUMNS = {REQUIRED_COLUMNS}")
+    print("  Pipeline: robots.txt -> fetch(retry+backoff) -> find table -> "
+          "parse -> normalize -> clean -> validate -> save CSV")
+    print(f"  Dependencies installed: {_DEPS_OK}")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Local BeautifulSoup/pandas scraping example (editable runner template).")
+    parser.add_argument("--sample", action="store_true", help="Print a sample summary and exit (no network).")
+    args = parser.parse_args()
+    if args.sample:
+        _print_sample()
+        sys.exit(0)
+    main()