@wenathlan/saddle 1.8.17 → 2.0.2

package/docs/plans/42.pesquisa.concorrencia.md

@@ -1,721 +0,0 @@
-# Competitive Research: webscrape vs. 7 Competitors
-
-> **Date:** 2026-06-08
-> **Objective:** Deep analysis of 7 direct/indirect competitors to identify gaps, opportunities, and architectural insights for the `@wenathlan/webscrape` toolkit.
-
----
-
-## Table of Contents
-
-1. [Executive Summary](#executive-summary)
-2. [Webscrape Current Capabilities](#webscrape-current-capabilities)
-3. [Competitor Deep Dives](#competitor-deep-dives)
-   - [1. Crawlee](#1-crawlee)
-   - [2. Playwright](#2-playwright)
-   - [3. Puppeteer](#3-puppeteer)
-   - [4. Cheerio](#4-cheerio)
-   - [5. Scrapy](#5-scrapy)
-   - [6. Firecrawl](#6-firecrawl)
-   - [7. Browserless](#7-browserless)
-4. [Feature Comparison Table](#feature-comparison-table)
-5. [Architecture Insights](#architecture-insights)
-6. [Critical Gaps in Webscrape](#critical-gaps-in-webscrape)
-7. [Recommendations](#recommendations)
-
----
-
-## Executive Summary
-
-The web scraping ecosystem is mature and fragmented. Competitors range from low-level browser APIs (Puppeteer, Playwright) to full orchestration frameworks (Crawlee, Scrapy) to hosted AI-first services (Firecrawl). 
-
-**webscrape's unique position:** A lightweight, cross-runtime (Node.js/Bun) toolkit that combines scraping, serialization, AI agent formatting, and an HTTP server — without requiring a cloud service. This is genuinely differentiated.
-
-**Biggest gaps:** Anti-bot/stealth capabilities, persistent queue/storage, persistent session management, Docker deployment, and MCP server integration.
-
----
-
-## Webscrape Current Capabilities
-
-Based on codebase analysis (`C:\allan2\devthink\webscrape\`):
-
-| Module | File | What it does |
-|--------|------|-------------|
-| Browser automation | `browser.ts` | Playwright wrapper (chromium/firefox/webkit), ad blocking, stealth mode |
-| HTTP scraping | `fetch.ts` | Node fetch with auto mode detection |
-| Smart scraping | `scrape.ts` | Auto-detect fetch vs browser, scroll, extract |
-| Content extraction | `extract.ts` | Cheerio-based: links, images, tables, metadata, JSON-LD |
-| Serialization | `serialize.ts` | Markdown, XML, JSON, Redis, plain text |
-| Crawler | `crawler.ts` | BFS/DFS crawling with depth, concurrency, dedup |
-| Batch processing | `batch.ts` | Parallel/sequential batch scraping with p-limit |
-| Rate limiting | `rate-limiter.ts` | Token bucket algorithm with concurrency control |
-| Proxy pool | `proxy.ts` | Round-robin/random rotation, failure tracking, auto-revive |
-| Cache | `cache.ts` | LRU cache with TTL |
-| Session | `session.ts` | CookieJar, session state |
-| Headers | `headers.ts` | 4 browser profiles (Chrome/Firefox/Safari) |
-| Middleware | `middleware.ts` | Pipeline pattern (logging, timeout, retry) |
-| HTTP server | `server.ts` | Hono-based REST API (/scrape, /batch, /crawl, /health) |
-| AI agent | `agent.ts` | Format output for LLMs, token estimation |
-| Chunking | `chunking.ts` | Split content for RAG pipelines |
-| Browser pool | `pool.ts` | Browser instance reuse with wait queue |
-| LLMs.txt | `llms-txt.ts` | Generate llms.txt for sites |
-| CLI | `cli.ts` | Commander-based CLI tool |
-
-**Runtime support:** Node.js >= 26.2.0, Bun >= 1.4.0
-
----
-
-## Competitor Deep Dives
-
-### 1. Crawlee
-
-**Website:** [crawlee.dev](https://crawlee.dev) | **GitHub:** [apify/crawlee](https://github.com/apify/crawlee)
-**Stars:** 23.7k | **License:** Apache-2.0 | **Language:** TypeScript
-**npm:** `crawlee` (~100 dependents) | **Maintained by:** Apify
-
-#### What It Is
-A comprehensive web scraping and browser automation framework. Crawlee is the open-source engine behind the Apify platform. It supports HTTP crawling, headless browser crawling (Playwright, Puppeteer, Cheerio, JSDOM), proxy rotation, session management, and persistent storage.
-
-#### Key Features
-- **Multi-crawler interface:** `PlaywrightCrawler`, `PuppeteerCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `BasicCrawler`
-- **Persistent request queue:** Breadth-first and depth-first crawling with disk/cloud storage
-- **Pluggable storage:** Dataset, Key-Value Store, Request Queue (local or Apify cloud)
-- **Automatic scaling:** Adjusts concurrency based on system resources
-- **Proxy rotation:** Integrated proxy configuration with session-based rotation
-- **Session management:** Per-session cookie and header persistence
-- **Browser fingerprint generation:** Human-like fingerprints by default
-- **Auto-retry:** Configurable retry with exponential backoff
-- **CLI scaffolding:** `npx crawlee create my-crawler`
-- **Docker deployment:** Pre-built Dockerfiles
-- **TypeScript-first:** Full generic support
-
-#### Architecture
-```
-Crawler (Playwright/Puppeteer/Cheerio/JSDOM/Basic)
-  → Request Handler (user logic)
-    → Request Queue (persistent, FIFO/LIFO)
-      → Storage (Dataset + KV Store)
-        → Local filesystem OR Apify Cloud
-```
-
-#### What Crawlee Does That Webscrape Doesn't
-1. **Persistent request queue** — survives restarts, handles millions of URLs
-2. **Pluggable storage adapters** — local disk, S3, Apify cloud
-3. **Built-in auto-scaling** — adjusts concurrency to CPU/memory
-4. **Session-based proxy rotation** — keeps same proxy per session
-5. **Human-like fingerprint generation** — automatic browser fingerprint randomization
-6. **Adaptive crawler** — automatically detects if page needs JS rendering
-7. **Docker deployment templates** — production-ready containers
-8. **CLI project scaffolding** — instant project setup
-9. **Python version** — crawlee-python for Python ecosystem
-10. **Apify platform integration** — cloud execution, monitoring, scheduling
-
-#### Success Factors
-- Backed by Apify (commercial company with cloud platform)
-- Battle-tested at scale (millions of pages)
-- Excellent documentation with interactive examples
-- Active community (Discord, Stack Overflow)
-- Regular releases (134 releases, v3.17.0)
-
----
-
-### 2. Playwright
-
-**Website:** [playwright.dev](https://playwright.dev) | **GitHub:** [microsoft/playwright](https://github.com/microsoft/playwright)
-**Stars:** 90.5k | **License:** Apache-2.0 | **Language:** TypeScript (91%)
-**npm:** `playwright` (~471k dependents) | **Maintained by:** Microsoft
-
-#### What It Is
-A browser automation framework for testing and scripting. Drives Chromium, Firefox, and WebKit with a single API. Not a scraping framework per se, but widely used for scraping due to its reliability and cross-browser support.
-
-#### Key Features
-- **Cross-browser:** Chromium, Firefox, WebKit with single API
-- **Auto-wait:** No artificial timeouts, waits for elements to be actionable
-- **Locators:** Resilient element finding (role, label, test-id, etc.)
-- **Test isolation:** Each test gets fresh browser context
-- **Tracing:** Full execution traces, screenshots, videos on failure
-- **Parallelism:** Tests run in parallel by default
-- **Network interception:** Route/mock network requests
-- **Mobile emulation:** Device profiles built-in
-- **PDF generation:** Page-to-PDF conversion
-- **Code generation:** Record interactions as code
-- **MCP server:** AI agent integration via Model Context Protocol
-- **CLI for coding agents:** Token-efficient browser automation for AI
-
-#### Architecture
-```
-Playwright Library
-  → Browser Process (Chromium/Firefox/WebKit)
-    → Browser Context (isolated profiles)
-      → Page (tab)
-        → Frame (iframe)
-          → DOM / Network / Storage
-```
-
-#### What Playwright Does That Webscrape Doesn't
-1. **True multi-browser** — Firefox, WebKit support (webscrape uses Playwright but defaults to Chromium)
-2. **Auto-wait and web-first assertions** — eliminates flaky timeouts
-3. **Locator system** — resilient element finding with role/text/label selectors
-4. **Tracing and debugging** — full execution traces with DOM snapshots
-5. **Mobile device emulation** — built-in device profiles
-6. **Network request mocking** — intercept and modify requests/responses
-7. **Video recording** — record browser sessions
-8. **MCP server integration** — `@playwright/mcp` for AI agents
-9. **VS Code extension** — code generation and debugging in editor
-10. **Massive ecosystem** — 471k dependents, used by 90%+ of browser automation projects
-
-#### Success Factors
-- Microsoft backing with dedicated team
-- Best-in-class cross-browser support
-- Superior developer experience (auto-wait, locators, tracing)
-- Works in CI/CD out of the box
-- Extensive documentation and examples
-- Active development (17k+ commits, v1.60.0)
-
----
-
-### 3. Puppeteer
-
-**Website:** [pptr.dev](https://pptr.dev) | **GitHub:** [puppeteer/puppeteer](https://github.com/puppeteer/puppeteer)
-**Stars:** 94.5k | **License:** Apache-2.0 | **Language:** TypeScript (93.8%)
-**npm:** `puppeteer` (~575k dependents) | **Maintained by:** Google Chrome team
-
-#### What It Is
-A JavaScript library providing high-level API to control Chrome or Firefox over the DevTools Protocol or WebDriver BiDi. The original headless Chrome automation tool.
-
-#### Key Features
-- **Chrome DevTools Protocol** — direct browser control
-- **WebDriver BiDi** — modern protocol support
-- **Screenshot and PDF** — full page or element screenshots, PDF generation
-- **Network interception** — request/response modification
-- **JavaScript execution** — evaluate arbitrary JS in page context
-- **Cookie management** — set/get/delete cookies
-- **Emulation** — viewport, user agent, geolocation, timezone
-- **Chrome extensions support** — load custom extensions
-- **MCP server** — `chrome-devtools-mcp` for AI agents
-- **Docker support** — official Docker images
-
-#### Architecture
-```
-Puppeteer API
-  → Chrome DevTools Protocol (CDP) / WebDriver BiDi
-    → Chrome/Firefox Browser Process
-      → Browser Context
-        → Page
-```
-
-#### What Puppeteer Does That Webscrape Doesn't
-1. **Direct CDP access** — low-level browser control when needed
-2. **WebDriver BiDi support** — modern W3C standard protocol
-3. **Chrome extensions** — load and use extensions in automation
-4. **Lighthouse integration** — performance auditing built-in
-5. **Chrome for Testing** — managed browser versions
-6. **Massive community** — 575k dependents, most-used browser automation library
-7. **MCP server** — `chrome-devtools-mcp` for AI agent integration
-
-#### Success Factors
-- Google Chrome team backing
-- First-mover advantage (created the headless Chrome automation space)
-- Simple, focused API
-- Excellent Chrome DevTools integration
-- Massive adoption and ecosystem
-
----
-
-### 4. Cheerio
-
-**Website:** [cheerio.js.org](https://cheerio.js.org) | **GitHub:** [cheeriojs/cheerio](https://github.com/cheeriojs/cheerio)
-**Stars:** 30.4k | **License:** MIT | **Language:** TypeScript (65%)
-**npm:** `cheerio` (~1.8m dependents) | **Maintained by:** Community
-
-#### What It Is
-A fast, flexible library for parsing and manipulating HTML and XML. Implements a subset of jQuery API. Server-side HTML parsing without a browser.
-
-#### Key Features
-- **jQuery-like API** — familiar `$('.selector')` syntax
-- **Fast parsing** — uses parse5 (HTML) or htmlparser2 (forgiving mode)
-- **HTML manipulation** — add/remove/change elements
-- **Rendering** — serialize back to HTML
-- **Server-side** — no browser required
-- **Flexible** — works in Node.js and browser environments
-- **XML support** — parse and manipulate XML documents
-
-#### Architecture
-```
-HTML String
-  → Parser (parse5 or htmlparser2)
-    → DOM Tree (in-memory)
-      → Cheerio API (jQuery-like)
-        → Manipulated DOM
-          → Serialization (HTML/Text)
-```
-
-#### What Cheerio Does That Webscrape Doesn't
-1. **HTML manipulation** — modify DOM before extraction (webscrape only reads)
-2. **XML parsing** — dedicated XML support
-3. **Lightweight** — no browser dependency, tiny footprint
-4. **Browser + server** — works in both environments
-5. **Massive adoption** — 1.8m dependents, de facto standard for HTML parsing
-
-#### Success Factors
-- jQuery familiarity — easy learning curve
-- Blazingly fast — no browser overhead
-- Rock-solid stability — 4,200+ commits, 62 releases
-- Used by virtually every Node.js scraping project
-- MIT license — maximum compatibility
-
----
-
-### 5. Scrapy
-
-**Website:** [scrapy.org](https://scrapy.org) | **GitHub:** [scrapy/scrapy](https://github.com/scrapy/scrapy)
-**Stars:** 62.1k | **License:** BSD-3-Clause | **Language:** Python (99.5%)
-**PyPI:** `Scrapy` (~55.2k dependents) | **Maintained by:** Zyte (formerly Scrapinghub)
-
-#### What It Is
-The gold standard for Python web scraping frameworks. A complete, batteries-included framework for extracting structured data from websites. Uses Twisted for async networking.
-
-#### Key Features
-- **Middleware pipeline** — request/response processing chain
-- **Item pipelines** — structured data processing and validation
-- **Built-in selectors** — CSS and XPath selectors
-- **Feed exports** — JSON, CSV, XML, databases
-- **Robust error handling** — auto-retry, dedup, crawl depth limits
-- ** robots.txt compliance** — automatic respect
-- **Link extraction** — automatic URL discovery
-- **Closeable items** — memory-efficient data structures
-- **Scrapyd** — deploy and run spiders as services
-- **Scrapy Cloud** — managed hosting (Zyte)
-- **Scrapyrt** — REST API for spiders
-- **Splash integration** — JavaScript rendering
-
-#### Architecture (Reference for webscrape)
-```
-Spider (user logic)
-  → Scheduler (priority queue, dedup)
-    → Downloader (async, concurrent)
-      → Downloader Middlewares (proxy, headers, cookies, retry)
-        → Spider Middlewares (response processing)
-          → Item Pipelines (validation, storage)
-            → Exporters (JSON, CSV, XML)
-```
-
-#### What Scrapy Does That Webscrape Doesn't
-1. **Middleware pipeline** — formalized request/response processing chain
-2. **Item pipelines** — structured data validation and post-processing
-3. **Feed exports** — multiple output formats with auto-export
-4. **XPath selectors** — CSS + XPath selector engine
-5. **Robots.txt compliance** — automatic crawl-delay, disallow respect
-6. **Scrapyd service** — deploy spiders as long-running services
-7. **Scrapy Shell** — interactive debugging environment
-8. **Crawl depth tracking** — per-domain statistics
-9. **Signal system** — event-driven architecture
-10. **Extension system** — plug in custom functionality
-
-#### Success Factors
-- 15+ years of production use
-- Python ecosystem dominance
-- Battle-tested at massive scale
-- Zyte commercial backing
-- Excellent documentation and books
-
----
-
-### 6. Firecrawl
-
-**Website:** [firecrawl.dev](https://firecrawl.dev) | **GitHub:** [firecrawl/firecrawl](https://github.com/firecrawl/firecrawl)
-**Stars:** 130k (highest!) | **License:** AGPL-3.0 | **Languages:** TypeScript (67%), Python (17%), Rust (5%)
-**npm:** `firecrawl` | **Maintained by:** Firecrawl team
-
-#### What It Is
-An AI-first web scraping API. Converts any website into LLM-ready markdown or structured data. Handles proxies, JS rendering, captchas, and rate limits. Available as a hosted API or self-hosted.
-
-#### Key Features
-- **AI Agent endpoint** — describe what you need, Firecrawl finds and extracts it
-- **Search endpoint** — search the web and get full page content
-- **Scrape endpoint** — convert URL to markdown/HTML/screenshot/JSON
-- **Crawl endpoint** — scrape entire websites
-- **Map endpoint** — discover all URLs on a site instantly
-- **Batch scrape** — thousands of URLs asynchronously
-- **Interact endpoint** — scrape, then interact with AI prompts
-- **Video extraction** — extract video from YouTube, TikTok
-- **PDF/DOCX parsing** — parse uploaded documents
-- **MCP server** — direct integration with AI agents
-- **CLI tool** — `firecrawl scrape/crawl/search`
-- **Skill system** — installable agent skills
-- **Structured output** — Zod schema for typed extraction
-- **Self-hosted option** — Docker deployment
-
-#### Architecture
-```
-API Request
-  → Authentication & Rate Limiting
-    → Orchestrator
-      → Browser Pool (Playwright)
-        → Proxy Rotation (residential)
-          → Anti-detection (stealth, fingerprints)
-            → Content Extraction (HTML → Markdown/JSON)
-              → LLM Processing (optional)
-                → Response
-```
-
-#### What Firecrawl Does That Webscrape Doesn't
-1. **AI Agent endpoint** — natural language data extraction without knowing URLs
-2. **Search endpoint** — web search with full content extraction
-3. **Map endpoint** — instant URL discovery
-4. **Residential proxy pool** — managed IP rotation at scale
-5. **Anti-bot bypass** — captcha solving, fingerprint rotation, JS-blocked content
-6. **Video extraction** — YouTube/TikTok content
-7. **Document parsing** — PDF, DOCX, HTML file processing
-8. **MCP server** — native AI agent integration
-9. **Hosted API** — zero infrastructure management
-10. **Spark models** — custom AI models for web extraction
-
-#### Success Factors
-- **AI-first positioning** — perfect timing for LLM/RAG boom
-- **130k GitHub stars** — fastest-growing in the space
-- **AGPL license + hosted service** — open source + commercial
-- **Multi-language SDKs** — Python, Node.js, Java, Elixir, Rust
-- **Developer experience** — simple API, great docs, playground
-- **Self-hosted option** — Docker deployment for data sovereignty
-
----
-
-### 7. Browserless
-
-**Website:** [browserless.io](https://browserless.io) | **GitHub:** [browserless/browserless](https://github.com/browserless/browserless)
-**Stars:** 13.3k | **License:** SSPL-1.0 / Commercial | **Language:** TypeScript (90%)
-**Docker:** `browserless/chrome` | **Maintained by:** Browserless team
-
-#### What It Is
-A Docker-deployed headless browser service. Provides Puppeteer/Playwright access via WebSocket or REST API. "Browser as a Service" — run Chrome in Docker with managed scaling, queues, and anti-detection.
-
-#### Key Features
-- **WebSocket API** — connect Puppeteer/Playwright remotely
-- **REST API** — screenshot, PDF, HTML, scrape without code
-- **Parallelism & queueing** — handle multiple sessions with concurrency limits
-- **Debug Viewer** — real-time browser session monitoring
-- **BrowserQL** — custom query language for anti-detection
-- **Persistent Sessions** — cookies/cache/localStorage across sessions (90-day retention)
-- **Session Replay** — record and debug browser sessions
-- **Chrome Extensions** — load custom extensions including ad blockers
-- **Advanced Captcha/Stealth** — fingerprint randomization, residential proxy rotation
-- **Inbuilt residential proxy** — automatic IP rotation
-- **/smart-scrape API** — cascading strategies (HTTP → proxy → browser → captcha solving)
-- **/crawl API** — async website crawling
-- **/map API** — URL discovery
-- **/search API** — web search with scraping
-- **MCP Server** — AI assistant integration
-- **ARM64 support** — Apple Silicon compatible
-
-#### Architecture
-```
-Client (Puppeteer/Playwright/REST)
-  → WebSocket / HTTP Gateway
-    → Queue Manager (concurrency control)
-      → Session Manager (isolation, persistence)
-        → Browser Pool (Chromium/Firefox/WebKit)
-          → Docker Container (managed Chrome)
-            → Anti-detection Layer (BrowserQL, fingerprints)
-```
-
-#### What Browserless Does That Webscrape Doesn't
-1. **BrowserQL** — custom query language for anti-detection
-2. **Persistent sessions** — 90-day cookie/cache retention
-3. **Session replay** — record and debug sessions
-4. **Chrome extensions in automation** — ad blockers, captcha solvers
-5. **Residential proxy integration** — built-in IP rotation
-6. **Smart scrape** — cascading strategies (try HTTP first, escalate to browser)
-7. **REST API** — scrape without writing code
-8. **Docker-first** — containerized browser service
-9. **Debug viewer** — real-time session monitoring
-10. **Enterprise features** — webhooks, monitoring, SLA
-
-#### Success Factors
-- **Browser-as-a-Service model** — removes infrastructure burden
-- **Anti-detection focus** — BrowserQL, fingerprint rotation
-- **Docker simplicity** — one command to deploy
-- **Enterprise ready** — SLA, monitoring, webhooks
-- **Battle-tested** — 6,187 commits, 655 releases
-
----
-
-## Feature Comparison Table
-
-| Feature | webscrape | Crawlee | Playwright | Puppeteer | Cheerio | Scrapy | Firecrawl | Browserless |
-|---------|-----------|---------|------------|-----------|---------|--------|-----------|-------------|
-| **Type** | Toolkit | Framework | Browser API | Browser API | Parser | Framework | API Service | BaaS |
-| **GitHub Stars** | — | 23.7k | 90.5k | 94.5k | 30.4k | 62.1k | 130k | 13.3k |
-| **Language** | TypeScript | TypeScript | TypeScript | TypeScript | TypeScript | Python | TS/Py/Rust | TypeScript |
-| **Runtime** | Node/Bun | Node.js | Node/Py/Java/.NET | Node.js | Node/Browser | Python | Any (API) | Docker |
-| **Browser Support** | Chromium | All (PW/Puppeteer) | Chromium/FF/WebKit | Chrome/Firefox | None | None (or Splash) | All (managed) | Chromium/FF/WebKit |
-| **HTTP Scraping** | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (REST) |
-| **HTML Parsing** | ✅ (Cheerio) | ✅ (Cheerio/JSDOM) | ❌ | ❌ | ✅ | ✅ (parsel) | ✅ | ❌ |
-| **Markdown Conversion** | ✅ (custom) | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (core feature) | ✅ (/smart-scrape) |
-| **Proxy Rotation** | ✅ (basic pool) | ✅ (session-based) | ❌ | ❌ | ❌ | ✅ (middleware) | ✅ (residential) | ✅ (residential) |
-| **Rate Limiting** | ✅ (token bucket) | ✅ (built-in) | ❌ | ❌ | ❌ | ✅ (auto) | ✅ (managed) | ✅ (managed) |
-| **Anti-Detection** | ❌ (basic headers) | ✅ (fingerprints) | ❌ | ❌ | ❌ | ❌ | ✅ (stealth) | ✅ (BrowserQL) |
-| **Session/Cookie Mgmt** | ✅ (CookieJar) | ✅ (per-session) | ✅ (contexts) | ✅ (cookies) | ❌ | ✅ (middleware) | ❌ (stateless API) | ✅ (persistent 90d) |
-| **Persistent Queue** | ❌ (in-memory) | ✅ (disk/cloud) | ❌ | ❌ | ❌ | ✅ (memory/SQLite) | ✅ (managed) | ✅ (managed) |
-| **Crawler** | ✅ (basic BFS) | ✅ (advanced) | ❌ | ❌ | ❌ | ✅ (spiders) | ✅ (API endpoint) | ✅ (/crawl API) |
-| **Batch Processing** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ (API endpoint) | ❌ |
-| **Serialization** | ✅ (MD/XML/JSON/Redis) | ✅ (Dataset) | ❌ | ❌ | ✅ (HTML) | ✅ (Feed exports) | ✅ (MD/HTML/JSON) | ✅ (HTML/PDF/Screenshot) |
-| **AI/LLM Integration** | ✅ (formatForAgent) | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (core feature) | ❌ |
-| **Token Estimation** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
-| **Chunking for RAG** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
-| **HTTP Server API** | ✅ (Hono) | ❌ (use Apify) | ❌ | ❌ | ❌ | ✅ (Scrapyrt) | ✅ (cloud API) | ✅ (REST/WS) |
-| **MCP Server** | ❌ | ❌ | ✅ (@playwright/mcp) | ✅ (chrome-devtools-mcp) | ❌ | ❌ | ✅ (firecrawl-mcp) | ✅ |
-| **CLI** | ✅ | ✅ (crawlee create) | ✅ (playwright-cli) | ❌ | ❌ | ✅ (scrapy CLI) | ✅ (firecrawl-cli) | ❌ |
-| **Docker Support** | ❌ | ✅ (templates) | ✅ | ✅ | ❌ | ✅ | ✅ (self-host) | ✅ (core feature) |
-| **Middleware System** | ✅ (pipeline) | ✅ (hooks) | ✅ (route) | ✅ (plugins) | ❌ | ✅ (middleware) | ❌ | ❌ |
-| **Browser Pool** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ (managed) | ✅ (managed) |
-| **Sitemap Discovery** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (/map) | ✅ (/map API) |
-| **Robots.txt** | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
-| **Screenshots** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ |
-| **PDF Generation** | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ |
-| **Video Extraction** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
-| **Mobile Emulation** | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
-| **Tracing/Debug** | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ (Debug Viewer) |
-
----
-
-## Architecture Insights
-
-### 1. Crawlee's Architecture (Best Reference for webscrape)
-
-```
-┌─────────────────────────────────────────────────┐
-│                   Crawlee                        │
-├─────────────┬───────────────┬───────────────────┤
-│  Crawler    │  Queue        │  Storage           │
-│  (strategy) │  (persistent) │  (pluggable)       │
-├─────────────┼───────────────┼───────────────────┤
-│ Playwright  │ Request Queue │ Dataset            │
-│ Puppeteer   │ (FIFO/LIFO)   │ Key-Value Store    │
-│ Cheerio     │               │ Request Queue      │
-│ JSDOM       │               │ (local/cloud)      │
-│ Basic HTTP  │               │                    │
-└─────────────┴───────────────┴───────────────────┘
-         ↓              ↓              ↓
-    Auto-scaling   Session Mgmt   Proxy Rotation
-```
-
-**Key insight:** Crawlee's power comes from the separation of concerns — crawlers, queues, and storage are independent, pluggable components. webscrape could benefit from this modularity.
-
-### 2. Scrapy's Middleware Pipeline (Most Mature Pattern)
-
-```
-Request → [Downloader Middlewares] → Downloader → [Spider Middlewares] → Spider → [Item Pipelines] → Export
-              ↓                                              ↓                    ↓
-         Proxy rotation                              Response processing    Data validation
-         Headers/cookies                             Link extraction        Deduplication
-         Retry logic                                 Character encoding     Storage
-         Redirect handling
-```
-
-**Key insight:** Scrapy's middleware pattern is the most battle-tested approach. webscrape's `MiddlewarePipeline` is a good start but lacks the three-tier separation (downloader → spider → pipeline).
-
-### 3. Firecrawl's API-First Architecture (Most Modern)
-
-```
-┌─────────────┐     ┌──────────────┐     ┌────────────────┐
-│  API Layer  │────→│  Orchestrator │────→│  Browser Pool   │
-│  (REST/WS)  │     │  (job queue)  │     │  (Playwright)   │
-└─────────────┘     └──────────────┘     └────────────────┘
-                           ↓                      ↓
-                    ┌──────────────┐     ┌────────────────┐
-                    │  AI Models   │     │  Proxy Pool     │
-                    │  (Spark)     │     │  (residential)  │
-                    └──────────────┘     └────────────────┘
-                           ↓                      ↓
-                    ┌──────────────────────────────────┐
-                    │  Output (Markdown/JSON/Screenshot)│
-                    └──────────────────────────────────┘
-```
-
-**Key insight:** Firecrawl's success comes from treating scraping as an API service, not a library. webscrape has a server (`server.ts`) but it's basic compared to Firecrawl's managed infrastructure.
-
-### 4. Browserless's Docker-First Architecture
-
-```
-┌──────────────────────────────────────┐
-│           Docker Container            │
-├──────────────┬───────────────────────┤
-│  Gateway     │  Queue Manager        │
-│  (WS/HTTP)   │  (concurrency limits) │
-├──────────────┴───────────────────────┤
-│  Session Manager                      │
-│  (isolation, persistence, replay)     │
-├──────────────────────────────────────┤
-│  Browser Pool                         │
-│  (Chromium/Firefox/WebKit)            │
-├──────────────────────────────────────┤
-│  Anti-Detection Layer                 │
-│  (BrowserQL, fingerprints, proxies)   │
-└──────────────────────────────────────┘
-```
-
-**Key insight:** Browserless succeeds by containerizing the browser and providing multiple access patterns (WebSocket, REST). webscrape's Docker support is a critical gap.
-
----
-
-## Critical Gaps in Webscrape
-
-### Gap 1: No Anti-Detection / Stealth (CRITICAL)
-
-**Current:** Only basic User-Agent rotation in `headers.ts` (4 profiles).
-**Competitors:** Crawlee generates human-like fingerprints. Firecrawl uses residential proxies + stealth. Browserless has BrowserQL.
-**Impact:** webscrape will fail against Cloudflare, DataDome, PerimeterX, and similar bot protection.
-
-**Recommendation:**
-- Integrate `puppeteer-extra-plugin-stealth` (already in optionalDependencies)
-- Add TLS fingerprint randomization
-- Add canvas/WebGL fingerprint spoofing
-- Add navigator property randomization
-
-### Gap 2: No Persistent Queue (HIGH)
-
-**Current:** Crawler uses in-memory `Set` and `Array` for visited tracking and queue.
-**Competitors:** Crawlee has persistent request queue (survives restarts). Scrapy uses SQLite/memory. Firecrawl manages queues server-side.
-**Impact:** Cannot resume crawls, cannot handle millions of URLs, no crash recovery.
-
-**Recommendation:**
-- Add SQLite-backed request queue (already has `better-sqlite3` dependency)
-- Add Redis queue option for distributed crawling
-- Implement crawl state persistence
-
-### Gap 3: No Docker Support (HIGH)
-
-**Current:** No Dockerfile, no container deployment.
-**Competitors:** All major competitors offer Docker support. Browserless is Docker-first.
-**Impact:** Cannot be deployed in containers, no Kubernetes support, no serverless deployment.
-
-**Recommendation:**
-- Add `Dockerfile` for the HTTP server
-- Add `docker-compose.yml` with optional Redis/SQLite
-- Add deployment docs for Railway, Fly.io, AWS ECS
-
-### Gap 4: No MCP Server (MEDIUM-HIGH)
-
-**Current:** No MCP integration.
-**Competitors:** Playwright, Firecrawl, and Browserless all have MCP servers.
-**Impact:** Cannot be used by AI agents (Claude, Cursor, Windsurf) natively.
-
-**Recommendation:**
-- Create `@wenathlan/webscrape-mcp` package
-- Expose scrape/crawl/batch endpoints as MCP tools
-- Integrate with Claude Desktop, VS Code, Cursor
-
-### Gap 5: No Persistent Session Management (MEDIUM)
-
-**Current:** `CookieJar` and `ScrapingSession` are in-memory only.
-**Competitors:** Crawlee persists sessions to disk. Browserless persists for 90 days.
-**Impact:** Cannot maintain login state across restarts, cannot share sessions.
-
-**Recommendation:**
-- Add file-based session persistence
-- Add Redis session storage option
-- Add session export/import
-
-### Gap 6: No XPath Support (MEDIUM)
-
-**Current:** Only CSS selectors via Cheerio.
-**Competitors:** Scrapy has built-in XPath. Playwright supports both CSS and XPath.
-**Impact:** Many scraping patterns require XPath (especially for complex DOM navigation).
-
-**Recommendation:**
-- XPath is already available via Cheerio's `cheerio-select` — just needs documentation
-- Consider adding `xpath` package for advanced XPath 2.0+ support
-
-### Gap 7: No PDF/Video Extraction (MEDIUM)
-
-**Current:** Screenshots only, no PDF generation, no video extraction.
-**Competitors:** Firecrawl extracts videos and PDFs. Browserless generates PDFs. Playwright/Puppeteer have PDF support.
-**Impact:** Cannot handle common scraping use cases (PDF reports, video content).
-
-**Recommendation:**
-- Add PDF generation via Playwright's `page.pdf()`
-- Add PDF content extraction via `pdf-parse`
-- Add video URL detection (YouTube/TikTok metadata)
-
-### Gap 8: No Structured Output with Zod Schemas (MEDIUM)
-
-**Current:** `formatForAgent` generates generic agent output.
-**Competitors:** Firecrawl allows Zod schema for typed extraction. Crawlee has structured output.
-**Impact:** Cannot extract specific data structures from pages.
-
-**Recommendation:**
-- Add `extractWithSchema(url, schema)` function
-- Use Zod for runtime validation
-- Generate typed TypeScript output
-
-### Gap 9: No Webhook Support (LOW-MEDIUM)
-
-**Current:** No webhook/event notification system.
-**Competitors:** Firecrawl has webhooks for async operations. Browserless has webhook integrations.
-**Impact:** Cannot notify external systems of crawl completion or errors.
-
-**Recommendation:**
-- Add webhook configuration to crawl/batch operations
-- Support HTTP POST callbacks on completion/error
-
-### Gap 10: No Adaptive Rendering Detection (LOW-MEDIUM)
-
-**Current:** `detectRenderingMode()` in `fetch.ts` is basic.
-**Competitors:** Crawlee's adaptive crawler persists rendering type detection results.
-**Impact:** Same pages may be re-evaluated on every crawl.
-
-**Recommendation:**
-- Cache rendering mode decisions per domain
-- Use heuristics: frameworks (React/Vue/Angular), API calls, etc.
-
----
-
-## Recommendations
-
-### Priority 1: Anti-Detection (Immediate)
-The `puppeteer-extra-plugin-stealth` is already in `optionalDependencies`. Make it a first-class feature:
-```typescript
-// Add to browser.ts
-import StealthPlugin from 'puppeteer-extra-plugin-stealth';
-```
-
-### Priority 2: Persistent Queue (Week 1)
-Use the existing `better-sqlite3` dependency:
-```typescript
-// New file: queue.ts
-class SqliteQueue {
-  private db: Database;
-  // ... persistent request queue with dedup
-}
-```
-
-### Priority 3: Docker Support (Week 1)
-Add Dockerfile for the HTTP server mode.
-
-### Priority 4: MCP Server (Week 2)
-Create MCP server package for AI agent integration.
-
-### Priority 5: Session Persistence (Week 2)
-Add file/SQLite-based session storage.
-
-### Priority 6: Structured Extraction (Week 3)
-Add Zod-based schema extraction.
-
----
-
-## What Webscrape Does BETTER Than Competitors
-
-1. **All-in-one toolkit** — scraping + serialization + AI formatting + server in one package
-2. **Cross-runtime** — Node.js AND Bun support (unique)
-3. **Multi-format serialization** — Markdown, XML, JSON, Redis, plain text
-4. **Token estimation** — built-in LLM token counting
-5. **Chunking for RAG** — content splitting for retrieval-augmented generation
-6. **LLMs.txt generation** — unique feature for AI discoverability
-7. **Lightweight** — no cloud dependency, no Docker required
-8. **Middleware pipeline** — composable processing chain
-9. **Browser pool** — efficient instance reuse
-10. **Server mode** — instant REST API from any scraping logic
-
----
-
-*This research was compiled from official documentation, GitHub repositories, and npm registries as of June 2026.*