rubycrawl 0.1.3 → 0.2.0

data/README.md

@@ -1,32 +1,56 @@
-# rubycrawl
+# RubyCrawl 🎭
 
-[![Gem Version](https://badge.fury.io/rb/rubycrawl.svg)](https://badge.fury.io/rb/rubycrawl)
+[![Gem Version](https://badge.fury.io/rb/rubycrawl.svg)](https://rubygems.org/gems/rubycrawl)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.0-red.svg)](https://www.ruby-lang.org/)
 
-**Playwright-based web crawler for Ruby** — Inspired by [crawl4ai](https://github.com/unclecode/crawl4ai) (Python), designed idiomatically for Ruby with production-ready features.
+**Production-ready web crawler for Ruby powered by Ferrum** — Full JavaScript rendering via Chrome DevTools Protocol, with first-class Rails support and no Node.js dependency.
 
-RubyCrawl provides accurate, JavaScript-enabled web scraping using Playwright's battle-tested browser automation, wrapped in a clean Ruby API. Perfect for extracting content from modern SPAs and dynamic websites.
+RubyCrawl provides **accurate, JavaScript-enabled web scraping** using a pure Ruby browser automation stack. Perfect for extracting content from modern SPAs, dynamic websites, and building RAG knowledge bases.
+
+**Why RubyCrawl?**
+
+- ✅ **Real browser** — Handles JavaScript, AJAX, and SPAs correctly
+- ✅ **Pure Ruby** — No Node.js, no npm, no external processes to manage
+- ✅ **Zero config** — Works out of the box, no Ferrum knowledge needed
+- ✅ **Production-ready** — Auto-retry, error handling, resource optimization
+- ✅ **Multi-page crawling** — BFS algorithm with smart URL deduplication
+- ✅ **Rails-friendly** — Generators, initializers, and ActiveJob integration
+
+```ruby
+# One line to crawl any JavaScript-heavy site
+result = RubyCrawl.crawl("https://docs.example.com")
+
+result.html           # Full HTML with JS rendered
+result.clean_text     # Noise-stripped plain text (no nav/footer/ads)
+result.clean_markdown # Markdown ready for RAG pipelines
+result.links          # All links with url, text, title, rel
+result.metadata       # Title, description, OG tags, etc.
+```
 
 ## Features
 
-- **Playwright-powered**: Real browser automation for JavaScript-heavy sites
-- **Production-ready**: Designed for Rails apps and production environments
-- **Simple API**: Clean, minimal Ruby interface — zero Playwright knowledge required
-- **Resource optimization**: Built-in resource blocking for faster crawls
-- **Auto-managed browsers**: Browser process reuse and automatic lifecycle management
-- **Content extraction**: HTML, links, and Markdown conversion
-- **Multi-page crawling**: BFS crawler with depth limits and deduplication
+- **Pure Ruby**: Ferrum drives Chromium directly via CDP — no Node.js or npm required
+- **Production-ready**: Designed for Rails apps with auto-retry and exponential backoff
+- **Simple API**: Clean Ruby interface — zero Ferrum or CDP knowledge required
+- **Resource optimization**: Built-in resource blocking for 2-3x faster crawls
+- **Auto-managed browsers**: Lazy Chrome singleton, isolated page per crawl
+- **Content extraction**: HTML, plain text, clean HTML, Markdown (lazy), links, metadata
+- **Multi-page crawling**: BFS crawler with configurable depth limits and URL deduplication
+- **Smart URL handling**: Automatic normalization, tracking parameter removal, same-host filtering
 - **Rails integration**: First-class Rails support with generators and initializers
 
 ## Table of Contents
 
 - [Installation](#installation)
 - [Quick Start](#quick-start)
+- [Use Cases](#use-cases)
 - [Usage](#usage)
   - [Basic Crawling](#basic-crawling)
   - [Multi-Page Crawling](#multi-page-crawling)
   - [Configuration](#configuration)
   - [Result Object](#result-object)
+  - [Error Handling](#error-handling)
 - [Rails Integration](#rails-integration)
 - [Production Deployment](#production-deployment)
 - [Architecture](#architecture)
@@ -40,7 +64,7 @@ RubyCrawl provides accurate, JavaScript-enabled web scraping using Playwright's
 ### Requirements
 
 - **Ruby** >= 3.0
-- **Node.js** LTS (v18+ recommended) — required for the bundled Playwright service
+- **Chrome or Chromium** — managed automatically by Ferrum (downloaded on first use)
 
 ### Add to Gemfile
 
@@ -54,9 +78,9 @@ Then install:
 bundle install
 ```
 
-### Install Playwright browsers
+### Install Chrome
 
-After bundling, install the Playwright browsers:
+Ferrum manages Chrome automatically. Run the install task to verify Chrome is available and generate a Rails initializer:
 
 ```bash
 bundle exec rake rubycrawl:install
@@ -64,9 +88,10 @@ bundle exec rake rubycrawl:install
 
 This command:
 
-- Installs Node.js dependencies in the bundled `node/` directory
-- Downloads Playwright browsers (Chromium, Firefox, WebKit)
-- Creates a Rails initializer (if using Rails)
+- ✅ Checks for Chrome/Chromium in your PATH
+- ✅ Creates a Rails initializer (if using Rails)
+
+**Note:** If Chrome is not in your PATH, install it via your system package manager or download from [google.com/chrome](https://www.google.com/chrome/).
 
 ## Quick Start
 
@@ -77,27 +102,37 @@ require "rubycrawl"
 result = RubyCrawl.crawl("https://example.com")
 
 # Access extracted content
-puts result.html      # Raw HTML content
-puts result.markdown  # Converted to Markdown
-puts result.links     # Extracted links from the page
-puts result.metadata  # Status code, final URL, etc.
+result.final_url      # Final URL after redirects
+result.clean_text     # Noise-stripped plain text (no nav/footer/ads)
+result.clean_html     # Noise-stripped HTML (same noise removed as clean_text)
+result.raw_text       # Full body.innerText (unfiltered)
+result.html           # Full raw HTML content
+result.links          # Extracted links with url, text, title, rel
+result.metadata       # Title, description, OG tags, etc.
+result.clean_markdown # Markdown converted from clean_html (lazy — first access only)
 ```
 
+## Use Cases
+
+RubyCrawl is perfect for:
+
+- **RAG applications**: Build knowledge bases for LLM/AI applications by crawling documentation sites
+- **Data aggregation**: Crawl product catalogs, job listings, or news articles
+- **SEO analysis**: Extract metadata, links, and content structure
+- **Content migration**: Convert existing sites to Markdown for static site generators
+- **Documentation scraping**: Create local copies of documentation with preserved links
+
 ## Usage
 
 ### Basic Crawling
 
-The simplest way to crawl a URL:
-
 ```ruby
 result = RubyCrawl.crawl("https://example.com")
 
-# Access the results
-result.html      # => "<html>...</html>"
-result.markdown  # => "# Example Domain\n\nThis domain is..." (lazy-loaded)
-result.links     # => [{ "url" => "https://...", "text" => "More info" }, ...]
-result.metadata  # => { "status" => 200, "final_url" => "https://example.com" }
-result.text      # => "" (coming soon)
+result.html           # => "<html>...</html>"
+result.clean_text     # => "Example Domain\n\nThis domain is..." (no nav/ads)
+result.raw_text       # => "Example Domain\nThis domain is..." (full body text)
+result.metadata       # => { "final_url" => "https://example.com", "title" => "..." }
 ```
 
 ### Multi-Page Crawling
@@ -109,50 +144,83 @@ Crawl an entire site following links with BFS (breadth-first search):
 RubyCrawl.crawl_site("https://example.com", max_pages: 100, max_depth: 3) do |page|
   # Each page is yielded as it's crawled (streaming)
   puts "Crawled: #{page.url} (depth: #{page.depth})"
-  
+
   # Save to database
   Page.create!(
-    url: page.url,
-    html: page.html,
-    markdown: page.markdown,
-    depth: page.depth
+    url:      page.url,
+    html:     page.html,
+    markdown: page.clean_markdown,
+    depth:    page.depth
   )
 end
 ```
 
+**Real-world example: Building a RAG knowledge base**
+
+```ruby
+require "rubycrawl"
+
+RubyCrawl.configure(
+  wait_until: "networkidle",  # Ensure JS content loads
+  block_resources: true       # Skip images/fonts for speed
+)
+
+pages_crawled = RubyCrawl.crawl_site(
+  "https://docs.example.com",
+  max_pages: 500,
+  max_depth: 5,
+  same_host_only: true
+) do |page|
+  VectorDB.upsert(
+    id:       Digest::SHA256.hexdigest(page.url),
+    content:  page.clean_markdown,
+    metadata: {
+      url:   page.url,
+      title: page.metadata["title"],
+      depth: page.depth
+    }
+  )
+end
+
+puts "Indexed #{pages_crawled} pages"
+```
+
 #### Multi-Page Options
 
-| Option | Default | Description |
-|--------|---------|-------------|
-| `max_pages` | 50 | Maximum number of pages to crawl |
-| `max_depth` | 3 | Maximum link depth from start URL |
-| `same_host_only` | true | Only follow links on the same domain |
-| `wait_until` | inherited | Page load strategy |
-| `block_resources` | inherited | Block images/fonts/CSS |
+| Option            | Default   | Description                          |
+| ----------------- | --------- | ------------------------------------ |
+| `max_pages`       | 50        | Maximum number of pages to crawl     |
+| `max_depth`       | 3         | Maximum link depth from start URL    |
+| `same_host_only`  | true      | Only follow links on the same domain |
+| `wait_until`      | inherited | Page load strategy                   |
+| `block_resources` | inherited | Block images/fonts/CSS               |
 
 #### Page Result Object
 
 The block receives a `PageResult` with:
 
 ```ruby
-page.url       # String: Final URL after redirects
-page.html      # String: Full HTML content  
-page.markdown  # String: Lazy-converted Markdown
-page.links     # Array: URLs extracted from page
-page.metadata  # Hash: HTTP status, final URL, etc.
-page.depth     # Integer: Link depth from start URL
+page.url            # String: Final URL after redirects
+page.html           # String: Full raw HTML content
+page.clean_html     # String: Noise-stripped HTML (no nav/header/footer/ads)
+page.clean_text     # String: Noise-stripped plain text (derived from clean_html)
+page.raw_text       # String: Full body.innerText (unfiltered)
+page.clean_markdown # String: Lazy-converted Markdown from clean_html
+page.links          # Array: URLs extracted from page
+page.metadata       # Hash: final_url, title, OG tags, etc.
+page.depth          # Integer: Link depth from start URL
 ```
 
 ### Configuration
 
 #### Global Configuration
 
-Set default options that apply to all crawls:
-
 ```ruby
 RubyCrawl.configure(
-  wait_until: "networkidle",  # Wait until network is idle
-  block_resources: true        # Block images, fonts, CSS for speed
+  wait_until:      "networkidle",
+  block_resources: true,
+  timeout:         60,
+  headless:        true
 )
 
 # All subsequent crawls use these defaults
@@ -161,8 +229,6 @@ result = RubyCrawl.crawl("https://example.com")
 
 #### Per-Request Options
 
-Override defaults for specific requests:
-
 ```ruby
 # Use global defaults
 result = RubyCrawl.crawl("https://example.com")
@@ -170,36 +236,41 @@ result = RubyCrawl.crawl("https://example.com")
 # Override for this request only
 result = RubyCrawl.crawl(
   "https://example.com",
-  wait_until: "domcontentloaded",
+  wait_until:      "domcontentloaded",
   block_resources: false
 )
 ```
 
 #### Configuration Options
 
-| Option            | Values                                          | Default  | Description                                       |
-| ----------------- | ----------------------------------------------- | -------- | ------------------------------------------------- |
-| `wait_until`      | `"load"`, `"domcontentloaded"`, `"networkidle"` | `"load"` | When to consider page loaded                      |
-| `block_resources` | `true`, `false`                                 | `true`   | Block images, fonts, CSS, media for faster crawls |
+| Option            | Values                                                      | Default | Description                                         |
+| ----------------- | ----------------------------------------------------------- | ------- | --------------------------------------------------- |
+| `wait_until`      | `"load"`, `"domcontentloaded"`, `"networkidle"`, `"commit"` | `nil`   | When to consider page loaded (nil = Ferrum default) |
+| `block_resources` | `true`, `false`                                             | `nil`   | Block images, fonts, CSS, media for faster crawls   |
+| `max_attempts`    | Integer                                                     | `3`     | Total number of attempts (including the first)      |
+| `timeout`         | Integer (seconds)                                           | `30`    | Browser navigation timeout                          |
+| `headless`        | `true`, `false`                                             | `true`  | Run Chrome headlessly                               |
 
 **Wait strategies explained:**
 
-- `load` — Wait for the load event (fastest, good for static sites)
-- `domcontentloaded` — Wait for DOM ready (medium speed)
-- `networkidle` — Wait until no network requests for 500ms (slowest, best for SPAs)
+- `load` — Wait for the load event (good for static sites)
+- `domcontentloaded` — Wait for DOM ready (faster)
+- `networkidle` — Wait until no network requests for 500ms (best for SPAs)
+- `commit` — Wait until the first response bytes are received (fastest)
 
 ### Result Object
 
-The crawl result is a `RubyCrawl::Result` object with these attributes:
-
 ```ruby
 result = RubyCrawl.crawl("https://example.com")
 
-result.html      # String: Raw HTML content from page
-result.markdown  # String: Markdown conversion (lazy-loaded on first access)
-result.links     # Array: Extracted links with url and text
-result.text      # String: Plain text (coming soon)
-result.metadata  # Hash: Comprehensive metadata (see below)
+result.html           # String: Full raw HTML
+result.clean_html     # String: Noise-stripped HTML (nav/header/footer/ads removed)
+result.clean_text     # String: Plain text derived from clean_html — ideal for RAG
+result.raw_text       # String: Full body.innerText (unfiltered)
+result.clean_markdown # String: Markdown from clean_html (lazy — computed on first access)
+result.links          # Array: Extracted links with url/text/title/rel
+result.metadata       # Hash: See below
+result.final_url      # String: Shortcut for metadata['final_url']
 ```
 
 #### Links Format
@@ -207,101 +278,89 @@ result.metadata  # Hash: Comprehensive metadata (see below)
 ```ruby
 result.links
 # => [
-#   { "url" => "https://example.com/about", "text" => "About Us" },
-#   { "url" => "https://example.com/contact", "text" => "Contact" },
-#   ...
+#   { "url" => "https://example.com/about", "text" => "About", "title" => nil, "rel" => nil },
+#   { "url" => "https://example.com/contact", "text" => "Contact", "title" => nil, "rel" => "nofollow" },
 # ]
 ```
 
+URLs are automatically resolved to absolute form by the browser.
+
 #### Markdown Conversion
 
-Markdown is **lazy-loaded** — conversion only happens when you access `.markdown`:
+Markdown is **lazy** — conversion only happens on first access of `.clean_markdown`:
 
 ```ruby
-result = RubyCrawl.crawl(url)
-result.html       # ✅ No overhead
-result.markdown   # ⬅️ Conversion happens here (first call only)
-result.markdown   # ✅ Cached, instant
+result.clean_html     # ✅ Already available, no overhead
+result.clean_markdown # Converts clean_html → Markdown here (first call only)
+result.clean_markdown # ✅ Cached, instant on subsequent calls
 ```
 
 Uses [reverse_markdown](https://github.com/xijo/reverse_markdown) with GitHub-flavored output.
 
 #### Metadata Fields
 
-The `metadata` hash includes HTTP and HTML metadata:
-
 ```ruby
 result.metadata
 # => {
-#   "status" => 200,                 # HTTP status code
-#   "final_url" => "https://...",    # Final URL after redirects
-#   "title" => "Page Title",         # <title> tag
-#   "description" => "...",          # Meta description
-#   "keywords" => "ruby, web",       # Meta keywords
-#   "author" => "Author Name",       # Meta author
-#   "og_title" => "...",             # Open Graph title
-#   "og_description" => "...",       # Open Graph description
-#   "og_image" => "https://...",     # Open Graph image
-#   "og_url" => "https://...",       # Open Graph URL
-#   "og_type" => "website",          # Open Graph type
-#   "twitter_card" => "summary",     # Twitter card type
-#   "twitter_title" => "...",        # Twitter title
-#   "twitter_description" => "...",  # Twitter description
-#   "twitter_image" => "https://...",# Twitter image
-#   "canonical" => "https://...",    # Canonical URL
-#   "lang" => "en",                  # Page language
-#   "charset" => "UTF-8"             # Character encoding
+#   "final_url"           => "https://example.com",
+#   "title"               => "Page Title",
+#   "description"         => "...",
+#   "keywords"            => "ruby, web",
+#   "author"              => "Author Name",
+#   "og_title"            => "...",
+#   "og_description"      => "...",
+#   "og_image"            => "https://...",
+#   "og_url"              => "https://...",
+#   "og_type"             => "website",
+#   "twitter_card"        => "summary",
+#   "twitter_title"       => "...",
+#   "twitter_description" => "...",
+#   "twitter_image"       => "https://...",
+#   "canonical"           => "https://...",
+#   "lang"                => "en",
+#   "charset"             => "UTF-8"
 # }
 ```
 
-Note: All HTML metadata fields may be `null` if not present on the page.
-
 ### Error Handling
 
-RubyCrawl provides specific exception classes for different error scenarios:
-
 ```ruby
 begin
   result = RubyCrawl.crawl(url)
 rescue RubyCrawl::ConfigurationError => e
-  # Invalid URL or configuration
-  puts "Configuration error: #{e.message}"
+  # Invalid URL or option value
 rescue RubyCrawl::TimeoutError => e
-  # Page load timeout or network timeout
-  puts "Timeout: #{e.message}"
+  # Page load timed out
 rescue RubyCrawl::NavigationError => e
-  # Page navigation failed (404, DNS error, SSL error, etc.)
-  puts "Navigation failed: #{e.message}"
+  # Navigation failed (404, DNS error, SSL error)
 rescue RubyCrawl::ServiceError => e
-  # Node service unavailable or crashed
-  puts "Service error: #{e.message}"
+  # Browser failed to start or crashed
 rescue RubyCrawl::Error => e
   # Catch-all for any RubyCrawl error
-  puts "Crawl error: #{e.message}"
 end
 ```
 
 **Exception Hierarchy:**
-- `RubyCrawl::Error` (base class)
-  - `RubyCrawl::ConfigurationError` - Invalid URL or configuration
-  - `RubyCrawl::TimeoutError` - Timeout during crawl
-  - `RubyCrawl::NavigationError` - Page navigation failed
-  - `RubyCrawl::ServiceError` - Node service issues
 
-**Automatic Retry:** RubyCrawl automatically retries transient failures (service errors, timeouts) up to 3 times with exponential backoff (2s, 4s, 8s). Configure with:
+```
+RubyCrawl::Error
+  ├── ConfigurationError  — invalid URL or option value
+  ├── TimeoutError        — page load timed out
+  ├── NavigationError     — navigation failed (HTTP error, DNS, SSL)
+  └── ServiceError        — browser failed to start or crashed
+```
+
+**Automatic Retry:** `ServiceError` and `TimeoutError` are retried with exponential backoff. `NavigationError` and `ConfigurationError` are not retried (they won't succeed on retry).
 
 ```ruby
-RubyCrawl.configure(max_retries: 5)
-# or per-request
-RubyCrawl.crawl(url, retries: 1)  # Disable retry
+RubyCrawl.configure(max_attempts: 5)     # 5 total attempts
+RubyCrawl.crawl(url, max_attempts: 1)    # Disable retries
 ```
 
 ## Rails Integration
 
 ### Installation
 
-Run the installer in your Rails app:
-
 ```bash
 bundle exec rake rubycrawl:install
 ```
@@ -309,264 +368,157 @@ bundle exec rake rubycrawl:install
 This creates `config/initializers/rubycrawl.rb`:
 
 ```ruby
-# frozen_string_literal: true
-
-# rubycrawl default configuration
 RubyCrawl.configure(
-  wait_until: "load",
+  wait_until:      "load",
   block_resources: true
 )
 ```
 
 ### Usage in Rails
 
+#### Background Jobs with ActiveJob
+
 ```ruby
-# In a controller, service, or background job
-class ContentScraperJob < ApplicationJob
+class CrawlPageJob < ApplicationJob
+  queue_as :crawlers
+
+  retry_on RubyCrawl::ServiceError, wait: :exponentially_longer, attempts: 5
+  retry_on RubyCrawl::TimeoutError, wait: :exponentially_longer, attempts: 3
+  discard_on RubyCrawl::ConfigurationError
+
   def perform(url)
     result = RubyCrawl.crawl(url)
 
-    # Save to database
-    ScrapedContent.create!(
-      url: url,
-      html: result.html,
-      status: result.metadata[:status]
+    Page.create!(
+      url:        result.final_url,
+      title:      result.metadata['title'],
+      content:    result.clean_text,
+      markdown:   result.clean_markdown,
+      crawled_at: Time.current
     )
   end
 end
 ```
 
+**Multi-page RAG knowledge base:**
+
+```ruby
+class BuildKnowledgeBaseJob < ApplicationJob
+  queue_as :crawlers
+
+  def perform(documentation_url)
+    RubyCrawl.crawl_site(documentation_url, max_pages: 500, max_depth: 5) do |page|
+      embedding = OpenAI.embed(page.clean_markdown)
+
+      Document.create!(
+        url:       page.url,
+        title:     page.metadata['title'],
+        content:   page.clean_markdown,
+        embedding: embedding,
+        depth:     page.depth
+      )
+    end
+  end
+end
+```
+
+#### Best Practices
+
+1. **Use background jobs** to avoid blocking web requests
+2. **Configure retry logic** based on error type
+3. **Store `clean_markdown`** for RAG applications (preserves heading structure for chunking)
+4. **Rate limit** external crawling to be respectful
+
 ## Production Deployment
 
 ### Pre-deployment Checklist
 
-1. **Install Node.js** on your production servers (LTS version recommended)
+1. **Ensure Chrome is installed** on your production servers
 2. **Run installer** during deployment:
    ```bash
    bundle exec rake rubycrawl:install
    ```
-3. **Set environment variables** (optional):
-   ```bash
-   export RUBYCRAWL_NODE_BIN=/usr/bin/node  # Custom Node.js path
-   export RUBYCRAWL_NODE_LOG=/var/log/rubycrawl.log  # Service logs
-   ```
 
 ### Docker Example
 
 ```dockerfile
 FROM ruby:3.2
 
-# Install Node.js LTS
-RUN curl -fsSL https://deb.nodesource.com/setup_lts.x | bash - \
-    && apt-get install -y nodejs
-
-# Install system dependencies for Playwright
-RUN npx playwright install-deps
+# Install Chrome
+RUN apt-get update && apt-get install -y \
+    chromium \
+    --no-install-recommends \
+    && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
 COPY Gemfile* ./
 RUN bundle install
 
-# Install Playwright browsers
-RUN bundle exec rake rubycrawl:install
-
 COPY . .
 CMD ["rails", "server"]
 ```
 
-### Heroku Deployment
-
-Add the Node.js buildpack:
-
-```bash
-heroku buildpacks:add heroku/nodejs
-heroku buildpacks:add heroku/ruby
-```
-
-Add to `package.json` in your Rails root:
+Ferrum will detect `chromium` automatically. To specify a custom path:
 
-```json
-{
-  "engines": {
-    "node": "18.x"
-  }
-}
+```ruby
+RubyCrawl.configure(
+  browser_options: { "browser-path": "/usr/bin/chromium" }
+)
 ```
 
-### Performance Tips
-
-- **Reuse instances**: Use the class-level `RubyCrawl.crawl` method (recommended) rather than creating new instances
-- **Resource blocking**: Keep `block_resources: true` for 2-3x faster crawls when you don't need images/CSS
-- **Concurrency**: Use background jobs (Sidekiq, etc.) for parallel crawling
-- **Browser reuse**: The first crawl is slower due to browser launch; subsequent crawls reuse the process
-
 ## Architecture
 
-RubyCrawl uses a **dual-process architecture**:
+RubyCrawl uses a single-process architecture:
 
 ```
-┌─────────────────────────────────────────────┐
-│  Ruby Process (Your Application)            │
-│  ┌─────────────────────────────────────┐   │
-│  │  RubyCrawl Gem                        │   │
-│  │  • Public API                        │   │
-│  │  • Result normalization              │   │
-│  │  • Error handling                    │   │
-│  └────────────┬────────────────────────┘   │
-└───────────────┼─────────────────────────────┘
-                │ HTTP/JSON (localhost:3344)
-┌───────────────┼─────────────────────────────┐
-│  Node.js Process (Auto-started)              │
-│  ┌────────────┴────────────────────────┐   │
-│  │  Playwright Service                  │   │
-│  │  • Browser management                │   │
-│  │  • Page navigation                   │   │
-│  │  • HTML extraction                   │   │
-│  │  • Resource blocking                 │   │
-│  └─────────────────────────────────────┘   │
-└─────────────────────────────────────────────┘
+RubyCrawl (public API)
+  ↓
+Browser (lib/rubycrawl/browser.rb)  ← Ferrum wrapper
+  ↓
+Ferrum::Browser                     ← Chrome DevTools Protocol (pure Ruby)
+  ↓
+Chromium                            ← headless browser
 ```
 
-**Why this architecture?**
-
-- **Separation of concerns**: Ruby handles orchestration, Node handles browsers
-- **Stability**: Playwright's official Node.js bindings are most reliable
-- **Performance**: Long-running browser process, reused across requests
-- **Simplicity**: No C extensions, pure Ruby + bundled Node service
-
-See [.github/copilot-instructions.md](.github/copilot-instructions.md) for detailed architecture documentation.
+- Chrome launches once lazily and is reused across all crawls
+- Each crawl gets an isolated page context (own cookies/storage)
+- JS extraction runs inside the browser via `page.evaluate()`
+- No separate processes, no HTTP boundary, no Node.js
 
 ## Performance
 
-### Benchmarks
-
-Typical crawl times (M1 Mac, fast network):
-
-| Page Type   | First Crawl | Subsequent | Config                      |
-| ----------- | ----------- | ---------- | --------------------------- |
-| Static HTML | ~2s         | ~500ms     | `block_resources: true`     |
-| SPA (React) | ~3s         | ~1.2s      | `wait_until: "networkidle"` |
-| Heavy site  | ~4s         | ~2s        | `block_resources: false`    |
-
-**Note**: First crawl includes browser launch time (~1.5s). Subsequent crawls reuse the browser.
-
-### Optimization Tips
-
-1. **Enable resource blocking** for content-only extraction:
-
-   ```ruby
-   RubyCrawl.configure(block_resources: true)
-   ```
-
-2. **Use appropriate wait strategy**:
-   - Static sites: `wait_until: "load"`
-   - SPAs: `wait_until: "networkidle"`
-
-3. **Batch processing**: Use background jobs for concurrent crawling:
-   ```ruby
-   urls.each { |url| CrawlJob.perform_later(url) }
-   ```
+- **Resource blocking**: Keep `block_resources: true` (default: nil) to skip images/fonts/CSS for 2-3x faster crawls
+- **Wait strategy**: Use `wait_until: "load"` for static sites, `"networkidle"` for SPAs
+- **Concurrency**: Use background jobs (Sidekiq, GoodJob, etc.) for parallel crawling
+- **Browser reuse**: The first crawl is slower (~2s) due to Chrome launch; subsequent crawls are much faster (~200-500ms)
 
 ## Development
 
-### Setup
-
 ```bash
 git clone git@github.com:craft-wise/rubycrawl.git
 cd rubycrawl
-bin/setup  # Installs dependencies and sets up Node service
-```
-
-### Running Tests
+bin/setup
 
-```bash
+# Run unit tests (no browser required)
 bundle exec rspec
-```
-
-### Manual Testing
 
-```bash
-# Terminal 1: Start Node service manually (optional)
-cd node
-npm start
+# Run integration tests (requires Chrome)
+INTEGRATION=1 bundle exec rspec
 
-# Terminal 2: Ruby console
+# Manual testing
 bin/console
-> result = RubyCrawl.crawl("https://example.com")
-> puts result.html
+> RubyCrawl.crawl("https://example.com")
+> RubyCrawl.crawl("https://example.com").clean_text
+> RubyCrawl.crawl("https://example.com").clean_markdown
 ```
 
-### Project Structure
-
-```
-rubycrawl/
-├── lib/
-│   ├── rubycrawl.rb              # Main gem entry point
-│   ├── rubycrawl/
-│   │   ├── version.rb           # Gem version
-│   │   ├── railtie.rb           # Rails integration
-│   │   └── tasks/
-│   │       └── install.rake     # Installation task
-├── node/
-│   ├── src/
-│   │   └── index.js             # Playwright HTTP service
-│   ├── package.json
-│   └── README.md
-├── spec/                        # RSpec tests
-├── .github/
-│   └── copilot-instructions.md  # GitHub Copilot guidelines
-├── CLAUDE.md                    # Claude AI guidelines
-└── README.md
-```
-
-## Roadmap
-
-### Current (v0.1.0)
-
-- [x] HTML extraction
-- [x] Link extraction
-- [x] Markdown conversion (lazy-loaded)
-- [x] Multi-page crawling with BFS
-- [x] URL normalization and deduplication
-- [x] Basic metadata (status, final URL)
-- [x] Resource blocking
-- [x] Rails integration
-
-### Coming Soon
-
-- [ ] Plain text extraction
-- [ ] Screenshot capture
-- [ ] Custom JavaScript execution
-- [ ] Session/cookie support
-- [ ] Proxy support
-- [ ] Robots.txt support
-
 ## Contributing
 
 Contributions are welcome! Please read our [contribution guidelines](.github/copilot-instructions.md) first.
 
-### Development Philosophy
-
 - **Simplicity over cleverness**: Prefer clear, explicit code
 - **Stability over speed**: Correctness first, optimization second
-- **Ruby-first**: Hide Node.js/Playwright complexity from users
-- **No vendor lock-in**: Pure open source, no SaaS dependencies
-
-## Comparison with crawl4ai
-
-| Feature             | crawl4ai (Python) | rubycrawl (Ruby) |
-| ------------------- | ----------------- | ---------------- |
-| Browser automation  | Playwright        | Playwright       |
-| Language            | Python            | Ruby             |
-| LLM extraction      | ✅                | Planned          |
-| Markdown extraction | ✅                | ✅               |
-| Link extraction     | ✅                | ✅               |
-| Multi-page crawling | ✅                | ✅               |
-| Rails integration   | N/A               | ✅               |
-| Resource blocking   | ✅                | ✅               |
-| Session management  | ✅                | Planned          |
-
-RubyCrawl aims to bring the same level of accuracy and reliability to the Ruby ecosystem.
+- **Hide complexity**: Users should never need to know Ferrum exists
 
 ## License
 
@@ -574,12 +526,12 @@ The gem is available as open source under the terms of the [MIT License](LICENSE
 
 ## Credits
 
-Inspired by [crawl4ai](https://github.com/unclecode/crawl4ai) by @unclecode.
+Built with [Ferrum](https://github.com/rubycdp/ferrum) — pure Ruby Chrome DevTools Protocol client.
 
-Built with [Playwright](https://playwright.dev/) by Microsoft.
+Powered by [reverse_markdown](https://github.com/xijo/reverse_markdown) for GitHub-flavored Markdown conversion.
 
 ## Support
 
 - **Issues**: [GitHub Issues](https://github.com/craft-wise/rubycrawl/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/your-org/rubycrawl/discussions)
+- **Discussions**: [GitHub Discussions](https://github.com/craft-wise/rubycrawl/discussions)
 - **Email**: ganesh.navale@zohomail.in