rubycrawl 0.1.2 → 0.1.4

data/README.md

@@ -1,39 +1,67 @@
-# rubycrawl
+# RubyCrawl 🎭
 
-[![Gem Version](https://badge.fury.io/rb/rubycrawl.svg)](https://badge.fury.io/rb/rubycrawl)
+[![Gem Version](https://badge.fury.io/rb/rubycrawl.svg)](https://rubygems.org/gems/rubycrawl)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.0-red.svg)](https://www.ruby-lang.org/)
+[![Node.js](https://img.shields.io/badge/node.js-18%2B-green.svg)](https://nodejs.org/)
 
-**Playwright-based web crawler for Ruby** — Inspired by [crawl4ai](https://github.com/unclecode/crawl4ai) (Python), designed idiomatically for Ruby with production-ready features.
+**Production-ready web crawler for Ruby powered by Playwright** — Bringing the power of modern browser automation to the Ruby ecosystem with first-class Rails support.
 
-RubyCrawl provides accurate, JavaScript-enabled web scraping using Playwright's battle-tested browser automation, wrapped in a clean Ruby API. Perfect for extracting content from modern SPAs and dynamic websites.
+RubyCrawl provides **accurate, JavaScript-enabled web scraping** using Playwright's battle-tested browser automation, wrapped in a clean Ruby API. Perfect for extracting content from modern SPAs, dynamic websites, and building RAG knowledge bases.
+
+**Why RubyCrawl?**
+
+- ✅ **Real browser** — Handles JavaScript, AJAX, and SPAs correctly
+- ✅ **Zero config** — Works out of the box, no Playwright knowledge needed
+- ✅ **Production-ready** — Auto-retry, error handling, resource optimization
+- ✅ **Multi-page crawling** — BFS algorithm with smart URL deduplication
+- ✅ **Rails-friendly** — Generators, initializers, and ActiveJob integration
+- ✅ **Modular architecture** — Clean, testable, maintainable codebase
+
+```ruby
+# One line to crawl any JavaScript-heavy site
+result = RubyCrawl.crawl("https://docs.example.com")
+
+result.html           # Full HTML with JS rendered
+result.links          # All links with metadata
+result.metadata       # Title, description, OG tags, etc.
+```
 
 ## Features
 
-- **Playwright-powered**: Real browser automation for JavaScript-heavy sites
-- **Production-ready**: Designed for Rails apps and production environments
-- **Simple API**: Clean, minimal Ruby interface — zero Playwright knowledge required
-- **Resource optimization**: Built-in resource blocking for faster crawls
-- **Auto-managed browsers**: Browser process reuse and automatic lifecycle management
-- **Content extraction**: HTML, links, and Markdown conversion
-- **Multi-page crawling**: BFS crawler with depth limits and deduplication
-- **Rails integration**: First-class Rails support with generators and initializers
+- **🎭 Playwright-powered**: Real browser automation for JavaScript-heavy sites and SPAs
+- **🚀 Production-ready**: Designed for Rails apps and production environments with auto-retry and error handling
+- **🎯 Simple API**: Clean, minimal Ruby interface — zero Playwright or Node.js knowledge required
+- **⚡ Resource optimization**: Built-in resource blocking for 2-3x faster crawls
+- **🔄 Auto-managed browsers**: Browser process reuse and automatic lifecycle management
+- **📄 Content extraction**: HTML, plain text, links (with metadata), and **clean markdown** via HTML conversion
+- **🌐 Multi-page crawling**: BFS (breadth-first search) crawler with configurable depth limits and URL deduplication
+- **🛡️ Smart URL handling**: Automatic normalization, tracking parameter removal, and same-host filtering
+- **🔧 Rails integration**: First-class Rails support with generators and initializers
+- **💎 Modular design**: Clean separation of concerns with focused, testable modules
 
 ## Table of Contents
 
+- [Features](#features)
 - [Installation](#installation)
 - [Quick Start](#quick-start)
+- [Use Cases](#use-cases)
 - [Usage](#usage)
   - [Basic Crawling](#basic-crawling)
   - [Multi-Page Crawling](#multi-page-crawling)
   - [Configuration](#configuration)
   - [Result Object](#result-object)
+  - [Error Handling](#error-handling)
 - [Rails Integration](#rails-integration)
 - [Production Deployment](#production-deployment)
 - [Architecture](#architecture)
 - [Performance](#performance)
 - [Development](#development)
+  - [Project Structure](#project-structure)
 - [Contributing](#contributing)
+- [Why Choose RubyCrawl?](#why-choose-rubycrawl)
 - [License](#license)
+- [Support](#support)
 
 ## Installation
 
@@ -64,9 +92,24 @@ bundle exec rake rubycrawl:install
 
 This command:
 
-- Installs Node.js dependencies in the bundled `node/` directory
-- Downloads Playwright browsers (Chromium, Firefox, WebKit)
-- Creates a Rails initializer (if using Rails)
+- ✅ Installs Node.js dependencies in the bundled `node/` directory
+- ✅ Downloads Playwright browsers (Chromium, Firefox, WebKit) — ~300MB download
+- ✅ Creates a Rails initializer (if using Rails)
+
+**Note:** You only need to run this once. The installation task is idempotent and safe to run multiple times.
+
+**Troubleshooting installation:**
+
+```bash
+# If installation fails, check Node.js version
+node --version  # Should be v18+ LTS
+
+# Enable verbose logging
+RUBYCRAWL_NODE_LOG=/tmp/rubycrawl.log bundle exec rake rubycrawl:install
+
+# Check installation status
+cd node && npm list
+```
 
 ## Quick Start
 
@@ -77,12 +120,24 @@ require "rubycrawl"
 result = RubyCrawl.crawl("https://example.com")
 
 # Access extracted content
-puts result.html      # Raw HTML content
-puts result.markdown  # Converted to Markdown
-puts result.links     # Extracted links from the page
-puts result.metadata  # Status code, final URL, etc.
+result.final_url       # Final URL after redirects
+result.text            # Plain text content (via innerText)
+result.html            # Raw HTML content
+result.links           # Extracted links with metadata
+result.metadata        # Title, description, OG tags, etc.
 ```
 
+## Use Cases
+
+RubyCrawl is perfect for:
+
+- **📊 Data aggregation**: Crawl product catalogs, job listings, or news articles
+- **🤖 RAG applications**: Build knowledge bases for LLM/AI applications by crawling documentation sites
+- **🔍 SEO analysis**: Extract metadata, links, and content structure
+- **📱 Content migration**: Convert existing sites to Markdown for static site generators
+- **🧪 Testing**: Verify deployed site structure and content
+- **📚 Documentation scraping**: Create local copies of documentation with preserved links
+
 ## Usage
 
 ### Basic Crawling
@@ -93,11 +148,9 @@ The simplest way to crawl a URL:
 result = RubyCrawl.crawl("https://example.com")
 
 # Access the results
-result.html      # => "<html>...</html>"
-result.markdown  # => "# Example Domain\n\nThis domain is..." (lazy-loaded)
-result.links     # => [{ "url" => "https://...", "text" => "More info" }, ...]
-result.metadata  # => { "status" => 200, "final_url" => "https://example.com" }
-result.text      # => "" (coming soon)
+result.html            # => "<html>...</html>"
+result.text            # => "Example Domain\nThis domain is..." (plain text via innerText)
+result.metadata        # => { "status" => 200, "final_url" => "https://example.com" }
 ```
 
 ### Multi-Page Crawling
@@ -109,38 +162,72 @@ Crawl an entire site following links with BFS (breadth-first search):
 RubyCrawl.crawl_site("https://example.com", max_pages: 100, max_depth: 3) do |page|
   # Each page is yielded as it's crawled (streaming)
   puts "Crawled: #{page.url} (depth: #{page.depth})"
-  
+
   # Save to database
   Page.create!(
     url: page.url,
     html: page.html,
-    markdown: page.markdown,
+    markdown: page.clean_markdown,
     depth: page.depth
   )
 end
 ```
 
+**Real-world example: Building a RAG knowledge base**
+
+```ruby
+# Crawl documentation site for AI/RAG application
+require "rubycrawl"
+
+RubyCrawl.configure(
+  wait_until: "networkidle",  # Ensure JS content loads
+  block_resources: true       # Skip images/fonts for speed
+)
+
+pages_crawled = RubyCrawl.crawl_site(
+  "https://docs.example.com",
+  max_pages: 500,
+  max_depth: 5,
+  same_host_only: true
+) do |page|
+  # Store in vector database for RAG
+  VectorDB.upsert(
+    id: Digest::SHA256.hexdigest(page.url),
+    content: page.clean_markdown,  # Clean markdown for better embeddings
+    metadata: {
+      url: page.url,
+      title: page.metadata["title"],
+      depth: page.depth
+    }
+  )
+
+  puts "✓ Indexed: #{page.metadata['title']} (#{page.depth} levels deep)"
+end
+
+puts "Crawled #{pages_crawled} pages into knowledge base"
+```
+
 #### Multi-Page Options
 
-| Option | Default | Description |
-|--------|---------|-------------|
-| `max_pages` | 50 | Maximum number of pages to crawl |
-| `max_depth` | 3 | Maximum link depth from start URL |
-| `same_host_only` | true | Only follow links on the same domain |
-| `wait_until` | inherited | Page load strategy |
-| `block_resources` | inherited | Block images/fonts/CSS |
+| Option            | Default   | Description                          |
+| ----------------- | --------- | ------------------------------------ |
+| `max_pages`       | 50        | Maximum number of pages to crawl     |
+| `max_depth`       | 3         | Maximum link depth from start URL    |
+| `same_host_only`  | true      | Only follow links on the same domain |
+| `wait_until`      | inherited | Page load strategy                   |
+| `block_resources` | inherited | Block images/fonts/CSS               |
 
 #### Page Result Object
 
 The block receives a `PageResult` with:
 
 ```ruby
-page.url       # String: Final URL after redirects
-page.html      # String: Full HTML content  
-page.markdown  # String: Lazy-converted Markdown
-page.links     # Array: URLs extracted from page
-page.metadata  # Hash: HTTP status, final URL, etc.
-page.depth     # Integer: Link depth from start URL
+page.url            # String: Final URL after redirects
+page.html           # String: Full HTML content
+page.clean_markdown # String: Lazy-converted Markdown
+page.links          # Array: URLs extracted from page
+page.metadata       # Hash: HTTP status, final URL, etc.
+page.depth          # Integer: Link depth from start URL
 ```
 
 ### Configuration
@@ -177,16 +264,55 @@ result = RubyCrawl.crawl(
 
 #### Configuration Options
 
-| Option            | Values                                          | Default  | Description                                       |
-| ----------------- | ----------------------------------------------- | -------- | ------------------------------------------------- |
-| `wait_until`      | `"load"`, `"domcontentloaded"`, `"networkidle"` | `"load"` | When to consider page loaded                      |
-| `block_resources` | `true`, `false`                                 | `true`   | Block images, fonts, CSS, media for faster crawls |
+| Option            | Values                                                                 | Default  | Description                                       |
+| ----------------- | ---------------------------------------------------------------------- | -------- | ------------------------------------------------- |
+| `wait_until`      | `"load"`, `"domcontentloaded"`, `"networkidle"`, `"commit"`           | `"load"` | When to consider page loaded                      |
+| `block_resources` | `true`, `false`                                                        | `true`   | Block images, fonts, CSS, media for faster crawls |
+| `max_attempts`    | Integer                                                                | `3`      | Total number of attempts (including the first)    |
 
 **Wait strategies explained:**
 
 - `load` — Wait for the load event (fastest, good for static sites)
 - `domcontentloaded` — Wait for DOM ready (medium speed)
 - `networkidle` — Wait until no network requests for 500ms (slowest, best for SPAs)
+- `commit` — Wait until the first response bytes are received (fastest possible)
+
+### Advanced Usage
+
+#### Session-Based Crawling
+
+Sessions allow reusing browser contexts for better performance when crawling multiple pages. They're automatically used by `crawl_site`, but you can manage them manually for advanced use cases:
+
+```ruby
+# Create a session (reusable browser context)
+session_id = RubyCrawl.create_session
+
+begin
+  # All crawls with this session_id share the same browser context
+  result1 = RubyCrawl.crawl("https://example.com/page1", session_id: session_id)
+  result2 = RubyCrawl.crawl("https://example.com/page2", session_id: session_id)
+  # Browser state (cookies, localStorage) persists between crawls
+ensure
+  # Always destroy session when done
+  RubyCrawl.destroy_session(session_id)
+end
+```
+
+**When to use sessions:**
+
+- Multiple sequential crawls to the same domain (better performance)
+- Preserving cookies/state set by the site between page visits
+- Avoiding browser context creation overhead
+
+**Important:** Sessions are for **performance optimization only**. RubyCrawl is designed for crawling **public websites**. It does not provide authentication or login functionality for protected content.
+
+**Note:** `crawl_site` automatically creates and manages a session internally, so you don't need manual session management for multi-page crawling.
+
+**Session lifecycle:**
+
+- Sessions automatically expire after 30 minutes of inactivity
+- Sessions are cleaned up every 5 minutes
+- Always call `destroy_session` when done to free resources immediately
 
 ### Result Object
 
@@ -195,33 +321,47 @@ The crawl result is a `RubyCrawl::Result` object with these attributes:
 ```ruby
 result = RubyCrawl.crawl("https://example.com")
 
-result.html      # String: Raw HTML content from page
-result.markdown  # String: Markdown conversion (lazy-loaded on first access)
-result.links     # Array: Extracted links with url and text
-result.text      # String: Plain text (coming soon)
-result.metadata  # Hash: Comprehensive metadata (see below)
+result.html           # String: Raw HTML content from page
+result.text           # String: Plain text via document.body.innerText
+result.clean_markdown # String: Markdown conversion (lazy-loaded on first access)
+result.links          # Array: Extracted links with url and text
+result.metadata       # Hash: Comprehensive metadata (see below)
 ```
 
 #### Links Format
 
+Links are extracted with full metadata:
+
 ```ruby
 result.links
 # => [
-#   { "url" => "https://example.com/about", "text" => "About Us" },
-#   { "url" => "https://example.com/contact", "text" => "Contact" },
+#   {
+#     "url" => "https://example.com/about",
+#     "text" => "About Us",
+#     "title" => "Learn more about us",  # <a title="...">
+#     "rel" => nil                        # <a rel="nofollow">
+#   },
+#   {
+#     "url" => "https://example.com/contact",
+#     "text" => "Contact",
+#     "title" => null,
+#     "rel" => "nofollow"
+#   },
 #   ...
 # ]
 ```
 
+**Note:** URLs are automatically converted to absolute URLs by the browser, so relative links like `/about` become `https://example.com/about`.
+
 #### Markdown Conversion
 
-Markdown is **lazy-loaded** — conversion only happens when you access `.markdown`:
+Markdown is **lazy-loaded** — conversion only happens when you access `.clean_markdown`:
 
 ```ruby
 result = RubyCrawl.crawl(url)
-result.html       # ✅ No overhead
-result.markdown   # ⬅️ Conversion happens here (first call only)
-result.markdown   # ✅ Cached, instant
+result.html           # ✅ No overhead
+result.clean_markdown # ⬅️ Conversion happens here (first call only)
+result.clean_markdown # ✅ Cached, instant
 ```
 
 Uses [reverse_markdown](https://github.com/xijo/reverse_markdown) with GitHub-flavored output.
@@ -282,18 +422,19 @@ end
 ```
 
 **Exception Hierarchy:**
+
 - `RubyCrawl::Error` (base class)
   - `RubyCrawl::ConfigurationError` - Invalid URL or configuration
   - `RubyCrawl::TimeoutError` - Timeout during crawl
   - `RubyCrawl::NavigationError` - Page navigation failed
   - `RubyCrawl::ServiceError` - Node service issues
 
-**Automatic Retry:** RubyCrawl automatically retries transient failures (service errors, timeouts) up to 3 times with exponential backoff (2s, 4s, 8s). Configure with:
+**Automatic Retry:** RubyCrawl automatically retries transient failures (service errors, timeouts) with exponential backoff. The default `max_attempts: 3` means 3 total attempts (2 retries). Configure with:
 
 ```ruby
-RubyCrawl.configure(max_retries: 5)
+RubyCrawl.configure(max_attempts: 5)
 # or per-request
-RubyCrawl.crawl(url, retries: 1)  # Disable retry
+RubyCrawl.crawl(url, max_attempts: 1)  # No retries
 ```
 
 ## Rails Integration
@@ -320,22 +461,177 @@ RubyCrawl.configure(
 
 ### Usage in Rails
 
+#### Basic Usage in Controllers
+
+```ruby
+class PagesController < ApplicationController
+  def show
+    result = RubyCrawl.crawl(params[:url])
+
+    @page = Page.create!(
+      url: result.final_url,
+      title: result.metadata['title'],
+      html: result.html,
+      text: result.text,
+      markdown: result.clean_markdown
+    )
+
+    redirect_to @page
+  end
+end
+```
+
+#### Background Jobs with ActiveJob
+
+**Simple Crawl Job:**
+
 ```ruby
-# In a controller, service, or background job
-class ContentScraperJob < ApplicationJob
-  def perform(url)
+class CrawlPageJob < ApplicationJob
+  queue_as :crawlers
+
+  # Automatic retry with exponential backoff for transient failures
+  retry_on RubyCrawl::ServiceError, wait: :exponentially_longer, attempts: 5
+  retry_on RubyCrawl::TimeoutError, wait: :exponentially_longer, attempts: 3
+
+  # Don't retry on configuration errors (bad URLs)
+  discard_on RubyCrawl::ConfigurationError
+
+  def perform(url, user_id: nil)
     result = RubyCrawl.crawl(url)
 
-    # Save to database
-    ScrapedContent.create!(
-      url: url,
+    Page.create!(
+      url: result.final_url,
+      title: result.metadata['title'],
+      text: result.text,
       html: result.html,
-      status: result.metadata[:status]
+      user_id: user_id,
+      crawled_at: Time.current
     )
+  rescue RubyCrawl::NavigationError => e
+    # Page not found or failed to load
+    Rails.logger.warn "Failed to crawl #{url}: #{e.message}"
+    FailedCrawl.create!(url: url, error: e.message, user_id: user_id)
   end
 end
+
+# Enqueue from anywhere
+CrawlPageJob.perform_later("https://example.com", user_id: current_user.id)
 ```
 
+**Multi-Page Site Crawler Job:**
+
+```ruby
+class CrawlSiteJob < ApplicationJob
+  queue_as :crawlers
+
+  def perform(start_url, max_pages: 50)
+    pages_crawled = RubyCrawl.crawl_site(
+      start_url,
+      max_pages: max_pages,
+      max_depth: 3,
+      same_host_only: true
+    ) do |page|
+      Page.create!(
+        url: page.url,
+        title: page.metadata['title'],
+        text: page.clean_markdown, # Store markdown for RAG applications
+        depth: page.depth,
+        crawled_at: Time.current
+      )
+    end
+
+    Rails.logger.info "Crawled #{pages_crawled} pages from #{start_url}"
+  end
+end
+```
+
+**Batch Crawling Pattern:**
+
+```ruby
+class BatchCrawlJob < ApplicationJob
+  queue_as :crawlers
+
+  def perform(urls)
+    # Create session for better performance
+    session_id = RubyCrawl.create_session
+
+    begin
+      urls.each do |url|
+        result = RubyCrawl.crawl(url, session_id: session_id)
+
+        Page.create!(
+          url: result.final_url,
+          html: result.html,
+          text: result.text
+        )
+      end
+    ensure
+      # Always destroy session when done
+      RubyCrawl.destroy_session(session_id)
+    end
+  end
+end
+
+# Enqueue batch
+BatchCrawlJob.perform_later(["https://example.com", "https://example.com/about"])
+```
+
+**Periodic Crawling with Sidekiq-Cron:**
+
+```ruby
+# config/schedule.yml (for sidekiq-cron)
+crawl_news_sites:
+  cron: "0 */6 * * *"  # Every 6 hours
+  class: "CrawlNewsSitesJob"
+
+# app/jobs/crawl_news_sites_job.rb
+class CrawlNewsSitesJob < ApplicationJob
+  queue_as :scheduled_crawlers
+
+  def perform
+    Site.where(active: true).find_each do |site|
+      CrawlSiteJob.perform_later(site.url, max_pages: site.max_pages)
+    end
+  end
+end
+```
+
+**RAG/AI Knowledge Base Pattern:**
+
+```ruby
+class BuildKnowledgeBaseJob < ApplicationJob
+  queue_as :crawlers
+
+  def perform(documentation_url)
+    RubyCrawl.crawl_site(
+      documentation_url,
+      max_pages: 500,
+      max_depth: 5
+    ) do |page|
+      # Store in vector database for RAG
+      embedding = OpenAI.embed(page.clean_markdown)
+
+      Document.create!(
+        url: page.url,
+        title: page.metadata['title'],
+        content: page.clean_markdown,
+        embedding: embedding,
+        depth: page.depth
+      )
+    end
+  end
+end
+```
+
+#### Best Practices
+
+1. **Use background jobs** for crawling to avoid blocking web requests
+2. **Configure retry logic** based on error types (retry ServiceError, discard ConfigurationError)
+3. **Use sessions** for batch crawling to improve performance
+4. **Monitor job failures** and set up alerts for repeated errors
+5. **Rate limit** external crawling to be respectful (use job throttling)
+6. **Store both HTML and text** for flexibility in data processing
+
 ## Production Deployment
 
 ### Pre-deployment Checklist
@@ -393,154 +689,41 @@ Add to `package.json` in your Rails root:
 }
 ```
 
-### Performance Tips
-
-- **Reuse instances**: Use the class-level `RubyCrawl.crawl` method (recommended) rather than creating new instances
-- **Resource blocking**: Keep `block_resources: true` for 2-3x faster crawls when you don't need images/CSS
-- **Concurrency**: Use background jobs (Sidekiq, etc.) for parallel crawling
-- **Browser reuse**: The first crawl is slower due to browser launch; subsequent crawls reuse the process
-
-## Architecture
-
-RubyCrawl uses a **dual-process architecture**:
-
-```
-┌─────────────────────────────────────────────┐
-│  Ruby Process (Your Application)            │
-│  ┌─────────────────────────────────────┐   │
-│  │  RubyCrawl Gem                        │   │
-│  │  • Public API                        │   │
-│  │  • Result normalization              │   │
-│  │  • Error handling                    │   │
-│  └────────────┬────────────────────────┘   │
-└───────────────┼─────────────────────────────┘
-                │ HTTP/JSON (localhost:3344)
-┌───────────────┼─────────────────────────────┐
-│  Node.js Process (Auto-started)              │
-│  ┌────────────┴────────────────────────┐   │
-│  │  Playwright Service                  │   │
-│  │  • Browser management                │   │
-│  │  • Page navigation                   │   │
-│  │  • HTML extraction                   │   │
-│  │  • Resource blocking                 │   │
-│  └─────────────────────────────────────┘   │
-└─────────────────────────────────────────────┘
-```
-
-**Why this architecture?**
-
-- **Separation of concerns**: Ruby handles orchestration, Node handles browsers
-- **Stability**: Playwright's official Node.js bindings are most reliable
-- **Performance**: Long-running browser process, reused across requests
-- **Simplicity**: No C extensions, pure Ruby + bundled Node service
-
-See [.github/copilot-instructions.md](.github/copilot-instructions.md) for detailed architecture documentation.
-
-## Performance
-
-### Benchmarks
+## How It Works
 
-Typical crawl times (M1 Mac, fast network):
+RubyCrawl uses a simple architecture:
 
-| Page Type   | First Crawl | Subsequent | Config                      |
-| ----------- | ----------- | ---------- | --------------------------- |
-| Static HTML | ~2s         | ~500ms     | `block_resources: true`     |
-| SPA (React) | ~3s         | ~1.2s      | `wait_until: "networkidle"` |
-| Heavy site  | ~4s         | ~2s        | `block_resources: false`    |
+- **Ruby Gem** provides the public API and handles orchestration
+- **Node.js Service** (bundled, auto-started) manages Playwright browsers
+- Communication via HTTP/JSON on localhost
 
-**Note**: First crawl includes browser launch time (~1.5s). Subsequent crawls reuse the browser.
+This design keeps things stable and easy to debug. The browser runs in a separate process, so crashes won't affect your Ruby application.
 
-### Optimization Tips
+## Performance Tips
 
-1. **Enable resource blocking** for content-only extraction:
-
-   ```ruby
-   RubyCrawl.configure(block_resources: true)
-   ```
-
-2. **Use appropriate wait strategy**:
-   - Static sites: `wait_until: "load"`
-   - SPAs: `wait_until: "networkidle"`
-
-3. **Batch processing**: Use background jobs for concurrent crawling:
-   ```ruby
-   urls.each { |url| CrawlJob.perform_later(url) }
-   ```
+- **Resource blocking**: Keep `block_resources: true` (default) for 2-3x faster crawls when you don't need images/CSS
+- **Wait strategy**: Use `wait_until: "load"` for static sites, `"networkidle"` for SPAs
+- **Concurrency**: Use background jobs (Sidekiq, etc.) for parallel crawling
+- **Browser reuse**: The first crawl is slower (~2s) due to browser launch; subsequent crawls are much faster (~500ms)
 
 ## Development
 
-### Setup
+Want to contribute? Check out the [contributor guidelines](.github/copilot-instructions.md).
 
 ```bash
+# Setup
 git clone git@github.com:craft-wise/rubycrawl.git
 cd rubycrawl
-bin/setup  # Installs dependencies and sets up Node service
-```
+bin/setup
 
-### Running Tests
-
-```bash
+# Run tests
 bundle exec rspec
-```
-
-### Manual Testing
-
-```bash
-# Terminal 1: Start Node service manually (optional)
-cd node
-npm start
 
-# Terminal 2: Ruby console
+# Manual testing
 bin/console
-> result = RubyCrawl.crawl("https://example.com")
-> puts result.html
-```
-
-### Project Structure
-
-```
-rubycrawl/
-├── lib/
-│   ├── rubycrawl.rb              # Main gem entry point
-│   ├── rubycrawl/
-│   │   ├── version.rb           # Gem version
-│   │   ├── railtie.rb           # Rails integration
-│   │   └── tasks/
-│   │       └── install.rake     # Installation task
-├── node/
-│   ├── src/
-│   │   └── index.js             # Playwright HTTP service
-│   ├── package.json
-│   └── README.md
-├── spec/                        # RSpec tests
-├── .github/
-│   └── copilot-instructions.md  # GitHub Copilot guidelines
-├── CLAUDE.md                    # Claude AI guidelines
-└── README.md
+> RubyCrawl.crawl("https://example.com")
 ```
 
-## Roadmap
-
-### Current (v0.1.0)
-
-- [x] HTML extraction
-- [x] Link extraction
-- [x] Markdown conversion (lazy-loaded)
-- [x] Multi-page crawling with BFS
-- [x] URL normalization and deduplication
-- [x] Basic metadata (status, final URL)
-- [x] Resource blocking
-- [x] Rails integration
-
-### Coming Soon
-
-- [ ] Plain text extraction
-- [ ] Screenshot capture
-- [ ] Custom JavaScript execution
-- [ ] Session/cookie support
-- [ ] Proxy support
-- [ ] Robots.txt support
-
 ## Contributing
 
 Contributions are welcome! Please read our [contribution guidelines](.github/copilot-instructions.md) first.
@@ -552,21 +735,46 @@ Contributions are welcome! Please read our [contribution guidelines](.github/cop
 - **Ruby-first**: Hide Node.js/Playwright complexity from users
 - **No vendor lock-in**: Pure open source, no SaaS dependencies
 
-## Comparison with crawl4ai
+## Why Choose RubyCrawl?
+
+RubyCrawl stands out in the Ruby ecosystem with its unique combination of features:
+
+### 🎯 **Built for Ruby Developers**
+
+- **Idiomatic Ruby API** — Feels natural to Rubyists, no need to learn Playwright
+- **Rails-first design** — Generators, initializers, and ActiveJob integration out of the box
+- **Modular architecture** — Clean, testable code following Ruby best practices
+
+### 🚀 **Production-Grade Reliability**
 
-| Feature             | crawl4ai (Python) | rubycrawl (Ruby) |
-| ------------------- | ----------------- | ---------------- |
-| Browser automation  | Playwright        | Playwright       |
-| Language            | Python            | Ruby             |
-| LLM extraction      | ✅                | Planned          |
-| Markdown extraction | ✅                | ✅               |
-| Link extraction     | ✅                | ✅               |
-| Multi-page crawling | ✅                | ✅               |
-| Rails integration   | N/A               | ✅               |
-| Resource blocking   | ✅                | ✅               |
-| Session management  | ✅                | Planned          |
+- **Automatic retry** with exponential backoff for transient failures
+- **Smart error handling** with custom exception hierarchy
+- **Process isolation** — Browser crashes don't affect your Ruby application
+- **Battle-tested** — Built on Playwright's proven browser automation
 
-RubyCrawl aims to bring the same level of accuracy and reliability to the Ruby ecosystem.
+### 💎 **Developer Experience**
+
+- **Zero configuration** — Works immediately after installation
+- **Lazy loading** — Markdown conversion only when you need it
+- **Smart URL handling** — Automatic normalization and deduplication
+- **Comprehensive docs** — Clear examples for common use cases
+
+### 🌐 **Rich Feature Set**
+
+- ✅ JavaScript-enabled crawling (SPAs, AJAX, dynamic content)
+- ✅ Multi-page crawling with BFS algorithm
+- ✅ Link extraction with metadata (url, text, title, rel)
+- ✅ Markdown conversion (GitHub-flavored)
+- ✅ Metadata extraction (OG tags, Twitter cards, etc.)
+- ✅ Resource blocking for 2-3x performance boost
+
+### 📊 **Perfect for Modern Use Cases**
+
+- **RAG applications** — Build AI knowledge bases from documentation
+- **Data aggregation** — Extract structured data from multiple pages
+- **Content migration** — Convert sites to Markdown for static generators
+- **SEO analysis** — Extract metadata and link structures
+- **Testing** — Verify deployed site content and structure
 
 ## License
 
@@ -574,12 +782,21 @@ The gem is available as open source under the terms of the [MIT License](LICENSE
 
 ## Credits
 
-Inspired by [crawl4ai](https://github.com/unclecode/crawl4ai) by @unclecode.
+Built with [Playwright](https://playwright.dev/) by Microsoft — the industry-standard browser automation framework.
 
-Built with [Playwright](https://playwright.dev/) by Microsoft.
+Powered by [reverse_markdown](https://github.com/xijo/reverse_markdown) for GitHub-flavored Markdown conversion.
 
 ## Support
 
 - **Issues**: [GitHub Issues](https://github.com/craft-wise/rubycrawl/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/your-org/rubycrawl/discussions)
+- **Discussions**: [GitHub Discussions](https://github.com/craft-wise/rubycrawl/discussions)
 - **Email**: ganesh.navale@zohomail.in
+
+## Acknowledgments
+
+Special thanks to:
+
+- [Microsoft Playwright](https://playwright.dev/) team for the robust, production-grade browser automation framework
+- The Ruby community for building an ecosystem that values developer happiness and code clarity
+- The Node.js community for excellent tooling and libraries that make cross-language integration seamless
+- Open source contributors worldwide who make projects like this possible