puppeteer-ruby 0.45.6 → 0.50.0.alpha5

data/CLAUDE/cdp_protocol.md

@@ -0,0 +1,230 @@
+# Chrome DevTools Protocol (CDP)
+
+This document covers CDP usage in puppeteer-ruby.
+
+## Overview
+
+The Chrome DevTools Protocol (CDP) is the low-level protocol used to communicate with Chrome/Chromium browsers. All browser automation in puppeteer-ruby is done through CDP.
+
+## CDP Domains
+
+CDP is organized into domains, each handling specific functionality:
+
+| Domain | Purpose | Key Commands |
+|--------|---------|--------------|
+| `Page` | Page lifecycle | `navigate`, `reload`, `setContent` |
+| `Runtime` | JavaScript execution | `evaluate`, `callFunctionOn` |
+| `DOM` | DOM manipulation | `querySelector`, `getContentQuads` |
+| `Input` | User input simulation | `dispatchKeyEvent`, `dispatchMouseEvent` |
+| `Network` | Network control | `enable`, `setRequestInterception` |
+| `Emulation` | Device emulation | `setDeviceMetricsOverride` |
+| `Target` | Target management | `createTarget`, `attachToTarget` |
+
+## Sending CDP Commands
+
+### Basic Pattern
+
+```ruby
+# Via CDPSession
+result = @client.send_message('Page.navigate', url: 'https://example.com')
+
+# Result is a hash with response data
+# { "frameId" => "...", "loaderId" => "..." }
+```
+
+### With Timeout
+
+```ruby
+# send_message has built-in timeout handling
+result = @client.send_message('Page.captureScreenshot', format: 'png')
+```
+
+### Error Handling
+
+```ruby
+begin
+  @client.send_message('Page.navigate', url: 'invalid-url')
+rescue Puppeteer::CDPSession::Error => e
+  # Handle CDP error
+  puts "CDP Error: #{e.message}"
+end
+```
+
+## Subscribing to Events
+
+### Basic Event Subscription
+
+```ruby
+# Subscribe to event (persistent)
+@client.on('Network.requestWillBeSent') do |event|
+  puts "Request: #{event['request']['url']}"
+end
+
+# Subscribe once (auto-removes after first event)
+@client.once('Page.loadEventFired') do |event|
+  puts "Page loaded!"
+end
+```
+
+### Enabling Domains
+
+Some CDP domains require explicit enabling before events are sent:
+
+```ruby
+# Enable the Network domain
+@client.send_message('Network.enable')
+
+# Now Network events will be received
+@client.on('Network.requestWillBeSent') { |e| ... }
+@client.on('Network.responseReceived') { |e| ... }
+```
+
+### Common Domains That Need Enabling
+
+- `Network.enable` - Network events
+- `Page.enable` - Page lifecycle events
+- `Runtime.enable` - Runtime events (console, exceptions)
+- `DOM.enable` - DOM events
+
+## Common CDP Patterns
+
+### JavaScript Evaluation
+
+```ruby
+# Evaluate expression
+result = @client.send_message('Runtime.evaluate',
+  expression: 'document.title',
+  returnByValue: true
+)
+# result['result']['value'] => "Page Title"
+
+# Call function on object
+result = @client.send_message('Runtime.callFunctionOn',
+  functionDeclaration: '(a, b) => a + b',
+  arguments: [{ value: 1 }, { value: 2 }],
+  executionContextId: context_id,
+  returnByValue: true
+)
+# result['result']['value'] => 3
+```
+
+### DOM Queries
+
+```ruby
+# Get document node
+doc = @client.send_message('DOM.getDocument')
+root_id = doc['root']['nodeId']
+
+# Query selector
+result = @client.send_message('DOM.querySelector',
+  nodeId: root_id,
+  selector: 'button'
+)
+button_node_id = result['nodeId']
+```
+
+### Input Simulation
+
+```ruby
+# Mouse click
+@client.send_message('Input.dispatchMouseEvent',
+  type: 'mousePressed',
+  x: 100,
+  y: 200,
+  button: 'left',
+  clickCount: 1
+)
+@client.send_message('Input.dispatchMouseEvent',
+  type: 'mouseReleased',
+  x: 100,
+  y: 200,
+  button: 'left',
+  clickCount: 1
+)
+
+# Key press
+@client.send_message('Input.dispatchKeyEvent',
+  type: 'keyDown',
+  key: 'Enter',
+  code: 'Enter'
+)
+@client.send_message('Input.dispatchKeyEvent',
+  type: 'keyUp',
+  key: 'Enter',
+  code: 'Enter'
+)
+```
+
+### Screenshots
+
+```ruby
+result = @client.send_message('Page.captureScreenshot',
+  format: 'png',
+  clip: {
+    x: 0,
+    y: 0,
+    width: 800,
+    height: 600,
+    scale: 1
+  }
+)
+image_data = Base64.decode64(result['data'])
+```
+
+## CDP Session Management
+
+### Session Hierarchy
+
+```
+Browser
+  └── Connection (WebSocket to browser DevTools)
+        ├── Browser-level CDPSession
+        └── Target-level CDPSessions (one per page/worker)
+```
+
+### Creating Target Sessions
+
+```ruby
+# New page creates its own session
+target = browser.wait_for_target { |t| t.url.include?('example.com') }
+session = target.create_cdp_session
+
+# Use session for that specific page
+session.send_message('Page.enable')
+```
+
+## CDP Events Reference
+
+### Page Events
+
+| Event | When Fired |
+|-------|------------|
+| `Page.loadEventFired` | Window load event |
+| `Page.domContentEventFired` | DOMContentLoaded event |
+| `Page.frameNavigated` | Frame navigation complete |
+| `Page.frameAttached` | New frame attached |
+| `Page.frameDetached` | Frame removed |
+
+### Network Events
+
+| Event | When Fired |
+|-------|------------|
+| `Network.requestWillBeSent` | Request about to be sent |
+| `Network.responseReceived` | Response headers received |
+| `Network.loadingFinished` | Response body loaded |
+| `Network.loadingFailed` | Request failed |
+
+### Runtime Events
+
+| Event | When Fired |
+|-------|------------|
+| `Runtime.consoleAPICalled` | console.log/warn/error |
+| `Runtime.exceptionThrown` | Uncaught exception |
+| `Runtime.executionContextCreated` | New JS context |
+| `Runtime.executionContextDestroyed` | Context destroyed |
+
+## Resources
+
+- [Chrome DevTools Protocol Documentation](https://chromedevtools.github.io/devtools-protocol/)
+- [CDP Protocol Viewer](https://chromedevtools.github.io/devtools-protocol/tot/)
+- [Puppeteer CDP Usage](https://github.com/puppeteer/puppeteer/tree/main/packages/puppeteer-core/src/cdp)

data/CLAUDE/concurrency.md

@@ -0,0 +1,216 @@
+# Concurrency Model
+
+This document explains the concurrency architecture in puppeteer-ruby using `socketry/async`.
+
+## Current State: socketry/async
+
+puppeteer-ruby uses Fiber-based concurrency with the `socketry/async` gem (version 2.35.1+).
+
+### Key Components
+
+| Component | Purpose |
+|-----------|---------|
+| `Async::Promise` | Promise that can be resolved/rejected later |
+| `Puppeteer::AsyncUtils` | Utility module for async operations |
+| `Puppeteer::ReactorRunner` | Dedicated Async reactor thread for sync API |
+
+### AsyncUtils Module
+
+Located in `lib/puppeteer/async_utils.rb`:
+
+```ruby
+module Puppeteer::AsyncUtils
+  # Wait for all promises to complete (like Promise.all)
+  def await_promise_all(*tasks)
+    # ...
+  end
+
+  # Wait for first promise to complete (like Promise.race)
+  def await_promise_race(*tasks)
+    # ...
+  end
+
+  # Timeout wrapper
+  def async_timeout(timeout_ms, &block)
+    # ...
+  end
+
+  # Sleep helper that works in Async context
+  def sleep_seconds(seconds)
+    # ...
+  end
+end
+```
+
+### ReactorRunner
+
+`ReactorRunner` manages a dedicated Async reactor thread and provides a bridge between synchronous API calls and the Async context:
+
+```ruby
+# From lib/puppeteer/reactor_runner.rb
+
+# ReactorRunner runs Async operations in a dedicated thread
+runner = Puppeteer::ReactorRunner.new
+
+# Wrap an object to proxy calls through the reactor
+browser = runner.wrap(actual_browser)
+
+# Calls are executed in the Async reactor context
+browser.new_page  # Runs in reactor thread
+```
+
+Key features:
+- Dedicated thread running Async reactor
+- Proxies method calls into reactor context
+- Handles result unwrapping and error propagation
+- `wait_until_idle` for graceful shutdown
+
+### Threading Model
+
+```
+Main Thread                   Reactor Thread (Async)
+    │                              │
+    ├── sync method call ─────────►│
+    │                              │ (execute in Async context)
+    │   ◄────────────────────────── result
+    │                              │
+    ├── browser.close ────────────►│
+    │                              │ wait_until_idle
+    │   ◄────────────────────────── cleanup complete
+    │                              │
+```
+
+### Promise Patterns
+
+#### Creating and Resolving Promises
+
+```ruby
+# Create a promise
+promise = Async::Promise.new
+
+# Resolve with value
+promise.resolve(result)
+
+# Reject with error
+promise.reject(error)
+
+# Wait for result
+result = promise.wait
+```
+
+#### Waiting for Events
+
+```ruby
+# Common pattern for waiting on events
+promise = Async::Promise.new.tap do |p|
+  page.once('load') { p.resolve(true) }
+end
+
+# Later, wait for the event
+promise.wait
+```
+
+#### Running Multiple Operations
+
+```ruby
+# Wait for all (like Promise.all)
+Puppeteer::AsyncUtils.await_promise_all(
+  page.async_goto('https://example1.com'),
+  page.async_goto('https://example2.com'),
+)
+
+# Wait for any (like Promise.race)
+Puppeteer::AsyncUtils.await_promise_race(
+  timeout_promise,
+  navigation_promise,
+)
+```
+
+#### Timeout Handling
+
+```ruby
+# With timeout (milliseconds)
+Puppeteer::AsyncUtils.async_timeout(5000) do
+  slow_operation
+end
+# Raises Async::TimeoutError if exceeded
+```
+
+### Async Method Pattern
+
+```ruby
+# Define synchronous method
+def wait_for_selector(selector, timeout: nil)
+  # Implementation...
+end
+
+# Generate async version that returns Async task
+define_async_method :async_wait_for_selector
+```
+
+Usage:
+
+```ruby
+# Synchronous (blocks)
+element = page.wait_for_selector('button')
+
+# Asynchronous (returns Async task)
+task = page.async_wait_for_selector('button')
+# Do other work...
+element = task.wait
+```
+
+## Key Implementation Files
+
+| File | Description |
+|------|-------------|
+| `lib/puppeteer/async_utils.rb` | Core async utility functions |
+| `lib/puppeteer/reactor_runner.rb` | Reactor thread management |
+| `lib/puppeteer/define_async_method.rb` | Async method generation |
+| `lib/puppeteer/connection.rb` | WebSocket with async messaging |
+| `lib/puppeteer/lifecycle_watcher.rb` | Navigation wait logic |
+
+## Guidelines for New Code
+
+### Do
+
+- Use `Async::Promise` for deferred results
+- Use `AsyncUtils` methods for combining promises
+- Keep synchronous and async logic separate
+- Handle `Async::TimeoutError` for timeout operations
+- Use `Mutex` for shared state between threads
+
+### Don't
+
+- Block the reactor thread with synchronous I/O
+- Create nested `Async` blocks unnecessarily
+- Ignore promise rejections
+- Use `sleep` directly (use `AsyncUtils.sleep_seconds`)
+
+### Example: Async-Ready Code
+
+```ruby
+class MyComponent
+  def perform_operation(timeout: 30000)
+    promise = Async::Promise.new
+
+    listener_id = @emitter.add_event_listener('complete') do |result|
+      promise.resolve(result) unless promise.resolved?
+    end
+
+    begin
+      Puppeteer::AsyncUtils.async_timeout(timeout) do
+        promise.wait
+      end
+    ensure
+      @emitter.remove_event_listener(listener_id)
+    end
+  end
+end
+```
+
+## Resources
+
+- [socketry/async documentation](https://github.com/socketry/async)
+- [Async::Promise](https://github.com/socketry/async)
+- [Ruby Fiber scheduler](https://ruby-doc.org/core-3.1.0/Fiber/Scheduler.html)