puppeteer-bidi 0.0.1.beta1

data/CLAUDE/selector_evaluation.md

@@ -0,0 +1,198 @@
+# Selector Evaluation Methods Implementation
+
+This document explains the implementation of `eval_on_selector` and `eval_on_selector_all` methods, including delegation patterns, handle lifecycle, and performance considerations.
+
+### Overview
+
+The `eval_on_selector` and `eval_on_selector_all` methods provide convenient shortcuts for querying elements and evaluating JavaScript functions on them, equivalent to Puppeteer's `$eval` and `$$eval`.
+
+### API Design
+
+#### Method Naming Convention
+
+Ruby cannot use `$` in method names, so we use descriptive alternatives:
+
+| Puppeteer | Ruby | Description |
+|-----------|------|-------------|
+| `$eval` | `eval_on_selector` | Evaluate on first matching element |
+| `$$eval` | `eval_on_selector_all` | Evaluate on all matching elements |
+
+#### Implementation Hierarchy
+
+Following Puppeteer's delegation pattern:
+
+```
+Page#eval_on_selector(_all)
+  ↓ delegates to
+Frame#eval_on_selector(_all)
+  ↓ delegates to
+ElementHandle#eval_on_selector(_all) (on document)
+  ↓ implementation
+  1. query_selector(_all) - Find element(s)
+  2. Validate results
+  3. evaluate() - Execute function
+  4. dispose - Clean up handles
+```
+
+### Implementation Details
+
+#### Page and Frame Methods
+
+```ruby
+# lib/puppeteer/bidi/page.rb
+def eval_on_selector(selector, page_function, *args)
+  main_frame.eval_on_selector(selector, page_function, *args)
+end
+
+# lib/puppeteer/bidi/frame.rb
+def eval_on_selector(selector, page_function, *args)
+  document.eval_on_selector(selector, page_function, *args)
+end
+```
+
+**Design rationale**: Page and Frame act as thin wrappers, delegating to the document element handle.
+
+#### ElementHandle#eval_on_selector
+
+```ruby
+def eval_on_selector(selector, page_function, *args)
+  assert_not_disposed
+
+  element_handle = query_selector(selector)
+  raise SelectorNotFoundError, selector unless element_handle
+
+  begin
+    element_handle.evaluate(page_function, *args)
+  ensure
+    element_handle.dispose
+  end
+end
+```
+
+**Key points**:
+- Throws `SelectorNotFoundError` if no element found (matches Puppeteer behavior)
+- Uses `begin/ensure` to guarantee handle disposal
+- Searches within element's subtree (not page-wide)
+
+#### ElementHandle#eval_on_selector_all
+
+```ruby
+def eval_on_selector_all(selector, page_function, *args)
+  assert_not_disposed
+
+  element_handles = query_selector_all(selector)
+
+  begin
+    # Create array handle in browser context
+    array_handle = @realm.call_function(
+      '(...elements) => elements',
+      false,
+      arguments: element_handles.map(&:remote_value)
+    )
+
+    array_js_handle = JSHandle.from(array_handle['result'], @realm)
+
+    begin
+      array_js_handle.evaluate(page_function, *args)
+    ensure
+      array_js_handle.dispose
+    end
+  ensure
+    element_handles.each(&:dispose)
+  end
+end
+```
+
+**Key points**:
+- Returns result for empty array without error (differs from `eval_on_selector`)
+- Creates array handle using spread operator trick: `(...elements) => elements`
+- Nested `ensure` blocks for proper resource cleanup
+- Disposes both individual element handles and array handle
+
+### Error Handling Differences
+
+| Method | Behavior when no elements found |
+|--------|--------------------------------|
+| `eval_on_selector` | Throws `SelectorNotFoundError` |
+| `eval_on_selector_all` | Returns evaluation result (e.g., `0` for `divs => divs.length`) |
+
+This matches Puppeteer's behavior:
+- `$eval`: Must find exactly one element
+- `$$eval`: Works with zero or more elements
+
+### Usage Examples
+
+```ruby
+# Basic usage
+page.set_content('<section id="test">Hello</section>')
+id = page.eval_on_selector('section', 'e => e.id')
+# => "test"
+
+# With arguments
+text = page.eval_on_selector('section', '(e, suffix) => e.textContent + suffix', '!')
+# => "Hello!"
+
+# ElementHandle arguments
+div = page.query_selector('div')
+result = page.eval_on_selector('section', '(e, div) => e.textContent + div.textContent', div)
+
+# eval_on_selector_all with multiple elements
+page.set_content('<div>A</div><div>B</div><div>C</div>')
+count = page.eval_on_selector_all('div', 'divs => divs.length')
+# => 3
+
+# Subtree search with ElementHandle
+tweet = page.query_selector('.tweet')
+likes = tweet.eval_on_selector('.like', 'node => node.innerText')
+# Only searches within .tweet element
+```
+
+### Test Coverage
+
+**Total**: 13 integration tests
+
+**Page.eval_on_selector** (4 tests):
+- Basic functionality (property access)
+- Argument passing
+- ElementHandle arguments
+- Error on missing selector
+
+**ElementHandle.eval_on_selector** (3 tests):
+- Basic functionality
+- Subtree isolation
+- Error on missing selector
+
+**Page.eval_on_selector_all** (4 tests):
+- Basic functionality (array length)
+- Extra arguments
+- ElementHandle arguments
+- Large element count (1001 elements)
+
+**ElementHandle.eval_on_selector_all** (2 tests):
+- Subtree retrieval
+- Empty result handling
+
+### Performance Considerations
+
+#### Handle Lifecycle
+
+- **eval_on_selector**: Creates 1 temporary handle per call
+- **eval_on_selector_all**: Creates N+1 handles (N elements + 1 array)
+- All handles automatically disposed after evaluation
+
+#### Large Element Sets
+
+Tested with 1001 elements without issues. The implementation efficiently:
+1. Queries all elements at once
+2. Creates single array handle
+3. Evaluates function in single round-trip
+4. Disposes all handles in parallel
+
+### Reference Implementation
+
+Based on Puppeteer's implementation:
+- [Page.$eval](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/api/Page.ts)
+- [Frame.$eval](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/api/Frame.ts)
+- [ElementHandle.$eval](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/api/ElementHandle.ts)
+- [Test specs](https://github.com/puppeteer/puppeteer/blob/main/test/src/queryselector.spec.ts)
+

data/CLAUDE/test_server_routes.md

@@ -0,0 +1,263 @@
+# Test Server Dynamic Routes
+
+This document describes the dynamic route handling functionality added to the test server infrastructure.
+
+## Overview
+
+The test server (`spec/support/test_server.rb`) has been extended to support dynamic route interception and request synchronization, matching Puppeteer's test server capabilities.
+
+## Features
+
+### 1. Dynamic Route Interception
+
+**Purpose**: Intercept specific routes and control the response timing/content.
+
+**API:**
+
+```ruby
+server.set_route(path) do |request, response|
+  # Control when/how to respond
+  response_holder[:response] = response
+end
+```
+
+**Usage Example:**
+
+```ruby
+it 'should intercept CSS loading' do
+  with_test_state do |page:, server:, **|
+    response_holder = {}
+
+    # Intercept CSS file to control when it loads
+    server.set_route('/one-style.css') do |_req, res|
+      response_holder[:response] = res
+    end
+
+    # Navigate to page (will wait for CSS)
+    page.goto("#{server.prefix}/one-style.html", wait: 'none')
+
+    # Do something while CSS is pending...
+
+    # Release the CSS response
+    response_holder[:response].finish
+  end
+end
+```
+
+### 2. Request Synchronization
+
+**Purpose**: Wait for a specific request to arrive at the server before proceeding.
+
+**API:**
+
+```ruby
+# Returns an Async task that resolves when request is received
+task = server.wait_for_request(path)
+task.wait  # Block until request arrives (with 5s timeout)
+```
+
+**Usage Example:**
+
+```ruby
+it 'should wait for image request' do
+  with_test_state do |page:, server:, **|
+    # Start navigation
+    page.goto("#{server.prefix}/page-with-image.html", wait: 'none')
+
+    # Wait for the image to be requested
+    begin
+      server.wait_for_request('/image.png').wait
+      puts "Image was requested!"
+    rescue Async::TimeoutError
+      puts "Image request never arrived"
+    end
+  end
+end
+```
+
+## Implementation Details
+
+### Async HTTP Server
+
+`TestServer::Server` now runs an `Async::HTTP::Server` inside a dedicated thread. The server keeps two shared hashes guarded by mutexes:
+
+- `@routes` maps request paths to custom handlers.
+- `@request_promises` stores waiters created via `wait_for_request`.
+
+Incoming requests execute the following flow:
+
+```ruby
+server = Async::HTTP::Server.for(endpoint) do |request|
+  if handler = lookup_route(request.path)
+    notify_request(request.path)
+    respond_with_handler(handler, request)
+  else
+    serve_static_asset(request)
+  end
+end
+```
+
+Static assets are served from `spec/assets`, while dynamic route handlers receive a lightweight wrapper (`RouteRequest`) exposing `path`, `headers`, `params`, and optional `body` accessors.
+
+### Response Writer
+
+Dynamic handlers interact with a `ResponseWriter` instance that buffers data until `finish` is invoked:
+
+```ruby
+server.set_route('/slow.css') do |_request, writer|
+  writer.add_header('content-type', 'text/css; charset=utf-8')
+  writer.write("body { background: red; }")
+  writer.finish
+end
+```
+
+The server task waits asynchronously for `writer.finish` before constructing the final `Protocol::HTTP::Response`. Handlers may capture the writer and complete it later from other tasks or threads, enabling Puppeteer-style resource gating.
+
+## Testing Navigation Events
+
+### Puppeteer Test Pattern
+
+A common Puppeteer test pattern tests the timing of `domcontentloaded` vs `load` events:
+
+```typescript
+it("should work with both domcontentloaded and load", async () => {
+  let response!: ServerResponse;
+  server.setRoute("/one-style.css", (_req, res) => {
+    return (response = res);
+  });
+
+  let bothFired = false;
+
+  const navigationPromise = page.goto(server.PREFIX + "/one-style.html");
+  const domContentLoadedPromise = page.waitForNavigation({
+    waitUntil: "domcontentloaded",
+  });
+  const loadFiredPromise = page
+    .waitForNavigation({ waitUntil: "load" })
+    .then(() => {
+      bothFired = true;
+    });
+
+  await server.waitForRequest("/one-style.css");
+  await domContentLoadedPromise;
+  expect(bothFired).toBe(false); // load hasn't fired yet
+
+  response.end(); // Release CSS
+  await loadFiredPromise; // Now load fires
+});
+```
+
+## Known Limitations and Challenges
+
+### 1. Navigation Timing Coordination
+
+**Challenge**: Testing `domcontentloaded` vs `load` timing requires careful coordination of:
+
+- Navigation start (must not wait for load)
+- Event listeners (must be registered before events fire)
+- Resource loading (must control when resources complete)
+
+**Issue**: In Ruby implementation, using `page.goto()` causes problems because:
+
+- `goto(url)` internally calls `navigate(url, wait: 'complete')` by default
+- This blocks until the `load` event fires
+- Cannot register `wait_for_navigation` listeners after navigation completes
+
+**Attempted Solutions:**
+
+1. **Using `wait: 'none'`**:
+
+   ```ruby
+   page.browsing_context.navigate(url, wait: 'none')
+   ```
+
+   - Doesn't block on navigation
+   - But bypasses high-level `Page` API
+   - Still has timing issues with event listener registration
+
+2. **Parallel Async tasks**:
+   ```ruby
+   navigation_task = Async { page.goto(url) }
+   dom_loaded_task = Async { page.wait_for_navigation(wait_until: 'domcontentloaded') }
+   load_task = Async { page.wait_for_navigation(wait_until: 'load') }
+   ```
+   - Race condition: `wait_for_navigation` might miss events if called after navigation completes
+   - Async task scheduling order is not guaranteed
+
+### 2. Request Promise Resolution
+
+**Issue**: `server.wait_for_request()` timeout errors are logged as warnings:
+
+```
+Async::TimeoutError: execution expired
+```
+
+This is expected behavior when the request arrives immediately (before `wait_for_request` is called), but the warning is noisy.
+
+**Current Workaround:**
+
+```ruby
+begin
+  server.wait_for_request('/one-style.css').wait
+rescue Async::TimeoutError
+  # Request might have already arrived - ignore
+end
+```
+
+### 3. Response Writer Semantics
+
+**Limitation**: The custom `ResponseWriter` currently buffers the entire body before sending it back through `Protocol::HTTP::Response`. True streaming responses are not yet implemented, so large payloads are held in memory until `finish` is called. Handlers should keep payloads small, or extend the writer to stream chunks if needed in future work.
+
+## Future Improvements
+
+### 1. Navigation API Enhancement
+
+Consider adding a `Page.navigate` method that exposes the `wait` parameter:
+
+```ruby
+# High-level API with wait control
+page.navigate(url, wait: 'none')  # Start navigation without waiting
+page.navigate(url, wait: 'interactive')  # Wait for domcontentloaded
+page.navigate(url, wait: 'complete')  # Wait for load (default)
+```
+
+### 2. Event Listener Preregistration
+
+Add API to register navigation listeners before starting navigation:
+
+```ruby
+page.with_navigation_listeners do |listeners|
+  listeners.on_dom_content_loaded { puts "DOM ready" }
+  listeners.on_load { puts "Page loaded" }
+
+  page.goto(url)  # Listeners already registered
+end
+```
+
+### 3. Test Server Request Queue
+
+Store all incoming requests in a queue for post-facto checking:
+
+```ruby
+# Record all requests
+server.enable_request_recording
+
+page.goto(url)
+
+# Check what was requested
+requests = server.recorded_requests
+expect(requests.map(&:path)).to include('/one-style.css')
+```
+
+## Related Files
+
+- `spec/support/test_server.rb` - Test server implementation
+- `spec/integration/navigation_spec.rb` - Navigation tests using dynamic routes
+- `spec/assets/one-style.html` - Test asset with external CSS
+- `spec/assets/one-style.css` - CSS file for testing resource loading
+
+## References
+
+- [Puppeteer test server](https://github.com/puppeteer/puppeteer/blob/main/test/src/server/index.ts)
+- [Puppeteer navigation tests](https://github.com/puppeteer/puppeteer/blob/main/test/src/navigation.spec.ts)
+- [async-http server guide](https://socketry.github.io/async-http/guides/getting-started/index.html#making-a-server)

data/CLAUDE/testing_strategy.md

@@ -0,0 +1,236 @@
+# Testing Strategy and Performance Optimization
+
+This document covers integration test organization, performance optimization strategies, golden image testing, and debugging techniques.
+
+
+#### Integration Tests Organization
+
+```
+spec/
+├── unit/                    # Fast unit tests (future)
+├── integration/             # Browser automation tests
+│   ├── examples/           # Example-based tests
+│   │   └── screenshot_spec.rb
+│   └── screenshot_spec.rb  # Feature test suites
+├── assets/                 # Test HTML/CSS/JS files
+│   ├── grid.html
+│   ├── scrollbar.html
+│   ├── empty.html
+│   └── digits/*.png
+├── golden-firefox/         # Reference images
+│   └── screenshot-*.png
+└── support/               # Test utilities
+    ├── test_server.rb
+    └── golden_comparator.rb
+```
+
+#### Implemented Screenshot Tests
+
+All 12 tests ported from [Puppeteer's screenshot.spec.ts](https://github.com/puppeteer/puppeteer/blob/main/test/src/screenshot.spec.ts):
+
+1. **should work** - Basic screenshot functionality
+2. **should clip rect** - Clipping specific region
+3. **should get screenshot bigger than the viewport** - Offscreen clip with captureBeyondViewport
+4. **should clip bigger than the viewport without "captureBeyondViewport"** - Viewport coordinate transformation
+5. **should run in parallel** - Thread-safe parallel screenshots on single page
+6. **should take fullPage screenshots** - Full page with document origin
+7. **should take fullPage screenshots without captureBeyondViewport** - Full page with viewport resize
+8. **should run in parallel in multiple pages** - Concurrent screenshots across multiple pages
+9. **should work with odd clip size on Retina displays** - Odd pixel dimensions (11x11)
+10. **should return base64** - Base64 encoding verification
+11. **should take fullPage screenshots when defaultViewport is null** - No explicit viewport
+12. **should restore to original viewport size** - Viewport restoration after fullPage
+
+Run tests:
+```bash
+bundle exec rspec spec/integration/screenshot_spec.rb
+# Expected: 12 examples, 0 failures (completes in ~8 seconds with optimized spec_helper)
+```
+
+#### Test Performance Optimization
+
+**Critical**: Integration tests are ~19x faster with browser reuse strategy.
+
+##### Before Optimization (Per-test Browser Launch)
+```ruby
+def with_test_state(**options)
+  server = TestServer::Server.new
+  server.start
+
+  with_browser(**options) do |browser|  # New browser per test!
+    context = browser.default_browser_context
+    page = browser.new_page
+    yield(page: page, server: server, browser: browser, context: context)
+  end
+ensure
+  server.stop
+end
+```
+
+**Performance**: ~195 seconds for 35 tests (browser launch overhead × 35)
+
+##### After Optimization (Shared Browser)
+```ruby
+# In spec_helper.rb
+config.before(:suite) do
+  if RSpec.configuration.files_to_run.any? { |f| f.include?('spec/integration') }
+    $shared_browser = Puppeteer::Bidi.launch(headless: headless_mode?)
+    $shared_test_server = TestServer::Server.new
+    $shared_test_server.start
+  end
+end
+
+def with_test_state(**options)
+  if $shared_browser && options.empty?
+    # Create new page (tab) per test
+    page = $shared_browser.new_page
+    context = $shared_browser.default_browser_context
+
+    begin
+      yield(page: page, server: $shared_test_server, browser: $shared_browser, context: context)
+    ensure
+      page.close unless page.closed?  # Clean up tab
+    end
+  else
+    # Fall back to per-test browser for custom options
+  end
+end
+```
+
+**Performance**: ~10 seconds for 35 tests (1 browser launch + 35 tab creations)
+
+##### Performance Results
+
+| Test Suite | Before | After | Improvement |
+|------------|--------|-------|-------------|
+| **evaluation_spec (23 tests)** | 127s | **7.17s** | **17.7x faster** |
+| **screenshot_spec (12 tests)** | 68s | **8.47s** | **8.0x faster** |
+| **Combined (35 tests)** | 195s | **10.33s** | **18.9x faster** 🚀 |
+
+**Key Benefits**:
+- Browser launch only once per suite
+- Each test gets fresh page (tab) for isolation
+- Cleanup handled automatically
+- Backward compatible (custom options fall back to per-test browser)
+
+#### Environment Variables
+
+```bash
+HEADLESS=false  # Run browser in non-headless mode for debugging
+```
+
+### Debugging Techniques
+
+#### 1. Save Screenshots for Inspection
+
+```ruby
+# In golden_comparator.rb
+def save_screenshot(screenshot_base64, filename)
+  output_dir = File.join(__dir__, '../output')
+  FileUtils.mkdir_p(output_dir)
+  File.binwrite(File.join(output_dir, filename),
+                Base64.decode64(screenshot_base64))
+end
+```
+
+#### 2. Compare Images Pixel-by-Pixel
+
+```ruby
+cat > /tmp/compare.rb << 'EOF'
+require 'chunky_png'
+
+golden = ChunkyPNG::Image.from_file('spec/golden-firefox/screenshot.png')
+actual = ChunkyPNG::Image.from_file('spec/output/debug.png')
+
+diff_count = 0
+(0...golden.height).each do |y|
+  (0...golden.width).each do |x|
+    if golden[x, y] != actual[x, y]
+      diff_count += 1
+      puts "Diff at (#{x}, #{y})" if diff_count <= 10
+    end
+  end
+end
+puts "Total: #{diff_count} pixels differ"
+EOF
+ruby /tmp/compare.rb
+```
+
+#### 3. Debug BiDi Responses
+
+```ruby
+# Temporarily add debugging
+result = @browsing_context.default_realm.evaluate(script, true)
+puts "BiDi result: #{result.inspect}"
+deserialize_result(result)
+```
+
+### Common Pitfalls and Solutions
+
+#### 1. BiDi Protocol Differences
+
+**Problem:** BiDi `origin` parameter behavior differs from expectations
+
+**Solution:** Consult BiDi spec and test both `'document'` and `'viewport'` origins
+
+```ruby
+# document: Absolute coordinates in full page
+# viewport: Relative to current viewport
+options[:origin] = capture_beyond_viewport ? 'document' : 'viewport'
+```
+
+#### 2. Image Comparison Failures
+
+**Problem:** Golden images don't match exactly (1-2 pixel differences)
+
+**Solution:** Implement tolerance in comparison
+
+```ruby
+# Allow small rendering differences (±1 RGB per channel)
+compare_with_golden(screenshot, 'golden.png', pixel_threshold: 1)
+```
+
+#### 3. Viewport State Management
+
+**Problem:** Viewport not restored after fullPage screenshot
+
+**Solution:** Use `ensure` block
+
+```ruby
+begin
+  set_viewport(full_page_dimensions)
+  screenshot = capture_screenshot(...)
+ensure
+  set_viewport(original_viewport) if original_viewport
+end
+```
+
+#### 4. Thread Safety
+
+**Problem:** Parallel screenshots cause race conditions
+
+**Solution:** BiDi protocol handles this naturally - test with threads
+
+```ruby
+threads = (0...3).map do |i|
+  Thread.new { page.screenshot(clip: {...}) }
+end
+screenshots = threads.map(&:value)
+```
+
+### Documentation References
+
+**Essential reading for implementation:**
+
+1. **WebDriver BiDi Spec**: https://w3c.github.io/webdriver-bidi/
+2. **Puppeteer Source**: https://github.com/puppeteer/puppeteer
+3. **Puppeteer BiDi Tests**: https://github.com/puppeteer/puppeteer/tree/main/test/src
+4. **Firefox BiDi Impl**: Check Firefox implementation notes for quirks
+
+**Reference implementation workflow:**
+1. Find corresponding Puppeteer test in `test/src/`
+2. Read TypeScript implementation in `packages/puppeteer-core/src/`
+3. Check BiDi spec for protocol details
+4. Implement Ruby version maintaining same logic
+5. Download golden images and verify pixel-perfect match (with tolerance)
+