puppeteer-bidi 0.0.3.beta1 → 0.0.3.beta2

data/CLAUDE/test_server_routes.md

@@ -1,263 +0,0 @@
-# Test Server Dynamic Routes
-
-This document describes the dynamic route handling functionality added to the test server infrastructure.
-
-## Overview
-
-The test server (`spec/support/test_server.rb`) has been extended to support dynamic route interception and request synchronization, matching Puppeteer's test server capabilities.
-
-## Features
-
-### 1. Dynamic Route Interception
-
-**Purpose**: Intercept specific routes and control the response timing/content.
-
-**API:**
-
-```ruby
-server.set_route(path) do |request, response|
-  # Control when/how to respond
-  response_holder[:response] = response
-end
-```
-
-**Usage Example:**
-
-```ruby
-it 'should intercept CSS loading' do
-  with_test_state do |page:, server:, **|
-    response_holder = {}
-
-    # Intercept CSS file to control when it loads
-    server.set_route('/one-style.css') do |_req, res|
-      response_holder[:response] = res
-    end
-
-    # Navigate to page (will wait for CSS)
-    page.goto("#{server.prefix}/one-style.html", wait: 'none')
-
-    # Do something while CSS is pending...
-
-    # Release the CSS response
-    response_holder[:response].finish
-  end
-end
-```
-
-### 2. Request Synchronization
-
-**Purpose**: Wait for a specific request to arrive at the server before proceeding.
-
-**API:**
-
-```ruby
-# Returns an Async task that resolves when request is received
-task = server.wait_for_request(path)
-task.wait  # Block until request arrives (with 5s timeout)
-```
-
-**Usage Example:**
-
-```ruby
-it 'should wait for image request' do
-  with_test_state do |page:, server:, **|
-    # Start navigation
-    page.goto("#{server.prefix}/page-with-image.html", wait: 'none')
-
-    # Wait for the image to be requested
-    begin
-      server.wait_for_request('/image.png').wait
-      puts "Image was requested!"
-    rescue Async::TimeoutError
-      puts "Image request never arrived"
-    end
-  end
-end
-```
-
-## Implementation Details
-
-### Async HTTP Server
-
-`TestServer::Server` now runs an `Async::HTTP::Server` inside a dedicated thread. The server keeps two shared hashes guarded by mutexes:
-
-- `@routes` maps request paths to custom handlers.
-- `@request_promises` stores waiters created via `wait_for_request`.
-
-Incoming requests execute the following flow:
-
-```ruby
-server = Async::HTTP::Server.for(endpoint) do |request|
-  if handler = lookup_route(request.path)
-    notify_request(request.path)
-    respond_with_handler(handler, request)
-  else
-    serve_static_asset(request)
-  end
-end
-```
-
-Static assets are served from `spec/assets`, while dynamic route handlers receive a lightweight wrapper (`RouteRequest`) exposing `path`, `headers`, `params`, and optional `body` accessors.
-
-### Response Writer
-
-Dynamic handlers interact with a `ResponseWriter` instance that buffers data until `finish` is invoked:
-
-```ruby
-server.set_route('/slow.css') do |_request, writer|
-  writer.add_header('content-type', 'text/css; charset=utf-8')
-  writer.write("body { background: red; }")
-  writer.finish
-end
-```
-
-The server task waits asynchronously for `writer.finish` before constructing the final `Protocol::HTTP::Response`. Handlers may capture the writer and complete it later from other tasks or threads, enabling Puppeteer-style resource gating.
-
-## Testing Navigation Events
-
-### Puppeteer Test Pattern
-
-A common Puppeteer test pattern tests the timing of `domcontentloaded` vs `load` events:
-
-```typescript
-it("should work with both domcontentloaded and load", async () => {
-  let response!: ServerResponse;
-  server.setRoute("/one-style.css", (_req, res) => {
-    return (response = res);
-  });
-
-  let bothFired = false;
-
-  const navigationPromise = page.goto(server.PREFIX + "/one-style.html");
-  const domContentLoadedPromise = page.waitForNavigation({
-    waitUntil: "domcontentloaded",
-  });
-  const loadFiredPromise = page
-    .waitForNavigation({ waitUntil: "load" })
-    .then(() => {
-      bothFired = true;
-    });
-
-  await server.waitForRequest("/one-style.css");
-  await domContentLoadedPromise;
-  expect(bothFired).toBe(false); // load hasn't fired yet
-
-  response.end(); // Release CSS
-  await loadFiredPromise; // Now load fires
-});
-```
-
-## Known Limitations and Challenges
-
-### 1. Navigation Timing Coordination
-
-**Challenge**: Testing `domcontentloaded` vs `load` timing requires careful coordination of:
-
-- Navigation start (must not wait for load)
-- Event listeners (must be registered before events fire)
-- Resource loading (must control when resources complete)
-
-**Issue**: In Ruby implementation, using `page.goto()` causes problems because:
-
-- `goto(url)` internally calls `navigate(url, wait: 'complete')` by default
-- This blocks until the `load` event fires
-- Cannot register `wait_for_navigation` listeners after navigation completes
-
-**Attempted Solutions:**
-
-1. **Using `wait: 'none'`**:
-
-   ```ruby
-   page.browsing_context.navigate(url, wait: 'none')
-   ```
-
-   - Doesn't block on navigation
-   - But bypasses high-level `Page` API
-   - Still has timing issues with event listener registration
-
-2. **Parallel Async tasks**:
-   ```ruby
-   navigation_task = Async { page.goto(url) }
-   dom_loaded_task = Async { page.wait_for_navigation(wait_until: 'domcontentloaded') }
-   load_task = Async { page.wait_for_navigation(wait_until: 'load') }
-   ```
-   - Race condition: `wait_for_navigation` might miss events if called after navigation completes
-   - Async task scheduling order is not guaranteed
-
-### 2. Request Promise Resolution
-
-**Issue**: `server.wait_for_request()` timeout errors are logged as warnings:
-
-```
-Async::TimeoutError: execution expired
-```
-
-This is expected behavior when the request arrives immediately (before `wait_for_request` is called), but the warning is noisy.
-
-**Current Workaround:**
-
-```ruby
-begin
-  server.wait_for_request('/one-style.css').wait
-rescue Async::TimeoutError
-  # Request might have already arrived - ignore
-end
-```
-
-### 3. Response Writer Semantics
-
-**Limitation**: The custom `ResponseWriter` currently buffers the entire body before sending it back through `Protocol::HTTP::Response`. True streaming responses are not yet implemented, so large payloads are held in memory until `finish` is called. Handlers should keep payloads small, or extend the writer to stream chunks if needed in future work.
-
-## Future Improvements
-
-### 1. Navigation API Enhancement
-
-Consider adding a `Page.navigate` method that exposes the `wait` parameter:
-
-```ruby
-# High-level API with wait control
-page.navigate(url, wait: 'none')  # Start navigation without waiting
-page.navigate(url, wait: 'interactive')  # Wait for domcontentloaded
-page.navigate(url, wait: 'complete')  # Wait for load (default)
-```
-
-### 2. Event Listener Preregistration
-
-Add API to register navigation listeners before starting navigation:
-
-```ruby
-page.with_navigation_listeners do |listeners|
-  listeners.on_dom_content_loaded { puts "DOM ready" }
-  listeners.on_load { puts "Page loaded" }
-
-  page.goto(url)  # Listeners already registered
-end
-```
-
-### 3. Test Server Request Queue
-
-Store all incoming requests in a queue for post-facto checking:
-
-```ruby
-# Record all requests
-server.enable_request_recording
-
-page.goto(url)
-
-# Check what was requested
-requests = server.recorded_requests
-expect(requests.map(&:path)).to include('/one-style.css')
-```
-
-## Related Files
-
-- `spec/support/test_server.rb` - Test server implementation
-- `spec/integration/navigation_spec.rb` - Navigation tests using dynamic routes
-- `spec/assets/one-style.html` - Test asset with external CSS
-- `spec/assets/one-style.css` - CSS file for testing resource loading
-
-## References
-
-- [Puppeteer test server](https://github.com/puppeteer/puppeteer/blob/main/test/src/server/index.ts)
-- [Puppeteer navigation tests](https://github.com/puppeteer/puppeteer/blob/main/test/src/navigation.spec.ts)
-- [async-http server guide](https://socketry.github.io/async-http/guides/getting-started/index.html#making-a-server)

data/CLAUDE/testing_strategy.md

@@ -1,236 +0,0 @@
-# Testing Strategy and Performance Optimization
-
-This document covers integration test organization, performance optimization strategies, golden image testing, and debugging techniques.
-
-
-#### Integration Tests Organization
-
-```
-spec/
-├── unit/                    # Fast unit tests (future)
-├── integration/             # Browser automation tests
-│   ├── examples/           # Example-based tests
-│   │   └── screenshot_spec.rb
-│   └── screenshot_spec.rb  # Feature test suites
-├── assets/                 # Test HTML/CSS/JS files
-│   ├── grid.html
-│   ├── scrollbar.html
-│   ├── empty.html
-│   └── digits/*.png
-├── golden-firefox/         # Reference images
-│   └── screenshot-*.png
-└── support/               # Test utilities
-    ├── test_server.rb
-    └── golden_comparator.rb
-```
-
-#### Implemented Screenshot Tests
-
-All 12 tests ported from [Puppeteer's screenshot.spec.ts](https://github.com/puppeteer/puppeteer/blob/main/test/src/screenshot.spec.ts):
-
-1. **should work** - Basic screenshot functionality
-2. **should clip rect** - Clipping specific region
-3. **should get screenshot bigger than the viewport** - Offscreen clip with captureBeyondViewport
-4. **should clip bigger than the viewport without "captureBeyondViewport"** - Viewport coordinate transformation
-5. **should run in parallel** - Thread-safe parallel screenshots on single page
-6. **should take fullPage screenshots** - Full page with document origin
-7. **should take fullPage screenshots without captureBeyondViewport** - Full page with viewport resize
-8. **should run in parallel in multiple pages** - Concurrent screenshots across multiple pages
-9. **should work with odd clip size on Retina displays** - Odd pixel dimensions (11x11)
-10. **should return base64** - Base64 encoding verification
-11. **should take fullPage screenshots when defaultViewport is null** - No explicit viewport
-12. **should restore to original viewport size** - Viewport restoration after fullPage
-
-Run tests:
-```bash
-bundle exec rspec spec/integration/screenshot_spec.rb
-# Expected: 12 examples, 0 failures (completes in ~8 seconds with optimized spec_helper)
-```
-
-#### Test Performance Optimization
-
-**Critical**: Integration tests are ~19x faster with browser reuse strategy.
-
-##### Before Optimization (Per-test Browser Launch)
-```ruby
-def with_test_state(**options)
-  server = TestServer::Server.new
-  server.start
-
-  with_browser(**options) do |browser|  # New browser per test!
-    context = browser.default_browser_context
-    page = browser.new_page
-    yield(page: page, server: server, browser: browser, context: context)
-  end
-ensure
-  server.stop
-end
-```
-
-**Performance**: ~195 seconds for 35 tests (browser launch overhead × 35)
-
-##### After Optimization (Shared Browser)
-```ruby
-# In spec_helper.rb
-config.before(:suite) do
-  if RSpec.configuration.files_to_run.any? { |f| f.include?('spec/integration') }
-    $shared_browser = Puppeteer::Bidi.launch_browser_instance(headless: headless_mode?)
-    $shared_test_server = TestServer::Server.new
-    $shared_test_server.start
-  end
-end
-
-def with_test_state(**options)
-  if $shared_browser && options.empty?
-    # Create new page (tab) per test
-    page = $shared_browser.new_page
-    context = $shared_browser.default_browser_context
-
-    begin
-      yield(page: page, server: $shared_test_server, browser: $shared_browser, context: context)
-    ensure
-      page.close unless page.closed?  # Clean up tab
-    end
-  else
-    # Fall back to per-test browser for custom options
-  end
-end
-```
-
-**Performance**: ~10 seconds for 35 tests (1 browser launch + 35 tab creations)
-
-##### Performance Results
-
-| Test Suite | Before | After | Improvement |
-|------------|--------|-------|-------------|
-| **evaluation_spec (23 tests)** | 127s | **7.17s** | **17.7x faster** |
-| **screenshot_spec (12 tests)** | 68s | **8.47s** | **8.0x faster** |
-| **Combined (35 tests)** | 195s | **10.33s** | **18.9x faster** 🚀 |
-
-**Key Benefits**:
-- Browser launch only once per suite
-- Each test gets fresh page (tab) for isolation
-- Cleanup handled automatically
-- Backward compatible (custom options fall back to per-test browser)
-
-#### Environment Variables
-
-```bash
-HEADLESS=false  # Run browser in non-headless mode for debugging
-```
-
-### Debugging Techniques
-
-#### 1. Save Screenshots for Inspection
-
-```ruby
-# In golden_comparator.rb
-def save_screenshot(screenshot_base64, filename)
-  output_dir = File.join(__dir__, '../output')
-  FileUtils.mkdir_p(output_dir)
-  File.binwrite(File.join(output_dir, filename),
-                Base64.decode64(screenshot_base64))
-end
-```
-
-#### 2. Compare Images Pixel-by-Pixel
-
-```ruby
-cat > /tmp/compare.rb << 'EOF'
-require 'chunky_png'
-
-golden = ChunkyPNG::Image.from_file('spec/golden-firefox/screenshot.png')
-actual = ChunkyPNG::Image.from_file('spec/output/debug.png')
-
-diff_count = 0
-(0...golden.height).each do |y|
-  (0...golden.width).each do |x|
-    if golden[x, y] != actual[x, y]
-      diff_count += 1
-      puts "Diff at (#{x}, #{y})" if diff_count <= 10
-    end
-  end
-end
-puts "Total: #{diff_count} pixels differ"
-EOF
-ruby /tmp/compare.rb
-```
-
-#### 3. Debug BiDi Responses
-
-```ruby
-# Temporarily add debugging
-result = @browsing_context.default_realm.evaluate(script, true)
-puts "BiDi result: #{result.inspect}"
-deserialize_result(result)
-```
-
-### Common Pitfalls and Solutions
-
-#### 1. BiDi Protocol Differences
-
-**Problem:** BiDi `origin` parameter behavior differs from expectations
-
-**Solution:** Consult BiDi spec and test both `'document'` and `'viewport'` origins
-
-```ruby
-# document: Absolute coordinates in full page
-# viewport: Relative to current viewport
-options[:origin] = capture_beyond_viewport ? 'document' : 'viewport'
-```
-
-#### 2. Image Comparison Failures
-
-**Problem:** Golden images don't match exactly (1-2 pixel differences)
-
-**Solution:** Implement tolerance in comparison
-
-```ruby
-# Allow small rendering differences (±1 RGB per channel)
-compare_with_golden(screenshot, 'golden.png', pixel_threshold: 1)
-```
-
-#### 3. Viewport State Management
-
-**Problem:** Viewport not restored after fullPage screenshot
-
-**Solution:** Use `ensure` block
-
-```ruby
-begin
-  set_viewport(full_page_dimensions)
-  screenshot = capture_screenshot(...)
-ensure
-  set_viewport(original_viewport) if original_viewport
-end
-```
-
-#### 4. Thread Safety
-
-**Problem:** Parallel screenshots cause race conditions
-
-**Solution:** BiDi protocol handles this naturally - test with threads
-
-```ruby
-threads = (0...3).map do |i|
-  Thread.new { page.screenshot(clip: {...}) }
-end
-screenshots = threads.map(&:value)
-```
-
-### Documentation References
-
-**Essential reading for implementation:**
-
-1. **WebDriver BiDi Spec**: https://w3c.github.io/webdriver-bidi/
-2. **Puppeteer Source**: https://github.com/puppeteer/puppeteer
-3. **Puppeteer BiDi Tests**: https://github.com/puppeteer/puppeteer/tree/main/test/src
-4. **Firefox BiDi Impl**: Check Firefox implementation notes for quirks
-
-**Reference implementation workflow:**
-1. Find corresponding Puppeteer test in `test/src/`
-2. Read TypeScript implementation in `packages/puppeteer-core/src/`
-3. Check BiDi spec for protocol details
-4. Implement Ruby version maintaining same logic
-5. Download golden images and verify pixel-perfect match (with tolerance)
-

data/CLAUDE/two_layer_architecture.md

@@ -1,180 +0,0 @@
-# Two-Layer Async Architecture
-
-This codebase implements a two-layer architecture to separate async complexity from user-facing APIs.
-
-## Architecture Overview
-
-```
-┌─────────────────────────────────────────────────────────┐
-│  Upper Layer (Puppeteer::Bidi)                          │
-│  - User-facing, synchronous API                         │
-│  - Calls .wait internally on Core layer methods         │
-│  - Examples: Page, Frame, JSHandle, ElementHandle       │
-├─────────────────────────────────────────────────────────┤
-│  Core Layer (Puppeteer::Bidi::Core)                     │
-│  - Returns Async::Task for all async operations         │
-│  - Uses async_send_command internally                   │
-│  - Examples: Session, BrowsingContext, Realm            │
-└─────────────────────────────────────────────────────────┘
-```
-
-## Design Principles
-
-1. **Core Layer (Puppeteer::Bidi::Core)**:
-   - All methods that communicate with BiDi protocol return `Async::Task`
-   - Uses `session.async_send_command` (not `send_command`)
-   - Methods are explicitly async and composable
-   - Examples: `BrowsingContext#navigate`, `Realm#call_function`
-
-2. **Upper Layer (Puppeteer::Bidi)**:
-   - All methods call `.wait` on Core layer async operations
-   - Provides synchronous, blocking API for users
-   - Users never see `Async::Task` directly
-   - Examples: `Page#goto`, `Frame#evaluate`, `JSHandle#get_property`
-
-## Implementation Patterns
-
-### Core Layer Pattern
-
-```ruby
-# lib/puppeteer/bidi/core/browsing_context.rb
-def navigate(url, wait: nil)
-  Async do
-    raise BrowsingContextClosedError, @reason if closed?
-    params = { context: @id, url: url }
-    params[:wait] = wait if wait
-    result = session.async_send_command('browsingContext.navigate', params).wait
-    result
-  end
-end
-
-def perform_actions(actions)
-  raise BrowsingContextClosedError, @reason if closed?
-  session.async_send_command('input.performActions', {
-    context: @id,
-    actions: actions
-  })
-end
-```
-
-**Key points:**
-- Returns `Async::Task` (implicitly from `Async do` block or explicitly from `async_send_command`)
-- Users of Core layer must call `.wait` to get results
-
-### Upper Layer Pattern
-
-```ruby
-# lib/puppeteer/bidi/frame.rb
-def goto(url, wait_until: 'load', timeout: 30000)
-  response = wait_for_navigation(timeout: timeout, wait_until: wait_until) do
-    @browsing_context.navigate(url, wait: 'interactive').wait  # .wait call
-  end
-  HTTPResponse.new(url: @browsing_context.url, status: 200)
-end
-
-# lib/puppeteer/bidi/keyboard.rb
-def perform_actions(action_list)
-  @browsing_context.perform_actions([
-    {
-      type: 'key',
-      id: 'default keyboard',
-      actions: action_list
-    }
-  ]).wait  # .wait call
-end
-```
-
-**Key points:**
-- Always calls `.wait` on Core layer methods
-- Returns plain Ruby objects (String, Hash, etc.), not Async::Task
-- User-facing API is synchronous
-
-## Common Mistakes and How to Fix Them
-
-### Mistake 1: Forgetting .wait on Core Layer Methods
-
-```ruby
-# WRONG: Missing .wait
-def query_selector(selector)
-  result = @realm.call_function(...)
-  if result['type'] == 'exception'  # Error: undefined method '[]' for Async::Task
-    # ...
-  end
-end
-
-# CORRECT: Add .wait
-def query_selector(selector)
-  result = @realm.call_function(...).wait  # Add .wait here
-  if result['type'] == 'exception'
-    # ...
-  end
-end
-```
-
-### Mistake 2: Using send_command Instead of async_send_command in Core Layer
-
-```ruby
-# WRONG: Using send_command (doesn't exist)
-def perform_actions(actions)
-  session.send_command('input.performActions', {...})  # Error: undefined method
-end
-
-# CORRECT: Use async_send_command
-def perform_actions(actions)
-  session.async_send_command('input.performActions', {...})
-end
-```
-
-### Mistake 3: Not Calling .wait on All Core Methods
-
-```ruby
-# WRONG: Multiple Core calls, only one .wait
-def get_properties
-  result = @realm.call_function(...).wait  # OK
-  props_result = @realm.call_function(...)  # Missing .wait!
-  if props_result['type'] == 'exception'  # Error
-    # ...
-  end
-end
-
-# CORRECT: Add .wait to all Core calls
-def get_properties
-  result = @realm.call_function(...).wait
-  props_result = @realm.call_function(...).wait  # Add .wait
-  if props_result['type'] == 'exception'
-    # ...
-  end
-end
-```
-
-## Checklist for Adding New Methods
-
-When adding new methods to the Upper Layer:
-
-1. Identify all calls to Core layer methods
-2. Add `.wait` to each Core layer method call
-3. Verify the method returns plain Ruby objects, not Async::Task
-4. Test with integration specs
-
-When adding new methods to the Core Layer:
-
-1. Use `session.async_send_command` (not `send_command`)
-2. Wrap in `Async do ... end` if needed
-3. Return `Async::Task` (don't call .wait)
-4. Document that callers must call .wait
-
-## Files Modified for Async Architecture
-
-**Core Layer:**
-- `lib/puppeteer/bidi/core/session.rb` - Changed `send_command` → `async_send_command`
-- `lib/puppeteer/bidi/core/browsing_context.rb` - All methods use `async_send_command`
-- `lib/puppeteer/bidi/core/realm.rb` - Methods return `Async::Task`
-
-**Upper Layer:**
-- `lib/puppeteer/bidi/realm.rb` - Added `.wait` to `execute_with_core`, `call_function`
-- `lib/puppeteer/bidi/frame.rb` - Added `.wait` to `goto`
-- `lib/puppeteer/bidi/js_handle.rb` - Added `.wait` to `dispose`, `get_property`, `get_properties`, `as_element`
-- `lib/puppeteer/bidi/element_handle.rb` - Added `.wait` to `query_selector_all`, `eval_on_selector_all`
-- `lib/puppeteer/bidi/keyboard.rb` - Added `.wait` to `perform_actions`
-- `lib/puppeteer/bidi/mouse.rb` - Added `.wait` to `perform_actions`
-- `lib/puppeteer/bidi/page.rb` - Added `.wait` to `capture_screenshot`, `close`, `set_viewport`, `set_javascript_enabled`