puppeteer-bidi 0.0.1.beta1

data/CLAUDE/frame_architecture.md

@@ -0,0 +1,346 @@
+# Frame Architecture Implementation
+
+## Overview
+
+This document details the Frame architecture implementation following Puppeteer's parent-based design pattern.
+
+## Architecture Change
+
+### Before (Incorrect)
+
+```ruby
+class Frame
+  def initialize(browsing_context, page = nil)
+    @browsing_context = browsing_context
+    @page = page
+  end
+end
+
+# Page creates frame
+Frame.new(@browsing_context, self)
+```
+
+**Problem**: Frame directly stores reference to Page, doesn't support nested frames (iframe).
+
+### After (Correct - Following Puppeteer)
+
+```ruby
+class Frame
+  def initialize(parent, browsing_context)
+    @parent = parent  # Page or Frame
+    @browsing_context = browsing_context
+  end
+
+  def page
+    @parent.is_a?(Page) ? @parent : @parent.page
+  end
+
+  def parent_frame
+    @parent.is_a?(Frame) ? @parent : nil
+  end
+end
+
+# Page creates frame
+Frame.new(self, @browsing_context)
+```
+
+**Benefits**:
+- Supports nested frames (iframe within iframe)
+- Matches Puppeteer's TypeScript implementation
+- Enables recursive page traversal
+- Simplifies parent_frame implementation
+
+## Reference Implementation
+
+Based on [Puppeteer's Frame.ts](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/bidi/Frame.ts):
+
+```typescript
+export class BidiFrame extends Frame {
+  #parent: BidiPage | BidiFrame;
+  #browsingContext: BrowsingContext;
+
+  constructor(
+    parent: BidiPage | BidiFrame,
+    browsingContext: BrowsingContext,
+  ) {
+    super();
+    this.#parent = parent;
+    this.#browsingContext = browsingContext;
+  }
+
+  override get page(): BidiPage {
+    let parent = this.#parent;
+    while (parent instanceof BidiFrame) {
+      parent = parent.#parent;
+    }
+    return parent;
+  }
+
+  override get parentFrame(): BidiFrame | null {
+    if (this.#parent instanceof BidiFrame) {
+      return this.#parent;
+    }
+    return null;
+  }
+}
+```
+
+## Implementation Details
+
+### Constructor Signature
+
+**Critical**: The first parameter is `parent` (Page or Frame), not `page`:
+
+```ruby
+def initialize(parent, browsing_context)
+  @parent = parent
+  @browsing_context = browsing_context
+end
+```
+
+### Page Traversal
+
+Recursive implementation using ternary operator:
+
+```ruby
+def page
+  @parent.is_a?(Page) ? @parent : @parent.page
+end
+```
+
+This is simpler than a while loop and matches Puppeteer's logic flow.
+
+### Parent Frame Access
+
+```ruby
+def parent_frame
+  @parent.is_a?(Frame) ? @parent : nil
+end
+```
+
+Returns:
+- `Frame` instance if this is a child frame
+- `nil` if this is a main frame (parent is Page)
+
+## Usage Examples
+
+### Main Frame
+
+```ruby
+page = browser.new_page
+main_frame = page.main_frame
+
+main_frame.page         # => page
+main_frame.parent_frame # => nil
+```
+
+### Nested Frames (Future)
+
+```ruby
+# When iframe support is added:
+iframe = main_frame.child_frames.first
+
+iframe.page         # => page (traverses up to Page)
+iframe.parent_frame # => main_frame
+```
+
+## Testing
+
+All 108 integration tests pass with this architecture:
+
+```bash
+bundle exec rspec spec/integration/
+# 108 examples, 0 failures, 4 pending
+```
+
+## Key Takeaways
+
+1. **Follow Puppeteer's constructor signature exactly** - `(parent, browsing_context)` not `(browsing_context, page)`
+2. **Use ternary operator for simplicity** - `@parent.is_a?(Page) ? @parent : @parent.page`
+3. **Enables future iframe support** - Architecture supports nested frame trees
+4. **Remove redundant attr_reader** - No need for `attr_reader :parent` when using private instance variable
+
+## Frame Events
+
+### Overview
+
+Frame lifecycle events are emitted on the Page object, following Puppeteer's pattern:
+
+- `:frameattached` - Fired when a new child frame is created
+- `:framedetached` - Fired when a frame's browsing context is closed
+- `:framenavigated` - Fired on DOMContentLoaded or fragment navigation
+
+### Event Emission Locations (Following Puppeteer Exactly)
+
+**Critical**: The location where each event is emitted matters for correct behavior.
+
+| Event | Location | Trigger |
+|-------|----------|---------|
+| `:frameattached` | `Frame#create_frame_target` | Child browsing context created |
+| `:framedetached` | `Frame#initialize_frame` | **Self's** browsing context closed |
+| `:framenavigated` | `Frame#initialize_frame` | DOMContentLoaded or fragment_navigated |
+
+### Puppeteer Reference Code
+
+From [Puppeteer's bidi/Frame.ts](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/bidi/Frame.ts):
+
+```typescript
+// In #initialize() - FrameDetached is emitted for THIS frame
+this.browsingContext.on('closed', () => {
+  this.page().trustedEmitter.emit(PageEvent.FrameDetached, this);
+});
+
+// In #createFrameTarget() - FrameAttached is emitted for child frame
+#createFrameTarget(browsingContext: BrowsingContext) {
+  const frame = BidiFrame.from(this, browsingContext);
+  this.#frames.set(browsingContext, frame);
+  this.page().trustedEmitter.emit(PageEvent.FrameAttached, frame);
+
+  // Note: FrameDetached is NOT emitted here
+  browsingContext.on('closed', () => {
+    this.#frames.delete(browsingContext);
+  });
+
+  return frame;
+}
+```
+
+### Ruby Implementation
+
+```ruby
+# Frame#initialize_frame
+def initialize_frame
+  # ... child frame setup ...
+
+  # FrameDetached: emit when THIS frame's context closes
+  @browsing_context.on(:closed) do
+    @frames.clear
+    page.emit(:framedetached, self)
+  end
+
+  # FrameNavigated: emit on navigation events
+  @browsing_context.on(:dom_content_loaded) do
+    page.emit(:framenavigated, self)
+  end
+
+  @browsing_context.on(:fragment_navigated) do
+    page.emit(:framenavigated, self)
+  end
+end
+
+# Frame#create_frame_target
+def create_frame_target(browsing_context)
+  frame = Frame.from(self, browsing_context)
+  @frames[browsing_context.id] = frame
+
+  # FrameAttached: emit for the new child frame
+  page.emit(:frameattached, frame)
+
+  # Only cleanup, NO FrameDetached here
+  browsing_context.once(:closed) do
+    @frames.delete(browsing_context.id)
+  end
+
+  frame
+end
+```
+
+### Common Mistake
+
+**Wrong**: Emitting `:framedetached` in `create_frame_target` when child's context closes.
+
+**Correct**: Each frame emits its own `:framedetached` in `initialize_frame` when its own browsing context closes.
+
+This matters because the event should be emitted by the frame instance that is being detached, not by its parent.
+
+## Page Event Emitter
+
+Page delegates to `Core::EventEmitter` for event handling:
+
+```ruby
+class Page
+  def initialize(...)
+    @emitter = Core::EventEmitter.new
+  end
+
+  def on(event, &block)
+    @emitter.on(event, &block)
+  end
+
+  def emit(event, data = nil)
+    @emitter.emit(event, data)
+  end
+end
+```
+
+## Files Changed
+
+- `lib/puppeteer/bidi/frame.rb`: Constructor signature, page method, parent_frame method, frame events
+- `lib/puppeteer/bidi/page.rb`: main_frame initialization, event emitter delegation
+
+## BiDi Protocol Limitations
+
+### Frame.frameElement with Shadow DOM
+
+**Status**: Not supported in BiDi protocol
+
+`Frame#frame_element` returns `nil` for iframes inside Shadow DOM (both open and closed).
+
+#### Root Cause
+
+| Protocol | Behavior | Mechanism |
+|----------|----------|-----------|
+| **CDP (Chrome)** | Works | Uses `DOM.getFrameOwner` command |
+| **BiDi (Firefox)** | Returns nil | Uses `document.querySelectorAll` (cannot traverse Shadow DOM) |
+
+#### Technical Details
+
+1. **CDP Implementation** (`cdp/Frame.js`):
+   ```javascript
+   const { backendNodeId } = await parent.client.send('DOM.getFrameOwner', {
+     frameId: this._id,
+   });
+   return await parent.mainRealm().adoptBackendNode(backendNodeId);
+   ```
+
+2. **BiDi Implementation** (base `api/Frame.js`):
+   ```javascript
+   const list = await parentFrame.isolatedRealm().evaluateHandle(() => {
+     return document.querySelectorAll('iframe,frame');
+   });
+   // Cannot find elements inside Shadow DOM
+   ```
+
+3. **WebDriver BiDi Specification**: No `DOM.getFrameOwner` equivalent command exists.
+
+#### Verification
+
+Tested with Puppeteer (Node.js) using both protocols:
+
+```
+=== Firefox (BiDi protocol) ===
+Frame element is NULL - Shadow DOM issue confirmed
+
+=== Chrome (CDP protocol) ===
+Frame element tagName: iframe
+```
+
+#### References
+
+- [Puppeteer Issue #13155](https://github.com/puppeteer/puppeteer/issues/13155) - Original bug report
+- [Puppeteer PR #13156](https://github.com/puppeteer/puppeteer/pull/13156) - CDP-only fix (October 2024)
+- [WebDriver BiDi Specification](https://w3c.github.io/webdriver-bidi/) - browsingContext module
+
+#### Test Status
+
+```ruby
+it 'should handle shadow roots', pending: 'BiDi protocol limitation: no DOM.getFrameOwner equivalent' do
+  # ...
+end
+```
+
+This is a **protocol limitation**, not an implementation bug in this library.
+
+## Commit Reference
+
+See commit: "Refactor Frame to use parent-based architecture following Puppeteer"

data/CLAUDE/javascript_evaluation.md

@@ -0,0 +1,341 @@
+# JavaScript Evaluation Implementation
+
+This document details the implementation of JavaScript evaluation in Page and Frame classes, including script detection logic, argument serialization, and result deserialization.
+
+### Page.evaluate and Frame.evaluate
+
+The `evaluate` method supports both JavaScript expressions and functions with proper argument serialization.
+
+#### Detection Logic
+
+The implementation distinguishes between three types of JavaScript code:
+
+1. **Expressions**: Simple JavaScript code
+2. **Functions**: Arrow functions or function declarations (use `script.callFunction`)
+3. **IIFE**: Immediately Invoked Function Expressions (use `script.evaluate`)
+
+```ruby
+# Expression - uses script.evaluate
+page.evaluate('7 * 3')  # => 21
+
+# Function - uses script.callFunction
+page.evaluate('(a, b) => a + b', 3, 4)  # => 7
+
+# IIFE - uses script.evaluate (not script.callFunction)
+page.evaluate('(() => document.title)()')  # => "Page Title"
+```
+
+#### IIFE Detection Pattern
+
+**Critical**: IIFE must be detected and treated as expressions:
+
+```ruby
+# Check if it's an IIFE - ends with () after the function body
+is_iife = script_trimmed.match?(/\)\s*\(\s*\)\s*\z/)
+
+# Only treat as function if not IIFE
+is_function = !is_iife && (
+  script_trimmed.match?(/\A\s*(?:async\s+)?(?:\(.*?\)|[a-zA-Z_$][\w$]*)\s*=>/) ||
+  script_trimmed.match?(/\A\s*(?:async\s+)?function\s*\w*\s*\(/)
+)
+```
+
+**Why this matters**: IIFE like `(() => {...})()` looks like a function but must be evaluated as an expression. Using `script.callFunction` on IIFE causes syntax errors.
+
+#### Argument Serialization
+
+Arguments are serialized to BiDi `LocalValue` format:
+
+```ruby
+# Special numbers
+{ type: 'number', value: 'NaN' }
+{ type: 'number', value: 'Infinity' }
+{ type: 'number', value: '-0' }
+
+# Collections
+{ type: 'array', value: [...] }
+{ type: 'object', value: [[key, value], ...] }
+{ type: 'map', value: [[key, value], ...] }
+```
+
+#### Result Deserialization
+
+BiDi returns results in special format that must be deserialized:
+
+```ruby
+# BiDi response format
+{
+  "type" => "success",
+  "realm" => "...",
+  "result" => {
+    "type" => "number",
+    "value" => 42
+  }
+}
+
+# Extract and deserialize
+actual_result = result['result'] || result
+deserialize_result(actual_result)  # => 42
+```
+
+#### Exception Handling
+
+Exceptions from JavaScript are returned in the result, not thrown by BiDi:
+
+```ruby
+if result['type'] == 'exception'
+  exception_details = result['exceptionDetails']
+  text = exception_details['text']  # "ReferenceError: notExistingObject is not defined"
+  raise text
+end
+```
+
+### Core::Realm Return Values
+
+**Important**: Core::Realm methods return the **complete BiDi result**, not just the value:
+
+```ruby
+# Core::Realm.call_function returns:
+{
+  "type" => "success" | "exception",
+  "realm" => "...",
+  "result" => {...} | nil,
+  "exceptionDetails" => {...} | nil
+}
+
+# NOT result['result'] (this was a bug that was fixed)
+```
+
+### Testing Strategy
+
+#### Integration Tests Organization
+
+```
+spec/
+├── unit/                    # Fast unit tests (future)
+├── integration/             # Browser automation tests
+│   ├── examples/           # Example-based tests
+│   │   └── screenshot_spec.rb
+│   └── screenshot_spec.rb  # Feature test suites
+├── assets/                 # Test HTML/CSS/JS files
+│   ├── grid.html
+│   ├── scrollbar.html
+│   ├── empty.html
+│   └── digits/*.png
+├── golden-firefox/         # Reference images
+│   └── screenshot-*.png
+└── support/               # Test utilities
+    ├── test_server.rb
+    └── golden_comparator.rb
+```
+
+#### Implemented Screenshot Tests
+
+All 12 tests ported from [Puppeteer's screenshot.spec.ts](https://github.com/puppeteer/puppeteer/blob/main/test/src/screenshot.spec.ts):
+
+1. **should work** - Basic screenshot functionality
+2. **should clip rect** - Clipping specific region
+3. **should get screenshot bigger than the viewport** - Offscreen clip with captureBeyondViewport
+4. **should clip bigger than the viewport without "captureBeyondViewport"** - Viewport coordinate transformation
+5. **should run in parallel** - Thread-safe parallel screenshots on single page
+6. **should take fullPage screenshots** - Full page with document origin
+7. **should take fullPage screenshots without captureBeyondViewport** - Full page with viewport resize
+8. **should run in parallel in multiple pages** - Concurrent screenshots across multiple pages
+9. **should work with odd clip size on Retina displays** - Odd pixel dimensions (11x11)
+10. **should return base64** - Base64 encoding verification
+11. **should take fullPage screenshots when defaultViewport is null** - No explicit viewport
+12. **should restore to original viewport size** - Viewport restoration after fullPage
+
+Run tests:
+```bash
+bundle exec rspec spec/integration/screenshot_spec.rb
+# Expected: 12 examples, 0 failures (completes in ~8 seconds with optimized spec_helper)
+```
+
+#### Test Performance Optimization
+
+**Critical**: Integration tests are ~19x faster with browser reuse strategy.
+
+##### Before Optimization (Per-test Browser Launch)
+```ruby
+def with_test_state(**options)
+  server = TestServer::Server.new
+  server.start
+
+  with_browser(**options) do |browser|  # New browser per test!
+    context = browser.default_browser_context
+    page = browser.new_page
+    yield(page: page, server: server, browser: browser, context: context)
+  end
+ensure
+  server.stop
+end
+```
+
+**Performance**: ~195 seconds for 35 tests (browser launch overhead × 35)
+
+##### After Optimization (Shared Browser)
+```ruby
+# In spec_helper.rb
+config.before(:suite) do
+  if RSpec.configuration.files_to_run.any? { |f| f.include?('spec/integration') }
+    $shared_browser = Puppeteer::Bidi.launch(headless: headless_mode?)
+    $shared_test_server = TestServer::Server.new
+    $shared_test_server.start
+  end
+end
+
+def with_test_state(**options)
+  if $shared_browser && options.empty?
+    # Create new page (tab) per test
+    page = $shared_browser.new_page
+    context = $shared_browser.default_browser_context
+
+    begin
+      yield(page: page, server: $shared_test_server, browser: $shared_browser, context: context)
+    ensure
+      page.close unless page.closed?  # Clean up tab
+    end
+  else
+    # Fall back to per-test browser for custom options
+  end
+end
+```
+
+**Performance**: ~10 seconds for 35 tests (1 browser launch + 35 tab creations)
+
+##### Performance Results
+
+| Test Suite | Before | After | Improvement |
+|------------|--------|-------|-------------|
+| **evaluation_spec (23 tests)** | 127s | **7.17s** | **17.7x faster** |
+| **screenshot_spec (12 tests)** | 68s | **8.47s** | **8.0x faster** |
+| **Combined (35 tests)** | 195s | **10.33s** | **18.9x faster** 🚀 |
+
+**Key Benefits**:
+- Browser launch only once per suite
+- Each test gets fresh page (tab) for isolation
+- Cleanup handled automatically
+- Backward compatible (custom options fall back to per-test browser)
+
+#### Environment Variables
+
+```bash
+HEADLESS=false  # Run browser in non-headless mode for debugging
+```
+
+### Debugging Techniques
+
+#### 1. Save Screenshots for Inspection
+
+```ruby
+# In golden_comparator.rb
+def save_screenshot(screenshot_base64, filename)
+  output_dir = File.join(__dir__, '../output')
+  FileUtils.mkdir_p(output_dir)
+  File.binwrite(File.join(output_dir, filename),
+                Base64.decode64(screenshot_base64))
+end
+```
+
+#### 2. Compare Images Pixel-by-Pixel
+
+```ruby
+cat > /tmp/compare.rb << 'EOF'
+require 'chunky_png'
+
+golden = ChunkyPNG::Image.from_file('spec/golden-firefox/screenshot.png')
+actual = ChunkyPNG::Image.from_file('spec/output/debug.png')
+
+diff_count = 0
+(0...golden.height).each do |y|
+  (0...golden.width).each do |x|
+    if golden[x, y] != actual[x, y]
+      diff_count += 1
+      puts "Diff at (#{x}, #{y})" if diff_count <= 10
+    end
+  end
+end
+puts "Total: #{diff_count} pixels differ"
+EOF
+ruby /tmp/compare.rb
+```
+
+#### 3. Debug BiDi Responses
+
+```ruby
+# Temporarily add debugging
+result = @browsing_context.default_realm.evaluate(script, true)
+puts "BiDi result: #{result.inspect}"
+deserialize_result(result)
+```
+
+### Common Pitfalls and Solutions
+
+#### 1. BiDi Protocol Differences
+
+**Problem:** BiDi `origin` parameter behavior differs from expectations
+
+**Solution:** Consult BiDi spec and test both `'document'` and `'viewport'` origins
+
+```ruby
+# document: Absolute coordinates in full page
+# viewport: Relative to current viewport
+options[:origin] = capture_beyond_viewport ? 'document' : 'viewport'
+```
+
+#### 2. Image Comparison Failures
+
+**Problem:** Golden images don't match exactly (1-2 pixel differences)
+
+**Solution:** Implement tolerance in comparison
+
+```ruby
+# Allow small rendering differences (±1 RGB per channel)
+compare_with_golden(screenshot, 'golden.png', pixel_threshold: 1)
+```
+
+#### 3. Viewport State Management
+
+**Problem:** Viewport not restored after fullPage screenshot
+
+**Solution:** Use `ensure` block
+
+```ruby
+begin
+  set_viewport(full_page_dimensions)
+  screenshot = capture_screenshot(...)
+ensure
+  set_viewport(original_viewport) if original_viewport
+end
+```
+
+#### 4. Thread Safety
+
+**Problem:** Parallel screenshots cause race conditions
+
+**Solution:** BiDi protocol handles this naturally - test with threads
+
+```ruby
+threads = (0...3).map do |i|
+  Thread.new { page.screenshot(clip: {...}) }
+end
+screenshots = threads.map(&:value)
+```
+
+### Documentation References
+
+**Essential reading for implementation:**
+
+1. **WebDriver BiDi Spec**: https://w3c.github.io/webdriver-bidi/
+2. **Puppeteer Source**: https://github.com/puppeteer/puppeteer
+3. **Puppeteer BiDi Tests**: https://github.com/puppeteer/puppeteer/tree/main/test/src
+4. **Firefox BiDi Impl**: Check Firefox implementation notes for quirks
+
+**Reference implementation workflow:**
+1. Find corresponding Puppeteer test in `test/src/`
+2. Read TypeScript implementation in `packages/puppeteer-core/src/`
+3. Check BiDi spec for protocol details
+4. Implement Ruby version maintaining same logic
+5. Download golden images and verify pixel-perfect match (with tolerance)
+