puppeteer-bidi 0.0.1.beta1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8a2905e58b0b31a75a0b286fa14986def369fe62d122dd62dc6432a763070ea9
+  data.tar.gz: b6cb21ba581490c0cbe28ae4515498660091ec9653df2c6d8c1e4b9d5f34197e
+SHA512:
+  metadata.gz: 68c9e714dd633886188df24ccedc71139ac4ed8f5525e470c99b0c802dcdc5ce261279fea56b2c60dd6456a245b056acf987d9cd43cd6283c33f3bca2f568794
+  data.tar.gz: beb1f011607f387f10329db6078a91d41eb5582631256f742655af39d7ea15638aa3d29c3e7c44a1fce4072585c141898147c783be65b1cf20deda1a73e25d4f

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,13 @@
+AllCops:
+  TargetRubyVersion: 2.6
+
+Style/StringLiterals:
+  Enabled: true
+  EnforcedStyle: double_quotes
+
+Style/StringLiteralsInInterpolation:
+  Enabled: true
+  EnforcedStyle: double_quotes
+
+Layout/LineLength:
+  Max: 120

data/CLAUDE/README.md

@@ -0,0 +1,158 @@
+# Detailed Implementation Documentation
+
+This directory contains detailed documentation for specific implementation topics in puppeteer-bidi.
+
+## Quick Reference
+
+| Document | Topic | Key Takeaway |
+|----------|-------|--------------|
+| [async_programming.md](async_programming.md) | Fiber-based async | Use Async, NOT concurrent-ruby |
+| [two_layer_architecture.md](two_layer_architecture.md) | Core vs Upper layer | Always call `.wait` on Core methods |
+| [porting_puppeteer.md](porting_puppeteer.md) | Implementing features | Study TypeScript first, port tests |
+| [query_handler.md](query_handler.md) | Selector handling | Override script methods, use PuppeteerUtil |
+| [javascript_evaluation.md](javascript_evaluation.md) | JS evaluation | IIFE detection is critical |
+| [jshandle_implementation.md](jshandle_implementation.md) | Handle management | resultOwnership must be 'root' |
+| [selector_evaluation.md](selector_evaluation.md) | Selector methods | Use `eval_on_selector` not `$eval` |
+| [error_handling.md](error_handling.md) | Custom exceptions | Type-safe error handling |
+| [click_implementation.md](click_implementation.md) | Click & mouse | session.subscribe is mandatory |
+| [wrapped_element_click.md](wrapped_element_click.md) | Wrapped elements | Use getClientRects() |
+| [navigation_waiting.md](navigation_waiting.md) | waitForNavigation | Event-driven with Async::Promise |
+| [testing_strategy.md](testing_strategy.md) | Test optimization | Browser reuse = 19x faster |
+| [frame_architecture.md](frame_architecture.md) | Frame hierarchy | `(parent, browsing_context)` |
+| [rspec_pending_vs_skip.md](rspec_pending_vs_skip.md) | Test documentation | Use `pending` for Firefox |
+| [test_server_routes.md](test_server_routes.md) | Dynamic routes | `server.set_route` for tests |
+
+## Architecture & Patterns
+
+### [Async Programming](async_programming.md)
+
+Guide to Fiber-based async programming with socketry/async.
+
+**Key concepts:**
+- Use Async (Fiber-based), NOT concurrent-ruby (Thread-based)
+- No Mutex needed - cooperative multitasking
+- WebSocket messages must use `Async do` for non-blocking processing
+
+### [Two-Layer Architecture](two_layer_architecture.md)
+
+Core vs Upper layer separation for async complexity management.
+
+**Key concepts:**
+- Core layer returns `Async::Task`
+- Upper layer calls `.wait` on all Core methods
+- User-facing API is synchronous
+
+### [Porting Puppeteer](porting_puppeteer.md)
+
+Best practices for implementing Puppeteer features in Ruby.
+
+**Key concepts:**
+- Study TypeScript implementation first
+- Port corresponding test cases
+- Use official test assets without modification
+
+## Implementation Details
+
+### [QueryHandler](query_handler.md)
+
+Extensible selector handling for CSS, XPath, text selectors.
+
+**Key concepts:**
+- Override `query_one_script` and `query_all_script`
+- TextQueryHandler uses PuppeteerUtil directly (closure dependency)
+- Handle adoption pattern for navigation
+
+### [JavaScript Evaluation](javascript_evaluation.md)
+
+Implementation of `evaluate()` and `evaluate_handle()`.
+
+**Key concepts:**
+- IIFE must be detected and treated as expressions
+- Always deserialize BiDi results before returning
+
+### [JSHandle and ElementHandle](jshandle_implementation.md)
+
+Handle management and BiDi protocol parameters.
+
+**Key concepts:**
+- Set `resultOwnership: 'root'` to get handles
+- Handle parameters must be arrays
+
+### [Selector Evaluation](selector_evaluation.md)
+
+Implementation of `eval_on_selector` methods.
+
+**Key concepts:**
+- Delegation pattern: Page → Frame → ElementHandle
+- Always dispose handles in ensure blocks
+
+### [Click Implementation](click_implementation.md)
+
+Mouse input and click functionality.
+
+**Key concepts:**
+- Must call `session.subscribe` for events
+- URL updates happen via events
+
+### [Wrapped Element Click](wrapped_element_click.md)
+
+Clicking wrapped/multi-line text elements.
+
+**Key concepts:**
+- Use getClientRects() for wrapped elements
+- Viewport clipping ensures clicks stay visible
+
+### [Navigation Waiting](navigation_waiting.md)
+
+waitForNavigation patterns.
+
+**Key concepts:**
+- Event-driven with Async::Promise
+- Attach to existing navigations before block execution
+
+### [Frame Architecture](frame_architecture.md)
+
+Parent-based frame hierarchy.
+
+**Key concepts:**
+- First parameter is `parent` (Page or Frame)
+- Enables future iframe support
+
+### [Error Handling](error_handling.md)
+
+Custom exception types.
+
+**Key concepts:**
+- Use custom exceptions for type-safe rescue
+- Include contextual data in exceptions
+
+## Testing
+
+### [Testing Strategy](testing_strategy.md)
+
+Test organization and optimization.
+
+**Key concepts:**
+- Reuse browser across tests for 19x speedup
+- Use official Puppeteer test assets
+
+### [RSpec: pending vs skip](rspec_pending_vs_skip.md)
+
+Documenting browser limitations.
+
+**Key concepts:**
+- Use `pending` for Firefox BiDi limitations
+- Use `skip` for unimplemented features
+
+### [Test Server Routes](test_server_routes.md)
+
+Dynamic route handling for tests.
+
+**Key concepts:**
+- `server.set_route(path)` for intercepting requests
+- `server.wait_for_request(path)` for synchronization
+
+## Related Documentation
+
+- **Main guide**: [CLAUDE.md](../CLAUDE.md) - High-level development guide
+- **Core layer**: [lib/puppeteer/bidi/core/README.md](../lib/puppeteer/bidi/core/README.md) - Core BiDi abstraction layer

data/CLAUDE/async_programming.md

@@ -0,0 +1,158 @@
+# Async Programming with socketry/async
+
+This project uses the [socketry/async](https://github.com/socketry/async) library for asynchronous operations.
+
+## Why Async Instead of concurrent-ruby?
+
+**IMPORTANT**: This project uses `Async` (Fiber-based), **NOT** `concurrent-ruby` (Thread-based).
+
+| Feature               | Async (Fiber-based)                                    | concurrent-ruby (Thread-based)         |
+| --------------------- | ------------------------------------------------------ | -------------------------------------- |
+| **Concurrency Model** | Cooperative multitasking (like JavaScript async/await) | Preemptive multitasking                |
+| **Race Conditions**   | Not possible within a Fiber                            | Requires Mutex, locks, etc.            |
+| **Synchronization**   | Not needed (cooperative)                               | Required (Mutex, Semaphore)            |
+| **Mental Model**      | Similar to JavaScript async/await                      | Traditional thread programming         |
+| **Bug Risk**          | Lower (no race conditions)                             | Higher (race conditions, deadlocks)    |
+
+**Key advantages:**
+
+- **No race conditions**: Fibers yield control cooperatively, so no concurrent access to shared state
+- **No Mutex needed**: Since there are no race conditions, no synchronization primitives required
+- **Similar to JavaScript**: If you understand `async/await` in JavaScript, you understand Async in Ruby
+- **Easier to reason about**: Code executes sequentially within a Fiber until it explicitly yields
+
+**Example:**
+
+```ruby
+# DON'T: Use concurrent-ruby (Thread-based, requires Mutex)
+require 'concurrent'
+@pending = Concurrent::Map.new  # Thread-safe map
+promise = Concurrent::Promises.resolvable_future
+promise.fulfill(value)
+
+# DO: Use Async (Fiber-based, no synchronization needed)
+require 'async/promise'
+@pending = {}  # Plain Hash is safe with Fibers
+promise = Async::Promise.new
+promise.resolve(value)
+```
+
+## Best Practices
+
+1. **Use `Sync` at top level**: When running async code at the top level of a thread or application, use `Sync { }` instead of `Async { }`
+
+   ```ruby
+   Thread.new do
+     Sync do
+       # async operations here
+     end
+   end
+   ```
+
+2. **Reactor lifecycle**: The reactor is automatically managed by `Sync { }`. No need to create explicit `Async::Reactor` instances in application code.
+
+3. **Background operations**: For long-running background tasks (like WebSocket connections), wrap `Sync { }` in a Thread:
+
+   ```ruby
+   connection_task = Thread.new do
+     Sync do
+       transport.connect  # Async operation that blocks until connection closes
+     end
+   end
+   ```
+
+4. **Promise usage**: Use `Async::Promise` for async coordination:
+
+   ```ruby
+   promise = Async::Promise.new
+
+   # Resolve the promise
+   promise.resolve(value)
+
+   # Wait for the promise with timeout
+   Async do |task|
+     task.with_timeout(5) do
+       result = promise.wait
+     end
+   end.wait
+   ```
+
+5. **No Mutex needed**: Since Async is Fiber-based, you don't need Mutex for shared state within the same event loop
+
+## AsyncUtils: Promise.all and Promise.race
+
+The `lib/puppeteer/bidi/async_utils.rb` module provides JavaScript-like Promise utilities:
+
+```ruby
+# Promise.all - Wait for all tasks to complete
+results = AsyncUtils.promise_all(
+  -> { sleep 0.1; 'first' },
+  -> { sleep 0.2; 'second' },
+  -> { sleep 0.05; 'third' }
+)
+# => ['first', 'second', 'third'] (in order, runs in parallel)
+
+# Promise.race - Return the first to complete
+result = AsyncUtils.promise_race(
+  -> { sleep 0.3; 'slow' },
+  -> { sleep 0.1; 'fast' },
+  -> { sleep 0.2; 'medium' }
+)
+# => 'fast' (cancels remaining tasks)
+```
+
+**When to use AsyncUtils:**
+
+- **Parallel task execution**: Running multiple independent async operations
+- **Racing timeouts**: First of multiple operations to complete
+- **NOT for event-driven waiting**: Use `Async::Promise` directly for event listeners
+
+## WebSocket Message Handling Pattern
+
+**CRITICAL**: BiDi message handling must use `Async do` to process messages asynchronously:
+
+```ruby
+# lib/puppeteer/bidi/transport.rb
+while (message = connection.read)
+  next if message.nil?
+
+  # DO: Use Async do for non-blocking message processing
+  Async do
+    data = JSON.parse(message)
+    debug_print_receive(data)
+    @on_message&.call(data)
+  rescue StandardError => e
+    # Handle errors
+  end
+end
+```
+
+**Why this matters:**
+
+- **Without `Async do`**: Message processing blocks the message loop, preventing other messages from being read
+- **With `Async do`**: Each message is processed in a separate fiber, allowing concurrent message handling
+- **Prevents deadlocks**: When multiple operations are waiting for responses, they can all be processed concurrently
+
+**Example of the problem this solves:**
+
+```ruby
+# Without Async do:
+# 1. Message A arrives and starts processing
+# 2. Processing A calls wait_for_navigation which waits for Message B
+# 3. Message B arrives but can't be read because Message A is still being processed
+# 4. DEADLOCK
+
+# With Async do:
+# 1. Message A arrives and starts processing in Fiber 1
+# 2. Fiber 1 yields when calling wait (cooperative multitasking)
+# 3. Message B can now be read and processed in Fiber 2
+# 4. Both messages complete successfully
+```
+
+This pattern is essential for the BiDi protocol's bidirectional communication model.
+
+## References
+
+- [Async Best Practices](https://socketry.github.io/async/guides/best-practices/)
+- [Async Documentation](https://socketry.github.io/async/)
+- [Async::Barrier Guide](https://socketry.github.io/async/guides/tasks/index.html)

data/CLAUDE/click_implementation.md

@@ -0,0 +1,340 @@
+# Click Implementation and Mouse Input
+
+This document provides comprehensive coverage of the click functionality implementation, including architecture, bug fixes, event handling, and BiDi protocol format requirements.
+
+### Overview
+
+Implemented full click functionality following Puppeteer's architecture, including mouse input actions, element visibility detection, and automatic scrolling.
+
+### Architecture
+
+```
+Page#click
+  ↓ delegates to
+Frame#click
+  ↓ delegates to
+ElementHandle#click
+  ↓ implementation
+  1. scroll_into_view_if_needed
+  2. clickable_point calculation
+  3. Mouse#click (BiDi input.performActions)
+```
+
+### Key Components
+
+#### Mouse Class (`lib/puppeteer/bidi/mouse.rb`)
+
+Implements mouse input actions via BiDi `input.performActions`:
+
+```ruby
+def click(x, y, button: LEFT, count: 1, delay: nil)
+  actions = []
+  if @x != x || @y != y
+    actions << {
+      type: 'pointerMove',
+      x: x.to_i,
+      y: y.to_i,
+      origin: 'viewport'  # BiDi expects string, not hash!
+    }
+  end
+  @x = x
+  @y = y
+  bidi_button = button_to_bidi(button)
+  count.times do
+    actions << { type: 'pointerDown', button: bidi_button }
+    actions << { type: 'pause', duration: delay.to_i } if delay
+    actions << { type: 'pointerUp', button: bidi_button }
+  end
+  perform_actions(actions)
+end
+```
+
+**Critical BiDi Protocol Detail**: The `origin` parameter must be the string `'viewport'`, NOT a hash like `{type: 'viewport'}`. This caused a protocol error during initial implementation.
+
+#### ElementHandle Click Methods
+
+##### scroll_into_view_if_needed
+
+Uses IntersectionObserver API to detect viewport visibility:
+
+```ruby
+def scroll_into_view_if_needed
+  return if intersecting_viewport?
+
+  scroll_info = evaluate(<<~JS)
+    element => {
+      if (!element.isConnected) return 'Node is detached from document';
+      if (element.nodeType !== Node.ELEMENT_NODE) return 'Node is not of type HTMLElement';
+
+      element.scrollIntoView({
+        block: 'center',
+        inline: 'center',
+        behavior: 'instant'
+      });
+      return false;
+    }
+  JS
+
+  raise scroll_info if scroll_info
+end
+```
+
+##### intersecting_viewport?
+
+Uses browser's IntersectionObserver for accurate visibility detection:
+
+```ruby
+def intersecting_viewport?(threshold: 0)
+  evaluate(<<~JS, threshold)
+    (element, threshold) => {
+      return new Promise(resolve => {
+        const observer = new IntersectionObserver(entries => {
+          resolve(entries[0].intersectionRatio > threshold);
+          observer.disconnect();
+        });
+        observer.observe(element);
+      });
+    }
+  JS
+end
+```
+
+##### clickable_point
+
+Calculates click coordinates with optional offset:
+
+```ruby
+def clickable_point(offset: nil)
+  box = clickable_box
+  if offset
+    { x: box[:x] + offset[:x], y: box[:y] + offset[:y] }
+  else
+    { x: box[:x] + box[:width] / 2, y: box[:y] + box[:height] / 2 }
+  end
+end
+```
+
+### Critical Bug Fixes
+
+#### 1. Missing session.subscribe Call
+
+**Problem**: Navigation events (browsingContext.load, etc.) were not firing, causing tests to timeout.
+
+**Root Cause**: Missing subscription to BiDi modules. Puppeteer subscribes to these modules on session creation:
+- browsingContext
+- network
+- log
+- script
+- input
+
+**Fix**: Added subscription in two places:
+
+```ruby
+# lib/puppeteer/bidi/browser.rb
+subscribe_modules = %w[
+  browsingContext
+  network
+  log
+  script
+  input
+]
+@session.subscribe(subscribe_modules)
+
+# lib/puppeteer/bidi/core/session.rb
+def initialize_session
+  subscribe_modules = %w[
+    browsingContext
+    network
+    log
+    script
+    input
+  ]
+  subscribe(subscribe_modules)
+end
+```
+
+**Impact**: This fix enabled all navigation-related functionality, including the "click links which cause navigation" test.
+
+#### 2. Event-Based URL Updates
+
+**Problem**: Initial implementation updated `@url` directly in `navigate()` method, which is not how Puppeteer works.
+
+**Puppeteer's Approach**: URL updates happen via BiDi events:
+- `browsingContext.historyUpdated`
+- `browsingContext.domContentLoaded`
+- `browsingContext.load`
+
+**Fix**: Removed direct URL assignment from navigate():
+
+```ruby
+# lib/puppeteer/bidi/core/browsing_context.rb
+def navigate(url, wait: nil)
+  raise BrowsingContextClosedError, @reason if closed?
+  params = { context: @id, url: url }
+  params[:wait] = wait if wait
+  result = session.send_command('browsingContext.navigate', params)
+  # URL will be updated via browsingContext.load event
+  result
+end
+```
+
+Event handlers (already implemented) update URL automatically:
+
+```ruby
+# History updated
+session.on('browsingContext.historyUpdated') do |info|
+  next unless info['context'] == @id
+  @url = info['url']
+  emit(:history_updated, nil)
+end
+
+# DOM content loaded
+session.on('browsingContext.domContentLoaded') do |info|
+  next unless info['context'] == @id
+  @url = info['url']
+  emit(:dom_content_loaded, nil)
+end
+
+# Page loaded
+session.on('browsingContext.load') do |info|
+  next unless info['context'] == @id
+  @url = info['url']
+  emit(:load, nil)
+end
+```
+
+**Why this matters**: Event-based updates ensure URL synchronization even when navigation is triggered by user actions (like clicking links) rather than explicit `navigate()` calls.
+
+### Test Coverage
+
+#### Click Tests (20 tests in spec/integration/click_spec.rb)
+
+Ported from [Puppeteer's click.spec.ts](https://github.com/puppeteer/puppeteer/blob/main/test/src/click.spec.ts):
+
+1. **Basic clicking**: button, svg, wrapped links
+2. **Edge cases**: window.Node removed, span with inline elements
+3. **Navigation**: click after navigation, click links causing navigation
+4. **Scrolling**: offscreen buttons, scrollable content
+5. **Multi-click**: double click, triple click (text selection)
+6. **Different buttons**: left, right (contextmenu), middle (auxclick)
+7. **Visibility**: partially obscured button, rotated button
+8. **Form elements**: checkbox toggle (input and label)
+9. **Error handling**: missing selector
+10. **Special cases**: disabled JavaScript, iframes (pending)
+
+#### Page Tests (3 tests in spec/integration/page_spec.rb)
+
+1. **Page.url**: Verify URL updates after navigation
+2. **Page.setJavaScriptEnabled**: Control JavaScript execution (pending - Firefox limitation)
+
+**All 108 integration tests pass** (4 pending due to Firefox BiDi limitations).
+
+### Firefox BiDi Limitations
+
+- `emulation.setScriptingEnabled`: Part of WebDriver BiDi spec but not yet implemented in Firefox
+- Tests gracefully skip with clear messages using RSpec's `skip` feature
+
+### Implementation Best Practices Learned
+
+#### 1. Always Consult Puppeteer's Implementation First
+
+**Workflow**:
+1. Read Puppeteer's TypeScript implementation
+2. Understand BiDi protocol calls being made
+3. Implement Ruby equivalent with same logic flow
+4. Port corresponding test cases
+
+**Example**: The click implementation journey revealed that Puppeteer's architecture (Page → Frame → ElementHandle delegation) is critical for proper functionality.
+
+#### 2. Stay Faithful to Puppeteer's Test Structure
+
+**Initial mistake**: Created complex polling logic for navigation test
+**Correction**: Simplified to match Puppeteer's simple approach:
+
+```ruby
+# Simple and correct (matches Puppeteer)
+page.set_content("<a href=\"#{server.empty_page}\">empty.html</a>")
+page.click('a')  # Should not hang
+```
+
+#### 3. Event Subscription is Critical
+
+**Key lesson**: BiDi requires explicit subscription to event modules. Without it:
+- Navigation events don't fire
+- URL updates don't work
+- Tests timeout mysteriously
+
+**Solution**: Subscribe early in browser/session initialization.
+
+#### 4. Use RSpec `it` Syntax
+
+Per Ruby/RSpec conventions, use `it` instead of `example`:
+
+```ruby
+# Correct
+it 'should click the button' do
+  # ...
+end
+
+# Incorrect
+example 'should click the button' do
+  # ...
+end
+```
+
+### BiDi Protocol Format Requirements
+
+#### Origin Parameter Format
+
+**Critical**: BiDi `input.performActions` expects `origin` as a string, not a hash:
+
+```ruby
+# CORRECT
+origin: 'viewport'
+
+# WRONG - causes protocol error
+origin: { type: 'viewport' }
+```
+
+**Error message if wrong**:
+```
+Expected "origin" to be undefined, "viewport", "pointer", or an element,
+got: [object Object] {"type":"viewport"}
+```
+
+### Performance and Reliability
+
+- **IntersectionObserver**: Fast and accurate visibility detection
+- **Auto-scrolling**: Ensures elements are clickable before interaction
+- **Event-driven**: URL updates via events enable proper async handling
+- **Thread-safe**: BiDi protocol handles concurrent operations naturally
+
+### Future Enhancements
+
+Potential improvements for click/mouse functionality:
+
+1. **Drag and drop**: Implement drag operations
+2. **Hover**: Mouse move without click
+3. **Wheel**: Mouse wheel scrolling
+4. **Touch**: Touch events for mobile emulation
+5. **Keyboard modifiers**: Click with Ctrl/Shift/Alt
+6. **Frame support**: Click inside iframes (currently pending)
+
+### Reference Implementation
+
+Based on Puppeteer's implementation:
+- [Page.click](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/api/Page.ts)
+- [Frame.click](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/api/Frame.ts)
+- [ElementHandle.click](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/api/ElementHandle.ts)
+- [Mouse input](https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/bidi/Input.ts)
+- [Test specs](https://github.com/puppeteer/puppeteer/blob/main/test/src/click.spec.ts)
+
+### Key Takeaways
+
+1. **session.subscribe is mandatory** for BiDi event handling - don't forget it!
+2. **Event-based state management** (URL updates via events, not direct assignment)
+3. **BiDi protocol details matter** (string vs hash for origin parameter)
+4. **Follow Puppeteer's architecture** (delegation patterns, event handling)
+5. **Test simplicity** - stay faithful to Puppeteer's test structure
+6. **Browser limitations** - gracefully handle unimplemented features (setScriptingEnabled)
+