puppeteer-bidi 0.0.1.beta1

data/CLAUDE/porting_puppeteer.md

@@ -0,0 +1,214 @@
+# Porting Puppeteer to Ruby
+
+Best practices for implementing Puppeteer features in puppeteer-bidi.
+
+## 1. Reference Implementation First
+
+**Always consult the official Puppeteer implementation before implementing features:**
+
+- **TypeScript source files**:
+  - `packages/puppeteer-core/src/bidi/Page.ts` - High-level Page API
+  - `packages/puppeteer-core/src/bidi/BrowsingContext.ts` - Core BiDi context
+  - `packages/puppeteer-core/src/api/Page.ts` - Common Page interface
+
+- **Test files**:
+  - `test/src/screenshot.spec.ts` - Screenshot test suite
+  - `test/golden-firefox/` - Golden images for visual regression testing
+
+**Example workflow:**
+
+```ruby
+# 1. Read Puppeteer's TypeScript implementation
+# 2. Understand the BiDi protocol calls being made
+# 3. Implement Ruby equivalent with same logic flow
+# 4. Port corresponding test cases
+```
+
+## 2. Test Infrastructure Setup
+
+**Use async-http for test servers** (lightweight + Async-friendly):
+
+```ruby
+# spec/support/test_server.rb
+endpoint = Async::HTTP::Endpoint.parse("http://127.0.0.1:#{@port}")
+
+server = Async::HTTP::Server.for(endpoint) do |request|
+  if handler = lookup_route(request.path)
+    notify_request(request.path)
+    respond_with_handler(handler, request)
+  else
+    serve_static_asset(request)
+  end
+end
+
+server.run
+```
+
+**Helper pattern for integration tests:**
+
+```ruby
+# Optimized helper - reuses shared browser, creates new page per test
+def with_test_state
+  page = $shared_browser.new_page
+  context = $shared_browser.default_browser_context
+
+  begin
+    yield(page: page, server: $shared_test_server, browser: $shared_browser, context: context)
+  ensure
+    page.close unless page.closed?
+  end
+end
+```
+
+## 3. BiDi Protocol Data Deserialization
+
+**BiDi returns values in special format - always deserialize:**
+
+```ruby
+# BiDi response format:
+# [["width", {"type" => "number", "value" => 500}],
+#  ["height", {"type" => "number", "value" => 1000}]]
+
+def deserialize_result(result)
+  value = result['value']
+  return value unless value.is_a?(Array)
+
+  # Convert to Ruby Hash
+  if value.all? { |item| item.is_a?(Array) && item.length == 2 }
+    value.each_with_object({}) do |(key, val), hash|
+      hash[key] = deserialize_value(val)
+    end
+  else
+    value
+  end
+end
+
+def deserialize_value(val)
+  case val['type']
+  when 'number' then val['value']
+  when 'string' then val['value']
+  when 'boolean' then val['value']
+  when 'undefined', 'null' then nil
+  else val['value']
+  end
+end
+```
+
+## 4. Implementing Puppeteer-Compatible APIs
+
+**Follow Puppeteer's exact logic flow:**
+
+Example: `fullPage` screenshot implementation
+
+```ruby
+# From Puppeteer's Page.ts:
+# if (options.fullPage) {
+#   if (!options.captureBeyondViewport) {
+#     // Resize viewport to full page
+#   }
+# } else {
+#   options.captureBeyondViewport = false;
+# }
+
+if full_page
+  unless capture_beyond_viewport
+    scroll_dimensions = evaluate(...)
+    set_viewport(scroll_dimensions)
+    begin
+      data = capture_screenshot(origin: 'viewport')
+    ensure
+      set_viewport(original_viewport)  # Always restore
+    end
+  else
+    options[:origin] = 'document'
+  end
+elsif !clip
+  capture_beyond_viewport = false  # Match Puppeteer behavior
+end
+```
+
+**Key principles:**
+
+- Use `begin/ensure` blocks for cleanup (viewport restoration, etc.)
+- Match Puppeteer's parameter defaults exactly
+- Follow the same conditional logic order
+
+## 5. Layer Architecture
+
+**Maintain clear separation:**
+
+```
+High-level API (lib/puppeteer/bidi/)
+├── Browser        - User-facing browser interface
+├── BrowserContext - Session management
+└── Page           - Page automation API
+
+Core Layer (lib/puppeteer/bidi/core/)
+├── Session        - BiDi session management
+├── Browser        - Low-level browser operations
+├── UserContext    - BiDi user context
+└── BrowsingContext - BiDi browsing context (tab/frame)
+```
+
+## 6. Setting Page Content
+
+**Use data URLs with base64 encoding:**
+
+```ruby
+def set_content(html, wait_until: 'load')
+  # Encode HTML in base64 to avoid URL encoding issues
+  encoded = Base64.strict_encode64(html)
+  data_url = "data:text/html;base64,#{encoded}"
+  goto(data_url, wait_until: wait_until)
+end
+```
+
+**Why base64:**
+
+- Avoids URL encoding issues with special characters
+- Handles multi-byte characters correctly
+- Standard approach in browser automation tools
+
+## 7. Viewport Restoration
+
+**Always restore viewport after temporary changes:**
+
+```ruby
+# Save current viewport (may be nil)
+original_viewport = viewport
+
+# If no viewport set, save window size
+unless original_viewport
+  original_size = evaluate('({ width: window.innerWidth, height: window.innerHeight })')
+  original_viewport = { width: original_size['width'].to_i, height: original_size['height'].to_i }
+end
+
+# Change viewport temporarily
+set_viewport(width: new_width, height: new_height)
+
+begin
+  # Do work
+ensure
+  # Always restore
+  set_viewport(**original_viewport) if original_viewport
+end
+```
+
+## 8. Test Assets Policy
+
+**CRITICAL**: Always use Puppeteer's official test assets without modification.
+
+- **Source**: https://github.com/puppeteer/puppeteer/tree/main/test/assets
+- **Rule**: Never modify test asset files (HTML, CSS, images) in `spec/assets/`
+- **Verification**: Before creating PR, verify all `spec/assets/` files match Puppeteer's official versions
+
+```bash
+# During development - OK to experiment
+vim spec/assets/test.html  # Temporary modification for debugging
+
+# Before PR - MUST revert to official
+curl -sL https://raw.githubusercontent.com/puppeteer/puppeteer/main/test/assets/test.html \
+  -o spec/assets/test.html
+```
+
+**Why this matters**: Test assets are designed to test specific edge cases (rotated elements, complex layouts, etc.). Using simplified versions defeats the purpose of these tests.

data/CLAUDE/query_handler.md

@@ -0,0 +1,194 @@
+# QueryHandler Implementation
+
+The QueryHandler system provides extensible selector handling for CSS, XPath, text, and other selector types.
+
+## Architecture
+
+```
+QueryHandler (singleton)
+├── get_query_handler_and_selector(selector)
+│   └── Returns: { updated_selector, polling, query_handler }
+
+BaseQueryHandler
+├── run_query_one(element, selector)  → ElementHandle | nil
+├── run_query_all(element, selector)  → Array<ElementHandle>
+└── wait_for(element_or_frame, selector, options)
+
+Implementations:
+├── CSSQueryHandler   - Default, uses cssQuerySelector/cssQuerySelectorAll
+├── XPathQueryHandler - xpath/ prefix, uses xpathQuerySelectorAll
+└── TextQueryHandler  - text/ prefix, uses textQuerySelectorAll
+```
+
+## Selector Prefixes
+
+| Prefix    | Handler            | Example                     |
+| --------- | ------------------ | --------------------------- |
+| (none)    | CSSQueryHandler    | `div.foo`, `#id`            |
+| `xpath/`  | XPathQueryHandler  | `xpath/html/body/div`       |
+| `text/`   | TextQueryHandler   | `text/Hello World`          |
+| `aria/`   | ARIAQueryHandler   | `aria/Submit[role="button"]`|
+| `pierce/` | PierceQueryHandler | `pierce/.shadow-element`    |
+
+## Implementation Pattern
+
+All query handlers follow the same pattern: override `query_one_script`, `query_all_script`, and `wait_for_selector_script` to define the JavaScript that runs in the browser.
+
+```ruby
+class CSSQueryHandler < BaseQueryHandler
+  private
+
+  def query_one_script
+    <<~JAVASCRIPT
+    (PuppeteerUtil, element, selector) => {
+      return PuppeteerUtil.cssQuerySelector(element, selector);
+    }
+    JAVASCRIPT
+  end
+
+  def query_all_script
+    <<~JAVASCRIPT
+    async (PuppeteerUtil, element, selector) => {
+      return [...PuppeteerUtil.cssQuerySelectorAll(element, selector)];
+    }
+    JAVASCRIPT
+  end
+
+  def wait_for_selector_script
+    <<~JAVASCRIPT
+    (PuppeteerUtil, selector, root, visibility) => {
+      const element = PuppeteerUtil.cssQuerySelector(root || document, selector);
+      return PuppeteerUtil.checkVisibility(element, visibility === null ? undefined : visibility);
+    }
+    JAVASCRIPT
+  end
+end
+```
+
+## TextQueryHandler - Special Case
+
+`textQuerySelectorAll` cannot be extracted via `toString()` because it references helper functions (`f`, `m`, `d`) that are only available within the PuppeteerUtil closure. So TextQueryHandler uses a different pattern: call `textQuerySelectorAll` directly from PuppeteerUtil instead of recreating the function.
+
+```ruby
+class TextQueryHandler < BaseQueryHandler
+  private
+
+  def query_one_script
+    <<~JAVASCRIPT
+    (PuppeteerUtil, element, selector) => {
+      for (const result of PuppeteerUtil.textQuerySelectorAll(element, selector)) {
+        return result;
+      }
+      return null;
+    }
+    JAVASCRIPT
+  end
+
+  def query_all_script
+    <<~JAVASCRIPT
+    async (PuppeteerUtil, element, selector) => {
+      return [...PuppeteerUtil.textQuerySelectorAll(element, selector)];
+    }
+    JAVASCRIPT
+  end
+end
+```
+
+## Handle Adoption Pattern
+
+After navigation, the sandbox realm is destroyed and `puppeteer_util` handles become stale. The solution is to adopt the element into the isolated realm BEFORE calling any query methods:
+
+```ruby
+def run_query_one(element, selector)
+  realm = element.frame.isolated_realm
+
+  # Adopt the element into the isolated realm first.
+  # This ensures the realm is valid and triggers puppeteer_util reset if needed
+  # after navigation (mirrors Puppeteer's @bindIsolatedHandle decorator pattern).
+  adopted_element = realm.adopt_handle(element)
+
+  result = realm.call_function(
+    query_one_script,
+    false,
+    arguments: [
+      Serializer.serialize(realm.puppeteer_util_lazy_arg),
+      adopted_element.remote_value,
+      Serializer.serialize(selector)
+    ]
+  )
+
+  # ... handle result ...
+ensure
+  adopted_element&.dispose
+end
+```
+
+**Why this matters:**
+
+1. After navigation, sandbox realm is destroyed
+2. `:updated` event that resets `puppeteer_util` isn't fired until we call INTO the sandbox
+3. Calling `adopt_handle` first ensures the realm exists and is valid
+4. Then `puppeteer_util` will be fresh (re-evaluated if realm was recreated)
+
+## Debugging
+
+Use `DEBUG_BIDI_COMMAND=1` to see protocol messages:
+
+```bash
+DEBUG_BIDI_COMMAND=1 bundle exec rspec spec/integration/queryhandler_spec.rb:8
+```
+
+This shows:
+1. PuppeteerUtil being evaluated in sandbox
+2. The query function being called with arguments
+3. The result being returned
+
+## Adding New Query Handlers
+
+1. Create a new class extending `BaseQueryHandler`
+2. Override `query_one_script`, `query_all_script`, and `wait_for_selector_script`
+3. Register in `BUILTIN_QUERY_HANDLERS` constant
+4. The script receives `(PuppeteerUtil, element, selector)` as arguments
+5. Return the element(s) found or null/empty array
+
+```ruby
+class MyQueryHandler < BaseQueryHandler
+  private
+
+  def query_one_script
+    <<~JAVASCRIPT
+    (PuppeteerUtil, element, selector) => {
+      // Use PuppeteerUtil.myQuerySelector if available
+      // Or implement custom logic
+      return element.querySelector(selector);
+    }
+    JAVASCRIPT
+  end
+
+  def query_all_script
+    <<~JAVASCRIPT
+    async (PuppeteerUtil, element, selector) => {
+      return [...element.querySelectorAll(selector)];
+    }
+    JAVASCRIPT
+  end
+
+  def wait_for_selector_script
+    <<~JAVASCRIPT
+    (PuppeteerUtil, selector, root, visibility) => {
+      const element = (root || document).querySelector(selector);
+      return PuppeteerUtil.checkVisibility(element, visibility === null ? undefined : visibility);
+    }
+    JAVASCRIPT
+  end
+end
+```
+
+## Test Coverage
+
+Tests are in `spec/integration/queryhandler_spec.rb`:
+
+- Text selectors: 12 tests (query_selector, query_selector_all, shadow DOM piercing, etc.)
+- XPath selectors: 6 tests (in Page and ElementHandle)
+
+Tests ported from Puppeteer's `test/src/queryhandler.spec.ts`.

data/CLAUDE/rspec_pending_vs_skip.md

@@ -0,0 +1,262 @@
+# RSpec: pending vs skip
+
+## Overview
+
+RSpec provides two mechanisms for handling tests that cannot or should not run: `pending` and `skip`. Understanding when to use each is critical for documenting browser limitations and future work.
+
+## Difference
+
+### skip
+
+**Completely skips the test** - does not run any code:
+
+```ruby
+it 'should work' do
+  skip 'feature not implemented'
+
+  # This code NEVER runs
+  page.do_something
+  expect(result).to be_truthy
+end
+```
+
+**Output**: Test marked as skipped, no execution, no error trace.
+
+### pending
+
+**Runs the test** and expects it to fail:
+
+```ruby
+it 'should work' do
+  pending 'feature not implemented'
+
+  # This code RUNS and is expected to fail
+  page.do_something  # Raises error
+  expect(result).to be_truthy
+end
+```
+
+**Output**: Test marked as pending with full error trace showing exactly what failed.
+
+## When to Use Each
+
+### Use `pending` for:
+
+1. **Browser limitations** - Features not yet supported by Firefox BiDi
+2. **Known failures** - Code exists but fails due to external issues
+3. **Documentation** - Want to show error trace to document what's missing
+
+### Use `skip` for:
+
+1. **Unimplemented features** - Code doesn't exist yet
+2. **Environment issues** - Test requires specific setup not available
+3. **Temporary exclusion** - Test is broken and needs fixing
+
+## Firefox BiDi Limitations
+
+For features that exist in BiDi spec but not yet implemented in Firefox, use `pending`:
+
+```ruby
+describe 'Page.setJavaScriptEnabled' do
+  it 'should work' do
+    # Pending: Firefox does not yet support emulation.setScriptingEnabled BiDi command
+    pending 'emulation.setScriptingEnabled not supported by Firefox yet'
+
+    with_test_state do |page:, **|
+      page.set_javascript_enabled(false)
+      expect(page.javascript_enabled?).to be false
+
+      page.goto('data:text/html, <script>var something = "forbidden"</script>')
+
+      error = nil
+      begin
+        page.evaluate('something')
+      rescue => e
+        error = e
+      end
+
+      expect(error).not_to be_nil
+      expect(error.message).to include('something is not defined')
+    end
+  end
+end
+```
+
+**Why pending, not skip**:
+- Code path exists (`page.set_javascript_enabled`)
+- BiDi command exists in spec (`emulation.setScriptingEnabled`)
+- Firefox just hasn't implemented it yet
+- Running the test shows exactly what error Firefox returns
+
+## Output Comparison
+
+### With `skip`
+
+```
+Page
+  Page.setJavaScriptEnabled
+    should work (SKIPPED)
+```
+
+No error information, no way to know what's missing.
+
+### With `pending`
+
+```
+Page
+  Page.setJavaScriptEnabled
+    should work (PENDING: emulation.setScriptingEnabled not supported by Firefox yet)
+
+Pending: (Failures listed here are expected and do not affect your suite's status)
+
+  1) Page Page.setJavaScriptEnabled should work
+     # emulation.setScriptingEnabled not supported by Firefox yet
+     Failure/Error: raise ProtocolError, "BiDi error (#{method}): #{result['error']['message']}"
+
+     Puppeteer::Bidi::Connection::ProtocolError:
+       BiDi error (emulation.setScriptingEnabled):
+     # ./lib/puppeteer/bidi/connection.rb:71:in 'send_command'
+     # ./lib/puppeteer/bidi/core/browsing_context.rb:331:in 'set_javascript_enabled'
+     # ./lib/puppeteer/bidi/page.rb:313:in 'set_javascript_enabled'
+```
+
+Full error trace shows:
+- Which BiDi command failed
+- Error message from Firefox
+- Complete stack trace
+- Where in our code it failed
+
+## Implementation Pattern
+
+### Before (Incorrect - Using skip in before block)
+
+```ruby
+describe 'Page.setJavaScriptEnabled' do
+  before do
+    skip 'emulation.setScriptingEnabled not supported by Firefox yet'
+  end
+
+  it 'should work' do
+    # Never runs
+  end
+
+  it 'setInterval should pause' do
+    # Never runs
+  end
+end
+```
+
+**Problems**:
+- Tests don't run at all
+- No error information
+- Not clear which BiDi command is missing
+
+### After (Correct - Using pending in individual tests)
+
+```ruby
+describe 'Page.setJavaScriptEnabled' do
+  it 'should work' do
+    pending 'emulation.setScriptingEnabled not supported by Firefox yet'
+
+    with_test_state do |page:, **|
+      # Test code runs and fails with proper error trace
+    end
+  end
+
+  it 'setInterval should pause' do
+    pending 'emulation.setScriptingEnabled not supported by Firefox yet'
+
+    with_test_state do |page:, **|
+      # Test code runs and fails with proper error trace
+    end
+  end
+end
+```
+
+**Benefits**:
+- Tests run and document exact failure
+- Each test can have specific pending message
+- Easy to identify when Firefox adds support (test will pass)
+- Full error trace available for debugging
+
+## Best Practices
+
+### 1. Be Specific in Pending Messages
+
+```ruby
+# Good
+pending 'emulation.setScriptingEnabled not supported by Firefox yet'
+
+# Bad
+pending 'not supported'
+```
+
+### 2. Include BiDi Command Name
+
+```ruby
+# Good
+pending 'browsingContext.setViewport not implemented'
+
+# Bad
+pending 'viewport not working'
+```
+
+### 3. Document When to Re-check
+
+```ruby
+# Good
+pending 'network.addIntercept requires Firefox 120+, current: 119'
+
+# Bad
+pending 'network interception broken'
+```
+
+### 4. Remove Pending When Fixed
+
+When Firefox adds support, the test will fail with:
+```
+Expected example to fail but it passed
+```
+
+This is your signal to remove the `pending` line!
+
+## Firefox BiDi Limitations (Current)
+
+As of this implementation, the following BiDi commands are not supported by Firefox:
+
+1. `emulation.setScriptingEnabled` - Control JavaScript execution
+   - Tests: `spec/integration/page_spec.rb` (2 tests)
+   - Tests: `spec/integration/click_spec.rb` (1 test)
+
+## Files Changed
+
+- `spec/integration/click_spec.rb`: Changed `skip` to `pending` (line 71)
+- `spec/integration/page_spec.rb`: Moved `skip` from before block to individual tests as `pending` (lines 19, 48)
+
+## Test Results
+
+```bash
+bundle exec rspec spec/integration/
+# 108 examples, 0 failures, 4 pending
+```
+
+All pending tests show proper error traces documenting Firefox limitations.
+
+## Key Takeaways
+
+1. **Use `pending` for browser limitations** - Shows what's missing with error trace
+2. **Use `skip` for unimplemented features** - Our code doesn't exist yet
+3. **Be specific in messages** - Include BiDi command name and reason
+4. **Pending in test body, not before block** - Each test should be explicit
+5. **Pending tests run code** - They document exact failure mode
+6. **Remove pending when fixed** - Test will fail with "expected to fail but passed"
+
+## References
+
+- [RSpec Documentation: Pending and Skipped Examples](https://rspec.info/features/3-12/rspec-core/pending-and-skipped-examples/)
+- [WebDriver BiDi Spec](https://w3c.github.io/webdriver-bidi/) - Check which commands are standardized
+- [Firefox BiDi Implementation Status](https://wiki.mozilla.org/WebDriver/RemoteProtocol/WebDriver_BiDi) - Check Firefox support
+
+## Commit Reference
+
+See commit: "test: Use pending instead of skip for Firefox unsupported features"