webtap-tool 0.1.1__py3-none-any.whl

webtap/cdp/README.md

@@ -0,0 +1,268 @@
+# Chrome DevTools Protocol (CDP) Integration
+
+This module handles the core CDP connection and event management for WebTap.
+
+## Overview
+
+The CDP module provides:
+- WebSocket connection to Chrome's debugging port
+- Event capture and storage in DuckDB
+- Dynamic field discovery for flexible querying
+- Native event storage (no transformation)
+
+## Architecture
+
+```
+Chrome Browser
+    ↓ (WebSocket)
+CDPSession (session.py)
+    ├── WebSocketApp (connection management)
+    ├── DuckDB (event storage)
+    └── Field Discovery (dynamic paths)
+         ↓
+Query Builder (query.py)
+    └── SQL Generation
+         ↓
+WebTap Commands
+```
+
+## Core Components
+
+### session.py
+The main CDP session manager:
+- Establishes WebSocket connection
+- Stores events as-is in DuckDB
+- Discovers field paths dynamically
+- Handles CDP command execution
+
+### query.py
+Dynamic query builder:
+- Fuzzy field matching
+- SQL generation for JSON queries
+- Cross-event correlation
+
+### schema/
+CDP protocol reference:
+- Protocol version information
+- Domain definitions (future)
+
+## Philosophy: Native Storage
+
+We store CDP events exactly as received:
+
+```python
+# CDP sends this
+{
+    "method": "Network.responseReceived",
+    "params": {
+        "requestId": "123.456",
+        "response": {
+            "status": 200,
+            "headers": {...}
+        }
+    }
+}
+
+# We store it as-is in DuckDB
+# No transformation, no data loss
+```
+
+## Event Domains
+
+Currently capturing events from:
+
+### Network Domain
+- `Network.requestWillBeSent`
+- `Network.responseReceived`
+- `Network.loadingFinished`
+- `Network.loadingFailed`
+
+### Page Domain
+- `Page.frameNavigated`
+- `Page.domContentEventFired`
+- `Page.loadEventFired`
+
+### Runtime Domain
+- `Runtime.consoleAPICalled`
+- `Runtime.exceptionThrown`
+
+### Fetch Domain
+- `Fetch.requestPaused`
+- `Fetch.authRequired`
+
+### Storage Domain
+- `Storage.cookiesChanged`
+- `Storage.cacheStorageContentUpdated`
+
+## Database Schema
+
+### events table
+```sql
+CREATE TABLE events (
+    rowid INTEGER PRIMARY KEY,
+    event JSON,
+    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+)
+```
+
+### Query Examples
+
+```sql
+-- Find all 404 responses
+SELECT * FROM events 
+WHERE json_extract_string(event, '$.params.response.status') = '404'
+
+-- Get request/response pairs
+SELECT 
+    e1.rowid as request_row,
+    e2.rowid as response_row,
+    json_extract_string(e1.event, '$.params.request.url') as url
+FROM events e1
+JOIN events e2 ON 
+    json_extract_string(e1.event, '$.params.requestId') = 
+    json_extract_string(e2.event, '$.params.requestId')
+WHERE 
+    json_extract_string(e1.event, '$.method') = 'Network.requestWillBeSent'
+    AND json_extract_string(e2.event, '$.method') = 'Network.responseReceived'
+```
+
+## Field Discovery
+
+The system automatically discovers all field paths:
+
+```python
+# When we see this event:
+{
+    "method": "Network.responseReceived",
+    "params": {
+        "response": {
+            "status": 200,
+            "url": "https://example.com"
+        }
+    }
+}
+
+# We discover these paths:
+# - method
+# - params.response.status
+# - params.response.url
+
+# Users can then query with fuzzy matching:
+events(status=200)  # Finds params.response.status
+events(url="example")  # Finds params.response.url
+```
+
+## Connection Management
+
+### Initialization
+```python
+cdp = CDPSession()
+await cdp.connect("localhost", 9222, page_id)
+```
+
+### Event Flow
+1. Chrome sends event over WebSocket
+2. CDPSession receives in `on_message()`
+3. Event stored in DuckDB immediately
+4. Field paths extracted for discovery
+5. Event available for querying
+
+## CDP Command Execution
+
+Direct command execution:
+```python
+# Get response body
+result = cdp.execute("Network.getResponseBody", {
+    "requestId": "123.456"
+})
+
+# Evaluate JavaScript
+result = cdp.execute("Runtime.evaluate", {
+    "expression": "document.title"
+})
+```
+
+## Performance Considerations
+
+- **Minimal Processing**: Events stored as-is
+- **Lazy Evaluation**: Field discovery on-demand
+- **Efficient Storage**: DuckDB's columnar format
+- **Fast Queries**: JSON functions optimized in DuckDB
+
+## Extension Points
+
+### Adding New Domains
+To capture events from additional CDP domains:
+
+1. Enable the domain:
+```python
+cdp.execute("DOMStorage.enable")
+```
+
+2. Events automatically captured and stored
+
+3. Query them:
+```python
+events(method="DOMStorage.*")
+```
+
+### Custom Event Processing
+While we store events as-is, you can add custom processors:
+
+```python
+def process_network_event(event):
+    # Custom logic here
+    pass
+
+# Register processor
+cdp.register_processor("Network.*", process_network_event)
+```
+
+## Integration with SDP
+
+The CDP module will work alongside the future SDP (Svelte Debug Protocol) module:
+
+```
+CDP Events (Network, DOM, Console)
+    +
+SDP Events (State, Components, Reactivity)
+    ↓
+Unified Event Stream in DuckDB
+    ↓
+Correlated Analysis
+```
+
+## Best Practices
+
+1. **Don't Transform**: Store CDP data as-is
+2. **Query Don't Parse**: Use SQL for extraction
+3. **Discover Don't Define**: Let field paths emerge
+4. **Correlate Don't Duplicate**: Link events by IDs
+
+## Debugging
+
+### Enable verbose logging
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+```
+
+### Check connection
+```python
+cdp.connected  # Should be True
+cdp.ws.sock.connected  # WebSocket status
+```
+
+### Inspect stored events
+```python
+cdp.query("SELECT COUNT(*) FROM events")
+cdp.query("SELECT * FROM events ORDER BY rowid DESC LIMIT 5")
+```
+
+## Future Enhancements
+
+- [ ] Event compression for long sessions
+- [ ] Streaming to external storage
+- [ ] Real-time event subscriptions
+- [ ] Custom domain definitions
+- [ ] Event replay functionality

webtap/cdp/__init__.py

@@ -0,0 +1,14 @@
+"""Chrome DevTools Protocol client with native event storage.
+
+Native CDP approach - store events as-is, query on-demand.
+Built on WebSocketApp + DuckDB for minimal overhead.
+
+PUBLIC API:
+  - CDPSession: Main CDP client with WebSocket connection and event storage
+  - build_query: Dynamic query builder with field discovery
+"""
+
+from webtap.cdp.query import build_query
+from webtap.cdp.session import CDPSession
+
+__all__ = ["CDPSession", "build_query"]

webtap/cdp/query.py

@@ -0,0 +1,107 @@
+"""Dynamic CDP query builder with field discovery.
+
+PUBLIC API:
+  - build_query: Build SQL queries with automatic field discovery
+"""
+
+
+def build_query(
+    session, query: dict, event_type: str | list[str] | None = None, limit: int = 20
+) -> tuple[str, dict[str, list[str]]]:
+    """Build SQL queries with automatic CDP field discovery.
+
+    Uses CDPSession's live field_paths lookup built from actual events.
+    Supports filtering, wildcard matching, and multi-field extraction.
+
+    Args:
+        session: CDPSession with field_paths lookup.
+        query: Field names and values - "*" extracts only, values filter.
+        event_type: Optional CDP event type(s) to filter.
+        limit: Maximum results. Defaults to 20.
+
+    Returns:
+        Tuple of (sql_query, discovered_fields_dict).
+
+    Examples:
+        build_query(session, {"url": "*"})  # Extract all URL fields
+        build_query(session, {"status": 200})  # Filter by status=200
+        build_query(session, {"url": "*youtube*", "status": 200})  # Multiple fields
+    """
+
+    # Field discovery using live field_paths
+    discovered = {}
+    for key in query.keys():
+        if key in ["limit"]:
+            continue
+        discovered[key] = session.discover_field_paths(key)
+
+    # Handle case where no fields found
+    if not any(discovered.values()):
+        return "SELECT NULL as no_fields_found FROM events LIMIT 0", discovered
+
+    # Build WHERE conditions
+    where_conditions = []
+
+    # Filter by CDP event type
+    if event_type:
+        if isinstance(event_type, str):
+            where_conditions.append(f"json_extract_string(event, '$.method') = '{event_type}'")
+        elif isinstance(event_type, list):
+            types_str = ", ".join(f"'{t}'" for t in event_type)
+            where_conditions.append(f"json_extract_string(event, '$.method') IN ({types_str})")
+
+    # Build field filters using discovered paths
+    for key, value in query.items():
+        if key in ["limit"] or value == "*":
+            continue
+
+        paths = discovered.get(key, [])
+        if not paths:
+            continue
+
+        # Create filter conditions for each path
+        path_conditions = []
+        for path in paths:
+            # Remove event type prefix for JSON path
+            actual_path = path.split(":", 1)[1] if ":" in path else path
+            json_path = "$." + actual_path
+
+            if isinstance(value, str):
+                # Convert wildcards to SQL LIKE patterns
+                if "*" in value or "?" in value:
+                    pattern = value.replace("*", "%").replace("?", "_")
+                else:
+                    pattern = value
+                path_conditions.append(f"json_extract_string(event, '{json_path}') LIKE '{pattern}'")
+            elif isinstance(value, (int, float)):
+                path_conditions.append(f"CAST(json_extract_string(event, '{json_path}') AS NUMERIC) = {value}")
+            elif isinstance(value, bool):
+                path_conditions.append(f"json_extract_string(event, '{json_path}') = '{str(value).lower()}'")
+            elif value is None:
+                path_conditions.append(f"json_extract_string(event, '{json_path}') IS NULL")
+
+        # OR conditions between different paths for same field
+        if path_conditions:
+            if len(path_conditions) == 1:
+                where_conditions.append(path_conditions[0])
+            else:
+                where_conditions.append(f"({' OR '.join(path_conditions)})")
+
+    # Build SELECT clause with rowid and discovered fields
+    select_parts = ["rowid"]
+    for key, paths in discovered.items():
+        for path in paths:
+            # Use actual path for JSON, full path for column alias
+            actual_path = path.split(":", 1)[1] if ":" in path else path
+            json_path = "$." + actual_path
+            select_parts.append(f"json_extract_string(event, '{json_path}') as \"{path}\"")
+
+    # Assemble final SQL query
+    sql = f"SELECT {', '.join(select_parts)} FROM events"
+
+    if where_conditions:
+        sql += " WHERE " + " AND ".join(where_conditions)
+
+    sql += f" ORDER BY rowid DESC LIMIT {limit}"
+
+    return sql, discovered

webtap/cdp/schema/README.md

@@ -0,0 +1,41 @@
+# Chrome DevTools Protocol Schema
+
+This directory contains the CDP protocol schema and version information fetched from Chrome.
+
+## Files
+
+- `cdp_protocol.json` - Full CDP protocol schema with all domains, commands, events, and types
+- `cdp_version.json` - Chrome version and protocol version information
+
+## Fetching Latest Schema
+
+To update these files with the latest protocol from your Chrome instance:
+
+```bash
+# Ensure Chrome is running with debugging enabled:
+# google-chrome --remote-debugging-port=9222
+
+# Fetch protocol schema
+curl -s http://localhost:9222/json/protocol > cdp_protocol.json
+
+# Fetch version info
+curl -s http://localhost:9222/json/version | jq '.' > cdp_version.json
+```
+
+## Using the Schema
+
+These files are useful for:
+- Understanding available CDP commands and their parameters
+- Debugging protocol issues
+- Validating command usage
+- Discovering new CDP features
+
+## Example: Finding Fetch Commands
+
+```bash
+# List all Fetch domain commands
+cat cdp_protocol.json | jq '.domains[] | select(.domain == "Fetch") | .commands[].name'
+
+# Get details for a specific command
+cat cdp_protocol.json | jq '.domains[] | select(.domain == "Fetch") | .commands[] | select(.name == "continueResponse")'
+```