webtap-tool 0.11.0__py3-none-any.whl

webtap/VISION.md

@@ -0,0 +1,246 @@
+# WebTap Vision: Work WITH Chrome DevTools Protocol
+
+## Core Philosophy
+
+**Store CDP events as-is. Transform minimally. Query on-demand.**
+
+Instead of transforming CDP's complex nested structures into our own models, we embrace CDP's native format. We store events mostly unchanged, extract minimal data for tables, and query additional data on-demand.
+
+## The Problem We're Solving
+
+CDP sends rich, nested event data. Previous approaches tried to:
+1. Transform everything into flat models
+2. Create abstraction layers over CDP
+3. Build complex query engines
+4. Format data for display
+
+This led to:
+- Loss of CDP's rich information
+- Complex transformation logic
+- Over-engineered abstractions
+- Unnecessary memory usage
+
+## The Solution: Native CDP Storage
+
+### 1. Store Events As-Is
+
+```python
+# CDP gives us this - we keep it!
+{
+    "method": "Network.responseReceived",
+    "params": {
+        "requestId": "123.456", 
+        "response": {
+            "status": 200,
+            "headers": {...},
+            "mimeType": "application/json",
+            "timing": {...}
+        }
+    }
+}
+```
+
+### 2. Minimal Summaries for Tables
+
+Extract only what's needed for table display:
+```python
+NetworkSummary(
+    id="123.456",
+    method="GET",
+    status=200,
+    url="https://api.example.com/data",
+    type="json",
+    size=1234
+)
+```
+
+Keep the full CDP events attached for detail views.
+
+### 3. On-Demand Queries
+
+Some data isn't in the event stream:
+- Response bodies: `Network.getResponseBody`
+- Cookies: `Storage.getCookies` 
+- LocalStorage: `DOMStorage.getDOMStorageItems`
+- JavaScript evaluation: `Runtime.evaluate`
+
+Query these when needed, not preemptively.
+
+## Architecture
+
+```
+                    ┌─────────────────┐
+                    │   Chrome Tab    │
+                    └────────┬────────┘
+                             │ CDP Events
+                    ┌────────▼────────┐
+                    │   WebSocket     │
+                    │  (WebSocketApp) │
+                    └────────┬────────┘
+                             │ Raw Events
+                    ┌────────▼────────┐
+                    │  DuckDB Storage │
+                    │  (events table) │
+                    └────────┬────────┘
+                             │ SQL Queries
+                ┌────────────┼────────────┐
+                │            │            │
+        ┌───────▼──────┐ ┌───▼───┐ ┌──────▼──────┐
+        │   Commands   │ │ Tables│ │Detail Views │
+        │network()     │ │       │ │             │
+        │console()     │ │Minimal│ │Full CDP Data│
+        │storage()     │ │Summary│ │+ On-Demand  │
+        └──────────────┘ └───────┘ └─────────────┘
+```
+
+## Data Flow Examples
+
+### Network Request Lifecycle
+
+```python
+# 1. Request sent - store as-is in DuckDB
+db.execute("INSERT INTO events VALUES (?)", [json.dumps({
+    "method": "Network.requestWillBeSent",
+    "params": {...}  # Full CDP data
+})])
+
+# 2. Response received - store as-is
+db.execute("INSERT INTO events VALUES (?)", [json.dumps({
+    "method": "Network.responseReceived", 
+    "params": {...}  # Full CDP data
+})])
+
+# 3. Query for table view - SQL on JSON
+db.execute("""
+    SELECT 
+        json_extract_string(event, '$.params.requestId') as id,
+        json_extract_string(event, '$.params.response.status') as status,
+        json_extract_string(event, '$.params.response.url') as url
+    FROM events 
+    WHERE json_extract_string(event, '$.method') = 'Network.responseReceived'
+""")
+
+# 4. Detail view - get all events for request
+db.execute("""
+    SELECT event FROM events 
+    WHERE json_extract_string(event, '$.params.requestId') = ?
+""", [request_id])
+
+# 5. Body fetch - on-demand CDP call
+cdp.execute("Network.getResponseBody", {"requestId": "123.456"})
+```
+
+### Console Message
+
+```python
+# 1. Store as-is
+console_events.append({
+    "method": "Runtime.consoleAPICalled",
+    "params": {
+        "type": "error",
+        "args": [{"type": "string", "value": "Failed to fetch"}],
+        "stackTrace": {...}
+    }
+})
+
+# 2. Table view - minimal summary
+ConsoleSummary(
+    id="console-123",
+    level="error",
+    message="Failed to fetch",
+    source="console"
+)
+
+# 3. Detail view - full CDP data
+{
+    "summary": summary,
+    "raw": console_events[i],  # Full CDP event with stack trace
+}
+```
+
+## Benefits
+
+1. **No Information Loss** - Full CDP data always available
+2. **Minimal Memory** - Only store what CDP sends
+3. **Simple Code** - No complex transformations
+4. **Fast Tables** - Minimal summaries render quickly
+5. **Rich Details** - Full CDP data for debugging
+6. **On-Demand Loading** - Expensive operations only when needed
+7. **Future Proof** - New CDP features automatically available
+
+## Implementation Principles
+
+### DO:
+- Store CDP events as-is
+- Build minimal summaries for tables
+- Query additional data on-demand
+- Group events by correlation ID (requestId)
+- Let Replkit2 handle display
+
+### DON'T:
+- Transform CDP structure unnecessarily
+- Fetch data preemptively
+- Create abstraction layers
+- Build complex query engines
+- Format data for display
+
+## File Structure
+
+```
+webtap/
+├── VISION.md           # This file
+├── __init__.py         # Module initialization
+├── app.py              # REPL app with WebTapState
+├── client.py           # RPCClient for JSON-RPC communication
+├── daemon.py           # Background daemon process
+├── filters.py          # Filter management system
+├── api/                # FastAPI server (runs in daemon)
+│   ├── __init__.py
+│   ├── server.py       # Server initialization & RPC wiring
+│   ├── state.py        # State snapshot for SSE broadcasts
+│   └── sse.py          # Server-sent events for live updates
+├── rpc/                # JSON-RPC 2.0 framework
+│   ├── __init__.py
+│   ├── framework.py    # RPCFramework class & method registration
+│   ├── machine.py      # ConnectionMachine (state transitions)
+│   ├── handlers.py     # All RPC method handlers
+│   └── errors.py       # RPCError and error codes
+├── cdp/
+│   ├── __init__.py
+│   ├── session.py      # CDPSession with DuckDB storage
+│   ├── har.py          # HAR view aggregation
+│   └── schema/         # CDP protocol reference
+│       └── README.md
+├── services/           # Service layer (business logic)
+│   ├── __init__.py
+│   ├── main.py         # WebTapService orchestrator
+│   ├── network.py      # Network request handling
+│   ├── console.py      # Console message handling
+│   ├── fetch.py        # Request interception
+│   └── dom.py          # DOM inspection & selection
+└── commands/           # Thin command wrappers
+    ├── __init__.py
+    ├── _builders.py    # Response builders & validators
+    ├── _utils.py       # Shared utilities & expression eval
+    ├── _code_generation.py  # JSON/code generation helpers
+    ├── _tips.py        # Documentation parser (TIPS.md)
+    ├── connection.py   # connect, disconnect, pages, clear
+    ├── navigation.py   # navigate, reload, back, forward
+    ├── network.py      # network() command
+    ├── console.py      # console() command
+    ├── request.py      # request() field selection + expr
+    ├── javascript.py   # js() execution
+    ├── fetch.py        # fetch(), requests(), resume(), fail()
+    ├── filters.py      # filters() management
+    ├── selections.py   # selections/browser() element selection
+    ├── to_model.py     # to_model() Pydantic generation
+    └── quicktype.py    # quicktype() type generation
+```
+
+## Success Metrics
+
+- **Lines of Code**: < 500 (excluding commands)
+- **Transformation Logic**: < 100 lines
+- **Memory Usage**: Only what CDP sends
+- **Response Time**: Instant for tables, < 100ms for details
+- **CDP Coverage**: 100% of CDP data accessible

webtap/__init__.py

@@ -0,0 +1,84 @@
+"""WebTap - Chrome DevTools Protocol REPL.
+
+Main entry point for WebTap browser debugging tool. Provides both REPL and MCP
+functionality for Chrome DevTools Protocol interaction with native CDP event
+storage and on-demand querying.
+
+PUBLIC API:
+  - app: Main ReplKit2 App instance
+  - main: Entry point function for CLI
+"""
+
+import atexit
+import sys
+
+from webtap.app import app
+
+# Register cleanup on exit to shutdown DB thread
+atexit.register(lambda: app.state.cleanup() if hasattr(app, "state") and app.state else None)
+
+
+def main():
+    """Entry point for WebTap.
+
+    Starts in one of five modes:
+    - Daemon mode (with --daemon flag):
+      - No args: Start daemon in foreground
+      - stop: Stop running daemon
+      - status: Show daemon status
+    - CLI mode (with --cli flag) for command-line interface
+    - MCP mode (with --mcp flag) for Model Context Protocol server
+    - REPL mode (default) for interactive shell
+
+    In REPL and MCP modes, the daemon is automatically started if not running.
+    CLI mode doesn't need the daemon (only for setup commands).
+    """
+    # Handle daemon management
+    if "--daemon" in sys.argv:
+        from webtap.daemon import start_daemon, stop_daemon, daemon_status
+
+        # Check for subcommands
+        if "stop" in sys.argv:
+            try:
+                stop_daemon()
+                print("Daemon stopped")
+            except RuntimeError as e:
+                print(f"Error: {e}")
+                sys.exit(1)
+        elif "status" in sys.argv:
+            status = daemon_status()
+            if status["running"]:
+                print(f"Daemon running (pid: {status['pid']})")
+                if status.get("connected"):
+                    print(f"Connected to: {status.get('page_title', 'Unknown')}")
+                    print(f"Events: {status.get('event_count', 0)}")
+                else:
+                    print("Not connected to any page")
+            else:
+                print("Daemon not running")
+                if status.get("error"):
+                    print(f"Error: {status['error']}")
+        else:
+            # Start daemon in foreground
+            start_daemon()
+        return
+
+    # CLI mode doesn't need daemon
+    if "--cli" in sys.argv:
+        sys.argv.remove("--cli")
+        app.cli()
+        return
+
+    # REPL and MCP modes need daemon
+    from webtap.daemon import ensure_daemon
+
+    ensure_daemon()
+
+    if "--mcp" in sys.argv:
+        app.mcp.run()
+    else:
+        # Run REPL
+        app.run(title="WebTap - Chrome DevTools Protocol REPL")
+
+
+__all__ = ["app", "main"]

webtap/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for python -m webtap."""
+
+from webtap import main
+
+if __name__ == "__main__":
+    main()

webtap/api/__init__.py

@@ -0,0 +1,9 @@
+"""WebTap API package.
+
+PUBLIC API:
+  - run_daemon_server: Run daemon server (blocking)
+"""
+
+from webtap.api.server import run_daemon_server
+
+__all__ = ["run_daemon_server"]

webtap/api/app.py

@@ -0,0 +1,26 @@
+"""FastAPI application and shared state.
+
+PUBLIC API:
+  - api: FastAPI application instance
+  - app_state: Global reference to DaemonState (set by server.py on startup)
+"""
+
+from typing import Any
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+
+# Create FastAPI app
+api = FastAPI(title="WebTap API", version="0.1.0")
+
+# Enable CORS for extension
+api.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],  # Chrome extensions have unique origins
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Global reference to WebTap state (set by server.py on startup)
+app_state: Any | None = None

webtap/api/models.py

@@ -0,0 +1,69 @@
+"""Pydantic request models for API endpoints.
+
+PUBLIC API:
+  - ConnectRequest: Chrome page connection parameters
+  - FetchRequest: Fetch interception configuration
+  - CDPRequest: CDP command relay parameters
+  - ResumeRequest: Resume paused request parameters
+  - FailRequest: Fail paused request parameters
+  - FulfillRequest: Fulfill paused request with custom response
+"""
+
+from typing import Any, Dict
+
+from pydantic import BaseModel
+
+
+class ConnectRequest(BaseModel):
+    """Request model for connecting to a Chrome page.
+
+    Supports either page index (for REPL/MCP) or page_id (for extension).
+    """
+
+    page: int | None = None  # Page index (0-based)
+    page_id: str | None = None  # Page ID from Chrome
+
+
+class FetchRequest(BaseModel):
+    """Request model for enabling/disabling fetch interception."""
+
+    enabled: bool
+    response_stage: bool = False
+
+
+class CDPRequest(BaseModel):
+    """Request model for CDP command relay."""
+
+    method: str
+    params: Dict[str, Any] = {}
+
+
+class ResumeRequest(BaseModel):
+    """Request model for resuming a paused request."""
+
+    rowid: int
+    modifications: Dict[str, Any] = {}
+    wait: float = 0.5
+
+
+class FailRequest(BaseModel):
+    """Request model for failing a paused request."""
+
+    rowid: int
+    reason: str = "BlockedByClient"
+
+
+class FulfillRequest(BaseModel):
+    """Request model for fulfilling a paused request with custom response."""
+
+    rowid: int
+    response_code: int = 200
+    response_headers: list[dict[str, str]] = []
+    body: str = ""
+
+
+class ClearRequest(BaseModel):
+    """Request model for clearing data stores."""
+
+    events: bool = True
+    console: bool = False

webtap/api/server.py

@@ -0,0 +1,111 @@
+"""Daemon server lifecycle management."""
+
+import asyncio
+import logging
+
+import uvicorn
+
+from webtap.api.app import api
+from webtap.api.sse import broadcast_processor, get_broadcast_queue, set_broadcast_ready_event, router as sse_router
+from webtap.daemon_state import DaemonState
+from webtap.rpc import RPCFramework
+from webtap.rpc.handlers import register_handlers
+
+logger = logging.getLogger(__name__)
+
+
+def run_daemon_server(host: str = "127.0.0.1", port: int = 8765):
+    """Run daemon server in foreground (blocking).
+
+    This function is called by daemon.py when running in --daemon mode.
+    It initializes daemon state with CDPSession and WebTapService,
+    then runs the API server.
+
+    Args:
+        host: Host to bind to
+        port: Port to bind to
+    """
+    import os
+    import webtap.api.app as app_module
+    from fastapi import Request
+
+    # Initialize daemon state
+    app_module.app_state = DaemonState()
+    logger.info("Daemon initialized with CDPSession and WebTapService")
+
+    # Initialize RPC framework and register handlers
+    rpc = RPCFramework(app_module.app_state.service)
+    register_handlers(rpc)
+    app_module.app_state.service.rpc = rpc
+    logger.info("RPC framework initialized with 22 handlers")
+
+    # Add single RPC endpoint
+    @api.post("/rpc")
+    async def handle_rpc(request: Request) -> dict:
+        """Handle JSON-RPC 2.0 requests."""
+        body = await request.json()
+        return await rpc.handle(body)
+
+    # Add health check endpoint
+    @api.get("/health")
+    async def health_check() -> dict:
+        """Quick health check endpoint for extension."""
+        return {"status": "ok", "pid": os.getpid()}
+
+    # Include SSE endpoint
+    api.include_router(sse_router)
+
+    async def run():
+        """Run server with proper shutdown handling."""
+        config = uvicorn.Config(
+            api,
+            host=host,
+            port=port,
+            log_level="warning",
+            access_log=False,
+        )
+        server = uvicorn.Server(config)
+
+        # Create event for broadcast processor ready signal
+        ready_event = asyncio.Event()
+        set_broadcast_ready_event(ready_event)
+
+        # Start broadcast processor in background
+        broadcast_task = asyncio.create_task(broadcast_processor())
+
+        # Wait for processor to be ready (with timeout)
+        try:
+            await asyncio.wait_for(ready_event.wait(), timeout=5.0)
+        except asyncio.TimeoutError:
+            logger.error("Broadcast processor failed to start")
+            broadcast_task.cancel()
+            return
+
+        # Wire broadcast queue to service
+        queue = get_broadcast_queue()
+        if queue and app_module.app_state:
+            app_module.app_state.service.set_broadcast_queue(queue)
+            logger.debug("Broadcast queue wired to WebTapService")
+
+        try:
+            await server.serve()
+        except (SystemExit, KeyboardInterrupt):
+            pass
+        finally:
+            if not broadcast_task.done():
+                broadcast_task.cancel()
+                try:
+                    await broadcast_task
+                except asyncio.CancelledError:
+                    pass
+
+    try:
+        asyncio.run(run())
+    except (SystemExit, KeyboardInterrupt):
+        pass
+    except Exception as e:
+        logger.error(f"Daemon server failed: {e}")
+    finally:
+        if app_module.app_state:
+            app_module.app_state.cleanup()
+        logger.info("Daemon cleanup complete")