webtap-tool 0.1.1__py3-none-any.whl

webtap/VISION.md

@@ -0,0 +1,234 @@
+# WebTap Vision: Work WITH Chrome DevTools Protocol
+
+## Core Philosophy
+
+**Store CDP events as-is. Transform minimally. Query on-demand.**
+
+Instead of transforming CDP's complex nested structures into our own models, we embrace CDP's native format. We store events mostly unchanged, extract minimal data for tables, and query additional data on-demand.
+
+## The Problem We're Solving
+
+CDP sends rich, nested event data. Previous approaches tried to:
+1. Transform everything into flat models
+2. Create abstraction layers over CDP
+3. Build complex query engines
+4. Format data for display
+
+This led to:
+- Loss of CDP's rich information
+- Complex transformation logic
+- Over-engineered abstractions
+- Unnecessary memory usage
+
+## The Solution: Native CDP Storage
+
+### 1. Store Events As-Is
+
+```python
+# CDP gives us this - we keep it!
+{
+    "method": "Network.responseReceived",
+    "params": {
+        "requestId": "123.456", 
+        "response": {
+            "status": 200,
+            "headers": {...},
+            "mimeType": "application/json",
+            "timing": {...}
+        }
+    }
+}
+```
+
+### 2. Minimal Summaries for Tables
+
+Extract only what's needed for table display:
+```python
+NetworkSummary(
+    id="123.456",
+    method="GET",
+    status=200,
+    url="https://api.example.com/data",
+    type="json",
+    size=1234
+)
+```
+
+Keep the full CDP events attached for detail views.
+
+### 3. On-Demand Queries
+
+Some data isn't in the event stream:
+- Response bodies: `Network.getResponseBody`
+- Cookies: `Storage.getCookies` 
+- LocalStorage: `DOMStorage.getDOMStorageItems`
+- JavaScript evaluation: `Runtime.evaluate`
+
+Query these when needed, not preemptively.
+
+## Architecture
+
+```
+                    ┌─────────────────┐
+                    │   Chrome Tab    │
+                    └────────┬────────┘
+                             │ CDP Events
+                    ┌────────▼────────┐
+                    │   WebSocket     │
+                    │  (WebSocketApp) │
+                    └────────┬────────┘
+                             │ Raw Events
+                    ┌────────▼────────┐
+                    │  DuckDB Storage │
+                    │  (events table) │
+                    └────────┬────────┘
+                             │ SQL Queries
+                ┌────────────┼────────────┐
+                │            │            │
+        ┌───────▼──────┐ ┌───▼───┐ ┌──────▼──────┐
+        │   Commands   │ │ Tables│ │Detail Views │
+        │network()     │ │       │ │             │
+        │console()     │ │Minimal│ │Full CDP Data│
+        │storage()     │ │Summary│ │+ On-Demand  │
+        └──────────────┘ └───────┘ └─────────────┘
+```
+
+## Data Flow Examples
+
+### Network Request Lifecycle
+
+```python
+# 1. Request sent - store as-is in DuckDB
+db.execute("INSERT INTO events VALUES (?)", [json.dumps({
+    "method": "Network.requestWillBeSent",
+    "params": {...}  # Full CDP data
+})])
+
+# 2. Response received - store as-is
+db.execute("INSERT INTO events VALUES (?)", [json.dumps({
+    "method": "Network.responseReceived", 
+    "params": {...}  # Full CDP data
+})])
+
+# 3. Query for table view - SQL on JSON
+db.execute("""
+    SELECT 
+        json_extract_string(event, '$.params.requestId') as id,
+        json_extract_string(event, '$.params.response.status') as status,
+        json_extract_string(event, '$.params.response.url') as url
+    FROM events 
+    WHERE json_extract_string(event, '$.method') = 'Network.responseReceived'
+""")
+
+# 4. Detail view - get all events for request
+db.execute("""
+    SELECT event FROM events 
+    WHERE json_extract_string(event, '$.params.requestId') = ?
+""", [request_id])
+
+# 5. Body fetch - on-demand CDP call
+cdp.execute("Network.getResponseBody", {"requestId": "123.456"})
+```
+
+### Console Message
+
+```python
+# 1. Store as-is
+console_events.append({
+    "method": "Runtime.consoleAPICalled",
+    "params": {
+        "type": "error",
+        "args": [{"type": "string", "value": "Failed to fetch"}],
+        "stackTrace": {...}
+    }
+})
+
+# 2. Table view - minimal summary
+ConsoleSummary(
+    id="console-123",
+    level="error",
+    message="Failed to fetch",
+    source="console"
+)
+
+# 3. Detail view - full CDP data
+{
+    "summary": summary,
+    "raw": console_events[i],  # Full CDP event with stack trace
+}
+```
+
+## Benefits
+
+1. **No Information Loss** - Full CDP data always available
+2. **Minimal Memory** - Only store what CDP sends
+3. **Simple Code** - No complex transformations
+4. **Fast Tables** - Minimal summaries render quickly
+5. **Rich Details** - Full CDP data for debugging
+6. **On-Demand Loading** - Expensive operations only when needed
+7. **Future Proof** - New CDP features automatically available
+
+## Implementation Principles
+
+### DO:
+- Store CDP events as-is
+- Build minimal summaries for tables
+- Query additional data on-demand
+- Group events by correlation ID (requestId)
+- Let Replkit2 handle display
+
+### DON'T:
+- Transform CDP structure unnecessarily
+- Fetch data preemptively
+- Create abstraction layers
+- Build complex query engines
+- Format data for display
+
+## File Structure
+
+```
+webtap/
+├── VISION.md           # This file
+├── __init__.py         # Module initialization + API server startup
+├── app.py              # REPL app with WebTapState
+├── api.py              # FastAPI server for Chrome extension
+├── filters.py          # Filter management system
+├── cdp/
+│   ├── __init__.py
+│   ├── session.py      # CDPSession with DuckDB storage
+│   ├── query.py        # Dynamic query builder
+│   └── schema/         # CDP protocol reference
+│       ├── README.md
+│       └── cdp_version.json
+├── services/           # Service layer (business logic)
+│   ├── __init__.py
+│   ├── main.py         # WebTapService orchestrator
+│   ├── network.py      # Network request handling
+│   ├── console.py      # Console message handling
+│   ├── fetch.py        # Request interception
+│   └── body.py         # Response body caching
+└── commands/           # Thin command wrappers
+    ├── __init__.py
+    ├── _errors.py      # Unified error handling
+    ├── _markdown.py    # Markdown elements
+    ├── _symbols.py     # ASCII symbol registry
+    ├── _utils.py       # Shared utilities
+    ├── connection.py   # connect, disconnect, pages
+    ├── navigation.py   # navigate, reload, back, forward
+    ├── network.py      # network() command
+    ├── console.py      # console() command
+    ├── events.py       # events() dynamic querying
+    ├── inspect.py      # inspect() event details
+    ├── javascript.py   # js() execution
+    ├── body.py         # body() response inspection
+    ├── fetch.py        # fetch(), requests(), resume()
+    └── filters.py      # filters() management
+```
+
+## Success Metrics
+
+- **Lines of Code**: < 500 (excluding commands)
+- **Transformation Logic**: < 100 lines
+- **Memory Usage**: Only what CDP sends
+- **Response Time**: Instant for tables, < 100ms for details
+- **CDP Coverage**: 100% of CDP data accessible

webtap/__init__.py

@@ -0,0 +1,56 @@
+"""WebTap - Chrome DevTools Protocol REPL.
+
+Main entry point for WebTap browser debugging tool. Provides both REPL and MCP
+functionality for Chrome DevTools Protocol interaction with native CDP event
+storage and on-demand querying.
+
+PUBLIC API:
+  - app: Main ReplKit2 App instance
+  - main: Entry point function for CLI
+"""
+
+import sys
+import logging
+
+from webtap.app import app
+from webtap.api import start_api_server
+
+logger = logging.getLogger(__name__)
+
+
+def main():
+    """Entry point for the WebTap REPL.
+
+    Starts in one of three modes:
+    - CLI mode (with --cli flag) for command-line interface
+    - MCP mode (with --mcp flag) for Model Context Protocol server
+    - REPL mode (default) for interactive shell
+
+    In REPL and MCP modes, the API server is started for Chrome extension
+    integration. The API server runs in background to handle extension requests.
+    """
+    # Start API server for Chrome extension (except in CLI mode)
+    if "--cli" not in sys.argv:
+        _start_api_server_safely()
+
+    if "--mcp" in sys.argv:
+        app.mcp.run()
+    elif "--cli" in sys.argv:
+        # Remove --cli from argv before passing to Typer
+        sys.argv.remove("--cli")
+        app.cli()  # Run CLI mode via Typer
+    else:
+        # Run REPL
+        app.run(title="WebTap - Chrome DevTools Protocol REPL")
+
+
+def _start_api_server_safely():
+    """Start API server with error handling."""
+    try:
+        start_api_server(app.state)
+        logger.info("API server started on http://localhost:8765")
+    except Exception as e:
+        logger.warning(f"Failed to start API server: {e}")
+
+
+__all__ = ["app", "main"]

webtap/api.py

@@ -0,0 +1,222 @@
+"""FastAPI endpoints for WebTap browser extension.
+
+PUBLIC API:
+  - start_api_server: Start API server in background thread
+"""
+
+import logging
+import threading
+from typing import Any, Dict
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+import uvicorn
+
+
+logger = logging.getLogger(__name__)
+
+
+# Request models
+class ConnectRequest(BaseModel):
+    """Request model for connecting to a Chrome page."""
+
+    page_id: str
+
+
+class FetchRequest(BaseModel):
+    """Request model for enabling/disabling fetch interception."""
+
+    enabled: bool
+    response_stage: bool = False  # Optional: also pause at Response stage
+
+
+# Create FastAPI app
+api = FastAPI(title="WebTap API", version="0.1.0")
+
+# Enable CORS for extension
+api.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],  # Chrome extensions have unique origins
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+
+# Global reference to WebTap state (set by start_api_server)
+app_state = None
+
+
+@api.get("/pages")
+async def list_pages() -> Dict[str, Any]:
+    """List available Chrome pages for extension selection."""
+    if not app_state:
+        return {"error": "WebTap not initialized", "pages": []}
+
+    return app_state.service.list_pages()
+
+
+@api.get("/status")
+async def get_status() -> Dict[str, Any]:
+    """Get current connection status and event count."""
+    if not app_state:
+        return {"connected": False, "error": "WebTap not initialized", "events": 0}
+
+    return app_state.service.get_status()
+
+
+@api.post("/connect")
+async def connect(request: ConnectRequest) -> Dict[str, Any]:
+    """Connect to a Chrome page by stable page ID."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    return app_state.service.connect_to_page(page_id=request.page_id)
+
+
+@api.post("/disconnect")
+async def disconnect() -> Dict[str, Any]:
+    """Disconnect from currently connected page."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    return app_state.service.disconnect()
+
+
+@api.post("/clear")
+async def clear_events() -> Dict[str, Any]:
+    """Clear all stored events from DuckDB."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    return app_state.service.clear_events()
+
+
+@api.post("/fetch")
+async def set_fetch_interception(request: FetchRequest) -> Dict[str, Any]:
+    """Enable or disable fetch request interception."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    if request.enabled:
+        result = app_state.service.fetch.enable(app_state.service.cdp, response_stage=request.response_stage)
+    else:
+        result = app_state.service.fetch.disable()
+    return result
+
+
+@api.get("/fetch/paused")
+async def get_paused_requests() -> Dict[str, Any]:
+    """Get list of currently paused fetch requests."""
+    if not app_state:
+        return {"error": "WebTap not initialized", "requests": []}
+
+    fetch_service = app_state.service.fetch
+    if not fetch_service.enabled:
+        return {"enabled": False, "requests": []}
+
+    paused_list = fetch_service.get_paused_list()
+    return {
+        "enabled": True,
+        "requests": paused_list,
+        "count": len(paused_list),
+        "response_stage": fetch_service.enable_response_stage,
+    }
+
+
+@api.get("/filters/status")
+async def get_filter_status() -> Dict[str, Any]:
+    """Get current filter configuration and enabled categories."""
+    if not app_state:
+        return {"error": "WebTap not initialized", "filters": {}, "enabled": []}
+
+    fm = app_state.service.filters
+    return {"filters": fm.filters, "enabled": list(fm.enabled_categories), "path": str(fm.filter_path)}
+
+
+@api.post("/filters/toggle/{category}")
+async def toggle_filter_category(category: str) -> Dict[str, Any]:
+    """Toggle a specific filter category on or off."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    fm = app_state.service.filters
+
+    if category not in fm.filters:
+        return {"error": f"Category '{category}' not found"}
+
+    if category in fm.enabled_categories:
+        fm.enabled_categories.discard(category)
+        enabled = False
+    else:
+        fm.enabled_categories.add(category)
+        enabled = True
+
+    fm.save()
+
+    return {"category": category, "enabled": enabled, "total_enabled": len(fm.enabled_categories)}
+
+
+@api.post("/filters/enable-all")
+async def enable_all_filters() -> Dict[str, Any]:
+    """Enable all available filter categories."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    fm = app_state.service.filters
+    fm.set_enabled_categories(None)
+    fm.save()
+
+    return {"enabled": list(fm.enabled_categories), "total": len(fm.enabled_categories)}
+
+
+@api.post("/filters/disable-all")
+async def disable_all_filters() -> Dict[str, Any]:
+    """Disable all filter categories."""
+    if not app_state:
+        return {"error": "WebTap not initialized"}
+
+    fm = app_state.service.filters
+    fm.set_enabled_categories([])
+    fm.save()
+
+    return {"enabled": [], "total": 0}
+
+
+def start_api_server(state, host: str = "127.0.0.1", port: int = 8765):
+    """Start the API server in a background thread.
+
+    Args:
+        state: WebTapState instance from the main app.
+        host: Host to bind to. Defaults to 127.0.0.1.
+        port: Port to bind to. Defaults to 8765.
+
+    Returns:
+        Thread instance running the server.
+    """
+    global app_state
+    app_state = state
+
+    thread = threading.Thread(target=run_server, args=(host, port), daemon=True)
+    thread.start()
+
+    logger.info(f"API server started on http://{host}:{port}")
+    return thread
+
+
+def run_server(host: str, port: int):
+    """Run the FastAPI server in a thread."""
+    try:
+        uvicorn.run(
+            api,
+            host=host,
+            port=port,
+            log_level="error",
+            access_log=False,
+        )
+    except Exception as e:
+        logger.error(f"API server failed: {e}")
+
+
+__all__ = ["start_api_server"]

webtap/app.py

@@ -0,0 +1,76 @@
+"""Main application entry point for WebTap browser debugger.
+
+PUBLIC API:
+  - WebTapState: Application state class with CDP session and service
+  - app: Main ReplKit2 App instance (imported by commands and __init__)
+"""
+
+import sys
+from dataclasses import dataclass, field
+
+from replkit2 import App
+
+from webtap.cdp import CDPSession
+from webtap.services import WebTapService
+
+
+@dataclass
+class WebTapState:
+    """Application state for WebTap browser debugging.
+
+    Maintains CDP session and connection state for browser interaction.
+    All data is stored in DuckDB via the CDP session - no caching needed.
+
+    Attributes:
+        cdp: Chrome DevTools Protocol session instance.
+        service: WebTapService orchestrating all domain services.
+    """
+
+    cdp: CDPSession = field(default_factory=CDPSession)
+    service: WebTapService = field(init=False)
+
+    def __post_init__(self):
+        """Initialize service with self reference after dataclass init."""
+        self.service = WebTapService(self)
+
+
+# Must be created before command imports for decorator registration
+app = App(
+    "webtap",
+    WebTapState,
+    uri_scheme="webtap",
+    fastmcp={
+        "description": "Chrome DevTools Protocol debugger",
+        "tags": {"browser", "debugging", "chrome", "cdp"},
+    },
+    typer_config={
+        "add_completion": False,  # Hide shell completion options
+        "help": "WebTap - Chrome DevTools Protocol CLI",
+    },
+)
+
+# Command imports trigger @app.command decorator registration
+if "--cli" in sys.argv:
+    # Only import CLI-compatible commands (no dict/list parameters)
+    from webtap.commands import setup  # noqa: E402, F401
+    from webtap.commands import launch  # noqa: E402, F401
+else:
+    # Import all commands for REPL/MCP mode
+    from webtap.commands import connection  # noqa: E402, F401
+    from webtap.commands import navigation  # noqa: E402, F401
+    from webtap.commands import javascript  # noqa: E402, F401
+    from webtap.commands import network  # noqa: E402, F401
+    from webtap.commands import console  # noqa: E402, F401
+    from webtap.commands import events  # noqa: E402, F401
+    from webtap.commands import filters  # noqa: E402, F401
+    from webtap.commands import inspect  # noqa: E402, F401
+    from webtap.commands import fetch  # noqa: E402, F401
+    from webtap.commands import body  # noqa: E402, F401
+    from webtap.commands import setup  # noqa: E402, F401
+    from webtap.commands import launch  # noqa: E402, F401
+
+
+# Entry point is in __init__.py:main() as specified in pyproject.toml
+
+
+__all__ = ["WebTapState", "app"]