webtap-tool 0.1.1__tar.gz

webtap_tool-0.1.1/.gitignore

@@ -0,0 +1,3 @@
+# Ignore previous version files and directories
+*.v[0-9]
+*.v[0-9][0-9]

webtap_tool-0.1.1/ARCHITECTURE.md

@@ -0,0 +1,150 @@
+# WebTap Architecture
+
+Implementation guide for WebTap commands following the VISION.
+
+## Core Components
+
+### CDPSession (cdp/session.py)
+- WebSocket connection to Chrome
+- DuckDB in-memory storage: `CREATE TABLE events (event JSON)`
+- Events stored AS-IS: `INSERT INTO events VALUES (?)`
+- Query interface: `query(sql)` - returns result rows
+- Body fetching: `fetch_body(request_id)` - CDP call on-demand
+
+### Command Pattern
+
+Commands query DuckDB and return data for Replkit2 display.
+
+```python
+@app.command
+def network(state, query: dict = None):
+    """Query network events with flexible filtering."""
+    
+    # Default query
+    default = {
+        'limit': 20,
+        'exclude_static': True,  # Skip images/fonts
+        'exclude_tracking': True  # Skip analytics
+    }
+    q = {**default, **(query or {})}
+    
+    # Build SQL from query dict
+    sql = build_network_sql(q)
+    
+    # Return for Replkit2 display
+    return state.cdp.query(sql)
+```
+
+## Command Implementation Guide
+
+### network(query: dict)
+```python
+# Query dict can contain:
+# - id: Single request detail
+# - status: Filter by status code
+# - method: Filter by HTTP method
+# - url_contains: Substring match
+# - limit: Result limit
+# - exclude_static: Hide images/css/fonts
+# - exclude_tracking: Hide analytics
+
+# Build SQL:
+SELECT 
+    json_extract_string(event, '$.params.requestId') as id,
+    json_extract_string(event, '$.params.response.status') as status,
+    json_extract_string(event, '$.params.response.url') as url
+FROM events
+WHERE json_extract_string(event, '$.method') = 'Network.responseReceived'
+    AND [additional filters from query dict]
+LIMIT 20
+```
+
+### console(query: dict)
+```python
+# Query dict can contain:
+# - level: 'error', 'warn', 'log'
+# - source: 'console', 'network', 'security'
+# - contains: Text search in message
+# - limit: Result limit
+
+# Build SQL:
+SELECT 
+    json_extract_string(event, '$.params.type') as level,
+    json_extract_string(event, '$.params.args[0].value') as message,
+    json_extract_string(event, '$.params.timestamp') as time
+FROM events
+WHERE json_extract_string(event, '$.method') IN ('Runtime.consoleAPICalled', 'Log.entryAdded')
+    AND [additional filters]
+```
+
+### body(id: str, expr: str = None)
+```python
+# Fetch body on-demand
+result = state.cdp.fetch_body(id)
+
+if not expr:
+    return result['body']  # Raw body
+
+# Evaluate Python expression on body (like inspect command)
+context = {
+    'data': result['body'],
+    'json': json.loads(result['body']) if parseable,
+    're': __import__('re')
+}
+return eval(expr, {}, context)
+```
+
+### inspect(query: dict, expr: str)
+```python
+# Query events then apply Python expression
+events = state.cdp.query(build_sql(query))
+
+# Apply expression to each event
+results = []
+for event in events:
+    context = {'event': json.loads(event[0])}
+    results.append(eval(expr, {}, context))
+return results
+```
+
+## SQL Patterns
+
+### Fuzzy field matching
+```sql
+-- Find any field containing 'status'
+SELECT * FROM events 
+WHERE json_extract_string(event, '$.params.response.status') = '404'
+   OR json_extract_string(event, '$.params.status') = '404'
+```
+
+### Correlation by requestId
+```sql
+-- Get all events for a request
+SELECT event FROM events
+WHERE json_extract_string(event, '$.params.requestId') = ?
+ORDER BY rowid
+```
+
+### Exclude noise
+```sql
+-- Skip tracking/analytics
+WHERE json_extract_string(event, '$.params.request.url') NOT LIKE '%google-analytics%'
+  AND json_extract_string(event, '$.params.request.url') NOT LIKE '%doubleclick%'
+  AND json_extract_string(event, '$.params.type') NOT IN ('Image', 'Font', 'Stylesheet')
+```
+
+## Display Strategy
+
+- **Lists**: Return list of dicts for Replkit2 table display
+- **Details**: Return single dict for box display
+- **Raw**: Return JSON strings for inspect/debug
+
+Commands should NOT format output - let Replkit2 handle display based on `@app.command(display="table"|"markdown"|"raw")`.
+
+## Future Commands
+
+- `storage()` - Query cookies/localStorage via CDP
+- `api()` - Discover API endpoints from traffic
+- `har()` - Export to HAR format
+- `intercept()` - Modify requests (requires Fetch domain)
+- `timeline()` - Request/response correlation view

webtap_tool-0.1.1/CHANGELOG.md

@@ -0,0 +1,69 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+### Changed
+
+### Fixed
+
+### Removed
+
+## [0.1.1] - 2025-09-05
+
+### Added
+
+### Changed
+
+### Fixed
+
+### Removed
+
+## [0.1.0] - 2025-09-05
+
+### Added
+- Chrome DevTools Protocol (CDP) integration for browser debugging
+- Native CDP Storage architecture using DuckDB for event storage
+- Dynamic field discovery with fuzzy matching across all CDP events
+- Network request/response monitoring with on-demand body fetching
+- Console message capture with error tracking
+- JavaScript execution in browser context via `js()` command
+- Request interception and modification via `fetch()` command
+- Chrome extension for visual page selection and debugging
+- Bootstrap commands for downloading filters and extension (`setup-filters`, `setup-extension`)
+- Chrome launcher command (`launch-chrome`) for debugging-enabled browser startup
+- FastAPI server on port 8765 for Chrome extension integration
+- Comprehensive filter system (ads, tracking, analytics, CDN, consent, monitoring)
+- Events query system for flexible CDP event exploration
+- Inspect command with Python environment for data analysis
+- Svelte Debug Protocol (SDP) experimental support for Svelte app debugging
+- Service layer architecture with clean dependency injection
+- Markdown-based output formatting for all commands
+- MCP (Model Context Protocol) support via ReplKit2
+- CLI mode with Typer integration
+
+### Changed
+- **BREAKING**: Removed single `bootstrap` command, replaced with separate setup commands
+- **BREAKING**: `eval()` and `exec()` commands replaced by unified `js()` command
+- **BREAKING**: All commands now return markdown dictionaries instead of plain text
+- Aligned with ReplKit2 v0.11.0 API changes (`typer_config` instead of `cli_config`)
+- Store CDP events as-is without transformation (Native CDP Storage philosophy)
+- Connection errors return error responses instead of raising exceptions
+- Standardized command pattern with unified builders and error handling
+
+### Fixed
+- CLI mode parameter handling for dict/list types
+- Type checking errors with proper null checks
+- Import order issues in CLI mode
+- Shell completion options properly hidden in CLI mode
+
+<!-- 
+When you run 'relkit bump', the [Unreleased] section will automatically 
+become the new version section. Make sure to add your changes above!
+-->

webtap_tool-0.1.1/PKG-INFO

@@ -0,0 +1,427 @@
+Metadata-Version: 2.4
+Name: webtap-tool
+Version: 0.1.1
+Summary: Terminal-based web page inspector for AI debugging sessions
+Author-email: Fredrik Angelsen <fredrikangelsen@gmail.com>
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
+Classifier: Topic :: Software Development :: Debuggers
+Requires-Python: >=3.12
+Requires-Dist: beautifulsoup4>=4.13.5
+Requires-Dist: cryptography>=45.0.6
+Requires-Dist: duckdb>=1.3.2
+Requires-Dist: fastapi>=0.116.1
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: lxml>=6.0.1
+Requires-Dist: msgpack-python>=0.5.6
+Requires-Dist: protobuf>=6.32.0
+Requires-Dist: pyjwt>=2.10.1
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: replkit2[all]>=0.11.0
+Requires-Dist: requests>=2.32.4
+Requires-Dist: uvicorn>=0.35.0
+Requires-Dist: websocket-client>=1.8.0
+Requires-Dist: websockets>=15.0.1
+Description-Content-Type: text/markdown
+
+# WebTap
+
+Browser debugging via Chrome DevTools Protocol with native event storage and dynamic querying.
+
+## Overview
+
+WebTap connects to Chrome's debugging protocol and stores CDP events as-is in DuckDB, enabling powerful SQL queries and dynamic field discovery without complex transformations.
+
+## Key Features
+
+- **Native CDP Storage** - Events stored exactly as received in DuckDB
+- **Dynamic Field Discovery** - Automatically indexes all field paths from events
+- **Smart Filtering** - Built-in filters for ads, tracking, analytics noise
+- **SQL Querying** - Direct DuckDB access for complex analysis
+- **Chrome Extension** - Visual page selector and connection management
+- **Python Inspection** - Full Python environment for data exploration
+
+## Installation
+
+```bash
+# Install with uv
+uv tool install webtap
+
+# Or from source
+cd packages/webtap
+uv sync
+```
+
+## Quick Start
+
+1. **Start Chrome with debugging**
+```bash
+# macOS
+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
+
+# Linux  
+google-chrome --remote-debugging-port=9222
+
+# Windows
+chrome.exe --remote-debugging-port=9222
+```
+
+2. **Launch WebTap**
+```bash
+webtap
+
+# You'll see:
+================================================================================
+                     WebTap - Chrome DevTools Protocol REPL
+--------------------------------------------------------------------------------
+Type help() for available commands
+>>>
+```
+
+3. **Connect and explore**
+```python
+>>> pages()                          # List available Chrome pages
+>>> connect(0)                       # Connect to first page
+>>> network()                        # View network requests (filtered)
+>>> console()                        # View console messages
+>>> events({"url": "*api*"})         # Query any CDP field dynamically
+```
+
+## Core Commands
+
+### Connection & Navigation
+```python
+pages()                      # List Chrome pages
+connect(0)                   # Connect by index (shorthand)
+connect(page=1)              # Connect by index (explicit)
+connect(page_id="xyz")       # Connect by page ID  
+disconnect()                 # Disconnect from current page
+navigate("https://...")      # Navigate to URL
+reload(ignore_cache=False)   # Reload page
+back() / forward()           # Navigate history
+page()                       # Show current page info
+```
+
+### Dynamic Event Querying
+```python
+# Query ANY field across ALL event types using dict filters
+events({"url": "*github*"})              # Find GitHub requests
+events({"status": 404})                  # Find all 404s
+events({"type": "xhr", "method": "POST"})   # Find AJAX POSTs  
+events({"headers": "*"})                 # Extract all headers
+
+# Field names are fuzzy-matched and case-insensitive
+events({"URL": "*api*"})     # Works! Finds 'url', 'URL', 'documentURL'
+events({"err": "*"})         # Finds 'error', 'errorText', 'err'
+```
+
+### Network Monitoring
+```python
+network()                              # Filtered network requests (default)
+network(no_filters=True)               # Show everything (noisy!)
+network(filters=["ads", "tracking"])   # Specific filter categories
+```
+
+### Filter Management
+```python
+# Manage noise filters
+filters()                                # Show current filters (default action="list")
+filters(action="load")                   # Load from .webtap/filters.json
+filters(action="add", config={"domain": "*doubleclick*", "category": "ads"})
+filters(action="save")                   # Persist to disk
+filters(action="toggle", config={"category": "ads"})  # Toggle category
+
+# Built-in categories: ads, tracking, analytics, telemetry, cdn, fonts, images
+```
+
+### Data Inspection
+```python
+# Inspect events by rowid
+inspect(49)                         # View event details by rowid
+inspect(50, expr="data['params']['response']['headers']")  # Extract field
+
+# Response body inspection with Python expressions
+body(49)                            # Get response body
+body(49, expr="import json; json.loads(body)")  # Parse JSON
+body(49, expr="len(body)")         # Check size
+
+# Request interception
+fetch("enable")                     # Enable request interception
+fetch("disable")                    # Disable request interception
+requests()                          # Show paused requests
+resume(123)                         # Continue paused request by ID
+fail(123)                           # Fail paused request by ID
+```
+
+### Console & JavaScript
+```python
+console()                           # View console messages
+js("document.title")                # Evaluate JavaScript (returns value)
+js("console.log('Hello')", wait_return=False)  # Execute without waiting
+clear()                             # Clear events (default)
+clear(console=True)                 # Clear browser console
+clear(events=True, console=True, cache=True)  # Clear everything
+```
+
+## Architecture
+
+### Native CDP Storage Philosophy
+
+```
+Chrome Tab
+    ↓ CDP Events (WebSocket)
+DuckDB Storage (events table)
+    ↓ SQL Queries + Field Discovery
+Service Layer (WebTapService)
+    ├── NetworkService - Request filtering
+    ├── ConsoleService - Message handling
+    ├── FetchService - Request interception
+    └── BodyService - Response caching
+    ↓
+Commands (Thin Wrappers)
+    ├── events() - Query any field
+    ├── network() - Filtered requests  
+    ├── console() - Messages
+    ├── body() - Response bodies
+    └── js() - JavaScript execution
+    ↓
+API Server (FastAPI on :8765)
+    └── Chrome Extension Integration
+```
+
+### How It Works
+
+1. **Events stored as-is** - No transformation, full CDP data preserved
+2. **Field paths indexed** - Every unique path like `params.response.status` tracked
+3. **Dynamic discovery** - Fuzzy matching finds fields without schemas
+4. **SQL generation** - User queries converted to DuckDB JSON queries
+5. **On-demand fetching** - Bodies, cookies fetched only when needed
+
+## Advanced Usage
+
+### Direct SQL Queries
+```python
+# Access DuckDB directly
+sql = """
+    SELECT json_extract_string(event, '$.params.response.url') as url,
+           json_extract_string(event, '$.params.response.status') as status
+    FROM events 
+    WHERE json_extract_string(event, '$.method') = 'Network.responseReceived'
+"""
+results = state.cdp.query(sql)
+```
+
+### Field Discovery
+```python
+# See what fields are available
+state.cdp.field_paths.keys()  # All discovered field names
+
+# Find all paths for a field
+state.cdp.discover_field_paths("url")
+# Returns: ['params.request.url', 'params.response.url', 'params.documentURL', ...]
+```
+
+### Direct CDP Access
+```python
+# Send CDP commands directly
+state.cdp.execute("Network.getResponseBody", {"requestId": "123"})
+state.cdp.execute("Storage.getCookies", {})
+state.cdp.execute("Runtime.evaluate", {"expression": "window.location.href"})
+```
+
+### Chrome Extension
+
+Install the extension from `packages/webtap/extension/`:
+1. Open `chrome://extensions/`
+2. Enable Developer mode
+3. Load unpacked → Select extension folder
+4. Click extension icon to connect to pages
+
+## Examples
+
+### List and Connect to Pages
+```python
+>>> pages()
+## Chrome Pages
+
+| Index | Title                | URL                            | ID     | Connected |
+|:------|:---------------------|:-------------------------------|:-------|:----------|
+| 0     | Messenger            | https://www.m...1743198803269/ | DC8... | No        |
+| 1     | GitHub - replkit2    | https://githu...elsen/replkit2 | DD4... | No        |
+| 2     | YouTube Music        | https://music.youtube.com/     | F83... | No        |
+
+_3 pages available_
+<pages: 1 fields>
+
+>>> connect(1)
+## Connection Established
+
+**Page:** GitHub - angelsen/replkit2
+
+**URL:** https://github.com/angelsen/replkit2
+<connect: 1 fields>
+```
+
+### Monitor Network Traffic
+```python
+>>> network()
+## Network Requests
+
+| ID   | ReqID        | Method | Status | URL                                             | Type     | Size |
+|:-----|:-------------|:-------|:-------|:------------------------------------------------|:---------|:-----|
+| 3264 | 682214.9033  | GET    | 200    | https://api.github.com/graphql                  | Fetch    | 22KB |
+| 2315 | 682214.8985  | GET    | 200    | https://api.github.com/repos/angelsen/replkit2  | Fetch    | 16KB |
+| 359  | 682214.8638  | GET    | 200    | https://github.githubassets.com/assets/app.js   | Script   | 21KB |
+
+_3 requests_
+
+### Next Steps
+
+- **Analyze responses:** `body(3264)` - fetch response body
+- **Parse HTML:** `body(3264, "bs4(body, 'html.parser').find('title').text")`
+- **Extract JSON:** `body(3264, "json.loads(body)['data']")`
+- **Find patterns:** `body(3264, "re.findall(r'/api/\\w+', body)")`
+- **Decode JWT:** `body(3264, "jwt.decode(body, options={'verify_signature': False})")`
+- **Search events:** `events({'url': '*api*'})` - find all API calls
+- **Intercept traffic:** `fetch('enable')` then `requests()` - pause and modify
+<network: 1 fields>
+```
+
+### View Console Messages
+```python
+>>> console()
+## Console Messages
+
+| ID   | Level      | Source   | Message                                                         | Time     |
+|:-----|:-----------|:---------|:----------------------------------------------------------------|:---------|
+| 5939 | WARNING    | security | An iframe which has both allow-scripts and allow-same-origin... | 11:42:46 |
+| 2319 | LOG        | console  | API request completed                                           | 11:42:40 |
+| 32   | ERROR      | network  | Failed to load resource: the server responded with a status...  | 12:47:41 |
+
+_3 messages_
+
+### Next Steps
+
+- **Inspect error:** `inspect(32)` - view full stack trace
+- **Find all errors:** `events({'level': 'error'})` - filter console errors
+- **Extract stack:** `inspect(32, "data.get('stackTrace', {})")`
+- **Search messages:** `events({'message': '*failed*'})` - pattern match
+- **Check network:** `network()` - may show failed requests causing errors
+<console: 1 fields>
+```
+
+### Find and Analyze API Calls
+```python
+>>> events({"url": "*api*", "method": "POST"})
+## Query Results
+
+| RowID | Method                      | URL                             | Status |
+|:------|:----------------------------|:--------------------------------|:-------|
+| 49    | Network.requestWillBeSent   | https://api.github.com/graphql  | -      |
+| 50    | Network.responseReceived    | https://api.github.com/graphql  | 200    |
+
+_2 events_
+<events: 1 fields>
+
+>>> body(50, expr="import json; json.loads(body)['data']")
+{'viewer': {'login': 'octocat', 'name': 'The Octocat'}}
+
+>>> inspect(49)  # View full request details
+```
+
+### Debug Failed Requests
+```python
+>>> events({"status": 404})
+## Query Results
+
+| RowID | Method                   | URL                               | Status |
+|:------|:-------------------------|:----------------------------------|:-------|
+| 32    | Network.responseReceived | https://api.example.com/missing   | 404    |
+| 29    | Network.responseReceived | https://api.example.com/notfound  | 404    |
+
+_2 events_
+<events: 1 fields>
+
+>>> events({"errorText": "*"})  # Find network errors
+>>> events({"type": "Failed"})  # Find failed resources
+```
+
+### Monitor Specific Domains
+```python
+>>> events({"url": "*myapi.com*"})  # Your API
+>>> events({"url": "*localhost*"})  # Local development
+>>> events({"url": "*stripe*"})     # Payment APIs
+```
+
+### Extract Headers and Cookies
+```python
+>>> events({"headers": "*authorization*"})  # Find auth headers
+>>> state.cdp.execute("Storage.getCookies", {})  # Get all cookies
+>>> events({"setCookie": "*"})  # Find Set-Cookie headers
+```
+
+## Filter Configuration
+
+WebTap includes aggressive default filters to reduce noise. Customize in `.webtap/filters.json`:
+
+```json
+{
+  "ads": {
+    "domains": ["*doubleclick*", "*googlesyndication*", "*adsystem*"],
+    "types": ["Ping", "Beacon"]
+  },
+  "tracking": {
+    "domains": ["*google-analytics*", "*segment*", "*mixpanel*"],
+    "types": ["Image", "Script"]
+  }
+}
+```
+
+## Design Principles
+
+1. **Store AS-IS** - No transformation of CDP events
+2. **Query On-Demand** - Extract only what's needed
+3. **Dynamic Discovery** - No predefined schemas
+4. **SQL-First** - Leverage DuckDB's JSON capabilities
+5. **Minimal Memory** - Store only CDP data
+
+## Requirements
+
+- Chrome/Chromium with debugging enabled
+- Python 3.12+
+- Dependencies: websocket-client, duckdb, replkit2, fastapi, uvicorn, beautifulsoup4
+
+## Development
+
+```bash
+# Run from source
+cd packages/webtap
+uv run webtap
+
+# API server starts automatically on port 8765
+# Chrome extension connects to http://localhost:8765
+
+# Type checking and linting
+basedpyright packages/webtap/src/webtap
+ruff check --fix packages/webtap/src/webtap
+ruff format packages/webtap/src/webtap
+```
+
+## API Server
+
+WebTap automatically starts a FastAPI server on port 8765 for Chrome extension integration:
+
+- `GET /status` - Connection status
+- `GET /pages` - List available Chrome pages
+- `POST /connect` - Connect to a page
+- `POST /disconnect` - Disconnect from current page
+- `POST /clear` - Clear events/console/cache
+- `GET /fetch/paused` - Get paused requests
+- `POST /filters/toggle/{category}` - Toggle filter categories
+
+The API server runs in a background thread and doesn't block the REPL.
+
+## License
+
+MIT - See [LICENSE](../../LICENSE) for details.