webtap-tool 0.1.1__py3-none-any.whl

webtap_tool-0.1.1.dist-info/METADATA

@@ -0,0 +1,427 @@
+Metadata-Version: 2.4
+Name: webtap-tool
+Version: 0.1.1
+Summary: Terminal-based web page inspector for AI debugging sessions
+Author-email: Fredrik Angelsen <fredrikangelsen@gmail.com>
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
+Classifier: Topic :: Software Development :: Debuggers
+Requires-Python: >=3.12
+Requires-Dist: beautifulsoup4>=4.13.5
+Requires-Dist: cryptography>=45.0.6
+Requires-Dist: duckdb>=1.3.2
+Requires-Dist: fastapi>=0.116.1
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: lxml>=6.0.1
+Requires-Dist: msgpack-python>=0.5.6
+Requires-Dist: protobuf>=6.32.0
+Requires-Dist: pyjwt>=2.10.1
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: replkit2[all]>=0.11.0
+Requires-Dist: requests>=2.32.4
+Requires-Dist: uvicorn>=0.35.0
+Requires-Dist: websocket-client>=1.8.0
+Requires-Dist: websockets>=15.0.1
+Description-Content-Type: text/markdown
+
+# WebTap
+
+Browser debugging via Chrome DevTools Protocol with native event storage and dynamic querying.
+
+## Overview
+
+WebTap connects to Chrome's debugging protocol and stores CDP events as-is in DuckDB, enabling powerful SQL queries and dynamic field discovery without complex transformations.
+
+## Key Features
+
+- **Native CDP Storage** - Events stored exactly as received in DuckDB
+- **Dynamic Field Discovery** - Automatically indexes all field paths from events
+- **Smart Filtering** - Built-in filters for ads, tracking, analytics noise
+- **SQL Querying** - Direct DuckDB access for complex analysis
+- **Chrome Extension** - Visual page selector and connection management
+- **Python Inspection** - Full Python environment for data exploration
+
+## Installation
+
+```bash
+# Install with uv
+uv tool install webtap
+
+# Or from source
+cd packages/webtap
+uv sync
+```
+
+## Quick Start
+
+1. **Start Chrome with debugging**
+```bash
+# macOS
+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
+
+# Linux  
+google-chrome --remote-debugging-port=9222
+
+# Windows
+chrome.exe --remote-debugging-port=9222
+```
+
+2. **Launch WebTap**
+```bash
+webtap
+
+# You'll see:
+================================================================================
+                     WebTap - Chrome DevTools Protocol REPL
+--------------------------------------------------------------------------------
+Type help() for available commands
+>>>
+```
+
+3. **Connect and explore**
+```python
+>>> pages()                          # List available Chrome pages
+>>> connect(0)                       # Connect to first page
+>>> network()                        # View network requests (filtered)
+>>> console()                        # View console messages
+>>> events({"url": "*api*"})         # Query any CDP field dynamically
+```
+
+## Core Commands
+
+### Connection & Navigation
+```python
+pages()                      # List Chrome pages
+connect(0)                   # Connect by index (shorthand)
+connect(page=1)              # Connect by index (explicit)
+connect(page_id="xyz")       # Connect by page ID  
+disconnect()                 # Disconnect from current page
+navigate("https://...")      # Navigate to URL
+reload(ignore_cache=False)   # Reload page
+back() / forward()           # Navigate history
+page()                       # Show current page info
+```
+
+### Dynamic Event Querying
+```python
+# Query ANY field across ALL event types using dict filters
+events({"url": "*github*"})              # Find GitHub requests
+events({"status": 404})                  # Find all 404s
+events({"type": "xhr", "method": "POST"})   # Find AJAX POSTs  
+events({"headers": "*"})                 # Extract all headers
+
+# Field names are fuzzy-matched and case-insensitive
+events({"URL": "*api*"})     # Works! Finds 'url', 'URL', 'documentURL'
+events({"err": "*"})         # Finds 'error', 'errorText', 'err'
+```
+
+### Network Monitoring
+```python
+network()                              # Filtered network requests (default)
+network(no_filters=True)               # Show everything (noisy!)
+network(filters=["ads", "tracking"])   # Specific filter categories
+```
+
+### Filter Management
+```python
+# Manage noise filters
+filters()                                # Show current filters (default action="list")
+filters(action="load")                   # Load from .webtap/filters.json
+filters(action="add", config={"domain": "*doubleclick*", "category": "ads"})
+filters(action="save")                   # Persist to disk
+filters(action="toggle", config={"category": "ads"})  # Toggle category
+
+# Built-in categories: ads, tracking, analytics, telemetry, cdn, fonts, images
+```
+
+### Data Inspection
+```python
+# Inspect events by rowid
+inspect(49)                         # View event details by rowid
+inspect(50, expr="data['params']['response']['headers']")  # Extract field
+
+# Response body inspection with Python expressions
+body(49)                            # Get response body
+body(49, expr="import json; json.loads(body)")  # Parse JSON
+body(49, expr="len(body)")         # Check size
+
+# Request interception
+fetch("enable")                     # Enable request interception
+fetch("disable")                    # Disable request interception
+requests()                          # Show paused requests
+resume(123)                         # Continue paused request by ID
+fail(123)                           # Fail paused request by ID
+```
+
+### Console & JavaScript
+```python
+console()                           # View console messages
+js("document.title")                # Evaluate JavaScript (returns value)
+js("console.log('Hello')", wait_return=False)  # Execute without waiting
+clear()                             # Clear events (default)
+clear(console=True)                 # Clear browser console
+clear(events=True, console=True, cache=True)  # Clear everything
+```
+
+## Architecture
+
+### Native CDP Storage Philosophy
+
+```
+Chrome Tab
+    ↓ CDP Events (WebSocket)
+DuckDB Storage (events table)
+    ↓ SQL Queries + Field Discovery
+Service Layer (WebTapService)
+    ├── NetworkService - Request filtering
+    ├── ConsoleService - Message handling
+    ├── FetchService - Request interception
+    └── BodyService - Response caching
+    ↓
+Commands (Thin Wrappers)
+    ├── events() - Query any field
+    ├── network() - Filtered requests  
+    ├── console() - Messages
+    ├── body() - Response bodies
+    └── js() - JavaScript execution
+    ↓
+API Server (FastAPI on :8765)
+    └── Chrome Extension Integration
+```
+
+### How It Works
+
+1. **Events stored as-is** - No transformation, full CDP data preserved
+2. **Field paths indexed** - Every unique path like `params.response.status` tracked
+3. **Dynamic discovery** - Fuzzy matching finds fields without schemas
+4. **SQL generation** - User queries converted to DuckDB JSON queries
+5. **On-demand fetching** - Bodies, cookies fetched only when needed
+
+## Advanced Usage
+
+### Direct SQL Queries
+```python
+# Access DuckDB directly
+sql = """
+    SELECT json_extract_string(event, '$.params.response.url') as url,
+           json_extract_string(event, '$.params.response.status') as status
+    FROM events 
+    WHERE json_extract_string(event, '$.method') = 'Network.responseReceived'
+"""
+results = state.cdp.query(sql)
+```
+
+### Field Discovery
+```python
+# See what fields are available
+state.cdp.field_paths.keys()  # All discovered field names
+
+# Find all paths for a field
+state.cdp.discover_field_paths("url")
+# Returns: ['params.request.url', 'params.response.url', 'params.documentURL', ...]
+```
+
+### Direct CDP Access
+```python
+# Send CDP commands directly
+state.cdp.execute("Network.getResponseBody", {"requestId": "123"})
+state.cdp.execute("Storage.getCookies", {})
+state.cdp.execute("Runtime.evaluate", {"expression": "window.location.href"})
+```
+
+### Chrome Extension
+
+Install the extension from `packages/webtap/extension/`:
+1. Open `chrome://extensions/`
+2. Enable Developer mode
+3. Load unpacked → Select extension folder
+4. Click extension icon to connect to pages
+
+## Examples
+
+### List and Connect to Pages
+```python
+>>> pages()
+## Chrome Pages
+
+| Index | Title                | URL                            | ID     | Connected |
+|:------|:---------------------|:-------------------------------|:-------|:----------|
+| 0     | Messenger            | https://www.m...1743198803269/ | DC8... | No        |
+| 1     | GitHub - replkit2    | https://githu...elsen/replkit2 | DD4... | No        |
+| 2     | YouTube Music        | https://music.youtube.com/     | F83... | No        |
+
+_3 pages available_
+<pages: 1 fields>
+
+>>> connect(1)
+## Connection Established
+
+**Page:** GitHub - angelsen/replkit2
+
+**URL:** https://github.com/angelsen/replkit2
+<connect: 1 fields>
+```
+
+### Monitor Network Traffic
+```python
+>>> network()
+## Network Requests
+
+| ID   | ReqID        | Method | Status | URL                                             | Type     | Size |
+|:-----|:-------------|:-------|:-------|:------------------------------------------------|:---------|:-----|
+| 3264 | 682214.9033  | GET    | 200    | https://api.github.com/graphql                  | Fetch    | 22KB |
+| 2315 | 682214.8985  | GET    | 200    | https://api.github.com/repos/angelsen/replkit2  | Fetch    | 16KB |
+| 359  | 682214.8638  | GET    | 200    | https://github.githubassets.com/assets/app.js   | Script   | 21KB |
+
+_3 requests_
+
+### Next Steps
+
+- **Analyze responses:** `body(3264)` - fetch response body
+- **Parse HTML:** `body(3264, "bs4(body, 'html.parser').find('title').text")`
+- **Extract JSON:** `body(3264, "json.loads(body)['data']")`
+- **Find patterns:** `body(3264, "re.findall(r'/api/\\w+', body)")`
+- **Decode JWT:** `body(3264, "jwt.decode(body, options={'verify_signature': False})")`
+- **Search events:** `events({'url': '*api*'})` - find all API calls
+- **Intercept traffic:** `fetch('enable')` then `requests()` - pause and modify
+<network: 1 fields>
+```
+
+### View Console Messages
+```python
+>>> console()
+## Console Messages
+
+| ID   | Level      | Source   | Message                                                         | Time     |
+|:-----|:-----------|:---------|:----------------------------------------------------------------|:---------|
+| 5939 | WARNING    | security | An iframe which has both allow-scripts and allow-same-origin... | 11:42:46 |
+| 2319 | LOG        | console  | API request completed                                           | 11:42:40 |
+| 32   | ERROR      | network  | Failed to load resource: the server responded with a status...  | 12:47:41 |
+
+_3 messages_
+
+### Next Steps
+
+- **Inspect error:** `inspect(32)` - view full stack trace
+- **Find all errors:** `events({'level': 'error'})` - filter console errors
+- **Extract stack:** `inspect(32, "data.get('stackTrace', {})")`
+- **Search messages:** `events({'message': '*failed*'})` - pattern match
+- **Check network:** `network()` - may show failed requests causing errors
+<console: 1 fields>
+```
+
+### Find and Analyze API Calls
+```python
+>>> events({"url": "*api*", "method": "POST"})
+## Query Results
+
+| RowID | Method                      | URL                             | Status |
+|:------|:----------------------------|:--------------------------------|:-------|
+| 49    | Network.requestWillBeSent   | https://api.github.com/graphql  | -      |
+| 50    | Network.responseReceived    | https://api.github.com/graphql  | 200    |
+
+_2 events_
+<events: 1 fields>
+
+>>> body(50, expr="import json; json.loads(body)['data']")
+{'viewer': {'login': 'octocat', 'name': 'The Octocat'}}
+
+>>> inspect(49)  # View full request details
+```
+
+### Debug Failed Requests
+```python
+>>> events({"status": 404})
+## Query Results
+
+| RowID | Method                   | URL                               | Status |
+|:------|:-------------------------|:----------------------------------|:-------|
+| 32    | Network.responseReceived | https://api.example.com/missing   | 404    |
+| 29    | Network.responseReceived | https://api.example.com/notfound  | 404    |
+
+_2 events_
+<events: 1 fields>
+
+>>> events({"errorText": "*"})  # Find network errors
+>>> events({"type": "Failed"})  # Find failed resources
+```
+
+### Monitor Specific Domains
+```python
+>>> events({"url": "*myapi.com*"})  # Your API
+>>> events({"url": "*localhost*"})  # Local development
+>>> events({"url": "*stripe*"})     # Payment APIs
+```
+
+### Extract Headers and Cookies
+```python
+>>> events({"headers": "*authorization*"})  # Find auth headers
+>>> state.cdp.execute("Storage.getCookies", {})  # Get all cookies
+>>> events({"setCookie": "*"})  # Find Set-Cookie headers
+```
+
+## Filter Configuration
+
+WebTap includes aggressive default filters to reduce noise. Customize in `.webtap/filters.json`:
+
+```json
+{
+  "ads": {
+    "domains": ["*doubleclick*", "*googlesyndication*", "*adsystem*"],
+    "types": ["Ping", "Beacon"]
+  },
+  "tracking": {
+    "domains": ["*google-analytics*", "*segment*", "*mixpanel*"],
+    "types": ["Image", "Script"]
+  }
+}
+```
+
+## Design Principles
+
+1. **Store AS-IS** - No transformation of CDP events
+2. **Query On-Demand** - Extract only what's needed
+3. **Dynamic Discovery** - No predefined schemas
+4. **SQL-First** - Leverage DuckDB's JSON capabilities
+5. **Minimal Memory** - Store only CDP data
+
+## Requirements
+
+- Chrome/Chromium with debugging enabled
+- Python 3.12+
+- Dependencies: websocket-client, duckdb, replkit2, fastapi, uvicorn, beautifulsoup4
+
+## Development
+
+```bash
+# Run from source
+cd packages/webtap
+uv run webtap
+
+# API server starts automatically on port 8765
+# Chrome extension connects to http://localhost:8765
+
+# Type checking and linting
+basedpyright packages/webtap/src/webtap
+ruff check --fix packages/webtap/src/webtap
+ruff format packages/webtap/src/webtap
+```
+
+## API Server
+
+WebTap automatically starts a FastAPI server on port 8765 for Chrome extension integration:
+
+- `GET /status` - Connection status
+- `GET /pages` - List available Chrome pages
+- `POST /connect` - Connect to a page
+- `POST /disconnect` - Disconnect from current page
+- `POST /clear` - Clear events/console/cache
+- `GET /fetch/paused` - Get paused requests
+- `POST /filters/toggle/{category}` - Toggle filter categories
+
+The API server runs in a background thread and doesn't block the REPL.
+
+## License
+
+MIT - See [LICENSE](../../LICENSE) for details.

webtap_tool-0.1.1.dist-info/RECORD

@@ -0,0 +1,43 @@
+webtap/VISION.md,sha256=kfoJfEPVc4chOrD9tNMDmYBY9rX9KB-286oZj70ALCE,7681
+webtap/__init__.py,sha256=iMD7Y3DMnmd-psxLdG9uLeo9qXfCDDzUQ52lVo5LWOU,1633
+webtap/api.py,sha256=I4CyOlgJW_peDn94DIeq5945G9BAbCzOqMmf_7YUm_g,6060
+webtap/app.py,sha256=RgLGWUc39o1AOnXxpXg7WVbNk3X8If3Q7nbGeqPaHMw,2586
+webtap/filters.py,sha256=nphF2bFRbFtS2ue-CbV1uzKpuK3IYBbwjLeLhMDdLEk,11034
+webtap/cdp/README.md,sha256=0TS0V_dRgRAzBqhddpXWD4S0YVi5wI4JgFJSll_KUBE,5660
+webtap/cdp/__init__.py,sha256=c6NFG0XJnAa5GTe9MLr9mDZcLZqoTQN7A1cvvOfLcgY,453
+webtap/cdp/query.py,sha256=ULfc8ddHnX_uHezaVLblPOzV6tTyu3ohOc6XlE3wPa0,4153
+webtap/cdp/session.py,sha256=BTajmDH6nel2dvia7IX9k6C-RinkN2NxY4rxtUJAJE0,12362
+webtap/cdp/schema/README.md,sha256=hnWCzbXYcYtWaZb_SgjVaFBiG81S9b9Y3x-euQFwQDo,1222
+webtap/cdp/schema/cdp_protocol.json,sha256=dp9_OLYLuVsQb1oV5r6MZfMzURscBLyAXUckdaPWyv4,1488452
+webtap/cdp/schema/cdp_version.json,sha256=OhGy1qpfQjSe3Z7OqL6KynBFlDFBXxKGPZCY-ZN_lVU,399
+webtap/commands/DEVELOPER_GUIDE.md,sha256=lii-bOUVNBiFjpZlYIipnFNNfUI9vI5VwAhGi2jvQAU,9870
+webtap/commands/TIPS.md,sha256=gE75m66QxU-7RG8NS9z5EchMoEN2OrQ0ZzHMETsAd_w,6854
+webtap/commands/__init__.py,sha256=rr3xM_bY0BgxkDOjsnsI8UBhjlz7nqiYlgJ8fjiJ1jQ,270
+webtap/commands/_builders.py,sha256=nUZLcMWXVWx6KmEV9KLajV-Gbxv2m2NB5lNZgR5Be1c,3842
+webtap/commands/_errors.py,sha256=W64bGcUDsvdncD3WiL4H7hVTVE58OiwWYC391SAk43s,3400
+webtap/commands/_tips.py,sha256=SleMpwdghrHNqdzR60Cu8T0NZqJfWfcfrgIcyWI6GIQ,4793
+webtap/commands/_utils.py,sha256=VLXDhhhJITrQjwEyeLRTU2ey0QcLzY-_OxTjtPJlhYM,6816
+webtap/commands/body.py,sha256=e67q_rT8vtRz5_ViQjl5a6BAolW4uwedRZkVp8TJ5Vo,7111
+webtap/commands/connection.py,sha256=ZYV2TmK1LRVFyMneNYswJmnaoi45rFTApQew5Gm-CC0,5465
+webtap/commands/console.py,sha256=moGLsZ-k5wtukjrPFkEXjBl-Jj_yj4bHEArPXVmZLUc,2180
+webtap/commands/events.py,sha256=yx3iJgTANKsoGXBMu1WfBOjEW_thmNKMmUTXtamqRtQ,4093
+webtap/commands/fetch.py,sha256=_TzOvJfVzPaw4ZmyI95Qb7rS3iKx2nmp_IL3jaQO_6g,7772
+webtap/commands/filters.py,sha256=trZvcbHTaF0FZC8dyMAmhmS2dlBoA82VXe5_DDS3eU8,8986
+webtap/commands/inspect.py,sha256=6PGN7iDT1oLzQJboNeYozLILrW2VsAzmtMpF3_XhD30,5746
+webtap/commands/javascript.py,sha256=QpQdqqoQwwTyz1lpibZ92XKOL89scu_ndgSjkhaYuDk,3195
+webtap/commands/launch.py,sha256=-DonCgu7LakyTIN5ksGjY-8CiQzQiQ2SAamBCOvLWrw,2687
+webtap/commands/navigation.py,sha256=Mapawp2AZTJQaws2uwlTgMUhqz7HlVTLxiZ06n_MQc0,6071
+webtap/commands/network.py,sha256=hwZshGGdVsJ_9MFjOKJXT07I990JjZInw2LLnKXLQ5Y,2910
+webtap/commands/setup.py,sha256=ewoOTvgCAzMPcFm6cbk-93nY_BI2IXqPZGGGpc1rJiw,4459
+webtap/services/README.md,sha256=rala_jtnNgSiQ1lFLM7x_UQ4SJZDceAm7dpkQMRTYaI,2346
+webtap/services/__init__.py,sha256=IjFqu0Ak6D-r18aokcQMtenDV3fbelvfjTCejGv6CZ0,570
+webtap/services/body.py,sha256=tU60N-efOvTMndDGLcz0rf921VHiRmczDEs-qtECMng,3934
+webtap/services/console.py,sha256=W1FGSuLl1bgZzpD9bVu98wlHGupe12vA7YqDLXFszYs,3789
+webtap/services/fetch.py,sha256=hKrhFda1x65GmLHO0XMkxbclMtkWt0uEv7Rp9Ui2na0,13623
+webtap/services/main.py,sha256=HcXdPuI7hzsxsNvfN0npGhj_M7HObc83Lr3fuy7BMeE,5673
+webtap/services/network.py,sha256=EJZIlaE98v113xypPIiiTE-LiG-eCXGNXoJjzDYebhU,3480
+webtap/services/setup.py,sha256=DF2471WsY5pJ5gEn64EMGRW5t9mUyuTQoCTejOJeQxg,7594
+webtap_tool-0.1.1.dist-info/METADATA,sha256=QYQTlMAdoYseHUo742JZfunjHy6KjKlO89YvKZJDBSE,14277
+webtap_tool-0.1.1.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+webtap_tool-0.1.1.dist-info/entry_points.txt,sha256=iFe575I0CIb1MbfPt0oX2VYyY5gSU_dA551PKVR83TU,39
+webtap_tool-0.1.1.dist-info/RECORD,,

webtap_tool-0.1.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.27.0
+Root-Is-Purelib: true
+Tag: py3-none-any

webtap_tool-0.1.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+webtap = webtap:main