webtap-tool 0.1.1__py3-none-any.whl

webtap/commands/TIPS.md

@@ -0,0 +1,153 @@
+# WebTap Command Documentation
+
+## Libraries
+All commands have these pre-imported (no imports needed!):
+- **Web:** bs4/BeautifulSoup, lxml, ElementTree/ET
+- **Data:** json, yaml, msgpack, protobuf_json/protobuf_text
+- **Security:** jwt, base64, hashlib, cryptography
+- **HTTP:** httpx, urllib
+- **Text:** re, difflib, textwrap, html
+- **Utils:** datetime, collections, itertools, pprint, ast
+
+## Commands
+
+### body
+Fetch and analyze HTTP response bodies with Python expressions.
+
+#### Examples
+```python
+body(123)                                  # Get body
+body(123, "json.loads(body)")              # Parse JSON
+body(123, "bs4(body, 'html.parser').find('title').text")  # HTML title
+body(123, "jwt.decode(body, options={'verify_signature': False})")  # Decode JWT
+body(123, "re.findall(r'/api/[^\"\\s]+', body)[:10]")  # Find API endpoints
+body(123, "httpx.get(json.loads(body)['next_url']).json()")  # Chain requests
+body(123, "msgpack.unpackb(body)")         # Binary formats
+```
+
+#### Tips
+- **Chain requests:** `body({id}, "httpx.get(json.loads(body)['next_url']).text[:100]")`
+- **Parse XML:** `body({id}, "ElementTree.fromstring(body).find('.//title').text")`
+- **Extract forms:** `body({id}, "[f['action'] for f in bs4(body, 'html.parser').find_all('form')]")`
+- **Decode protobuf:** `body({id}, "protobuf_json.Parse(body, YourMessage())")`
+- **Find related:** `events({'requestId': request_id})` - related events
+
+### inspect
+Inspect CDP events with full Python debugging.
+
+Available objects: 'data' (when inspecting event), 'cdp' and 'state' (when no event).
+
+#### Examples
+```python
+inspect(456)                               # Full event
+inspect(456, "data['method']")             # Event type
+inspect(456, "list(data.keys())")          # Top-level keys
+inspect(456, "data.get('params', {}).get('response', {}).get('status')")
+inspect(456, "re.findall(r'session=(\\w+)', str(data))")  # Extract patterns
+inspect(456, "base64.b64decode(data['params']['response']['body'])")
+inspect(456, "jwt.decode(auth.replace('Bearer ', ''), options={'verify_signature': False})")
+inspect(expr="len(cdp.events)")           # Direct CDP access
+inspect(expr="[e for e in cdp.events if 'error' in str(e).lower()][:3]")
+```
+
+#### Tips
+- **Find related:** `events({'requestId': data.get('params', {}).get('requestId')})`
+- **Compare events:** `inspect(other_id, "data.get('method')")`
+- **Extract timing:** `inspect({id}, "data['params']['timing']")`
+- **Decode cookies:** `inspect({id}, "[c.split('=') for c in data.get('params', {}).get('cookies', '').split(';')]")`
+- **Get body:** `body({id})` - if this is a response event
+
+### network
+Show network requests with full data.
+
+#### Tips
+- **Analyze responses:** `body({id})` - fetch response body
+- **Parse HTML:** `body({id}, "bs4(body, 'html.parser').find('title').text")`
+- **Extract JSON:** `body({id}, "json.loads(body)['data']")`
+- **Find patterns:** `body({id}, "re.findall(r'/api/\\w+', body)")`
+- **Decode JWT:** `body({id}, "jwt.decode(body, options={'verify_signature': False})")`
+- **Search events:** `events({'url': '*api*'})` - find all API calls
+- **Intercept traffic:** `fetch('enable')` then `requests()` - pause and modify
+
+### console
+Show console messages with full data.
+
+#### Tips
+- **Inspect error:** `inspect({id})` - view full stack trace
+- **Find all errors:** `events({'level': 'error'})` - filter console errors
+- **Extract stack:** `inspect({id}, "data.get('stackTrace', {})")`
+- **Search messages:** `events({'message': '*failed*'})` - pattern match
+- **Check network:** `network()` - may show failed requests causing errors
+
+### events
+Query CDP events by field values with automatic discovery.
+
+Searches across ALL event types - network, console, page, etc.
+Field names are discovered automatically and case-insensitive.
+
+#### Examples
+```python
+events()                                    # Recent 20 events
+events({"method": "Runtime.*"})            # Runtime events
+events({"requestId": "123"}, limit=100)    # Specific request
+events({"url": "*api*"})                   # Find all API calls
+events({"status": 200})                    # Successful responses
+events({"level": "error"})                # Console errors
+```
+
+#### Tips
+- **Inspect full event:** `inspect({id})` - view complete CDP data
+- **Extract nested data:** `inspect({id}, "data['params']['response']['headers']")`
+- **Find patterns:** `inspect({id}, "re.findall(r'token=(\\w+)', str(data))")`
+- **Get response body:** `body({id})` - if this is a network response
+- **Decode data:** `inspect({id}, "base64.b64decode(data.get('params', {}).get('body', ''))")`
+
+### js
+Execute JavaScript in the browser with optional promise handling.
+
+#### Examples
+```python
+js("document.title")                           # Get page title
+js("document.body.innerText.length")           # Get text length
+js("[...document.links].map(a => a.href)")    # Get all links
+js("fetch('/api').then(r => r.json())", await_promise=True)  # Async
+js("document.querySelectorAll('.ad').forEach(e => e.remove())", wait_return=False)
+js("window.fetch = new Proxy(window.fetch, {get: (t, p) => console.log(p)})", wait_return=False)
+```
+
+#### Tips
+- **Extract all links:** `js("[...document.links].map(a => a.href)")`
+- **Get page text:** `js("document.body.innerText")`
+- **Find data attributes:** `js("[...document.querySelectorAll('[data-id]')].map(e => e.dataset)")`
+- **Monitor DOM:** `js("new MutationObserver(console.log).observe(document, {childList: true, subtree: true})", wait_return=False)`
+- **Hook fetch:** `js("window.fetch = new Proxy(fetch, {apply: (t, _, a) => {console.log(a); return t(...a)}})", wait_return=False)`
+- **Check console:** `console()` - see logged messages from JS execution
+
+### fetch
+Control request interception for debugging and modification.
+
+#### Examples
+```python
+fetch("status")                           # Check status
+fetch("enable")                           # Enable request stage
+fetch("enable", {"response": true})       # Both stages
+fetch("disable")                          # Disable
+```
+
+#### Tips
+- **View paused:** `requests()` - see all intercepted requests
+- **Inspect request:** `inspect({id})` - view full CDP event data
+- **Analyze body:** `body({id})` - fetch and examine response body
+- **Resume request:** `resume({id})` - continue the request
+- **Modify request:** `resume({id}, modifications={'url': '...'})`
+- **Block request:** `fail({id}, 'BlockedByClient')` - reject the request
+
+### requests
+Show paused requests and responses.
+
+#### Tips
+- **Inspect request:** `inspect({id})` - view full CDP event data
+- **Analyze body:** `body({id})` - fetch and examine response body
+- **Resume request:** `resume({id})` - continue the request
+- **Modify request:** `resume({id}, modifications={'url': '...'})`
+- **Fail request:** `fail({id}, 'BlockedByClient')` - block the request

webtap/commands/__init__.py

@@ -0,0 +1,7 @@
+"""WebTap command modules for browser automation.
+
+Commands are imported directly by app.py to register with the ReplKit2 app.
+In CLI mode, only CLI-compatible commands are imported to avoid Typer issues.
+"""
+
+# No imports needed here - app.py imports commands directly

webtap/commands/_builders.py

@@ -0,0 +1,127 @@
+"""Response builders using ReplKit2 v0.10.0+ markdown elements."""
+
+from typing import Any
+
+
+def table_response(
+    title: str | None = None,
+    headers: list[str] | None = None,
+    rows: list[dict] | None = None,
+    summary: str | None = None,
+    warnings: list[str] | None = None,
+    tips: list[str] | None = None,
+) -> dict:
+    """Build table response with full data.
+
+    Args:
+        title: Optional table title
+        headers: Column headers
+        rows: Data rows with FULL data
+        summary: Optional summary text
+        warnings: Optional warning messages
+        tips: Optional developer tips/guidance
+    """
+    elements = []
+
+    if title:
+        elements.append({"type": "heading", "content": title, "level": 2})
+
+    if warnings:
+        for warning in warnings:
+            elements.append({"type": "alert", "message": warning, "level": "warning"})
+
+    if headers and rows:
+        elements.append({"type": "table", "headers": headers, "rows": rows})
+    elif rows:  # Headers can be inferred from row keys
+        elements.append({"type": "table", "rows": rows})
+    else:
+        elements.append({"type": "text", "content": "_No data available_"})
+
+    if summary:
+        elements.append({"type": "text", "content": f"_{summary}_"})
+
+    if tips:
+        elements.append({"type": "heading", "content": "Next Steps", "level": 3})
+        elements.append({"type": "list", "items": tips})
+
+    return {"elements": elements}
+
+
+def info_response(
+    title: str | None = None,
+    fields: dict | None = None,
+    extra: str | None = None,
+) -> dict:
+    """Build info display with key-value pairs.
+
+    Args:
+        title: Optional info title
+        fields: Dict of field names to values
+        extra: Optional extra content (raw markdown)
+    """
+    elements = []
+
+    if title:
+        elements.append({"type": "heading", "content": title, "level": 2})
+
+    if fields:
+        for key, value in fields.items():
+            if value is not None:
+                elements.append({"type": "text", "content": f"**{key}:** {value}"})
+
+    if extra:
+        elements.append({"type": "raw", "content": extra})
+
+    if not elements:
+        elements.append({"type": "text", "content": "_No information available_"})
+
+    return {"elements": elements}
+
+
+def error_response(message: str, suggestions: list[str] | None = None) -> dict:
+    """Build error response with optional suggestions.
+
+    Args:
+        message: Error message
+        suggestions: Optional list of suggestions
+    """
+    elements: list[dict[str, Any]] = [{"type": "alert", "message": message, "level": "error"}]
+
+    if suggestions:
+        elements.append({"type": "text", "content": "**Try:**"})
+        elements.append({"type": "list", "items": suggestions})
+
+    return {"elements": elements}
+
+
+def success_response(message: str, details: dict | None = None) -> dict:
+    """Build success response with optional details.
+
+    Args:
+        message: Success message
+        details: Optional dict of additional details
+    """
+    elements = [{"type": "alert", "message": message, "level": "success"}]
+
+    if details:
+        for key, value in details.items():
+            if value is not None:
+                elements.append({"type": "text", "content": f"**{key}:** {value}"})
+
+    return {"elements": elements}
+
+
+def warning_response(message: str, suggestions: list[str] | None = None) -> dict:
+    """Build warning response with optional suggestions.
+
+    Args:
+        message: Warning message
+        suggestions: Optional list of suggestions
+    """
+    elements: list[dict[str, Any]] = [{"type": "alert", "message": message, "level": "warning"}]
+
+    if suggestions:
+        elements.append({"type": "text", "content": "**Try:**"})
+        elements.append({"type": "list", "items": suggestions})
+
+    return {"elements": elements}

webtap/commands/_errors.py

@@ -0,0 +1,108 @@
+"""Unified error handling for WebTap commands."""
+
+from typing import Optional
+from replkit2.textkit import markdown
+
+
+ERRORS = {
+    "not_connected": {
+        "message": "Not connected to Chrome",
+        "details": "Use `connect()` to connect to a page",
+        "help": [
+            "Run `pages()` to see available tabs",
+            "Use `connect(0)` to connect to first tab",
+            "Or `connect(page_id='...')` for specific tab",
+        ],
+    },
+    "fetch_disabled": {
+        "message": "Fetch interception not enabled",
+        "details": "Enable with `fetch(enable=True)` to pause requests",
+    },
+    "no_data": {"message": "No data available", "details": "The requested data is not available"},
+}
+
+
+def check_connection(state) -> Optional[dict]:
+    """Check CDP connection and return error response if not connected.
+
+    Args:
+        state: Application state containing CDP session.
+    """
+    if not (state.cdp and state.cdp.is_connected):
+        return error_response("not_connected")
+    return None
+
+
+def error_response(error_key: str, custom_message: str | None = None, **kwargs) -> dict:
+    """Build consistent error response in markdown.
+
+    Args:
+        error_key: Key from error templates or custom identifier.
+        custom_message: Override default message.
+        **kwargs: Additional context to add to error response.
+    """
+    error_info = ERRORS.get(error_key, {})
+    message = custom_message or error_info.get("message", "Error occurred")
+
+    builder = markdown().element("alert", message=message, level="error")
+
+    if details := error_info.get("details"):
+        builder.text(details)
+
+    if help_items := error_info.get("help"):
+        builder.text("**How to fix:**")
+        builder.list_(help_items)
+
+    for key, value in kwargs.items():
+        if value:
+            builder.text(f"_{key}: {value}_")
+
+    return builder.build()
+
+
+def warning_response(message: str, details: str | None = None) -> dict:
+    """Build warning response for non-fatal issues.
+
+    Args:
+        message: Warning message text.
+        details: Additional details.
+    """
+    builder = markdown().element("alert", message=message, level="warning")
+    if details:
+        builder.text(details)
+    return builder.build()
+
+
+def invalid_options_error(message: str, expected: dict = None) -> dict:  # pyright: ignore[reportArgumentType]
+    """Create error response for invalid options dict.
+
+    Args:
+        message: Error message describing the issue.
+        expected: Optional example of expected format.
+    """
+    builder = markdown().element("alert", message=message, level="error")
+
+    if expected:
+        builder.text("**Expected format:**")
+        import json
+
+        builder.code_block(json.dumps(expected, indent=2), language="json")
+
+    return builder.build()
+
+
+def missing_param_error(param: str, command: str = None) -> dict:  # pyright: ignore[reportArgumentType]
+    """Create error response for missing required parameter.
+
+    Args:
+        param: Name of the missing parameter.
+        command: Optional command name for context.
+    """
+    message = f"Required parameter '{param}' not provided"
+    if command:
+        message = f"{command}: {message}"
+
+    builder = markdown().element("alert", message=message, level="error")
+    builder.text(f"The '{param}' parameter is required for this operation")
+
+    return builder.build()

webtap/commands/_tips.py

@@ -0,0 +1,147 @@
+"""Parser for TIPS.md documentation.
+
+This module reads TIPS.md and provides:
+- MCP descriptions for commands
+- Developer tips for command responses
+- Pre-imported libraries documentation
+"""
+
+import re
+from pathlib import Path
+from typing import Dict, List, Optional
+
+
+class TipsParser:
+    """Parse TIPS.md for command documentation."""
+
+    def __init__(self):
+        # TIPS.md is in the same directory as this file
+        self.tips_path = Path(__file__).parent / "TIPS.md"
+        self.content = self.tips_path.read_text() if self.tips_path.exists() else ""
+        self._cache = {}
+
+    def _get_libraries(self) -> str:
+        """Extract the libraries section."""
+        if "libraries" not in self._cache:
+            match = re.search(r"## Libraries\n(.*?)(?=\n##)", self.content, re.DOTALL)
+            self._cache["libraries"] = match.group(1).strip() if match else ""
+        return self._cache["libraries"]
+
+    def _get_command_section(self, command: str) -> Optional[str]:
+        """Get the full section for a command."""
+        # Simple and explicit - look for the exact command name
+        # Use negative lookahead to ensure we match ### but not ####
+        pattern = rf"### {re.escape(command)}\n(.*?)(?=\n###(?!#)|\Z)"
+        match = re.search(pattern, self.content, re.DOTALL)
+        return match.group(1).strip() if match else None
+
+    def _get_description(self, command: str) -> Optional[str]:
+        """Get command description (text before #### sections)."""
+        section = self._get_command_section(command)
+        if not section:
+            return None
+
+        # Extract text before first #### heading
+        match = re.match(r"(.*?)(?=\n####|\Z)", section, re.DOTALL)
+        return match.group(1).strip() if match else ""
+
+    def _get_examples(self, command: str) -> Optional[str]:
+        """Get examples section for a command."""
+        section = self._get_command_section(command)
+        if not section:
+            return None
+
+        # Extract Examples section
+        match = re.search(r"#### Examples\n```python\n(.*?)\n```", section, re.DOTALL)
+        return match.group(1).strip() if match else None
+
+    def _get_tips(self, command: str, context: Optional[Dict] = None) -> Optional[List[str]]:
+        """Get tips list for a command."""
+        section = self._get_command_section(command)
+        if not section:
+            return None
+
+        # Extract Tips section
+        match = re.search(r"#### Tips\n(.*?)(?=\n###|\n##|\Z)", section, re.DOTALL)
+        if not match:
+            return None
+
+        tips_text = match.group(1)
+        # Parse bullet points
+        tips = re.findall(r"^- (.+)$", tips_text, re.MULTILINE)
+
+        # Apply context substitutions
+        if context and tips:
+            formatted_tips = []
+            for tip in tips:
+                for key, value in context.items():
+                    tip = tip.replace(f"{{{key}}}", str(value))
+                formatted_tips.append(tip)
+            return formatted_tips
+
+        return tips
+
+    def _get_mcp_description(self, command: str) -> Optional[str]:
+        """Build MCP description from markdown."""
+        description = self._get_description(command)
+        if not description:
+            return None
+
+        # Build complete MCP description
+        parts = [description]
+
+        # Add libraries section for certain commands
+        if command in ["body", "inspect"]:
+            parts.append("")
+            parts.append(self._get_libraries())
+
+        # Add examples if available
+        examples = self._get_examples(command)
+        if examples:
+            parts.append("")
+            parts.append("Examples:")
+            # Indent examples
+            for line in examples.split("\n"):
+                parts.append(f"  {line}" if line else "")
+
+        return "\n".join(parts)
+
+
+# Global parser instance
+parser = TipsParser()
+
+
+# Public API
+def get_mcp_description(command: str) -> Optional[str]:
+    """Get MCP description for a command from TIPS.md.
+
+    Args:
+        command: Name of the command.
+    """
+    return parser._get_mcp_description(command)
+
+
+def get_tips(command: str, context: Optional[Dict] = None) -> Optional[List[str]]:
+    """Get developer tips for a command from TIPS.md.
+
+    Args:
+        command: Name of the command.
+        context: Optional context for variable substitution.
+    """
+    return parser._get_tips(command, context)
+
+
+def get_all_tips() -> Dict[str, List[str]]:
+    """Get all available tips from TIPS.md."""
+    all_tips = {}
+
+    # Find all command sections
+    pattern = r"### ([^\n]+)\n"
+    matches = re.findall(pattern, parser.content)
+
+    for command in matches:
+        tips = parser._get_tips(command)
+        if tips:
+            all_tips[command] = tips
+
+    return all_tips