ta-studio-mcp 1.3.0 → 1.4.0

package/README.md

@@ -1,144 +1,27 @@
-# ta-studio-mcp 🚀
-
-**The definitive domain knowledge layer for AI agents building mobile test automation at TA Studio.**
-
-AI agents often struggle with project-specific context, unique navigation patterns, and "tribal knowledge" about past bugs. `ta-studio-mcp` solves this by giving your agent (Claude, Cursor, Windsurf) structured, programmatic access to the team's methodologies, codebase maps, and verified bug fixes.
-
----
-
-## ⚡ Quick Start
-
-```bash
-npx ta-studio-mcp
-```
-
----
-
-## 🎨 Figma Flow Analysis Pipeline
-
-The cornerstone of TA Studio's design-to-test workflow is a sophisticated 3-phase analysis pipeline that converts complex Figma documents into actionable test plans.
-
-### Phase 1: Direct Extraction
-We use the Figma REST API with a specific recursive traversal logic:
-- **Depth-3 Extraction**: `DOC` -> `CANVAS` -> `SECTION` -> `FRAME`.
-- **Logic**: We stop at the Frame level to capture screens. Critical insight: standard `depth=2` extractions only reach the Section level, often missing the actual UI Frames nested within.
-- **Node Analysis**: Every frame is analyzed for its name, `transitionNodeID` (prototype links), and spatial coordinates.
-
-### Phase 2: Multi-Signal Priority Clustering
-To group screens into logical user flows (e.g., "Onboarding", "Checkout"), we use a cascade of grouping signals in order of reliability:
-1. **Section-Based**: (Highest Priority) Uses Figma's native `Section` grouping if available.
-2. **Prototype Connections**: Uses Union-Find algorithm on `transitionNodeID` links to group screens that are functionally connected.
-3. **Name-Prefix Matching**: Split by common naming separators like ` / `, ` - `, or ` — `.
-4. **Spatial Clustering**: (Lowest Priority) Groups by proximity using Y-binning and X-gap splitting.
-
-### Phase 3: Visual Overlay & CV Fallback
-- **PIL Visualizer**: Generates a high-contrast overlay with 12 distinct colors, semi-transparent fills (alpha=40), and strong outlines (alpha=200).
-- **Rate-Limit Fallback**: When the Figma Images API is rate-limited (common with `429` errors), we switch to a Computer Vision (CV) pipeline:
-    - **Brightness Thresholding**: Identify sections and frames by detecting canvas brightness deltas (>80 for sections, >100 for frames).
-    - **Morphological Opening**: Uses `scipy.ndimage` to clean up noise and bridge gaps in detected outlines.
-    - **Connected Component Analysis**: Reconstructs frame hierarchies from pixel data when API metadata is unavailable.
-
-Key files:
-- `backend/app/figma/flow_analyzer.py` (707 lines)
-- `scripts/figma_cv_overlay.py` (162 lines)
-
----
-
-## 📱 Device Testing & Simulation Lifecycle
-
-The TA Studio backend manages thousands of automated simulation steps across a fleet of Android emulators.
-
-### Concurrency & Thread Safety
-- **Async Execution**: Each device task runs in its own `asyncio.Task`.
-- **Semaphore Guard**: A global `asyncio.Semaphore(max_concurrent)` prevents host resource exhaustion.
-- **Simulation Lock**: Per-simulation `asyncio.Lock` ensures that result indexing is thread-safe and deterministic.
-
-### Simulation States
-`queued` -> `running` -> `completed` | `failed` | `cancelled`
-- **Auto-Purge**: 24h age-out or 100 total simulations retention limit to prevent memory bloat.
-- **Device Auto-Select**: Automatically ranks devices (`emulator-5554` first, then general emulators, then physical devices).
-
-Key file: `backend/app/agents/coordinator/coordinator_service.py`
-
----
-
-## 👁️ Vision Click & Agentic Visual Reasoning
-
-When the accessibility tree (`list_elements_on_screen`) fails—common in canvas-based UIs or custom views—we switch to Agentic Vision.
-
-### Two-Layer Architecture
-1. **Layer 1: SoM Annotation**: Deterministic, fast (<100ms) annotation of the screenshot using color-coded bounding boxes derived from the element list.
-2. **Layer 2: GPT-5.2 Agentic Vision**:
-    - **Think-Act-Observe**: GPT-5.2 generates specialized Python code and runs it in a `LocalCodeExecutor`.
-    - **Analysis**: The agent analyzes the SoM-annotated image + the element metadata to find the target.
-    - **Feedback**: Results are fed back into the OAVR loop for final execution.
-
-Key file: `backend/app/agents/device_testing/agentic_vision_service.py` (847 lines)
-
----
-
-## 🔗 Mobile MCP & ADB Fallback
-
-`ta-studio-mcp` provides the bridge to `mobile-mcp` but with a critical safety layer for production stability.
-
-### The v0.0.36 Bug Fix
-Mobile MCP v0.0.36 has a critical flaw: it fails to detect *any* device if *one* device in the list is offline.
-- **The Solution**: Our `MobileMCPClient` implements a 1:1 ADB bridge fallback.
-- **ADB Commands**: Uses `exec-out screencap -p` for fast PNG streaming and `uiautomator dump /dev/tty` for zero-file-I/O UI tree extraction.
-
-Key file: `backend/app/agents/device_testing/mobile_mcp_client.py`
-
----
-
-## ⚡ Flicker Detection & Performance Metrics
-
-We detect regressions that occur faster than standard screenshot intervals (16-200ms).
-
-### 4-Layer Flicker Pipeline
-1. **Trigger**: High-speed `screenrecord` (10s limit).
-2. **Extraction**: `ffmpeg` scene filter (`gt(scene,0.003)`).
-3. **Analysis**: Consecutive frame SSIM (Structural Similarity Index) calculation. SSIM drops > 0.15 are flagged.
-4. **Verification**: GPT-5.2 Vision confirms the flicker and generates a video artifact for regression triage.
-
-Key file: `backend/app/agents/device_testing/flicker_detection_service.py`
-
----
-
-## 🐞 Critical Bug Fixes (Implementation Level)
-
-| Severity | Issue | Root Cause & Expert Fix |
-|----------|-------|-------------------------|
-| **CRITICAL** | Bbox Misalignment | **RC**: 45% Scaling Delta between native vs JPEG. **Fix**: Map coords via `img_width / native_width`. |
-| **CRITICAL** | Async to_thread | **RC**: Collision when passing `async` functions to `to_thread`. **Fix**: Ensure target is a plain function. |
-| **CRITICAL** | Race Condition | **RC**: Parallel tool calls in device sessions. **Fix**: `parallel_tool_calls=False` is mandatory. |
-| **HIGH** | Simulation Leak | **RC**: Context persistence leading to memory fail. **Fix**: 24h retention + indexing lock. |
-| **HIGH** | Figma API 429 | **RC**: Heavy polling on Figma API. **Fix**: Integrated CV Brightness thresholding fallback. |
-| **MEDIUM** | SSIM False Positive | **RC**: Normal UI transitions. **Fix**: Increased sensitivity threshold to 0.15. |
-
----
-
-## 🔄 Core Workflows
-
-### The Ralph Loop (Closed-Loop Verification)
-1. **CODE** -> Implement the feature or fix.
-2. **LINT** -> Run `mypy` and `eslint`.
-3. **UNIT TEST** -> Verify the module in isolation.
-4. **CHECK ASYNC** -> Explicitly audit `to_thread` safety and concurrency locks.
-5. **VERIFY HUD** -> Launch the emulator and watch the real-time Stream HUD during autonomous execution.
-
----
-
-## 📦 Installation & Setup
-
-### Claude Desktop
-```bash
-claude mcp add ta-studio -- npx -y ta-studio-mcp@latest
+# ta-studio-mcp
+
+Thin local relay for TA Studio. Connects your machine to the TA Studio server via outbound WebSocket.
+
+## Quick Start
+
+Add to your Claude Code `.mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "ta-studio": {
+      "command": "npx",
+      "args": ["ta-studio-mcp@latest"],
+      "env": {
+        "TA_API_KEY": "sk-your-key"
+      }
+    }
+  }
+}
 ```
 
-### Cursor / Windsurf
-Add `npx -y ta-studio-mcp@latest` as a command-type MCP server.
-
----
+## Requirements
 
-## 📜 License
-MIT © 2026 TA Studios.
+- Python 3.10+
+- `websockets` pip package (`pip install websockets`)
+- Android SDK with ADB (for emulator control)

package/__init__.py

@@ -0,0 +1,14 @@
+"""
+TA Studio MCP — Thin local relay package.
+
+This package runs on the user's machine and connects OUT to the TA Studio
+server via outbound WebSocket. No ports are opened. No tunnel required.
+
+Components:
+  - ws_client: Outbound WebSocket connection with auto-reconnect
+  - auth: API key authentication for the handshake
+  - emulator_relay: Receives and executes ADB commands from server
+  - stream: Captures and streams emulator frames to server
+"""
+
+__version__ = "1.0.0"

package/auth.py

@@ -0,0 +1,53 @@
+"""
+TA Studio — API key authentication for outbound WebSocket handshake.
+
+The thin relay authenticates with the TA Studio server by sending an API key
+during the WebSocket upgrade. The server validates the key and associates the
+connection with the user's account.
+"""
+
+from __future__ import annotations
+
+import os
+import json
+import logging
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# API key resolution
+# ---------------------------------------------------------------------------
+
+def get_api_key() -> str:
+    """Return the TA Studio API key from environment or local config.
+
+    Resolution order:
+      1. TA_API_KEY environment variable
+      2. TA_MCP_TOKEN environment variable (legacy compat)
+      3. ~/.ta-studio/config.json → api_key field
+    """
+    key = os.getenv("TA_API_KEY") or os.getenv("TA_MCP_TOKEN", "")
+    if key:
+        return key.strip()
+
+    config_path = os.path.expanduser("~/.ta-studio/config.json")
+    try:
+        with open(config_path) as f:
+            data = json.load(f)
+            return (data.get("api_key") or "").strip()
+    except (FileNotFoundError, json.JSONDecodeError, KeyError):
+        pass
+
+    return ""
+
+
+def build_auth_headers(api_key: Optional[str] = None) -> dict[str, str]:
+    """Return headers dict for the WebSocket upgrade request."""
+    key = api_key or get_api_key()
+    if not key:
+        raise ValueError(
+            "No TA Studio API key found. Set TA_API_KEY env var or run the "
+            "install script to configure ~/.ta-studio/config.json"
+        )
+    return {"Authorization": f"Bearer {key}"}

package/bin/cli.js

@@ -0,0 +1,69 @@
+#!/usr/bin/env node
+"use strict";
+
+const { spawn, execFileSync } = require("child_process");
+const path = require("path");
+
+const PKG_DIR = path.resolve(__dirname, "..");
+
+// --- Locate a working Python 3 interpreter -------------------------------- //
+
+function findPython() {
+  for (const bin of ["python3", "python"]) {
+    try {
+      const ver = execFileSync(bin, ["--version"], {
+        encoding: "utf8",
+        stdio: ["ignore", "pipe", "ignore"],
+      }).trim();
+      const major = parseInt((ver.match(/(\d+)\./) || [])[1], 10);
+      if (major >= 3) return bin;
+    } catch {
+      // not found — try next
+    }
+  }
+  return null;
+}
+
+// --- Main ----------------------------------------------------------------- //
+
+const python = findPython();
+
+if (!python) {
+  process.stderr.write(
+    [
+      "",
+      "  ta-studio-mcp: Python 3 not found.",
+      "",
+      "  This relay requires Python 3.10+ with the 'websockets' package.",
+      "  Install Python from https://www.python.org/downloads/ and then run:",
+      "",
+      "    pip install websockets",
+      "",
+      "",
+    ].join("\n")
+  );
+  process.exit(1);
+}
+
+const child = spawn(python, [path.join(PKG_DIR, "main.py")], {
+  stdio: "inherit",
+  env: {
+    ...process.env,
+    PYTHONPATH: PKG_DIR + (process.env.PYTHONPATH ? path.delimiter + process.env.PYTHONPATH : ""),
+  },
+});
+
+// Forward signals so the Python process shuts down cleanly.
+for (const sig of ["SIGINT", "SIGTERM"]) {
+  process.on(sig, () => {
+    child.kill(sig);
+  });
+}
+
+child.on("exit", (code, signal) => {
+  if (signal) {
+    process.kill(process.pid, signal);
+  } else {
+    process.exit(code ?? 1);
+  }
+});

package/emulator_relay.py

@@ -0,0 +1,241 @@
+"""
+TA Studio — Emulator relay.
+
+Receives emulator commands from the TA Studio server via WebSocket and
+executes them locally via ADB. Results are sent back through the same
+connection.
+
+Supported commands:
+  - adb_shell: run an ADB shell command
+  - adb_install: install an APK
+  - launch_emulator: start an emulator AVD
+  - list_devices: list connected ADB devices
+  - tap / swipe / type_text / press_key: input events
+  - screenshot: capture and return a screenshot
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import os
+import shutil
+import subprocess
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+def _adb_path() -> str:
+    """Resolve ADB binary path."""
+    android_home = os.environ.get(
+        "ANDROID_HOME",
+        os.path.expanduser("~/Library/Android/sdk"),
+    )
+    adb = os.path.join(android_home, "platform-tools", "adb")
+    if os.path.isfile(adb):
+        return adb
+    # Fallback to PATH
+    found = shutil.which("adb")
+    return found or "adb"
+
+
+async def _run_adb(*args: str, device: str | None = None) -> dict[str, Any]:
+    """Run an ADB command and return stdout/stderr/returncode."""
+    cmd = [_adb_path()]
+    if device:
+        cmd += ["-s", device]
+    cmd += list(args)
+
+    proc = await asyncio.create_subprocess_exec(
+        *cmd,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    stdout, stderr = await proc.communicate()
+    return {
+        "stdout": stdout.decode(errors="replace"),
+        "stderr": stderr.decode(errors="replace"),
+        "returncode": proc.returncode,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Command handlers
+# ---------------------------------------------------------------------------
+
+
+async def handle_list_devices(_payload: dict[str, Any]) -> dict[str, Any]:
+    """List connected ADB devices."""
+    result = await _run_adb("devices", "-l")
+    lines = result["stdout"].strip().split("\n")[1:]  # skip header
+    devices = []
+    for line in lines:
+        parts = line.split()
+        if len(parts) >= 2:
+            devices.append({"serial": parts[0], "state": parts[1]})
+    return {"devices": devices}
+
+
+async def handle_adb_shell(payload: dict[str, Any]) -> dict[str, Any]:
+    """Run an ADB shell command."""
+    command = payload.get("command", "")
+    device = payload.get("device")
+    if not command:
+        return {"error": "No command provided"}
+    return await _run_adb("shell", command, device=device)
+
+
+async def handle_adb_install(payload: dict[str, Any]) -> dict[str, Any]:
+    """Install an APK on a device."""
+    apk_path = payload.get("apk_path", "")
+    device = payload.get("device")
+    if not apk_path or not os.path.isfile(apk_path):
+        return {"error": f"APK not found: {apk_path}"}
+    return await _run_adb("install", "-r", apk_path, device=device)
+
+
+async def handle_launch_emulator(payload: dict[str, Any]) -> dict[str, Any]:
+    """Launch an Android emulator AVD."""
+    avd_name = payload.get("avd_name", "")
+    android_home = os.environ.get(
+        "ANDROID_HOME",
+        os.path.expanduser("~/Library/Android/sdk"),
+    )
+    emulator_bin = os.path.join(android_home, "emulator", "emulator")
+    if not os.path.isfile(emulator_bin):
+        return {"error": "Emulator binary not found"}
+
+    if not avd_name:
+        # List available AVDs and pick the first
+        proc = await asyncio.create_subprocess_exec(
+            emulator_bin, "-list-avds",
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+        stdout, _ = await proc.communicate()
+        avds = [l.strip() for l in stdout.decode().strip().split("\n") if l.strip()]
+        if not avds:
+            return {"error": "No AVDs available"}
+        avd_name = avds[0]
+
+    # Launch in background
+    proc = await asyncio.create_subprocess_exec(
+        emulator_bin,
+        "-avd", avd_name,
+        "-no-snapshot",
+        "-gpu", "swiftshader_indirect",
+        "-no-boot-anim",
+        stdout=asyncio.subprocess.DEVNULL,
+        stderr=asyncio.subprocess.DEVNULL,
+    )
+    return {"avd_name": avd_name, "pid": proc.pid, "status": "launching"}
+
+
+async def handle_screenshot(payload: dict[str, Any]) -> dict[str, Any]:
+    """Capture a screenshot and return as base64 PNG."""
+    device = payload.get("device")
+    result = await _run_adb("exec-out", "screencap", "-p", device=device)
+    if result["returncode"] != 0:
+        return {"error": result["stderr"]}
+    # screencap -p via exec-out returns raw PNG bytes in stdout
+    # Re-run with raw output
+    cmd = [_adb_path()]
+    if device:
+        cmd += ["-s", device]
+    cmd += ["exec-out", "screencap", "-p"]
+
+    proc = await asyncio.create_subprocess_exec(
+        *cmd,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    stdout, stderr = await proc.communicate()
+    if proc.returncode != 0:
+        return {"error": stderr.decode(errors="replace")}
+
+    return {"image_base64": base64.b64encode(stdout).decode(), "format": "png"}
+
+
+async def handle_tap(payload: dict[str, Any]) -> dict[str, Any]:
+    """Tap at coordinates."""
+    x, y = payload.get("x", 0), payload.get("y", 0)
+    device = payload.get("device")
+    return await _run_adb("shell", f"input tap {x} {y}", device=device)
+
+
+async def handle_swipe(payload: dict[str, Any]) -> dict[str, Any]:
+    """Swipe between coordinates."""
+    x1, y1 = payload.get("x1", 0), payload.get("y1", 0)
+    x2, y2 = payload.get("x2", 0), payload.get("y2", 0)
+    duration = payload.get("duration", 300)
+    device = payload.get("device")
+    return await _run_adb(
+        "shell", f"input swipe {x1} {y1} {x2} {y2} {duration}",
+        device=device,
+    )
+
+
+async def handle_type_text(payload: dict[str, Any]) -> dict[str, Any]:
+    """Type text on the device."""
+    text = payload.get("text", "")
+    device = payload.get("device")
+    # Escape spaces for ADB input
+    escaped = text.replace(" ", "%s")
+    return await _run_adb("shell", f"input text '{escaped}'", device=device)
+
+
+async def handle_press_key(payload: dict[str, Any]) -> dict[str, Any]:
+    """Press a key event (e.g., KEYCODE_HOME)."""
+    keycode = payload.get("keycode", "KEYCODE_HOME")
+    device = payload.get("device")
+    return await _run_adb("shell", f"input keyevent {keycode}", device=device)
+
+
+# ---------------------------------------------------------------------------
+# Command dispatcher
+# ---------------------------------------------------------------------------
+
+COMMAND_HANDLERS: dict[str, Any] = {
+    "list_devices": handle_list_devices,
+    "adb_shell": handle_adb_shell,
+    "adb_install": handle_adb_install,
+    "launch_emulator": handle_launch_emulator,
+    "screenshot": handle_screenshot,
+    "tap": handle_tap,
+    "swipe": handle_swipe,
+    "type_text": handle_type_text,
+    "press_key": handle_press_key,
+}
+
+
+async def dispatch_command(msg: dict[str, Any]) -> dict[str, Any]:
+    """Dispatch an incoming command message to the appropriate handler.
+
+    Expected message format:
+        {"type": "command", "command": "<name>", "request_id": "...", ...payload}
+
+    Returns a response dict with the same request_id.
+    """
+    command = msg.get("command", "")
+    request_id = msg.get("request_id", "")
+    handler = COMMAND_HANDLERS.get(command)
+
+    if not handler:
+        return {
+            "type": "response",
+            "request_id": request_id,
+            "error": f"Unknown command: {command}",
+        }
+
+    try:
+        result = await handler(msg)
+        return {"type": "response", "request_id": request_id, **result}
+    except Exception as e:
+        logger.exception("Command '%s' failed", command)
+        return {
+            "type": "response",
+            "request_id": request_id,
+            "error": str(e),
+        }

package/main.py

@@ -0,0 +1,105 @@
+"""
+TA Studio MCP — Entry point for the thin local relay.
+
+Usage:
+    python -m ta-studio-mcp
+    # or
+    npx ta-studio-mcp@latest
+
+Connects outbound to the TA Studio server via WebSocket. Receives emulator
+commands, executes them locally via ADB, and streams results back.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import sys
+
+from .auth import build_auth_headers
+from .ws_client import TAWebSocketClient, DEFAULT_SERVER_URL
+from .emulator_relay import dispatch_command
+from .stream import FrameStreamer
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+logger = logging.getLogger("ta-studio-mcp")
+
+
+async def main() -> None:
+    """Start the thin relay — connect out to TA Studio server."""
+    server_url = os.getenv("TA_STUDIO_URL", "").rstrip("/")
+    if server_url:
+        # Convert HTTP URL to WebSocket URL
+        ws_url = server_url.replace("https://", "wss://").replace("http://", "ws://")
+        if not ws_url.endswith("/ws/agent-relay"):
+            ws_url += "/ws/agent-relay"
+    else:
+        ws_url = DEFAULT_SERVER_URL
+
+    logger.info("TA Studio thin relay starting")
+    logger.info("Server: %s", ws_url)
+
+    # Build auth headers
+    try:
+        auth_headers = build_auth_headers()
+    except ValueError as e:
+        logger.error(str(e))
+        sys.exit(1)
+
+    client = TAWebSocketClient(server_url=ws_url, auth_headers=auth_headers)
+    streamer = FrameStreamer(send_fn=client.send)
+
+    # -- Handle commands from server -----------------------------------------
+
+    async def on_command(msg: dict) -> None:
+        """Handle command messages from the TA agent."""
+        response = await dispatch_command(msg)
+        await client.send(response)
+
+    async def on_stream_start(msg: dict) -> None:
+        """Start frame streaming when server requests it."""
+        session_id = msg.get("session_id", "default")
+        device = msg.get("device")
+        fps = msg.get("fps", 2)
+        streamer._device = device
+        streamer._fps = min(fps, 10)
+        await streamer.start(session_id)
+
+    async def on_stream_stop(_msg: dict) -> None:
+        """Stop frame streaming."""
+        await streamer.stop()
+
+    async def on_connected(_msg: dict) -> None:
+        """Send identity info on connection."""
+        from .emulator_relay import handle_list_devices
+        devices = await handle_list_devices({})
+        await client.send({
+            "type": "relay_ready",
+            "version": "1.0.0",
+            "devices": devices.get("devices", []),
+        })
+
+    client.on("command", on_command)
+    client.on("stream_start", on_stream_start)
+    client.on("stream_stop", on_stream_stop)
+    client.on("connected", on_connected)
+
+    # Run until interrupted
+    try:
+        await client.connect()
+    except KeyboardInterrupt:
+        await client.disconnect()
+
+
+def run() -> None:
+    """Synchronous entry point."""
+    asyncio.run(main())
+
+
+if __name__ == "__main__":
+    run()