ta-studio-mcp 1.2.4 → 1.4.0

package/README.md

@@ -1,98 +1,27 @@
-# ta-studio-mcp 🚀
-
-**The definitive domain knowledge layer for AI agents building mobile test automation at TA Studio.**
-
-AI agents often struggle with project-specific context, unique navigation patterns, and "tribal knowledge" about past bugs. `ta-studio-mcp` solves this by giving your agent (Claude, Cursor, Windsurf) structured, programmatic access to the team's methodologies, codebase maps, and verified bug fixes.
-
----
-
-## 📋 Prerequisites
-
-- **Node.js**: `v18.0.0` or higher.
-- **MCP Client**: An IDE or tool that supports the [Model Context Protocol](https://modelcontextprotocol.io) (e.g., Claude Desktop, Cursor, Windsurf, VS Code).
-
----
-
-## ⚡ Quick Start
-
-```bash
-npx ta-studio-mcp
+# ta-studio-mcp
+
+Thin local relay for TA Studio. Connects your machine to the TA Studio server via outbound WebSocket.
+
+## Quick Start
+
+Add to your Claude Code `.mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "ta-studio": {
+      "command": "npx",
+      "args": ["ta-studio-mcp@latest"],
+      "env": {
+        "TA_API_KEY": "sk-your-key"
+      }
+    }
+  }
+}
 ```
 
----
-
-## 🧠 Expert Knowledge & Deep Technical Lore
-
-This section documents the state-of-the-art implementations used by the TA Studio team.
-
-### 1. Set-of-Mark (SoM) Screenshot Annotation
-Based on OmniParser's SoM approach, we use color-coded, type-aware bounding boxes to provide visual anchors.
-- **Type-Aware Palette**: 9 distinct colors (e.g., **Dodger Blue** for buttons, **Orange** for inputs, **Purple** for toggles).
-- **PIL Threading**: `asyncio.to_thread(_draw_bounding_boxes_threaded)` for non-blocking UI drawing.
-- **TOON Optimization**: **Token Optimized Object Notation** reduces prompt tokens by 40% by stripping redundant XML.
-- **Scaling Correction**: Screenshots are compressed to ~45% resolution. We apply `native_coord * (img_width / native_width)` for pixel-perfect alignment.
-
-### 2. Deep Subagent Handoff Protocol
-Our "Deep Agent Pattern" orchestrates specialized specialists via a strict chain of custody:
-1. **Perceptor** (`Screen Classifier`): Returns structured state and **TOON** elements.
-2. **Planner** (`Device Agent`): Proposes action based on the identified UI state.
-3. **Guardrail** (`Action Verifier`): Applies **Boolean Verification** (Safe/Relevant/Executable).
-4. **Actor** (`Mobile MCP`): Executes the approved action on the target device.
-5. **Doctor** (`Failure Diagnosis`): Categorizes failures and suggests recovery (Jitter/Wait/Backtrack).
-
-### 3. Boolean Verification vs. Numerical Scoring
-We reject "0.85 confidence" scores. Every action must pass three binary checks:
-- **is_safe**: Does this action cause data loss or unauthorized access? (YES/NO)
-- **is_relevant**: Does this move the needle on the task goal? (YES/NO)
-- **is_executable**: Is the target realistically reachable on the current screen? (YES/NO)
-- **Logic**: Action executes ONLY if ALL checks are YES. Reject and propose an `alternative_action` otherwise.
-
-### 4. Real-Time HUD & Parallel Execution
-- **Observation Pipeline**: Achieve `<200ms` lag via `on_step` async callbacks that emit SSE events to the frontend.
-- **Concurrency Control**: `asyncio.Semaphore` and per-simulation `asyncio.Lock` manage multiple parallel device streams without resource collision.
-- **Retention**: Automated 24h age or 100 total simulations cleanup before auto-purge.
-
-### 5. Model Tiering (Jan 2026 Standard)
-- **Thinking Tier (GPT-5.2)**: High-level orchestration (Coordinator) and complex visual reasoning. reasoning effort: `high`.
-- **Core Tier (GPT-5-mini)**: Specialized agents (Classifier, Verifier, Diagnosis). *Never use nano for classification.*
-- **Utility Tier (GPT-5-nano)**: MCP tool call formatting and data distillation.
-
----
-
-## 🐞 Critical Bug Fixes (Implementation Level)
-
-| Severity | Issue | Root Cause & Expert Fix |
-|----------|-------|-------------------------|
-| **CRITICAL** | Bbox Misalignment | **RC**: 45% Scaling Delta. **Fix**: Apply `img_width / native_width` factor. |
-| **CRITICAL** | Async to_thread | **RC**: CORO vs CALL mismatch. **Fix**: Remove `async` from functions passed to `asyncio.to_thread`. |
-| **CRITICAL** | Race Condition | **RC**: Parallel session state collision. **Fix**: Set `parallel_tool_calls=False`. |
-| **HIGH** | Simulation Leak | **RC**: Memory persistence. **Fix**: 24h/100-run auto-purge with `asyncio.Lock`. |
-| **HIGH** | Mobile MCP Bug | **RC**: Offline device fail (v0.0.36). **Fix**: Full ADB bridge fallback (screencap/uiautomator). |
-
----
-
-## 🔄 Core Workflows
-
-### The Ralph Loop (Closed-Loop Verification)
-1. **CODE** → Implement feature or fix.
-2. **LINT** → `mypy` / `eslint` verification.
-3. **UNIT TEST** → Specific module verification.
-4. **CHECK ASYNC** → Confirm `to_thread` safety.
-5. **VERIFY HUD** → Watch the emulator stream while agent runs autonomously.
-
----
-
-## 📦 Installation & Setup
-
-### Claude Desktop
-```bash
-claude mcp add ta-studio -- npx -y ta-studio-mcp@latest
-```
-
-### Cursor / Windsurf
-Add `npx -y ta-studio-mcp@latest` as a command-type MCP server.
-
----
+## Requirements
 
-## 📜 License
-MIT © 2026 TA Studios.
+- Python 3.10+
+- `websockets` pip package (`pip install websockets`)
+- Android SDK with ADB (for emulator control)

package/__init__.py

@@ -0,0 +1,14 @@
+"""
+TA Studio MCP — Thin local relay package.
+
+This package runs on the user's machine and connects OUT to the TA Studio
+server via outbound WebSocket. No ports are opened. No tunnel required.
+
+Components:
+  - ws_client: Outbound WebSocket connection with auto-reconnect
+  - auth: API key authentication for the handshake
+  - emulator_relay: Receives and executes ADB commands from server
+  - stream: Captures and streams emulator frames to server
+"""
+
+__version__ = "1.0.0"

package/auth.py

@@ -0,0 +1,53 @@
+"""
+TA Studio — API key authentication for outbound WebSocket handshake.
+
+The thin relay authenticates with the TA Studio server by sending an API key
+during the WebSocket upgrade. The server validates the key and associates the
+connection with the user's account.
+"""
+
+from __future__ import annotations
+
+import os
+import json
+import logging
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# API key resolution
+# ---------------------------------------------------------------------------
+
+def get_api_key() -> str:
+    """Return the TA Studio API key from environment or local config.
+
+    Resolution order:
+      1. TA_API_KEY environment variable
+      2. TA_MCP_TOKEN environment variable (legacy compat)
+      3. ~/.ta-studio/config.json → api_key field
+    """
+    key = os.getenv("TA_API_KEY") or os.getenv("TA_MCP_TOKEN", "")
+    if key:
+        return key.strip()
+
+    config_path = os.path.expanduser("~/.ta-studio/config.json")
+    try:
+        with open(config_path) as f:
+            data = json.load(f)
+            return (data.get("api_key") or "").strip()
+    except (FileNotFoundError, json.JSONDecodeError, KeyError):
+        pass
+
+    return ""
+
+
+def build_auth_headers(api_key: Optional[str] = None) -> dict[str, str]:
+    """Return headers dict for the WebSocket upgrade request."""
+    key = api_key or get_api_key()
+    if not key:
+        raise ValueError(
+            "No TA Studio API key found. Set TA_API_KEY env var or run the "
+            "install script to configure ~/.ta-studio/config.json"
+        )
+    return {"Authorization": f"Bearer {key}"}

package/bin/cli.js

@@ -0,0 +1,69 @@
+#!/usr/bin/env node
+"use strict";
+
+const { spawn, execFileSync } = require("child_process");
+const path = require("path");
+
+const PKG_DIR = path.resolve(__dirname, "..");
+
+// --- Locate a working Python 3 interpreter -------------------------------- //
+
+function findPython() {
+  for (const bin of ["python3", "python"]) {
+    try {
+      const ver = execFileSync(bin, ["--version"], {
+        encoding: "utf8",
+        stdio: ["ignore", "pipe", "ignore"],
+      }).trim();
+      const major = parseInt((ver.match(/(\d+)\./) || [])[1], 10);
+      if (major >= 3) return bin;
+    } catch {
+      // not found — try next
+    }
+  }
+  return null;
+}
+
+// --- Main ----------------------------------------------------------------- //
+
+const python = findPython();
+
+if (!python) {
+  process.stderr.write(
+    [
+      "",
+      "  ta-studio-mcp: Python 3 not found.",
+      "",
+      "  This relay requires Python 3.10+ with the 'websockets' package.",
+      "  Install Python from https://www.python.org/downloads/ and then run:",
+      "",
+      "    pip install websockets",
+      "",
+      "",
+    ].join("\n")
+  );
+  process.exit(1);
+}
+
+const child = spawn(python, [path.join(PKG_DIR, "main.py")], {
+  stdio: "inherit",
+  env: {
+    ...process.env,
+    PYTHONPATH: PKG_DIR + (process.env.PYTHONPATH ? path.delimiter + process.env.PYTHONPATH : ""),
+  },
+});
+
+// Forward signals so the Python process shuts down cleanly.
+for (const sig of ["SIGINT", "SIGTERM"]) {
+  process.on(sig, () => {
+    child.kill(sig);
+  });
+}
+
+child.on("exit", (code, signal) => {
+  if (signal) {
+    process.kill(process.pid, signal);
+  } else {
+    process.exit(code ?? 1);
+  }
+});

package/emulator_relay.py

@@ -0,0 +1,241 @@
+"""
+TA Studio — Emulator relay.
+
+Receives emulator commands from the TA Studio server via WebSocket and
+executes them locally via ADB. Results are sent back through the same
+connection.
+
+Supported commands:
+  - adb_shell: run an ADB shell command
+  - adb_install: install an APK
+  - launch_emulator: start an emulator AVD
+  - list_devices: list connected ADB devices
+  - tap / swipe / type_text / press_key: input events
+  - screenshot: capture and return a screenshot
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import os
+import shutil
+import subprocess
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+def _adb_path() -> str:
+    """Resolve ADB binary path."""
+    android_home = os.environ.get(
+        "ANDROID_HOME",
+        os.path.expanduser("~/Library/Android/sdk"),
+    )
+    adb = os.path.join(android_home, "platform-tools", "adb")
+    if os.path.isfile(adb):
+        return adb
+    # Fallback to PATH
+    found = shutil.which("adb")
+    return found or "adb"
+
+
+async def _run_adb(*args: str, device: str | None = None) -> dict[str, Any]:
+    """Run an ADB command and return stdout/stderr/returncode."""
+    cmd = [_adb_path()]
+    if device:
+        cmd += ["-s", device]
+    cmd += list(args)
+
+    proc = await asyncio.create_subprocess_exec(
+        *cmd,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    stdout, stderr = await proc.communicate()
+    return {
+        "stdout": stdout.decode(errors="replace"),
+        "stderr": stderr.decode(errors="replace"),
+        "returncode": proc.returncode,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Command handlers
+# ---------------------------------------------------------------------------
+
+
+async def handle_list_devices(_payload: dict[str, Any]) -> dict[str, Any]:
+    """List connected ADB devices."""
+    result = await _run_adb("devices", "-l")
+    lines = result["stdout"].strip().split("\n")[1:]  # skip header
+    devices = []
+    for line in lines:
+        parts = line.split()
+        if len(parts) >= 2:
+            devices.append({"serial": parts[0], "state": parts[1]})
+    return {"devices": devices}
+
+
+async def handle_adb_shell(payload: dict[str, Any]) -> dict[str, Any]:
+    """Run an ADB shell command."""
+    command = payload.get("command", "")
+    device = payload.get("device")
+    if not command:
+        return {"error": "No command provided"}
+    return await _run_adb("shell", command, device=device)
+
+
+async def handle_adb_install(payload: dict[str, Any]) -> dict[str, Any]:
+    """Install an APK on a device."""
+    apk_path = payload.get("apk_path", "")
+    device = payload.get("device")
+    if not apk_path or not os.path.isfile(apk_path):
+        return {"error": f"APK not found: {apk_path}"}
+    return await _run_adb("install", "-r", apk_path, device=device)
+
+
+async def handle_launch_emulator(payload: dict[str, Any]) -> dict[str, Any]:
+    """Launch an Android emulator AVD."""
+    avd_name = payload.get("avd_name", "")
+    android_home = os.environ.get(
+        "ANDROID_HOME",
+        os.path.expanduser("~/Library/Android/sdk"),
+    )
+    emulator_bin = os.path.join(android_home, "emulator", "emulator")
+    if not os.path.isfile(emulator_bin):
+        return {"error": "Emulator binary not found"}
+
+    if not avd_name:
+        # List available AVDs and pick the first
+        proc = await asyncio.create_subprocess_exec(
+            emulator_bin, "-list-avds",
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+        stdout, _ = await proc.communicate()
+        avds = [l.strip() for l in stdout.decode().strip().split("\n") if l.strip()]
+        if not avds:
+            return {"error": "No AVDs available"}
+        avd_name = avds[0]
+
+    # Launch in background
+    proc = await asyncio.create_subprocess_exec(
+        emulator_bin,
+        "-avd", avd_name,
+        "-no-snapshot",
+        "-gpu", "swiftshader_indirect",
+        "-no-boot-anim",
+        stdout=asyncio.subprocess.DEVNULL,
+        stderr=asyncio.subprocess.DEVNULL,
+    )
+    return {"avd_name": avd_name, "pid": proc.pid, "status": "launching"}
+
+
+async def handle_screenshot(payload: dict[str, Any]) -> dict[str, Any]:
+    """Capture a screenshot and return as base64 PNG."""
+    device = payload.get("device")
+    result = await _run_adb("exec-out", "screencap", "-p", device=device)
+    if result["returncode"] != 0:
+        return {"error": result["stderr"]}
+    # screencap -p via exec-out returns raw PNG bytes in stdout
+    # Re-run with raw output
+    cmd = [_adb_path()]
+    if device:
+        cmd += ["-s", device]
+    cmd += ["exec-out", "screencap", "-p"]
+
+    proc = await asyncio.create_subprocess_exec(
+        *cmd,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    stdout, stderr = await proc.communicate()
+    if proc.returncode != 0:
+        return {"error": stderr.decode(errors="replace")}
+
+    return {"image_base64": base64.b64encode(stdout).decode(), "format": "png"}
+
+
+async def handle_tap(payload: dict[str, Any]) -> dict[str, Any]:
+    """Tap at coordinates."""
+    x, y = payload.get("x", 0), payload.get("y", 0)
+    device = payload.get("device")
+    return await _run_adb("shell", f"input tap {x} {y}", device=device)
+
+
+async def handle_swipe(payload: dict[str, Any]) -> dict[str, Any]:
+    """Swipe between coordinates."""
+    x1, y1 = payload.get("x1", 0), payload.get("y1", 0)
+    x2, y2 = payload.get("x2", 0), payload.get("y2", 0)
+    duration = payload.get("duration", 300)
+    device = payload.get("device")
+    return await _run_adb(
+        "shell", f"input swipe {x1} {y1} {x2} {y2} {duration}",
+        device=device,
+    )
+
+
+async def handle_type_text(payload: dict[str, Any]) -> dict[str, Any]:
+    """Type text on the device."""
+    text = payload.get("text", "")
+    device = payload.get("device")
+    # Escape spaces for ADB input
+    escaped = text.replace(" ", "%s")
+    return await _run_adb("shell", f"input text '{escaped}'", device=device)
+
+
+async def handle_press_key(payload: dict[str, Any]) -> dict[str, Any]:
+    """Press a key event (e.g., KEYCODE_HOME)."""
+    keycode = payload.get("keycode", "KEYCODE_HOME")
+    device = payload.get("device")
+    return await _run_adb("shell", f"input keyevent {keycode}", device=device)
+
+
+# ---------------------------------------------------------------------------
+# Command dispatcher
+# ---------------------------------------------------------------------------
+
+COMMAND_HANDLERS: dict[str, Any] = {
+    "list_devices": handle_list_devices,
+    "adb_shell": handle_adb_shell,
+    "adb_install": handle_adb_install,
+    "launch_emulator": handle_launch_emulator,
+    "screenshot": handle_screenshot,
+    "tap": handle_tap,
+    "swipe": handle_swipe,
+    "type_text": handle_type_text,
+    "press_key": handle_press_key,
+}
+
+
+async def dispatch_command(msg: dict[str, Any]) -> dict[str, Any]:
+    """Dispatch an incoming command message to the appropriate handler.
+
+    Expected message format:
+        {"type": "command", "command": "<name>", "request_id": "...", ...payload}
+
+    Returns a response dict with the same request_id.
+    """
+    command = msg.get("command", "")
+    request_id = msg.get("request_id", "")
+    handler = COMMAND_HANDLERS.get(command)
+
+    if not handler:
+        return {
+            "type": "response",
+            "request_id": request_id,
+            "error": f"Unknown command: {command}",
+        }
+
+    try:
+        result = await handler(msg)
+        return {"type": "response", "request_id": request_id, **result}
+    except Exception as e:
+        logger.exception("Command '%s' failed", command)
+        return {
+            "type": "response",
+            "request_id": request_id,
+            "error": str(e),
+        }

package/main.py

@@ -0,0 +1,105 @@
+"""
+TA Studio MCP — Entry point for the thin local relay.
+
+Usage:
+    python -m ta-studio-mcp
+    # or
+    npx ta-studio-mcp@latest
+
+Connects outbound to the TA Studio server via WebSocket. Receives emulator
+commands, executes them locally via ADB, and streams results back.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import sys
+
+from .auth import build_auth_headers
+from .ws_client import TAWebSocketClient, DEFAULT_SERVER_URL
+from .emulator_relay import dispatch_command
+from .stream import FrameStreamer
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+logger = logging.getLogger("ta-studio-mcp")
+
+
+async def main() -> None:
+    """Start the thin relay — connect out to TA Studio server."""
+    server_url = os.getenv("TA_STUDIO_URL", "").rstrip("/")
+    if server_url:
+        # Convert HTTP URL to WebSocket URL
+        ws_url = server_url.replace("https://", "wss://").replace("http://", "ws://")
+        if not ws_url.endswith("/ws/agent-relay"):
+            ws_url += "/ws/agent-relay"
+    else:
+        ws_url = DEFAULT_SERVER_URL
+
+    logger.info("TA Studio thin relay starting")
+    logger.info("Server: %s", ws_url)
+
+    # Build auth headers
+    try:
+        auth_headers = build_auth_headers()
+    except ValueError as e:
+        logger.error(str(e))
+        sys.exit(1)
+
+    client = TAWebSocketClient(server_url=ws_url, auth_headers=auth_headers)
+    streamer = FrameStreamer(send_fn=client.send)
+
+    # -- Handle commands from server -----------------------------------------
+
+    async def on_command(msg: dict) -> None:
+        """Handle command messages from the TA agent."""
+        response = await dispatch_command(msg)
+        await client.send(response)
+
+    async def on_stream_start(msg: dict) -> None:
+        """Start frame streaming when server requests it."""
+        session_id = msg.get("session_id", "default")
+        device = msg.get("device")
+        fps = msg.get("fps", 2)
+        streamer._device = device
+        streamer._fps = min(fps, 10)
+        await streamer.start(session_id)
+
+    async def on_stream_stop(_msg: dict) -> None:
+        """Stop frame streaming."""
+        await streamer.stop()
+
+    async def on_connected(_msg: dict) -> None:
+        """Send identity info on connection."""
+        from .emulator_relay import handle_list_devices
+        devices = await handle_list_devices({})
+        await client.send({
+            "type": "relay_ready",
+            "version": "1.0.0",
+            "devices": devices.get("devices", []),
+        })
+
+    client.on("command", on_command)
+    client.on("stream_start", on_stream_start)
+    client.on("stream_stop", on_stream_stop)
+    client.on("connected", on_connected)
+
+    # Run until interrupted
+    try:
+        await client.connect()
+    except KeyboardInterrupt:
+        await client.disconnect()
+
+
+def run() -> None:
+    """Synchronous entry point."""
+    asyncio.run(main())
+
+
+if __name__ == "__main__":
+    run()

package/package.json

@@ -1,59 +1,25 @@
 {
   "name": "ta-studio-mcp",
-  "version": "1.2.4",
-  "description": "TA Studio MCP — Domain knowledge, patterns, bug fixes, and workflows for AI agents working on the TA Studio mobile test automation platform.",
-  "type": "module",
+  "version": "1.4.0",
+  "description": "TA Studio MCP — thin local relay for AI-driven mobile QA",
   "bin": {
-    "ta-studio-mcp": "dist/index.js"
+    "ta-studio-mcp": "bin/cli.js"
   },
-  "main": "./dist/index.js",
-  "files": [
-    "dist",
-    "README.md",
-    "LICENSE"
-  ],
-  "scripts": {
-    "build": "tsc",
-    "dev": "tsc --watch",
-    "start": "node dist/index.js",
-    "prepublishOnly": "npm run build"
-  },
-  "repository": {
-    "type": "git",
-    "url": "git+https://github.com/TA-Studios-AI-Avengers/ta-agent-examples.git",
-    "directory": "packages/ta-studio-mcp"
-  },
-  "homepage": "https://github.com/TA-Studios-AI-Avengers/ta-agent-examples/tree/main/packages/ta-studio-mcp#readme",
-  "bugs": {
-    "url": "https://github.com/TA-Studios-AI-Avengers/ta-agent-examples/issues"
-  },
-  "author": "TA Studios",
   "keywords": [
     "mcp",
-    "model-context-protocol",
-    "mobile-testing",
+    "testing",
+    "qa",
     "android",
-    "test-automation",
-    "ai-agents",
-    "ta-studio",
-    "claude",
-    "openai",
-    "agentic",
-    "oavr",
-    "som-annotation",
-    "device-testing",
-    "screenshot-annotation"
+    "emulator",
+    "claude-code"
   ],
   "license": "MIT",
-  "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.21.1",
-    "zod": "^3.23.8"
-  },
-  "devDependencies": {
-    "@types/node": "^22.0.0",
-    "typescript": "^5.5.4"
-  },
   "engines": {
     "node": ">=18"
-  }
-}
+  },
+  "files": [
+    "bin/",
+    "*.py",
+    "README.md"
+  ]
+}