agent-portal-2 0.1.0

package/docs/MCP_SERVER.md

@@ -0,0 +1,68 @@
+# MCP Server
+
+## What MCP Is
+
+MCP, or Model Context Protocol, is a structured way for AI clients to discover and call tools through a standard interface.
+
+## Why Agent Portal Uses MCP
+
+Agent Portal already has a local runtime, policy engine, browser control layer, reporting, and steering controls. MCP turns those capabilities into a consistent tool surface that other AI clients can connect to without custom one-off integrations.
+
+## Start The Runtime
+
+```bash
+agent-portal start
+```
+
+## Start The MCP Server
+
+```bash
+agent-portal mcp start
+```
+
+or:
+
+```bash
+agent-portal-mcp start
+```
+
+## Available Tool Groups
+
+- Browser
+- Navigation
+- Inspection
+- Agent Steering
+- Reports
+
+## Security Model
+
+- localhost-first runtime URL by default
+- optional bearer token through `AGENT_PORTAL_TOKEN` or `agent-portal.config.json`
+- risky actions are classified before execution
+- blocked actions are never auto-executed
+- approval-required actions stay in the queue until approved
+- secret tokens are redacted from MCP errors
+
+## Approval Flow
+
+1. A risky MCP tool proposes an action through the Agent Portal runtime.
+2. The runtime applies its policy engine and assigns a risk level.
+3. Safe actions can auto-execute.
+4. Higher-risk actions become `pending_approval`.
+5. `approve_action` can approve and execute a queued action.
+6. `reject_action` rejects the queued action.
+
+## Example Tool Calls
+
+- `browser_open`
+- `navigate_to_url`
+- `click_element`
+- `capture_screenshot`
+- `get_action_queue`
+- `generate_report`
+
+## Troubleshooting
+
+- If the runtime is offline, start it with `agent-portal start`.
+- If token auth is enabled, set `AGENT_PORTAL_TOKEN`.
+- If tools are not visible, run `agent-portal mcp doctor`.

package/docs/architecture.md

@@ -0,0 +1,214 @@
+# Agent Portal Architecture
+
+## Product Shape
+
+Agent Portal is best treated as a platform, not a single app. The architecture needs to support:
+
+- local-first execution
+- visual awareness
+- controlled actions
+- multi-agent coordination
+- replayable sessions
+- durable memory
+- explicit permissions
+
+## Core Layers
+
+### 1. Desktop Shell
+
+Responsibilities:
+
+- native window management
+- embedded browser surfaces
+- agent console UI
+- session inspector
+- settings and permissions
+
+Recommended implementation path:
+
+- start with Electron for speed and ecosystem reach
+- evaluate Tauri later if footprint becomes a major constraint
+
+### 2. Agent Runtime
+
+Responsibilities:
+
+- spawn and isolate agents
+- manage task state
+- coordinate multi-agent execution
+- stream events to the UI
+- attach tools and permissions
+
+Key concepts:
+
+- agent definitions
+- task queue
+- execution policies
+- event bus
+- cancellation and recovery
+
+### 3. Vision Pipeline
+
+Responsibilities:
+
+- capture screenshots
+- read DOM structure
+- parse accessibility trees
+- run OCR
+- detect actionable UI elements
+- emit state diffs over time
+
+Output contract:
+
+- normalized `VisualSnapshot`
+- element list with labels, selectors, roles, and bounds
+- changed/appeared/disappeared annotations
+
+### 3.5 Vision Core
+
+Responsibilities:
+
+- fuse screenshots, DOM, accessibility, console, and network evidence
+- classify the current interface
+- infer likely user intent
+- generate root-cause hypotheses when flows fail
+- hand structured context to goal planning and memory
+
+Initial implementation path:
+
+- heuristics and typed contracts first
+- model-assisted reasoning layer second
+- confidence scoring and self-healing action selection third
+
+### 4. Browser Control Layer
+
+Responsibilities:
+
+- navigation
+- clicking
+- typing
+- waiting
+- file upload and download
+- script execution
+- trace capture
+
+Recommended implementation path:
+
+- Playwright as the first execution backend
+- adapter abstraction so Selenium or remote browsers can be added later
+
+### 5. Desktop Control Layer
+
+Responsibilities:
+
+- window discovery
+- input simulation
+- native app inspection
+- application-specific adapters
+
+Recommended implementation path:
+
+- define the abstraction now
+- ship browser-first before broad desktop automation
+- add Windows-first adapters for VS Code, Terminal, and File Explorer
+
+### 6. Memory Engine
+
+Responsibilities:
+
+- persist project context
+- store recent findings
+- keep reusable workflows
+- retain user and team preferences
+
+Storage model:
+
+- workspace metadata
+- task history
+- vector or semantic memory later
+- durable file-backed logs immediately
+
+### 6.5 Portal Graph
+
+Responsibilities:
+
+- map pages, links, and recurring flows
+- keep a remembered topology of the app
+- support self-healing navigation when selectors change
+- provide reusable structure for future agents
+
+### 7. Session Recorder
+
+Responsibilities:
+
+- append action timeline
+- attach screenshots and console logs
+- reconstruct replay views
+- generate bug reports
+
+Initial storage approach:
+
+- append-only event log
+- screenshot references
+- per-session manifest JSON
+
+### 8. Security Manager
+
+Responsibilities:
+
+- tool permissions
+- file and network boundaries
+- secrets handling
+- action approvals
+- auditability
+
+This is a product boundary, not a bolt-on feature.
+
+## Monorepo Strategy
+
+`packages/core`
+
+- shared contracts
+- runtime state shapes
+- orchestration primitives
+
+`packages/sdk`
+
+- external developer API
+- stable ergonomic wrapper over runtime actions
+
+`packages/mcp-server`
+
+- tool-facing surface for agent frameworks
+- command validation
+- capability exposure
+
+`apps/desktop`
+
+- first-party desktop operator experience
+- live state inspection
+- future visual workspace UI
+
+## Suggested First Vertical Slice
+
+Build the smallest end-to-end system that proves the concept:
+
+1. Launch desktop shell
+2. Open browser page
+3. Capture screenshot and DOM
+4. Detect clickable elements
+5. Execute one click or type action
+6. Record the session timeline
+7. Generate a simple report
+
+If this loop feels great, the rest of the platform has a solid foundation.
+
+## Phase 3 Direction
+
+The next platform shift is from action execution to understanding:
+
+1. Capture multimodal evidence
+2. Classify the interface and workflow
+3. Generate a goal plan from intent
+4. Write understanding into memory
+5. Reuse graph and memory across future sessions

package/docs/roadmap.md

@@ -0,0 +1,125 @@
+# Agent Portal Roadmap
+
+## Phase 1: Foundation
+
+Goal:
+Ship a browser-first local runtime that can observe, act, and report.
+
+Deliverables:
+
+- desktop shell scaffold
+- agent runtime contracts
+- browser adapter abstraction
+- Playwright integration
+- session event log
+- screenshot capture
+- simple report generation
+
+Success criteria:
+
+- an agent can open a local app, inspect it, perform a small workflow, and produce a replayable action history
+
+## Phase 2: Visual Intelligence
+
+Goal:
+Make the agent reliably understand what it sees.
+
+Deliverables:
+
+- DOM and accessibility tree fusion
+- OCR pipeline
+- UI element classification
+- visual diffing
+- confidence scoring for actions
+
+Success criteria:
+
+- the agent can explain what changed on screen and choose the right target more consistently than raw selectors alone
+
+## Phase 3: Agent Intelligence Layer
+
+Codename:
+Project Aperture
+
+Goal:
+Move from browser control to interface understanding, goal planning, and durable project memory.
+
+Deliverables:
+
+- `VisionCore` for unified page understanding
+- `GoalPlanner` for turning intent into execution steps
+- `PortalGraph` for remembered application structure
+- memory records for previous findings and plans
+- project awareness detection for frameworks and services
+- root-cause hypothesis generation from console and network evidence
+
+Success criteria:
+
+- the agent can explain what page it is on, what the user is trying to do, what likely failed, and what to do next
+
+## Phase 4: Multi-Agent Collaboration
+
+Goal:
+Allow specialized agents to work together inside one project workspace.
+
+Deliverables:
+
+- shared task board
+- agent-to-agent event bus
+- role-specific permission sets
+- shared memory context
+- result handoff model
+
+Success criteria:
+
+- a QA agent can find an issue, a frontend agent can fix it, and a reporter agent can summarize the result in one session
+
+## Phase 5: Desktop Control
+
+Goal:
+Expand beyond browser-only automation.
+
+Deliverables:
+
+- window discovery
+- native input dispatch
+- app adapters for VS Code, Terminal, and File Explorer
+- screenshot plus accessibility inspection for desktop apps
+
+Success criteria:
+
+- agents can complete a mixed workflow spanning editor, terminal, browser, and local files
+
+## Phase 6: Platform Surface
+
+Goal:
+Turn Agent Portal into an ecosystem others can build on.
+
+Deliverables:
+
+- public SDKs
+- MCP server
+- CLI tools
+- plugin marketplace
+- API and WebSocket surfaces
+
+Success criteria:
+
+- third-party developers can create agents, plugins, and workflows without modifying the core app
+
+## Phase 7: Enterprise and Reality Mode
+
+Goal:
+Differentiate with trust, resilience, and adversarial testing.
+
+Deliverables:
+
+- shared team workspaces
+- permissions and audit logs
+- SSO and deployment modes
+- reality mode stress behaviors
+- policy-driven autonomous QA
+
+Success criteria:
+
+- teams can safely run persistent agents against real products with clear oversight and reproducible evidence

package/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "agent-portal-2",
+  "version": "0.1.0",
+  "private": false,
+  "description": "Desktop-native operating environment for AI agents.",
+  "workspaces": [
+    "apps/*",
+    "packages/*"
+  ],
+  "scripts": {
+    "build": "npm run build --workspace @agent-portal/core && npm run build --workspace @agent-portal/sdk && npm run build --workspace @agent-portal/mcp-server && npm run build --workspace @agent-portal/desktop && npm run build --workspace apps/vscode-extension",
+    "dev": "npm run dev --workspace @agent-portal/desktop",
+    "check": "npm run build --workspace @agent-portal/core && npm run typecheck --workspace @agent-portal/core && npm run typecheck --workspace @agent-portal/sdk && npm run typecheck --workspace @agent-portal/mcp-server && npm run typecheck --workspace @agent-portal/desktop && npm run typecheck --workspace apps/vscode-extension",
+    "test": "python -m unittest discover -s python/tests -v && python -m unittest discover -s packages/agent-portal-mcp/tests -v && npm run build && node --test tests/*.test.mjs",
+    "release:python": "python scripts/release_python.py",
+    "release:desktop": "node scripts/package_desktop.mjs"
+  },
+  "devDependencies": {
+    "typescript": "^5.9.2"
+  }
+}

package/packages/agent-portal-mcp/README.md

@@ -0,0 +1,12 @@
+# Agent Portal MCP Server
+
+This package exposes Agent Portal through a local MCP-compatible stdio server.
+
+It connects to the local Agent Portal runtime and turns browser, inspection,
+steering, and report operations into structured MCP tools.
+
+Start it with:
+
+```bash
+agent-portal-mcp start
+```

package/packages/agent-portal-mcp/agent_portal_mcp/__init__.py

@@ -0,0 +1,3 @@
+from .server import AgentPortalMcpServer
+
+__all__ = ["AgentPortalMcpServer"]

package/packages/agent-portal-mcp/agent_portal_mcp/bridge/__init__.py

@@ -0,0 +1 @@
+__all__ = []

package/packages/agent-portal-mcp/agent_portal_mcp/bridge/runtime_client.py

@@ -0,0 +1,180 @@
+from __future__ import annotations
+
+import json
+import os
+import time
+from pathlib import Path
+from typing import Any
+from urllib.error import HTTPError, URLError
+from urllib.request import Request, urlopen
+
+
+class RuntimeClientError(RuntimeError):
+    pass
+
+
+class AgentPortalRuntimeClient:
+    def __init__(
+        self,
+        runtime_url: str = "http://127.0.0.1:8765",
+        token: str | None = None,
+        retries: int = 2,
+        timeout_seconds: float = 4.0,
+    ) -> None:
+        self.runtime_url = runtime_url.rstrip("/")
+        self.token = token or self._load_token()
+        self.retries = retries
+        self.timeout_seconds = timeout_seconds
+
+    def health(self) -> dict[str, Any]:
+        return self.get("/health")
+
+    def status(self) -> dict[str, Any]:
+        return self.get("/status")
+
+    def propose_action(
+        self,
+        action_type: str,
+        reason: str,
+        target: str | None = None,
+        payload: str | None = None,
+        risk_level: str = "low",
+    ) -> dict[str, Any]:
+        return self.post(
+            "/control/propose-action",
+            {
+                "actionType": action_type,
+                "reason": reason,
+                "target": target,
+                "payload": payload,
+                "riskLevel": risk_level,
+            },
+        )
+
+    def approve_action(self, action_id: str, execute: bool = False) -> dict[str, Any]:
+        return self.post("/control/approve-action", {"actionId": action_id, "execute": execute})
+
+    def reject_action(self, action_id: str, reason: str) -> dict[str, Any]:
+        return self.post("/control/reject-action", {"actionId": action_id, "reason": reason})
+
+    def action_queue(self) -> dict[str, Any]:
+        return self.post("/control/action-queue", {})
+
+    def set_goal(self, goal: str) -> dict[str, Any]:
+        return self.post("/control/goal", {"goal": goal})
+
+    def current_goal(self) -> dict[str, Any]:
+        return self.post("/control/goal/current", {})
+
+    def pause(self) -> dict[str, Any]:
+        return self.post("/control/pause", {})
+
+    def resume(self) -> dict[str, Any]:
+        return self.post("/control/resume", {})
+
+    def stop(self) -> dict[str, Any]:
+        return self.post("/control/stop", {})
+
+    def set_action_mode(self, mode: str) -> dict[str, Any]:
+        return self.post("/control/action-mode", {"mode": mode})
+
+    def browser_start(self) -> dict[str, Any]:
+        return self.post("/browser/start", {})
+
+    def browser_status(self) -> dict[str, Any]:
+        return self.post("/browser/status", {})
+
+    def browser_close(self) -> dict[str, Any]:
+        return self.post("/browser/close", {})
+
+    def browser_refresh(self) -> dict[str, Any]:
+        return self.post("/browser/refresh", {})
+
+    def browser_back(self) -> dict[str, Any]:
+        return self.post("/browser/back", {})
+
+    def browser_forward(self) -> dict[str, Any]:
+        return self.post("/browser/forward", {})
+
+    def capture(self, label: str = "capture") -> dict[str, Any]:
+        return self.post("/browser/capture", {"label": label})
+
+    def read_text(self, selector: str, reason: str) -> dict[str, Any]:
+        return self.post("/browser/read-text", {"selector": selector, "reason": reason})
+
+    def read_dom(self) -> dict[str, Any]:
+        return self.post("/browser/read-dom", {})
+
+    def read_accessibility_tree(self) -> dict[str, Any]:
+        return self.post("/browser/read-accessibility-tree", {})
+
+    def read_console_errors(self) -> dict[str, Any]:
+        return self.post("/browser/read-console-errors", {})
+
+    def read_network_failures(self) -> dict[str, Any]:
+        return self.post("/browser/read-network-failures", {})
+
+    def inspect(self) -> dict[str, Any]:
+        return self.post("/browser/inspect", {})
+
+    def inspect_element(self, selector: str) -> dict[str, Any]:
+        return self.post("/browser/inspect-element", {"selector": selector})
+
+    def generate_report(self) -> dict[str, Any]:
+        return self.post("/report/generate", {})
+
+    def list_reports(self) -> dict[str, Any]:
+        return self.post("/report/list", {})
+
+    def read_report(self, report_name: str) -> dict[str, Any]:
+        return self.post("/report/read", {"report": report_name})
+
+    def export_report(self, report_name: str, destination: str | None = None) -> dict[str, Any]:
+        return self.post("/report/export", {"report": report_name, "destination": destination})
+
+    def get(self, route: str) -> dict[str, Any]:
+        return self._request(route, method="GET")
+
+    def post(self, route: str, payload: dict[str, Any]) -> dict[str, Any]:
+        return self._request(route, method="POST", payload=payload)
+
+    def _request(
+        self,
+        route: str,
+        method: str,
+        payload: dict[str, Any] | None = None,
+    ) -> dict[str, Any]:
+        url = f"{self.runtime_url}{route}"
+        data = json.dumps(payload or {}).encode("utf8") if method == "POST" else None
+        headers = {"Content-Type": "application/json"}
+        if self.token:
+            headers["Authorization"] = f"Bearer {self.token}"
+
+        last_error: Exception | None = None
+        for attempt in range(self.retries + 1):
+            try:
+                request = Request(url, data=data, headers=headers, method=method)
+                with urlopen(request, timeout=self.timeout_seconds) as response:
+                    return json.loads(response.read().decode("utf8"))
+            except (HTTPError, URLError) as exc:
+                last_error = exc
+                if attempt < self.retries:
+                    time.sleep(0.2 * (attempt + 1))
+                    continue
+        raise RuntimeClientError(
+            "Agent Portal runtime is not running. Start it with: agent-portal start"
+        ) from last_error
+
+    def _load_token(self) -> str | None:
+        env_token = os.getenv("AGENT_PORTAL_TOKEN")
+        if env_token:
+            return env_token
+        config_path = Path.cwd() / "agent-portal.config.json"
+        if not config_path.exists():
+            return None
+        try:
+            raw = json.loads(config_path.read_text(encoding="utf8"))
+        except json.JSONDecodeError:
+            return None
+        token = raw.get("api_token")
+        return token if isinstance(token, str) and token else None

package/packages/agent-portal-mcp/agent_portal_mcp/cli.py

@@ -0,0 +1,32 @@
+from __future__ import annotations
+
+import argparse
+import json
+from dataclasses import asdict
+
+from agent_portal_mcp.doctor import run_doctor
+from agent_portal_mcp.server import AgentPortalMcpServer
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(prog="agent-portal-mcp")
+    parser.add_argument("--runtime-url", default="http://127.0.0.1:8765")
+    parser.add_argument("--json", action="store_true")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+    subparsers.add_parser("start")
+    subparsers.add_parser("doctor")
+    args = parser.parse_args()
+
+    if args.command == "start":
+        AgentPortalMcpServer(runtime_url=args.runtime_url).serve_stdio()
+        return
+
+    result = run_doctor(runtime_url=args.runtime_url)
+    payload = {"checks": [asdict(check) for check in result.checks]}
+    if args.json:
+        print(json.dumps(payload, indent=2))
+        return
+    for check in result.checks:
+        print(f"{check.status.upper():7} {check.name}: {check.details}")
+        if check.suggested_fix:
+            print(f"        fix: {check.suggested_fix}")