@widgex/wgx 0.0.0-stage → 0.4.2

package/python/XACTOR.md

@@ -0,0 +1,146 @@
+## Requires Python 3.10+
+
+This tree uses PEP 604 unions (`dict | None`) in signatures Python evaluates at
+definition time, so 3.9 cannot even import `bridge`. The macOS system
+interpreter at `/usr/bin/python3` **is** 3.9 — the extension never spawns a bare
+`python3` for that reason. `resolvePythonBin` (`src/ide-widgets/widgexactor/pythonBin.ts`)
+picks the interpreter and enforces the floor; `widgex.pythonPath` overrides it.
+
+# What the XACTOR is
+
+Three layers that let an agent drive a website it has never seen, without
+looking at pictures of it.
+
+The problem being solved is not "how do I click a button". It is that **the
+agent knows what it wants and cannot find it**, because the UI is graphical and
+the agent is not. Everything below follows from that.
+
+---
+
+# Invariant intent
+
+These are binding. Code that violates one is wrong even if it works.
+
+## Layer 0 — actuation
+
+Receives serial actions whose intent the agent has **already formed**. Resolves
+each descriptor at dispatch time, performs it with humanised input, and reports
+what happened.
+
+- It **searches** — it will rank candidates against a description.
+- It **never chooses** between candidates that are genuinely close. Ambiguity is
+  returned with its scores, not resolved by guessing. The decision margin is the
+  caller's parameter.
+- **An agent must never need to understand Layer 0 to drive a website.** No
+  concept from this layer — coordinates, bezier paths, network classes, epochs —
+  should ever be something an agent has to reason about.
+- Every action settles. There is no code path that accepts work and then
+  discards it, and no failure that is logged instead of returned.
+
+## Layer 1 — elementary interactions
+
+The composable set the agent uses to turn a general intuition about identifiable
+features into a concrete reference. `python/xactor/api.py`.
+
+- Its job is **composition**: coarse look, then narrow by scope, then narrow by
+  predicate, then act.
+- **The information Layer 1 shows the agent must be consistent with how Layer 0
+  resolves.** Same vocabulary, same matching rules, same names. An agent reading
+  one model and acting through another is the bug this rule exists to prevent.
+- Locating and acting are separate verbs, because which candidate was meant is
+  the agent's call.
+
+## Layer 2 — widgets
+
+Caches Layer 1 compositions into declared transitions, so a modelled site can be
+driven to an interactive objective in a single pass with the least cognitive
+effort — explicitly written down, and built to be reached rather than explored.
+
+## Cross-cutting — keep the agent out of the loop
+
+Every design choice that adds an agent round trip is a cost that has to be
+justified. A screenshot to look at, a checkpoint mid-chain, an observe-after-act
+— each one turns a script back into a conversation. Load the whole chain, let it
+run, read one report.
+
+---
+
+# How it actually works
+
+## Referring to an element
+
+A **uid** (`e7f3a1`) or a **path**:
+
+```
+main/table/row"prod-key"/button"Delete"
+dialog"Create Token"/button"Create"
+form#login/textbox"Username"
+```
+
+Paths run through an *anchor-projected* tree — landmarks, collections and their
+members, interactive leaves, headings. Wrapper `div`s are transparent, so a
+framework inserting one changes no path.
+
+Nothing is written into the page. A reference is a **description**, re-derived
+from the live DOM by the same function that produced it, so a stale one matches
+nothing or matches several — it can never quietly resolve to the wrong element.
+
+*A uid is a memory. A path is an assertion.*
+
+## Finding something
+
+```python
+# see the candidates, with their scores
+cands = await api.search(role="button", like="create")
+
+# exactly one, or an error carrying what tied
+btn = await api.one(role="button", textExact="Create", under='dialog"Create Token"')
+
+# a search deferred into an action -- the usual form
+click(api.find(role="button", like="create token"))
+```
+
+The UI is graphical, so structure is not always enough. Anchor on what is
+visible instead:
+
+```python
+api.find(role="textbox", right_of="Name")
+api.find(role="button", in_row='row"prod-key"')
+api.find(role="button", below='heading"Danger zone"')
+```
+
+A screenshot is the last resort, not the first. It costs a round trip through
+the agent for every lookup.
+
+## Driving a chain
+
+```python
+click(api.find(role="button", like="create token"))
+type_text(api.find(role="textbox", right_of="Name"), "ci-deploy")
+click(api.one(role="button", textExact="Create", under='dialog"Create Token"'))
+report = await api.drain()
+```
+
+One report for the whole chain: what each step resolved to, whether it worked,
+and what changed. **A failure stops the chain** — the remainder is reported as
+skipped rather than attempted, because once a step fails the page is no longer
+in the state the rest was written against.
+
+## Readiness
+
+Not a concept the agent handles. An action waits for its own target; the engine
+knows what is still loading from events it already receives, and only blocks on
+things that could move what is about to be clicked. A stylesheet blocks. An
+image does not.
+
+---
+
+# What was removed, and why
+
+| Gone | Because |
+|---|---|
+| `xh_N` ids | Page-resident counters. A re-render destroyed the node and the id silently dangled — or worse, matched whatever was minted into that slot next. |
+| `expects_transition` | Its detection required the URL to change, which a modal or an in-place SPA update never does. It burned a hard 5s floor and then reported success anyway. Readiness is intrinsic now. |
+| `hybrid_snapshot` / `query` | A flat list. Eight identical "Delete" buttons were indistinguishable, and every node carried five raw CSS strings nobody read. |
+| The 5-strategy resolver cascade | Blind guessing, one CDP round trip per rung, implemented four times in four places that disagreed. The guessing is now an explicit ranked search the agent can see. |
+| Adaptive timeouts | They recorded their own failures as samples, so a page that timed out at 5s taught the engine to wait 7.5s, then 11.25s, without bound. |

package/python/browser_actor_host.py

@@ -0,0 +1,456 @@
+#!/usr/bin/env python3
+"""
+browser_actor_host.py — Persistent Python daemon for browser automation.
+
+Reads JSON-RPC requests from stdin, executes commands against a browser page
+via raw CDP (WebSocket), writes JSON responses to stdout.
+
+Protocol:
+  Request:  {"id": 1, "method": "execute", "params": {"script": "...", "timeout": 30}}
+  Response: {"id": 1, "result": {...}} or {"id": 1, "error": {"message": "..."}}
+  Event:    {"event": "ready"} | {"event": "navigation", "url": "...", "title": "..."}
+
+Methods:
+  create_session(ws_url)                — Connect to a CDP page target via WebSocket
+  stop_session()                        — Close CDP connection
+  execute(script, timeout?)             — Run Python code with `page`, `result` in scope
+  get_state()                           — Return {url, title}
+  screenshot()                          — Return {base64} PNG screenshot
+  navigate(url)                         — Navigate to a URL
+  start_recording()                     — Enable action recording, clear buffer
+  stop_recording()                      — Disable recording, return buffer
+  flush_recording()                     — Return + clear buffer without stopping
+  shutdown()                            — Close connection and exit
+"""
+
+import ast
+import asyncio
+import base64
+import json
+import sys
+import time
+import traceback
+from typing import Any
+
+from cdp_page import CdpPage
+
+
+# ── Injected recorder JS ──────────────────────────────────────────────────────
+# Captures click, input, and submit events in the page and emits them as
+# structured JSON via console.log with a __WX_ACTION__ prefix.
+
+RECORDER_JS = """
+(function() {
+  if (window.__wx_recorder_installed) return;
+  window.__wx_recorder_installed = true;
+
+  function getAttributes(el) {
+    const attrs = {};
+    for (const name of ['data-testid', 'aria-label', 'name', 'type', 'href', 'role', 'placeholder']) {
+      const val = el.getAttribute(name);
+      if (val != null) attrs[name] = val;
+    }
+    return attrs;
+  }
+
+  function getSelector(el) {
+    if (el.id) return '#' + el.id;
+    const testId = el.getAttribute('data-testid');
+    if (testId) return '[data-testid="' + testId + '"]';
+    const ariaLabel = el.getAttribute('aria-label');
+    if (ariaLabel) return '[aria-label="' + ariaLabel + '"]';
+    const name = el.getAttribute('name');
+    if (name) return el.tagName.toLowerCase() + '[name="' + name + '"]';
+    return null;
+  }
+
+  // Click handler
+  document.addEventListener('click', function(e) {
+    const el = e.target;
+    const tag = el.tagName ? el.tagName.toLowerCase() : '';
+    const text = (el.innerText || '').trim().substring(0, 80);
+    const action = {
+      type: 'click',
+      timestamp: Date.now(),
+      x: e.clientX,
+      y: e.clientY,
+      tag: tag,
+      id: el.id || null,
+      classes: el.className ? String(el.className).substring(0, 200) : null,
+      text: text || null,
+      selector: getSelector(el),
+      attributes: getAttributes(el)
+    };
+    console.log('__WX_ACTION__' + JSON.stringify(action));
+  }, true);
+
+  // Input handler (debounced — fires on change, not per keystroke)
+  document.addEventListener('change', function(e) {
+    const el = e.target;
+    const tag = el.tagName ? el.tagName.toLowerCase() : '';
+    // Skip password fields
+    if (el.type === 'password') return;
+    const action = {
+      type: 'input',
+      timestamp: Date.now(),
+      tag: tag,
+      id: el.id || null,
+      name: el.getAttribute('name') || null,
+      value: (el.value || '').substring(0, 500),
+      inputType: el.type || null,
+      selector: getSelector(el),
+      attributes: getAttributes(el)
+    };
+    console.log('__WX_ACTION__' + JSON.stringify(action));
+  }, true);
+
+  // Submit handler
+  document.addEventListener('submit', function(e) {
+    const form = e.target;
+    const action = {
+      type: 'submit',
+      timestamp: Date.now(),
+      formId: form.id || null,
+      formAction: form.action || null,
+      selector: form.id ? '#' + form.id : null
+    };
+    console.log('__WX_ACTION__' + JSON.stringify(action));
+  }, true);
+})();
+"""
+
+
+def emit_event(event_type: str, **kwargs):
+    msg = {"event": event_type, **kwargs}
+    sys.stdout.write(json.dumps(msg) + "\n")
+    sys.stdout.flush()
+
+
+def emit_response(req_id: Any, result: Any = None, error: str | None = None):
+    if error:
+        msg = {"id": req_id, "error": {"message": error}}
+    else:
+        msg = {"id": req_id, "result": result}
+    sys.stdout.write(json.dumps(msg) + "\n")
+    sys.stdout.flush()
+
+
+async def main():
+    page: CdpPage | None = None
+
+    # ── Action recording state ─────────────────────────────────────
+    recorded_actions: list[dict] = []
+    recording_enabled: bool = True
+
+    def on_console_message(params: dict):
+        """CDP Runtime.consoleAPICalled handler — intercepts __WX_ACTION__ logs."""
+        if not recording_enabled:
+            return
+        try:
+            args = params.get("args", [])
+            if not args:
+                return
+            text = args[0].get("value", "")
+            if isinstance(text, str) and text.startswith("__WX_ACTION__"):
+                payload = json.loads(text[len("__WX_ACTION__"):])
+                recorded_actions.append(payload)
+        except (json.JSONDecodeError, KeyError, IndexError):
+            pass
+
+    def on_network_request(params: dict):
+        """CDP Network.requestWillBeSent handler — captures document navigations."""
+        if not recording_enabled:
+            return
+        try:
+            req_type = params.get("type", "")
+            if req_type == "Document":
+                url = params.get("request", {}).get("url", "")
+                recorded_actions.append({
+                    "type": "navigation",
+                    "url": url,
+                    "timestamp": int(time.time() * 1000),
+                })
+        except (KeyError, AttributeError):
+            pass
+
+    async def setup_recording(p: CdpPage):
+        """Set up CDP listeners for action recording."""
+        try:
+            p.on("Runtime.consoleAPICalled", on_console_message)
+            p.on("Network.requestWillBeSent", on_network_request)
+
+            # Inject recorder into all new documents
+            await p.send("Page.addScriptToEvaluateOnNewDocument", {
+                "source": RECORDER_JS
+            })
+
+            # Inject into current page
+            await p.evaluate(RECORDER_JS)
+        except Exception as e:
+            emit_event("warning", message=f"Recording setup failed: {e}")
+
+    async def handle_create_session(req_id: Any, params: dict):
+        """Connect to a CDP page target via its WebSocket URL."""
+        nonlocal page
+
+        ws_url = params.get("ws_url", "")
+        if not ws_url:
+            emit_response(req_id, error="ws_url is required")
+            return
+
+        try:
+            emit_event("log", message=f"Connecting to CDP target: {ws_url}")
+            p = CdpPage(ws_url)
+            await p.connect()
+            page = p
+
+            emit_event("log", message=f"Connected. URL: {p.url}")
+
+            # Set up recording
+            await setup_recording(p)
+
+            emit_response(req_id, result={
+                "session_id": "cdp",
+                "status": "connected",
+                "mode": "cdp",
+            })
+        except Exception:
+            emit_response(req_id, error=traceback.format_exc())
+
+    async def handle_stop_session(req_id: Any):
+        """Close CDP connection."""
+        nonlocal page
+
+        try:
+            if page:
+                await page.close()
+                page = None
+            emit_response(req_id, result={"status": "stopped"})
+        except Exception:
+            emit_response(req_id, error=traceback.format_exc())
+
+    async def get_page():
+        """Get the current CdpPage."""
+        if page is None:
+            emit_event("error", message="No browser session. Call create_session first.")
+            raise RuntimeError("No browser session")
+        return page
+
+    async def handle_execute(req_id: Any, params: dict):
+        nonlocal recorded_actions
+        script = params.get("script", "")
+        timeout = params.get("timeout", 60)
+
+        try:
+            p = await get_page()
+        except RuntimeError as e:
+            emit_response(req_id, error=str(e))
+            return
+
+        # Mark script start in recording buffer
+        script_start_idx = len(recorded_actions)
+        if recording_enabled:
+            recorded_actions.append({"type": "script_start", "timestamp": int(time.time() * 1000)})
+
+        result_dict: dict[str, Any] = {}
+        scope = {
+            "page": p,
+            "result": result_dict,
+            "asyncio": asyncio,
+        }
+
+        try:
+            # Wrapped on the AST rather than by indenting the source, for the
+            # reason `xactor/bridge.py`'s `compile_exec` gives at length: text
+            # indentation reaches inside triple-quoted strings and silently
+            # adds four spaces to every line of the caller's own data.
+            module = ast.parse("async def __actor_main__():\n    return result")
+            module.body[0].body[:0] = ast.parse(script).body
+            ast.fix_missing_locations(module)
+            exec(compile(module, "<actor-script>", "exec"), scope)
+            await asyncio.wait_for(scope["__actor_main__"](), timeout=timeout)
+        except asyncio.TimeoutError:
+            emit_response(req_id, error=f"Script timed out after {timeout}s")
+            return
+        except Exception:
+            emit_response(req_id, error=traceback.format_exc())
+            return
+
+        # Mark script end
+        if recording_enabled:
+            recorded_actions.append({"type": "script_end", "timestamp": int(time.time() * 1000)})
+
+        script_actions = recorded_actions[script_start_idx:] if recording_enabled else []
+
+        try:
+            current_url = p.url
+            current_title = await asyncio.wait_for(p.title(), timeout=5)
+        except Exception:
+            current_url = p.url
+            current_title = ""
+
+        response = {
+            "data": result_dict,
+            "url": current_url,
+            "title": current_title,
+        }
+        if recording_enabled and script_actions:
+            response["recorded_actions"] = script_actions
+
+        emit_response(req_id, result=response)
+
+    async def handle_get_state(req_id: Any):
+        try:
+            p = await get_page()
+        except RuntimeError as e:
+            emit_response(req_id, error=str(e))
+            return
+        try:
+            url = p.url
+            title = await asyncio.wait_for(p.title(), timeout=5)
+        except Exception:
+            url = p.url if p else ""
+            title = ""
+        emit_response(req_id, result={"url": url, "title": title})
+
+    async def handle_screenshot(req_id: Any):
+        try:
+            p = await get_page()
+        except RuntimeError as e:
+            emit_response(req_id, error=str(e))
+            return
+        try:
+            png_bytes = await p.screenshot()
+            b64 = base64.b64encode(png_bytes).decode("ascii")
+            emit_response(req_id, result={"base64": b64})
+        except Exception:
+            emit_response(req_id, error=traceback.format_exc())
+
+    async def handle_navigate(req_id: Any, params: dict):
+        url = params.get("url", "")
+        try:
+            p = await get_page()
+        except RuntimeError as e:
+            emit_response(req_id, error=str(e))
+            return
+        try:
+            await p.goto(url)
+            current_url = p.url
+            current_title = await p.title()
+            emit_event("navigation", url=current_url, title=current_title)
+            emit_response(req_id, result={"url": current_url, "title": current_title})
+        except Exception:
+            emit_response(req_id, error=traceback.format_exc())
+
+    async def handle_start_recording(req_id: Any):
+        nonlocal recording_enabled, recorded_actions
+        recording_enabled = True
+        recorded_actions = []
+        emit_response(req_id, result={"status": "recording"})
+
+    async def handle_stop_recording(req_id: Any):
+        nonlocal recording_enabled
+        recording_enabled = False
+        actions = list(recorded_actions)
+        emit_response(req_id, result={"recorded_actions": actions, "count": len(actions)})
+
+    async def handle_flush_recording(req_id: Any):
+        nonlocal recorded_actions
+        actions = list(recorded_actions)
+        recorded_actions = []
+        emit_response(req_id, result={"recorded_actions": actions, "count": len(actions)})
+
+    async def handle_shutdown(req_id: Any):
+        nonlocal page
+
+        if page:
+            try:
+                await page.close()
+            except Exception:
+                pass
+            page = None
+
+        emit_response(req_id, result={"status": "shutdown"})
+
+    # Main stdin read loop
+    loop = asyncio.get_event_loop()
+
+    if sys.platform == "win32":
+        import concurrent.futures
+        _stdin_executor = concurrent.futures.ThreadPoolExecutor(max_workers=1, thread_name_prefix="stdin")
+
+        def _read_stdin_line() -> bytes:
+            """Blocking read of one line from stdin (runs in thread)."""
+            return sys.stdin.buffer.readline()
+
+        async def _readline() -> bytes:
+            return await loop.run_in_executor(_stdin_executor, _read_stdin_line)
+    else:
+        reader = asyncio.StreamReader()
+        protocol = asyncio.StreamReaderProtocol(reader)
+        await loop.connect_read_pipe(lambda: protocol, sys.stdin)
+
+        async def _readline() -> bytes:
+            return await reader.readline()
+
+    # Signal ready
+    sys.stdout.write(json.dumps({"event": "ready"}) + "\n")
+    sys.stdout.flush()
+
+    while True:
+        line = await _readline()
+        if not line:
+            break  # stdin closed
+
+        line_str = line.decode("utf-8").strip()
+        if not line_str:
+            continue
+
+        try:
+            req = json.loads(line_str)
+        except json.JSONDecodeError as e:
+            sys.stdout.write(json.dumps({"id": None, "error": {"message": f"Invalid JSON: {e}"}}) + "\n")
+            sys.stdout.flush()
+            continue
+
+        req_id = req.get("id")
+        method = req.get("method", "")
+        params = req.get("params", {})
+
+        try:
+            if method == "create_session":
+                await handle_create_session(req_id, params)
+            elif method == "stop_session":
+                await handle_stop_session(req_id)
+            elif method == "execute":
+                await handle_execute(req_id, params)
+            elif method == "get_state":
+                await handle_get_state(req_id)
+            elif method == "screenshot":
+                await handle_screenshot(req_id)
+            elif method == "navigate":
+                await handle_navigate(req_id, params)
+            elif method == "start_recording":
+                await handle_start_recording(req_id)
+            elif method == "stop_recording":
+                await handle_stop_recording(req_id)
+            elif method == "flush_recording":
+                await handle_flush_recording(req_id)
+            elif method == "shutdown":
+                await handle_shutdown(req_id)
+                break
+            else:
+                emit_response(req_id, error=f"Unknown method: {method}")
+        except Exception:
+            emit_response(req_id, error=traceback.format_exc())
+
+    # Cleanup
+    if page:
+        try:
+            await page.close()
+        except Exception:
+            pass
+
+
+if __name__ == "__main__":
+    asyncio.run(main())