@rebind.gg/mcp-server 0.1.5 → 0.2.0

package/README.md

@@ -1,19 +1,15 @@
 # @rebind.gg/mcp-server
 
-Stdio MCP server that exposes Rebind as computer-use tools: `screenshot`,
-`screenshot_window`, `zoom`, `click` (verified), `move_mouse`, `type`, `key`,
-`scroll`, `run_lua`, `wait`, window control, and `calibrate`. Wraps
-`@rebind.gg/client-ts`; requires the **Rebind** app
-(https://rebind.gg/download) running its Remote Access relay — `remote.luau`
-(recommended) or `remote_access.lua` (protocol >= 1.1.0). The `click` tool clicks via `hid.press`, which is
-coroutine-only; use `remote.luau` or a current `remote_access.lua` (older copies
-called it outside a coroutine, so clicks silently failed).
+Stdio MCP server that turns [Rebind](https://rebind.gg) into a computer-use toolkit: screenshots, verified clicks, keyboard input, window control, and direct Luau scripting on the connected machine. Input uses the relay's active transport: genuine USB HID in hardware mode or OS injection in software mode.
+
+**Requirements**
+
+- [Bun](https://bun.sh) >= 1.1 (the server runs via `bunx`)
+- The [Rebind app](https://rebind.gg/download) running the free [Remote Control](https://www.rebind.gg/marketplace/script_AAHteeDcYPGOT7GfXZ_OK) package (`rebind install @rebind/remote-control`)
 
 ## Install
 
-One command writes the correct config for whichever clients you name — Claude
-Code, Cursor, Codex, OpenCode, VS Code, Windsurf, Zed, and more — each in its
-native format, via [`add-mcp`](https://github.com/neondatabase/add-mcp):
+One command writes the correct config for whichever clients you name — Claude Code, Cursor, Codex, OpenCode, VS Code, Windsurf, Zed, and more — via [`add-mcp`](https://github.com/neondatabase/add-mcp):
 
 ```sh
 npx add-mcp "bunx @rebind.gg/mcp-server" --name rebind \
@@ -21,13 +17,9 @@ npx add-mcp "bunx @rebind.gg/mcp-server" --name rebind \
   -g -a claude-code -a cursor -a codex -a opencode
 ```
 
-Drop `-a <agent>` for the clients you want, or pass `--all`. `-g` installs
-globally; omit it for project-local config. Restart the client afterward.
-
-Requires **Bun** (the server runs via `bunx`) and the **Rebind** app
-(https://rebind.gg/download) running its Remote Access relay.
+Drop `-a <agent>` for the clients you want, or pass `--all`. `-g` installs globally; omit it for project-local config. Restart the client afterward.
 
-## Configure manually
+### Manual configuration
 
 Any MCP client takes the stdio block directly:
 
@@ -43,37 +35,74 @@ Any MCP client takes the stdio block directly:
 }
 ```
 
-`REBIND_TOKEN` is required when the server has an auth token set (the **Auth
-token** panel field on `remote.luau`, or `AUTH_TOKEN` on `remote_access.lua`).
-
-Check the relay is reachable before wiring it up:
+### Verify the relay is reachable
 
 ```sh
 bunx @rebind.gg/mcp-server --selftest
 ```
 
-Set `REBIND_AGENT_LOG=<path>` to append one JSONL line per tool call
-(`{ts, tool, ok, latency_ms, note}`) — no screenshot bytes, no user content.
-Off unless set.
-
-`click` and `move_mouse` reach an absolute screen point by reading the real OS
-cursor and sending relative HID deltas (which map 1:1 to pixels), correcting
-against the true cursor until it lands. This sidesteps the relay's own absolute
-move, whose delta is computed from a cached cursor position that goes stale and
-flings the pointer to a screen edge. A mouse scale probe runs on the first
-click/move_mouse; if pointer ballistics are detected (enhance pointer precision,
-DPI mismatch) the probe reports it, since acceleration would break the 1:1
-assumption.
-
-Tool calls are serialized through a single queue: there is one physical cursor
-and keyboard, so parallel tool calls from the model (e.g. two `move_mouse` in
-one turn) run one after another instead of interleaving HID input. All results
-are reported in the coordinate space of the last screenshot, the same space the
-model emits.
+Connects, takes a capture, and reports the display — the fastest way to catch the #1 first-run failure (relay not loaded).
+
+## Tools
+
+| Tool | What it does |
+| --- | --- |
+| `screenshot` | Capture a display (cursor marked with a crosshair). All coordinates the model emits are pixels in the most recent screenshot. |
+| `screenshot_window` | Capture one window by title substring (or the active window) — smaller, cheaper, monitor-proof. |
+| `zoom` | Re-capture a region of the last screenshot at native resolution, for small text. Read-only. |
+| `click` | Move to a point and click (`left`/`right`/`middle`, optional double). Landing position is verified and corrected before pressing. |
+| `move_mouse` | Move without clicking, with the same verification. |
+| `type` | Type text into the focused element. |
+| `key` | Press a key or combo, e.g. `Enter`, `LCtrl+L`, `LAlt+Tab`. |
+| `scroll` | Scroll at the cursor. Positive scrolls down, negative up. `unit` is `wheel` (default, fine wheel clicks) or `page` (one PageDown or PageUp per unit, capped at 10). |
+| `read_page` | Extract the focused browser's current page as structured text (URL, forms, interactive elements, plain text) via an address-bar bookmarklet; the full dump, including cleaned HTML, lands in the clipboard. The result is read back through AppleScript (`URL of active tab of front window`), so the relay host must be macOS and the front browser must answer that query. Safari is detected but blocks the bookmarklet path. |
+| `meditate` | Research a problem page until a reasoning model has the data to solve it: fetches the URL directly, pulls the script bundles behind a JS shell so its API endpoints are visible, fetches those endpoints and assets, and reads the live page with vision. Returns a markdown dossier plus JSON `{error, problem, requirements, plan, submittable, form, live_captcha_hint, confidence, sources, assets, pending_fetch, notes, cost_usd}`. `error` is non-null when no problem could be identified or the page could not be read; partial data is still returned. Gated behind `REBIND_ALLOW_DELEGATION=1` (spends OpenRouter credits). See the pipeline below. |
+| `run_lua` | Run a Luau script on the connected machine and return its `return` value as JSON — the fast path for any deterministic sequence (launch apps, place windows, shell commands, pixel search). |
+| `lua_docs` | Full Luau SDK type definitions for `run_lua`. |
+| `list_displays` / `list_windows` / `focus_window` / `position_window` / `active_window` | Display and window enumeration, focus, and placement. |
+| `get_cursor` / `wait` / `calibrate` | Cursor readback, pauses, and the mouse scale probe. |
+| `delegate_task` / `rebind_versus` | Server-side reasoning helpers with no desktop access: one bounded text task, or parallel critics that attack a plan. Registered only with `REBIND_ALLOW_DELEGATION=1` (spends OpenRouter credits). |
+
+## Configuration
+
+| Env var | Default | Purpose |
+| --- | --- | --- |
+| `REBIND_URL` | `ws://127.0.0.1:19561` | Relay WebSocket URL. |
+| `REBIND_TOKEN` | — | Required when the relay has an auth token set (the **Auth token** field in Remote Control's Options). |
+| `REBIND_AGENT_LOG` | off | Path to append one JSONL line per tool call (`{ts, tool, ok, latency_ms, note}`). No screenshot bytes, no user content. |
+| `REBIND_MODEL` | `google/gemini-3.5-flash-lite` | OpenRouter slug for `delegate_task`/`rebind_versus`/`meditate` reasoning (only used with `REBIND_ALLOW_DELEGATION=1`). |
+| `REBIND_VISION_MODEL` | falls back to `REBIND_MODEL` | Multimodal slug for `meditate`'s live-page/captcha vision pass. Set this when `REBIND_MODEL` is a text-only model. |
+| `REBIND_ALLOW_DELEGATION` | off | Set to `1` to register the credit-spending tools: `delegate_task`, `rebind_versus`, `meditate`. |
+
+## How it works
+
+**Verified clicks.** `click` and `move_mouse` reach an absolute screen point by reading the real OS cursor and sending relative HID deltas (which map 1:1 to pixels), correcting against the true cursor until it lands. This sidesteps the relay's own absolute move, whose delta comes from a cached cursor position that goes stale and flings the pointer to a screen edge.
+
+**Calibration.** A mouse scale probe runs on the first `click`/`move_mouse`. If pointer ballistics are detected (enhance pointer precision, DPI mismatch), the probe reports it, since acceleration would break the 1:1 assumption. Re-run it any time with `calibrate`.
+
+**One queue.** There is one physical cursor and keyboard, so tool calls are serialized: parallel calls from the model run one after another instead of interleaving HID input.
+
+**One coordinate space.** All results are reported in the pixel space of the last screenshot — the same space the model emits — so no rescaling math ever falls on the model.
+
+**Machine-aware orientation.** At startup the server probes the connected machine through the relay (OS, shortcut modifier, available shell tools, installed apps) and serves it as the MCP `instructions` block, along with a policy: prefer `run_lua` for deterministic work, keyboard over mouse, and screenshots last. The full Luau SDK type definitions are embedded inline so the model never has to guess `run_lua` APIs (including the coroutine-safe HID path — `HID.Down`/`Up`/`MoveTo`/`Combo` work in a one-shot chunk, `HID.Press` does not). The OS is read from the relay, not `process.platform`, so the guidance is correct even when the server and the relay run on different machines.
+
+## Solve pipeline
+
+`meditate` gathers; it does not submit. The intended flow, all gated behind `REBIND_ALLOW_DELEGATION=1`:
+
+1. **`meditate`** → a dossier. Evidence is gathered by **fetching, not by driving the GUI**: a direct GET returns the server's own bytes, and when that comes back a JS shell the script bundles are pulled in so the endpoints the page calls are visible to the model, which then requests them by id. Same-origin URLs are fetched **from inside the page** — the browser's own session, cookies and TLS answer them, so an endpoint that refuses a bare curl responds normally; cross-origin falls back to relay-side curl. The rendered DOM is read when a fetch cannot see the content. It then leaves the browser **open on the problem page** and returns `form` (per-field `{value, source, op}`, traced to evidence) and `submittable`. It never bakes in the captcha — that can rotate, so it is read live at submit time, and its absence does not make a dossier unsubmittable.
+2. **`rebind_versus`** → feed it the dossier to attack the reconciled values before anything irreversible happens.
+3. **Solve (caller-driven, by sight)** on the still-open page: re-navigate to the anchor and confirm the URL; screenshot → click field → type each `form` value; tick attachment checkboxes; read the **live** captcha off the page; then **re-extract the filled form and diff it against the dossier** — on any mismatch or unresolved conflict, stop (do not submit). Only on a clean diff, click submit and verify. When `submittable` is false there is no form to fill; surface the answer instead.
+
+The verify-diff before an irreversible submit is a discipline the caller enforces — auto-submitting reconciled data fails closed, not open.
 
 ## Develop
 
 ```sh
-bun test packages/mcp-server/tests/
+bun run --cwd packages/mcp-server test        # src/ and tests/
 bun run --cwd packages/mcp-server typecheck
 ```
+
+## License
+
+MIT © [US Input Company](https://usinput.com)

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@rebind.gg/mcp-server",
-  "version": "0.1.5",
-  "description": "MCP server exposing Rebind as computer-use tools: screenshot, verified click, type, key, scroll, and window control over the Remote Access protocol.",
+  "version": "0.2.0",
+  "description": "MCP server exposing Rebind as computer-use tools: screenshot, verified click, type, key, scroll, and window control over the Remote Control protocol.",
   "type": "module",
   "bin": {
     "rebind-mcp-server": "src/index.ts"
@@ -30,7 +30,6 @@
     "mcp",
     "model-context-protocol",
     "computer-use",
-    "openclaw",
     "hid",
     "automation",
     "keyboard",
@@ -42,12 +41,16 @@
   "scripts": {
     "start": "bun run src/index.ts",
     "selftest": "bun run src/index.ts --selftest",
-    "test": "bun test tests/",
-    "typecheck": "tsc --noEmit"
+    "test": "bun test src/ tests/",
+    "typecheck": "tsc --noEmit",
+    "bench": "bun run bench/runner.ts",
+    "bench:selftest": "bun run bench/runner.ts --selftest",
+    "bench:setup": "bun run bench/runner.ts --setup-only",
+    "bench:demo": "bash bench/demo.sh"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.12.0",
-    "@rebind.gg/client-ts": "0.2.0",
+    "@rebind.gg/client-ts": "0.3.0",
     "sharp": "^0.34.2",
     "zod": "^3.25.0"
   },

package/src/bookmarklet.js

@@ -0,0 +1,182 @@
+(() => {
+  const clone = document.documentElement.cloneNode(true);
+  clone
+    .querySelectorAll('script, style, link[rel="stylesheet"], svg, noscript')
+    .forEach((el) => {
+      el.remove();
+    });
+  clone.querySelectorAll("*").forEach((el) => {
+    [...el.attributes].forEach((attr) => {
+      if (attr.name.startsWith("on") || attr.name === "style")
+        el.removeAttribute(attr.name);
+    });
+  });
+  clone.querySelectorAll("[href]").forEach((el) => {
+    try {
+      el.setAttribute(
+        "href",
+        new URL(el.getAttribute("href"), document.baseURI).href,
+      );
+    } catch (e) {}
+  });
+  clone.querySelectorAll("[src]").forEach((el) => {
+    try {
+      el.setAttribute(
+        "src",
+        new URL(el.getAttribute("src"), document.baseURI).href,
+      );
+    } catch (e) {}
+  });
+  clone.querySelectorAll("[action]").forEach((el) => {
+    try {
+      el.setAttribute(
+        "action",
+        new URL(el.getAttribute("action"), document.baseURI).href,
+      );
+    } catch (e) {}
+  });
+  const cleanedHTML = clone.outerHTML
+    .replace(/\s{2,}/g, " ")
+    .replace(/>\s+</g, ">\n<");
+  const forms = document.querySelectorAll("form");
+  let formOut = "";
+  const formElementsSet = new Set();
+  forms.forEach((form, fi) => {
+    const rect = form.getBoundingClientRect();
+    const scrollY = Math.round(rect.top + window.scrollY);
+    formOut += `[Form ${fi}] action="${form.action || ""}" method="${form.method}" scrollY=${scrollY}\n`;
+    form
+      .querySelectorAll("input, select, textarea, button")
+      .forEach((el, ei) => {
+        formElementsSet.add(el);
+        const r = el.getBoundingClientRect();
+        const sY = Math.round(r.top + window.scrollY);
+        const label = (
+          el.getAttribute("aria-label") ||
+          el.value ||
+          el.textContent ||
+          ""
+        )
+          .trim()
+          .replace(/\s+/g, " ")
+          .slice(0, 60);
+        formOut += `   [${fi}.${ei}] <${el.tagName.toLowerCase()}> type="${el.type || ""}" name="${el.name || ""}" scrollY=${sY} label="${label}"\n`;
+      });
+    formOut += "\n";
+  });
+  let otherOut = "";
+  let idx = 0;
+  document
+    .querySelectorAll("a, button, input, select, textarea, [role]")
+    .forEach((el) => {
+      if (formElementsSet.has(el) || el.closest("form")) return;
+      const rect = el.getBoundingClientRect();
+      const scrollY = Math.round(rect.top + window.scrollY);
+      const label = (
+        el.getAttribute("aria-label") ||
+        el.textContent ||
+        el.value ||
+        ""
+      )
+        .trim()
+        .replace(/\s+/g, " ")
+        .slice(0, 60);
+      const href = el.href ? el.href : "";
+      const role = el.getAttribute("role") || "";
+      const tag = el.tagName.toLowerCase();
+      const ac = el.getAttribute("aria-controls") || "";
+      const ae = el.getAttribute("aria-expanded");
+      const asel = el.getAttribute("aria-selected");
+      const submitish =
+        el.type === "submit" ||
+        el.type === "reset" ||
+        el.name === "intent" ||
+        /\b(submit|save|reset|cancel|delete)\b/i.test(label);
+      const reveal =
+        !href &&
+        !submitish &&
+        (role === "tab" ||
+          !!ac ||
+          ae !== null ||
+          asel !== null ||
+          tag === "button" ||
+          role === "button");
+      otherOut += `[${idx}] <${tag}> role="${role}" scrollY=${scrollY} name="${label}"${href ? ` href="${href}"` : ""}${ac ? ` aria-controls="${ac}"` : ""}${ae !== null ? ` aria-expanded="${ae}"` : ""}${asel !== null ? ` aria-selected="${asel}"` : ""}${reveal ? " reveal=1" : ""}\n`;
+      idx++;
+    });
+  const walker = document.createTreeWalker(
+    document.body,
+    NodeFilter.SHOW_TEXT,
+    null,
+    false,
+  );
+  const textParts = [];
+  let node = walker.nextNode();
+  while (node) {
+    const parent = node.parentElement;
+    if (!parent || !["SCRIPT", "STYLE", "NOSCRIPT"].includes(parent.tagName)) {
+      const t = node.nodeValue.replace(/\s+/g, " ").trim();
+      if (t) textParts.push(t);
+    }
+    node = walker.nextNode();
+  }
+  const plainText = textParts
+    .join(" ")
+    .replace(/\s{2,}/g, " ")
+    .trim();
+  let pageURL = "";
+  try {
+    pageURL = window.top.location.href;
+  } catch (e) {
+    pageURL = location.href;
+  }
+  pageURL = pageURL.split("#RB:")[0];
+  const combined =
+    "=== Page URL ===\n" +
+    pageURL +
+    "\n\n=== Forms & Form Elements ===\n" +
+    formOut +
+    "\n=== Other Interactive Elements ===\n" +
+    otherOut +
+    "\n=== Plain Text (including hidden) ===\n" +
+    plainText +
+    "\n\n=== Cleaned HTML (absolute links) ===\n" +
+    cleanedHTML;
+  function fallbackCopy(text) {
+    const ta = document.createElement("textarea");
+    ta.value = text;
+    ta.style.position = "fixed";
+    ta.style.left = "-9999px";
+    document.body.appendChild(ta);
+    ta.select();
+    try {
+      document.execCommand("copy");
+    } catch (err) {}
+    document.body.removeChild(ta);
+  } /* Delivery. The clipboard CANNOT be relied on here: after the omnibox runs this,
+   focus usually stays in the address bar, so navigator.clipboard.writeText
+   rejects with NotAllowedError ("Document is not focused") roughly half the time
+   and the dump is silently lost. A page cannot focus itself, so there is no fix
+   from in here. location.hash needs no focus and no permission, and the caller
+   reads it straight off the tab URL - that is the reliable channel. The
+   clipboard write stays as a best-effort bonus (documented behavior) and its
+   failure no longer matters. The hash is cleared afterwards so the address bar
+   is not left holding the whole dump. */
+  try {
+    location.hash = `RB:${encodeURIComponent(combined)}`;
+    setTimeout(() => {
+      try {
+        history.replaceState(null, "", location.pathname + location.search);
+      } catch (e) {}
+    }, 20000);
+  } catch (e) {}
+  try {
+    if (navigator.clipboard?.writeText) {
+      navigator.clipboard.writeText(combined).catch(() => {
+        fallbackCopy(combined);
+      });
+    } else {
+      fallbackCopy(combined);
+    }
+  } catch (e) {}
+})();