@rebind.gg/mcp-server 0.1.6 → 0.2.0

package/README.md

@@ -1,13 +1,11 @@
 # @rebind.gg/mcp-server
 
-Stdio MCP server that turns [Rebind](https://rebind.gg) into a computer-use toolkit: screenshots, verified clicks, keyboard input, window control, and direct Luau scripting on the connected machine — driven over real HID, not synthetic OS events.
+Stdio MCP server that turns [Rebind](https://rebind.gg) into a computer-use toolkit: screenshots, verified clicks, keyboard input, window control, and direct Luau scripting on the connected machine. Input uses the relay's active transport: genuine USB HID in hardware mode or OS injection in software mode.
 
 **Requirements**
 
 - [Bun](https://bun.sh) >= 1.1 (the server runs via `bunx`)
-- The [Rebind app](https://rebind.gg/download) running its Remote Access relay: `remote.luau` (recommended) or `remote_access.lua` (protocol >= 1.1.0)
-
-> Older copies of `remote_access.lua` called `hid.press` outside a coroutine, so clicks silently failed. Use `remote.luau` or a current `remote_access.lua`.
+- The [Rebind app](https://rebind.gg/download) running the free [Remote Control](https://www.rebind.gg/marketplace/script_AAHteeDcYPGOT7GfXZ_OK) package (`rebind install @rebind/remote-control`)
 
 ## Install
 
@@ -56,19 +54,25 @@ Connects, takes a capture, and reports the display — the fastest way to catch
 | `move_mouse` | Move without clicking, with the same verification. |
 | `type` | Type text into the focused element. |
 | `key` | Press a key or combo, e.g. `Enter`, `LCtrl+L`, `LAlt+Tab`. |
-| `scroll` | Scroll the wheel at the cursor (positive = up). |
+| `scroll` | Scroll at the cursor. Positive scrolls down, negative up. `unit` is `wheel` (default, fine wheel clicks) or `page` (one PageDown or PageUp per unit, capped at 10). |
+| `read_page` | Extract the focused browser's current page as structured text (URL, forms, interactive elements, plain text) via an address-bar bookmarklet; the full dump, including cleaned HTML, lands in the clipboard. The result is read back through AppleScript (`URL of active tab of front window`), so the relay host must be macOS and the front browser must answer that query. Safari is detected but blocks the bookmarklet path. |
+| `meditate` | Research a problem page until a reasoning model has the data to solve it: fetches the URL directly, pulls the script bundles behind a JS shell so its API endpoints are visible, fetches those endpoints and assets, and reads the live page with vision. Returns a markdown dossier plus JSON `{error, problem, requirements, plan, submittable, form, live_captcha_hint, confidence, sources, assets, pending_fetch, notes, cost_usd}`. `error` is non-null when no problem could be identified or the page could not be read; partial data is still returned. Gated behind `REBIND_ALLOW_DELEGATION=1` (spends OpenRouter credits). See the pipeline below. |
 | `run_lua` | Run a Luau script on the connected machine and return its `return` value as JSON — the fast path for any deterministic sequence (launch apps, place windows, shell commands, pixel search). |
 | `lua_docs` | Full Luau SDK type definitions for `run_lua`. |
-| `list_displays` / `list_windows` / `focus_window` / `active_window` | Display and window enumeration and control. |
+| `list_displays` / `list_windows` / `focus_window` / `position_window` / `active_window` | Display and window enumeration, focus, and placement. |
 | `get_cursor` / `wait` / `calibrate` | Cursor readback, pauses, and the mouse scale probe. |
+| `delegate_task` / `rebind_versus` | Server-side reasoning helpers with no desktop access: one bounded text task, or parallel critics that attack a plan. Registered only with `REBIND_ALLOW_DELEGATION=1` (spends OpenRouter credits). |
 
 ## Configuration
 
 | Env var | Default | Purpose |
 | --- | --- | --- |
 | `REBIND_URL` | `ws://127.0.0.1:19561` | Relay WebSocket URL. |
-| `REBIND_TOKEN` | — | Required when the relay has an auth token set (the **Auth token** panel field on `remote.luau`, or `AUTH_TOKEN` on `remote_access.lua`). |
+| `REBIND_TOKEN` | — | Required when the relay has an auth token set (the **Auth token** field in Remote Control's Options). |
 | `REBIND_AGENT_LOG` | off | Path to append one JSONL line per tool call (`{ts, tool, ok, latency_ms, note}`). No screenshot bytes, no user content. |
+| `REBIND_MODEL` | `google/gemini-3.5-flash-lite` | OpenRouter slug for `delegate_task`/`rebind_versus`/`meditate` reasoning (only used with `REBIND_ALLOW_DELEGATION=1`). |
+| `REBIND_VISION_MODEL` | falls back to `REBIND_MODEL` | Multimodal slug for `meditate`'s live-page/captcha vision pass. Set this when `REBIND_MODEL` is a text-only model. |
+| `REBIND_ALLOW_DELEGATION` | off | Set to `1` to register the credit-spending tools: `delegate_task`, `rebind_versus`, `meditate`. |
 
 ## How it works
 
@@ -80,10 +84,22 @@ Connects, takes a capture, and reports the display — the fastest way to catch
 
 **One coordinate space.** All results are reported in the pixel space of the last screenshot — the same space the model emits — so no rescaling math ever falls on the model.
 
+**Machine-aware orientation.** At startup the server probes the connected machine through the relay (OS, shortcut modifier, available shell tools, installed apps) and serves it as the MCP `instructions` block, along with a policy: prefer `run_lua` for deterministic work, keyboard over mouse, and screenshots last. The full Luau SDK type definitions are embedded inline so the model never has to guess `run_lua` APIs (including the coroutine-safe HID path — `HID.Down`/`Up`/`MoveTo`/`Combo` work in a one-shot chunk, `HID.Press` does not). The OS is read from the relay, not `process.platform`, so the guidance is correct even when the server and the relay run on different machines.
+
+## Solve pipeline
+
+`meditate` gathers; it does not submit. The intended flow, all gated behind `REBIND_ALLOW_DELEGATION=1`:
+
+1. **`meditate`** → a dossier. Evidence is gathered by **fetching, not by driving the GUI**: a direct GET returns the server's own bytes, and when that comes back a JS shell the script bundles are pulled in so the endpoints the page calls are visible to the model, which then requests them by id. Same-origin URLs are fetched **from inside the page** — the browser's own session, cookies and TLS answer them, so an endpoint that refuses a bare curl responds normally; cross-origin falls back to relay-side curl. The rendered DOM is read when a fetch cannot see the content. It then leaves the browser **open on the problem page** and returns `form` (per-field `{value, source, op}`, traced to evidence) and `submittable`. It never bakes in the captcha — that can rotate, so it is read live at submit time, and its absence does not make a dossier unsubmittable.
+2. **`rebind_versus`** → feed it the dossier to attack the reconciled values before anything irreversible happens.
+3. **Solve (caller-driven, by sight)** on the still-open page: re-navigate to the anchor and confirm the URL; screenshot → click field → type each `form` value; tick attachment checkboxes; read the **live** captcha off the page; then **re-extract the filled form and diff it against the dossier** — on any mismatch or unresolved conflict, stop (do not submit). Only on a clean diff, click submit and verify. When `submittable` is false there is no form to fill; surface the answer instead.
+
+The verify-diff before an irreversible submit is a discipline the caller enforces — auto-submitting reconciled data fails closed, not open.
+
 ## Develop
 
 ```sh
-bun test packages/mcp-server/tests/
+bun run --cwd packages/mcp-server test        # src/ and tests/
 bun run --cwd packages/mcp-server typecheck
 ```
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@rebind.gg/mcp-server",
-  "version": "0.1.6",
-  "description": "MCP server exposing Rebind as computer-use tools: screenshot, verified click, type, key, scroll, and window control over the Remote Access protocol.",
+  "version": "0.2.0",
+  "description": "MCP server exposing Rebind as computer-use tools: screenshot, verified click, type, key, scroll, and window control over the Remote Control protocol.",
   "type": "module",
   "bin": {
     "rebind-mcp-server": "src/index.ts"
@@ -30,7 +30,6 @@
     "mcp",
     "model-context-protocol",
     "computer-use",
-    "openclaw",
     "hid",
     "automation",
     "keyboard",
@@ -42,12 +41,16 @@
   "scripts": {
     "start": "bun run src/index.ts",
     "selftest": "bun run src/index.ts --selftest",
-    "test": "bun test tests/",
-    "typecheck": "tsc --noEmit"
+    "test": "bun test src/ tests/",
+    "typecheck": "tsc --noEmit",
+    "bench": "bun run bench/runner.ts",
+    "bench:selftest": "bun run bench/runner.ts --selftest",
+    "bench:setup": "bun run bench/runner.ts --setup-only",
+    "bench:demo": "bash bench/demo.sh"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.12.0",
-    "@rebind.gg/client-ts": "0.2.1",
+    "@rebind.gg/client-ts": "0.3.0",
     "sharp": "^0.34.2",
     "zod": "^3.25.0"
   },

package/src/bookmarklet.js

@@ -0,0 +1,182 @@
+(() => {
+  const clone = document.documentElement.cloneNode(true);
+  clone
+    .querySelectorAll('script, style, link[rel="stylesheet"], svg, noscript')
+    .forEach((el) => {
+      el.remove();
+    });
+  clone.querySelectorAll("*").forEach((el) => {
+    [...el.attributes].forEach((attr) => {
+      if (attr.name.startsWith("on") || attr.name === "style")
+        el.removeAttribute(attr.name);
+    });
+  });
+  clone.querySelectorAll("[href]").forEach((el) => {
+    try {
+      el.setAttribute(
+        "href",
+        new URL(el.getAttribute("href"), document.baseURI).href,
+      );
+    } catch (e) {}
+  });
+  clone.querySelectorAll("[src]").forEach((el) => {
+    try {
+      el.setAttribute(
+        "src",
+        new URL(el.getAttribute("src"), document.baseURI).href,
+      );
+    } catch (e) {}
+  });
+  clone.querySelectorAll("[action]").forEach((el) => {
+    try {
+      el.setAttribute(
+        "action",
+        new URL(el.getAttribute("action"), document.baseURI).href,
+      );
+    } catch (e) {}
+  });
+  const cleanedHTML = clone.outerHTML
+    .replace(/\s{2,}/g, " ")
+    .replace(/>\s+</g, ">\n<");
+  const forms = document.querySelectorAll("form");
+  let formOut = "";
+  const formElementsSet = new Set();
+  forms.forEach((form, fi) => {
+    const rect = form.getBoundingClientRect();
+    const scrollY = Math.round(rect.top + window.scrollY);
+    formOut += `[Form ${fi}] action="${form.action || ""}" method="${form.method}" scrollY=${scrollY}\n`;
+    form
+      .querySelectorAll("input, select, textarea, button")
+      .forEach((el, ei) => {
+        formElementsSet.add(el);
+        const r = el.getBoundingClientRect();
+        const sY = Math.round(r.top + window.scrollY);
+        const label = (
+          el.getAttribute("aria-label") ||
+          el.value ||
+          el.textContent ||
+          ""
+        )
+          .trim()
+          .replace(/\s+/g, " ")
+          .slice(0, 60);
+        formOut += `   [${fi}.${ei}] <${el.tagName.toLowerCase()}> type="${el.type || ""}" name="${el.name || ""}" scrollY=${sY} label="${label}"\n`;
+      });
+    formOut += "\n";
+  });
+  let otherOut = "";
+  let idx = 0;
+  document
+    .querySelectorAll("a, button, input, select, textarea, [role]")
+    .forEach((el) => {
+      if (formElementsSet.has(el) || el.closest("form")) return;
+      const rect = el.getBoundingClientRect();
+      const scrollY = Math.round(rect.top + window.scrollY);
+      const label = (
+        el.getAttribute("aria-label") ||
+        el.textContent ||
+        el.value ||
+        ""
+      )
+        .trim()
+        .replace(/\s+/g, " ")
+        .slice(0, 60);
+      const href = el.href ? el.href : "";
+      const role = el.getAttribute("role") || "";
+      const tag = el.tagName.toLowerCase();
+      const ac = el.getAttribute("aria-controls") || "";
+      const ae = el.getAttribute("aria-expanded");
+      const asel = el.getAttribute("aria-selected");
+      const submitish =
+        el.type === "submit" ||
+        el.type === "reset" ||
+        el.name === "intent" ||
+        /\b(submit|save|reset|cancel|delete)\b/i.test(label);
+      const reveal =
+        !href &&
+        !submitish &&
+        (role === "tab" ||
+          !!ac ||
+          ae !== null ||
+          asel !== null ||
+          tag === "button" ||
+          role === "button");
+      otherOut += `[${idx}] <${tag}> role="${role}" scrollY=${scrollY} name="${label}"${href ? ` href="${href}"` : ""}${ac ? ` aria-controls="${ac}"` : ""}${ae !== null ? ` aria-expanded="${ae}"` : ""}${asel !== null ? ` aria-selected="${asel}"` : ""}${reveal ? " reveal=1" : ""}\n`;
+      idx++;
+    });
+  const walker = document.createTreeWalker(
+    document.body,
+    NodeFilter.SHOW_TEXT,
+    null,
+    false,
+  );
+  const textParts = [];
+  let node = walker.nextNode();
+  while (node) {
+    const parent = node.parentElement;
+    if (!parent || !["SCRIPT", "STYLE", "NOSCRIPT"].includes(parent.tagName)) {
+      const t = node.nodeValue.replace(/\s+/g, " ").trim();
+      if (t) textParts.push(t);
+    }
+    node = walker.nextNode();
+  }
+  const plainText = textParts
+    .join(" ")
+    .replace(/\s{2,}/g, " ")
+    .trim();
+  let pageURL = "";
+  try {
+    pageURL = window.top.location.href;
+  } catch (e) {
+    pageURL = location.href;
+  }
+  pageURL = pageURL.split("#RB:")[0];
+  const combined =
+    "=== Page URL ===\n" +
+    pageURL +
+    "\n\n=== Forms & Form Elements ===\n" +
+    formOut +
+    "\n=== Other Interactive Elements ===\n" +
+    otherOut +
+    "\n=== Plain Text (including hidden) ===\n" +
+    plainText +
+    "\n\n=== Cleaned HTML (absolute links) ===\n" +
+    cleanedHTML;
+  function fallbackCopy(text) {
+    const ta = document.createElement("textarea");
+    ta.value = text;
+    ta.style.position = "fixed";
+    ta.style.left = "-9999px";
+    document.body.appendChild(ta);
+    ta.select();
+    try {
+      document.execCommand("copy");
+    } catch (err) {}
+    document.body.removeChild(ta);
+  } /* Delivery. The clipboard CANNOT be relied on here: after the omnibox runs this,
+   focus usually stays in the address bar, so navigator.clipboard.writeText
+   rejects with NotAllowedError ("Document is not focused") roughly half the time
+   and the dump is silently lost. A page cannot focus itself, so there is no fix
+   from in here. location.hash needs no focus and no permission, and the caller
+   reads it straight off the tab URL - that is the reliable channel. The
+   clipboard write stays as a best-effort bonus (documented behavior) and its
+   failure no longer matters. The hash is cleared afterwards so the address bar
+   is not left holding the whole dump. */
+  try {
+    location.hash = `RB:${encodeURIComponent(combined)}`;
+    setTimeout(() => {
+      try {
+        history.replaceState(null, "", location.pathname + location.search);
+      } catch (e) {}
+    }, 20000);
+  } catch (e) {}
+  try {
+    if (navigator.clipboard?.writeText) {
+      navigator.clipboard.writeText(combined).catch(() => {
+        fallbackCopy(combined);
+      });
+    } else {
+      fallbackCopy(combined);
+    }
+  } catch (e) {}
+})();