@mmerterden/multi-agent-toolkit-mcp 3.13.0 → 3.13.1

package/CHANGELOG.md

@@ -19,6 +19,76 @@ Releases before this file exists are recorded in the git tags and commit history
 
 ---
 
+## 3.13.1 - 2026-09-21
+
+Fixes to the web and context families, each held by a gate proven to fail
+without it. The acceptance numbers at the end were measured against live sites.
+
+### Fixed
+
+- **A question asked of the index returned nothing at all.** FTS5 reads the
+  space between two terms as AND, so a nine-word question demanded all nine
+  words from one chunk. `context_search` now runs the every-term query first,
+  because a caller who typed all the terms meant all of them, and falls back to
+  an any-term query when that finds nothing; `bm25` then ranks by how many of
+  the terms a chunk carries and how rare each one is. A log line still matches
+  exactly as before.
+- **`web_crawl`, `web_extract` and `web_map` crawled `about:blank`.** Run before
+  the session has navigated, extraction succeeded and returned nothing: the
+  crawl reported one page, the index gained an empty document, and the next
+  search ranked it. The three now name the missing `web_goto` instead of
+  returning an empty success.
+- **A bounded crawl spent its budget on site furniture.** Links were followed in
+  DOM order, so a global index, a feedback form and a breadcrumb came before the
+  second body link. Links inside the page's content region are handed back
+  first, and a link that differs from another only by its query string is
+  treated as the same document. Measured on a 20-page crawl of the Python
+  standard library index: 14 material pages before, 18 after.
+- **Refs from inside a frame were never registered.** Playwright stamps a frame
+  generation onto refs handed out past the first navigation (`f2e8`, not `e8`),
+  so the registry accepted nothing from the second page onward and refused every
+  ref the snapshot plainly showed.
+- **`web_click` reported a timeout without saying what caused it.** A refusal
+  now names the reason: disabled, not displayed, transparent, zero-sized,
+  off-viewport, `pointer-events: none`, or the element that is on top of it at
+  the click point.
+- **Readable extraction fell back to the page source.** The Readability and
+  Turndown browser bundles are resolved by their browser filenames and injected
+  into the page; when neither is available the fallback reads the live content
+  container's rendered text and drops chrome line by line.
+- **`context_search` said which lines matched but not which file.** Each hit now
+  carries its path.
+
+### Added
+
+- **`web_crawl` feeds the full-text index.** Each page is written beside the
+  index and indexed under its URL, so twenty pages arrive searchable instead of
+  as a wall of text. Failure to index never fails the crawl.
+- **An offloaded payload's summary carries its signal lines.** A head-and-tail
+  window keeps the two places a large payload is least likely to carry its
+  answer; the tool name is already known, so up to twenty matching lines are
+  surfaced with their line numbers - `FATAL` and `Abort trap` for a crash log,
+  `error:` and `Undefined symbol` for a build, tappable nodes for a UI dump,
+  measurements for a memory report. Patterns only, no model.
+- **`@mozilla/readability` and `turndown` as optional peers**, alongside
+  `playwright`. Absent, extraction uses the fallback above.
+- **`MCP_TOOLKIT_INDEX_DIR`** overrides where the index lives, so a gate ranks a
+  corpus it owns.
+
+### Acceptance
+
+Run against live sites before release:
+
+| Criterion                                                                                | Result                |
+| ---------------------------------------------------------------------------------------- | --------------------- |
+| Click by ref on ten real sites, asserting the page moved                                 | 10/10                 |
+| Article body extracted from ten pages, no nav, cookie or footer text                     | 10/10                 |
+| `max_pages: 20` crawl of a documentation site, then five questions ranked from the index | 5/5 first-hit correct |
+
+None of the five questions names the module it should find.
+
+---
+
 ## 3.13.0 - 2026-09-20
 
 ### Added
@@ -177,8 +247,7 @@ because the thing that broke was the assumption that the host is a Mac.
   `pass_inspect`, `pass_certificates`. Build, sign, validate and inspect Apple
   Wallet passes; offline, deterministic, and generic by construction. All five
   of Apple's styles are first-class with a per-style image contract, and no
-  brand, airline or project appears anywhere in the directory. 95 tools become
-  99. Three are read-only; `pass_build` writes the file the caller named.
+  brand, airline or project appears anywhere in the directory. 95 tools become 99. Three are read-only; `pass_build` writes the file the caller named.
 
   **A literal passphrase is not an accepted input.** The caller names an
   environment variable (`passphrase_env`) or a keychain entry, and the value
@@ -576,7 +645,7 @@ unverifiable without a device or emulator to run adb against.
 
 - **The accessibility audits reported a clean screen they had never read.** On a
   tree that came back empty they returned `elements_scanned: 0, total_issues: 0,
-  critical: 0, important: 0, warning: 0` - a full clean bill of health,
+critical: 0, important: 0, warning: 0` - a full clean bill of health,
   indistinguishable from an accessible screen, and the first reading is the one
   anyone believes. Both audits now carry `measurable` and a `reason`, and an
   unmeasurable run reports `total_issues: null`, never 0. A count of zero is a
@@ -704,7 +773,7 @@ declare `>= 3.1.0`.
   `~/.claude/logs/multi-agent-toolkit/<tool>-<timestamp>.txt`, and the tool
   returns a head + tail window with the line count and the path.
   `agent_query_output {pattern, path?, context_lines?, max_matches?,
-  ignore_case?}` searches that file and returns matching lines with numbered
+ignore_case?}` searches that file and returns matching lines with numbered
   context, so a follow-up question does not mean re-running an expensive tool -
   and for a UI dump a second run is not even the same evidence.
 
@@ -787,7 +856,7 @@ removed (still 83); input handling is tightened but stays backward-compatible.
 - `ios_record_video` and `android_record_screen` actually record. Both previously
   returned a literal "Run: ..." instruction for the caller to execute by hand. Now
   `action:"start"` spawns the recorder (`simctl io recordVideo` / `adb shell
-  screenrecord`) in the background with PID tracking and returns immediately;
+screenrecord`) in the background with PID tracking and returns immediately;
   `action:"stop"` interrupts it (SIGINT, so the mp4 container is finalized), pulls
   the file off the device on Android, and returns the local path. One recording
   per device; a stop with nothing running is an error.
@@ -1006,10 +1075,11 @@ Documentation that claimed more than it checked, and the gate that let it.
 
 **Editor support is now stated as verified vs compatible.** The README opened with
 "Works with Claude Code, Copilot CLI, Cursor, Antigravity, and VS Code Copilot Chat
+
 - the multi-agent-pipeline's full-orchestration targets". That parenthetical stopped
-being true in pipeline v10.7.0, when the Cursor / Antigravity / Codex / Copilot Chat
-adapters were deleted; the pipeline has targeted Claude Code and Copilot CLI only
-ever since.
+  being true in pipeline v10.7.0, when the Cursor / Antigravity / Codex / Copilot Chat
+  adapters were deleted; the pipeline has targeted Claude Code and Copilot CLI only
+  ever since.
 
 The list is not simply wrong, though: this is a plain stdio MCP server, so it does
 run in any MCP client, and the config snippets for those clients stay. What was
@@ -1021,12 +1091,12 @@ of the 78 - in the primary host, for two releases, with every gate green.
 **Gate 3b: per-family counts, not just the headline.** Gate 3 checked the two
 headline numbers and nothing else, so every subcount drifted behind a green check:
 
-| README said | reality |
-|---|---|
-| Device Control (42 tools) | 58 |
-| Store Compliance (4 tools) | 3 |
-| iOS Tools (34), diagram (31) | 35 |
-| 5 categories summing to 57 | 78 |
+| README said                  | reality |
+| ---------------------------- | ------- |
+| Device Control (42 tools)    | 58      |
+| Store Compliance (4 tools)   | 3       |
+| iOS Tools (34), diagram (31) | 35      |
+| 5 categories summing to 57   | 78      |
 
 The last row is the interesting one: the whole `design_*` family, six tools, was
 missing from the category list entirely, which is why the listed categories could
@@ -1253,12 +1323,14 @@ content beside it. 14 gates, all green.
 ## [3.0.0] - 2026-08-22
 
 ### Changed
+
 - **Renamed to `@mmerterden/multi-agent-toolkit-mcp`.** The old name read as internal scaffolding for one pipeline; this server is a standalone MCP over stdio with three runtime dependencies and no coupling to any orchestrator, and the name now says which family it belongs to. Major, because a package rename breaks every consumer that resolves it by name.
 - **The MCP server identity reported over the protocol is now `multi-agent-toolkit-mcp`.** Hosts key their registration off this, so an existing `dev-toolkit` entry does not upgrade in place - it has to be removed and re-added. The pipeline installer does that automatically; a hand-registered client needs `<cli> mcp remove dev-toolkit` once.
 - **`dev-toolkit-mcp` is kept as a second `bin` alias** so a script that invokes the old binary name keeps working through the transition.
 - Package description and keywords lead with what the server does (iOS Simulator, Android Emulator, headless web, 83 tools) rather than with the family it ships in, because that is what someone searching for it will search for.
 
 ### Migration
+
 - `npm i -g @mmerterden/multi-agent-toolkit-mcp` then remove the old registration: `claude mcp remove dev-toolkit` (same for `copilot` / `codex`).
 - `@mmerterden/dev-toolkit-mcp` stays published at 2.26.0 and is deprecated with a pointer to the new name. Nothing is unpublished; a pinned consumer keeps resolving.
 
@@ -1480,7 +1552,7 @@ rarely the same size.
   size finding that buried the real defects. The comparison now reports
   per-element edge insets (left/right) rather than raw width, gaps between
   consecutive elements, vertical placement, and font size, family and text
-  colour sampled from the text ink rather than the box average -  the box average
+  colour sampled from the text ink rather than the box average - the box average
   previously returned the background colour and so passed every check. Pass
   `responsive: false` to restore the previous absolute-delta behaviour.
 - **Height is advisory.** It is still reported, but a content-driven height
@@ -1512,20 +1584,20 @@ because no failure ever carried `isError`, no host could tell.
 ### Fixed
 
 - **`isError` on every failure.** The `CallTool` dispatch returned command
-  failures as ordinary text, so a host -  and the pipeline gates reading these
-  results -  saw failure as success. Failures now return `isError: true`. Failing
+  failures as ordinary text, so a host - and the pipeline gates reading these
+  results - saw failure as success. Failures now return `isError: true`. Failing
   command output is also capped at 600 chars: a failed `simctl` call was
   inlining its entire ~3 KB usage page into the caller's context.
 - **`ios_biometric` was a no-op that claimed success.** `simctl keychain
-  <device> biometric-enroll` / `biometric-match` do not exist -  `keychain`
+<device> biometric-enroll` / `biometric-match` do not exist - `keychain`
   supports only `add-root-cert`, `add-cert`, `reset`. Now drives the BiometricKit
   notification via `notifyutil` inside the simulator, and because `notifyutil`
   exits 0 even when it cannot post the name, its output is inspected: a "Failed
   with code N" line is reported as a failure with next steps instead of
   "success simulated".
 - **`ios_go_home` was a no-op.** `simctl io <device> pressButton` does not exist
-  -  `io` supports only `enumerate`, `poll`, `recordVideo`, `screenshot`,
-  `screenConfig`. Routed through `idb ui button HOME`, like tap/swipe/type.
+  - `io` supports only `enumerate`, `poll`, `recordVideo`, `screenshot`,
+    `screenConfig`. Routed through `idb ui button HOME`, like tap/swipe/type.
 - **`android_set_locale` reported success unconditionally.** It broadcast the
   dead pre-Android-7 `SET_LOCALE` intent with stderr sent to `/dev/null`, and
   `am broadcast` exits 0 even when nothing handles the intent. Now uses the
@@ -1550,7 +1622,7 @@ because no failure ever carried `isError`, no host could tell.
 
 - **`sdk-floor` audit rule** (ITMS-90725), bringing `ios_app_store_audit` to
   **18 rules**. Asserts `DTSDKName` / `DTPlatformVersion` major >= 26 and
-  `DTXcode` >= 2600, the iOS 26 / Xcode 26 floor in force since 2026-04-28 -  a
+  `DTXcode` >= 2600, the iOS 26 / Xcode 26 floor in force since 2026-04-28 - a
   hard upload rejection that nothing checked. Grouped as `core` because it costs
   nothing to run. Reports a WARNING when an archive carries no build receipts at
   all, rather than passing silently.

package/index.js

@@ -1589,15 +1589,82 @@ function fenceUntrusted(source, body) {
   return `${UNTRUSTED_OPEN} (${source}) <<<\n${body}\n${UNTRUSTED_CLOSE}`;
 }
 
+// Why an element that exists refused to be acted on. Returns null when nothing
+// obvious is wrong, so the caller still gets Playwright's own message rather
+// than a guess dressed up as a diagnosis.
+async function describeUnactionable(locator) {
+  try {
+    if ((await locator.count()) === 0) return "no element matches it any more";
+    const state = await locator.evaluate((el) => {
+      const cs = getComputedStyle(el);
+      const r = el.getBoundingClientRect();
+      return {
+        display: cs.display,
+        visibility: cs.visibility,
+        opacity: cs.opacity,
+        pointerEvents: cs.pointerEvents,
+        w: r.width,
+        h: r.height,
+        offLeft: r.right <= 0,
+        offTop: r.bottom <= 0,
+        disabled: el.hasAttribute("disabled") || el.getAttribute("aria-disabled") === "true",
+        label: (el.innerText || el.getAttribute("aria-label") || "").trim().slice(0, 40),
+      };
+    });
+    const named = state.label ? ` ("${state.label}")` : "";
+    if (state.disabled) return `it is disabled${named}`;
+    if (state.display === "none") return `it is display:none${named}`;
+    if (state.visibility === "hidden") return `it is visibility:hidden${named}`;
+    if (Number(state.opacity) === 0) return `it is fully transparent${named}`;
+    if (state.w === 0 || state.h === 0) return `it has no size${named}`;
+    if (state.offLeft || state.offTop) {
+      return `it sits outside the viewport${named} - a skip link or an off-screen menu, visible only once focused`;
+    }
+    if (state.pointerEvents === "none") return `it ignores pointer events${named}`;
+
+    // Everything above says the element is there and paintable, which is where
+    // Playwright's timeout usually comes from: it waits to deliver a pointer
+    // event and something else is on top. Asking the document what is actually
+    // at that point names the blocker instead of leaving the caller with a
+    // duration.
+    const covering = await locator.evaluate((el) => {
+      const r = el.getBoundingClientRect();
+      const x = r.left + r.width / 2;
+      const y = r.top + r.height / 2;
+      if (x < 0 || y < 0 || x > innerWidth || y > innerHeight) return "__outside__";
+      const hit = document.elementFromPoint(x, y);
+      if (!hit || hit === el || el.contains(hit) || hit.contains(el)) return null;
+      const name = hit.tagName.toLowerCase();
+      const id = hit.id ? `#${hit.id}` : "";
+      const cls = hit.className && typeof hit.className === "string"
+        ? `.${hit.className.trim().split(/\s+/).slice(0, 2).join(".")}`
+        : "";
+      return `${name}${id}${cls}`;
+    });
+    if (covering === "__outside__") {
+      return `its centre is outside the viewport${named} - scroll to it, or pick a target that is on screen`;
+    }
+    if (covering) {
+      return `${covering} is on top of it${named} - a sticky header or an overlay is taking the click`;
+    }
+    return null;
+  } catch {
+    return null;
+  }
+}
+
 async function extractReadable(page) {
   const paths = [];
   for (const mod of ["@mozilla/readability", "turndown"]) {
     try { paths.push(requireFrom.resolve(mod)); } catch { paths.push(null); }
   }
-  if (paths[0] && paths[1]) {
+  const bundles = paths[0] && paths[1]
+    ? [readabilityBundlePath(paths[0]), turndownBundlePath(paths[1])]
+    : [null, null];
+  if (bundles[0] && bundles[1]) {
     try {
-      await page.addScriptTag({ path: readabilityBundlePath(paths[0]) });
-      await page.addScriptTag({ path: turndownBundlePath(paths[1]) });
+      await page.addScriptTag({ path: bundles[0] });
+      await page.addScriptTag({ path: bundles[1] });
       const out = await page.evaluate(() => {
         const clone = document.cloneNode(true);
         // eslint-disable-next-line no-undef
@@ -1612,35 +1679,127 @@ async function extractReadable(page) {
       // fall through to the plain extraction
     }
   }
+  // The fallback used to clone the body, strip chrome from the clone and read
+  // its innerText. innerText is computed from layout, and a detached node has
+  // none - so on a real page it returned a fraction of the text or nothing,
+  // while a small fixture with everything above the fold looked fine.
+  //
+  // So: find the LIVE container that holds the article, read its innerText, and
+  // drop chrome by line. Reading the live node is what makes the text real; the
+  // line filter is what keeps "Jump to content" out of it.
   return page.evaluate(() => {
-    const drop = ["nav", "header", "footer", "aside", "script", "style", "noscript", "[role=navigation]", "[class*=cookie]", "[class*=banner]"];
-    const clone = document.body.cloneNode(true);
-    for (const sel of drop) for (const el of clone.querySelectorAll(sel)) el.remove();
-    const title = document.title || "";
-    const text = clone.innerText.replace(/\n{3,}/g, "\n\n").trim();
-    return { markdown: `# ${title}\n\n${text}`, text };
+    const CONTAINERS = [
+      "main article",
+      "article",
+      "main",
+      "[role=main]",
+      "#mw-content-text",
+      "#content",
+      ".markdown-body",
+      ".post-content",
+    ];
+    let host = null;
+    for (const sel of CONTAINERS) {
+      const el = document.querySelector(sel);
+      if (el && (el.innerText || "").trim().length > 200) {
+        host = el;
+        break;
+      }
+    }
+    if (!host) host = document.body;
+
+    const CHROME = [
+      /^jump to (content|navigation|search)$/i,
+      /^skip to (main|content)/i,
+      /^(main )?menu$/i,
+      /^toggle the table of contents$/i,
+      /^(search|sign in|log in|sign up|subscribe|donate)$/i,
+      /^(accept|reject) (all )?cookies?/i,
+      /^cookie (settings|preferences)/i,
+      /^edit (source|this page)$/i,
+      /^\[edit\]$/i,
+      /^(privacy policy|terms of use|contact us)$/i,
+    ];
+    const kept = (host.innerText || "")
+      .split("\n")
+      .filter((line) => {
+        const t = line.trim();
+        if (!t) return true;
+        return !CHROME.some((rx) => rx.test(t));
+      })
+      .join("\n")
+      .replace(/\n{3,}/g, "\n\n")
+      .trim();
+
+    // document.title carries the site suffix ("Bloom filter - Wikipedia").
+    // The first h1 inside the container is the article's own name.
+    const h1 = host.querySelector("h1") || document.querySelector("h1");
+    const title = (h1 && h1.innerText.trim()) || document.title || "";
+    return { markdown: `# ${title}\n\n${kept}`, text: kept };
   });
 }
 
+// Both packages ship a Node entry point and a browser bundle, and only the
+// browser bundle can be injected into a page: the Node entry is CommonJS, so
+// addScriptTag loads it, `module` is undefined, and the whole extraction falls
+// through to the plain path without ever saying why.
+//
+// The file is checked rather than assumed. A layout change upstream returns
+// null here, which keeps the fallback rather than injecting a file that throws.
 function readabilityBundlePath(resolved) {
-  return resolved.replace(/index\.js$/, "Readability.js");
+  const candidate = join(dirname(resolved), "Readability.js");
+  return existsSync(candidate) ? candidate : null;
 }
 function turndownBundlePath(resolved) {
-  return resolved.replace(/turndown\.cjs$|index\.js$/, "turndown.js");
+  for (const name of ["turndown.browser.umd.js", "turndown.umd.js", "turndown.js"]) {
+    const candidate = join(dirname(resolved), name);
+    if (existsSync(candidate)) return candidate;
+  }
+  return null;
+}
+
+// Every tool below reads the page that is already open. A session that has not
+// navigated yet is sitting on `about:blank`, where extraction succeeds and
+// returns nothing: the crawl reports "1 page", the index gains an empty
+// document, and the search that follows ranks it. Name the missing step
+// instead of returning an empty success.
+function blankPageError(page, tool) {
+  const url = page.url();
+  if (url && url !== "about:blank" && !url.startsWith("chrome-error://")) return null;
+  return `ERROR: ${tool} reads the page that is currently open, and nothing has been opened yet (${url || "no url"}). Call web_goto first.`;
 }
 
 async function sameOriginLinks(page) {
   return page.evaluate(() => {
     const here = location.origin;
-    const out = new Set();
-    for (const a of document.querySelectorAll("a[href]")) {
-      let u;
-      try { u = new URL(a.getAttribute("href"), location.href); } catch { continue; }
-      if (u.origin !== here) continue;
-      u.hash = "";
-      out.add(u.toString());
-    }
-    return [...out];
+    // A documentation page carries two kinds of link: the ones in its body,
+    // which are the material, and the site chrome - a global index, a
+    // "report a bug" form, the breadcrumb back to the root - which is the
+    // same handful of pages on every page of the site. A bounded crawl that
+    // takes them in DOM order spends a third of its budget on chrome before
+    // it reaches the second body link, so body links are handed back first.
+    const CONTENT = ["main article", "article", "main", "[role=main]", "#content", ".markdown-body", ".body"];
+    let host = null;
+    for (const sel of CONTENT) {
+      const el = document.querySelector(sel);
+      if (el && el.querySelector("a[href]")) { host = el; break; }
+    }
+    const collect = (root) => {
+      const found = [];
+      for (const a of root.querySelectorAll("a[href]")) {
+        let u;
+        try { u = new URL(a.getAttribute("href"), location.href); } catch { continue; }
+        if (u.origin !== here) continue;
+        // A link that differs from another only by its query string is the
+        // same document with a feedback form or a tracking tag attached.
+        if (u.search && /^(https?:)?[^?]*\.(html?|md|txt)$/i.test(u.pathname)) u.search = "";
+        u.hash = "";
+        found.push(u.toString());
+      }
+      return found;
+    };
+    const ordered = host ? [...collect(host), ...collect(document)] : collect(document);
+    return [...new Set(ordered)];
   });
 }
 
@@ -1724,7 +1883,7 @@ const WEB_TOOLS = [
   { name: "web_storage_state", description: "Read cookies and localStorage for the current page, or restore a previously read state.", inputSchema: { type: "object", properties: { restore: { type: "string", description: "A state JSON string from a previous call. Omit to read." } } } },
   { name: "web_extract", description: "The page's article content as Markdown, with navigation, footers and cookie banners dropped. Falls back to a plain text extraction when the readability helpers are not installed.", inputSchema: { type: "object", properties: { as: { type: "string", enum: ["markdown", "text"], description: "Default markdown." } } } },
   { name: "web_map", description: "Same-origin links reachable from the current page, deduplicated. A cheap site map without fetching anything.", inputSchema: { type: "object", properties: { limit: { type: "number", description: "Default 200." } } } },
-  { name: "web_crawl", description: "Follow same-origin links from the current page, extracting each one. Bounded: max_pages default 20 (cap 200), max_depth default 2, one request at a time with a delay, robots.txt respected, and a self-identifying User-Agent.", inputSchema: { type: "object", properties: { max_pages: { type: "number" }, max_depth: { type: "number" }, delay_ms: { type: "number", description: "Default 250." }, respect_robots: { type: "boolean", description: "Default true." } } } },
+  { name: "web_crawl", description: "Follow same-origin links from the current page, extracting each one and indexing it so context_search can rank the crawl afterwards. Bounded: max_pages default 20 (cap 200), max_depth default 2, one request at a time with a delay, robots.txt respected, and a self-identifying User-Agent.", inputSchema: { type: "object", properties: { max_pages: { type: "number" }, max_depth: { type: "number" }, delay_ms: { type: "number", description: "Default 250." }, index: { type: "boolean", description: "Index each page into the full-text index. Default true - a crawl whose pages cannot be searched afterwards returns a wall of text nobody can query." }, respect_robots: { type: "boolean", description: "Default true." } } } },
   { name: "web_press_key", description: "Press a key, optionally focusing an element first.", inputSchema: { type: "object", properties: { key: { type: "string", description: "Playwright key name: Enter, Escape, Tab, ArrowDown, Control+A ..." }, selector: { type: "string" }, ref: { type: "string" } }, required: ["key"] } },
   { name: "web_select_option", description: "Choose an option in a <select>, by value or by visible label.", inputSchema: { type: "object", properties: { selector: { type: "string" }, ref: { type: "string" }, value: { type: "string" }, label: { type: "string" } } } },
 ];
@@ -1768,7 +1927,21 @@ async function handleWeb(name, args) {
     case "web_click": {
       const t = locate(page, args);
       if (t.error) return t.error;
-      await t.locator.first().click({ timeout: args.timeout_ms || 5000 });
+      try {
+        await t.locator.first().click({ timeout: args.timeout_ms || 5000 });
+      } catch (e) {
+        // Playwright waits for the element to be actionable and then reports a
+        // timeout, which says nothing about WHY. The commonest reason on a real
+        // page is a target the snapshot listed and the layout hides: the first
+        // link on an accessibility-first site is a "Skip to content" link
+        // positioned off-screen until it takes focus. A caller told only
+        // "Timeout 5000ms exceeded" re-tries the same ref; one told the element
+        // is hidden picks a different one.
+        const why = await describeUnactionable(t.locator.first());
+        return why
+          ? `${ERROR_PREFIX}${t.label} could not be clicked: ${why}`
+          : `${ERROR_PREFIX}${t.label} could not be clicked: ${String(e.message || e).split("\n")[0]}`;
+      }
       return `Clicked: ${t.label}`;
     }
     case "web_type": {
@@ -1799,7 +1972,14 @@ async function handleWeb(name, args) {
       const tree = await target.ariaSnapshot({ mode: "ai" });
       // Register every ref the snapshot handed out, stamped with this
       // snapshot's id so a later navigation invalidates them as a set.
-      for (const m of String(tree).matchAll(/\[ref=(e\d+)\]/g)) {
+      //
+      // An element inside an iframe is handed out as `f<frame>e<n>`, not `e<n>`.
+      // Matching only the bare form registers nothing on a page that frames
+      // anything - a cookie banner, an embedded video, an analytics pixel - so
+      // every ref on it is refused as unknown while the snapshot plainly shows
+      // it. Playwright resolves both spellings through the same `aria-ref=`
+      // engine; only this registry needed to accept them.
+      for (const m of String(tree).matchAll(/\[ref=((?:f\d+)?e\d+)\]/g)) {
         _refs.set(m[1], { snapshot: _snapshotId });
       }
       return `${_refs.size} refs\n${tree}`;
@@ -1864,24 +2044,43 @@ async function handleWeb(name, args) {
     }
 
     case "web_extract": {
+      const blank = blankPageError(page, "web_extract");
+      if (blank) return blank;
       const text = await extractReadable(page);
       const body = args.as === "text" ? text.text : text.markdown;
       return fenceUntrusted(page.url(), body);
     }
 
     case "web_map": {
+      const blank = blankPageError(page, "web_map");
+      if (blank) return blank;
       const links = await sameOriginLinks(page);
       const limit = args.limit || 200;
       return links.slice(0, limit).join("\n") || "(no same-origin links on this page)";
     }
 
     case "web_crawl": {
+      const blank = blankPageError(page, "web_crawl");
+      if (blank) return blank;
       const maxPages = Math.min(Number(args.max_pages) || 20, 200);
       const maxDepth = Number(args.max_depth) || 2;
       const delay = Number(args.delay_ms) || 250;
       const respectRobots = args.respect_robots !== false;
       const start = page.url();
       const origin = new URL(start).origin;
+      const wantIndex = args.index !== false;
+      let indexed = 0;
+      let indexDb = null;
+      let crawlDir = null;
+      if (wantIndex) {
+        try {
+          indexDb = ctxIndex.openIndex();
+          crawlDir = join(ctxIndex.INDEX_DIR, "crawl");
+          if (!existsSync(crawlDir)) mkdirSync(crawlDir, { recursive: true });
+        } catch {
+          indexDb = null;
+        }
+      }
       const disallowed = respectRobots ? await robotsDisallow(page, origin) : [];
       const seen = new Set([start]);
       const queue = [{ url: start, depth: 0 }];
@@ -1897,6 +2096,23 @@ async function handleWeb(name, args) {
         }
         const doc = await extractReadable(page);
         out.push(`--- ${url}\n${doc.markdown}`);
+        // A crawl that cannot be searched afterwards is a wall of text: twenty
+        // pages arrive at once and the caller has no way to ask which of them
+        // answers the question. Each page is written beside the index and
+        // indexed under its URL, so context_search ranks the crawl the same way
+        // it ranks an offloaded payload. Failure here never fails the crawl -
+        // the pages are already in the reply.
+        if (wantIndex && indexDb) {
+          try {
+            const slug = url.replace(/[^a-z0-9]+/gi, "-").slice(0, 120);
+            const file = join(crawlDir, `${slug || "page"}.md`);
+            writeFileSync(file, `# ${url}\n\n${doc.markdown}\n`);
+            ctxIndex.indexFile(indexDb, file);
+            indexed += 1;
+          } catch {
+            /* the page is in the reply; the index is the bonus */
+          }
+        }
         if (depth < maxDepth) {
           for (const link of await sameOriginLinks(page)) {
             if (seen.has(link) || seen.size >= maxPages * 4) continue;
@@ -1909,7 +2125,12 @@ async function handleWeb(name, args) {
         if (queue.length && out.length < maxPages) await page.waitForTimeout(delay);
       }
       resetRefs();
-      return `${out.length} page(s)\n\n${fenceUntrusted(`${out.length} page(s) from ${origin}`, out.join("\n\n"))}`;
+      const indexNote = indexed
+        ? ` - ${indexed} indexed, searchable with context_search`
+        : wantIndex && indexDb === null
+          ? " - the index could not be opened, so these pages are not searchable"
+          : "";
+      return `${out.length} page(s)${indexNote}\n\n${fenceUntrusted(`${out.length} page(s) from ${origin && origin !== "null" ? origin : start}`, out.join("\n\n"))}`;
     }
 
     case "web_press_key": {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mmerterden/multi-agent-toolkit-mcp",
-  "version": "3.13.0",
+  "version": "3.13.1",
   "description": "MCP server for iOS Simulator, Android Emulator and headless web control. 115 tools: device automation (tap/swipe/type), accessibility audits, visual diff, crash logs, App Store / Play Store pre-submission compliance. Runs standalone over stdio with any MCP client.",
   "type": "module",
   "main": "index.js",
@@ -85,11 +85,24 @@
     "pngjs": "^7.0.0"
   },
   "peerDependencies": {
-    "playwright": ">=1.60.0 <2"
+    "@mozilla/readability": "^0.6.0",
+    "playwright": ">=1.60.0 <2",
+    "turndown": "^7.2.0"
   },
   "peerDependenciesMeta": {
     "playwright": {
       "optional": true
+    },
+    "@mozilla/readability": {
+      "optional": true
+    },
+    "turndown": {
+      "optional": true
     }
+  },
+  "devDependencies": {
+    "@mozilla/readability": "^0.6.0",
+    "playwright": "^1.63.0",
+    "turndown": "^7.2.4"
   }
 }

package/tools/context/index.js

@@ -108,30 +108,47 @@ export function indexFile(db, path) {
 // a log line into the query gets a SQL error instead of results. Quoting each
 // term makes every query a literal phrase search, which is what a caller
 // searching a log actually means.
-export function toMatchQuery(query) {
+//
+// Space between two FTS5 terms is AND. That is right for a log line, where
+// every word is a fact about the one entry being looked for, and wrong for a
+// question: "pack numbers into a bytes object using a format string" demands
+// all nine words from a single chunk and returns nothing at all rather than
+// the page that carries seven of them. `mode: "or"` builds the same query
+// with OR, which is what `search` falls back to; bm25 then ranks by how many
+// of the terms a chunk carries and how rare each one is.
+export function toMatchQuery(query, { mode = "and" } = {}) {
   const terms = String(query)
     .split(/\s+/)
     .map((t) => t.replace(/"/g, ""))
     .filter(Boolean);
   if (!terms.length) return null;
-  return terms.map((t) => `"${t}"`).join(" ");
+  return terms.map((t) => `"${t}"`).join(mode === "or" ? " OR " : " ");
 }
 
 export function search(db, query, { limit = 5, path = null } = {}) {
-  const match = toMatchQuery(query);
-  if (!match) return [];
   const where = path
     ? "where chunks match ? and doc_id = (select id from docs where path = ?)"
     : "where chunks match ?";
-  const params = path ? [match, path] : [match];
-  const rows = db
-    .prepare(
-      `select chunks.rowid, doc_id, first_line, last_line, body,
-              (select path from docs where docs.id = chunks.doc_id) as path,
-              bm25(chunks) as score
-       from chunks ${where} order by score limit ?`,
-    )
-    .all(...params, limit);
+  const sql = db.prepare(
+    `select chunks.rowid, doc_id, first_line, last_line, body,
+            (select path from docs where docs.id = chunks.doc_id) as path,
+            bm25(chunks) as score
+     from chunks ${where} order by score limit ?`,
+  );
+  const run = (match) => {
+    const params = path ? [match, path] : [match];
+    return sql.all(...params, limit);
+  };
+  // Every term first, because a caller who typed all of them meant all of
+  // them. Any term second, so a question that no single chunk answers word
+  // for word still comes back ranked instead of empty.
+  let rows = [];
+  for (const mode of ["and", "or"]) {
+    const match = toMatchQuery(query, { mode });
+    if (!match) return [];
+    rows = run(match);
+    if (rows.length) break;
+  }
   return rows.map((r) => ({
     id: r.rowid,
     docId: r.doc_id,

package/tools/offload/index.js

@@ -142,6 +142,60 @@ export function offloadedErrorSummary(text, path, headChars = 400, tailChars = 2
   );
 }
 
+// What a payload IS decides what is worth keeping out of it.
+//
+// A head-plus-tail window keeps the two places a large payload is least likely
+// to carry its answer. A logcat's meaning is one FATAL line somewhere in the
+// middle; a build log's is its `error:` lines; a UI dump's is the handful of
+// nodes you can actually tap. The tool name is already here, so the window can
+// be chosen rather than fixed - and the lines it surfaces come with their line
+// numbers, so the caller can go straight to them in the saved file.
+//
+// Patterns only, no model: the cost of being wrong is a few extra lines in a
+// window that already exists, which is the right trade for a dependency-free
+// check that runs on every offload.
+const SIGNAL_FAMILIES = [
+  {
+    when: /logcat|crash|crashes/i,
+    label: "crash",
+    rx: /\b(FATAL|AndroidRuntime|Exception|SIGABRT|SIGSEGV|SIGILL|Thread \d+ Crashed|Abort trap)\b/,
+  },
+  {
+    when: /xcodebuild|xcresult|build|test/i,
+    label: "failure",
+    rx: /(^|\s)(error:|FAILED|failed:|XCTAssert\w*|Undefined symbol|linker command failed|\*\* BUILD FAILED)/,
+  },
+  {
+    when: /ui_tree|accessibility|inspect/i,
+    label: "interactive",
+    rx: /\b(AXButton|AXTextField|AXSwitch|clickable=true|enabled=true|focusable=true)\b/,
+  },
+  { when: /leaks|meminfo|memory/i, label: "measurement", rx: /\b(Leak|leaked|TOTAL|Total PSS|bytes)\b/ },
+];
+
+export const SIGNAL_LINE_CAP = 20;
+const SIGNAL_LINE_CHARS = 200;
+
+// Lines from the part the window hides, that carry what this kind of payload is
+// read for. Returns "" when the family is unknown or nothing matched, so the
+// window is exactly what it was before.
+export function signalLines(tool, lines, headCount, tailCount, cap = SIGNAL_LINE_CAP) {
+  const family = SIGNAL_FAMILIES.find((f) => f.when.test(String(tool)));
+  if (!family) return "";
+  const from = headCount;
+  const to = Math.max(headCount, lines.length - tailCount);
+  const hits = [];
+  for (let i = from; i < to && hits.length < cap; i++) {
+    if (family.rx.test(lines[i])) {
+      const body = lines[i].length > SIGNAL_LINE_CHARS ? `${lines[i].slice(0, SIGNAL_LINE_CHARS)}...` : lines[i];
+      hits.push(`  ${i + 1}: ${body}`);
+    }
+  }
+  if (!hits.length) return "";
+  const more = hits.length === cap ? ` (first ${cap})` : "";
+  return `\n\n[${family.label} lines from the hidden middle${more}]\n${hits.join("\n")}`;
+}
+
 export function offloadLargeText(tool, text, opts = {}) {
   const dir = opts.dir ?? OFFLOAD_DIR;
   const minChars = opts.minChars ?? OFFLOAD_MIN_CHARS;
@@ -166,13 +220,15 @@ export function offloadLargeText(tool, text, opts = {}) {
   const head = lines.slice(0, HEAD_LINES).join("\n");
   const tail = lines.slice(-TAIL_LINES).join("\n");
   const hidden = Math.max(0, lines.length - HEAD_LINES - TAIL_LINES);
+  const signal = signalLines(tool, lines, HEAD_LINES, TAIL_LINES);
   return {
     offloaded: true,
     path,
     text:
       `${head}\n\n[... ${hidden} line(s) not shown. Full output (${lines.length} lines, ` +
       `${text.length} bytes) saved to ${path} - search it with ` +
-      `agent_query_output {pattern: "..."} instead of re-running this tool ...]\n\n${tail}`,
+      `agent_query_output {pattern: "..."} instead of re-running this tool ...]` +
+      `${signal}\n\n${tail}`,
   };
 }