poltertab 1.3.0 → 1.4.0

package/README.md

@@ -86,10 +86,11 @@ Running from a clone instead of npm? Point the client at the file directly:
 
 ## Supported MCP Tools
 
-The server exposes 21 tools to the LLM (all prefixed with `browser_`). Every
+The server exposes 23 tools to the LLM (all prefixed with `browser_`). Every
 page-facing tool also takes optional `tabId` and `session` to target a specific
-tab — omit both and PolterTab uses the last tab it navigated, falling back to
-your active tab.
+tab — omit both and PolterTab uses the last tab it navigated. The first
+`browser_navigate` of a session opens its own tab rather than commandeering the
+one you are looking at.
 
 ### Page actions
 
@@ -104,15 +105,77 @@ your active tab.
 
 ### Reading the page
 
+Every tool here takes `output_file`, which writes the payload to
+`~/.poltertab/downloads/` and returns only a summary — row count, field names,
+fill rates and two sample records. Use it for anything bigger than a glance; the
+raw payload should never be the default path into a context window.
+
 | Action | Parameters | Description |
 | -------- | ----------- | ------------- |
-| `browser_scrape` | `selector`, `attribute`, `multiple` | Scrape given elements, or the whole page (title, meta, links, headings, body text) |
-| `browser_snapshot` | — | Ref-tagged tree of visible and interactive nodes, capped at 400 |
-| `browser_get_text` | `selector` (required) | Text content of one element |
+| `browser_extract` | `record`, `fields` (required), `anchor`, `max_text`, `probe`, `output_file` | Repeating records with fields grouped per record — see below |
+| `browser_extract_all` | `url_template`, `record`, `fields` (required), `key`, `limit`, `offset`, `start_page`, `max_pages`, `fill_tolerance`, `output_file` | Paginate and extract in one call, no model round-trip per page |
+| `browser_scrape` | `selector`, `attribute`, `multiple`, `fields`, `max_text`, `output_file` | Scrape given elements, or the whole page. `attribute` accepts `text` as well as any attribute name; `href`/`src` come back absolute. `fields` picks which parts of a full-page scrape to return — `["meta","jsonld"]` gets og: tags and schema.org blobs without 50KB of body text |
+| `browser_snapshot` | `interactive_only`, `max_nodes`, `max_depth`, `output_file` | Ref-tagged tree of visible and interactive nodes, capped at 400 |
+| `browser_get_text` | `selector` (required), `max_text`, `output_file` | Text content of one element. Flags it in the result when text was cut |
 | `browser_get_title` | — | Page title and URL |
 | `browser_get_url` | — | Page URL and title |
 | `browser_screenshot` | — | Visible tab as base64 PNG (focuses the tab) |
 
+### Extracting records
+
+`browser_scrape` returns a flat list, which loses track of which value belongs
+to which record. That is fine for one element and dangerous for ninety: the
+moment a field is optional, a flat scrape of names against phone numbers shifts
+every later phone up a row and mis-assigns the rest of the page.
+
+`browser_extract` takes a record container and resolves each field *inside* it:
+
+```json
+{
+  "record": ".agent-card",
+  "fields": {
+    "name":    { "sel": "a.agent-card-name", "get": "text" },
+    "url":     { "sel": "a.agent-card-name", "get": "href" },
+    "phone":   { "sel": "a[href^='tel:']",   "get": "href", "strip": "tel:" },
+    "socials": { "sel": "a.social-button",   "get": "href", "many": true }
+  },
+  "anchor": "url",
+  "output_file": "agents.csv"
+}
+```
+
+- **A missing field yields `null`.** It never shifts.
+- `sel` is relative to the record root; omit it (or use `"."`) for the root itself.
+- `get` is `text` (default), `href`, `src`, or any attribute name.
+- `anchor` names the field a real record always has. Records missing it — the
+  placeholder cards that pad the last page of a listing — are dropped and
+  counted in `dropped` rather than emitted as rows of nulls.
+
+The result reports what happened, so an empty column cannot pass for an answer:
+
+```
+{ "count": 12, "dropped": 0,
+  "fill_rates": { "name": 12, "phone": 10, "socials": 0 },
+  "warnings": ["socials: 0/12 within record scope, but 40 matches page-wide
+                for a.social-button — record boundary likely too narrow"] }
+```
+
+That warning is the loosening probe. Any field that comes back entirely empty is
+re-checked page-wide, because the container that *looks* like the record often
+excludes a sibling holding some of the fields — and "0 socials" reads as "these
+people have no social accounts" rather than "your selector scope is wrong".
+
+**`browser_extract_all`** takes the same spec plus a `url_template` containing
+`{page}`, and runs the loop itself. It dedups on `key` and halts on the first of:
+`limit_reached`, `empty_page`, `duplicate_page` (a page whose records repeat an
+earlier one — what a site does when it ignores an unrecognised page-size
+param), `fill_rate_deviation` (a page whose fill rates collapse against the
+first page's baseline, which is how a spec learned on page 1 quietly degrades on
+page 10), or `max_pages`. It always reports which condition fired and returns
+everything collected up to that point, so a halt never costs you the run.
+
+`.csv` and `.jsonl` output files are written in those formats.
+
 ### Network capture
 
 For sites whose DOM is obfuscated or virtualized, read the JSON driving the UI
@@ -271,7 +334,7 @@ Nothing the server accumulates is stored inside the package:
 ```
 ~/.poltertab/
 ├── navigation_memory/   # Per-domain obstacles and fixes (browser_save_site_memory)
-└── downloads/           # output_file payloads from browser_get_network_state
+└── downloads/           # output_file payloads from any read tool
 ```
 
 This matters for a global npm install. The package sits in `node_modules/`, so

package/mcp-server/index.js

@@ -654,19 +654,139 @@ const BROWSER_TOOLS = [
   },
   {
     name: "browser_scrape",
-    description: "Scrape the page or specific elements",
+    description:
+      "Scrape the page or specific elements. For repeating records (cards, rows, listings) use browser_extract instead — it keeps fields grouped per record.",
     inputSchema: {
       type: "object",
       properties: {
         selector: { type: "string" },
-        attribute: { type: "string" },
+        attribute: {
+          type: "string",
+          description:
+            "Attribute to read, or 'text' for the element's text. 'href'/'src' come back absolute.",
+        },
         multiple: { type: "boolean" },
+        fields: {
+          type: "array",
+          items: {
+            type: "string",
+            enum: ["meta", "jsonld", "links", "headings", "bodyText"],
+          },
+          description:
+            "Full-page scrape only (no selector): which parts to return. title and url are always included. ['meta','jsonld'] is the cheap structured-data path — og: tags and schema.org blobs without 50KB of body text. Omit for everything.",
+        },
+        max_text: {
+          type: "number",
+          description: "Max characters of text per element (default 500)",
+        },
+        output_file: {
+          type: "string",
+          description:
+            "Filename to write the payload under ~/.poltertab/downloads/, returning only a summary. A path is reduced to its basename — output cannot be written elsewhere.",
+        },
         tabId: { type: "number" },
         session: { type: "string" },
       },
       required: [],
     },
   },
+  {
+    name: "browser_extract",
+    description:
+      "Extract repeating records (cards, rows, listings) with fields grouped per record. Fields resolve INSIDE each record root and a missing field yields null instead of shifting later records' values. Returns fill rates and warns when a field is empty inside the record scope but matches page-wide (record boundary too narrow).",
+    inputSchema: {
+      type: "object",
+      properties: {
+        record: {
+          type: "string",
+          description:
+            "Selector for the repeating container, e.g. '.agent-card'. Verify with browser_snapshot that it encloses ALL the fields you want — the element that looks like the card often excludes siblings holding some of them.",
+        },
+        fields: {
+          type: "object",
+          description:
+            "Map of field name -> {sel, get, many, strip}. sel is relative to the record root; omit it (or use '.') for the root itself. get: 'text' (default) | 'href' | 'src' | any attribute name. many: true collects all matches into an array. strip removes a leading prefix, e.g. 'tel:'.",
+          additionalProperties: {
+            type: "object",
+            properties: {
+              sel: { type: "string" },
+              get: { type: "string" },
+              many: { type: "boolean" },
+              strip: { type: "string" },
+            },
+          },
+        },
+        anchor: {
+          type: "string",
+          description:
+            "Name of the field that is always present on a real record (usually the detail-page link). Records missing it are dropped as placeholders and counted in 'dropped'.",
+        },
+        max_text: {
+          type: "number",
+          description: "Max characters per text field (default 500)",
+        },
+        probe: {
+          type: "boolean",
+          description:
+            "Page-wide re-check of any field that came back entirely empty (default true)",
+        },
+        output_file: {
+          type: "string",
+          description:
+            "Filename to write rows under ~/.poltertab/downloads/, returning only a summary. .jsonl and .csv are written in those formats, anything else as JSON. A path is reduced to its basename — output cannot be written elsewhere.",
+        },
+        tabId: { type: "number" },
+        session: { type: "string" },
+      },
+      required: ["record", "fields"],
+    },
+  },
+  {
+    name: "browser_extract_all",
+    description:
+      "Paginate and extract in one call, with no model round-trip per page. Takes browser_extract's spec plus a URL template, walks pages, dedups on a key, and halts on: limit reached, empty page, a page whose records repeat an earlier page's (the trap where ignored page-size params silently return page 1 again), fill rates collapsing against page 1's baseline, or max_pages. Always reports which condition fired and returns everything collected so far.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        url_template: {
+          type: "string",
+          description:
+            "URL with a {page} placeholder, e.g. 'https://www.kw.com/agents?page={page}'",
+        },
+        record: { type: "string" },
+        fields: { type: "object" },
+        anchor: { type: "string" },
+        key: {
+          type: "string",
+          description:
+            "Field name to dedup on — use a stable per-record identifier such as the detail URL. Falls back to whole-row equality.",
+        },
+        limit: {
+          type: "number",
+          description: "Stop after this many records (default 200)",
+        },
+        offset: {
+          type: "number",
+          description:
+            "Skip this many records from the start of the stream. Pages are re-fetched to reach the offset; pass start_page to skip cheaply.",
+        },
+        start_page: { type: "number", description: "First page (default 1)" },
+        max_pages: {
+          type: "number",
+          description: "Hard guard on pages fetched (default 50)",
+        },
+        fill_tolerance: {
+          type: "number",
+          description:
+            "Halt when a field well-populated on the baseline page falls below this fraction of it (default 0.5). 0 disables the check.",
+        },
+        max_text: { type: "number" },
+        output_file: { type: "string" },
+        session: { type: "string" },
+      },
+      required: ["url_template", "record", "fields"],
+    },
+  },
   {
     name: "browser_screenshot",
     description:
@@ -718,6 +838,12 @@ const BROWSER_TOOLS = [
       type: "object",
       properties: {
         selector: { type: "string" },
+        max_text: {
+          type: "number",
+          description:
+            "Max characters (default 10000). The result flags it when text was cut.",
+        },
+        output_file: { type: "string" },
         tabId: { type: "number" },
         session: { type: "string" },
       },
@@ -750,10 +876,21 @@ const BROWSER_TOOLS = [
   },
   {
     name: "browser_snapshot",
-    description: "Get a snapshot of the DOM",
+    description:
+      "Get a snapshot of the DOM. Large pages run to tens of KB — narrow it with interactive_only/max_nodes, or send it to output_file, before spending the context on it.",
     inputSchema: {
       type: "object",
       properties: {
+        interactive_only: {
+          type: "boolean",
+          description: "Only clickable/typable elements",
+        },
+        max_nodes: {
+          type: "number",
+          description: "Cap on nodes returned (default 400)",
+        },
+        max_depth: { type: "number" },
+        output_file: { type: "string" },
         tabId: { type: "number" },
         session: { type: "string" },
       },
@@ -910,6 +1047,281 @@ server.setRequestHandler(ListToolsRequestSchema, async () => {
   };
 });
 
+// --- Output handling ---
+//
+// The raw payload must never be the default path into the context window. Two
+// costs dominated a 100-record scrape and neither was the data: envelopes
+// several lines of JSON deep to carry one short string, and the agent then
+// re-typing every record by hand into a file. Writing straight to disk removes
+// both.
+
+function toCsv(rows) {
+  if (!rows.length) return "";
+  const cols = [...new Set(rows.flatMap((r) => Object.keys(r)))];
+  const cell = (v) => {
+    if (v === null || v === undefined) return "";
+    const s = Array.isArray(v)
+      ? v.join(" | ")
+      : typeof v === "object"
+        ? JSON.stringify(v) // beats a column of "[object Object]"
+        : String(v);
+    // Quote only when it would otherwise break the row, and double any
+    // embedded quote — the two mistakes that produce a file which imports
+    // silently and wrongly.
+    return /[",\r\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;
+  };
+  return [
+    cols.join(","),
+    ...rows.map((r) => cols.map((c) => cell(r[c])).join(",")),
+  ].join("\n");
+}
+
+// Site memory is keyed by hostname, and that key arrives from a model — so it
+// is untrusted input rather than a filename. Two failures this closes: a note
+// saved under kw.com was invisible to a lookup for www.kw.com (the same site),
+// and the raw value was interpolated straight into a path, so "../.." reached
+// outside MEMORY_DIR.
+function memoryFile(rawHost) {
+  let host = String(rawHost).trim().toLowerCase();
+
+  // The parameter is also documented as accepting `url`, so a full URL turning
+  // up here is expected rather than a caller mistake.
+  if (host.includes("/")) {
+    try {
+      host = new URL(host.includes("://") ? host : `https://${host}`).hostname;
+    } catch {
+      host = host.split("/")[0];
+    }
+  }
+
+  host = host.replace(/[^a-z0-9.-]/g, "").replace(/^\.+/, "");
+  if (!host) throw new Error(`Not a usable hostname: ${rawHost}`);
+
+  // Existing notes live under whichever spelling first created them — the store
+  // already holds both kw.com.json and www.linkedin.com.json — so try the
+  // variants before concluding this is a new file.
+  const bare = host.replace(/^www\./, "");
+  for (const name of [bare, host, `www.${bare}`]) {
+    const p = path.join(MEMORY_DIR, `${name}.json`);
+    if (fs.existsSync(p)) return p;
+  }
+  return path.join(MEMORY_DIR, `${bare}.json`);
+}
+
+const isBlank = (v) =>
+  v === null || v === undefined || v === "" || (Array.isArray(v) && !v.length);
+
+// `rows` is passed separately when the payload has a records array: .jsonl and
+// .csv are formats for records, not for envelopes.
+function writeOutput(name, payload, rows) {
+  const requested = String(name);
+  const safeName = path.basename(requested);
+  const parts = safeName.split(".");
+  const ext = parts.length > 1 ? `.${parts.pop()}` : "";
+  const base = parts.join(".");
+  const finalName = `${base}_${Date.now()}${ext}`;
+
+  fs.mkdirSync(DOWNLOADS_DIR, { recursive: true });
+  const safePath = path.join(DOWNLOADS_DIR, finalName);
+
+  let body;
+  if (rows && ext === ".jsonl") {
+    body = rows.map((r) => JSON.stringify(r)).join("\n");
+  } else if (rows && ext === ".csv") {
+    body = toCsv(rows);
+  } else {
+    body = JSON.stringify(payload, null, 2);
+  }
+
+  fs.writeFileSync(safePath, body);
+  const out = { file: safePath, bytes: Buffer.byteLength(body) };
+
+  // Output stays inside DOWNLOADS_DIR: this path comes from a model, and a tool
+  // that writes to an arbitrary absolute path is a different capability than
+  // one that saves a scrape. But relocating without a word is how a caller ends
+  // up looking for a file that was never going to be there.
+  if (safeName !== requested) {
+    out.note = `output_file is confined to ${DOWNLOADS_DIR} — "${requested}" was written as ${path.basename(safePath)}, not to the path requested.`;
+  }
+  return out;
+}
+
+// What comes back inline when the payload went to disk: enough to know the call
+// worked and the shape is right, and nothing more.
+function summarizeOutput(payload, rows, written) {
+  const summary = { ...written };
+  if (rows) {
+    summary.rows = rows.length;
+    summary.fields = rows.length ? Object.keys(rows[0]) : [];
+    summary.sample = rows.slice(0, 2);
+    for (const k of ["fill_rates", "dropped", "warnings", "stopped_because", "pages_fetched"]) {
+      if (payload && payload[k] !== undefined) summary[k] = payload[k];
+    }
+  } else if (Array.isArray(payload)) {
+    summary.items = payload.length;
+    summary.sample = payload.slice(0, 2);
+  } else if (payload && typeof payload === "object") {
+    summary.keys = Object.keys(payload);
+  }
+  return summary;
+}
+
+// Payload shapes that carry a records array worth writing as jsonl/csv.
+function rowsOf(payload) {
+  if (payload && Array.isArray(payload.rows)) return payload.rows;
+  if (Array.isArray(payload)) return payload;
+  return null;
+}
+
+// --- The pagination loop ---
+//
+// This exists so the model stops being the for-loop. It supplies the spec once
+// and receives records; every page in between costs a browser round-trip
+// instead of a model round-trip.
+//
+// Every halt condition below is a case where continuing would produce data
+// that looks complete and is not, so each one names itself in the result. A
+// paused run costs far less than a confidently wrong dataset.
+async function extractAll(args) {
+  const {
+    url_template,
+    record,
+    fields,
+    anchor,
+    key,
+    limit = 200,
+    offset = 0,
+    start_page = 1,
+    max_pages = 50,
+    fill_tolerance = 0.5,
+    max_text,
+    session,
+  } = args;
+
+  if (!url_template.includes("{page}")) {
+    throw new Error("url_template must contain a {page} placeholder");
+  }
+
+  const spec = { record, fields, anchor, max_text, session, probe: true };
+  const target = offset + limit;
+  const rows = [];
+  const seen = new Set();
+  const warnings = [];
+  const pages = [];
+
+  let baseline = null;
+  let stopped_because = "max_pages";
+  let page = start_page;
+  let fetched = 0;
+
+  const keyOf = (row) =>
+    key ? String(row[key] ?? "") : JSON.stringify(row);
+
+  while (fetched < max_pages) {
+    const url = url_template.replace("{page}", String(page));
+    const nav = await sendCommand("navigate", { url, session });
+    fetched++;
+
+    const res = await sendCommand("extract", spec);
+    const pageRows = (res && res.rows) || [];
+    pages.push({
+      page,
+      url: nav && nav.url,
+      found: res ? res.records_found : 0,
+      kept: pageRows.length,
+      dropped: res ? res.dropped : 0,
+    });
+    if (res && res.warnings && res.warnings.length) {
+      warnings.push(`page ${page}: ${res.warnings.join("; ")}`);
+    }
+
+    if (!pageRows.length) {
+      stopped_because = "empty_page";
+      break;
+    }
+
+    // A site that ignores an unrecognised page param answers every request with
+    // page 1. Identical content reads as real data, which is how a "next 100"
+    // silently becomes the same 12 records eight times over.
+    const fresh = pageRows.filter((r) => !seen.has(keyOf(r)));
+    if (!fresh.length) {
+      stopped_because = "duplicate_page";
+      warnings.push(
+        `page ${page} returned only records already seen — pagination is not advancing`,
+      );
+      break;
+    }
+
+    // Fill rates against the first page. A spec learned on page 1 degrades
+    // quietly later: variant card layouts, a column that stops being populated.
+    // Halting beats emitting rows that are 40% empty.
+    const ratios = {};
+    for (const [name, n] of Object.entries(res.fill_rates || {})) {
+      ratios[name] = n / pageRows.length;
+    }
+    if (!baseline) {
+      baseline = ratios;
+    } else if (fill_tolerance > 0) {
+      const collapsed = Object.keys(baseline).filter(
+        (name) =>
+          baseline[name] >= 0.5 &&
+          ratios[name] < baseline[name] * fill_tolerance,
+      );
+      if (collapsed.length) {
+        stopped_because = "fill_rate_deviation";
+        warnings.push(
+          `page ${page}: ${collapsed
+            .map(
+              (n) =>
+                `${n} ${(ratios[n] * 100).toFixed(0)}% vs baseline ${(baseline[n] * 100).toFixed(0)}%`,
+            )
+            .join(", ")} — page layout likely differs from the learned spec`,
+        );
+        break;
+      }
+    }
+
+    for (const r of fresh) {
+      seen.add(keyOf(r));
+      rows.push(r);
+    }
+
+    if (rows.length >= target) {
+      stopped_because = "limit_reached";
+      break;
+    }
+    page++;
+  }
+
+  const sliced = rows.slice(offset, offset + limit);
+  if (offset && fetched) {
+    warnings.push(
+      `offset ${offset} was reached by fetching from page ${start_page}; pass start_page to skip pages instead of re-reading them`,
+    );
+  }
+
+  // extract reports fill as counts, so extract_all does too — the per-page
+  // baseline is a fraction because it is compared across pages of differing
+  // size, and carries the unit in its name rather than looking like a count.
+  const fill_rates = {};
+  for (const name of Object.keys(fields)) {
+    fill_rates[name] = sliced.filter((r) => !isBlank(r[name])).length;
+  }
+
+  return {
+    rows: sliced,
+    count: sliced.length,
+    collected: rows.length,
+    pages_fetched: fetched,
+    last_page: page,
+    stopped_because,
+    fill_rates,
+    baseline_fill_ratios: baseline,
+    pages,
+    warnings,
+  };
+}
+
 const handleToolCall = async (request) => {
   const { name, arguments: args } = request.params;
 
@@ -951,24 +1363,12 @@ const handleToolCall = async (request) => {
       // payload would flood the very context window this parameter exists to
       // protect.
       if (opts.output_file) {
-        // Sanitize the file name to prevent Path Traversal
-        const safeName = path.basename(opts.output_file);
-        // Add timestamp to prevent collisions
-        const parts = safeName.split(".");
-        const ext = parts.length > 1 ? `.${parts.pop()}` : "";
-        const base = parts.join(".");
-        const finalName = `${base}_${Date.now()}${ext}`;
-
-        fs.mkdirSync(DOWNLOADS_DIR, { recursive: true });
-
-        const safePath = path.join(DOWNLOADS_DIR, finalName);
-        fs.writeFileSync(safePath, JSON.stringify(responsePayload, null, 2));
-
+        const written = writeOutput(opts.output_file, responsePayload);
         return {
           content: [
             {
               type: "text",
-              text: `Data successfully written to ${safePath}. Captured ${responsePayload.capturedRequests} requests.`,
+              text: `Data successfully written to ${written.file}. Captured ${responsePayload.capturedRequests} requests.`,
             },
           ],
         };
@@ -1029,7 +1429,7 @@ const handleToolCall = async (request) => {
     if (action === "get_site_memory") {
       const host = args.hostname || args.domain || args.url;
       if (!host) throw new Error("Missing 'hostname' parameter");
-      const file = path.join(MEMORY_DIR, `${host}.json`);
+      const file = memoryFile(host);
       let data = [];
       if (fs.existsSync(file)) {
         data = JSON.parse(fs.readFileSync(file, "utf8"));
@@ -1042,7 +1442,7 @@ const handleToolCall = async (request) => {
     if (action === "save_site_memory") {
       const host = args.hostname || args.domain || args.url;
       if (!host) throw new Error("Missing 'hostname' parameter");
-      const file = path.join(MEMORY_DIR, `${host}.json`);
+      const file = memoryFile(host);
       let data = [];
       if (fs.existsSync(file)) {
         data = JSON.parse(fs.readFileSync(file, "utf8"));
@@ -1058,6 +1458,32 @@ const handleToolCall = async (request) => {
       };
     }
 
+    // Loops in the server, not in the model. One tool call covers every page.
+    if (action === "extract_all") {
+      const payload = await extractAll(args || {});
+      const opts = args || {};
+
+      if (opts.output_file) {
+        const written = writeOutput(opts.output_file, payload, payload.rows);
+        const { rows, pages, ...rest } = payload;
+        const summary = {
+          ...rest,
+          ...written,
+          fields: rows.length ? Object.keys(rows[0]) : [],
+          sample: rows.slice(0, 2),
+        };
+        return {
+          content: [
+            { type: "text", text: JSON.stringify(summary, null, 2) },
+          ],
+        };
+      }
+
+      return {
+        content: [{ type: "text", text: JSON.stringify(payload, null, 2) }],
+      };
+    }
+
     const result = await sendCommand(action, args || {});
 
     if (!isSecondary) {
@@ -1071,6 +1497,25 @@ const handleToolCall = async (request) => {
       }
     }
 
+    // Any read tool can send its payload to disk. Placed after tab tracking so
+    // taking the file path does not cost the session its tab bookkeeping.
+    if (args && args.output_file && result && typeof result === "object") {
+      const rows = rowsOf(result);
+      const written = writeOutput(args.output_file, result, rows);
+      return {
+        content: [
+          {
+            type: "text",
+            text: JSON.stringify(
+              summarizeOutput(result, rows, written),
+              null,
+              2,
+            ),
+          },
+        ],
+      };
+    }
+
     // Check if error result string (graceful error handling)
     if (
       typeof result === "string" &&

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "poltertab",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "Let any AI assistant drive and scrape your existing Chrome profile — no headless browser, no complex setup.",
   "bin": {
     "poltertab": "./bin/poltertab.js"

package/skills/browser-navigation-strategy/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: browser-navigation-strategy
-description: Use when driving a real browser with PolterTab's browser_* tools — navigating, scraping, or clicking through a site whose DOM is virtualized, obfuscated, shadow-rooted, or iframe-sandboxed, and selectors keep going stale.
+description: Use when driving a real browser with PolterTab's browser_* tools — extracting repeating records, paginating a listing, navigating, scraping, or clicking through a site whose DOM is virtualized, obfuscated, shadow-rooted, or iframe-sandboxed, and selectors keep going stale.
 ---
 
 # Browser Navigation Strategy
@@ -9,49 +9,140 @@ PolterTab drives the user's real Chrome profile. Sessions are already logged in,
 and every action happens in a window the user can see. That makes it powerful and
 unforgiving: a wrong click lands on a real account.
 
-## Read the page before you touch it
+## Pick the right tool before you start
 
-`browser_snapshot` is the first call on any unfamiliar page. It returns visible,
-interactive elements with `@e1`-style refs, and it descends into shadow roots and
-child frames — so what it shows you is the page as it really is, not just the
-top-level light DOM.
+| What you want | Use |
+| --- | --- |
+| Many similar records on a page (cards, rows, listings) | `browser_extract` |
+| Those records across many pages | `browser_extract_all` |
+| One element's text | `browser_get_text` |
+| Page title, og: tags, schema.org data | `browser_scrape` with `fields` |
+| To find out what a page contains | `browser_snapshot` |
+| Data the DOM does not hold (virtualized/obfuscated) | intercept + `browser_get_network_state` |
 
-Never guess a selector from a URL or a screenshot. Snapshot, then act on what
-came back.
+Getting this wrong is the most expensive mistake available. Scraping 100 records
+one field at a time costs a round-trip per page and leaves you reassembling
+records by hand; `browser_extract_all` is one call.
 
-## Prefer the network over the DOM
+## Repeating records: use browser_extract
 
-On a virtualized list (LinkedIn, most infinite feeds), the DOM holds only the
-rows currently on screen, under class names that change between deploys. Scraping
-it yields a fraction of the data and breaks next week.
+A flat scrape returns a list per selector, so you have to work out which value
+belongs to which record. That works until a field is optional — then one missing
+value shifts every later record and the result looks completely plausible.
+
+`browser_extract` takes a container and resolves each field *inside* it:
+
+```json
+{
+  "record": ".agent-card",
+  "fields": {
+    "name":    { "sel": "a.agent-card-name", "get": "text" },
+    "url":     { "sel": "a.agent-card-name", "get": "href" },
+    "phone":   { "sel": "a[href^='tel:']",   "get": "href", "strip": "tel:" },
+    "socials": { "sel": "a.social-button",   "get": "href", "many": true }
+  },
+  "anchor": "url",
+  "output_file": "agents.csv"
+}
+```
+
+- A missing field is `null`. It never shifts.
+- `sel` is relative to the record; omit it (or `"."`) for the record itself.
+- `get`: `text` (default), `href`, `src`, or any attribute name. `href`/`src`
+  come back absolute.
+- `anchor` names the field every real record has. Records without it — the
+  placeholder cards padding a last page — are dropped and counted.
+
+**Read the response before trusting it.** `fill_rates` and `warnings` tell you
+whether the spec is right:
+
+```
+fill_rates: { name: 12, phone: 10, socials: 0 }
+warnings: ["socials: 0/12 within record scope, but 40 matches page-wide
+            for a.social-button — record boundary likely too narrow"]
+```
+
+That warning means the container you picked excludes a sibling holding the
+field. Widen `record` and re-run. Without it, an empty column reads as "these
+records have no socials" — confidently wrong.
+
+A field at 0 with *no* page-wide matches means the selector is wrong, not the
+boundary. The message distinguishes the two; act on which one you got.
+
+## Paginating: use browser_extract_all
+
+Same spec plus a `url_template` containing `{page}`. It runs the loop itself, so
+pages cost a browser round-trip instead of a model round-trip.
+
+Dedup on `key` — a stable per-record id, the detail URL or an element `id`, never
+position. It halts on the first of `limit_reached`, `empty_page`,
+`duplicate_page`, `fill_rate_deviation`, or `max_pages`, and always says which.
+
+Two of those are worth understanding:
+
+- **`duplicate_page`** — the page returned records already seen. This is what a
+  site does with a param it does not recognise: it serves page 1 again. Without
+  the check you would collect the same 12 records eight times.
+- **`fill_rate_deviation`** — a field well-populated on page 1 collapsed on a
+  later page, so the learned spec no longer fits. Everything collected so far is
+  returned; re-run from `start_page` with a spec covering both layouts. Set
+  `fill_tolerance: 0` if a site is legitimately heterogeneous.
+
+## Take the free structured data first
 
-The reliable path:
+Before writing any selector, try:
+
+```json
+{ "fields": ["meta", "jsonld"] }
+```
+
+on `browser_scrape`. `title`, every `og:`/`twitter:` tag, and any schema.org
+`ld+json` blob come back with no selectors at all. Real estate, jobs, products,
+events and profile pages very often ship a complete typed record this way —
+name, telephone, email, postal address, employer — which beats any selector you
+could write and does not break on redesign.
+
+This is the cheapest call available on a detail page. Try it before building an
+enrichment scrape.
+
+## Prefer the network over the DOM — when the DOM is the problem
+
+On a virtualized list (LinkedIn, most infinite feeds), the DOM holds only the
+rows on screen, under class names that change between deploys. There, the
+network is the reliable path:
 
 1. `browser_set_intercept_patterns` if the site uses endpoints outside the
    defaults (`graphql`, `/api/`, `voyager`, `feed`).
 2. `browser_smart_scroll` to trigger lazy loading. It yields until new requests
    land, so you do not have to guess a wait.
-3. `browser_get_network_state` to read the structured JSON the page itself
-   received.
+3. `browser_get_network_state` to read the JSON the page itself received.
+
+**But check first that there is a network path at all.** Plenty of sites are
+server-rendered: the records ship in the initial HTML and no XHR ever fires.
+Chasing the network there costs several calls and returns an empty buffer that
+looks identical to "you set the wrong pattern". If `browser_extract` already
+returns full records, or `fields: ["jsonld"]` has the data, the page is
+server-rendered — use the DOM and move on.
 
-Pass `output_file` on `browser_get_network_state` for anything large. The payload
-goes to disk and you get a summary — a 4 MB GraphQL response does not belong in
-the context window.
+The default intercept patterns include `/api/`, which matches analytics and maps
+traffic on most sites. An almost-empty-looking buffer full of Google requests is
+that, not your data.
 
 ## Shadow DOM, iframes, and late modals
 
 PolterTab handles these without special flags, which is worth knowing so you do
 not work around problems it already solves:
 
-- **Shadow roots** — including closed ones — are pierced automatically. A
-  selector that matches inside a web component just works.
+- **Shadow roots** — including closed ones — are pierced automatically, in
+  `browser_extract`'s record scope too.
 - **Child frames** are searched in parallel when the top frame misses, so an
   element inside a sandboxed iframe is reachable by plain selector.
 - **Late-rendering elements** get retried for ~3s. A modal that mounts a beat
   after the click that opened it does not need a manual wait.
 
-If an element genuinely is not there, you get `Element not found` after the
-retry window. Treat that as "not on this page", not "try again immediately".
+`Element not found` after that window means "not on this page", not "try again".
+`browser_extract` does not throw for this — it returns `records_found: 0` with
+the warnings above, which tell you more than an error would.
 
 ## Selectors, in order of durability
 
@@ -63,32 +154,69 @@ retry window. Treat that as "not on this page", not "try again immediately".
 Long generated class chains (`.css-1x2y3z4`) are the worst of both worlds:
 brittle and unreadable. Reach for a `@e` ref from the last snapshot instead.
 
+## Keep payloads out of the context window
+
+Every read tool takes `output_file`. The payload goes to
+`~/.poltertab/downloads/` and you get back row count, field names, fill rates
+and two sample records. Use it for anything bigger than a glance.
+
+`.csv` and `.jsonl` are written in those formats, so the file is the deliverable
+— never re-type extracted records into a file by hand, which costs more than the
+extraction did.
+
+`browser_snapshot` on a real page runs to tens of KB. Narrow it with
+`interactive_only` or `max_nodes`, or send it to `output_file`.
+
+Text fields are cut at `max_text` (default 500) and the result says when it
+happened. Raise it rather than assuming you have the whole value.
+
+## Known limits — do not fight these
+
+- **Records spanning sibling elements.** `browser_extract` needs one container
+  per record. A table where an item's fields live in two sibling `<tr>`s (Hacker
+  News is the canonical case) has no such container. Extract twice and join on a
+  shared id — never on position.
+- **Pagination that is not a URL parameter.** `browser_extract_all` only drives
+  `{page}` in a URL. Infinite scroll, a "Load more" button, and cursor tokens
+  need `browser_smart_scroll` or clicking, in a loop you run yourself.
+- **Virtualized lists.** `browser_extract` reads the DOM, so it sees only the
+  rendered window. Use the network path.
+- **`output_file` writes only to `~/.poltertab/downloads/`.** A path is reduced
+  to its basename; the returned `file` is where it actually went.
+
 ## Carry knowledge between sessions
 
 `browser_get_site_memory(hostname)` at the start of work on a site, and
-`browser_save_site_memory` the moment you solve something non-obvious — a cookie
-wall that intercepts the first click, a required query parameter, the shape of a
-GraphQL response worth reusing.
+`browser_save_site_memory` the moment you solve something non-obvious. Save the
+record root, the field spec that worked, the pagination parameter, and any
+obstacle with its fix. `www.` is normalized and a full URL is accepted, so
+lookups do not miss.
 
-The point is to not rediscover the same obstacle next week. Save the obstacle and
-the fix together; a note saying "this page is tricky" helps nobody.
+A note saying "this page is tricky" helps nobody. Save what you would need to
+skip the discovery entirely next time.
 
 ## Multiple tabs
 
 `browser_session_create` gives a named tab you can return to. Every page-facing
-tool takes `session` (or `tabId`) to target it. Omit both and PolterTab uses the
-last tab it navigated, falling back to the active tab — fine for single-track
-work, ambiguous the moment two flows interleave. Name your sessions when running
-more than one.
+tool takes `session` (or `tabId`).
 
-Close what you opened with `browser_session_close`.
+Omit both and the first `browser_navigate` opens its own tab — it does not
+commandeer the tab the user is looking at — and later calls reuse it. That is
+fine for single-track work and ambiguous the moment two flows interleave, so
+name your sessions when running more than one. Close what you opened with
+`browser_session_close`.
 
 ## Pitfalls
 
+- Do not scrape repeating records field-by-field. Use `browser_extract`, and
+  read its `warnings`.
+- Do not zip flat arrays into records by position. One optional field silently
+  ruins the whole dataset.
+- Do not loop pages yourself when `url_template` covers the site.
 - Do not use `browser_get_text` or `browser_scrape` on virtualized lists. Use
   `browser_smart_scroll` then `browser_get_network_state`.
-- Do not invent intercept patterns. Check the defaults, then set only what the
-  site actually needs.
+- Do not invent intercept patterns, and do not assume an empty buffer means you
+  asked wrong — the site may be server-rendered.
 - Do not paste large scraped payloads into the conversation. Use `output_file`.
 - Do not retry a failed selector unchanged. The retry already happened inside
   the tool — snapshot again and find out what the page really contains.