symbiot 0.41.4 → 0.41.6

package/README.md

@@ -58,8 +58,11 @@ Tabs:
 - **Dashboard** — everything you [watch](#screens-blueprints-for-screen-automation)
   side by side: a card for each page (your inbox, GitHub, WhatsApp, any other) with
   how many new since yesterday (or in the last 3 or 7 days), its latest brief, the
-  newest items, and **Open**, **Draft a reply** and **Check now** on them. The tab
-  shows the count, so you see what's waiting from any tab.
+  newest items, and **Open**, **Draft a reply** and **Check now** on them. **Seen**
+  on a card sets its count back to 0 (what it found stays under Watching, and the
+  other cards keep theirs). The tab shows the count, so you see what's waiting from
+  any tab, and once you watch something the app opens on the Dashboard instead of
+  the Map. `symbiot watch board` prints the same cards as JSON.
 - **Drift** — the [`symbiot drift`](#whats-out-of-sync-symbiot-drift) report, with
   a "fetch latest" toggle (and an [experimental](#experimental) "check CI").
 - **Week / Standup / Todo** — the write-ups (these use your chosen AI).
@@ -108,7 +111,8 @@ repo is now archived), without Electron:
   default). The app writes your week with your chosen AI, saves it to
   `~/.config/symbiot/weeks/<date>.md`, and sends a desktop notification
   (`notify-send` on Linux, Notification Center on macOS, a tray balloon on
-  Windows). The Week tab shows it until you write a new one. It runs while
+  Windows). The Week tab shows it until you write a new one, and a week you
+  write there with its button is saved to `weeks/` the same way. It runs while
   `symbiot app` is running. A week missed while the computer was off is written
   when the app next starts, once. **Write it now** checks that it works.
 - **Start Symbiot in the background when I log in** adds an autostart entry
@@ -263,6 +267,16 @@ Tasks waiting for review aren't re-sent to the agent, and while the repo's agent
 is still running, Approve waits ("agent still working"), so half-done work isn't
 committed.
 
+**Tasks about tasks.** An agent can't edit your task list, so a task like
+`Drop the "gosolr's own WhatsApp number" task` or ``Merge the two `WA_WABA_ID`
+tasks into one`` is done once you approve it. The tasks it names then leave your
+list, so a later **Send to repos** doesn't bring them back: a dropped one is
+archived, and of merged ones the newest stays and the rest are archived. Symbiot
+finds them by the quoted name before the word "task": tasks in that repo, from
+before the approval, with all of its words ("'s" aside). If it names more than
+four, it's too vague, and nothing is removed. In the archived view they're marked
+**dropped** or **merged**, and one you restore stays.
+
 **Releases.** The review card warns about merged work that isn't released yet,
 measured the way the repo releases: from its last `v*` tag, or, for a repo whose
 GitHub workflow runs `npm publish` on every push to its default branch, from the
@@ -311,6 +325,18 @@ preset only fills in the command box; the saved command is what runs.
   isn't on your PATH). They just open the repo. Nothing comes back for review unless
   you run an agent there yourself, and Neovim needs a terminal to open in.
 
+**Connectors.** The connectors you've linked to Claude (claude.ai's Google Drive,
+Gmail, Notion…, and servers added with `claude mcp add`) are only usable by an
+unattended run when `--allowedTools` names them, so a run asked to check your
+Drive or mail used to be refused. Symbiot now adds each one's rule
+(`mcp__claude_ai_Google_Drive` covers `mcp__claude_ai_Google_Drive__search_files`
+and the rest of its tools) to a Claude command on every run. It reads them from
+`~/.claude.json`, so linking one takes effect on the next run, and your saved
+command stays as you typed it. A connector still waiting to be authorized at
+claude.ai isn't added. Settings → Handoff lists them, and says so when your
+command isn't Claude and its runs can't use them. Claude in Orca's tab asks you
+before using one.
+
 Each run is logged to `.symbiot/agent.log` in the repo and shown live in the
 **Agents** tab. The command is saved as `agentCmd` in
 `~/.config/symbiot/config.json` — it's your command, Symbiot only fills in
@@ -365,6 +391,16 @@ keep going with everything that doesn't depend on an answer, and to ask instead
 doing anything destructive. Both files live in `.symbiot/`, so they're never
 committed.
 
+**A run that stopped on questions isn't repeated for nothing.** When a run ends
+having asked questions you haven't answered, with tasks still unticked, Symbiot
+keeps what it ran with in `.symbiot/blocked.json`. The next send to that repo
+starts no agent until something it could act on changes: `.env` (or another
+`.env.*` file there), `ANSWERS.md`, the agent command (a command you allowed, a
+connector), or a task it didn't have. Otherwise the run would only ask the same
+questions again. The send says so, with **Start it anyway** for when what changed
+is somewhere else (`symbiot push --open --force` from the terminal). **Draft a
+reply** always starts its run.
+
 ## What's out of sync? `symbiot drift`
 
 ```bash
@@ -589,7 +625,13 @@ the window, so a map only has the part that fits: when there's more, the screen
 says so and **Scroll down** (and **Scroll up**) scrolls the page in the hidden
 browser and maps what's in the window then, as a new screen. It scrolls what your
 mouse wheel would: the page, or the part of it that scrolls on its own (Gmail's
-list of mail). Scrolling only looks, so it never asks first. **Press** on a region clicks it
+list of mail). Scrolling only looks, so it never asks first. Or **Whole page**
+maps all of it at once: one tall screenshot (up to 16,000 pixels) with every
+button, link and field on it marked where it is on the page. It's for a page that
+scrolls as a whole; a list that scrolls inside the page, like Gmail's, still takes
+Scroll down. The hidden browser scrolls down a window at a time and puts the
+screenshots together, so a menu bar that stays at the top shows once, at the top,
+and a sticky side menu shows in the first window only. **Press** on a region clicks it
 in that hidden browser and maps the page it leads to as a new screen, so map,
 press, map is how an agent finds its way around a site. **Type** on a field types
 your text into it there (replacing what was in it) and, with **Type, then Enter**
@@ -621,6 +663,7 @@ symbiot screens type <id> "Search mail" "invoice" --enter --yes   # type into a
 symbiot screens type <id> To "sam@example.com" --yes   # without --enter: the text stays in the field...
 symbiot screens press <new id> Send --yes  # ...for a press on the screen that just printed
 symbiot screens scroll <id>            # "more": "below"? scroll down and map the next part (or up, top, bottom)
+symbiot screens whole <id>             # all of that page in one tall screen (or: symbiot screens map <site> --whole)
 symbiot screens show <id>              # a saved screen's blueprint
 symbiot screens signin gmail           # sign in once, in Symbiot's browser window
 symbiot screens                        # list them
@@ -651,8 +694,10 @@ Your coding agent reads it too, so it can act on it with `symbiot screens`:
 symbiot watch add <screen id> --every 15   # watch a mapped page (or: symbiot watch add gmail)
 symbiot watch add github                   # your GitHub notifications (see below)
 symbiot watch new --hours 24               # what's new, newest first (JSON)
+symbiot watch board                        # the Dashboard's cards (JSON; .total is the count, for a status bar)
+symbiot watch seen <watch id>              # set a card back to 0, like its Seen button
 symbiot watch check                        # read them all now (JSON)
-symbiot watch draft <id>                   # Draft a reply to a new email (see below)
+symbiot watch draft <id>                   # Draft a reply to a new email or chat message (see below)
 symbiot watch brief on                     # your AI says what needs you (off to stop)
 symbiot watch                              # what you watch, and what's new
 symbiot watch remove <id>
@@ -689,6 +734,15 @@ know, say) is answered there. The agent sees the email's text, so it reaches
 whatever AI your agent uses. From a terminal, `symbiot watch new` marks the
 emails `"mail": true`, and `symbiot watch draft <id>` does the same as the button.
 
+A new WhatsApp message (a watched `web.whatsapp.com`) has **Draft a reply** too.
+There, the agent opens the chat in the hidden browser and types the reply into
+its message box, and leaves it there unsent. WhatsApp keeps it as that chat's
+draft in Symbiot's browser: type `web.whatsapp.com` under Screens and click
+**Sign in** to read it and send it. The same guard holds, plus one for chats: a
+draft's run can't press Enter (in a chat, Enter sends), and in WhatsApp its line
+breaks are typed as spaces. Add `web.whatsapp.com` under Trusted sites for it.
+`symbiot watch new` marks those `"chat": true`.
+
 **In Standup.** Standup ends with what's waiting on you since yesterday, counted
 from what Watch found: `Waiting on you: 3 emails, 2 GitHub notifications`. Symbiot
 counts them itself, so the numbers are right; the AI only sees them to know

package/agents.mjs

@@ -4,9 +4,9 @@
 import { spawn } from "node:child_process";
 import { homedir } from "node:os";
 import { join } from "node:path";
-import { readFileSync, writeFileSync, mkdirSync, existsSync, openSync, writeSync, unlinkSync } from "node:fs";
-import { randomBytes } from "node:crypto";
-import { loadConfig, saveConfig, loadTasks, sameTask, sh, hasCmd } from "./core.mjs";
+import { readFileSync, writeFileSync, mkdirSync, existsSync, openSync, writeSync, unlinkSync, readdirSync, statSync } from "node:fs";
+import { randomBytes, createHash } from "node:crypto";
+import { loadConfig, saveConfig, loadTasks, sameTask, taskWords, sh, hasCmd } from "./core.mjs";
 
 const HANDOFFS = []; // live registry of agents Symbiot has handed work to
 // ---- hand a repo (+ its tasks) to the user's agent — generic, settings-based
@@ -48,21 +48,50 @@ function grantRule(tool) {
   t = t.replace(/[()"'`]/g, "").replace(/:\*$/, "").trim();
   return t ? `Bash(${t}:*)` : "";
 }
+// Add a rule to a Claude command's --allowedTools (or start the flag), once.
+function allowTool(cmd, t) {
+  if (!t || cmd.includes(`"${t}"`)) return cmd;
+  if (/--allowedTools\b/.test(cmd)) return cmd.replace(/(--allowedTools\s+(?:"[^"]*"\s*)+)/, (m) => m.trimEnd() + ` "${t}" `);
+  return cmd + ` --allowedTools "${t}"`;
+}
+const isClaudeCmd = (cmd) => /^\s*claude\b/.test(String(cmd || ""));
 function grantAgent({ tool, dir } = {}) {
   const cfg = loadConfig(); let cmd = (cfg.agentCmd || CLAUDE_CMD).trim();
-  if (!/^\s*claude\b/.test(cmd)) return { error: "Grants apply to the Claude agent command. Pick a Claude preset first, or edit the command directly." };
-  if (tool) {
-    const t = grantRule(tool);
-    if (t && !cmd.includes(`"${t}"`)) {
-      if (/--allowedTools\b/.test(cmd)) cmd = cmd.replace(/(--allowedTools\s+(?:"[^"]*"\s*)+)/, (m) => m.trimEnd() + ` "${t}" `);
-      else cmd += ` --allowedTools "${t}"`;
-    }
-  }
+  if (!isClaudeCmd(cmd)) return { error: "Grants apply to the Claude agent command. Pick a Claude preset first, or edit the command directly." };
+  if (tool) cmd = allowTool(cmd, grantRule(tool));
   if (dir) { const d = String(dir).trim(); if (d && !cmd.includes(`--add-dir "${d}"`)) cmd += ` --add-dir "${d}"`; }
   cmd = cmd.replace(/\s+/g, " ").trim();
   cfg.agentCmd = cmd; saveConfig(cfg);
   return { ok: true, cmd };
 }
+// ---- connectors: what the user linked to Claude, for its runs -------------
+// Connectors linked to Claude (claude.ai's Google Drive, Gmail, Notion…, and
+// servers added with `claude mcp add`) are tools to an unattended run only when
+// --allowedTools names them: `claude -p` can't ask, so a run told to check your
+// Drive or mail was refused and said it had no access. From ~/.claude.json: the
+// claude.ai connectors that have connected (claudeAiMcpEverConnected), the
+// user's own servers and this folder's. One waiting to be authorized
+// (mcpNeedsAuthNoticed) is listed but not ready. Its rule, mcp__<name>, allows
+// all its tools; Claude names them with anything but letters, digits, _ and -
+// made _ ("claude.ai Google Drive" -> mcp__claude_ai_Google_Drive__search_files).
+function claudeConnectors(dir = "", file = join(homedir(), ".claude.json")) {
+  let j = {}; try { j = JSON.parse(readFileSync(file, "utf8")) || {}; } catch {}
+  const names = (x) => Array.isArray(x) ? x.filter((s) => typeof s === "string" && s) : [];
+  const servers = (o) => o && typeof o === "object" && !Array.isArray(o) ? Object.keys(o) : [];
+  const proj = dir && j.projects && typeof j.projects === "object" ? j.projects[dir] : null;
+  const needsAuth = new Set(names(j.mcpNeedsAuthNoticed));
+  return [...new Set([...names(j.claudeAiMcpEverConnected), ...servers(j.mcpServers), ...servers(proj && proj.mcpServers)])]
+    .map((name) => ({ name, rule: "mcp__" + name.replace(/[^A-Za-z0-9_-]/g, "_"), ready: !needsAuth.has(name) }));
+}
+// A Claude command with the ready connectors' rules added, for this run only:
+// the saved command stays as typed, so linking or unlinking one takes effect on
+// the next run. Any other agent's command runs as-is (Settings says so).
+function withConnectors(tmpl, dir, file) {
+  if (!isClaudeCmd(tmpl)) return tmpl;
+  return claudeConnectors(dir, file).filter((c) => c.ready).reduce((cmd, c) => allowTool(cmd, c.rule), tmpl).replace(/\s+$/, "");
+}
+// For Settings → Handoff: the connectors, and whether this command's runs get them.
+function connectorsInfo(file) { return { claude: isClaudeCmd(handoffCmd()), list: claudeConnectors("", file).map(({ name, ready }) => ({ name, ready })) }; }
 const fillHandoff = (tmpl, repoPath) => tmpl.replace(/\{dir\}/g, shSingle(repoPath)).replace(/\{prompt\}/g, escDq(HANDOFF_PROMPT));
 // One agent per folder: two identical runs once started on the same repo 6s
 // apart and raced each other. The registry catches a second click in this
@@ -74,15 +103,21 @@ const fillHandoff = (tmpl, repoPath) => tmpl.replace(/\{dir\}/g, shSingle(repoPa
 // A folder's .symbiot/handoff.json ({ name, env }) names its job and joins its
 // environment, on every run there, a rerun with answers too: a draft reply's
 // folder (watch.mjs) carries SYMBIOT_DRAFT, which refuses a press on Send.
-function runHandoff(repoPath) {
-  const tmpl = handoffCmd(); if (!tmpl || !repoPath) return null;
+// A folder whose last run stopped on questions, with nothing changed since
+// that could answer them, gets no run: { blocked, questions, note }
+// (blockedAgain). force starts one anyway (something changed elsewhere).
+function runHandoff(repoPath, { force = false } = {}) {
+  let tmpl = handoffCmd(); if (!tmpl || !repoPath) return null;
   let opts = {}; try { opts = JSON.parse(readSymbiot(repoPath, "handoff.json")) || {}; } catch {}
   const busy = runningHandoff(repoPath); if (busy) return { busy: true, id: busy.id || "", pid: busy.pid, auto: !!busy.auto };
   releaseHeldTasks(repoPath); // held for an agent another process started, which has since exited
+  tmpl = withConnectors(tmpl, repoPath);
+  const blocked = !force && blockedAgain(repoPath, tmpl); if (blocked) return blocked;
   const lock = join(repoPath, ".symbiot", LOCK);
   const env = opts.env && typeof opts.env === "object" ? Object.fromEntries(Object.entries(opts.env).map(([k, v]) => [k, String(v)])) : null;
-  const e = track(typeof opts.name === "string" && opts.name ? opts.name.slice(0, 80) : repoPath.split("/").pop(), fillHandoff(tmpl, repoPath), repoPath, () => {
+  const e = track(typeof opts.name === "string" && opts.name ? opts.name.slice(0, 80) : repoPath.split("/").pop(), fillHandoff(tmpl, repoPath), repoPath, (code) => {
     try { if (JSON.parse(readFileSync(lock, "utf8")).pid === e.pid) unlinkSync(lock); } catch {}
+    noteBlocked(repoPath, tmpl, e.startedAt, code);
     startHeldTasks(repoPath); // tasks sent while it ran land now; start on them as that Send would have
   }, env);
   if (!e) return null;
@@ -90,6 +125,38 @@ function runHandoff(repoPath) {
   if (e.pid) try { writeFileSync(lock, JSON.stringify({ pid: e.pid, id: e.id, startedAt: e.startedAt, owner: process.pid })); } catch {}
   return e;
 }
+// ---- a run that would only report the same blockers again -------------------
+// A run that ends having asked questions (QUESTIONS.md written during it, some
+// still unanswered) with tasks left unticked is blocked on the user: noteBlocked
+// keeps what it ran with in .symbiot/blocked.json. Another run on the same
+// inputs could only say the same things again, so blockedAgain refuses one
+// until something it could act on changes: .env (the values it asked for),
+// ANSWERS.md, the agent command (a grant, a connector) or a task it didn't have.
+const BLOCKED = "blocked.json";
+const digest = (s) => createHash("sha1").update(String(s)).digest("hex").slice(0, 16);
+const openTasks = (path) => readSymbiot(path, "TASKS.md").split("\n").filter((l) => /^\s*-\s*\[ \]/.test(l)).map((l) => taskWords(l.replace(/^\s*-\s*\[ \]\s*/, ""))).filter(Boolean);
+function runInputs(path, cmd) {
+  let files = [], env = ""; try { files = readdirSync(path).filter((f) => /^\.env(\..+)?$/.test(f) && !/\.(example|sample|template)$/.test(f)).sort(); } catch {}
+  for (const f of files) { try { env += f + "\0" + readFileSync(join(path, f), "utf8") + "\0"; } catch {} }
+  return { env: digest(env), answers: digest(readSymbiot(path, "ANSWERS.md")), cmd: digest(cmd), open: openTasks(path) };
+}
+function noteBlocked(path, cmd, startedAt, code) {
+  const f = join(path, ".symbiot", BLOCKED);
+  let asked = false; try { asked = statSync(join(path, ".symbiot", "QUESTIONS.md")).mtimeMs >= startedAt; } catch {}
+  const questions = asked ? agentQuestions(path, "").questions.length : 0, inputs = runInputs(path, cmd);
+  try {
+    if (code === 0 && questions && inputs.open.length) writeFileSync(f, JSON.stringify({ at: Date.now(), questions, ...inputs }));
+    else unlinkSync(f);
+  } catch {}
+}
+function blockedAgain(path, cmd) {
+  let b = null; try { b = JSON.parse(readSymbiot(path, BLOCKED)); } catch {}
+  if (!b || !Array.isArray(b.open)) return null;
+  const now = runInputs(path, cmd), was = new Set(b.open);
+  if (now.env !== b.env || now.answers !== b.answers || now.cmd !== b.cmd || !now.open.length || now.open.some((t) => !was.has(t))) return null;
+  const n = b.questions || 0, them = n === 1 ? "it" : "them";
+  return { blocked: true, questions: n, note: `Not started: the last run here stopped on ${n} question${n === 1 ? "" : "s"}, and .env and ANSWERS.md haven't changed since, so another run would only ask again. Answer ${them} or change .env, then send again.` };
+}
 const LOCK = "agent.pid";
 const LOCK_MAX_AGE = 12 * 3600 * 1000; // older than this, the pid has probably been reused
 const pidAlive = (pid) => { try { process.kill(pid, 0); return true; } catch (err) { return !!err && err.code === "EPERM"; } };
@@ -132,7 +199,7 @@ function releaseHeldTasks(path) {
 // once, so this can't loop. Returns the new job, or null.
 function startHeldTasks(path) {
   if (!releaseHeldTasks(path) || !/^\s*-\s*\[ \]/m.test(readSymbiot(path, "TASKS.md"))) return null;
-  const n = runHandoff(path); if (!n || n.busy) return null;
+  const n = runHandoff(path); if (!n || n.busy || n.blocked) return null;
   n.fromHeld = true; return n;
 }
 // Presets. [cmd, label, macAppName] — macApp used to launch GUI editors on macOS
@@ -348,7 +415,7 @@ function answerQuestions(path, answers, opts = {}) {
     writeFileSync(join(path, ".symbiot", "ANSWERS.md"), prev.replace(/\s*$/, "\n") + rows.map((x) => `\n### ${x.q}\n${x.a}\n_answered ${day}_\n`).join(""));
   } catch (e) { return { error: "Couldn't write ANSWERS.md: " + ((e && e.message) || e) }; }
   const out = { ok: true, saved: rows.length };
-  if (opts.rerun) { const e = runHandoff(path); if (e && e.busy) out.note = "Answers saved. An agent is still running in that folder, so another wasn't started. Send them again once it finishes."; else if (e) out.rerun = e.id; else out.note = "Answers saved. Set an agent command in Settings to have the agent pick them up automatically."; }
+  if (opts.rerun) { const e = runHandoff(path); if (e && e.busy) out.note = "Answers saved. An agent is still running in that folder, so another wasn't started. Send them again once it finishes."; else if (e && e.blocked) out.note = "Answers saved. " + e.note; else if (e) out.rerun = e.id; else out.note = "Answers saved. Set an agent command in Settings to have the agent pick them up automatically."; }
   return out;
 }
 // The open tasks' titles in a held brief (TASKS.next.md), or null if nothing is held.
@@ -367,4 +434,4 @@ function agentsList() {
   });
 }
 
-export { HANDOFFS, HANDOFF_PROMPT, QUESTIONS_MAX, shSingle, CLAUDE_CMD, ORCA_CLAUDE_CMD, handoffCmd, setHandoffCmd, grantAgent, grantRule, fillHandoff, runHandoff, runningHandoff, writeTasks, releaseHeldTasks, startHeldTasks, detectHandoffs, orcaHandoffCmd, migrateOrcaCmd, migrateClaudeCmd, track, agentChanges, parseQuestions, suggestionTarget, agentQuestions, answerQuestions, agentsList };
+export { HANDOFFS, HANDOFF_PROMPT, QUESTIONS_MAX, shSingle, CLAUDE_CMD, ORCA_CLAUDE_CMD, handoffCmd, setHandoffCmd, grantAgent, grantRule, allowTool, claudeConnectors, withConnectors, connectorsInfo, fillHandoff, runHandoff, blockedAgain, runningHandoff, writeTasks, releaseHeldTasks, startHeldTasks, detectHandoffs, orcaHandoffCmd, migrateOrcaCmd, migrateClaudeCmd, track, agentChanges, parseQuestions, suggestionTarget, agentQuestions, answerQuestions, agentsList };

package/desktop.mjs

@@ -57,18 +57,20 @@ function weeklyState() { return { ...weeklyCfg(), latest: latestWeek() }; }
 // Write the week with `produce` (index.mjs), save it and notify. `last` is set
 // first, so a slow or failing run is never retried every minute. notify: false
 // skips the notification, for a caller that tells you itself (the saved file
-// and what's returned are the same either way).
+// and what's returned are the same either way): the Week tab's button does, so
+// a week written there is saved too. What's returned is produce's answer (text,
+// footer), plus the file it was saved to, or an error.
 async function runWeekly(produce, { notify: on = true } = {}) {
   const notify = (body) => on && desktopNotify("Symbiot", body);
   const cfg = loadConfig(); cfg.weekly = { ...weeklyCfg(cfg), last: Date.now() }; saveConfig(cfg);
   const r = await produce("week");
   if (r.error === "not-connected") { notify("Time for your weekly update. Connect an AI in Symbiot's Settings and it writes it for you."); return { error: r.error }; }
-  if (/^\(?couldn't reach the model/i.test(String(r.text || ""))) { notify("Couldn't write your week: the model didn't answer. Try the Week tab."); return { error: r.text }; }
+  if (/^\(?couldn't reach the model/i.test(String(r.text || ""))) { notify("Couldn't write your week: the model didn't answer. Try the Week tab."); return { ...r, error: r.text }; }
   const file = join(WEEKS_DIR, localDay() + ".md");
   try { mkdirSync(WEEKS_DIR, { recursive: true }); writeFileSync(file, String(r.text || "").trim() + (r.footer ? FOOT + r.footer : "") + "\n"); }
-  catch (e) { return { error: "Couldn't save the write-up: " + ((e && e.message) || e) }; }
+  catch (e) { return { ...r, error: "Couldn't save the write-up: " + ((e && e.message) || e) }; }
   notify("Your week is written. Open the Week tab in Symbiot, or " + file);
-  return { ok: true, file, text: r.text };
+  return { ...r, ok: true, file };
 }
 // Checks once a minute while the app runs. Returns a stop function.
 function startWeekly(produce) {

package/headless.mjs

@@ -7,6 +7,8 @@
 // site. On a site you trust (Settings), that goes ahead without asking. In the
 // app the browser stays open for a few minutes after each action, so the next
 // one carries on from the page as it is (type into a field, then press Send).
+// A map has what fits in the window: Scroll down maps the next part, and Whole
+// page maps all of a page that scrolls as a whole in one tall screenshot.
 // readPage reads a page without saving a screen, for Watch (watch.mjs).
 //
 // It drives Chrome / Chromium / Edge / Brave over the DevTools protocol on a pipe
@@ -19,7 +21,7 @@ import { spawn } from "node:child_process";
 import { join } from "node:path";
 import { mkdirSync, existsSync } from "node:fs";
 import { CONFIG_DIR, chromeBinary, loadConfig, saveConfig } from "./core.mjs";
-import { loadScreens, addPageScreen, center } from "./screens.mjs";
+import { loadScreens, addPageScreen, center, stitchPng } from "./screens.mjs";
 
 const PROFILE = join(CONFIG_DIR, "browser");
 const VIEW = { w: 1280, h: 800 }; // the page's size: a laptop-sized window
@@ -207,10 +209,14 @@ const SCROLLER = `function scroller() {
 // link, a part of the same kind is the same thing, so only the outer one counts.
 // And how far down it's scrolled: { y, max, selector? } (no selector: the page
 // itself), when there's more than the window shows.
-const COLLECT = `(() => { ${SCROLLER}
+// whole: called again a window further down each time (snapshotWhole), it adds
+// what's new on screen to what it found before (kept in the page) and gives all
+// of it, each where it is on the page, not the window.
+const collectJs = (whole) => `(() => { ${SCROLLER}
   const SEL = 'a[href],button,input:not([type=hidden]),select,textarea,summary,[contenteditable=""],[contenteditable=true],[onclick],' +
     ['button','link','tab','menuitem','menuitemcheckbox','menuitemradio','option','checkbox','radio','switch','combobox','textbox','searchbox','row','treeitem'].map((r) => '[role=' + r + ']').join(',');
-  const W = innerWidth, H = innerHeight, out = [], kinds = new Map();
+  const W = innerWidth, H = innerHeight, X = ${whole ? "scrollX" : "0"}, Y = ${whole ? "scrollY" : "0"};
+  const kept = ${whole ? "(window.__symbiotWhole = window.__symbiotWhole || { out: [], kinds: new Map() })" : "{ out: [], kinds: new Map() }"}, out = kept.out, kinds = kept.kinds;
   const clean = (s) => String(s || '').replace(/\\s+/g, ' ').trim();
   const kindOf = (e) => {
     const t = e.tagName.toLowerCase(), r = (e.getAttribute('role') || '').toLowerCase(), ty = (e.getAttribute('type') || 'text').toLowerCase();
@@ -236,6 +242,7 @@ const COLLECT = `(() => { ${SCROLLER}
     const img = e.querySelector('img[alt],svg title,[aria-label]');
     return img ? clean(img.getAttribute('alt') || img.getAttribute('aria-label') || img.textContent) : '';
   };
+  const stuck = (e) => { for (let n = e; n && n !== document.body; n = n.parentElement) if (/^(fixed|sticky)$/.test(getComputedStyle(n).position)) return true; return false; };
   const unique = (sel) => { try { return document.querySelectorAll(sel).length === 1; } catch (x) { return false; } };
   const cssPath = (e) => {
     const parts = [];
@@ -250,7 +257,7 @@ const COLLECT = `(() => { ${SCROLLER}
   };
   for (const e of document.querySelectorAll(SEL)) {
     if (out.length >= 200) break;
-    if (e.disabled || e.closest('[aria-hidden=true],[inert]')) continue;
+    if (kinds.has(e) || e.disabled || e.closest('[aria-hidden=true],[inert]')) continue;
     const kind = kindOf(e), up = e.parentElement && e.parentElement.closest(SEL);
     if (up && kinds.get(up) === kind && kind !== 'field') continue;
     const b = e.getBoundingClientRect();
@@ -261,9 +268,11 @@ const COLLECT = `(() => { ${SCROLLER}
     // on top: what's at its centre is it, or inside it (not a banner over it)
     const top = document.elementFromPoint((x + x2) / 2, (y + y2) / 2);
     if (!top || !(top === e || e.contains(top) || top.contains(e))) continue;
+    ${whole ? "if (y >= H - 200 && stuck(e)) continue; // stuck to the window's bottom: left out of the whole page's picture" : ""}
     kinds.set(e, kind);
     const full = labelOf(e);
-    const r = { label: full.slice(0, 80) || kind, kind, x: Math.round(x), y: Math.round(y), w: Math.round(x2 - x), h: Math.round(y2 - y), selector: cssPath(e) };
+    const box = ${whole ? "{ x: Math.max(0, b.left + X), y: Math.max(0, b.top + Y), w: b.width, h: b.height }" : "{ x, y, w: x2 - x, h: y2 - y }"};
+    const r = { label: full.slice(0, 80) || kind, kind, x: Math.round(box.x), y: Math.round(box.y), w: Math.round(box.w), h: Math.round(box.h), selector: cssPath(e) };
     if (full.length > 80) r.text = full.slice(0, 400); // all of a long one (an inbox row), for Watch
     if (e.href && /^https?:/.test(e.href)) r.href = String(e.href).slice(0, 500);
     out.push(r);
@@ -272,10 +281,11 @@ const COLLECT = `(() => { ${SCROLLER}
   const scroll = max > 0 ? { y: Math.round(sc.scrollTop), max, ...(sc === (document.scrollingElement || document.documentElement) ? {} : { selector: cssPath(sc) }) } : null;
   return { url: location.href, title: document.title, items: out, ...(scroll ? { scroll } : {}) };
 })()`;
+const COLLECT = collectJs(false), COLLECT_WHOLE = collectJs(true);
 
 // What's on the page now: { url, title, items }.
-async function collect(page) {
-  const { result, exceptionDetails } = await page.send("Runtime.evaluate", { expression: COLLECT, returnByValue: true });
+async function collect(page, expression = COLLECT) {
+  const { result, exceptionDetails } = await page.send("Runtime.evaluate", { expression, returnByValue: true });
   if (exceptionDetails) throw new Error("Couldn't read the page: " + ((exceptionDetails.exception && exceptionDetails.exception.description) || exceptionDetails.text));
   return (result && result.value) || { url: "", title: "", items: [] };
 }
@@ -285,22 +295,106 @@ function signInPage(url) {
   return /(^|\.)(accounts\.google|login\.(microsoftonline|live)|signin\.aws|auth0|okta)\./i.test(host + ".") || /\/(log-?in|sign-?in|auth|sso)\b/i.test(String(url || ""));
 }
 
+// A screenshot and what's on it, saved as a screen named `name`, else for the page.
+function saveScreen(name, data, info, suffix = "") {
+  const host = hostOf(info.url), base = String(name || "").trim() || info.title || host || "Page";
+  const s = addPageScreen(suffix ? base.slice(0, 80 - suffix.length) + suffix : base, Buffer.isBuffer(data) ? data : Buffer.from(data, "base64"), info, info.items);
+  if (s.error) return s;
+  // a sign-in page instead of the site: say how to get past it once
+  return signInPage(info.url) ? { ...s, note: `This looks like a sign-in page (${host}). Click Sign in, sign in once in the window that opens, close it, then map again: the hidden browser keeps that sign-in.` } : s;
+}
 // The page as it is now, saved as a screen with its regions.
 async function snapshot(page, name) {
   const info = await collect(page);
   const { data } = await page.send("Page.captureScreenshot", { format: "png" });
-  const host = hostOf(info.url);
-  const s = addPageScreen(String(name || "").trim() || info.title || host || "Page", Buffer.from(data, "base64"), info, info.items);
-  if (s.error) return s;
-  // a sign-in page instead of the site: say how to get past it once
-  return signInPage(info.url) ? { ...s, note: `This looks like a sign-in page (${host}). Click Sign in, sign in once in the window that opens, close it, then map again: the hidden browser keeps that sign-in.` } : s;
+  return saveScreen(name, data, info);
+}
+
+// The whole page at once, in one screenshot as tall as the page (up to MAX_TALL),
+// with every button, link and field on it as a region, where it is on the page.
+// The window stays the laptop size (so a part as tall as the window stays that
+// tall): it scrolls down a window at a time, finds what's on each part and takes
+// its screenshot, and the screenshots are put together (screens.mjs stitchPng).
+// What's fixed or sticky (a menu bar, a cookie banner) is hidden once it's in one,
+// so it shows once, where it was, not on every part. The parts overlap by
+// OVERLAP pixels, and the later one's are kept, so what stuck to the bottom of the
+// window is left out (and not marked: collectJs). Only for a page that scrolls
+// as a whole: a list that scrolls inside the page (Gmail's) wouldn't show more, so
+// that gives { error, inside } and is left to Scroll down.
+const MAX_TALL = 16000, OVERLAP = 200; // pixels
+const PAGE_HEIGHT = `(() => { ${SCROLLER}
+  const d = document.scrollingElement || document.documentElement, sc = scroller();
+  return { h: Math.max(d.scrollHeight, document.body ? document.body.scrollHeight : 0), inside: !!sc && sc !== d };
+})()`;
+const HIDE_STUCK = `(() => { const H = innerHeight;
+  if (!document.getElementById('symbiot-hide')) { const st = document.createElement('style'); st.id = 'symbiot-hide'; st.textContent = '[data-symbiot-hide]{visibility:hidden!important}'; document.documentElement.appendChild(st); }
+  for (const e of document.querySelectorAll('body *')) {
+    const p = getComputedStyle(e).position; if (p !== 'fixed' && p !== 'sticky') continue;
+    const b = e.getBoundingClientRect(); if (b.bottom > 0 && b.top < H) e.setAttribute('data-symbiot-hide', '');
+  }
+})()`;
+const UNHIDE = `document.querySelectorAll('[data-symbiot-hide]').forEach((e) => e.removeAttribute('data-symbiot-hide')); const st = document.getElementById('symbiot-hide'); if (st) st.remove(); window.__symbiotWhole = null; scrollTo({ top: 0, behavior: 'instant' });`;
+async function evaluate(page, expression) {
+  const { result, exceptionDetails } = await page.send("Runtime.evaluate", { expression, returnByValue: true });
+  if (exceptionDetails) throw new Error("Couldn't read the page: " + ((exceptionDetails.exception && exceptionDetails.exception.description) || exceptionDetails.text));
+  return result && result.value;
+}
+async function snapshotWhole(page, name) {
+  const first = await evaluate(page, PAGE_HEIGHT);
+  if (first.inside) return { error: "This page doesn't scroll as a whole: a list inside it does (like Gmail's mail), and a tall screenshot wouldn't show more of that. Use Scroll down to map the next part of the list.", inside: true };
+  if (first.h <= VIEW.h + 1) { const s = await snapshot(page, name); return s.error || s.note ? s : { ...s, note: "It all fits in the window, so this map is the whole page." }; }
+  // a window at a time, overlapping a little; the page can load more as it goes (up to MAX_TALL)
+  let info = null, h = first.h, end = 0; const shots = [];
+  await evaluate(page, "window.__symbiotWhole = null");
+  try {
+    for (let y = 0, i = 0; i < 60; i++) {
+      await evaluate(page, `scrollTo({ top: ${y}, behavior: 'instant' })`); await sleep(150);
+      const at = Number(await evaluate(page, "scrollY")) || 0;
+      if (!info || info.items.length < 200) info = await collect(page, COLLECT_WHOLE);
+      const { data } = await page.send("Page.captureScreenshot", { format: "png" });
+      shots.push({ png: Buffer.from(data, "base64"), y: at }); end = at + VIEW.h;
+      await evaluate(page, HIDE_STUCK);
+      h = (await evaluate(page, PAGE_HEIGHT)).h;
+      if (end >= Math.min(MAX_TALL, h) || at < y) break; // the bottom (or it wouldn't scroll further)
+      y = Math.min(at + VIEW.h - OVERLAP, Math.min(MAX_TALL, h) - VIEW.h);
+    }
+  } finally { await evaluate(page, UNHIDE).catch(() => {}); }
+  const tall = Math.min(MAX_TALL, end), png = stitchPng(shots, tall);
+  if (!png) return { error: "Couldn't put the page's screenshots together." };
+  const items = info.items.filter((r) => r.y < tall);
+  const s = saveScreen(name, png, { url: info.url, title: info.title, full: true, items }, " (whole page)");
+  if (s.error || s.note) return s;
+  const note = [h > tall + 1 && `The page goes on past ${tall} pixels, so this map stops there.`, items.length >= 200 && "It marks the first 200 buttons, links and fields, from the top."].filter(Boolean).join(" ");
+  return note ? { ...s, note } : s;
 }
 
 // Map a site: open it in the hidden browser and save what's on it as a screen.
-function mapPage(input, name) {
+// whole: all of the page in one tall screen (snapshotWhole); a page whose list
+// scrolls inside it gets the usual map, with a note that says why.
+function mapPage(input, name, { whole = false } = {}) {
   const url = siteUrl(input);
   if (!url) return Promise.resolve({ error: "Give a site to map: a name (gmail), a host (github.com/pulls) or a web address." });
-  return oneAtATime(() => withPage(async (page) => { await open(page, url); return snapshot(page, name); })).catch((e) => ({ error: String((e && e.message) || e) }));
+  return oneAtATime(() => withPage(async (page) => {
+    await open(page, url);
+    if (!whole) return snapshot(page, name);
+    const s = await snapshotWhole(page, name);
+    if (!s.inside) return s;
+    const one = await snapshot(page, name);
+    return one.error || one.note ? one : { ...one, note: s.error };
+  })).catch((e) => ({ error: String((e && e.message) || e) }));
+}
+
+// The whole of a page you've mapped, as one tall screen (snapshotWhole). Only
+// looks, so it never asks first.
+function wholePage(id) {
+  const s = loadScreens().find((x) => x.id === id); if (!s) return Promise.resolve({ error: "not found" });
+  if (!s.page || !s.page.url) return Promise.resolve({ error: "Whole page works on a mapped page." });
+  return oneAtATime(() => withPage(async (page, b) => {
+    const here = await showScreen(page, b, s);
+    const next = await snapshotWhole(page, s.name.replace(/ ↓ \d+%$/, "").replace(/ \(whole page\)$/, ""));
+    if (next.error) { delete next.inside; return next; }
+    return { ...next, kept: here };
+  })).catch((e) => ({ error: String((e && e.message) || e) }));
 }
 
 // Read a site as it is now, without saving a screen: { url, title, items, login }.
@@ -408,7 +502,11 @@ function actOnRegion(id, regionId, verb, confirmed, check, act) {
       e.scrollIntoView({ block: 'center', inline: 'center' }); const b = e.getBoundingClientRect();
       return b.width && b.height ? { x: b.left + b.width / 2, y: b.top + b.height / 2 } : null; })()`;
     const { result } = await page.send("Runtime.evaluate", { expression: find, returnByValue: true });
-    const at = (result && result.value) || center(r), found = !!(result && result.value);
+    const found = !!(result && result.value);
+    // not found by its selector: the same spot. On a whole-page screen that spot is
+    // on the page, not the window: scroll it into the window first.
+    let at = found ? result.value : center(r);
+    if (!found && s.page.full) { const y = await evaluate(page, `(scrollTo(0, ${at.y} - innerHeight / 2), scrollY)`); at = { x: at.x, y: at.y - (Number(y) || 0) }; }
     const navigating = page.until("Page.frameStartedLoading", 1500);
     const extra = await act(page, at, r);
     if (await navigating) await page.until("Page.loadEventFired", 30000);
@@ -432,10 +530,15 @@ function pressRegion(id, regionId, { confirmed = false, noSend = false } = {}) {
 // asked (which is how a search or a one-line form is sent). Without Enter, press
 // the form's button next, on the screen this maps, while the browser is still
 // open (the app's): what's typed is still there. Otherwise it's gone by then.
+// noSend (a draft reply's run) refuses Enter, which sends in a chat, and in a
+// WhatsApp chat types line breaks as spaces, in case a new line sends there too.
 const MAX_TEXT = 2000;
-function typeRegion(id, regionId, text, { enter = false, confirmed = false } = {}) {
+const CHAT_HOSTS = new Set(["web.whatsapp.com"]);
+function typeRegion(id, regionId, text, { enter = false, confirmed = false, noSend = false } = {}) {
   text = String(text == null ? "" : text);
+  if (noSend) { const s = loadScreens().find((x) => x.id === id); if (s && s.page && CHAT_HOSTS.has(hostOf(s.page.url))) text = text.replace(/\s*[\r\n]+\s*/g, " ").trim(); }
   return actOnRegion(id, regionId, "Type", confirmed, (r) => {
+    if (noSend && enter) return "Enter sends in a chat, and this run only drafts: type without --enter, and the reply stays unsent for the user to send.";
     if (r.kind !== "field") return `"${r.label}" isn't a field (it's a ${r.kind || "region"}). Type works on a field; use Press for the rest.`;
     if (!text && !enter) return "Give the text to type.";
     if (text.length > MAX_TEXT) return `That's more than ${MAX_TEXT} characters.`;
@@ -463,4 +566,4 @@ async function signIn(input) {
   return { ok: true, url };
 }
 
-export { siteUrl, browserArgs, mapPage, readPage, isSend, pressRegion, typeRegion, scrollPage, SCROLLS, signIn, keepBrowserOpen, closeBrowser, browserOpen, trustedSites, isTrusted, trustSite, untrustSite, PROFILE };
+export { siteUrl, browserArgs, mapPage, wholePage, readPage, isSend, pressRegion, typeRegion, scrollPage, SCROLLS, signIn, keepBrowserOpen, closeBrowser, browserOpen, trustedSites, isTrusted, trustSite, untrustSite, PROFILE };