dsh-inner-browser-use 0.18.0 → 0.27.0

package/docs/BENCH.md

@@ -232,3 +232,112 @@ Measured with `/tmp/probe-unseen.mjs` (tool layer, example.com, profile
 `probe-tool`): 1344 ms with nothing watching and no column asked for, 3236 ms in
 the first version of the same wait that did not skip the case where no column was
 ever requested — the whole difference was the wasted 1.2 s.
+
+## Phase 9 — finding the field behind the words, 0.19.0
+
+A read of a `/bare` form (two div labels, two bare inputs, no aria anywhere on
+the fields) and one `{field: "Work email"}` step:
+
+| what | result |
+| --- | --- |
+| the click lands on the field, not on the div | yes, value `me@x.y` in `input[name=wk]` |
+| the second, untouched field | still empty |
+| the tool's own words | `typed into "Work email"` |
+| worst case when nothing matches | one extra in-page query, then the old behaviour |
+
+The query walks the open shadow roots (capped at 400) and, per visible editable
+element, builds a list of candidate names from real labels, `aria-labelledby`,
+`aria-label`, `placeholder`, `title`, `name`, `label[for]`, the element right
+before it, and its wrapper's text — then scores exact (3) / contains (2) /
+reverse-contains (1).
+
+## Phase 10 — typing by name, 0.20.0
+
+| what the caller has | what it used to take | now |
+| --- | --- | --- |
+| a name for the field, no number | read the page, find the number, then type | one `sidecar_type` with `field`, one round trip |
+| the field's words live in a div | click landed on the div, text went wherever focus was | resolved to the field, `typed into "Work email"` |
+| a banner over the field | typed under the banner | banner cleared first, then typed |
+
+Breadth check `type-by-words` on the `/bare` form: value lands in
+`input[name=wk]`, the page does not move (no stray Enter).
+
+## Phase 11 — choosing a dropdown by name, 0.21.0
+
+`/bare` now carries a `select[name=seats]` whose only name on screen is a div
+saying `Seats`. One `sidecar_select` with `field: "Seats"`, `value: "Two"`:
+
+| what | result |
+| --- | --- |
+| the value after the call | `2` |
+| the tool's own words | `chose Two in Seats` (on-screen name, not `name="seats"`) |
+| a wrong option | answered with the first 12 options instead of a page read |
+| round trips to do it | one; no read beforehand, no read after |
+
+The naming rules (`labelSpot` and `selectSpot`) now live in one shared in-page
+block, so the two steps cannot drift apart.
+
+## Phase 12 — words split across nodes, 0.22.0
+
+`/words` now carries `<button><span>Sign</span><span>in</span></button>`, which
+the page reports as `Signin`.
+
+| what | result |
+| --- | --- |
+| `{text: "Sign in"}` before | not found; caller had to fall back to a number |
+| `{text: "Sign in"}` now | clicked, title changes, `ok click "Sign in"` |
+| extra work | none: the same single control list, one comparison changed |
+
+Exact equality still wins over the squeezed form, so `Sign in` never lands on
+`Sign up` when both exist.
+
+## Phase 13 — the field behind the box, 0.23.0
+
+`/bare` gains the date-picker shape: a visible rounded box with a transparent
+`<input>` laid over it, its words in a div two levels up.
+
+| what | before | now |
+| --- | --- | --- |
+| `{field: "Birthday"}` | not found, fell back to clicking the div | found the hidden input, clicked its visible box |
+| value after one `sidecar_do` step | whatever already had focus | `2001-02-03` in `#bd`, `typed into "Birthday"` |
+| extra round trips | read the page, guess a number | none |
+
+The visible-box rule: a candidate is usable when it, or its nearest visible
+ancestor, has a real box — so a 0-opacity input still counts if it covers
+something the person can see.
+
+## Phase 15 — the answer is the picture, 0.25.0
+
+`/pic` is one 140x90 PNG and nothing else.
+
+| what | result |
+| --- | --- |
+| `sidecar_grab` | 6 ms, `picture-check.png`, 140x90 |
+| before | three round trips: list the images, ask for the full address, fetch it |
+
+A picture page has no text at all, so a read of it costs one line and no tokens;
+the picture comes back as a file path the person can open.
+
+## Phase 16 — say what the picture looks like, 0.26.0
+
+Two 140x90 PNGs: one a gradient, one a single colour.
+
+| page | what `sidecar_grab` answers | time |
+| --- | --- | --- |
+| `/pic` | `... 140x90 a picture with detail` | 21 ms (first, page cold) |
+| `/flat` | `... 140x90 mostly one colour` | 4 ms |
+
+The sampling is 32x32 inside the page, so it costs no round trip and its result
+travels in the same line as the path.
+
+## Phase 17 — the answer is a picture, in the read itself, 0.27.0
+
+| page | read line | extra round trips |
+| --- | --- | --- |
+| `/pic` (gradient, no words) | `picture 140x90 a picture with detail "Chart" — sidecar_grab saves it as a file` | 0 |
+| `/flat` (single colour) | `picture 140x90 mostly one colour "Chart" — …` | 0 |
+| `/words` (text page) | no picture line | 0 |
+
+The sampling is 32x32 inside the page, in the same call that reads the page, so a
+blank chart is visible before anything is saved.
+

package/docs/ROADMAP.md

@@ -174,3 +174,61 @@ Next in this direction: the drawing half could re-attach by itself. A tab that
 reloads re-subscribes on its own, but a tab that never loaded the half cannot be
 told to — that is a load-order problem, so the fix belongs in how the panel half
 is delivered, not in the wait.
+
+## Phase 10 — words are not where the field is, 0.19.0
+
+The tool used to treat "the control that says Email" as "the field called
+Email". Real forms separate the two, so the writing step now resolves the words
+to the element that takes text, and says which one it used.
+
+Next in this direction: the same resolution for `sidecar_type` when the caller
+names a field by its words instead of a ref, and for date/time pickers where the
+visible box is a div and the real input is hidden beside it.
+
+## Phase 11 — say where, not which number, 0.20.0
+
+Typing by name is in. The same resolution is what other steps should be able to
+use when the caller knows the destination but not the number — the next place
+worth doing this is `sidecar_select` (choose an option by the field it belongs
+to), and after that the date/time case where the visible box is a div and the
+real input sits hidden beside it.
+
+## Phase 12 — the same words, the same place, 0.21.0
+
+`sidecar_select` joined `sidecar_type` and `sidecar_do` on the shared resolver.
+Left in this direction: the date/time case where the box the user sees is a div
+and the real input is hidden beside it, and `sidecar_click` on a control whose
+words are split across sibling nodes (`<span>Sign</span><span>in</span>`).
+
+## Phase 13 — the page's spelling is not the caller's, 0.22.0
+
+Whitespace is normalised in both halves now (tool side and in-page side). Next
+in this line: the date/time box where the visible box is a div and the real
+input is hidden beside it, which needs the resolver to accept a hidden target
+and click the thing in front of it.
+
+## Phase 14 — behind the box, 0.23.0
+
+Both halves now resolve a name to a place: visible field, hidden field behind a
+visible box, words split over nodes, words outside the wrapper. Next: a page
+whose visible box is a `<canvas>` or an `<iframe>` — nothing in the tree to
+name, so the fallback has to be a picture plus a click, not a name lookup.
+
+## Phase 16 — a file, not a path, 0.25.0
+
+`sidecar_grab` returns a path. Next: say one line about the picture (size, and
+whether it is mostly one colour) so a model that cannot open files still learns
+"the chart is blank" without a full read.
+
+## Phase 17 — a picture nobody can see, 0.26.0
+
+The description is coarse on purpose. Next: when a page's answer is a picture,
+let a read say so in one line ("the answer is a picture, and it is blank") so a
+model that cannot open files does not need the tool at all to notice a failure.
+
+## Phase 18 — a picture that never arrived, 0.27.0
+
+A picture is only an answer if it loaded. Next: report a large picture element
+whose source failed (`naturalWidth === 0`) as a broken picture rather than
+staying silent, since that is the commonest way a generated image "disappears".
+

package/docs/VERIFY.md

@@ -149,3 +149,17 @@ hand on the mirror: while the person is using it a read still works
 (`hold-refuses-action`), and `sidecar_resume` hands the page back so the same
 click goes through (`resume-lets-go`).
 
+## Who the mirror serves
+
+The bar that matters most cannot be reached from `node --test`, because it is
+about a column that belongs to one conversation. With DSH running:
+
+```sh
+node scripts/live-safety.mjs http://127.0.0.1:49552
+```
+
+It reads the state, then tries the two ways in — a frame stream and a
+navigation — as a session that is not the owner. Both must answer 403, and the
+owner's stream must answer 200. Three `true` lines is the bar; `open · owner
+none` as the first line only means nothing is on screen, and the two refusals
+still have to hold.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-inner-browser-use",
-  "version": "0.18.0",
+  "version": "0.27.0",
   "description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
   "type": "module",
   "main": "index.js",
@@ -48,7 +48,10 @@
     "live": "node scripts/live.mjs"
   },
   "license": "MIT",
-  "author": "regrevia",
+  "author": {
+    "name": "regrevia",
+    "url": "https://github.com/regrevia"
+  },
   "repository": {
     "type": "git",
     "url": "git+https://github.com/regrevia/dsh-inner-browser-use.git"

package/read.js

@@ -43,6 +43,9 @@ const SELECTOR = [
   '[role="option"]',
   '[role="textbox"]',
   '[role="searchbox"]',
+  /* A drawing surface has no words in the tree, but it is the whole page in
+     some products: number it, and let a picture be taken of it. */
+  'canvas',
 ].join(',')
 
 /** How a role is written in the compact read: short enough to cost little. */
@@ -133,6 +136,10 @@ function snapshot(mode, budget, max) {
     const own = named || el.getAttribute?.('aria-label') || el.getAttribute?.('title') || el.getAttribute?.('placeholder') || ''
     const text = String(el.innerText ?? '').replace(/\\s+/g, ' ').trim()
     const value = el.tagName === 'INPUT' && el.type !== 'password' ? String(el.value ?? '') : ''
+    if (el.tagName === 'CANVAS' && label === '' && own === '' && text === '' && value === '') {
+      const b = el.getBoundingClientRect()
+      return 'drawing ' + Math.round(b.width) + 'x' + Math.round(b.height)
+    }
     return String(label || own || text || value || el.getAttribute?.('alt') || el.getAttribute?.('name') || '').replace(/\\s+/g, ' ').trim().slice(0, 140)
   }
   const box = (el) => el.getBoundingClientRect()
@@ -240,6 +247,59 @@ function snapshot(mode, budget, max) {
       }
     }
   }
+  /* A page with almost no words and one big picture is answering with the
+     picture. Say so here, in the same round trip, so a model that cannot open
+     files still learns that something came back: and sample it small to tell a
+     blank chart from a chart. */
+  const picture = (() => {
+    if (full >= 40) return null
+    let best = null
+    const missing = []
+    for (const el of document.querySelectorAll('img,canvas')) {
+      const b = box(el)
+      if (b.width < 64 || b.height < 64 || b.width * b.height < 8000) continue
+      const s = getComputedStyle(el)
+      if (s.visibility === 'hidden' || s.display === 'none') continue
+      /* A picture slot that never got its picture is the commonest way a
+         generated image goes missing, and it is silent by default. */
+      if (el.tagName === 'IMG' && el.complete && el.naturalWidth === 0) {
+        missing.push({ width: Math.round(b.width), height: Math.round(b.height), name: String(el.getAttribute('alt') ?? '').replace(/\s+/g, ' ').trim().slice(0, 60) })
+        continue
+      }
+      const w = el.tagName === 'CANVAS' ? el.width : el.naturalWidth || Math.round(b.width)
+      const h = el.tagName === 'CANVAS' ? el.height : el.naturalHeight || Math.round(b.height)
+      if (best === null || w * h > best.w * best.h) best = { el, w, h, kind: String(el.tagName).toLowerCase() }
+    }
+    if (best === null) return missing.length === 0 ? null : { missing }
+    /* What the read hands back about one picture: not a screen of fields. */
+    const said = { name: String(best.el.getAttribute?.('alt') ?? best.el.getAttribute?.('aria-label') ?? '').replace(/\s+/g, ' ').trim().slice(0, 60) }
+    said.kind = best.kind
+    said.width = best.w
+    said.height = best.h
+    said.colors = 0
+    said.flat = 0
+    try {
+      const board = document.createElement('canvas')
+      board.width = 32
+      board.height = 32
+      const pen = board.getContext('2d', { willReadFrequently: true })
+      if (pen === null) return said
+      pen.drawImage(best.el, 0, 0, 32, 32)
+      const dots = pen.getImageData(0, 0, 32, 32).data
+      const tally = new Map()
+      for (let at = 0; at < dots.length; at += 4) {
+        const key = (dots[at] >> 5) * 64 + (dots[at + 1] >> 5) * 8 + (dots[at + 2] >> 5)
+        tally.set(key, (tally.get(key) || 0) + 1)
+      }
+      let most = 0
+      for (const count of tally.values()) most = Math.max(most, count)
+      said.colors = tally.size
+      said.flat = Math.round((most / 1024) * 100) / 100
+    } catch {
+      /* A picture from elsewhere cannot be sampled: report its size only. */
+    }
+    return said
+  })()
   return JSON.stringify({
     url: location.href, // ui-copy-ignore
     title: document.title, // ui-copy-ignore
@@ -252,6 +312,7 @@ function snapshot(mode, budget, max) {
       return { ref: reg.id(el), kind: role(el), name: name(el) }
     })(),
     controls: picked,
+    picture,
   })
 })()`
 }
@@ -278,6 +339,119 @@ export function resolve(ref) {
 })()`
 }
 
+/**
+ * How big the control is and where it sits, so a picture can be of it alone.
+ * @param {number} ref - ref from a read.
+ * @returns {string} an expression that answers with the rectangle.
+ */
+export function bounds(ref) {
+  return `(() => {
+  const reg = window.${REGISTRY}
+  if (!reg) return JSON.stringify({ gone: true })
+  const hit = reg.find(${Number(ref)})
+  if (!hit) return JSON.stringify({ gone: true })
+  const el = hit.el
+  if (el.scrollIntoView) el.scrollIntoView({ block: 'center', inline: 'center' })
+  const b = el.getBoundingClientRect()
+  if (b.width < 2 || b.height < 2) return JSON.stringify({ gone: true, off: true })
+  return JSON.stringify({ gone: false, x: b.left, y: b.top, width: b.width, height: b.height })
+})()`
+}
+
+/**
+ * The pictures on the page, as files a person can open.
+ * A generated image or a chart is often the whole answer, and there is no
+ * wording in the tree that stands for it: take the picture itself.
+ * @param {number} [ref] - only pictures at or inside this control.
+ * @param {number} [count] - how many, largest first. Default 1.
+ * @returns {string} an expression that answers with `{pictures: [...]}`.
+ */
+export function pictures(ref, count) {
+  const scope = Number(ref)
+  const limit = Math.max(1, Math.min(4, Number(count) || 1))
+  return `(async () => {
+  const reg = window.${REGISTRY}
+  const where = ${Number.isFinite(scope) ? `(reg ? (reg.find(${scope}) || {}).el : null)` : 'null'}
+  const picked = []
+  const seen = new Set()
+  const add = (el, kind) => {
+    if (el === null || el === undefined || seen.has(el)) return
+    seen.add(el)
+    const b = el.getBoundingClientRect()
+    const w = kind === 'canvas' ? el.width : el.naturalWidth || b.width
+    const h = kind === 'canvas' ? el.height : el.naturalHeight || b.height
+    if (w < 48 || h < 48) return
+    if (b.width < 8 || b.height < 8) return
+    const s = getComputedStyle(el)
+    if (s.visibility === 'hidden' || s.display === 'none' || Number(s.opacity) < 0.05) return
+    picked.push({ el, kind, w, h, area: w * h })
+  }
+  const roots = where ? [where, ...where.querySelectorAll('*')] : [document, ...document.querySelectorAll('*')]
+  for (const node of roots) {
+    if (node === document || node === where) continue
+    const tag = node.tagName
+    if (tag === 'IMG') add(node, 'img')
+    else if (tag === 'CANVAS') add(node, 'canvas')
+  }
+  if (where && where.tagName === 'IMG') add(where, 'img')
+  if (where && where.tagName === 'CANVAS') add(where, 'canvas')
+  picked.sort((a, b) => b.area - a.area)
+  /* A picture nobody can look at still has to be describable: how much of it is
+     one colour tells a blank chart from a chart. Sampled at 32x32, which is
+     enough for that and costs nothing. */
+  const look = async (data) => {
+    try {
+      const img = new Image()
+      img.src = data
+      await img.decode()
+      const board = document.createElement('canvas')
+      board.width = 32
+      board.height = 32
+      const pen = board.getContext('2d', { willReadFrequently: true })
+      if (pen === null) return null
+      pen.drawImage(img, 0, 0, 32, 32)
+      const dots = pen.getImageData(0, 0, 32, 32).data
+      const tally = new Map()
+      for (let at = 0; at < dots.length; at += 4) {
+        const key = (dots[at] >> 5) * 64 + (dots[at + 1] >> 5) * 8 + (dots[at + 2] >> 5)
+        tally.set(key, (tally.get(key) || 0) + 1)
+      }
+      let most = 0
+      for (const count of tally.values()) most = Math.max(most, count)
+      return { colors: tally.size, flat: Math.round((most / 1024) * 100) / 100 }
+    } catch {
+      return null
+    }
+  }
+  const out = []
+  for (const item of picked.slice(0, ${limit})) {
+    let data = ''
+    let type = 'image/png'
+    try {
+      if (item.kind === 'canvas') {
+        data = item.el.toDataURL('image/png')
+      } else {
+        const url = item.el.currentSrc || item.el.src || ''
+        const answer = await fetch(url, { credentials: 'include' })
+        const blob = await answer.blob()
+        type = blob.type || type
+        data = await new Promise((done, fail) => {
+          const reader = new FileReader()
+          reader.onload = () => done(String(reader.result))
+          reader.onerror = () => fail(new Error('unreadable'))
+          reader.readAsDataURL(blob)
+        })
+      }
+    } catch {
+      data = ''
+    }
+    const feel = data ? await look(data) : null
+    out.push({ kind: item.kind, width: item.w, height: item.h, type, data, colors: feel?.colors ?? 0, flat: feel?.flat ?? 0 })
+  }
+  return JSON.stringify({ pictures: out })
+})()`
+}
+
 /**
  * What sits on top of a control at click time, when it is not the control.
  * A cookie banner or a modal over the thing the agent meant to click is the
@@ -521,6 +695,7 @@ export async function readPage(page, options = {}) {
     title: String(read?.title ?? page.title),
     text: String(read?.text ?? ''),
     textChars: Number(read?.textChars ?? 0),
+    picture: read?.picture ?? null,
     scroll: read?.scroll,
     focused: read?.focused ?? null,
     controls: Array.isArray(read?.controls) ? read.controls : [],
@@ -591,6 +766,18 @@ export function renderPage(read, options = {}) {
     lines.push(body)
     if (read.textChars > body.length) lines.push(`… ${read.textChars - body.length} more characters`)
   }
+  if (read.picture && Array.isArray(read.picture.missing)) {
+    /* Something was meant to be here and is not: say that instead of nothing. */
+    for (const gone of read.picture.missing.slice(0, 2)) {
+      const size = `${Math.round(Number(gone.width) || 0)}x${Math.round(Number(gone.height) || 0)}`
+      lines.push(`picture ${size}${gone.name === '' ? '' : ` "${gone.name}"`} did not load — the page may need a reload, or the picture is gone`)
+    }
+  } else if (read.picture) {
+    const pic = read.picture
+    const look = Number(pic.flat) >= 0.9 ? 'mostly one colour' : Number(pic.colors) <= 3 ? 'a few colours' : 'a picture with detail'
+    const size = `${Math.round(Number(pic.width) || 0)}x${Math.round(Number(pic.height) || 0)}`
+    lines.push(`picture ${size} ${look}${pic.name === '' ? '' : ` "${pic.name}"`} — sidecar_grab saves it as a file`)
+  }
   if (read.scroll && read.scroll.height > read.scroll.viewport + 40) {
     lines.push(`scroll ${read.scroll.y}/${read.scroll.height - read.scroll.viewport}`)
   }
@@ -634,23 +821,8 @@ export function sameThing(want, control) {
   return true
 }
 
-/**
- * The field a label belongs to, when nothing formally connects the two.
- *
- * `{field: "Email"}` used to mean "click the control that says Email". On a real
- * form the words live in a div above a bare input, so the click landed on the
- * text and the typing went wherever focus already was — the classic silent wrong
- * field. This finds the *editable* element the words point at: a real label, an
- * aria name, a placeholder, a name attribute, the text right before it, or the
- * text of the small wrapper it sits in.
- * @param {string} words - what the field is called.
- * @returns {string} an expression that evaluates to a spot, or {} when nothing fits.
- */
-export function labelSpot(words) {
-  const wanted = JSON.stringify(String(words ?? '').toLowerCase())
-  return `(() => {
-  const want = ${wanted}
-  if (want === '') return JSON.stringify({})
+/** The in-page half shared by `labelSpot` and `selectSpot`: shadow roots, what says what. */
+const NAME_PRELUDE = `
   const roots = [document]
   for (const el of document.querySelectorAll('*')) {
     if (roots.length > 400) break
@@ -685,34 +857,125 @@ export function labelSpot(words) {
     add(el.getAttribute('aria-label'))
     add(el.getAttribute('placeholder'))
     add(el.getAttribute('title'))
-    add(el.getAttribute('name'))
     if (el.id) for (const l of document.querySelectorAll('label[for]')) if (l.htmlFor === el.id) add(l.innerText ?? l.textContent)
     const prev = el.previousElementSibling
     if (prev && editable(prev) !== true) add(prev.innerText ?? prev.textContent)
     const parent = el.parentElement
     if (parent && parent.querySelectorAll('input,textarea,select,[contenteditable=true]').length <= 1) add(parent.innerText ?? parent.textContent)
+    /* Wrapped a couple of levels deep (a styled box hiding a real input), the
+       words sit before the wrapper, not before the input. */
+    let up = parent
+    for (let depth = 0; depth < 3 && up; depth += 1) {
+      const before = up.previousElementSibling
+      if (before && editable(before) !== true) add(before.innerText ?? before.textContent)
+      up = up.parentElement
+    }
+    /* The attribute is the last resort for a name on screen: it is often the
+       developer's word for the field, not the one the user reads. */
+    add(el.getAttribute('name'))
     return out
   }
-  let best = null
-  for (const root of roots) {
-    for (const el of root.querySelectorAll('input,textarea,select,[contenteditable=""],[contenteditable=true],[role=textbox],[role=searchbox],[role=combobox]')) {
-      if (editable(el) !== true || shown(el) !== true) continue
-      let score = 0
-      let name = ''
-      for (const nm of names(el)) {
-        const low = nm.toLowerCase()
-        if (low === want) score = Math.max(score, 3)
-        else if (want.length >= 2 && low.includes(want)) score = Math.max(score, 2)
-        else if (low.length >= 3 && want.includes(low)) score = Math.max(score, 1)
-        if (score > 0 && name === '') name = nm
+  const squeeze = (s) => String(s).toLowerCase().replace(/\s+/g, '')
+  const scored = (el, want) => {
+    let score = 0
+    let name = ''
+    const flat = squeeze(want)
+    for (const nm of names(el)) {
+      const low = nm.toLowerCase()
+      if (low === want || squeeze(low) === flat) score = Math.max(score, 3)
+      else if (want.length >= 2 && low.includes(want)) score = Math.max(score, 2)
+      else if (flat.length >= 2 && squeeze(low).includes(flat)) score = Math.max(score, 2)
+      else if (low.length >= 3 && want.includes(low)) score = Math.max(score, 1)
+      if (score > 0 && name === '') name = nm
+    }
+    return { score, name }
+  }
+  const boxOf = (el) => {
+    for (let node = el; node; node = node.parentElement) {
+      const b = node.getBoundingClientRect()
+      if (b.width < 8 || b.height < 8) continue
+      const s = getComputedStyle(node)
+      if (s.display !== 'none' && s.visibility !== 'hidden' && Number(s.opacity) > 0.05) return b
+    }
+    return null
+  }
+  const bestOf = (selector, want, hiddenOk) => {
+    let best = null
+    for (const root of roots) {
+      for (const el of root.querySelectorAll(selector)) {
+        if (hiddenOk !== true && shown(el) !== true) continue
+        const hit = scored(el, want)
+        if (hit.score === 0) continue
+        if (best === null || hit.score > best.score) best = { el, score: hit.score, name: hit.name }
       }
-      if (score === 0) continue
-      if (best === null || score > best.score) best = { el, score, name }
     }
+    return best
+  }
+`
+
+/**
+ * The field a label belongs to, when nothing formally connects the two.
+ * ...
+ */
+/**
+ * The field a label belongs to, when nothing formally connects the two.
+ *
+ * `{field: "Email"}` used to mean "click the control that says Email". On a real
+ * form the words live in a div above a bare input, so the click landed on the
+ * text and the typing went wherever focus already was — the classic silent wrong
+ * field. This finds the *editable* element the words point at: a real label, an
+ * aria name, a placeholder, a name attribute, the text right before it, or the
+ * text of the small wrapper it sits in.
+ * @param {string} words - what the field is called.
+ * @returns {string} an expression that evaluates to a spot, or {} when nothing fits.
+ */
+export function labelSpot(words) {
+  const wanted = JSON.stringify(String(words ?? '').toLowerCase())
+  return `(() => {
+  const want = ${wanted}
+  if (want === '') return JSON.stringify({})
+  ${NAME_PRELUDE}
+  const selector = 'input,textarea,select,[contenteditable=""],[contenteditable=true],[role=textbox],[role=searchbox],[role=combobox]'
+  let best = bestOf(selector, want)
+  let box = best === null ? null : boxOf(best.el)
+  let behind = false
+  if (best === null || box === null) {
+    /* The field a date picker shows is often a transparent box over a styled
+       div: the input stays out of sight, but the words still name it. */
+    const under = bestOf(selector, want, true)
+    const place = under === null ? null : boxOf(under.el)
+    if (under !== null && place !== null) { best = under; box = place; behind = true }
   }
-  if (best === null) return JSON.stringify({})
+  if (best === null || box === null) return JSON.stringify({})
+  if (editable(best.el) !== true) return JSON.stringify({})
   best.el.scrollIntoView({ block: 'center', inline: 'nearest' })
-  const b = best.el.getBoundingClientRect()
-  return JSON.stringify({ x: Math.round(b.left + b.width / 2), y: Math.round(b.top + b.height / 2), name: best.name, score: best.score })
+  const fresh = boxOf(best.el) ?? box
+  return JSON.stringify({ x: Math.round(fresh.left + fresh.width / 2), y: Math.round(fresh.top + fresh.height / 2), name: best.name, score: best.score, behind })
+})()`
+}
+
+/**
+ * The dropdown the words belong to, with the wanted option already chosen.
+ * @param {string} words - what the dropdown is called on screen.
+ * @param {string} value - option label or value.
+ * @returns {string} an expression that picks the option and says what it did.
+ */
+export function selectSpot(words, value) {
+  const wanted = JSON.stringify(String(words ?? '').toLowerCase())
+  const wanted2 = JSON.stringify(String(value ?? ''))
+  return `(() => {
+  const want = ${wanted}
+  const wanted = ${wanted2}
+  if (want === '') return 'no dropdown was named'
+  ${NAME_PRELUDE}
+  const best = bestOf('select', want)
+  if (best === null) return 'no dropdown says ' + JSON.stringify(want)
+  const select = best.el
+  const option = Array.from(select.options).find((item) => item.value === wanted || item.text.trim() === wanted || item.text.trim().toLowerCase() === wanted.toLowerCase())
+  if (!option) return 'no option ' + wanted + ' in ' + best.name + '; options are ' + Array.from(select.options).slice(0, 12).map((item) => item.text.trim()).join(' | ')
+  select.value = option.value
+  select.dispatchEvent(new Event('input', { bubbles: true }))
+  select.dispatchEvent(new Event('change', { bubbles: true }))
+  return 'chose ' + option.text.trim() + ' in ' + best.name
 })()`
 }

package/tools.js

@@ -13,7 +13,7 @@ import { homedir } from 'node:os'
 import { join } from 'node:path'
 import { mkdir, writeFile } from 'node:fs/promises'
 import { VIEW } from './sidecar.js?v=1'
-import { readPage, renderPage, locate, covering, contextFor, identityOf, sameThing, labelSpot, REF_GONE } from './read.js?v=1'
+import { readPage, renderPage, locate, covering, contextFor, identityOf, sameThing, labelSpot, selectSpot, bounds, pictures, REF_GONE } from './read.js?v=1'
 
 /** Where a screenshot from `sidecar_shot` lands, under the user's home. */
 export const SHOT_DIR = join('.dsh-inner-browser-use', 'screenshots')
@@ -419,9 +419,15 @@ async function until(page, expression, timeout) {
 async function findWords(page, words) {
   const read = await readPage(page, { mode: 'text', maxChars: 1200, max: 200 })
   const shown = read.controls.filter((control) => control.view !== false)
-  const wanted = words.toLowerCase()
-  const hit = shown.find((control) => control.name.toLowerCase() === wanted) ?? shown.find((control) => control.name.toLowerCase().includes(wanted))
-  if (hit === undefined) return null
+  /* The words may sit in separate inline nodes: <span>Sign</span><span>in</span>
+     reads as "Signin" to the page, so compare with the spaces taken out too. */
+  const squeeze = (text) => String(text).toLowerCase().replace(/\s+/g, '')
+  const wanted = squeeze(words)
+  const found = shown.find((control) => squeeze(control.name) === wanted) ?? shown.find((control) => squeeze(control.name).includes(wanted))
+  if (found === undefined) return null
+  /* When only the squeezed form matched, say the words the caller used: the
+     page spread them over nodes and "Signin" is not what anyone wrote. */
+  const hit = squeeze(found.name) === wanted && found.name.toLowerCase() !== words.toLowerCase() ? { ...found, name: words } : found
   const spot = await locate(page, hit.ref)
   if (spot.gone === true) return null
   return { ...hit, x: Number(spot.x), y: Number(spot.y) }
@@ -438,12 +444,50 @@ const EDITABLE_KINDS = ['input', 'textarea', 'select', 'combobox', 'textbox', 's
  */
 async function fieldSpot(page, words) {
   const said = await evaluate(page, labelSpot(words))
-  if (said.error !== undefined || said.result === undefined) return null
-  const spot = said.result
-  if (spot === undefined || spot === null || spot.name === undefined) return null
+  if (said === undefined || said === null || said.error !== undefined) return null
+  const spot = typeof said === 'string' ? JSON.parse(said) : said
+  if (spot === null || spot.name === undefined || spot.name === '') return null
   return { x: Number(spot.x), y: Number(spot.y), name: String(spot.name) }
 }
 
+/**
+ * Where the field called these words is, clearing a banner over it first.
+ * @param {any} page - sidecar page.
+ * @param {string} words - what the field is called on screen.
+ * @returns {Promise<{x: number, y: number, name: string, lead: string}|null>} the click.
+ */
+async function fieldByName(page, words) {
+  const hit = await findWords(page, words)
+  const lead = hit === null ? '' : await coverLead(page, hit.ref)
+  if (hit !== null && EDITABLE_KINDS.includes(String(hit.kind))) {
+    const spot = await locate(page, hit.ref)
+    if (spot.gone !== true) return { x: Number(spot.x), y: Number(spot.y), name: String(hit.name), lead }
+  }
+  const field = await fieldSpot(page, words)
+  if (field !== null) return { ...field, lead }
+  if (hit === null) return null
+  const spot = await locate(page, hit.ref)
+  if (spot.gone === true) return null
+  return { x: Number(spot.x), y: Number(spot.y), name: String(hit.name), lead }
+}
+
+/**
+ * Click a spot, type into it, and optionally clear it first.
+ * @param {any} page - sidecar page.
+ * @param {{x: number, y: number}} point - where to click.
+ * @param {string} text - what to type.
+ * @param {{clear?: boolean, submit?: boolean}} options - how.
+ */
+async function writeAt(page, point, text, options = {}) {
+  await page.mouse('mouseMoved', point)
+  await page.mouse('mousePressed', { ...point, clickCount: 1 })
+  await page.mouse('mouseReleased', { ...point, clickCount: 1 })
+  await wait(80)
+  if (options.clear === true) await clearFocus(page)
+  await page.insertText(text)
+  if (options.submit === true) await pressKey(page, 'Enter')
+}
+
 /**
  * Where the control behind a ref is now, following it across a re-render.
  * The module-level sibling of findAgain, for runStep, which has no closure.
@@ -502,40 +546,10 @@ async function runStep(page, step, known) {
   if (step.field !== undefined) {
     /* The words may sit on the field or in a div above a bare input. Only a
        click that lands on the field itself can be trusted with the typing. */
-    const hit = await findWords(page, String(step.field))
-    let lead = ''
-    let point = null
-    let called = ''
-    if (hit !== null && EDITABLE_KINDS.includes(String(hit.kind))) {
-      lead = await coverLead(page, hit.ref)
-      const spot = await locate(page, hit.ref)
-      if (spot.gone !== true) {
-        point = { x: Number(spot.x), y: Number(spot.y) }
-        called = String(hit.name)
-      }
-    }
-    if (point === null) {
-      const field = await fieldSpot(page, String(step.field))
-      if (field !== null) {
-        point = { x: field.x, y: field.y }
-        called = field.name
-      }
-    }
-    if (point === null) {
-      if (hit === null) return `no control says ${JSON.stringify(String(step.field))}`
-      const spot = await locate(page, hit.ref)
-      if (spot.gone === true) return `ref ${String(hit.ref)}: ${REF_GONE}`
-      point = { x: Number(spot.x), y: Number(spot.y) }
-      called = String(hit.name)
-    }
-    await page.mouse('mouseMoved', point)
-    await page.mouse('mousePressed', { ...point, clickCount: 1 })
-    await page.mouse('mouseReleased', { ...point, clickCount: 1 })
-    await wait(80)
-    if (step.clear === true) await clearFocus(page)
-    await page.insertText(String(step.text ?? ''))
-    if (step.submit === true) await pressKey(page, 'Enter')
-    return `${lead}typed into "${called.slice(0, 60)}"`
+    const target = await fieldByName(page, String(step.field))
+    if (target === null) return `${String(step.field)}: ${REF_GONE}`
+    await writeAt(page, { x: target.x, y: target.y }, String(step.text ?? ''), { clear: step.clear, submit: step.submit })
+    return `${target.lead}typed into "${target.name.slice(0, 60)}"`
   }
   if (step.tap !== undefined) {
     const [x, y] = String(step.tap).split(',').map(Number)
@@ -893,6 +907,7 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
         properties: {
           text: { type: 'string', description: 'Text to type.' },
           ref: { type: 'integer', description: 'Control to click before typing.' },
+          field: { type: 'string', description: 'The field is called this on screen, when you have no number for it.' },
           submit: { type: 'boolean', description: 'Press Enter after typing.' },
           clear: { type: 'boolean', description: 'Select all and delete before typing.' },
           read: { type: 'boolean', description: 'Read the page after the action, instead of a one-line confirmation.' },
@@ -907,6 +922,15 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
       execute: async (args) => withPage(async (page) => {
         let cover = ''
         let followed = ''
+        if (args?.field !== undefined && args?.ref === undefined) {
+          const target = await fieldByName(page, String(args.field))
+          if (target === null) return `${String(args.field)}: ${REF_GONE}`
+          cover = target.lead
+          await writeAt(page, { x: target.x, y: target.y }, String(args?.text ?? ''), { clear: args?.clear, submit: args?.submit === true })
+          await settle(page)
+          const called = await acted(page, args, () => look(page, { mode: 'text', maxChars: 2000 }), 'type')
+          return called + cover + ` (typed into "${target.name.slice(0, 50)}")` // ui-copy-ignore
+        }
         if (args?.ref !== undefined) {
           const gate = await clearCover(page, Number(args.ref))
           cover = gate.note
@@ -1163,9 +1187,10 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
         type: 'object', // ui-copy-ignore
         properties: {
           ref: { type: 'integer', description: 'Number of the select control from sidecar_view.' },
+          field: { type: 'string', description: 'The dropdown is called this on screen, when you have no number for it.' },
           value: { type: 'string', description: 'Option label or value to choose.' },
         },
-        required: ['ref', 'value'], // ui-copy-ignore
+        required: ['value'], // ui-copy-ignore
         additionalProperties: false,
       },
       output: TEXT_OUTPUT, // ui-copy-ignore
@@ -1173,6 +1198,15 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
       isConcurrencySafe: () => false,
       /** @param {any} args - arguments. @returns {Promise<{text: string}>} the page after. */
       execute: async (args) => withPage(async (page) => {
+        if (args?.ref === undefined) {
+          if (args?.field === undefined) return 'name the dropdown: pass its number from sidecar_view, or the words on screen that name it.' // ui-copy-ignore
+          const chosen = await evaluate(page, selectSpot(String(args.field), String(args?.value ?? '')))
+          const said = typeof chosen === 'string' ? chosen : chosen?.error === undefined ? '' : String(chosen.error)
+          if (said === '' || said.startsWith('no dropdown')) return said === '' ? 'the page did not answer; read it and try again.' : said // ui-copy-ignore
+          await settle(page, 3000)
+          if (args?.read === true) return `${said}\n${await look(page, { mode: 'text', maxChars: 2000 })}`
+          return `${said} · ${await brief(page)}`
+        }
         const spot = await findAgain(page, Number(args?.ref))
         if (spot.gone === true) return REF_GONE
         const wanted = JSON.stringify(String(args?.value ?? ''))
@@ -1253,7 +1287,18 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
         if (args?.ref !== undefined) {
           const spot = await findAgain(page, Number(args.ref))
           if (spot.gone === true) return REF_GONE
-          clip = { x: Math.max(0, Number(spot.x) - 20), y: Math.max(0, Number(spot.y) - 20), width: 220, height: 140 }
+          /* The picture is of the control, not of a guess around its middle:
+             a canvas can be most of the page, a checkbox a few pixels. */
+          const box = await evaluate(page, bounds(Number(args.ref)))
+          const rect = typeof box === 'string' ? JSON.parse(box) : box
+          if (rect?.gone === true) return REF_GONE
+          const pad = 8
+          clip = {
+            x: Math.max(0, Math.round(Number(rect.x) - pad)),
+            y: Math.max(0, Math.round(Number(rect.y) - pad)),
+            width: Math.max(8, Math.round(Number(rect.width) + pad * 2)),
+            height: Math.max(8, Math.round(Number(rect.height) + pad * 2)),
+          }
         }
         const data = await page.shot({ full: args?.full === true, rect: clip, scale })
         const dir = join(homedir(), SHOT_DIR)
@@ -1264,6 +1309,58 @@ export function sidecarTools(sidecar, refs, host = {}) { // ui-copy-ignore
         return file
       }),
     },
+    {
+      name: 'sidecar_grab', // ui-copy-ignore
+      description: 'Save the picture on the page as a file: an image, a chart, a drawing surface. Picks the largest one, or the one inside a given control.',
+      parameters: {
+        type: 'object', // ui-copy-ignore
+        properties: {
+          ref: { type: 'integer', description: 'Only the picture at or inside this control.' },
+          name: { type: 'string', description: 'File name without extension.' },
+          count: { type: 'integer', description: 'How many, largest first. Default 1, at most 4.' },
+        },
+        additionalProperties: false,
+      },
+      output: TEXT_OUTPUT, // ui-copy-ignore
+      timeoutMs: 40_000,
+      isConcurrencySafe: () => false,
+      /** @param {any} args - arguments. @returns {Promise<string>} the saved paths, one per line. */
+      execute: async (args) => withRead(async (page) => {
+        const said = await evaluate(page, pictures(args?.ref, args?.count))
+        let found
+        try {
+          found = JSON.parse(typeof said === 'string' ? said : JSON.stringify(said))
+        } catch {
+          found = { pictures: [] }
+        }
+        const rows = []
+        const dir = join(homedir(), SHOT_DIR)
+        await mkdir(dir, { recursive: true })
+        const base = String(args?.name ?? '').replace(/[^\w.-]/g, '') || `sidecar-picture-${Date.now()}`
+        const list = Array.isArray(found?.pictures) ? found.pictures : []
+        for (let index = 0; index < list.length; index += 1) {
+          const item = list[index]
+          const body = String(item?.data ?? '')
+          const mark = body.indexOf(',')
+          if (mark < 0) continue
+          const type = String(item?.type ?? 'image/png')
+          const ext = type.includes('jpeg') ? 'jpg' : type.includes('webp') ? 'webp' : type.includes('svg') ? 'svg' : 'png'
+          const file = join(dir, `${base}${list.length > 1 ? `-${index + 1}` : ''}.${ext}`)
+          await writeFile(file, Buffer.from(body.slice(mark + 1), 'base64'))
+          const size = `${Math.round(Number(item.width) || 0)}x${Math.round(Number(item.height) || 0)}`
+          /* Say what the picture looks like in words: a model that cannot open
+             a file still needs to know a chart came back blank. */
+          const flat = Number(item.flat) || 0
+          const colors = Number(item.colors) || 0
+          const look = flat >= 0.9 ? 'mostly one colour' : colors <= 3 ? 'a few colours' : 'a picture with detail'
+          rows.push(`${file} ${size} ${look}`)
+        }
+        if (rows.length === 0) {
+          return list.length === 0 ? 'There is no picture on the page.' : 'That picture could not be saved.'
+        }
+        return rows.join('\n')
+      }),
+    },
     {
       name: 'sidecar_state', // ui-copy-ignore
       description: 'Report the sidecar browser: whether it runs, its profile, address, title, how many people are watching its mirror, and how many times a call had to find a control again after the page redrew it.',