dsh-moli 1.1.15 → 1.1.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -209,7 +209,7 @@ An unresolvable bundle is **skipped silently** apart from one stderr line
209
209
 
210
210
  ## Usage
211
211
 
212
- `moli_fetch` parameters: `url` (required), `dump`, `writeTo`, `eval`, `scroll`, `wait`,
212
+ `moli_fetch` parameters: `url` (required), `dump`, `writeTo`, `selector`, `eval`, `scroll`, `wait`,
213
213
  `waitSelector`, `waitScript`, `waitResponseUrl`, `waitResponseJson`, `delayMs`, `timeoutMs`,
214
214
  `maxChars`, `headers`, `userAgent`, `layout`, `scrollbars`, `noJs`, `noCss`, `resources`,
215
215
  `withFrames`, `insecureTls`, `failOnHttpError`, `profileDir`, `cookieFile`, `proxy`.
@@ -218,6 +218,7 @@ An unresolvable bundle is **skipped silently** apart from one stderr line
218
218
  moli_fetch url=https://example.com dump=markdown # token-efficient text
219
219
  moli_fetch url=https://example.com dump=semantic_tree_text # nodes with backend ids
220
220
  moli_fetch url=https://example.com eval="document.title" # one computed value
221
+ moli_fetch url=https://example.com selector=".video-card" # one container, not the page
221
222
  moli_fetch url=https://example.com dump=screenshot writeTo=/abs/path/a.png
222
223
  moli_fetch url=https://example.com dump=pdf writeTo=/abs/path/a.pdf
223
224
  moli_fetch url=https://example.com dump=markdown scroll=800 # trigger lazy loading first
@@ -225,6 +226,35 @@ moli_fetch url=https://example.com dump=markdown scroll=800 # trigger lazy loa
225
226
 
226
227
  Screenshot and PDF dumps need `writeTo`; the bytes are written verbatim rather than printed.
227
228
 
229
+ ### `selector` — read one container, not the page
230
+
231
+ A listing page is mostly noise if you only want the items. `selector` returns the text
232
+ of the elements a CSS selector matches, which on a node-dense feed measured **1,829
233
+ characters against 95,605** for the same page as Markdown.
234
+
235
+ ```
236
+ moli_fetch url=https://example.com/feed selector=".video-card"
237
+ moli_fetch url=https://example.com/feed selector="h2" scroll=2000 # scroll first, then read
238
+ ```
239
+
240
+ All matches are joined with newlines and the result reports how many there were:
241
+ "matched 48 element(s)" is the difference between a complete read and a partial one,
242
+ and it is the only signal you get that the selector was too broad or too narrow.
243
+
244
+ Two cases are deliberately errors rather than empty answers, because the engine
245
+ cannot tell an empty selection from a page with no text — an expression returning
246
+ `""` exits 0 with a blank line:
247
+
248
+ - **nothing matched** — `ok: false`, "no element matches". A selector that is not on
249
+ the page is almost always a spelling mistake or a page that has not loaded yet.
250
+ - **matched, but no text** — `ok: false`, "match but none of them has any text", which
251
+ is what an icon or an image container gives you. Use `eval` if you need attributes.
252
+
253
+ `selector` cannot be combined with `dump` or `eval`: a selector returns one
254
+ container's text, a dump returns the whole page, and only one expression value can be
255
+ returned — so one of them could only be silently ignored, which is the failure mode
256
+ this plugin keeps having to fix.
257
+
228
258
  ### `scroll` — reach what only appears on a scroll
229
259
 
230
260
  The engine serializes the whole document, so a plain fetch already contains everything the DOM holds.
@@ -242,8 +272,29 @@ Measured on a page whose `scroll` handler fills a slot: without `scroll` the dum
242
272
  `STILL-EMPTY`, with `scroll=1000` it says `LAZY-CONTENT-1000`. It also moves the screenshot, so a
243
273
  scrolled `dump=screenshot` shows that part of the page.
244
274
 
245
- It runs as part of the engine's own pre-capture evaluation and therefore composes with `eval`: the
246
- scroll happens first, and the expression still returns its own value.
275
+ `scroll` is a side effect, so it composes and stands alone:
276
+
277
+ ```
278
+ moli_fetch url=... dump=markdown scroll=600 # scroll, then dump the page
279
+ moli_fetch url=... eval="document.title" scroll=600 # scroll, then read a value
280
+ moli_fetch url=... scroll=600 # just scroll; you still get the page
281
+ ```
282
+
283
+ The last form is worth spelling out because it was broken. The scroll runs in the engine's own
284
+ pre-capture evaluation, and without `dump` the engine prints whatever that evaluation returns — so an
285
+ earlier version that returned the scroll position made this call answer `600` instead of the document,
286
+ while the identical call without `scroll` returned 1,989 characters of HTML. A scroll reports nothing
287
+ now; it only happens.
288
+
289
+ ### `eval` with `dump` — the expression value is dropped
290
+
291
+ The engine documents this and it is kept, because using an expression to change a page before
292
+ capturing it is a real pattern. What is not acceptable is that it used to look like a clean answer:
293
+ `eval='JSON.stringify({probe:1})'` with `dump=markdown` returned 900 characters of Markdown, reported
294
+ `ok: true`, and said nothing. The result now carries `evalDiscarded: true` and a note, and the rendered
295
+ output leads with the warning rather than burying it under the body.
296
+
297
+ If you want the expression's value, pass `eval` without `dump`.
247
298
 
248
299
  ### `failOnHttpError` — stop reading an error page as content
249
300
 
@@ -468,10 +519,17 @@ Eight suites live in [`verify/`](verify/), and all are runnable against a workin
468
519
  `bottom`, nonsense rejected, the caller's `eval` value preserved) and then, when an engine is
469
520
  present, fetches a page whose `scroll` handler fills a slot and asserts the content appears with
470
521
  `scroll` and the placeholder appears without it — a property of the output, not of an internal call.
522
+ - `selector-verify.mjs` covers the two silent-wrong-answer cases a container read invites: a selector
523
+ matching nothing is an error rather than empty content, and a selector matching several elements
524
+ reports the count and joins them. It also asserts that the selector is quoted as data rather than
525
+ spliced into the expression, that `selector` with `dump` or `eval` is refused instead of one of them
526
+ being ignored, and that it composes with `scroll`.
471
527
  - `http-verify.mjs` asserts the parameter-to-flag mapping directly, because a parameter that never
472
528
  reaches the engine is indistinguishable from a feature that does not work. It then fetches a 404 and
473
529
  checks that `failOnHttpError` reports it while keeping the body, that a healthy page is unaffected,
474
530
  and that a screenshot of an error page is still written as a valid PNG rather than decoded as text.
531
+ It also covers the two silent-answer cases: `eval` with `dump` is flagged `evalDiscarded` and the
532
+ warning renders first, and `scroll` alone returns the document rather than the scroll offset.
475
533
  The end-to-end half needs an HTTP error from a real server: loopback is unreachable from inside this
476
534
  sandbox — `curl.exe` times out on `127.0.0.1` while the Node host reaches it — so it uses a path
477
535
  under `example.com` and skips when the network is away.
package/README.zh.md CHANGED
@@ -195,15 +195,16 @@ profile 插件需要两半都齐:依赖边(`dependencies`)+ `dsh.profile.b
195
195
 
196
196
  ## 用法
197
197
 
198
- `moli_fetch` 参数:`url`(必填)、`dump`、`writeTo`、`eval`、`scroll`、`wait`、`waitSelector`、
199
- `waitScript`、`waitResponseUrl`、`waitResponseJson`、`delayMs`、`timeoutMs`、`maxChars`、`headers`、
200
- `userAgent`、`layout`、`scrollbars`、`noJs`、`noCss`、`resources`、`withFrames`、`insecureTls`、
201
- `failOnHttpError`、`profileDir`、`cookieFile`、`proxy`。
198
+ `moli_fetch` 参数:`url`(必填)、`dump`、`writeTo`、`selector`、`eval`、`scroll`、`wait`、
199
+ `waitSelector`、`waitScript`、`waitResponseUrl`、`waitResponseJson`、`delayMs`、`timeoutMs`、
200
+ `maxChars`、`headers`、`userAgent`、`layout`、`scrollbars`、`noJs`、`noCss`、`resources`、
201
+ `withFrames`、`insecureTls`、`failOnHttpError`、`profileDir`、`cookieFile`、`proxy`。
202
202
 
203
203
  ```
204
204
  moli_fetch url=https://example.com dump=markdown # 省 token 的正文
205
205
  moli_fetch url=https://example.com dump=semantic_tree_text # 带节点 id 的结构
206
206
  moli_fetch url=https://example.com eval="document.title" # 取一个值
207
+ moli_fetch url=https://example.com selector=".video-card" # 只取一个容器,不要整页
207
208
  moli_fetch url=https://example.com dump=screenshot writeTo=/绝对路径/a.png
208
209
  moli_fetch url=https://example.com dump=pdf writeTo=/绝对路径/a.pdf
209
210
  moli_fetch url=https://example.com dump=markdown scroll=800 # 先触发懒加载再抓取
@@ -211,6 +212,29 @@ moli_fetch url=https://example.com dump=markdown scroll=800 # 先触发懒加
211
212
 
212
213
  截图与 PDF 必须给 `writeTo`;字节是原样落盘的,不会打印到输出里。
213
214
 
215
+ ### `selector` —— 只取一个容器,不要整页
216
+
217
+ 列表页里如果只要条目,正文大部分都是噪声。`selector` 返回 CSS 选择器匹配到的元素文本——在一个节点密集的
218
+ 信息流上实测**1,829 字符,而同一页面走 markdown 是 95,605**。
219
+
220
+ ```
221
+ moli_fetch url=https://example.com/feed selector=".video-card"
222
+ moli_fetch url=https://example.com/feed selector="h2" scroll=2000 # 先滚再取
223
+ ```
224
+
225
+ 所有匹配项以换行拼接,并且结果会报告匹配到多少个:**"matched 48 element(s)"** 是"读全了"和"只读到一部分"
226
+ 的区别,也是你判断选择器太宽或太窄的唯一信号。
227
+
228
+ 两种情况**刻意做成错误而不是空答案**,因为引擎分不清"没匹配到"和"页面没有文本"——表达式返回 `""` 时是
229
+ exit 0 加一个空行:
230
+
231
+ - **没有任何匹配** —— `ok: false`,"no element matches"。页面上不存在的选择器几乎总是拼写错误或页面还没加载。
232
+ - **匹配到了但没有文本** —— `ok: false`,说明匹配到了但都没文本,这通常会发生在图标或图片容器上。需要属性就
233
+ 用 `eval`。
234
+
235
+ `selector` 不能与 `dump` 或 `eval` 同传:选择器返回一个容器的文本、dump 返回整页、而一次只能返回一个表达式
236
+ 值——所以其中一个只能被**静默忽略**,而这正是这个插件一路在修的那类失败。
237
+
214
238
  ### `scroll` —— 拿到"只有滚动后才出现"的内容
215
239
 
216
240
  引擎会序列化整个文档,所以普通抓取**已经包含 DOM 里的一切**。它拿不到的是**尚未存在的内容**:首次
@@ -227,8 +251,26 @@ moli_fetch url=https://example.com/feed dump=markdown scroll=-400 # 往回滚
227
251
  `scroll=1000` 时是 `LAZY-CONTENT-1000`。它同样会改变截图,所以带滚动的 `dump=screenshot` 拍到的是页面的
228
252
  那一段。
229
253
 
230
- 它作为引擎自身"抓取前求值"的一部分执行,因此与 `eval` 可组合:**先滚动,表达式再读取页面,并且仍然返回
231
- 它自己的值**。
254
+ `scroll` 是副作用,因此既可组合也可单独使用:
255
+
256
+ ```
257
+ moli_fetch url=... dump=markdown scroll=600 # 先滚动,再 dump 页面
258
+ moli_fetch url=... eval="document.title" scroll=600 # 先滚动,再取一个值
259
+ moli_fetch url=... scroll=600 # 只滚动,你仍然拿到页面
260
+ ```
261
+
262
+ **最后一种写法值得单独说明,因为它曾经是坏的。** 滚动在引擎自身的"抓取前求值"里执行,而在没有 `dump` 时,
263
+ 引擎打印的就是这个求值的返回值——所以早期版本返回滚动位置,导致这一调用回的是 `600` 而不是页面,而**同样的
264
+ 调用不加 `scroll` 会返回 1989 个字符的 HTML**。现在滚动不报告任何东西,它只是发生。
265
+
266
+ ### `eval` 与 `dump` 同用 —— 表达式值会被丢弃
267
+
268
+ 这是引擎写明并有意的行为,予以保留——**用表达式在抓取前改变页面**是真实用法。不能接受的是它**看起来像一个
269
+ 干净答案**:`eval='JSON.stringify({probe:1})'` 配 `dump=markdown` 会返回 900 个字符的 Markdown、报
270
+ `ok: true`、什么都不说。现在结果会带 `evalDiscarded: true` 和说明,而且渲染输出**把警告放在最前面**,不会
271
+ 被正文埋掉。
272
+
273
+ 要拿表达式的值,就只传 `eval`、不要传 `dump`。
232
274
 
233
275
  ### `failOnHttpError` —— 别再把错误页当正文读
234
276
 
@@ -426,11 +468,15 @@ profile。无论生效的是哪个代理,loopback 地址一律绕过它。
426
468
  - `scroll-verify.mjs` 在无引擎时校验表达式构造(像素值、负数、`top`、`bottom`、无效输入被拒、调用方的 `eval`
427
469
  值被保留),有引擎时抓取一个"滚动处理器填充槽位"的页面,断言带 `scroll` 时内容出现、不带时只有占位符——
428
470
  断言的是**输出**的性质,而不是内部调用。
471
+ - `selector-verify.mjs` 覆盖"按容器取值"最容易出现的两个静默错答案:选择器无匹配时报错而不是空内容、匹配多个
472
+ 元素时报告数量并拼接。它还断言选择器是**作为数据被引号包裹**而不是拼进表达式、`selector` 与 `dump`/`eval`
473
+ 同传会被拒绝(而不是静默忽略其中一个)、以及它能与 `scroll` 组合。
429
474
  - `http-verify.mjs` 直接断言"参数 → 引擎选项"的映射,因为**参数没送到引擎和功能不生效,从外面看是一样的**。
430
475
  随后抓一个 404,验证 `failOnHttpError` 会报告失败同时保留响应体、健康页面不受影响、以及错误页的截图仍然被
431
- 写为合法 PNG 而不是被当文本解码。端到端部分需要一个真实服务器的 HTTP 错误:**本沙箱内无法访问 loopback**
432
- ——`curl.exe` 连 `127.0.0.1` 会超时,而 Node 宿主进程可以——所以它用 `example.com` 下的一个路径,网络不可用
433
- 时跳过。
476
+ 写为合法 PNG 而不是被当文本解码。它还覆盖那两个"静默给错答案"的情形:`eval` 配 `dump` 会带
477
+ `evalDiscarded` 标记且警告渲染在最前,`scroll` 单独用返回的是文档而不是滚动偏移。端到端部分需要一个真实
478
+ 服务器的 HTTP 错误:**本沙箱内无法访问 loopback**——`curl.exe` 连 `127.0.0.1` 会超时,而 Node 宿主进程
479
+ 可以——所以它用 `example.com` 下的一个路径,网络不可用时跳过。
434
480
 
435
481
  ### 测试前置条件
436
482
 
package/index.js CHANGED
@@ -94,12 +94,16 @@ const FETCH_PARAMETERS = {
94
94
  dump: {
95
95
  type: 'string',
96
96
  enum: ['html', 'markdown', 'json', 'semantic_tree', 'semantic_tree_text', 'screenshot', 'screenshot_full', 'pdf'],
97
- description: 'Output format. Omit for rendered HTML. `markdown` and `semantic_tree_text` are the token-efficient choices for reading a page; `json` carries status and final_url; screenshot/pdf need writeTo.',
97
+ description: 'Output format. Omit for rendered HTML. `markdown` is the cheapest way to read a page. `semantic_tree_text` is for locating things by structure (role, name, backend id), not for saving tokens: its size follows the page\'s node density and it is often larger than markdown — measured at 1.8x on a node-dense page — so prefer it only when you need the structure. `json` carries status and final_url; screenshot/pdf need writeTo.',
98
98
  },
99
99
  writeTo: {
100
100
  type: 'string',
101
101
  description: 'Absolute path to save the output. Required for screenshot/pdf; optional for text dumps larger than maxChars.',
102
102
  },
103
+ selector: {
104
+ type: 'string',
105
+ description: 'Return the text of the elements this CSS selector matches instead of the whole page. This is the cheap way to read one container out of a noisy listing page: the cards on a feed measured 1,829 characters against 95,605 for the same page as markdown. All matches are joined with newlines and the result reports how many there were; matching nothing is an error, not empty content, because the engine cannot distinguish an empty selection from a page with no text. Not compatible with `dump` or `eval` — one of them could only be ignored.',
106
+ },
103
107
  eval: {
104
108
  type: 'string',
105
109
  description: 'JavaScript expression evaluated once the page is ready; its value is returned as text or JSON instead of the page. Combining it with `dump` does NOT return both: the engine discards the expression value and dumps the resulting page state instead (its own documented semantics). Use `eval` alone to read a value, or `dump` alone to read the page.',
@@ -174,6 +178,9 @@ const FETCH_OUTPUT_SCHEMA = {
174
178
  empty: { type: 'boolean' },
175
179
  omitted: { type: 'integer' },
176
180
  evalFailed: { type: 'boolean' },
181
+ evalDiscarded: { type: 'boolean' },
182
+ selector: { type: 'string' },
183
+ matched: { type: 'integer' },
177
184
  truncated: { type: 'boolean' },
178
185
  error: { type: 'string' },
179
186
  hint: { type: 'string' },
@@ -226,13 +233,18 @@ function renderFetch(_args, value) {
226
233
  if (value.unreachable === true) lines.push('Cause: the host did not answer; a local forward proxy is the usual fix on this network.');
227
234
  else if (value.possiblyUnreachable === true) lines.push('Cause: the host did not answer in time; it may be blocked, or the page may simply be slow.');
228
235
  if (typeof value.hint === 'string' && value.hint !== '') lines.push(`Next: ${value.hint}`);
229
- // An HTTP error keeps its body under `failOnHttpError`. Reporting the failure
236
+ // An HTTP error keeps its body under `failOnHttpError`. Reporting the failure
230
237
  // without the body would throw away the error page, which is the part worth
231
- // reading -- and for a binary dump the body is the file that was written.
232
- if (value.kind === 'binary' && typeof value.path === 'string') {
238
+ // reading -- and for a binary dump the body is the file that was written. A
239
+ // selector failure is different: there is no server body to show, and the
240
+ // message already says what did not match.
241
+ if (value.httpError === true && value.kind === 'binary' && typeof value.path === 'string') {
233
242
  lines.push(`The error page was still written to ${value.path} (${String(value.bytes)} bytes).`);
234
- } else if (typeof value.content === 'string' && value.content !== '') {
243
+ } else if (value.httpError === true && typeof value.content === 'string' && value.content !== '') {
235
244
  lines.push('', 'The server\'s response body:', '', value.content);
245
+ } else if (typeof value.selector === 'string') {
246
+ lines.push(`Selector: ${JSON.stringify(value.selector)}`
247
+ + (typeof value.matched === 'number' ? ` (matched ${String(value.matched)})` : ''));
236
248
  }
237
249
  return [{ type: 'text', text: lines.join('\n') }];
238
250
  }
@@ -243,6 +255,18 @@ function renderFetch(_args, value) {
243
255
  }];
244
256
  }
245
257
  const parts = [];
258
+ // A discarded `eval` value is the one case where the output looks like a clean
259
+ // answer to a question that was never answered, so the warning goes first rather
260
+ // than after the body, where a 900-character page would bury it.
261
+ if (value.evalDiscarded === true) {
262
+ parts.push('Note: the `eval` expression was dropped because `dump` was given too — the engine discards the expression value in that combination. What follows is the page, not the expression.');
263
+ }
264
+ // A selector answer is a list, so say how many elements produced it: "48 matched,
265
+ // 34 had text" is the difference between a complete read and a partial one, and it
266
+ // is the only signal the caller has that the selector was too broad or too narrow.
267
+ if (typeof value.selector === 'string' && typeof value.matched === 'number') {
268
+ parts.push(`Selector ${JSON.stringify(value.selector)} matched ${String(value.matched)} element(s).`);
269
+ }
246
270
  if (typeof value.note === 'string' && value.note !== '') parts.push(value.note);
247
271
  parts.push(String(value.content ?? ''));
248
272
  return [{ type: 'text', text: parts.join('\n\n') }];
package/lib/moli.js CHANGED
@@ -44,6 +44,44 @@ const SCROLL_TOKENS = {
44
44
  bottom: 'document.documentElement.scrollHeight',
45
45
  };
46
46
 
47
+ /**
48
+ * Build the expression that returns the text of the elements a CSS selector matches.
49
+ *
50
+ * This exists because reading one container should not cost the whole page: on a
51
+ * node-dense listing page the full markdown dump measured 95,605 characters while
52
+ * the same page's video cards came to 1,829. `--eval` can already do this, but it
53
+ * requires writing the JavaScript; a selector is the same idea with no syntax to get
54
+ * wrong, which is what a model that is not fluent in the DOM needs.
55
+ *
56
+ * It works within one navigation -- the value is returned by the engine's own
57
+ * evaluation, so the "page is gone by the time the result is assembled" limitation
58
+ * does not apply.
59
+ *
60
+ * The result is prefixed with a machine-readable count, because the engine prints an
61
+ * empty string for an expression that yields `""`, and an empty selection is
62
+ * otherwise indistinguishable from a page with no text. A selector that matches
63
+ * nothing is reported as a failure rather than as empty content: the caller picked a
64
+ * selector that is not on the page, and telling them so is the whole point.
65
+ *
66
+ * @param selector - a CSS selector.
67
+ * @returns a JavaScript expression, or undefined when no selector was given.
68
+ */
69
+ export function selectorScript(selector) {
70
+ if (typeof selector !== 'string' || selector.trim() === '') return undefined;
71
+ // JSON.stringify quotes the selector safely for embedding; a selector is data, not
72
+ // code, and must never be spliced into the expression raw.
73
+ const quoted = JSON.stringify(selector);
74
+ return `(() => {
75
+ const nodes = [...document.querySelectorAll(${quoted})];
76
+ const texts = nodes.map(node => (node.innerText ?? node.textContent ?? '').trim()).filter(text => text !== '');
77
+ return '[dsh-moli:selector]' + JSON.stringify({
78
+ matched: nodes.length,
79
+ withText: texts.length,
80
+ joined: texts.join('\\n'),
81
+ });
82
+ })()`;
83
+ }
84
+
47
85
  /**
48
86
  * Build the expression that scrolls the page, to be run by the engine's own
49
87
  * `--eval` before the dump.
@@ -56,6 +94,14 @@ const SCROLL_TOKENS = {
56
94
  * content that "below the fold" already contains; it is about triggering the
57
95
  * handler, the intersection observer, or the lazy image that only runs on a scroll.
58
96
  *
97
+ * **The expression must not return the scroll position.** Without `--dump` the
98
+ * engine prints whatever `--eval` evaluates to, so returning `String(window.scrollY)`
99
+ * here replaced the page with the number `600`: `moli_fetch url=... scroll=600`
100
+ * answered `600`, while the identical call without `scroll` returned the document's
101
+ * HTML. The scroll is a side effect; the value it wants to report is recorded on
102
+ * `window` instead, where a later read can pick it up without changing what the
103
+ * engine prints.
104
+ *
59
105
  * @param scroll - pixels down (negative scrolls up), or `top` / `bottom`.
60
106
  * @returns a JavaScript expression, or undefined when there is nothing to do.
61
107
  */
@@ -74,26 +120,69 @@ export function scrollScript(scroll) {
74
120
  return undefined;
75
121
  }
76
122
  // Dispatch a scroll event as well as moving: `window.scrollTo` updates the
77
- // position, but a page listening for the event is the reason this exists.
78
- return `(() => { window.scrollTo(0, ${target}); window.dispatchEvent(new Event('scroll')); return String(window.scrollY); })()`;
123
+ // position, but a page listening for the event is the reason this exists. The
124
+ // height is captured on both sides of the move so a caller can tell whether the
125
+ // page grew -- which is the one signal that says lazy loading actually happened.
126
+ return `(() => {
127
+ const before = document.body === null ? 0 : document.body.scrollHeight;
128
+ window.scrollTo(0, ${target});
129
+ window.dispatchEvent(new Event('scroll'));
130
+ window.__dshMoliScroll = {
131
+ requested: ${target},
132
+ y: window.scrollY,
133
+ heightBefore: before,
134
+ heightAfter: document.body === null ? 0 : document.body.scrollHeight,
135
+ };
136
+ })()`;
79
137
  }
80
138
 
81
139
  /**
82
- * Combine the caller's `eval` with a scroll, if one was asked for.
140
+ * Build the one expression the engine will run from `scroll`, `selector` and `eval`.
141
+ *
142
+ * The engine accepts a single `--eval`, so everything that has to happen before the
143
+ * page is captured is composed here, in the order the caller expects: scroll first,
144
+ * then read (a selector or the caller's expression).
83
145
  *
84
- * The two have to share one expression because the engine accepts a single
85
- * `--eval`. The scroll goes first so the page is in its final state before the
86
- * caller's expression reads it, and the caller's value still becomes the result.
146
+ * What gets returned depends on whether anything else will produce the output, and
147
+ * getting this wrong breaks the feature in one direction or the other:
87
148
  *
88
- * @returns the expression to pass, or undefined when neither was given.
149
+ * - **A `dump` was requested.** The dump produces the output, so the expression is
150
+ * there only for its side effects and its value is discarded by the engine anyway.
151
+ * It must still be passed, or the scroll never runs.
152
+ * - **No `dump`.** The expression's value *is* the output, whether it is a selector
153
+ * or the caller's own expression.
154
+ * - **No `dump`, no selector, no caller expression, only `scroll`.** The engine
155
+ * prints whatever `--eval` evaluates to, so passing the scroll expression here
156
+ * would replace the page with the scroll's return value. `undefined` is returned
157
+ * instead, leaving the engine's default output -- the serialized document --
158
+ * intact. This is the case that shipped broken: `moli_fetch url=... scroll=600`
159
+ * answered `600`.
160
+ *
161
+ * @param evalExpression - the caller's expression, if any.
162
+ * @param scroll - the caller's `scroll` value, if any.
163
+ * @param hasDump - whether the caller also asked for a dump.
164
+ * @param selector - the caller's CSS selector, if any.
165
+ * @returns the expression to pass, or undefined to let the engine decide.
89
166
  */
90
- export function combineEval(evalExpression, scroll) {
167
+ export function combineEval(evalExpression, scroll, hasDump = false, selector = undefined) {
91
168
  const move = scrollScript(scroll);
92
- if (move === undefined) return evalExpression;
93
- if (evalExpression === undefined || evalExpression === '') return move;
94
- // The caller's expression is kept last and wrapped so its value is the one the
95
- // engine reports. Losing it would contradict the documented `eval` behaviour.
96
- return `${move}; (() => (${evalExpression}))()`;
169
+ const read = selectorScript(selector);
170
+ const caller = evalExpression === undefined || evalExpression === '' ? undefined : evalExpression;
171
+
172
+ // A selector is a read, so it replaces the caller's expression rather than being
173
+ // chained with it; `buildArgs` rejects the two together, so only one is ever set.
174
+ // The selector script is already a call expression; a caller's expression has to be
175
+ // wrapped so its value is what the engine reports.
176
+ const body = read !== undefined ? { code: read, wrapped: true } : (caller === undefined ? undefined : { code: caller, wrapped: false });
177
+
178
+ if (move === undefined) return body === undefined ? undefined : body.code;
179
+ if (body === undefined) {
180
+ // No read at all: keep the scroll only when a dump will produce the output.
181
+ return hasDump ? move : undefined;
182
+ }
183
+ // The read goes last so its value is the one the engine reports; losing it would
184
+ // contradict the documented `eval` behaviour.
185
+ return `${move}; ${body.wrapped ? body.code : `(() => (${body.code}))()`}`;
97
186
  }
98
187
 
99
188
  /**
@@ -332,9 +421,10 @@ export function buildArgs(args) {
332
421
  // Screenshots and PDFs silently produce nothing without the layout renderer.
333
422
  if (needsLayout) argv.push('--layout');
334
423
  if (args.scrollbars === true) argv.push('--scrollbars');
335
- // `scroll` rides along in the same expression: the engine takes a single `--eval`,
336
- // and the scroll has to happen before the page is captured.
337
- const evaluation = combineEval(args.eval, args.scroll);
424
+ // `scroll` and `selector` ride along in the same expression: the engine takes a
425
+ // single `--eval`. `hasDump` matters -- see combineEval -- because with a dump the
426
+ // expression exists only for its side effect, and without one it becomes the output.
427
+ const evaluation = combineEval(args.eval, args.scroll, args.dump !== undefined, args.selector);
338
428
  if (evaluation !== undefined) argv.push('--eval', evaluation);
339
429
  if (args.wait !== undefined && args.wait !== '') argv.push('--wait', args.wait);
340
430
  if (args.waitSelector !== undefined && args.waitSelector !== '') argv.push('--wait-selector', args.waitSelector);
@@ -378,6 +468,30 @@ export function runFetch(options) {
378
468
  const args = options.insecureTls === true && options.args.insecureTls === undefined
379
469
  ? { ...options.args, insecureTls: true }
380
470
  : options.args;
471
+ // `selector` replaces the whole-page output, so combining it with either reader is a
472
+ // call that cannot do what it says: one of the two would be silently ignored. That
473
+ // is the failure mode this plugin keeps having to fix, so it is refused up front and
474
+ // as a structured result, the same way a binary dump with no writeTo is.
475
+ if (typeof args.selector === 'string' && args.selector.trim() !== '') {
476
+ if (args.dump !== undefined) {
477
+ return {
478
+ ok: false,
479
+ url: args.url,
480
+ selector: args.selector,
481
+ error: '`selector` and `dump` cannot be combined: a selector returns one container\'s text and a dump returns the whole page, so one of them could only be ignored',
482
+ hint: 'Use `selector` to read part of the page, or `dump` to read all of it.',
483
+ };
484
+ }
485
+ if (args.eval !== undefined && args.eval !== '') {
486
+ return {
487
+ ok: false,
488
+ url: args.url,
489
+ selector: args.selector,
490
+ error: '`selector` and `eval` cannot be combined: only one expression value can be returned',
491
+ hint: 'Put the extra work in `eval` and drop `selector`, or keep `selector` and read the text it returns.',
492
+ };
493
+ }
494
+ }
381
495
  const argv = buildArgs(args);
382
496
  const timeoutMs = args.timeoutMs ?? 30000;
383
497
  const isBinary = BINARY_DUMPS.has(args.dump);
@@ -561,6 +675,64 @@ export function runFetch(options) {
561
675
  }
562
676
 
563
677
  const text = outcome.stdout.toString('utf8');
678
+ // A selector result comes back as our own sentinel, because an empty selection and
679
+ // a page with no text are both an empty string from the engine -- verified: an
680
+ // expression returning `""` exits 0 with an empty line. Reporting "matched
681
+ // nothing" is the entire value of asking by selector, so it must not be silent.
682
+ if (typeof args.selector === 'string' && args.selector.trim() !== '') {
683
+ const payload = parseSelectorResult(text);
684
+ if (payload === undefined) {
685
+ return {
686
+ ok: false,
687
+ url: args.url,
688
+ selector: args.selector,
689
+ error: `the selector result could not be read: the engine returned ${JSON.stringify(text.slice(0, 120))}`,
690
+ hint: 'The page may not have finished rendering. Try waitSelector, or wait for the content with waitResponseUrl.',
691
+ };
692
+ }
693
+ if (payload.matched === 0) {
694
+ return {
695
+ ok: false,
696
+ url: args.url,
697
+ selector: args.selector,
698
+ matched: 0,
699
+ error: `no element matches ${JSON.stringify(args.selector)}`,
700
+ hint: 'The selector found nothing on this page. Check the spelling, wait for the content to arrive (`waitSelector`, `waitResponseUrl`), or use `dump` to see what the page actually contains.',
701
+ };
702
+ }
703
+ if (payload.withText === 0) {
704
+ return {
705
+ ok: false,
706
+ url: args.url,
707
+ selector: args.selector,
708
+ matched: payload.matched,
709
+ error: `${String(payload.matched)} element(s) match ${JSON.stringify(args.selector)} but none of them has any text`,
710
+ hint: 'The elements are probably images, icons or empty containers. Use `eval` if you need their attributes rather than their text.',
711
+ };
712
+ }
713
+ return {
714
+ ok: true,
715
+ url: args.url,
716
+ kind: 'text',
717
+ selector: args.selector,
718
+ content: payload.joined,
719
+ chars: payload.joined.length,
720
+ matched: payload.matched,
721
+ ...(payload.matched > payload.withText
722
+ ? { note: `${String(payload.matched - payload.withText)} of the ${String(payload.matched)} matched elements had no text and were left out.` }
723
+ : {}),
724
+ };
725
+ }
726
+ // `eval` together with `dump` is the engine's documented behaviour -- it discards
727
+ // the expression's value and dumps the page state the expression produced -- but
728
+ // it reads as a successful answer to a question that was never answered. The
729
+ // caller asked for `{probe:1}` and got 900 characters of Markdown. The discard is
730
+ // kept, because mutating a page before capturing it is a real use, so the result
731
+ // has to say what happened instead of looking clean.
732
+ const evalDiscarded = args.eval !== undefined && args.eval !== '' && args.dump !== undefined;
733
+ const evalNote = evalDiscarded
734
+ ? 'The `eval` expression produced no value here: `dump` was given too, and the engine discards the expression result in that combination. The output above is the page, not the expression. Run again with `eval` alone for the value, or keep both deliberately when the expression only changes the page before it is captured.'
735
+ : undefined;
564
736
  // A JSON dump is the only format that reports the HTTP status, so an error
565
737
  // page is otherwise indistinguishable from content. Surface it rather than
566
738
  // letting the model read a 404 body as if the fetch succeeded.
@@ -591,7 +763,10 @@ export function runFetch(options) {
591
763
  path: args.writeTo,
592
764
  chars: text.length,
593
765
  bytes,
594
- note: 'Full output saved to file; read it for the complete text.',
766
+ note: evalNote === undefined
767
+ ? 'Full output saved to file; read it for the complete text.'
768
+ : `Full output saved to file; read it for the complete text. ${evalNote}`,
769
+ ...(evalDiscarded ? { evalDiscarded: true } : {}),
595
770
  };
596
771
  }
597
772
 
@@ -613,10 +788,21 @@ export function runFetch(options) {
613
788
  const headLen = Math.ceil(budget * 0.6);
614
789
  const tailLen = budget - headLen;
615
790
  omitted = text.length - budget;
616
- const mark = `\n\n… [dsh-moli] ${String(omitted)} characters omitted here; the head and tail are kept. Pass writeTo to keep all ${String(text.length)}. …\n\n`;
791
+ // The marker inside the body is only a position signal: where the gap is and how
792
+ // big it is. Everything actionable -- the head/tail split, the total, the writeTo
793
+ // escape -- lives in the note, which is rendered once at the top. Both used to
794
+ // say the same thing, and on a 2,000-character budget the two copies cost 12.6%
795
+ // of everything the model received.
796
+ const mark = `\n\n… [dsh-moli] ${String(omitted)} characters omitted …\n\n`;
617
797
  content = text.slice(0, headLen) + mark + text.slice(text.length - tailLen);
618
- note = `Output truncated: showing the first ${String(headLen)} and the last ${String(tailLen)} of ${String(text.length)} characters, with ${String(omitted)} omitted in the middle. Pass writeTo to keep all ${String(text.length)}.`;
798
+ note = `Truncated to fit maxChars: the first ${String(headLen)} and the last ${String(tailLen)} of ${String(text.length)} characters are shown, with ${String(omitted)} omitted in the middle. Pass writeTo to keep all ${String(text.length)}.`;
619
799
  }
800
+ // `eval` together with `dump` is the engine's documented behaviour -- it discards
801
+ // the expression's value and dumps the page state the expression produced -- but
802
+ // it reads as a successful answer to a question that was never answered. The
803
+ // caller asked for `{probe:1}` and got 900 characters of Markdown. The value is
804
+ // still discarded (that combination is genuinely useful for mutating a page before
805
+ // capturing it), so the result has to say so rather than look clean.
620
806
  return {
621
807
  ok: true,
622
808
  url: args.url,
@@ -626,10 +812,43 @@ export function runFetch(options) {
626
812
  chars: text.length,
627
813
  bytes,
628
814
  truncated,
629
- ...(truncated ? { omitted, note } : {}),
815
+ ...(truncated ? { omitted } : {}),
816
+ ...(evalDiscarded ? { evalDiscarded: true } : {}),
817
+ ...(note === undefined && evalNote === undefined
818
+ ? {}
819
+ : { note: [note, evalNote].filter(part => part !== undefined).join(' ') }),
630
820
  };
631
821
  }
632
822
 
823
+ /**
824
+ * Read the payload {@link selectorScript} prefixes onto its result.
825
+ *
826
+ * The engine prints the expression value verbatim, so the script adds a sentinel to
827
+ * tell a selector answer apart from page text. Anything without the sentinel is not
828
+ * a selector answer and comes back as undefined, which the caller reports as a
829
+ * failure rather than dressing it up as content.
830
+ *
831
+ * @param text - the engine's stdout.
832
+ * @returns `{ matched, withText, joined }`, or undefined.
833
+ */
834
+ function parseSelectorResult(text) {
835
+ const marker = '[dsh-moli:selector]';
836
+ const at = String(text).indexOf(marker);
837
+ if (at < 0) return undefined;
838
+ try {
839
+ const raw = JSON.parse(String(text).slice(at + marker.length).trim());
840
+ if (raw === null || typeof raw !== 'object') return undefined;
841
+ if (typeof raw.matched !== 'number' || typeof raw.joined !== 'string') return undefined;
842
+ return {
843
+ matched: raw.matched,
844
+ withText: typeof raw.withText === 'number' ? raw.withText : 0,
845
+ joined: raw.joined,
846
+ };
847
+ } catch {
848
+ return undefined;
849
+ }
850
+ }
851
+
633
852
  /**
634
853
  * Read the HTTP status out of the engine's stderr reason.
635
854
  *
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-moli",
3
- "version": "1.1.15",
3
+ "version": "1.1.17",
4
4
  "type": "module",
5
5
  "description": "DeepSeek Harness tools for the local Moli headless browser: rendered fetches, markdown, semantic trees, screenshots, and PDFs.",
6
6
  "license": "MIT",