dsh-moli 1.1.16 → 1.1.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -1
- package/README.zh.md +31 -4
- package/index.js +22 -5
- package/lib/moli.js +183 -28
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -209,7 +209,7 @@ An unresolvable bundle is **skipped silently** apart from one stderr line
|
|
|
209
209
|
|
|
210
210
|
## Usage
|
|
211
211
|
|
|
212
|
-
`moli_fetch` parameters: `url` (required), `dump`, `writeTo`, `eval`, `scroll`, `wait`,
|
|
212
|
+
`moli_fetch` parameters: `url` (required), `dump`, `writeTo`, `selector`, `eval`, `scroll`, `wait`,
|
|
213
213
|
`waitSelector`, `waitScript`, `waitResponseUrl`, `waitResponseJson`, `delayMs`, `timeoutMs`,
|
|
214
214
|
`maxChars`, `headers`, `userAgent`, `layout`, `scrollbars`, `noJs`, `noCss`, `resources`,
|
|
215
215
|
`withFrames`, `insecureTls`, `failOnHttpError`, `profileDir`, `cookieFile`, `proxy`.
|
|
@@ -218,6 +218,7 @@ An unresolvable bundle is **skipped silently** apart from one stderr line
|
|
|
218
218
|
moli_fetch url=https://example.com dump=markdown # token-efficient text
|
|
219
219
|
moli_fetch url=https://example.com dump=semantic_tree_text # nodes with backend ids
|
|
220
220
|
moli_fetch url=https://example.com eval="document.title" # one computed value
|
|
221
|
+
moli_fetch url=https://example.com selector=".video-card" # one container, not the page
|
|
221
222
|
moli_fetch url=https://example.com dump=screenshot writeTo=/abs/path/a.png
|
|
222
223
|
moli_fetch url=https://example.com dump=pdf writeTo=/abs/path/a.pdf
|
|
223
224
|
moli_fetch url=https://example.com dump=markdown scroll=800 # trigger lazy loading first
|
|
@@ -225,6 +226,35 @@ moli_fetch url=https://example.com dump=markdown scroll=800 # trigger lazy loa
|
|
|
225
226
|
|
|
226
227
|
Screenshot and PDF dumps need `writeTo`; the bytes are written verbatim rather than printed.
|
|
227
228
|
|
|
229
|
+
### `selector` — read one container, not the page
|
|
230
|
+
|
|
231
|
+
A listing page is mostly noise if you only want the items. `selector` returns the text
|
|
232
|
+
of the elements a CSS selector matches, which on a node-dense feed measured **1,829
|
|
233
|
+
characters against 95,605** for the same page as Markdown.
|
|
234
|
+
|
|
235
|
+
```
|
|
236
|
+
moli_fetch url=https://example.com/feed selector=".video-card"
|
|
237
|
+
moli_fetch url=https://example.com/feed selector="h2" scroll=2000 # scroll first, then read
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
All matches are joined with newlines and the result reports how many there were:
|
|
241
|
+
"matched 48 element(s)" is the difference between a complete read and a partial one,
|
|
242
|
+
and it is the only signal you get that the selector was too broad or too narrow.
|
|
243
|
+
|
|
244
|
+
Two cases are deliberately errors rather than empty answers, because the engine
|
|
245
|
+
cannot tell an empty selection from a page with no text — an expression returning
|
|
246
|
+
`""` exits 0 with a blank line:
|
|
247
|
+
|
|
248
|
+
- **nothing matched** — `ok: false`, "no element matches". A selector that is not on
|
|
249
|
+
the page is almost always a spelling mistake or a page that has not loaded yet.
|
|
250
|
+
- **matched, but no text** — `ok: false`, "match but none of them has any text", which
|
|
251
|
+
is what an icon or an image container gives you. Use `eval` if you need attributes.
|
|
252
|
+
|
|
253
|
+
`selector` cannot be combined with `dump` or `eval`: a selector returns one
|
|
254
|
+
container's text, a dump returns the whole page, and only one expression value can be
|
|
255
|
+
returned — so one of them could only be silently ignored, which is the failure mode
|
|
256
|
+
this plugin keeps having to fix.
|
|
257
|
+
|
|
228
258
|
### `scroll` — reach what only appears on a scroll
|
|
229
259
|
|
|
230
260
|
The engine serializes the whole document, so a plain fetch already contains everything the DOM holds.
|
|
@@ -489,6 +519,11 @@ Eight suites live in [`verify/`](verify/), and all are runnable against a workin
|
|
|
489
519
|
`bottom`, nonsense rejected, the caller's `eval` value preserved) and then, when an engine is
|
|
490
520
|
present, fetches a page whose `scroll` handler fills a slot and asserts the content appears with
|
|
491
521
|
`scroll` and the placeholder appears without it — a property of the output, not of an internal call.
|
|
522
|
+
- `selector-verify.mjs` covers the two silent-wrong-answer cases a container read invites: a selector
|
|
523
|
+
matching nothing is an error rather than empty content, and a selector matching several elements
|
|
524
|
+
reports the count and joins them. It also asserts that the selector is quoted as data rather than
|
|
525
|
+
spliced into the expression, that `selector` with `dump` or `eval` is refused instead of one of them
|
|
526
|
+
being ignored, and that it composes with `scroll`.
|
|
492
527
|
- `http-verify.mjs` asserts the parameter-to-flag mapping directly, because a parameter that never
|
|
493
528
|
reaches the engine is indistinguishable from a feature that does not work. It then fetches a 404 and
|
|
494
529
|
checks that `failOnHttpError` reports it while keeping the body, that a healthy page is unaffected,
|
package/README.zh.md
CHANGED
|
@@ -195,15 +195,16 @@ profile 插件需要两半都齐:依赖边(`dependencies`)+ `dsh.profile.b
|
|
|
195
195
|
|
|
196
196
|
## 用法
|
|
197
197
|
|
|
198
|
-
`moli_fetch` 参数:`url`(必填)、`dump`、`writeTo`、`eval`、`scroll`、`wait
|
|
199
|
-
`waitScript`、`waitResponseUrl`、`waitResponseJson`、`delayMs`、`timeoutMs
|
|
200
|
-
`userAgent`、`layout`、`scrollbars`、`noJs`、`noCss`、`resources
|
|
201
|
-
`failOnHttpError`、`profileDir`、`cookieFile`、`proxy`。
|
|
198
|
+
`moli_fetch` 参数:`url`(必填)、`dump`、`writeTo`、`selector`、`eval`、`scroll`、`wait`、
|
|
199
|
+
`waitSelector`、`waitScript`、`waitResponseUrl`、`waitResponseJson`、`delayMs`、`timeoutMs`、
|
|
200
|
+
`maxChars`、`headers`、`userAgent`、`layout`、`scrollbars`、`noJs`、`noCss`、`resources`、
|
|
201
|
+
`withFrames`、`insecureTls`、`failOnHttpError`、`profileDir`、`cookieFile`、`proxy`。
|
|
202
202
|
|
|
203
203
|
```
|
|
204
204
|
moli_fetch url=https://example.com dump=markdown # 省 token 的正文
|
|
205
205
|
moli_fetch url=https://example.com dump=semantic_tree_text # 带节点 id 的结构
|
|
206
206
|
moli_fetch url=https://example.com eval="document.title" # 取一个值
|
|
207
|
+
moli_fetch url=https://example.com selector=".video-card" # 只取一个容器,不要整页
|
|
207
208
|
moli_fetch url=https://example.com dump=screenshot writeTo=/绝对路径/a.png
|
|
208
209
|
moli_fetch url=https://example.com dump=pdf writeTo=/绝对路径/a.pdf
|
|
209
210
|
moli_fetch url=https://example.com dump=markdown scroll=800 # 先触发懒加载再抓取
|
|
@@ -211,6 +212,29 @@ moli_fetch url=https://example.com dump=markdown scroll=800 # 先触发懒加
|
|
|
211
212
|
|
|
212
213
|
截图与 PDF 必须给 `writeTo`;字节是原样落盘的,不会打印到输出里。
|
|
213
214
|
|
|
215
|
+
### `selector` —— 只取一个容器,不要整页
|
|
216
|
+
|
|
217
|
+
列表页里如果只要条目,正文大部分都是噪声。`selector` 返回 CSS 选择器匹配到的元素文本——在一个节点密集的
|
|
218
|
+
信息流上实测**1,829 字符,而同一页面走 markdown 是 95,605**。
|
|
219
|
+
|
|
220
|
+
```
|
|
221
|
+
moli_fetch url=https://example.com/feed selector=".video-card"
|
|
222
|
+
moli_fetch url=https://example.com/feed selector="h2" scroll=2000 # 先滚再取
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
所有匹配项以换行拼接,并且结果会报告匹配到多少个:**"matched 48 element(s)"** 是"读全了"和"只读到一部分"
|
|
226
|
+
的区别,也是你判断选择器太宽或太窄的唯一信号。
|
|
227
|
+
|
|
228
|
+
两种情况**刻意做成错误而不是空答案**,因为引擎分不清"没匹配到"和"页面没有文本"——表达式返回 `""` 时是
|
|
229
|
+
exit 0 加一个空行:
|
|
230
|
+
|
|
231
|
+
- **没有任何匹配** —— `ok: false`,"no element matches"。页面上不存在的选择器几乎总是拼写错误或页面还没加载。
|
|
232
|
+
- **匹配到了但没有文本** —— `ok: false`,说明匹配到了但都没文本,这通常会发生在图标或图片容器上。需要属性就
|
|
233
|
+
用 `eval`。
|
|
234
|
+
|
|
235
|
+
`selector` 不能与 `dump` 或 `eval` 同传:选择器返回一个容器的文本、dump 返回整页、而一次只能返回一个表达式
|
|
236
|
+
值——所以其中一个只能被**静默忽略**,而这正是这个插件一路在修的那类失败。
|
|
237
|
+
|
|
214
238
|
### `scroll` —— 拿到"只有滚动后才出现"的内容
|
|
215
239
|
|
|
216
240
|
引擎会序列化整个文档,所以普通抓取**已经包含 DOM 里的一切**。它拿不到的是**尚未存在的内容**:首次
|
|
@@ -444,6 +468,9 @@ profile。无论生效的是哪个代理,loopback 地址一律绕过它。
|
|
|
444
468
|
- `scroll-verify.mjs` 在无引擎时校验表达式构造(像素值、负数、`top`、`bottom`、无效输入被拒、调用方的 `eval`
|
|
445
469
|
值被保留),有引擎时抓取一个"滚动处理器填充槽位"的页面,断言带 `scroll` 时内容出现、不带时只有占位符——
|
|
446
470
|
断言的是**输出**的性质,而不是内部调用。
|
|
471
|
+
- `selector-verify.mjs` 覆盖"按容器取值"最容易出现的两个静默错答案:选择器无匹配时报错而不是空内容、匹配多个
|
|
472
|
+
元素时报告数量并拼接。它还断言选择器是**作为数据被引号包裹**而不是拼进表达式、`selector` 与 `dump`/`eval`
|
|
473
|
+
同传会被拒绝(而不是静默忽略其中一个)、以及它能与 `scroll` 组合。
|
|
447
474
|
- `http-verify.mjs` 直接断言"参数 → 引擎选项"的映射,因为**参数没送到引擎和功能不生效,从外面看是一样的**。
|
|
448
475
|
随后抓一个 404,验证 `failOnHttpError` 会报告失败同时保留响应体、健康页面不受影响、以及错误页的截图仍然被
|
|
449
476
|
写为合法 PNG 而不是被当文本解码。它还覆盖那两个"静默给错答案"的情形:`eval` 配 `dump` 会带
|
package/index.js
CHANGED
|
@@ -94,12 +94,16 @@ const FETCH_PARAMETERS = {
|
|
|
94
94
|
dump: {
|
|
95
95
|
type: 'string',
|
|
96
96
|
enum: ['html', 'markdown', 'json', 'semantic_tree', 'semantic_tree_text', 'screenshot', 'screenshot_full', 'pdf'],
|
|
97
|
-
description: 'Output format. Omit for rendered HTML. `markdown`
|
|
97
|
+
description: 'Output format. Omit for rendered HTML. `markdown` is the cheapest way to read a page. `semantic_tree_text` is for locating things by structure (role, name, backend id), not for saving tokens: its size follows the page\'s node density and it is often larger than markdown — measured at 1.8x on a node-dense page — so prefer it only when you need the structure. `json` carries status and final_url; screenshot/pdf need writeTo.',
|
|
98
98
|
},
|
|
99
99
|
writeTo: {
|
|
100
100
|
type: 'string',
|
|
101
101
|
description: 'Absolute path to save the output. Required for screenshot/pdf; optional for text dumps larger than maxChars.',
|
|
102
102
|
},
|
|
103
|
+
selector: {
|
|
104
|
+
type: 'string',
|
|
105
|
+
description: 'Return the text of the elements this CSS selector matches instead of the whole page. This is the cheap way to read one container out of a noisy listing page: the cards on a feed measured 1,829 characters against 95,605 for the same page as markdown. All matches are joined with newlines and the result reports how many there were; matching nothing is an error, not empty content, because the engine cannot distinguish an empty selection from a page with no text. Not compatible with `dump` or `eval` — one of them could only be ignored.',
|
|
106
|
+
},
|
|
103
107
|
eval: {
|
|
104
108
|
type: 'string',
|
|
105
109
|
description: 'JavaScript expression evaluated once the page is ready; its value is returned as text or JSON instead of the page. Combining it with `dump` does NOT return both: the engine discards the expression value and dumps the resulting page state instead (its own documented semantics). Use `eval` alone to read a value, or `dump` alone to read the page.',
|
|
@@ -175,6 +179,8 @@ const FETCH_OUTPUT_SCHEMA = {
|
|
|
175
179
|
omitted: { type: 'integer' },
|
|
176
180
|
evalFailed: { type: 'boolean' },
|
|
177
181
|
evalDiscarded: { type: 'boolean' },
|
|
182
|
+
selector: { type: 'string' },
|
|
183
|
+
matched: { type: 'integer' },
|
|
178
184
|
truncated: { type: 'boolean' },
|
|
179
185
|
error: { type: 'string' },
|
|
180
186
|
hint: { type: 'string' },
|
|
@@ -227,13 +233,18 @@ function renderFetch(_args, value) {
|
|
|
227
233
|
if (value.unreachable === true) lines.push('Cause: the host did not answer; a local forward proxy is the usual fix on this network.');
|
|
228
234
|
else if (value.possiblyUnreachable === true) lines.push('Cause: the host did not answer in time; it may be blocked, or the page may simply be slow.');
|
|
229
235
|
if (typeof value.hint === 'string' && value.hint !== '') lines.push(`Next: ${value.hint}`);
|
|
230
|
-
|
|
236
|
+
// An HTTP error keeps its body under `failOnHttpError`. Reporting the failure
|
|
231
237
|
// without the body would throw away the error page, which is the part worth
|
|
232
|
-
// reading -- and for a binary dump the body is the file that was written.
|
|
233
|
-
|
|
238
|
+
// reading -- and for a binary dump the body is the file that was written. A
|
|
239
|
+
// selector failure is different: there is no server body to show, and the
|
|
240
|
+
// message already says what did not match.
|
|
241
|
+
if (value.httpError === true && value.kind === 'binary' && typeof value.path === 'string') {
|
|
234
242
|
lines.push(`The error page was still written to ${value.path} (${String(value.bytes)} bytes).`);
|
|
235
|
-
} else if (typeof value.content === 'string' && value.content !== '') {
|
|
243
|
+
} else if (value.httpError === true && typeof value.content === 'string' && value.content !== '') {
|
|
236
244
|
lines.push('', 'The server\'s response body:', '', value.content);
|
|
245
|
+
} else if (typeof value.selector === 'string') {
|
|
246
|
+
lines.push(`Selector: ${JSON.stringify(value.selector)}`
|
|
247
|
+
+ (typeof value.matched === 'number' ? ` (matched ${String(value.matched)})` : ''));
|
|
237
248
|
}
|
|
238
249
|
return [{ type: 'text', text: lines.join('\n') }];
|
|
239
250
|
}
|
|
@@ -250,6 +261,12 @@ function renderFetch(_args, value) {
|
|
|
250
261
|
if (value.evalDiscarded === true) {
|
|
251
262
|
parts.push('Note: the `eval` expression was dropped because `dump` was given too — the engine discards the expression value in that combination. What follows is the page, not the expression.');
|
|
252
263
|
}
|
|
264
|
+
// A selector answer is a list, so say how many elements produced it: "48 matched,
|
|
265
|
+
// 34 had text" is the difference between a complete read and a partial one, and it
|
|
266
|
+
// is the only signal the caller has that the selector was too broad or too narrow.
|
|
267
|
+
if (typeof value.selector === 'string' && typeof value.matched === 'number') {
|
|
268
|
+
parts.push(`Selector ${JSON.stringify(value.selector)} matched ${String(value.matched)} element(s).`);
|
|
269
|
+
}
|
|
253
270
|
if (typeof value.note === 'string' && value.note !== '') parts.push(value.note);
|
|
254
271
|
parts.push(String(value.content ?? ''));
|
|
255
272
|
return [{ type: 'text', text: parts.join('\n\n') }];
|
package/lib/moli.js
CHANGED
|
@@ -44,6 +44,44 @@ const SCROLL_TOKENS = {
|
|
|
44
44
|
bottom: 'document.documentElement.scrollHeight',
|
|
45
45
|
};
|
|
46
46
|
|
|
47
|
+
/**
|
|
48
|
+
* Build the expression that returns the text of the elements a CSS selector matches.
|
|
49
|
+
*
|
|
50
|
+
* This exists because reading one container should not cost the whole page: on a
|
|
51
|
+
* node-dense listing page the full markdown dump measured 95,605 characters while
|
|
52
|
+
* the same page's video cards came to 1,829. `--eval` can already do this, but it
|
|
53
|
+
* requires writing the JavaScript; a selector is the same idea with no syntax to get
|
|
54
|
+
* wrong, which is what a model that is not fluent in the DOM needs.
|
|
55
|
+
*
|
|
56
|
+
* It works within one navigation -- the value is returned by the engine's own
|
|
57
|
+
* evaluation, so the "page is gone by the time the result is assembled" limitation
|
|
58
|
+
* does not apply.
|
|
59
|
+
*
|
|
60
|
+
* The result is prefixed with a machine-readable count, because the engine prints an
|
|
61
|
+
* empty string for an expression that yields `""`, and an empty selection is
|
|
62
|
+
* otherwise indistinguishable from a page with no text. A selector that matches
|
|
63
|
+
* nothing is reported as a failure rather than as empty content: the caller picked a
|
|
64
|
+
* selector that is not on the page, and telling them so is the whole point.
|
|
65
|
+
*
|
|
66
|
+
* @param selector - a CSS selector.
|
|
67
|
+
* @returns a JavaScript expression, or undefined when no selector was given.
|
|
68
|
+
*/
|
|
69
|
+
export function selectorScript(selector) {
|
|
70
|
+
if (typeof selector !== 'string' || selector.trim() === '') return undefined;
|
|
71
|
+
// JSON.stringify quotes the selector safely for embedding; a selector is data, not
|
|
72
|
+
// code, and must never be spliced into the expression raw.
|
|
73
|
+
const quoted = JSON.stringify(selector);
|
|
74
|
+
return `(() => {
|
|
75
|
+
const nodes = [...document.querySelectorAll(${quoted})];
|
|
76
|
+
const texts = nodes.map(node => (node.innerText ?? node.textContent ?? '').trim()).filter(text => text !== '');
|
|
77
|
+
return '[dsh-moli:selector]' + JSON.stringify({
|
|
78
|
+
matched: nodes.length,
|
|
79
|
+
withText: texts.length,
|
|
80
|
+
joined: texts.join('\\n'),
|
|
81
|
+
});
|
|
82
|
+
})()`;
|
|
83
|
+
}
|
|
84
|
+
|
|
47
85
|
/**
|
|
48
86
|
* Build the expression that scrolls the page, to be run by the engine's own
|
|
49
87
|
* `--eval` before the dump.
|
|
@@ -99,38 +137,52 @@ export function scrollScript(scroll) {
|
|
|
99
137
|
}
|
|
100
138
|
|
|
101
139
|
/**
|
|
102
|
-
*
|
|
140
|
+
* Build the one expression the engine will run from `scroll`, `selector` and `eval`.
|
|
103
141
|
*
|
|
104
|
-
* The
|
|
142
|
+
* The engine accepts a single `--eval`, so everything that has to happen before the
|
|
143
|
+
* page is captured is composed here, in the order the caller expects: scroll first,
|
|
144
|
+
* then read (a selector or the caller's expression).
|
|
105
145
|
*
|
|
106
|
-
* What gets returned depends on whether anything else will produce the output,
|
|
107
|
-
*
|
|
146
|
+
* What gets returned depends on whether anything else will produce the output, and
|
|
147
|
+
* getting this wrong breaks the feature in one direction or the other:
|
|
108
148
|
*
|
|
109
149
|
* - **A `dump` was requested.** The dump produces the output, so the expression is
|
|
110
|
-
* there only
|
|
150
|
+
* there only for its side effects and its value is discarded by the engine anyway.
|
|
111
151
|
* It must still be passed, or the scroll never runs.
|
|
112
|
-
* - **No `dump
|
|
113
|
-
*
|
|
114
|
-
* - **No `dump`, no caller expression
|
|
115
|
-
* evaluates to, so passing the scroll expression here
|
|
116
|
-
* the scroll's return value. `undefined` is returned
|
|
117
|
-
* default output -- the serialized document --
|
|
118
|
-
* shipped broken: `moli_fetch url=... scroll=600`
|
|
152
|
+
* - **No `dump`.** The expression's value *is* the output, whether it is a selector
|
|
153
|
+
* or the caller's own expression.
|
|
154
|
+
* - **No `dump`, no selector, no caller expression, only `scroll`.** The engine
|
|
155
|
+
* prints whatever `--eval` evaluates to, so passing the scroll expression here
|
|
156
|
+
* would replace the page with the scroll's return value. `undefined` is returned
|
|
157
|
+
* instead, leaving the engine's default output -- the serialized document --
|
|
158
|
+
* intact. This is the case that shipped broken: `moli_fetch url=... scroll=600`
|
|
159
|
+
* answered `600`.
|
|
119
160
|
*
|
|
120
161
|
* @param evalExpression - the caller's expression, if any.
|
|
121
162
|
* @param scroll - the caller's `scroll` value, if any.
|
|
122
163
|
* @param hasDump - whether the caller also asked for a dump.
|
|
164
|
+
* @param selector - the caller's CSS selector, if any.
|
|
123
165
|
* @returns the expression to pass, or undefined to let the engine decide.
|
|
124
166
|
*/
|
|
125
|
-
export function combineEval(evalExpression, scroll, hasDump = false) {
|
|
167
|
+
export function combineEval(evalExpression, scroll, hasDump = false, selector = undefined) {
|
|
126
168
|
const move = scrollScript(scroll);
|
|
169
|
+
const read = selectorScript(selector);
|
|
127
170
|
const caller = evalExpression === undefined || evalExpression === '' ? undefined : evalExpression;
|
|
128
|
-
|
|
129
|
-
//
|
|
130
|
-
//
|
|
131
|
-
|
|
132
|
-
//
|
|
133
|
-
|
|
171
|
+
|
|
172
|
+
// A selector is a read, so it replaces the caller's expression rather than being
|
|
173
|
+
// chained with it; `buildArgs` rejects the two together, so only one is ever set.
|
|
174
|
+
// The selector script is already a call expression; a caller's expression has to be
|
|
175
|
+
// wrapped so its value is what the engine reports.
|
|
176
|
+
const body = read !== undefined ? { code: read, wrapped: true } : (caller === undefined ? undefined : { code: caller, wrapped: false });
|
|
177
|
+
|
|
178
|
+
if (move === undefined) return body === undefined ? undefined : body.code;
|
|
179
|
+
if (body === undefined) {
|
|
180
|
+
// No read at all: keep the scroll only when a dump will produce the output.
|
|
181
|
+
return hasDump ? move : undefined;
|
|
182
|
+
}
|
|
183
|
+
// The read goes last so its value is the one the engine reports; losing it would
|
|
184
|
+
// contradict the documented `eval` behaviour.
|
|
185
|
+
return `${move}; ${body.wrapped ? body.code : `(() => (${body.code}))()`}`;
|
|
134
186
|
}
|
|
135
187
|
|
|
136
188
|
/**
|
|
@@ -369,13 +421,10 @@ export function buildArgs(args) {
|
|
|
369
421
|
// Screenshots and PDFs silently produce nothing without the layout renderer.
|
|
370
422
|
if (needsLayout) argv.push('--layout');
|
|
371
423
|
if (args.scrollbars === true) argv.push('--scrollbars');
|
|
372
|
-
// `scroll`
|
|
373
|
-
//
|
|
374
|
-
//
|
|
375
|
-
|
|
376
|
-
// see combineEval -- because with a dump the expression exists only for its side
|
|
377
|
-
// effect, and without one it would become the output.
|
|
378
|
-
const evaluation = combineEval(args.eval, args.scroll, args.dump !== undefined);
|
|
424
|
+
// `scroll` and `selector` ride along in the same expression: the engine takes a
|
|
425
|
+
// single `--eval`. `hasDump` matters -- see combineEval -- because with a dump the
|
|
426
|
+
// expression exists only for its side effect, and without one it becomes the output.
|
|
427
|
+
const evaluation = combineEval(args.eval, args.scroll, args.dump !== undefined, args.selector);
|
|
379
428
|
if (evaluation !== undefined) argv.push('--eval', evaluation);
|
|
380
429
|
if (args.wait !== undefined && args.wait !== '') argv.push('--wait', args.wait);
|
|
381
430
|
if (args.waitSelector !== undefined && args.waitSelector !== '') argv.push('--wait-selector', args.waitSelector);
|
|
@@ -419,6 +468,30 @@ export function runFetch(options) {
|
|
|
419
468
|
const args = options.insecureTls === true && options.args.insecureTls === undefined
|
|
420
469
|
? { ...options.args, insecureTls: true }
|
|
421
470
|
: options.args;
|
|
471
|
+
// `selector` replaces the whole-page output, so combining it with either reader is a
|
|
472
|
+
// call that cannot do what it says: one of the two would be silently ignored. That
|
|
473
|
+
// is the failure mode this plugin keeps having to fix, so it is refused up front and
|
|
474
|
+
// as a structured result, the same way a binary dump with no writeTo is.
|
|
475
|
+
if (typeof args.selector === 'string' && args.selector.trim() !== '') {
|
|
476
|
+
if (args.dump !== undefined) {
|
|
477
|
+
return {
|
|
478
|
+
ok: false,
|
|
479
|
+
url: args.url,
|
|
480
|
+
selector: args.selector,
|
|
481
|
+
error: '`selector` and `dump` cannot be combined: a selector returns one container\'s text and a dump returns the whole page, so one of them could only be ignored',
|
|
482
|
+
hint: 'Use `selector` to read part of the page, or `dump` to read all of it.',
|
|
483
|
+
};
|
|
484
|
+
}
|
|
485
|
+
if (args.eval !== undefined && args.eval !== '') {
|
|
486
|
+
return {
|
|
487
|
+
ok: false,
|
|
488
|
+
url: args.url,
|
|
489
|
+
selector: args.selector,
|
|
490
|
+
error: '`selector` and `eval` cannot be combined: only one expression value can be returned',
|
|
491
|
+
hint: 'Put the extra work in `eval` and drop `selector`, or keep `selector` and read the text it returns.',
|
|
492
|
+
};
|
|
493
|
+
}
|
|
494
|
+
}
|
|
422
495
|
const argv = buildArgs(args);
|
|
423
496
|
const timeoutMs = args.timeoutMs ?? 30000;
|
|
424
497
|
const isBinary = BINARY_DUMPS.has(args.dump);
|
|
@@ -602,6 +675,54 @@ export function runFetch(options) {
|
|
|
602
675
|
}
|
|
603
676
|
|
|
604
677
|
const text = outcome.stdout.toString('utf8');
|
|
678
|
+
// A selector result comes back as our own sentinel, because an empty selection and
|
|
679
|
+
// a page with no text are both an empty string from the engine -- verified: an
|
|
680
|
+
// expression returning `""` exits 0 with an empty line. Reporting "matched
|
|
681
|
+
// nothing" is the entire value of asking by selector, so it must not be silent.
|
|
682
|
+
if (typeof args.selector === 'string' && args.selector.trim() !== '') {
|
|
683
|
+
const payload = parseSelectorResult(text);
|
|
684
|
+
if (payload === undefined) {
|
|
685
|
+
return {
|
|
686
|
+
ok: false,
|
|
687
|
+
url: args.url,
|
|
688
|
+
selector: args.selector,
|
|
689
|
+
error: `the selector result could not be read: the engine returned ${JSON.stringify(text.slice(0, 120))}`,
|
|
690
|
+
hint: 'The page may not have finished rendering. Try waitSelector, or wait for the content with waitResponseUrl.',
|
|
691
|
+
};
|
|
692
|
+
}
|
|
693
|
+
if (payload.matched === 0) {
|
|
694
|
+
return {
|
|
695
|
+
ok: false,
|
|
696
|
+
url: args.url,
|
|
697
|
+
selector: args.selector,
|
|
698
|
+
matched: 0,
|
|
699
|
+
error: `no element matches ${JSON.stringify(args.selector)}`,
|
|
700
|
+
hint: 'The selector found nothing on this page. Check the spelling, wait for the content to arrive (`waitSelector`, `waitResponseUrl`), or use `dump` to see what the page actually contains.',
|
|
701
|
+
};
|
|
702
|
+
}
|
|
703
|
+
if (payload.withText === 0) {
|
|
704
|
+
return {
|
|
705
|
+
ok: false,
|
|
706
|
+
url: args.url,
|
|
707
|
+
selector: args.selector,
|
|
708
|
+
matched: payload.matched,
|
|
709
|
+
error: `${String(payload.matched)} element(s) match ${JSON.stringify(args.selector)} but none of them has any text`,
|
|
710
|
+
hint: 'The elements are probably images, icons or empty containers. Use `eval` if you need their attributes rather than their text.',
|
|
711
|
+
};
|
|
712
|
+
}
|
|
713
|
+
return {
|
|
714
|
+
ok: true,
|
|
715
|
+
url: args.url,
|
|
716
|
+
kind: 'text',
|
|
717
|
+
selector: args.selector,
|
|
718
|
+
content: payload.joined,
|
|
719
|
+
chars: payload.joined.length,
|
|
720
|
+
matched: payload.matched,
|
|
721
|
+
...(payload.matched > payload.withText
|
|
722
|
+
? { note: `${String(payload.matched - payload.withText)} of the ${String(payload.matched)} matched elements had no text and were left out.` }
|
|
723
|
+
: {}),
|
|
724
|
+
};
|
|
725
|
+
}
|
|
605
726
|
// `eval` together with `dump` is the engine's documented behaviour -- it discards
|
|
606
727
|
// the expression's value and dumps the page state the expression produced -- but
|
|
607
728
|
// it reads as a successful answer to a question that was never answered. The
|
|
@@ -667,9 +788,14 @@ export function runFetch(options) {
|
|
|
667
788
|
const headLen = Math.ceil(budget * 0.6);
|
|
668
789
|
const tailLen = budget - headLen;
|
|
669
790
|
omitted = text.length - budget;
|
|
670
|
-
|
|
791
|
+
// The marker inside the body is only a position signal: where the gap is and how
|
|
792
|
+
// big it is. Everything actionable -- the head/tail split, the total, the writeTo
|
|
793
|
+
// escape -- lives in the note, which is rendered once at the top. Both used to
|
|
794
|
+
// say the same thing, and on a 2,000-character budget the two copies cost 12.6%
|
|
795
|
+
// of everything the model received.
|
|
796
|
+
const mark = `\n\n… [dsh-moli] ${String(omitted)} characters omitted …\n\n`;
|
|
671
797
|
content = text.slice(0, headLen) + mark + text.slice(text.length - tailLen);
|
|
672
|
-
note = `
|
|
798
|
+
note = `Truncated to fit maxChars: the first ${String(headLen)} and the last ${String(tailLen)} of ${String(text.length)} characters are shown, with ${String(omitted)} omitted in the middle. Pass writeTo to keep all ${String(text.length)}.`;
|
|
673
799
|
}
|
|
674
800
|
// `eval` together with `dump` is the engine's documented behaviour -- it discards
|
|
675
801
|
// the expression's value and dumps the page state the expression produced -- but
|
|
@@ -694,6 +820,35 @@ export function runFetch(options) {
|
|
|
694
820
|
};
|
|
695
821
|
}
|
|
696
822
|
|
|
823
|
+
/**
|
|
824
|
+
* Read the payload {@link selectorScript} prefixes onto its result.
|
|
825
|
+
*
|
|
826
|
+
* The engine prints the expression value verbatim, so the script adds a sentinel to
|
|
827
|
+
* tell a selector answer apart from page text. Anything without the sentinel is not
|
|
828
|
+
* a selector answer and comes back as undefined, which the caller reports as a
|
|
829
|
+
* failure rather than dressing it up as content.
|
|
830
|
+
*
|
|
831
|
+
* @param text - the engine's stdout.
|
|
832
|
+
* @returns `{ matched, withText, joined }`, or undefined.
|
|
833
|
+
*/
|
|
834
|
+
function parseSelectorResult(text) {
|
|
835
|
+
const marker = '[dsh-moli:selector]';
|
|
836
|
+
const at = String(text).indexOf(marker);
|
|
837
|
+
if (at < 0) return undefined;
|
|
838
|
+
try {
|
|
839
|
+
const raw = JSON.parse(String(text).slice(at + marker.length).trim());
|
|
840
|
+
if (raw === null || typeof raw !== 'object') return undefined;
|
|
841
|
+
if (typeof raw.matched !== 'number' || typeof raw.joined !== 'string') return undefined;
|
|
842
|
+
return {
|
|
843
|
+
matched: raw.matched,
|
|
844
|
+
withText: typeof raw.withText === 'number' ? raw.withText : 0,
|
|
845
|
+
joined: raw.joined,
|
|
846
|
+
};
|
|
847
|
+
} catch {
|
|
848
|
+
return undefined;
|
|
849
|
+
}
|
|
850
|
+
}
|
|
851
|
+
|
|
697
852
|
/**
|
|
698
853
|
* Read the HTTP status out of the engine's stderr reason.
|
|
699
854
|
*
|
package/package.json
CHANGED