dsh-moli 1.1.3 → 1.1.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +105 -0
- package/README.zh.md +88 -0
- package/index.js +253 -3
- package/lib/actions.js +380 -0
- package/lib/moli.js +41 -7
- package/lib/session.js +372 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -223,6 +223,52 @@ moli_fetch url=https://example.com dump=pdf writeTo=/abs/path/a.pdf
|
|
|
223
223
|
|
|
224
224
|
Screenshot and PDF dumps need `writeTo`; the bytes are written verbatim rather than printed.
|
|
225
225
|
|
|
226
|
+
### `moli_session` — an interactive browser
|
|
227
|
+
|
|
228
|
+
`moli_fetch` loads a page and prints it. That covers reading, and nothing else: it cannot dismiss a
|
|
229
|
+
consent wall, fill a search box, scroll a lazy list, or wait for a client-side route to change.
|
|
230
|
+
`moli_session` drives a persistent browser instead, over the Chrome DevTools Protocol endpoint that
|
|
231
|
+
`moli serve` publishes. State — cookies, navigation, typed input — survives between calls until you
|
|
232
|
+
stop it.
|
|
233
|
+
|
|
234
|
+
```
|
|
235
|
+
moli_session action=open url=https://example.com # start the browser and navigate
|
|
236
|
+
moli_session action=snapshot # the rendered page as text
|
|
237
|
+
moli_session action=click selector="#accept-cookies"
|
|
238
|
+
moli_session action=type selector="#q" text="moli" submit=true
|
|
239
|
+
moli_session action=scroll dy=1200
|
|
240
|
+
moli_session action=waitFor selector=".results"
|
|
241
|
+
moli_session action=eval expression="document.title"
|
|
242
|
+
moli_session action=screenshot writeTo=/abs/path/a.png
|
|
243
|
+
moli_session action=status # is it running, and on what page
|
|
244
|
+
moli_session action=close # stop it and free the port
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Elements are addressed by **CSS selector**, never by pixel coordinates: the tool measures the element
|
|
248
|
+
and dispatches real mouse and key events, so page handlers run exactly as they do for a person.
|
|
249
|
+
|
|
250
|
+
**One thing to know before relying on `click`.** This engine's element geometry is document-absolute
|
|
251
|
+
and does not follow scrolling, and its hit test is bounded by the viewport height. Measured on Moli
|
|
252
|
+
1.1.16: an element at document `y=1008` reports the same `y` at `scrollY` 0, 400 and 800, while
|
|
253
|
+
`elementFromPoint` stops answering past `y=1080`. The consequence is that synthetic input cannot reach
|
|
254
|
+
an element below the first viewport, on any page long enough to scroll. `click` says so, naming the
|
|
255
|
+
element, and points at the route that does work:
|
|
256
|
+
|
|
257
|
+
```
|
|
258
|
+
moli_session action=eval expression="document.querySelector('#far-down').click()"
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
`eval` runs a click inside the page and bypasses the input layer, so it reaches elements the input API
|
|
262
|
+
cannot. Note that it also skips hover and focus behaviour, which is the difference between clicking a
|
|
263
|
+
control and calling its handler.
|
|
264
|
+
|
|
265
|
+
The session needs **Node 21 or newer** for the built-in `WebSocket`; on Node 20 the tool answers with
|
|
266
|
+
that reason instead of failing obscurely. `moli_fetch` and `moli_probe` still work on 20.12.
|
|
267
|
+
|
|
268
|
+
`moli_session` also takes `port` (default 9222). If that port is already served by a `moli serve` you
|
|
269
|
+
started yourself, the tool connects to it rather than failing to bind — and `close` then leaves it
|
|
270
|
+
alone, because it is not ours to stop.
|
|
271
|
+
|
|
226
272
|
## Relationship to Moli's own agent skills
|
|
227
273
|
|
|
228
274
|
Moli ships official agent skills in its repository — [`moli-webfetch`](https://github.com/lexmount/moli/tree/main/skills/moli-webfetch),
|
|
@@ -310,8 +356,24 @@ Override any field under the plugin's `config` entry in the profile patch layer.
|
|
|
310
356
|
maxChars: 200000 # inline text truncation
|
|
311
357
|
proxy: 'http://127.0.0.1:7890' # forward proxy for moli, downloads, and the installer's engine download
|
|
312
358
|
insecureTls: true # see the TLS notes below
|
|
359
|
+
sessionPort: 9222 # CDP port for moli_session
|
|
360
|
+
blockPrivateNetworks: false # refuse private addresses (see the note below)
|
|
361
|
+
blockCidrs: [] # extra CIDRs to refuse
|
|
313
362
|
```
|
|
314
363
|
|
|
364
|
+
The plugin-level `proxy` above is the default; `moli_fetch` also takes a `proxy` argument, so a single
|
|
365
|
+
call can route through a different proxy without changing the profile. A loopback URL always bypasses
|
|
366
|
+
whichever proxy is in effect.
|
|
367
|
+
|
|
368
|
+
`blockPrivateNetworks` is **off by default, matching the engine**: `moli fetch http://10.0.0.1/` reaches
|
|
369
|
+
a private address, so the plugin does not silently narrow what the moli you installed already does.
|
|
370
|
+
Turn it on, or list `blockCidrs`, when the agent is following links on pages you do not control and
|
|
371
|
+
you want the engine to refuse private destinations. Two caveats worth knowing before you rely on it:
|
|
372
|
+
moli documents the policy as applying to **directly connected** targets, so a configured `proxy`
|
|
373
|
+
routes around it, and only `blockCidrs` (which the proxy must enforce) covers that path. The sharpest
|
|
374
|
+
target this protects is the cloud metadata address `169.254.169.254`, which hands out instance
|
|
375
|
+
credentials on a VM.
|
|
376
|
+
|
|
315
377
|
Without `exePath`, the plugin probes `exePath` → `MOLI_BIN` → `%LOCALAPPDATA%\Moli\bin\moli.exe` →
|
|
316
378
|
`%USERPROFILE%\.local\bin\moli.exe`, then scans inside `%LOCALAPPDATA%\Moli` for a `moli.exe`, so an
|
|
317
379
|
installation under a versioned subdirectory is still found.
|
|
@@ -367,6 +429,23 @@ Four suites live in [`verify/`](verify/), and all are runnable against a working
|
|
|
367
429
|
- `win-verify.mjs` (Windows hosts) runs the same pure assertions, then executes the `DefaultWebProxy`
|
|
368
430
|
prelude in a real `powershell.exe` and inspects the proxy object and credentials it set, exercises
|
|
369
431
|
the Windows download routes, and runs `installMoli` end to end.
|
|
432
|
+
- `empty-verify.mjs` asserts, against three compiled stub engines, that a fetch which exits 0 with no
|
|
433
|
+
output is a successful empty result rather than an error, that a non-zero exit still fails, and that
|
|
434
|
+
a complaint on stderr with no stdout is not mistaken for an empty page.
|
|
435
|
+
- `report-verify.mjs` guards the findings from a usage report: `eval` documents that pairing it with
|
|
436
|
+
`dump` discards its value, an expression that throws is classified as such instead of being blamed
|
|
437
|
+
on the installation, truncation keeps the tail as well as the head, `proxy` is a per-call parameter,
|
|
438
|
+
and `moli_probe` states which process its `httpsOk` describes.
|
|
439
|
+
- `session-verify.mjs` drives `moli_session` end to end against a `data:` URL fixture, so it needs no
|
|
440
|
+
network and no site stays still underneath it: lifecycle and port release, snapshot, click, type
|
|
441
|
+
with and without clearing, press, scroll, `waitFor` in both directions, eval including a throwing
|
|
442
|
+
expression, screenshot byte signature, navigation by clicking a link, restart after close, and the
|
|
443
|
+
viewport limit described above — which is asserted as a limit, with the `eval` route that works
|
|
444
|
+
around it asserted next to it.
|
|
445
|
+
|
|
446
|
+
`.github/workflows/verify.yml` runs the first two on Linux, Windows and macOS (they need no engine and
|
|
447
|
+
no network) and attempts the engine-dependent suites on Linux and Windows, reporting a missing engine
|
|
448
|
+
as a skip rather than a failure.
|
|
370
449
|
|
|
371
450
|
Beyond those, `lib/moli.js` and the tool layer were exercised end to end against Moli 1.1.16 on
|
|
372
451
|
Windows: registration, probe, Markdown and JSON dumps, `eval` (including a tight-timeout regression
|
|
@@ -380,6 +459,32 @@ Confirmed at runtime too: a JavaScript-rendered page (GitHub search) returned 14
|
|
|
380
459
|
Markdown containing 30 real repository matches when fetched through a local proxy. A throwaway
|
|
381
460
|
profile with the plugin installed composes its config with no `JsonSchemaError` and no skipped bundle.
|
|
382
461
|
|
|
462
|
+
## Releasing
|
|
463
|
+
|
|
464
|
+
Publishing goes through GitHub Actions' OIDC identity, so **no npm token exists to expire**:
|
|
465
|
+
|
|
466
|
+
```bash
|
|
467
|
+
npm version patch # or minor / major
|
|
468
|
+
git push && git tag v1.1.6 && git push origin v1.1.6
|
|
469
|
+
```
|
|
470
|
+
|
|
471
|
+
`.github/workflows/publish.yml` checks that the tag matches `package.json`, runs the offline suites,
|
|
472
|
+
and publishes. One-time setup on npmjs.com: **package → Settings → Trusted Publisher → GitHub
|
|
473
|
+
Actions**, with user `27painter`, repository `dsh-moli`, workflow filename `publish.yml`, and the
|
|
474
|
+
`npm publish` action allowed.
|
|
475
|
+
|
|
476
|
+
**Do not publish with a stored token.** This machine was configured with a classic token that bypasses
|
|
477
|
+
2FA, and npm is retiring those; when it was revoked, the publish failed with
|
|
478
|
+
|
|
479
|
+
```
|
|
480
|
+
E404 Not Found - PUT https://registry.npmjs.org/dsh-moli
|
|
481
|
+
The requested resource 'dsh-moli@1.1.5' could not be found or you do not have permission
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
which reads as a missing package and is actually an authorisation failure. Run
|
|
485
|
+
`node scripts/preflight-publish.mjs` before any local publish: it reports whether the credential can
|
|
486
|
+
authenticate and names that misleading 404 when it cannot.
|
|
487
|
+
|
|
383
488
|
## License
|
|
384
489
|
|
|
385
490
|
MIT. See [LICENSE](LICENSE).
|
package/README.zh.md
CHANGED
|
@@ -209,6 +209,46 @@ moli_fetch url=https://example.com dump=pdf writeTo=/绝对路径/a.pdf
|
|
|
209
209
|
|
|
210
210
|
截图与 PDF 必须给 `writeTo`;字节是原样落盘的,不会打印到输出里。
|
|
211
211
|
|
|
212
|
+
### `moli_session` —— 可交互的浏览器
|
|
213
|
+
|
|
214
|
+
`moli_fetch` 是"加载一页、打印出来",只能读。它点不掉同意墙、填不了搜索框、滚不动懒加载列表,也等不到
|
|
215
|
+
客户端路由切换。`moli_session` 驱动的是一个**常驻浏览器**,走 `moli serve` 暴露的 Chrome DevTools
|
|
216
|
+
Protocol。状态(cookie、导航位置、已输入的内容)在多次调用之间保持,直到你显式停止。
|
|
217
|
+
|
|
218
|
+
```
|
|
219
|
+
moli_session action=open url=https://example.com # 启动浏览器并导航
|
|
220
|
+
moli_session action=snapshot # 渲染后的页面文本
|
|
221
|
+
moli_session action=click selector="#accept-cookies"
|
|
222
|
+
moli_session action=type selector="#q" text="moli" submit=true
|
|
223
|
+
moli_session action=scroll dy=1200
|
|
224
|
+
moli_session action=waitFor selector=".results"
|
|
225
|
+
moli_session action=eval expression="document.title"
|
|
226
|
+
moli_session action=screenshot writeTo=/绝对路径/a.png
|
|
227
|
+
moli_session action=status # 是否在运行、当前在哪一页
|
|
228
|
+
moli_session action=close # 停止并释放端口
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
元素一律用 **CSS 选择器**定位,不接受像素坐标:工具自己测量元素并派发真实的鼠标与键盘事件,所以页面的
|
|
232
|
+
处理器会像人操作时一样执行。
|
|
233
|
+
|
|
234
|
+
**依赖 `click` 之前需要知道一件事。** 这个引擎的元素几何是**文档绝对坐标且不随滚动更新**,而它的命中测试
|
|
235
|
+
**受视口高度限制**。在 Moli 1.1.16 上实测:文档 `y=1008` 的元素在 `scrollY` 为 0、400、800 时报告的 `y`
|
|
236
|
+
完全相同,而 `elementFromPoint` 在 `y=1080` 之后一律返回 null。后果是:**在长到需要滚动的页面上,合成输入
|
|
237
|
+
无法触达首个视口以下的元素。** `click` 会如实报出这一点并点名元素,同时给出真正可行的路径:
|
|
238
|
+
|
|
239
|
+
```
|
|
240
|
+
moli_session action=eval expression="document.querySelector('#far-down').click()"
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
`eval` 在页面内部执行点击,绕开输入层,因此能触达输入 API 达不到的元素。注意它同时跳过了 hover 与 focus
|
|
244
|
+
行为——这正是"点击一个控件"与"调用它的处理器"的区别。
|
|
245
|
+
|
|
246
|
+
本工具需要 **Node 21 或更新版本**(内置 `WebSocket`);在 Node 20 上会直接说明原因,而不是给出费解的失败。
|
|
247
|
+
`moli_fetch` 与 `moli_probe` 仍支持 20.12。
|
|
248
|
+
|
|
249
|
+
`moli_session` 也接受 `port`(默认 9222)。如果该端口已经由你自己启动的 `moli serve` 占用,工具会连接它
|
|
250
|
+
而不是绑定失败——此时 `close` **不会**停掉它,因为那不是我们的进程。
|
|
251
|
+
|
|
212
252
|
## 与 Moli 官方 agent skill 的关系
|
|
213
253
|
|
|
214
254
|
Moli 仓库自带官方 agent skill:[`moli-webfetch`](https://github.com/lexmount/moli/tree/main/skills/moli-webfetch)、
|
|
@@ -284,8 +324,20 @@ cookie 驱动的查询参数。那次实测同时覆盖了代理下载路由与
|
|
|
284
324
|
maxChars: 200000 # 内联文本截断阈值
|
|
285
325
|
proxy: 'http://127.0.0.1:7890' # 转给本地正向代理(moli、下载、安装器内部引擎下载都走它)
|
|
286
326
|
insecureTls: true # 见下方 TLS 说明
|
|
327
|
+
sessionPort: 9222 # moli_session 的 CDP 端口
|
|
328
|
+
blockPrivateNetworks: false # 拒绝私网地址(见下方说明)
|
|
329
|
+
blockCidrs: [] # 额外拒绝的 CIDR
|
|
287
330
|
```
|
|
288
331
|
|
|
332
|
+
上面配置里的 `proxy` 是默认值;`moli_fetch` 也能接收 `proxy` 参数,因此单次调用可以走另一个代理而不必改
|
|
333
|
+
profile。无论生效的是哪个代理,loopback 地址一律绕过它。
|
|
334
|
+
|
|
335
|
+
`blockPrivateNetworks` **默认关闭,与引擎一致**:`moli fetch http://10.0.0.1/` 本来就能访问私网地址,插件
|
|
336
|
+
不会悄悄收窄你已安装的 moli 的能力。当 agent 会在你无法控制的页面上跟着链接走、而你希望引擎拒绝私网目标时,
|
|
337
|
+
再打开它或配置 `blockCidrs`。依赖它之前有两点需要知道:moli 文档写明该策略只作用于**直连**目标,因此配置了
|
|
338
|
+
`proxy` 就会绕过它,只有 `blockCidrs`(由代理执行)能覆盖那条路径。这个开关保护的最锐利目标是云元数据地址
|
|
339
|
+
`169.254.169.254`——在虚拟机上它会交出实例凭据。
|
|
340
|
+
|
|
289
341
|
未设置 `exePath` 时探测顺序:`exePath` → 环境变量 `MOLI_BIN` → `%LOCALAPPDATA%\Moli\bin\moli.exe`
|
|
290
342
|
→ `%USERPROFILE%\.local\bin\moli.exe`,随后还会在 `%LOCALAPPDATA%\Moli` 内递归查找 `moli.exe`,
|
|
291
343
|
所以装在带版本号的子目录里也能找到。
|
|
@@ -327,6 +379,18 @@ cookie 驱动的查询参数。那次实测同时覆盖了代理下载路由与
|
|
|
327
379
|
这正是异步路径的验收标准。
|
|
328
380
|
- `win-verify.mjs`(Windows 主机)跑同样的纯函数断言,然后在真实 `powershell.exe` 里执行 `DefaultWebProxy`
|
|
329
381
|
prelude 并检查它设置的代理对象与凭据,练习 Windows 下载路由,并端到端跑 `installMoli`。
|
|
382
|
+
- `empty-verify.mjs` 用三个编译好的桩引擎断言:退出码 0 且无输出属于"成功但内容为空",而不是错误;非零退出
|
|
383
|
+
仍然失败;stderr 有抱怨而 stdout 为空不会被误当成空页面。
|
|
384
|
+
- `report-verify.mjs` 守住一份使用体验报告里的发现:`eval` 与 `dump` 同用会丢弃表达式值(已在参数描述里
|
|
385
|
+
写明)、表达式抛异常时被正确归类而不是甩锅给"安装有问题"、截断同时保留头尾、`proxy` 是可按调用传入的参数、
|
|
386
|
+
以及 `moli_probe` 会说明它的 `httpsOk` 描述的是哪个进程。
|
|
387
|
+
- `session-verify.mjs` 用 `data:` URL 夹具端到端驱动 `moli_session`,因此不依赖网络、也不会因为站点变化而
|
|
388
|
+
失效:生命周期与端口释放、snapshot、click、带与不带清空的 type、press、scroll、两个方向的 `waitFor`、
|
|
389
|
+
含抛异常的 eval、截图字节签名、点击链接导航、关闭后重启,以及上面那条视口限制——限制被当作**限制**断言,
|
|
390
|
+
紧随其后断言真正能绕过它的 `eval` 路径。
|
|
391
|
+
|
|
392
|
+
`.github/workflows/verify.yml` 在 Linux、Windows、macOS 上跑前两个(无需引擎、无需网络),在 Linux 与 Windows
|
|
393
|
+
上尝试依赖引擎的几套;若 runner 上没有引擎,则报告为跳过而不是失败。
|
|
330
394
|
|
|
331
395
|
此外,`lib/moli.js` 与工具层在 Windows 上对 Moli 1.1.16 做了端到端验证:注册、probe、Markdown 与 JSON
|
|
332
396
|
dump、`eval`(含针对 `-t` 参数位置的紧超时回归防护)、PNG 与 PDF 产物签名、截断、全部拒绝路径(二元 dump
|
|
@@ -337,6 +401,30 @@ dump、`eval`(含针对 `-t` 参数位置的紧超时回归防护)、PNG 与
|
|
|
337
401
|
30 处真实结果匹配。另有一个装了本插件的一次性 profile,组合其配置时既无 `JsonSchemaError` 也无被跳过的
|
|
338
402
|
bundle。
|
|
339
403
|
|
|
404
|
+
## 发布
|
|
405
|
+
|
|
406
|
+
发布走 GitHub Actions 的 OIDC 身份,**本机不存任何 npm token,因此没有会失效的凭据**:
|
|
407
|
+
|
|
408
|
+
```bash
|
|
409
|
+
npm version patch # 或 minor / major
|
|
410
|
+
git push && git tag v1.1.6 && git push origin v1.1.6
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
`.github/workflows/publish.yml` 会先校验 tag 与 `package.json` 版本一致、跑离线测试套件,然后发布。
|
|
414
|
+
npmjs.com 上需要一次性配置:**package → Settings → Trusted Publisher → GitHub Actions**,用户
|
|
415
|
+
`27painter`、仓库 `dsh-moli`、workflow 文件名 `publish.yml`,并允许 `npm publish` 动作。
|
|
416
|
+
|
|
417
|
+
**不要再用本机存储的 token 发布。** 这台机器原先配的是一个绕过 2FA 的 classic token,npm 正在淘汰这类
|
|
418
|
+
token;它被吊销后,发布会报:
|
|
419
|
+
|
|
420
|
+
```
|
|
421
|
+
E404 Not Found - PUT https://registry.npmjs.org/dsh-moli
|
|
422
|
+
The requested resource 'dsh-moli@1.1.5' could not be found or you do not have permission
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
**这看起来像"包不存在",实际是认证失败。** 任何本地发布之前先跑
|
|
426
|
+
`node scripts/preflight-publish.mjs`:它会告诉你凭据能否认证,不能时明确点出这个误导性的 404。
|
|
427
|
+
|
|
340
428
|
## 许可
|
|
341
429
|
|
|
342
430
|
MIT,见 [LICENSE](LICENSE)。
|
package/index.js
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
/** Host half of dsh-moli: expose the local `moli` headless browser as callable tools. */
|
|
2
|
+
import { performAction } from './lib/actions.js';
|
|
2
3
|
import { MANUAL_INSTALL, installMoli } from './lib/install.js';
|
|
3
4
|
import { findMoli, moliCandidates, probe, runFetch } from './lib/moli.js';
|
|
5
|
+
import { SessionManager, sessionSupported, sessionUnsupportedReason } from './lib/session.js';
|
|
4
6
|
|
|
5
7
|
export const name = 'dsh-moli';
|
|
6
8
|
export const inject = ['tools'];
|
|
@@ -65,7 +67,7 @@ const FETCH_PARAMETERS = {
|
|
|
65
67
|
},
|
|
66
68
|
eval: {
|
|
67
69
|
type: 'string',
|
|
68
|
-
description: 'JavaScript expression evaluated once the page is ready; its value is returned as text or JSON instead of the page.',
|
|
70
|
+
description: 'JavaScript expression evaluated once the page is ready; its value is returned as text or JSON instead of the page. Combining it with `dump` does NOT return both: the engine discards the expression value and dumps the resulting page state instead (its own documented semantics). Use `eval` alone to read a value, or `dump` alone to read the page.',
|
|
69
71
|
},
|
|
70
72
|
wait: {
|
|
71
73
|
type: 'string',
|
|
@@ -85,6 +87,10 @@ const FETCH_PARAMETERS = {
|
|
|
85
87
|
resources: { type: 'boolean', description: 'Also fetch images, fonts, audio, video, and text tracks.' },
|
|
86
88
|
withFrames: { type: 'boolean', description: 'Include subframes in the output.' },
|
|
87
89
|
insecureTls: { type: 'boolean', description: 'Skip TLS host verification. Use only for a known-broken local endpoint.' },
|
|
90
|
+
proxy: {
|
|
91
|
+
type: 'string',
|
|
92
|
+
description: 'Forward proxy for this call, e.g. "http://127.0.0.1:7890". Overrides the plugin configuration for one call; loopback URLs always bypass it.',
|
|
93
|
+
},
|
|
88
94
|
profileDir: { type: 'string', description: 'Moli profile directory carrying imported cookies or storage.' },
|
|
89
95
|
cookieFile: { type: 'string', description: 'Netscape/curl cookie-jar file to send.' },
|
|
90
96
|
},
|
|
@@ -111,6 +117,8 @@ const FETCH_OUTPUT_SCHEMA = {
|
|
|
111
117
|
chars: { type: 'integer' },
|
|
112
118
|
bytes: { type: 'integer' },
|
|
113
119
|
empty: { type: 'boolean' },
|
|
120
|
+
omitted: { type: 'integer' },
|
|
121
|
+
evalFailed: { type: 'boolean' },
|
|
114
122
|
truncated: { type: 'boolean' },
|
|
115
123
|
error: { type: 'string' },
|
|
116
124
|
hint: { type: 'string' },
|
|
@@ -132,6 +140,7 @@ const PROBE_OUTPUT_SCHEMA = {
|
|
|
132
140
|
exe: { type: 'string' },
|
|
133
141
|
version: { type: 'string' },
|
|
134
142
|
httpsOk: { type: 'boolean' },
|
|
143
|
+
httpsOkScope: { type: 'string' },
|
|
135
144
|
sandboxTlsBlock: { type: 'boolean' },
|
|
136
145
|
revocationBlock: { type: 'boolean' },
|
|
137
146
|
detail: { type: 'string' },
|
|
@@ -219,6 +228,151 @@ function renderProbe(_args, value) {
|
|
|
219
228
|
return [{ type: 'text', text: lines.join('\n') }];
|
|
220
229
|
}
|
|
221
230
|
|
|
231
|
+
/** Actions `moli_session` accepts. */
|
|
232
|
+
const SESSION_ACTIONS = ['start', 'open', 'snapshot', 'click', 'type', 'press', 'scroll', 'waitFor', 'eval', 'screenshot', 'status', 'close'];
|
|
233
|
+
|
|
234
|
+
/**
|
|
235
|
+
* Parameters of `moli_session`.
|
|
236
|
+
*
|
|
237
|
+
* Raw JSON Schema with a root `type: "object"`, like the other tools: `schemaOf`
|
|
238
|
+
* projects this straight to the provider, so the author DSL would arrive as
|
|
239
|
+
* `type: null` and be rejected.
|
|
240
|
+
*/
|
|
241
|
+
const SESSION_PARAMETERS = {
|
|
242
|
+
type: 'object',
|
|
243
|
+
additionalProperties: false,
|
|
244
|
+
required: ['action'],
|
|
245
|
+
properties: {
|
|
246
|
+
action: {
|
|
247
|
+
type: 'string',
|
|
248
|
+
enum: SESSION_ACTIONS,
|
|
249
|
+
description: 'start: ensure the browser is running. open: navigate. snapshot: read the rendered page. '
|
|
250
|
+
+ 'click/type/press/scroll: interact. waitFor: wait for a selector to appear or disappear. '
|
|
251
|
+
+ 'eval: run JavaScript in the page. screenshot: write a PNG. status: report the session. close: stop the browser.',
|
|
252
|
+
},
|
|
253
|
+
url: { type: 'string', description: 'open: absolute http:// or https:// URL to navigate to.' },
|
|
254
|
+
selector: { type: 'string', description: 'CSS selector for click, type, scroll and waitFor.' },
|
|
255
|
+
text: { type: 'string', description: 'type: the text to enter.' },
|
|
256
|
+
submit: { type: 'boolean', description: 'type: press Enter after entering the text.' },
|
|
257
|
+
clear: { type: 'boolean', description: 'type: empty the field before entering text.' },
|
|
258
|
+
key: { type: 'string', description: 'press: key name, e.g. Enter, Tab, Escape, ArrowDown.' },
|
|
259
|
+
dy: { type: 'number', description: 'scroll: pixels to move down (negative moves up). Defaults to 800.' },
|
|
260
|
+
block: { type: 'string', enum: ['start', 'center', 'end', 'nearest'], description: 'scroll: where to align the element. Defaults to center.' },
|
|
261
|
+
state: { type: 'string', enum: ['attached', 'detached'], description: 'waitFor: wait for the selector to appear (attached) or vanish (detached).' },
|
|
262
|
+
expression: { type: 'string', description: 'eval: a JavaScript expression; promises are awaited and the value is returned.' },
|
|
263
|
+
writeTo: { type: 'string', description: 'screenshot: absolute path for the PNG. Required, since images are not returned inline.' },
|
|
264
|
+
format: { type: 'string', enum: ['png', 'jpeg'], description: 'screenshot: image format. Defaults to png.' },
|
|
265
|
+
includeHtml: { type: 'boolean', description: 'snapshot: also return the rendered HTML.' },
|
|
266
|
+
maxChars: { type: 'integer', description: 'snapshot: truncate the text to this many characters.' },
|
|
267
|
+
timeoutMs: { type: 'integer', description: 'Deadline for this action. Defaults to the plugin configuration.' },
|
|
268
|
+
port: { type: 'integer', description: 'start: CDP port. Defaults to the plugin configuration (9222).' },
|
|
269
|
+
},
|
|
270
|
+
};
|
|
271
|
+
|
|
272
|
+
const SESSION_OUTPUT_SCHEMA = {
|
|
273
|
+
type: 'object',
|
|
274
|
+
additionalProperties: true,
|
|
275
|
+
required: ['ok'],
|
|
276
|
+
properties: {
|
|
277
|
+
ok: { type: 'boolean' },
|
|
278
|
+
action: { type: 'string' },
|
|
279
|
+
error: { type: 'string' },
|
|
280
|
+
hint: { type: 'string' },
|
|
281
|
+
url: { type: 'string' },
|
|
282
|
+
title: { type: 'string' },
|
|
283
|
+
text: { type: 'string' },
|
|
284
|
+
chars: { type: 'integer' },
|
|
285
|
+
truncated: { type: 'boolean' },
|
|
286
|
+
links: { type: 'array', items: { type: 'object', additionalProperties: true } },
|
|
287
|
+
fields: { type: 'array', items: { type: 'object', additionalProperties: true } },
|
|
288
|
+
html: { type: 'string' },
|
|
289
|
+
selector: { type: 'string' },
|
|
290
|
+
clicked: { type: 'object', additionalProperties: true },
|
|
291
|
+
navigated: { type: 'boolean' },
|
|
292
|
+
submitted: { type: 'boolean' },
|
|
293
|
+
value: {},
|
|
294
|
+
serialized: { type: 'string' },
|
|
295
|
+
key: { type: 'string' },
|
|
296
|
+
y: { type: 'number' },
|
|
297
|
+
height: { type: 'number' },
|
|
298
|
+
viewport: { type: 'number' },
|
|
299
|
+
state: { type: 'string' },
|
|
300
|
+
waitedMs: { type: 'integer' },
|
|
301
|
+
path: { type: 'string' },
|
|
302
|
+
bytes: { type: 'integer' },
|
|
303
|
+
format: { type: 'string' },
|
|
304
|
+
running: { type: 'boolean' },
|
|
305
|
+
ownedByThisPlugin: { type: 'boolean' },
|
|
306
|
+
external: { type: 'boolean' },
|
|
307
|
+
pid: { type: 'integer' },
|
|
308
|
+
endpoint: { type: 'string' },
|
|
309
|
+
logPath: { type: 'string' },
|
|
310
|
+
settled: { type: 'boolean' },
|
|
311
|
+
settledMs: { type: 'integer' },
|
|
312
|
+
stopped: { type: 'boolean' },
|
|
313
|
+
},
|
|
314
|
+
};
|
|
315
|
+
|
|
316
|
+
/**
|
|
317
|
+
* Render a `moli_session` value for the model.
|
|
318
|
+
*
|
|
319
|
+
* A snapshot is the one result worth formatting rather than dumping as JSON: the
|
|
320
|
+
* link and field tables are what the next action usually depends on.
|
|
321
|
+
*
|
|
322
|
+
* @param _args - the call arguments, unused.
|
|
323
|
+
* @param value - the canonical result value.
|
|
324
|
+
* @returns result content blocks.
|
|
325
|
+
*/
|
|
326
|
+
function renderSession(_args, value) {
|
|
327
|
+
if (value.ok !== true) {
|
|
328
|
+
const lines = [`moli_session failed${typeof value.action === 'string' ? ` (${value.action})` : ''}: ${value.error ?? 'unknown error'}`];
|
|
329
|
+
if (typeof value.hint === 'string' && value.hint !== '') lines.push(value.hint);
|
|
330
|
+
if (typeof value.bytes === 'number') lines.push(`(the image was ${String(value.bytes)} bytes but could not be written)`);
|
|
331
|
+
return [{ type: 'text', text: lines.join('\n') }];
|
|
332
|
+
}
|
|
333
|
+
if (value.action === 'snapshot') {
|
|
334
|
+
const lines = [`${String(value.title)} — ${String(value.url)}`, ''];
|
|
335
|
+
lines.push(String(value.text ?? ''));
|
|
336
|
+
if (Array.isArray(value.links) && value.links.length > 0) {
|
|
337
|
+
lines.push('', `Links (${String(value.links.length)}):`);
|
|
338
|
+
for (const link of value.links.slice(0, 25)) lines.push(` ${String(link.text)} -> ${String(link.href)}`);
|
|
339
|
+
}
|
|
340
|
+
if (Array.isArray(value.fields) && value.fields.length > 0) {
|
|
341
|
+
lines.push('', `Fields (${String(value.fields.length)}):`);
|
|
342
|
+
for (const field of value.fields.slice(0, 25)) lines.push(` <${String(field.tag)}${field.type === '' ? '' : ` type=${String(field.type)}`}${field.name === '' ? '' : ` name=${String(field.name)}`}${field.id === '' ? '' : ` id=${String(field.id)}`}>`);
|
|
343
|
+
}
|
|
344
|
+
return [{ type: 'text', text: lines.join('\n') }];
|
|
345
|
+
}
|
|
346
|
+
if (value.action === 'status') {
|
|
347
|
+
return [{
|
|
348
|
+
type: 'text',
|
|
349
|
+
text: value.running === true
|
|
350
|
+
? `Session running at ${String(value.endpoint)}${value.external === true ? ' (started outside this plugin, so close will not stop it)' : ''}${value.url === undefined ? '' : `\nCurrent page: ${String(value.title)} — ${String(value.url)}`}`
|
|
351
|
+
: 'No session is running. Call moli_session with action "start" or "open" to launch one.',
|
|
352
|
+
}];
|
|
353
|
+
}
|
|
354
|
+
if (value.action === 'click') {
|
|
355
|
+
return [{
|
|
356
|
+
type: 'text',
|
|
357
|
+
text: `Clicked ${String(value.clicked?.tag ?? 'element')}${value.clicked?.text === undefined || value.clicked.text === '' ? '' : ` (“${String(value.clicked.text)}”)`}`
|
|
358
|
+
+ `${value.navigated === true ? `, which navigated to ${String(value.url)}` : '; the page did not navigate'}`,
|
|
359
|
+
}];
|
|
360
|
+
}
|
|
361
|
+
if (value.action === 'type') {
|
|
362
|
+
return [{ type: 'text', text: `Entered text${value.submitted === true ? ' and pressed Enter' : ''}; the field now reads ${JSON.stringify(String(value.value ?? ''))}` }];
|
|
363
|
+
}
|
|
364
|
+
if (value.action === 'screenshot') {
|
|
365
|
+
return [{ type: 'text', text: `Screenshot written to ${String(value.path)} (${String(value.bytes)} bytes). Read that file to see it.` }];
|
|
366
|
+
}
|
|
367
|
+
if (value.action === 'eval') {
|
|
368
|
+
return [{ type: 'text', text: String(value.serialized ?? '') }];
|
|
369
|
+
}
|
|
370
|
+
if (value.action === 'close') {
|
|
371
|
+
return [{ type: 'text', text: value.stopped === true ? 'Session stopped.' : 'Nothing to stop: no session was running that this plugin started.' }];
|
|
372
|
+
}
|
|
373
|
+
return [{ type: 'text', text: JSON.stringify(value) }];
|
|
374
|
+
}
|
|
375
|
+
|
|
222
376
|
/**
|
|
223
377
|
* Register the Moli tools.
|
|
224
378
|
*
|
|
@@ -265,11 +419,13 @@ export function apply(ctx, config) {
|
|
|
265
419
|
manualInstall: MANUAL_INSTALL,
|
|
266
420
|
};
|
|
267
421
|
}
|
|
422
|
+
// A per-call proxy wins over the plugin configuration, so one host behind a
|
|
423
|
+
// different proxy does not require a reconfiguration.
|
|
268
424
|
return runFetch({
|
|
269
425
|
exe,
|
|
270
426
|
args: { ...args, timeoutMs: args.timeoutMs ?? defaultTimeoutMs },
|
|
271
427
|
maxChars,
|
|
272
|
-
proxy,
|
|
428
|
+
proxy: typeof args.proxy === 'string' && args.proxy !== '' ? args.proxy : proxy,
|
|
273
429
|
insecureTls,
|
|
274
430
|
});
|
|
275
431
|
},
|
|
@@ -335,6 +491,10 @@ export function apply(ctx, config) {
|
|
|
335
491
|
exe: result.exe,
|
|
336
492
|
version: result.version,
|
|
337
493
|
httpsOk: result.httpsOk,
|
|
494
|
+
// Says which process the HTTPS result describes. Under the file sandbox a
|
|
495
|
+
// restricted shell cannot reach the network while this path can, so the
|
|
496
|
+
// caller should not read httpsOk as "this machine has internet".
|
|
497
|
+
httpsOkScope: result.httpsOkScope,
|
|
338
498
|
sandboxTlsBlock: result.sandboxTlsBlock,
|
|
339
499
|
revocationBlock: result.revocationBlock,
|
|
340
500
|
detail: result.detail,
|
|
@@ -342,5 +502,95 @@ export function apply(ctx, config) {
|
|
|
342
502
|
},
|
|
343
503
|
});
|
|
344
504
|
|
|
345
|
-
|
|
505
|
+
// ---------------------------------------------------------------------------
|
|
506
|
+
// moli_session: an interactive browser, for the work a one-shot fetch cannot do
|
|
507
|
+
// ---------------------------------------------------------------------------
|
|
508
|
+
//
|
|
509
|
+
// This is the plugin's only stateful tool. `moli serve` is a long-lived child
|
|
510
|
+
// that has to outlive a single call, so one manager is created per plugin
|
|
511
|
+
// instance and its lifecycle is explicit (`start` implicitly, `close` by hand).
|
|
512
|
+
// Registration stays synchronous: the server is spawned on first use, never at
|
|
513
|
+
// load time, so a plugin load cannot delay Host startup or open a port.
|
|
514
|
+
const session = new SessionManager({
|
|
515
|
+
get exe() { return cachedExe; },
|
|
516
|
+
port: options.sessionPort ?? 9222,
|
|
517
|
+
host: '127.0.0.1',
|
|
518
|
+
insecureTls,
|
|
519
|
+
proxy: options.proxy,
|
|
520
|
+
userAgent: options.sessionUserAgent,
|
|
521
|
+
profileDir: options.profileDir,
|
|
522
|
+
resources: options.sessionResources === true,
|
|
523
|
+
// Off, matching the engine's own default: `moli fetch <private-ip>` can reach
|
|
524
|
+
// a private address, so the plugin does not silently narrow what a user's
|
|
525
|
+
// installed engine already does. Set it to true, or pass blockCidrs, to have
|
|
526
|
+
// the engine refuse private addresses for directly connected targets.
|
|
527
|
+
blockPrivateNetworks: options.blockPrivateNetworks === true,
|
|
528
|
+
blockCidrs: Array.isArray(options.blockCidrs) ? options.blockCidrs : [],
|
|
529
|
+
startTimeoutMs: options.sessionStartTimeoutMs ?? 20000,
|
|
530
|
+
});
|
|
531
|
+
|
|
532
|
+
ctx.tools.register({
|
|
533
|
+
name: 'moli_session',
|
|
534
|
+
description: 'Drive an interactive browser through a persistent session, for what a one-shot fetch cannot'
|
|
535
|
+
+ ' do: click through consent walls, fill and submit a search box, scroll a lazy list, wait for a'
|
|
536
|
+
+ ' client-side route change, or run a script in the page. Keeps one browser alive across calls, so'
|
|
537
|
+
+ ' state (cookies, navigation, form input) persists until you stop it with action "close".'
|
|
538
|
+
+ ' Start with "open"; read the page with "snapshot"; interact with "click", "type", "press" and'
|
|
539
|
+
+ ' "scroll"; use "eval" as the escape hatch. Elements are addressed by CSS selector, never by pixel'
|
|
540
|
+
+ ' coordinates.',
|
|
541
|
+
parameters: SESSION_PARAMETERS,
|
|
542
|
+
output: { schema: SESSION_OUTPUT_SCHEMA, render: renderSession },
|
|
543
|
+
// One browser, one page: overlapping actions would interleave input events and
|
|
544
|
+
// navigate the page out from under each other.
|
|
545
|
+
isConcurrencySafe: () => false,
|
|
546
|
+
async execute(args, exec) {
|
|
547
|
+
// `status` and `close` are meaningful with nothing running, so they are
|
|
548
|
+
// answered before the engine is required.
|
|
549
|
+
if (args.action === 'close') {
|
|
550
|
+
const outcome = await session.close();
|
|
551
|
+
return { ok: true, action: 'close', ...outcome };
|
|
552
|
+
}
|
|
553
|
+
if (args.action === 'status') {
|
|
554
|
+
const state = await session.status();
|
|
555
|
+
return { ok: true, action: 'status', ...state };
|
|
556
|
+
}
|
|
557
|
+
if (args.action === 'start') {
|
|
558
|
+
if (!sessionSupported()) return { ok: false, action: 'start', error: sessionUnsupportedReason() };
|
|
559
|
+
const { external } = await session.ensure();
|
|
560
|
+
const state = await session.status();
|
|
561
|
+
return { ok: true, action: 'start', started: true, external, ...state };
|
|
562
|
+
}
|
|
563
|
+
|
|
564
|
+
if (engine() === undefined) {
|
|
565
|
+
return {
|
|
566
|
+
ok: false,
|
|
567
|
+
action: String(args.action),
|
|
568
|
+
error: 'moli was not found, so a session cannot be started',
|
|
569
|
+
hint: 'Call moli_probe with action "install", or set the plugin config field `exePath`.',
|
|
570
|
+
};
|
|
571
|
+
}
|
|
572
|
+
if (!sessionSupported()) return { ok: false, action: String(args.action), error: sessionUnsupportedReason() };
|
|
573
|
+
|
|
574
|
+
if (exec?.signal?.aborted === true) return { ok: false, action: String(args.action), error: 'the call was aborted before it started' };
|
|
575
|
+
|
|
576
|
+
try {
|
|
577
|
+
const { connection } = await session.ensure();
|
|
578
|
+
const result = await performAction(connection, args, { maxChars });
|
|
579
|
+
// A closed socket means the browser went away under us; drop it so the
|
|
580
|
+
// next call starts a fresh one instead of failing forever.
|
|
581
|
+
if (connection.closed === true) void session.close();
|
|
582
|
+
return result;
|
|
583
|
+
} catch (error) {
|
|
584
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
585
|
+
const hint = /timed out/u.test(message)
|
|
586
|
+
? 'The action exceeded its deadline. Raise timeoutMs, or check `waitFor` for a selector that never appears.'
|
|
587
|
+
: /ECONNREFUSED|closed|WebSocket/u.test(message)
|
|
588
|
+
? 'The browser connection was lost. Call moli_session with action "status", then "start" to relaunch it.'
|
|
589
|
+
: undefined;
|
|
590
|
+
return { ok: false, action: String(args.action), error: message, ...(hint === undefined ? {} : { hint }) };
|
|
591
|
+
}
|
|
592
|
+
},
|
|
593
|
+
});
|
|
594
|
+
|
|
595
|
+
ctx.logger?.info?.(`dsh-moli: registered moli_fetch, moli_probe and moli_session (${cachedExe ?? 'moli missing'})`);
|
|
346
596
|
}
|