dsh-moli 1.1.13 → 1.1.14

package/README.md

@@ -210,8 +210,9 @@ An unresolvable bundle is **skipped silently** apart from one stderr line
 ## Usage
 
 `moli_fetch` parameters: `url` (required), `dump`, `writeTo`, `eval`, `scroll`, `wait`,
-`waitSelector`, `delayMs`, `timeoutMs`, `maxChars`, `headers`, `userAgent`, `layout`, `scrollbars`,
-`noJs`, `noCss`, `resources`, `withFrames`, `insecureTls`, `profileDir`, `cookieFile`, `proxy`.
+`waitSelector`, `waitScript`, `waitResponseUrl`, `waitResponseJson`, `delayMs`, `timeoutMs`,
+`maxChars`, `headers`, `userAgent`, `layout`, `scrollbars`, `noJs`, `noCss`, `resources`,
+`withFrames`, `insecureTls`, `failOnHttpError`, `profileDir`, `cookieFile`, `proxy`.
 
 ```
 moli_fetch url=https://example.com dump=markdown             # token-efficient text
@@ -244,6 +245,43 @@ scrolled `dump=screenshot` shows that part of the page.
 It runs as part of the engine's own pre-capture evaluation and therefore composes with `eval`: the
 scroll happens first, and the expression still returns its own value.
 
+### `failOnHttpError` — stop reading an error page as content
+
+By default a 404 is returned like any other page, so a failed fetch looks successful. Setting
+`failOnHttpError` turns a final status of 400 or above into `ok: false` with the status named — and
+**keeps the body**, because an error page is often the thing you need to read.
+
+```
+moli_fetch url=https://example.com/nope dump=markdown failOnHttpError=true
+```
+
+| | default | `failOnHttpError: true` |
+|---|---|---|
+| a 404 | `ok: true`, the error page as content | `ok: false`, `status: 404`, body still returned |
+| a screenshot of a 404 | written as a normal image | still written, and reported as a failure |
+
+It is off by default so existing calls keep their behaviour. A JSON dump already reported the status
+on its own; markdown and HTML dumps did not, and that is the gap this closes.
+
+### Waiting for what the page actually fetches
+
+`wait: "networkidle"` plus `delayMs` is a guess about how long a list takes to arrive. These wait for
+the request itself:
+
+```
+# the response that carries the data
+moli_fetch url=https://example.com/feed dump=markdown waitResponseUrl="/api/feed"
+
+# the payload, when the URL is not distinctive
+moli_fetch url=https://example.com/feed dump=markdown waitResponseJson="data.list.0.id=42"
+
+# the page's own idea of readiness
+moli_fetch url=https://example.com/app dump=markdown waitScript="window.appReady === true"
+```
+
+`waitResponseJson` takes `PATH=VALUE`, where `PATH` addresses the field and `VALUE` is compared
+literally. Use these when a fixed delay either wastes time or reads the page too early.
+
 ## Relationship to Moli's own agent skills
 
 Moli ships official agent skills in its repository — [`moli-webfetch`](https://github.com/lexmount/moli/tree/main/skills/moli-webfetch),
@@ -403,6 +441,13 @@ Four suites live in [`verify/`](verify/), and all are runnable against a working
   `bottom`, nonsense rejected, the caller's `eval` value preserved) and then, when an engine is
   present, fetches a page whose `scroll` handler fills a slot and asserts the content appears with
   `scroll` and the placeholder appears without it — a property of the output, not of an internal call.
+- `http-verify.mjs` asserts the parameter-to-flag mapping directly, because a parameter that never
+  reaches the engine is indistinguishable from a feature that does not work. It then fetches a 404 and
+  checks that `failOnHttpError` reports it while keeping the body, that a healthy page is unaffected,
+  and that a screenshot of an error page is still written as a valid PNG rather than decoded as text.
+  The end-to-end half needs an HTTP error from a real server: loopback is unreachable from inside this
+  sandbox — `curl.exe` times out on `127.0.0.1` while the Node host reaches it — so it uses a path
+  under `example.com` and skips when the network is away.
 
 `.github/workflows/verify.yml` runs `schema-verify`, `empty-verify` and `scroll-verify` on Linux,
 Windows and macOS (they need no engine and no network) and attempts the engine-dependent suites on
@@ -467,6 +512,21 @@ which reads as a missing package and is actually an authorisation failure. Run
 `node scripts/preflight-publish.mjs` before any local publish: it reports whether the credential can
 authenticate and names that misleading 404 when it cannot.
 
+### Deprecating a version: do it on the website
+
+Publishing is scripted; deprecating deliberately is not. Use the package's page on npmjs.com —
+**package → Settings → Deprecate** — and name the versions and the reason.
+
+`npm deprecate dsh-moli@1.1.11 "…"` produces the same misleading failure as above when the local
+credential is dead, and there is a second reason not to reach for a token here. npm restricted
+2FA-bypass granular tokens from package-management operations on 31 July 2026, so the kind of token
+that used to make `deprecate` scriptable is being withdrawn, while `publish` has a supported
+token-free path through OIDC. A release process that needs a token only for the rare act of
+deprecating is a process that will be broken the next time it is needed, which is exactly what
+happened with 1.1.11 and 1.1.12.
+
+The 1.0.0 and 1.0.1 deprecations were done this way, from the website.
+
 ## License
 
 MIT. See [LICENSE](LICENSE).

package/README.zh.md

@@ -196,8 +196,9 @@ profile 插件需要两半都齐：依赖边（`dependencies`）+ `dsh.profile.b
 ## 用法
 
 `moli_fetch` 参数：`url`（必填）、`dump`、`writeTo`、`eval`、`scroll`、`wait`、`waitSelector`、
-`delayMs`、`timeoutMs`、`maxChars`、`headers`、`userAgent`、`layout`、`scrollbars`、`noJs`、
-`noCss`、`resources`、`withFrames`、`insecureTls`、`profileDir`、`cookieFile`、`proxy`。
+`waitScript`、`waitResponseUrl`、`waitResponseJson`、`delayMs`、`timeoutMs`、`maxChars`、`headers`、
+`userAgent`、`layout`、`scrollbars`、`noJs`、`noCss`、`resources`、`withFrames`、`insecureTls`、
+`failOnHttpError`、`profileDir`、`cookieFile`、`proxy`。
 
 ```
 moli_fetch url=https://example.com dump=markdown            # 省 token 的正文
@@ -229,6 +230,41 @@ moli_fetch url=https://example.com/feed dump=markdown scroll=-400    # 往回滚
 它作为引擎自身"抓取前求值"的一部分执行，因此与 `eval` 可组合：**先滚动，表达式再读取页面，并且仍然返回
 它自己的值**。
 
+### `failOnHttpError` —— 别再把错误页当正文读
+
+默认情况下 404 和普通页面一样被返回，于是**抓取失败看起来是成功的**。加上 `failOnHttpError` 后，最终状态码
+400 及以上会变成 `ok: false` 并报出状态码——**而且保留响应体**，因为错误页往往正是你需要读的东西。
+
+```
+moli_fetch url=https://example.com/nope dump=markdown failOnHttpError=true
+```
+
+| | 默认 | `failOnHttpError: true` |
+|---|---|---|
+| 一个 404 | `ok: true`，错误页作为正文 | `ok: false`、`status: 404`，响应体仍然返回 |
+| 404 的截图 | 当普通图片写出 | 仍然写出，但报告为失败 |
+
+默认关闭，因此现有调用行为不变。JSON dump 本来就能自己报出状态码；markdown 与 HTML 不能，这个开关补的正是
+这个缺口。
+
+### 等待页面真正取回的东西
+
+`wait: "networkidle"` 加 `delayMs` 是在**猜**列表要多久才到。下面这几个等的是请求本身：
+
+```
+# 等待承载数据的那次响应
+moli_fetch url=https://example.com/feed dump=markdown waitResponseUrl="/api/feed"
+
+# URL 不够有特征时，按载荷内容等
+moli_fetch url=https://example.com/feed dump=markdown waitResponseJson="data.list.0.id=42"
+
+# 让页面用自己的方式表示"就绪"
+moli_fetch url=https://example.com/app dump=markdown waitScript="window.appReady === true"
+```
+
+`waitResponseJson` 接收 `PATH=VALUE`：`PATH` 定位字段，`VALUE` 按字面比较。当固定延时不是浪费时间就是读得太
+早时，用这几个。
+
 ## 与 Moli 官方 agent skill 的关系
 
 Moli 仓库自带官方 agent skill：[`moli-webfetch`](https://github.com/lexmount/moli/tree/main/skills/moli-webfetch)、
@@ -358,6 +394,11 @@ profile。无论生效的是哪个代理，loopback 地址一律绕过它。
 - `scroll-verify.mjs` 在无引擎时校验表达式构造（像素值、负数、`top`、`bottom`、无效输入被拒、调用方的 `eval`
   值被保留），有引擎时抓取一个"滚动处理器填充槽位"的页面，断言带 `scroll` 时内容出现、不带时只有占位符——
   断言的是**输出**的性质，而不是内部调用。
+- `http-verify.mjs` 直接断言"参数 → 引擎选项"的映射，因为**参数没送到引擎和功能不生效，从外面看是一样的**。
+  随后抓一个 404，验证 `failOnHttpError` 会报告失败同时保留响应体、健康页面不受影响、以及错误页的截图仍然被
+  写为合法 PNG 而不是被当文本解码。端到端部分需要一个真实服务器的 HTTP 错误：**本沙箱内无法访问 loopback**
+  ——`curl.exe` 连 `127.0.0.1` 会超时，而 Node 宿主进程可以——所以它用 `example.com` 下的一个路径，网络不可用
+  时跳过。
 
 `.github/workflows/verify.yml` 在 Linux、Windows、macOS 上跑 `schema-verify`、`empty-verify` 与
 `scroll-verify`（无需引擎、无需网络），在 Linux 与 Windows 上尝试依赖引擎的几套；若 runner 上没有引擎，
@@ -414,6 +455,19 @@ E404 Not Found - PUT https://registry.npmjs.org/dsh-moli
 **这看起来像"包不存在"，实际是认证失败。** 任何本地发布之前先跑
 `node scripts/preflight-publish.mjs`：它会告诉你凭据能否认证，不能时明确点出这个误导性的 404。
 
+### 废弃某个版本：去网站操作，别用命令行
+
+发布是脚本化的，**废弃则刻意不做成脚本**。用 npmjs.com 上的包页面——**package → Settings →
+Deprecate**——选版本、写理由。
+
+`npm deprecate dsh-moli@1.1.11 "…"` 在本机凭据失效时会给出上面那个同样误导的失败，而且这里还有第二个
+理由不要依赖 token：npm 在 **2026 年 7 月 31 日**起禁止 2FA-bypass 类型的 granular token 执行**包管理**
+操作，也就是说"以前能让 `deprecate` 脚本化"的那类凭据正在被取消；而 `publish` 有受支持的、免 token 的
+OIDC 路径。**一个只在"偶尔废弃版本"时才需要 token 的流程，注定在下次需要它时是坏的**——1.1.11 与 1.1.12
+遇到的正是这种情况。
+
+1.0.0 与 1.0.1 的废弃就是通过网站完成的。
+
 ## 许可
 
 MIT，见 [LICENSE](LICENSE)。

package/index.js

@@ -77,6 +77,22 @@ const FETCH_PARAMETERS = {
       description: 'Readiness stage to wait for. Defaults to `done`.',
     },
     waitSelector: { type: 'string', description: 'Wait until this CSS selector matches.' },
+    waitScript: {
+      type: 'string',
+      description: 'Wait until this JavaScript expression is truthy; a function result is called as the predicate on each poll. More precise than a fixed delay when the page signals readiness in its own terms.',
+    },
+    waitResponseUrl: {
+      type: 'string',
+      description: 'Wait for a response whose original or final URL contains this substring. This is the reliable way to await an AJAX-built list: `wait: "networkidle"` plus a delay is a guess, whereas this waits for the request that actually carries the data.',
+    },
+    waitResponseJson: {
+      type: 'string',
+      description: 'Wait for a response whose JSON field equals a value, written as PATH=VALUE (for example "data.list.0.id=42"). The counterpart to waitResponseUrl when the condition is in the payload rather than the URL.',
+    },
+    failOnHttpError: {
+      type: 'boolean',
+      description: 'Treat a final main-document HTTP status of 400 or above as a failure. Without it an error page comes back as ordinary content and a 404 reads like a successful fetch; with it the result is ok:false and reports the status, while still carrying the rendered body so the error page can be read. Off by default, so existing calls keep their behaviour.',
+    },
     delayMs: { type: 'integer', description: 'Extra milliseconds to wait after readiness.' },
     timeoutMs: { type: 'integer', description: 'Total readiness deadline in milliseconds. Defaults to the plugin configuration.' },
     maxChars: { type: 'integer', description: 'Truncate inline text output to this many characters. Defaults to the plugin configuration.' },
@@ -125,6 +141,7 @@ const FETCH_OUTPUT_SCHEMA = {
     error: { type: 'string' },
     hint: { type: 'string' },
     status: { type: 'integer' },
+    httpError: { type: 'boolean' },
     sandboxTlsBlock: { type: 'boolean' },
     unreachable: { type: 'boolean' },
     possiblyUnreachable: { type: 'boolean' },
@@ -172,6 +189,14 @@ function renderFetch(_args, value) {
     if (value.unreachable === true) lines.push('Cause: the host did not answer; a local forward proxy is the usual fix on this network.');
     else if (value.possiblyUnreachable === true) lines.push('Cause: the host did not answer in time; it may be blocked, or the page may simply be slow.');
     if (typeof value.hint === 'string' && value.hint !== '') lines.push(`Next: ${value.hint}`);
+    // An HTTP error keeps its body under `failOnHttpError`. Reporting the failure
+    // without the body would throw away the error page, which is the part worth
+    // reading -- and for a binary dump the body is the file that was written.
+    if (value.kind === 'binary' && typeof value.path === 'string') {
+      lines.push(`The error page was still written to ${value.path} (${String(value.bytes)} bytes).`);
+    } else if (typeof value.content === 'string' && value.content !== '') {
+      lines.push('', 'The server\'s response body:', '', value.content);
+    }
     return [{ type: 'text', text: lines.join('\n') }];
   }
   if (value.kind === 'binary') {

package/lib/moli.js

@@ -31,6 +31,13 @@ const TEXT_DUMPS = new Set([undefined, 'html', 'markdown', 'json', 'semantic_tre
 /** Accepted `--wait` lifecycle stages. */
 const WAIT_STAGES = new Set(['domcontentloaded', 'load', 'networkidle', 'domstable', 'done']);
 
+/**
+ * The exit code the engine uses for "the final main-document status was >= 400".
+ * Its help states it for `--fail` and `--fail-with-body`; treating it specially is
+ * what lets an HTTP error be reported without discarding the body.
+ */
+const ENGINE_HTTP_ERROR_STATUS = 22;
+
 /** `scroll` keywords, mapped to what they mean in the page. */
 const SCROLL_TOKENS = {
   top: '0',
@@ -300,7 +307,18 @@ export function fetchOnce(options) {
  * @param args - validated tool arguments.
  * @returns the argv tail (everything after the executable).
  */
-function buildArgs(args) {
+/**
+ * Build the `moli fetch` argument vector for one call.
+ *
+ * Exported so the mapping from tool parameters to engine flags can be asserted
+ * directly. That matters more than it sounds: a parameter that silently fails to
+ * reach the engine is indistinguishable from a feature that does not work, and the
+ * only fast, deterministic way to tell them apart is to check the vector.
+ *
+ * @param args - tool arguments.
+ * @returns the argv tail (without the executable).
+ */
+export function buildArgs(args) {
   if (args.dump !== undefined && !TEXT_DUMPS.has(args.dump) && !BINARY_DUMPS.has(args.dump)) {
     throw new Error(`unsupported dump: ${String(args.dump)}`);
   }
@@ -320,6 +338,16 @@ function buildArgs(args) {
   if (evaluation !== undefined) argv.push('--eval', evaluation);
   if (args.wait !== undefined && args.wait !== '') argv.push('--wait', args.wait);
   if (args.waitSelector !== undefined && args.waitSelector !== '') argv.push('--wait-selector', args.waitSelector);
+  if (args.waitScript !== undefined && args.waitScript !== '') argv.push('--wait-script', args.waitScript);
+  // Response waits: the right way to await an AJAX list, where a fixed delay is a
+  // guess. The engine takes one of each, and a caller supplying several is asking
+  // for "any of these", which is the engine's own semantics for multiple waits.
+  if (args.waitResponseUrl !== undefined && args.waitResponseUrl !== '') argv.push('--wait-response-url', args.waitResponseUrl);
+  if (args.waitResponseJson !== undefined && args.waitResponseJson !== '') argv.push('--wait-response-json', args.waitResponseJson);
+  // The engine's own switch, but only when the caller asked for a failure to be
+  // surfaced: adding it unconditionally would turn every 404 into `ok: false` and
+  // change what existing calls do.
+  if (args.failOnHttpError === true) argv.push('--fail-with-body');
   if (args.delayMs !== undefined) argv.push('--delay-ms', String(args.delayMs));
   if (args.noJs === true) argv.push('--disable-js');
   if (args.noCss === true) argv.push('--disable-css');
@@ -413,6 +441,54 @@ export function runFetch(options) {
     };
   }
 
+  // The engine signals an HTTP error with exit code 22 and, under
+  // `--fail-with-body`, keeps whatever the output format produced. That is why the
+  // body is still here: the caller is told the fetch failed *and* gets to read what
+  // came back, which is what an error page is for.
+  //
+  // This branch is reached for binary dumps too -- a screenshot of an error page is
+  // still a screenshot -- so it must not decode stdout as text.
+  if (outcome.status === ENGINE_HTTP_ERROR_STATUS && bytes > 0) {
+    // The status is in the stderr reason. A JSON dump also carries it in the body,
+    // which is the fallback when the reason wording changes.
+    const reported = httpStatusFrom(stderr);
+    if (isBinary) {
+      const failure = ensureWritable(args.writeTo);
+      if (failure !== undefined) {
+        return { ok: false, url: args.url, httpError: true, error: failure, hint: 'Pass an absolute path in an existing directory.' };
+      }
+      writeFileSync(args.writeTo, outcome.stdout);
+      return {
+        ok: false,
+        url: args.url,
+        kind: 'binary',
+        dump: args.dump,
+        path: args.writeTo,
+        bytes,
+        httpError: true,
+        ...(reported === undefined ? {} : { status: reported }),
+        error: stderr.trim() === '' ? 'the server answered with an HTTP error status' : stderr.trim(),
+        hint: `The image the server returned was written to ${String(args.writeTo)}. Drop \`failOnHttpError\` to treat this as ordinary output.`,
+      };
+    }
+    const text = outcome.stdout.toString('utf8');
+    const status = reported
+      ?? (args.dump === 'json' && text.trimStart().startsWith('{') ? parseStatus(text) : undefined);
+    return {
+      ok: false,
+      url: args.url,
+      httpError: true,
+      ...(status === undefined ? {} : { status }),
+      error: stderr.trim() === ''
+        ? `the server answered with an HTTP error status${status === undefined ? '' : ` (${String(status)})`}`
+        : stderr.trim(),
+      content: text,
+      chars: text.length,
+      hint: 'The request reached a server and it refused. The content above is the error page as the engine rendered it, '
+        + 'so it is often worth reading. Drop `failOnHttpError` to treat this as ordinary content instead.',
+    };
+  }
+
   if (outcome.status !== 0) {
     const sandboxTls = isSandboxTlsFailure(stderr);
     // A blocked host fails at connect time ("Failed to connect" / RST) or, when
@@ -426,16 +502,24 @@ export function runFetch(options) {
     // through to the generic hint below used to tell the caller to re-check the
     // installation with moli_probe, which cannot possibly help.
     const evalFailure = args.eval !== undefined && /JavaScript evaluation failed/u.test(stderr);
+    // A binary dump that failed produced no file at all. Saying only "exited with
+    // status 1" leaves the caller to guess whether an image exists; the useful fact
+    // is that it does not, and that a text dump would have captured this page.
+    const binaryFailure = isBinary && httpStatusFrom(stderr) !== undefined;
     const hint = evalFailure
       ? 'The eval expression threw. Check that the page has reached the state you expect (`wait` or `waitSelector`), that the selector matches an element, and that the expression is valid JavaScript. The engine error names the type and position.'
-      : sandboxTls
-        ? 'The DSH file sandbox blocks Schannel client credentials, so moli cannot do TLS at low integrity. Retry the same call with sandbox_permissions "danger-full-access" (with a justification), use an http:// URL, or pass proxy to reach a local forward proxy.'
-        : possiblyUnreachable
-          ? 'The host did not answer in time. Reachability to some hosts varies on this network, so retrying may succeed; if a local forward proxy is available, setting the plugin config field `proxy` is the other option. A certificate problem would report a schannel error instead of a timeout.'
-          : 'Run moli_probe to confirm the install, then retry; use waitSelector or eval for a page that renders late.';
+      : binaryFailure
+        ? 'The engine could not render this page, so no file was written. An HTTP error still produces a body for a text dump (markdown, html or json), so ask for one of those if you need to see what the server returned.'
+        : sandboxTls
+          ? 'The DSH file sandbox blocks Schannel client credentials, so moli cannot do TLS at low integrity. Retry the same call with sandbox_permissions "danger-full-access" (with a justification), use an http:// URL, or pass proxy to reach a local forward proxy.'
+          : possiblyUnreachable
+            ? 'The host did not answer in time. Reachability to some hosts varies on this network, so retrying may succeed; if a local forward proxy is available, setting the plugin config field `proxy` is the other option. A certificate problem would report a schannel error instead of a timeout.'
+            : 'Run moli_probe to confirm the install, then retry; use waitSelector or eval for a page that renders late.';
+    const failureStatus = httpStatusFrom(stderr);
     return {
       ok: false,
       url: args.url,
+      ...(failureStatus === undefined ? {} : { status: failureStatus, httpError: true }),
       error: stderr === '' ? `moli exited with status ${String(outcome.status)} and no output` : stderr,
       sandboxTlsBlock: sandboxTls,
       unreachable,
@@ -450,6 +534,20 @@ export function runFetch(options) {
     if (failure !== undefined) {
       return { ok: false, url: args.url, error: failure, sandboxTlsBlock: false, hint: 'Pass an absolute path in an existing directory.' };
     }
+    // A binary dump produces nothing for an HTTP error: the engine exits 1 before
+    // rendering anything, unlike the text dumps which exit 22 with the body kept.
+    // Nothing is written, so say that rather than leaving an empty file behind.
+    if (outcome.status !== 0) {
+      const status = httpStatusFrom(stderr);
+      return {
+        ok: false,
+        url: args.url,
+        ...(status === undefined ? {} : { status }),
+        httpError: outcome.status === ENGINE_HTTP_ERROR_STATUS || status !== undefined,
+        error: stderr.trim() === '' ? `moli exited with status ${String(outcome.status)} before producing ${String(args.dump)} output` : stderr.trim(),
+        hint: 'The engine could not render this page, so no file was written. A text dump (markdown, html or json) still captures an HTTP error body; binary output does not.',
+      };
+    }
     writeFileSync(args.writeTo, outcome.stdout);
     return {
       ok: true,
@@ -532,6 +630,25 @@ export function runFetch(options) {
   };
 }
 
+/**
+ * Read the HTTP status out of the engine's stderr reason.
+ *
+ * The engine reports it there, in the form
+ *   `Reason: final main document returned HTTP status 404 at ...`
+ * for `--fail` and `--fail-with-body`, whatever the dump format. That makes stderr
+ * the reliable source: a JSON dump also carries the status in its body, but a
+ * markdown or HTML dump does not, and this branch has to name the status for those
+ * too. Matching narrowly keeps an unrelated number in the text from being read as a
+ * status.
+ *
+ * @param stderr - the engine's stderr text.
+ * @returns the numeric status, or undefined when it is not reported.
+ */
+function httpStatusFrom(stderr) {
+  const match = /HTTP status (\d{3})\b/u.exec(String(stderr));
+  return match === null ? undefined : Number(match[1]);
+}
+
 /**
  * Read the HTTP status out of a JSON dump without trusting its shape.
  *

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-moli",
-  "version": "1.1.13",
+  "version": "1.1.14",
   "type": "module",
   "description": "DeepSeek Harness tools for the local Moli headless browser: rendered fetches, markdown, semantic trees, screenshots, and PDFs.",
   "license": "MIT",