dsh-moli 1.1.15 → 1.1.16

package/README.md

@@ -242,8 +242,29 @@ Measured on a page whose `scroll` handler fills a slot: without `scroll` the dum
 `STILL-EMPTY`, with `scroll=1000` it says `LAZY-CONTENT-1000`. It also moves the screenshot, so a
 scrolled `dump=screenshot` shows that part of the page.
 
-It runs as part of the engine's own pre-capture evaluation and therefore composes with `eval`: the
-scroll happens first, and the expression still returns its own value.
+`scroll` is a side effect, so it composes and stands alone:
+
+```
+moli_fetch url=... dump=markdown scroll=600      # scroll, then dump the page
+moli_fetch url=... eval="document.title" scroll=600   # scroll, then read a value
+moli_fetch url=... scroll=600                    # just scroll; you still get the page
+```
+
+The last form is worth spelling out because it was broken. The scroll runs in the engine's own
+pre-capture evaluation, and without `dump` the engine prints whatever that evaluation returns — so an
+earlier version that returned the scroll position made this call answer `600` instead of the document,
+while the identical call without `scroll` returned 1,989 characters of HTML. A scroll reports nothing
+now; it only happens.
+
+### `eval` with `dump` — the expression value is dropped
+
+The engine documents this and it is kept, because using an expression to change a page before
+capturing it is a real pattern. What is not acceptable is that it used to look like a clean answer:
+`eval='JSON.stringify({probe:1})'` with `dump=markdown` returned 900 characters of Markdown, reported
+`ok: true`, and said nothing. The result now carries `evalDiscarded: true` and a note, and the rendered
+output leads with the warning rather than burying it under the body.
+
+If you want the expression's value, pass `eval` without `dump`.
 
 ### `failOnHttpError` — stop reading an error page as content
 
@@ -472,6 +493,8 @@ Eight suites live in [`verify/`](verify/), and all are runnable against a workin
   reaches the engine is indistinguishable from a feature that does not work. It then fetches a 404 and
   checks that `failOnHttpError` reports it while keeping the body, that a healthy page is unaffected,
   and that a screenshot of an error page is still written as a valid PNG rather than decoded as text.
+  It also covers the two silent-answer cases: `eval` with `dump` is flagged `evalDiscarded` and the
+  warning renders first, and `scroll` alone returns the document rather than the scroll offset.
   The end-to-end half needs an HTTP error from a real server: loopback is unreachable from inside this
   sandbox — `curl.exe` times out on `127.0.0.1` while the Node host reaches it — so it uses a path
   under `example.com` and skips when the network is away.

package/README.zh.md

@@ -227,8 +227,26 @@ moli_fetch url=https://example.com/feed dump=markdown scroll=-400    # 往回滚
 `scroll=1000` 时是 `LAZY-CONTENT-1000`。它同样会改变截图，所以带滚动的 `dump=screenshot` 拍到的是页面的
 那一段。
 
-它作为引擎自身"抓取前求值"的一部分执行，因此与 `eval` 可组合：**先滚动，表达式再读取页面，并且仍然返回
-它自己的值**。
+`scroll` 是副作用，因此既可组合也可单独使用：
+
+```
+moli_fetch url=... dump=markdown scroll=600            # 先滚动，再 dump 页面
+moli_fetch url=... eval="document.title" scroll=600    # 先滚动，再取一个值
+moli_fetch url=... scroll=600                          # 只滚动，你仍然拿到页面
+```
+
+**最后一种写法值得单独说明，因为它曾经是坏的。** 滚动在引擎自身的"抓取前求值"里执行，而在没有 `dump` 时，
+引擎打印的就是这个求值的返回值——所以早期版本返回滚动位置，导致这一调用回的是 `600` 而不是页面，而**同样的
+调用不加 `scroll` 会返回 1989 个字符的 HTML**。现在滚动不报告任何东西，它只是发生。
+
+### `eval` 与 `dump` 同用 —— 表达式值会被丢弃
+
+这是引擎写明并有意的行为，予以保留——**用表达式在抓取前改变页面**是真实用法。不能接受的是它**看起来像一个
+干净答案**：`eval='JSON.stringify({probe:1})'` 配 `dump=markdown` 会返回 900 个字符的 Markdown、报
+`ok: true`、什么都不说。现在结果会带 `evalDiscarded: true` 和说明，而且渲染输出**把警告放在最前面**，不会
+被正文埋掉。
+
+要拿表达式的值，就只传 `eval`、不要传 `dump`。
 
 ### `failOnHttpError` —— 别再把错误页当正文读
 
@@ -428,9 +446,10 @@ profile。无论生效的是哪个代理，loopback 地址一律绕过它。
   断言的是**输出**的性质，而不是内部调用。
 - `http-verify.mjs` 直接断言"参数 → 引擎选项"的映射，因为**参数没送到引擎和功能不生效，从外面看是一样的**。
   随后抓一个 404，验证 `failOnHttpError` 会报告失败同时保留响应体、健康页面不受影响、以及错误页的截图仍然被
-  写为合法 PNG 而不是被当文本解码。端到端部分需要一个真实服务器的 HTTP 错误：**本沙箱内无法访问 loopback**
-  ——`curl.exe` 连 `127.0.0.1` 会超时，而 Node 宿主进程可以——所以它用 `example.com` 下的一个路径，网络不可用
-  时跳过。
+  写为合法 PNG 而不是被当文本解码。它还覆盖那两个"静默给错答案"的情形：`eval` 配 `dump` 会带
+  `evalDiscarded` 标记且警告渲染在最前，`scroll` 单独用返回的是文档而不是滚动偏移。端到端部分需要一个真实
+  服务器的 HTTP 错误：**本沙箱内无法访问 loopback**——`curl.exe` 连 `127.0.0.1` 会超时，而 Node 宿主进程
+  可以——所以它用 `example.com` 下的一个路径，网络不可用时跳过。
 
 ### 测试前置条件
 

package/index.js

@@ -174,6 +174,7 @@ const FETCH_OUTPUT_SCHEMA = {
     empty: { type: 'boolean' },
     omitted: { type: 'integer' },
     evalFailed: { type: 'boolean' },
+    evalDiscarded: { type: 'boolean' },
     truncated: { type: 'boolean' },
     error: { type: 'string' },
     hint: { type: 'string' },
@@ -243,6 +244,12 @@ function renderFetch(_args, value) {
     }];
   }
   const parts = [];
+  // A discarded `eval` value is the one case where the output looks like a clean
+  // answer to a question that was never answered, so the warning goes first rather
+  // than after the body, where a 900-character page would bury it.
+  if (value.evalDiscarded === true) {
+    parts.push('Note: the `eval` expression was dropped because `dump` was given too — the engine discards the expression value in that combination. What follows is the page, not the expression.');
+  }
   if (typeof value.note === 'string' && value.note !== '') parts.push(value.note);
   parts.push(String(value.content ?? ''));
   return [{ type: 'text', text: parts.join('\n\n') }];

package/lib/moli.js

@@ -56,6 +56,14 @@ const SCROLL_TOKENS = {
  * content that "below the fold" already contains; it is about triggering the
  * handler, the intersection observer, or the lazy image that only runs on a scroll.
  *
+ * **The expression must not return the scroll position.** Without `--dump` the
+ * engine prints whatever `--eval` evaluates to, so returning `String(window.scrollY)`
+ * here replaced the page with the number `600`: `moli_fetch url=... scroll=600`
+ * answered `600`, while the identical call without `scroll` returned the document's
+ * HTML. The scroll is a side effect; the value it wants to report is recorded on
+ * `window` instead, where a later read can pick it up without changing what the
+ * engine prints.
+ *
  * @param scroll - pixels down (negative scrolls up), or `top` / `bottom`.
  * @returns a JavaScript expression, or undefined when there is nothing to do.
  */
@@ -74,26 +82,55 @@ export function scrollScript(scroll) {
     return undefined;
   }
   // Dispatch a scroll event as well as moving: `window.scrollTo` updates the
-  // position, but a page listening for the event is the reason this exists.
-  return `(() => { window.scrollTo(0, ${target}); window.dispatchEvent(new Event('scroll')); return String(window.scrollY); })()`;
+  // position, but a page listening for the event is the reason this exists. The
+  // height is captured on both sides of the move so a caller can tell whether the
+  // page grew -- which is the one signal that says lazy loading actually happened.
+  return `(() => {
+    const before = document.body === null ? 0 : document.body.scrollHeight;
+    window.scrollTo(0, ${target});
+    window.dispatchEvent(new Event('scroll'));
+    window.__dshMoliScroll = {
+      requested: ${target},
+      y: window.scrollY,
+      heightBefore: before,
+      heightAfter: document.body === null ? 0 : document.body.scrollHeight,
+    };
+  })()`;
 }
 
 /**
  * Combine the caller's `eval` with a scroll, if one was asked for.
  *
- * The two have to share one expression because the engine accepts a single
- * `--eval`. The scroll goes first so the page is in its final state before the
- * caller's expression reads it, and the caller's value still becomes the result.
+ * The two share one expression because the engine accepts a single `--eval`.
+ *
+ * What gets returned depends on whether anything else will produce the output,
+ * and getting this wrong breaks the feature in one direction or the other:
  *
- * @returns the expression to pass, or undefined when neither was given.
+ * - **A `dump` was requested.** The dump produces the output, so the expression is
+ *   there only to scroll first and its value is discarded by the engine anyway.
+ *   It must still be passed, or the scroll never runs.
+ * - **No `dump`, and the caller gave an expression.** The expression value *is* the
+ *   output, and the scroll runs as part of it.
+ * - **No `dump`, no caller expression.** The engine prints whatever `--eval`
+ *   evaluates to, so passing the scroll expression here would replace the page with
+ *   the scroll's return value. `undefined` is returned instead, leaving the engine's
+ *   default output -- the serialized document -- intact. This is the case that
+ *   shipped broken: `moli_fetch url=... scroll=600` answered `600`.
+ *
+ * @param evalExpression - the caller's expression, if any.
+ * @param scroll - the caller's `scroll` value, if any.
+ * @param hasDump - whether the caller also asked for a dump.
+ * @returns the expression to pass, or undefined to let the engine decide.
  */
-export function combineEval(evalExpression, scroll) {
+export function combineEval(evalExpression, scroll, hasDump = false) {
   const move = scrollScript(scroll);
-  if (move === undefined) return evalExpression;
-  if (evalExpression === undefined || evalExpression === '') return move;
-  // The caller's expression is kept last and wrapped so its value is the one the
-  // engine reports. Losing it would contradict the documented `eval` behaviour.
-  return `${move}; (() => (${evalExpression}))()`;
+  const caller = evalExpression === undefined || evalExpression === '' ? undefined : evalExpression;
+  if (move === undefined) return caller;
+  // The caller's expression goes last and is wrapped so its value is the one the
+  // engine reports; losing it would contradict the documented `eval` behaviour.
+  if (caller !== undefined) return `${move}; (() => (${caller}))()`;
+  // No caller expression: keep the scroll only when a dump will produce the output.
+  return hasDump ? move : undefined;
 }
 
 /**
@@ -334,7 +371,11 @@ export function buildArgs(args) {
   if (args.scrollbars === true) argv.push('--scrollbars');
   // `scroll` rides along in the same expression: the engine takes a single `--eval`,
   // and the scroll has to happen before the page is captured.
-  const evaluation = combineEval(args.eval, args.scroll);
+  // `scroll` rides along in the same expression: the engine takes a single `--eval`,
+  // and the scroll has to happen before the page is captured. `hasDump` matters --
+  // see combineEval -- because with a dump the expression exists only for its side
+  // effect, and without one it would become the output.
+  const evaluation = combineEval(args.eval, args.scroll, args.dump !== undefined);
   if (evaluation !== undefined) argv.push('--eval', evaluation);
   if (args.wait !== undefined && args.wait !== '') argv.push('--wait', args.wait);
   if (args.waitSelector !== undefined && args.waitSelector !== '') argv.push('--wait-selector', args.waitSelector);
@@ -561,6 +602,16 @@ export function runFetch(options) {
   }
 
   const text = outcome.stdout.toString('utf8');
+  // `eval` together with `dump` is the engine's documented behaviour -- it discards
+  // the expression's value and dumps the page state the expression produced -- but
+  // it reads as a successful answer to a question that was never answered. The
+  // caller asked for `{probe:1}` and got 900 characters of Markdown. The discard is
+  // kept, because mutating a page before capturing it is a real use, so the result
+  // has to say what happened instead of looking clean.
+  const evalDiscarded = args.eval !== undefined && args.eval !== '' && args.dump !== undefined;
+  const evalNote = evalDiscarded
+    ? 'The `eval` expression produced no value here: `dump` was given too, and the engine discards the expression result in that combination. The output above is the page, not the expression. Run again with `eval` alone for the value, or keep both deliberately when the expression only changes the page before it is captured.'
+    : undefined;
   // A JSON dump is the only format that reports the HTTP status, so an error
   // page is otherwise indistinguishable from content. Surface it rather than
   // letting the model read a 404 body as if the fetch succeeded.
@@ -591,7 +642,10 @@ export function runFetch(options) {
       path: args.writeTo,
       chars: text.length,
       bytes,
-      note: 'Full output saved to file; read it for the complete text.',
+      note: evalNote === undefined
+        ? 'Full output saved to file; read it for the complete text.'
+        : `Full output saved to file; read it for the complete text. ${evalNote}`,
+      ...(evalDiscarded ? { evalDiscarded: true } : {}),
     };
   }
 
@@ -617,6 +671,12 @@ export function runFetch(options) {
     content = text.slice(0, headLen) + mark + text.slice(text.length - tailLen);
     note = `Output truncated: showing the first ${String(headLen)} and the last ${String(tailLen)} of ${String(text.length)} characters, with ${String(omitted)} omitted in the middle. Pass writeTo to keep all ${String(text.length)}.`;
   }
+  // `eval` together with `dump` is the engine's documented behaviour -- it discards
+  // the expression's value and dumps the page state the expression produced -- but
+  // it reads as a successful answer to a question that was never answered. The
+  // caller asked for `{probe:1}` and got 900 characters of Markdown. The value is
+  // still discarded (that combination is genuinely useful for mutating a page before
+  // capturing it), so the result has to say so rather than look clean.
   return {
     ok: true,
     url: args.url,
@@ -626,7 +686,11 @@ export function runFetch(options) {
     chars: text.length,
     bytes,
     truncated,
-    ...(truncated ? { omitted, note } : {}),
+    ...(truncated ? { omitted } : {}),
+    ...(evalDiscarded ? { evalDiscarded: true } : {}),
+    ...(note === undefined && evalNote === undefined
+      ? {}
+      : { note: [note, evalNote].filter(part => part !== undefined).join(' ') }),
   };
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-moli",
-  "version": "1.1.15",
+  "version": "1.1.16",
   "type": "module",
   "description": "DeepSeek Harness tools for the local Moli headless browser: rendered fetches, markdown, semantic trees, screenshots, and PDFs.",
   "license": "MIT",