weflow-cli 1.11.1 → 1.12.0

package/CHANGELOG.md

@@ -17,6 +17,168 @@ purpose** - a hand-written "latest is X" is exactly the claim that goes stale he
 
 All notable user-facing changes are recorded here. This project follows [Semantic Versioning](https://semver.org/).
 
+## Unreleased
+
+## 1.12.0
+
+- **One word per question, across every artifact that describes exported media.** The auxiliary `*_media.json`
+  called a document entry `exported` when it meant "the local copy was **found**", while `manifest.json` used
+  `exported` for "it is **in the assets directory**" - two different questions wearing one word, and a copy
+  failure would have made "已导出" false. Now: the report answers *locatable in the source data* (`found` /
+  `missing`), the manifest answers *materialized in the export directory* (`written` = what is there now,
+  `copied` = what this run added, `unavailable` = not written, whether because the file is absent, oversized,
+  or the copy itself failed). `manifest.documents` gained the clearer `written` (it was `exported`) and the
+  CLI's JSON for all three branches (html / md / `--media`) is now produced by **one** function
+  (`media_json_summary`) instead of three hand-written blocks - so the formats cannot describe the same fact
+  differently, and the summary carries the report's file name everywhere (previously only the HTML branch did).
+  Pinned by tests: a file whose copy fails reports `found` in the report and `unavailable` in the manifest
+  (never "exported"), and the summary is asserted to carry counts, the report name, and **no absolute path**.
+
+- **The article half of the semantic index was embedding YAML metadata, not article text.** `collect_articles`
+  found the body by looking for a `## 正文` marker; measured on 2026-10-15, **none** of the 10,759 daily `.md` files
+  in the default window contains it (the real shape is `frontmatter → # title → > source → --- → ## AI 摘要 →
+  --- → repeated title → body`), so every record silently fell back to `content[:1000]` - the frontmatter. Sampled
+  300 of them: 100% contain `title:`, median CJK ratio **0.16**; the only content signal left was the title, which
+  is what "the article line ranks by title" actually meant. The extractor now reads the real structure (median CJK
+  ratio 0.16 → **0.80**, 299/300 files changed) and a test fails if it is reverted to the old fallback. Note that
+  an existing index does **not** pick this up by itself: an incremental build embeds new ids only, so it takes
+  `search-index --full`.
+- **Building the index can be interrupted without losing work already paid for, and it says where it is.** A full
+  rebuild of ~16k records takes over twenty minutes while the CLI gave its subprocess **five** - so `--full` had
+  never completed through the CLI at all, and since nothing was written until the very end, a timeout threw away
+  the whole run. It now flushes every 100 batches (vectors first, then `meta.json`, each through a temporary file
+  and `replace`), so a killed run leaves a **valid** partial index and a plain re-run continues from there, because
+  the flushed ids are already in `meta.json`. Batch failures (measured: `IncompleteRead` from the embedding
+  endpoint) are now **excluded** instead of being written as all-zero vectors - such ids would otherwise sit in
+  `meta` forever, since an incremental build only ever embeds new ones - and the count is reported as
+  `embedFailed`. Progress lines (`[索引] 已嵌入 N/M`) go to stderr.
+- **A mismatched index no longer degrades into "nothing found".** When `vectors.npy` holds more rows than
+  `meta.json` (a build killed between the two writes, or files edited by hand), the top-k indices ran past the end
+  of `meta` and the bare `except` in `search()` turned that `IndexError` into a silent fallback to keyword search -
+  a real query that presents as "no hits". The two are now truncated to the shorter one, which costs a few results
+  instead of the whole vector path.
+
+- **`search --hybrid`: an opt-in second ranker, off by default because it was measured not to earn it.** The new
+  flag fuses the vector ranking with a fully local BM25 ranking (Chinese is tokenised into **character bigrams**,
+  so no segmenter and no word list; one pass over the indexed texts, no extra index file, ~31 ms for 16k records)
+  using Reciprocal Rank Fusion. It fixes one concrete failure - the rarest word in a question being overridden by
+  the most frequent one: `卫星遥感反演气溶胶光学厚度` ranks the satellite-based PM2.5 paper 3rd on vectors alone
+  and 2nd with the flag. But measured on the same day, it is **not better** on four of five realistic questions
+  (it pushed the correct first hit out of the top five for `有没有讲遥感或卫星数据用在空气质量监测上的文章`, and
+  `后向轨迹 HYSPLIT 怎么用` was dragged off by function words), and on 16 verbatim-fragment queries BM25 alone
+  scored 16/16 against the fusion's 12/16 - RRF gives both sides equal voting power, and Chinese function words turn
+  BM25 into noise. So the default stays as measured-better and the flag is for questions that really do contain a
+  rare term. Note it also makes the printed `score` (cosine) non-monotonic, because the order now comes from the
+  fused ranks. The decision and both A/B tables are in D-110; making it a default would first need idf-gated
+  voting and a re-measurement on the same harness.
+
+- **`search --from <account>`: filter by source account, because that is an attribute query and vectors cannot
+  answer it.** Measured on the 2026-09-12..10-13 window (10,759 articles, 413 accounts): searching the account name
+  itself returns only 0-3 of the top 10 from that account for 红星新闻 (908 articles), 政事儿 (632) and 新京报 (529),
+  while the account name is present in **every** one of its records' `id` (the daily names files
+  `<account>-<title>.md`). The flag filters on that id as a substring (so `--from 红星` works), applies to the article
+  line only - a chat message has no source account - respects the same mask as the keyword side when `--hybrid` is on,
+  and says "the index has no records for this account" rather than returning an empty list silently. It gives the
+  right pool, not a better order within it: ranking inside an account is still semantic nearest-neighbour, so a
+  general-news account can return its own unrelated stories for a topical query.
+
+- **A recall/precision evaluation harness now exists, and it immediately paid for itself: gated keyword fusion is
+  the new default.** `scripts/recall_eval.py` measures `search` with the standard metric set (Recall@1/5/10, MRR@10,
+  nDCG@10, Hit@1) plus a **coverage** count, and builds its ground truth at run time (a uniqueness-checked 12-character
+  fragment from the middle of a record's own text, and its title / first 30 characters) so no corpus text is committed;
+  hand-written questions go in an out-of-repo TSV (`question<TAB>anchor<TAB>line`). It requires `--yes` because each
+  question costs an embedding and leaves the machine, and it writes nothing. With that harness, the keyword side of
+  `--hybrid` gained two gates - only documents matching a **rare** term (df/N <= 5%) vote, their BM25 score must be
+  within 0.5x of the best, and a query with no rare term makes the keyword side **abstain** - and on the same three
+  batteries (40 fragment + 40 title + 12 hand-written questions) that turns the ungated version's zero gain into:
+  fragment recall@10 0.91 -> **1.00**, recall@5 0.89 -> 0.97, hit@1 0.60 -> **0.80**, title hit@1 0.83 -> 0.86, and
+  no change on the question battery. Nothing got worse, so **fusion is on by default** (`--no-hybrid` restores plain
+  vector order, and note the printed `score`, a cosine, is no longer monotonic). The ungated variant scoring exactly
+  like pure vectors is what explains D-110's earlier "harmful" measurement: the problem was function words voting,
+  not fusion. Threshold sensitivity is flat (0.02/0.7 and 0.10/0.3 bracket the same result), so the value is not
+  load-bearing; it must not be re-tuned on this battery.
+
+- **The rerank half of the pipeline works again - it now falls back to the configured chat model, and its state is
+  visible in the output.** Rerank was never *running* on this machine: the Jev/Typesafe contract returns
+  `HTTP 451 not available in your region`, and the failure was only a stderr WARN, which the CLI does not forward -
+  so a search reported nothing and looked like "fine". `rerank_dispatch` now tries the contract first and, if that
+  client fails or is absent, reranks with the configured OpenAI-compatible chat model; both failing keeps the original
+  order, and `RERANK_STATE` (`provider` / `degraded` / `error` / `scored`) travels into the JSON so "not reranked" can
+  no longer be mistaken for "reranked". Passages are cleansed before they are sent (bold markers, 公众号 boilerplate,
+  bylines, image/link syntax), because the excerpt is only 300 characters. Measured on the same 40 questions: hit@1
+  0.55 -> **0.65**, recall@5 0.65 -> **0.80**, MRR@10 0.59 -> **0.72**, nDCG@10 0.63 -> **0.74**, with the weakest
+  cell (verbatim fragment on the article line) going recall@5 0.30 -> **0.70** - in the same range the literature
+  reports for retrieve-and-rerank (MS MARCO nDCG@10 0.396 dense -> 0.452 with a reranker; a public example puts
+  hybrid-RRF 32.95 -> 36.16 with rerank). Two consequences worth knowing: **every search now spends one model call**
+  (~2-3k tokens, ~¥0.004, plus a few seconds) and the assistant, MCP and `chat` share this path, so `--no-rerank`
+  is the escape; and one regression is recorded rather than smoothed over - title/article recall@5 slipped 1.00 to
+  0.90 (one case at n=10), whose fix (bigger candidate pool, or protecting the retrieval-side top hit) is not done.
+
+- **The rerank candidate pool is now 100 by default, because the pool was the biggest lever all along.** Rerank can
+  only reorder what retrieval hands it: with a pool of 20, a correct document ranked 21st by the hybrid ranking could
+  never appear. Measured on the same 40 questions with only the pool changed - 20 / 50 / 100 / 200 gives recall@10
+  **0.82 / 0.88 / 0.95 / 0.95**, MRR@10 0.72 / 0.75 / **0.80** / 0.82, nDCG@10 0.75 / 0.78 / **0.83** / 0.85, and in
+  the weakest cell (verbatim fragment on the article line) recall@10 0.70 / 0.80 / **1.00** / 1.00 - so 100 is the new
+  default and 200 buys nothing measurable. Against published production thresholds (Recall@10 >= 0.90, MRR >= 0.75)
+  this now clears both. The pool is adjustable per query (`--rerank-pool`, CLI and script; when the flag is absent
+  nothing is forwarded, so the default lives in exactly one place - the first version parsed the commander option
+  unconditionally and **made a missing flag a fatal invalid argument**), and the cost is stated plainly: the rerank
+  prompt grows from ~2.5k to ~20k tokens (~¥0.01 per search, a few seconds). Two corrections to the previous entry:
+  the title/article recall@5 "regression" was **noise** (rerank is not perfectly deterministic even at temperature 0,
+  and each cell holds 10 questions, so one case is 0.10) - it re-measures at 1.00; and per-cell numbers at n=10 must
+  not be used to judge a single change.
+
+- **Results are collapsed by identical text now, because multi-account reposts were eating slots.** Measured on the
+  current defaults (rerank with a pool of 100): **7%** of 100 top-5 slots held a duplicate of another item in the same
+  result list, and **6 of 20** questions showed one - e.g. the 环境监测新规 query had a five-department notice carried
+  by three accounts plus a meeting report carried by three more, so four slots went to two pieces of content. The
+  first (best-ranked) copy is kept and the others are counted into a new `duplicateCount` field, so the fact that
+  several accounts carried it stays visible instead of silently shrinking the list; `--no-dedup` returns the raw
+  list. This deliberately happens at presentation time and **not at index time**: the index keeps every copy because
+  `--from <account>` resolves an account through the record's `id`, and dropping copies there would make per-account
+  lookups lose records. It changes no recall metric (the ground truth is content) - only how much distinct material
+  one result list actually carries.
+
+- **`search --with-text` returns the article itself, not the excerpt it was matched on.** What a search used to hand
+  back was the title plus the ~500-character slice that the embedding was computed from - which is not the content
+  anyone wants. The content is already local: the record's `id` carries the daily markdown's relative path, and the
+  fetch cache (`output/.cache/fetch/<md5(url)>.md`) holds the **full** body. The flag adds `fullText` (readable: bold
+  markers, image syntax and 公众号 boilerplate lines removed, paragraphs kept - deliberately different from the
+  300-character, whitespace-collapsed excerpt the reranker receives), `sourceFile` (always given, so the raw file can
+  be compared against what was cleaned) and `url`; the longer of the daily markdown and the cache wins. Chat records
+  need none of this - the indexed text *is* the whole message. It is **off by default** for a context reason rather
+  than a cost one: the assistant, MCP and RAG share this `search()`, and ten results at 4,000 characters would push
+  ~20k characters into those paths. Unreadable records return `fullText: null` with `sourceFile` still set (never a
+  silent empty string). Measured on the demo query, the first hit goes from a 500-character excerpt to 1,723
+  characters of real content (citation, three highlights, abstract, and the results paragraph: R² 0.87/0.86/0.83,
+  AOD feature importance 40.63%); before image syntax was stripped it was 3,860 characters, ~2,100 of which was one
+  line of `![](https://mmbiz.qpic.cn/…)`.
+
+- **`--fetch-missing` fills in a thin body by fetching the article, through the daily pipeline's own cache.**
+  Some daily markdown files are stubs (a "好文回顾" record held 201 characters), and `--with-text` could only hand
+  back that stub. With this flag, any record whose local body is under 300 characters **and** that carries a `url`
+  is fetched once; the result replaces `fullText`, is marked `fetchedFromUrl`, and is written to the **same** cache
+  the daily and backfill use (`output/.cache/fetch/<md5(url)>.md`, success-only, cache-before-network), so whichever
+  path fetches it, the other gets it free next time. It is off by default (a search should not generate upstream
+  requests on its own), capped (3 records unless a number is given), and only touches thin bodies - measured on one
+  query, a 201-character stub became 965 characters of abstract and findings while a 3,020-character neighbour was
+  left alone. Failures keep the local text and are reported rather than blanking the field. It deliberately does
+  **not** sleep between fetches the way the daily's batch path sleeps 8-12 seconds: that is politeness for hundreds
+  of articles, this is three on demand. Same caveat as the daily: the cache is the version fetched at that time, so a
+  revised source article will not be reflected.
+
+- **MMR redundancy removal was built, measured, and left off by default - it removes useful material here.** The
+  new `search --mmr` flag re-orders the final results with Maximal Marginal Relevance (lambda 0.7, the value WeKnora
+  uses) and the evaluation harness grew a **diversity metric** (mean pairwise cosine inside the returned top-k). It
+  does what it says on the ranking side: diversity goes from 0.627 to 0.559 with recall@10 unchanged at 0.95. But
+  judged precision (same criterion and judge as the precision baseline: 10 questions, top-5 each) **drops** - strict
+  relevant 40% -> 30%, loose 68% -> 64% - and recall@5 slips 0.93 -> 0.90. The reason is worth keeping: in this
+  corpus the "duplicates" are mostly **several accounts carrying the same story**, i.e. multi-source confirmation
+  rather than noise, so pushing redundancy down starts deleting the very items that answer the question. The cheap
+  half of that problem is already handled by default same-text collapsing (D-114), which costs nothing and loses no
+  information. Tuning lambda is not the fix (larger just degenerates to plain relevance, smaller deletes more): the
+  direction is wrong, not the weight.
+
 ## 1.11.1
 
 - **Five defects in 1.11.0's document export, all found by looking at a real export instead of the code.**

package/OPERATIONS.md

@@ -410,6 +410,11 @@ weflow-cli config set dashscopeApiKey "sk-..."
 而不会拿一个空 key 去请求。`--api-key` 参数或 `DASHSCOPE_API_KEY` 环境变量仍然优先，
 临时用一次时更方便。
 
+**这一步会把片段发到阿里云。** 建索引和每次检索都要走百炼的 embeddings 接口，送出去的是**每条素材的一小段纯文本**——
+文章是「标题 + 正文前 500 字」、聊天消息是**消息内容前 500 字**；数据库、密钥、图片、文件路径、好友名都不在里面。
+实测按默认窗口建一份 16,027 条的索引，累计送出约 587 万字（≈ ¥2，估算口径与 `search-index --dry-run` 相同）；
+之后每查一次只送一个问句（十几个 token，不到一分钱）。要完全不联网检索就用 `vault search`。
+
 顺带一提，`favPassphrase`（解锁收藏与 biz 库的口令）现在也能这样设置了——
 
 ```powershell
@@ -432,6 +437,13 @@ weflow-cli search-index --days 3650 --article-days 365 --yes   # 要"整段历
 - 默认值与从前**完全一致**（90 / 30）——加参数不是改代价，只是把原来写死、改不了的数字
   变成改得了的。想建"整份聊天记录"的知识库，就显式给一个大窗口。
 - **天数越多，发给向量服务的文本越多**，这是出境成本，所以确认语里会写明这次用的窗口。
+
+**全量重建要二十分钟以上，而且现在可以被打断。** 实测 16,000 条左右（聊天 90 天 + 日报 30 天）约 20–25 分钟，
+比 CLI 给子进程的超时还长（原先只有 5 分钟，已改成 30 分钟）——所以它是**每 100 批往盘上落一次**的：中途被杀
+也只丢最后不到一千条，再跑一次 `search-index`（**不加 `--full`**）就从断点接着嵌，已经在盘上的 id 不会重复付费。
+跑起来后 stderr 每落一次盘打一行 `[索引] 已嵌入 N/M`。某批请求失败时（实测见过 `IncompleteRead`）那些记录
+**不写进索引**（写进去的话它们会永久占位、再也不会被重嵌），数量在结果的 `embedFailed` 里，重跑即可补上。
+
 - 上限 36500 天（≈100 年）；`0` 会被拒——0 天等于建出一个**空**索引，而它看起来像"建好了"。
 
 ### 知识库框架：一图看懂
@@ -593,6 +605,8 @@ weflow-cli wiki lint --json     # 机器可读
 ```powershell
 weflow-cli search "PM2.5 对植物用水效率的影响"
 weflow-cli search "..." --no-rerank     # 只按向量/关键词相似度，不调决策模型
+weflow-cli search "HYSPLIT 后向轨迹" --no-hybrid   # 只按向量排（默认融合本地关键词名次，见下）
+weflow-cli search "暗访" --from 红星新闻        # 只查这个公众号（属性筛，不走语义）
 weflow-cli search "项目排期" --line chat   # 只看聊天那条线
 weflow-cli search "..." --line all         # 两条线一起（默认只搜文章线）
 ```
@@ -611,7 +625,64 @@ weflow-cli search "..." --line all         # 两条线一起（默认只搜文
 - 返回里 `score` 仍是相似度（含义没变），重排的分数在 `rerankScore`。
 - 没配 `typesafeApiKey`、连接失败或服务过载（529）时，**原序返回并打一行 WARN**，
   检索本身不会因此失败。瞬时错误会自动重试最多 2 次。
+- **本机实测重排是不生效的**（2026-10-15）：Typesafe 返回 `HTTP 451: not available in your region`，
+  所以顺序就是裸相似度，等同 `--no-rerank`。而这行 WARN 只有**直接跑 Python 脚本**时看得见——CLI 没转发
+  子进程的 stderr，`search` 看起来一切正常。这是已知缺口，别把"没报错"读成"重排过了"。
 - 想确认重排有没有帮上忙：同一个查询跑一次带重排、一次 `--no-rerank` 对比顺序即可。
+- **重排现在是真在工作**（2026-10-15 起）：先试 Jev/Typesafe 契约，失败或不可用就退到**配置里的对话模型**
+  （本机能连上的那个）。两段都失败就保持原序，并把 `provider` / `degraded` / `error` / `scored` 写进 JSON 的
+  `rerank` 字段 —— 别把"没重排"读成"重排过了"。送进模型前会先**清洗段落**（去加粗标记、公众号套话、署名行、
+  图片与链接语法），因为摘录只有 300 字。
+- **代价**：每次检索多一次模型调用（约 2–3k tokens ≈ ¥0.004）与几秒延迟；`search`、助手的 `search_semantic`、
+  MCP 的 `search_knowledge`、`chat` 共享同一条路，所以一起受影响。要省/要快就 `--no-rerank`。
+  实测效果（同一批 40 条题）：hit@1 0.55 → **0.65**、recall@5 0.65 → **0.80**、nDCG@10 0.63 → **0.74**；
+  最弱的"碎片/文章"那格 recall@5 0.30 → **0.70**。
+- **默认会融合一次本地关键词名次**（门控 BM25 + RRF，2026-10-15 起；`--no-hybrid` 关掉）。它解决的是一种具体的失效：问句里
+  最稀有的那个词被出现最多的词带走 —— 实测「卫星遥感反演气溶胶光学厚度」在纯向量下把"基于遥感的 PM2.5 反演"
+  排到第 3，开了它到第 2。但它在**另外 4 条实测问句上都不优于纯向量**（「有没有讲遥感…」被挤出前五、
+  「后向轨迹 HYSPLIT 怎么用」被"怎么用"这类虚词带偏），所以默认不关它是不划算的：**问句里确实有稀有词时才开**。
+  两点要知道：关键词侧全本地、不联网、不花钱（16k 条约 31 ms）；开了之后 `score`（余弦）可能不再单调，
+  顺序来自融合名次而不是分数。
+
+- **重排看多少条候选：`--rerank-pool`（默认 100）**。这是 2026-10-16 量出来的**最大杠杆**：重排只能重排
+  它拿到的候选，池子 20 时正确答案只要在融合排名 21 名以外就永远进不来。同一批 40 题：池子 20 → 100 让
+  **recall@10 0.82 → 0.95、MRR@10 0.72 → 0.80、最弱的"碎片/文章"格 0.70 → 1.00**；100 → 200 只剩噪声级增益。
+  代价是每次重排的提示从约 2.5k tokens 涨到约 20k（≈ ¥0.01/次 + 几秒）；嫌贵就 `--rerank-pool 20`，
+  这比整个 `--no-rerank` 温和。
+- **同文本转载默认折叠**：同一篇文章被几个号转发时，结果里只出现一次（`duplicateCount` 告诉你有几个号发过）。
+  实测 20 条问句的 top-5 里 7% 是这类重复（"环境监测新规"那类通稿能一次占掉 4 个名额）。`--no-dedup` 关掉。
+  它**只改展示、不改排序**（不影响 recall 指标），也不需要在建索引时去重 —— 索引侧去重会破坏 `--from <公众号>`。
+- **要正文就加 `--with-text`**：默认返回的是"用来匹配的摘录"（标题 + 前 500 字），不是内容。加上它，每条会多给
+  `fullText`（能读的正文：去标记/图片语法/公众号套话，保留段落）、`sourceFile`（本地日报 md 路径）、`url`（原文链接）。
+  正文本地就有两处、**取更长的那份**：日报 md，以及正文缓存 `output/.cache/fetch/`（回填/日报抓来的完整正文）。
+  聊天记录里索引那段**本来就是消息全文**。`--with-text 2000` 可限每条字数（默认 4000）；读不到的会留 `None`
+  并保留 `sourceFile`，不会假装成"这篇没正文"。常用组合：`search "…" --top-k 3 --with-text`。
+- **正文太薄就按 url 抓：`--fetch-missing [N]`**（默认关；不写字数时最多抓 3 条）。判据是"本地正文短于 300 字
+  且记录里有 `url`"，抓到的写回 `fullText` 并标 `fetchedFromUrl`；它**复用日报的同一份缓存**
+  （`output/.cache/fetch/`，只缓存成功），所以抓过的日报重跑直接命中。实测：某条 201 字的"好文回顾"→ 965 字
+  （完整摘要与结论），而同一趟里本地已 3,020 字的那条**没有被碰**（只抓薄的）。
+  与日报的 8–12 秒批量节流不同：那是几百篇的礼貌，这是几条的按需 —— 这个差异是刻意的。
+- **`--mmr` 压近似邻居（默认关，2026-10-16 量过）**：把最终结果按 MMR（λ=0.7，照 WeKnora）去冗余，
+  压掉"讲同一件事"的近似条目。它确实让结果更不重复（top-10 两两余弦 0.627 → 0.559），但**判官口径下的准确率
+  掉了**（严格相关 40% → 30%，宽松 68% → 64%），recall@5 也掉 0.03 —— 因为这份语料里的"重复"多是
+  **多个号转载同一件事**，MMR 把有用的重复一起删了。所以默认关；要"同一件事只留一条"用默认的
+  **同文本折叠**（免费、不损信息），别用 MMR。
+### 检索质量怎么量（评测台）
+
+改了检索（换模型、调窗口、动排名）之后，"变好还是变坏"要用同一批题量，不能靠手工试问：
+
+```powershell
+python scripts/recall_eval.py --records 30 --yes      # 自动造题：碎片 + 标题，各线各 30 条
+python scripts/recall_eval.py --records 30 --dedup --yes   # 顺便量"同文本多号转载"吃掉多少名额
+python scripts/recall_eval.py --queries-file D:/q.tsv --records 0 --yes   # 只用你手写的问句
+```
+
+指标是标准口径：Recall@1/5/10、MRR@10、nDCG@10、Hit@1，外加**覆盖**（答案在库里根本不存在时不算
+排序失败）。`--queries-file` 的格式是 TSV：`问题<TAB>答案里必然出现的字面量<TAB>线`，**请放在仓库外**
+（锚串取自你的语料）。它要花钱（每题一次 embedding）且问句出境，所以必须 `--yes`；不写任何文件。
+
+2026-10-15 的基线（索引 15,995 条）：标题/开头 30 字 recall@10 = 0.97–1.00；12 字正文碎片
+recall@10 = 1.00、hit@1 = 0.80（**这是门控融合开了之后的数**，纯向量是 0.91/0.60）。
 
 ### 语音消息转文字
 
@@ -641,6 +712,12 @@ python scripts/wechat_voice.py --db "<message_0.db>" --key <key> --salt <salt> `
   --passphrase <passphrase> --talker "<会话id>" --cache-dir "output\.voice-cache"
 ```
 
+- **`--from <公众号>` 按号筛，走的是属性不是语义。** 号名不住在嵌入文本里（它住在 `id`/文件名
+  `<公众号>-<标题>.md` 里），所以按号查必须精确筛。实测窗口 2026-09-12..10-13（10,759 篇 / 413 个号）：
+  红星新闻 908 篇、政事儿 632 篇、新京报 529 篇，而不筛号直接搜号名时，前 10 条里只有 0–3 条真是那个号。
+  三点要知道：`--from` 是**子串匹配**（给"红星"也认）；聊天消息没有"公众号"这个属性，所以它实际只作用于
+  文章线；筛不到时会明确说"索引里没有这个号的记录"，而不是安静返回空。**它给对的池子，不保证池子里的排序**
+  —— 号内检索仍是语义最近邻（实测一个综合新闻号里搜"暗访"，前五条有两三条是别的主题）。
 - **有 NVIDIA GPU 时自动走 GPU**，约快 30 倍（实测 0.07 秒/条 vs 2.0 秒/条）；GPU 不可用则退回 CPU，不会报错。需要 `pip install nvidia-cublas-cu12 nvidia-cudnn-cu12`。
 - 语言**自动检测**。强制 `--language yue` 在粤语模型上会返回空，别加。
 - 中断后重跑会从缓存未命中的地方继续，不会从头再来。
@@ -863,6 +940,42 @@ schtasks /create /tn "WeFlow Daily" ^
   不需要手工删。
 - **别把触发条件设成"上次成功才跑"**：那样第一次失败之后就再也不会被触发了。
 
+#### 两种形态：完整日报 vs 历史回填（先认清你要哪种）
+
+`output/biz-daily/<日期>/` 里可能是两种东西，**从 README 第一行就能分辨**：
+
+| | 完整日报（`daily`） | 历史回填（`backfill_articles.py`） |
+| --- | --- | --- |
+| README 开头 | `# 公众号日报` | `# 公众号文章 — <日期>（历史回填）` |
+| 内容 | AI 摘要 + 图片本地化 + `index.html` 阅读页 + 行动建议 | 只抓正文，**不下载图片、不调用模型** |
+| 成本 | 一天约 1 小时（含 8–12 秒/篇的节流）+ AI 费用 | 并发抓取，几分钟一天，**不花钱** |
+| 排期命令 | `schtasks /create /tn "WeFlow Daily" /tr "weflow-cli daily --yes" /sc daily /st 08:20` | `schtasks /create /tn "WeFlow Backfill" /tr "python <仓库>\scripts\backfill_articles.py --since <昨天> --until <昨天> --workers 4 --yes" /sc daily /st 08:20` |
+
+两条**可以并存**，但要知道区别：回填那条便宜得多、跑得快，代价是没有摘要/图片/网页。
+本机实测（2026-10-14）：归档里 359 天中只有 12 天是完整日报形态，其余都是回填形态——
+也就是说"日报断了半个月"这类现象，多数时候断的是**回填**那条，而它同样需要有人触发。
+
+> 回填与完整日报**共用一份正文缓存**（`output/.cache/fetch/`，D-108）：回填抓过的正文，之后
+> `daily --date X` 会直接命中、不再联网、也不等节流。但**别把接力读成"那天就免费了"**——日报对
+> 每篇有 url 的文章都要抓完整正文，回填只抓"本地正文过短"的那些，所以 2026-10-13 那天 485 篇里
+> 只有 145 篇在缓存里。想强制重新抓正文用 `--no-fetch-cache`（两边同名同义：跳过"读"，成功仍写回）。
+
+#### 怎么看出某天"没做完"
+
+按顺序看三个地方，一步就能定性：
+
+1. **目录是不是空壳**：只有目录、没有 `README.md`/`.articles.json` → 那次在抓取阶段就断了。
+2. **`.articles.json` 里有没有 `fetchFailed`**：这是"这一轮有 N 篇正文没抓到"，`day_done` 把它读作
+   **没做完**，所以再跑一次只会重试那几篇（其余走缓存）。README 里也会写明篇数。
+   ⚠️ 与此对应的是 `topicFilter`（只回填了部分主题）和 `truncated`（被 `--limit-per-day` 截过）——
+   三种都表示"这天不完整"，但只有 `fetchFailed` 会在下次零失败的跑动后消失。
+   **怎么判断剩下的失败是"重试可得"还是"永久没了"：连着跑两次，看失败数。** 2026-10-14 实测一轮
+   13 天：首次 68 篇失败 → 重试后 43 篇，**再重试仍是 43 篇且逐日数字完全相同** ⇒ 触底了，那些
+   正文在源站已经取不到（同仓库更早的实测：84 次失败里 28 次重试可得、44 篇是被删除或违规下架）。
+   重试很便宜（命中缓存的那些不发请求），所以"再跑一次"是安全的确认动作，而不是赌运气。
+3. **计划任务的上次结果**：`schtasks /query /tn "WeFlow Daily" /fo LIST /v`（或换成你给任务起的名字）。
+   任务不存在 / 从没运行 / 上次失败，是三种不同的情况，别混成"日报坏了"。
+
 ## 6. AI 和助手
 
 日报关闭 AI 不会自动关闭其他命令的 AI。报告、RAG、助手和证据线索分析分别按命令参数和配置决定是否调用模型。云端分析前应确认输入范围、供应商和隐私设置；优先使用本地模型处理聊天正文。

package/bin/weflow-cli.ts

@@ -14,6 +14,7 @@ import { configService } from '../src/services/configService.js'
 import { chatService } from '../src/services/chatService.js'
 import { emptyMessagesNote, looksLikeTalkerId } from '../src/services/messageQuery.js'
 import { exportService } from '../src/services/exportService.js'
+import type { ExportResult } from '../src/services/exportService.js'
 import { writeEvidencePackage } from '../src/services/evidenceService.js'
 import { resolveTalker as resolveTalkerCore } from '../src/utils/talkerUtils.js'
 import { getPythonCommand } from '../src/utils/python.js'
@@ -1874,8 +1875,12 @@ program
     }
     // 图片要靠 Python 那条路匹配（缓存缩略图、资源 md5、原图那档）。`--media` 就是"先抽一次"，
     // 抽完 json/txt/excel 各自消费同一份清单 —— 匹配只跑一遍，也不会在 TS 里重写一遍。
+    // 抽取那一步的媒体摘要留在这里：`--media` 的调用者（机器调用）应当能从结果里直接读到
+    // "各状态多少条、报告叫什么"，而不必去翻 Python 的 stdout。
+    let extracted: ExportResult | undefined
     if (opts.media && (format === 'json' || format === 'txt' || format === 'excel')) {
       const media = await exportService.extractMedia(talker, opts.output, !!opts.json, { fullImages: opts.fullImages === true }, limit, opts.date || '')
+      extracted = media
       if (!media.success) {
         if (opts.json) console.log(JSON.stringify({ success: false, code: 'MEDIA_EXPORT_FAILED', error: media.error || '媒体抽取失败' }))
         else console.log(chalk.red(`✗ 媒体抽取失败: ${media.error}`))
@@ -1909,7 +1914,16 @@ program
     }
 
     if (result?.success) {
-      if (opts.json) console.log(JSON.stringify({ success: true, format, contract: format === 'json' ? opts.contract : null, path: result.path, count: result.count ?? null }, null, 2))
+      if (opts.json) console.log(JSON.stringify({
+        success: true,
+        format,
+        contract: format === 'json' ? opts.contract : null,
+        path: result.path,
+        count: result.count ?? null,
+        // 媒体摘要：html/md 那条由导出自己产出（Python 的 media_json_summary），
+        // json/txt/excel 的 `--media` 则由上面那次抽取产出 —— 两条都是同一份词汇。
+        media: result.media ?? extracted?.media ?? null,
+      }, null, 2))
       else console.log(chalk.green(`✓ 导出成功: ${result.path}`))
     } else {
       if (opts.json) console.log(JSON.stringify({ success: false, code: 'EXPORT_FAILED', error: result?.error || '导出失败' }))
@@ -4985,6 +4999,13 @@ program
     .option('--top-k <n>', '返回数量', '10')
     .option('--line <line>', 'wiki=文章线(默认) / chat=聊天记录 / all=两条都搜', 'wiki')
     .option('--no-rerank', '不调用决策模型重排，只按向量/关键词相似度返回（回退到引入重排之前）')
+    .option('--no-hybrid', '不融合本地 BM25 名次（默认融合；门控后实测只增不减）')
+    .option('--rerank-pool <n>', '重排看多少条候选（默认 100；越大越准也越贵）')
+    .option('--no-dedup', '不折叠同文本的多号转载（默认折叠，结果里带 duplicateCount）')
+    .option('--with-text [n]', '给每条命中挂上本地正文（fullText / sourceFile / url）；n 为每条最多多少字')
+    .option('--fetch-missing [n]', '正文太薄时按 url 抓一次（最多 n 条，默认 3；复用日报缓存）')
+    .option('--mmr', '对结果做 MMR 去冗余（压掉近似的同领域邻居，可能牺牲单点召回）')
+    .option('--from <account>', '只查这个公众号的文章（按文件名匹配，不走语义）')
     .option('--api-key <key>', 'DashScope embedding API key')
     .option('--dry-run', '仅预览，不读取索引或调用向量服务')
     .option('--yes', '确认执行可能调用云端向量服务的搜索')
@@ -4996,6 +5017,10 @@ program
     const pkgRoot = resolvePackageRoot()
       const script = join(pkgRoot, 'scripts', 'semantic_search.py')
       const topK = parseCliInteger(opts.topK, 'top-k', 1, 100, opts.json)
+      // 没传就**不转发**（默认值只在 Python 侧定义一处）：commander 对未提供的选项给 undefined，
+      // 直接 parseCliInteger 会被自己的校验判成非法参数 —— 实测就是这么红的。
+      const rerankPool = opts.rerankPool
+        ? parseCliInteger(opts.rerankPool, 'rerank-pool', 5, 500, opts.json) : null
       requireCliChoice(opts.line, ['wiki', 'chat', 'all'], 'line', opts.json)
       const preview = {
         success: true,
@@ -5004,6 +5029,13 @@ program
         topK,
         line: opts.line,
         noRerank: opts.rerank === false,
+        noHybrid: opts.hybrid === false,
+        rerankPool,
+        noDedup: opts.dedup === false,
+        withText: opts.withText || null,
+        fetchMissing: opts.fetchMissing || null,
+        mmr: opts.mmr === true,
+        from: opts.from || null,
         readsLocalIndex: true,
         mayUseCloudEmbedding: true,
       }
@@ -5032,7 +5064,15 @@ program
       // 接这条通道之前，`OPERATIONS.md` 一直在教用户敲 `search --no-rerank`，而 CLI 会答 `unknown option`
       // —— 同一份文档里 `--line` 是通的，所以它看起来像"能用的旗标"（2026-10-11 由文档命令面那道检查发现）。
       const args: string[] = [script, 'search', '--top-k', String(topK), '--line', opts.line,
-        ...(opts.rerank === false ? ['--no-rerank'] : [])]
+        ...(opts.rerank === false ? ['--no-rerank'] : []),
+        ...(opts.hybrid === false ? ['--no-hybrid'] : []),
+        ...(opts.from ? ['--from', opts.from] : []),
+        ...(rerankPool ? ['--rerank-pool', String(rerankPool)] : []),
+        ...(opts.dedup === false ? ['--no-dedup'] : []),
+        ...(opts.withText ? ['--with-text', opts.withText === true ? '4000' : String(opts.withText)] : []),
+        ...(opts.fetchMissing ? ['--fetch-missing',
+          opts.fetchMissing === true ? '3' : String(opts.fetchMissing)] : []),
+        ...(opts.mmr ? ['--mmr'] : [])]
       try {
         const { stdout } = await execFileAsync(getPythonCommand(), args, {
           timeout: 60_000, maxBuffer: 10 * 1024 * 1024,
@@ -5113,7 +5153,10 @@ program
       if (opts.full) args.push('--full')
       try {
         const { stdout } = await execFileAsync(getPythonCommand(), args, {
-          timeout: 300_000, maxBuffer: 10 * 1024 * 1024,
+          // 30 分钟：全量重建实测要跑二十分钟以上（16k 条 / 1500+ 批），而这里原来给的是 5 分钟 --
+          // 那个值让 `--full` 在真实索引上**从来没跑完过**（超时被杀；且脚本改成了边嵌边落盘之前，
+          // 中途被杀等于连已经嵌好的那部分也一起丢）。
+          timeout: 1_800_000, maxBuffer: 10 * 1024 * 1024,
           env: pythonProcessEnv(opts.apiKey, 'DASHSCOPE_API_KEY'),
         })
         if (opts.json) console.log(stdout.trim())

package/dist/bin/weflow-cli.js

@@ -1902,8 +1902,12 @@ program
     }
     // 图片要靠 Python 那条路匹配（缓存缩略图、资源 md5、原图那档）。`--media` 就是"先抽一次"，
     // 抽完 json/txt/excel 各自消费同一份清单 —— 匹配只跑一遍，也不会在 TS 里重写一遍。
+    // 抽取那一步的媒体摘要留在这里：`--media` 的调用者（机器调用）应当能从结果里直接读到
+    // "各状态多少条、报告叫什么"，而不必去翻 Python 的 stdout。
+    let extracted;
     if (opts.media && (format === 'json' || format === 'txt' || format === 'excel')) {
         const media = await exportService.extractMedia(talker, opts.output, !!opts.json, { fullImages: opts.fullImages === true }, limit, opts.date || '');
+        extracted = media;
         if (!media.success) {
             if (opts.json)
                 console.log(JSON.stringify({ success: false, code: 'MEDIA_EXPORT_FAILED', error: media.error || '媒体抽取失败' }));
@@ -1940,7 +1944,16 @@ program
     }
     if (result?.success) {
         if (opts.json)
-            console.log(JSON.stringify({ success: true, format, contract: format === 'json' ? opts.contract : null, path: result.path, count: result.count ?? null }, null, 2));
+            console.log(JSON.stringify({
+                success: true,
+                format,
+                contract: format === 'json' ? opts.contract : null,
+                path: result.path,
+                count: result.count ?? null,
+                // 媒体摘要：html/md 那条由导出自己产出（Python 的 media_json_summary），
+                // json/txt/excel 的 `--media` 则由上面那次抽取产出 —— 两条都是同一份词汇。
+                media: result.media ?? extracted?.media ?? null,
+            }, null, 2));
         else
             console.log(chalk.green(`✓ 导出成功: ${result.path}`));
     }
@@ -5200,6 +5213,13 @@ program
     .option('--top-k <n>', '返回数量', '10')
     .option('--line <line>', 'wiki=文章线(默认) / chat=聊天记录 / all=两条都搜', 'wiki')
     .option('--no-rerank', '不调用决策模型重排，只按向量/关键词相似度返回（回退到引入重排之前）')
+    .option('--no-hybrid', '不融合本地 BM25 名次（默认融合；门控后实测只增不减）')
+    .option('--rerank-pool <n>', '重排看多少条候选（默认 100；越大越准也越贵）')
+    .option('--no-dedup', '不折叠同文本的多号转载（默认折叠，结果里带 duplicateCount）')
+    .option('--with-text [n]', '给每条命中挂上本地正文（fullText / sourceFile / url）；n 为每条最多多少字')
+    .option('--fetch-missing [n]', '正文太薄时按 url 抓一次（最多 n 条，默认 3；复用日报缓存）')
+    .option('--mmr', '对结果做 MMR 去冗余（压掉近似的同领域邻居，可能牺牲单点召回）')
+    .option('--from <account>', '只查这个公众号的文章（按文件名匹配，不走语义）')
     .option('--api-key <key>', 'DashScope embedding API key')
     .option('--dry-run', '仅预览，不读取索引或调用向量服务')
     .option('--yes', '确认执行可能调用云端向量服务的搜索')
@@ -5211,6 +5231,10 @@ program
     const pkgRoot = resolvePackageRoot();
     const script = join(pkgRoot, 'scripts', 'semantic_search.py');
     const topK = parseCliInteger(opts.topK, 'top-k', 1, 100, opts.json);
+    // 没传就**不转发**（默认值只在 Python 侧定义一处）：commander 对未提供的选项给 undefined，
+    // 直接 parseCliInteger 会被自己的校验判成非法参数 —— 实测就是这么红的。
+    const rerankPool = opts.rerankPool
+        ? parseCliInteger(opts.rerankPool, 'rerank-pool', 5, 500, opts.json) : null;
     requireCliChoice(opts.line, ['wiki', 'chat', 'all'], 'line', opts.json);
     const preview = {
         success: true,
@@ -5219,6 +5243,13 @@ program
         topK,
         line: opts.line,
         noRerank: opts.rerank === false,
+        noHybrid: opts.hybrid === false,
+        rerankPool,
+        noDedup: opts.dedup === false,
+        withText: opts.withText || null,
+        fetchMissing: opts.fetchMissing || null,
+        mmr: opts.mmr === true,
+        from: opts.from || null,
         readsLocalIndex: true,
         mayUseCloudEmbedding: true,
     };
@@ -5249,7 +5280,15 @@ program
     // 接这条通道之前，`OPERATIONS.md` 一直在教用户敲 `search --no-rerank`，而 CLI 会答 `unknown option`
     // —— 同一份文档里 `--line` 是通的，所以它看起来像"能用的旗标"（2026-10-11 由文档命令面那道检查发现）。
     const args = [script, 'search', '--top-k', String(topK), '--line', opts.line,
-        ...(opts.rerank === false ? ['--no-rerank'] : [])];
+        ...(opts.rerank === false ? ['--no-rerank'] : []),
+        ...(opts.hybrid === false ? ['--no-hybrid'] : []),
+        ...(opts.from ? ['--from', opts.from] : []),
+        ...(rerankPool ? ['--rerank-pool', String(rerankPool)] : []),
+        ...(opts.dedup === false ? ['--no-dedup'] : []),
+        ...(opts.withText ? ['--with-text', opts.withText === true ? '4000' : String(opts.withText)] : []),
+        ...(opts.fetchMissing ? ['--fetch-missing',
+            opts.fetchMissing === true ? '3' : String(opts.fetchMissing)] : []),
+        ...(opts.mmr ? ['--mmr'] : [])];
     try {
         const { stdout } = await execFileAsync(getPythonCommand(), args, {
             timeout: 60_000, maxBuffer: 10 * 1024 * 1024,
@@ -5333,7 +5372,10 @@ program
         args.push('--full');
     try {
         const { stdout } = await execFileAsync(getPythonCommand(), args, {
-            timeout: 300_000, maxBuffer: 10 * 1024 * 1024,
+            // 30 分钟：全量重建实测要跑二十分钟以上（16k 条 / 1500+ 批），而这里原来给的是 5 分钟 --
+            // 那个值让 `--full` 在真实索引上**从来没跑完过**（超时被杀；且脚本改成了边嵌边落盘之前，
+            // 中途被杀等于连已经嵌好的那部分也一起丢）。
+            timeout: 1_800_000, maxBuffer: 10 * 1024 * 1024,
             env: pythonProcessEnv(opts.apiKey, 'DASHSCOPE_API_KEY'),
         });
         if (opts.json)