weflow-cli 1.11.1 → 1.13.0

package/CHANGELOG.md

@@ -17,6 +17,384 @@ purpose** - a hand-written "latest is X" is exactly the claim that goes stale he
 
 All notable user-facing changes are recorded here. This project follows [Semantic Versioning](https://semver.org/).
 
+## 1.13.0
+
+- **`sessions` shows readable previews instead of raw message bytes.** The preview column used to
+  print whatever sat in `source` / `message_content`, so card and mini-program messages came out as
+  `(/ }25984…` fragments and text messages with a corrupt column showed binary. It now labels by
+  message type (`[图片]` / `[语音]` / `[视频]` / `[表情]` / `[链接/文件]` / `[系统消息]`, honouring
+  4.x's packed `local_type` where the subtype lives in the high bits), picks whichever column decodes
+  as real text, and strips a leading run of ASCII symbols that group messages carry. Chinese
+  punctuation and emoji openings are left alone.
+- **`check` says when the configured database path is a junction.** WeChat moved to another drive
+  leaves `C:\Users\<user>\xwechat_files` pointing at the real location; printing only the link made
+  it look like the config pointed somewhere wrong. It now adds `↳ 软链指向 <real path>`.
+- **README: `init` needs `--path` when the data directory is not in a default location**, because
+  "close WeChat first" removes the only remaining way auto-discovery could find it.
+- **Silent failures now say what broke, where.** Three places that used to lose data without a word:
+  a message shard that cannot be opened during `export` (it skipped the shard and still reported
+  success - you got a shorter transcript and no reason), a corrupt `.image_map.json` in the daily
+  report (images from earlier passes disappeared from the HTML), and an encrypted config value that
+  will not decrypt (`get_*_key` returned `''`, which reads as "not configured" even though it *is*
+  configured and merely broken). Each now prints one line naming the file and the exception type;
+  the config warnings are de-duplicated because those helpers run on every request. A guard test
+  (`test/silent_failure_guard_test.py`) pins the count of bare `except:` + `pass` bodies at 53 so
+  the number cannot creep back up.
+- **`docs/POSITIONING.md`**: why this project exists next to WeChat itself. Written after 4.1.15
+  added "ask AI about a file / an article" - i.e. WeChat now does the simplest half of what a
+  "WeChat + AI" tool does. It lists the five moats (data that can leave the client, cross-time
+  queries, cross-corpus joins, local inference, traceable answers), the features that will be eaten
+  (single-document summaries, in-chat keyword search), and four questions that decide whether a
+  feature belongs here.
+- **Two scheduled passes instead of one: a noon preview and a complete end-of-day run.**
+  `scripts/daily_task.ps1 noon` produces *today so far* (13:00); the bare invocation still
+  produces *yesterday* (now 00:05 + at logon). Both write the same date directory, and the
+  second pass is incremental - `.run_state.json` makes `biz_daily` skip what the noon pass
+  already handled, while README/`index.html` are rebuilt from every markdown file on disk.
+  `scripts/install_tasks.ps1` registers both (it needs an elevated terminal: Task Scheduler
+  writes are refused in a sandboxed shell while reads still work).
+- **`.articles.json` is merged, not overwritten.** With two passes a day the second one only
+  carries the articles that arrived after noon, so overwriting dropped the morning batch's
+  titles/summaries/topics from the structured file while the page still listed the whole day -
+  downstream reports and statistics silently saw half a day. Entries are now merged by URL
+  (title+time when a push has no URL), `decisionModel` survives a `--no-ai` rerun, and a
+  corrupt old file starts from empty instead of failing the day.
+- **The scheduled guard no longer trusts `index.html` alone.** The noon pass writes one too,
+  so "index.html exists" made the 00:05 run skip the afternoon and leave a complete-looking
+  half day behind. It now reads `.articles.json`'s `generated_at`: finished means generated
+  after the target day (the usual 00:0x pass) or after 23:00 that night.
+- **`weflow-cli check --report`: a paste-ready, redacted diagnosis.** It prints the environment
+  (Node / `python --version` / `sqlcipher3` & friends), the WeChat client actually loaded and whether
+  its data directory matches the configured one, whether a key/passphrase is configured, and then a
+  **verdict** — "Python unusable", "missing sqlcipher3", "not initialised", "client not running",
+  "reading a different database" — each with the next command to run. Usernames, `wxid`s and
+  key-shaped strings become `<用户>` / `wxid_<已隐藏>` / `<密钥>`; message bodies and key values never
+  enter it. `sessions` printing "未找到会话" and a failed `init` now point at it, because that is the
+  point where "no sessions" and "cannot read them" used to look identical (the same root cause was
+  reported as four different problems between 2026-08 and 2026-10).
+- **`wechatClient.running` and its explanation no longer contradict each other.** It used to be
+  `any WeChat process`, while the note was keyed on `Weixin.exe`, so "only WeChatAppEx is running"
+  produced `运行中: 是` next to `客户端没在运行`. Both now use the launcher process; the note names
+  the helper processes when that is what it saw.
+- **A same-day rerun of the scheduled daily no longer erases the log of the run that failed.**
+  `scripts/daily_task.ps1` redirects through `Start-Process`, which **overwrites** its targets, so
+  retrying a failed day replaced the only evidence of what broke. Existing `daily-<date>.log` /
+  `.err.log` are now rotated to `daily-<date>.run<N>[.err].log` (first free index) before the new
+  run starts. Verified by planting a marker in both files and running the wrapper: the markers
+  landed in `.run1` and the fresh log did not contain them.
+- **`search --phrase` now refuses the vector-leg flags instead of accepting and ignoring them.**
+  `--no-rerank`, `--rerank-pool`, `--no-hybrid`, `--no-dedup`, `--mmr`, `--with-text` and
+  `--fetch-missing` describe the vector/rerank path; on the verbatim (FTS5) leg they were forwarded
+  to the script and **silently dropped**, so a reader who passed one got the default behaviour while
+  believing they had changed it. The CLI now exits with `INVALID_ARGUMENT` naming the offending
+  flags (JSON-aware), and calling `semantic_search.py` directly prints
+  `[WARN] --phrase only locates verbatim text; ignored: ...`. `--top-k`, `--line` and `--from` still
+  apply. A test spawns the real CLI to prove the rejection happens before any script call.
+- **Client-version detection keeps working on newer WeChat: the command line is no longer the only
+  observation.** Since **4.1.15.53** the `Weixin.exe` main process is launched with **no switches at
+  all** (`"D:\WeiXin\Weixin.exe"`), so the `--user-lib-dir` judgement that `check` relied on returned
+  "version unknown" on every machine that updated - the one field worth looking at first when reads
+  fail. It now falls back to the directory of the **`Weixin.dll` the process actually has mapped**
+  (same quantity, second observation), and the `note` says which source was used. Measured after the
+  update: `check --json` reports `4.1.15.53` instead of `null`. The no-guessing rules are unchanged -
+  multiple directories still yield no single version, an unreadable module table is still `null`
+  rather than an inference from what is on disk, and the hardcoded version literal is still not read
+  (D-125 amends D-107).
+- **`init` no longer says "初始化完成" when it has no usable key, and a Windows `9009` now says what to
+  do.** A 2026-10-18 report ended at `NT 数据库扫描: NT scan failed (exit 9009)`, "✓ 已沿用配置中的密钥",
+  "初始化完成! (4.x 模式)" and "密钥: 已安全保存" - with `sessions` empty on every run. Two defects
+  behind it: `getPythonCommand()` falls back to the **bare name `python`** when every candidate fails its
+  probe (runnable *and* `import sqlcipher3`), and on Windows that bare name is often the Microsoft Store
+  placeholder, which exits `9009`; and the closing summary was unconditional, so a scan that never ran
+  still looked like a finished setup. The exit code is now translated into the actual requirement
+  (Python 3.10+ with `sqlcipher3`), and the summary is gated on whether **this run** obtained a key
+  (memory-scan match *or* passphrase derivation) - otherwise it prints a failed-init heading with the
+  three steps that fix it, instead of "密钥已保存" next to a key that was never checked against this
+  database.
+- **`judgeProvider=chat` now honours `aiEngine`; a local-engine user is no longer shipped abroad
+  silently.** The chat branch (D-120) read `deepseekApiKey` unconditionally, so with `aiEngine=local`
+  turning on the judge sent article text to DeepSeek while nothing in the UI changed - not a cost
+  question, but a user who chose to stay local being sent out anyway. `local`/`ollama` now goes through
+  `detect_local_engine()` (`provider='chat-local'`, same contract checks), `claude` is **refused
+  explicitly** rather than swapped for another vendor, and an undetected local service returns `None`
+  with a WARN instead of falling back to the cloud (D-124). The local path is covered by unit tests
+  with a stub engine; it has **not** been run against a live Ollama/LM Studio.
+- **A search that did not use the vector leg now says so, and `--mmr` can no longer crash on that
+  path.** `search()` used to wrap its whole vector path in a bare `except: pass`, so *any* failure
+  (an embedding upstream error, numpy missing, a broken `meta.json`) silently became a keyword-only
+  search while the envelope still reported `hybrid: true` - two very different quality levels that
+  looked identical in the output, and the reason every "faster/slower, better/worse" comparison could
+  land on a different code path without anyone noticing. The response now carries
+  `vector: {used, fallback}`, where `fallback` names the actual exception (or "embedding empty/all
+  zero"). Two further defects came out of the same read: `vectors` was assigned **only inside** the
+  `try`, so a failure before `np.load` (e.g. `require_numpy()` raising) made the fallback's
+  `mmr_select(..., vectors, ...)` throw `UnboundLocalError` under `--mmr` - now initialized to `None`
+  and guarded; and `main()` parsed the 13.5 MB `meta.json` a **second** time for `line_counts()`
+  (~125 ms/query, 1.2% of a search) - the counts are now computed from the meta `search()` already
+  has. The bare `except` also swallowed `KeyboardInterrupt`, which made cancelling a long run
+  unreliable on that path. The rerank state gained a **`transport`** field alongside `provider`,
+  because `provider: typesafe` names the *contract* that ran, not who served it - with
+  `judgeProvider=chat` the Jev contract is satisfied by DeepSeek, so the old field read as "the native
+  API worked" (it did not; the other half of the same batch fell back to the LLM path). Eight tests
+  pin the fallback behaviour, and the misleading-first-test episode is recorded in D-122: a
+  regression test that failed *after* `vectors` was assigned stayed green when the initialization was
+  removed, and a mutation check is what exposed it.
+
+
+- **A third retrieval leg answers a question the other two cannot: "which account wrote this exact
+  phrase, and when?"** `search --phrase` locates a string **verbatim** using SQLite's bundled
+  **FTS5 with the `trigram` tokenizer** (no new dependency), returning hits with the matched run
+  highlighted plus `groups` counting them by account × date. It is a *different question*, not a
+  better ranking: BM25 scores terms and vectors score meaning, so neither can answer a code name
+  (`std::vector`), a proper noun, a policy phrase or a quotation check - and folding "the exact
+  string is absent" into a similarity ranking is a failure the reader cannot see. Four measured facts
+  set the shape: the default `unicode61` tokenizer is **useless for Chinese** (whole sentences become
+  one token - `沙箱逃逸` matched **0**); `trigram` matches **contiguous substrings, not a term AND**
+  (`光伏补贴政策` only hits the verbatim occurrence, while `补贴政策` also hits "的补贴政策" - which is
+  exactly what this leg is for, and exactly why it cannot replace the keyword leg, since
+  `光伏补贴政策` returns **0 of 10,759** articles); it has a **3-character floor**, so a short query
+  returns an explicit `tooShort` instead of "0 hits" (saying "cannot search" to mean "absent" is this
+  leg's worst failure); and query punctuation is parsed as **syntax** (`PM2.5` was a syntax error,
+  `GPT-4` a "no such column"), so the whole string is quoted as a literal (inner quotes doubled) -
+  after which `PM2.5` / `C++` / `GPT-4` all work. On this corpus: `沙箱逃逸` 2 hits across two
+  accounts, `PM2.5` **58** (including the chat line), `腾讯混元` 7, `分布式光伏` 1; `std::vector` 0.
+  **One table for the corpus, not one per article** - a table per article would mean ~96,000 tables
+  with FTS5's shadow tables, `bm25()`'s corpus-level statistics degenerating to one row each, and the
+  inverted index's only advantage thrown away. The index is **derived** from the semantic index's
+  `meta.json` (a second record enumeration would drift, silently). Build with
+  `search-index --phrase`: **local, free, no key, no confirmation gate** (that gate exists for
+  egress) - the CLI's preview says `mayUseCloudEmbedding: false` so this is not mistaken for a
+  vector call.
+
+
+- **A Jev-like decision model that works from this region is one config line away.** `judgeProvider` now
+  selects among three transports for the *whole* decision layer (the daily's topic/relevance/inclusion
+  judgement, search reranking, reply debt, drafted replies and route-card term selection all share one
+  client): `native` (unchanged default), **`chat`** - the conversational model already configured for
+  summaries (`aiBaseUrl` relay / `aiModel` / `deepseekApiKey`), so it needs **no new vendor and no new
+  key** - and `openrouter` (a new encrypted `openrouterApiKey` plus `openrouterJudgeModel`, default
+  `typesafe/jev-router`). The native endpoint is region-blocked here (**HTTP 451**), and the failure shape
+  is silence rather than an error - 0 of 12,542 articles over the last 40 complete days carry a
+  `relevanceScore` - so the point of the switch is to route around a block, not to add a vendor.
+  Measured with the same prompt and the same validators against one real article: **`deepseek-chat`
+  passes in 1.0 s** (1,649 prompt + 69 completion tokens, cost on the order of ¥1/day for 500 articles),
+  `deepseek-reasoner` in 7.2 s, `qwen-plus` in 2.3 s.
+  **A passing contract is not the same judgement**: `deepseek-chat` scored that article relevance 1 with
+  an inclusion probability of 0.15 while `qwen-plus` scored it 2 and 0.85 - and the report's thresholds
+  were set against Jev, so switching judges means switching rulers. That is why the raw scores and
+  `usage.model` are stored; it also immediately paid off, since the served model for a `deepseek-chat`
+  request came back as **`deepseek-flash`**.
+
+
+- **The decision model has a second transport, the region block is now a measured fact rather than a
+  guess, and the switch that uses it ships off.** `api.typesafe.ai` answers **HTTP 451 "not available in
+  your region"** from this machine, and the failure shape is silence rather than an error: of 12,542
+  articles across the last 40 complete days, **0** carry a `relevanceScore` - and the same client serves the
+  reranker, reply debt, drafted replies and route-card term selection, so the whole layer degrades together.
+  New config: `judgeProvider` (`native` default / `openrouter`), `openrouterApiKey` (encrypted at rest) and
+  `openrouterJudgeModel` (default `typesafe/jev-router`). One switch replaces the layer, which is exactly why
+  **the default stays `native`**.
+  **The region filter is per model, not per site** (measured): `typesafe/jev-router` and `:free` models pass
+  from here, while `openai/gpt-4o-mini` returns **403 "This model is not available in your region."** - so a
+  model swap has to be re-tested on the machine that will use it.
+  The contract moves to our side (the native endpoint returns structured answers; a chat model has to be
+  *asked* for them), so the prompt and four validators live in `jev_client.py` and drift raises `JevError`
+  instead of producing a plausible-looking default - including the two shapes that used to fail *quietly*:
+  `noul` arriving as a boolean (once read as "no" for every question) and `score` arriving as a string
+  (which raised `ValueError`, something the callers' `except JevError` could not catch). Cost is measured:
+  **$0.014** for one article with four topic choices on a call the router sent to `x-ai/grok-4.7` (1,535
+  reasoning tokens), and since the router chooses a different upstream per call, the worst case is what has
+  to be budgeted (**~$7 for a 500-article day**); `max_tokens` does **not** cap it. Four failures now say
+  four different things because they need four different fixes: **402** = credits, **403 + "region"** =
+  change the model (changing the key does nothing), **401** = auth, **empty content** = reasoning ate the
+  budget. **Off by default, and not usable on the key tested here**: `/credits` reports `total_credits: 0`
+  and calls are refused with 402.
+
+- **The daily report can now be scheduled to run itself, and it knows what "yesterday" means.**
+  `daily` (and the `pipeline.py` it drives) gained `--yesterday`. A scheduled run has to be *the
+  previous day*, not "today": WeChat accounts keep publishing until midnight, so a same-evening run
+  silently ships a truncated day that nobody ever goes back to fix - while "yesterday" is
+  **time-insensitive** (any wake-up after midnight reads the same complete data, so a machine that
+  was asleep only needs to be allowed to start late, not have its date re-derived). Passing both
+  `--date` and `--yesterday` is an error rather than a silent pick, because then "which day" has no
+  single answer. `OPERATIONS.md` now carries a corrected recipe for the scheduled task - it uses
+  `Register-ScheduledTask` with `-StartWhenAvailable`, because `schtasks /create` cannot express
+  "start late if you missed it", and without that a sleeping machine means a **silently** missing
+  day.
+
+- **`daily --prefetch-bodies [n]` (default off) fetches the day's article bodies concurrently
+  *before* the fetch phase.** That phase is serial and sleeps 8-12 s for every article that misses
+  the body cache - deliberate throttling, and on the order of an hour for a 450-article day - while
+  a **cache hit neither connects nor sleeps**. Warming the cache first turns the fetch phase into
+  cache hits. The **number of requests to upstream is unchanged**; only the window they are spread
+  over changes, which is exactly why this is opt-in rather than the new default. Measured on the
+  first real run (2026-10-16, a 523-article day): **516 bodies warmed in 10 minutes** and the whole
+  `biz_daily` step took **15m49s**, where the same articles serially with throttling take over an
+  hour. The scheduled task uses it, so a report that used to hold the machine for an hour finishes
+  in a fraction of that. (Related: the index does **not** read this cache - measured no gain, see
+  `docs/DECISIONS.md` D-118 - so the cache serves the display path (`search --with-text`) and this
+  speed-up only.)
+
+## 1.12.0
+
+- **One word per question, across every artifact that describes exported media.** The auxiliary `*_media.json`
+  called a document entry `exported` when it meant "the local copy was **found**", while `manifest.json` used
+  `exported` for "it is **in the assets directory**" - two different questions wearing one word, and a copy
+  failure would have made "已导出" false. Now: the report answers *locatable in the source data* (`found` /
+  `missing`), the manifest answers *materialized in the export directory* (`written` = what is there now,
+  `copied` = what this run added, `unavailable` = not written, whether because the file is absent, oversized,
+  or the copy itself failed). `manifest.documents` gained the clearer `written` (it was `exported`) and the
+  CLI's JSON for all three branches (html / md / `--media`) is now produced by **one** function
+  (`media_json_summary`) instead of three hand-written blocks - so the formats cannot describe the same fact
+  differently, and the summary carries the report's file name everywhere (previously only the HTML branch did).
+  Pinned by tests: a file whose copy fails reports `found` in the report and `unavailable` in the manifest
+  (never "exported"), and the summary is asserted to carry counts, the report name, and **no absolute path**.
+
+- **The article half of the semantic index was embedding YAML metadata, not article text.** `collect_articles`
+  found the body by looking for a `## 正文` marker; measured on 2026-10-15, **none** of the 10,759 daily `.md` files
+  in the default window contains it (the real shape is `frontmatter → # title → > source → --- → ## AI 摘要 →
+  --- → repeated title → body`), so every record silently fell back to `content[:1000]` - the frontmatter. Sampled
+  300 of them: 100% contain `title:`, median CJK ratio **0.16**; the only content signal left was the title, which
+  is what "the article line ranks by title" actually meant. The extractor now reads the real structure (median CJK
+  ratio 0.16 → **0.80**, 299/300 files changed) and a test fails if it is reverted to the old fallback. Note that
+  an existing index does **not** pick this up by itself: an incremental build embeds new ids only, so it takes
+  `search-index --full`.
+- **Building the index can be interrupted without losing work already paid for, and it says where it is.** A full
+  rebuild of ~16k records takes over twenty minutes while the CLI gave its subprocess **five** - so `--full` had
+  never completed through the CLI at all, and since nothing was written until the very end, a timeout threw away
+  the whole run. It now flushes every 100 batches (vectors first, then `meta.json`, each through a temporary file
+  and `replace`), so a killed run leaves a **valid** partial index and a plain re-run continues from there, because
+  the flushed ids are already in `meta.json`. Batch failures (measured: `IncompleteRead` from the embedding
+  endpoint) are now **excluded** instead of being written as all-zero vectors - such ids would otherwise sit in
+  `meta` forever, since an incremental build only ever embeds new ones - and the count is reported as
+  `embedFailed`. Progress lines (`[索引] 已嵌入 N/M`) go to stderr.
+- **A mismatched index no longer degrades into "nothing found".** When `vectors.npy` holds more rows than
+  `meta.json` (a build killed between the two writes, or files edited by hand), the top-k indices ran past the end
+  of `meta` and the bare `except` in `search()` turned that `IndexError` into a silent fallback to keyword search -
+  a real query that presents as "no hits". The two are now truncated to the shorter one, which costs a few results
+  instead of the whole vector path.
+
+- **`search --hybrid`: an opt-in second ranker, off by default because it was measured not to earn it.** The new
+  flag fuses the vector ranking with a fully local BM25 ranking (Chinese is tokenised into **character bigrams**,
+  so no segmenter and no word list; one pass over the indexed texts, no extra index file, ~31 ms for 16k records)
+  using Reciprocal Rank Fusion. It fixes one concrete failure - the rarest word in a question being overridden by
+  the most frequent one: `卫星遥感反演气溶胶光学厚度` ranks the satellite-based PM2.5 paper 3rd on vectors alone
+  and 2nd with the flag. But measured on the same day, it is **not better** on four of five realistic questions
+  (it pushed the correct first hit out of the top five for `有没有讲遥感或卫星数据用在空气质量监测上的文章`, and
+  `后向轨迹 HYSPLIT 怎么用` was dragged off by function words), and on 16 verbatim-fragment queries BM25 alone
+  scored 16/16 against the fusion's 12/16 - RRF gives both sides equal voting power, and Chinese function words turn
+  BM25 into noise. So the default stays as measured-better and the flag is for questions that really do contain a
+  rare term. Note it also makes the printed `score` (cosine) non-monotonic, because the order now comes from the
+  fused ranks. The decision and both A/B tables are in D-110; making it a default would first need idf-gated
+  voting and a re-measurement on the same harness.
+
+- **`search --from <account>`: filter by source account, because that is an attribute query and vectors cannot
+  answer it.** Measured on the 2026-09-12..10-13 window (10,759 articles, 413 accounts): searching the account name
+  itself returns only 0-3 of the top 10 from that account for 红星新闻 (908 articles), 政事儿 (632) and 新京报 (529),
+  while the account name is present in **every** one of its records' `id` (the daily names files
+  `<account>-<title>.md`). The flag filters on that id as a substring (so `--from 红星` works), applies to the article
+  line only - a chat message has no source account - respects the same mask as the keyword side when `--hybrid` is on,
+  and says "the index has no records for this account" rather than returning an empty list silently. It gives the
+  right pool, not a better order within it: ranking inside an account is still semantic nearest-neighbour, so a
+  general-news account can return its own unrelated stories for a topical query.
+
+- **A recall/precision evaluation harness now exists, and it immediately paid for itself: gated keyword fusion is
+  the new default.** `scripts/recall_eval.py` measures `search` with the standard metric set (Recall@1/5/10, MRR@10,
+  nDCG@10, Hit@1) plus a **coverage** count, and builds its ground truth at run time (a uniqueness-checked 12-character
+  fragment from the middle of a record's own text, and its title / first 30 characters) so no corpus text is committed;
+  hand-written questions go in an out-of-repo TSV (`question<TAB>anchor<TAB>line`). It requires `--yes` because each
+  question costs an embedding and leaves the machine, and it writes nothing. With that harness, the keyword side of
+  `--hybrid` gained two gates - only documents matching a **rare** term (df/N <= 5%) vote, their BM25 score must be
+  within 0.5x of the best, and a query with no rare term makes the keyword side **abstain** - and on the same three
+  batteries (40 fragment + 40 title + 12 hand-written questions) that turns the ungated version's zero gain into:
+  fragment recall@10 0.91 -> **1.00**, recall@5 0.89 -> 0.97, hit@1 0.60 -> **0.80**, title hit@1 0.83 -> 0.86, and
+  no change on the question battery. Nothing got worse, so **fusion is on by default** (`--no-hybrid` restores plain
+  vector order, and note the printed `score`, a cosine, is no longer monotonic). The ungated variant scoring exactly
+  like pure vectors is what explains D-110's earlier "harmful" measurement: the problem was function words voting,
+  not fusion. Threshold sensitivity is flat (0.02/0.7 and 0.10/0.3 bracket the same result), so the value is not
+  load-bearing; it must not be re-tuned on this battery.
+
+- **The rerank half of the pipeline works again - it now falls back to the configured chat model, and its state is
+  visible in the output.** Rerank was never *running* on this machine: the Jev/Typesafe contract returns
+  `HTTP 451 not available in your region`, and the failure was only a stderr WARN, which the CLI does not forward -
+  so a search reported nothing and looked like "fine". `rerank_dispatch` now tries the contract first and, if that
+  client fails or is absent, reranks with the configured OpenAI-compatible chat model; both failing keeps the original
+  order, and `RERANK_STATE` (`provider` / `degraded` / `error` / `scored`) travels into the JSON so "not reranked" can
+  no longer be mistaken for "reranked". Passages are cleansed before they are sent (bold markers, 公众号 boilerplate,
+  bylines, image/link syntax), because the excerpt is only 300 characters. Measured on the same 40 questions: hit@1
+  0.55 -> **0.65**, recall@5 0.65 -> **0.80**, MRR@10 0.59 -> **0.72**, nDCG@10 0.63 -> **0.74**, with the weakest
+  cell (verbatim fragment on the article line) going recall@5 0.30 -> **0.70** - in the same range the literature
+  reports for retrieve-and-rerank (MS MARCO nDCG@10 0.396 dense -> 0.452 with a reranker; a public example puts
+  hybrid-RRF 32.95 -> 36.16 with rerank). Two consequences worth knowing: **every search now spends one model call**
+  (~2-3k tokens, ~¥0.004, plus a few seconds) and the assistant, MCP and `chat` share this path, so `--no-rerank`
+  is the escape; and one regression is recorded rather than smoothed over - title/article recall@5 slipped 1.00 to
+  0.90 (one case at n=10), whose fix (bigger candidate pool, or protecting the retrieval-side top hit) is not done.
+
+- **The rerank candidate pool is now 100 by default, because the pool was the biggest lever all along.** Rerank can
+  only reorder what retrieval hands it: with a pool of 20, a correct document ranked 21st by the hybrid ranking could
+  never appear. Measured on the same 40 questions with only the pool changed - 20 / 50 / 100 / 200 gives recall@10
+  **0.82 / 0.88 / 0.95 / 0.95**, MRR@10 0.72 / 0.75 / **0.80** / 0.82, nDCG@10 0.75 / 0.78 / **0.83** / 0.85, and in
+  the weakest cell (verbatim fragment on the article line) recall@10 0.70 / 0.80 / **1.00** / 1.00 - so 100 is the new
+  default and 200 buys nothing measurable. Against published production thresholds (Recall@10 >= 0.90, MRR >= 0.75)
+  this now clears both. The pool is adjustable per query (`--rerank-pool`, CLI and script; when the flag is absent
+  nothing is forwarded, so the default lives in exactly one place - the first version parsed the commander option
+  unconditionally and **made a missing flag a fatal invalid argument**), and the cost is stated plainly: the rerank
+  prompt grows from ~2.5k to ~20k tokens (~¥0.01 per search, a few seconds). Two corrections to the previous entry:
+  the title/article recall@5 "regression" was **noise** (rerank is not perfectly deterministic even at temperature 0,
+  and each cell holds 10 questions, so one case is 0.10) - it re-measures at 1.00; and per-cell numbers at n=10 must
+  not be used to judge a single change.
+
+- **Results are collapsed by identical text now, because multi-account reposts were eating slots.** Measured on the
+  current defaults (rerank with a pool of 100): **7%** of 100 top-5 slots held a duplicate of another item in the same
+  result list, and **6 of 20** questions showed one - e.g. the 环境监测新规 query had a five-department notice carried
+  by three accounts plus a meeting report carried by three more, so four slots went to two pieces of content. The
+  first (best-ranked) copy is kept and the others are counted into a new `duplicateCount` field, so the fact that
+  several accounts carried it stays visible instead of silently shrinking the list; `--no-dedup` returns the raw
+  list. This deliberately happens at presentation time and **not at index time**: the index keeps every copy because
+  `--from <account>` resolves an account through the record's `id`, and dropping copies there would make per-account
+  lookups lose records. It changes no recall metric (the ground truth is content) - only how much distinct material
+  one result list actually carries.
+
+- **`search --with-text` returns the article itself, not the excerpt it was matched on.** What a search used to hand
+  back was the title plus the ~500-character slice that the embedding was computed from - which is not the content
+  anyone wants. The content is already local: the record's `id` carries the daily markdown's relative path, and the
+  fetch cache (`output/.cache/fetch/<md5(url)>.md`) holds the **full** body. The flag adds `fullText` (readable: bold
+  markers, image syntax and 公众号 boilerplate lines removed, paragraphs kept - deliberately different from the
+  300-character, whitespace-collapsed excerpt the reranker receives), `sourceFile` (always given, so the raw file can
+  be compared against what was cleaned) and `url`; the longer of the daily markdown and the cache wins. Chat records
+  need none of this - the indexed text *is* the whole message. It is **off by default** for a context reason rather
+  than a cost one: the assistant, MCP and RAG share this `search()`, and ten results at 4,000 characters would push
+  ~20k characters into those paths. Unreadable records return `fullText: null` with `sourceFile` still set (never a
+  silent empty string). Measured on the demo query, the first hit goes from a 500-character excerpt to 1,723
+  characters of real content (citation, three highlights, abstract, and the results paragraph: R² 0.87/0.86/0.83,
+  AOD feature importance 40.63%); before image syntax was stripped it was 3,860 characters, ~2,100 of which was one
+  line of `![](https://mmbiz.qpic.cn/…)`.
+
+- **`--fetch-missing` fills in a thin body by fetching the article, through the daily pipeline's own cache.**
+  Some daily markdown files are stubs (a "好文回顾" record held 201 characters), and `--with-text` could only hand
+  back that stub. With this flag, any record whose local body is under 300 characters **and** that carries a `url`
+  is fetched once; the result replaces `fullText`, is marked `fetchedFromUrl`, and is written to the **same** cache
+  the daily and backfill use (`output/.cache/fetch/<md5(url)>.md`, success-only, cache-before-network), so whichever
+  path fetches it, the other gets it free next time. It is off by default (a search should not generate upstream
+  requests on its own), capped (3 records unless a number is given), and only touches thin bodies - measured on one
+  query, a 201-character stub became 965 characters of abstract and findings while a 3,020-character neighbour was
+  left alone. Failures keep the local text and are reported rather than blanking the field. It deliberately does
+  **not** sleep between fetches the way the daily's batch path sleeps 8-12 seconds: that is politeness for hundreds
+  of articles, this is three on demand. Same caveat as the daily: the cache is the version fetched at that time, so a
+  revised source article will not be reflected.
+
+- **MMR redundancy removal was built, measured, and left off by default - it removes useful material here.** The
+  new `search --mmr` flag re-orders the final results with Maximal Marginal Relevance (lambda 0.7, the value WeKnora
+  uses) and the evaluation harness grew a **diversity metric** (mean pairwise cosine inside the returned top-k). It
+  does what it says on the ranking side: diversity goes from 0.627 to 0.559 with recall@10 unchanged at 0.95. But
+  judged precision (same criterion and judge as the precision baseline: 10 questions, top-5 each) **drops** - strict
+  relevant 40% -> 30%, loose 68% -> 64% - and recall@5 slips 0.93 -> 0.90. The reason is worth keeping: in this
+  corpus the "duplicates" are mostly **several accounts carrying the same story**, i.e. multi-source confirmation
+  rather than noise, so pushing redundancy down starts deleting the very items that answer the question. The cheap
+  half of that problem is already handled by default same-text collapsing (D-114), which costs nothing and loses no
+  information. Tuning lambda is not the fix (larger just degenerates to plain relevance, smaller deletes more): the
+  direction is wrong, not the weight.
+
 ## 1.11.1
 
 - **Five defects in 1.11.0's document export, all found by looking at a real export instead of the code.**