weflow-cli 1.11.0 → 1.12.0

package/docs/PROJECT_STATE.md

@@ -36,7 +36,7 @@ WeFlow CLI is a local-first command-line tool and MCP server for user-authorized
 | Knowledge workflows | Wiki compilation (`compile_wiki` aggregates `[[concept]]` links into Vault pages; **the article corpus produced none as of 2026-09-26** (the 1,633 article notes then in the Vault carried only their own topic link and `## 关联网络` path links, measured over the whole corpus), so an article-side concept-extraction step was missing; `article-notes` (added the same day) is that step, and it works: 30 article notes produced 110 concepts and 10 concept pages on this machine, which are now the first knowledge pages the assistant can search (`vault search`, `search_knowledge`). The corpus has grown since - 7,119 concept cards, and 22,374 concept pages across both lines as of 2026-09-28. The degeneracy reported in that trial was an artefact of a 120-card sample: with all 710 September articles carded, the topic field splits into 学术/新闻/政治 (62/33/25 of 108 pages), so the earlier "the classifier puts everything in one bucket" judgement is withdrawn. Two upstream noise sources showed up in that trial. The first - WeChat reader UI text inside article summaries - is **fixed**, and the backfill this line used to defer has since been run: measured over the corpus it was 1,484 of 1,633 notes (the corpus has grown to 34,092 since), the cleaner runs on both write paths, and `create_reading_notes --clean-summaries` took the existing notes to **0**; the Vault's `Sources/WeChat` copies were cleaned on 2026-09-28 as well (D-055), so the residue now survives only in `output/biz-daily`, which is the archive. The second is **measured but not fixed**: of 224 notes whose titles read as news, 40 are labelled `学术` - that classifier belongs to the daily pipeline and changing its criteria without labels would be a guess. Fed by both sources: `output/biz-daily` articles and, since 2026-09-26, `output/chat-notes` conversation cards produced by the confirmed `chat-notes` command - which, since 2026-10-01, carries voice messages as **text** rather than a bare `[语音]`: it reads the local faster-whisper transcript cache (content-addressed, `output/.voice-cache` - one definition, in `wechat_voice`) and labels each one `[语音·本机转写·未校对]`, because a machine transcript of someone's speech may be wrong and the model must not quote it as fact. Measured on the 30-day window: **54 of 125** voice messages carry text that used to be a placeholder; the ones still missing are counted and printed, and `--transcribe-voice` fills them locally (the cache is the progress, so an interrupted run resumes); **the graph is also kept free of duplicate nodes by `wiki compile --merge-duplicates`** (74 groups merged, 80 pages removed, and 98 links that pointed at the removed names still resolve through `aliases` - see D-053); **most of the article line's pages were never written by a model** - `wiki compile --pages-from-cards` assembles a page from material the cards already carry (the definition is the card's own `desc`, the related concepts are the other concepts from the same article, the source is that article) and marks it `summary_by: card`. That label is deliberate and is not the same as "no model was involved": the `desc` line was written by `article_notes`, so the quality is that line's quality, and a few read as meta-descriptions of where the term appeared rather than as definitions. 17,383 pages at ¥0; the model-based route was priced at ¥31 and rejected, because every one of those concepts is mentioned by exactly one article (D-054); **the pages accumulate only if `wiki compile --refresh-sources` is run** - `wiki compile` itself only builds pages for concepts that do not have one, so an existing page never gains an edge (measured: one run added 3,164 edges across 324 article-line pages; on 2026-10-01, after the whole
 pre-2026-03-03 window had been carded, another run added **8,551** edges across **3,506** pages - verified against a
 snapshot of all 48,426 pages: the file count changed, the frontmatter that changed is **0**, and the files that lost
-a source line are **0**, so the rewrite is what it claims to be, additive; it rewrites the body only, never the frontmatter, and never deletes a source line no current card accounts for - see D-052), semantic search (two-stage: similarity, then a decision-model rerank over the top 20 candidates; `--no-rerank` reverts) - the index covers 90 days of chat and 30 date-directories of articles **by default**, and `search-index --days / --article-days` changes that (the numbers were hardcoded at the call sites until 2026-09-25, and neither the command nor its preview said so), RAG, WeRead sync, reviews, reading notes, and staged Vault promotion. The article line was extended on 2026-10-01 with the **whole**
+a source line are **0**, so the rewrite is what it claims to be, additive; it rewrites the body only, never the frontmatter, and never deletes a source line no current card accounts for - see D-052), semantic search (two-stage: similarity, then a decision-model rerank over the top 20 candidates; `--no-rerank` reverts) - the index covers 90 days of chat and 30 date-directories of articles **by default**, and `search-index --days / --article-days` changes that (the numbers were hardcoded at the call sites until 2026-09-25, and neither the command nor its preview said so; a **gated local BM25 + vector fusion** is the default since 2026-10-15 (the gate is what makes it work: fragment recall@10 0.91 -> 1.00, hit@1 0.60 -> 0.80, nothing worse - D-111; `--no-hybrid` reverts), an index was then built on this machine on 2026-10-15: 16,027 records (10,759 articles + 5,268 chat messages) out of ~5.87M characters, ≈¥2 at list price), RAG, WeRead sync, reviews, reading notes, and staged Vault promotion. The article line was extended on 2026-10-01 with the **whole**
 pre-2026-03-03 backlog, not only its AI part: **8,754** articles over 157 days (1,702 AI + 7,052 others, of which
 2,709 needed fetching), **8,670** reading notes (free and local - 94 fewer than the index because a note file is named
 `<date>-<title>` and one title republished by two accounts the same day makes a single file; measured per day, that
@@ -93,7 +93,7 @@ mistook a **missing mechanism** for a **tool that cannot be tested**: the harnes
 `assistantPrivacy` per case all along, it just had not generalised it. With a general `config` override
 (`weread-notebooks`, asserting the book title really reaches the answer, because reading `b.title` off the
 notebooks response yields `undefined` while the output still looks fine), what genuinely remains uncovered is
-**one** tool: `search_semantic` (needs `dashscopeApiKey` **and** a built index - neither exists on this machine). That
+**one** tool: `search_semantic` (it needs `dashscopeApiKey` **and** a built index; as of 2026-10-15 this machine has both - the index holds 16,027 records - so it stays uncovered for a different reason: its query vector is a **second egress**, and the harness admits only the model call). That
 list is no longer prose that can rot: it is the `EVAL_UNCOVERED` map, and `test/assistant-eval.test.ts` asserts both
 directions - every tool is either claimed by some case's assertions or registered there with a reason, and no
 registered name has stopped existing - so adding a tool means writing a case or writing a reason, and skipping both
@@ -133,6 +133,7 @@ few weeks. Each row is an index entry - the measurements and the reasoning live
 | The MMKV stores under `%APPDATA%\Tencent\xwechat\radium\mmkv\` (D-101) | Their format holds (crc32 precondition passes; one is empty), but the only key we can construct does not open them, and the `config/` root key is random and was never captured. | **Deliberately not attempted**: the remaining route is injecting into the client process, which this repo does not do (D-101). |
 | **Why** the data root uses hard links (CONTACT_DB_SCHEMA §6.2) | The effect is measured (395 groups, 407 MB double-counted if you sum sizes); the cause is the report's `hardlink.dat` inference (multi-account / multi-version reuse) and has not been established. The file itself was read on 2026-10-11: it is **32 hex characters** (a 16-byte value), one copy on this machine, older than both installed client versions - so it is a marker or seed, not the link inventory (16 bytes cannot describe 395 groups). | Finding what consumes it. One testable guess, not yet tried: use those 16 bytes as an AES-128 key for the `radium\mmkv\` stores, judged by the D-101 oracle (crc32 consistent **and** the whole record sequence walks). The stores are on this machine; the tooling is not. |
 | Issue #7 (HTML emoji export) | Open pending reporter verification. | The reporter's confirmation. |
+| Issue #11 (documents in the non-HTML formats) | **Answered 2026-10-13 and fixed in 1.11.0/1.11.1** ([comment](https://github.com/zhuobichen/weflow-cli/issues/11#issuecomment-6160700152)); the issue is left open for the reporter, as with #7. The fix itself was re-checked against a real export and five defects were found that way (names, control characters, reserved device names, a manifest that a narrower run overwrote - see CHANGELOG 1.11.1). Still open: **the document-body path has only ever been measured on this machine** - one account, 1,855 file messages, 88.6% resolved, and the remaining 11.4% *look* like files that were never downloaded here but that reading is itself unverified. | The reporter's confirmation, plus a second machine or account - that is what would separate "our matching misses files that exist" from "those files are not here". |
 | The CLI's **runtime** error paths under `--json` (~100 prose sites in `bin/weflow-cli.ts`) | The argument-validation family **is** audited (80 sites, 0 defects, 2026-10-11) and `weread`'s nine one-line failure branches are fixed, but the rest are runtime failures - "not configured", "no WeChat process", permission denied - whose reachability under `--json` depends on that command's interactive gate. A window around a line cannot tell "mentions `opts.json`" from "is governed by `opts.json`": that filter answered wrongly in **both** directions on 2026-10-11 (305 candidates, then 12, of which the real family was nine). | Reading each site, or driving each command into its failure state where that needs no credentials and no network - `weread`'s empty-`HOME` case is the worked example (`test/json-error-surface.test.ts`). Two named gaps: `export --contract` is verified **on this machine only**, because its check runs after `resolveTalker()`; and the human-facing error arm is not uniform across the CLI (most sites `console.error`, some `console.log`) - a separate, user-visible change that this work deliberately did not make. |
 
 **What this is not**: a bug list. Everything above works on the paths this machine exercises; what is missing is
@@ -164,10 +165,10 @@ Documentation was synchronized with the current source baseline on 2026-09-19. C
 
 - Issue #7 remains open pending reporter verification. HTML emoji export preserves the emoji label and matches resource records only by nonzero server-message identity; local IDs are not unique across shards or conversations. WeChat 4.x V2 image containers are decoded with an account-specific media key derived from local `kvcomm` data and verified against a real cached V2 header. Remote emoticons use AES-CBC with the message key as both key and IV. Entity-escaped emoji XML is normalized before extracting MD5 and media URLs; failed encrypted downloads fall back across available thumbnail/CDN fields. Signature-only default `[打脸]` messages use the bundled official Facepalm asset when no message-specific resource is available. Forwarded app cards remain structured links with available covers, and Bilibili share pages can resolve covers through their BV metadata endpoint when no `og:image` is present. Forty-two synthetic Python regression tests (`export_chat_media_test.py`, `export_chat_media_report_test.py`) cover these paths; run `python -m unittest discover -s test -p '*_test.py' -v`. CI also runs these tests. This does not establish compatibility with every real WeChat media format.
 
-- **Media leaves the export as files now, images and documents alike, and the matching happens once.** `export … md` writes images to `<output>/<talker>_assets/` and references them relatively; `export … json|txt|excel --media` runs the same Python extraction and then writes the requested format against the same `manifest.json`. The switch existed because a reporter asked whether the non-HTML formats omitted images *and documents* - images had just been fixed, documents had not, and **no format exported a document body** (the HTML page showed `[文件] <name>`). Documents are file-message appmsg `type=6`: the row carries only name/md5/size, and the body sits in the account directory's `msg/file/YYYY-MM/` **with no per-conversation directory**, so the index is a stat-only `(normalized name, byte size)` lookup and md5 disambiguates only when a name is ambiguous (WeChat appends its own `(1)`). Measured on this machine: 1,855 file messages, 1,489 unique matches + 154 md5-resolved = **88.6%**, and the remaining 212 have no local body at all (never downloaded, or cleaned up) - those stay as `[文件] name` labels and are *counted*, not guessed (D-014). Bodies are copied under their original names and extensions; the manifest is `weflow-media-manifest/v2` (values carry `kind`) and v1 files still read. Two traps are worth keeping: `<refermsg>` sits **inside** `<appmsg>`, so searching the XML for the first matching tag reported 7,083 "files" that were nearly all quoted text (the parser now reads `<appmsg>`'s direct children only); and a count of newly copied files was reported as the number of documents exported, so a second run into the same directory printed "0 document(s)" with 67 documents on disk. **HTML deliberately embeds no document bodies** (a multi-MB PDF as base64 in a page nobody can open), and voice/video still have labels only.
+- **Media leaves the export as files now, images and documents alike, and the matching happens once.** `export … md` writes images to `<output>/<talker>_assets/` and references them relatively; `export … json|txt|excel --media` runs the same Python extraction and then writes the requested format against the same `manifest.json`. The switch existed because a reporter asked whether the non-HTML formats omitted images *and documents* - images had just been fixed, documents had not, and **no format exported a document body** (the HTML page showed `[文件] <name>`). Documents are file-message appmsg `type=6`: the row carries only name/md5/size, and the body sits in the account directory's `msg/file/YYYY-MM/` **with no per-conversation directory**, so the index is a stat-only `(normalized name, byte size)` lookup and md5 disambiguates only when a name is ambiguous (WeChat appends its own `(1)`). Measured on this machine: 1,855 file messages, 1,489 unique matches + 154 md5-resolved = **88.6%**, and the remaining 212 have no local body at all (never downloaded, or cleaned up) - those stay as `[文件] name` labels and are *counted*, not guessed (D-014). Bodies are copied under their original names and extensions; the manifest is `weflow-media-manifest/v2` (values carry `kind`) and v1 files still read. Two traps are worth keeping: `<refermsg>` sits **inside** `<appmsg>`, so searching the XML for the first matching tag reported 7,083 "files" that were nearly all quoted text (the parser now reads `<appmsg>`'s direct children only); and a count of newly copied files was reported as the number of documents exported, so a second run into the same directory printed "0 document(s)" with 67 documents on disk. **HTML deliberately embeds no document bodies** (a multi-MB PDF as base64 in a page nobody can open), and voice/video still have labels only. **One vocabulary now covers all three artifacts** (`*_media.json`, `manifest.json`, the CLI's JSON): each word answers exactly one question, so no two of them can be read as the same claim. `found`/`missing` (report statuses, documents) say whether the item could be **located in the source data**; `written`/`copied`/`unavailable` (manifest `documents`, and the same three inside the CLI's `media.documents`) say whether it was **materialized into the export directory** - `written` is what is there now, `copied` is what this run added, `unavailable` is what could not be written (not found, too large, or the copy failed). The report says `found` for a file whose copy then fails, and the manifest says `unavailable` for it: two different questions, two true answers, where the old shared word `exported` would have made "已导出" false. All three branches (html / md / `--media`) now emit that summary from one function (`media_json_summary`), so they cannot disagree, and it carries the report's file name (html was the only one that used to).
 
 - WeChat platform behavior, database formats, account restrictions, and terms can change without notice. Local operation is not a legal, account-safety, or platform-compatibility guarantee.
-- **The daily is no longer all-or-nothing.** It fetches every article before writing anything and the bodies lived only in memory, so any interruption threw the whole day away - a 400-article day is over an hour of fetching, and the 2026-09-22 run stopped at 105/403 with nothing written. Bodies are now cached under `output/.cache/fetch/` keyed by URL, so a re-run fetches only the missing ones (measured: a resumed batch of 3 goes from 64 s to 2.0 s, and cache hits take **no** throttle sleep because they send no request - the throttle exists to spare the upstream, and this is the same direction). Only successes are cached, so a failure is retried next time rather than remembered. `--no-fetch-cache` forces a refetch. **The lesson that made this urgent**: a long single-shot run is a gamble, and `cmd | tee log` reports the wrapper's exit code, not the program's - the 105/403 failure looked like `exit 0`.
+- **The daily is no longer all-or-nothing.** It fetches every article before writing anything and the bodies lived only in memory, so any interruption threw the whole day away - a 400-article day is over an hour of fetching, and the 2026-09-22 run stopped at 105/403 with nothing written. Bodies are now cached under `output/.cache/fetch/` keyed by URL, so a re-run fetches only the missing ones (measured: a resumed batch of 3 goes from 64 s to 2.0 s, and cache hits take **no** throttle sleep because they send no request - the throttle exists to spare the upstream, and this is the same direction). Only successes are cached, so a failure is retried next time rather than remembered. `--no-fetch-cache` forces a refetch. **Since 2026-10-14 the historical backfill writes and reads this same cache** (it calls `fetch_article_cached`, and carries the same `--no-fetch-cache`), so a backfilled day can be produced by `daily --date X` afterwards **without re-fetching what the backfill already got** - measured on 2026-10-13: backfilling populated 145 cache entries (57 -> 202 files) and those 145 became cache hits for the daily. **The overlap is partial and should not be read as "the day becomes free"**: the daily fetches every article with a url (it wants full bodies for summaries) while the backfill only fetches the ones whose local body is under 100 characters, so on that same day **340 of 485 articles still needed the network** (~55 min at the deliberate throttle). Making the backfill pre-fetch everything would remove that, but it means *more* upstream requests and `docs/NEXT_ACTIONS.md` says not to add requests through a default change - so the default stays as it is, and an explicit opt-in is the only shape that work could take. Not implemented; recorded in D-108. **The lesson that made this urgent**: a long single-shot run is a gamble, and `cmd | tee log` reports the wrapper's exit code, not the program's - the 105/403 failure looked like `exit 0`.
 - Where a many-source day spends its time is now measured rather than assumed, and two of the three biggest items have been fixed. A 190-article day **was** ~3 h 10 m: fetching bodies ~130 min, the deliberate 8-12 s inter-article throttle ~32 min, image downloads ~18 min (47 image links per three articles, 0.37 s and 135 KB each, sequential), and the serial summary loop ~9 min (measured 2.27/2.92/2.45 s per article, plus 0.3 s each). It is now ~1 h 5 m: fetching ~33 min (gzip, below), throttle ~32 min, images ~3 min (6-way concurrent), summaries ~2 min (6-way concurrent). Classification was never a factor - already concurrent. **The throttle is now the largest single item, and it is the one thing left that is deliberate** ("reduce upstream pressure"); the other two costs were implementation choices, not policy. Both new concurrency settings are 6 (`IMAGE_WORKERS`, `SUMMARY_WORKERS`), matching `JEV_WORKERS`.
 - **Article bodies are now fetched with `Accept-Encoding: gzip, deflate`**, which `urllib` does not send by default. Measured on the same four articles: 3.3-3.5 MB and 33-40 s before, 0.7-0.8 MB and 6-10 s after, with the decompressed page identical in size and `js_content` / `rich_media_content` still present - so ~130 min of fetching becomes ~33 min on a 190-article day, and **fewer bytes cross the wire than before**, which is the same direction as the throttle. `_decode_body` recognises only the two encodings the request asks for, and passes anything else through unchanged: raising there would turn a transfer optimisation into a fetch failure. The relative cost of the remaining parts is unchanged, so the throttle is now the largest single line item.
 - The article body is **not** available locally, so fetching cannot be skipped: a 公众号 message in the local database carries title, digest, URL and metadata, and stripping its XML tags leaves 0 Chinese characters (the ~5-12 KB that remains is URLs, base64 and ids).
@@ -270,7 +271,15 @@ fallback is a defect report, not a statistic** - it was in the output and read p
 `'学术' != 'AI'` in a test made it undeniable. `test/backfill_articles_test.py` now pins it, including a mutation
 check (replacing the guess with the default fails three tests).
 
-**Historical backfill is a separate line from the daily pipeline, and stays separate.** Neither of the two
+**Historical backfill is a separate line from the daily pipeline, and stays separate - but the two now share one
+body cache.** Since 2026-10-14 the backfill fetches through the daily's own `biz_daily.fetch_article_cached`
+(write-through to `output/.cache/fetch/<md5(url)>.md`, successes only), so a backfilled day can be produced the
+official way afterwards without touching the network: `daily --date X` reads those bodies and its 8-12 s
+inter-article throttle never fires on a cache hit (the sleep lives in the daily's *call loop*, not in the fetch
+function). The pipelines themselves stay separate for the reasons below - different outputs (text vs
+HTML+images), different fetch policy (concurrent, no throttle here vs serial and deliberately throttled there) -
+and each side keeps its own `--no-fetch-cache` for a forced refetch (same name, same meaning: skip the read,
+still write the result back). Neither of the two
 image-carrying outputs is needed for the knowledge base, and the difference is not marginal: the daily path stores
 `output/biz-daily/<date>/` (article HTML plus images, ~1.1 MB/article) *and* an identical copy under
 `wechat-vault/Sources/WeChat/<date>/` (`pipeline.py:133`), so the same 35,463 articles would cost ~65 GB the daily
@@ -280,7 +289,14 @@ were symptoms of the same mismatch, that this path was built for one day at a ti
 reads a title and an account name, never the body, which is what makes pre-filtering possible at all and also what
 bounds the claim: the AI set is the keyword's, not the daily line's model's, and these articles have never been
 through a model. Partially backfilled days are marked (`topicFilter`, plus `truncated` for a `--limit-per-day`
-slice) so `day_done` will not mistake one for a finished day - D-060. **Run the same day**: the AI subset is
+slice, plus `fetchFailed` when a body fetch failed **in that run**) so `day_done` will not mistake one for a
+finished day - D-060. `fetchFailed` differs from the other two in one way worth knowing: it is **not sticky**,
+because a later full run retries the failed articles (they are still short-bodied, so they stay in the todo list,
+and the ones that succeeded are cache hits) - so a run with zero failures clears the marker. Keeping a fetch
+failure and a genuinely short body apart matters: the first is retryable, the second is a property of the
+article, and before the split a single network hiccup was written down as "skipped N (body too short)" with the
+day still marked complete, i.e. those articles were buried for good (measured elsewhere in this file: 28 of 84
+failures were transient and did come back on retry). **Run the same day**: the AI subset is
 fetched and on disk - **1,702 of the 1,758** in-scope articles across 156 days (96.8%), needing 1,208 network fetches;
 `output/biz-daily/<day>/AI/` holds them, no other topic directory was created, every one of those days carries
 `topicFilter`, and a full run would skip **0** of them. The 56 not pulled are not a fetch defect: re-probing all 84
@@ -359,10 +375,75 @@ line too (D-105): `search`, `chat`, `vault search`, `vault rag` and the assistan
 `search_knowledge` all take `line: wiki|chat|all`, default to the article line where the user sees it (the
 *functions* default to `all` so a caller that forgets cannot silently halve the search), and every output says which
 line it searched. The semantic side carries this as a mask applied **before** the top-k sort over one shared index.
-**Limitation to keep in view**: the vector path of semantic search has never run against a real index on this
-machine - the index does not exist (building it needs `dashscopeApiKey` and spends the user's embedding credit), so
-the mask is pinned against a hand-made fixture index and the real-data check was done on the keyword fallback (which
-is the path this machine actually takes). Two things about this were
+**Limitation, updated 2026-10-15**: the vector path now runs against a **real** index on this machine (the user
+supplied a `dashscopeApiKey`; the index holds 16,027 records - 10,759 articles + 5,268 chat messages - and real
+searches were run through the CLI). What the measurement changed is the *shape* of the limit: no longer "never
+exercised", but "the article line ranks weakly" - raw cosine only, because rerank is blocked here (Typesafe returns
+`HTTP 451: not available in your region`), scores clustering inside 0.49-0.58 and **not comparable across queries**
+(0.58 on one query need not mean "closer" than 0.50 on another), and a compound query losing its rarer half
+("卫星遥感反演气溶胶光学厚度" returned 气溶胶 papers first and 遥感 at rank 4). **The cause was found the same
+day, and it was not the truncation**: none of the 10,759 daily `.md` files in that window contains `## 正文`, so
+every article fell back to `content[:1000]` and only YAML metadata was embedded (sampled 300: 100% contain
+`title:`, median CJK ratio 0.16). The extractor now reads the real structure instead (0.16 → 0.80 median CJK
+ratio, 299/300 files) with a test that fails if it is reverted; **the index needs a `--full` rebuild to pick that
+up**, because an incremental build embeds new ids only (see D-109). Cost was measured the same day: the build shipped ~5.87M characters (~¥2
+estimated), one query ~12 tokens. The mask itself is still pinned against a hand-made fixture index, since a fixture
+**A recall baseline now exists (measured 2026-10-15, index of 15,995 records), so the next change has something to
+be compared against.** Three batteries, all with a known correct answer: (a) 80 auto queries built from a record's
+own text - 12-character verbatim fragments recall@10 = **13/20** on the article line and **12/20** on chat; titles /
+first-30-characters recall@10 = **20/20** on the article line and 16/20 on chat; (b) 10 hand-written questions, each
+with a literal "the answer must contain this" anchor: **10/10 present in the index**, first anchor-bearing record at
+rank 1 for **9/10** (the miss sat at rank 14); (c) a plumbing self-check that returns the stored text to rank 1
+16/16. The split matters: **title-like lookups are near-perfect, verbatim-fragment lookups are not** (median rank
+5-6), which is the price of one vector per record with the first ~500 characters embedded. Two readings that are
+easy to get wrong: **coverage is not recall** (the article line is the last 30 date directories, 2026-09-12..10-13,
+chat is 90 days, so a miss outside that window is a coverage gap), and **"how many records contain the anchor" is not
+precision** - one of the ten anchors was satisfied by a "recommended reading" footer on an unrelated article.
+
+**Per-account lookup needed an attribute filter, not vectors (same day).** The window holds 10,759 articles from
+**413 accounts** (largest: one tabloid with 908, then 632 and 529 - the top three alone are ~19% of the window, and
+they are general news, not air quality). Searching an account's name returns only 0-3 of the top 10 from that account,
+because the name lives in the record's `id` (files are named `<account>-<title>.md`) and not in the embedded text -
+the frontmatter's `source:` left the embedded text when the article body extraction was fixed. `search --from <name>`
+now filters on that id (substring; article line only, since a chat message has no source account) and says so when an
+account has no records instead of returning nothing. Verified on the live index: without the filter, `暗访` returns
+0 of 5 from the tabloid; with `--from 红星新闻`, 5 of 5 are its own. It hands back the right **pool**, not a better
+order inside it - ranking within a general-news account is still nearest-neighbour, so a topical query can return
+that account's unrelated stories.
+
+**A precision baseline was measured the same day, and it has to be read split - the raw number is misleading on its
+own.** Ten fixed questions, top-5 each, every item labelled relevant / partial / irrelevant by the configured
+DeepSeek model against one stated criterion ("does this item answer what was asked"), **not** by human label - and
+not independently spot-checked yet beyond the author's own read of all 50. Raw strict precision@5 = 12/50 = **24%**
+(loose, counting `partial`, 62%); after collapsing identical texts it is 12/46 = 26%, so multi-account reposts cost
+only ~8% of the slots. The split is what matters: **content questions ("what does X say") score 12/32 = 38%**, while
+**procedure questions ("how do I use X / where do I find Y") score 0/14**. Two of the three worst queries are
+**coverage, not ranking**: the index contains **zero** records mentioning `HYSPLIT` (and zero with `后向轨迹`), one
+with `在线监测`, two with `政策文件` - a corpus of 公众号 articles has no manuals in it, so no ranking change can fix
+those. Even inside the content half the shape is the same one the recall battery showed: precision@1 is 5/10, and the
+entries below the first are usually same-domain **neighbours** rather than answers (a monitoring question returns
+aerosol-formation papers; an air-quality-forecast question returns wind-power and tropical-cyclone forecast papers).
+That is the signature of one vector per record with only the first ~500 characters embedded and no rerank available.
+
+**A repeatable harness now carries these numbers.** `scripts/recall_eval.py` builds its ground truth at run time
+from the index itself (a uniqueness-checked 12-character fragment from the middle of a record's own text, and the
+record's title / first 30 characters), reports the standard metric set (Recall@1/5/10, MRR@10, nDCG@10, Hit@1) plus
+**coverage** - so a query whose answer is not in the corpus at all is reported as a coverage gap rather than counted
+as a ranking failure - and requires `--yes` because each question spends an embedding and leaves the machine. Running
+it immediately found a recall win: adding two gates to the keyword side of the fusion (only documents matching a rare
+term vote, and only if their BM25 score is within 0.5x of the best; no rare term in the query means the keyword side
+abstains) took fragment recall@10 from 0.91 to **1.00** and hit@1 from 0.60 to **0.80**, moved title hit@1 from 0.83
+to 0.86, and left the hand-written-question battery unchanged - where the **ungated** version of the same fusion
+measured exactly like pure vectors on all three batteries. Fusion is on by default, and **rerank is live** since the same day (it falls back to the configured chat model, since the Jev contract is region-blocked here): on the same 40 questions hit@1 0.55 -> **0.75**, recall@5 0.65 -> **0.93**, recall@10 0.75 -> **0.95** and nDCG@10 0.63 -> **0.85**; the last step of that was the rerank **candidate pool**, raised from 20 to 100 by default after the pool curve 20/50/100/200 measured recall@10 0.82 -> 0.88 -> 0.95 -> 0.95 - which clears the published production thresholds (Recall@10 >= 0.90, MRR@10 >= 0.75) - see D-112 and D-113 (the latter also corrects a "regression" in D-112 that re-measured as noise); D-111 (which
+also corrects D-110's "harmful" conclusion: the fault was function words voting, not fusion).
+
+**One lever is measured dead, not merely unproven: filtering the index by `relevance`.** In this window every one of
+the 10,759 articles carries `relevance: 中` - the window is mostly backfilled days, and the backfill path deliberately
+does no AI work, so the field holds a default rather than a judgement (see the `result` sub-scores in the quality
+notes: `relevanceScore` only exists where the judge actually ran). It therefore has no discriminative power here and
+must not be proposed as a precision fix without first checking that the judge ran for those days.
+
+is what a unit test can hold. Two things about this were
 wrong on 2026-09-27 and are now fixed: the concept directory was **declared in two places** (`VAULT_DIRS` said
 `007_Wiki/Concepts`, every writer and reader used the top-level `Wiki/Concepts`), so `007_Wiki/` was created and
 never filled; and the historical import skipped `Sources/` entirely (it would have carried ~33 GB of images), which

package/docs/RELEASING.md

@@ -21,6 +21,19 @@ npm 包与 GitHub 分开发布，中间隔多久都不奇怪——`1.5.0` 与主
 已备好但**从未发布**的版本号可以并进下一个：`1.5.1` 准备过却没发，
 它的修复就直接并进了 `1.6.0`。
 
+**改 `package.json` 的版本号时必须同时改 `package-lock.json`**（lockfileVersion 3 里有两个地方：
+顶层的 `version` 与 `packages[""].version`）。**2026-10-13 发 1.11.0 时就是这么红了一次**：包已经发出去、
+tag 也打了，`release-consistency` 才报「版本号与锁文件必须一致」—— 我本地只跑了 build/test/文档命令检查，
+而这三样都不看锁文件。发布物本身没问题（`files` 白名单里没有锁文件），但 master 的 CI 红了。
+
+```powershell
+node -p "require('./package.json').version"     # 两个必须相等
+node -p "require('./package-lock.json').version"
+```
+
+顺手验一下锁文件没跑偏（只写锁、不动 `node_modules`）：
+`npm install --package-lock-only --dry-run` —— 报 `up to date` 即可，那也正是 `npm ci` 会不会炸的判据。
+
 ## 2. 整理 CHANGELOG
 
 - 把 `## Unreleased` 改名为 `## <新版本号>`
@@ -229,6 +242,40 @@ npm view weflow-cli version --registry=https://registry.npmjs.org  # 用官方
   3cf5605，它不在包里那一份 tarball 的对应关系里 —— 写下来是因为 1.8.0 那次正是"事后不知道 tag 对应哪个
   提交"）。GitHub Release `v1.10.0` 同样指向它，正文由 `## 1.10.0` 段机械切出（92,213 字符，未触及
   125,000 上限 ⇒ 未截断），List 上已是 **Latest**。
+- **1.11.0 的出处核过了（2026-10-13，发布当天，四种证据）**：
+  1. 本地 `npm pack` 的 shasum = `21db27cbb151987331c7993c625eb4626c23f58a`，与**两个源**上
+     `weflow-cli@1.11.0` 的 `dist.shasum` **三者相同** ⇒ 发出去的就是我构建的那一份，且镜像同步的是同一份；
+  2. 从 registry（不是本地 tarball）装进一个**空目录**，用**装好的那份**跑 `cli.cjs --version` 得到 `1.11.0`、
+     `--help` 72 行含 `Commands:`；1.9.0 缺过的 `logo.png` 与 `scripts/data/wechat_emoji.json` 都在，
+     开发用探针 `.mjs` 不在包里（有意，见 `test/pack-contents.test.ts`）；
+  3. **本次真正改动的那个脚本逐字节核过**：装好的那份 `scripts/export_chat_html.py` 的 sha256
+     = `73117B7A35347D68FE239D660C9A0531E231BA8CA7A0A31F8DA0C07D5D54C041`，与仓库里那份**相同**
+     ⇒ 用户拿到的是这一版脚本，不是旧的。这是比"文件在不在"更强的一层：**在，而且是我这次写的那份**；
+  4. 两个源都是 1.11.0：官方源与 `registry.npmmirror.com`（`curl -X PUT …/syncs` 受理后，发布后约 1 分钟
+     查 shasum 仍取不到、20 分钟内再查时两源都已生效 —— 这次的窗口比 1.10.0 那次（约 4 分钟）长，
+     但**没有逐点采样**，所以只记"两次查询的区间"，不写一个假的分钟数）。
+  **发布日志 0 条告警**；**tag 打在被发布的那个提交上**：`v1.11.0` → `bd82316`（`821479b` 是功能提交，
+  `bd82316` 只改了 `package.json` 版本号与 CHANGELOG 段名）。GitHub Release `v1.11.0` 指向 `bd82316`，
+  正文由 `## 1.11.0` 段机械切出（9,689 字符、7 条 bullet、未触及 125,000 上限 ⇒ 未截断），
+  建完在 List 上已是 **Latest**（把 1.10.0 挤下去是预期行为）。
+  ⚠️ **这次发布本身有一次 CI 失败，记在这里免得事后只看到"一切正常"**：`chore: 1.11.0` 那次提交的
+  `release-consistency` 报「版本号与锁文件必须一致」—— 我改了 `package.json` 忘了 `package-lock.json`，
+  连红三次（含紧随其后的文档提交）。**发布物不受影响**（白名单里没有锁文件、shasum 与装出来的那份都验过），
+  但 master 的 CI 是红的；修在 `f9e55c8`，之后两次推送都是 success。清单第 1 步已补上这条检查。
+- **1.11.1 的出处核过了（2026-10-13，发布当天，四种证据）**：这是 1.11.0 发布后**当天**的补丁版
+  （文档导出五处修复，详见 CHANGELOG）。
+  1. 本地 `npm pack` 的 shasum = `5527b538e3ed5ec49520e14e563a8bff31ee4ea4`，与**两个源**上
+     `weflow-cli@1.11.1` 的 `dist.shasum` **三者相同**；
+  2. 从 registry 装进空目录，用**装好的那份**跑 `cli.cjs --version` 得到 `1.11.1`，且它的
+     `scripts/export_chat_html.py` 与仓库里那份 **sha256 逐字节相同**（并确认含本次新增的
+     `_load_existing_manifest` / `_clean_asset_component`）——"在，而且是我这次写的那份"；
+  3. 两个源都是 1.11.1：官方源**约 2 分钟**就有了，`registry.npmmirror.com` 约 **5 分钟**（`curl -X PUT …/syncs`
+     受理后）；两个源上该版本的 shasum 也一致；
+  4. **版本号三处一起改**（`package.json` + `package-lock.json` 的顶层 `version` 与 `packages[""].version`），
+     所以这一次 `release-consistency` 没有再红 —— 1.11.0 那次就是漏了锁文件。
+  **tag 打在被发布的那个提交上**：`v1.11.1` → `888d8fc`（只改了 `package.json`/`package-lock.json`/CHANGELOG
+  段名，功能提交是 `2b03b4f`）。GitHub Release `v1.11.1` 指向它，正文由 `## 1.11.1` 段机械切出
+  （3,154 字符、1 条 bullet、未触及上限），建完是 **Latest**。
 
 
 ## 发布后

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "weflow-cli",
-  "version": "1.11.0",
+  "version": "1.12.0",
   "description": "让 AI 读懂你的微信 — 公众号日报·聊天导出·知识库搜索·MCP Server",
   "type": "module",
   "bin": {

package/scripts/backfill_articles.py

@@ -54,7 +54,8 @@ from _utils import load_config  # noqa: E402
 from biz_daily import (DEFAULT_RELEVANCE, DEFAULT_TOPIC, TOPICS,  # noqa: E402
                        _group_by_topic, _guess_topic, _serializable_article,
                        _tags_for_write, extract_article_info, fetch_article,
-                       get_db_keys, sanitize_filename, summary_section)
+                       fetch_article_cached, get_db_keys, sanitize_filename,
+                       summary_section)
 from _utils import strip_wx_ads_document, write_with_frontmatter  # noqa: E402
 
 TZ = timezone(timedelta(hours=8))
@@ -159,20 +160,32 @@ def select_topics(articles, topics):
     return [a for a in articles if a.get('topic') in topics]
 
 
-def fetch_bodies(articles, workers: int, log=None):
+def fetch_bodies(articles, workers: int, log=None, use_cache: bool = True):
     """并发把这批文章的正文抓回来，写进 `fetched_md`。
 
     **只抓正文，不碰图片**——`fetch_article` 返回的 markdown 里图片是链接，本脚本
     就此打住，不再像 `biz_daily` 那样逐张下到本地（那一步占了实测 30 秒/篇里的绝大部分）。
+
+    **走 `fetch_article_cached`（日报那条同一个缓存）**，理由是把"补历史"和"正日报"
+    接起来：补完的正文落在 `output/.cache/fetch/<md5(url)>.md`，随后逐天
+    `daily --date X` 全是缓存命中——而日报的 8–12 秒节流在它的**调用循环**里
+    （`biz_daily` 抓取那一段），命中缓存既不联网也不睡，所以正式路径能快速产出
+    与平常完全一样的完整日报（含摘要与配图）。反过来也一样：日报跑过的天，这里重抓
+    时直接命中，不会再去打扰上游。
+
+    `use_cache=False`（`--no-fetch-cache`）**只跳过"读"**：仍然照常联网，成功结果照常
+    写回缓存。这样"强制重抓"既是"这次不看旧缓存"，也是"把旧缓存刷新成新的"。
+    **失败不写缓存**（由 `fetch_article_cached` 保证），所以重抓失败不会把已有的有效
+    缓存清掉或弄脏。
     """
     todo = [a for a in articles
             if len(body_without_media(a.get('local_text'))) < MIN_BODY_CHARS and a.get('url')]
     if not todo:
-        return {'attempted': 0, 'ok': 0}
+        return {'attempted': 0, 'ok': 0, 'failed': 0}
 
     def one(article):
         try:
-            body = fetch_article(article['url'])
+            body, _hit = fetch_article_cached(article['url'], use_cache=use_cache)
         except Exception:
             return False
         if body:
@@ -184,9 +197,11 @@ def fetch_bodies(articles, workers: int, log=None):
     with ThreadPoolExecutor(max_workers=max(workers, 1)) as pool:
         for result in pool.map(one, todo):
             ok += 1 if result else 0
+    failed = len(todo) - ok
     if log:
-        log('    抓正文: 尝试 %d，成功 %d，并发 %d' % (len(todo), ok, workers))
-    return {'attempted': len(todo), 'ok': ok}
+        log('    抓正文: 尝试 %d，成功 %d%s，并发 %d'
+            % (len(todo), ok, ('，失败 %d' % failed) if failed else '', workers))
+    return {'attempted': len(todo), 'ok': ok, 'failed': failed}
 
 
 def _read_index(day_dir) -> dict:
@@ -226,7 +241,7 @@ def _merge_articles(kept, fresh):
     return sorted(merged.values(), key=lambda e: (str(e.get('time', '')), str(e.get('title', ''))))
 
 
-def write_day(articles, day, out_root, topic_filter=None, truncated=False):
+def write_day(articles, day, out_root, topic_filter=None, truncated=False, fetch_failed=0):
     """落盘：`<日期>/<主题>/<来源>-<标题>.md` + `.articles.json` + README。
 
     格式与 `biz_daily` 一致——`create_reading_notes` 与 `compile_wiki` 都按那个形状读。
@@ -324,6 +339,17 @@ def write_day(articles, day, out_root, topic_filter=None, truncated=False):
     if truncated or prior.get('truncated'):
         # `truncated` 一旦记上就不撤销：这一次没截，不代表上一次截掉的那些补回来了。
         payload['truncated'] = True
+    # **抓取失败要与"正文本来就短"分开记**，否则一句"跳过 N 篇（正文太短）"把两件事混成
+    # 一件：前者这一轮可重试、后者是文章自己的属性。这个标记让 `day_done` 把这一天判成
+    # **没做完**，于是下次重跑会再试一遍——正文命中缓存的那批不花钱，只重试失败的。
+    #
+    # 它**不是 sticky 的**（与 `truncated` 不同）：全量跑完一轮、一篇都没失败，就说明
+    # 上一次失败的那些也补齐了，标记该消失。但**局部**跑（`--topic`）只覆盖一部分文章，
+    # 它没重试到的那些（别的主题）还在，所以那一档只保证"至少还有这么多"，取两轮的较大者。
+    carried_failed = int(prior.get('fetchFailed') or 0)
+    failed_total = fetch_failed if topic_filter is None else max(fetch_failed, carried_failed)
+    if failed_total:
+        payload['fetchFailed'] = failed_total
     (out_dir / '.articles.json').write_text(
         json.dumps(payload, ensure_ascii=False, indent=2), encoding='utf-8')
     if scope_filter:
@@ -333,11 +359,19 @@ def write_day(articles, day, out_root, topic_filter=None, truncated=False):
         scope = ''
     # README 说的是**这一天**有几篇（合并后的），不是这一轮写了几篇 —— 分两次回填的那天，
     # 前者才是读它的人想要的数。
+    failed_note = ''
+    if failed_total:
+        # 让人**看得见**这天没做完：README 是这个目录里唯一给人看的文件，
+        # 只写"共 N 篇"会让"少了 N 篇"读起来像"这天就这些文章"。
+        failed_note = ('\n其中 **%d 篇正文没抓到**（网络/来源问题），这一天不算完成；'
+                       '重跑会只重试这些篇（其余走缓存）。\n' % failed_total)
     (out_dir / 'README.md').write_text(
         '# 公众号文章 — %s（历史回填）\n\n共 %d 篇。正文由 `backfill_articles.py` 抓取，'
-        '**未下载图片、未调用模型**。\n%s' % (day, len(payload['articles']), scope), encoding='utf-8')
+        '**未下载图片、未调用模型**。\n%s%s' % (day, len(payload['articles']), scope, failed_note),
+        encoding='utf-8')
     return {'written': len(written), 'skipped': len(skipped),
-            'fallbackTopics': fallbacks, 'skipSample': skipped[:3]}
+            'fallbackTopics': fallbacks, 'skipSample': skipped[:3],
+            'fetchFailed': failed_total}
 
 
 def copy_to_vault(day: str, out_root: str, vault_root: str) -> dict:
@@ -429,6 +463,11 @@ def day_done(out_root, day, topics=None) -> bool:
         return False
     if payload.get('truncated'):
         return False
+    if payload.get('fetchFailed'):
+        # 上一次这天有文章**抓取失败**（不是"正文本来就短"）。不算做完 ⇒ 下次重跑会再试。
+        # 不计这一条的话，一次网络抖动就会被"这天已回填"永久埋掉：文件在、索引在、md 也有，
+        # 少的那几篇谁都看不出来。仓库里量过同类事故：84 次失败里 28 次重试就能拿到。
+        return False
     stored = payload.get('topicFilter')
     if stored is None:
         return True            # 完整的一天，所有主题都在
@@ -460,7 +499,13 @@ def main():
                         help='只回填这些主题（可重复或逗号分隔，合法值 %s），不写=全部。'
                              '只回填过一部分主题的一天，之后跑全量时不会被跳过。'
                              % '/'.join(TOPICS))
-    parser.add_argument('--refresh', action='store_true', help='已回填过的天也重做（默认跳过）')
+    parser.add_argument('--no-fetch-cache', action='store_true',
+                        help='忽略正文抓取缓存，强制重新抓取（默认命中缓存，用于续跑与重跑）。'
+                             '与日报的同名开关行为一致；**与 --refresh 不是一回事**，'
+                             '后者重做"哪几天"，它重抓"哪几篇的正文"')
+    parser.add_argument('--refresh', action='store_true',
+                        help='已回填过的天也重做（默认跳过）。它决定**处理哪些天**，'
+                             '不改变正文的抓取方式——要绕过缓存重抓正文用 --no-fetch-cache')
     parser.add_argument('--max-days', type=int, default=0, help='最多处理几天（0=不限，用于试跑）')
     parser.add_argument('--dry-run', action='store_true', help='只报到要抓多少，不抓不写')
     parser.add_argument('--yes', action='store_true', help='确认抓取并写入文件')
@@ -592,16 +637,21 @@ def main():
     report = []
     for index, (day, articles, truncated) in enumerate(todo, 1):
         print('\n[%d/%d] %s — 选中 %d 篇' % (index, len(todo), day, len(articles)))
-        fetch = fetch_bodies(articles, args.workers, log=print)
-        result = write_day(articles, day, args.out, topic_filter=topics, truncated=truncated)
+        fetch = fetch_bodies(articles, args.workers, log=print,
+                             use_cache=not args.no_fetch_cache)
+        result = write_day(articles, day, args.out, topic_filter=topics, truncated=truncated,
+                            fetch_failed=fetch['failed'])
         print('    写入 %d 篇，跳过 %d 篇（正文太短），主题兜底 %d 篇'
               % (result['written'], result['skipped'], result['fallbackTopics']))
+        if fetch['failed']:
+            print('    ⚠ 有 %d 篇正文没抓到：这天不算完成，重跑会只重试这些篇'
+                  % fetch['failed'])
         if args.vault_copy:
             sync = copy_to_vault(day, args.out, args.vault_root)
             print('    入 Vault 的 Sources: %d 个 md' % sync['copied'])
         report.append({'date': day, 'inDb': len(articles), 'fetched': fetch['ok'],
-                       'written': result['written'], 'skippedThin': result['skipped'],
-                       'truncated': truncated})
+                       'fetchFailed': fetch['failed'], 'written': result['written'],
+                       'skippedThin': result['skipped'], 'truncated': truncated})
     conn.close()
 
     total_written = sum(r['written'] for r in report)