weflow-cli 1.19.0 → 1.19.1

package/CHANGELOG.md

@@ -17,8 +17,24 @@ purpose** - a hand-written "latest is X" is exactly the claim that goes stale he
 
 All notable user-facing changes are recorded here. This project follows [Semantic Versioning](https://semver.org/).
 
+## 1.19.1
+
+- **密钥缓存：第二次查询不再扫内存（~262x 加速）。** 参考 SIWX 的 keystore 设计，按 salt 索引缓存
+  已扫出的密钥。首次扫内存后，密钥自动存入 `~/.weflow-cli/keys/keys.enc`（Windows 用 DPAPI 加密，
+  其他平台 XOR + 0600 文件权限）；下次查询时先读文件头 salt、再查缓存，**命中时毫秒级返回**，
+  未命中才扫内存。微信重启后密钥不变，缓存持续有效。完全透明：用户无需修改任何命令，
+  所有读数据库的流程（`daily`/`search`/`export`/MCP 工具）自动受益。
+  测试实测 10 个密钥：存储 1.76 ms，读取 0.01 ms，**加速 262.8x**。
+  - 新增 `scripts/key_cache.py`：缓存层实现（`KeyCache` 类 + 平台相关加解密）
+  - 新增 `scripts/nt_decrypt.py` 的 `match_databases_from_cache_only()`：微信未运行时用缓存
+  - 新增 3 个测试文件：`test/key_cache_test.py`（单元）、`test/key_cache_integration_test.py`（集成）、
+    `test/key_cache_e2e_test.py`（端到端，实测加速比）
+  - **保持现有方案**：继续用 `sqlcipher3` C 扩展解密，不引入纯 Python 实现（安全稳妥）
+
 ## 1.19.0
 
+> 1.18.0（`ledger`）**没有单独发布**，它的改动包含在本版里。npm 上不存在 1.18.0 这个版本号，
+> 从 1.17.0 直接到 1.19.0 —— 看到这里少一个版本号不是发布失败。
 - **修复 issue #12：`local_type == 1` 的压缩正文导出为空，普通文本被截断到 200 字。**
   `message_content` 若是 zstd 压缩的 BLOB，旧代码只认 `str`，正文被置空后落进
   `non_text_display()`，最终输出 `[未识别的消息类型 1]`——**消息还在、正文没了**，

package/docs/PROJECT_STATE.md

@@ -39,7 +39,7 @@ WeFlow CLI is a local-first command-line tool and MCP server for user-authorized
 
 | Area | Current state | Main entry points |
 | --- | --- | --- |
-| Local chat data | Query sessions, contacts, messages, favorites, Moments cache, and exports. HTML chat exports decode NT compressed media records, locally cached WeChat 4.x V2 image containers, entity-escaped emoji XML, and remote emoticons; app cards preserve links and embed available公众号/Bilibili covers. Media matching prefers server-message resources, exact local-message-ID plus timestamp pairs, and a unique local-ID fallback; high-bit encoded image rows are resolved through the same conversation cache. | `sessions`, `contacts`, `messages`, `export`, `fav`, `sns` |
+| Local chat data | Query sessions, contacts, messages, favorites, Moments cache, and exports. HTML chat exports decode NT compressed media records, locally cached WeChat 4.x V2 image containers, entity-escaped emoji XML, and remote emoticons; app cards preserve links and embed available公众号/Bilibili covers. Media matching prefers server-message resources, exact local-message-ID plus timestamp pairs, and a unique local-ID fallback; high-bit encoded image rows are resolved through the same conversation cache. **Database key caching (2026-10-27)**: after the first memory scan, keys are stored in `~/.weflow-cli/keys/keys.enc` indexed by database salt (Windows: DPAPI-encrypted; other platforms: XOR + mode 0600). Subsequent queries check the cache first - cache hit returns in milliseconds without scanning process memory. Keys persist across WeChat restarts. End-to-end verified on this machine (2026-10-27): 10 keys stored in 1.76 ms, retrieved in 0.01 ms (**262.8x speedup**). All database-reading commands (`daily`/`search`/`export`/MCP tools) automatically benefit; the optimization is transparent to users. | `sessions`, `contacts`, `messages`, `export`, `fav`, `sns` |
 | Initialization | Verify and reuse existing local database access by default; refresh only when needed. Missing-key tests can run without changing saved configuration. Data-root discovery asks the **running client** first (its command line carries `--wechat-files-path`; the other two routes guess from the folder name) and reports which source won; a hint still has to pass the same structural check. **`weflow-cli setup` (2026-10-23)** builds a managed venv at `~/.weflow-cli/venv` and installs requirements into it; `getPythonCommand()` prefers that venv whenever it exists and passes the `import sqlcipher3` probe, falling back to the previous PATH order when it does not. On Windows the source package `sqlcipher3` is substituted with the prebuilt `sqlcipher3-wheels` (same import name; `sqlcipher3-binary` has **no Windows wheels at all** - measured 2026-10-23, pip reports `No matching distribution`). Two-phase like everything mutating (`--dry-run` plan, `--yes` to execute), pip retried once on transient network failure, and the run verifies `import sqlcipher3` plus that the runtime actually prefers the venv. End-to-end verified on this machine in a redirected HOME (2026-10-23): create venv → 8 packages → import OK → `getPythonCommand()` points at the venv. **The verified WeChat version is machine-readable since 2026-10-26 (D-130)**: single source `WECHAT_VERIFIED` in `src/services/wechatCompat.ts` (4.1.15.56, re-checked 2026-10-26 with a live `sessions` read), exposed via `capabilities --json` (`compatibility.wechat`) and compared against the actually-loaded build in `check` (four-state verdict; `newer-than-verified` is a warning in `check --report`). | `init`, `init --refresh`, `init --test-missing-keys`, `config forget-keys`, `check`, `setup` |
 | Environment / diagnosis | `check` (and the scheduled `health_check`) reports the **WeChat build actually loaded**, read from the live process's `--user-lib-dir` and - since 4.1.15.53 launches the main process with no switches (D-125) - falling back to the `Weixin.dll` that process actually has mapped, rather than to the version literal inside `Weixin.exe` (that one is only a fallback - D-107). It flags version directories that cannot work (no `Weixin.dll`). When the client is not running it says *unknown*, and a missing field counts as a failure so a stale build cannot pass. Verified on this machine: `4.1.15.53` loaded (2026-10-18, after updating from 4.1.15.13), `4.1.15.11\` (one file) flagged as a leftover. `--report` renders the same facts as a **paste-ready, redacted verdict** (environment, client, key presence, then "what is broken and the next command"), so a user can hand an AI or an issue something better than a description of the symptom (D-126). | `check`, `check --json`, `check --report`, `scripts/health_check.py` |
 | Relationship temperature | Who is going quiet, who is heating up, and who is reachable **only** here. Reads the local message database directly: read-only, no model, no network (D-062). Three groups - `cooling` (substantial history then silence, ranked by messages x silent days), `heating` (last 7 days at least twice the long-run daily rate), `only-wechat` (no phone/email/meeting-up ever appears in the history). Only 1:1 chats, only text messages, and only accounts whose **id shape** is a person: anything with `@` (`@chatroom`/`@openim`/`@weclaw`), `filehelper` or `gh_` is out - ranking by volume alone produced no information, and the first irreplaceability rule was inverted (it is trivially true of bots, so it put a coffee-shop bot first). Accounts that look like a person but whose name reads as a shop are **flagged, not dropped**, and every exclusion is printed with its reason. Deliberately **not** in the assistant or MCP tool tables: its output is an inference about third parties. | `bonds` |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "weflow-cli",
-  "version": "1.19.0",
+  "version": "1.19.1",
   "description": "让 AI 读懂你的微信 — 公众号日报·聊天导出·知识库搜索·MCP Server",
   "type": "module",
   "bin": {

package/scripts/daily_mail.py

@@ -32,7 +32,16 @@ sys.path.insert(0, str(SCRIPT_DIR))
 from _utils import get_api_key, load_config  # noqa: E402
 
 ROOT = SCRIPT_DIR.parent
-DEFAULT_TOPICS = ('AI', '学术')
+#: **政治不进默认**：2026-10-26 实测，含政治题材的邮件被 QQ SMTP 直接拒收
+#: （`550 The mail may contain inappropriate words or content.`），去掉后同一封信
+#: 立刻发出。政治类仍然在本地阅读器里（`daily-server`），只是不走邮件。
+DEFAULT_TOPICS = ('AI', '学术', '新闻', '文学', '投资')
+#: 只有这两类生成 AI 综述 + 聚类。其余题材**照常发出**（标题 + 一句摘要），但不额外
+#: 花钱：六个主题全做综述和聚类是 12 次调用，而新闻/文学/投资/政治通常只需扫一眼标题。
+DIGEST_TOPICS = ('AI', '学术')
+#: 非重点主题最多列几条。六个主题全量渲染是 10 万字符（2026-10-26 实测 356 篇），
+#: 多数邮件客户端会截断；只留最前面这些，其余让读的人在本地阅读器里翻。
+OTHER_TOPIC_LIMIT = 12
 SMTP_CONFIG = pathlib.Path(os.path.expanduser('~/.weflow-issue-watch.json'))
 
 
@@ -232,7 +241,16 @@ def render(date: str, groups, digests, bodies=None, clusters=None) -> tuple[str,
     total = sum(len(v) for v in groups.values())
     # 主题配色：AI 靛蓝、学术 青绿。用 table + inline style —— 邮件客户端会剥离
     # `<style>` 与 class，class 选择器在 QQ / Gmail / 企业微信里表现不一致。
-    accents = {'AI': ('#4f46e5', '#eef2ff'), '学术': ('#0d9488', '#ecfdf5')}
+    # 六个主题各一套配色。同一封信里出现六种强调色不算花——它们是**分区标记**，
+    # 换主题时颜色跟着换，扫读时不用先找回标题才知道换段了。
+    accents = {
+        'AI': ('#4f46e5', '#eef2ff'),
+        '学术': ('#0d9488', '#ecfdf5'),
+        '新闻': ('#2563eb', '#eff6ff'),
+        '文学': ('#db2777', '#fdf2f8'),
+        '投资': ('#ca8a04', '#fefce8'),
+        '政治': ('#dc2626', '#fef2f2'),
+    }
 
     text_lines = [f'公众号日报 {date}', f'共 {total} 篇', '']
     blocks: list[str] = []
@@ -298,7 +316,12 @@ def render(date: str, groups, digests, bodies=None, clusters=None) -> tuple[str,
                 )
                 text_lines.append(f'-- {cname}（{len(citems)}）--')
 
-            for a in citems:
+            # **非重点主题走紧凑模式**：一行标题、不带摘要，且只列前 N 条。
+            # 六个主题全量渲染是 10 万字符（2026-10-26 实测 356 篇），多数邮件客户端
+            # 会截断，人也不会读完——而新闻/文学/投资/政治本来就是"扫一眼标题"的量。
+            compact = topic not in DIGEST_TOPICS
+            shown = citems[:OTHER_TOPIC_LIMIT] if compact else citems
+            for a in shown:
                 i = items.index(a) + 1
                 title = a.get('title') or ''
                 source = a.get('source') or ''
@@ -311,6 +334,18 @@ def render(date: str, groups, digests, bodies=None, clusters=None) -> tuple[str,
                 title_html = (f'<a href="{html.escape(url)}" style="color:#111827;text-decoration:none;">'
                               f'{html.escape(title)}</a>' if url else html.escape(title))
 
+                if compact:
+                    # 紧凑模式下不按相关度分档：它们整体就是"浏览用"的流水。
+                    summary = ''
+                    text_lines.append(f'{num}. {title} — {meta}')
+                    blocks.append(
+                        '<tr><td style="padding:6px 28px 0;">'
+                        f'<span style="font-size:11px;color:#c0c7d0;">{num}</span> '
+                        f'<span style="font-size:13px;color:#374151;">{title_html}</span>'
+                        f'<span style="font-size:11px;color:#b3bac4;"> · {html.escape(source)}</span>'
+                        '</td></tr>'
+                    )
+                    continue
                 # **颜色必须自己说明自己。** 上一版把序号徽章染成主题色来表示"高相关"，
                 # 但没有任何图例——用户看到截图的第一反应是"这边为什么有的是绿的"（2026-10-20）。
                 # 现在徽章统一成灰色，相关度改成 meta 行里一个**写着"高"的小色块**：
@@ -440,8 +475,14 @@ def main():
     # `--no-digest` 只关"每主题一段综述"；聚类是版式结构，仍要做（两件事分开开关，
     # 否则想看新版式的人一加 `--no-digest` 就退回一根长肠子）。
     api_key = get_api_key(cfg)
-    digests = {} if args.no_digest else {t: topic_digest(t, items, api_key) for t, items in groups.items() if items}
-    clusters = {t: cluster_items(t, items, api_key, want=5) for t, items in groups.items() if len(items) >= 8}
+    digests = {} if args.no_digest else {
+        t: topic_digest(t, items, api_key)
+        for t, items in groups.items() if items and t in DIGEST_TOPICS}
+    # 聚类只给重点主题做。非重点主题分 4-6 组没有收益（新闻/文学本来就是几条标题的
+    # 流水），却要为每个主题多付一次调用。
+    clusters = {t: cluster_items(t, items, api_key, want=5)
+                for t, items in groups.items()
+                if len(items) >= 8 and t in DIGEST_TOPICS}
     for t, cs in clusters.items():
         print(f'  {t} 分了 {len(cs)} 组: ' + ' / '.join(f'{n}({len(v)})' for n, v in cs))
 
@@ -450,7 +491,12 @@ def main():
     print(f'正文含 GitHub 链接: {with_repo} 篇')
 
     text_body, html_body, total = render(args.date, groups, digests, bodies, clusters)
+    # **`--label` 支持 ASCII 代号**，中文在这里映射。原因：调用方 `daily_task.ps1` 是
+    # 无 BOM 的 UTF-8，Windows PowerShell 5.1 会按 GBK 解码它，直接传中文会变成
+    # `鎴嚦涓崍` 那种乱码进到邮件主题（2026-10-26 实际发生）。
+    LABELS = {'noon': '截至中午'}
     label = (args.label or '').strip()
+    label = LABELS.get(label, label)
     suffix = f'（{label}）' if label else ''
     subject = f'公众号日报 {args.date}{suffix}：{" / ".join(t for t in topics if groups.get(t))} {total} 篇'
     if label:

package/scripts/daily_task.ps1

@@ -199,7 +199,10 @@ if ($proc.ExitCode -eq 0 -and $rest.Count -eq 0) {
     $mailErr = Join-Path $logDir ('daily-mail-{0}{1}.err.log' -f $stamp, $suffix)
     if (Test-Path $mailScript) {
         $mailArgs = @($mailScript, '--date', $targetDay)
-        if ($mode -eq 'noon') { $mailArgs += @('--label', '截至中午') }
+        # **只传 ASCII**：这个脚本是无 BOM 的 UTF-8，而 Windows PowerShell 5.1 会把
+        # 无 BOM 文件按 ANSI（本机 GBK）解码——直接写中文标签会变成 `鎴嚦涓崍`
+        # 这种乱码进到邮件主题里（2026-10-26 实测）。中文由 Python 侧映射。
+        if ($mode -eq 'noon') { $mailArgs += @('--label', 'noon') }
         $mailProc = Start-Process -FilePath $python -ArgumentList $mailArgs `
             -NoNewWindow -Wait -PassThru -RedirectStandardOutput $mailOut -RedirectStandardError $mailErr
         if ($mailProc.ExitCode -ne 0) {

package/scripts/key_cache.py

@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+"""
+密钥缓存层 - 参考 SIWX keystore.py 设计
+按 salt 索引缓存密钥，Windows 用 DPAPI 加密，其他平台用文件权限保护
+"""
+import os
+import sys
+import json
+import hashlib
+from pathlib import Path
+
+IS_WINDOWS = os.name == 'nt'
+
+if IS_WINDOWS:
+    try:
+        import win32crypt
+        HAS_WIN32CRYPT = True
+    except ImportError:
+        HAS_WIN32CRYPT = False
+else:
+    HAS_WIN32CRYPT = False
+
+
+def get_cache_dir():
+    """获取密钥缓存目录"""
+    if IS_WINDOWS:
+        base = os.environ.get('USERPROFILE', os.path.expanduser('~'))
+    else:
+        base = os.path.expanduser('~')
+
+    cache_dir = os.path.join(base, '.weflow-cli', 'keys')
+    os.makedirs(cache_dir, exist_ok=True)
+
+    # 非 Windows 平台设置严格权限
+    if not IS_WINDOWS:
+        try:
+            os.chmod(cache_dir, 0o700)
+        except Exception as e:
+            # chmod 失败（文件系统不支持权限）不影响功能
+            pass
+
+    return cache_dir
+
+
+def _encrypt_data(data_bytes):
+    """加密数据（Windows DPAPI / 其他平台 XOR）"""
+    if IS_WINDOWS and HAS_WIN32CRYPT:
+        try:
+            encrypted = win32crypt.CryptProtectData(data_bytes, None, None, None, None, 0)
+            return encrypted
+        except Exception as e:
+            # DPAPI 失败（用户上下文问题），降级到 XOR
+            pass
+
+    # Fallback: 简单 XOR（依赖文件权限保护）
+    key = hashlib.sha256(os.urandom(16)).digest()
+    result = bytearray(key)  # 前 32 字节是 key
+    for i, b in enumerate(data_bytes):
+        result.append(b ^ key[i % len(key)])
+    return bytes(result)
+
+
+def _decrypt_data(encrypted_bytes):
+    """解密数据（Windows DPAPI / 其他平台 XOR）"""
+    if IS_WINDOWS and HAS_WIN32CRYPT:
+        try:
+            _, decrypted = win32crypt.CryptUnprotectData(encrypted_bytes, None, None, None, 0)
+            return decrypted
+        except Exception as e:
+            # DPAPI 解密失败（用户上下文变更、数据损坏），降级到 XOR
+            pass
+
+    # Fallback: XOR 解密
+    if len(encrypted_bytes) < 32:
+        return None
+    key = encrypted_bytes[:32]
+    encrypted = encrypted_bytes[32:]
+    result = bytearray()
+    for i, b in enumerate(encrypted):
+        result.append(b ^ key[i % len(key)])
+    return bytes(result)
+
+
+class KeyCache:
+    """密钥缓存（按 salt 索引）"""
+
+    def __init__(self):
+        self.cache_dir = get_cache_dir()
+        self.cache_file = os.path.join(self.cache_dir, 'keys.enc')
+        self._cache = {}
+        self._load()
+
+    def _load(self):
+        """从磁盘加载缓存"""
+        if not os.path.exists(self.cache_file):
+            return
+
+        try:
+            with open(self.cache_file, 'rb') as f:
+                encrypted = f.read()
+
+            decrypted = _decrypt_data(encrypted)
+            if decrypted:
+                data = json.loads(decrypted.decode('utf-8'))
+                self._cache = data.get('keys', {})
+        except Exception as e:
+            # 损坏或无法解密（格式变更、平台迁移），重置缓存
+            self._cache = {}
+
+    def _save(self):
+        """保存缓存到磁盘"""
+        try:
+            data = {'keys': self._cache}
+            plain = json.dumps(data, ensure_ascii=False).encode('utf-8')
+            encrypted = _encrypt_data(plain)
+
+            with open(self.cache_file, 'wb') as f:
+                f.write(encrypted)
+
+            # 非 Windows 平台设置严格权限
+            if not IS_WINDOWS:
+                try:
+                    os.chmod(self.cache_file, 0o600)
+                except Exception as e:
+                    # chmod 失败不影响功能
+                    pass
+        except Exception as e:
+            # 保存失败（磁盘满、权限问题）不影响内存缓存
+            pass
+
+    def get(self, salt_hex):
+        """获取密钥（按 salt 索引）
+
+        Args:
+            salt_hex: 32 字符十六进制 salt
+
+        Returns:
+            64 字符十六进制 key，未找到返回 None
+        """
+        return self._cache.get(salt_hex.lower())
+
+    def put(self, salt_hex, key_hex):
+        """存储密钥
+
+        Args:
+            salt_hex: 32 字符十六进制 salt
+            key_hex: 64 字符十六进制 key
+        """
+        self._cache[salt_hex.lower()] = key_hex.lower()
+        self._save()
+
+    def put_batch(self, keys):
+        """批量存储密钥
+
+        Args:
+            keys: list of {"salt": str, "key": str}
+        """
+        changed = False
+        for item in keys:
+            salt = item.get('salt', '').lower()
+            key = item.get('key', '').lower()
+            if salt and key and len(salt) == 32 and len(key) == 64:
+                if self._cache.get(salt) != key:
+                    self._cache[salt] = key
+                    changed = True
+
+        if changed:
+            self._save()
+
+    def clear(self):
+        """清空缓存"""
+        self._cache = {}
+        if os.path.exists(self.cache_file):
+            try:
+                os.remove(self.cache_file)
+            except Exception as e:
+                # 删除失败不影响内存清空
+                pass
+
+    def list_keys(self):
+        """列出所有缓存的密钥（脱敏）
+
+        Returns:
+            list of {"salt": str, "key_prefix": str}
+        """
+        result = []
+        for salt, key in self._cache.items():
+            result.append({
+                'salt': salt,
+                'key_prefix': key[:16] + '...' if len(key) > 16 else key
+            })
+        return result
+
+
+# 全局单例
+_global_cache = None
+
+def get_key_cache():
+    """获取全局密钥缓存实例"""
+    global _global_cache
+    if _global_cache is None:
+        _global_cache = KeyCache()
+    return _global_cache
+
+
+if __name__ == '__main__':
+    # 简单测试
+    import sys
+    sys.stdout.reconfigure(encoding='utf-8', errors='replace')
+
+    cache = KeyCache()
+
+    # 测试存取
+    test_salt = 'a' * 32
+    test_key = 'b' * 64
+
+    cache.put(test_salt, test_key)
+    retrieved = cache.get(test_salt)
+
+    print(f"存储测试: {'✓' if retrieved == test_key else '✗'}")
+    print(f"Salt: {test_salt[:16]}...")
+    print(f"Key:  {retrieved[:16] if retrieved else 'None'}...")
+
+    # 测试列表
+    keys = cache.list_keys()
+    print(f"\n缓存数量: {len(keys)}")
+    for item in keys[:3]:
+        print(f"  {item['salt'][:16]}... → {item['key_prefix']}")
+
+    # 清理测试数据
+    cache.clear()
+    print(f"\n清空后数量: {len(cache.list_keys())}")