@forsion/tangu-computer-use 0.5.8 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # 更新日志
 
+## 0.6.0 — 2026-09-27
+
+- 新增「电脑历史」采集（macOS，协议 13）：Forsion 桌面端开启该功能后，helper 通过新的 `recordSubscribe` 订阅，把前台 App 切换、窗口标题、浏览器网址（去掉查询串）、焦点输入框里新打的字、点击的控件名和 ⌘ / ⌃ 快捷键推给桌面端。只需要「辅助功能」授权，不截图、不录原始按键。helper 自己不落盘，保存、保留期、清除、暂停都由桌面端负责；最后一个订阅者断开后，所有观察者随即拆除。
+- 始终不记录：密码管理器与钥匙串、安全输入框和 Secure Input 期间的内容，以及无痕 / 私密窗口（只留一条不带标题的切换）。桌面端下发的排除 App、排除站点和只记标题的 App 在读取辅助功能数据之前就生效。agent 通过 Computer Use 操作期间产生的事件会标成 agent 所为。
+- 排除的 App 只留切到它的那一条不带标题的切换，App 里的窗口变化一概不发，连时间点和次数都不外泄。排除站点另留一条不带标题和网址的窗口标记，用来结束同一浏览器里上一个页面的计时。
+- 无痕窗口、排除的 App 或站点这类没记录的时段过后，第一条可记录的 App 切换或窗口事件带 `resumed: true`。它不含那段的任何时间或内容；桌面端和引擎折叠时间线时在这里断开，没记录的时间不会算进前后的页面。
+- 标题不变的导航也会被识别（同名的登录页换了站点、单页应用换了路由）：记录文字、点击和快捷键之前，以及一段输入停手时，helper 会按记住的网页区域或地址栏重读一次网址（1–2 次辅助功能读取，不重新遍历）。网址变了就当换了情境：还没报出的输入先按原来的页面报出，新页面是排除站点就直接丢掉。
+- 查找浏览器网址的有上限遍历先读节点角色，再决定要不要读它的子节点，绝不向网页区域（AXWebArea）要子节点，不会逼浏览器生成整棵网页的无障碍树。
+- `diagnostics` 新增无痕扫描计数 `recorderPrivateScans`、`recorderPrivateScanIncomplete`、`recorderPrivateScanStops`（只有计数，不含内容）。遍历找到网页区域后不再展开，碰到深度、节点数或时间上限也会停下，这时工具栏上的无痕标记可能没被看到；这类窗口目前仍按普通窗口记录，等真机数据再决定要不要改。
+- Safari 暂时只记不带标题的切换：Safari 私密窗口的标题里没有任何标记，在真机上找到可靠的判定办法之前，整个 Safari 按私密窗口处理（`node scripts/probe-recorder.mjs` 可用来找这个信号）。其他浏览器的无痕窗口按标题和工具栏上的无痕标记识别。
+- 设置了排除站点时，浏览器（含 Chrome / Edge 的 Beta、Dev、Canary 渠道，以及 Opera、Arc、Dia、Zen、Orion、DuckDuckGo、Firefox Developer Edition 等）的网址读不到，就按排除站点处理：只记一条不带标题的切换。
+- 点击只在那一点最上面的窗口属于前台 App 时才记录。点在 Spotlight、Raycast、通知、Dock、菜单栏图标这类不抢前台的窗口上，不会被记成前台 App 里的点击。
+- 已知限制：`recordSubscribe` 只校验连接方和 helper 是同一个 macOS 用户，不校验对端的代码签名，helper 也不知道桌面端是否开着电脑历史。只要 helper 在运行（Computer Use 和电脑历史都会拉起它），同一用户下的任何进程都能自己订阅、带上自己的策略：即使桌面端的电脑历史已关闭或暂停，这条订阅照样收到事件，桌面端设置的排除 App 和排除站点也管不到它。helper 内置的排除（密码管理器、钥匙串、安全输入框、Secure Input、无痕窗口）照常生效。同一用户的进程本来就能在同一个 socket 上调用 `getUserContext`、`axReadText`、`focusedElement`、`look` 等辅助功能命令读取屏幕上的内容，这是同一个暴露面，所以这一版接受这个风险。对端签名校验留作后续工作。
+- 新增 `npm run check:recorder-logic`（单测，已并入 `npm run check`）与 `npm run check:recorder`（起一个临时 helper 验证订阅接线，无需授权）。
+
+The macOS helper can now record Computer History for Forsion Desktop (protocol 13). While the feature is on, Desktop opens a `recordSubscribe` stream and the helper pushes app switches, window titles, browser URLs without query strings, newly typed text in the focused field, the names of clicked controls and ⌘/⌃ shortcuts. It needs only the Accessibility permission: no screenshots and no raw keystrokes. The helper stores nothing itself. Desktop owns storage, retention, clearing and pausing, and the helper removes every observer once the last subscriber disconnects. Password managers, Keychain, secure text fields, Secure Input and private browser windows are never recorded (a private window leaves only an untitled switch). Excluded apps, excluded sites and title-only apps from Desktop apply before any Accessibility data is read, and events caused by an agent's Computer Use actions are marked as such. An excluded app leaves only the untitled switch to it. Window changes inside it aren't sent at all, so not even their timing or count leaves the helper. An excluded site also leaves one window marker without a title or URL, which ends the timing of the previous page in the same browser. After a stretch that wasn't recorded (a private window, an excluded app or an excluded site), the first recordable app or window event carries `resumed: true`. It carries nothing about that stretch. Desktop and the engine break the timeline there, so the unrecorded time isn't counted toward the pages around it.
+
+Navigation that keeps the page title is caught too, such as a sign-in page with the same name on another site or a route change in a single-page app. Before it records text, a click or a shortcut, and when a stretch of typing settles, the helper reads the URL again from the web area or address field it remembered for that window. That's one or two Accessibility reads, with no new walk. A changed URL counts as a new context: pending text is reported for the page it was typed on, or dropped if the new page is an excluded site. The capped walk that looks for the browser URL now reads each node's role before deciding whether to read its children, and it never asks a web area (AXWebArea) for its children, so the browser isn't pushed into building the accessibility tree of the whole page. `diagnostics` now reports private-scan counts: `recorderPrivateScans`, `recorderPrivateScanIncomplete` and `recorderPrivateScanStops` (counts only, no content). The walk stops expanding once it finds the web area, and it also stops at its depth, node and time limits, so a private-mode marker in the toolbar can be missed. Such windows are still recorded as normal windows for now, until real-machine data settles whether that should change.
+
+Safari is recorded as untitled app switches only for now. Its private windows carry no marker in the title, so every Safari window is treated as private until a reliable signal is found on a real machine (`node scripts/probe-recorder.mjs` helps look for one). Private windows in other browsers are detected from the title and from private-mode labels in the toolbar. When sites are excluded and a browser's URL can't be read, that moment is treated as an excluded site and leaves only an untitled switch. This covers Chrome and Edge Beta, Dev and Canary as well as Opera, Arc, Dia, Zen, Orion, DuckDuckGo, Firefox Developer Edition and similar browsers. A click is recorded only when the topmost window at that point belongs to the frontmost app. Clicks on Spotlight, Raycast, notifications, the Dock or menu bar icons are not recorded as clicks in the frontmost app.
+
+Known limitation: `recordSubscribe` only checks that the client runs as the same macOS user as the helper. It doesn't check the client's code signature, and the helper doesn't know whether Computer History is on in Forsion Desktop. While the helper is running (Computer Use and Computer History both start it), any process of that user can open its own subscription with its own policy. That subscription gets events even when Computer History is off or paused in Desktop, and Desktop's excluded apps and sites don't apply to it. The helper's built-in exclusions (password managers, Keychain, secure text fields, Secure Input and private windows) still apply. Any process of that user can already call the helper's Accessibility commands on the same socket, such as `getUserContext`, `axReadText`, `focusedElement` and `look`, to read what's on screen. That's the same exposure, so this release accepts the risk. A peer signature check is future work.
+
 ## 0.5.8 — 2026-09-26
 
 - 修复 Windows 助手在没装过 VC++ 运行库的干净 Windows 上起不来：助手改为静态链接 MSVC 运行库，不再依赖 `VCRUNTIME140.dll`。发布前的包内容检查会拦下仍依赖 VC++ 运行库的 Windows 助手。

package/README.md

@@ -148,6 +148,14 @@ Build the helper with `npm run build:native` and pair it with Genesis's Mini Pan
 
 macOS helper 会向 socket 同目录发布短时前台输入信号，供 Genesis Mini Panel 跟随光标。后台 AX/PID 调用不触发；必须配套更新 Genesis 并按现有签名流程重建 helper。验证命令为 `npm run check:mini-foreground`，不会操控用户应用。
 
+### Computer History recorder / 电脑历史采集（`recordSubscribe`, protocol 13, macOS）
+
+Forsion Desktop opens a **dedicated** connection to the helper socket and sends one line: `{"id":"…","cmd":"recordSubscribe","policy":{"excludeBundleIds":[],"titleOnlyBundleIds":[],"excludeDomains":[],"text":true,"clicks":true,"keys":true}}` (every policy field is optional; a trailing `.*` in a bundle id is a prefix match). The first reply line is `{"id":"…","ok":true,"result":{"subscribed":true,"protocolVersion":13,"axTrusted":true}}`, or `ok:false` with `accessibility_denied` (that connection then stays an ordinary request connection). After an ok reply the connection only carries events, one `{"ev":{…}}` per line (`kind` = `app` / `window` / `text` / `click` / `key` / `system`; the shape lives in Genesis `desktop/shared/computerHistory.ts`). After a stretch that was not recorded (a private window, an excluded app or site), the first recordable `app` or `window` event carries `resumed: true`. It says nothing about that stretch, and consumers must not merge spans across it. The subscription ends when the client closes the socket; to change the policy, close and subscribe again. Several subscribers may be open at once, and each is filtered by its own policy. A slow reader never blocks the recorder: its backlog is capped and overflow is reported later as `{"kind":"system","state":"dropped","count":N}`. When the last subscriber leaves, the helper removes every observer and global monitor.
+
+The recorder (`native/macos/activity_recorder.swift`) watches only the frontmost app through one AXObserver on its own thread, never enables AXEnhancedUserInterface / AXManualAccessibility, and never walks whole trees; the browser URL comes from a capped walk that never enters web content. Password managers, Keychain, the helper itself, secure text fields, Secure Input and private windows are never recorded, whatever the policy says. Safari is recorded as untitled switches only, because its private windows carry no title marker and no other signal has been verified yet (`node scripts/probe-recorder.mjs` dumps candidate signals). When a subscriber excludes sites and a browser's URL can't be read, that moment counts as an excluded site. An excluded app leaves only the untitled switch to it (no window events inside it); an excluded site also leaves one window marker without a title or URL. Before text, clicks and shortcuts are recorded, the helper re-reads the browser URL through the web area or address field it remembered for that window, so a navigation that keeps the title is still treated as a new context. The URL walk reads a node's role before its children and never asks a web area for its children. `diagnostics` reports private-scan counts (`recorderPrivateScans`, `recorderPrivateScanIncomplete`, `recorderPrivateScanStops`). A click is recorded only when the topmost window at that point belongs to the frontmost app. The helper writes nothing to disk. Only the peer's user is checked, not its code signature: while the helper runs, any process of the same user can subscribe with its own policy, even when Computer History is off or paused in Desktop and regardless of Desktop's exclusions (the built-in exclusions still apply). That is the same exposure as the helper's existing Accessibility commands such as `getUserContext` and `axReadText`. `npm run check:recorder-logic` runs the pure-logic tests; `npm run check:recorder` starts a throwaway helper from `prebuilt/` on a temporary socket and checks the subscription wiring without any grant.
+
+Forsion 桌面端开启「电脑历史」后，用一条**专用**连接发 `recordSubscribe`：首行是回包，之后这条连接只推事件（一行一条 `{"ev":{…}}`），客户端关闭即退订；改策略就断开重订。无痕窗口、排除的 App 或站点这类没记录的时段之后，第一条可记录的 `app` / `window` 事件带 `resumed: true`：它不含那段的任何时间或内容，消费方折叠时不能跨过它合并。多个订阅者可并存、各按自己的策略过滤；读得慢的订阅者只会丢事件并在之后收到 `dropped` 计数，不会拖住采集。最后一个订阅者断开后，所有观察者随即拆除。helper 不落盘；密码管理器、钥匙串、安全输入框、Secure Input 与无痕窗口在任何策略下都不记录。Safari 私密窗口的标题没有标记、别的信号也还没验证过，所以 Safari 暂时只记不带标题的切换（`node scripts/probe-recorder.mjs` 可导出候选信号）。设了排除站点而浏览器网址读不到时，按排除站点处理；排除的 App 只留切到它的那一条不带标题的切换（App 里的窗口变化不发），排除站点另留一条不带标题和网址的窗口标记。记录文字、点击和快捷键之前，helper 会按记住的网页区域或地址栏重读一次网址，所以标题不变的导航也算换了情境。查网址的遍历先读角色再读子节点，绝不向网页区域要子节点；`diagnostics` 报无痕扫描计数（`recorderPrivateScans`、`recorderPrivateScanIncomplete`、`recorderPrivateScanStops`）。点击只在那一点最上面的窗口属于前台 App 时记录。只校验对端是同一用户、不校验代码签名：helper 在运行时，同一用户的任何进程都能带自己的策略订阅，桌面端关闭或暂停电脑历史、设置排除都管不到它（helper 内置的排除照常生效）。这和 `getUserContext`、`axReadText` 等现有辅助功能命令是同一个暴露面。验证：`npm run check:recorder-logic`（纯逻辑）与 `npm run check:recorder`（临时 helper 接线，无需授权）。
+
 ### macOS installation and signing / 安装与签名
 
 Run `node scripts/build-native.mjs --arch all` before packaging, then `node scripts/verify-macos-bundles.mjs` and `npm run check:installer`. Both architectures must include `tangu-computer-use.app.zip` and its JSON checksum manifest. ZIP transport preserves the helper signature through Electron's recursive signing. Runtime setup only verifies and copies these archives; missing or damaged artifacts fail without trying a user's signing identity or creating keys. `setup-helper.mjs --check` is read-only (exit 0: current, 10: installation/repair needed, 1: package error).

package/UPSTREAM.md

@@ -114,7 +114,7 @@ Accessibility/Screen-Recording (TCC) grants to them.
 > stays put, everything else goes to `~/Applications` (no admin needed). Users whose `/Applications`
 > is not writable will be asked to re-grant TCC once, because the path moved.
 
-**Helper protocol version — bumped by us.** Upstream v0.5.1 is still at 6; we run **12**, in two places that must
+**Helper protocol version — bumped by us.** Upstream v0.5.1 is still at 6; we run **13**, in two places that must
 always match: `native/macos/bridge.swift` (`private let protocolVersion`) and
 `src/vendor/platform/macos/helper.ts` (`HELPER_PROTOCOL_VERSION`). Bump it whenever the helper gains a
 command **or changes native behaviour** — `src/onboarding.ts` now notices a stale binary *on disk*, but a
@@ -274,4 +274,5 @@ button; `install.sh dev|prod` stays as the developer loop for iterating on the b
    it walks every window shape and asserts canvas-aspect / fill / snug / no-overflow. The DOM shim in
    `check.mjs` does no layout, so CSS regressions are only visible there.
 
+- Computer History recorder (protocol 13, 0.6.0): `native/macos/activity_recorder.swift` + `activity_recorder_tests.swift` (`check:recorder-logic`), listed in `macosSourcePaths` before `bridge.swift`. Three `bridge.swift` hooks to replay after a resync: `processClient` hands a `recordSubscribe` line to `subscribeRecorder()` and, once subscribed, only waits for EOF and then calls `ActivityRecorder.shared.unsubscribe`; `handleRequest` wraps `act` / `actBatch` / `focusWindow` / `restoreUserFocus` in `agentEnter()` / `agentLeave()` (events become `origin:"agent"`) and rejects `recordSubscribe` on stdin; `diagnostics` reports `recorderSubscribers`. `grep -n "ActivityRecorder" native/macos/bridge.swift` must find them. The recorder never calls `ensureEnhancedAccessibility`.
 - Mini Panel foreground activity: `native/macos/foreground_activity.swift` emits a bounded, data-only lease beside the daemon socket. Hooked into actual HID/focus delivery and recursive input scopes, not the requested delivery policy. Included in both native-build and source-install inputs. No change to upstream tool schemas or input behavior.

package/manifest.json

@@ -2,7 +2,7 @@
   "id": "tangu-computer-use",
   "name": "电脑操作",
   "nameEn": "Computer Use",
-  "version": "0.5.8",
+  "version": "0.6.0",
   "apiVersion": 1,
   "minAppVersion": "2.11.2",
   "description": "让 agent 观察并操作桌面应用:看屏幕、点按、输入、滚动(macOS / Windows / Linux)。捆绑包内含引擎侧的 12 个工具、配套技能,以及一个「被操控的窗口」实时画面视图 —— agent 动手时,被操控的那个窗口会亮起边缘光效,画面同步出现在 Agent Desk 上。",