@forsion/tangu-computer-use 0.6.1 → 0.6.2

package/CHANGELOG.md

@@ -1,5 +1,32 @@
 # 更新日志
 
+## 0.6.2 — 2026-10-06
+
+- 新增「侧边拼接」（macOS，协议 14）：在 Forsion 命令面板里运行「把对话贴到应用旁边」，从屏上的窗口里挑一个，对话面板就贴在它旁边。拖动、移动、缩放目标窗口时面板跟着走；拖面板或拉它贴着目标的那条边，目标窗口也跟着动。目标最小化、隐藏、去了别的桌面或全屏时面板藏起，回来再贴上；目标关掉时面板一并收起。
+- 面板里是一段新对话，每条消息默认带上这个 App 的引用（可以去掉）。agent 据此把它当作默认的操作对象，用电脑操作的工具直接看、直接操作它。点进面板时，目标 App 里划选的文字（没有文字选区时是列表里选中的项）自动挂成引用；密码框一律不读。
+- helper 新增三条只读命令：`dockCandidates`（屏上可贴靠的窗口，前→后）、`dockProbe`（一个窗口此刻的位置与前台 App，只读 CGWindowList、不碰辅助功能，桌面端约 60Hz 轮询）、`selection`（焦点元素的划线文本或选中项）；`setWindowFrame` 的宽高改为可省（只挪不缩放）。
+- 已知限制：跨进程的窗口没法真正挂在一起，面板的层级靠「目标 App 到前台时把面板提上来」近似；Windows 暂不支持（那边的 helper 桌面端连不上）。
+
+Docked chat panel (macOS, protocol 14). Run "Dock a chat beside an app" from the Forsion command palette and pick a window on screen. The chat panel docks beside it and follows when you drag, move or resize that window. Dragging the panel, or the panel edge that touches the window, moves or resizes the window too. The panel hides while the window is minimized, hidden, on another desktop or in full screen, and docks again when it comes back. Closing the window closes the panel. The panel holds a new chat whose messages reference that app by default (you can remove the reference), so the agent treats it as the default target and can see and operate it with the Computer Use tools. When you click into the panel, the text you selected in that app, or the items selected in a list when there's no text selection, is attached as a quote. Password fields are never read. The helper adds three read-only commands: `dockCandidates` (windows on screen you can dock beside, front to back), `dockProbe` (one window's position and the frontmost app, read from CGWindowList without Accessibility, polled by Desktop at about 60Hz) and `selection` (selected text or selected items of the focused element). `setWindowFrame` now accepts a missing width and height, which moves the window without resizing it. Known limits: windows of different apps can't truly be attached, so the panel's stacking is approximated by raising it whenever the target app comes to the front. Windows isn't supported yet, because Desktop can't reach the helper there.
+
+- 目标窗口是 Forsion 自己（拉起引擎的那个桌面应用）时，`find_roots` / `observe_ui` / `act_ui` 的结果末尾附一行 `[forsion]` 提示，让模型改用界面命令（`list_ui_commands` → `run_ui_command` / `set_ui_setting`）。只提示不拦截：没有对应界面命令的地方仍可操作。技能说明同步加了这一条。
+- `npm run check:no-foreground` 增加这条提示的 9 项判据。
+
+When the target window belongs to Forsion itself (the desktop app that launched the engine), `find_roots`, `observe_ui` and `act_ui` results now end with a `[forsion]` line pointing the model to the UI tools (`list_ui_commands`, then `run_ui_command` / `set_ui_setting`). It is a hint, not a block: controls without a UI command can still be driven. The skill text carries the same rule.
+
+- 「电脑历史」采集支持 Windows（协议仍是 13，与 macOS 同一套 `recordSubscribe` 订阅格式）。新增 `windows-bridge.exe serve --pipe <名字>` 常驻模式：采集器跑在独立进程里，监听只有当前用户和 SYSTEM 能连、拒绝远程客户端的命名管道，最后一个订阅者断开 60 秒后自行退出；`windows-bridge.exe recorder-protocol` 打印协议号。原有的 stdio 助手和它的协议版本不变。
+- 记录内容与 macOS 相同：前台 App 切换（exe 名 + 文件说明里的显示名）、窗口标题、浏览器地址栏网址（去掉查询串）、焦点可编辑输入框里新打的字（UI Automation 的 Value / Text 模式）、点击的控件类型和按钮名、Ctrl / Win 组合快捷键。不截图，不录原始按键；键盘钩子只用来判断「有没有在打字」和识别组合键。锁屏、睡眠期间暂停。
+- 始终不记录：密码框（UI Automation 的 IsPassword，报出前再复核一次）、密码管理器与 Windows 凭据界面、无痕 / InPrivate 窗口。Edge 与 Firefox 系的无痕窗口按标题标记识别；Chrome 等 Chromium 系浏览器在 Windows 上无痕窗口的标题没有标记，只能看工具栏上的无痕按钮，遍历没走完时这个窗口先按无痕处理（只留一条不带标题的切换），之后自动重走，最多 5 次。
+- agent 通过 Computer Use 操作（`act`、`actBatch`、`focusWindow`、`openBrowserLocation`）期间及之后 750 毫秒，stdio 助手持有命名互斥体 `Local\tangu-computer-use-agent-active`，采集器据此把这段时间的事件标成 agent 所为。
+- 已知限制：与 macOS 相同，管道只校验同一 Windows 用户，不校验对端身份。第一次读某个 Chromium 窗口时，Chrome 会在那一刻打开自己的无障碍支持，这一次可能要 1 秒左右。
+- 新增 `scripts/probe-recorder-windows.mjs` 与 CI 工作流 `windows-recorder-probe`：在 Windows 真桌面上操作记事本、Chrome、Edge，核对事件与隐私边界。
+
+Computer History recording now works on Windows too, with the same `recordSubscribe` format and protocol 13 as macOS. The new `windows-bridge.exe serve --pipe <name>` mode runs the recorder as its own resident process. It listens on a named pipe that only the current user and SYSTEM can open, rejects remote clients, and exits 60 seconds after its last subscriber disconnects. `windows-bridge.exe recorder-protocol` prints the protocol number. The stdio helper and its protocol version are unchanged. The recorder captures the same things as on macOS: app switches (exe name plus the display name from the file description), window titles, browser URLs from the address bar without query strings, newly typed text in the focused editable field (UI Automation Value or Text pattern), the type and label of clicked controls, and Ctrl and Win shortcuts. It takes no screenshots and records no raw keystrokes. The keyboard hook only tells whether someone is typing and recognizes key combinations. Recording pauses while the session is locked or asleep.
+
+Password fields (UI Automation IsPassword, checked again before text is reported), password managers, Windows credential prompts and private or InPrivate windows are never recorded. Private windows in Edge and Firefox-based browsers are recognized by their title. On Windows, private windows in Chrome and other Chromium browsers have no marker in the title, so the recorder looks for the private-mode button in the toolbar. Until that walk finishes, the window is treated as private and leaves only an untitled switch, and the walk is retried automatically up to 5 times. While an agent acts through Computer Use (`act`, `actBatch`, `focusWindow`, `openBrowserLocation`) and for 750 ms afterwards, the stdio helper holds the named mutex `Local\tangu-computer-use-agent-active`, and the recorder marks events from that time as agent actions.
+
+Known limitations: as on macOS, the pipe only checks that the client runs as the same Windows user. It doesn't check who the client is. The first time the recorder reads a Chromium window, Chrome turns on its accessibility support, which can take about a second once. The new `scripts/probe-recorder-windows.mjs` and the `windows-recorder-probe` CI workflow drive Notepad, Chrome and Edge on a real Windows desktop and check the events and privacy limits.
+
 ## 0.6.1 — 2026-10-02
 
 - 修复：在 Forsion 里停用再启用 Computer Use 插件（引擎不重启的热插拔）时，每次启用都会多挂一组进程退出监听，开关拨多了会报 `MaxListenersExceededWarning`，退出时收尾也会重复执行。现在停用会摘掉这组监听，再启用也只挂一组。

package/UPSTREAM.md

@@ -114,7 +114,7 @@ Accessibility/Screen-Recording (TCC) grants to them.
 > stays put, everything else goes to `~/Applications` (no admin needed). Users whose `/Applications`
 > is not writable will be asked to re-grant TCC once, because the path moved.
 
-**Helper protocol version — bumped by us.** Upstream v0.5.1 is still at 6; we run **13**, in two places that must
+**Helper protocol version — bumped by us.** Upstream v0.5.1 is still at 6; we run **14**, in two places that must
 always match: `native/macos/bridge.swift` (`private let protocolVersion`) and
 `src/vendor/platform/macos/helper.ts` (`HELPER_PROTOCOL_VERSION`). Bump it whenever the helper gains a
 command **or changes native behaviour** — `src/onboarding.ts` now notices a stale binary *on disk*, but a
@@ -275,4 +275,6 @@ button; `install.sh dev|prod` stays as the developer loop for iterating on the b
    `check.mjs` does no layout, so CSS regressions are only visible there.
 
 - Computer History recorder (protocol 13, 0.6.0): `native/macos/activity_recorder.swift` + `activity_recorder_tests.swift` (`check:recorder-logic`), listed in `macosSourcePaths` before `bridge.swift`. Three `bridge.swift` hooks to replay after a resync: `processClient` hands a `recordSubscribe` line to `subscribeRecorder()` and, once subscribed, only waits for EOF and then calls `ActivityRecorder.shared.unsubscribe`; `handleRequest` wraps `act` / `actBatch` / `focusWindow` / `restoreUserFocus` in `agentEnter()` / `agentLeave()` (events become `origin:"agent"`) and rejects `recordSubscribe` on stdin; `diagnostics` reports `recorderSubscribers`. `grep -n "ActivityRecorder" native/macos/bridge.swift` must find them. The recorder never calls `ensureEnhancedAccessibility`.
+- Computer History recorder, **Windows** (recordSubscribe contract 13): `native/windows/bridge-rs/src/recorder.rs` (pipe server, hook thread, UIA thread — `#[cfg(windows)]`), `recorder_logic.rs` (platform-neutral, tests in `tests/recorder_logic_tests.rs`, run anywhere with `cargo test`) and `agent_marker.rs`. Replay after a resync: `lib.rs` declares the three modules; `main.rs` branches to `recorder-protocol` / `serve --pipe` **before** `window::start_root_event_journal()` (serve mode must not start the journal or the stdio loop) and wraps `act` / `actBatch` / `focusWindow` / `openBrowserLocation` in `agent_marker::enter()` (a session-local named mutex, held 750ms past the command; the serve process probes it to mark `origin:"agent"` — actor and recorder are different processes on Windows). `Cargo.toml` adds the `implement` feature + a direct `windows-core` dependency (the `#[implement]` macro expands to `windows_core::…`) and the Security / Pipes / IO / RemoteDesktop / StationsAndDesktops / LibraryLoader / Storage features. `grep -n "agent_marker\|serve_recorder" native/windows/bridge-rs/src/main.rs` must find them. The **stdio** protocol number (`protocol::PROTOCOL_VERSION`, vendor TS checks it with `!==`) is untouched; `RECORDER_PROTOCOL` is a separate constant. Desktop runs serve from its own hashed copy of the exe (never the `~/.pi/...` path setup-helper overwrites). Real-machine check: `.github/workflows/windows-recorder-probe.yml` (push `ci/windows-recorder*`) runs `scripts/probe-recorder-windows.mjs`.
+- App side panel (protocol 14, 0.6.2): three read-only commands in `bridge.swift` under `MARK: - Tangu:App 侧边拼接` — `dockCandidates` (on-screen layer-0 windows of `.regular` apps, front to back), `dockProbe` (one window's live CGWindowList bounds + `frontPid`, no AX, polled ~60Hz by the desktop) and `selection` (focused element's selected text, walking up to the AXWebArea and falling back to `AXSelectedTextMarkerRange`; else selected rows/children; secure fields — and secure descendants of selected rows — never read; with `windowId` it reads only when that window is the focused one). Plus `setWindowFrame` now treats `width`/`height` as optional (move without resize). Dispatch cases sit next to `getMousePosition`.
 - Mini Panel foreground activity: `native/macos/foreground_activity.swift` emits a bounded, data-only lease beside the daemon socket. Hooked into actual HID/focus delivery and recursive input scopes, not the requested delivery policy. Included in both native-build and source-install inputs. No change to upstream tool schemas or input behavior.

package/main.js

@@ -534,6 +534,21 @@ ctx.registerView({
   },
 })
 
+// ── 侧边拼接:把一扇对话面板贴在某个 App 窗口旁边,拖动 / 移动 / 缩放都像一个窗口 ──────────────
+// 面板、跟随、读划线都在宿主(electron/appDock.ts,经本包 helper 的 dockProbe / selection 命令);这里只是入口。
+// 面板打开后先列出屏上的窗口让用户挑,所以命令本身不带目标。老宿主没有这条 IPC、或不是 macOS(跟随靠 helper 的
+// CGWindowList / AX,宿主那边直接回 unsupported_platform)→ 不注册,免得给出一个点了没反应的命令。
+const dockOpen = globalThis.window && window.tangu && window.tangu.platform === 'darwin' && window.tangu.appDockOpen
+if (dockOpen) {
+  const en = () => (ctx.getLocale ? ctx.getLocale() : 'zh') === 'en'
+  ctx.registerCommand({
+    id: 'dock-chat',
+    title: () => (en() ? 'Dock a chat beside an app' : '把对话贴到应用旁边'),
+    keywords: 'side panel dock attach app window chat 侧边 贴边 拼接 应用 窗口 对话 tiebian pinjie',
+    run: () => { void dockOpen() },
+  })
+}
+
 // 纯函数暴露给 check.mjs(宿主装载时 ctx 存在,check 里只取这个挂点)。
 globalThis.__CU_LIVE_TEST__ = {
   explain, ago, describe, PLUGIN_ID, VIEW_ID, TICK_MS, IDLE_TICK_MS,

package/manifest.json

@@ -2,11 +2,11 @@
   "id": "tangu-computer-use",
   "name": "电脑操作",
   "nameEn": "Computer Use",
-  "version": "0.6.1",
+  "version": "0.6.2",
   "apiVersion": 1,
   "minAppVersion": "2.11.2",
-  "description": "让 agent 观察并操作桌面应用:看屏幕、点按、输入、滚动(macOS / Windows / Linux)。捆绑包内含引擎侧的 12 个工具、配套技能,以及一个「被操控的窗口」实时画面视图 —— agent 动手时,被操控的那个窗口会亮起边缘光效,画面同步出现在 Agent Desk 上。",
-  "descriptionEn": "Let the agent observe and control desktop apps: see the screen, click, type and scroll (macOS / Windows / Linux). The bundle includes 12 engine tools, a companion skill and a live view of the window being controlled. While the agent works, that window gets a glowing edge and its picture appears on Agent Desk.",
+  "description": "让 agent 观察并操作桌面应用:看屏幕、点按、输入、滚动(macOS / Windows / Linux)。捆绑包内含引擎侧的 12 个工具、配套技能,一个「被操控的窗口」实时画面视图,以及「把对话贴到应用旁边」的侧边面板(macOS)—— agent 动手时,被操控的那个窗口会亮起边缘光效,画面同步出现在 Agent Desk 上。",
+  "descriptionEn": "Let the agent observe and control desktop apps: see the screen, click, type and scroll (macOS / Windows / Linux). The bundle includes 12 engine tools, a companion skill, a live view of the window being controlled and a chat panel you can dock beside any app (macOS). While the agent works, that window gets a glowing edge and its picture appears on Agent Desk.",
   "main": "main.js",
   "onboarding": {
     "intro": "工具、技能与实时画面视图都在这个包里,装一次就位。本机 helper 随包自带、首次使用时自动装好;你只需要在 macOS 系统设置里授予「辅助功能」与「屏幕录制」——授权给的是那个 helper,不是 Forsion 主程序。",

package/native/macos/bridge.swift

@@ -369,7 +369,7 @@ final class InputSuppressionGuard {
 }
 
 final class Bridge {
-	private let protocolVersion = 13
+	private let protocolVersion = 14
 	private let refStore = AXRefStore()
 	private let inputSuppressionGuard = InputSuppressionGuard()
 	private let physicalInputLock = NSRecursiveLock()
@@ -791,6 +791,12 @@ final class Bridge {
 			return try axReadText(request)
 		case "getMousePosition":
 			return getMousePosition()
+		case "dockCandidates":
+			return dockCandidates(request)
+		case "dockProbe":
+			return try dockProbe(request)
+		case "selection":
+			return try selection(request)
 		case "recordSubscribe":
 			// 只在 serve socket 的专用连接上有意义(processClient 截走了那条路);stdin 模式没有能长开的连接。
 			throw BridgeFailure(message: "recordSubscribe needs a dedicated connection to the serve socket", code: "unsupported")
@@ -1271,20 +1277,29 @@ final class Bridge {
 		}
 		let x = try doubleArg(request, "x")
 		let y = try doubleArg(request, "y")
-		let width = max(100.0, try doubleArg(request, "width"))
-		let height = max(80.0, try doubleArg(request, "height"))
+		// Tangu(协议 14):width/height 可一起省 = 只挪不缩放。贴边面板拖动时每帧都在挪目标窗口,
+		// 同尺寸再写一遍 kAXSize 会让一些 App 每帧重排一次。参数全部校验完才动窗口:报错了就不该已经挪过。
+		var size: CGSize?
+		if request["width"] != nil || request["height"] != nil {
+			size = CGSize(width: max(100.0, try doubleArg(request, "width")), height: max(80.0, try doubleArg(request, "height")))
+		}
 		var origin = CGPoint(x: x, y: y)
-		var size = CGSize(width: width, height: height)
-		guard let originValue = AXValueCreate(.cgPoint, &origin), let sizeValue = AXValueCreate(.cgSize, &size) else {
+		guard let originValue = AXValueCreate(.cgPoint, &origin) else {
 			throw BridgeFailure(message: "Failed to create AX frame values", code: "frame_value_failed")
 		}
 		let positionStatus = AXUIElementSetAttributeValue(window, kAXPositionAttribute as CFString, originValue)
-		let sizeStatus = AXUIElementSetAttributeValue(window, kAXSizeAttribute as CFString, sizeValue)
+		var sizeStatus: AXError?
+		if var size {
+			guard let sizeValue = AXValueCreate(.cgSize, &size) else {
+				throw BridgeFailure(message: "Failed to create AX frame values", code: "frame_value_failed")
+			}
+			sizeStatus = AXUIElementSetAttributeValue(window, kAXSizeAttribute as CFString, sizeValue)
+		}
 		let frame = frameForWindow(window)
 		return [
 			"ok": positionStatus == .success || sizeStatus == .success,
 			"positionStatus": Int(positionStatus.rawValue),
-			"sizeStatus": Int(sizeStatus.rawValue),
+			"sizeStatus": sizeStatus.map { Int($0.rawValue) } ?? NSNull(),
 			"framePoints": ["x": frame.origin.x, "y": frame.origin.y, "w": frame.width, "h": frame.height],
 		]
 	}
@@ -3038,6 +3053,155 @@ final class Bridge {
 		return ["x": position.x, "y": position.y]
 	}
 
+	// MARK: - Tangu:App 侧边拼接(协议 14)
+	// Forsion 的对话面板贴在某个 App 窗口旁边、像一个窗口那样一起动。跟随走 CGWindowList —— WindowServer 的
+	// 实时 bounds,用户拖窗口的过程中也在变(AX 的 kAXPosition 不保证拖动中更新);一次 ~0.1ms、零 AX 往返,
+	// 桌面端按 60Hz 轮询 dockProbe。反方向(拖面板带着目标走)复用 setWindowFrame。
+
+	/// 可贴靠的窗口,前→后(z 序:最上面那个多半就是用户刚在用的)。只收普通 App(.regular)的屏上 layer-0 窗口;
+	/// excludePids = 调用方自己(Forsion 的窗口不该出现在候选里)。
+	private func dockCandidates(_ request: [String: Any]) -> [String: Any] {
+		let excluded = Set(((request["excludePids"] as? [Any]) ?? []).compactMap { ($0 as? NSNumber)?.int32Value } + [getpid()])
+		var regular: [Int32: NSRunningApplication] = [:]
+		for app in NSWorkspace.shared.runningApplications where app.activationPolicy == .regular {
+			regular[app.processIdentifier] = app
+		}
+		let entries = (CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]]) ?? []
+		var windows: [[String: Any]] = []
+		for entry in entries {
+			guard let pid = (entry[kCGWindowOwnerPID as String] as? NSNumber)?.int32Value,
+				!excluded.contains(pid), let app = regular[pid],
+				((entry[kCGWindowLayer as String] as? NSNumber)?.intValue ?? 0) == 0,
+				let windowId = (entry[kCGWindowNumber as String] as? NSNumber)?.uint32Value,
+				let boundsDict = entry[kCGWindowBounds as String] as? [String: Any],
+				let bounds = CGRect(dictionaryRepresentation: boundsDict as CFDictionary),
+				bounds.width >= 200, bounds.height >= 150
+			else { continue }
+			// kCGWindowName 要屏幕录制权限;没有就退到 AX 标题(只要辅助功能)。
+			var title = (entry[kCGWindowName as String] as? String) ?? ""
+			if title.isEmpty, let window = windowElement(pid: pid, windowId: windowId) {
+				title = stringAttribute(window, attribute: kAXTitleAttribute as CFString) ?? ""
+			}
+			var item: [String: Any] = [
+				"pid": Int(pid), "windowId": Int(windowId), "app": app.localizedName ?? processName(pid: pid) ?? "", "title": title,
+				"x": bounds.origin.x, "y": bounds.origin.y, "w": bounds.width, "h": bounds.height,
+			]
+			if let bundleId = app.bundleIdentifier { item["bundleId"] = bundleId }
+			windows.append(item)
+			if windows.count == 24 { break }
+		}
+		return ["windows": windows]
+	}
+
+	/// 一个窗口此刻的位置/在不在屏上 + 前台 App 的 pid。只读 CGWindowList,不碰 AX。
+	/// onScreen=false:最小化、App 被隐藏、在别的桌面(Space)。exists=false:窗口已关。
+	private func dockProbe(_ request: [String: Any]) throws -> [String: Any] {
+		let windowId = UInt32(try intArg(request, "windowId"))
+		let frontPid = Int(NSWorkspace.shared.frontmostApplication?.processIdentifier ?? 0)
+		// ⚠️别用 CGWindowListCreateDescriptionFromArray:对别的进程的窗口恒返回空数组(07-29 实测)。
+		// ⚠️.optionIncludingWindow 单用只回**屏上**的窗口:最小化 / App 被隐藏 / 在别的桌面时回空(10-02 实测),
+		// 拿空当「窗口关了」会把贴边面板一并关掉。回空时再到全量表里找一次(几 ms,只在窗口不在屏上时走;桌面端此时也降频)。
+		let match = { (list: [[String: Any]]?) in list?.first(where: { ($0[kCGWindowNumber as String] as? NSNumber)?.uint32Value == windowId }) }
+		guard let entry = match(CGWindowListCopyWindowInfo([.optionIncludingWindow], windowId) as? [[String: Any]])
+			?? match(CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]]),
+			let boundsDict = entry[kCGWindowBounds as String] as? [String: Any],
+			let bounds = CGRect(dictionaryRepresentation: boundsDict as CFDictionary)
+		else {
+			return ["exists": false, "frontPid": frontPid]
+		}
+		return [
+			"exists": true,
+			"onScreen": (entry[kCGWindowIsOnscreen as String] as? NSNumber)?.boolValue ?? false,
+			"x": bounds.origin.x, "y": bounds.origin.y, "w": bounds.width, "h": bounds.height,
+			"frontPid": frontPid,
+		]
+	}
+
+	/// 目标 App 此刻的划线文本,没有就给选中项(列表/表格里选中的行:Finder 里的文件、邮件列表里的邮件)。只读。
+	/// App 失去前台后焦点元素与选区仍在,所以用户点进 Forsion 面板那一刻再读也来得及。密码框一律不读。
+	private func selection(_ request: [String: Any]) throws -> [String: Any] {
+		let pid = Int32(try intArg(request, "pid"))
+		ensureEnhancedAccessibility(pid: pid)
+		let app = AXUIElementCreateApplication(pid)
+		AXUIElementSetMessagingTimeout(app, 1.0)
+		var result: [String: Any] = [:]
+		let focusedWindow = copyAttribute(app, attribute: kAXFocusedWindowAttribute as CFString).flatMap(asAXElement)
+		// 给了 windowId(贴边面板总会给)就只读那一扇窗:同一 App 的另一扇窗里的选区不属于这段对话。
+		// 对不上(或配不出 AX 窗口)一律不读 —— 宁可漏引用,不可张冠李戴。
+		if let windowId = optionalIntArg(request, "windowId").map({ UInt32($0) }) {
+			guard let expected = windowElement(pid: pid, windowId: windowId), let focusedWindow, CFEqual(expected, focusedWindow) else {
+				return ["otherWindow": true]
+			}
+		}
+		if let focusedWindow {
+			result["windowTitle"] = stringAttribute(focusedWindow, attribute: kAXTitleAttribute as CFString) ?? ""
+		}
+		guard let element = copyAttribute(app, attribute: kAXFocusedUIElementAttribute as CFString).flatMap(asAXElement) else {
+			return result
+		}
+		let role = stringAttribute(element, attribute: kAXRoleAttribute as CFString) ?? ""
+		let subrole = stringAttribute(element, attribute: kAXSubroleAttribute as CFString) ?? ""
+		if isSecureTextElement(role: role, subrole: subrole) {
+			result["element"] = ["role": role, "secure": true]
+			return result
+		}
+		result["element"] = [
+			"role": role,
+			"subrole": subrole,
+			"title": stringAttribute(element, attribute: kAXTitleAttribute as CFString) ?? "",
+			"description": stringAttribute(element, attribute: kAXDescriptionAttribute as CFString) ?? "",
+			"value": String(displayValue(element, role: role, subrole: subrole).prefix(200)),
+		]
+		if let text = selectedText(around: element), !text.isEmpty {
+			result["text"] = String(text.prefix(20_000))
+			return result
+		}
+		let rows = axElementArrayIfPresent(element, attribute: kAXSelectedRowsAttribute as CFString)
+			?? axElementArrayIfPresent(element, attribute: kAXSelectedChildrenAttribute as CFString) ?? []
+		let items = rows.prefix(20).map(describeSelected).filter { !$0.isEmpty }
+		if !items.isEmpty { result["items"] = Array(items) }
+		return result
+	}
+
+	/// 焦点元素自己没有选区时往上找(网页里焦点常落在链接/段落上,选区挂在 AXWebArea)。
+	/// 原生控件给 kAXSelectedText;WebKit/Chromium 的正文选区只认 text marker。
+	private func selectedText(around start: AXUIElement) -> String? {
+		var current: AXUIElement? = start
+		for _ in 0..<8 {
+			guard let element = current else { return nil }
+			if let text = stringAttribute(element, attribute: kAXSelectedTextAttribute as CFString), !text.isEmpty { return text }
+			if let range = copyAttribute(element, attribute: "AXSelectedTextMarkerRange" as CFString) {
+				var value: AnyObject?
+				if AXUIElementCopyParameterizedAttributeValue(element, "AXStringForTextMarkerRange" as CFString, range, &value) == .success,
+					let text = value as? String, !text.isEmpty
+				{
+					return text
+				}
+			}
+			current = copyAttribute(element, attribute: kAXParentAttribute as CFString).flatMap(asAXElement)
+		}
+		return nil
+	}
+
+	/// 选中行 → 一行人话:自己的标题/值/描述,没有就拼前几个子孙里的文字(表格行的文字在单元格里)。
+	/// 密码框一律跳过(行本身、以及行里的每个子孙):第三方 App 可能把明文挂在 AX value 上。
+	private func describeSelected(_ element: AXUIElement) -> String {
+		let isSecure = { (el: AXUIElement) in
+			self.isSecureTextElement(role: self.stringAttribute(el, attribute: kAXRoleAttribute as CFString) ?? "",
+				subrole: self.stringAttribute(el, attribute: kAXSubroleAttribute as CFString) ?? "")
+		}
+		if isSecure(element) { return "" }
+		for attribute in [kAXTitleAttribute, kAXValueAttribute, kAXDescriptionAttribute] {
+			if let text = stringAttribute(element, attribute: attribute as CFString), !text.isEmpty { return String(text.prefix(200)) }
+		}
+		let texts = collectDescendants(startingAt: element, maxDepth: 3).compactMap { child -> String? in
+			let role = stringAttribute(child, attribute: kAXRoleAttribute as CFString) ?? ""
+			guard role == "AXStaticText" || role == "AXTextField", !isSecure(child) else { return nil }
+			return stringAttribute(child, attribute: kAXValueAttribute as CFString).flatMap { $0.isEmpty ? nil : $0 }
+		}
+		return String(texts.prefix(4).joined(separator: " · ").prefix(200))
+	}
+
 	private func copyAttribute(_ element: AXUIElement, attribute: CFString) -> AnyObject? {
 		var value: AnyObject?
 		let status = AXUIElementCopyAttributeValue(element, attribute, &value)

package/native/windows/bridge-rs/Cargo.lock

@@ -269,6 +269,7 @@ dependencies = [
  "serde_json",
  "thiserror",
  "windows",
+ "windows-core",
 ]
 
 [[package]]

package/native/windows/bridge-rs/Cargo.toml

@@ -12,7 +12,18 @@ image = { version = "0.25", default-features = false, features = ["png"] }
 base64 = "0.22"
 
 [target.'cfg(windows)'.dependencies]
+# Tangu 新增:`#[implement]`(电脑历史的 UIA 事件处理器)展开后引用 windows_core,须直接依赖、版本与 windows 一致。
+windows-core = "0.58"
 windows = { version = "0.58", features = [
+    "implement",
+    "Win32_Security",
+    "Win32_Security_Authorization",
+    "Win32_Storage_FileSystem",
+    "Win32_System_IO",
+    "Win32_System_LibraryLoader",
+    "Win32_System_Pipes",
+    "Win32_System_RemoteDesktop",
+    "Win32_System_StationsAndDesktops",
     "Win32_UI_Accessibility",
     "Win32_UI_WindowsAndMessaging",
     "Win32_UI_Input_KeyboardAndMouse",

package/native/windows/bridge-rs/src/agent_marker.rs

@@ -0,0 +1,97 @@
+//! Tangu 新增:跨进程的「agent 正在操作」标记,给电脑历史的事件打 `origin:"agent"`。
+//!
+//! macOS 上动作与采集在同一个 helper 进程里,一个深度计数就够;Windows 上执行动作的是引擎拉起的 stdio 助手,
+//! 采集的是桌面拉起的 `serve` 进程,所以用一个会话内的命名互斥量当标记:stdio 助手执行 act 等命令期间(以及结束后
+//! 750ms —— UIA 事件与输入是异步到的)持有它的句柄,采集器只探它在不在(OpenMutexW),从不创建。
+//! 非 Windows 平台全是空操作。
+
+#[cfg(windows)]
+use std::sync::Mutex;
+
+#[cfg(windows)]
+const MARKER_NAME: &str = "Local\\tangu-computer-use-agent-active";
+#[cfg(windows)]
+const GRACE_MS: u64 = 750;
+
+#[cfg(windows)]
+#[derive(Default)]
+struct MarkerState {
+    depth: u32,
+    /// 互斥量句柄(isize:HANDLE 不是 Send)。
+    handle: Option<isize>,
+    generation: u64,
+}
+
+#[cfg(windows)]
+fn state() -> &'static Mutex<MarkerState> {
+    static STATE: std::sync::OnceLock<Mutex<MarkerState>> = std::sync::OnceLock::new();
+    STATE.get_or_init(|| Mutex::new(MarkerState::default()))
+}
+
+/// 持有期间标记亮着;drop 后再亮 750ms。
+pub struct AgentGuard(());
+
+pub fn enter() -> AgentGuard {
+    #[cfg(windows)]
+    if let Ok(mut s) = state().lock() {
+        s.depth += 1;
+        if s.handle.is_none() {
+            use windows::core::HSTRING;
+            use windows::Win32::System::Threading::CreateMutexW;
+            if let Ok(h) = unsafe { CreateMutexW(None, false, &HSTRING::from(MARKER_NAME)) } {
+                s.handle = Some(h.0 as isize);
+            }
+        }
+    }
+    AgentGuard(())
+}
+
+impl Drop for AgentGuard {
+    fn drop(&mut self) {
+        #[cfg(windows)]
+        {
+            let generation = match state().lock() {
+                Ok(mut s) if s.depth > 0 => {
+                    s.depth -= 1;
+                    if s.depth > 0 {
+                        return;
+                    }
+                    s.generation += 1;
+                    s.generation
+                }
+                _ => return,
+            };
+            std::thread::spawn(move || {
+                std::thread::sleep(std::time::Duration::from_millis(GRACE_MS));
+                let Ok(mut s) = state().lock() else { return };
+                if s.depth == 0 && s.generation == generation {
+                    if let Some(raw) = s.handle.take() {
+                        use windows::Win32::Foundation::{CloseHandle, HANDLE};
+                        let _ = unsafe { CloseHandle(HANDLE(raw as *mut _)) };
+                    }
+                }
+            });
+        }
+    }
+}
+
+/// 别的进程(或本进程)此刻是否亮着标记。只探不建。
+pub fn active() -> bool {
+    #[cfg(windows)]
+    {
+        use windows::core::HSTRING;
+        use windows::Win32::Foundation::CloseHandle;
+        use windows::Win32::System::Threading::{OpenMutexW, SYNCHRONIZATION_SYNCHRONIZE};
+        match unsafe { OpenMutexW(SYNCHRONIZATION_SYNCHRONIZE, false, &HSTRING::from(MARKER_NAME)) } {
+            Ok(h) => {
+                let _ = unsafe { CloseHandle(h) };
+                true
+            }
+            Err(_) => false,
+        }
+    }
+    #[cfg(not(windows))]
+    {
+        false
+    }
+}

package/native/windows/bridge-rs/src/lib.rs

@@ -3,10 +3,14 @@
 //! This crate provides the JSON-lines protocol types used to communicate
 //! between the TypeScript host and the Rust helper binary.
 
+pub mod agent_marker;
 pub mod capture;
 pub mod error;
 pub mod input;
 pub mod protocol;
+#[cfg(windows)]
+pub mod recorder;
+pub mod recorder_logic;
 pub mod refs;
 pub mod state;
 pub mod uia;

package/native/windows/bridge-rs/src/main.rs

@@ -128,6 +128,16 @@ fn with_physical_input<T>(
 fn main() {
     #[cfg(windows)]
     set_dpi_awareness();
+    // Tangu 新增:电脑历史采集器的入口(桌面端拉起,见 src/recorder.rs);不走下面的 stdio 循环与窗口事件日志。
+    let args = std::env::args().skip(1).collect::<Vec<_>>();
+    match args.first().map(String::as_str) {
+        Some("recorder-protocol") => {
+            println!("{}", windows_bridge::recorder_logic::RECORDER_PROTOCOL);
+            return;
+        }
+        Some("serve") => std::process::exit(serve_recorder(&args)),
+        _ => {}
+    }
     window::start_root_event_journal();
 
     let stdin = io::stdin();
@@ -192,6 +202,25 @@ fn main() {
     }
 }
 
+fn serve_recorder(args: &[String]) -> i32 {
+    let pipe = args
+        .iter()
+        .position(|a| a == "--pipe")
+        .and_then(|i| args.get(i + 1))
+        .cloned()
+        .unwrap_or_default();
+    #[cfg(windows)]
+    {
+        windows_bridge::recorder::serve(&pipe)
+    }
+    #[cfg(not(windows))]
+    {
+        let _ = pipe;
+        eprintln!("serve is only available on Windows");
+        2
+    }
+}
+
 #[cfg(windows)]
 fn set_dpi_awareness() {
     use windows::Win32::UI::HiDpi::{
@@ -211,6 +240,9 @@ fn handle_request(request: &Request) -> Response {
         return Response::err(&request.id, ProtocolError::new(format!("Unsupported Windows helper protocol {}; expected {}. Restart Pi to use the installed helper.", request.protocol_version, PROTOCOL_VERSION), ErrorCode::InvalidRequest));
     }
 
+    // Tangu 新增:执行动作期间亮起跨进程标记,电脑历史据此给这段时间的事件打 origin:"agent"(见 agent_marker.rs)。
+    let _agent = matches!(request.cmd.as_str(), "act" | "actBatch" | "focusWindow" | "openBrowserLocation")
+        .then(windows_bridge::agent_marker::enter);
     let result = match request.cmd.as_str() {
         "diagnostics" => Ok(diagnostics()),
         "listRoots" | "listWindows" => handle_list_roots(&request.args),