mio-cua 0.1.0__tar.gz

mio_cua-0.1.0/PKG-INFO

@@ -0,0 +1,13 @@
+Metadata-Version: 2.4
+Name: mio-cua
+Version: 0.1.0
+Summary: Mio Computer-Use Agent: Windows desktop automation AI agent (SDK + CLI)
+Requires-Python: >=3.10
+Requires-Dist: requests>=2.31
+Requires-Dist: mss>=9.0
+Requires-Dist: Pillow>=10.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: pynput>=1.7
+Requires-Dist: pywin32>=306
+Requires-Dist: pywinauto>=0.6.8
+Requires-Dist: rapidocr_onnxruntime>=1.3.24

mio_cua-0.1.0/README.md

@@ -0,0 +1,110 @@
+# mio-cua
+
+Mio Computer-Use Agent：Windows 桌面自动化 AI Agent（Python SDK + CLI）。
+
+<!-- mcp-name: io.github.mldlbs/mio-cua -->
+
+## 安装
+
+```bash
+pip install -e .        # 从源码安装（依赖见 pyproject.toml）
+# Windows 下建议以管理员身份运行
+```
+
+### 配置
+
+设置 LLM API Key：
+
+```bash
+set OPENAI_API_KEY=sk-xxx        # Windows CMD
+$env:OPENAI_API_KEY = "sk-xxx"   # PowerShell
+```
+
+示例脚本还支持 `DESKTOP_AGENT_PROVIDER` / `DESKTOP_AGENT_MODEL` 环境变量覆盖模型。
+
+### 当前状态
+
+v0.5：Scene Graph 感知层 + 网页纯视觉感知（Regions 版面 + OmniParser 控件识别）；五个端到端场景（记事本/计算器/资源管理器/跨应用/网页）已在真实 Windows 11 桌面完成 5/5 全绿终验（2026-08-08）。
+
+- **Scene Graph 感知层**：感知层把 OCR+UIA 融合为一张场景图（`mio_cua/scene/`）——每个 UI 对象是一个 Node（类型/文本/状态/bbox），配空间关系（leftOf/above/labelFor）和 **Action candidates**（感知层已验证的可点击动作，如 `click node 7 ('7') {'value': '7'} expects {'display': True}`）。LLM 从候选动作里选择，不再自行推断坐标/UI。
+- **网页纯视觉感知（像人一样上网）**：浏览器窗口额外走两条视觉链路，不依赖 DOM——
+  - **Regions 版面分析**（`rapid_layout`，可选依赖）：识别页面结构（导航/标题/正文/表格/图片区域），可选依赖未装则优雅降级。
+  - **OmniParser 控件识别**（`scene/omniparser.py`，可选依赖）：YOLOv9 交互元素检测 + Florence-2 语义描述，把网页截图解析成按钮/链接/输入框（`button`/`text` 节点 + 可点击候选）。需 torch CUDA；模型默认在项目内 `models/omniparser/`，也可用 `OMNIPARSER_DIR`/`OMNIPARSER_WEIGHTS` 环境变量覆盖。
+- **显示区验证**：计算器显示区被识别为 `display` 节点，点击数字键后通过 Scene Diff 检测显示值变化（`0`→`7`），用于动作成功校验。
+- **跨应用（crossapp）场景**：`smoke/crossapp.yaml`——读 `smoke_numbers.txt`（12/34/56）→ 计算器求和 102 → 保存 `smoke_sum_result.txt`。全绿验收（2026-08-08，19-21 步）。关键修复：**一动作一感知**（每步重新感知再决策，避免多动作基于陈旧场景执行）；**计算器 `+` 键发送**（`key(keys="+")` 曾被当分隔符 split 失败）；**UWP 计算器聚焦**；其余见 SMOKE.md 修复链。
+- **网页纯视觉场景（web）**：`smoke/web.yaml`——Edge 打开本地 HTML，纯视觉（OmniParser，不接 DOM）点击按钮/输入文本并视觉确认。PASS（2026-08-08，9 步 165s）。OmniParser 走本地 HF 缓存离线推理（`HF_HUB_OFFLINE=1`，首帧加载约 20s，后续每帧 ~0.3s）。
+- **完整套件**：calculator/crossapp/explorer/notepad/web 五场景全绿终验（2026-08-08，crossapp 21 步 SUCCESS、其余 SUCCESS/校验通过）。deepseek-v4-flash 即可完成。调优中修复 6 个工程 bug（陈旧场景执行、UWP 聚焦、`+` 键 split、浏览器 PATH、重命名引导、元素 id 稳定）。
+- **GPU OCR**：经 DirectML（onnxruntime-directml）加速，单次 OCR 平均 ~1.6s；`DESKTOP_AGENT_OCR_DEVICE=cpu` 可回退纯 CPU。
+- **窗口区域截图**：只抓活动窗口区域（而非全屏），截图体积缩减约 80%，更聚焦目标窗口。
+- **Artifact 自动清理**：任务结束后自动修剪，目录默认上限 200MB（配置项 `artifact_max_bytes`）。
+- **循环守卫**：动作无变化 / 重复相同动作 ≥6 次直接判定 FAIL，避免空转。
+- **真实验收（deepseek-v4-flash + DirectML）**：记事本（输入并保存 `hello world`）、计算器（`123*456=56088`）、资源管理器（新建并重命名 `smoke_demo_folder`）三个场景全部 PASS。
+
+发送给 LLM 的截图带编号框（overlay），编号与元素列表 `id` 一一对应。每步操作与截图会落盘为 artifact，任务状态支持 `resume`。可重试失败会自动触发 Recovery（聚焦窗口后重试，每次 action 最多重试 2 次）。
+
+> 提示：首次在真实桌面运行前，请先完成一次小任务的冒烟测试（如"打开记事本输入 hello"），并确认 F9 急停可用。冒烟测试见 `SMOKE.md`。
+
+## CLI
+
+```bash
+mio-cua run "打开计算器，计算 3*4" --model gpt-4o
+mio-cua run "删除所有文件" --dry-run    # 只打印计划，不执行
+mio-cua resume <task_id>                # 从上次状态重新执行任务
+mio-cua replay <task_id>                # 从 artifact 回放任务步骤（调试用，--full 显示参数）
+mio-cua providers
+```
+
+## Artifacts / 状态
+
+- 每步：`~/.mio_cua/artifacts/<ts>.json`（observation + action + result）
+- 每步截图：`~/.mio_cua/artifacts/<ts>.png`（overlay 标注图）与 `.raw.png`
+- 任务状态：`~/.mio_cua/artifacts/state/<task_id>.json`
+
+## SDK
+
+```python
+from mio_cua import Agent, AgentConfig, Task
+
+agent = Agent(AgentConfig(model="gpt-4o", max_steps=50))
+result = agent.run(Task(instruction="打开记事本，输入 hello"))
+print(result.status, result.steps)
+```
+
+## MCP（接入 Claude / Cursor / ChatGPT）
+
+mio-cua 也作为 **MCP server** 暴露，让任意 MCP 客户端直接控制桌面：
+
+```bash
+pip install -e .
+```
+
+```json
+{ "mcpServers": { "mio-cua": { "command": "mio-cua-mcp", "args": [] } } }
+```
+
+10 个工具：文件操作（list_dir/make_dir/move_file/move_files）、窗口（launch/focus_window/get_active_window）、输入（click/type/key）。详见 [MCP.md](MCP.md)。
+
+## 安全
+
+- F9 急停
+- 步数上限 / 任务超时
+- 每步截图留痕（artifact）
+
+## 验收套件（smoke）
+
+五个端到端场景在**独立虚拟桌面**上隔离运行，不打扰主桌面：
+
+```bash
+# 完整套件（calculator/crossapp/explorer/notepad/web）
+python scripts/run_smoke_vdesk.py --only calculator,crossapp,explorer,notepad,web --model deepseek-v4-flash --base-url https://api.deepseek.com/v1
+
+# 单场景
+python scripts/run_smoke_vdesk.py --only crossapp --model deepseek-v4-flash --base-url https://api.deepseek.com/v1
+```
+
+运行前需：`$env:OPENAI_API_KEY="sk-xxx"`；桌面已解锁；用户约 20 分钟不碰键鼠（vdesk 隔离在用户操作时失效）。日志 `%TEMP%\smoke_vdesk.log`，结果以 `[PASS]/[FAIL]` 汇总。
+
+前置文件：`~/Desktop/smoke_numbers.txt`（12/34/56，crossapp 输入）、`~/Desktop/vision_test.html`（web 测试页）。场景间自动清理测试应用进程。
+
+- 判定规则：`status=SUCCESS` 且（若场景有产物校验）所有 file/dir/contains 校验通过才算 PASS——防止 agent 假 success。
+- 收尾提示：屏幕稳定且已按 Enter/Save 确认后，提示 agent 调 `success`，避免任务完成却超时。

mio_cua-0.1.0/mio_cua/__init__.py

@@ -0,0 +1,14 @@
+__version__ = "0.1.0"
+
+from mio_cua.config import AgentConfig
+from mio_cua.models.task import Task
+
+
+def __getattr__(name):
+    if name == "Agent":
+        from mio_cua.agent_factory import Agent
+        return Agent
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+__all__ = ["Agent", "AgentConfig", "Task", "__version__"]

mio_cua-0.1.0/mio_cua/agent/diff.py

@@ -0,0 +1,81 @@
+from mio_cua.models.observation import Change, ObservationDiff
+from mio_cua.scene.diff import diff as scene_diff_func
+
+
+def _key(e):
+    return e.role or "unknown", e.text or ""
+
+
+def _nearby(a, b, tol: int = 5) -> bool:
+    """True if bbox centers are within tol pixels of each other."""
+    ax, ay, aw, ah = a
+    bx, by, bw, bh = b
+    ac = (ax + aw / 2, ay + ah / 2)
+    bc = (bx + bw / 2, by + bh / 2)
+    return abs(ac[0] - bc[0]) <= tol and abs(ac[1] - bc[1]) <= tol
+
+
+def _role(e):
+    return e.role or "unknown"
+
+
+def compute_diff(prev, current) -> ObservationDiff:
+    if prev is None:
+        return ObservationDiff(prev=None, current=current, changes=[])
+    # Prefer the scene graph diff when both observations carry one: it detects
+    # display readout changes (e.g. calculator 0 -> 7) which the element-level
+    # diff treats as unrelated "removed" + "added" noise.
+    prev_scene = getattr(prev, "scene", None)
+    curr_scene = getattr(current, "scene", None)
+    if prev_scene is not None and curr_scene is not None \
+            and getattr(prev_scene, "nodes", None) and getattr(curr_scene, "nodes", None):
+        sc = scene_diff_func(prev_scene, curr_scene)
+        if sc:
+            return ObservationDiff(prev=prev, current=current, changes=[
+                Change(c.kind, c.node_id, c.description) for c in sc
+            ])
+    return _diff_elements(prev, current)
+
+
+def _diff_elements(prev, current) -> ObservationDiff:
+    prev_elements = list(prev.elements)
+    changes = []
+    matched_prev = set()
+    consumed_cur = set()
+
+    # stable: current element matched to prev by (role, text) identity within bbox tolerance
+    for ci, e in enumerate(current.elements):
+        for i, p in enumerate(prev_elements):
+            if i in matched_prev:
+                continue
+            if _key(p) == _key(e) and _nearby(p.bbox, e.bbox):
+                matched_prev.add(i)
+                consumed_cur.add(ci)
+                break
+
+    # text_changed: same role at nearly same position, but text differs
+    for ci, e in enumerate(current.elements):
+        if ci in consumed_cur:
+            continue
+        for i, p in enumerate(prev_elements):
+            if i in matched_prev:
+                continue
+            if _role(p) == _role(e) and _nearby(p.bbox, e.bbox) and p.text != e.text:
+                matched_prev.add(i)
+                consumed_cur.add(ci)
+                changes.append(Change("text_changed", e.id, f"{p.text} -> {e.text}"))
+                break
+
+    # added: current elements never matched
+    for ci, e in enumerate(current.elements):
+        if ci not in consumed_cur:
+            changes.append(Change("added", e.id, e.text or e.role))
+
+    # removed: prev elements never matched
+    for i, p in enumerate(prev_elements):
+        if i not in matched_prev:
+            label = p.text or p.role or "unknown"
+            changes.append(Change("removed", None, label))
+
+    return ObservationDiff(prev=prev, current=current, changes=changes)
+

mio_cua-0.1.0/mio_cua/agent/dispatcher.py

@@ -0,0 +1,39 @@
+from typing import Callable, List
+
+from mio_cua.models.action import Plan
+from mio_cua.models.action_result import ActionResult
+
+
+class Dispatcher:
+    """Executes a plan of actions through a tool registry.
+
+    The `recover` callable must accept `(action, result, ctx)` and return an
+    ActionResult. If None, a default recovery that re-dispatches the action
+    through the registry (bound to ctx) is used.
+    """
+
+    def __init__(self, registry, recover: Callable = None):
+        self.registry = registry
+        self.recover = recover
+
+    def _default_recover(self, action, result, ctx):
+        return result
+
+    def execute(self, plan: Plan, safety, ctx) -> List[ActionResult]:
+        results = []
+        for action in plan.actions:
+            if safety.should_stop():
+                break
+            ctx.current_action_id = action.id
+            try:
+                result = self.registry.call(action.type, action.params, ctx)
+            except Exception as e:
+                result = ActionResult(action.id, success=False, message=str(e), retryable=True)
+            if not result.success and result.retryable:
+                if self.recover is not None:
+                    result = self.recover(action, result, ctx)
+                else:
+                    result = self._default_recover(action, result, ctx)
+            results.append(result)
+            safety.record_step()
+        return results

mio_cua-0.1.0/mio_cua/agent/expected.py

@@ -0,0 +1,48 @@
+"""Programmatic action verification against an affordance's ``expected``.
+
+The perception layer annotates click candidates with an ``expected`` dict
+(e.g. ``{'display': True}`` for calculator digits, ``{'display': 'unchanged'}``
+for operators). The LLM sees this hint but the loop never checks whether the
+screen actually changed as expected. This module closes that gap: after an
+action, diff the display nodes and report whether the expected change happened,
+so the loop can tell the agent "that click registered" vs "it did not".
+
+This reduces the classic failure where the agent clicks a key, the display did
+not change, and it either repeats the click (thinking it missed) or moves on
+(not noticing it missed).
+"""
+
+from typing import Optional, Tuple
+
+from mio_cua.scene.diff import display_text
+
+
+class ExpectedVerifier:
+    """Verify an action's outcome against its expected screen change."""
+
+    def verify(self, prev_scene, curr_scene, expected: dict) -> Tuple[bool, str]:
+        """Return (ok, detail).
+
+        ``ok`` is True when the display change matches ``expected`` (or there
+        is nothing to verify). ``detail`` is a human-readable reason.
+        """
+        if not expected:
+            return True, "no expectation"
+        if "display" not in expected:
+            return True, "no display expectation"
+        return self._verify_display(prev_scene, curr_scene, expected["display"])
+
+    def _verify_display(self, prev_scene, curr_scene, want) -> Tuple[bool, str]:
+        prev_text = display_text(prev_scene) if prev_scene is not None else ""
+        curr_text = display_text(curr_scene) if curr_scene is not None else ""
+        changed = bool(prev_text and prev_text != curr_text)
+
+        if want is True:
+            if changed:
+                return True, f"display changed: {prev_text!r} -> {curr_text!r}"
+            return False, f"display did not change (still {curr_text!r})"
+        if want == "unchanged":
+            if changed:
+                return False, f"display changed unexpectedly: {prev_text!r} -> {curr_text!r}"
+            return True, f"display unchanged ({curr_text!r})"
+        return True, "unknown expectation"

mio_cua-0.1.0/mio_cua/agent/loop.py

@@ -0,0 +1,358 @@
+import logging
+import os
+import time
+import uuid
+
+from mio_cua.agent.diff import compute_diff
+from mio_cua.automation.input_controller import InputController
+from mio_cua.events import ObservationCreated, ActionStarted, ActionFinished, TaskFinished
+from mio_cua.models.action_result import ActionResult
+from mio_cua.models.task import Task, TaskResult
+from mio_cua.scene.memory import SceneMemory
+
+logger = logging.getLogger(__name__)
+
+
+def _keys_eq(sig, want):
+    """True if a key() action sig's keys value is exactly ``want``.
+
+    The sig looks like ``key([('keys', 'ctrl+s')])`` (real loop) or
+    ``key({'keys': 'ctrl+s'})`` (older/test). Compare the exact token so
+    ``ctrl+shift+n`` is not mistaken for ``ctrl+s`` (substring trap).
+    """
+    for sep in ("'keys', '", "'keys': '"):
+        marker = "'keys'" + sep[6:]
+        idx = sig.find(sep)
+        if idx < 0:
+            continue
+        start = idx + len(sep)
+        end = sig.find("'", start)
+        if end < 0:
+            continue
+        if sig[start:end] == want:
+            return True
+    return False
+
+
+class AgentLoop:
+    def __init__(self, perception, planner, registry, safety, events,
+                 recover=None, config=None, history=None, controller=None,
+                 artifact_store=None, state_dir=None):
+        self.perception = perception
+        self.planner = planner
+        self.registry = registry
+        self.safety = safety
+        self.events = events
+        self.recover = recover
+        self.config = config
+        self.history = history
+        self.controller = controller or InputController()
+        self.artifact_store = artifact_store
+        self.state_dir = state_dir
+        self._task = None
+        self._artifact_paths = []
+        self._task_id = uuid.uuid4().hex[:8]
+        self.scene_memory = SceneMemory()
+
+    def _make_ctx(self, obs):
+        from mio_cua.tools.context import ToolContext
+        self.controller.current_observation = obs
+        return ToolContext(
+            controller=self.controller,
+            perception=self.perception,
+            config=self.config,
+            events=self.events,
+            current_observation=obs,
+        )
+
+    def _save_artifact(self, obs, action, result):
+        if self.artifact_store is None:
+            return
+        p = self.artifact_store.save_artifact(obs=obs, action=action, result=result,
+                                              task_id=self._task_id)
+        self._artifact_paths.append(str(p))
+
+    def _save_state(self, obs, step):
+        if self.state_dir is None:
+            return
+        from mio_cua.memory.state import TaskState, state_path
+        shot = obs.screenshot_path if obs else ""
+        TaskState(state_path(self.state_dir, self._task_id)).save(
+            task_id=self._task_id,
+            instruction=self._task.instruction if self._task else "",
+            step=step,
+            screenshot=shot,
+        )
+
+    def _prune_artifacts(self):
+        if self.artifact_store is None:
+            return
+        limit = getattr(self.config, "artifact_max_bytes", 200 * 1024 * 1024)
+        try:
+            freed = self.artifact_store.prune(limit)
+            if freed:
+                logger.info("pruned %d bytes of old artifacts", freed)
+        except Exception as e:
+            logger.warning("artifact prune failed: %s", e)
+
+    def run(self, task: Task) -> TaskResult:
+        start = time.time()
+        self._task = task
+        self.safety.start()
+        steps = 0
+        finished_status = None
+        finished_summary = ""
+        terminal = "RUNNING"
+        try:
+            from collections import deque
+            from mio_cua.agent.expected import ExpectedVerifier
+            prev = None
+            no_change = 0
+            repeat_count = 0
+            self._recent_sigs = deque(maxlen=8)
+            self._verifier = ExpectedVerifier()
+            self._pending_verify = None  # (node_id, expected, prev_scene) awaiting the next observation
+            while not self.safety.should_stop():
+                obs = self.perception.observe()
+                self.events.publish(ObservationCreated(obs))
+                self._save_state(obs, steps)
+                self.scene_memory.push(getattr(obs, "scene", None))
+                diff = compute_diff(prev, obs)
+                if prev is not None and not diff.changes:
+                    no_change += 1
+                else:
+                    no_change = 0
+                hints = []
+                if self._pending_verify is not None:
+                    vh = self._verify_pending(obs)
+                    if vh:
+                        hints.append(vh)
+                    self._pending_verify = None
+                mem_summary = self.scene_memory.summarize(
+                    recent_actions=[h["type"] for h in (self.history.recent(6) if self.history else [])]
+                    if self.history else None,
+                )
+                if mem_summary:
+                    hints.append("MEMORY (what you have already seen/done):\n" + mem_summary +
+                                 "\nUse this to continue the task -- do not re-read or re-open what you already saw.")
+                if no_change >= 2:
+                    hints.append('the screen did not change after your recent actions — the last action had no visible effect. Do NOT repeat it. To confirm a dialog, call key(keys="enter") or click the Save/OK button.')
+                confirm_hint = self._confirm_hint()
+                if confirm_hint:
+                    hints.append(confirm_hint)
+                rename_hint = self._rename_hint()
+                if rename_hint:
+                    hints.append(rename_hint)
+                finish_hint = self._completion_hint(no_change)
+                if finish_hint:
+                    hints.append(finish_hint)
+                if len(self._recent_sigs) >= 4 and self._recent_sigs.count(self._recent_sigs[-1]) >= 4:
+                    hints.append(f"you have called `{self._recent_sigs[-1]}` repeatedly with no effect. STOP repeating it and choose a different action now.")
+                if hints:
+                    logger.debug("hints@%d: %s", steps, " | ".join(hints))
+                plan = self.planner.plan(task, obs, diff, self.registry.schemas(), history=self.history, hints=hints)
+                if not plan.actions:
+                    break
+                ctx = self._make_ctx(obs)
+                for i, action in enumerate(plan.actions):
+                    if self.safety.should_stop():
+                        break
+                    self.events.publish(ActionStarted(action))
+                    ctx.current_action_id = action.id
+                    try:
+                        result = self.registry.call(action.type, action.params, ctx)
+                    except Exception as e:
+                        result = ActionResult(action.id, success=False, message=str(e), retryable=True)
+                    if not result.success and result.retryable and self.recover is not None:
+                        result = self.recover(action, result, ctx)
+                    if result.success and action.type == "click":
+                        self._pending_verify = self._capture_expected(obs, action)
+                    self._save_artifact(obs, action, result)
+                    self.events.publish(ActionFinished(result))
+                    if self.history is not None:
+                        self.history.record(action.id, action.type, result.success, result.message)
+                    self.safety.record_step()
+                    steps += 1
+                    if action.type not in ("success", "fail"):
+                        sig = f"{action.type}({sorted(action.params.items())})"
+                        if self._recent_sigs and self._recent_sigs[-1] == sig:
+                            repeat_count += 1
+                        else:
+                            repeat_count = 1
+                        self._recent_sigs.append(sig)
+                        if repeat_count >= 6:
+                            finished_status = "FAIL"
+                            finished_summary = f"stuck: repeated {sig} {repeat_count} times with no effect"
+                            break
+                    if action.type == "success":
+                        blocker = self._unconfirmed_edit()
+                        if blocker:
+                            # hard guard: an element_id-less type (rename box /
+                            # filename field) was not confirmed with Enter, so a
+                            # success() would claim an edit that never applied.
+                            hints.append(blocker)
+                            self._save_artifact(obs, action, result)
+                            self.events.publish(ActionFinished(ActionResult(
+                                action.id, False, blocker, retryable=True)))
+                            if self.history is not None:
+                                self.history.record(action.id, action.type, False, blocker)
+                            self.safety.record_step()
+                            steps += 1
+                            continue
+                        finished_status = "SUCCESS"
+                        finished_summary = str(action.params.get("result", ""))
+                        break
+                    if action.type == "fail":
+                        finished_status = "FAIL"
+                        finished_summary = str(action.params.get("reason", ""))
+                        break
+                    # Only ONE action per observation: the screen changes after
+                    # every action, so subsequent actions in the same plan were
+                    # decided against a stale scene. Re-observe + replan first.
+                    # (Terminal success/fail already broke above.)
+                    if i + 1 < len(plan.actions):
+                        break
+                if finished_status in ("SUCCESS", "FAIL"):
+                    break
+                prev = obs
+        except Exception as e:
+            finished_status = "FAIL"
+            finished_summary = f"loop error: {e}"
+        finally:
+            # capture status BEFORE stopping so stop() doesn't flip it to ABORTED
+            terminal = self.safety.status()
+            self.safety.stop()
+
+        status = finished_status or terminal
+        if status == "RUNNING":
+            status = "FAIL"
+        self._prune_artifacts()
+        result = TaskResult(
+            status=status,
+            summary=finished_summary,
+            task_id=self._task_id,
+            steps=steps,
+            duration=time.time() - start,
+            artifacts=self._artifact_paths,
+        )
+        self.events.publish(TaskFinished(result))
+        return result
+
+    def _capture_expected(self, obs, action):
+        """Record the clicked node's expected screen change for verification."""
+        scene = getattr(obs, "scene", None)
+        if scene is None:
+            return None
+        node_id = action.params.get("element_id")
+        if node_id is None:
+            return None
+        aff = scene.affordance_for(int(node_id), "click")
+        if aff is None or not aff.expected:
+            return None
+        return (int(node_id), dict(aff.expected), scene)
+
+    def _verify_pending(self, obs):
+        """Check whether the previous click produced its expected change."""
+        node_id, expected, prev_scene = self._pending_verify
+        curr_scene = getattr(obs, "scene", None)
+        if curr_scene is None:
+            return None
+        ok, detail = self._verifier.verify(prev_scene, curr_scene, expected)
+        if ok:
+            return None
+        return (f"VERIFICATION: your last click on node {node_id} did not have the "
+                f"expected effect ({detail}). It likely missed or the target changed. "
+                f"Do NOT repeat it blindly -- re-inspect and pick a fresh target.")
+
+    def _unconfirmed_edit(self):
+        """A rename/filename type (type WITHOUT element_id) that has NOT been
+        confirmed with Enter. success() must be blocked then, or the agent
+        claims an edit that never applied (folder/file name unchanged)."""
+        sigs = list(getattr(self, "_recent_sigs", None) or [])
+        typed = [s for s in sigs if s.startswith("type(") and "element_id" not in s]
+        if not typed:
+            return None
+        if any(self._is_confirming_action(s) for s in sigs):
+            return None
+        return ("BLOCKED: you typed a name but never pressed `enter` to apply it "
+                "-- the edit is NOT saved. Call `key(keys=\"enter\")` to confirm, "
+                "then call `success`.")
+
+    def _rename_hint(self):
+        """If the agent pressed ctrl+shift+n (created a new folder) but has not
+        typed a name for it, the folder stays '新建文件夹' and the task is not
+        done. Tell it to type the name (type WITHOUT element_id -- the rename
+        box has focus)."""
+        sigs = list(getattr(self, "_recent_sigs", None) or [])
+        if not sigs:
+            return None
+        created = any("ctrl+shift+n" in s for s in sigs[-4:])
+        if not created:
+            return None
+        if any(s.startswith("type(") for s in sigs[-4:]):
+            return None
+        return ("You created a new folder (ctrl+shift+n) but have NOT typed its "
+                "name. Its name is selected and the rename box has focus -- call "
+                "`type` WITHOUT element_id (e.g. `type(text=\"smoke_demo_folder\")`) "
+                "to name it, then press `enter`.")
+
+    def _confirm_hint(self):
+        """If the agent typed WITHOUT an element_id (a focused rename box or
+        filename field -- the explorer/filename case) and has not pressed Enter
+        to confirm, the edit is still pending. Tell it to press Enter instead
+        of re-typing or moving on. The classic explorer failure: type the
+        folder name, never press Enter, call success.
+
+        Types WITH an element_id (e.g. notepad body) do not need Enter, so we
+        only react to element_id-less types to avoid false positives.
+        """
+        sigs = list(getattr(self, "_recent_sigs", None) or [])
+        if not sigs:
+            return None
+        typed_without_target = [s for s in sigs
+                                if s.startswith("type(") and "element_id" not in s]
+        if not typed_without_target:
+            return None
+        # ...and no Enter/Save confirmation after the latest such type
+        if any(self._is_confirming_action(s) for s in sigs[-3:]):
+            return None
+        return ("You typed into a rename/filename box (type without element_id) "
+                "but have NOT pressed `enter` to confirm it -- the change is NOT "
+                "applied until `enter`. Call `key(keys=\"enter\")` now, then the "
+                "task is done.")
+
+    def _completion_hint(self, no_change):
+        """Hint the agent to finish when it has done the closing steps but
+        keeps verifying instead of calling success (task completed but
+        status=TIMEOUT).
+
+        Requires a *confirming* action (Enter / Save) reasonably recent AND the
+        screen to have settled (no_change >= 1). Not just any type: typing
+        without Enter leaves the edit unconfirmed, so that alone is not
+        completion (see _confirm_hint). A confirming action later in the
+        trailing history means a save/rename was applied.
+        """
+        if no_change < 1:
+            return None
+        sigs = list(getattr(self, "_recent_sigs", None) or [])
+        if len(sigs) < 2:
+            return None
+        confirmed = [s for s in sigs if self._is_confirming_action(s)]
+        if not confirmed:
+            return None
+        # The confirming action should not be buried too far behind unrelated work.
+        last_index = max(i for i, s in enumerate(sigs) if self._is_confirming_action(s))
+        if len(sigs) - 1 - last_index > 2:
+            return None
+        return ("The screen has settled and you already pressed Enter / Save to "
+                "apply the change. If the task's goal is met, STOP and call "
+                "`success` with a summary now -- do not keep acting.")
+
+    @staticmethod
+    def _is_confirming_action(sig):
+        """Actions that CONFIRM a save/rename: Enter, Save click, Ctrl+S."""
+        if sig.startswith("key("):
+            return any(_keys_eq(sig, k) for k in ("enter", "ctrl+s"))
+        if sig.startswith("click("):
+            return "Save" in sig or "保存" in sig
+        return False