@tanskong/office-assistant 1.0.4 → 1.0.6

package/README.md

@@ -1,57 +1,91 @@
-# Office Assistant
+# Office Assistant - 智能办公助手
 
-AI-powered Office automation via MCP protocol.
+基于 MCP 协议的 AI 驱动 Office 自动化工具。
 
-## Features
+## 功能特性
 
-- **RunPython**: Execute Python code to control Office applications freely
-- **Smart Workspace**: Desktop folder for safe file processing
-- **Full Office Suite**: Excel, Word, PowerPoint, Outlook, Visio, WPS, and more
-- **License Protection**: Machine-bound license activation
+- **RunPython 主力工具**：自由执行 Python 代码控制 Office 应用程序
+- **智能工作区**：桌面文件夹安全处理文件
+- **全 Office 套件支持**：Excel、Word、PowerPoint、Outlook、Visio、WPS 等
+- **UI 自动化引擎**：基于 Windows UI Automation 框架的精准截图与控件操作
+- **许可证保护**：机器绑定的许可证激活机制
 
-## Installation
+## 安装
 
 ```bash
-npx @yourname/office-assistant
+npx @tanskong/office-assistant
 ```
 
-## Activation
+## 激活
 
-1. Set environment variable:
+1. 设置环境变量：
    ```bash
    set OFFICE_ASSISTANT_LICENSE=OFFICE-ASSISTANT-XXXX-XXXX-XXXX-XXXX
    ```
 
-2. Run activation:
+2. 运行激活：
    ```bash
-   npx @yourname/office-assistant activate
+   npx @tanskong/office-assistant activate
    ```
 
-3. Start the server:
+3. 启动服务：
    ```bash
-   npx @yourname/office-assistant
+   npx @tanskong/office-assistant
    ```
 
-## Disclaimer
+## 免责声明
 
-Please place files in `Desktop/智能办公区/数据` before processing.
-We are not responsible for data loss when operating on original file locations.
+请将需要处理的文件复制到 `桌面/智能办公区/数据` 文件夹后再进行操作。
+直接在原文件位置操作导致的数据丢失，本软件概不负责。
 
-## Tools
+## 工具列表
 
-| Tool | Description |
-|------|-------------|
-| `run_python` | Execute Python code (PRIMARY TOOL) |
-| `available_apps` | List installed Office applications |
-| `launch_app` | Launch an Office application |
-| `quit_app` | Quit an Office application |
-| `open_file` | Open an Office file |
-| `save_file` | Save current document |
-| `screenshot` | Capture screenshot |
-| `download` | Download file from URL |
-| `speak` | Text-to-speech |
-| `demonstrate` | Run demonstration |
+### Office 控制工具
 
-## License
+| 工具 | 说明 |
+|------|------|
+| `run_python` | 执行 Python 代码（主力工具） |
+| `available_apps` | 列出已安装的 Office 应用程序 |
+| `launch_app` | 启动 Office 应用程序 |
+| `quit_app` | 退出 Office 应用程序 |
+| `open_file` | 打开 Office 文件 |
+| `save_file` | 保存当前文档 |
 
-Commercial license required. See activation instructions above.
+### UIA 自动化工具
+
+| 工具 | 说明 |
+|------|------|
+| `screenshot` | 精准截图（全屏/活动窗口/指定窗口） |
+| `screenshot_element` | 截取指定 UI 元素的截图 |
+| `find_ui_element` | 查找 UI 元素并返回属性信息 |
+| `get_ui_tree` | 获取窗口的 UI 树结构 |
+| `click_ui_element` | 点击指定 UI 元素 |
+| `get_window_list` | 列出所有顶层窗口 |
+| `activate_window` | 激活指定窗口 |
+
+### 系统工具
+
+| 工具 | 说明 |
+|------|------|
+| `download` | 从 URL 下载文件 |
+| `demonstrate` | 运行功能演示 |
+
+## 截图功能示例
+
+```python
+# 截取全屏
+screenshot()
+
+# 截取活动窗口
+screenshot('active')
+
+# 截取指定窗口
+screenshot('Excel')
+
+# 截取 UI 元素
+screenshot_element('Calculate', 'Button', 'Calculator')
+```
+
+## 许可证
+
+需要商业许可证。请参阅上方的激活说明。

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tanskong/office-assistant",
-  "version": "1.0.4",
+  "version": "1.0.6",
   "description": "Office Assistant - AI-powered Office automation via MCP protocol with license protection",
   "main": "bin/office-assistant.js",
   "bin": {

package/requirements.txt

@@ -2,3 +2,4 @@ fastmcp>=2.3.3
 pywin32>=310
 Pillow>=10.0.0
 psutil>=5.9.0
+uiautomation>=2.0.29

package/src/officer.py

@@ -6,6 +6,8 @@ import win32com.client
 import pythoncom
 from pathlib import Path
 
+from uia_tools import UIATools
+
 
 class TheOfficer:
     def __init__(self):
@@ -31,6 +33,7 @@ class TheOfficer:
         self.ComObjects = {}
         self._printable = False
         self.Data = OfficerData()
+        self.UIA = UIATools()
 
         self.MicrosoftApplications = [
             'Word', 'Excel', 'PowerPoint',
@@ -265,7 +268,26 @@ class TheOfficer:
         os.makedirs(folder, exist_ok=True)
         return os.path.join(folder, file_name)
 
-    def ScreenShot(self, save_path: str = None) -> str:
+    def ScreenShot(self, save_path: str = None, target: str = None) -> str:
+        """
+        截取屏幕截图。
+
+        Args:
+            save_path: 保存路径，None 则自动生成
+            target: 截图目标，None=全屏, 'active'=活动窗口, 其他=指定窗口名称
+
+        Returns:
+            截图文件路径
+        """
+        if self.UIA.available:
+            if target is None:
+                return self.UIA.screenshot_desktop(save_path)
+            elif target == 'active':
+                return self.UIA.screenshot_window(None, save_path)
+            else:
+                return self.UIA.screenshot_window(target, save_path)
+
+        # 降级到原有 GDI 截图（当 UIA 不可用时）
         import win32gui
         import win32ui
         import win32con

package/src/server.py

@@ -11,6 +11,7 @@ We are not responsible for data loss when operating on original file locations.
 
 import sys
 import os
+from io import StringIO
 
 from license_manager import verify_license
 verify_license()
@@ -184,27 +185,157 @@ def run_python(code: str, app_name: str = "Excel") -> dict:
     doc = _get_active_doc(app, app_name)
     globals_dict = _build_globals(app, doc, app_name)
 
+    # Capture stdout to return print output
+    old_stdout = sys.stdout
+    sys.stdout = buffer = StringIO()
+    
     try:
         exec(code, globals_dict)
-        return {"success": True}
+        output = buffer.getvalue()
+        sys.stdout = old_stdout
+        return {"success": True, "output": output or "(无输出)"}
     except Exception as e:
+        sys.stdout = old_stdout
         return {"success": False, "error": str(e)}
 
 
-# ========== System Tools ==========
+# ========== UIA Tools ==========
 
 @mcp.tool()
-def screenshot(save_name: str = None) -> dict:
-    """Capture a screenshot and save to the export folder."""
+def screenshot(target: str = None, save_name: str = None) -> dict:
+    """
+    Capture a screenshot with UIA precision.
+
+    Args:
+        target: Screenshot target. None=full desktop, 'active'=active window, other=window name
+        save_name: Filename to save (auto-generated if None)
+
+    Examples:
+        screenshot()                    # Full desktop
+        screenshot('active')            # Current active window
+        screenshot('Excel')             # Window with name containing 'Excel'
+    """
+    try:
+        path = Officer.ScreenShot(save_name, target)
+        return {"success": True, "path": path}
+    except Exception as e:
+        return {"success": False, "error": str(e)}
+
+
+@mcp.tool()
+def screenshot_element(element_name: str, control_type: str = None,
+                       parent_name: str = None, save_name: str = None) -> dict:
+    """
+    Capture screenshot of a specific UI element.
+
+    Args:
+        element_name: Name of the UI element
+        control_type: Control type, e.g., 'Button', 'Edit', 'Document'
+        parent_name: Parent window name to limit search scope
+        save_name: Filename to save
+
+    Examples:
+        screenshot_element('Calculate', 'Button', 'Calculator')
+        screenshot_element('Sheet1', 'TabItem', 'Excel')
+    """
     try:
-        path = Officer.ScreenShot(save_name)
+        path = Officer.UIA.screenshot_element(element_name, control_type, parent_name, save_name)
         return {"success": True, "path": path}
-    except ImportError as e:
-        return {"success": False, "error": "缺少 Pillow 模块，请运行: pip install Pillow", "detail": str(e)}
     except Exception as e:
         return {"success": False, "error": str(e)}
 
 
+@mcp.tool()
+def find_ui_element(name: str, control_type: str = None,
+                    parent_name: str = None) -> dict:
+    """
+    Find a UI element and return its properties.
+
+    Args:
+        name: Element name
+        control_type: Control type filter
+        parent_name: Parent window name
+
+    Returns:
+        Element properties including name, type, rect, automation_id
+    """
+    try:
+        element = Officer.UIA.find_element(name, control_type, parent_name)
+        if element is None:
+            return {"success": False, "error": f"Element not found: {name}"}
+        return {
+            "success": True,
+            "name": element.Name,
+            "type": element.ControlTypeName,
+            "automation_id": element.AutomationId,
+            "class_name": element.ClassName,
+            "rect": element.BoundingRectangle,
+            "enabled": element.IsEnabled,
+            "visible": not element.IsOffscreen
+        }
+    except Exception as e:
+        return {"success": False, "error": str(e)}
+
+
+@mcp.tool()
+def get_ui_tree(window_name: str = None, max_depth: int = 3) -> dict:
+    """
+    Get the UI automation tree of a window.
+
+    Args:
+        window_name: Window name, None for active window
+        max_depth: Maximum traversal depth
+
+    Returns:
+        Hierarchical tree structure of UI elements
+    """
+    try:
+        tree = Officer.UIA.get_element_tree(window_name, max_depth)
+        return {"success": True, "tree": tree}
+    except Exception as e:
+        return {"success": False, "error": str(e)}
+
+
+@mcp.tool()
+def click_ui_element(name: str, control_type: str = None,
+                     parent_name: str = None) -> dict:
+    """
+    Click a UI element.
+
+    Args:
+        name: Element name
+        control_type: Control type
+        parent_name: Parent window name
+    """
+    try:
+        result = Officer.UIA.click_element(name, control_type, parent_name)
+        return {"success": result}
+    except Exception as e:
+        return {"success": False, "error": str(e)}
+
+
+@mcp.tool()
+def get_window_list() -> dict:
+    """List all top-level windows."""
+    try:
+        windows = Officer.UIA.get_window_list()
+        return {"success": True, "windows": windows}
+    except Exception as e:
+        return {"success": False, "error": str(e)}
+
+
+@mcp.tool()
+def activate_window(window_name: str) -> dict:
+    """Activate a window by name."""
+    try:
+        result = Officer.UIA.activate_window(window_name)
+        return {"success": result}
+    except Exception as e:
+        return {"success": False, "error": str(e)}
+
+
+# ========== System Tools ==========
+
 @mcp.tool()
 def download(url: str, save_name: str = None) -> str:
     """Download a file from URL to the download folder."""

package/src/uia_tools.py

@@ -0,0 +1,366 @@
+#!/usr/bin/env python3
+"""
+UI Automation Tools for Office Assistant
+========================================
+基于 Windows UI Automation (UIA) 框架的自动化工具模块。
+提供精准截图、UI 元素探测、控件操作等功能。
+
+依赖: uiautomation >= 2.0
+"""
+
+import os
+import datetime
+from typing import Optional, List, Dict, Any
+from pathlib import Path
+
+try:
+    import uiautomation as auto
+except ImportError:
+    auto = None
+
+
+class UIATools:
+    """UI Automation 工具封装类"""
+
+    def __init__(self):
+        self._uia_available = auto is not None
+
+    @property
+    def available(self) -> bool:
+        """检查 UIA 是否可用"""
+        return self._uia_available
+
+    def _ensure_available(self):
+        if not self._uia_available:
+            raise ImportError("uiautomation 模块未安装，请运行: pip install uiautomation")
+
+    def _get_save_path(self, file_name: str = None, subfolder: str = "导出") -> str:
+        """获取保存路径"""
+        workspace = Path.home() / "Desktop" / "智能办公区"
+        if not workspace.exists():
+            workspace = Path.home() / "OfficeMCP_Data"
+        folder = workspace / subfolder
+        folder.mkdir(parents=True, exist_ok=True)
+        if file_name is None:
+            now = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
+            file_name = f"screenshot_{now}.png"
+        return str(folder / file_name)
+
+    # ========== 截图功能 ==========
+
+    def screenshot_window(self, window_name: str = None, save_path: str = None) -> str:
+        """
+        截取指定窗口的截图。
+
+        Args:
+            window_name: 窗口标题或名称，None 则截取活动窗口
+            save_path: 保存路径，None 则自动生成
+
+        Returns:
+            截图文件的完整路径
+        """
+        self._ensure_available()
+
+        if window_name:
+            window = auto.WindowControl(searchDepth=1, Name=window_name)
+            if not window.Exists():
+                raise ValueError(f"未找到窗口: {window_name}")
+        else:
+            window = auto.GetForegroundControl()
+
+        if save_path is None:
+            save_path = self._get_save_path()
+        else:
+            save_path = self._get_save_path(save_path)
+
+        window.CaptureToImage(save_path)
+        return save_path
+
+    def screenshot_element(self, element_name: str, control_type: str = None,
+                          parent_name: str = None, save_path: str = None) -> str:
+        """
+        截取指定 UI 元素的截图。
+
+        Args:
+            element_name: 元素名称 (Name 属性)
+            control_type: 控件类型，如 'Button', 'Edit', 'Document' 等
+            parent_name: 父窗口名称，用于限定搜索范围
+            save_path: 保存路径
+
+        Returns:
+            截图文件的完整路径
+        """
+        self._ensure_available()
+
+        element = self.find_element(element_name, control_type, parent_name)
+        if element is None:
+            raise ValueError(f"未找到元素: {element_name}")
+
+        if save_path is None:
+            save_path = self._get_save_path()
+        else:
+            save_path = self._get_save_path(save_path)
+
+        element.CaptureToImage(save_path)
+        return save_path
+
+    def screenshot_desktop(self, save_path: str = None) -> str:
+        """
+        截取全屏截图（替代原有的 GDI 截图）。
+
+        Args:
+            save_path: 保存路径
+
+        Returns:
+            截图文件的完整路径
+        """
+        self._ensure_available()
+
+        if save_path is None:
+            save_path = self._get_save_path()
+        else:
+            save_path = self._get_save_path(save_path)
+
+        desktop = auto.GetRootControl()
+        desktop.CaptureToImage(save_path)
+        return save_path
+
+    # ========== 元素查找功能 ==========
+
+    def find_element(self, name: str, control_type: str = None,
+                     parent_name: str = None, timeout: int = 3):
+        """
+        查找 UI 元素。
+
+        Args:
+            name: 元素名称
+            control_type: 控件类型
+            parent_name: 父窗口名称
+            timeout: 超时时间（秒）
+
+        Returns:
+            找到的元素对象，未找到返回 None
+        """
+        self._ensure_available()
+
+        if parent_name:
+            parent = auto.WindowControl(searchDepth=1, Name=parent_name)
+            if not parent.Exists(timeout):
+                return None
+            scope = parent
+        else:
+            scope = auto.GetRootControl()
+
+        kwargs = {"Name": name, "searchDepth": 10}
+        if control_type:
+            kwargs["ControlType"] = getattr(auto.ControlType, control_type, 0)
+
+        element = scope.FindFirst(**kwargs)
+        return element
+
+    def find_elements(self, name: str = None, control_type: str = None,
+                      parent_name: str = None, timeout: int = 3) -> List[Any]:
+        """
+        查找多个匹配的 UI 元素。
+
+        Args:
+            name: 元素名称（支持部分匹配）
+            control_type: 控件类型
+            parent_name: 父窗口名称
+            timeout: 超时时间
+
+        Returns:
+            元素列表
+        """
+        self._ensure_available()
+
+        if parent_name:
+            parent = auto.WindowControl(searchDepth=1, Name=parent_name)
+            if not parent.Exists(timeout):
+                return []
+            scope = parent
+        else:
+            scope = auto.GetRootControl()
+
+        kwargs = {"searchDepth": 10}
+        if name:
+            kwargs["Name"] = name
+        if control_type:
+            kwargs["ControlType"] = getattr(auto.ControlType, control_type, 0)
+
+        return scope.FindAll(**kwargs)
+
+    def get_element_tree(self, window_name: str = None, max_depth: int = 3) -> Dict:
+        """
+        获取 UI 元素树结构。
+
+        Args:
+            window_name: 窗口名称，None 则获取当前活动窗口
+            max_depth: 最大遍历深度
+
+        Returns:
+            树结构的字典表示
+        """
+        self._ensure_available()
+
+        if window_name:
+            root = auto.WindowControl(searchDepth=1, Name=window_name)
+        else:
+            root = auto.GetForegroundControl()
+
+        if not root.Exists():
+            return {}
+
+        def build_tree(control, depth: int) -> Dict:
+            if depth > max_depth:
+                return {"name": control.Name, "type": control.ControlTypeName, "truncated": True}
+
+            children = []
+            for child in control.GetChildren():
+                children.append(build_tree(child, depth + 1))
+
+            return {
+                "name": control.Name,
+                "type": control.ControlTypeName,
+                "automation_id": control.AutomationId,
+                "class_name": control.ClassName,
+                "rect": control.BoundingRectangle,
+                "children": children
+            }
+
+        return build_tree(root, 1)
+
+    # ========== 元素操作功能 ==========
+
+    def click_element(self, name: str, control_type: str = None,
+                      parent_name: str = None) -> bool:
+        """
+        点击指定元素。
+
+        Args:
+            name: 元素名称
+            control_type: 控件类型
+            parent_name: 父窗口名称
+
+        Returns:
+            是否成功
+        """
+        self._ensure_available()
+
+        element = self.find_element(name, control_type, parent_name)
+        if element is None:
+            return False
+
+        element.Click()
+        return True
+
+    def send_keys_to_element(self, name: str, text: str,
+                             control_type: str = "Edit",
+                             parent_name: str = None) -> bool:
+        """
+        向输入框发送文本。
+
+        Args:
+            name: 元素名称
+            text: 要发送的文本
+            control_type: 控件类型，默认 Edit
+            parent_name: 父窗口名称
+
+        Returns:
+            是否成功
+        """
+        self._ensure_available()
+
+        element = self.find_element(name, control_type, parent_name)
+        if element is None:
+            return False
+
+        element.SendKeys(text)
+        return True
+
+    def get_element_text(self, name: str, control_type: str = None,
+                         parent_name: str = None) -> str:
+        """
+        获取元素的文本内容。
+
+        Args:
+            name: 元素名称
+            control_type: 控件类型
+            parent_name: 父窗口名称
+
+        Returns:
+            文本内容
+        """
+        self._ensure_available()
+
+        element = self.find_element(name, control_type, parent_name)
+        if element is None:
+            return ""
+
+        return element.Name or ""
+
+    def wait_for_element(self, name: str, control_type: str = None,
+                         parent_name: str = None, timeout: int = 10) -> bool:
+        """
+        等待元素出现。
+
+        Args:
+            name: 元素名称
+            control_type: 控件类型
+            parent_name: 父窗口名称
+            timeout: 超时时间（秒）
+
+        Returns:
+            是否找到
+        """
+        self._ensure_available()
+
+        element = self.find_element(name, control_type, parent_name)
+        if element is None:
+            return False
+        return element.Exists(timeout)
+
+    # ========== 窗口操作 ==========
+
+    def get_window_list(self) -> List[Dict[str, str]]:
+        """
+        获取所有顶层窗口列表。
+
+        Returns:
+            窗口信息列表
+        """
+        self._ensure_available()
+
+        windows = []
+        root = auto.GetRootControl()
+        for window in root.GetChildren():
+            if window.ControlType == auto.ControlType.WindowControl:
+                windows.append({
+                    "name": window.Name,
+                    "class": window.ClassName,
+                    "handle": window.NativeWindowHandle,
+                    "rect": window.BoundingRectangle
+                })
+        return windows
+
+    def activate_window(self, window_name: str) -> bool:
+        """
+        激活指定窗口。
+
+        Args:
+            window_name: 窗口名称
+
+        Returns:
+            是否成功
+        """
+        self._ensure_available()
+
+        window = auto.WindowControl(searchDepth=1, Name=window_name)
+        if window.Exists():
+            window.SwitchToThisWindow()
+            return True
+        return False
+
+
+# 全局实例
+UIA = UIATools()