PyClickerRPA 0.3.1__tar.gz → 0.4.0__tar.gz

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: PyClickerRPA
-Version: 0.3.1
-Summary: Visual UI automation & testing tool: Windows + Android(ADB) automation, robust/self-healing image matching, OCR, node flow editor, recorder, a visual test framework (assertions, visual regression, HTML/JUnit reports) and a web console.
+Version: 0.4.0
+Summary: Visual UI automation & testing tool: Windows + Android(ADB) automation, robust/self-healing image matching, OCR, node flow editor, a smart recorder that records images instead of coordinates, a visual test framework (assertions, visual regression, HTML/JUnit reports) and a web console.
 Author: Equinox
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/Equinox/PyClickerRPA
@@ -82,7 +82,7 @@ autoclicker-web --suites-dir examples          # web console at http://127.0.0.1
 - **Smart wait**: `wait_stable` waits until the screen (or an ROI) stops
   changing — replaces guessed `sleep`s (fast when ready, patient when slow).
 - **~30+ step types**: click image/text/position, click-all, relative/offset
-  clicks, wait / random / wait-stable, drag / long-press / pinch / scroll,
+  clicks, wait / random / wait-stable, drag / **drag-from-image** / long-press / pinch / scroll,
   key / type / hotkey, loop / goto / while / do-until, if image/text/color/
   count/var with **true/false branching** (if-else), set-var, read-text→variable,
   activate-window, HTTP request, run-Python-script, call sub-task, groups, and a
@@ -93,12 +93,33 @@ autoclicker-web --suites-dir examples          # web console at http://127.0.0.1
 Drag-and-drop **node flow graph** (condition nodes have **true/false branch
 ports** for visual if-else; a terminal **stop** node ends the flow) and an
 interoperable **step list** with a full
-per-step dialog; an action **recorder** (global hooks), snippet library, task
+per-step dialog; snippet library, task
 packages (`.zip` with templates), on-preview coordinate/color picking, template
 crop, single-step test run; global hotkeys (F6 run/stop, ESC stop), scheduler,
 and an image/color **monitor/trigger** with tray icon; a **Test** tab to load and
 run visual test suites.
 
+### Smart recorder
+
+The action **recorder** (global hooks) doesn't just replay coordinates and
+sleeps — with **智能录制 / smart mode** on (default) it records what things *look
+like* instead of where they *were*:
+
+- **Clicks become image steps.** A patch is cropped around the click and grown
+  (48 → 192 px) until it is both textured enough and **unique on screen**, then
+  emitted as a `click_image` step — so the flow still works after the window
+  moves or the resolution changes. Identical patches are de-duplicated by
+  content hash.
+- **Drags become `drag_image`.** The drag's *start* is anchored to an image and
+  the *displacement* is stored, so start and end move together.
+- **Pauses become `wait_stable`.** A long pause is only converted if the
+  recorder actually **observed the screen settle** during it; on a permanently
+  animating screen it keeps the fixed wait (a `wait_stable` that never settles
+  would fail the run).
+
+Anything that can't be made robust falls back to the plain coordinate/fixed-wait
+step and says why in the log.
+
 ## Visual test framework (`autoclicker.testkit`)
 
 Turn automation into tests:

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/PyClickerRPA.egg-info/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: PyClickerRPA
-Version: 0.3.1
-Summary: Visual UI automation & testing tool: Windows + Android(ADB) automation, robust/self-healing image matching, OCR, node flow editor, recorder, a visual test framework (assertions, visual regression, HTML/JUnit reports) and a web console.
+Version: 0.4.0
+Summary: Visual UI automation & testing tool: Windows + Android(ADB) automation, robust/self-healing image matching, OCR, node flow editor, a smart recorder that records images instead of coordinates, a visual test framework (assertions, visual regression, HTML/JUnit reports) and a web console.
 Author: Equinox
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/Equinox/PyClickerRPA
@@ -82,7 +82,7 @@ autoclicker-web --suites-dir examples          # web console at http://127.0.0.1
 - **Smart wait**: `wait_stable` waits until the screen (or an ROI) stops
   changing — replaces guessed `sleep`s (fast when ready, patient when slow).
 - **~30+ step types**: click image/text/position, click-all, relative/offset
-  clicks, wait / random / wait-stable, drag / long-press / pinch / scroll,
+  clicks, wait / random / wait-stable, drag / **drag-from-image** / long-press / pinch / scroll,
   key / type / hotkey, loop / goto / while / do-until, if image/text/color/
   count/var with **true/false branching** (if-else), set-var, read-text→variable,
   activate-window, HTTP request, run-Python-script, call sub-task, groups, and a
@@ -93,12 +93,33 @@ autoclicker-web --suites-dir examples          # web console at http://127.0.0.1
 Drag-and-drop **node flow graph** (condition nodes have **true/false branch
 ports** for visual if-else; a terminal **stop** node ends the flow) and an
 interoperable **step list** with a full
-per-step dialog; an action **recorder** (global hooks), snippet library, task
+per-step dialog; snippet library, task
 packages (`.zip` with templates), on-preview coordinate/color picking, template
 crop, single-step test run; global hotkeys (F6 run/stop, ESC stop), scheduler,
 and an image/color **monitor/trigger** with tray icon; a **Test** tab to load and
 run visual test suites.
 
+### Smart recorder
+
+The action **recorder** (global hooks) doesn't just replay coordinates and
+sleeps — with **智能录制 / smart mode** on (default) it records what things *look
+like* instead of where they *were*:
+
+- **Clicks become image steps.** A patch is cropped around the click and grown
+  (48 → 192 px) until it is both textured enough and **unique on screen**, then
+  emitted as a `click_image` step — so the flow still works after the window
+  moves or the resolution changes. Identical patches are de-duplicated by
+  content hash.
+- **Drags become `drag_image`.** The drag's *start* is anchored to an image and
+  the *displacement* is stored, so start and end move together.
+- **Pauses become `wait_stable`.** A long pause is only converted if the
+  recorder actually **observed the screen settle** during it; on a permanently
+  animating screen it keeps the fixed wait (a `wait_stable` that never settles
+  would fail the run).
+
+Anything that can't be made robust falls back to the plain coordinate/fixed-wait
+step and says why in the log.
+
 ## Visual test framework (`autoclicker.testkit`)
 
 Turn automation into tests:

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/PyClickerRPA.egg-info/SOURCES.txt

@@ -19,6 +19,7 @@ autoclicker/models.py
 autoclicker/ocr.py
 autoclicker/run.py
 autoclicker/runner.py
+autoclicker/templatize.py
 autoclicker/windows.py
 autoclicker/gui/__init__.py
 autoclicker/gui/flow_connection.py
@@ -56,7 +57,9 @@ tests/test_adb.py
 tests/test_flow_branch.py
 tests/test_matcher.py
 tests/test_models.py
+tests/test_recorder_gui.py
 tests/test_runner.py
+tests/test_templatize.py
 tests/test_testkit.py
 tests/test_testkit_gui.py
 tests/test_web.py

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/README.md

@@ -44,7 +44,7 @@ autoclicker-web --suites-dir examples          # web console at http://127.0.0.1
 - **Smart wait**: `wait_stable` waits until the screen (or an ROI) stops
   changing — replaces guessed `sleep`s (fast when ready, patient when slow).
 - **~30+ step types**: click image/text/position, click-all, relative/offset
-  clicks, wait / random / wait-stable, drag / long-press / pinch / scroll,
+  clicks, wait / random / wait-stable, drag / **drag-from-image** / long-press / pinch / scroll,
   key / type / hotkey, loop / goto / while / do-until, if image/text/color/
   count/var with **true/false branching** (if-else), set-var, read-text→variable,
   activate-window, HTTP request, run-Python-script, call sub-task, groups, and a
@@ -55,12 +55,33 @@ autoclicker-web --suites-dir examples          # web console at http://127.0.0.1
 Drag-and-drop **node flow graph** (condition nodes have **true/false branch
 ports** for visual if-else; a terminal **stop** node ends the flow) and an
 interoperable **step list** with a full
-per-step dialog; an action **recorder** (global hooks), snippet library, task
+per-step dialog; snippet library, task
 packages (`.zip` with templates), on-preview coordinate/color picking, template
 crop, single-step test run; global hotkeys (F6 run/stop, ESC stop), scheduler,
 and an image/color **monitor/trigger** with tray icon; a **Test** tab to load and
 run visual test suites.
 
+### Smart recorder
+
+The action **recorder** (global hooks) doesn't just replay coordinates and
+sleeps — with **智能录制 / smart mode** on (default) it records what things *look
+like* instead of where they *were*:
+
+- **Clicks become image steps.** A patch is cropped around the click and grown
+  (48 → 192 px) until it is both textured enough and **unique on screen**, then
+  emitted as a `click_image` step — so the flow still works after the window
+  moves or the resolution changes. Identical patches are de-duplicated by
+  content hash.
+- **Drags become `drag_image`.** The drag's *start* is anchored to an image and
+  the *displacement* is stored, so start and end move together.
+- **Pauses become `wait_stable`.** A long pause is only converted if the
+  recorder actually **observed the screen settle** during it; on a permanently
+  animating screen it keeps the fixed wait (a `wait_stable` that never settles
+  would fail the run).
+
+Anything that can't be made robust falls back to the plain coordinate/fixed-wait
+step and says why in the log.
+
 ## Visual test framework (`autoclicker.testkit`)
 
 Turn automation into tests:

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/autoclicker/gui/main_window.py

@@ -12,10 +12,10 @@ import tempfile
 import uuid
 import zipfile
 
-from PySide6.QtCore import Qt, QThread, Signal
+from PySide6.QtCore import Qt, QSettings, QThread, Signal
 from PySide6.QtGui import QAction, QGuiApplication, QKeySequence
 from PySide6.QtWidgets import (
-    QComboBox, QDockWidget, QFileDialog, QHBoxLayout, QInputDialog, QLabel,
+    QCheckBox, QComboBox, QDockWidget, QFileDialog, QHBoxLayout, QInputDialog, QLabel,
     QMainWindow, QMessageBox, QPushButton, QSplitter, QTableWidget,
     QTableWidgetItem, QTabWidget, QTextEdit, QVBoxLayout, QWidget,
 )
@@ -69,8 +69,11 @@ class MainWindow(QMainWindow):
         self._runner.runFinished.connect(self._on_finished)
         self._runner.pausedAt.connect(self._on_paused)
 
+        self._settings = QSettings("AutoClicker", "AutoClicker")
+
         self._recorder = Recorder(self)
         self._recorder.stepRecorded.connect(self._on_step_recorded)
+        self._recorder.note.connect(self._append_log)
 
         self._build_ui()
         self._build_menu()
@@ -229,6 +232,16 @@ class MainWindow(QMainWindow):
         capture_btn.clicked.connect(self.capture)
         self.record_btn = QPushButton("● 录制")
         self.record_btn.clicked.connect(self.toggle_record)
+        self.template_mode_chk = QCheckBox("智能录制")
+        self.template_mode_chk.setToolTip(
+            "录出来的任务尽量不依赖绝对坐标和绝对时间:\n"
+            "· 点击处裁一小块图当模板 → 位置无关的「找图点击」\n"
+            "· 长间隔改成「等画面稳定」→ 换台机器也不会等不够\n"
+            "抽不出好模板、或录制时画面一直在动,则各自退回坐标点击/固定等待。")
+        self.template_mode_chk.setChecked(
+            self._settings.value("recorder/templateMode", True, type=bool))
+        self.template_mode_chk.toggled.connect(
+            lambda on: self._settings.setValue("recorder/templateMode", on))
         self.run_btn = QPushButton("▶ 运行")
         self.run_btn.clicked.connect(self.run_task)
         self.pause_btn = QPushButton("⏸ 暂停")
@@ -250,6 +263,7 @@ class MainWindow(QMainWindow):
         bar.addWidget(refresh_btn)
         bar.addWidget(capture_btn)
         bar.addStretch()
+        bar.addWidget(self.template_mode_chk)
         bar.addWidget(self.record_btn)
         bar.addWidget(self.run_btn)
         bar.addWidget(self.pause_btn)
@@ -513,10 +527,12 @@ class MainWindow(QMainWindow):
             self._append_log("录制结束")
             return
         hwnd = self._target_hwnd()
-        self._recorder.start(hwnd)
+        tmpl = self.template_mode_chk.isChecked()
+        self._recorder.start(hwnd, template_mode=tmpl)
         self.record_btn.setText("■ 停止录制")
         self.tabs.setCurrentWidget(self.task_widget)
-        self._append_log("开始录制(在目标窗口内操作;再次点击结束)")
+        self._append_log("开始录制(在目标窗口内操作;再次点击结束)"
+                         + ("[智能录制]" if tmpl else ""))
 
     def _on_step_recorded(self, step: Step):
         self.task_widget.append_step(step)

pyclickerrpa-0.4.0/autoclicker/gui/recorder.py

@@ -0,0 +1,290 @@
+"""Action recorder — port of recorder.cpp using pynput global hooks.
+
+Records left clicks / drags (client-relative to the target window) and key
+presses into Steps, inserting Wait steps for gaps >= 150ms. Emits stepRecorded
+from the listener thread; the signal is delivered to the GUI thread by Qt.
+
+In *template mode* a click additionally gets a small image patch cropped around
+it and becomes a position-independent ClickImage step (see `templatize`); when no
+usable template can be cut it falls back to the plain ClickPosition step.
+"""
+from __future__ import annotations
+
+import math
+import time
+
+from PySide6.QtCore import QObject, QTimer, Signal
+
+from autoclicker.models import Step, StepType
+
+try:
+    import win32gui
+    _HAS_WIN32 = True
+except ImportError:
+    _HAS_WIN32 = False
+
+from pynput import mouse, keyboard
+
+# pynput special key -> our key name (mirrors Recorder::vkToName specials)
+_SPECIAL_KEYS = {
+    keyboard.Key.enter: "ENTER", keyboard.Key.esc: "ESC", keyboard.Key.tab: "TAB",
+    keyboard.Key.space: "SPACE", keyboard.Key.backspace: "BACKSPACE",
+    keyboard.Key.delete: "DELETE", keyboard.Key.up: "UP", keyboard.Key.down: "DOWN",
+    keyboard.Key.left: "LEFT", keyboard.Key.right: "RIGHT", keyboard.Key.home: "HOME",
+    keyboard.Key.end: "END", keyboard.Key.page_up: "PAGEUP", keyboard.Key.page_down: "PAGEDOWN",
+}
+_MODIFIERS = {
+    keyboard.Key.ctrl, keyboard.Key.ctrl_l, keyboard.Key.ctrl_r,
+    keyboard.Key.shift, keyboard.Key.shift_l, keyboard.Key.shift_r,
+    keyboard.Key.alt, keyboard.Key.alt_l, keyboard.Key.alt_r, keyboard.Key.alt_gr,
+    keyboard.Key.cmd, keyboard.Key.cmd_l, keyboard.Key.cmd_r,
+}
+
+
+class Recorder(QObject):
+    stepRecorded = Signal(object)   # emits a Step
+    recordingStopped = Signal()
+    note = Signal(str)              # 图像模式的取模板结果/回退原因，接到主窗口日志
+
+    # 抓帧间隔。只是为了在点击时手里有一张"刚刚"的画面，不需要很快
+    GRAB_INTERVAL_MS = 120
+
+    # 智能等待：只有间隔够长(人是真在等界面)、且录制时**观察到画面确实静下来过**，
+    # 才把固定 Wait 换成 WaitStable。有持续动画的画面永远稳不下来，
+    # 那种地方换成 WaitStable 会超时 Fail，比固定等待更糟。
+    STABLE_TOL = 0.005          # 和 runner._exec_wait_stable 的默认容差一致
+    STABLE_MIN_GAP_MS = 700     # 短于此的间隔只是手速，不是在等界面
+    STABLE_MIN_QUIET_MS = 250   # 至少观察到这么久的静止，才敢认为它能稳
+    STABLE_MAX_MS = 500         # 运行时要求的稳定时长上限(= Step.stableMs 默认值)
+
+    def __init__(self, parent=None):
+        super().__init__(parent)
+        self._hwnd = None
+        self._recording = False
+        self._mouse_listener = None
+        self._key_listener = None
+        self._t0 = 0.0
+        self._last_event = None
+        self._down = False
+        self._down_pos = (0, 0)
+        self._down_time = 0.0
+        self._template_mode = False
+        self._grab = None
+        self._frame = None        # 最近抓到的一帧
+        self._down_frame = None   # 按下那一刻的帧(点击前的画面)
+        self._quiet_since = None  # 画面连续静止的起点(elapsed ms)，动了就清空
+        self._timer = QTimer(self)
+        self._timer.timeout.connect(self._grab_frame)
+
+    def is_recording(self) -> bool:
+        return self._recording
+
+    def start(self, hwnd, *, template_mode: bool = False, grab=None):
+        """grab 可注入(默认抓目标窗口客户区/全屏)，为的是离屏测试能塞假帧。"""
+        if self._recording:
+            return
+        self._hwnd = hwnd
+        self._recording = True
+        self._template_mode = template_mode
+        self._grab = grab or self._default_grab
+        self._frame = None
+        self._down_frame = None
+        self._quiet_since = None
+        self._t0 = time.monotonic()
+        self._last_event = None
+        self._down = False
+        self._mouse_listener = mouse.Listener(on_click=self._on_click)
+        self._key_listener = keyboard.Listener(on_press=self._on_press)
+        self._mouse_listener.start()
+        self._key_listener.start()
+        if template_mode:
+            self._grab_frame()   # 先垫一帧，别让第一次点击没图可用
+            self._timer.start(self.GRAB_INTERVAL_MS)
+
+    def stop(self):
+        if not self._recording:
+            return
+        self._recording = False
+        self._timer.stop()
+        if self._mouse_listener:
+            self._mouse_listener.stop()
+            self._mouse_listener = None
+        if self._key_listener:
+            self._key_listener.stop()
+            self._key_listener = None
+        self._frame = None
+        self._down_frame = None
+        self.recordingStopped.emit()
+
+    # ---- 抓帧(跑在 Qt 线程的 QTimer 上) ----
+    def _default_grab(self):
+        from autoclicker import capture
+        return capture.capture_window(self._hwnd) if self._hwnd else capture.capture_screen()
+
+    def _grab_frame(self):
+        """故意不在 pynput 回调里抓图：回调跑在低级钩子线程上，PrintWindow 一次要几十毫秒，
+        阻塞太久 Windows 会直接把钩子摘掉，录制就莫名失灵了。"""
+        try:
+            frame = self._grab()
+        except Exception as e:
+            self._frame = None
+            self._quiet_since = None
+            self.note.emit(f"图像模式抓屏失败: {e}")
+            return
+        frame = frame if frame is not None and frame.size else None
+        prev, self._frame = self._frame, frame
+        if frame is None:
+            self._quiet_since = None
+            return
+        # 顺手记录画面有没有静下来 —— 这是后面决定能不能用 WaitStable 的依据
+        from autoclicker import matcher
+        if prev is None or matcher.diff_ratio(prev, frame) > self.STABLE_TOL:
+            self._quiet_since = None
+        elif self._quiet_since is None:
+            self._quiet_since = self._elapsed_ms()
+
+    def _click_step(self, cx: int, cy: int) -> Step:
+        """图像模式下尽量产出位置无关的 ClickImage，抽不出好模板就退回坐标点击。"""
+        if not self._template_mode:
+            return Step(type=StepType.ClickPosition, x=cx, y=cy)
+        frame = self._down_frame
+        if frame is None:
+            self.note.emit(f"({cx},{cy}) 没抓到画面 → 退回坐标点击")
+            return Step(type=StepType.ClickPosition, x=cx, y=cy)
+        from autoclicker import templatize
+        try:
+            choice = templatize.pick_template(frame, cx, cy)
+            if choice is None:
+                self.note.emit(f"({cx},{cy}) 该处不够独特(太平淡或全屏有多处相同) → "
+                               f"退回坐标点击;可事后用「裁剪模板」手工指定")
+                return Step(type=StepType.ClickPosition, x=cx, y=cy)
+            path = templatize.save_template(choice.patch)
+            step = templatize.build_click_image_step(path, choice.off_x, choice.off_y)
+        except Exception as e:
+            self.note.emit(f"({cx},{cy}) 取模板失败({e}) → 退回坐标点击")
+            return Step(type=StepType.ClickPosition, x=cx, y=cy)
+        self.note.emit(f"({cx},{cy}) 已取模板 {choice.size}px 纹理{choice.score:.0f} → {path}")
+        return step
+
+    def _drag_step(self, cx: int, cy: int, duration_ms: int) -> Step:
+        """智能录制下把拖拽的**起点**图像化(DragImage 存的是位移，所以终点跟着起点走);
+        起点抽不出模板就退回绝对坐标的 Drag。"""
+        sx, sy = self._down_pos
+        plain = Step(type=StepType.Drag, x=sx, y=sy, x2=cx, y2=cy, durationMs=duration_ms)
+        if not self._template_mode or self._down_frame is None:
+            return plain
+        from autoclicker import templatize
+        try:
+            choice = templatize.pick_template(self._down_frame, sx, sy)
+            if choice is None:
+                self.note.emit(f"拖拽起点({sx},{sy})不够独特 → 退回坐标拖拽")
+                return plain
+            path = templatize.save_template(choice.patch)
+        except Exception as e:
+            self.note.emit(f"拖拽起点取模板失败({e}) → 退回坐标拖拽")
+            return plain
+        self.note.emit(f"拖拽起点({sx},{sy}) 已取模板 {choice.size}px → 位移({cx - sx},{cy - sy})")
+        return Step(type=StepType.DragImage, imagePath=path,
+                    x=choice.off_x, y=choice.off_y,
+                    x2=cx - sx, y2=cy - sy, durationMs=duration_ms)
+
+    # ---- helpers ----
+    def _elapsed_ms(self) -> float:
+        return (time.monotonic() - self._t0) * 1000.0
+
+    def _inside(self, x, y) -> bool:
+        if not _HAS_WIN32 or not self._hwnd:
+            return True
+        try:
+            l, t, r, b = win32gui.GetWindowRect(self._hwnd)
+        except Exception:
+            return False
+        return l <= x <= r and t <= y <= b
+
+    def _to_client(self, x, y):
+        if _HAS_WIN32 and self._hwnd:
+            try:
+                return win32gui.ScreenToClient(self._hwnd, (x, y))
+            except Exception:
+                return x, y
+        return x, y
+
+    def _emit_wait_if_needed(self):
+        now = self._elapsed_ms()
+        # 用 None 而不是 0 表示"还没有上一个事件"：Windows 的 monotonic 精度约 15.6ms,
+        # 第一次操作和 start() 落在同一刻度时 elapsed 恰好是 0.0,用 >0 判断会把
+        # 第一个间隔静默丢掉。
+        if self._last_event is not None:
+            gap = int(now - self._last_event)
+            if gap >= 150:
+                self.stepRecorded.emit(self._wait_step(gap, now))
+        self._last_event = now
+
+    def _wait_step(self, gap: int, now: float) -> Step:
+        """长间隔 + 录制时确实看到画面静下来过 → 等画面稳定(换台机器也不会等不够);
+        否则照旧记录固定间隔。"""
+        quiet = self._quiet_since
+        if not self._template_mode or gap < self.STABLE_MIN_GAP_MS or quiet is None:
+            return Step(type=StepType.Wait, waitMs=gap)
+        quiet_ms = int(now - quiet)
+        if quiet_ms < self.STABLE_MIN_QUIET_MS:
+            return Step(type=StepType.Wait, waitMs=gap)
+        # 运行时要求的稳定时长不超过我们实际观察到的，免得它永远等不到
+        stable_ms = max(200, min(self.STABLE_MAX_MS, quiet_ms))
+        timeout_ms = min(15000, max(3000, gap * 3))
+        self.note.emit(f"间隔 {gap}ms 且画面静止过 {quiet_ms}ms → 改为等画面稳定"
+                       f"(稳定 {stable_ms}ms / 超时 {timeout_ms}ms)")
+        return Step(type=StepType.WaitStable, stableMs=stable_ms, timeoutMs=timeout_ms)
+
+    # ---- pynput callbacks (run on listener thread) ----
+    def _on_click(self, x, y, button, pressed):
+        if not self._recording or button != mouse.Button.left:
+            return
+        if not self._inside(x, y):
+            return
+        cx, cy = self._to_client(x, y)
+        if pressed:
+            self._down = True
+            self._down_pos = (cx, cy)
+            self._down_time = self._elapsed_ms()
+            # 按下这一刻的画面才是"点击前"的样子；用释放时的帧会抓到已经按下去
+            # 或者菜单已经弹出来的状态，裁出来的模板就没用了
+            self._down_frame = self._frame
+        elif self._down:
+            self._down = False
+            dx, dy = cx - self._down_pos[0], cy - self._down_pos[1]
+            dist = math.hypot(dx, dy)
+            self._emit_wait_if_needed()
+            if dist > 10:
+                dur = int(self._elapsed_ms() - self._down_time)
+                self.stepRecorded.emit(self._drag_step(cx, cy, dur if dur >= 50 else 200))
+            else:
+                self.stepRecorded.emit(self._click_step(cx, cy))
+
+    def _on_press(self, key):
+        if not self._recording or key in _MODIFIERS:
+            return
+        if _HAS_WIN32 and self._hwnd:
+            try:
+                if win32gui.GetForegroundWindow() != self._hwnd:
+                    return
+            except Exception:
+                pass
+        name = self._key_name(key)
+        if name:
+            self._emit_wait_if_needed()
+            self.stepRecorded.emit(Step(type=StepType.KeyPress, keyName=name))
+
+    @staticmethod
+    def _key_name(key) -> str:
+        if key in _SPECIAL_KEYS:
+            return _SPECIAL_KEYS[key]
+        if isinstance(key, keyboard.Key):
+            k = str(key).replace("Key.", "")
+            if k.startswith("f") and k[1:].isdigit():
+                return k.upper()
+            return ""
+        ch = getattr(key, "char", None)
+        if ch and (ch.isalnum()):
+            return ch.upper()
+        return ""

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/autoclicker/gui/task_widget.py

@@ -50,6 +50,7 @@ TYPE_LABELS = [
     (StepType.WaitImage, "等待图片出现"),
     (StepType.WaitImageGone, "等待图片消失"),
     (StepType.MergeImage, "合成拖拽"),
+    (StepType.DragImage, "找图拖拽"),
     (StepType.ClickAll, "点击所有匹配"),
     (StepType.ClickRelative, "相对点击(last+偏移)"),
     (StepType.ReadText, "读取文字→变量"),
@@ -65,13 +66,13 @@ TYPE_LABELS = [
 ]
 
 _IMAGE_TYPES = {StepType.ClickImage, StepType.WaitImage, StepType.WaitImageGone,
-                StepType.MergeImage, StepType.ClickAll, StepType.IfImageFound,
-                StepType.IfImageCount}
+                StepType.MergeImage, StepType.DragImage, StepType.ClickAll,
+                StepType.IfImageFound, StepType.IfImageCount}
 _IF_TYPES = {StepType.IfImageFound, StepType.IfTextFound, StepType.IfElse,
              StepType.IfColorMatch, StepType.IfImageCount, StepType.IfVar}
 _ACTION_TYPES = {StepType.ClickImage, StepType.ClickText, StepType.ClickPosition,
                  StepType.CallSub, StepType.WaitImage, StepType.WaitImageGone,
-                 StepType.MergeImage, StepType.ClickAll}
+                 StepType.MergeImage, StepType.DragImage, StepType.ClickAll}
 _UNTIL_TYPES = {StepType.DoUntil, StepType.WhileCondition}
 
 # 变量比较运算符 (显示标签, 存储 token)。前 6 个 label==token 兼容旧任务。
@@ -174,8 +175,16 @@ class StepEditorDialog(QDialog):
             mm.setCurrentIndex({"robust": 0, "template": 1, "feature": 2}.get(s.matchMode, 0))
             self.w["matchMode"] = mm
             self.form.addRow("匹配方式", mm)
-        if t in (StepType.ClickImage, StepType.WaitImage, StepType.WaitImageGone, StepType.MergeImage):
+        if t in (StepType.ClickImage, StepType.WaitImage, StepType.WaitImageGone,
+                 StepType.MergeImage, StepType.DragImage):
             self.form.addRow("超时(ms)", self._spin("timeoutMs", s.timeoutMs, 0, 3600000))
+        if t == StepType.DragImage:
+            # 存的是相对命中点的位移,不是终点绝对坐标 —— 窗口移位时终点会跟着走
+            self.form.addRow("起点微调 X", self._spin("x", s.x))
+            self.form.addRow("起点微调 Y", self._spin("y", s.y))
+            self.form.addRow("拖拽位移 X2", self._spin("x2", s.x2))
+            self.form.addRow("拖拽位移 Y2", self._spin("y2", s.y2))
+            self.form.addRow("拖拽时长(ms)", self._spin("durationMs", s.durationMs, 0, 60000))
         if t == StepType.IfImageCount:
             self.form.addRow("最少匹配数", self._spin("minMatchCount", s.minMatchCount, 1, 100))
         if t == StepType.ClickAll:

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/autoclicker/models.py

@@ -51,6 +51,7 @@ class StepType(str, Enum):
     ClickRelative = "click_relative"
     HttpRequest = "http_request"
     RunScript = "run_script"
+    DragImage = "drag_image"       # 找到起点模板后按 x2/y2 位移拖拽(位置无关的拖拽)
     WaitStable = "wait_stable"     # 等画面(或 ROI)稳定后继续
     ActivateWindow = "activate_window"  # 刷新/激活目标窗口(点击会切前台窗口时用)
     Stop = "stop"                  # 结束/停止任务(终止循环)
@@ -433,6 +434,8 @@ class Step:
         elif t == StepType.ReadText:
             mode = "数字" if self.varValue == "number" else "文字"
             base = f"读取{mode}→变量: {self.varName}"
+        elif t == StepType.DragImage:
+            base = f"找图拖拽: {fname(self.imagePath)} →偏移({self.x2},{self.y2})"
         elif t == StepType.ClickRelative:
             base = f"相对点击: last+({self.x},{self.y})"
         elif t == StepType.HttpRequest:

{pyclickerrpa-0.3.1 → pyclickerrpa-0.4.0}/autoclicker/runner.py

@@ -504,6 +504,8 @@ class TaskRunner(threading.Thread):
             self.cb.log(f"  相对点击 last+({step.x},{step.y}) = ({cx}, {cy})")
             self.clicker.click(self.target_hwnd, cx, cy)
             return StepResult.Success
+        if t == StepType.DragImage:
+            return self._exec_drag_image(step)
         if t == StepType.WaitStable:
             return self._exec_wait_stable(step)
         if t == StepType.ActivateWindow:
@@ -729,6 +731,42 @@ class TaskRunner(threading.Thread):
         self._sleep_ms(200)   # 给前台切换一点时间
         return StepResult.Success
 
+    def _exec_drag_image(self, step: Step) -> StepResult:
+        """找到起点模板，再从命中点按 x2/y2 的**位移**拖拽。
+
+        存位移而不是终点绝对坐标，这样窗口整体移位时起点和终点一起跟着走。
+        """
+        templates = self._template_list(step)
+        if not templates:
+            self.cb.log("  错误: 未指定图片路径")
+            return StepResult.Fail
+        templs = [(img, mask) for (img, mask) in
+                  (self._load_template(p, step.display_name()) for p in templates) if img is not None]
+        if not templs:
+            return StepResult.Fail
+
+        start = _now_ms()
+        while self._running and _now_ms() - start < step.timeoutMs:
+            screen = self._screenshot()
+            for img, mask in templs:
+                results = self.matcher.find_image_roi(screen, img, step.threshold, step.roiRect,
+                                                      step.multiScale, mask=mask, mode=step.matchMode)
+                if not results:
+                    continue
+                best = results[0]
+                sx = self._rand_offset(best.cx + step.x, step.randomOffset)
+                sy = self._rand_offset(best.cy + step.y, step.randomOffset)
+                ex, ey = sx + step.x2, sy + step.y2
+                self.cb.match_found(sx, sy, best.confidence)
+                self.cb.log(f"  找图拖拽 ({sx},{sy}) → ({ex},{ey}) 置信度: {best.confidence:.3f}")
+                self.clicker.drag(self.target_hwnd, sx, sy, ex, ey, step.durationMs)
+                self._vars["last_x"] = str(ex)
+                self._vars["last_y"] = str(ey)
+                return StepResult.Success
+            self._sleep_ms(500)
+        self.cb.log("  超时: 未找到拖拽起点图片")
+        return StepResult.Fail
+
     def _exec_wait_stable(self, step: Step, tol: float = 0.005, interval_ms: int = 150) -> StepResult:
         """轮询截图，直到监视区域连续稳定 step.stableMs（相邻帧变化比例 <= tol）。
         超时(step.timeoutMs)仍未稳定则 Fail。"""