pctr 0.3.4__tar.gz → 0.3.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pctr-0.3.4/src/pctr.egg-info → pctr-0.3.5}/PKG-INFO +63 -10
- {pctr-0.3.4 → pctr-0.3.5}/README.md +56 -5
- {pctr-0.3.4 → pctr-0.3.5}/pyproject.toml +7 -6
- pctr-0.3.5/src/pctr/__init__.py +1 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr/cli.py +353 -15
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr/data/AGENTS.md +4 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr/data/SKILL.md +59 -5
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr/mcp_server.py +18 -3
- pctr-0.3.5/src/pctr/py.typed +0 -0
- {pctr-0.3.4 → pctr-0.3.5/src/pctr.egg-info}/PKG-INFO +63 -10
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr.egg-info/SOURCES.txt +1 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr.egg-info/requires.txt +3 -0
- pctr-0.3.4/src/pctr/__init__.py +0 -1
- {pctr-0.3.4 → pctr-0.3.5}/LICENSE +0 -0
- {pctr-0.3.4 → pctr-0.3.5}/setup.cfg +0 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr/__main__.py +0 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr.egg-info/dependency_links.txt +0 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr.egg-info/entry_points.txt +0 -0
- {pctr-0.3.4 → pctr-0.3.5}/src/pctr.egg-info/top_level.txt +0 -0
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pctr
|
|
3
|
-
Version: 0.3.
|
|
3
|
+
Version: 0.3.5
|
|
4
4
|
Summary: Element-based Windows UI Automation CLI for AI agents. Find controls by name/type/id and click, type, hold, drag, and screenshot - no pixel coordinates.
|
|
5
|
-
Author:
|
|
5
|
+
Author: NTUserDev
|
|
6
6
|
License-Expression: MIT
|
|
7
|
-
Project-URL: Homepage, https://github.com/
|
|
8
|
-
Project-URL: Repository, https://github.com/
|
|
9
|
-
Project-URL: Issues, https://github.com/
|
|
7
|
+
Project-URL: Homepage, https://github.com/NTUserDev/pctr
|
|
8
|
+
Project-URL: Repository, https://github.com/NTUserDev/pctr
|
|
9
|
+
Project-URL: Issues, https://github.com/NTUserDev/pctr/issues
|
|
10
10
|
Keywords: automation,windows,uia,pywinauto,desktop,agent,computer-use,rpa
|
|
11
11
|
Classifier: Development Status :: 4 - Beta
|
|
12
12
|
Classifier: Environment :: Win32 (MS Windows)
|
|
@@ -31,6 +31,8 @@ Provides-Extra: vision
|
|
|
31
31
|
Requires-Dist: ultralytics>=8.2; extra == "vision"
|
|
32
32
|
Provides-Extra: mcp
|
|
33
33
|
Requires-Dist: mcp>=2; extra == "mcp"
|
|
34
|
+
Provides-Extra: all
|
|
35
|
+
Requires-Dist: pctr[mcp,vision]; extra == "all"
|
|
34
36
|
Dynamic: license-file
|
|
35
37
|
|
|
36
38
|
# pctr
|
|
@@ -46,11 +48,19 @@ mouse and keyboard input. On top of UIA it also offers **OCR**
|
|
|
46
48
|
(`Windows.Media.Ocr`) and **YOLO-World** open-vocabulary detection for text and
|
|
47
49
|
objects that aren't exposed as controls.
|
|
48
50
|
|
|
51
|
+

|
|
52
|
+
|
|
53
|
+
*Driven entirely by pctr itself - list windows, map virtual desktops, dump a
|
|
54
|
+
control tree, `set` a field, read it back with OCR, screenshot it, and YOLO-World
|
|
55
|
+
detect. Recorded with OBS, which pctr also piloted.*
|
|
56
|
+
|
|
49
57
|
## Install
|
|
50
58
|
|
|
51
59
|
```bash
|
|
52
|
-
pip install pctr
|
|
53
|
-
pip install "pctr[vision]" #
|
|
60
|
+
pip install pctr # core (UIA + OCR)
|
|
61
|
+
pip install "pctr[vision]" # + ultralytics, for `pctr look`
|
|
62
|
+
pip install "pctr[mcp]" # + MCP SDK, for the MCP server
|
|
63
|
+
pip install "pctr[all]" # everything
|
|
54
64
|
```
|
|
55
65
|
|
|
56
66
|
Windows only.
|
|
@@ -77,6 +87,18 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
|
|
|
77
87
|
`pctr_shot`, `pctr_ocr`, `pctr_ocrfind`, `pctr_ocrclick`, `pctr_look`,
|
|
78
88
|
`pctr_desktop`.
|
|
79
89
|
|
|
90
|
+
Or run/manage it from the CLI:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
pctr mcp status # SDK + server state
|
|
94
|
+
pctr mcp start --transport streamable-http --port 8765 # background HTTP server
|
|
95
|
+
pctr mcp restart | pctr mcp stop
|
|
96
|
+
pctr mcp help
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`stdio` runs in the foreground (what a client spawns); `sse` / `streamable-http`
|
|
100
|
+
run in the background, reachable at `http://127.0.0.1:8765/mcp` (or `…/sse`).
|
|
101
|
+
|
|
80
102
|
## Commands
|
|
81
103
|
|
|
82
104
|
| Command | What it does |
|
|
@@ -97,13 +119,17 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
|
|
|
97
119
|
| `pctr move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
|
|
98
120
|
| `pctr drag --start x,y --end x,y [--duration S]` | Drag between two points. |
|
|
99
121
|
| `pctr keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
|
|
100
|
-
| `pctr ocr [--title RE]` | OCR the screen/window; list words with boxes. |
|
|
101
|
-
| `pctr ocrfind --text RE` | OCR then list matches with click centers (single words). |
|
|
102
|
-
| `pctr ocrclick --text RE [--nth N]` | OCR then click a word. |
|
|
122
|
+
| `pctr ocr [--title RE] [--lang TAG]` | OCR the screen/window; list words with boxes. |
|
|
123
|
+
| `pctr ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers (single words). |
|
|
124
|
+
| `pctr ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
|
|
103
125
|
| `pctr look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
|
|
104
126
|
| `pctr lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
|
|
127
|
+
| `pctr mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
|
|
105
128
|
| `pctr skill [--install]` / `pctr setup` | Print/install the agent skill + an AGENTS.md section. |
|
|
106
129
|
|
|
130
|
+
Global flags: `--json` (machine-readable output) and `--file FILE` (run a batch
|
|
131
|
+
of commands from a file).
|
|
132
|
+
|
|
107
133
|
Common filters: `--title` (regex), `--name` (regex), `--control-type`
|
|
108
134
|
(`Button`, `Edit`, `Pane`, `MenuItem`, …), `--auto-id`, `--nth`, `--process`.
|
|
109
135
|
|
|
@@ -118,6 +144,33 @@ pctr click --title "Notepad" --name "^File$"
|
|
|
118
144
|
pctr set --title "Notepad" --name "Text Editor" --text "hello"
|
|
119
145
|
```
|
|
120
146
|
|
|
147
|
+
## JSON output
|
|
148
|
+
|
|
149
|
+
Add `--json` (global, or per-command) for machine-readable output from
|
|
150
|
+
`windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
pctr windows --json
|
|
154
|
+
# [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]
|
|
155
|
+
|
|
156
|
+
pctr desktop list --json
|
|
157
|
+
# [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]
|
|
158
|
+
|
|
159
|
+
pctr look --for "a car" --json
|
|
160
|
+
# [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
Progress and warnings go to stderr, so stdout stays valid JSON.
|
|
164
|
+
|
|
165
|
+
## Batch files
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
pctr --file script.txt # one command per line; # comments and blanks skipped
|
|
169
|
+
pctr --json --file script.txt # --json applies to every line
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Stops on the first failing line and reports its line number.
|
|
173
|
+
|
|
121
174
|
## Virtual desktops
|
|
122
175
|
|
|
123
176
|
```bash
|
|
@@ -11,11 +11,19 @@ mouse and keyboard input. On top of UIA it also offers **OCR**
|
|
|
11
11
|
(`Windows.Media.Ocr`) and **YOLO-World** open-vocabulary detection for text and
|
|
12
12
|
objects that aren't exposed as controls.
|
|
13
13
|
|
|
14
|
+

|
|
15
|
+
|
|
16
|
+
*Driven entirely by pctr itself - list windows, map virtual desktops, dump a
|
|
17
|
+
control tree, `set` a field, read it back with OCR, screenshot it, and YOLO-World
|
|
18
|
+
detect. Recorded with OBS, which pctr also piloted.*
|
|
19
|
+
|
|
14
20
|
## Install
|
|
15
21
|
|
|
16
22
|
```bash
|
|
17
|
-
pip install pctr
|
|
18
|
-
pip install "pctr[vision]" #
|
|
23
|
+
pip install pctr # core (UIA + OCR)
|
|
24
|
+
pip install "pctr[vision]" # + ultralytics, for `pctr look`
|
|
25
|
+
pip install "pctr[mcp]" # + MCP SDK, for the MCP server
|
|
26
|
+
pip install "pctr[all]" # everything
|
|
19
27
|
```
|
|
20
28
|
|
|
21
29
|
Windows only.
|
|
@@ -42,6 +50,18 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
|
|
|
42
50
|
`pctr_shot`, `pctr_ocr`, `pctr_ocrfind`, `pctr_ocrclick`, `pctr_look`,
|
|
43
51
|
`pctr_desktop`.
|
|
44
52
|
|
|
53
|
+
Or run/manage it from the CLI:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
pctr mcp status # SDK + server state
|
|
57
|
+
pctr mcp start --transport streamable-http --port 8765 # background HTTP server
|
|
58
|
+
pctr mcp restart | pctr mcp stop
|
|
59
|
+
pctr mcp help
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
`stdio` runs in the foreground (what a client spawns); `sse` / `streamable-http`
|
|
63
|
+
run in the background, reachable at `http://127.0.0.1:8765/mcp` (or `…/sse`).
|
|
64
|
+
|
|
45
65
|
## Commands
|
|
46
66
|
|
|
47
67
|
| Command | What it does |
|
|
@@ -62,13 +82,17 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
|
|
|
62
82
|
| `pctr move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
|
|
63
83
|
| `pctr drag --start x,y --end x,y [--duration S]` | Drag between two points. |
|
|
64
84
|
| `pctr keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
|
|
65
|
-
| `pctr ocr [--title RE]` | OCR the screen/window; list words with boxes. |
|
|
66
|
-
| `pctr ocrfind --text RE` | OCR then list matches with click centers (single words). |
|
|
67
|
-
| `pctr ocrclick --text RE [--nth N]` | OCR then click a word. |
|
|
85
|
+
| `pctr ocr [--title RE] [--lang TAG]` | OCR the screen/window; list words with boxes. |
|
|
86
|
+
| `pctr ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers (single words). |
|
|
87
|
+
| `pctr ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
|
|
68
88
|
| `pctr look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
|
|
69
89
|
| `pctr lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
|
|
90
|
+
| `pctr mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
|
|
70
91
|
| `pctr skill [--install]` / `pctr setup` | Print/install the agent skill + an AGENTS.md section. |
|
|
71
92
|
|
|
93
|
+
Global flags: `--json` (machine-readable output) and `--file FILE` (run a batch
|
|
94
|
+
of commands from a file).
|
|
95
|
+
|
|
72
96
|
Common filters: `--title` (regex), `--name` (regex), `--control-type`
|
|
73
97
|
(`Button`, `Edit`, `Pane`, `MenuItem`, …), `--auto-id`, `--nth`, `--process`.
|
|
74
98
|
|
|
@@ -83,6 +107,33 @@ pctr click --title "Notepad" --name "^File$"
|
|
|
83
107
|
pctr set --title "Notepad" --name "Text Editor" --text "hello"
|
|
84
108
|
```
|
|
85
109
|
|
|
110
|
+
## JSON output
|
|
111
|
+
|
|
112
|
+
Add `--json` (global, or per-command) for machine-readable output from
|
|
113
|
+
`windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
pctr windows --json
|
|
117
|
+
# [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]
|
|
118
|
+
|
|
119
|
+
pctr desktop list --json
|
|
120
|
+
# [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]
|
|
121
|
+
|
|
122
|
+
pctr look --for "a car" --json
|
|
123
|
+
# [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Progress and warnings go to stderr, so stdout stays valid JSON.
|
|
127
|
+
|
|
128
|
+
## Batch files
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
pctr --file script.txt # one command per line; # comments and blanks skipped
|
|
132
|
+
pctr --json --file script.txt # --json applies to every line
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Stops on the first failing line and reports its line number.
|
|
136
|
+
|
|
86
137
|
## Virtual desktops
|
|
87
138
|
|
|
88
139
|
```bash
|
|
@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "pctr"
|
|
7
|
-
version = "0.3.
|
|
7
|
+
version = "0.3.5"
|
|
8
8
|
description = "Element-based Windows UI Automation CLI for AI agents. Find controls by name/type/id and click, type, hold, drag, and screenshot - no pixel coordinates."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.10"
|
|
11
11
|
license = "MIT"
|
|
12
|
-
authors = [{ name = "
|
|
12
|
+
authors = [{ name = "NTUserDev" }]
|
|
13
13
|
keywords = ["automation", "windows", "uia", "pywinauto", "desktop", "agent", "computer-use", "rpa"]
|
|
14
14
|
classifiers = [
|
|
15
15
|
"Development Status :: 4 - Beta",
|
|
@@ -35,18 +35,19 @@ dependencies = [
|
|
|
35
35
|
[project.optional-dependencies]
|
|
36
36
|
vision = ["ultralytics>=8.2"]
|
|
37
37
|
mcp = ["mcp>=2"]
|
|
38
|
+
all = ["pctr[vision,mcp]"]
|
|
38
39
|
|
|
39
40
|
[project.scripts]
|
|
40
41
|
pctr = "pctr.cli:main"
|
|
41
42
|
pctr-mcp = "pctr.mcp_server:main"
|
|
42
43
|
|
|
43
44
|
[project.urls]
|
|
44
|
-
Homepage = "https://github.com/
|
|
45
|
-
Repository = "https://github.com/
|
|
46
|
-
Issues = "https://github.com/
|
|
45
|
+
Homepage = "https://github.com/NTUserDev/pctr"
|
|
46
|
+
Repository = "https://github.com/NTUserDev/pctr"
|
|
47
|
+
Issues = "https://github.com/NTUserDev/pctr/issues"
|
|
47
48
|
|
|
48
49
|
[tool.setuptools.packages.find]
|
|
49
50
|
where = ["src"]
|
|
50
51
|
|
|
51
52
|
[tool.setuptools.package-data]
|
|
52
|
-
pctr = ["data/*.md"]
|
|
53
|
+
pctr = ["data/*.md", "py.typed"]
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "0.3.5"
|
|
@@ -130,6 +130,29 @@ def _describe(el):
|
|
|
130
130
|
)
|
|
131
131
|
|
|
132
132
|
|
|
133
|
+
def _print_json(obj):
|
|
134
|
+
import json
|
|
135
|
+
print(json.dumps(obj, ensure_ascii=False))
|
|
136
|
+
|
|
137
|
+
|
|
138
|
+
def _is_json(args):
|
|
139
|
+
return bool(getattr(args, "json_out", False))
|
|
140
|
+
|
|
141
|
+
|
|
142
|
+
def _el_dict(el, depth=None):
|
|
143
|
+
info = el.element_info
|
|
144
|
+
r = info.rectangle
|
|
145
|
+
d = {
|
|
146
|
+
"name": info.name,
|
|
147
|
+
"type": info.control_type,
|
|
148
|
+
"id": info.automation_id,
|
|
149
|
+
"rect": [r.left, r.top, r.right, r.bottom],
|
|
150
|
+
}
|
|
151
|
+
if depth is not None:
|
|
152
|
+
d["depth"] = depth
|
|
153
|
+
return d
|
|
154
|
+
|
|
155
|
+
|
|
133
156
|
def _get_element(args):
|
|
134
157
|
win = _find_window(args.title, getattr(args, "process", None), getattr(args, "timeout", 5.0))
|
|
135
158
|
els = _matches(win, args.name, getattr(args, "control_type", None), getattr(args, "auto_id", None))
|
|
@@ -148,12 +171,18 @@ def _get_element(args):
|
|
|
148
171
|
def cmd_windows(args):
|
|
149
172
|
rx = _compile(args.filter, "filter") if args.filter else None
|
|
150
173
|
proc = getattr(args, "process", None)
|
|
174
|
+
rows = []
|
|
151
175
|
for hwnd, title, pid in _enum_top_windows():
|
|
152
176
|
if rx and rx.search(title) is None:
|
|
153
177
|
continue
|
|
154
178
|
if proc is not None and pid != proc:
|
|
155
179
|
continue
|
|
156
|
-
|
|
180
|
+
rows.append({"hwnd": hwnd, "pid": pid, "title": title})
|
|
181
|
+
if _is_json(args):
|
|
182
|
+
_print_json(rows)
|
|
183
|
+
return
|
|
184
|
+
for r in rows:
|
|
185
|
+
print("pid=%s | %s" % (r["pid"], r["title"]))
|
|
157
186
|
|
|
158
187
|
|
|
159
188
|
def _walk(root, max_depth):
|
|
@@ -180,27 +209,44 @@ def _walk(root, max_depth):
|
|
|
180
209
|
|
|
181
210
|
def cmd_tree(args):
|
|
182
211
|
win = _find_window(args.title, args.process, args.timeout)
|
|
183
|
-
print("WINDOW: %s (pid=%s)" % (win.window_text(), win.process_id()))
|
|
184
|
-
count = 0
|
|
185
212
|
depth = getattr(args, "depth", None)
|
|
186
213
|
depth_mode = depth is not None and 0 <= depth < 99
|
|
187
214
|
it = _walk(win, depth) if depth_mode else ((el, 1) for el in win.descendants())
|
|
215
|
+
rows = []
|
|
216
|
+
truncated = False
|
|
188
217
|
for el, d in it:
|
|
189
218
|
try:
|
|
190
219
|
el.element_info
|
|
191
220
|
except Exception:
|
|
192
221
|
continue
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
print("... (limit %d reached)" % args.limit)
|
|
222
|
+
rows.append(_el_dict(el, d if depth_mode else None))
|
|
223
|
+
if len(rows) >= args.limit:
|
|
224
|
+
truncated = True
|
|
197
225
|
break
|
|
226
|
+
if _is_json(args):
|
|
227
|
+
_print_json({
|
|
228
|
+
"window": win.window_text(),
|
|
229
|
+
"pid": win.process_id(),
|
|
230
|
+
"elements": rows,
|
|
231
|
+
"truncated": truncated,
|
|
232
|
+
})
|
|
233
|
+
return
|
|
234
|
+
print("WINDOW: %s (pid=%s)" % (win.window_text(), win.process_id()))
|
|
235
|
+
for r in rows:
|
|
236
|
+
prefix = " " * (r["depth"] - 1) if depth_mode else ""
|
|
237
|
+
print(" %s%s | type=%s | id=%s | rect=%d,%d,%d,%d" % (
|
|
238
|
+
prefix, r["name"], r["type"], r["id"], *r["rect"]))
|
|
239
|
+
if truncated:
|
|
240
|
+
print("... (limit %d reached)" % args.limit)
|
|
198
241
|
|
|
199
242
|
|
|
200
243
|
def cmd_find(args):
|
|
201
244
|
win = _find_window(args.title, args.process, args.timeout)
|
|
202
245
|
els = _matches(win, args.name, args.control_type, args.auto_id)
|
|
203
246
|
if not els:
|
|
247
|
+
if _is_json(args):
|
|
248
|
+
_print_json({"window": win.window_text(), "matches": [], "count": 0})
|
|
249
|
+
raise SystemExit(1)
|
|
204
250
|
print("no matches")
|
|
205
251
|
raise SystemExit(1)
|
|
206
252
|
nth = getattr(args, "nth", 0) or 0
|
|
@@ -208,9 +254,16 @@ def cmd_find(args):
|
|
|
208
254
|
if nth >= len(els):
|
|
209
255
|
_fail("matched %d element(s); nth=%d out of range" % (len(els), nth))
|
|
210
256
|
els = [els[nth]]
|
|
211
|
-
print("WINDOW: %s" % win.window_text())
|
|
212
257
|
limit = getattr(args, "limit", None)
|
|
213
258
|
shown = els[:limit] if limit else els
|
|
259
|
+
if _is_json(args):
|
|
260
|
+
_print_json({
|
|
261
|
+
"window": win.window_text(),
|
|
262
|
+
"count": len(els),
|
|
263
|
+
"matches": [dict(_el_dict(el), index=i) for i, el in enumerate(shown)],
|
|
264
|
+
})
|
|
265
|
+
return
|
|
266
|
+
print("WINDOW: %s" % win.window_text())
|
|
214
267
|
for i, el in enumerate(shown):
|
|
215
268
|
print("[%d] %s" % (i, _describe(el)))
|
|
216
269
|
if limit and len(els) > limit:
|
|
@@ -474,6 +527,12 @@ def _desktop_list(args):
|
|
|
474
527
|
counts = {}
|
|
475
528
|
for _hwnd, _title, d in _list_windows_with_desktop():
|
|
476
529
|
counts[d] = counts.get(d, 0) + 1
|
|
530
|
+
if _is_json(args):
|
|
531
|
+
_print_json([
|
|
532
|
+
{"index": i, "guid": d, "current": d == cur, "windows": counts.get(d, 0)}
|
|
533
|
+
for i, d in enumerate(guids)
|
|
534
|
+
])
|
|
535
|
+
return
|
|
477
536
|
for i, d in enumerate(guids):
|
|
478
537
|
print("#%d %s%s (%d windows)" % (i, d, " [current]" if d == cur else "", counts.get(d, 0)))
|
|
479
538
|
|
|
@@ -481,11 +540,17 @@ def _desktop_list(args):
|
|
|
481
540
|
def _desktop_windows(args):
|
|
482
541
|
guids = _vdesk_guids()
|
|
483
542
|
want = getattr(args, "desktop", None)
|
|
543
|
+
rows = []
|
|
484
544
|
for hwnd, title, d in _list_windows_with_desktop():
|
|
485
545
|
i = guids.index(d) if d in guids else -1
|
|
486
546
|
if want is not None and i != want:
|
|
487
547
|
continue
|
|
488
|
-
|
|
548
|
+
rows.append({"desktop": i, "hwnd": hwnd, "title": title})
|
|
549
|
+
if _is_json(args):
|
|
550
|
+
_print_json(rows)
|
|
551
|
+
return
|
|
552
|
+
for r in rows:
|
|
553
|
+
print("desktop=#%s | hwnd=%s | %s" % (r["desktop"], r["hwnd"], r["title"]))
|
|
489
554
|
|
|
490
555
|
|
|
491
556
|
def _desktop_where(args, bool_only):
|
|
@@ -498,11 +563,17 @@ def _desktop_where(args, bool_only):
|
|
|
498
563
|
hwnd = wintypes.HWND(int(h))
|
|
499
564
|
if bool_only:
|
|
500
565
|
on = vdm.IsWindowOnCurrentVirtualDesktop(hwnd)
|
|
566
|
+
if _is_json(args):
|
|
567
|
+
_print_json({"on_current": bool(on)})
|
|
568
|
+
return
|
|
501
569
|
print("on_current=%s" % ("True" if on else "False"))
|
|
502
570
|
return
|
|
503
571
|
guids = _vdesk_guids()
|
|
504
572
|
d = _guid_norm(vdm.GetWindowDesktopId(hwnd))
|
|
505
573
|
i = guids.index(d) if d in guids else -1
|
|
574
|
+
if _is_json(args):
|
|
575
|
+
_print_json({"desktop": i, "guid": d})
|
|
576
|
+
return
|
|
506
577
|
print("desktop=#%d %s" % (i, d))
|
|
507
578
|
|
|
508
579
|
|
|
@@ -596,12 +667,20 @@ $file = Await ([Windows.Storage.StorageFile]::GetFileFromPathAsync($Path)) ([Win
|
|
|
596
667
|
$stream = Await ($file.OpenAsync([Windows.Storage.FileAccessMode]::Read)) ([Windows.Storage.Streams.IRandomAccessStream])
|
|
597
668
|
$decoder = Await ([Windows.Graphics.Imaging.BitmapDecoder]::CreateAsync($stream)) ([Windows.Graphics.Imaging.BitmapDecoder])
|
|
598
669
|
$bitmap = Await ($decoder.GetSoftwareBitmapAsync()) ([Windows.Graphics.Imaging.SoftwareBitmap])
|
|
599
|
-
$engine =
|
|
600
|
-
|
|
670
|
+
$engine = $null
|
|
671
|
+
$lang = $env:PCTR_OCR_LANG
|
|
672
|
+
if ($lang) {
|
|
601
673
|
$null = [Windows.Globalization.Language,Windows.Foundation,ContentType=WindowsRuntime]
|
|
602
|
-
$engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language
|
|
674
|
+
$engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language $lang))
|
|
675
|
+
if (-not $engine) { Write-Error ("OCR language not available on this system: " + $lang); exit 2 }
|
|
676
|
+
} else {
|
|
677
|
+
$engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromUserProfileLanguages()
|
|
678
|
+
if (-not $engine) {
|
|
679
|
+
$null = [Windows.Globalization.Language,Windows.Foundation,ContentType=WindowsRuntime]
|
|
680
|
+
$engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language 'en-US'))
|
|
681
|
+
}
|
|
682
|
+
if (-not $engine) { Write-Error 'no OCR engine available'; exit 2 }
|
|
603
683
|
}
|
|
604
|
-
if (-not $engine) { Write-Error 'no OCR engine available'; exit 2 }
|
|
605
684
|
$res = Await ($engine.RecognizeAsync($bitmap)) ([Windows.Media.Ocr.OcrResult])
|
|
606
685
|
$out = foreach ($line in $res.Lines) { foreach ($w in $line.Words) { [pscustomobject]@{ text = $w.Text; x = [int]$w.BoundingRect.X; y = [int]$w.BoundingRect.Y; w = [int]$w.BoundingRect.Width; h = [int]$w.BoundingRect.Height } } }
|
|
607
686
|
ConvertTo-Json -Compress -InputObject @($out)
|
|
@@ -616,7 +695,7 @@ def _pctr_tempdir():
|
|
|
616
695
|
return d
|
|
617
696
|
|
|
618
697
|
|
|
619
|
-
def _windows_ocr(image_path):
|
|
698
|
+
def _windows_ocr(image_path, lang=None):
|
|
620
699
|
import base64
|
|
621
700
|
import json
|
|
622
701
|
import os
|
|
@@ -628,6 +707,10 @@ def _windows_ocr(image_path):
|
|
|
628
707
|
enc = base64.b64encode(ps.encode("utf-16-le")).decode("ascii")
|
|
629
708
|
env = dict(os.environ)
|
|
630
709
|
env["PCTR_OCR_PATH"] = image_path
|
|
710
|
+
if lang:
|
|
711
|
+
env["PCTR_OCR_LANG"] = lang
|
|
712
|
+
else:
|
|
713
|
+
env.pop("PCTR_OCR_LANG", None)
|
|
631
714
|
out = subprocess.run(
|
|
632
715
|
["powershell", "-NoProfile", "-NonInteractive", "-EncodedCommand", enc],
|
|
633
716
|
capture_output=True, text=True, env=env,
|
|
@@ -656,7 +739,7 @@ def _ocr_capture(args):
|
|
|
656
739
|
img = pyautogui.screenshot()
|
|
657
740
|
try:
|
|
658
741
|
img.save(path)
|
|
659
|
-
words = _windows_ocr(path)
|
|
742
|
+
words = _windows_ocr(path, getattr(args, "lang", None))
|
|
660
743
|
finally:
|
|
661
744
|
try:
|
|
662
745
|
os.remove(path)
|
|
@@ -667,6 +750,12 @@ def _ocr_capture(args):
|
|
|
667
750
|
|
|
668
751
|
def cmd_ocr(args):
|
|
669
752
|
words, ox, oy = _ocr_capture(args)
|
|
753
|
+
if _is_json(args):
|
|
754
|
+
_print_json([
|
|
755
|
+
{"text": w["text"], "x": ox + w["x"], "y": oy + w["y"], "w": w["w"], "h": w["h"]}
|
|
756
|
+
for w in words
|
|
757
|
+
])
|
|
758
|
+
return
|
|
670
759
|
for w in words:
|
|
671
760
|
print("%s | %d,%d %dx%d" % (w["text"], ox + w["x"], oy + w["y"], w["w"], w["h"]))
|
|
672
761
|
print("(%d words)" % len(words))
|
|
@@ -677,6 +766,13 @@ def cmd_ocrfind(args):
|
|
|
677
766
|
words, ox, oy = _ocr_capture(args)
|
|
678
767
|
rx = re.compile(args.text, re.I)
|
|
679
768
|
hits = [w for w in words if rx.search(w["text"])]
|
|
769
|
+
if _is_json(args):
|
|
770
|
+
_print_json([
|
|
771
|
+
{"text": w["text"], "x": ox + w["x"], "y": oy + w["y"], "w": w["w"], "h": w["h"],
|
|
772
|
+
"cx": ox + w["x"] + w["w"] // 2, "cy": oy + w["y"] + w["h"] // 2}
|
|
773
|
+
for w in hits
|
|
774
|
+
])
|
|
775
|
+
return
|
|
680
776
|
for i, w in enumerate(hits):
|
|
681
777
|
print("[%d] %s | center=%d,%d" % (i, w["text"], ox + w["x"] + w["w"] // 2, oy + w["y"] + w["h"] // 2))
|
|
682
778
|
if not hits:
|
|
@@ -762,6 +858,13 @@ def _look_detect(args):
|
|
|
762
858
|
|
|
763
859
|
def cmd_look(args):
|
|
764
860
|
dets = _look_detect(args)
|
|
861
|
+
if _is_json(args):
|
|
862
|
+
_print_json([
|
|
863
|
+
{"label": d["label"], "conf": round(d["conf"], 4), "x": d["x"], "y": d["y"],
|
|
864
|
+
"box": list(d["box"]), "index": i}
|
|
865
|
+
for i, d in enumerate(dets)
|
|
866
|
+
])
|
|
867
|
+
return
|
|
765
868
|
for i, d in enumerate(dets):
|
|
766
869
|
print("[%d] %s %.2f | center=%d,%d | box=%d,%d,%d,%d" % (i, d["label"], d["conf"], d["x"], d["y"], *d["box"]))
|
|
767
870
|
if not dets:
|
|
@@ -829,6 +932,200 @@ def cmd_setup(args):
|
|
|
829
932
|
print("done. restart your agent so it loads the skill.")
|
|
830
933
|
|
|
831
934
|
|
|
935
|
+
MCP_DEFAULT_PORT = 8765
|
|
936
|
+
|
|
937
|
+
|
|
938
|
+
def _mcp_statefile():
|
|
939
|
+
import os
|
|
940
|
+
return os.path.join(_pctr_tempdir(), "mcp.json")
|
|
941
|
+
|
|
942
|
+
|
|
943
|
+
def _pid_alive(pid):
|
|
944
|
+
import ctypes
|
|
945
|
+
k = ctypes.windll.kernel32
|
|
946
|
+
h = k.OpenProcess(0x1000, False, int(pid)) # PROCESS_QUERY_LIMITED_INFORMATION
|
|
947
|
+
if not h:
|
|
948
|
+
return False
|
|
949
|
+
k.CloseHandle(h)
|
|
950
|
+
return True
|
|
951
|
+
|
|
952
|
+
|
|
953
|
+
def _mcp_state():
|
|
954
|
+
import json
|
|
955
|
+
import os
|
|
956
|
+
p = _mcp_statefile()
|
|
957
|
+
if not os.path.exists(p):
|
|
958
|
+
return None
|
|
959
|
+
try:
|
|
960
|
+
with open(p, encoding="utf-8") as f:
|
|
961
|
+
return json.load(f)
|
|
962
|
+
except Exception:
|
|
963
|
+
return None
|
|
964
|
+
|
|
965
|
+
|
|
966
|
+
def _mcp_running():
|
|
967
|
+
st = _mcp_state()
|
|
968
|
+
if st and st.get("pid") and _pid_alive(st["pid"]):
|
|
969
|
+
return st
|
|
970
|
+
return None
|
|
971
|
+
|
|
972
|
+
|
|
973
|
+
def _mcp_port_open(host, port, timeout=0.5):
|
|
974
|
+
import socket
|
|
975
|
+
s = socket.socket()
|
|
976
|
+
s.settimeout(timeout)
|
|
977
|
+
try:
|
|
978
|
+
s.connect((host, int(port)))
|
|
979
|
+
return True
|
|
980
|
+
except OSError:
|
|
981
|
+
return False
|
|
982
|
+
finally:
|
|
983
|
+
s.close()
|
|
984
|
+
|
|
985
|
+
|
|
986
|
+
def _mcp_sdk_version():
|
|
987
|
+
try:
|
|
988
|
+
from importlib.metadata import version
|
|
989
|
+
return version("mcp")
|
|
990
|
+
except Exception:
|
|
991
|
+
return None
|
|
992
|
+
|
|
993
|
+
|
|
994
|
+
def _mcp_help():
|
|
995
|
+
print("pctr mcp - manage the pctr MCP server\n")
|
|
996
|
+
print("usage: pctr mcp [start|stop|restart|status|help] [options]\n")
|
|
997
|
+
print(" start start the server (stdio foreground, or http/sse in the background)")
|
|
998
|
+
print(" stop stop a background server started with `start`")
|
|
999
|
+
print(" restart stop (if running) then start")
|
|
1000
|
+
print(" status show SDK + server state")
|
|
1001
|
+
print(" help this text\n")
|
|
1002
|
+
print("options:")
|
|
1003
|
+
print(" --transport stdio|sse|streamable-http (default: stdio)")
|
|
1004
|
+
print(" --host HOST (default: 127.0.0.1)")
|
|
1005
|
+
print(" --port PORT (default: %d)" % MCP_DEFAULT_PORT)
|
|
1006
|
+
print(" --foreground run http/sse in the foreground")
|
|
1007
|
+
print()
|
|
1008
|
+
print("register with an MCP client (stdio):")
|
|
1009
|
+
print(' { "command": "pctr-mcp" }')
|
|
1010
|
+
print("or (http):")
|
|
1011
|
+
print(' { "url": "http://127.0.0.1:%d/mcp" }' % MCP_DEFAULT_PORT)
|
|
1012
|
+
print()
|
|
1013
|
+
print('needs the SDK: pip install "pctr[mcp]"')
|
|
1014
|
+
|
|
1015
|
+
|
|
1016
|
+
def _mcp_status(args):
|
|
1017
|
+
ver = _mcp_sdk_version()
|
|
1018
|
+
print("MCP SDK: %s" % (("mcp " + ver) if ver else 'NOT installed (pip install "pctr[mcp]")'))
|
|
1019
|
+
st = _mcp_running()
|
|
1020
|
+
if not st:
|
|
1021
|
+
print("server: not running")
|
|
1022
|
+
return
|
|
1023
|
+
host = st.get("host", "127.0.0.1")
|
|
1024
|
+
port = st.get("port")
|
|
1025
|
+
reach = _mcp_port_open(host, port) if port else False
|
|
1026
|
+
print("server: running pid=%s transport=%s %s:%s reachable=%s" % (
|
|
1027
|
+
st.get("pid"), st.get("transport"), host, port, "yes" if reach else "no"))
|
|
1028
|
+
|
|
1029
|
+
|
|
1030
|
+
def _mcp_start(args):
|
|
1031
|
+
import os
|
|
1032
|
+
import subprocess
|
|
1033
|
+
running = _mcp_running()
|
|
1034
|
+
if running:
|
|
1035
|
+
_fail("pctr MCP already running (pid=%s); use `pctr mcp restart`" % running["pid"])
|
|
1036
|
+
transport = args.transport
|
|
1037
|
+
if transport == "stdio":
|
|
1038
|
+
from pctr import mcp_server
|
|
1039
|
+
mcp_server.server.run(transport="stdio")
|
|
1040
|
+
return
|
|
1041
|
+
log = os.path.join(_pctr_tempdir(), "mcp.log")
|
|
1042
|
+
cmd = [sys.executable, "-m", "pctr.mcp_server", "--transport", transport,
|
|
1043
|
+
"--host", args.host, "--port", str(args.port)]
|
|
1044
|
+
flags = 0x00000008 | 0x00000200 # DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP
|
|
1045
|
+
with open(log, "ab") as lf:
|
|
1046
|
+
p = subprocess.Popen(cmd, creationflags=flags, close_fds=True, stdin=subprocess.DEVNULL,
|
|
1047
|
+
stdout=lf, stderr=subprocess.STDOUT)
|
|
1048
|
+
import json
|
|
1049
|
+
with open(_mcp_statefile(), "w", encoding="utf-8") as f:
|
|
1050
|
+
json.dump({"pid": p.pid, "transport": transport, "host": args.host, "port": args.port}, f)
|
|
1051
|
+
for _ in range(24):
|
|
1052
|
+
time.sleep(0.25)
|
|
1053
|
+
if _mcp_port_open(args.host, args.port):
|
|
1054
|
+
break
|
|
1055
|
+
reach = _mcp_port_open(args.host, args.port)
|
|
1056
|
+
path = "/sse" if transport == "sse" else "/mcp"
|
|
1057
|
+
print("pctr MCP started: pid=%s transport=%s http://%s:%s%s reachable=%s" % (
|
|
1058
|
+
p.pid, transport, args.host, args.port, path, "yes" if reach else "no"))
|
|
1059
|
+
print("logs: %s" % log)
|
|
1060
|
+
|
|
1061
|
+
|
|
1062
|
+
def _mcp_stop(args):
|
|
1063
|
+
import os
|
|
1064
|
+
import subprocess
|
|
1065
|
+
st = _mcp_running()
|
|
1066
|
+
if not st:
|
|
1067
|
+
try:
|
|
1068
|
+
os.remove(_mcp_statefile())
|
|
1069
|
+
except OSError:
|
|
1070
|
+
pass
|
|
1071
|
+
print("pctr MCP not running")
|
|
1072
|
+
return
|
|
1073
|
+
pid = st["pid"]
|
|
1074
|
+
subprocess.run(["taskkill", "/PID", str(pid), "/F"], capture_output=True)
|
|
1075
|
+
try:
|
|
1076
|
+
os.remove(_mcp_statefile())
|
|
1077
|
+
except OSError:
|
|
1078
|
+
pass
|
|
1079
|
+
print("pctr MCP stopped (pid=%s)" % pid)
|
|
1080
|
+
|
|
1081
|
+
|
|
1082
|
+
def cmd_mcp(args):
|
|
1083
|
+
action = getattr(args, "action", None) or "help"
|
|
1084
|
+
if action in ("help", "info"):
|
|
1085
|
+
return _mcp_help()
|
|
1086
|
+
if action == "status":
|
|
1087
|
+
return _mcp_status(args)
|
|
1088
|
+
if action == "start":
|
|
1089
|
+
return _mcp_start(args)
|
|
1090
|
+
if action == "stop":
|
|
1091
|
+
return _mcp_stop(args)
|
|
1092
|
+
if action == "restart":
|
|
1093
|
+
_mcp_stop(args)
|
|
1094
|
+
return _mcp_start(args)
|
|
1095
|
+
|
|
1096
|
+
|
|
1097
|
+
def _run_file(path, json_default=False):
|
|
1098
|
+
import shlex
|
|
1099
|
+
ap = build_parser()
|
|
1100
|
+
try:
|
|
1101
|
+
with open(path, encoding="utf-8") as f:
|
|
1102
|
+
lines = f.read().splitlines()
|
|
1103
|
+
except OSError as e:
|
|
1104
|
+
_fail("cannot read --file %r: %s" % (path, e))
|
|
1105
|
+
ran = 0
|
|
1106
|
+
for i, raw in enumerate(lines, 1):
|
|
1107
|
+
line = raw.strip()
|
|
1108
|
+
if not line or line.startswith("#"):
|
|
1109
|
+
continue
|
|
1110
|
+
try:
|
|
1111
|
+
argv = shlex.split(line)
|
|
1112
|
+
except ValueError as e:
|
|
1113
|
+
_fail("--file line %d: %s" % (i, e))
|
|
1114
|
+
try:
|
|
1115
|
+
a = ap.parse_args(argv)
|
|
1116
|
+
except SystemExit:
|
|
1117
|
+
_fail("--file line %d: bad command: %s" % (i, line))
|
|
1118
|
+
if not getattr(a, "cmd", None):
|
|
1119
|
+
_fail("--file line %d: no subcommand: %s" % (i, line))
|
|
1120
|
+
if json_default and not getattr(a, "json_out", False):
|
|
1121
|
+
a.json_out = True
|
|
1122
|
+
print("pctr> %s" % line, file=sys.stderr)
|
|
1123
|
+
_validate(a)
|
|
1124
|
+
a.func(a)
|
|
1125
|
+
ran += 1
|
|
1126
|
+
print("ran %d command(s) from %s" % (ran, path), file=sys.stderr)
|
|
1127
|
+
|
|
1128
|
+
|
|
832
1129
|
def add_target(p, require_selector=False, require_window=False):
|
|
833
1130
|
p.add_argument("--title", default=None, help="Window title regex")
|
|
834
1131
|
p.add_argument("--process", type=int, default=None, help="Filter by process id")
|
|
@@ -858,25 +1155,41 @@ def _validate(args):
|
|
|
858
1155
|
_fail("provide at least one selector: --title / --process / --name / --control-type / --auto-id")
|
|
859
1156
|
|
|
860
1157
|
|
|
1158
|
+
def _add_json(p):
|
|
1159
|
+
p.add_argument("--json", action="store_true", dest="json_out", default=argparse.SUPPRESS,
|
|
1160
|
+
help="Emit machine-readable JSON")
|
|
1161
|
+
|
|
1162
|
+
|
|
1163
|
+
def _add_lang(p):
|
|
1164
|
+
p.add_argument("--lang", default=None, help="OCR language BCP-47 tag, e.g. en-US or de-DE")
|
|
1165
|
+
|
|
1166
|
+
|
|
861
1167
|
def build_parser():
|
|
862
1168
|
ap = argparse.ArgumentParser(prog="pctr", description=DESCRIPTION, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
863
1169
|
ap.add_argument("--version", action="version", version="pctr %s" % _VERSION)
|
|
1170
|
+
ap.add_argument("--json", action="store_true", dest="json_out",
|
|
1171
|
+
help="Emit machine-readable JSON (windows/tree/find/ocr/ocrfind/look/desktop)")
|
|
1172
|
+
ap.add_argument("--file", default=None,
|
|
1173
|
+
help="Run pctr commands from a file, one per line (# comments allowed)")
|
|
864
1174
|
sub = ap.add_subparsers(dest="cmd", required=False)
|
|
865
1175
|
|
|
866
1176
|
w = sub.add_parser("windows", help="List top-level windows")
|
|
867
1177
|
w.add_argument("--filter", default=None)
|
|
868
1178
|
w.add_argument("--process", type=int, default=None, help="Only this process id")
|
|
1179
|
+
_add_json(w)
|
|
869
1180
|
w.set_defaults(func=cmd_windows)
|
|
870
1181
|
|
|
871
1182
|
t = sub.add_parser("tree", help="Dump UIA tree")
|
|
872
1183
|
add_target(t, require_window=True)
|
|
873
1184
|
t.add_argument("--depth", type=int, default=99)
|
|
874
1185
|
t.add_argument("--limit", type=int, default=400)
|
|
1186
|
+
_add_json(t)
|
|
875
1187
|
t.set_defaults(func=cmd_tree)
|
|
876
1188
|
|
|
877
1189
|
f = sub.add_parser("find", help="Find elements")
|
|
878
1190
|
add_target(f, require_window=True)
|
|
879
1191
|
f.add_argument("--limit", type=int, default=None, help="Show at most N matches")
|
|
1192
|
+
_add_json(f)
|
|
880
1193
|
f.set_defaults(func=cmd_find)
|
|
881
1194
|
|
|
882
1195
|
c = sub.add_parser("click", help="Click an element")
|
|
@@ -982,6 +1295,7 @@ def build_parser():
|
|
|
982
1295
|
dp.add_argument("--timeout", type=float, default=5.0)
|
|
983
1296
|
dp.add_argument("--desktop", type=int, default=None, help="Filter `windows` to this desktop index")
|
|
984
1297
|
dp.add_argument("--to", type=int, default=None, help="Target desktop index for move-window")
|
|
1298
|
+
_add_json(dp)
|
|
985
1299
|
dp.set_defaults(func=cmd_desktop)
|
|
986
1300
|
|
|
987
1301
|
dr = sub.add_parser("drag", help="Drag from --start to --end (x,y pairs)")
|
|
@@ -996,6 +1310,8 @@ def build_parser():
|
|
|
996
1310
|
oc.add_argument("--title", default=None)
|
|
997
1311
|
oc.add_argument("--process", type=int, default=None)
|
|
998
1312
|
oc.add_argument("--timeout", type=float, default=5.0)
|
|
1313
|
+
_add_json(oc)
|
|
1314
|
+
_add_lang(oc)
|
|
999
1315
|
oc.set_defaults(func=cmd_ocr)
|
|
1000
1316
|
|
|
1001
1317
|
ocf = sub.add_parser("ocrfind", help="OCR then list words matching --text (with click centers)")
|
|
@@ -1003,6 +1319,8 @@ def build_parser():
|
|
|
1003
1319
|
ocf.add_argument("--process", type=int, default=None)
|
|
1004
1320
|
ocf.add_argument("--timeout", type=float, default=5.0)
|
|
1005
1321
|
ocf.add_argument("--text", required=True)
|
|
1322
|
+
_add_json(ocf)
|
|
1323
|
+
_add_lang(ocf)
|
|
1006
1324
|
ocf.set_defaults(func=cmd_ocrfind)
|
|
1007
1325
|
|
|
1008
1326
|
occ = sub.add_parser("ocrclick", help="OCR then click the word matching --text")
|
|
@@ -1011,6 +1329,7 @@ def build_parser():
|
|
|
1011
1329
|
occ.add_argument("--timeout", type=float, default=5.0)
|
|
1012
1330
|
occ.add_argument("--text", required=True)
|
|
1013
1331
|
occ.add_argument("--nth", type=int, default=0)
|
|
1332
|
+
_add_lang(occ)
|
|
1014
1333
|
occ.set_defaults(func=cmd_ocrclick)
|
|
1015
1334
|
|
|
1016
1335
|
lk = sub.add_parser("look", help="Open-vocab detect objects by text prompt (YOLO-World)")
|
|
@@ -1020,6 +1339,7 @@ def build_parser():
|
|
|
1020
1339
|
lk.add_argument("--timeout", type=float, default=5.0)
|
|
1021
1340
|
lk.add_argument("--conf", type=float, default=None)
|
|
1022
1341
|
lk.add_argument("--model", default=None)
|
|
1342
|
+
_add_json(lk)
|
|
1023
1343
|
lk.set_defaults(func=cmd_look)
|
|
1024
1344
|
|
|
1025
1345
|
lkc = sub.add_parser("lookclick", help="Open-vocab detect then click the top match")
|
|
@@ -1043,12 +1363,30 @@ def build_parser():
|
|
|
1043
1363
|
st.add_argument("--project", default=None)
|
|
1044
1364
|
st.set_defaults(func=cmd_setup)
|
|
1045
1365
|
|
|
1366
|
+
mc = sub.add_parser("mcp", help="Manage the pctr MCP server: start / stop / restart / status / help")
|
|
1367
|
+
mc.add_argument("action", nargs="?", default="help",
|
|
1368
|
+
choices=["start", "stop", "restart", "status", "help", "info"])
|
|
1369
|
+
mc.add_argument("--transport", choices=["stdio", "sse", "streamable-http"], default="stdio")
|
|
1370
|
+
mc.add_argument("--host", default="127.0.0.1")
|
|
1371
|
+
mc.add_argument("--port", type=int, default=MCP_DEFAULT_PORT)
|
|
1372
|
+
mc.add_argument("--foreground", action="store_true", help="Run http/sse in the foreground")
|
|
1373
|
+
mc.set_defaults(func=cmd_mcp)
|
|
1374
|
+
|
|
1046
1375
|
return ap
|
|
1047
1376
|
|
|
1048
1377
|
|
|
1049
1378
|
def main():
|
|
1050
1379
|
ap = build_parser()
|
|
1051
1380
|
args = ap.parse_args()
|
|
1381
|
+
if getattr(args, "file", None):
|
|
1382
|
+
try:
|
|
1383
|
+
return _run_file(args.file, json_default=bool(getattr(args, "json_out", False)))
|
|
1384
|
+
except SystemExit:
|
|
1385
|
+
raise
|
|
1386
|
+
except KeyboardInterrupt:
|
|
1387
|
+
raise SystemExit(130)
|
|
1388
|
+
except Exception as e:
|
|
1389
|
+
_fail("%s: %s" % (type(e).__name__, e))
|
|
1052
1390
|
if not getattr(args, "cmd", None):
|
|
1053
1391
|
ap.print_help()
|
|
1054
1392
|
return
|
|
@@ -20,5 +20,9 @@ pctr look --for "a red error icon" # fallback: open-vocab object detection
|
|
|
20
20
|
- Games ignore synthetic input: add `--direct` to mouse/key commands.
|
|
21
21
|
- Electron apps expose only the frame - use `focus` + global keys, or CDP.
|
|
22
22
|
- Virtual desktops: `pctr desktop list/windows/where/on-current` (read-only), `next/prev/new/close` (switches the active desktop), `move-window --to N`. UIA actions (`invoke`/`set`) reach windows on other desktops without switching; raw mouse/keys do not. `move-window` is blocked on Win11.
|
|
23
|
+
- `--json` (global or per-command) gives machine-readable output for `windows/tree/find/ocr/ocrfind/look/desktop`; progress goes to stderr so stdout stays clean.
|
|
24
|
+
- `--file FILE` runs a batch of commands (one per line, `#` comments, stops on first error).
|
|
25
|
+
- `pctr ocr --lang de-DE` picks the OCR language (BCP-47).
|
|
23
26
|
|
|
24
27
|
Run `pctr -h` for all commands. Full skill: `pctr skill`.
|
|
28
|
+
MCP: `pip install "pctr[mcp]"`, then `pctr mcp start` (background http) or register `pctr-mcp` as a stdio server. Manage with `pctr mcp start|stop|restart|status|help`.
|
|
@@ -49,13 +49,16 @@ back as `--name`, `--control-type`, `--auto-id`.
|
|
|
49
49
|
| `move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
|
|
50
50
|
| `drag --start x,y --end x,y [--duration S]` | Drag between points. |
|
|
51
51
|
| `keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
|
|
52
|
-
| `ocr [--title RE]` | OCR the screen (or window); list words with boxes. |
|
|
53
|
-
| `ocrfind --text RE` | OCR then list matches with click centers. Matches **single words**. |
|
|
54
|
-
| `ocrclick --text RE [--nth N]` | OCR then click a word. |
|
|
52
|
+
| `ocr [--title RE] [--lang TAG]` | OCR the screen (or window); list words with boxes. |
|
|
53
|
+
| `ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers. Matches **single words**. |
|
|
54
|
+
| `ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
|
|
55
55
|
| `look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
|
|
56
56
|
| `lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
|
|
57
|
+
| `mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
|
|
57
58
|
| `skill [--install]` / `setup` | Print/install this skill + an AGENTS.md section. |
|
|
58
59
|
|
|
60
|
+
Global flags: `--json` (machine-readable output) and `--file FILE` (batch).
|
|
61
|
+
|
|
59
62
|
Filters: `--title` (regex), `--name` (regex), `--control-type`, `--auto-id`,
|
|
60
63
|
`--nth`, `--process`, `--timeout`.
|
|
61
64
|
|
|
@@ -74,6 +77,33 @@ Filters: `--title` (regex), `--name` (regex), `--control-type`, `--auto-id`,
|
|
|
74
77
|
|
|
75
78
|
Errors print a single `error: ...` line (no tracebacks).
|
|
76
79
|
|
|
80
|
+
### JSON output
|
|
81
|
+
|
|
82
|
+
Add `--json` (global, or per-command) to get machine-readable output from
|
|
83
|
+
`windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
pctr windows --json # [{"hwnd":..,"pid":..,"title":".."}, ...]
|
|
87
|
+
pctr find --title App --json # {"window":"..","count":N,"matches":[{"name":..,"rect":[l,t,r,b]}, ..]}
|
|
88
|
+
pctr desktop list --json # [{"index":0,"guid":"..","current":true,"windows":11}, ..]
|
|
89
|
+
pctr look --for "a car" --json # [{"label":"..","conf":0.86,"x":..,"y":..,"box":[..]}, ..]
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Human-readable progress goes to stderr, so JSON on stdout stays clean.
|
|
93
|
+
|
|
94
|
+
### Batch files
|
|
95
|
+
|
|
96
|
+
Run many commands from a file - one per line, `#` comments allowed, blank lines
|
|
97
|
+
skipped:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
pctr --file script.txt # e.g. lines: windows --filter Notepad
|
|
101
|
+
pctr --json --file script.txt # --json applies to every command in the file
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Stops on the first failing line and reports its line number. Each line is parsed
|
|
105
|
+
exactly like a normal `pctr ...` invocation (same flags, same quoting rules).
|
|
106
|
+
|
|
77
107
|
## Patterns
|
|
78
108
|
|
|
79
109
|
- **Text fields -> use `set`, not `type`.** `set` uses the ValuePattern: atomic, exact, can't drop chars. `type`'s element path goes through pywinauto `type_keys`, which interprets `{ } + ^ % ~ ( )` as key syntax - so `--text "a{b}"` is **not** literal and an unmatched `{` fails. Prefer `set`; use `type` only for plain text on fields with no ValuePattern.
|
|
@@ -116,9 +146,33 @@ pctr desktop move-window --title RE --to N
|
|
|
116
146
|
`PCTR_YOLO_MODEL` env var to avoid the first-run download. Progress bars are
|
|
117
147
|
sent to stderr, so `look`'s stdout stays parseable.
|
|
118
148
|
|
|
149
|
+
## MCP server
|
|
150
|
+
|
|
151
|
+
`pip install "pctr[mcp]"` then register `pctr-mcp` as a stdio MCP server
|
|
152
|
+
(Claude Desktop / Cursor / opencode). Exposes `pctr_windows`, `pctr_tree`,
|
|
153
|
+
`pctr_find`, `pctr_click`, `pctr_set`, `pctr_type`, `pctr_keys`, `pctr_hotkey`,
|
|
154
|
+
`pctr_focus`, `pctr_wait`, `pctr_shot`, `pctr_ocr`, `pctr_ocrfind`,
|
|
155
|
+
`pctr_ocrclick`, `pctr_look`, `pctr_desktop`.
|
|
156
|
+
|
|
157
|
+
Manage or run it from the CLI:
|
|
158
|
+
|
|
159
|
+
```bash
|
|
160
|
+
pctr mcp status # SDK + server state
|
|
161
|
+
pctr mcp start --transport streamable-http --port 8765 # background HTTP server
|
|
162
|
+
pctr mcp restart | pctr mcp stop
|
|
163
|
+
pctr mcp help
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
`stdio` runs in the foreground (what an MCP client spawns); `sse` /
|
|
167
|
+
`streamable-http` run in the background. HTTP endpoint:
|
|
168
|
+
`http://127.0.0.1:8765/mcp` (streamable-http) or `…/sse` (sse).
|
|
169
|
+
|
|
119
170
|
## Setup
|
|
120
171
|
|
|
121
172
|
```bash
|
|
122
|
-
pip install pctr
|
|
123
|
-
pctr
|
|
173
|
+
pip install pctr # core (UIA + OCR)
|
|
174
|
+
pip install "pctr[vision]" # + ultralytics, for `pctr look`
|
|
175
|
+
pip install "pctr[mcp]" # + MCP SDK, for `pctr-mcp` / `pctr mcp`
|
|
176
|
+
pip install "pctr[all]" # all of the above
|
|
177
|
+
pctr setup # installs this skill for opencode / claude / agents + appends to ./AGENTS.md
|
|
124
178
|
```
|
|
@@ -27,7 +27,7 @@ except Exception: # pragma: no cover
|
|
|
27
27
|
|
|
28
28
|
server = MCPServer(
|
|
29
29
|
"pctr",
|
|
30
|
-
version=_VERSION or "0.
|
|
30
|
+
version=_VERSION or "0.0.0",
|
|
31
31
|
instructions=(
|
|
32
32
|
"Element-based Windows UI automation. Find controls by name / control "
|
|
33
33
|
"type / automation id instead of pixel coordinates, then click/type/etc. "
|
|
@@ -197,8 +197,23 @@ def pctr_desktop(action: str, title: str | None = None, process: int | None = No
|
|
|
197
197
|
return _run(a)
|
|
198
198
|
|
|
199
199
|
|
|
200
|
-
def main():
|
|
201
|
-
|
|
200
|
+
def main(argv=None):
|
|
201
|
+
import argparse
|
|
202
|
+
ap = argparse.ArgumentParser(
|
|
203
|
+
prog="pctr-mcp",
|
|
204
|
+
description="pctr MCP server - element-based Windows UI automation over MCP.",
|
|
205
|
+
)
|
|
206
|
+
ap.add_argument("--transport", choices=["stdio", "sse", "streamable-http"], default="stdio",
|
|
207
|
+
help="stdio (default, for MCP clients) or an HTTP transport")
|
|
208
|
+
ap.add_argument("--host", default="127.0.0.1")
|
|
209
|
+
ap.add_argument("--port", type=int, default=8765)
|
|
210
|
+
a = ap.parse_args(argv)
|
|
211
|
+
if a.transport == "streamable-http":
|
|
212
|
+
server.run(transport="streamable-http", host=a.host, port=a.port)
|
|
213
|
+
elif a.transport == "sse":
|
|
214
|
+
server.run(transport="sse", host=a.host, port=a.port)
|
|
215
|
+
else:
|
|
216
|
+
server.run(transport="stdio")
|
|
202
217
|
|
|
203
218
|
|
|
204
219
|
if __name__ == "__main__":
|
|
File without changes
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pctr
|
|
3
|
-
Version: 0.3.
|
|
3
|
+
Version: 0.3.5
|
|
4
4
|
Summary: Element-based Windows UI Automation CLI for AI agents. Find controls by name/type/id and click, type, hold, drag, and screenshot - no pixel coordinates.
|
|
5
|
-
Author:
|
|
5
|
+
Author: NTUserDev
|
|
6
6
|
License-Expression: MIT
|
|
7
|
-
Project-URL: Homepage, https://github.com/
|
|
8
|
-
Project-URL: Repository, https://github.com/
|
|
9
|
-
Project-URL: Issues, https://github.com/
|
|
7
|
+
Project-URL: Homepage, https://github.com/NTUserDev/pctr
|
|
8
|
+
Project-URL: Repository, https://github.com/NTUserDev/pctr
|
|
9
|
+
Project-URL: Issues, https://github.com/NTUserDev/pctr/issues
|
|
10
10
|
Keywords: automation,windows,uia,pywinauto,desktop,agent,computer-use,rpa
|
|
11
11
|
Classifier: Development Status :: 4 - Beta
|
|
12
12
|
Classifier: Environment :: Win32 (MS Windows)
|
|
@@ -31,6 +31,8 @@ Provides-Extra: vision
|
|
|
31
31
|
Requires-Dist: ultralytics>=8.2; extra == "vision"
|
|
32
32
|
Provides-Extra: mcp
|
|
33
33
|
Requires-Dist: mcp>=2; extra == "mcp"
|
|
34
|
+
Provides-Extra: all
|
|
35
|
+
Requires-Dist: pctr[mcp,vision]; extra == "all"
|
|
34
36
|
Dynamic: license-file
|
|
35
37
|
|
|
36
38
|
# pctr
|
|
@@ -46,11 +48,19 @@ mouse and keyboard input. On top of UIA it also offers **OCR**
|
|
|
46
48
|
(`Windows.Media.Ocr`) and **YOLO-World** open-vocabulary detection for text and
|
|
47
49
|
objects that aren't exposed as controls.
|
|
48
50
|
|
|
51
|
+

|
|
52
|
+
|
|
53
|
+
*Driven entirely by pctr itself - list windows, map virtual desktops, dump a
|
|
54
|
+
control tree, `set` a field, read it back with OCR, screenshot it, and YOLO-World
|
|
55
|
+
detect. Recorded with OBS, which pctr also piloted.*
|
|
56
|
+
|
|
49
57
|
## Install
|
|
50
58
|
|
|
51
59
|
```bash
|
|
52
|
-
pip install pctr
|
|
53
|
-
pip install "pctr[vision]" #
|
|
60
|
+
pip install pctr # core (UIA + OCR)
|
|
61
|
+
pip install "pctr[vision]" # + ultralytics, for `pctr look`
|
|
62
|
+
pip install "pctr[mcp]" # + MCP SDK, for the MCP server
|
|
63
|
+
pip install "pctr[all]" # everything
|
|
54
64
|
```
|
|
55
65
|
|
|
56
66
|
Windows only.
|
|
@@ -77,6 +87,18 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
|
|
|
77
87
|
`pctr_shot`, `pctr_ocr`, `pctr_ocrfind`, `pctr_ocrclick`, `pctr_look`,
|
|
78
88
|
`pctr_desktop`.
|
|
79
89
|
|
|
90
|
+
Or run/manage it from the CLI:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
pctr mcp status # SDK + server state
|
|
94
|
+
pctr mcp start --transport streamable-http --port 8765 # background HTTP server
|
|
95
|
+
pctr mcp restart | pctr mcp stop
|
|
96
|
+
pctr mcp help
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`stdio` runs in the foreground (what a client spawns); `sse` / `streamable-http`
|
|
100
|
+
run in the background, reachable at `http://127.0.0.1:8765/mcp` (or `…/sse`).
|
|
101
|
+
|
|
80
102
|
## Commands
|
|
81
103
|
|
|
82
104
|
| Command | What it does |
|
|
@@ -97,13 +119,17 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
|
|
|
97
119
|
| `pctr move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
|
|
98
120
|
| `pctr drag --start x,y --end x,y [--duration S]` | Drag between two points. |
|
|
99
121
|
| `pctr keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
|
|
100
|
-
| `pctr ocr [--title RE]` | OCR the screen/window; list words with boxes. |
|
|
101
|
-
| `pctr ocrfind --text RE` | OCR then list matches with click centers (single words). |
|
|
102
|
-
| `pctr ocrclick --text RE [--nth N]` | OCR then click a word. |
|
|
122
|
+
| `pctr ocr [--title RE] [--lang TAG]` | OCR the screen/window; list words with boxes. |
|
|
123
|
+
| `pctr ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers (single words). |
|
|
124
|
+
| `pctr ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
|
|
103
125
|
| `pctr look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
|
|
104
126
|
| `pctr lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
|
|
127
|
+
| `pctr mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
|
|
105
128
|
| `pctr skill [--install]` / `pctr setup` | Print/install the agent skill + an AGENTS.md section. |
|
|
106
129
|
|
|
130
|
+
Global flags: `--json` (machine-readable output) and `--file FILE` (run a batch
|
|
131
|
+
of commands from a file).
|
|
132
|
+
|
|
107
133
|
Common filters: `--title` (regex), `--name` (regex), `--control-type`
|
|
108
134
|
(`Button`, `Edit`, `Pane`, `MenuItem`, …), `--auto-id`, `--nth`, `--process`.
|
|
109
135
|
|
|
@@ -118,6 +144,33 @@ pctr click --title "Notepad" --name "^File$"
|
|
|
118
144
|
pctr set --title "Notepad" --name "Text Editor" --text "hello"
|
|
119
145
|
```
|
|
120
146
|
|
|
147
|
+
## JSON output
|
|
148
|
+
|
|
149
|
+
Add `--json` (global, or per-command) for machine-readable output from
|
|
150
|
+
`windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
pctr windows --json
|
|
154
|
+
# [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]
|
|
155
|
+
|
|
156
|
+
pctr desktop list --json
|
|
157
|
+
# [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]
|
|
158
|
+
|
|
159
|
+
pctr look --for "a car" --json
|
|
160
|
+
# [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
Progress and warnings go to stderr, so stdout stays valid JSON.
|
|
164
|
+
|
|
165
|
+
## Batch files
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
pctr --file script.txt # one command per line; # comments and blanks skipped
|
|
169
|
+
pctr --json --file script.txt # --json applies to every line
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Stops on the first failing line and reports its line number.
|
|
173
|
+
|
|
121
174
|
## Virtual desktops
|
|
122
175
|
|
|
123
176
|
```bash
|
pctr-0.3.4/src/pctr/__init__.py
DELETED
|
@@ -1 +0,0 @@
|
|
|
1
|
-
__version__ = "0.3.4"
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|