pctr 0.3.4__tar.gz → 0.3.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,12 +1,12 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: pctr
3
- Version: 0.3.4
3
+ Version: 0.3.5
4
4
  Summary: Element-based Windows UI Automation CLI for AI agents. Find controls by name/type/id and click, type, hold, drag, and screenshot - no pixel coordinates.
5
- Author: Space-lab515
5
+ Author: NTUserDev
6
6
  License-Expression: MIT
7
- Project-URL: Homepage, https://github.com/Space-lab515/pctr
8
- Project-URL: Repository, https://github.com/Space-lab515/pctr
9
- Project-URL: Issues, https://github.com/Space-lab515/pctr/issues
7
+ Project-URL: Homepage, https://github.com/NTUserDev/pctr
8
+ Project-URL: Repository, https://github.com/NTUserDev/pctr
9
+ Project-URL: Issues, https://github.com/NTUserDev/pctr/issues
10
10
  Keywords: automation,windows,uia,pywinauto,desktop,agent,computer-use,rpa
11
11
  Classifier: Development Status :: 4 - Beta
12
12
  Classifier: Environment :: Win32 (MS Windows)
@@ -31,6 +31,8 @@ Provides-Extra: vision
31
31
  Requires-Dist: ultralytics>=8.2; extra == "vision"
32
32
  Provides-Extra: mcp
33
33
  Requires-Dist: mcp>=2; extra == "mcp"
34
+ Provides-Extra: all
35
+ Requires-Dist: pctr[mcp,vision]; extra == "all"
34
36
  Dynamic: license-file
35
37
 
36
38
  # pctr
@@ -46,11 +48,19 @@ mouse and keyboard input. On top of UIA it also offers **OCR**
46
48
  (`Windows.Media.Ocr`) and **YOLO-World** open-vocabulary detection for text and
47
49
  objects that aren't exposed as controls.
48
50
 
51
+ ![pctr driving by element name: enumerating windows, mapping virtual desktops, dumping the control tree, filling a field, then reading it back with OCR](docs/demo.gif)
52
+
53
+ *Driven entirely by pctr itself - list windows, map virtual desktops, dump a
54
+ control tree, `set` a field, read it back with OCR, screenshot it, and YOLO-World
55
+ detect. Recorded with OBS, which pctr also piloted.*
56
+
49
57
  ## Install
50
58
 
51
59
  ```bash
52
- pip install pctr
53
- pip install "pctr[vision]" # adds ultralytics for `pctr look`
60
+ pip install pctr # core (UIA + OCR)
61
+ pip install "pctr[vision]" # + ultralytics, for `pctr look`
62
+ pip install "pctr[mcp]" # + MCP SDK, for the MCP server
63
+ pip install "pctr[all]" # everything
54
64
  ```
55
65
 
56
66
  Windows only.
@@ -77,6 +87,18 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
77
87
  `pctr_shot`, `pctr_ocr`, `pctr_ocrfind`, `pctr_ocrclick`, `pctr_look`,
78
88
  `pctr_desktop`.
79
89
 
90
+ Or run/manage it from the CLI:
91
+
92
+ ```bash
93
+ pctr mcp status # SDK + server state
94
+ pctr mcp start --transport streamable-http --port 8765 # background HTTP server
95
+ pctr mcp restart | pctr mcp stop
96
+ pctr mcp help
97
+ ```
98
+
99
+ `stdio` runs in the foreground (what a client spawns); `sse` / `streamable-http`
100
+ run in the background, reachable at `http://127.0.0.1:8765/mcp` (or `…/sse`).
101
+
80
102
  ## Commands
81
103
 
82
104
  | Command | What it does |
@@ -97,13 +119,17 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
97
119
  | `pctr move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
98
120
  | `pctr drag --start x,y --end x,y [--duration S]` | Drag between two points. |
99
121
  | `pctr keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
100
- | `pctr ocr [--title RE]` | OCR the screen/window; list words with boxes. |
101
- | `pctr ocrfind --text RE` | OCR then list matches with click centers (single words). |
102
- | `pctr ocrclick --text RE [--nth N]` | OCR then click a word. |
122
+ | `pctr ocr [--title RE] [--lang TAG]` | OCR the screen/window; list words with boxes. |
123
+ | `pctr ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers (single words). |
124
+ | `pctr ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
103
125
  | `pctr look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
104
126
  | `pctr lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
127
+ | `pctr mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
105
128
  | `pctr skill [--install]` / `pctr setup` | Print/install the agent skill + an AGENTS.md section. |
106
129
 
130
+ Global flags: `--json` (machine-readable output) and `--file FILE` (run a batch
131
+ of commands from a file).
132
+
107
133
  Common filters: `--title` (regex), `--name` (regex), `--control-type`
108
134
  (`Button`, `Edit`, `Pane`, `MenuItem`, …), `--auto-id`, `--nth`, `--process`.
109
135
 
@@ -118,6 +144,33 @@ pctr click --title "Notepad" --name "^File$"
118
144
  pctr set --title "Notepad" --name "Text Editor" --text "hello"
119
145
  ```
120
146
 
147
+ ## JSON output
148
+
149
+ Add `--json` (global, or per-command) for machine-readable output from
150
+ `windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
151
+
152
+ ```bash
153
+ pctr windows --json
154
+ # [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]
155
+
156
+ pctr desktop list --json
157
+ # [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]
158
+
159
+ pctr look --for "a car" --json
160
+ # [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]
161
+ ```
162
+
163
+ Progress and warnings go to stderr, so stdout stays valid JSON.
164
+
165
+ ## Batch files
166
+
167
+ ```bash
168
+ pctr --file script.txt # one command per line; # comments and blanks skipped
169
+ pctr --json --file script.txt # --json applies to every line
170
+ ```
171
+
172
+ Stops on the first failing line and reports its line number.
173
+
121
174
  ## Virtual desktops
122
175
 
123
176
  ```bash
@@ -11,11 +11,19 @@ mouse and keyboard input. On top of UIA it also offers **OCR**
11
11
  (`Windows.Media.Ocr`) and **YOLO-World** open-vocabulary detection for text and
12
12
  objects that aren't exposed as controls.
13
13
 
14
+ ![pctr driving by element name: enumerating windows, mapping virtual desktops, dumping the control tree, filling a field, then reading it back with OCR](docs/demo.gif)
15
+
16
+ *Driven entirely by pctr itself - list windows, map virtual desktops, dump a
17
+ control tree, `set` a field, read it back with OCR, screenshot it, and YOLO-World
18
+ detect. Recorded with OBS, which pctr also piloted.*
19
+
14
20
  ## Install
15
21
 
16
22
  ```bash
17
- pip install pctr
18
- pip install "pctr[vision]" # adds ultralytics for `pctr look`
23
+ pip install pctr # core (UIA + OCR)
24
+ pip install "pctr[vision]" # + ultralytics, for `pctr look`
25
+ pip install "pctr[mcp]" # + MCP SDK, for the MCP server
26
+ pip install "pctr[all]" # everything
19
27
  ```
20
28
 
21
29
  Windows only.
@@ -42,6 +50,18 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
42
50
  `pctr_shot`, `pctr_ocr`, `pctr_ocrfind`, `pctr_ocrclick`, `pctr_look`,
43
51
  `pctr_desktop`.
44
52
 
53
+ Or run/manage it from the CLI:
54
+
55
+ ```bash
56
+ pctr mcp status # SDK + server state
57
+ pctr mcp start --transport streamable-http --port 8765 # background HTTP server
58
+ pctr mcp restart | pctr mcp stop
59
+ pctr mcp help
60
+ ```
61
+
62
+ `stdio` runs in the foreground (what a client spawns); `sse` / `streamable-http`
63
+ run in the background, reachable at `http://127.0.0.1:8765/mcp` (or `…/sse`).
64
+
45
65
  ## Commands
46
66
 
47
67
  | Command | What it does |
@@ -62,13 +82,17 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
62
82
  | `pctr move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
63
83
  | `pctr drag --start x,y --end x,y [--duration S]` | Drag between two points. |
64
84
  | `pctr keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
65
- | `pctr ocr [--title RE]` | OCR the screen/window; list words with boxes. |
66
- | `pctr ocrfind --text RE` | OCR then list matches with click centers (single words). |
67
- | `pctr ocrclick --text RE [--nth N]` | OCR then click a word. |
85
+ | `pctr ocr [--title RE] [--lang TAG]` | OCR the screen/window; list words with boxes. |
86
+ | `pctr ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers (single words). |
87
+ | `pctr ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
68
88
  | `pctr look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
69
89
  | `pctr lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
90
+ | `pctr mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
70
91
  | `pctr skill [--install]` / `pctr setup` | Print/install the agent skill + an AGENTS.md section. |
71
92
 
93
+ Global flags: `--json` (machine-readable output) and `--file FILE` (run a batch
94
+ of commands from a file).
95
+
72
96
  Common filters: `--title` (regex), `--name` (regex), `--control-type`
73
97
  (`Button`, `Edit`, `Pane`, `MenuItem`, …), `--auto-id`, `--nth`, `--process`.
74
98
 
@@ -83,6 +107,33 @@ pctr click --title "Notepad" --name "^File$"
83
107
  pctr set --title "Notepad" --name "Text Editor" --text "hello"
84
108
  ```
85
109
 
110
+ ## JSON output
111
+
112
+ Add `--json` (global, or per-command) for machine-readable output from
113
+ `windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
114
+
115
+ ```bash
116
+ pctr windows --json
117
+ # [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]
118
+
119
+ pctr desktop list --json
120
+ # [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]
121
+
122
+ pctr look --for "a car" --json
123
+ # [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]
124
+ ```
125
+
126
+ Progress and warnings go to stderr, so stdout stays valid JSON.
127
+
128
+ ## Batch files
129
+
130
+ ```bash
131
+ pctr --file script.txt # one command per line; # comments and blanks skipped
132
+ pctr --json --file script.txt # --json applies to every line
133
+ ```
134
+
135
+ Stops on the first failing line and reports its line number.
136
+
86
137
  ## Virtual desktops
87
138
 
88
139
  ```bash
@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "pctr"
7
- version = "0.3.4"
7
+ version = "0.3.5"
8
8
  description = "Element-based Windows UI Automation CLI for AI agents. Find controls by name/type/id and click, type, hold, drag, and screenshot - no pixel coordinates."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
11
11
  license = "MIT"
12
- authors = [{ name = "Space-lab515" }]
12
+ authors = [{ name = "NTUserDev" }]
13
13
  keywords = ["automation", "windows", "uia", "pywinauto", "desktop", "agent", "computer-use", "rpa"]
14
14
  classifiers = [
15
15
  "Development Status :: 4 - Beta",
@@ -35,18 +35,19 @@ dependencies = [
35
35
  [project.optional-dependencies]
36
36
  vision = ["ultralytics>=8.2"]
37
37
  mcp = ["mcp>=2"]
38
+ all = ["pctr[vision,mcp]"]
38
39
 
39
40
  [project.scripts]
40
41
  pctr = "pctr.cli:main"
41
42
  pctr-mcp = "pctr.mcp_server:main"
42
43
 
43
44
  [project.urls]
44
- Homepage = "https://github.com/Space-lab515/pctr"
45
- Repository = "https://github.com/Space-lab515/pctr"
46
- Issues = "https://github.com/Space-lab515/pctr/issues"
45
+ Homepage = "https://github.com/NTUserDev/pctr"
46
+ Repository = "https://github.com/NTUserDev/pctr"
47
+ Issues = "https://github.com/NTUserDev/pctr/issues"
47
48
 
48
49
  [tool.setuptools.packages.find]
49
50
  where = ["src"]
50
51
 
51
52
  [tool.setuptools.package-data]
52
- pctr = ["data/*.md"]
53
+ pctr = ["data/*.md", "py.typed"]
@@ -0,0 +1 @@
1
+ __version__ = "0.3.5"
@@ -130,6 +130,29 @@ def _describe(el):
130
130
  )
131
131
 
132
132
 
133
+ def _print_json(obj):
134
+ import json
135
+ print(json.dumps(obj, ensure_ascii=False))
136
+
137
+
138
+ def _is_json(args):
139
+ return bool(getattr(args, "json_out", False))
140
+
141
+
142
+ def _el_dict(el, depth=None):
143
+ info = el.element_info
144
+ r = info.rectangle
145
+ d = {
146
+ "name": info.name,
147
+ "type": info.control_type,
148
+ "id": info.automation_id,
149
+ "rect": [r.left, r.top, r.right, r.bottom],
150
+ }
151
+ if depth is not None:
152
+ d["depth"] = depth
153
+ return d
154
+
155
+
133
156
  def _get_element(args):
134
157
  win = _find_window(args.title, getattr(args, "process", None), getattr(args, "timeout", 5.0))
135
158
  els = _matches(win, args.name, getattr(args, "control_type", None), getattr(args, "auto_id", None))
@@ -148,12 +171,18 @@ def _get_element(args):
148
171
  def cmd_windows(args):
149
172
  rx = _compile(args.filter, "filter") if args.filter else None
150
173
  proc = getattr(args, "process", None)
174
+ rows = []
151
175
  for hwnd, title, pid in _enum_top_windows():
152
176
  if rx and rx.search(title) is None:
153
177
  continue
154
178
  if proc is not None and pid != proc:
155
179
  continue
156
- print("pid=%s | %s" % (pid, title))
180
+ rows.append({"hwnd": hwnd, "pid": pid, "title": title})
181
+ if _is_json(args):
182
+ _print_json(rows)
183
+ return
184
+ for r in rows:
185
+ print("pid=%s | %s" % (r["pid"], r["title"]))
157
186
 
158
187
 
159
188
  def _walk(root, max_depth):
@@ -180,27 +209,44 @@ def _walk(root, max_depth):
180
209
 
181
210
  def cmd_tree(args):
182
211
  win = _find_window(args.title, args.process, args.timeout)
183
- print("WINDOW: %s (pid=%s)" % (win.window_text(), win.process_id()))
184
- count = 0
185
212
  depth = getattr(args, "depth", None)
186
213
  depth_mode = depth is not None and 0 <= depth < 99
187
214
  it = _walk(win, depth) if depth_mode else ((el, 1) for el in win.descendants())
215
+ rows = []
216
+ truncated = False
188
217
  for el, d in it:
189
218
  try:
190
219
  el.element_info
191
220
  except Exception:
192
221
  continue
193
- print(" %s%s" % (" " * (d - 1) if depth_mode else "", _describe(el)))
194
- count += 1
195
- if count >= args.limit:
196
- print("... (limit %d reached)" % args.limit)
222
+ rows.append(_el_dict(el, d if depth_mode else None))
223
+ if len(rows) >= args.limit:
224
+ truncated = True
197
225
  break
226
+ if _is_json(args):
227
+ _print_json({
228
+ "window": win.window_text(),
229
+ "pid": win.process_id(),
230
+ "elements": rows,
231
+ "truncated": truncated,
232
+ })
233
+ return
234
+ print("WINDOW: %s (pid=%s)" % (win.window_text(), win.process_id()))
235
+ for r in rows:
236
+ prefix = " " * (r["depth"] - 1) if depth_mode else ""
237
+ print(" %s%s | type=%s | id=%s | rect=%d,%d,%d,%d" % (
238
+ prefix, r["name"], r["type"], r["id"], *r["rect"]))
239
+ if truncated:
240
+ print("... (limit %d reached)" % args.limit)
198
241
 
199
242
 
200
243
  def cmd_find(args):
201
244
  win = _find_window(args.title, args.process, args.timeout)
202
245
  els = _matches(win, args.name, args.control_type, args.auto_id)
203
246
  if not els:
247
+ if _is_json(args):
248
+ _print_json({"window": win.window_text(), "matches": [], "count": 0})
249
+ raise SystemExit(1)
204
250
  print("no matches")
205
251
  raise SystemExit(1)
206
252
  nth = getattr(args, "nth", 0) or 0
@@ -208,9 +254,16 @@ def cmd_find(args):
208
254
  if nth >= len(els):
209
255
  _fail("matched %d element(s); nth=%d out of range" % (len(els), nth))
210
256
  els = [els[nth]]
211
- print("WINDOW: %s" % win.window_text())
212
257
  limit = getattr(args, "limit", None)
213
258
  shown = els[:limit] if limit else els
259
+ if _is_json(args):
260
+ _print_json({
261
+ "window": win.window_text(),
262
+ "count": len(els),
263
+ "matches": [dict(_el_dict(el), index=i) for i, el in enumerate(shown)],
264
+ })
265
+ return
266
+ print("WINDOW: %s" % win.window_text())
214
267
  for i, el in enumerate(shown):
215
268
  print("[%d] %s" % (i, _describe(el)))
216
269
  if limit and len(els) > limit:
@@ -474,6 +527,12 @@ def _desktop_list(args):
474
527
  counts = {}
475
528
  for _hwnd, _title, d in _list_windows_with_desktop():
476
529
  counts[d] = counts.get(d, 0) + 1
530
+ if _is_json(args):
531
+ _print_json([
532
+ {"index": i, "guid": d, "current": d == cur, "windows": counts.get(d, 0)}
533
+ for i, d in enumerate(guids)
534
+ ])
535
+ return
477
536
  for i, d in enumerate(guids):
478
537
  print("#%d %s%s (%d windows)" % (i, d, " [current]" if d == cur else "", counts.get(d, 0)))
479
538
 
@@ -481,11 +540,17 @@ def _desktop_list(args):
481
540
  def _desktop_windows(args):
482
541
  guids = _vdesk_guids()
483
542
  want = getattr(args, "desktop", None)
543
+ rows = []
484
544
  for hwnd, title, d in _list_windows_with_desktop():
485
545
  i = guids.index(d) if d in guids else -1
486
546
  if want is not None and i != want:
487
547
  continue
488
- print("desktop=#%s | hwnd=%s | %s" % (i, hwnd, title))
548
+ rows.append({"desktop": i, "hwnd": hwnd, "title": title})
549
+ if _is_json(args):
550
+ _print_json(rows)
551
+ return
552
+ for r in rows:
553
+ print("desktop=#%s | hwnd=%s | %s" % (r["desktop"], r["hwnd"], r["title"]))
489
554
 
490
555
 
491
556
  def _desktop_where(args, bool_only):
@@ -498,11 +563,17 @@ def _desktop_where(args, bool_only):
498
563
  hwnd = wintypes.HWND(int(h))
499
564
  if bool_only:
500
565
  on = vdm.IsWindowOnCurrentVirtualDesktop(hwnd)
566
+ if _is_json(args):
567
+ _print_json({"on_current": bool(on)})
568
+ return
501
569
  print("on_current=%s" % ("True" if on else "False"))
502
570
  return
503
571
  guids = _vdesk_guids()
504
572
  d = _guid_norm(vdm.GetWindowDesktopId(hwnd))
505
573
  i = guids.index(d) if d in guids else -1
574
+ if _is_json(args):
575
+ _print_json({"desktop": i, "guid": d})
576
+ return
506
577
  print("desktop=#%d %s" % (i, d))
507
578
 
508
579
 
@@ -596,12 +667,20 @@ $file = Await ([Windows.Storage.StorageFile]::GetFileFromPathAsync($Path)) ([Win
596
667
  $stream = Await ($file.OpenAsync([Windows.Storage.FileAccessMode]::Read)) ([Windows.Storage.Streams.IRandomAccessStream])
597
668
  $decoder = Await ([Windows.Graphics.Imaging.BitmapDecoder]::CreateAsync($stream)) ([Windows.Graphics.Imaging.BitmapDecoder])
598
669
  $bitmap = Await ($decoder.GetSoftwareBitmapAsync()) ([Windows.Graphics.Imaging.SoftwareBitmap])
599
- $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromUserProfileLanguages()
600
- if (-not $engine) {
670
+ $engine = $null
671
+ $lang = $env:PCTR_OCR_LANG
672
+ if ($lang) {
601
673
  $null = [Windows.Globalization.Language,Windows.Foundation,ContentType=WindowsRuntime]
602
- $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language 'en-US'))
674
+ $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language $lang))
675
+ if (-not $engine) { Write-Error ("OCR language not available on this system: " + $lang); exit 2 }
676
+ } else {
677
+ $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromUserProfileLanguages()
678
+ if (-not $engine) {
679
+ $null = [Windows.Globalization.Language,Windows.Foundation,ContentType=WindowsRuntime]
680
+ $engine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage((New-Object Windows.Globalization.Language 'en-US'))
681
+ }
682
+ if (-not $engine) { Write-Error 'no OCR engine available'; exit 2 }
603
683
  }
604
- if (-not $engine) { Write-Error 'no OCR engine available'; exit 2 }
605
684
  $res = Await ($engine.RecognizeAsync($bitmap)) ([Windows.Media.Ocr.OcrResult])
606
685
  $out = foreach ($line in $res.Lines) { foreach ($w in $line.Words) { [pscustomobject]@{ text = $w.Text; x = [int]$w.BoundingRect.X; y = [int]$w.BoundingRect.Y; w = [int]$w.BoundingRect.Width; h = [int]$w.BoundingRect.Height } } }
607
686
  ConvertTo-Json -Compress -InputObject @($out)
@@ -616,7 +695,7 @@ def _pctr_tempdir():
616
695
  return d
617
696
 
618
697
 
619
- def _windows_ocr(image_path):
698
+ def _windows_ocr(image_path, lang=None):
620
699
  import base64
621
700
  import json
622
701
  import os
@@ -628,6 +707,10 @@ def _windows_ocr(image_path):
628
707
  enc = base64.b64encode(ps.encode("utf-16-le")).decode("ascii")
629
708
  env = dict(os.environ)
630
709
  env["PCTR_OCR_PATH"] = image_path
710
+ if lang:
711
+ env["PCTR_OCR_LANG"] = lang
712
+ else:
713
+ env.pop("PCTR_OCR_LANG", None)
631
714
  out = subprocess.run(
632
715
  ["powershell", "-NoProfile", "-NonInteractive", "-EncodedCommand", enc],
633
716
  capture_output=True, text=True, env=env,
@@ -656,7 +739,7 @@ def _ocr_capture(args):
656
739
  img = pyautogui.screenshot()
657
740
  try:
658
741
  img.save(path)
659
- words = _windows_ocr(path)
742
+ words = _windows_ocr(path, getattr(args, "lang", None))
660
743
  finally:
661
744
  try:
662
745
  os.remove(path)
@@ -667,6 +750,12 @@ def _ocr_capture(args):
667
750
 
668
751
  def cmd_ocr(args):
669
752
  words, ox, oy = _ocr_capture(args)
753
+ if _is_json(args):
754
+ _print_json([
755
+ {"text": w["text"], "x": ox + w["x"], "y": oy + w["y"], "w": w["w"], "h": w["h"]}
756
+ for w in words
757
+ ])
758
+ return
670
759
  for w in words:
671
760
  print("%s | %d,%d %dx%d" % (w["text"], ox + w["x"], oy + w["y"], w["w"], w["h"]))
672
761
  print("(%d words)" % len(words))
@@ -677,6 +766,13 @@ def cmd_ocrfind(args):
677
766
  words, ox, oy = _ocr_capture(args)
678
767
  rx = re.compile(args.text, re.I)
679
768
  hits = [w for w in words if rx.search(w["text"])]
769
+ if _is_json(args):
770
+ _print_json([
771
+ {"text": w["text"], "x": ox + w["x"], "y": oy + w["y"], "w": w["w"], "h": w["h"],
772
+ "cx": ox + w["x"] + w["w"] // 2, "cy": oy + w["y"] + w["h"] // 2}
773
+ for w in hits
774
+ ])
775
+ return
680
776
  for i, w in enumerate(hits):
681
777
  print("[%d] %s | center=%d,%d" % (i, w["text"], ox + w["x"] + w["w"] // 2, oy + w["y"] + w["h"] // 2))
682
778
  if not hits:
@@ -762,6 +858,13 @@ def _look_detect(args):
762
858
 
763
859
  def cmd_look(args):
764
860
  dets = _look_detect(args)
861
+ if _is_json(args):
862
+ _print_json([
863
+ {"label": d["label"], "conf": round(d["conf"], 4), "x": d["x"], "y": d["y"],
864
+ "box": list(d["box"]), "index": i}
865
+ for i, d in enumerate(dets)
866
+ ])
867
+ return
765
868
  for i, d in enumerate(dets):
766
869
  print("[%d] %s %.2f | center=%d,%d | box=%d,%d,%d,%d" % (i, d["label"], d["conf"], d["x"], d["y"], *d["box"]))
767
870
  if not dets:
@@ -829,6 +932,200 @@ def cmd_setup(args):
829
932
  print("done. restart your agent so it loads the skill.")
830
933
 
831
934
 
935
+ MCP_DEFAULT_PORT = 8765
936
+
937
+
938
+ def _mcp_statefile():
939
+ import os
940
+ return os.path.join(_pctr_tempdir(), "mcp.json")
941
+
942
+
943
+ def _pid_alive(pid):
944
+ import ctypes
945
+ k = ctypes.windll.kernel32
946
+ h = k.OpenProcess(0x1000, False, int(pid)) # PROCESS_QUERY_LIMITED_INFORMATION
947
+ if not h:
948
+ return False
949
+ k.CloseHandle(h)
950
+ return True
951
+
952
+
953
+ def _mcp_state():
954
+ import json
955
+ import os
956
+ p = _mcp_statefile()
957
+ if not os.path.exists(p):
958
+ return None
959
+ try:
960
+ with open(p, encoding="utf-8") as f:
961
+ return json.load(f)
962
+ except Exception:
963
+ return None
964
+
965
+
966
+ def _mcp_running():
967
+ st = _mcp_state()
968
+ if st and st.get("pid") and _pid_alive(st["pid"]):
969
+ return st
970
+ return None
971
+
972
+
973
+ def _mcp_port_open(host, port, timeout=0.5):
974
+ import socket
975
+ s = socket.socket()
976
+ s.settimeout(timeout)
977
+ try:
978
+ s.connect((host, int(port)))
979
+ return True
980
+ except OSError:
981
+ return False
982
+ finally:
983
+ s.close()
984
+
985
+
986
+ def _mcp_sdk_version():
987
+ try:
988
+ from importlib.metadata import version
989
+ return version("mcp")
990
+ except Exception:
991
+ return None
992
+
993
+
994
+ def _mcp_help():
995
+ print("pctr mcp - manage the pctr MCP server\n")
996
+ print("usage: pctr mcp [start|stop|restart|status|help] [options]\n")
997
+ print(" start start the server (stdio foreground, or http/sse in the background)")
998
+ print(" stop stop a background server started with `start`")
999
+ print(" restart stop (if running) then start")
1000
+ print(" status show SDK + server state")
1001
+ print(" help this text\n")
1002
+ print("options:")
1003
+ print(" --transport stdio|sse|streamable-http (default: stdio)")
1004
+ print(" --host HOST (default: 127.0.0.1)")
1005
+ print(" --port PORT (default: %d)" % MCP_DEFAULT_PORT)
1006
+ print(" --foreground run http/sse in the foreground")
1007
+ print()
1008
+ print("register with an MCP client (stdio):")
1009
+ print(' { "command": "pctr-mcp" }')
1010
+ print("or (http):")
1011
+ print(' { "url": "http://127.0.0.1:%d/mcp" }' % MCP_DEFAULT_PORT)
1012
+ print()
1013
+ print('needs the SDK: pip install "pctr[mcp]"')
1014
+
1015
+
1016
+ def _mcp_status(args):
1017
+ ver = _mcp_sdk_version()
1018
+ print("MCP SDK: %s" % (("mcp " + ver) if ver else 'NOT installed (pip install "pctr[mcp]")'))
1019
+ st = _mcp_running()
1020
+ if not st:
1021
+ print("server: not running")
1022
+ return
1023
+ host = st.get("host", "127.0.0.1")
1024
+ port = st.get("port")
1025
+ reach = _mcp_port_open(host, port) if port else False
1026
+ print("server: running pid=%s transport=%s %s:%s reachable=%s" % (
1027
+ st.get("pid"), st.get("transport"), host, port, "yes" if reach else "no"))
1028
+
1029
+
1030
+ def _mcp_start(args):
1031
+ import os
1032
+ import subprocess
1033
+ running = _mcp_running()
1034
+ if running:
1035
+ _fail("pctr MCP already running (pid=%s); use `pctr mcp restart`" % running["pid"])
1036
+ transport = args.transport
1037
+ if transport == "stdio":
1038
+ from pctr import mcp_server
1039
+ mcp_server.server.run(transport="stdio")
1040
+ return
1041
+ log = os.path.join(_pctr_tempdir(), "mcp.log")
1042
+ cmd = [sys.executable, "-m", "pctr.mcp_server", "--transport", transport,
1043
+ "--host", args.host, "--port", str(args.port)]
1044
+ flags = 0x00000008 | 0x00000200 # DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP
1045
+ with open(log, "ab") as lf:
1046
+ p = subprocess.Popen(cmd, creationflags=flags, close_fds=True, stdin=subprocess.DEVNULL,
1047
+ stdout=lf, stderr=subprocess.STDOUT)
1048
+ import json
1049
+ with open(_mcp_statefile(), "w", encoding="utf-8") as f:
1050
+ json.dump({"pid": p.pid, "transport": transport, "host": args.host, "port": args.port}, f)
1051
+ for _ in range(24):
1052
+ time.sleep(0.25)
1053
+ if _mcp_port_open(args.host, args.port):
1054
+ break
1055
+ reach = _mcp_port_open(args.host, args.port)
1056
+ path = "/sse" if transport == "sse" else "/mcp"
1057
+ print("pctr MCP started: pid=%s transport=%s http://%s:%s%s reachable=%s" % (
1058
+ p.pid, transport, args.host, args.port, path, "yes" if reach else "no"))
1059
+ print("logs: %s" % log)
1060
+
1061
+
1062
+ def _mcp_stop(args):
1063
+ import os
1064
+ import subprocess
1065
+ st = _mcp_running()
1066
+ if not st:
1067
+ try:
1068
+ os.remove(_mcp_statefile())
1069
+ except OSError:
1070
+ pass
1071
+ print("pctr MCP not running")
1072
+ return
1073
+ pid = st["pid"]
1074
+ subprocess.run(["taskkill", "/PID", str(pid), "/F"], capture_output=True)
1075
+ try:
1076
+ os.remove(_mcp_statefile())
1077
+ except OSError:
1078
+ pass
1079
+ print("pctr MCP stopped (pid=%s)" % pid)
1080
+
1081
+
1082
+ def cmd_mcp(args):
1083
+ action = getattr(args, "action", None) or "help"
1084
+ if action in ("help", "info"):
1085
+ return _mcp_help()
1086
+ if action == "status":
1087
+ return _mcp_status(args)
1088
+ if action == "start":
1089
+ return _mcp_start(args)
1090
+ if action == "stop":
1091
+ return _mcp_stop(args)
1092
+ if action == "restart":
1093
+ _mcp_stop(args)
1094
+ return _mcp_start(args)
1095
+
1096
+
1097
+ def _run_file(path, json_default=False):
1098
+ import shlex
1099
+ ap = build_parser()
1100
+ try:
1101
+ with open(path, encoding="utf-8") as f:
1102
+ lines = f.read().splitlines()
1103
+ except OSError as e:
1104
+ _fail("cannot read --file %r: %s" % (path, e))
1105
+ ran = 0
1106
+ for i, raw in enumerate(lines, 1):
1107
+ line = raw.strip()
1108
+ if not line or line.startswith("#"):
1109
+ continue
1110
+ try:
1111
+ argv = shlex.split(line)
1112
+ except ValueError as e:
1113
+ _fail("--file line %d: %s" % (i, e))
1114
+ try:
1115
+ a = ap.parse_args(argv)
1116
+ except SystemExit:
1117
+ _fail("--file line %d: bad command: %s" % (i, line))
1118
+ if not getattr(a, "cmd", None):
1119
+ _fail("--file line %d: no subcommand: %s" % (i, line))
1120
+ if json_default and not getattr(a, "json_out", False):
1121
+ a.json_out = True
1122
+ print("pctr> %s" % line, file=sys.stderr)
1123
+ _validate(a)
1124
+ a.func(a)
1125
+ ran += 1
1126
+ print("ran %d command(s) from %s" % (ran, path), file=sys.stderr)
1127
+
1128
+
832
1129
  def add_target(p, require_selector=False, require_window=False):
833
1130
  p.add_argument("--title", default=None, help="Window title regex")
834
1131
  p.add_argument("--process", type=int, default=None, help="Filter by process id")
@@ -858,25 +1155,41 @@ def _validate(args):
858
1155
  _fail("provide at least one selector: --title / --process / --name / --control-type / --auto-id")
859
1156
 
860
1157
 
1158
+ def _add_json(p):
1159
+ p.add_argument("--json", action="store_true", dest="json_out", default=argparse.SUPPRESS,
1160
+ help="Emit machine-readable JSON")
1161
+
1162
+
1163
+ def _add_lang(p):
1164
+ p.add_argument("--lang", default=None, help="OCR language BCP-47 tag, e.g. en-US or de-DE")
1165
+
1166
+
861
1167
  def build_parser():
862
1168
  ap = argparse.ArgumentParser(prog="pctr", description=DESCRIPTION, formatter_class=argparse.RawDescriptionHelpFormatter)
863
1169
  ap.add_argument("--version", action="version", version="pctr %s" % _VERSION)
1170
+ ap.add_argument("--json", action="store_true", dest="json_out",
1171
+ help="Emit machine-readable JSON (windows/tree/find/ocr/ocrfind/look/desktop)")
1172
+ ap.add_argument("--file", default=None,
1173
+ help="Run pctr commands from a file, one per line (# comments allowed)")
864
1174
  sub = ap.add_subparsers(dest="cmd", required=False)
865
1175
 
866
1176
  w = sub.add_parser("windows", help="List top-level windows")
867
1177
  w.add_argument("--filter", default=None)
868
1178
  w.add_argument("--process", type=int, default=None, help="Only this process id")
1179
+ _add_json(w)
869
1180
  w.set_defaults(func=cmd_windows)
870
1181
 
871
1182
  t = sub.add_parser("tree", help="Dump UIA tree")
872
1183
  add_target(t, require_window=True)
873
1184
  t.add_argument("--depth", type=int, default=99)
874
1185
  t.add_argument("--limit", type=int, default=400)
1186
+ _add_json(t)
875
1187
  t.set_defaults(func=cmd_tree)
876
1188
 
877
1189
  f = sub.add_parser("find", help="Find elements")
878
1190
  add_target(f, require_window=True)
879
1191
  f.add_argument("--limit", type=int, default=None, help="Show at most N matches")
1192
+ _add_json(f)
880
1193
  f.set_defaults(func=cmd_find)
881
1194
 
882
1195
  c = sub.add_parser("click", help="Click an element")
@@ -982,6 +1295,7 @@ def build_parser():
982
1295
  dp.add_argument("--timeout", type=float, default=5.0)
983
1296
  dp.add_argument("--desktop", type=int, default=None, help="Filter `windows` to this desktop index")
984
1297
  dp.add_argument("--to", type=int, default=None, help="Target desktop index for move-window")
1298
+ _add_json(dp)
985
1299
  dp.set_defaults(func=cmd_desktop)
986
1300
 
987
1301
  dr = sub.add_parser("drag", help="Drag from --start to --end (x,y pairs)")
@@ -996,6 +1310,8 @@ def build_parser():
996
1310
  oc.add_argument("--title", default=None)
997
1311
  oc.add_argument("--process", type=int, default=None)
998
1312
  oc.add_argument("--timeout", type=float, default=5.0)
1313
+ _add_json(oc)
1314
+ _add_lang(oc)
999
1315
  oc.set_defaults(func=cmd_ocr)
1000
1316
 
1001
1317
  ocf = sub.add_parser("ocrfind", help="OCR then list words matching --text (with click centers)")
@@ -1003,6 +1319,8 @@ def build_parser():
1003
1319
  ocf.add_argument("--process", type=int, default=None)
1004
1320
  ocf.add_argument("--timeout", type=float, default=5.0)
1005
1321
  ocf.add_argument("--text", required=True)
1322
+ _add_json(ocf)
1323
+ _add_lang(ocf)
1006
1324
  ocf.set_defaults(func=cmd_ocrfind)
1007
1325
 
1008
1326
  occ = sub.add_parser("ocrclick", help="OCR then click the word matching --text")
@@ -1011,6 +1329,7 @@ def build_parser():
1011
1329
  occ.add_argument("--timeout", type=float, default=5.0)
1012
1330
  occ.add_argument("--text", required=True)
1013
1331
  occ.add_argument("--nth", type=int, default=0)
1332
+ _add_lang(occ)
1014
1333
  occ.set_defaults(func=cmd_ocrclick)
1015
1334
 
1016
1335
  lk = sub.add_parser("look", help="Open-vocab detect objects by text prompt (YOLO-World)")
@@ -1020,6 +1339,7 @@ def build_parser():
1020
1339
  lk.add_argument("--timeout", type=float, default=5.0)
1021
1340
  lk.add_argument("--conf", type=float, default=None)
1022
1341
  lk.add_argument("--model", default=None)
1342
+ _add_json(lk)
1023
1343
  lk.set_defaults(func=cmd_look)
1024
1344
 
1025
1345
  lkc = sub.add_parser("lookclick", help="Open-vocab detect then click the top match")
@@ -1043,12 +1363,30 @@ def build_parser():
1043
1363
  st.add_argument("--project", default=None)
1044
1364
  st.set_defaults(func=cmd_setup)
1045
1365
 
1366
+ mc = sub.add_parser("mcp", help="Manage the pctr MCP server: start / stop / restart / status / help")
1367
+ mc.add_argument("action", nargs="?", default="help",
1368
+ choices=["start", "stop", "restart", "status", "help", "info"])
1369
+ mc.add_argument("--transport", choices=["stdio", "sse", "streamable-http"], default="stdio")
1370
+ mc.add_argument("--host", default="127.0.0.1")
1371
+ mc.add_argument("--port", type=int, default=MCP_DEFAULT_PORT)
1372
+ mc.add_argument("--foreground", action="store_true", help="Run http/sse in the foreground")
1373
+ mc.set_defaults(func=cmd_mcp)
1374
+
1046
1375
  return ap
1047
1376
 
1048
1377
 
1049
1378
  def main():
1050
1379
  ap = build_parser()
1051
1380
  args = ap.parse_args()
1381
+ if getattr(args, "file", None):
1382
+ try:
1383
+ return _run_file(args.file, json_default=bool(getattr(args, "json_out", False)))
1384
+ except SystemExit:
1385
+ raise
1386
+ except KeyboardInterrupt:
1387
+ raise SystemExit(130)
1388
+ except Exception as e:
1389
+ _fail("%s: %s" % (type(e).__name__, e))
1052
1390
  if not getattr(args, "cmd", None):
1053
1391
  ap.print_help()
1054
1392
  return
@@ -20,5 +20,9 @@ pctr look --for "a red error icon" # fallback: open-vocab object detection
20
20
  - Games ignore synthetic input: add `--direct` to mouse/key commands.
21
21
  - Electron apps expose only the frame - use `focus` + global keys, or CDP.
22
22
  - Virtual desktops: `pctr desktop list/windows/where/on-current` (read-only), `next/prev/new/close` (switches the active desktop), `move-window --to N`. UIA actions (`invoke`/`set`) reach windows on other desktops without switching; raw mouse/keys do not. `move-window` is blocked on Win11.
23
+ - `--json` (global or per-command) gives machine-readable output for `windows/tree/find/ocr/ocrfind/look/desktop`; progress goes to stderr so stdout stays clean.
24
+ - `--file FILE` runs a batch of commands (one per line, `#` comments, stops on first error).
25
+ - `pctr ocr --lang de-DE` picks the OCR language (BCP-47).
23
26
 
24
27
  Run `pctr -h` for all commands. Full skill: `pctr skill`.
28
+ MCP: `pip install "pctr[mcp]"`, then `pctr mcp start` (background http) or register `pctr-mcp` as a stdio server. Manage with `pctr mcp start|stop|restart|status|help`.
@@ -49,13 +49,16 @@ back as `--name`, `--control-type`, `--auto-id`.
49
49
  | `move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
50
50
  | `drag --start x,y --end x,y [--duration S]` | Drag between points. |
51
51
  | `keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
52
- | `ocr [--title RE]` | OCR the screen (or window); list words with boxes. |
53
- | `ocrfind --text RE` | OCR then list matches with click centers. Matches **single words**. |
54
- | `ocrclick --text RE [--nth N]` | OCR then click a word. |
52
+ | `ocr [--title RE] [--lang TAG]` | OCR the screen (or window); list words with boxes. |
53
+ | `ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers. Matches **single words**. |
54
+ | `ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
55
55
  | `look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
56
56
  | `lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
57
+ | `mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
57
58
  | `skill [--install]` / `setup` | Print/install this skill + an AGENTS.md section. |
58
59
 
60
+ Global flags: `--json` (machine-readable output) and `--file FILE` (batch).
61
+
59
62
  Filters: `--title` (regex), `--name` (regex), `--control-type`, `--auto-id`,
60
63
  `--nth`, `--process`, `--timeout`.
61
64
 
@@ -74,6 +77,33 @@ Filters: `--title` (regex), `--name` (regex), `--control-type`, `--auto-id`,
74
77
 
75
78
  Errors print a single `error: ...` line (no tracebacks).
76
79
 
80
+ ### JSON output
81
+
82
+ Add `--json` (global, or per-command) to get machine-readable output from
83
+ `windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
84
+
85
+ ```bash
86
+ pctr windows --json # [{"hwnd":..,"pid":..,"title":".."}, ...]
87
+ pctr find --title App --json # {"window":"..","count":N,"matches":[{"name":..,"rect":[l,t,r,b]}, ..]}
88
+ pctr desktop list --json # [{"index":0,"guid":"..","current":true,"windows":11}, ..]
89
+ pctr look --for "a car" --json # [{"label":"..","conf":0.86,"x":..,"y":..,"box":[..]}, ..]
90
+ ```
91
+
92
+ Human-readable progress goes to stderr, so JSON on stdout stays clean.
93
+
94
+ ### Batch files
95
+
96
+ Run many commands from a file - one per line, `#` comments allowed, blank lines
97
+ skipped:
98
+
99
+ ```bash
100
+ pctr --file script.txt # e.g. lines: windows --filter Notepad
101
+ pctr --json --file script.txt # --json applies to every command in the file
102
+ ```
103
+
104
+ Stops on the first failing line and reports its line number. Each line is parsed
105
+ exactly like a normal `pctr ...` invocation (same flags, same quoting rules).
106
+
77
107
  ## Patterns
78
108
 
79
109
  - **Text fields -> use `set`, not `type`.** `set` uses the ValuePattern: atomic, exact, can't drop chars. `type`'s element path goes through pywinauto `type_keys`, which interprets `{ } + ^ % ~ ( )` as key syntax - so `--text "a{b}"` is **not** literal and an unmatched `{` fails. Prefer `set`; use `type` only for plain text on fields with no ValuePattern.
@@ -116,9 +146,33 @@ pctr desktop move-window --title RE --to N
116
146
  `PCTR_YOLO_MODEL` env var to avoid the first-run download. Progress bars are
117
147
  sent to stderr, so `look`'s stdout stays parseable.
118
148
 
149
+ ## MCP server
150
+
151
+ `pip install "pctr[mcp]"` then register `pctr-mcp` as a stdio MCP server
152
+ (Claude Desktop / Cursor / opencode). Exposes `pctr_windows`, `pctr_tree`,
153
+ `pctr_find`, `pctr_click`, `pctr_set`, `pctr_type`, `pctr_keys`, `pctr_hotkey`,
154
+ `pctr_focus`, `pctr_wait`, `pctr_shot`, `pctr_ocr`, `pctr_ocrfind`,
155
+ `pctr_ocrclick`, `pctr_look`, `pctr_desktop`.
156
+
157
+ Manage or run it from the CLI:
158
+
159
+ ```bash
160
+ pctr mcp status # SDK + server state
161
+ pctr mcp start --transport streamable-http --port 8765 # background HTTP server
162
+ pctr mcp restart | pctr mcp stop
163
+ pctr mcp help
164
+ ```
165
+
166
+ `stdio` runs in the foreground (what an MCP client spawns); `sse` /
167
+ `streamable-http` run in the background. HTTP endpoint:
168
+ `http://127.0.0.1:8765/mcp` (streamable-http) or `…/sse` (sse).
169
+
119
170
  ## Setup
120
171
 
121
172
  ```bash
122
- pip install pctr
123
- pctr setup # installs this skill for opencode / claude / agents + appends to ./AGENTS.md
173
+ pip install pctr # core (UIA + OCR)
174
+ pip install "pctr[vision]" # + ultralytics, for `pctr look`
175
+ pip install "pctr[mcp]" # + MCP SDK, for `pctr-mcp` / `pctr mcp`
176
+ pip install "pctr[all]" # all of the above
177
+ pctr setup # installs this skill for opencode / claude / agents + appends to ./AGENTS.md
124
178
  ```
@@ -27,7 +27,7 @@ except Exception: # pragma: no cover
27
27
 
28
28
  server = MCPServer(
29
29
  "pctr",
30
- version=_VERSION or "0.3.3",
30
+ version=_VERSION or "0.0.0",
31
31
  instructions=(
32
32
  "Element-based Windows UI automation. Find controls by name / control "
33
33
  "type / automation id instead of pixel coordinates, then click/type/etc. "
@@ -197,8 +197,23 @@ def pctr_desktop(action: str, title: str | None = None, process: int | None = No
197
197
  return _run(a)
198
198
 
199
199
 
200
- def main():
201
- server.run(transport="stdio")
200
+ def main(argv=None):
201
+ import argparse
202
+ ap = argparse.ArgumentParser(
203
+ prog="pctr-mcp",
204
+ description="pctr MCP server - element-based Windows UI automation over MCP.",
205
+ )
206
+ ap.add_argument("--transport", choices=["stdio", "sse", "streamable-http"], default="stdio",
207
+ help="stdio (default, for MCP clients) or an HTTP transport")
208
+ ap.add_argument("--host", default="127.0.0.1")
209
+ ap.add_argument("--port", type=int, default=8765)
210
+ a = ap.parse_args(argv)
211
+ if a.transport == "streamable-http":
212
+ server.run(transport="streamable-http", host=a.host, port=a.port)
213
+ elif a.transport == "sse":
214
+ server.run(transport="sse", host=a.host, port=a.port)
215
+ else:
216
+ server.run(transport="stdio")
202
217
 
203
218
 
204
219
  if __name__ == "__main__":
File without changes
@@ -1,12 +1,12 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: pctr
3
- Version: 0.3.4
3
+ Version: 0.3.5
4
4
  Summary: Element-based Windows UI Automation CLI for AI agents. Find controls by name/type/id and click, type, hold, drag, and screenshot - no pixel coordinates.
5
- Author: Space-lab515
5
+ Author: NTUserDev
6
6
  License-Expression: MIT
7
- Project-URL: Homepage, https://github.com/Space-lab515/pctr
8
- Project-URL: Repository, https://github.com/Space-lab515/pctr
9
- Project-URL: Issues, https://github.com/Space-lab515/pctr/issues
7
+ Project-URL: Homepage, https://github.com/NTUserDev/pctr
8
+ Project-URL: Repository, https://github.com/NTUserDev/pctr
9
+ Project-URL: Issues, https://github.com/NTUserDev/pctr/issues
10
10
  Keywords: automation,windows,uia,pywinauto,desktop,agent,computer-use,rpa
11
11
  Classifier: Development Status :: 4 - Beta
12
12
  Classifier: Environment :: Win32 (MS Windows)
@@ -31,6 +31,8 @@ Provides-Extra: vision
31
31
  Requires-Dist: ultralytics>=8.2; extra == "vision"
32
32
  Provides-Extra: mcp
33
33
  Requires-Dist: mcp>=2; extra == "mcp"
34
+ Provides-Extra: all
35
+ Requires-Dist: pctr[mcp,vision]; extra == "all"
34
36
  Dynamic: license-file
35
37
 
36
38
  # pctr
@@ -46,11 +48,19 @@ mouse and keyboard input. On top of UIA it also offers **OCR**
46
48
  (`Windows.Media.Ocr`) and **YOLO-World** open-vocabulary detection for text and
47
49
  objects that aren't exposed as controls.
48
50
 
51
+ ![pctr driving by element name: enumerating windows, mapping virtual desktops, dumping the control tree, filling a field, then reading it back with OCR](docs/demo.gif)
52
+
53
+ *Driven entirely by pctr itself - list windows, map virtual desktops, dump a
54
+ control tree, `set` a field, read it back with OCR, screenshot it, and YOLO-World
55
+ detect. Recorded with OBS, which pctr also piloted.*
56
+
49
57
  ## Install
50
58
 
51
59
  ```bash
52
- pip install pctr
53
- pip install "pctr[vision]" # adds ultralytics for `pctr look`
60
+ pip install pctr # core (UIA + OCR)
61
+ pip install "pctr[vision]" # + ultralytics, for `pctr look`
62
+ pip install "pctr[mcp]" # + MCP SDK, for the MCP server
63
+ pip install "pctr[all]" # everything
54
64
  ```
55
65
 
56
66
  Windows only.
@@ -77,6 +87,18 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
77
87
  `pctr_shot`, `pctr_ocr`, `pctr_ocrfind`, `pctr_ocrclick`, `pctr_look`,
78
88
  `pctr_desktop`.
79
89
 
90
+ Or run/manage it from the CLI:
91
+
92
+ ```bash
93
+ pctr mcp status # SDK + server state
94
+ pctr mcp start --transport streamable-http --port 8765 # background HTTP server
95
+ pctr mcp restart | pctr mcp stop
96
+ pctr mcp help
97
+ ```
98
+
99
+ `stdio` runs in the foreground (what a client spawns); `sse` / `streamable-http`
100
+ run in the background, reachable at `http://127.0.0.1:8765/mcp` (or `…/sse`).
101
+
80
102
  ## Commands
81
103
 
82
104
  | Command | What it does |
@@ -97,13 +119,17 @@ Tools exposed: `pctr_windows`, `pctr_tree`, `pctr_find`, `pctr_click`,
97
119
  | `pctr move --x N --y N` / `down` / `up` / `hold --ms N` | Raw mouse control (`--direct` for games). |
98
120
  | `pctr drag --start x,y --end x,y [--duration S]` | Drag between two points. |
99
121
  | `pctr keydown / keyup --key a` / `keyhold --key w --ms 1500` | Hold keyboard keys (`--direct` for games). |
100
- | `pctr ocr [--title RE]` | OCR the screen/window; list words with boxes. |
101
- | `pctr ocrfind --text RE` | OCR then list matches with click centers (single words). |
102
- | `pctr ocrclick --text RE [--nth N]` | OCR then click a word. |
122
+ | `pctr ocr [--title RE] [--lang TAG]` | OCR the screen/window; list words with boxes. |
123
+ | `pctr ocrfind --text RE [--lang TAG]` | OCR then list matches with click centers (single words). |
124
+ | `pctr ocrclick --text RE [--nth N] [--lang TAG]` | OCR then click a word. |
103
125
  | `pctr look --for "a dog" [--title RE] [--conf X]` | YOLO-World detect objects by text prompt. |
104
126
  | `pctr lookclick --for "a dog" [--nth N]` | Detect then click the top match. |
127
+ | `pctr mcp <start\|stop\|restart\|status\|help>` | Manage the MCP server (stdio or background http/sse). |
105
128
  | `pctr skill [--install]` / `pctr setup` | Print/install the agent skill + an AGENTS.md section. |
106
129
 
130
+ Global flags: `--json` (machine-readable output) and `--file FILE` (run a batch
131
+ of commands from a file).
132
+
107
133
  Common filters: `--title` (regex), `--name` (regex), `--control-type`
108
134
  (`Button`, `Edit`, `Pane`, `MenuItem`, …), `--auto-id`, `--nth`, `--process`.
109
135
 
@@ -118,6 +144,33 @@ pctr click --title "Notepad" --name "^File$"
118
144
  pctr set --title "Notepad" --name "Text Editor" --text "hello"
119
145
  ```
120
146
 
147
+ ## JSON output
148
+
149
+ Add `--json` (global, or per-command) for machine-readable output from
150
+ `windows`, `tree`, `find`, `ocr`, `ocrfind`, `look`, and `desktop`:
151
+
152
+ ```bash
153
+ pctr windows --json
154
+ # [{"hwnd":1234,"pid":5678,"title":"Untitled - Notepad"}, ...]
155
+
156
+ pctr desktop list --json
157
+ # [{"index":0,"guid":"ae37…","current":true,"windows":11}, ...]
158
+
159
+ pctr look --for "a car" --json
160
+ # [{"label":"a car","conf":0.86,"x":512,"y":300,"box":[400,220,620,380]}, ...]
161
+ ```
162
+
163
+ Progress and warnings go to stderr, so stdout stays valid JSON.
164
+
165
+ ## Batch files
166
+
167
+ ```bash
168
+ pctr --file script.txt # one command per line; # comments and blanks skipped
169
+ pctr --json --file script.txt # --json applies to every line
170
+ ```
171
+
172
+ Stops on the first failing line and reports its line number.
173
+
121
174
  ## Virtual desktops
122
175
 
123
176
  ```bash
@@ -5,6 +5,7 @@ src/pctr/__init__.py
5
5
  src/pctr/__main__.py
6
6
  src/pctr/cli.py
7
7
  src/pctr/mcp_server.py
8
+ src/pctr/py.typed
8
9
  src/pctr.egg-info/PKG-INFO
9
10
  src/pctr.egg-info/SOURCES.txt
10
11
  src/pctr.egg-info/dependency_links.txt
@@ -6,6 +6,9 @@ pyautogui>=0.9.54
6
6
  pydirectinput>=1.0.4
7
7
  pillow>=9.0
8
8
 
9
+ [all]
10
+ pctr[mcp,vision]
11
+
9
12
  [mcp]
10
13
  mcp>=2
11
14
 
@@ -1 +0,0 @@
1
- __version__ = "0.3.4"
File without changes
File without changes
File without changes