mctrl 0.1.0__py3-none-any.whl

mctrl-0.1.0.dist-info/METADATA

@@ -0,0 +1,692 @@
+Metadata-Version: 2.5
+Name: mctrl
+Version: 0.1.0
+Summary: Control your Mac with hand and eye movements picked up by any camera you own.
+Project-URL: Homepage, https://github.com/GriffinCanCode/mctrl
+Project-URL: Repository, https://github.com/GriffinCanCode/mctrl
+License: MIT
+Requires-Python: <3.13,>=3.11
+Requires-Dist: mediapipe<1.0,>=0.10.18
+Requires-Dist: numpy>=1.26
+Requires-Dist: opencv-python>=4.10.0
+Requires-Dist: pyobjc-framework-applicationservices>=10.3
+Requires-Dist: pyobjc-framework-avfoundation>=10.3
+Requires-Dist: pyobjc-framework-cocoa>=10.3
+Requires-Dist: pyobjc-framework-quartz>=10.3
+Requires-Dist: rumps>=0.4.0
+Description-Content-Type: text/markdown
+
+# mindcontrol
+
+Control your Mac with your hands and eyes, using the cameras you already own.
+
+It runs as a menu-bar background app. Your gaze aims the cursor across the
+screen, your hand refines the last inch and clicks, and the moment you touch the
+real mouse or keyboard the hands get out of the way. Stand up, walk around, sit
+back down — control follows whichever camera can see you.
+
+## The gesture vocabulary
+
+The whole system is built on one distinction: **a hand that is talking to the
+computer looks different from a hand that is just there.** Only the poses below
+do anything. Your hand can rest on the desk, hold a coffee, or gesture while you
+talk, and nothing happens.
+
+### Getting in and out
+
+**Open palm, held still for 1 second** — engage or disengage hand control. This
+is the only gesture recognised while control is off, so you can always put your
+hands down safely.
+
+**Touch the mouse, trackpad or keyboard** — control suspends instantly,
+mid-motion. It comes back on its own about three seconds after you stop, or
+immediately if you palm-toggle. You never "exit" hand mode; you just reach for
+the trackpad and it yields.
+
+### Pointing
+
+**Gaze** aims. Look somewhere and the cursor jumps to that region once your eyes
+settle there for ~150ms. Eyes are excellent at crossing a screen and poor at
+holding still, which is why they only ever do the coarse half.
+
+**Ready pose** — index finger and thumb both out, hand relaxed in a loose C —
+turns your hand into a trackpad floating in the air. Move it and the cursor
+moves relative to where it already is, so small wrist motions do fine work.
+Pointer acceleration means slow movement is precise and fast movement is
+sweeping.
+
+Gaze politely stands down whenever the hand is moving, dragging or scrolling, so
+the two never fight over the cursor.
+
+### Acting
+
+**Pinch thumb to index, quick tap** — left click.
+
+**Two quick pinch taps** — double click. Sent as a real chained click, so Finder
+and everything else treat it exactly like a trackpad double click.
+
+**Pinch thumb to middle finger** — right click.
+
+**Pinch and hold, then move** — grab and drag. Hold the pinch past a quarter
+second and you are now holding the thing under the cursor; release to drop it.
+This is the main way to manipulate anything directly.
+
+**Close your hand into a fist and move** — scroll and pan, as if you had grabbed
+the page and were pulling it. Open your hand to let go.
+
+### System control
+
+**Open palm, swipe left or right** — switch desktops.
+
+**Open palm, push up** — Mission Control.
+
+**Thumb and pinky out ("telephone hand"), held** — toggle dictation, for typing
+by voice. macOS has no shortcut for this that can be synthesised reliably, so
+assign a real key in System Settings > Keyboard > Dictation and mirror it under
+`[keys]` in `config.toml` (default `f5`).
+
+An open palm that *moves* is a swipe and an open palm that *sits still* is the
+engage toggle, so those two can never be confused. Every held gesture fires once
+and then waits for you to change pose, so holding a palm for three seconds
+toggles control exactly once.
+
+## Install
+
+Requires macOS on Apple silicon and Python 3.11 or 3.12.
+
+```bash
+uv sync            # installs the exact versions in uv.lock
+```
+
+MediaPipe is pinned below 1.0 deliberately: the 1.x macOS arm64 wheels abort
+inside `TensorsToDetectionsCalculator` asking for a Metal service the wheel does
+not ship. `uv.lock` is checked in because that is not the only sharp edge in this
+dependency set — the 0.10 line has also disagreed with itself about NumPy 2 — so
+the combination known to work is recorded rather than re-resolved.
+
+The two model bundles (~11 MB) download themselves into
+`~/.cache/mindcontrol/models/` the first time you run.
+
+Then build the native helper, which is what makes the cursor smooth and lets it
+snap to what you are aiming at:
+
+```bash
+mindcontrol bridge      # compiles native/ and reports where it stands
+```
+
+Needs the Xcode command line tools (`xcode-select --install`). If you skip this
+the app still runs — it posts events straight from Python instead — but you get
+one cursor move per camera frame and nothing snaps or highlights. The menu-bar
+status line ends in `snapping` or `raw pointer` so you can tell which you have.
+
+### As a Mac app
+
+To get an icon you can keep in the Dock instead of a shell you have to keep open:
+
+```bash
+make app        # build/MindControl.app, from its own CPython up
+make install    # copy it into /Applications
+make dmg        # dist/MindControl-<version>.dmg, to install it somewhere else
+```
+
+The first build takes a few minutes because it fetches a whole interpreter and
+every pinned wheel. Afterwards use `make update`, which reinstalls just this
+project and the helper into the existing bundle, syncs the handful of changed
+files into `/Applications` and relaunches — about fifteen seconds. `make help`
+lists the rest.
+
+The bundle carries its own CPython and the whole pinned dependency set, so once
+it is installed it reads nothing out of the checkout it was built from. The
+native helper is built into it and pointed at through `MINDCONTROL_BRIDGE`, and
+`config.toml` is copied into `~/.config/mindcontrol/` the first time it runs,
+after which that copy is yours to edit.
+
+It is a status-bar app, so opening it puts the glyph in the menu bar and nothing
+in the Dock. There is no terminal for it to complain to either, so everything it
+would have printed goes to `~/.local/state/mindcontrol/app.log`.
+
+The icon is drawn rather than drawn on: `packaging/icon.py` renders it with
+CoreGraphics at build time. It is deliberately full-bleed, because macOS 26
+rounds, masks and shadows an app icon itself and reads artwork that arrives with
+its own corners as a picture to inset into a plate. Pass `--plate` for the older
+look.
+
+## Permissions
+
+Three grants are needed, all for whichever app launches the process — your
+terminal, if you start it from a shell, and `MindControl.app` itself if you
+installed the bundle, which is the tidier of the two:
+
+**Camera** — macOS asks the first time. If you miss the prompt, System Settings
+> Privacy & Security > Camera.
+
+**Accessibility** — required to move the cursor and to notice when you touch the
+real mouse. System Settings > Privacy & Security > Accessibility. Without it the
+app runs and tracks your hands perfectly while silently failing to move
+anything, so it says so loudly at startup.
+
+**Menu Bar** — macOS 26 hosts every third-party status item through Control
+Center. System Settings > Menu Bar > Allow in the Menu Bar. Without it the
+process is running and there is nothing to click. If MindControl is not in that
+list, the bundle never registered; rebuild with `make app`.
+
+The native helper needs the same grant **in its own right**, because macOS
+attaches the permission to a binary rather than to a project. That is a feature
+here: granted to `mindcontrol-bridge` it survives rebuilding your virtualenv,
+which a grant made to `.venv/bin/python` does not. The helper also uses it for a
+second purpose — asking the window server what is on screen, which is how it
+knows what to highlight. Without it, motion and clicks still work and snapping
+silently does not, so it says so on startup too.
+
+`make permissions` opens those panes. Installed as a bundle there is only one
+entry to enable, `MindControl`: the helper is spawned by the app and lives
+inside it, so macOS holds the app responsible for what it asks for and the grant
+covers both. Running from a checkout is where the helper needs its own entry,
+because there the responsible process is your terminal.
+
+### Keeping the grants across updates
+
+An ad-hoc signature — the default, because it needs nothing set up — pins the
+bundle's designated requirement to a hash of its own contents:
+
+```
+$ codesign -d -r- /Applications/MindControl.app
+designated => cdhash H"e3bbe92e…"
+```
+
+Every update changes that hash, so macOS sees a different application, asks for
+the camera again and quietly stops honouring the Accessibility entry — while
+still showing its switch as on, because the entry belongs to the copy you built
+last time. That failure is worth recognising: the app runs, tracks your hands,
+and moves nothing, which looks like a bug in the tracking and is a signature.
+
+Two copies of the bundle do the same thing to each other. `build/MindControl.app`
+and `/Applications/MindControl.app` are two applications with one identifier, so
+grant the one you run, which `make run` and `make update` take to be the
+installed one.
+
+Signing with a certificate pins the requirement to the certificate instead, which
+does not change. Any code-signing certificate on your keychain is found and used
+without being asked for:
+
+```
+$ packaging/identity.sh
+Developer ID Application: …
+```
+
+With none, builds are ad-hoc and the grants are given again each time.
+`SIGN_IDENTITY` overrides the choice, and `SIGN_IDENTITY=-` forces ad-hoc. If you
+have no certificate, Keychain Access > Certificate Assistant > Create a
+Certificate… makes one: identity type **Self Signed Root**, certificate type
+**Code Signing**. Grant the three permissions once to a build signed that way and
+later `make update`s keep them.
+
+## Running
+
+```bash
+mindcontrol                      # menu-bar background app
+mindcontrol --debug              # foreground, with the tuning overlay
+mindcontrol --debug --no-overlay # foreground, status line only
+mindcontrol calibrate            # nine-point gaze calibration
+mindcontrol cameras              # list capture devices
+mindcontrol record               # capture a labelled gesture session
+mindcontrol autotune             # fit thresholds to that session
+mindcontrol replay               # run a session back through the engine
+mindcontrol bridge               # build the native helper
+```
+
+When snapping picks the wrong thing, ask the helper what it can see under the
+cursor:
+
+```bash
+native/.build/release/mindcontrol-bridge --inspect
+```
+
+It prints every candidate with its role, its distance, and its size, and marks
+the one it would choose with `->`. This is how every selection bug so far was
+found rather than reasoned about: a container directly under the cursor scoring
+zero and so beating the button inside it, word lookup asking for a fixed 96
+characters of context in documents shorter than that, a panel-sized `AXGroup`
+winning over empty space, and the whole Dock going dark because `AXDockItem` was a
+role the ranking had never been shown. It takes no single-instance claim, so it
+works while the real helper is running.
+
+When the cursor ends up somewhere you did not ask for, the goal, the destination
+and the posted position are three different numbers, and the only useful question
+is which of them disagreed:
+
+```bash
+native/.build/release/mindcontrol-bridge --trace   # one line per tick, on stderr
+```
+
+Every button also prints where it actually landed and what it was aimed at, which
+the per-tick lines cannot show: a click is precisely the moment the cursor stops
+being where the tick said it was. Reading a press resolve to a word 130 px away
+is how the stale-position bug above was found, after two rounds of guessing at it.
+
+The menu-bar glyph is your state at a glance: `◉` engaged, `◐` suspended because
+you touched hardware, `○` off. The menu carries a live status line (mode, hands
+seen, cameras, frame rate, whether gaze is calibrated), an engage toggle, the
+overlay toggle, and calibration.
+
+MediaPipe announces its GL version, its XNNPACK delegate and two feedback
+managers on every start, which buries the output you actually want to read during
+a guided recording. Those lines are suppressed while the models load:
+
+```bash
+MINDCONTROL_VERBOSE=1 mindcontrol record   # put them back
+```
+
+Suppression only covers model construction, and only when it succeeds. If a model
+fails to load, everything it logged on the way down is printed, since that is the
+one time those lines are worth having.
+
+### Calibrating gaze
+
+Run `mindcontrol calibrate`, or pick "Calibrate gaze..." from the menu. Nine
+dots appear; look at each until its ring closes. Escape cancels and leaves any
+existing calibration untouched.
+
+Calibration learns your main display specifically — the one the camera watched
+you look at. Hand movement can roam across every monitor; gaze warps land on
+that one screen. Recalibrate if you move the camera or change seat.
+
+Until you calibrate, everything except gaze works, and the pointer is purely
+hand-driven.
+
+## Using more than one camera
+
+Find out what you have:
+
+```bash
+mindcontrol cameras            # names, resolutions, and which indices work
+mindcontrol cameras --preview  # a frame from each, to see which is which
+```
+
+Names come from AVFoundation, whose ordering has been seen to disagree with
+OpenCV's indices, and there is no shared identifier to reconcile them. `--preview`
+is the only way to be certain, and being certain matters: `primary_gaze` pointed
+at the wrong camera means gaze estimated from a view of the wall.
+
+Then say which cameras to use and which one watches your eyes:
+
+```toml
+[cameras]
+devices = [0, 1]
+primary_gaze = 0
+```
+
+Nothing else changes. Each camera gets its own thread and its own inference, and
+the results are merged by how confident each camera is. The payoff is occlusion:
+a pinch hidden behind your palm from the laptop is obvious from a camera at your
+side, and either one can carry the gesture.
+
+Hand *shape* is combined across cameras by confidence-weighted vote. Hand
+*position* is not averaged — each camera sees you from a different place, so an
+average would describe a hand that exists nowhere. One camera leads for position,
+and when the lead changes the new leader's coordinates are shifted onto where the
+pointer already was, so the handover costs neither a jump nor the motion in that
+frame. Losing the motion mattered: a swipe is judged on accumulated travel, and a
+handover mid-sweep used to erase it.
+
+Gaze runs on the primary camera only. It is the expensive model, and only a
+camera near the screen you are looking at can say anything useful.
+
+More cameras is not automatically better, and the reason is the vote. A camera
+that sees the hand nearly edge-on still gets a say in the blended shape, and its
+say is wrong in a particular direction: fingers foreshorten, so an open palm can
+read as something closed. On one recorded session a phone added as a third view
+was worth two extra clicks and cost five swipes — 7 down to 2 — because the palm
+that has to stay open through a sweep kept flickering. Judge a camera by replaying
+a session with and without it, not by how good its own picture looks.
+
+Three cameras measured at 19ms median per poll on an M4 Max, against a 33ms
+budget at 30fps, so time is not what limits how many you add. Two things to know
+if a camera might come and go — an iPhone over Continuity does:
+
+List it **last**. Indices are positional, so a device that disappears from the
+middle renumbers everything after it, and a config naming cameras by number would
+quietly start pointing at different lenses. Lose the trailing index and the rest
+keep their meaning.
+
+Expect a wait on start. Cameras are given up to twelve seconds to deliver their
+first frame before recording begins, because a Continuity camera has been
+measured taking most of five, and frames captured while one is still waking are
+indistinguishable afterwards from a camera that saw nothing. Any camera that
+never wakes is named, and the session goes ahead without it.
+
+## Tuning
+
+The shipped thresholds are reasoned, not measured. They were chosen against hand
+proportions in the literature, and nobody's hands are the literature. Rather than
+nudging numbers until things feel right, you can measure yours once:
+
+```bash
+mindcontrol record     # ~1 minute: hold each pose when prompted
+mindcontrol autotune   # see what your hands imply, change nothing
+mindcontrol autotune --apply
+```
+
+`record` walks you through the poses and stores the raw landmarks together with
+the pose it asked for. That pairing is what makes it useful: the label says which
+cluster each sample belongs to, so a boundary can be *found* instead of guessed.
+
+**Use one hand and keep the other out of frame.** A second hand resting in shot
+is not idle data — it is a different pose wearing the same label, and it corrupts
+every threshold fitted from that prompt. Both `autotune` and `replay` check for
+this and tell you when a recording cannot answer the question you asked of it.
+
+`autotune` reads those clusters and puts each threshold in the gap between them —
+your pinched distances on one side, your open ones on the other. Percentiles
+rather than extremes, so a single bad frame cannot move a threshold. **If two
+clusters overlap it declines and tells you**, because a tuner that always emits a
+number is indistinguishable from one that emits noise. Without `--apply` it only
+reports. With it, `config.toml` is edited in place, comments intact.
+
+`replay` runs a recording back through the gesture engine offline:
+
+```bash
+mindcontrol replay
+```
+
+It prints what each prompt actually classified as, and what the state machine did
+about it — how many clicks your taps produced, whether a held pinch became one
+drag or a burst of clicks, whether anything fired while control was off. Because
+it reads recorded timestamps rather than the clock, every run is identical, so
+this is also how you tell whether a config change helped or hurt.
+
+Use it before `--apply`, not after. A fit can be sound and still be a regression,
+because several of these thresholds trade one pose against another: raising
+`thumb_extended` until a fist is recognised also makes it harder for a thumb to
+count as *out*, which is what the telephone pose needs. Replay each candidate,
+then write only the ones that earned it:
+
+```bash
+mindcontrol autotune --apply --only thumb_extended,pinch_close
+```
+
+When one gesture needs another attempt and the rest of the script already worked,
+re-record just that part:
+
+```bash
+mindcontrol record --focus pinch   # ~27s
+mindcontrol record --focus swipe   # ~16s
+mindcontrol record --focus poses
+```
+
+Each group carries the prompts its fit *depends on*, not only the failing one. A
+threshold is a boundary between two clusters, so recording just the pinch would
+leave nothing to separate it from — `--focus pinch` therefore also captures the
+open hands that form the other side of that boundary.
+
+Recordings live in `~/.local/state/mindcontrol/sessions/`. They are worth keeping:
+they turn "it feels wrong" into a number, and they let the test suite check real
+gestures long after the moment you performed them.
+
+### Tuning by hand
+
+Open the overlay (`mindcontrol --debug`) and watch the real numbers from your own
+hands in your own light. It shows each hand's skeleton, its classified pose, its
+live pinch distance, `x2` when cameras are being merged, and a small screen proxy
+showing where gaze thinks you are looking.
+
+Every threshold lives in `config.toml`, and "Reload config" applies edits without
+restarting. Distances are in **palm units** — divided by the span from your wrist
+to your middle knuckle — so they hold whether you are at the keyboard or across
+the room.
+
+The knobs worth reaching for first:
+
+- **Cursor too twitchy** — lower `pointer.filter_fc_min`. **Too laggy** — raise it.
+- **Cursor too slow to cross the screen** — raise `pointer.sensitivity` or
+  `pointer.gain_max`.
+- **Clicks not registering** — raise `gestures.pinch_close` toward your measured
+  pinch distance. **Clicks firing on their own** — lower it.
+- **Clicks sticking down** — lower `gestures.pinch_open`. Watch your resting
+  pinch number on the overlay; it must sit clearly above this value.
+- **Taps turning into drags** — raise `gestures.tap_max_ms`.
+- **Open palms triggering when your hand is sideways** — raise
+  `gestures.palm_facing` toward 0.3. It defaults to 0.0, accepting any
+  orientation, because a too-strict setting here would stop you engaging at all.
+- **Accidental swipes** — raise `gestures.swipe_min_speed` or
+  `swipe_min_travel`.
+- **Swipes never firing, though the palm is recognised when you hold it still** —
+  raise `gestures.swipe_grace_ms`. A sweeping palm is blurred and rotating, so the
+  pose drops out partway through; without a grace window the accumulated travel is
+  wiped mid-gesture and no swipe ever completes. 600 ms is enough for most hands.
+  Lower it if a fist just after a swipe fails to scroll.
+- **Adding a camera made swipes *worse*** — check the pose report from
+  `mindcontrol replay` for the prompt that is failing. Handing the lead between
+  cameras is free (the new leader's own movement is stitched onto the track, so
+  `rebases` should stay near zero), but a camera with a poor view still pulls the
+  *shape* average around, and a swipe needs the open palm to survive the sweep.
+  A camera that sees your hand edge-on helps a still pinch and hurts a sweep.
+- **Pinches doing nothing, or scrolling instead of clicking** — your pinch is
+  probably curling the other three fingers, which makes it a fist, and a fist
+  scrolls by design. Keep the middle, ring and little fingers out. No threshold
+  can separate the two: on a curled pinch they measure identically.
+- **Gaze fighting your hand** — lower `gaze.hand_quiet_speed`. **Gaze warping on
+  small corrections** — raise `gaze.warp_min_distance`.
+
+Set `pointer.mode` to `hands` to switch gaze off entirely, or `gaze` to lean on
+it harder.
+
+## How it fits together
+
+```
+Python -- what a hand meant
+  cameras (one thread each)
+  -> hand landmarks per camera        tracking/hands.py
+  -> gaze features on the primary     tracking/gaze.py
+  -> merge cameras into one view      fusion.py
+  -> classify shape, run the machine  geometry.py, gestures/engine.py
+  -> send intents over a socket       control/bridge.py
+
+Swift -- what the cursor does about it        native/Sources/BridgeCore/
+  <- 48-byte datagrams                Protocol.swift, Transport.swift
+  -> integrate motion at display rate Motion.swift
+  -> ask what is on screen            Probe.swift, Targets.swift
+  -> post real input events           Cursor.swift
+  -> draw the highlight               Overlay.swift
+  -> be the only helper running       Solitary.swift
+  -> assemble the above               Run.swift
+```
+
+`BridgeCore` is a library and `native/Sources/Bridge/main.swift` is a two-line
+executable over it, so target selection and the wire format can be tested without
+a screen, an Accessibility grant, or another application to interrogate:
+`swift test --package-path native`.
+
+`pipeline.py` runs the camera loop on a worker thread. `app.py` keeps the main
+thread for the menu bar, because a macOS status item needs the Cocoa run loop.
+`control/modes.py` arbitrates between your hands and your hardware, watching for
+physical input on a private run loop of its own.
+
+Every event the app injects is tagged, and the watcher ignores anything carrying
+that tag. Without it the app would see its own cursor motion, conclude a human
+had grabbed the mouse, and suspend itself the instant it started working. Both
+sides stamp the same tag, so `control/events.py` and `Cursor.swift` have to agree
+about it.
+
+### Why the cursor moved to Swift
+
+Not for speed in the abstract. Three things could not be done from Python, and
+each was measured before it was moved.
+
+**Interpolation.** The camera has an opinion thirty times a second; the display
+can show a new position a hundred and twenty times. Posting one move per camera
+frame is four refreshes of stillness and then a jump, which is what "not smooth"
+was. The helper accumulates deltas into a goal and walks the cursor there with a
+critically damped spring, on a thread that asks for the user-interactive band and
+is never behind the GIL while MediaPipe is running inference. Counted with an
+event tap: **4.9 events per intent at 114 Hz**, 8.4 ms apart, largest step 7 px,
+against 1.0 event per intent 33 ms apart and 20 px each.
+
+**One writer.** Cursor motion used to be posted from wherever a frame arrived, so
+a gaze warp and a hand delta could each be computed from a position the other had
+already invalidated. That is what "overlapping with itself" was. Now one thread
+owns the cursor and *clicks are queued to it too*, which makes the ordering total:
+a press and its release always bracket exactly the motion between them. It costs a
+click one tick — eight milliseconds — and no measured reversal survives it.
+
+One thread is only one writer if there is also one process, and that does not come
+for free. Binding the socket unlinks whatever was there, so a second helper takes
+every frame and looks perfectly healthy while the first sits on its last goal with
+a live motion thread, a live probe, and a second highlight window — started two
+deliberately and got exactly that. So the claim is held as an `flock`, which the
+kernel drops however the holder dies, and the newcomer evicts the incumbent rather
+than refusing to start: an orphan nobody can see must not be able to block every
+future launch. A helper also exits on its own once `getppid()` is 1, because a
+parent that crashes never gets to clean up. `--inspect` deliberately takes no
+claim, since diagnosing a *running* cursor is most of what it is for.
+
+One writer is still not the *only* writer, because the rest of the system has
+never agreed to that. Another application warping the pointer, a dialog taking
+focus, Mission Control — after any of them the helper's idea of where the cursor
+is describes somewhere it no longer is, and every target and pull computed from it
+is wrong in a way nothing on screen reveals until a click lands in the wrong
+place. Measured: two identical drags in a row, the second pressing on a word 130
+px from where the cursor had been put, because the probe was still answering about
+the first. So the real position is compared against the posted one every refresh,
+and any disagreement is adopted as truth — the goal, the velocity, the held
+target and every probe taken before that moment all discarded together.
+
+Only while the cursor is at rest, though, and that restriction is the whole
+reason it is safe: mid-flight the window server is legitimately a frame behind
+what was just posted, and at the top speed allowed here a frame is two hundred
+pixels, indistinguishable from a real jump. At rest there is nothing to mistake.
+Worth knowing what "at rest" has to mean — the cursor stopped moving, *not* the
+cursor reached its goal, because with a target in range the spring settles at the
+goal plus the pull. Comparing against the goal held the check off permanently at
+a 7 px steady-state offset, which is exactly how the bug above survived its first
+fix.
+
+**Knowing what is on screen.** Snapping needs to ask, and asking is synchronous
+IPC into another application's main thread:
+
+| | pyobjc | native Swift |
+|---|---|---|
+| one attribute read | 1140 µs | 382 µs |
+| whole-window tree walk (2419 nodes) | 2490 ms | 4251 ms |
+| single-point hit test | — | 0.43 ms (p95 3.46 ms) |
+| four attributes, batched vs separate | — | 0.14 vs 0.31 ms |
+
+The middle row is the important one: building a target list by walking a window is
+hopeless in *either* language — four seconds, by which time the layout has changed.
+So nothing walks the tree. The primitive that works is the single-point hit test,
+which answers "what is here" rather than "what exists", and nearness is
+reconstructed by asking about the cursor, the point it is heading for, and a ring
+around that. Swift is only 1.2× faster per node, because the cost is the IPC — but
+it is 3× faster per call, and the p95 is why the probe runs on a thread nobody
+waits on, with a messaging timeout so an application wedged on its own main thread
+costs one skipped probe rather than a frozen cursor.
+
+### What snapping actually does
+
+Choosing a target is not finding the closest rectangle. Four corrections turn
+proximity into intent, and they are tunable in `[native]`:
+
+- **Role.** A hit test lands on whatever is deepest at the point, which is usually
+  the group containing the button — 18 of 25 sampled screen points returned an
+  `AXGroup`. A button outranks its container even when the container is nearer.
+- **Scenery is refused outright.** Outranking a container is not enough when it is
+  the only candidate: a Finder window with the cursor in the gap between two icons
+  offered a 614×756 `AXGroup` at zero distance, which won, drew a highlight over
+  most of the window, and pulled nowhere at all — a large target's anchor clamps to
+  where the cursor already was. Nothing is the right answer over empty space.
+  Containers are therefore named and excluded rather than merely outranked. A role
+  the table has never seen is judged by its shape instead of refused, because
+  refusing all of them silently killed the entire Dock, where an icon is an
+  `AXDockItem`.
+- **Heading.** Where the hand is travelling is better evidence than where it
+  currently is, so a target ahead of the cursor beats an equidistant one behind.
+- **Stickiness.** The target already held keeps a bonus, or a cursor resting on the
+  boundary between two buttons alternates every probe and the highlight strobes.
+  This is the pinch detector's hysteresis applied to space instead of time.
+
+The pull is a force, not a jump: it fades to nothing at the edge of the snap
+radius so crossing that boundary is not felt, and a large target's anchor is the
+nearest point on it, so the pull vanishes once you are inside and the hand has
+full freedom again. Exactness comes at click time instead — a click resolves to
+the highlighted target itself, so the highlight never promises something the click
+does not honour. A drag freezes its target, except when it began on a word, in
+which case it keeps snapping to words.
+
+That last exception is what makes selecting a range land on whole ones, and it
+needs the two ends resolved differently. A click on a word wants its middle, which
+is where a caret belongs inside a word — but a press is the start of a selection,
+so it takes the word's *leading* edge, and the release takes the far edge of the
+word it ends on, measured against the word it began on so that dragging leftwards
+takes the first word entirely rather than the last. Without it, dragging from
+"quick" to "lazy" in TextEdit selected `ick brown fox jumps over the la`; with it,
+the same drag selects whole words, and a drag that stays inside one word selects
+that word.
+
+The highlight is a single Core Animation layer, retargeted rather than redrawn.
+That is why it cannot overlap itself, and why it glides at display rate even when
+the probe underneath it is stuttering.
+
+### Text, and where it stops working
+
+Aiming at a word uses three calls: the point maps to a character index, a clamped
+window of text around that index gives the word's extent, and one bounds query
+turns the extent into a rectangle. Verified in TextEdit: a 23×14 box on the word
+directly under the cursor, and neighbouring words picked up from the probe ring
+when the cursor is on a space between them.
+
+Two things had to be true for that to be trustworthy. Prose is aimed *inside*, not
+at, so a text element is only a target in its own right if it is short enough to be
+a field — otherwise a 656×384 document view wins by default and outlines the very
+text it is supposed to be pointing at. And there is no error for "no glyph here":
+a position past the end of the text answers with index zero, indistinguishable from
+a real hit on the first character, which offered the document's opening word to a
+point measured 92 px away in the bottom margin. Since the bounds of the answer are
+already being fetched, they are checked against the question — if the box is not on
+the line that was asked about, the answer is discarded. Clicking blank space still
+places the caret there; it simply no longer lights up a word to promise otherwise.
+
+Chromium and Electron windows answer the character count with nothing, so no
+amount of asking will find a word inside one — including in this project's own
+editor. Whole-element snapping still works there; word snapping does not.
+`PixelTextLocator` in `Probe.swift` is the seam for the way around it, reading
+glyph boxes from the pixels instead. It is deliberately unimplemented: Vision's
+fast recogniser is affordable at 4 ms on a 240×120 patch, but one-shot capture
+measured 53 ms regardless of region size, so doing it properly means holding a
+persistent `SCStream` on the focused window and asking for Screen Recording
+permission on top of Accessibility. Every caller already copes with a locator that
+declines to answer.
+
+## Tests
+
+```bash
+uv sync --group dev
+uv run pytest
+```
+
+Three layers, deliberately separated:
+
+- **Logic** — the state machine and the geometry, driven by stated measurements
+  and by synthetic hands built to human proportions. Fast, and independent of
+  cameras, lighting, and hands.
+- **Machinery** — a scripted session replayed end to end, covering session I/O,
+  re-measurement, and every gesture the engine can emit. Deterministic, so it
+  runs anywhere.
+- **Reality** — the same assertions against your own recording. These *skip*
+  until you have run `mindcontrol record`, rather than invent input, because a
+  test that fabricates its own data would report success while checking nothing
+  about the person using this.
+
+When a reality test fails but the matching machinery test passes, the pipeline is
+fine and the thresholds do not suit those hands: run `mindcontrol autotune`.
+
+## Limits worth knowing
+
+Gaze from a webcam is good for regions, not for pixels; expect a few percent of
+screen error, which is exactly why the hand does the last inch.
+
+There is no air keyboard. Typing goes through dictation.
+
+Gaze is calibrated for one display and one seating position.
+
+Hand tracking needs your hand reasonably lit and reasonably unoccluded. A second
+camera helps more than better thresholds.