@vortex-os/computer-use 0.2.1 → 0.7.0

package/README.md

@@ -2,11 +2,11 @@
 
 Read-only **screen perception** for VortEX agents, exposed as an MCP server. It lets an agent *see* what is on screen — read a window's structure, capture a region as an image, and watch for on-screen changes — without ever moving the mouse or typing. It layers on `@vortex-os/base` but also works standalone.
 
-> **Status: 0.1.0, Windows-first, read-only.** Mouse/keyboard **control is intentionally out of scope** for this release — this package only *perceives*. macOS/Linux backends are not yet implemented.
+> **Status: 0.5.0, Windows-first, read-only.** Mouse/keyboard **control is intentionally out of scope** for this release — this package only *perceives*. macOS/Linux backends are not yet implemented.
 
 ## What it is
 
-An MCP (Model Context Protocol) server that exposes six perception tools over stdio:
+An MCP (Model Context Protocol) server that exposes nine perception tools over stdio:
 
 | Tool | What it does | Cost |
 |---|---|---|
@@ -15,15 +15,115 @@ An MCP (Model Context Protocol) server that exposes six perception tools over st
 | `capture_screen` | Pixel capture (PNG) for what structure can't reach — canvases, games, remote desktops. Target by window, region, monitor, or cursor box. | image |
 | `watch_capture` | Captures N frames at an interval in one process; with `changeOnly`, keeps only changed frames. | image(s) |
 | `poll_change` | One non-blocking "did it change?" probe; returns a change percentage and (optionally) an image. Poll it on an interval to watch without blocking. | metadata, image optional |
+| `start_watch` | Watch a fixed target in the **background** (non-blocking) with a built-in **noise filter** that keeps only meaningful, settled changes. Works on video/games that change every frame. | runs in background |
+| `get_events` | Collect the buffered changes a `start_watch` has accumulated — batched (a few looks for a long watch); each event carries the settled frame. | metadata + image(s) |
+| `stop_watch` | Stop a background watch and discard its buffer. | — |
 | `beep` | A system beep, to get the user's attention while they look elsewhere. | — |
 
 The design favors **structure first, pixels as fallback**: `read_ui` is cheap and precise for ordinary apps; `capture_screen` is for content that has no accessibility tree (games, custom canvases).
 
+## Watching for changes (the noise filter + event buffer)
+
+`poll_change` is the manual primitive — *you* poll it on a loop. `start_watch` is the hands-off version: it runs a **background loop with a noise filter** and buffers what matters, so you can watch a busy screen without drowning in frames.
+
+The problem it solves: on video, games, or scrolling, the screen changes *every* frame, so raw change-detection fires constantly on the ripples of one activity. The filter combines **debounce** (wait for motion to settle, then capture a clean frame — quality) with **cooldown** (at most one event per N seconds — frequency) and **hysteresis** (ambient jitter never even wakes it), plus an anti-starvation `maxWait` so a continuously-moving screen still yields periodic snapshots instead of going silent.
+
+```
+start_watch { region|window|monitor, watchId? }   -> returns immediately, watch runs in the background
+   ... do other things ...
+get_events  { watchId? }                           -> the settled changes so far (frames + metadata), batched
+stop_watch  { watchId? }                           -> end it (omit watchId to stop all)
+```
+
+**Calibration.** The defaults assume a meaningful target: a playing video jitters ~2.5–4% frame-to-frame and a scene cut jumps ~16% — so `activityThreshold` (default 8%) sits between them, ignoring the jitter and reporting the cut. Because the change metric is whole-frame, a tiny local change (a clock, a toast) is a small fraction of the frame; **target the region where the change happens** so it reads as a large change. Tune `activityThreshold` / `quietThreshold` / `debounceQuietMs` / `cooldownMs` / `maxWaitMs` per call. The buffer is **memory-only** (no screen history on disk) with count, byte, and 5-minute TTL caps; watches auto-stop after 30 minutes.
+
+## Reflex alerts — sub-second voice, no cloud round-trip
+
+`get_events` is the *brain* path (the agent looks, judges, and replies — seconds, because it makes a cloud LLM call). For things you need to hear the instant they happen, `start_watch` takes `triggers` that fire **locally, with no LLM in the loop**, so the alert reaches you in well under a second:
+
+```
+start_watch {
+  window: "MyGame",
+  triggers: [
+    { action: "beep",  threshold: 20 },                          // a sound
+    { action: "say",   threshold: 20, say: "적 출현" },           // speak a FIXED phrase (Korean TTS)
+    { action: "ocr",   threshold: 12, dwellMs: 700 }             // OCR the region and read the text aloud
+  ]
+}
+```
+
+A trigger fires the moment the watched region changes past its `threshold` (with hysteresis + a per-trigger `cooldownMs`). The fast alert is the *reflex*; the agent's judged commentary still follows on the next `get_events`. Speaking uses the built-in Windows voice (System.Speech); reading text uses the built-in offline OCR (no install, no GPU).
+
+**Safety (this matters).** OCR text is *screen content* — untrusted. It is never spoken raw: it gets a spoken **provenance prefix** (`화면 글자: …`), control/secret-token shaping, and a **global speech budget** (capped utterances and seconds per minute, no overlapping speech, auto-mute on sustained noise) so an on-screen string can never voice fake instructions or flood you. A fixed `say` phrase is the safest action (you author the words; a trigger only controls *when*). If the voice/OCR engine isn't available, triggers degrade quietly (a `beep` still works).
+
+### Optional: higher-quality neural voice (Supertonic) + audio ducking
+
+The default voice is the built-in Windows one (System.Speech / Heami) — zero install, but robotic. For a much more natural voice, install **Supertonic 3** (Supertone): an on-device ONNX neural TTS (Korean + 30 more languages; code MIT, weights OpenRAIL-M — commercial use OK). One-time model download (~380 MB), then fully offline and fast (~0.5 s/sentence on CPU):
+
+```
+node scripts/fetch-supertonic.mjs       # downloads models to ~/.vortex/computer-use/supertonic-3
+npm i onnxruntime-node                   # the runtime (optionalDependency)
+```
+
+Once the models are present, the speak path uses them automatically (`engine: "auto"`) and falls back to Heami if anything is missing — it never goes mute.
+
+**Audio ducking.** While the companion speaks, other apps' audio (game / music / video) is briefly lowered per-app and **restored exactly** when it finishes, so the voice stands out. On by default. *DRM-protected audio (e.g. Netflix) cannot be ducked* — that protected path bypasses Windows volume control; normal app/game audio ducks fine.
+
+Configure in `computer-use.config.json` (`tts` section) or via env (env wins). Defaults shown:
+
+```json
+{ "tts": { "engine": "auto", "voice": "F1", "duck": true, "duckFactor": 0.3 } }
+```
+
+`engine` `auto|supertonic|heami` · `voice` `F1..F5/M1..M5` · `duckFactor` `0..1` (others drop to this fraction; lower = quieter). Env: `VORTEX_CU_TTS_ENGINE` / `VORTEX_CU_TTS_VOICE` / `VORTEX_CU_DUCK=off` / `VORTEX_CU_DUCK_FACTOR`. Restart the server after changing.
+
+### Optional: a local vision model (the `vision` trigger)
+
+A `vision` trigger describes the *scene* (not just its text) via a **local** vision-language model — smarter than OCR, still no cloud round-trip. It is **off by default and GPU-gated**: it runs only when you point it at a reachable, fast-enough local endpoint. Everything above works with **no GPU**; this just adds a smarter local description where the hardware allows. On a machine that can't run it, a `vision` trigger **degrades to `ocr`**.
+
+Point it at any OpenAI-compatible vision endpoint — `llama.cpp`'s `llama-server` (with `--mmproj`), `llamafile`, Ollama, LM Studio — via env (machine-local, never synced):
+
+```
+VORTEX_CU_VLM_ENDPOINT=http://127.0.0.1:8080/v1   # set this to enable; presence = on
+VORTEX_CU_VLM_MODEL=gemma-4-e2b-it                # any small VLM (e2b ~2GB is a good light default)
+VORTEX_CU_VLM_KEY=...                             # optional bearer token
+VORTEX_CU_VLM_ALLOW_REMOTE=1                      # only if the endpoint is NOT on this machine (off by default)
+VORTEX_CU_VLM_SLA_MS=6000                         # gate: if even a tiny probe is slower than this, stay off
+```
+
+How it stays safe (design §23.2/§24): only the *intent* (endpoint set or not) is configuration — the address/secret are machine-local env, never synced, so the same repo on a CPU-only machine simply runs without it. A **loopback** endpoint (same machine) is allowed by default; a **cross-network** one (LAN/VPN/another box) is **off unless you opt in**. The session **probes** the endpoint with a *synthetic* 1×1 image (never a real screen crop) to measure latency before trusting it; a real crop is sent only on an actual `vision` trigger, through the same denylist gate. The model's reply is **untrusted** — spoken with a `로컬 비전: …` prefix, shaped, and rate-limited, and the prompt tells the model to describe only (never follow on-screen instructions). `probe` reports the VLM's availability when you've configured one.
+
+## Adaptive companion — classify the activity, branch the help
+
+`classify_activity` lets the companion **figure out what you're doing from the first screenshot** and adapt, instead of being told. One read-only call returns:
+
+```json
+{ "class": "GAME", "process": "eldenring", "title": "...", "notificationState": "BUSY",
+  "interruptible": false, "canvas": true, "uiaCount": 1, "fullscreen": true,
+  "profile": { "proactive": true, "cadenceSec": 30, "mode": "periodic" }, "needsChangeRate": true }
+```
+
+It combines cheap signals — foreground **process** + **window title**, the Windows **interruptibility state** (`SHQueryUserNotificationState`), **UIA element count** (a near-empty tree on a screen-filling window = a GPU game/video canvas), and whether it **fills the screen** — into a class: `GAME · DEV · MEDIA · BROWSING · PRODUCTIVITY · UNKNOWN`, each with a help **profile**.
+
+The profiles branch the behavior (full design in [`docs/adaptive-companion.md`](docs/adaptive-companion.md)):
+
+| Class | Default | Speaks when | Cadence |
+|---|---|---|---|
+| **Strategy / sim game** | proactive | quiet stretch (your turn) + risk/opportunity | ~30 s during active play |
+| **Fast-action game** | break-gated | a menu/pause/death screen opens | one cue per break; *says up front it can't coach mid-fight* |
+| **Software dev** | silent | an error/failed build/stack trace appears | event-driven, not periodic |
+| **Media / browsing / docs** | silent | only on request | never proactive |
+
+For a `GAME`, `needsChangeRate` tells the agent to take a couple of `poll_change` reads to split **fast-action** (too fast to coach — break-gated only) from **strategy** (coachable, periodic). Honesty is built in: it never pretends to coach a game it can't follow, and it won't talk over media. The **interruptibility state** gates every utterance, on top of the global speech budget. Explicit user requests ("tell me when X happens", "be quieter") layer on as reflex triggers / cadence overrides.
+
+Tune it in `computer-use.config.json` (`companion` section): `uiaCanvasMax` (the canvas cutoff) and per-class `profiles` (e.g. `GAME.cadenceSec: 20` for chattier coaching). Env: `VORTEX_CU_UIA_CANVAS_MAX`.
+
 ## What it is NOT
 
 - **Not control.** No clicking, typing, or app automation. Perception only.
+- **Not real-time for *judgment*.** Reflex `triggers` deliver a sub-second beep / fixed phrase / OCR readout, but anything the agent has to *think* about (a judged message) is seconds-scale — it makes a cloud call. Good for alerts, translation, and watching-alongside; not for reflex-speed decisions.
 - **Not comprehensive secret protection.** See *Privacy & redaction* below — the denylist is the real control; field-level masking is best-effort and does not catch plaintext secrets sitting in arbitrary windows.
-- **Not cross-platform yet.** Windows only in 0.1.0.
+- **Not cross-platform yet.** Windows only in 0.3.0.
 
 ## Install
 
@@ -68,7 +168,10 @@ Each perception call appends one metadata line (timestamp, tool, output size, a
 ## Verify
 
 ```
-npm run verify   # node scripts/verify.mjs — needs a desktop session; captures the real screen
+npm run verify        # node scripts/verify.mjs — needs a desktop session; captures the real screen
+npm run test:filter   # node scripts/test-noise-filter.mjs — pure unit tests, no screen needed
+npm run test:speech   # node scripts/test-speech-safety.mjs — pure unit tests (provenance/shaping/budget)
+npm run test:vlm      # node scripts/test-vlm.mjs — pure unit tests (config/trust-tier/protocol)
 ```
 
-Exercises every tool plus the redaction/audit gate (denylist blocking across all capture modes, no over-block, no title leak, audit written with no plaintext).
+`verify` exercises every tool plus the redaction/audit gate (denylist blocking across all capture modes, no over-block, no title leak, audit written with no plaintext). The `test:*` scripts check the noise-filter, speech-safety, and VLM-protocol logic deterministically (no screen/audio/network). Three live harnesses drive the real screen: `node scripts/verify-watch.mjs` (background watch: settle → event → frame, denylist blindness, cleanup), `node scripts/verify-reflex.mjs` (reflex triggers → local speech, rendered to WAV so it stays silent), and `node scripts/verify-vlm.mjs` (the `vision` path against a mock local endpoint: synthetic-probe, real-crop only on a trigger, degrade-to-OCR).

package/computer-use.config.example.json

@@ -8,5 +8,21 @@
   "_examples": {
     "denyWindowTitles": ["Bitwarden", "1Password", "KeePass", "Online Banking"],
     "denyProcesses": ["Bitwarden", "1Password", "KeePassXC", "keeper"]
+  },
+  "_tts_comment": "Spoken-voice settings. engine 'auto' uses the higher-quality Supertonic neural voice when its models are present (run `node scripts/fetch-supertonic.mjs` once, ~380MB), else falls back to the built-in Windows voice ('heami'). 'duck' lowers OTHER apps' audio while the companion speaks (per-app, restored after); 'duckFactor' is how far they drop (0.3 = to 30%; lower = quieter). NOTE: DRM-protected audio (e.g. Netflix) can't be ducked — that's the app's protection, not a limitation here. Env overrides win over this file: VORTEX_CU_TTS_ENGINE / VORTEX_CU_TTS_VOICE / VORTEX_CU_DUCK(=off) / VORTEX_CU_DUCK_FACTOR. Restart the server after changing.",
+  "tts": {
+    "engine": "auto",
+    "voice": "F1",
+    "duck": true,
+    "duckFactor": 0.3
+  },
+  "_companion_comment": "Adaptive companion (classify_activity). 'uiaCanvasMax' is the UIA element-count cutoff below which a screen-filling window is treated as a GPU canvas (game/video) — raise it if a game with some on-screen widgets is misread as an app. 'profiles' overrides per-class cadence/proactivity, e.g. make game coaching chattier with GAME.cadenceSec=20. Classes: GAME, DEV, MEDIA, BROWSING, PRODUCTIVITY, UNKNOWN. Env override: VORTEX_CU_UIA_CANVAS_MAX. See docs/adaptive-companion.md. Restart the server after changing.",
+  "companion": {
+    "uiaCanvasMax": 5,
+    "profiles": {
+      "GAME": { "cadenceSec": 30, "proactive": true },
+      "DEV": { "proactive": false },
+      "MEDIA": { "proactive": false }
+    }
   }
 }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@vortex-os/computer-use",
-  "version": "0.2.1",
-  "description": "Add-on — read-only screen perception (structured UIA tree + pixel fallback + change watch) exposed as an MCP server, layered on @vortex-os/base. Windows-first. Control (mouse/keyboard) is intentionally out of scope.",
+  "version": "0.7.0",
+  "description": "Add-on — read-only screen perception (structured UIA tree + pixel fallback + noise-filtered background watch with an event buffer + sub-second reflex alerts: beep / fixed-phrase / OCR or optional local-VLM description spoken locally, optional higher-quality Supertonic neural TTS with per-app audio ducking, adaptive companion that classifies the on-screen activity and branches its help) exposed as an MCP server, layered on @vortex-os/base. Windows-first. Control (mouse/keyboard) is intentionally out of scope.",
   "license": "MIT",
   "author": "vortex-os-project",
   "homepage": "https://github.com/vortex-os-project/vortex#readme",
@@ -13,10 +13,20 @@
   "type": "module",
   "files": [
     "scripts/mcp-stdio.mjs",
+    "scripts/noise-filter.mjs",
+    "scripts/speech-safety.mjs",
+    "scripts/vlm.mjs",
+    "scripts/ocr.ps1",
+    "scripts/speak.ps1",
+    "scripts/speak-supertonic.mjs",
+    "scripts/audio-duck.ps1",
+    "scripts/fetch-supertonic.mjs",
     "scripts/worker.ps1",
     "scripts/lib.ps1",
     "scripts/probe.ps1",
     "scripts/read-ui.ps1",
+    "scripts/classify.ps1",
+    "scripts/activity.mjs",
     "scripts/point-to-ask.ps1",
     "computer-use.config.example.json",
     "README.md"
@@ -33,7 +43,11 @@
     }
   },
   "scripts": {
-    "verify": "node scripts/verify.mjs"
+    "verify": "node scripts/verify.mjs",
+    "test:filter": "node scripts/test-noise-filter.mjs",
+    "test:speech": "node scripts/test-speech-safety.mjs",
+    "test:vlm": "node scripts/test-vlm.mjs",
+    "test:activity": "node scripts/test-activity.mjs"
   },
   "peerDependencies": {
     "@vortex-os/base": ">=0.3.0 <1.0.0"
@@ -44,7 +58,8 @@
     }
   },
   "optionalDependencies": {
-    "@modelcontextprotocol/sdk": "^1.21.0"
+    "@modelcontextprotocol/sdk": "^1.21.0",
+    "onnxruntime-node": "^1.19.2"
   },
   "engines": {
     "node": ">=22"

package/scripts/activity.mjs

@@ -0,0 +1,92 @@
+// computer-use — activity classifier (pure, testable). Turns the raw signals from classify.ps1
+// (foreground process/title, notification state, UIA count, fullscreen) into an activity CLASS and a help
+// PROFILE the companion uses to pick its cadence/triggers/style. See docs/adaptive-companion.md.
+//
+// Deliberately conservative: process name is the strongest prior; UIA-emptiness marks a GPU canvas
+// (game/video) only when the process isn't a known browser/player; everything is overridable by config and by
+// explicit user requests. All thresholds are starting points to calibrate, surfaced via opts.
+
+// Known-app tables — lowercase process names, no ".exe". Extend via opts.apps.{dev,media,browser,productivity}.
+const DEV = ['code', 'cursor', 'devenv', 'rider64', 'idea64', 'pycharm64', 'webstorm64', 'clion64', 'goland64',
+  'sublime_text', 'notepad++', 'windowsterminal', 'wt', 'powershell', 'pwsh', 'cmd', 'conhost', 'alacritty',
+  'wezterm', 'hx', 'nvim', 'vim', 'emacs'];
+const MEDIA = ['mpv', 'vlc', 'mpc-hc64', 'mpc-be64', 'potplayermini64', 'potplayer', 'wmplayer', 'smplayer',
+  'kodi', 'plex', 'plexmediaplayer', 'netflix'];
+const BROWSER = ['chrome', 'msedge', 'firefox', 'brave', 'opera', 'vivaldi', 'arc', 'librewolf'];
+const PRODUCTIVITY = ['winword', 'excel', 'powerpnt', 'onenote', 'acrobat', 'acrord32', 'sumatrapdf', 'foxitpdfreader',
+  'notepad', 'obsidian', 'notion', 'hwp', 'soffice'];
+
+// Window-title hints that a browser tab is actually video (so a browser → MEDIA, not BROWSING).
+const VIDEO_TITLE = /youtube|netflix|twitch|vimeo|prime\s*video|disney\+?|wavve|tving|watcha|\bwatch\b/i;
+
+// SHQueryUserNotificationState values.
+const NS_NAME = { 1: 'NOT_PRESENT', 2: 'BUSY', 3: 'D3D_FULLSCREEN', 4: 'PRESENTATION', 5: 'ACCEPTS', 6: 'QUIET_TIME', 7: 'APP' };
+
+// Per-class help profile defaults (overridable via config `companion` section).
+export const PROFILES = {
+  GAME: { proactive: true, cadenceSec: 30, mode: 'periodic',
+    note: 'sample change-rate (poll_change) to split fast-action vs strategy; fast-action = break-gated only, announce the limit once' },
+  DEV: { proactive: false, cadenceSec: 0, mode: 'event-error',
+    note: 'silent until an error/build failure/stack trace; scaffolding help on request' },
+  MEDIA: { proactive: false, cadenceSec: 0, mode: 'silent',
+    note: 'on explicit request only; do not talk over it; DRM audio cannot be ducked' },
+  BROWSING: { proactive: false, cadenceSec: 0, mode: 'silent',
+    note: 'on request: summarize / translate / explain' },
+  PRODUCTIVITY: { proactive: false, cadenceSec: 0, mode: 'quiet',
+    note: 'on request; optional gentle risk flags' },
+  UNKNOWN: { proactive: false, cadenceSec: 0, mode: 'low-intrusion',
+    note: 'offer help once, then stay quiet until asked' },
+};
+
+const has = (list, p) => list.includes(p);
+
+/**
+ * Derive activity class + profile from raw signals.
+ * @param {object} raw  { process, title, notificationState, uiaCount, uiaCapped, fullscreen, ... }
+ * @param {object} opts { uiaCanvasMax=5, apps?:{dev,media,browser,productivity}, profiles?:{<CLASS>:{cadenceSec?,proactive?,mode?}} }
+ */
+export function classifyActivity(raw = {}, opts = {}) {
+  const uiaCanvasMax = opts.uiaCanvasMax ?? 5;
+  const apps = opts.apps || {};
+  const dev = apps.dev || DEV, media = apps.media || MEDIA, browser = apps.browser || BROWSER, prod = apps.productivity || PRODUCTIVITY;
+
+  const proc = String(raw.process || '').toLowerCase();
+  const title = String(raw.title || '');
+  const ns = Number(raw.notificationState || 0);
+  const nsName = NS_NAME[ns] || 'UNKNOWN';
+  const interruptible = ns === 5; // only ACCEPTS_NOTIFICATIONS is unconditionally clear to speak
+  // uiaCount is null when the UIA walk FAILED (≠ "found 0") — so a UIA error never reads as an empty canvas.
+  const hasUia = typeof raw.uiaCount === 'number';
+  // A capped count means there were MORE elements than the backend cap → never a sparse canvas, even if a large
+  // uiaCanvasMax is configured. So canvas requires a real, uncapped, below-threshold count.
+  const canvas = hasUia && !raw.uiaCapped && raw.uiaCount < uiaCanvasMax;
+
+  let cls;
+  if (has(dev, proc)) cls = 'DEV';
+  else if (has(media, proc)) cls = 'MEDIA';
+  else if (has(browser, proc)) cls = VIDEO_TITLE.test(title) ? 'MEDIA' : 'BROWSING';
+  else if (has(prod, proc)) cls = 'PRODUCTIVITY';
+  else if (canvas && proc) cls = 'GAME';                 // screen-canvas app that isn't a known player/browser
+  else if (hasUia && raw.uiaCount >= uiaCanvasMax) cls = 'PRODUCTIVITY'; // rich tree, unknown app → reading/productivity
+  else cls = 'UNKNOWN';
+
+  // Profile defaults, with optional per-class overrides from the `companion` config (cadence/proactive/mode).
+  const profile = { ...PROFILES[cls], ...((opts.profiles && opts.profiles[cls]) || {}) };
+
+  return {
+    class: cls,
+    process: raw.process || '',
+    title,
+    notificationState: nsName,
+    interruptible,
+    canvas,
+    uiaCount: hasUia ? raw.uiaCount : null,
+    fullscreen: !!raw.fullscreen,
+    profile,
+    // GAME needs a change-rate sample to split fast-action (break-gated) vs strategy (periodic) — the agent
+    // takes a couple of poll_change reads and applies the thresholds in docs/adaptive-companion.md.
+    needsChangeRate: cls === 'GAME',
+  };
+}
+
+export default classifyActivity;

package/scripts/audio-duck.ps1

@@ -0,0 +1,180 @@
+# computer-use — audio ducking helper (pwsh; WASAPI Core Audio via inline C#, NO install).
+#
+# When the companion speaks, briefly lower OTHER apps' audio sessions (game / video / music) so the voice
+# stands out, then restore EXACTLY on completion. Per-app, not master volume — our own voice is excluded.
+#
+# SAFETY: restore is the whole point. Volumes are snapshotted and always restored in a finally block, even if
+# playback throws or the process is asked to stop — a failed restore would leave the user's other apps quiet.
+#
+# Two uses:
+#   1. Dot-source and call [CU.AudioDuck]::Duck(factor, excludePids) -> handle ; [CU.AudioDuck]::Restore(handle).
+#      (speak.ps1 / Heami wraps System.Speech this way, excluding its own $PID.)
+#   2. -WavPath <wav>: duck (excluding THIS process), play the WAV via SoundPlayer, restore in finally.
+#      (speak-supertonic.mjs spawns this to play its synthesized WAV.)
+#
+# factor is the multiplier applied to other sessions (0.45 = reduce to 45%). 1.0 disables ducking.
+param(
+  [string]$WavPath = '',
+  [double]$Factor = 0.45,
+  [int[]]$ExcludePid = @()
+)
+$ErrorActionPreference = 'Stop'
+try { [Console]::OutputEncoding = [System.Text.UTF8Encoding]::new($false) } catch {}
+
+if (-not ('CU.AudioDuck' -as [type])) {
+Add-Type -TypeDefinition @'
+using System;
+using System.Collections.Generic;
+using System.Runtime.InteropServices;
+
+namespace CU {
+  [ComImport, Guid("BCDE0395-E52F-467C-8E3D-C4579291692E")] class MMDeviceEnumeratorComObject { }
+
+  enum EDataFlow { eRender = 0, eCapture = 1, eAll = 2 }
+  enum ERole { eConsole = 0, eMultimedia = 1, eCommunications = 2 }
+
+  [ComImport, Guid("A95664D2-9614-4F35-A746-DE8DB63617E6"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IMMDeviceEnumerator {
+    [PreserveSig] int EnumAudioEndpoints(EDataFlow dataFlow, int dwStateMask, out IMMDeviceCollection ppDevices);
+    [PreserveSig] int GetDefaultAudioEndpoint(EDataFlow dataFlow, ERole role, out IMMDevice ppEndpoint);
+  }
+
+  [ComImport, Guid("0BD7A1BE-7A1A-44DB-8397-CC5392387B5E"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IMMDeviceCollection {
+    [PreserveSig] int GetCount(out uint pcDevices);
+    [PreserveSig] int Item(uint nDevice, out IMMDevice ppDevice);
+  }
+
+  [ComImport, Guid("D666063F-1587-4E43-81F1-B948E807363F"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IMMDevice {
+    [PreserveSig] int Activate(ref Guid iid, int dwClsCtx, IntPtr pActivationParams,
+      [MarshalAs(UnmanagedType.IUnknown)] out object ppInterface);
+  }
+
+  [ComImport, Guid("77AA99A0-1BD6-484F-8BC7-2C654C9A9B6F"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IAudioSessionManager2 {
+    [PreserveSig] int NotImpl1();
+    [PreserveSig] int NotImpl2();
+    [PreserveSig] int GetSessionEnumerator(out IAudioSessionEnumerator SessionEnum);
+  }
+
+  [ComImport, Guid("E2F5BB11-0570-40CA-ACDD-3AA01277DEE8"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IAudioSessionEnumerator {
+    [PreserveSig] int GetCount(out int SessionCount);
+    [PreserveSig] int GetSession(int SessionCount, out IAudioSessionControl Session);
+  }
+
+  [ComImport, Guid("F4B1A599-7266-4319-A8CA-E70ACB11E8CD"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IAudioSessionControl { }
+
+  [ComImport, Guid("bfb7ff88-7239-4fc9-8fa2-07c950be9c6d"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface IAudioSessionControl2 {
+    // 9 inherited IAudioSessionControl slots (unused) ...
+    [PreserveSig] int N1();  [PreserveSig] int N2();  [PreserveSig] int N3();
+    [PreserveSig] int N4();  [PreserveSig] int N5();  [PreserveSig] int N6();
+    [PreserveSig] int N7();  [PreserveSig] int N8();  [PreserveSig] int N9();
+    // IAudioSessionControl2 own slots:
+    [PreserveSig] int GetSessionIdentifier([MarshalAs(UnmanagedType.LPWStr)] out string s);
+    [PreserveSig] int GetSessionInstanceIdentifier([MarshalAs(UnmanagedType.LPWStr)] out string s);
+    [PreserveSig] int GetProcessId(out uint pid);
+    [PreserveSig] int IsSystemSoundsSession();
+  }
+
+  [ComImport, Guid("87CE5498-68D6-44E5-9215-6DA47EF883D8"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
+  interface ISimpleAudioVolume {
+    [PreserveSig] int SetMasterVolume(float level, ref Guid ctx);
+    [PreserveSig] int GetMasterVolume(out float level);
+    [PreserveSig] int SetMute(bool mute, ref Guid ctx);
+    [PreserveSig] int GetMute(out bool mute);
+  }
+
+  public class Handle { internal ISimpleAudioVolume Vol; public float Original; public uint Pid; }
+
+  public static class AudioDuck {
+    static Guid IID_IAudioSessionManager2 = new Guid("77AA99A0-1BD6-484F-8BC7-2C654C9A9B6F");
+    static Guid CTX = Guid.Empty;
+
+    static IAudioSessionEnumerator SessionsForDevice(IMMDevice dev) {
+      object o; if (dev.Activate(ref IID_IAudioSessionManager2, 23 /*CLSCTX_ALL*/, IntPtr.Zero, out o) != 0) return null;
+      var mgr = (IAudioSessionManager2)o;
+      IAudioSessionEnumerator en; if (mgr.GetSessionEnumerator(out en) != 0) return null;
+      return en;
+    }
+
+    // Lower every session (except excludePids and system-sounds) to original*factor, across ALL active render
+    // devices (not just the default endpoint) so virtual-audio routing (e.g. VoiceMeeter) is still caught.
+    // Returns restore handles. DEVICE_STATE_ACTIVE = 0x1.
+    public static List<Handle> Duck(double factor, int[] excludePids) {
+      var handles = new List<Handle>();
+      var deo = (IMMDeviceEnumerator)(new MMDeviceEnumeratorComObject());
+      IMMDeviceCollection coll;
+      if (deo.EnumAudioEndpoints(EDataFlow.eRender, 0x1, out coll) != 0 || coll == null) return handles;
+      uint dcount; if (coll.GetCount(out dcount) != 0) return handles;
+      var excl = new HashSet<uint>(); foreach (var p in excludePids) excl.Add((uint)p);
+      for (uint di = 0; di < dcount; di++) {
+        IMMDevice dev; if (coll.Item(di, out dev) != 0 || dev == null) continue;
+        var en = SessionsForDevice(dev); if (en == null) continue;
+        int count; if (en.GetCount(out count) != 0) continue;
+        for (int i = 0; i < count; i++) {
+          IAudioSessionControl ctl; if (en.GetSession(i, out ctl) != 0 || ctl == null) continue;
+          try {
+            var c2 = (IAudioSessionControl2)ctl;
+            if (c2.IsSystemSoundsSession() == 0) continue; // S_OK(0) means IS system sounds -> skip it
+            uint pid; if (c2.GetProcessId(out pid) != 0) continue;
+            if (excl.Contains(pid)) continue;
+            var vol = (ISimpleAudioVolume)ctl;
+            float cur; if (vol.GetMasterVolume(out cur) != 0) continue;
+            var h = new Handle { Vol = vol, Original = cur, Pid = pid };
+            vol.SetMasterVolume((float)(cur * factor), ref CTX);
+            handles.Add(h);
+          } catch { }
+        }
+      }
+      return handles;
+    }
+
+    public static void Restore(List<Handle> handles) {
+      if (handles == null) return;
+      foreach (var h in handles) { try { h.Vol.SetMasterVolume(h.Original, ref CTX); } catch { } }
+    }
+
+    // Inspection helpers (for the isolation test): current live level / snapshotted original per handle.
+    public static float[] Levels(List<Handle> handles) {
+      var a = new float[handles.Count];
+      for (int i = 0; i < handles.Count; i++) { float c = 0; try { handles[i].Vol.GetMasterVolume(out c); } catch { } a[i] = c; }
+      return a;
+    }
+    public static float[] Originals(List<Handle> handles) {
+      var a = new float[handles.Count];
+      for (int i = 0; i < handles.Count; i++) a[i] = handles[i].Original;
+      return a;
+    }
+    public static uint[] Pids(List<Handle> handles) {
+      var a = new uint[handles.Count];
+      for (int i = 0; i < handles.Count; i++) a[i] = handles[i].Pid;
+      return a;
+    }
+  }
+}
+'@
+}
+
+function Invoke-Duck([double]$factor, [int[]]$exclude) {
+  if ($factor -ge 1.0 -or $factor -lt 0) { return $null }
+  try { return [CU.AudioDuck]::Duck($factor, $exclude) } catch { return $null }
+}
+function Restore-Duck($handles) {
+  if ($null -eq $handles) { return }
+  try { [CU.AudioDuck]::Restore($handles) } catch { }
+}
+
+# -WavPath mode: duck others (excluding THIS process, which owns the playback), play, restore in finally.
+if ($WavPath) {
+  $h = Invoke-Duck $Factor @($PID)
+  try {
+    $p = New-Object System.Media.SoundPlayer $WavPath
+    $p.PlaySync()
+  } finally {
+    Restore-Duck $h
+  }
+}

package/scripts/classify.ps1

@@ -0,0 +1,8 @@
+# computer-use — classify_activity raw-signal adapter over lib.ps1::Get-AxClassifyActivity.
+# Output = a single JSON blob of raw signals (foreground process/title, notification state, UIA count, fullscreen).
+# The JS side (activity.mjs) derives the activity class from these. Read-only, no images.
+param([int]$UiaCap = 60)
+$ErrorActionPreference = 'Stop'
+. (Join-Path $PSScriptRoot 'lib.ps1')
+Initialize-AxEnv
+Get-AxClassifyActivity -UiaCap $UiaCap | ConvertTo-Json -Depth 4

package/scripts/fetch-supertonic.mjs

@@ -0,0 +1,65 @@
+#!/usr/bin/env node
+// computer-use — Supertonic 3 model downloader (one-time, ~380 MB) for the optional neural TTS engine.
+//
+// Downloads Supertone/supertonic-3 (OpenRAIL-M weights — commercial use permitted) into the model cache the
+// speak path expects. Idempotent: existing non-empty files are skipped, so re-running resumes a partial fetch.
+// Usage: node fetch-supertonic.mjs [targetDir]   (default: VORTEX_CU_TTS_MODEL_DIR or ~/.vortex/computer-use/supertonic-3)
+//
+// The engine code (speak-supertonic.mjs) is adapted from Supertone's MIT Node example; only the weights are
+// downloaded here, never bundled, keeping the npm package small and the license boundary clean.
+
+import { createWriteStream, existsSync, statSync, mkdirSync, renameSync, unlinkSync } from 'node:fs';
+import { Readable } from 'node:stream';
+import { pipeline } from 'node:stream/promises';
+import { join, dirname } from 'node:path';
+import { homedir } from 'node:os';
+
+const HF = 'https://huggingface.co/Supertone/supertonic-3/resolve/main';
+const FILES = [
+  'onnx/duration_predictor.onnx',
+  'onnx/text_encoder.onnx',
+  'onnx/vector_estimator.onnx',
+  'onnx/vocoder.onnx',
+  'onnx/tts.json',
+  'onnx/unicode_indexer.json',
+  'config.json',
+  ...['F1', 'F2', 'F3', 'F4', 'F5', 'M1', 'M2', 'M3', 'M4', 'M5'].map((v) => `voice_styles/${v}.json`),
+];
+
+const targetDir = process.argv[2] || process.env.VORTEX_CU_TTS_MODEL_DIR || join(homedir(), '.vortex', 'computer-use', 'supertonic-3');
+
+async function fetchToFile(url, dest) {
+  const res = await fetch(url, { redirect: 'follow' });
+  if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);
+  const tmp = dest + '.part';
+  mkdirSync(dirname(dest), { recursive: true });
+  await pipeline(Readable.fromWeb(res.body), createWriteStream(tmp));
+  renameSync(tmp, dest);
+  return statSync(dest).size;
+}
+
+async function main() {
+  console.log(`Supertonic 3 model cache: ${targetDir}`);
+  let downloaded = 0, skipped = 0, bytes = 0;
+  for (const rel of FILES) {
+    const dest = join(targetDir, rel);
+    if (existsSync(dest) && statSync(dest).size > 0) { skipped++; continue; }
+    process.stdout.write(`  ↓ ${rel} ... `);
+    try {
+      const t = Date.now();
+      const sz = await fetchToFile(`${HF}/${rel}`, dest);
+      bytes += sz;
+      downloaded++;
+      console.log(`${(sz / 1048576).toFixed(1)} MB (${((Date.now() - t) / 1000).toFixed(1)}s)`);
+    } catch (e) {
+      console.log(`FAILED: ${e.message}`);
+      try { unlinkSync(dest + '.part'); } catch {}
+      console.error(`\nDownload failed for ${rel}. Re-run to resume.`);
+      process.exit(1);
+    }
+  }
+  console.log(`\nDone — ${downloaded} downloaded (${(bytes / 1048576).toFixed(0)} MB), ${skipped} already present.`);
+  console.log('Neural TTS is ready. Set VORTEX_CU_TTS_ENGINE=auto (default) to use it.');
+}
+
+main().catch((e) => { console.error(e); process.exit(1); });

package/scripts/lib.ps1

@@ -26,6 +26,9 @@ public static class AxNative {
   [DllImport("user32.dll")] public static extern int GetWindowThreadProcessId(IntPtr h, out int pid);
   [DllImport("user32.dll")] public static extern int GetWindowTextLength(IntPtr h);
   [DllImport("user32.dll", CharSet=CharSet.Unicode)] public static extern int GetWindowTextW(IntPtr h, StringBuilder s, int max);
+  // "Is it OK to interrupt the user right now?" — the global interruptibility gate (1=NOT_PRESENT, 2=BUSY,
+  //   3=RUNNING_D3D_FULL_SCREEN, 4=PRESENTATION_MODE, 5=ACCEPTS_NOTIFICATIONS, 6=QUIET_TIME, 7=APP). Returns HRESULT.
+  [DllImport("shell32.dll")] public static extern int SHQueryUserNotificationState(out int state);
   private delegate bool AxEnumProc(IntPtr h, IntPtr l);
   [DllImport("user32.dll", SetLastError=true)] private static extern bool EnumWindows(AxEnumProc cb, IntPtr l);
   // Enumerate every visible (not minimized, area>0) top-level window as (pid, rect, title) — so the denylist checks not just the
@@ -164,6 +167,65 @@ function Get-AxProbe {
 }
 
 # ---------------- capture ----------------
+# Raw signals for activity classification (the JS side in activity.mjs derives the class). Read-only, fast:
+# foreground process/title, the interruptibility notification-state, a capped UIA control-view descendant count
+# (near-empty = GPU canvas i.e. game/video; rich = normal app), and whether the window fills its monitor.
+function Get-AxClassifyActivity([int]$UiaCap = 60) {
+  $hwnd = [AxNative]::GetForegroundWindow()
+  $procId = 0; [void][AxNative]::GetWindowThreadProcessId($hwnd, [ref]$procId)
+  $title = ''
+  try {
+    $len = [AxNative]::GetWindowTextLength($hwnd)
+    if ($len -gt 0) { $sb = New-Object System.Text.StringBuilder ($len + 2); [void][AxNative]::GetWindowTextW($hwnd, $sb, $sb.Capacity); $title = $sb.ToString() }
+  } catch {}
+  $ns = 0; try { [void][AxNative]::SHQueryUserNotificationState([ref]$ns) } catch {}
+  $fs = $false
+  try {
+    $r = New-Object AxRECT
+    if ([AxNative]::GetWindowRect($hwnd, [ref]$r)) {
+      $scr = [System.Windows.Forms.Screen]::FromHandle($hwnd).Bounds
+      if (($r.Right - $r.Left) -ge $scr.Width -and ($r.Bottom - $r.Top) -ge $scr.Height) { $fs = $true }
+    }
+  } catch {}
+  # Denylist (same control as captures): never leak — or classify — a sensitive foreground window. A non-null
+  #   result means denied (incl. fail-closed when a rule is configured but title/pid can't be resolved).
+  $deny = $null
+  try { $deny = Test-AxDenylistElement $title $procId } catch { $deny = @{ reason = 'denylist check failed — fail-closed'; match = '' } }
+  if ($null -ne $deny) {
+    return [ordered]@{
+      redacted = $true; reason = [string]$deny.reason; process = ''; procId = $procId; title = ''
+      hwnd = [int64]$hwnd; notificationState = $ns; uiaCount = $null; uiaOk = $false; uiaCapped = $false; fullscreen = $fs
+    }
+  }
+  $proc = ''
+  try { $proc = (Get-Process -Id $procId -ErrorAction Stop).ProcessName } catch {}
+  # UIA control-view descendant count (capped) — near-empty on a GPU canvas (game/video), rich on normal apps.
+  # $uiaOk distinguishes "walked, found N" from "couldn't walk" (so a UIA failure isn't read as an empty canvas).
+  # A single hard $budget bounds BOTH pops and sibling enumeration so a huge/odd tree can't blow up the walk;
+  # the per-call spawnSync timeout (caller) is the backstop against a single hung COM call.
+  $uia = 0; $capped = $false; $uiaOk = $false; $budget = [Math]::Max(16, $UiaCap * 4); $iter = 0
+  try {
+    $root = [System.Windows.Automation.AutomationElement]::FromHandle($hwnd)
+    if ($null -ne $root) {
+      $walker = [System.Windows.Automation.TreeWalker]::ControlViewWalker
+      $stack = New-Object System.Collections.Stack
+      $c = $walker.GetFirstChild($root)
+      while ($null -ne $c -and $iter -lt $budget) { $stack.Push($c); $iter++; $c = $walker.GetNextSibling($c) }
+      while ($stack.Count -gt 0 -and $uia -lt $UiaCap -and $iter -lt $budget) {
+        $el = $stack.Pop(); $uia++
+        $cc = $walker.GetFirstChild($el)
+        while ($null -ne $cc -and $iter -lt $budget) { $stack.Push($cc); $iter++; $cc = $walker.GetNextSibling($cc) }
+      }
+      if ($uia -ge $UiaCap -or $iter -ge $budget) { $capped = $true }
+      $uiaOk = $true
+    }
+  } catch { $uiaOk = $false }
+  return [ordered]@{
+    redacted = $false; process = $proc; procId = $procId; title = $title; hwnd = [int64]$hwnd
+    notificationState = $ns; uiaCount = $(if ($uiaOk) { $uia } else { $null }); uiaOk = $uiaOk; uiaCapped = $capped; fullscreen = $fs
+  }
+}
+
 function New-AxOutPath([string]$OutDir, $Frame = $null) {
   # Avoid multi-instance / concurrent-capture collisions — guarantee uniqueness with PID + milliseconds + random number.
   $stamp = (Get-Date).ToString('HHmmssfff')