chartremotely 0.2.12__tar.gz → 0.2.13__tar.gz

{chartremotely-0.2.12 → chartremotely-0.2.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: chartremotely
-Version: 0.2.12
+Version: 0.2.13
 Summary: Remote control of desktop charting applications on macOS
 Author: Lonnie VanZandt
 License: 
@@ -215,6 +215,7 @@ Requires-Dist: pyobjc-framework-ApplicationServices>=10; extra == "macos"
 Requires-Dist: pyobjc-framework-Cocoa>=10; extra == "macos"
 Requires-Dist: pyobjc-framework-Quartz>=10; extra == "macos"
 Requires-Dist: pyobjc-framework-Security>=10; extra == "macos"
+Requires-Dist: mlx-whisper>=0.4.3; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "macos"
 Provides-Extra: setup
 Requires-Dist: tollbooth-dpyc[nostr]==0.90.0; extra == "setup"
 Provides-Extra: dev
@@ -315,7 +316,34 @@ ignores everything sent to it.
 
 ## Voice
 
-"Hey Siri, ChartRemotely" asks three questions:
+"Hey Siri, ChartRemotely", then **one sentence**:
+
+> "Palantir, half, on mac mini." · "Shopify thirty minutes." · "Nvidia daily." · "GE hourly on the studio."
+
+Say the company, the scale, and - if it is not this Mac - "on" the display.
+The Shortcut records for about six seconds (tap to finish sooner where your
+device offers it) and sends the recording to the Mac that made it. That Mac
+hears it with **Whisper, running locally** on its own GPU (`mlx-whisper`,
+Apple silicon only) - nothing is sent to Apple or anyone else to transcribe -
+primed with the words it expects: the ten scale mnemonics, bar phrases like
+"thirty minutes", the companies you chart lately and the names of your
+displays. The recording is deleted as soon as it is heard; only the text is
+logged.
+
+If something is missing or unclear you are asked, and the Shortcut listens
+again (up to three times), keeping what it already understood: "I heard
+Nvidia but no scale. Say the scale, like half or daily. I'm listening." -
+"Daily." Or "Did you mean PS (Pershing Square) or MSGS (Madison Square
+Garden)? Say the one you meant. I'm listening."
+
+**First run.** `chartremotely setup` fetches the speech model,
+`whisper-large-v3-turbo` (about 1.6 GB, once, from Hugging Face). The
+listener loads it when it starts, so a sentence takes about a second to hear.
+Setup also learns your display names (the free `chart_agent_status`, asked at
+most every four hours) so "on the studio" is understood.
+
+**ChartRemotely Ask** is the fallback Shortcut, installed alongside: it asks
+three questions and takes typed or dictated answers -
 
 1. **Which company?** — a spoken name or a spelled ticker.
 2. **What scale?** — a time frame mnemonic (below), or "as is".
@@ -355,7 +383,8 @@ Palantir; "shop if y" is Shopify); when two companies are too close to call
 you are asked - "Did you mean PS (Pershing Square) or MSGS (Madison Square
 Garden)?" - rather than shown the wrong chart, and the symbols you chart
 lately settle a near-tie. `understand()` parses a whole sentence the same way
-("Palantir half on the mini").
+("Palantir half on the mini") - that is what the ChartRemotely Shortcut
+uses on the sentence Whisper heard.
 
 Each command handled is logged as one line — time, the command's first word,
 the "Where?" as said, what was heard for a company or scale lookup, and the
@@ -397,7 +426,9 @@ and runs `chartremotely setup`, which:
 3. gives the Mac its tailnet address with `tailscale serve`;
 4. installs the listener and the relay as launchd agents;
 5. asks macOS for Accessibility and Screen Recording for the Python that runs them;
-6. makes this Mac's ChartRemotely voice Shortcut and opens it for import.
+6. fetches the Whisper speech model (about 1.6 GB, once) and your display names;
+7. makes this Mac's two Shortcuts, "ChartRemotely" (one spoken sentence) and
+   "ChartRemotely Ask" (three questions), and opens them for import.
 
 Run it again at any time: finished steps are skipped. `chartremotely doctor`
 reports anything still missing.

{chartremotely-0.2.12 → chartremotely-0.2.13}/README.md

@@ -91,7 +91,34 @@ ignores everything sent to it.
 
 ## Voice
 
-"Hey Siri, ChartRemotely" asks three questions:
+"Hey Siri, ChartRemotely", then **one sentence**:
+
+> "Palantir, half, on mac mini." · "Shopify thirty minutes." · "Nvidia daily." · "GE hourly on the studio."
+
+Say the company, the scale, and - if it is not this Mac - "on" the display.
+The Shortcut records for about six seconds (tap to finish sooner where your
+device offers it) and sends the recording to the Mac that made it. That Mac
+hears it with **Whisper, running locally** on its own GPU (`mlx-whisper`,
+Apple silicon only) - nothing is sent to Apple or anyone else to transcribe -
+primed with the words it expects: the ten scale mnemonics, bar phrases like
+"thirty minutes", the companies you chart lately and the names of your
+displays. The recording is deleted as soon as it is heard; only the text is
+logged.
+
+If something is missing or unclear you are asked, and the Shortcut listens
+again (up to three times), keeping what it already understood: "I heard
+Nvidia but no scale. Say the scale, like half or daily. I'm listening." -
+"Daily." Or "Did you mean PS (Pershing Square) or MSGS (Madison Square
+Garden)? Say the one you meant. I'm listening."
+
+**First run.** `chartremotely setup` fetches the speech model,
+`whisper-large-v3-turbo` (about 1.6 GB, once, from Hugging Face). The
+listener loads it when it starts, so a sentence takes about a second to hear.
+Setup also learns your display names (the free `chart_agent_status`, asked at
+most every four hours) so "on the studio" is understood.
+
+**ChartRemotely Ask** is the fallback Shortcut, installed alongside: it asks
+three questions and takes typed or dictated answers -
 
 1. **Which company?** — a spoken name or a spelled ticker.
 2. **What scale?** — a time frame mnemonic (below), or "as is".
@@ -131,7 +158,8 @@ Palantir; "shop if y" is Shopify); when two companies are too close to call
 you are asked - "Did you mean PS (Pershing Square) or MSGS (Madison Square
 Garden)?" - rather than shown the wrong chart, and the symbols you chart
 lately settle a near-tie. `understand()` parses a whole sentence the same way
-("Palantir half on the mini").
+("Palantir half on the mini") - that is what the ChartRemotely Shortcut
+uses on the sentence Whisper heard.
 
 Each command handled is logged as one line — time, the command's first word,
 the "Where?" as said, what was heard for a company or scale lookup, and the
@@ -173,7 +201,9 @@ and runs `chartremotely setup`, which:
 3. gives the Mac its tailnet address with `tailscale serve`;
 4. installs the listener and the relay as launchd agents;
 5. asks macOS for Accessibility and Screen Recording for the Python that runs them;
-6. makes this Mac's ChartRemotely voice Shortcut and opens it for import.
+6. fetches the Whisper speech model (about 1.6 GB, once) and your display names;
+7. makes this Mac's two Shortcuts, "ChartRemotely" (one spoken sentence) and
+   "ChartRemotely Ask" (three questions), and opens them for import.
 
 Run it again at any time: finished steps are skipped. `chartremotely doctor`
 reports anything still missing.

chartremotely-0.2.13/chartremotely/assets/shortcut-voice.plist

@@ -0,0 +1,259 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>WFWorkflowActions</key>
+	<array>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.repeat.count</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>GroupingIdentifier</key>
+				<string>2A5B6C7D-8E9F-4A0B-8C1D-2E3F4A5B6C7D</string>
+				<key>WFControlFlowMode</key>
+				<integer>0</integer>
+				<key>WFRepeatCount</key>
+				<integer>3</integer>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.recordaudio</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>CustomOutputName</key>
+				<string>Sentence</string>
+				<key>UUID</key>
+				<string>7B3E2C1A-4D5F-4A6B-8C9D-0E1F2A3B4C5D</string>
+				<key>WFRecordingCompression</key>
+				<string>Normal</string>
+				<key>WFRecordingEnd</key>
+				<string>After Time</string>
+				<key>WFRecordingStart</key>
+				<string>Immediately</string>
+				<key>WFRecordingTimeInterval</key>
+				<dict>
+					<key>Value</key>
+					<dict>
+						<key>Magnitude</key>
+						<string>6</string>
+						<key>Unit</key>
+						<string>sec</string>
+					</dict>
+					<key>WFSerializationType</key>
+					<string>WFQuantityFieldValue</string>
+				</dict>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.downloadurl</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>Advanced</key>
+				<true/>
+				<key>CustomOutputName</key>
+				<string>Reply</string>
+				<key>ShowHeaders</key>
+				<true/>
+				<key>UUID</key>
+				<string>9C4F3D2B-5E6A-4B7C-9D0E-1F2A3B4C5D6E</string>
+				<key>WFHTTPBodyType</key>
+				<string>File</string>
+				<key>WFHTTPHeaders</key>
+				<dict>
+					<key>Value</key>
+					<dict>
+						<key>WFDictionaryFieldValueItems</key>
+						<array>
+							<dict>
+								<key>WFItemType</key>
+								<integer>0</integer>
+								<key>WFKey</key>
+								<dict>
+									<key>Value</key>
+									<dict>
+										<key>string</key>
+										<string>X-Token</string>
+									</dict>
+									<key>WFSerializationType</key>
+									<string>WFTextTokenString</string>
+								</dict>
+								<key>WFValue</key>
+								<dict>
+									<key>Value</key>
+									<dict>
+										<key>string</key>
+										<string>__CHARTREMOTELY_TOKEN__</string>
+									</dict>
+									<key>WFSerializationType</key>
+									<string>WFTextTokenString</string>
+								</dict>
+							</dict>
+						</array>
+					</dict>
+					<key>WFSerializationType</key>
+					<string>WFDictionaryFieldValue</string>
+				</dict>
+				<key>WFHTTPMethod</key>
+				<string>POST</string>
+				<key>WFRequestVariable</key>
+				<dict>
+					<key>Value</key>
+					<dict>
+						<key>OutputName</key>
+						<string>Sentence</string>
+						<key>OutputUUID</key>
+						<string>7B3E2C1A-4D5F-4A6B-8C9D-0E1F2A3B4C5D</string>
+						<key>Type</key>
+						<string>ActionOutput</string>
+					</dict>
+					<key>WFSerializationType</key>
+					<string>WFTextTokenAttachment</string>
+				</dict>
+				<key>WFURL</key>
+				<string>__CHARTREMOTELY_URL__?hear=1</string>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.speaktext</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>WFSpeakTextWait</key>
+				<true/>
+				<key>WFText</key>
+				<dict>
+					<key>Value</key>
+					<dict>
+						<key>attachmentsByRange</key>
+						<dict>
+							<key>{0, 1}</key>
+							<dict>
+								<key>OutputName</key>
+								<string>Reply</string>
+								<key>OutputUUID</key>
+								<string>9C4F3D2B-5E6A-4B7C-9D0E-1F2A3B4C5D6E</string>
+								<key>Type</key>
+								<string>ActionOutput</string>
+							</dict>
+						</dict>
+						<key>string</key>
+						<string></string>
+					</dict>
+					<key>WFSerializationType</key>
+					<string>WFTextTokenString</string>
+				</dict>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.conditional</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>GroupingIdentifier</key>
+				<string>4D6E7F80-9A1B-4C2D-8E3F-4A5B6C7D8E9F</string>
+				<key>WFCondition</key>
+				<integer>99</integer>
+				<key>WFConditionalActionString</key>
+				<string>listening</string>
+				<key>WFControlFlowMode</key>
+				<integer>0</integer>
+				<key>WFInput</key>
+				<dict>
+					<key>Type</key>
+					<string>Variable</string>
+					<key>Variable</key>
+					<dict>
+						<key>Value</key>
+						<dict>
+							<key>Aggrandizements</key>
+							<array>
+								<dict>
+									<key>CoercionItemClass</key>
+									<string>WFStringContentItem</string>
+									<key>Type</key>
+									<string>WFCoercionVariableAggrandizement</string>
+								</dict>
+							</array>
+							<key>OutputName</key>
+							<string>Reply</string>
+							<key>OutputUUID</key>
+							<string>9C4F3D2B-5E6A-4B7C-9D0E-1F2A3B4C5D6E</string>
+							<key>Type</key>
+							<string>ActionOutput</string>
+						</dict>
+						<key>WFSerializationType</key>
+						<string>WFTextTokenAttachment</string>
+					</dict>
+				</dict>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.conditional</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>GroupingIdentifier</key>
+				<string>4D6E7F80-9A1B-4C2D-8E3F-4A5B6C7D8E9F</string>
+				<key>WFControlFlowMode</key>
+				<integer>1</integer>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.exit</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict/>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.conditional</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>GroupingIdentifier</key>
+				<string>4D6E7F80-9A1B-4C2D-8E3F-4A5B6C7D8E9F</string>
+				<key>UUID</key>
+				<string>6F8091A2-BC3D-4E4F-A051-6C7D8E9FA0B1</string>
+				<key>WFControlFlowMode</key>
+				<integer>2</integer>
+			</dict>
+		</dict>
+		<dict>
+			<key>WFWorkflowActionIdentifier</key>
+			<string>is.workflow.actions.repeat.count</string>
+			<key>WFWorkflowActionParameters</key>
+			<dict>
+				<key>GroupingIdentifier</key>
+				<string>2A5B6C7D-8E9F-4A0B-8C1D-2E3F4A5B6C7D</string>
+				<key>UUID</key>
+				<string>5E7F8091-AB2C-4D3E-9F40-5B6C7D8E9FA0</string>
+				<key>WFControlFlowMode</key>
+				<integer>2</integer>
+			</dict>
+		</dict>
+	</array>
+	<key>WFWorkflowClientVersion</key>
+	<string>2605.0.3</string>
+	<key>WFWorkflowIcon</key>
+	<dict>
+		<key>WFWorkflowIconGlyphNumber</key>
+		<integer>59511</integer>
+		<key>WFWorkflowIconStartColor</key>
+		<integer>463140863</integer>
+	</dict>
+	<key>WFWorkflowImportQuestions</key>
+	<array/>
+	<key>WFWorkflowInputContentItemClasses</key>
+	<array/>
+	<key>WFWorkflowMinimumClientVersion</key>
+	<integer>900</integer>
+	<key>WFWorkflowMinimumClientVersionString</key>
+	<string>900</string>
+	<key>WFWorkflowTypes</key>
+	<array>
+		<string>Watch</string>
+	</array>
+</dict>
+</plist>

chartremotely-0.2.13/chartremotely/displays.py

@@ -0,0 +1,129 @@
+"""The owner's display names, so a sentence can say where: "... on mac mini".
+
+:func:`understand.understand` splits "on <display>" off a sentence only when
+it knows the names, and :mod:`hearing` primes Whisper with them. They come
+from the operator's free ``chart_agent_status`` tool, asked as the owner
+(their npub and voice sign-in), and are cached in
+``~/.local/share/chartremotely/displays.json`` - labels only, never a token.
+
+The list changes only when a display is paired or forgotten, so it is asked
+for at most every :data:`MAX_AGE` seconds, after setup, and in the background:
+a voice request never waits for it, and uses what is cached (this Mac's own
+label at least) meanwhile.
+
+Stdlib only.
+"""
+
+from __future__ import annotations
+
+import json
+import threading
+import time
+from collections.abc import Callable
+
+from . import config, keystore, mcpclient, requestlog
+from .config import DATA_DIR
+
+PATH = DATA_DIR / "displays.json"
+MAX_AGE = 4 * 3600
+TOOL = "chart_agent_status"
+
+
+def _read(path=None) -> dict:
+    try:
+        data = json.loads((path or PATH).read_text())
+    except (OSError, ValueError):
+        return {}
+    return data if isinstance(data, dict) else {}
+
+
+def load(path=None) -> list[str]:
+    """The cached display labels, this Mac's first; [] when none are cached."""
+    data = _read(path)
+    labels = data.get("labels")
+    return [str(x) for x in labels if isinstance(x, str) and x.strip()] if isinstance(labels, list) else []
+
+
+def stale(path=None, now: Callable[[], float] = time.time) -> bool:
+    fetched = _read(path).get("fetched_at")
+    return not isinstance(fetched, (int, float)) or now() - fetched > MAX_AGE
+
+
+def labels_from(payload: dict, agent_id: str | None) -> list[str]:
+    """Display labels from a ``chart_agent_status`` answer, this Mac's first."""
+    rows = payload.get("displays") if isinstance(payload, dict) else None
+    if not isinstance(rows, list):
+        return []
+    mine, others = [], []
+    for row in rows:
+        if not isinstance(row, dict):
+            continue
+        label = " ".join(str(row.get("label") or "").split())[:64]
+        if not label:
+            continue
+        (mine if agent_id and row.get("agent_id") == agent_id else others).append(label)
+    out: list[str] = []
+    for label in mine + others:
+        if label not in out:
+            out.append(label)
+    return out
+
+
+def refresh(cfg: dict | None = None, client_for=None, path=None) -> list[str] | None:
+    """Ask the operator for the owner's displays and cache them.
+
+    Returns the labels, or None when they could not be had (not paired, not
+    signed in, operator unreachable) - the cache is then left as it was.
+    Never raises.
+    """
+    cfg = cfg or config.load()
+    base = (cfg.get("operator_url") or "").rstrip("/")
+    npub = cfg.get("owner_npub")
+    if not (base and npub and cfg.get("agent_id")):
+        return None
+    try:
+        token = keystore.load_token(npub)
+        if not token:
+            return None
+        payload = (client_for or mcpclient.Client)(base, timeout=30.0).call(TOOL, {"npub": npub, "dpop_token": token})
+    except Exception as exc:  # noqa: BLE001 - a missing list must never cost a reply
+        requestlog.outcome("serve", TOOL, "", f"display list failed: {type(exc).__name__}")
+        return None
+    if payload.get("error") or payload.get("success") is False:
+        requestlog.outcome("serve", TOOL, "", str(payload.get("error") or "refused"))
+        return None
+    labels = labels_from(payload, cfg.get("agent_id"))
+    target = path or PATH
+    try:
+        target.parent.mkdir(parents=True, exist_ok=True)
+        tmp = target.with_suffix(".tmp")
+        tmp.write_text(json.dumps({"fetched_at": time.time(), "labels": labels}))
+        tmp.replace(target)
+    except OSError:
+        pass
+    return labels
+
+
+_refreshing = threading.Lock()
+
+
+def refresh_in_background(**kwargs) -> bool:
+    """Start one :func:`refresh`; False when one is already running."""
+    if not _refreshing.acquire(blocking=False):
+        return False
+
+    def work() -> None:
+        try:
+            refresh(**kwargs)
+        finally:
+            _refreshing.release()
+
+    threading.Thread(target=work, name="refresh-displays", daemon=True).start()
+    return True
+
+
+def current() -> list[str]:
+    """The labels to use now; starts a background refresh when they are old."""
+    if stale():
+        refresh_in_background()
+    return load()