twokey 1.0.11 → 1.0.13

package/README.md

@@ -12,9 +12,15 @@ The core idea matches the video workflow:
 ## What Works Now
 
 - Global hold hotkey on X11 for voice capture.
+- Wayland compositor detection (GNOME/KDE/Sway/Hyprland) with compositor-specific runtime capability messaging.
+- Wayland automation integration paths:
+  - text insertion via `wtype` or `ydotool`
+  - selected-text read via `wtype` + `wl-paste` when available
 - Double-tap hotkey to cycle modes.
 - Single-tap hotkey to open file-context picker.
 - Voice-triggered toolchains for multi-step desktop actions.
+- Visual toolchain editor in settings with dry-run and manual execution controls.
+- Toolchain safety guardrails (dangerous shell pattern blocking).
 - Four modes:
   - Conversation
   - Edit Text
@@ -43,6 +49,14 @@ The core idea matches the video workflow:
 - Tray icon and settings window.
 - GitHub release check from settings.
 - In-app AppImage update download and launch.
+- Optional in-app background update download to staged AppImage file.
+- Local Whisper auto-setup on first use:
+  - managed `ffmpeg` in `~/.local/share/twokey/bin/`
+  - managed Whisper venv in `~/.local/share/twokey/whisper-venv/`
+- Local Whisper tuning in settings:
+  - model selection (`tiny` .. `large-v3`)
+  - beam size tuning (speed vs quality)
+- Visible install/save status in settings (including install spinner).
 
 ## Current Video Parity
 
@@ -58,14 +72,22 @@ Implemented from video behavior:
 
 Still not fully equivalent to the video vision:
 
-- Toolchains are implemented, but no visual workflow builder exists yet.
+- Toolchains have a visual settings editor; richer action templates and safety controls are still pending.
 - Wayland still has compositor-specific limits for global hold hotkeys and full automation.
-- Update install is available for AppImage, but no signed rollback-capable updater pipeline yet.
+- Update install is AppImage-first with channel-aware checks (`stable`/`beta`/`dev`), checksum validation and local rollback fallback.
+
+Current workflow action catalog:
+
+- `open_url`
+- `open_app`
+- `shell` (with safety blocking for known dangerous patterns)
+- `wait_ms`
+- `check_command`
 
 ## Install
 
 ```bash
-npm install twokey
+npm install -g twokey
 ```
 
 Run:
@@ -78,6 +100,7 @@ Default behavior:
 
 - starts native desktop app in background
 - if no native binary is installed, tries to download latest AppImage release
+- prepares runtime dependencies in user space (best effort, no system package mutation)
 
 Useful options:
 
@@ -86,8 +109,16 @@ twokey --help
 twokey --cli
 twokey --once "Erklaere X11 vs Wayland kurz"
 twokey --desktop
+twokey --prepare-runtime
+twokey --prepare-runtime-only
 ```
 
+Updater behavior:
+
+- Uses update channel from settings (`stable`, `beta`, `dev`).
+- Verifies AppImage checksum when a `.sha256` asset is available.
+- Restores previous local AppImage if updated binary cannot be started.
+
 ## Development
 
 Install dependencies:
@@ -114,6 +145,14 @@ npm run build
 cd src-tauri && cargo check
 ```
 
+Runtime smoke tests:
+
+```bash
+cargo test --manifest-path src-tauri/Cargo.toml runtime_e2e -- --nocapture
+TWOKEY_E2E_SESSION=x11 cargo test --manifest-path src-tauri/Cargo.toml runtime_e2e -- --nocapture
+TWOKEY_E2E_SESSION=wayland cargo test --manifest-path src-tauri/Cargo.toml runtime_e2e -- --nocapture
+```
+
 ## Hotkey Behavior
 
 Default hotkey can be changed in settings. Examples:
@@ -146,6 +185,8 @@ Configured in settings (`sttProvider`):
 
 `external-command` needs `TWOKEY_STT_COMMAND` with `{audio}` placeholder.
 
+For `local-whisper`, TwoKey attempts runtime setup in user space and checks required binaries before transcription.
+
 Example:
 
 ```bash
@@ -171,6 +212,11 @@ Optional runtime tools:
 - PDF extraction: `poppler-utils` (`pdftotext`)
 - TTS backends: `spd-say` or `espeak-ng`/`espeak`
 
+Notes:
+
+- `ffmpeg` is auto-provisioned to `~/.local/share/twokey/bin/ffmpeg` if missing.
+- Local Whisper CLI is auto-provisioned to `~/.local/share/twokey/whisper-venv/bin/whisper` if missing.
+
 ## Data Paths
 
 - Config: `~/.config/twokey-ai/`
@@ -178,11 +224,17 @@ Optional runtime tools:
 - Cache: `~/.cache/twokey-ai/`
 - History DB: `~/.local/share/twokey-ai/history.db`
 - Toolchains: `~/.config/twokey-ai/toolchains.json`
+- Managed runtime bin: `~/.local/share/twokey/bin/`
+- Managed Whisper venv: `~/.local/share/twokey/whisper-venv/`
 
 ## Repo
 
 https://github.com/meinzeug/twokey
 
+## Additional Docs
+
+- Wayland matrix: `docs/WAYLAND_COMPATIBILITY_MATRIX.md`
+
 ## License
 
 MIT

package/bin/postinstall.js

@@ -16,10 +16,6 @@ const isRoot = typeof process.getuid === "function" && process.getuid() === 0;
 const sudoUser = process.env.SUDO_USER;
 
 if (isRoot && sudoUser && sudoUser !== "root") {
-  if (isDesktopRunningForUser(sudoUser)) {
-    process.exit(0);
-  }
-
   const uid = resolveUid(sudoUser);
   const homeDir = resolveHomeDir(sudoUser);
   const env = { ...process.env };
@@ -32,9 +28,24 @@ if (isRoot && sudoUser && sudoUser !== "root") {
     env.DBUS_SESSION_BUS_ADDRESS = `unix:path=${env.XDG_RUNTIME_DIR}/bus`;
   }
 
+  const runtimePrep = spawn(
+    "sudo",
+    ["-u", sudoUser, "-H", process.execPath, cliPath, "--prepare-runtime-only", "--quiet"],
+    {
+      stdio: "ignore",
+      shell: false,
+      env,
+    },
+  );
+  runtimePrep.on("error", () => undefined);
+
+  if (isDesktopRunningForUser(sudoUser)) {
+    process.exit(0);
+  }
+
   const delegated = spawn(
     "sudo",
-    ["-u", sudoUser, "-H", process.execPath, cliPath, "--desktop", "--enable-autostart"],
+    ["-u", sudoUser, "-H", process.execPath, cliPath, "--prepare-runtime", "--desktop", "--enable-autostart"],
     {
       stdio: "ignore",
       shell: false,
@@ -45,11 +56,22 @@ if (isRoot && sudoUser && sudoUser !== "root") {
   delegated.on("error", () => process.exit(0));
   delegated.on("close", () => process.exit(0));
 } else {
+  if (isRoot) {
+    ensureSystemTtsBackend();
+  }
+
+  const runtimePrep = spawn(process.execPath, [cliPath, "--prepare-runtime-only", "--quiet"], {
+    stdio: "ignore",
+    shell: false,
+  });
+
+  runtimePrep.on("error", () => undefined);
+
   if (isDesktopRunningForUser(process.env.USER || "")) {
     process.exit(0);
   }
 
-  const child = spawn(process.execPath, [cliPath, "--desktop", "--enable-autostart"], {
+  const child = spawn(process.execPath, [cliPath, "--prepare-runtime", "--desktop", "--enable-autostart"], {
     stdio: "ignore",
     shell: false,
   });
@@ -85,3 +107,34 @@ function isDesktopRunningForUser(username) {
   const out = spawnSync("pgrep", ["-u", username, "-f", "twokey-ai(\\.AppImage)?"], { encoding: "utf8" });
   return out.status === 0;
 }
+
+function ensureSystemTtsBackend() {
+  if (hasAnyTtsBackend()) {
+    return;
+  }
+
+  if (!commandExists("apt-get")) {
+    return;
+  }
+
+  try {
+    const result = spawnSync("apt-get", ["install", "-y", "espeak-ng"], {
+      stdio: "ignore",
+      shell: false,
+    });
+    if (result.status !== 0) {
+      return;
+    }
+  } catch {
+    // best effort
+  }
+}
+
+function hasAnyTtsBackend() {
+  return commandExists("spd-say") || commandExists("espeak-ng") || commandExists("espeak");
+}
+
+function commandExists(name) {
+  const out = spawnSync("sh", ["-c", `command -v ${name} >/dev/null 2>&1`], { encoding: "utf8" });
+  return out.status === 0;
+}

package/bin/twokey.js

@@ -6,16 +6,21 @@ import os from "node:os";
 import path from "node:path";
 import readline from "node:readline";
 
-const VERSION = process.env.npm_package_version || "1.0.11";
+const VERSION = process.env.npm_package_version || "1.0.12";
 const DEFAULT_MODEL = process.env.TWOKEY_OLLAMA_MODEL || "qwen2.5:3b";
 const DEFAULT_OLLAMA_URL = process.env.TWOKEY_OLLAMA_URL || "http://127.0.0.1:11434";
 const LATEST_RELEASE_API = "https://api.github.com/repos/meinzeug/twokey/releases/latest";
 const APPIMAGE_DIR = path.join(os.homedir(), ".local", "share", "twokey", "bin");
 const APPIMAGE_PATH = path.join(APPIMAGE_DIR, "twokey-ai.AppImage");
 const APPIMAGE_META_PATH = path.join(APPIMAGE_DIR, "twokey-ai.meta.json");
+const MANAGED_FFMPEG_PATH = path.join(APPIMAGE_DIR, "ffmpeg");
+const WHISPER_VENV_DIR = path.join(os.homedir(), ".local", "share", "twokey", "whisper-venv");
+const MANAGED_WHISPER_PATH = path.join(WHISPER_VENV_DIR, "bin", "whisper");
 
 const args = process.argv.slice(2);
 const QUIET = args.includes("--quiet");
+const PREPARE_RUNTIME = args.includes("--prepare-runtime");
+const PREPARE_RUNTIME_ONLY = args.includes("--prepare-runtime-only");
 
 if (args.includes("--help") || args.includes("-h")) {
   printHelp();
@@ -27,50 +32,16 @@ if (args.includes("--version") || args.includes("-v")) {
   process.exit(0);
 }
 
-if (args.includes("--desktop")) {
-  launchDesktopApp().then(async (startedCommand) => {
-    if (startedCommand) {
-      if (args.includes("--enable-autostart")) {
-        try {
-          await ensureUserService(startedCommand);
-        } catch (error) {
-          if (!QUIET) {
-            const message = error instanceof Error ? error.message : String(error);
-            console.warn(`Autostart setup skipped: ${message}`);
-          }
-        }
-      }
+if (PREPARE_RUNTIME_ONLY) {
+  ensureRuntimeDependencies()
+    .then(() => process.exit(0))
+    .catch((error) => {
       if (!QUIET) {
-        console.log("TwoKey desktop app started in background.");
+        console.warn(`Runtime setup skipped: ${error.message || String(error)}`);
       }
       process.exit(0);
-    }
-    if (!QUIET) {
-      console.error("No native desktop binary found in PATH.");
-      console.error("Install the .deb/.AppImage release and ensure 'twokey-ai' is available in PATH.");
-    }
-    process.exit(1);
-  });
-}
-
-const onceIndex = args.findIndex((value) => value === "--once");
-if (onceIndex >= 0) {
-  const prompt = args.slice(onceIndex + 1).join(" ").trim();
-  if (!prompt) {
-    console.error("Missing prompt after --once");
-    process.exit(1);
-  }
-
-  runSinglePrompt(prompt).catch((error) => {
-    console.error(error.message || String(error));
-    process.exit(1);
-  });
-} else if (args.includes("--cli")) {
-  startRepl().catch((error) => {
-    console.error(error.message || String(error));
-    process.exit(1);
-  });
-} else {
+    });
+} else if (args.includes("--desktop")) {
   launchDesktopApp().then(async (startedCommand) => {
     if (startedCommand) {
       if (args.includes("--enable-autostart")) {
@@ -96,6 +67,51 @@ if (onceIndex >= 0) {
     }
     process.exit(1);
   });
+} else {
+  const onceIndex = args.findIndex((value) => value === "--once");
+  if (onceIndex >= 0) {
+    const prompt = args.slice(onceIndex + 1).join(" ").trim();
+    if (!prompt) {
+      console.error("Missing prompt after --once");
+      process.exit(1);
+    }
+
+    runSinglePrompt(prompt).catch((error) => {
+      console.error(error.message || String(error));
+      process.exit(1);
+    });
+  } else if (args.includes("--cli")) {
+    startRepl().catch((error) => {
+      console.error(error.message || String(error));
+      process.exit(1);
+    });
+  } else {
+    launchDesktopApp().then(async (startedCommand) => {
+      if (startedCommand) {
+        if (args.includes("--enable-autostart")) {
+          try {
+            await ensureUserService(startedCommand);
+          } catch (error) {
+            if (!QUIET) {
+              const message = error instanceof Error ? error.message : String(error);
+              console.warn(`Autostart setup skipped: ${message}`);
+            }
+          }
+        }
+        if (!QUIET) {
+          console.log("TwoKey desktop app started in background.");
+        }
+        process.exit(0);
+      }
+
+      if (!QUIET) {
+        console.error("Could not start desktop app.");
+        console.error("Tried system binaries and auto-download from GitHub Releases.");
+        console.error("Use 'twokey --cli' to run terminal mode.");
+      }
+      process.exit(1);
+    });
+  }
 }
 
 async function runSinglePrompt(prompt) {
@@ -191,6 +207,17 @@ async function askOllama(prompt) {
 }
 
 async function launchDesktopApp() {
+  if (PREPARE_RUNTIME) {
+    try {
+      await ensureRuntimeDependencies();
+    } catch (error) {
+      if (!QUIET) {
+        const message = error instanceof Error ? error.message : String(error);
+        console.warn(`Runtime dependencies could not be fully prepared: ${message}`);
+      }
+    }
+  }
+
   let appImageReady = false;
   try {
     appImageReady = await ensureLocalAppImage();
@@ -330,6 +357,7 @@ function spawnDetached(command) {
       detached: true,
       stdio: "ignore",
       shell: false,
+      env: withManagedBinPath(process.env),
     });
 
     child.once("spawn", () => {
@@ -343,6 +371,132 @@ function spawnDetached(command) {
   });
 }
 
+async function ensureRuntimeDependencies() {
+  await fs.promises.mkdir(APPIMAGE_DIR, { recursive: true });
+  await ensureManagedFfmpeg();
+  await ensureManagedWhisperCli();
+  await ensureManagedTtsBackend();
+}
+
+async function ensureManagedFfmpeg() {
+  if (await hasExecutable(MANAGED_FFMPEG_PATH)) {
+    return;
+  }
+
+  let sourcePath = "";
+  try {
+    const module = await import("@ffmpeg-installer/ffmpeg");
+    sourcePath =
+      module?.default?.path
+      || module?.path
+      || module?.default?.default?.path
+      || "";
+  } catch {
+    sourcePath = "";
+  }
+
+  if (!sourcePath) {
+    throw new Error("@ffmpeg-installer/ffmpeg konnte nicht geladen werden");
+  }
+
+  await fs.promises.copyFile(sourcePath, MANAGED_FFMPEG_PATH);
+  await fs.promises.chmod(MANAGED_FFMPEG_PATH, 0o755);
+}
+
+async function ensureManagedWhisperCli() {
+  if (await hasExecutable(MANAGED_WHISPER_PATH)) {
+    return;
+  }
+
+  if (!(await commandExists("python3"))) {
+    throw new Error("python3 fehlt, Whisper-CLI konnte nicht automatisch installiert werden");
+  }
+
+  await fs.promises.mkdir(path.dirname(WHISPER_VENV_DIR), { recursive: true });
+
+  const venvPython = path.join(WHISPER_VENV_DIR, "bin", "python");
+  if (!(await hasExecutable(venvPython))) {
+    await runCommand("python3", ["-m", "venv", WHISPER_VENV_DIR]);
+  }
+
+  await runCommand(venvPython, ["-m", "pip", "install", "--upgrade", "pip", "setuptools", "wheel"]);
+  await runCommand(venvPython, ["-m", "pip", "install", "--upgrade", "openai-whisper"]);
+
+  if (!(await hasExecutable(MANAGED_WHISPER_PATH))) {
+    throw new Error("Whisper wurde installiert, aber das CLI-Binary fehlt");
+  }
+}
+
+async function ensureManagedTtsBackend() {
+  if (await commandExists("spd-say") || await commandExists("espeak-ng") || await commandExists("espeak")) {
+    return;
+  }
+
+  if (!isRoot()) {
+    return;
+  }
+
+  if (await commandExists("apt-get")) {
+    try {
+      await runCommand("apt-get", ["install", "-y", "espeak-ng"]);
+    } catch {
+      // best effort only
+    }
+  }
+}
+
+async function commandExists(name) {
+  return new Promise((resolve) => {
+    const child = spawn("sh", ["-c", `command -v ${name} >/dev/null 2>&1`], {
+      stdio: "ignore",
+      shell: false,
+    });
+    child.on("error", () => resolve(false));
+    child.on("close", (code) => resolve(code === 0));
+  });
+}
+
+function isRoot() {
+  return typeof process.getuid === "function" && process.getuid() === 0;
+}
+
+async function runCommand(program, commandArgs) {
+  return new Promise((resolve, reject) => {
+    const child = spawn(program, commandArgs, {
+      shell: false,
+      stdio: QUIET ? "ignore" : "pipe",
+      env: withManagedBinPath(process.env),
+    });
+
+    let stderr = "";
+    if (child.stderr) {
+      child.stderr.on("data", (chunk) => {
+        stderr += String(chunk);
+      });
+    }
+
+    child.on("error", (error) => reject(error));
+    child.on("close", (code) => {
+      if (code === 0) {
+        resolve();
+        return;
+      }
+      reject(new Error(stderr.trim() || `${program} failed with code ${code}`));
+    });
+  });
+}
+
+function withManagedBinPath(baseEnv) {
+  const env = { ...baseEnv };
+  const currentPath = String(env.PATH || "");
+  const parts = currentPath.split(":").filter(Boolean);
+  if (!parts.includes(APPIMAGE_DIR)) {
+    parts.unshift(APPIMAGE_DIR);
+  }
+  env.PATH = parts.join(":");
+  return env;
+}
+
 function printHelp() {
   console.log("twokey <command/options>");
   console.log("");
@@ -352,6 +506,8 @@ function printHelp() {
   console.log("  --cli            Start interactive terminal mode");
   console.log("  --once <prompt>  Send one prompt to Ollama and print response");
   console.log("  --desktop        Start native desktop app in background");
+  console.log("  --prepare-runtime       Install/check local runtime dependencies");
+  console.log("  --prepare-runtime-only  Only install/check dependencies and exit");
   console.log("");
   console.log("Without options, twokey starts the native desktop app in background.");
   console.log("If no desktop binary is installed, twokey tries to download an AppImage from latest GitHub release.");
@@ -369,6 +525,7 @@ async function ensureUserService(command) {
     "After=graphical-session.target",
     "",
     "[Service]",
+    `Environment=PATH=${path.join(os.homedir(), ".local", "share", "twokey", "bin")}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`,
     `ExecStart=/bin/sh -lc ${shellEscape(command)}`,
     "Restart=on-failure",
     "RestartSec=3",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "twokey",
-  "version": "1.0.11",
+  "version": "1.0.13",
   "description": "Linux-first desktop AI assistant built with Tauri, React, and TypeScript.",
   "license": "MIT",
   "repository": {
@@ -48,6 +48,7 @@
     "tauri:build": "GTK_MODULES='' LIBGL_ALWAYS_SOFTWARE=1 WEBKIT_DISABLE_DMABUF_RENDERER=1 tauri build"
   },
   "dependencies": {
+    "@ffmpeg-installer/ffmpeg": "^1.1.0",
     "@tauri-apps/api": "^2.5.0",
     "lucide-react": "^0.468.0",
     "react": "^18.3.1",