twokey 1.0.3 → 1.0.4

package/README.md

@@ -1,210 +1,188 @@
 # TwoKey Linux AI Assistant
 
-TwoKey is a Linux-first desktop assistant prototype. The goal is a small always-available overlay that will later let users hold two keys, speak, and have the result processed by local or online AI providers without switching to a browser or chat window.
-
-Phase 1 implements the foundation only:
-
-- Tauri + React project structure
-- Minimal dark overlay pill
-- Dummy modes: Conversation, Edit Text, Dictation, Feedback
-- Clickable mode menu
-- Placeholder settings window
-- Initial architecture and product documentation
-- X11 Phase 2 prototype: hold `Ctrl+Space` to record audio, double tap `Ctrl+Space` to cycle modes
-- Phase 3 prototype: mock STT and external STT command adapter
-- Phase 4 prototype: local Ollama conversation mode with `qwen2.5:3b`
-
-No TTS or text injection is implemented yet.
-
-## Installation
+TwoKey is a Linux-first desktop AI assistant with a small floating pill overlay.
+The core idea matches the video workflow:
+
+- hold a global hotkey, speak, release
+- audio is recorded and transcribed
+- transcript is sent to the selected AI provider
+- result is used directly in your current desktop workflow
+- optional TTS reads the answer out loud
+
+## What Works Now
+
+- Global hold hotkey on X11 for voice capture.
+- Double-tap hotkey to cycle modes.
+- Single-tap hotkey to open file-context picker.
+- Voice-triggered toolchains for multi-step desktop actions.
+- Four modes:
+  - Conversation
+  - Edit Text
+  - Dictation
+  - Feedback
+- Conversation mode:
+  - transcript -> provider answer
+  - optional TTS playback
+- Edit mode:
+  - read selected text
+  - apply spoken transform via AI
+  - replace selected text directly
+- Dictation mode:
+  - insert transcript at cursor
+- Feedback mode:
+  - local feedback persistence in history DB
+- File context:
+  - TXT/MD/JSON/CSV/PDF content context
+  - image context routing to vision-capable providers
+- Provider routing:
+  - local Ollama
+  - OpenAI-compatible
+  - OpenRouter-compatible
+  - secure API key storage via local keyring
+- Local audit/history persistence in SQLite.
+- Tray icon and settings window.
+- GitHub release check from settings.
+- In-app AppImage update download and launch.
+
+## Current Video Parity
+
+Implemented from video behavior:
+
+- Minimal floating pill UI.
+- Hold-to-talk workflow.
+- Mode switch via double tap.
+- Text workflow without browser/chat-tab context switch.
+- File attach + ask flow.
+- Hybrid local/online providers.
+- Optional TTS answer playback.
+
+Still not fully equivalent to the video vision:
+
+- Toolchains are implemented, but no visual workflow builder exists yet.
+- Wayland still has compositor-specific limits for global hold hotkeys and full automation.
+- Update install is available for AppImage, but no signed rollback-capable updater pipeline yet.
+
+## Install
 
 ```bash
 npm install twokey
 ```
 
-Start the tool directly:
+Run:
 
 ```bash
 twokey
 ```
 
-Default behavior: start the native desktop app in background.
-If no desktop binary is installed yet, `twokey` attempts to download an AppImage from the latest GitHub release into `~/.local/share/twokey/bin/` and starts it.
+Default behavior:
 
-Useful CLI options:
+- starts native desktop app in background
+- if no native binary is installed, tries to download latest AppImage release
+
+Useful options:
 
 ```bash
 twokey --help
 twokey --cli
-twokey --once "Erklaere kurz den Unterschied zwischen X11 und Wayland"
+twokey --once "Erklaere X11 vs Wayland kurz"
 twokey --desktop
 ```
 
-## Minimal Usage
-
-```ts
-import { getPackageInfo } from "twokey";
-
-const info = getPackageInfo();
-console.log(info.name);
-console.log(info.runtimeStatus.waylandGlobalHotkeys);
-```
-
-The package exports runtime status metadata. Current status includes planned/limited areas such as Wayland global hotkeys, TTS, tray menu, SQLite history/audit, and online provider execution until secure API-key storage is implemented.
+## Development
 
-## Requirements
-
-- Linux desktop
-- Node.js 20+
-- npm
-- Rust toolchain with `cargo` for running the Tauri app
-- Tauri Linux system dependencies, for example on Ubuntu/Debian:
-
-```bash
-sudo apt install libwebkit2gtk-4.1-dev build-essential curl wget file libxdo-dev libssl-dev libayatana-appindicator3-dev librsvg2-dev
-```
-
-This workspace currently has Node/npm available. `cargo` is required before the native app can be started.
-
-## Start Development
-
-Install JavaScript dependencies:
+Install dependencies:
 
 ```bash
 npm install
 ```
 
-Run only the web UI:
-
-```bash
-npm run dev
-```
-
-Run the Linux desktop app:
+Run desktop dev stack:
 
 ```bash
 npm run tauri:dev
 ```
 
-Phase 2 hotkey behavior on X11:
-
-- Hold `Ctrl+Space`: start recording after a short hold delay.
-- Release `Ctrl+Space`: stop recording and save a WAV file under `~/.cache/twokey-ai/recordings/`.
-- Double tap `Ctrl+Space`: cycle to the next mode.
-- Press `Escape`: cancel an active recording.
-
-On Wayland, generic global hold-hotkeys are reported as unavailable instead of failing silently.
-
-Phase 3 STT behavior:
-
-- Default STT provider is a deterministic mock transcriber.
-- To test a real local or custom STT command, set `TWOKEY_STT_COMMAND`.
-- The command must print the transcript to stdout and include `{audio}` as placeholder.
+Important:
 
-Example:
-
-```bash
-TWOKEY_STT_COMMAND='whisper-cli -f {audio} --language de --no-timestamps' npm run tauri:dev
-```
+- `target/debug/twokey-ai` alone in dev mode will fail if Vite is not running.
+- For development, use `npm run tauri:dev` so frontend + Tauri run together.
 
-Phase 4 Ollama behavior:
-
-- Ollama runs as a systemd service on `127.0.0.1:11434`.
-- Default model: `qwen2.5:3b`.
-- Conversation mode sends finished transcripts to Ollama and displays the answer in the overlay.
-- Override model or endpoint with:
+Build:
 
 ```bash
-TWOKEY_OLLAMA_MODEL='llama3.2:3b' TWOKEY_OLLAMA_URL='http://127.0.0.1:11434' npm run tauri:dev
+npm run build
+cd src-tauri && cargo check
 ```
 
-Phase 5 dictation behavior:
-
-- In dictation mode, a finished transcript is pasted at the active cursor position.
-- X11 uses `xclip` or `xsel` plus `xdotool`.
-- Clipboard content is read before insertion and restored afterward when possible.
-- Wayland insertion is deliberately reported as unsupported for now.
+## Hotkey Behavior
 
-Phase 6 text editing behavior:
+Default hotkey can be changed in settings. Examples:
 
-- In edit mode, TwoKey reads the current X11 selection with `Ctrl+C`.
-- The spoken instruction and selected text are sent to Ollama.
-- The overlay shows a replacement preview.
-- The selected text is replaced only after pressing `Ersetzen`.
+- `Ctrl+Space`
+- `Ctrl+Super`
+- mouse-button combinations except left/right mouse button in recorder UI
 
-Phase 7 settings behavior:
+Runtime semantics on X11:
 
-- Settings are persisted at `~/.config/twokey-ai/settings.json`.
-- The settings window saves general, hotkey, STT, Ollama, privacy, and update-channel values.
-- Some settings are stored before they are fully applied at runtime; later phases will wire them into the helper/provider layers.
+- hold hotkey: start recording
+- release hotkey: stop recording -> transcribe -> mode action
+- short single tap: open file picker
+- short double tap: switch mode
+- `Escape` while recording: cancel
 
-Phase 8 provider behavior:
+Wayland fallback:
 
-- Ollama chat is routed through a provider abstraction.
-- Provider metadata is exposed to the settings UI.
-- OpenAI-compatible and OpenRouter-compatible providers are visible as planned online providers, but disabled until API-key storage and routing are implemented.
+- if global hotkeys are blocked, manual start/stop recording is available from the menu
+- this keeps the voice pipeline usable on restricted desktops
 
-Phase 9 file context behavior:
+## STT Providers
 
-- File context can be added from the overlay menu.
-- `txt`, `md`, `markdown`, `json`, and `csv` files are read directly.
-- PDFs are extracted with `pdftotext` from `poppler-utils`.
-- Images are registered as context metadata for future vision providers.
-- Extracted text is cached under `~/.cache/twokey-ai/file-contexts/`.
+Configured in settings (`sttProvider`):
 
-Phase 10 packaging behavior:
+- `mock`
+- `local-whisper`
+- `external-command`
+- `openai-compatible`
 
-- `npm run tauri:build` creates AppImage and `.deb` bundles.
-- Settings autostart writes `~/.config/autostart/twokey-ai.desktop`.
-- Generated bundles live under `src-tauri/target/release/bundle/`.
+`external-command` needs `TWOKEY_STT_COMMAND` with `{audio}` placeholder.
 
-Phase 11 update behavior:
-
-- Settings can check GitHub Releases for a newer version.
-- The app only reports availability; it does not auto-download or auto-install.
-
-Current stabilization status:
-
-- AppImage and `.deb` builds complete successfully.
-- Ollama runs locally as a systemd service with `qwen2.5:3b`.
-- Production dependency audit reports no vulnerabilities.
-- Wayland limitations, TTS, tray menu, SQLite history, and online provider execution remain future hardening work.
-
-Build the frontend:
+Example:
 
 ```bash
-npm run build
+TWOKEY_STT_COMMAND='whisper-cli -f {audio} -l de -nt' npm run tauri:dev
 ```
 
-Build the desktop bundle:
+## Linux Requirements
 
-```bash
-npm run tauri:build
-```
+- Node.js 20+
+- npm
+- Rust + cargo
+- Linux desktop dependencies for Tauri
 
-## Project Layout
+Ubuntu/Debian example:
 
-```text
-docs/                 Product, architecture, security, and status docs
-src/                  React frontend
-src-tauri/            Rust/Tauri desktop shell
+```bash
+sudo apt install libwebkit2gtk-4.1-dev build-essential curl wget file libxdo-dev libssl-dev libayatana-appindicator3-dev librsvg2-dev
 ```
 
-## Repository
-
-GitHub: https://github.com/meinzeug/twokey
+Optional runtime tools:
 
-## License
+- X11 automation: `xdotool`, `xclip` or `xsel`
+- PDF extraction: `poppler-utils` (`pdftotext`)
+- TTS backends: `spd-say` or `espeak-ng`/`espeak`
 
-MIT. See `LICENSE`.
-
-## Linux Notes
-
-TwoKey is designed around Linux conventions:
+## Data Paths
 
 - Config: `~/.config/twokey-ai/`
 - Data: `~/.local/share/twokey-ai/`
 - Cache: `~/.cache/twokey-ai/`
-- Logs/state: `~/.local/state/twokey-ai/`
+- History DB: `~/.local/share/twokey-ai/history.db`
+- Toolchains: `~/.config/twokey-ai/toolchains.json`
+
+## Repo
+
+https://github.com/meinzeug/twokey
+
+## License
 
-X11 and Wayland will be handled separately in later phases. Phase 1 only detects and displays the current session type.
+MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "twokey",
-  "version": "1.0.3",
+  "version": "1.0.4",
   "description": "Linux-first desktop AI assistant built with Tauri, React, and TypeScript.",
   "license": "MIT",
   "repository": {
@@ -43,8 +43,8 @@
     "lint": "npm run typecheck",
     "test": "npm run typecheck",
     "prepublishOnly": "npm run build && npm pack --dry-run",
-    "tauri:dev": "tauri dev",
-    "tauri:build": "tauri build"
+    "tauri:dev": "GTK_MODULES='' LIBGL_ALWAYS_SOFTWARE=1 WEBKIT_DISABLE_DMABUF_RENDERER=1 tauri dev",
+    "tauri:build": "GTK_MODULES='' LIBGL_ALWAYS_SOFTWARE=1 WEBKIT_DISABLE_DMABUF_RENDERER=1 tauri build"
   },
   "dependencies": {
     "@tauri-apps/api": "^2.5.0",