openvoiceui 1.0.0

package/SETUP.md

@@ -0,0 +1,360 @@
+# OpenVoiceUI Setup Guide
+
+This guide covers three install paths:
+- [Docker](#docker-quick-start) — easiest, works on any OS
+- [VPS / Linux server](#vps-setup) — for production self-hosting
+- [Local development](#local-development) — for contributors
+
+---
+
+## Prerequisites
+
+| Requirement | Notes |
+|---|---|
+| **OpenClaw `2026.3.13`** | The AI gateway that powers conversations. Required. [Download here](https://openclaw.ai) |
+| **Groq API key** | For Orpheus TTS (fast, high quality). Free tier available. [Get key](https://console.groq.com) |
+| Python 3.10+ | For local / VPS installs |
+| Docker + Compose | For Docker install only |
+
+> **OpenClaw is the most important dependency.** Without it the server starts but cannot respond to any voice input. OpenVoiceUI is tested with **openclaw@2026.3.13** — other versions may have breaking changes. See [OpenClaw Requirements](docs/openclaw-requirements.md) for full compatibility details.
+
+---
+
+## Upgrading from Pre-2.0
+
+If you have an existing installation, runtime data directories have moved under `runtime/`:
+
+| Old Location | New Location |
+|---|---|
+| `uploads/` | `runtime/uploads/` |
+| `canvas-pages/` | `runtime/canvas-pages/` |
+| `known_faces/` | `runtime/known_faces/` |
+| `music/` | `runtime/music/` |
+| `generated_music/` | `runtime/generated_music/` |
+| `faces/` | `runtime/faces/` |
+| `transcripts/` | `runtime/transcripts/` |
+| `usage.db` | `runtime/usage.db` |
+
+To migrate, move your existing data into the new paths:
+```bash
+mkdir -p runtime
+for dir in uploads canvas-pages known_faces music generated_music faces transcripts; do
+  [ -d "$dir" ] && mv "$dir" "runtime/$dir"
+done
+[ -f usage.db ] && mv usage.db runtime/usage.db
+```
+
+Docker users: `docker compose down`, pull the latest code, then `docker compose up --build`. Volume mounts in `docker-compose.yml` already point to `runtime/`.
+
+---
+
+## OpenClaw Setup
+
+> **Docker users:** Skip this section — `docker compose up` installs and configures
+> OpenClaw automatically with the correct version and settings.
+
+1. Install the tested version: `npm i -g openclaw@2026.3.13`
+2. Run the setup wizard: `openclaw onboard` (choose your LLM provider and API key)
+3. Start the gateway: `openclaw gateway` (listens on `ws://127.0.0.1:18791`)
+4. Copy your auth token — you'll need it for `CLAWDBOT_AUTH_TOKEN` in `.env`
+
+**Using an existing OpenClaw install?** See [OpenClaw Requirements](docs/openclaw-requirements.md)
+for the full list of version and configuration requirements. OpenVoiceUI needs
+specific gateway settings to work — don't skip this if you already have OpenClaw
+running with other agents.
+
+---
+
+## Docker Quick Start
+
+**Fastest path. Recommended for trying OpenVoiceUI.**
+
+```bash
+git clone https://github.com/MCERQUA/OpenVoiceUI.git
+cd OpenVoiceUI
+cp .env.example .env
+```
+
+---
+
+### Do you already have OpenClaw running?
+
+#### No — start everything fresh (recommended)
+
+The compose stack starts three containers for you:
+- **openclaw** — AI gateway on port 18791
+- **openvoiceui** — the UI/API server on port 5001
+- **supertonic** — local TTS engine
+
+Edit `.env` and set at minimum:
+```bash
+CLAWDBOT_AUTH_TOKEN=your-openclaw-token   # from openclaw gateway config
+GROQ_API_KEY=your-groq-key
+SECRET_KEY=any-random-string-here
+```
+
+**Optional: enable the coding-agent skill**
+
+The coding-agent skill lets the AI write code, create files, and run commands
+autonomously. It requires a coding CLI installed in the openclaw container.
+Set `CODING_CLI` in your `.env` before building — same options as openclaw's
+setup wizard:
+
+```bash
+# Choose one (or leave unset to skip):
+CODING_CLI=codex      # OpenAI Codex — also needs OPENAI_API_KEY
+CODING_CLI=claude     # Anthropic Claude Code — also needs ANTHROPIC_API_KEY
+CODING_CLI=opencode   # OpenCode — bring your own provider key
+CODING_CLI=pi         # Pi coding agent — bring your own provider key
+```
+
+> If you already ran openclaw's interactive setup wizard, it asked you this
+> question — you don't need to set it here.
+
+```bash
+docker compose up --build
+```
+
+#### Yes — connect to your existing OpenClaw
+
+> **Important:** OpenVoiceUI is tested with **openclaw@2026.3.13**. If your existing
+> install is a different version, voice features may not work. See
+> [OpenClaw Requirements](docs/openclaw-requirements.md) for the full compatibility
+> checklist.
+
+Point openvoiceui at your running OpenClaw gateway instead of starting a new one.
+
+1. Make sure your existing openclaw gateway has `bind: "lan"` (not `"loopback"`) so it
+   accepts connections from other containers, and the required auth settings:
+   ```json
+   "gateway": {
+     "bind": "lan",
+     "auth": { "mode": "token" },
+     "controlUi": {
+       "dangerouslyDisableDeviceAuth": true,
+       "dangerouslyAllowHostHeaderOriginFallback": true
+     }
+   }
+   ```
+
+2. Share the canvas-pages directory between your existing openclaw container and openvoiceui
+   (both need to read/write the same pages). Add a bind mount to **both** containers:
+   ```yaml
+   # your existing openclaw container (add to its volumes):
+   - ./canvas-pages:/path/to/openclaw/workspace/canvas-pages
+
+   # openvoiceui (already in docker-compose.yml):
+   - ./canvas-pages:/app/runtime/canvas-pages
+   ```
+   Pre-create the canvas manifest file before starting (Docker would otherwise create it as a directory):
+   ```bash
+   mkdir -p canvas-pages
+   echo '{"pages":{},"categories":{},"order":[]}' > canvas-manifest.json
+   ```
+
+3. Edit `.env`:
+   ```bash
+   CLAWDBOT_GATEWAY_URL=ws://<your-openclaw-host>:<port>   # e.g. ws://192.168.1.10:18791
+   CLAWDBOT_AUTH_TOKEN=your-openclaw-token
+   GROQ_API_KEY=your-groq-key
+   SECRET_KEY=any-random-string-here
+   ```
+
+4. Start only the openvoiceui and supertonic services (skip the built-in openclaw):
+   ```bash
+   docker compose up --build openvoiceui supertonic
+   ```
+
+---
+
+> Leave `CANVAS_PAGES_DIR` unset for Docker — it defaults correctly to the mounted volume.
+
+Open [http://localhost:5001](http://localhost:5001) in your browser. Allow microphone access and speak.
+
+**To stop:**
+```bash
+docker compose down
+```
+
+**Persistent data** (canvas pages, music, uploads, transcripts) lives in Docker named volumes and survives container restarts.
+
+---
+
+## VPS Setup
+
+For a production install on a Linux VPS with nginx + SSL.
+
+### 1. Clone and configure
+
+```bash
+git clone https://github.com/MCERQUA/OpenVoiceUI.git
+cd OpenVoiceUI
+cp .env.example .env
+nano .env   # or your preferred editor
+```
+
+Set these in `.env`:
+```bash
+PORT=5001
+DOMAIN=your-domain.com
+SECRET_KEY=<run: python3 -c "import secrets; print(secrets.token_hex(32))">
+CLAWDBOT_AUTH_TOKEN=your-openclaw-token
+CLAWDBOT_GATEWAY_URL=ws://127.0.0.1:18791
+GROQ_API_KEY=your-groq-key
+CANVAS_PAGES_DIR=/var/www/openvoiceui/canvas-pages
+```
+
+### 2. Create Python virtual environment
+
+```bash
+python3 -m venv venv
+venv/bin/pip install -r requirements.txt
+```
+
+### 3. Test the server runs
+
+```bash
+set -a && source .env && set +a
+venv/bin/python3 server.py
+```
+
+Open `http://your-server-ip:5001` to verify. Press Ctrl+C when done.
+
+### 4. Run the setup script (nginx + SSL + systemd)
+
+Edit the top of `deploy/setup-sudo.sh` to set your domain and email, then:
+
+```bash
+sudo bash deploy/setup-sudo.sh
+```
+
+This creates:
+- `/etc/nginx/sites-available/your-domain.com` — nginx reverse proxy config
+- `/etc/systemd/system/openvoiceui.service` — systemd service
+- `/var/www/openvoiceui/canvas-pages` — canvas page storage directory
+- Let's Encrypt SSL certificate
+
+### 5. Verify
+
+```bash
+sudo systemctl status openvoiceui
+sudo journalctl -u openvoiceui -f
+```
+
+Open `https://your-domain.com` in your browser.
+
+---
+
+## Local Development
+
+For contributors running without Docker or a VPS.
+
+```bash
+git clone https://github.com/MCERQUA/OpenVoiceUI.git
+cd OpenVoiceUI
+python3 -m venv venv
+venv/bin/pip install -r requirements.txt
+cp .env.example .env
+# Edit .env — set CLAWDBOT_AUTH_TOKEN and GROQ_API_KEY at minimum
+venv/bin/python3 server.py
+```
+
+Open [http://localhost:5001](http://localhost:5001).
+
+The system prompt (`prompts/voice-system-prompt.md`) hot-reloads — edit it without restarting the server.
+
+---
+
+## Configuration Reference
+
+All configuration is via `.env`. Key variables:
+
+| Variable | Required | Default | Description |
+|---|---|---|---|
+| `CLAWDBOT_AUTH_TOKEN` | **Yes** | — | OpenClaw gateway auth token |
+| `CLAWDBOT_GATEWAY_URL` | No | `ws://127.0.0.1:18791` | OpenClaw WebSocket URL |
+| `OPENCLAW_VERSION` | No | `2026.3.13` | Docker build arg: OpenClaw version to install |
+| `GROQ_API_KEY` | Recommended | — | Groq Orpheus TTS |
+| `SECRET_KEY` | Recommended | random | Flask session key |
+| `PORT` | No | `5001` | Server port |
+| `CANVAS_PAGES_DIR` | No | `canvas-pages/` in app dir | Where canvas HTML pages are stored |
+| `GATEWAY_SESSION_KEY` | No | `voice-main-1` | Session prefix (change for multiple instances) |
+| `SUPERTONIC_MODEL_PATH` | No | — | Path to local ONNX TTS model |
+| `FAL_KEY` | No | — | fal.ai key for Qwen3-TTS |
+| `HUME_API_KEY` | No | — | Hume EVI TTS |
+| `HUME_SECRET_KEY` | No | — | Hume EVI TTS secret |
+| `GEMINI_API_KEY` | No | — | Vision / screenshot analysis |
+| `SUNO_API_KEY` | No | — | AI music generation |
+| `CLERK_PUBLISHABLE_KEY` | No | — | Auth (leave unset for open access) |
+
+---
+
+## Useful Commands
+
+```bash
+# VPS: view live logs
+sudo journalctl -u openvoiceui -f
+
+# VPS: restart
+sudo systemctl restart openvoiceui
+
+# VPS: status
+systemctl status openvoiceui
+
+# Docker: view logs
+docker compose logs -f
+
+# Docker: restart
+docker compose restart
+
+# Run tests
+venv/bin/python -m pytest tests/
+```
+
+---
+
+## Troubleshooting
+
+**Voice input not working**
+- Allow microphone in browser (HTTPS required in production, HTTP localhost is fine for dev)
+- Check browser console for WebSpeech API errors
+- Chrome/Edge recommended; Firefox has limited WebSpeech support
+
+**Agent not responding**
+- Check OpenClaw is running: `ss -tlnp | grep 18791`
+- Check `CLAWDBOT_AUTH_TOKEN` is set in `.env` and matches your OpenClaw token
+- Check logs: `sudo journalctl -u openvoiceui -f` or `docker compose logs -f`
+- Look for `### Persistent WS connected` in logs — if missing, gateway connection failed
+
+**TTS audio not playing**
+- Check `GROQ_API_KEY` is set and valid
+- Try a different TTS provider in the Settings panel
+- Check logs for `tts_error` events
+
+**502 Bad Gateway (nginx)**
+- Verify the server is running: `systemctl status openvoiceui`
+- Verify PORT in `.env` matches nginx proxy port (default 5001)
+- Check nginx error log: `sudo tail -f /var/log/nginx/error.log`
+
+**Canvas pages not loading / black screen**
+- Verify `CANVAS_PAGES_DIR` path exists and is writable by the server user
+- Docker: leave `CANVAS_PAGES_DIR` unset so it uses the mounted volume
+- Docker: both `openclaw` and `openvoiceui` share the `canvas-pages` named volume — if you
+  customised the compose file make sure both services mount it at the same paths as the
+  default `docker-compose.yml`
+- Check logs for canvas route errors
+
+**Permission errors on VPS**
+- Canvas dir and uploads must be owned by the service user: `sudo chown -R $USER /var/www/openvoiceui`
+
+**Separate openclaw container (not using docker-compose)**
+- If you run openclaw outside of this compose stack (e.g. an existing installation), make sure
+  openclaw's gateway `bind` is set to `"lan"` (not `"loopback"`) so openvoiceui can reach it:
+  ```json
+  "gateway": {
+    "bind": "lan",
+    "controlUi": { "dangerouslyAllowHostHeaderOriginFallback": true }
+  }
+  ```
+- Share the canvas-pages directory between the two containers via a bind mount so openclaw
+  can write pages that openvoiceui serves.

package/app.py

@@ -0,0 +1,232 @@
+"""
+Flask application factory for ai-eyes2.
+
+Usage:
+    from app import create_app
+    app, sock = create_app()
+
+This factory pattern allows:
+- Blueprint registration (Phase 2 tasks P2-T2 through P2-T8)
+- Test isolation via config_override
+- Clean extension initialization
+
+ADR-009 (simple manager pattern): factory returns app + sock tuple so
+server.py module-level decorators (@app.route, @sock.route) keep working.
+"""
+import logging
+import os
+
+from flask import Flask, jsonify, redirect, request
+from flask_cors import CORS
+from flask_limiter import Limiter
+from flask_limiter.util import get_remote_address
+from flask_sock import Sock
+from werkzeug.middleware.proxy_fix import ProxyFix
+
+logger = logging.getLogger(__name__)
+
+# Match static_files.py and nginx (100 MB) — bulk uploads need full limit
+_MAX_UPLOAD_BYTES = 100 * 1024 * 1024  # 100 MB
+
+
+def create_app(config_override: dict = None):
+    """
+    Create and configure the Flask application.
+
+    Args:
+        config_override: Optional dict of Flask config values to apply.
+                         Primarily used in tests to inject TESTING=True etc.
+
+    Returns:
+        tuple: (app, sock) — configured Flask app and Flask-Sock instance.
+    """
+    app = Flask(
+        __name__,
+        # Serve static files from project root (index.html etc.) via explicit routes
+        static_folder=None,
+    )
+
+    # Core Flask config
+    secret_key = os.getenv('SECRET_KEY')
+    if not secret_key:
+        import secrets as _secrets
+        secret_key = _secrets.token_hex(32)
+        logger.warning(
+            'No SECRET_KEY set — generated a random key for this session. '
+            'Sessions will NOT persist across restarts. '
+            'Set SECRET_KEY in .env for production.'
+        )
+    app.config['SECRET_KEY'] = secret_key
+    app.config['MAX_CONTENT_LENGTH'] = _MAX_UPLOAD_BYTES
+
+    # Apply test / caller overrides last so they take precedence
+    if config_override:
+        app.config.update(config_override)
+
+    # Trust one level of X-Forwarded-* headers (nginx / reverse proxy).
+    # Without this, request.remote_addr is always 127.0.0.1 behind nginx,
+    # breaking per-IP rate limiting (all users share one bucket).
+    app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1)
+
+    # Initialize Flask-Sock for WebSocket support
+    sock = Sock(app)
+
+    # Configure CORS — allow your production domain and any localhost port for dev
+    # Anchored regex prevents partial matches like http://localhostX.evil.com
+    # Add extra origins via CORS_ORIGINS env var (comma-separated, e.g. https://yourdomain.com)
+    _extra_origins = [o.strip() for o in os.getenv('CORS_ORIGINS', '').split(',') if o.strip()]
+    CORS(app, origins=[
+        r'^http://localhost:\d+$',
+        *_extra_origins,
+    ], supports_credentials=True)
+
+    # ── Rate limiting ─────────────────────────────────────────────────────────
+    # Per-IP limits protect expensive endpoints from abuse.
+    # Override default via RATELIMIT_DEFAULT env var (e.g. "100 per minute").
+    # Disable for tests: config_override={'RATELIMIT_ENABLED': False}.
+    limiter = Limiter(
+        get_remote_address,
+        app=app,
+        default_limits=[os.getenv('RATELIMIT_DEFAULT', '200 per minute')],
+        storage_uri='memory://',
+    )
+    app.limiter = limiter
+
+    # ── Clerk auth gate ────────────────────────────────────────────────────────
+    # Auth is only active when CLERK_PUBLISHABLE_KEY is set in .env.
+    # Without it, the app runs fully open (single-user / local mode).
+    _clerk_key = (os.getenv('CLERK_PUBLISHABLE_KEY') or os.getenv('VITE_CLERK_PUBLISHABLE_KEY', '')).strip()
+    _auth_enabled = bool(_clerk_key)
+
+    if not _auth_enabled:
+        logger.info('No CLERK_PUBLISHABLE_KEY set — auth disabled (local mode)')
+    else:
+        # Routes that never require authentication:
+        _PUBLIC_PREFIXES = (
+            '/src/',       # static JS/CSS (needed to render the login screen)
+            '/sounds/',
+            '/music/',
+            '/images/',    # canvas images (individual pages check their own flag)
+            '/uploads/',   # uploaded/generated files — served from VPS filesystem (no secrets)
+            '/static/',    # PWA icons, app icons
+            '/pages/',     # canvas pages — served without auth (CANVAS_REQUIRE_AUTH opt-in)
+            '/api/canvas/',  # canvas API — creation, manifest, context (no per-user auth needed)
+            '/api/uploads',   # uploads list — files are already public at /uploads/, listing is fine
+            '/api/profiles',  # read-only profile config — loaded before Clerk init
+            '/api/chat',      # LLM proxy (Groq) — used by canvas pages for inline AI
+            '/api/tts/',      # TTS provider list — loaded before Clerk init
+            '/api/theme',     # theme config — loaded before Clerk init
+            '/api/music',     # music track list — loaded before Clerk init
+            '/api/faces',     # face list — loaded before Clerk init
+            '/api/icons/',    # icon library + generated icons — static images, no secrets
+        )
+        _PUBLIC_EXACT = {
+            '/',           # main page — hosts the Clerk login gate itself
+            '/pi',         # Pi-optimized page — same login gate, different entry point
+            '/health/live',
+            '/health/ready',
+            '/api/auth/check',      # Auth check endpoint — does its own token verification
+            '/api/suno/callback',   # Suno's servers POST here from external IPs (no Clerk token)
+            '/sw.js',           # PWA service worker — browser fetches this before auth
+            '/manifest.json',   # PWA manifest — browser fetches this before auth
+            '/favicon.ico',     # Browser favicon request — before auth
+            '/ws/clawdbot',     # WebSocket — browsers can't send Clerk token in WS headers;
+                                # handler secures itself via CLAWDBOT_AUTH_TOKEN to the gateway
+            '/openclaw-ui',     # WebSocket upgrade for OpenClaw Control UI proxy;
+                                # handler does its own Clerk auth via __session cookie
+        }
+
+        # Detect whether Clerk auth is configured at startup.
+        # Auth is opt-in: when no key is set, all routes are accessible (README § Authentication).
+        _clerk_key = (os.getenv('CLERK_PUBLISHABLE_KEY') or os.getenv('VITE_CLERK_PUBLISHABLE_KEY', '')).strip()
+
+        # Internal agent API key — allows openclaw agents to call Flask APIs
+        # without a Clerk JWT. Set AGENT_API_KEY in the container .env.
+        _agent_api_key = os.getenv('AGENT_API_KEY', '').strip()
+
+        @app.before_request
+        def require_auth():
+            """Block unauthenticated requests to all non-exempt routes.
+
+            Skipped entirely when Clerk is not configured (no CLERK_PUBLISHABLE_KEY),
+            matching the documented opt-in auth behaviour.
+            """
+            if not _clerk_key:
+                return  # No Clerk configured — open access (single-user / self-hosted)
+
+            path = request.path
+
+            # Always allow health probes and static assets
+            if path in _PUBLIC_EXACT:
+                return
+            if any(path.startswith(p) for p in _PUBLIC_PREFIXES):
+                return
+            # Canvas pages and images have their own auth logic (public flag)
+            # handled inside canvas_bp — let them through here
+            if path.startswith('/pages/') or path.startswith('/canvas-proxy') or path.startswith('/website-dev'):
+                return
+
+            # Internal agent API key — openclaw agents calling Flask APIs from inside Docker network
+            if _agent_api_key and request.headers.get('X-Agent-Key') == _agent_api_key:
+                return
+
+            from services.auth import get_token_from_request, verify_clerk_token
+            token = get_token_from_request()
+            user_id = verify_clerk_token(token) if token else None
+
+            if not user_id:
+                # For API calls return JSON 401; for page navigations redirect to /
+                if path.startswith('/api/') or request.headers.get('X-Requested-With'):
+                    return jsonify({'error': 'Unauthorized', 'code': 'auth_required'}), 401
+                # HTML page request — redirect to root (login gate)
+                return redirect('/')
+
+    # ── JSON error handler for 413 (file too large) ────────────────────────
+    @app.errorhandler(413)
+    def handle_413(e):
+        return jsonify({'error': 'File too large (100 MB max)'}), 413
+
+    # ── Security headers (P7-T3 security audit) ──────────────────────────────
+    @app.after_request
+    def add_security_headers(response):
+        """Add defensive HTTP security headers to every response."""
+        response.headers.setdefault('X-Content-Type-Options', 'nosniff')
+        response.headers.setdefault('X-Frame-Options', 'SAMEORIGIN')
+        response.headers.setdefault('X-XSS-Protection', '1; mode=block')
+        response.headers.setdefault('Referrer-Policy', 'strict-origin-when-cross-origin')
+        # Allow microphone and camera for voice/vision app; block geolocation
+        response.headers.setdefault(
+            'Permissions-Policy', 'camera=(self), microphone=(self), geolocation=()'
+        )
+        response.headers.setdefault(
+            'Content-Security-Policy',
+            "default-src 'self'; "
+            "script-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net https://*.clerk.accounts.dev https://*.jam-bot.com; "
+            "style-src 'self' 'unsafe-inline'; "
+            "img-src 'self' data: blob: https://img.clerk.com https://images.clerk.dev https://*.clerk.accounts.dev https://lh3.googleusercontent.com https://avatars.githubusercontent.com; "
+            "media-src 'self' blob:; "
+            "connect-src 'self' wss: https:; "
+            "frame-src 'self' https://*.clerk.accounts.dev https://*.jam-bot.com https:; "
+            "worker-src 'self' blob:"
+        )
+        return response
+
+    # ── CDN cache cleanup (MUST run after flask_cors) ──────────────────────
+    # flask_cors adds Vary:Origin + Access-Control-* to ALL responses, which
+    # causes Cloudflare to mark them cf-cache-status:DYNAMIC (uncacheable).
+    # Canvas media files don't need CORS — strip those headers so CDN caches them.
+    # Inserted at position 0 in after_request list so it runs LAST in LIFO order.
+    def _strip_cdn_blocking_headers(response):
+        _media_exts = ('.mp4', '.webm', '.mp3', '.wav', '.ogg', '.png', '.jpg',
+                       '.jpeg', '.gif', '.svg', '.webp', '.pdf')
+        if request.path.startswith('/pages/') and any(request.path.endswith(e) for e in _media_exts):
+            for h in ['Vary', 'Access-Control-Allow-Origin',
+                      'Access-Control-Allow-Credentials',
+                      'Content-Security-Policy', 'X-Frame-Options',
+                      'Permissions-Policy', 'X-XSS-Protection',
+                      'Referrer-Policy']:
+                response.headers.pop(h, None)
+        return response
+    app.after_request_funcs.setdefault(None, []).insert(0, _strip_cdn_blocking_headers)
+
+    return app, sock