flawed-avatar 0.2.1 → 0.2.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Peter Steinberger
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,217 @@
+<p align="center">
+  <img src="assets/icon.png" width="80" alt="flawed-avatar icon" />
+</p>
+
+<h1 align="center">flawed-avatar</h1>
+
+<p align="center">
+  A 3D avatar overlay that gives your <a href="https://github.com/nichochar/open-claw">OpenClaw</a> agent a face.<br/>
+  Real-time expressions, lip-sync, text-to-speech, and eye tracking — all running locally.
+</p>
+
+<p align="center">
+  <a href="https://www.npmjs.com/package/flawed-avatar"><img src="https://img.shields.io/npm/v/flawed-avatar?color=cb3837&label=npm" alt="npm version" /></a>
+  <a href="LICENSE"><img src="https://img.shields.io/github/license/RyuuTheChosen/flawed-openclaw?color=blue" alt="MIT License" /></a>
+  <a href="https://github.com/RyuuTheChosen/flawed-openclaw"><img src="https://img.shields.io/badge/platform-Windows%20%7C%20macOS%20%7C%20Linux-lightgrey" alt="Platforms" /></a>
+</p>
+
+---
+
+## What it does
+
+The avatar sits in a transparent, always-on-top overlay and reacts to your agent in real time:
+
+| Agent state | Avatar behavior |
+|---|---|
+| **Idle** | Breathing animation, relaxed posture, ambient eye saccades |
+| **Thinking** | Surprised expression, amplified head sway |
+| **Speaking** | Happy expression, lip-sync driven by TTS audio or text |
+| **Working** | Relaxed expression, subtle working tilt, head nod |
+
+It connects to the OpenClaw gateway over WebSocket and listens for agent lifecycle events — no polling, no config wiring.
+
+## Features
+
+**Avatar & Animation**
+- VRM model rendering via Three.js and [@pixiv/three-vrm](https://github.com/pixiv/three-vrm)
+- Compound facial expressions with cubic-eased blend shape transitions
+- Procedural breathing, head sway, speaking nod, and working tilt
+- FBX animation clips per phase (idle, thinking, speaking, working) with Mixamo retargeting
+- Spring bone physics for hair and accessories
+- Image-based lighting (IBL) with spherical harmonics
+
+**Lip-sync & TTS**
+- Audio-driven viseme blending via [wLipSync](https://github.com/hecomi/uLipSync)
+- Local neural TTS via [Kokoro](https://github.com/hexgrad/kokoro) (11 voices, offline ONNX)
+- Browser Web Speech API as a lightweight alternative
+- Text-based lip-sync fallback when TTS is off
+
+**Eye & Gaze**
+- Eyes track your cursor across the entire screen
+- Micro-saccades with configurable yaw/pitch range and hold durations
+- Hover awareness — avatar reacts when you mouse over it
+
+**Desktop Integration**
+- Transparent, click-through Electron overlay (mouse passes through empty pixels)
+- Native drag to reposition
+- Scroll-wheel zoom (0.5x to 6.0x)
+- System tray with show/hide, model picker, and settings
+- Chat window to message the active agent directly
+- Settings panel for scale, lighting, TTS engine, and voice selection
+- All preferences persisted between sessions
+
+**Multi-agent**
+- Per-agent VRM model assignment (different agents get different avatars)
+- Automatic model switching when the active agent changes
+
+## Install
+
+### macOS / Linux
+
+```bash
+openclaw plugins install flawed-avatar
+openclaw plugins enable flawed-avatar
+```
+
+### Windows
+
+> `openclaw plugins install <npm-package>` is currently broken on Windows + Node.js v22+ due to an upstream OpenClaw bug. Use the tarball workaround:
+
+```bash
+npm pack flawed-avatar
+openclaw plugins install ./flawed-avatar-0.2.1.tgz
+cd %USERPROFILE%\.openclaw\extensions\flawed-avatar
+npm install --omit=dev
+openclaw plugins enable flawed-avatar
+```
+
+Then restart the gateway:
+
+```bash
+openclaw gateway restart
+```
+
+The avatar overlay will appear automatically.
+
+## Configuration
+
+Add plugin settings to `~/.openclaw/openclaw.json`:
+
+```json
+{
+  "plugins": {
+    "entries": {
+      "flawed-avatar": {
+        "enabled": true,
+        "config": {
+          "autoStart": true,
+          "vrmPath": "/path/to/custom-model.vrm",
+          "gatewayUrl": "ws://127.0.0.1:18789",
+          "agents": {
+            "agent:researcher:main": { "vrmPath": "/models/researcher.vrm" },
+            "agent:coder:main": { "vrmPath": "/models/coder.vrm" }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `autoStart` | `boolean` | `true` | Launch overlay when OpenClaw starts |
+| `vrmPath` | `string` | bundled model | Path to a default VRM model |
+| `gatewayUrl` | `string` | `ws://127.0.0.1:18789` | OpenClaw gateway WebSocket URL |
+| `agents` | `object` | — | Per-agent VRM overrides keyed by session key |
+
+## Controls
+
+| Input | Action |
+|---|---|
+| **Scroll wheel** | Zoom in/out |
+| **Drag handle** | Reposition the overlay |
+| Chat icon | Toggle chat window |
+| Speaker icon | Toggle TTS |
+| Gear icon | Open settings panel |
+| Tray icon | Show/hide, change model, quit |
+
+## Settings panel
+
+- **Scale** — 0.5x to 2.0x avatar size
+- **Lighting** — Studio, Warm, Cool, Neutral, or Custom profiles
+- **TTS Engine** — Web Speech (browser) or Kokoro (local neural)
+- **TTS Voice** — 11 Kokoro voices (American/British, male/female) or system voices
+
+## Architecture
+
+```
+Plugin Service (Node.js)
+  │
+  ├── Spawns Electron child process
+  │     │
+  │     ├── Main Process
+  │     │     ├── Gateway WebSocket client (agent events)
+  │     │     ├── Window manager (avatar + chat + settings)
+  │     │     ├── System tray
+  │     │     └── Persistence (settings, chat history)
+  │     │
+  │     └── Renderer (Three.js)
+  │           ├── VRM loader + spring bones + IBL
+  │           ├── Animator (expressions, breathing, gaze, lip-sync)
+  │           └── Audio pipeline (Kokoro TTS → wLipSync → visemes)
+  │
+  └── Stdin IPC (show/hide/shutdown/model-switch)
+```
+
+## Development
+
+```bash
+git clone https://github.com/RyuuTheChosen/flawed-openclaw.git
+cd flawed-openclaw
+npm install
+npm run dev     # build + launch Electron
+```
+
+| Script | Description |
+|---|---|
+| `npm run build` | TypeScript + Rolldown bundle + copy renderer assets |
+| `npm run dev` | Build and launch in one step |
+| `npm run start` | Launch the last build without recompiling |
+
+### Project structure
+
+```
+index.ts                        Plugin registration (OpenClaw SDK)
+src/service.ts                  Electron process lifecycle manager
+src/main/
+  main.ts                       Electron entry point
+  gateway-client.ts             WebSocket client (protocol v3)
+  window-manager.ts             Multi-window coordination
+  tray.ts                       System tray menu
+  persistence/                  JSON file store with migrations
+src/renderer/
+  renderer.ts                   Boot sequence and render loop
+  avatar/                       VRM, animator, expressions, eye gaze, spring bones
+  audio/                        TTS engines, lip-sync, phoneme mapping
+  ui/                           Chat bubble, typing indicator
+  chat-window/                  Chat panel renderer
+  settings-window/              Settings panel renderer
+src/shared/
+  config.ts                     All tunable constants
+  types.ts                      AgentPhase, AgentState
+  ipc-channels.ts               Electron IPC channel definitions
+assets/
+  models/                       Bundled VRM avatars
+  animations/{idle,thinking,speaking,working}/   FBX motion clips
+```
+
+## Requirements
+
+- Node.js >= 18
+- OpenClaw (peer dependency)
+- Desktop environment with display server (auto-skips on headless Linux)
+
+## License
+
+[MIT](LICENSE)

package/dist/chat-renderer-bundle/chat-index.html

@@ -3,7 +3,7 @@
 <head>
   <meta charset="UTF-8" />
   <meta http-equiv="Content-Security-Policy" content="default-src 'self' file: data: blob:; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' file: data: blob:; connect-src 'self' file: data: blob:" />
-  <title>OpenClaw Chat</title>
+  <title>Flawed Avatar Chat</title>
   <link rel="stylesheet" href="./styles/chat.css" />
 </head>
 <body>

package/dist/main/main/device-identity.d.ts

@@ -0,0 +1,19 @@
+export type DeviceIdentity = {
+    deviceId: string;
+    publicKeyPem: string;
+    privateKeyPem: string;
+};
+export declare function loadDeviceIdentity(): DeviceIdentity | null;
+export declare function loadStoredAuthToken(deviceId: string, role: string): string | null;
+export declare function signPayload(privateKeyPem: string, payload: string): string;
+export declare function publicKeyToBase64Url(publicKeyPem: string): string;
+export declare function buildAuthPayload(params: {
+    deviceId: string;
+    clientId: string;
+    clientMode: string;
+    role: string;
+    scopes: string[];
+    signedAtMs: number;
+    token: string | null;
+    nonce: string | undefined;
+}): string;

package/dist/main/main/device-identity.js

@@ -0,0 +1,83 @@
+import * as crypto from "node:crypto";
+import * as fs from "node:fs";
+import * as path from "node:path";
+import * as os from "node:os";
+const IDENTITY_DIR = path.join(os.homedir(), ".openclaw", "identity");
+const DEVICE_FILE = path.join(IDENTITY_DIR, "device.json");
+const DEVICE_AUTH_FILE = path.join(IDENTITY_DIR, "device-auth.json");
+const ED25519_SPKI_PREFIX = Buffer.from("302a300506032b6570032100", "hex");
+function base64UrlEncode(buf) {
+    return buf.toString("base64").replaceAll("+", "-").replaceAll("/", "_").replace(/=+$/g, "");
+}
+function derivePublicKeyRaw(publicKeyPem) {
+    const spki = crypto.createPublicKey(publicKeyPem).export({ type: "spki", format: "der" });
+    if (spki.length === ED25519_SPKI_PREFIX.length + 32 &&
+        spki.subarray(0, ED25519_SPKI_PREFIX.length).equals(ED25519_SPKI_PREFIX)) {
+        return spki.subarray(ED25519_SPKI_PREFIX.length);
+    }
+    return spki;
+}
+export function loadDeviceIdentity() {
+    try {
+        if (!fs.existsSync(DEVICE_FILE))
+            return null;
+        const raw = fs.readFileSync(DEVICE_FILE, "utf8");
+        const parsed = JSON.parse(raw);
+        if (parsed?.version === 1 &&
+            typeof parsed.deviceId === "string" &&
+            typeof parsed.publicKeyPem === "string" &&
+            typeof parsed.privateKeyPem === "string") {
+            return {
+                deviceId: parsed.deviceId,
+                publicKeyPem: parsed.publicKeyPem,
+                privateKeyPem: parsed.privateKeyPem,
+            };
+        }
+    }
+    catch {
+        // Corrupt or unreadable
+    }
+    return null;
+}
+export function loadStoredAuthToken(deviceId, role) {
+    try {
+        if (!fs.existsSync(DEVICE_AUTH_FILE))
+            return null;
+        const raw = fs.readFileSync(DEVICE_AUTH_FILE, "utf8");
+        const parsed = JSON.parse(raw);
+        if (parsed?.version !== 1 || parsed.deviceId !== deviceId)
+            return null;
+        if (!parsed.tokens || typeof parsed.tokens !== "object")
+            return null;
+        const entry = parsed.tokens[role.trim()];
+        if (!entry || typeof entry.token !== "string")
+            return null;
+        return entry.token;
+    }
+    catch {
+        return null;
+    }
+}
+export function signPayload(privateKeyPem, payload) {
+    const key = crypto.createPrivateKey(privateKeyPem);
+    return base64UrlEncode(crypto.sign(null, Buffer.from(payload, "utf8"), key));
+}
+export function publicKeyToBase64Url(publicKeyPem) {
+    return base64UrlEncode(derivePublicKeyRaw(publicKeyPem));
+}
+export function buildAuthPayload(params) {
+    const version = params.nonce ? "v2" : "v1";
+    const base = [
+        version,
+        params.deviceId,
+        params.clientId,
+        params.clientMode,
+        params.role,
+        params.scopes.join(","),
+        String(params.signedAtMs),
+        params.token ?? "",
+    ];
+    if (version === "v2")
+        base.push(params.nonce ?? "");
+    return base.join("|");
+}

package/dist/main/main/gateway-client.d.ts

@@ -1,4 +1,5 @@
 import type { AgentState } from "../shared/types.js";
+import type { DeviceIdentity } from "./device-identity.js";
 /**
  * Lightweight gateway WebSocket client for the Electron main process.
  * Implements the minimal protocol v3 handshake (without device auth)
@@ -6,7 +7,7 @@ import type { AgentState } from "../shared/types.js";
  */
 export declare function createGatewayClient(gatewayUrl: string, onStateChange: (state: AgentState) => void, onModelSwitch: (vrmPath: string) => void, agentConfigs?: Record<string, {
     vrmPath?: string;
-}>, authToken?: string): {
+}>, authToken?: string, deviceIdentity?: DeviceIdentity | null): {
     destroy: () => void;
     sendChat: (text: string, sessionKey: string | null) => void;
     getCurrentAgentId: () => string | null;

package/dist/main/main/gateway-client.js

@@ -1,13 +1,28 @@
 import WebSocket from "ws";
 import { randomUUID } from "node:crypto";
+import * as fs from "node:fs";
+import * as path from "node:path";
+import { fileURLToPath } from "node:url";
 import { GATEWAY_RECONNECT_BASE_MS, GATEWAY_RECONNECT_MAX_MS } from "../shared/config.js";
+import { loadStoredAuthToken, buildAuthPayload, signPayload, publicKeyToBase64Url } from "./device-identity.js";
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+const PKG_VERSION = (() => {
+    try {
+        const raw = fs.readFileSync(path.resolve(__dirname, "..", "..", "..", "package.json"), "utf-8");
+        return JSON.parse(raw).version;
+    }
+    catch {
+        return "0.0.0";
+    }
+})();
 const PROTOCOL_VERSION = 3;
 /**
  * Lightweight gateway WebSocket client for the Electron main process.
  * Implements the minimal protocol v3 handshake (without device auth)
  * and listens for "agent" event frames to drive avatar animations.
  */
-export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, agentConfigs, authToken) {
+export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, agentConfigs, authToken, deviceIdentity) {
     let ws = null;
     let destroyed = false;
     let backoffMs = GATEWAY_RECONNECT_BASE_MS;
@@ -25,7 +40,6 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
         const { stream, data, sessionKey } = evt;
         // Track session changes - agent events contain the actual sessionKey
         if (sessionKey && sessionKey !== currentSessionKey) {
-            console.log("flawed-avatar: setting currentSessionKey from agent event:", sessionKey);
             currentSessionKey = sessionKey;
             if (agentConfigs?.[sessionKey]?.vrmPath) {
                 onModelSwitch(agentConfigs[sessionKey].vrmPath);
@@ -86,7 +100,6 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
                             const firstSession = sessions[0];
                             if (firstSession.key) {
                                 currentSessionKey = firstSession.key;
-                                console.log("flawed-avatar: auto-detected session from sessions.list:", currentSessionKey, "displayName:", firstSession.displayName);
                             }
                         }
                     }
@@ -100,13 +113,11 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
                             const agentId = firstAgent.id ?? "main";
                             const sessionKey = `agent:${agentId}:main`;
                             currentSessionKey = sessionKey;
-                            console.log("flawed-avatar: fallback to agent main session:", currentSessionKey);
                         }
                     }
                     // Connect success - request active sessions once
                     if (connectSent && !connectionSetupDone && !sessionsListRequestId && !agentsListRequestId) {
                         connectionSetupDone = true;
-                        console.log("flawed-avatar: gateway connected successfully");
                         backoffMs = GATEWAY_RECONNECT_BASE_MS;
                         // Request recently active sessions
                         requestSessionsList();
@@ -136,6 +147,35 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
             clearTimeout(connectTimer);
             connectTimer = null;
         }
+        const role = "operator";
+        const scopes = ["operator.admin"];
+        const storedToken = deviceIdentity ? loadStoredAuthToken(deviceIdentity.deviceId, role) : null;
+        const effectiveToken = storedToken ?? authToken ?? undefined;
+        const auth = effectiveToken ? { token: effectiveToken } : undefined;
+        const nonce = connectNonce ?? undefined;
+        const signedAtMs = Date.now();
+        const device = (() => {
+            if (!deviceIdentity)
+                return undefined;
+            const payload = buildAuthPayload({
+                deviceId: deviceIdentity.deviceId,
+                clientId: "gateway-client",
+                clientMode: "backend",
+                role,
+                scopes,
+                signedAtMs,
+                token: effectiveToken ?? null,
+                nonce,
+            });
+            const signature = signPayload(deviceIdentity.privateKeyPem, payload);
+            return {
+                id: deviceIdentity.deviceId,
+                publicKey: publicKeyToBase64Url(deviceIdentity.publicKeyPem),
+                signature,
+                signedAt: signedAtMs,
+                nonce,
+            };
+        })();
         const frame = {
             type: "req",
             id: randomUUID(),
@@ -146,14 +186,15 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
                 client: {
                     id: "gateway-client",
                     displayName: "Flawed Avatar",
-                    version: "0.1.0",
+                    version: PKG_VERSION,
                     platform: process.platform,
                     mode: "backend",
                 },
                 caps: [],
-                role: "operator",
-                scopes: ["operator.admin"],
-                auth: authToken ? { token: authToken } : {},
+                role,
+                scopes,
+                auth,
+                device,
             },
         };
         ws.send(JSON.stringify(frame));
@@ -182,7 +223,6 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
                 limit: 10,
             },
         };
-        console.log("flawed-avatar: requesting sessions list");
         ws.send(JSON.stringify(frame));
     }
     function requestAgentsList() {
@@ -195,7 +235,6 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
             method: "agents.list",
             params: {},
         };
-        console.log("flawed-avatar: requesting agents list (fallback)");
         ws.send(JSON.stringify(frame));
     }
     function connect() {
@@ -203,7 +242,6 @@ export function createGatewayClient(gatewayUrl, onStateChange, onModelSwitch, ag
             return;
         ws = new WebSocket(gatewayUrl, { maxPayload: 25 * 1024 * 1024 });
         ws.on("open", () => {
-            console.log("flawed-avatar: ws open, sending connect frame");
             queueConnect();
         });
         ws.on("message", (data) => {

package/dist/main/main/main.js

@@ -7,6 +7,7 @@ import { createWindowManager } from "./window-manager.js";
 import { createTray } from "./tray.js";
 import { createStdinListener } from "./stdin-listener.js";
 import { createGatewayClient } from "./gateway-client.js";
+import { loadDeviceIdentity } from "./device-identity.js";
 import { IPC } from "../shared/ipc-channels.js";
 import { GATEWAY_URL_DEFAULT, CHAT_INPUT_MAX_LENGTH } from "../shared/config.js";
 import { getVrmModelPath } from "./persistence/index.js";
@@ -75,7 +76,7 @@ app.whenReady().then(() => {
     const wm = createWindowManager();
     createTray(wm);
     // Return VRM model path (CLI override > persisted > default)
-    const defaultVrmPath = path.join(__dirname, "..", "..", "..", "assets", "models", "default-avatar.vrm");
+    const defaultVrmPath = path.join(__dirname, "..", "..", "..", "assets", "models", "CaptainLobster.vrm");
     ipcMain.handle(IPC.GET_VRM_PATH, () => {
         if (cliVrmPath)
             return cliVrmPath;
@@ -130,18 +131,15 @@ app.whenReady().then(() => {
     // Connect to gateway WebSocket for agent event streaming
     const gatewayUrl = cliGatewayUrl ?? GATEWAY_URL_DEFAULT;
     const authToken = resolveAuthToken();
-    console.log(`flawed-avatar: connecting to ${gatewayUrl} (auth=${authToken ? "token" : "none"})`);
-    const gw = createGatewayClient(gatewayUrl, (state) => wm.sendAgentState(state), (vrmPath) => wm.sendToAvatar(IPC.VRM_MODEL_CHANGED, vrmPath), agentConfigs, authToken);
+    const deviceIdentity = loadDeviceIdentity();
+    console.log(`flawed-avatar: connecting to ${gatewayUrl} (auth=${authToken ? "token" : "none"}, device=${deviceIdentity ? deviceIdentity.deviceId.slice(0, 8) + "…" : "none"})`);
+    const gw = createGatewayClient(gatewayUrl, (state) => wm.sendAgentState(state), (vrmPath) => wm.sendToAvatar(IPC.VRM_MODEL_CHANGED, vrmPath), agentConfigs, authToken, deviceIdentity);
     // IPC: send chat message to active agent
     ipcMain.on(IPC.SEND_CHAT, (_event, text) => {
-        console.log("flawed-avatar: SEND_CHAT received:", text);
         if (typeof text !== "string" || text.trim().length === 0 || text.length > CHAT_INPUT_MAX_LENGTH) {
-            console.log("flawed-avatar: SEND_CHAT rejected (validation failed)");
             return;
         }
         const agentId = gw.getCurrentAgentId();
-        console.log("flawed-avatar: current agentId:", agentId);
-        console.log("flawed-avatar: sending chat to gateway");
         gw.sendChat(text.trim(), agentId);
     });
     // Clean up resources on quit

package/dist/main/main/persistence/types.d.ts

@@ -38,8 +38,8 @@ export declare const ChatMessageSchema: z.ZodObject<{
     id: z.ZodString;
     timestamp: z.ZodNumber;
     role: z.ZodEnum<{
-        user: "user";
         assistant: "assistant";
+        user: "user";
     }>;
     text: z.ZodString;
     agentId: z.ZodOptional<z.ZodString>;
@@ -50,8 +50,8 @@ export declare const ChatHistorySchema: z.ZodObject<{
         id: z.ZodString;
         timestamp: z.ZodNumber;
         role: z.ZodEnum<{
-            user: "user";
             assistant: "assistant";
+            user: "user";
         }>;
         text: z.ZodString;
         agentId: z.ZodOptional<z.ZodString>;