zrb-face 0.1.0__tar.gz

zrb_face-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+app/node_modules/
+app/dist/
+dist/
+build/
+*.egg-info/
+*.vrm

zrb_face-0.1.0/PKG-INFO

@@ -0,0 +1,279 @@
+Metadata-Version: 2.5
+Name: zrb-face
+Version: 0.1.0
+Summary: A 3D character for zrb in a small, see-through, always-on-top window
+Project-URL: Homepage, https://github.com/state-alchemists/zrb-face
+Requires-Python: >=3.10
+Provides-Extra: zrb
+Requires-Dist: zrb; extra == 'zrb'
+Description-Content-Type: text/markdown
+
+# zrb-face
+
+A companion for [zrb](https://github.com/state-alchemists/zrb): a 3D character in a small, see-through, always-on-top window. She idles naturally (breathing, shifting her weight, glancing around, stretching), shows what zrb is doing and how it feels, moves her mouth with zrb's voice, dances while you play music, gestures with your own animations, and can start a conversation herself.
+
+## Quick start
+
+```sh
+pipx install zrb-face          # or: pip install zrb-face, next to zrb; from a checkout: pip install -e .
+
+# once, e.g. in ~/.zshrc: load the zrb side into every zrb chat
+export ZRB_INIT_SCRIPTS="$ZRB_INIT_SCRIPTS:$(zrb-face init-path)"
+
+mkdir -p ~/.config/zrb-face/models && cp my.vrm ~/.config/zrb-face/models/   # a model: see Changing the model
+zrb-face my                    # (re)start the server, open a window with my.vrm
+zrb chat
+```
+
+The window is a small desktop app; download it from the [releases](https://github.com/state-alchemists/zrb-face/releases) or build it (see [The desktop app](#the-desktop-app)). Without it, `zrb-face` opens the page in your browser instead.
+
+`zrb-face` opens the Windows build from WSL, the installed app on macOS, or Electron from `app/` in a source checkout (`ZRB_FACE_APP` overrides the program). It restarts the server each time, so the server always runs the current code with your current settings; open windows reload by themselves. The server keeps running across chats; its output goes to `zrb-face.log` in the temp folder, and `pkill -f "[z]rb_face/server.py"` stops it. The order doesn't matter: the window waits for the server, reloads itself when the server restarts, and zrb skips sending while either is missing. If a model can't be loaded, the window says so in red and tries again every few seconds.
+
+## How it fits together
+
+```
+ zrb chat  ── POST /event ──▶  zrb-face server  ── page, models, animations, live events ──▶  window (face.html)
+ + zrb_init.py                 (port 7777)                                                    in the app or a browser
+```
+
+| File | Role |
+| --- | --- |
+| `zrb_face/zrb_init.py` | The zrb side, loaded through `ZRB_INIT_SCRIPTS`: zrb's speech (moving the mouth, silent for "nothing to say"), the face hooks, the `FaceAction` tool, and the optional triggers that let her speak first. |
+| `zrb_face/client.py` | Sends events to the server: state and emotion from zrb's hooks, mouth openness from the loudness of the speech audio. Skips sending when the server is down. |
+| `zrb_face/server.py` | The server (Python standard library only): serves the page, the models and the animations from their folders, and relays every event to the open windows over Server-Sent Events. |
+| `zrb_face/face.html` | Everything visible, with three.js and three-vrm: loads the model, plays the animations, blinks, lip sync, eye contact, smiles, emotions, the camera. |
+| `zrb_face/__main__.py` | The `zrb-face` command. |
+| `zrb_face/anim/` | The animations that ship with the package. |
+| `app/` | The desktop window: an Electron shell (`main.js`) that shows the page see-through and always on top. |
+| `make_idle.py` | Generates the idle and dance animations in `zrb_face/anim/`. |
+| `test_face.py` | The self-check. |
+
+zrb runs in WSL on Windows, but a window that stays on top has to be a Windows program. Windows can reach a server in WSL (not the other way around in WSL's default NAT mode), so the server runs next to zrb and the window connects to it.
+
+## Running by hand
+
+What `zrb-face` does, per platform:
+
+```sh
+zrb-face serve &                                                                          # the server
+/mnt/c/Users/$USER/AppData/Local/zrb-face/zrb-face.exe rem </dev/null &>/dev/null &!      # Windows, from WSL
+open -n -a zrb-face --args rem                                                            # macOS, installed app
+~/zrb-face/app/node_modules/.bin/electron ~/zrb-face/app rem                              # Linux, or any OS from source
+```
+
+The window takes a model name (`rem` or `rem.vrm`; without one, the first model alphabetically) and optionally another server URL (`http://localhost:7779`). Start the Windows build detached as above, or zsh stops it ("suspended (tty input)"). Instead of the app you can open `http://localhost:7777/?model=rem` in a browser: the same page, but not see-through or always on top.
+
+## Using the window
+
+- **Move**: drag it. **Close**: Alt+F4 on Windows, Cmd+Q on macOS.
+- **Camera angle**: click the window, then use the keys in [Keyboard](#keyboard-click-the-window-first). The angle is remembered.
+
+## What she shows
+
+| Glow | State | Sent by |
+| --- | --- | --- |
+| cyan | idle | `Stop`, `StopFailure`, `SessionStart` |
+| violet (head tilts) | thinking | `UserPromptSubmit`, `PostToolUse`, `PostToolUseFailure` |
+| green | running a tool | `PreToolUse` |
+| red | needs attention | `Notification`, `PermissionRequest` |
+| bright cyan | speaking | mouth events |
+
+- **Voice**: the mouth follows the loudness of the speech, in 40 ms steps, and she nods slightly while speaking. It needs a voice zrb plays itself (piper, openai, gemini); `say` and `termux` play the audio themselves, so zrb never sees it. The mouth starts slightly ahead of the voice, because zrb reads audio ahead of playing it; fixing that exactly needs a frame observer in zrb's `PcmUtterance._fill`.
+- **Face**: she blinks every few seconds, her eyes follow you (the camera), and she smiles softly every 25–50 seconds and after finishing a task.
+- **Body**: each state loops its own animations, falling back to the idles; see [Animations](#animations). On `SessionStart` she plays the `greeting` action if there is one.
+- **Emotion**: with zrb's emotion on (`ZRB_LLM_EMOTION_ENABLED=true`), zrb opens each reply with an emoji, and she shows it for 6 s with the matching VRM expression (`happy`, `sad`, `angry`, `surprised`, `relaxed`), and plays the action of the same name if there is one (`actions/happy.vrma`). The user's own emotion isn't shown: her reply answers it.
+- **Gestures**: with animations in `actions/`, zrb gets a `FaceAction` tool and plays them itself, alongside its words (wave, nod, bow, …).
+- **Music**: when something plays music (a browser, Spotify, …) she dances with the `dance/` animations while idle; anything zrb does takes over, and she goes back to dancing after. The server watches the system's media status: Windows' media controls (also from WSL) or `playerctl` on Linux. Any playing media counts, a talk video too. On macOS, or to override it, use `zrb-face music on` / `off`; `ZRB_FACE_MUSIC=off` stops the watching.
+- **Talking first** (off by default): with `ZRB_FACE_TALK_FIRST=idle` she speaks up after you've been quiet for a while during waking hours (8:00–22:00); with `camera`, when you sit back down after being away. She never interrupts a running turn or her own speech, waits a cooldown between such turns, and replies `…` (shown, not spoken) when there's nothing worth saying.
+
+The mouth, the blink and the smile need the model's `aa`, `blink` and `relaxed` expressions. A model without them still nods while speaking.
+
+## Wiring it into zrb
+
+`zrb_face/zrb_init.py` is all zrb needs; load it through `ZRB_INIT_SCRIPTS` as in [Quick start](#quick-start) (`zrb-face init-path` prints where it is). It owns zrb's speech (`enable_speech`), so other init scripts should only choose the voice (`CFG.DEFAULT_LLM_SPEECH_BACKEND`, `CFG.DEFAULT_LLM_SPEECH_VOICE`) and not call `enable_speech` themselves; whichever script calls it last would win.
+
+## Configuration
+
+### Environment variables
+
+The zrb-side names follow zrb's `CFG.ENV_PREFIX` (`ZRB` by default). The server and the `zrb-face` command run without zrb, so their names are fixed.
+
+| Variable | Default | Read by | What it does |
+| --- | --- | --- | --- |
+| `ZRB_FACE_TALK_FIRST` | empty (never) | `zrb_init.py` | What may start a conversation: `idle`, `camera`, or `idle,camera`. |
+| `ZRB_FACE_IDLE_MINUTES` | `45` | `zrb_init.py` | Quiet minutes before `idle` speaks. |
+| `ZRB_FACE_COOLDOWN` | `1200` | `zrb_init.py` | Seconds between conversations she starts herself. |
+| `ZRB_FACE_CAMERA_INTERVAL` | `20` | `zrb_init.py` | Seconds between camera looks; each one lights the camera. |
+| `ZRB_FACE_CAMERA_PRESENT_BYTES` | `25000` | `zrb_init.py` | Photo size above which someone counts as present; tune it for your camera and room. |
+| `ZRB_FACE_MOUTH_GAIN` | `4` | `client.py` | How wide the mouth opens for a given loudness: raise it if the mouth barely opens, lower it if it stays wide open. |
+| `ZRB_FACE_URL` | `http://127.0.0.1:7777/event` | `client.py` | Where zrb sends events. |
+| `ZRB_FACE_HOST` | `127.0.0.1` | server | The address the server listens on. |
+| `ZRB_FACE_PORT` | `7777` | server, `zrb-face` | The server's port (change `ZRB_FACE_URL` to match). |
+| `ZRB_FACE_HOME` | `~/.config/zrb-face` | server, `zrb_init.py` | Your folder: models in `models/`, animations in `anim/`. |
+| `ZRB_FACE_MODELS` | empty | server | More model folders, searched before `<home>/models`; separated by `:` (`;` on Windows). `~` is expanded. |
+| `ZRB_FACE_ANIMS` | empty | server, `zrb_init.py` | More animation folders, searched before `<home>/anim` and the shipped ones. |
+| `ZRB_FACE_MUSIC` | `auto` | server | `auto` dances while media plays; `off` only on `zrb-face music on`. |
+| `ZRB_FACE_APP` | per platform | `zrb-face` | The program to open as the window, instead of the Windows build, the macOS app or Electron from source. |
+
+### zrb settings it depends on
+
+| Setting | Why it matters |
+| --- | --- |
+| `ZRB_INIT_SCRIPTS` | Must include `$(zrb-face init-path)`; that's what turns the face on. |
+| `ZRB_LLM_EMOTION_ENABLED` | `true` makes zrb tag replies with an emotion, which she shows. Off by default. Needs a zrb with the `Emotion` hook. |
+| `ZRB_LLM_SPEECH_BACKEND` / `ZRB_LLM_SPEECH_VOICE` (or `CFG.DEFAULT_LLM_SPEECH_BACKEND` / `_VOICE` in an init script) | The voice. For the mouth to move it must be one zrb plays itself (piper, openai, gemini), not `say` or `termux`. |
+| `ZRB_LLM_SPEECH_PLAYER` | Leave it at `auto` (the default), with zrb's voice extra installed (`zrb[voice]`, for sounddevice): then zrb plays the audio itself and the mouth can follow it. `command` hands the audio to a player program, hidden from zrb. |
+| `ZRB_HOOKS_ENABLED` | Must stay on (the default): the states and speech ride on zrb's hooks. |
+| zrb's camera settings (`ZRB_LLM_CAMERA_*`) | Used by `ZRB_FACE_TALK_FIRST=camera` to take its photos. |
+
+### Command line and URL
+
+| Where | Option | Example | What it does |
+| --- | --- | --- | --- |
+| `zrb-face` | model name | `zrb-face rem` | (Re)starts the server and opens a window with that model; without a name, the first model alphabetically. |
+| `zrb-face serve` | | | Runs the server in the foreground. |
+| `zrb-face action <name>` | | `zrb-face action wave` | Plays `actions/<name>` once; handy for trying a new clip. |
+| `zrb-face music on\|off` | | `zrb-face music on` | Dance while idle, or stop. |
+| `zrb-face dirs` | | | Lists the model and animation folders in search order, and what they hold. |
+| `zrb-face init-path` | | | Prints the path of `zrb_init.py`, for `ZRB_INIT_SCRIPTS`. |
+| `zrb-face.exe`, the macOS app, `electron app` | model name | `zrb-face.exe rem` or `rem.vrm` | The model to show (letters, digits, `-`, `_`; `.vrm` optional). |
+| same | server URL | `zrb-face.exe http://localhost:7779` | Another server than `http://localhost:7777`. |
+| the page | `?model=<name>` | `http://localhost:7777/?model=rem` | The model, in a browser. |
+| the page | `?app` | (set by the app) | See-through background, dragging moves the window, no status text. |
+
+### Keyboard (click the window first)
+
+| Key | What it does |
+| --- | --- |
+| ← → | Orbit around her. |
+| ↑ ↓ | Look from higher or lower. |
+| `+` / `-` | Zoom in or out (0.5× to 4×). |
+| `0` | Reset the camera. |
+
+The camera angle is saved in the window's storage and comes back next time.
+
+### Files and folders
+
+Models and animations are looked up in folders, in this order; the first file with a given name wins, so your own files override the shipped ones. `zrb-face dirs` shows the list.
+
+| Kind | Folders |
+| --- | --- |
+| models | `$ZRB_FACE_MODELS`, `~/.config/zrb-face/models` |
+| animations | `$ZRB_FACE_ANIMS`, `~/.config/zrb-face/anim`, the package's `anim/` |
+
+| Path | Convention |
+| --- | --- |
+| `<models folder>/<name>.vrm` | A model, opened by `<name>`. Only files directly in the folder are served. |
+| `<anim folder>/<pool>/<name>.vrma` | An animation; the pool (a subfolder) says when it plays. See [Animations](#animations). |
+
+### In the code
+
+Constants with no setting of their own; edit the file to change them.
+
+| File | Constant | Value | What it does |
+| --- | --- | --- | --- |
+| `face.html` | `FRAMING` | `'body'` | `'body'` frames her head to toe (with room for the reach of every looping clip), `'face'` head and shoulders. |
+| `face.html` | `LOOK` | cyan, violet, green, red, bright cyan | The glow colour per state: idle, thinking, tool, attention, speaking. |
+| `face.html` | blinking | every 2–6 s, 0.12 s long | How often and how long she blinks. |
+| `face.html` | smile | every 25–50 s for 3 s at 0.5; 2 s after a task | The soft `relaxed` smile. |
+| `face.html` | thinking tilt / speaking nod | 0.12 / 0.04 rad | How far the head tilts while thinking and nods with the voice. |
+| `face.html` | crossfade | 0.6 s (actions 0.3 s) | Blend time between animations. |
+| `client.py` | `MOUTH_STEP` | 0.04 s | Audio per mouth value. |
+| `zrb_init.py` | `AWAKE_HOURS` | 8:00–21:59 | When she may speak first. |
+| `zrb_init.py` | idle check / away time | every 60 s / 5 min | How often `idle` checks; how long you must be away before `camera` greets you back. |
+| `zrb_init.py` | `NOTHING_TO_SAY` | `…` | The reply she gives, unspoken, when nothing is worth saying. |
+| `app/main.js` | window | 440×640, bottom right | Size and place of the window (320×420 suits `'face'` framing). |
+| `app/main.js` | on top | `screen-saver` level | Stays above full-screen windows too. |
+| `face.html` | emotion | 0.6 for 6 s | How strong and how long she shows an emotion. |
+| `make_idle.py` | `CLIPS`, `DANCES` | 8 idles, 4 dances | The animations as eased keyframes of poses; run the script to regenerate `zrb_face/anim/idle/` and `dance/`. |
+| `make_idle.py` | `FPS` | 15 | Keyframes per second in the generated clips. |
+
+## Changing the model
+
+Put any VRM file (VRM 0.x or 1.0) in `~/.config/zrb-face/models/` (or a folder in `ZRB_FACE_MODELS`) as `<name>.vrm` and open it by name: `zrb-face <name>`. Without a name, the window shows the first model alphabetically. No model ships with the package. Model files are kept out of git (`*.vrm` is ignored). Nothing in the code depends on the model: the camera fits it, and the pose and idle come from the animations, which three-vrm retargets onto any VRM, as VRoid Hub's playground does.
+
+Where to get models:
+
+- **[VRoid Studio](https://vroid.com/en/studio)**: free, for Windows and macOS. Make your own and export it as VRM. You own it, and it has every expression.
+- **[VRoid Hub](https://hub.vroid.com/en)**: thousands of models. Only the ones marked downloadable can be downloaded, and each has its own conditions (avatar use, commercial use, redistribution, changes). Read them before you change or share the model.
+- **[BOOTH](https://booth.pm/en/search/VRM)**: search for `VRM` and filter for free items. The terms are on each item's page, often only in Japanese.
+- **[Open Source Avatars](https://github.com/toxsam/open-source-avatars)**: free VRM avatars with direct download links. The license is set per collection (CC0, CC-BY, …), so check it.
+- **[OpenGameArt: VRoid Studio CC0 models](https://opengameart.org/content/vroid-studio-cc0-models)**: public domain sample models from early VRoid Studio.
+
+Fan models of existing characters (Cortana, Kusanagi, and others) are fine on your own desktop, but don't commit or share them.
+
+## Animations
+
+Every `<pool>/<name>.vrma` (VRM Animation) in the animation folders loads when the page starts. The pool, the subfolder, says when it plays:
+
+| Pool | Plays |
+| --- | --- |
+| `idle/` | Loops while idle, and for every state with no pool of its own. |
+| `thinking/`, `tool/`, `attention/`, `speaking/` | Loops in that state. None ship; add your own. |
+| `dance/` | Loops instead of `idle/` while music plays. |
+| `actions/` | Once, on `{"type": "action", "value": "<name>"}`, then back to looping. zrb plays them through its `FaceAction` tool; `greeting` plays when a chat starts, and `happy`, `sad`, … with that emotion. |
+
+Any other pool loads too, and plays once something asks for it.
+
+To add your own, drop the file in, e.g. `~/.config/zrb-face/anim/actions/wave.vrma`, and restart with `zrb-face` (and zrb, so the `FaceAction` tool knows the new name). A file named like a shipped one replaces it: `~/.config/zrb-face/anim/idle/rest.vrma`. Try an action with `zrb-face action wave`.
+
+Within a pool, clips play in random order with crossfades, picked by group: the first part of the name (`stretch-side` belongs to `stretch`). A random group, then a random clip in it, so the stretches together come up as often as any single other idle. On top of whatever plays: the head tilts while thinking, nods while speaking, the eyes follow you, and the face shows zrb's emotion.
+
+The idles and dances are ours (CC0), generated by `python3 make_idle.py`. The dances are 8-beat loops at 120 BPM: `bounce` (down on every beat, arms pumping), `sway` (side to side every two beats), `step-touch` (stepping side to side, hands up on the touch) and `hands-up` (arms high, waving). They don't follow the song's tempo yet. The idles:
+
+| Clip | What she does |
+| --- | --- |
+| `rest` | Relaxed, a slow weight shift. |
+| `hands` | Hands clasped in front, swaying; the free foot twists on its toes. |
+| `weight` | Shifts from hip to hip with the free knee bent, taking a small step each time. |
+| `look` | Glances left and right. |
+| `stretch` | The yawn-stretch: hands rise past the face, then up and out into a Y, shoulders shrugged, back arched, up on the toes, eyes closed, yawning; then a slow release, shoulders dropping. |
+| `stretch-side` | One arm arcs over her head, the other hand slides down the thigh, the hips shift the other way; each side. |
+| `stretch-neck` | Ear to each shoulder, then chin down; shoulders dropped, eyes closed. |
+| `stretch-twist` | Turns from the hips up, the head leading, the arms swinging loosely, the back heel lifting. |
+
+Every clip also has breathing, uneven head drift, knees that flex a little, and fingers that relax and tighten. The hands are loosely curled, the arms hang a little away from the body, and the weight shifts move the hips over the standing leg. The motion follows VRoid's own VRMA clips (every joint moving, fingers included), common VTuber idle loops (breathing, sway, head pitch, roll and yaw), and how people actually stretch (the whole-body yawn-stretch; a side bend with the hips shifting the other way).
+
+To change or add idles, edit `CLIPS` in `make_idle.py` (eased keyframes of poses) and run it again. Its docstring lists which way each joint turns. Poses can also move the hips (`hipsMove`) and set the `blink` and `aa` expressions; the page takes the larger of the clip's value and its own blinking and lip sync.
+
+More free VRMA:
+
+- **VRoid's official 7-motion pack** (show full body, greeting, peace sign, shoot, spin, model pose, squat), free on [BOOTH](https://vroid.com/en/news/6HozzBIV0KkcKf9dc1fZGW). Free to use, commercial use with the credit "Animation credits to pixiv Inc.'s VRoid Project", but its terms forbid redistributing the files, so keep them out of git. E.g. `cp VRMA_02.vrma ~/.config/zrb-face/anim/actions/greeting.vrma`.
+- **Dances**: motion capture through bvh2vrma, e.g. the [AIST++](https://google.github.io/aistplusplus_dataset/) dance dataset (CC BY 4.0, credit needed) or the [CMU](http://mocap.cs.cmu.edu/) library (free). Mixamo's dances can be converted too, but its terms forbid sharing the files, so keep them in your own folder.
+- **Your own**: [bvh2vrma](https://vrm-c.github.io/bvh2vrma/) (VRM Consortium, MIT) converts BVH motion capture to VRMA, and VRoid Studio exports poses and animations as VRMA.
+
+## The desktop app
+
+Each [release](https://github.com/state-alchemists/zrb-face/releases) has a zip per platform:
+
+- **Windows** (`zrb-face-win32-x64.zip`): unzip into `%LOCALAPPDATA%\zrb-face\`, so `zrb-face` finds `zrb-face.exe` there from WSL.
+- **macOS** (`zrb-face-darwin-arm64.zip` for Apple silicon, `-x64` for Intel): move `zrb-face.app` to `/Applications`. The app isn't signed, so clear macOS's block once with `xattr -dr com.apple.quarantine /Applications/zrb-face.app`.
+- **Linux** (`zrb-face-linux-x64.zip`): unzip anywhere and set `ZRB_FACE_APP` to its `zrb-face` program. On Wayland (GNOME), a window can't keep itself on top, so pin it by hand.
+
+To build it yourself:
+
+```sh
+cd app
+npm install
+npm start                                   # run from source
+npm run pack:win                            # or pack:mac / pack:linux -> dist/
+cp -r dist/zrb-face-win32-x64 /mnt/c/Users/$USER/AppData/Local/zrb-face
+```
+
+If GitHub downloads are slow, prefix the build with `ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/`, then check the zip in `~/.cache/electron` against GitHub's `SHASUMS256.txt`.
+
+## Test
+
+```sh
+python3 test_face.py                        # with a Python that has zrb installed
+```
+
+## Releasing
+
+Bump the version in `pyproject.toml` and `app/package.json`, then push a tag: `git tag 0.1.0 && git push --tags`. `.github/workflows/release.yml` publishes the package to PyPI and attaches the app zips to the GitHub release. PyPI publishing is trusted publishing, so first add this repository, the workflow `release.yml` and the environment `pypi` as a trusted publisher of `zrb-face` on pypi.org.
+
+## Credits
+
+`zrb_face/anim/` and `make_idle.py`: ours, CC0.
+
+The models used during development, not included: *ram* and *rem made in mexico* (Re:Zero fan models) by pon3mx on [VRoid Hub](https://hub.vroid.com/en); search for them there. Ram: use, changes, commercial use and redistribution allowed, no credit needed. Rem: the same but no commercial use. Both have mouth, blink and emotion expressions. They are fan models of the Re:Zero characters, so keep them on your own machine.

zrb_face-0.1.0/README.md

@@ -0,0 +1,269 @@
+# zrb-face
+
+A companion for [zrb](https://github.com/state-alchemists/zrb): a 3D character in a small, see-through, always-on-top window. She idles naturally (breathing, shifting her weight, glancing around, stretching), shows what zrb is doing and how it feels, moves her mouth with zrb's voice, dances while you play music, gestures with your own animations, and can start a conversation herself.
+
+## Quick start
+
+```sh
+pipx install zrb-face          # or: pip install zrb-face, next to zrb; from a checkout: pip install -e .
+
+# once, e.g. in ~/.zshrc: load the zrb side into every zrb chat
+export ZRB_INIT_SCRIPTS="$ZRB_INIT_SCRIPTS:$(zrb-face init-path)"
+
+mkdir -p ~/.config/zrb-face/models && cp my.vrm ~/.config/zrb-face/models/   # a model: see Changing the model
+zrb-face my                    # (re)start the server, open a window with my.vrm
+zrb chat
+```
+
+The window is a small desktop app; download it from the [releases](https://github.com/state-alchemists/zrb-face/releases) or build it (see [The desktop app](#the-desktop-app)). Without it, `zrb-face` opens the page in your browser instead.
+
+`zrb-face` opens the Windows build from WSL, the installed app on macOS, or Electron from `app/` in a source checkout (`ZRB_FACE_APP` overrides the program). It restarts the server each time, so the server always runs the current code with your current settings; open windows reload by themselves. The server keeps running across chats; its output goes to `zrb-face.log` in the temp folder, and `pkill -f "[z]rb_face/server.py"` stops it. The order doesn't matter: the window waits for the server, reloads itself when the server restarts, and zrb skips sending while either is missing. If a model can't be loaded, the window says so in red and tries again every few seconds.
+
+## How it fits together
+
+```
+ zrb chat  ── POST /event ──▶  zrb-face server  ── page, models, animations, live events ──▶  window (face.html)
+ + zrb_init.py                 (port 7777)                                                    in the app or a browser
+```
+
+| File | Role |
+| --- | --- |
+| `zrb_face/zrb_init.py` | The zrb side, loaded through `ZRB_INIT_SCRIPTS`: zrb's speech (moving the mouth, silent for "nothing to say"), the face hooks, the `FaceAction` tool, and the optional triggers that let her speak first. |
+| `zrb_face/client.py` | Sends events to the server: state and emotion from zrb's hooks, mouth openness from the loudness of the speech audio. Skips sending when the server is down. |
+| `zrb_face/server.py` | The server (Python standard library only): serves the page, the models and the animations from their folders, and relays every event to the open windows over Server-Sent Events. |
+| `zrb_face/face.html` | Everything visible, with three.js and three-vrm: loads the model, plays the animations, blinks, lip sync, eye contact, smiles, emotions, the camera. |
+| `zrb_face/__main__.py` | The `zrb-face` command. |
+| `zrb_face/anim/` | The animations that ship with the package. |
+| `app/` | The desktop window: an Electron shell (`main.js`) that shows the page see-through and always on top. |
+| `make_idle.py` | Generates the idle and dance animations in `zrb_face/anim/`. |
+| `test_face.py` | The self-check. |
+
+zrb runs in WSL on Windows, but a window that stays on top has to be a Windows program. Windows can reach a server in WSL (not the other way around in WSL's default NAT mode), so the server runs next to zrb and the window connects to it.
+
+## Running by hand
+
+What `zrb-face` does, per platform:
+
+```sh
+zrb-face serve &                                                                          # the server
+/mnt/c/Users/$USER/AppData/Local/zrb-face/zrb-face.exe rem </dev/null &>/dev/null &!      # Windows, from WSL
+open -n -a zrb-face --args rem                                                            # macOS, installed app
+~/zrb-face/app/node_modules/.bin/electron ~/zrb-face/app rem                              # Linux, or any OS from source
+```
+
+The window takes a model name (`rem` or `rem.vrm`; without one, the first model alphabetically) and optionally another server URL (`http://localhost:7779`). Start the Windows build detached as above, or zsh stops it ("suspended (tty input)"). Instead of the app you can open `http://localhost:7777/?model=rem` in a browser: the same page, but not see-through or always on top.
+
+## Using the window
+
+- **Move**: drag it. **Close**: Alt+F4 on Windows, Cmd+Q on macOS.
+- **Camera angle**: click the window, then use the keys in [Keyboard](#keyboard-click-the-window-first). The angle is remembered.
+
+## What she shows
+
+| Glow | State | Sent by |
+| --- | --- | --- |
+| cyan | idle | `Stop`, `StopFailure`, `SessionStart` |
+| violet (head tilts) | thinking | `UserPromptSubmit`, `PostToolUse`, `PostToolUseFailure` |
+| green | running a tool | `PreToolUse` |
+| red | needs attention | `Notification`, `PermissionRequest` |
+| bright cyan | speaking | mouth events |
+
+- **Voice**: the mouth follows the loudness of the speech, in 40 ms steps, and she nods slightly while speaking. It needs a voice zrb plays itself (piper, openai, gemini); `say` and `termux` play the audio themselves, so zrb never sees it. The mouth starts slightly ahead of the voice, because zrb reads audio ahead of playing it; fixing that exactly needs a frame observer in zrb's `PcmUtterance._fill`.
+- **Face**: she blinks every few seconds, her eyes follow you (the camera), and she smiles softly every 25–50 seconds and after finishing a task.
+- **Body**: each state loops its own animations, falling back to the idles; see [Animations](#animations). On `SessionStart` she plays the `greeting` action if there is one.
+- **Emotion**: with zrb's emotion on (`ZRB_LLM_EMOTION_ENABLED=true`), zrb opens each reply with an emoji, and she shows it for 6 s with the matching VRM expression (`happy`, `sad`, `angry`, `surprised`, `relaxed`), and plays the action of the same name if there is one (`actions/happy.vrma`). The user's own emotion isn't shown: her reply answers it.
+- **Gestures**: with animations in `actions/`, zrb gets a `FaceAction` tool and plays them itself, alongside its words (wave, nod, bow, …).
+- **Music**: when something plays music (a browser, Spotify, …) she dances with the `dance/` animations while idle; anything zrb does takes over, and she goes back to dancing after. The server watches the system's media status: Windows' media controls (also from WSL) or `playerctl` on Linux. Any playing media counts, a talk video too. On macOS, or to override it, use `zrb-face music on` / `off`; `ZRB_FACE_MUSIC=off` stops the watching.
+- **Talking first** (off by default): with `ZRB_FACE_TALK_FIRST=idle` she speaks up after you've been quiet for a while during waking hours (8:00–22:00); with `camera`, when you sit back down after being away. She never interrupts a running turn or her own speech, waits a cooldown between such turns, and replies `…` (shown, not spoken) when there's nothing worth saying.
+
+The mouth, the blink and the smile need the model's `aa`, `blink` and `relaxed` expressions. A model without them still nods while speaking.
+
+## Wiring it into zrb
+
+`zrb_face/zrb_init.py` is all zrb needs; load it through `ZRB_INIT_SCRIPTS` as in [Quick start](#quick-start) (`zrb-face init-path` prints where it is). It owns zrb's speech (`enable_speech`), so other init scripts should only choose the voice (`CFG.DEFAULT_LLM_SPEECH_BACKEND`, `CFG.DEFAULT_LLM_SPEECH_VOICE`) and not call `enable_speech` themselves; whichever script calls it last would win.
+
+## Configuration
+
+### Environment variables
+
+The zrb-side names follow zrb's `CFG.ENV_PREFIX` (`ZRB` by default). The server and the `zrb-face` command run without zrb, so their names are fixed.
+
+| Variable | Default | Read by | What it does |
+| --- | --- | --- | --- |
+| `ZRB_FACE_TALK_FIRST` | empty (never) | `zrb_init.py` | What may start a conversation: `idle`, `camera`, or `idle,camera`. |
+| `ZRB_FACE_IDLE_MINUTES` | `45` | `zrb_init.py` | Quiet minutes before `idle` speaks. |
+| `ZRB_FACE_COOLDOWN` | `1200` | `zrb_init.py` | Seconds between conversations she starts herself. |
+| `ZRB_FACE_CAMERA_INTERVAL` | `20` | `zrb_init.py` | Seconds between camera looks; each one lights the camera. |
+| `ZRB_FACE_CAMERA_PRESENT_BYTES` | `25000` | `zrb_init.py` | Photo size above which someone counts as present; tune it for your camera and room. |
+| `ZRB_FACE_MOUTH_GAIN` | `4` | `client.py` | How wide the mouth opens for a given loudness: raise it if the mouth barely opens, lower it if it stays wide open. |
+| `ZRB_FACE_URL` | `http://127.0.0.1:7777/event` | `client.py` | Where zrb sends events. |
+| `ZRB_FACE_HOST` | `127.0.0.1` | server | The address the server listens on. |
+| `ZRB_FACE_PORT` | `7777` | server, `zrb-face` | The server's port (change `ZRB_FACE_URL` to match). |
+| `ZRB_FACE_HOME` | `~/.config/zrb-face` | server, `zrb_init.py` | Your folder: models in `models/`, animations in `anim/`. |
+| `ZRB_FACE_MODELS` | empty | server | More model folders, searched before `<home>/models`; separated by `:` (`;` on Windows). `~` is expanded. |
+| `ZRB_FACE_ANIMS` | empty | server, `zrb_init.py` | More animation folders, searched before `<home>/anim` and the shipped ones. |
+| `ZRB_FACE_MUSIC` | `auto` | server | `auto` dances while media plays; `off` only on `zrb-face music on`. |
+| `ZRB_FACE_APP` | per platform | `zrb-face` | The program to open as the window, instead of the Windows build, the macOS app or Electron from source. |
+
+### zrb settings it depends on
+
+| Setting | Why it matters |
+| --- | --- |
+| `ZRB_INIT_SCRIPTS` | Must include `$(zrb-face init-path)`; that's what turns the face on. |
+| `ZRB_LLM_EMOTION_ENABLED` | `true` makes zrb tag replies with an emotion, which she shows. Off by default. Needs a zrb with the `Emotion` hook. |
+| `ZRB_LLM_SPEECH_BACKEND` / `ZRB_LLM_SPEECH_VOICE` (or `CFG.DEFAULT_LLM_SPEECH_BACKEND` / `_VOICE` in an init script) | The voice. For the mouth to move it must be one zrb plays itself (piper, openai, gemini), not `say` or `termux`. |
+| `ZRB_LLM_SPEECH_PLAYER` | Leave it at `auto` (the default), with zrb's voice extra installed (`zrb[voice]`, for sounddevice): then zrb plays the audio itself and the mouth can follow it. `command` hands the audio to a player program, hidden from zrb. |
+| `ZRB_HOOKS_ENABLED` | Must stay on (the default): the states and speech ride on zrb's hooks. |
+| zrb's camera settings (`ZRB_LLM_CAMERA_*`) | Used by `ZRB_FACE_TALK_FIRST=camera` to take its photos. |
+
+### Command line and URL
+
+| Where | Option | Example | What it does |
+| --- | --- | --- | --- |
+| `zrb-face` | model name | `zrb-face rem` | (Re)starts the server and opens a window with that model; without a name, the first model alphabetically. |
+| `zrb-face serve` | | | Runs the server in the foreground. |
+| `zrb-face action <name>` | | `zrb-face action wave` | Plays `actions/<name>` once; handy for trying a new clip. |
+| `zrb-face music on\|off` | | `zrb-face music on` | Dance while idle, or stop. |
+| `zrb-face dirs` | | | Lists the model and animation folders in search order, and what they hold. |
+| `zrb-face init-path` | | | Prints the path of `zrb_init.py`, for `ZRB_INIT_SCRIPTS`. |
+| `zrb-face.exe`, the macOS app, `electron app` | model name | `zrb-face.exe rem` or `rem.vrm` | The model to show (letters, digits, `-`, `_`; `.vrm` optional). |
+| same | server URL | `zrb-face.exe http://localhost:7779` | Another server than `http://localhost:7777`. |
+| the page | `?model=<name>` | `http://localhost:7777/?model=rem` | The model, in a browser. |
+| the page | `?app` | (set by the app) | See-through background, dragging moves the window, no status text. |
+
+### Keyboard (click the window first)
+
+| Key | What it does |
+| --- | --- |
+| ← → | Orbit around her. |
+| ↑ ↓ | Look from higher or lower. |
+| `+` / `-` | Zoom in or out (0.5× to 4×). |
+| `0` | Reset the camera. |
+
+The camera angle is saved in the window's storage and comes back next time.
+
+### Files and folders
+
+Models and animations are looked up in folders, in this order; the first file with a given name wins, so your own files override the shipped ones. `zrb-face dirs` shows the list.
+
+| Kind | Folders |
+| --- | --- |
+| models | `$ZRB_FACE_MODELS`, `~/.config/zrb-face/models` |
+| animations | `$ZRB_FACE_ANIMS`, `~/.config/zrb-face/anim`, the package's `anim/` |
+
+| Path | Convention |
+| --- | --- |
+| `<models folder>/<name>.vrm` | A model, opened by `<name>`. Only files directly in the folder are served. |
+| `<anim folder>/<pool>/<name>.vrma` | An animation; the pool (a subfolder) says when it plays. See [Animations](#animations). |
+
+### In the code
+
+Constants with no setting of their own; edit the file to change them.
+
+| File | Constant | Value | What it does |
+| --- | --- | --- | --- |
+| `face.html` | `FRAMING` | `'body'` | `'body'` frames her head to toe (with room for the reach of every looping clip), `'face'` head and shoulders. |
+| `face.html` | `LOOK` | cyan, violet, green, red, bright cyan | The glow colour per state: idle, thinking, tool, attention, speaking. |
+| `face.html` | blinking | every 2–6 s, 0.12 s long | How often and how long she blinks. |
+| `face.html` | smile | every 25–50 s for 3 s at 0.5; 2 s after a task | The soft `relaxed` smile. |
+| `face.html` | thinking tilt / speaking nod | 0.12 / 0.04 rad | How far the head tilts while thinking and nods with the voice. |
+| `face.html` | crossfade | 0.6 s (actions 0.3 s) | Blend time between animations. |
+| `client.py` | `MOUTH_STEP` | 0.04 s | Audio per mouth value. |
+| `zrb_init.py` | `AWAKE_HOURS` | 8:00–21:59 | When she may speak first. |
+| `zrb_init.py` | idle check / away time | every 60 s / 5 min | How often `idle` checks; how long you must be away before `camera` greets you back. |
+| `zrb_init.py` | `NOTHING_TO_SAY` | `…` | The reply she gives, unspoken, when nothing is worth saying. |
+| `app/main.js` | window | 440×640, bottom right | Size and place of the window (320×420 suits `'face'` framing). |
+| `app/main.js` | on top | `screen-saver` level | Stays above full-screen windows too. |
+| `face.html` | emotion | 0.6 for 6 s | How strong and how long she shows an emotion. |
+| `make_idle.py` | `CLIPS`, `DANCES` | 8 idles, 4 dances | The animations as eased keyframes of poses; run the script to regenerate `zrb_face/anim/idle/` and `dance/`. |
+| `make_idle.py` | `FPS` | 15 | Keyframes per second in the generated clips. |
+
+## Changing the model
+
+Put any VRM file (VRM 0.x or 1.0) in `~/.config/zrb-face/models/` (or a folder in `ZRB_FACE_MODELS`) as `<name>.vrm` and open it by name: `zrb-face <name>`. Without a name, the window shows the first model alphabetically. No model ships with the package. Model files are kept out of git (`*.vrm` is ignored). Nothing in the code depends on the model: the camera fits it, and the pose and idle come from the animations, which three-vrm retargets onto any VRM, as VRoid Hub's playground does.
+
+Where to get models:
+
+- **[VRoid Studio](https://vroid.com/en/studio)**: free, for Windows and macOS. Make your own and export it as VRM. You own it, and it has every expression.
+- **[VRoid Hub](https://hub.vroid.com/en)**: thousands of models. Only the ones marked downloadable can be downloaded, and each has its own conditions (avatar use, commercial use, redistribution, changes). Read them before you change or share the model.
+- **[BOOTH](https://booth.pm/en/search/VRM)**: search for `VRM` and filter for free items. The terms are on each item's page, often only in Japanese.
+- **[Open Source Avatars](https://github.com/toxsam/open-source-avatars)**: free VRM avatars with direct download links. The license is set per collection (CC0, CC-BY, …), so check it.
+- **[OpenGameArt: VRoid Studio CC0 models](https://opengameart.org/content/vroid-studio-cc0-models)**: public domain sample models from early VRoid Studio.
+
+Fan models of existing characters (Cortana, Kusanagi, and others) are fine on your own desktop, but don't commit or share them.
+
+## Animations
+
+Every `<pool>/<name>.vrma` (VRM Animation) in the animation folders loads when the page starts. The pool, the subfolder, says when it plays:
+
+| Pool | Plays |
+| --- | --- |
+| `idle/` | Loops while idle, and for every state with no pool of its own. |
+| `thinking/`, `tool/`, `attention/`, `speaking/` | Loops in that state. None ship; add your own. |
+| `dance/` | Loops instead of `idle/` while music plays. |
+| `actions/` | Once, on `{"type": "action", "value": "<name>"}`, then back to looping. zrb plays them through its `FaceAction` tool; `greeting` plays when a chat starts, and `happy`, `sad`, … with that emotion. |
+
+Any other pool loads too, and plays once something asks for it.
+
+To add your own, drop the file in, e.g. `~/.config/zrb-face/anim/actions/wave.vrma`, and restart with `zrb-face` (and zrb, so the `FaceAction` tool knows the new name). A file named like a shipped one replaces it: `~/.config/zrb-face/anim/idle/rest.vrma`. Try an action with `zrb-face action wave`.
+
+Within a pool, clips play in random order with crossfades, picked by group: the first part of the name (`stretch-side` belongs to `stretch`). A random group, then a random clip in it, so the stretches together come up as often as any single other idle. On top of whatever plays: the head tilts while thinking, nods while speaking, the eyes follow you, and the face shows zrb's emotion.
+
+The idles and dances are ours (CC0), generated by `python3 make_idle.py`. The dances are 8-beat loops at 120 BPM: `bounce` (down on every beat, arms pumping), `sway` (side to side every two beats), `step-touch` (stepping side to side, hands up on the touch) and `hands-up` (arms high, waving). They don't follow the song's tempo yet. The idles:
+
+| Clip | What she does |
+| --- | --- |
+| `rest` | Relaxed, a slow weight shift. |
+| `hands` | Hands clasped in front, swaying; the free foot twists on its toes. |
+| `weight` | Shifts from hip to hip with the free knee bent, taking a small step each time. |
+| `look` | Glances left and right. |
+| `stretch` | The yawn-stretch: hands rise past the face, then up and out into a Y, shoulders shrugged, back arched, up on the toes, eyes closed, yawning; then a slow release, shoulders dropping. |
+| `stretch-side` | One arm arcs over her head, the other hand slides down the thigh, the hips shift the other way; each side. |
+| `stretch-neck` | Ear to each shoulder, then chin down; shoulders dropped, eyes closed. |
+| `stretch-twist` | Turns from the hips up, the head leading, the arms swinging loosely, the back heel lifting. |
+
+Every clip also has breathing, uneven head drift, knees that flex a little, and fingers that relax and tighten. The hands are loosely curled, the arms hang a little away from the body, and the weight shifts move the hips over the standing leg. The motion follows VRoid's own VRMA clips (every joint moving, fingers included), common VTuber idle loops (breathing, sway, head pitch, roll and yaw), and how people actually stretch (the whole-body yawn-stretch; a side bend with the hips shifting the other way).
+
+To change or add idles, edit `CLIPS` in `make_idle.py` (eased keyframes of poses) and run it again. Its docstring lists which way each joint turns. Poses can also move the hips (`hipsMove`) and set the `blink` and `aa` expressions; the page takes the larger of the clip's value and its own blinking and lip sync.
+
+More free VRMA:
+
+- **VRoid's official 7-motion pack** (show full body, greeting, peace sign, shoot, spin, model pose, squat), free on [BOOTH](https://vroid.com/en/news/6HozzBIV0KkcKf9dc1fZGW). Free to use, commercial use with the credit "Animation credits to pixiv Inc.'s VRoid Project", but its terms forbid redistributing the files, so keep them out of git. E.g. `cp VRMA_02.vrma ~/.config/zrb-face/anim/actions/greeting.vrma`.
+- **Dances**: motion capture through bvh2vrma, e.g. the [AIST++](https://google.github.io/aistplusplus_dataset/) dance dataset (CC BY 4.0, credit needed) or the [CMU](http://mocap.cs.cmu.edu/) library (free). Mixamo's dances can be converted too, but its terms forbid sharing the files, so keep them in your own folder.
+- **Your own**: [bvh2vrma](https://vrm-c.github.io/bvh2vrma/) (VRM Consortium, MIT) converts BVH motion capture to VRMA, and VRoid Studio exports poses and animations as VRMA.
+
+## The desktop app
+
+Each [release](https://github.com/state-alchemists/zrb-face/releases) has a zip per platform:
+
+- **Windows** (`zrb-face-win32-x64.zip`): unzip into `%LOCALAPPDATA%\zrb-face\`, so `zrb-face` finds `zrb-face.exe` there from WSL.
+- **macOS** (`zrb-face-darwin-arm64.zip` for Apple silicon, `-x64` for Intel): move `zrb-face.app` to `/Applications`. The app isn't signed, so clear macOS's block once with `xattr -dr com.apple.quarantine /Applications/zrb-face.app`.
+- **Linux** (`zrb-face-linux-x64.zip`): unzip anywhere and set `ZRB_FACE_APP` to its `zrb-face` program. On Wayland (GNOME), a window can't keep itself on top, so pin it by hand.
+
+To build it yourself:
+
+```sh
+cd app
+npm install
+npm start                                   # run from source
+npm run pack:win                            # or pack:mac / pack:linux -> dist/
+cp -r dist/zrb-face-win32-x64 /mnt/c/Users/$USER/AppData/Local/zrb-face
+```
+
+If GitHub downloads are slow, prefix the build with `ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/`, then check the zip in `~/.cache/electron` against GitHub's `SHASUMS256.txt`.
+
+## Test
+
+```sh
+python3 test_face.py                        # with a Python that has zrb installed
+```
+
+## Releasing
+
+Bump the version in `pyproject.toml` and `app/package.json`, then push a tag: `git tag 0.1.0 && git push --tags`. `.github/workflows/release.yml` publishes the package to PyPI and attaches the app zips to the GitHub release. PyPI publishing is trusted publishing, so first add this repository, the workflow `release.yml` and the environment `pypi` as a trusted publisher of `zrb-face` on pypi.org.
+
+## Credits
+
+`zrb_face/anim/` and `make_idle.py`: ours, CC0.
+
+The models used during development, not included: *ram* and *rem made in mexico* (Re:Zero fan models) by pon3mx on [VRoid Hub](https://hub.vroid.com/en); search for them there. Ram: use, changes, commercial use and redistribution allowed, no credit needed. Rem: the same but no commercial use. Both have mouth, blink and emotion expressions. They are fan models of the Re:Zero characters, so keep them on your own machine.

zrb_face-0.1.0/pyproject.toml

@@ -0,0 +1,26 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "zrb-face"
+version = "0.1.0"
+description = "A 3D character for zrb in a small, see-through, always-on-top window"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = []  # the server and the CLI are stdlib only; zrb_init.py runs inside zrb
+
+[project.optional-dependencies]
+zrb = ["zrb"]
+
+[project.scripts]
+zrb-face = "zrb_face.__main__:main"
+
+[project.urls]
+Homepage = "https://github.com/state-alchemists/zrb-face"
+
+[tool.hatch.build.targets.wheel]
+packages = ["zrb_face"]
+
+[tool.hatch.build.targets.sdist]
+include = ["zrb_face", "README.md"]

zrb_face-0.1.0/zrb_face/__init__.py

@@ -0,0 +1 @@
+"""zrb-face: a 3D character for zrb in a small, see-through, always-on-top window."""