@tensorkit/raydr 0.0.0-stage → 0.1.0

package/README.md

@@ -1,3 +1,554 @@
-# Temporary Holding Version
+# raydr
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Long planning conversations with a coding agent keep you at the desk, reading. The agent works
+for a while, then stops to ask a question or wait for a permission, and if you have walked away
+it sits there until you come back and notice. Voice assistants that promise to fix this put a
+second model between you and the agent: they summarise, paraphrase, and answer on your behalf,
+so what you hear is no longer what the agent said.
+
+raydr takes the other route. It speaks the agent's own words, verbatim, and only when the agent
+has something for you: a finished reply, a menu of choices, a permission prompt. You hear it on
+your phone or across the room, and you can answer from the phone with a tap or by talking. The
+agent's terminal stays the source of truth; raydr only reads what Claude Code reports and types
+the keystrokes you would have typed.
+
+It runs on your machine, talks to ElevenLabs for speech, and needs nothing else.
+
+<p>
+  <img src="docs/screenshots/phone-waiting.png" width="260" alt="Phone: a session is waiting for you with a three-option question in the tray">
+  <img src="docs/screenshots/phone-listening.png" width="260" alt="Phone: a reply being spoken, the orb breathing, the transcript in ink">
+</p>
+<img src="docs/screenshots/desktop.png" width="820" alt="Desktop: session rail on the left, transcript in the middle, prompts from every session on the right">
+
+## Contents
+
+- [What raydr is](#what-raydr-is)
+- [How it works](#how-it-works)
+- [Where things live](#where-things-live)
+- [Requirements](#requirements)
+- [Install](#install)
+- [First run, step by step](#first-run-step-by-step)
+- [Daily use](#daily-use)
+- [The interface](#the-interface)
+- [Reaching it from a phone](#reaching-it-from-a-phone)
+- [When things happen](#when-things-happen)
+- [Command reference](#command-reference)
+- [HTTP API](#http-api)
+- [Configuration](#configuration)
+- [Security model](#security-model)
+- [Troubleshooting](#troubleshooting)
+- [Known limits](#known-limits)
+- [Development](#development)
+
+## What raydr is
+
+raydr is a bridge between Claude Code and a web page. Claude Code reports events through its
+hook system; raydr turns the ones worth hearing into speech and streams them to a small app
+that works on a phone. From that app you can answer the agent's menus, approve or deny a
+permission prompt, and send a typed or spoken message into the session.
+
+What you hear:
+
+- **Replies.** When Claude finishes a turn, its final message is spoken. Markdown, file paths,
+  URLs, hashes and code blocks are rendered into something a voice can say ("page dot tsx",
+  "code block, 4 lines") while the transcript keeps the original text.
+- **Questions.** When Claude calls `AskUserQuestion`, the question and its options are spoken
+  ("Question: … Options: one, …; two, …") and the options appear as buttons.
+- **Permission prompts.** When Claude Code shows a permission dialog you hear "Claude needs your
+  permission" and get Allow and Deny.
+
+What raydr is not:
+
+- **No model in the loop.** Nothing is summarised or rephrased by another LLM. Brief mode cuts a
+  reply to its first paragraph; it does not rewrite it.
+- **No cloud relay.** The server binds to your machine's loopback interface. Audio goes to
+  ElevenLabs for synthesis and, when you talk back, for transcription; nothing else leaves.
+- **No chat channel.** There is no Telegram, Slack or email integration; the page is the client.
+- **No autonomy.** raydr never answers anything by itself. Every answer is a tap or a
+  recording you made.
+
+## How it works
+
+```
+Claude Code ──command hooks──▶ raydr server ──SSE──▶ web app (phone / desktop)
+                                    │  ▲                       │
+                          ElevenLabs│  │ElevenLabs Scribe      │ taps, voice
+                            speech  ▼  │                       ▼
+                             mp3 cache  └──────────── POST /commands, /transcribe
+                                                              │
+                                              herdr agent send-keys / prompt
+                                                              ▼
+                                                    the Claude Code pane
+```
+
+**Output path.** `raydr hooks install` adds three hooks to your Claude Code settings: `Stop`,
+`PreToolUse` with matcher `AskUserQuestion`, and `Notification`. Each hook runs `raydr hook
+<event>`, a tiny command that forwards the JSON Claude Code writes to its stdin to
+`POST /hooks/<event>` on the server, with the access key as a bearer token. It always exits 0
+without output, so a stopped server never surfaces an error inside Claude Code.
+
+The server answers the hook at once, then decides what the event means: a `Stop` becomes an
+`assistant_message`, an `AskUserQuestion` a `question`, a `permission_prompt` notification a
+`permission`. It renders the speakable text, asks ElevenLabs for an mp3, and publishes the event
+over a server-sent events stream with the URL of the clip. The clip starts streaming to the
+browser as soon as the first bytes arrive from ElevenLabs; a client that requests it mid-synthesis
+gets the whole file. Clips are cached on disk for a day.
+
+**Input path.** The app sends commands (`answer_question`, `permission_reply`, `say`) to
+`POST /commands`. The server maps the session to the [Herdr](https://herdr.dev) pane that runs
+it (the hook carries the pane id from `HERDR_PANE_ID`) and injects keystrokes with `herdr agent
+send-keys`, or types a message with `herdr agent prompt`. The outcome comes back as a
+`command_result` event. Voice goes through `POST /transcribe` (ElevenLabs Scribe) and then out
+as a `say`. Outside Herdr, raydr still speaks; it just cannot type back.
+
+## Where things live
+
+| What | Where | Notes |
+|---|---|---|
+| Configuration | `~/.config/raydr/env` | `KEY=VALUE` lines, mode 600; created by `raydr init` |
+| Pid file and log | `~/.local/state/raydr/server.pid`, `server.log` | written by `raydr start`; the log is never rotated |
+| Per-directory settings | `~/.local/state/raydr/sessions.json` | brief mode, keyed by working directory |
+| Audio cache | `~/.cache/raydr/audio/` | `<id>.mp3` per clip, `label-<name>.mp3` per session name; pruned after 24 h at startup |
+| Hooks | `~/.claude/settings.json` (or `./.claude/settings.local.json` with `--project`) | three command hooks pointing at this checkout's CLI |
+| `/raydr` command | `~/.claude/commands/raydr.md` (or `./.claude/commands/`) | copied by `hooks install`; also `raydr --command` |
+| Session opt-ins | `~/.local/state/raydr/sessions/<session id>.json` | written by `/raydr`; the hooks forward only sessions with a file |
+| Web bundle | `apps/web/dist` in the checkout | served by the server; rebuilt by `pnpm -r build` |
+| Server | `http://127.0.0.1:47391` | loopback only; `RAYDR_HOST` / `RAYDR_PORT` move it |
+| Sessions | any Claude Code session whose settings carry the hooks | named after its working directory |
+
+`XDG_CONFIG_HOME`, `XDG_STATE_HOME` and `XDG_CACHE_HOME` are honoured; `RAYDR_ENV_FILE`,
+`RAYDR_STATE_DIR` and `RAYDR_CACHE_DIR` override the individual locations.
+
+## Requirements
+
+- **Node 24** or newer and **pnpm** (`corepack enable` gives you the pinned version).
+- An **ElevenLabs API key** with text-to-speech permission. Without it every event still flows
+  to the page, just silently; with it, speech and transcription work.
+- **Herdr** on the same machine if you want to answer from the phone. Claude Code must run inside
+  a Herdr pane for commands to reach it. Listening works without Herdr.
+- **Claude Code** with hook support (2.1 or newer).
+- A browser. Safari on iOS and Chrome on Android and desktop are what the app is built for.
+
+## Install
+
+raydr is one npm package, `@tensorkit/raydr`: the CLI, the server and the web app together.
+
+```sh
+npm i -g @tensorkit/raydr
+raydr --version
+```
+
+To work on raydr itself, install from a clone instead:
+
+```sh
+git clone git@github.com:tensorkithq/raydr.git
+cd raydr
+pnpm install
+pnpm -r build
+```
+
+Expected: `pnpm -r build` ends with `apps/web build: ✓ built in …ms` and `Done` lines for the
+server and the `raydr` package. Then put `raydr` on your PATH, either way:
+
+```sh
+pnpm link --global              # needs a one-time `pnpm setup` so pnpm has a global bin directory
+cd packages/cli && npm link     # or: the same thing through npm, no setup step
+```
+
+`raydr --help` prints the usage block and `raydr doctor` reports which install it is running
+from (`npm package` or `linked checkout`). Running the CLI straight from GitHub with
+`pnpm dlx github:tensorkithq/raydr` does not work: pnpm refuses to run a git-hosted package's
+build script unless it is on a workspace allowlist, which `dlx` cannot be given.
+
+The command file for Claude Code is part of the install: `raydr --command` prints it and
+`raydr hooks install` copies it into place, so no other repository is needed.
+
+## First run, step by step
+
+1. **Create the configuration.**
+   ```
+   $ raydr init
+   Speech provider (elevenlabs / openai / kokoro) [elevenlabs]:
+   Transcription provider (elevenlabs / openai / whisper) [elevenlabs]:
+   ElevenLabs API key (hidden):
+   ElevenLabs voice id (blank for default):
+   Port (blank for 47391):
+   wrote ~/.config/raydr/env (mode 600): speech elevenlabs, transcription elevenlabs
+   next: raydr hooks install, then raydr start
+   ```
+   Only the keys the chosen providers need are asked for, typed without echo. Choosing `kokoro`
+   or `whisper` offers to run `raydr local setup` right away. `init` also generates
+   `RAYDR_ACCESS_KEY`, the password for the page. Prefer the prompts over `--elevenlabs-key`
+   and `--openai-key`, which would land in your shell history; `--speech` and `--transcription`
+   script the choice.
+
+2. **Install the hooks and the `/raydr` command.**
+   ```
+   $ raydr hooks install
+   backup written to ~/.claude/settings.json.raydr-backup-2026-…
+   installed hooks in ~/.claude/settings.json
+   installed /raydr command at ~/.claude/commands/raydr.md
+   they will post to http://127.0.0.1:47391
+   ```
+   A timestamped backup is written beside the settings file first. Existing hooks are untouched.
+   The command file calls `raydr` by name, so `raydr` must be on the PATH Claude Code sees
+   (`pnpm link --global` does that; the installer warns if it is not). The file is also
+   available on its own: `raydr --command > ~/.claude/commands/raydr.md`.
+
+3. **Start the server.**
+   ```
+   $ raydr start
+   raydr started (pid 12345) on http://127.0.0.1:47391
+   log: ~/.local/state/raydr/server.log
+   ```
+   Without an ElevenLabs key `start` and `serve` stop with `no ElevenLabs key configured; run
+   raydr init`; pass `--no-speech` to run the API without speech on purpose. Any command run
+   before the env file exists ends with a reminder to run `raydr init`.
+
+4. **Check everything.**
+   ```
+   $ raydr doctor
+   [ ok ] node            v24.19.0
+   [ ok ] herdr           status: running
+   [ ok ] env file        ~/.config/raydr/env (mode 600)
+   [ ok ] elevenlabs key  present
+   [ ok ] hooks           ~/.claude/settings.json (Stop, PreToolUse, Notification)
+   [ ok ] server          http://127.0.0.1:47391 pid 12345, speech elevenlabs
+   [ ok ] elevenlabs api  key accepted (2-character synthesis)
+   ```
+   The last line spends two characters of your ElevenLabs quota; `--offline` skips it. Anything
+   marked `FAIL` exits with status 1.
+
+5. **Open the page.** Visit `http://127.0.0.1:47391`, paste the access key (it is the
+   `RAYDR_ACCESS_KEY` line in the env file), and tap Connect. The key is kept in the browser's
+   local storage.
+
+6. **Opt a session in.** Sessions are silent until they join. In any Claude Code session type
+   `/raydr`; it prints `raydr: on for this session (annotator, pane w1:p5) · server running on
+   http://127.0.0.1:47391` and the chip appears on the page, named after the directory. `/raydr`
+   again turns it off, `/raydr status` just reports. Tap the chip: that first tap also unlocks
+   audio, which browsers require before a page may play sound.
+
+7. **Hear a reply.** The next time Claude finishes a turn in that session, you hear it. The
+   transcript shows the same words; Said and Written toggles show the spoken rendering and the
+   original text when they differ.
+
+## Daily use
+
+**Start per login.** On desktop Linux, user processes end at logout, so run `raydr start` once
+per login session (or from your shell profile). `raydr status` tells you whether it is up,
+`raydr logs -f` follows the log, `raydr stop` ends it. `raydr serve` runs it in the foreground
+instead, which is handy when developing.
+
+**Answer a menu.** A question rises into the tray above the dock. Tap an option (or several, for
+a multi-select question), then Send. The server presses the matching keys in the terminal and the
+card collapses to "Answered: …" when the keystrokes land.
+
+**Answer a permission.** Tap Allow, then tap it again within 1.5 s while it reads "Tap again to
+allow"; or tap Deny once. Allow presses `1` in the dialog; Deny presses Escape, which in Claude Code ends the
+turn without a reply.
+
+**Talk back.** Hold the black microphone button; playback pauses while it is down. Release to see
+the transcript, then Send. The text is typed into the session as a new message. Slide up to
+cancel; presses under 300 ms send nothing. The keyboard button reveals a text field for typing.
+
+**Brief mode.** Long-press a chip (or use the switch on the focused chip) and choose Brief:
+only the first paragraph of each reply is spoken, with a leading heading joined to it. The choice
+is stored on the server per working directory, so it survives restarts.
+
+**Also speak.** Long-press a chip that is not focused and choose Also speak: its replies play even
+while you follow another session, each introduced by the session's spoken name (a short tone if
+the name clip is missing).
+
+**Skip and speed.** The skip button drops the current clip; holding it jumps to the latest. Speed
+(1×, 1.25×, 1.5×) is in the settings sheet behind the header, and beside it on desktop.
+
+**Keyboard map (any width).** Space held is the microphone, → skips, 1–9 focus sessions in rail
+order. Keys typed into a text field stay in the field.
+
+**Opt in and out.** `/raydr` inside a session toggles it; the choice is a small file under the
+state directory that the hooks check on every event, so it costs nothing when off. Set
+`RAYDR_SESSIONS=all` in the env file to hear every session without opting in (the old
+behaviour); `RAYDR_OFF=1` in a pane's environment silences that pane regardless.
+
+**Limit to a project.** `raydr hooks install --project` writes the hooks into
+`./.claude/settings.local.json` of the current directory instead of the user-wide file, so only
+sessions in that project report. `raydr hooks status` lists every settings file that carries
+raydr hooks.
+
+**Uninstall completely.**
+```sh
+raydr hooks uninstall            # removes exactly the raydr entries; add --project for a project file
+raydr stop
+rm -rf ~/.local/state/raydr ~/.cache/raydr ~/.config/raydr
+pnpm unlink --global              # from the checkout, if you linked it
+```
+
+## The interface
+
+The focused session is the page; every other session is a chip. On a phone, top to bottom: the
+header (an orb beside the session name and a sub-line such as "speaking · clip 2 of 3"), the
+chips row, the transcript, the attention tray, and the dock. Above 900 px the chips become a
+left rail that also shows the keyboard map, the tray becomes a right column listing prompts from
+every session, and the text field is always visible.
+
+**Sessions and focus.** Sessions arrive silent. Each chip shows a count of unheard replies and a
+small orb when the session has a prompt waiting. Tap a chip to focus it: only the focused session
+is heard, its transcript is shown, and the microphone and text field send to it. Focus and
+also-speak are remembered by working directory, so restarting Claude Code in the same project
+restores them even though the session id changes.
+
+**The orb** is the single status instrument: it breathes and turns while a clip plays, sits still
+at 70 % while a prompt waits on you, and fades to 35 % when quiet. Under
+`prefers-reduced-motion` only the opacity changes.
+
+**Ink as time.** In the transcript the clip being spoken is full black, clips already heard are
+grey, and clips that have arrived but not yet played are lighter still. Only the current clip
+carries a timestamp.
+
+**Tray and guardrails.** Questions and permissions never sit in the stream; they rise into the
+tray, one at a time with a "1 of n" counter, focused session first and then oldest first.
+Prompts from other sessions are answerable from the tray without changing focus, labelled
+"<session> asks". A confirmed answer leaves a grey one-line record; a refused one stays in the
+tray with the reason so it can be retried. Menu answers are select-then-Send and permissions are
+tap-twice-to-allow (the first tap arms the button for 1.5 s, the second sends), so a pocket tap
+never approves anything.
+
+**History.** Transcripts live in the browser only: each session's items and last-played marker
+are kept in the page's local storage keyed by working directory (300 items per session, 20
+sessions), so a refresh shows them before the server replays anything. "Clear history on this
+device" in the settings sheet wipes them; the server keeps nothing beyond its own event replay
+buffer.
+
+**Catch-up.** At most two clips wait in the queue. When a third arrives the oldest are dropped and
+the transcript notes "skipped N older replies", so a long absence never means a backlog.
+
+## Providers
+
+Speech (what you hear) and transcription (what you say) are chosen independently at
+`raydr init` and can be switched later by editing two lines in the env file and restarting.
+
+| Provider | Kind | What it needs | Cost | Notes |
+|---|---|---|---|---|
+| `elevenlabs` (default) | speech and transcription | `ELEVENLABS_API_KEY` | premium: a paid plan, billed per character and per minute | the most natural voices; Scribe for transcription |
+| `openai` | speech and transcription | `OPENAI_API_KEY` | `gpt-4o-mini-tts` about $12 per 1M output characters, `gpt-4o-mini-transcribe` about $0.003 per minute (OpenAI's pricing page, September 2026) | instructable voice: `RAYDR_SPEECH_INSTRUCTIONS` sets the delivery, `RAYDR_OPENAI_VOICE` the voice (default `alloy`) |
+| `kokoro` | speech | `raydr local setup` (Python 3.10–3.13 or `uv`, about 1.5 GB venv, 330 MB model) | free after download | runs on CPU in a local sidecar; about 2× realtime on 8 cores; `RAYDR_KOKORO_VOICE` (default `af_heart`) |
+| `whisper` | transcription | the same setup (faster-whisper, 460 MB for `small.en`) | free after download | int8 on CPU; a 7 s take transcribes in about 2 s; `RAYDR_WHISPER_MODEL` (default `small.en`) |
+
+The local sidecar (`packages/cli/local/raydr_local.py`) is started and stopped with the server
+(`raydr start` / `stop`; `serve` runs it inline) and answers on `RAYDR_LOCAL_PORT` (47392) on
+loopback only. `raydr local setup` is idempotent: it checks Python, the venv, each package, the
+model caches and the bundled espeak-ng before touching anything, and prints a checklist. No
+system package is required; if the bundled espeak-ng cannot load on your system, Kokoro still
+speaks every dictionary word and skips the rest, and `apt install espeak-ng` /
+`brew install espeak-ng` restores the fallback.
+
+**Choosing a voice.** Provider tables do not tell you how a voice sounds in your room. Put the
+same paragraph through each candidate (`raydr init --force --speech openai`, restart, trigger a
+reply; then `--speech kokoro`, and so on) and listen on the phone you will use before deciding.
+Speed, warmth and how names are pronounced differ more than the price does.
+
+## Reaching it from a phone
+
+The server binds to loopback and should stay that way. To listen from a phone, put something in
+front of it that terminates TLS and reaches `127.0.0.1:47391` from inside the machine: a
+[Caddy](https://caddyserver.com) site, Tailscale Serve, or an SSH tunnel from the phone. With
+Caddy and a hostname that resolves to the machine:
+
+```
+raydr.example.com {
+    reverse_proxy 127.0.0.1:47391
+}
+```
+
+Open `https://raydr.example.com` on the phone and paste the access key. The access key is the
+only thing standing between the internet and your sessions, so do not expose the port itself and
+do not put the page behind a hostname you would not want guessed. Microphone capture requires a
+secure context, so plain `http://` from another device gives you listening but not talking back.
+
+## When things happen
+
+| Moment | What happens |
+|---|---|
+| You change server code (`apps/server`, `packages/cli`) | rebuild with `pnpm -r build`, then `raydr stop` and `raydr start`; a running server keeps its old code |
+| You change the web app (`apps/web`) | rebuild; a page reload is enough, the server serves `apps/web/dist` from disk |
+| You run `raydr hooks install` | sessions already running pick the hooks up at their next event; nothing to restart |
+| A session starts | nothing is heard until `/raydr` is run in it (or `RAYDR_SESSIONS=all`); the chip appears the moment it joins and disappears when it leaves |
+| You run `/raydr` | the opt-in file is written and the server is told at once; the next Claude Code event from that session is forwarded. Claude Code reads command files at startup, so a session started before `hooks install` needs a restart to see `/raydr` |
+| Claude finishes a turn | the reply is spoken if the session is focused or set to also speak; otherwise the chip's unheard count rises |
+| Claude asks a question | a `question` event arrives immediately; the chip gains a small orb and the card enters the tray |
+| Claude shows a permission dialog | Claude Code sends the notification about **6 seconds** after the dialog appears; the Allow control only works while the dialog is still open, and a reply sent for a prompt that was answered in the terminal fails with "not showing a permission dialog right now" |
+| You tap Allow or an option | the server presses keys in the pane; the card collapses on the `command_result`, usually within a second |
+| The microphone or text field is disabled | the session has no Herdr pane (started outside Herdr), Claude is waiting on a dialog in the terminal, or the browser has no `MediaRecorder` (an insecure context); the dock says which |
+| The page reloads | the transcript resumes from the last event this browser saw; replayed history is shown but not spoken |
+| You log out of the machine | the background server ends; `raydr start` again after login |
+
+## Command reference
+
+| Command | What it does | Flags |
+|---|---|---|
+| `raydr init` | Pick providers, write `~/.config/raydr/env` (mode 600) and generate the access key | `--speech P`, `--transcription P`, `--elevenlabs-key K`, `--openai-key K` (scripts only), `--voice-id V`, `--port P`, `--force` (rewrite, keep keys), `--rotate-key` |
+| `raydr local setup` | Install the Python sidecar for `kokoro` / `whisper`: Python, venv, packages, models, espeak probe; idempotent | |
+| `raydr local serve` | Run the sidecar in the foreground | |
+| `raydr start` | Run the server detached; pid and log under the state directory. Refuses if the port is taken and says whether that is raydr, and without an ElevenLabs key unless `--no-speech` | `--no-speech` |
+| `raydr stop` | SIGTERM the background server and wait up to 5 s. Only signals a pid it can prove is raydr | |
+| `raydr status` | Server address, pid, uptime, speech provider, connected clients, pid-file state, log path, hook files | |
+| `raydr logs` | Last 50 lines of the server log | `-f` to follow |
+| `raydr serve` | Run the server in the foreground (Ctrl-C stops it); same key check as `start` | `--no-speech` |
+| `raydr doctor` | Check node, herdr, the env file and its mode, each selected provider (key accepted by a two-character synthesis, or venv, models and sidecar for local ones), hooks and the server. Exit 1 on any failure | `--offline` skips the API calls |
+| `raydr hooks install` | Add the three command hooks after writing a timestamped backup and copy the `/raydr` command beside the settings file; replaces any raydr hooks it finds first, so reinstalling never duplicates | `--settings <path>`, `--project`, `--url <url>` |
+| `raydr command` | Print the `/raydr` command file (`--command` and `--skill` are aliases) | `--install` writes it to `~/.claude/commands/raydr.md`, with `--project` to `./.claude/commands/` |
+| `raydr session on\|off\|toggle\|status` | What `/raydr` runs: keep or remove this session's opt-in file and tell the server; one line of output | reads `CLAUDE_CODE_SESSION_ID`, `HERDR_PANE_ID`, `PWD` |
+| `raydr hooks uninstall` | Remove exactly the raydr entries | `--settings <path>`, `--project` |
+| `raydr hooks status` | List every settings file that carries raydr hooks and the server address | |
+| `raydr hook <event>` | Used by the installed hooks: forwards stdin to the server with the access key and pane id, 4 s budget, always exits 0 silently; drops sessions that have not opted in unless `RAYDR_SESSIONS=all`; `RAYDR_OFF=1` skips the post | `--url <url>` |
+
+`--project` wins over `--settings`. Every command reads `~/.config/raydr/env` plus the process
+environment; only `serve` and `start` will create the access key if it is missing.
+
+## HTTP API
+
+For building on the server. Every route except `/healthz` and the web app needs the access key,
+as `Authorization: Bearer <key>` or `?key=<key>` (what `<audio>` and `EventSource` can send).
+
+| Route | Purpose |
+|---|---|
+| `POST /hooks/{pretooluse,stop,notification}` | Claude Code hook payload as JSON; answers 200 at once and processes afterwards |
+| `POST /sessions/announce` | `{ sessionId, cwd, label, paneId?, state: "joined" \| "left" }` from `raydr session`; publishes `session_seen` or `session_left` |
+| `GET /events` | SSE stream: `session_seen`, `session_left`, `assistant_message`, `question`, `permission`, `command_result`. Replays recent history, honours `Last-Event-ID`, then sends `event: live` |
+| `GET /sessions` | Sessions seen since startup, with pane id, label clip and speak mode |
+| `PUT /sessions/:id/settings` | `{ "speakMode": "full" \| "summary" }` for that session's directory |
+| `POST /commands` | `answer_question { toolUseId, selections: number[][] }`, `permission_reply { decision }`, `say { text }`, each with a client `commandId` and `sessionId`; answers `202 { commandId }`, result arrives as a `command_result` event |
+| `POST /transcribe` | multipart `file` (up to 10 MB) → `{ text }`; 400 for bad audio, 502 when ElevenLabs fails, 503 when unconfigured |
+| `GET /audio/:id.mp3` | The clip; streams while still being synthesised |
+| `GET /healthz` | `{ ok, speech, clients, pid, port, uptimeSeconds }`, no key |
+| `GET /*` | The web app |
+
+Event and command shapes are typed in `packages/protocol/src/index.ts`. Ids are monotonic across
+server restarts, so a client can dedupe on them.
+
+## Configuration
+
+Read from `~/.config/raydr/env`; a variable in the process environment overrides the file.
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `RAYDR_ACCESS_KEY` | generated by `init` (or the server on first run) | Required by every authenticated route and sent by the hooks |
+| `RAYDR_SPEECH_PROVIDER` | `elevenlabs` | `elevenlabs`, `openai` or `kokoro` |
+| `RAYDR_TRANSCRIPTION_PROVIDER` | `elevenlabs` | `elevenlabs`, `openai` or `whisper` |
+| `ELEVENLABS_API_KEY` | | Needed by the `elevenlabs` providers |
+| `ELEVENLABS_VOICE_ID` | ElevenLabs' "Rachel" | Any voice id from your account |
+| `OPENAI_API_KEY` | | Needed by the `openai` providers |
+| `RAYDR_OPENAI_VOICE` | `alloy` | OpenAI voice name |
+| `RAYDR_SPEECH_INSTRUCTIONS` | a calm, unhurried colleague | Delivery instruction for OpenAI's instructable voices |
+| `RAYDR_KOKORO_VOICE` | `af_heart` | Kokoro voice |
+| `RAYDR_WHISPER_MODEL` | `small.en` | faster-whisper model name |
+| `RAYDR_LOCAL_PORT` | `47392` | Loopback port of the local sidecar |
+| `RAYDR_LOCAL_DIR` | `~/.local/share/raydr` | venv, model caches and setup record for the sidecar |
+| `RAYDR_HOST` | `127.0.0.1` | Bind address; keep it on loopback |
+| `RAYDR_PORT` | `47391` | Bind port |
+| `RAYDR_MAX_SPEECH_CHARS` | `1500` | Longer replies are cut at a sentence boundary before synthesis |
+| `RAYDR_ENV_FILE` | `~/.config/raydr/env` | Read a different env file (process environment only) |
+| `RAYDR_STATE_DIR` | `~/.local/state/raydr` | Pid file, log, per-directory settings |
+| `RAYDR_CACHE_DIR` | `~/.cache/raydr/audio` | mp3 cache |
+| `RAYDR_WEB_DIR` | `apps/web/dist` | Built web app to serve |
+| `RAYDR_SESSIONS` | `opt-in` | `all` forwards every session without `/raydr` |
+| `RAYDR_OFF` | | `1` or `true` in a Claude Code pane's environment silences that pane's hooks |
+
+## Security model
+
+The server listens on loopback only. Every route except `/healthz` and the static page requires
+the access key: the browser sends it, and so do the hooks, which read it from the env file. Hook
+bodies must be JSON, so a web page cannot inject events or trigger paid synthesis with a
+cross-origin form post. The env file is created with mode 600 and the server warns when it is
+readable by others.
+
+Commands only ever become keystrokes in a Herdr pane the server has seen a hook from, and only
+while Herdr reports that pane as waiting on a dialog (for permissions) or the question is still
+the latest one (for menus). Transcript files named in hook payloads are read only from Claude
+Code's own transcript directory, `.jsonl` only, last 256 KB only.
+
+To use it from a phone, keep the loopback bind and put TLS in front (see above). The access key
+lives in the browser's local storage; rotate it with `raydr init --force --rotate-key`, restart
+the server, and reconnect.
+
+## Troubleshooting
+
+| Symptom | Likely cause and fix |
+|---|---|
+| Events show in the transcript but nothing plays | Audio is not unlocked: tap anywhere once. Or the clip failed: `raydr logs` shows `speech failed: ElevenLabs 401 … payment_issue` or `quota_exceeded` (settle or top up the ElevenLabs account), `OpenAI 429` (rate or quota), or `local sidecar unreachable` (`raydr status` shows the sidecar; `raydr local setup` if it never came up) |
+| `raydr start` says a provider is not configured | The env file selects a provider without its key or setup; the line names the variable or command. `raydr init --force` re-picks providers keeping existing keys |
+| The local sidecar takes long to answer | Models load on first start (about 7 s warm-up here); `raydr status` shows `models loading` until then. First-ever setup downloads about 800 MB |
+| Claude Code prints `Stop hook error: connect ECONNREFUSED` | Hooks from an older version that used HTTP hooks; run `raydr hooks install` again. Current command hooks never print anything |
+| The page says the key was rejected | The key in the browser and the env file differ (rotated key, or a different env file). Reconnect with the current `RAYDR_ACCESS_KEY` |
+| A session never appears | It has not opted in: run `/raydr` in it (`raydr session status` inside the session says). Otherwise `raydr hooks status` shows no hooks, the pane has `RAYDR_OFF=1`, or the session started before the hooks and command were installed (restart it) |
+| `/raydr` says raydr is not installed | `raydr` is not on the PATH Claude Code sees; `npm i -g @tensorkit/raydr`, or from a checkout `cd packages/cli && npm link`, then start a new session |
+| A reply is spoken twice or old replies replay | Two servers are running against the same hooks (`raydr status` and `raydr start` refuse a second one on the same port); or the browser's `raydr.lastEventId` was cleared |
+| The phone cannot reach the page | The server is on loopback by design; reach it through a TLS proxy or tunnel. Do not set `RAYDR_HOST=0.0.0.0` |
+| The microphone button is greyed out | The dock says why: no Herdr pane for the session, Claude waiting on a terminal dialog, or no `MediaRecorder` (needs `https://` or localhost) |
+| Allow does nothing or says Claude is not showing a dialog | The dialog was already answered in the terminal, or the notification has not arrived yet (about 6 s after the dialog); wait for the card, then tap |
+| `raydr stop` says the pid file was stale | The server had already exited or the pid belongs to something else; the file was removed. Use `raydr status` |
+| `raydr start` says the port is in use by another program | Set `RAYDR_PORT` in the env file and reinstall the hooks so they post to the new port |
+
+## Known limits
+
+- Several questions in a single `AskUserQuestion` call are answered in order with a fixed 300 ms
+  gap and remain unverified live; single questions of either kind are verified.
+- Multi-select answers assume the menu cursor starts on option 1, as it does for a fresh menu; if
+  it was moved in the terminal first, the wrong options are toggled.
+- Free-text "Other" answers to menus are not supported.
+- Hooks installed both user-wide and with `--project` in the same project fire twice, so each
+  reply is reported twice; pick one level per project.
+- Denying a permission ends Claude's turn without a reply (that is how Escape behaves in Claude
+  Code); Allow lets it continue.
+- The server log is appended and never rotated.
+- Install from GitHub with `pnpm dlx` is blocked by pnpm's build-script policy; clone instead.
+- The local providers need Python 3.10–3.13 (Kokoro's dependencies have no wheels for 3.14 yet);
+  `raydr local setup` uses `uv` to fetch 3.12 when it is on PATH. OpenAI adapters were written
+  against the documented API and tested with a fake server, not a live key.
+- Sessions are known only for the life of the server process; a restart starts with an empty
+  list until each session reports again.
+
+## Development
+
+```
+packages/protocol   event and command types shared by server and web (types only)
+packages/cli        the raydr binary: lifecycle, init, doctor, hooks, the hook forwarder
+apps/server         node:http server, hook parsing, speech rendering, audio store, Herdr driver
+apps/web            Vite + React + Tailwind + Zustand client; see apps/web/README.md
+```
+
+The four checks, run from the root:
+
+```sh
+pnpm -r typecheck
+pnpm check          # Biome lint and format
+pnpm -r build
+pnpm -r test        # node:test; 41 server, 54 web, 20 cli
+```
+
+`pnpm --filter ./apps/web dev` runs Vite with the API proxied to a local server. Server and CLI
+tests run on the TypeScript sources directly; the CLI test script builds first because the hook
+forwarder is exercised as the real `dist/index.js`. Conventional commits, small and by intent.
+
+The publishable package is `packages/cli` (npm name `@tensorkit/raydr`, binary `raydr`). Its build embeds the `/raydr`
+command file and the version, bundles the CLI, the hook forwarder and the server into `dist/`
+with esbuild (the forwarder stays a separate small entry so hook latency does not pay for the
+server), and copies `apps/web/dist` in as `web/`. Nothing from the workspace survives as a
+runtime dependency.
+
+## Releasing
+
+```sh
+# from a clean main with the four checks green
+# bump "version" in packages/cli/package.json, then regenerate and check the embedded version
+pnpm -r build
+git diff --exit-code packages/cli/src/version.ts || git add packages/cli/src/version.ts
+pnpm --filter @tensorkit/raydr exec npm pack --dry-run   # inspect the file list
+# commit as "chore(release): 0.1.1" (package.json and version.ts together)
+git tag v0.1.1 && git push --tags
+pnpm publish --filter @tensorkit/raydr --access public                   # runs the full build through prepack
+```
+
+A GitHub Actions workflow that publishes on a `v*` tag (with an npm token in the repository
+secrets) would make the last step automatic; it is optional and not set up yet.