@markusylisiurunen/tau 0.3.73 → 0.3.74

package/dist/core/static/tau_docs/speech-generation.md

@@ -0,0 +1,181 @@
+# Speech generation
+
+`tau tool speech-generate` turns a caller-chunked script into one WAV using ElevenLabs Eleven v4 Turbo (`eleven_v4_turbo`) or Eleven v4 (`eleven_v4`). It supports single-speaker narration and multi-speaker dialogue.
+
+```bash
+tau tool speech-generate --help
+```
+
+## Credentials and requirements
+
+Requires an ElevenLabs API key. See [feature-specific credentials](credentials.md#feature-specific-keys) for setup.
+
+```bash
+tau tool speech-generate --list-voices
+```
+
+Voice listing prints one JSON object per voice, containing `voice_id` and `name`. Use those IDs in the script.
+
+## Models
+
+`--model` is required; the CLI has no implicit default. Use `eleven_v4_turbo` for lower-latency speech, as shown in the examples. Use `eleven_v4` when maximum quality matters more than latency. Both support narration and multi-speaker dialogue.
+
+## Input and examples
+
+Supply a UTF-8 JSON document with exactly `voices` and `chunks`. `voices` maps speaker names to voice IDs; `chunks` is an ordered, nonempty array of nonempty arrays of speaker turns. Each turn contains exactly `speaker` and nonblank `text`.
+
+```json
+{
+  "voices": {
+    "host": "VOICE_ID_A",
+    "guest": "VOICE_ID_B"
+  },
+  "chunks": [
+    [
+      {
+        "speaker": "host",
+        "text": "[calm] Welcome. What are we exploring today?"
+      },
+      {
+        "speaker": "guest",
+        "text": "How a forest changes through the seasons."
+      }
+    ],
+    [
+      { "speaker": "host", "text": "Let's begin with spring." },
+      {
+        "speaker": "guest",
+        "text": "The first changes happen before the leaves appear."
+      }
+    ]
+  ]
+}
+```
+
+Save as `dialogue.json`, replacing the placeholder voice IDs:
+
+```bash
+tau tool speech-generate --model eleven_v4_turbo \
+  --input ./dialogue.json --output ./dialogue.wav
+
+# Alternatively, use Eleven v4 for maximum quality.
+tau tool speech-generate --model eleven_v4 \
+  --input ./dialogue.json --output ./dialogue-v4.wav
+```
+
+Single-speaker narration uses the same shape with one voice. All chunks are validated before generation:
+
+- Each chunk contains at most 2,000 Unicode code points across its text, including whitespace and delivery tags.
+- Every speaker must resolve to a voice. At most 10 distinct voices may be used.
+
+Tau packs consecutive whole chunks into requests of at most 2,000 characters, following the [dialogue endpoint's reliability recommendation](https://elevenlabs.io/docs/api-reference/text-to-dialogue/convert). The caller chooses chunk boundaries; text and turn order are preserved.
+
+## Conversational assistant replies
+
+For conversational assistant replies with Eleven v4 Turbo or Eleven v4, use `[Brisk but relaxed, speaking naturally to a colleague]` as a starting point. Put the tag at the beginning of the turn's `text`, followed by the words to speak. Tags guide delivery; they are not intended to be spoken aloud.
+
+Choose a delivery that fits the content:
+
+| Use case | Suggested tag |
+| --- | --- |
+| Everyday assistant reply | `[Brisk but relaxed, speaking naturally to a colleague]` |
+| Faster update | `[Brisk but relaxed, slightly faster, brief natural pauses]` |
+| Step-by-step explanation | `[Calm and clear, measured pace, brief pauses between steps]` |
+| Story narration | `[Warm, expressive storytelling, unhurried pace]` |
+| Quiet aside in dialogue | `[whispering]` |
+
+These are custom prompting suggestions, not official named presets or guaranteed speed controls. Each speaker turn can use a different tag; place a tag before a phrase when the delivery should change within a turn.
+
+ElevenLabs documents open-ended audio tags in its [v4 prompting guide](https://elevenlabs.io/docs/overview/capabilities/text-to-speech/best-practices#prompting-eleven-v4), also available as [plain Markdown](https://elevenlabs.io/docs/overview/capabilities/text-to-speech/best-practices.md#prompting-eleven-v4). Its [documentation index](https://elevenlabs.io/docs/llms.txt) lists other guides in an agent-friendly format.
+
+Documented tags include `[curious]`, `[excited]`, `[sarcastic]`, `[whispering]`, `[laughs]`, `[sighs]`, and `[short pause]`. For example, a turn's `text` can contain:
+
+```text
+[curious] What happens if we try the other route?
+
+[excited] That worked! [short pause] Here's what changed.
+
+[Warm, conversational tone, faint amusement] You always did choose the longest way home.
+```
+
+Tags describe audible delivery or vocal actions, not visual actions such as smiling or standing. Clear descriptions reduce ambiguity between a delivery instruction and a sound effect. Results vary by voice and language; neither Eleven v4 Turbo nor Eleven v4 treats tags as deterministic controls.
+
+Save this as `assistant-reply.json`, replacing `VOICE_ID` with an ID from `--list-voices`:
+
+```json
+{
+  "voices": { "assistant": "VOICE_ID" },
+  "chunks": [
+    [
+      {
+        "speaker": "assistant",
+        "text": "[Brisk but relaxed, speaking naturally to a colleague] I found the issue. The app was using an old setting, so your changes never reached the server. I've fixed that and checked the result. You don't need to reinstall anything. Try saving once more. If the warning comes back, send me the exact message and I'll trace it from there."
+      }
+    ]
+  ]
+}
+```
+
+```bash
+tau tool speech-generate --model eleven_v4_turbo \
+  --input ./assistant-reply.json --output ./assistant-reply.wav
+
+# Alternatively, use Eleven v4 for maximum quality.
+tau tool speech-generate --model eleven_v4 \
+  --input ./assistant-reply.json --output ./assistant-reply-v4.wav
+```
+
+For a Finnish reply, replace the turn's `text` with:
+
+```text
+[Brisk but relaxed, speaking naturally to a colleague] Löysin ongelman. Sovellus käytti vanhaa asetusta, joten muutoksesi eivät päätyneet palvelimelle. Korjasin asetuksen ja tarkistin, että tallennus toimii. Sinun ei tarvitse asentaa mitään uudelleen. Kokeile tallentaa vielä kerran. Jos varoitus palaa, lähetä minulle sen tarkka teksti, niin selvitän syyn.
+```
+
+To compare pacing, replace the leading tag with `[Brisk but relaxed, slightly faster, brief natural pauses]` and generate to a fresh output path, keeping the voice and spoken text fixed.
+
+[Eleven v4 Turbo and Eleven v4](https://elevenlabs.io/docs/overview/capabilities/text-to-speech/eleven-v4) control delivery through audio tags rather than a numeric speed setting. Results vary by voice and language.
+
+## Long recordings and assembly
+
+Responses are streamed to disk as mono 24 kHz, signed 16-bit little-endian PCM. Batches are concatenated in order into one WAV. Approximately 30 minutes occupies 86.4 MB of PCM. Each batch is capped at 128 MiB and the assembled file must fit standard RIFF/WAV's roughly 4 GiB limit. Provider requests have a ten-minute deadline.
+
+Each request receives up to three previous completed request IDs for ElevenLabs request stitching. A missing ID stops subsequent generation. Stitching uses normal ElevenLabs request logging.
+
+## Outputs and recovery
+
+The output parent directory must exist. Neither the requested WAV nor `<output>.parts` may exist. The `.parts` directory contains:
+
+- `manifest.json`: model, PCM format, output path, request plan, text and voice IDs, one-based chunk indices, completion flags, and byte counts. `requestId` and `characterCost` retain the provider's `request-id` and `character-cost` headers, including when a subsequent audio download fails.
+- `batch-0001.pcm`, etc.: completed raw PCM batches in request order.
+- `.partial` files: interrupted downloads.
+- `assembled.wav`: the assembled output.
+
+Stdout on success is one JSON object with absolute `output`, `artifacts`, and the number of `batches`. The manifest contains the script text; keep artifacts private for sensitive scripts.
+
+Failures exit nonzero and report the retained directory. Requests are not automatically retried because a failed request may already have incurred a charge.
+
+To recover after a later batch fails, inspect the manifest's `completed` flags and chunk mapping. Generate only the unfinished chunks to a fresh output path, then assemble the completed PCM files from both runs in script order. Request stitching does not carry over between invocations.
+
+For example, with FFmpeg installed:
+
+```bash
+cat ./first.wav.parts/batch-0001.pcm \
+    ./first.wav.parts/batch-0002.pcm \
+    ./remaining.wav.parts/batch-0001.pcm > ./complete.pcm
+ffmpeg -n -f s16le -ar 24000 -ac 1 -i ./complete.pcm -c:a pcm_s16le ./complete.wav
+```
+
+Use only completed `.pcm` batches, not `.partial` files.
+
+## Approximate cost
+
+Standard rates from the [ElevenLabs API pricing page](https://elevenlabs.io/pricing/api):
+
+| Model | Per 1,000 characters | 2,000 characters | 27,000 characters (roughly 30 minutes) |
+| --- | --- | --- | --- |
+| `eleven_v4_turbo` | $0.04 | $0.08 | $1.08 |
+| `eleven_v4` | $0.08 | $0.16 | $2.16 |
+
+Check current API pricing for plan and voice-specific rates. The manifest's `characterCost` preserves the provider's billing header verbatim.
+
+See [command-line tools](tools.md) for command discovery and execution ownership.

package/dist/core/static/tau_docs/telegram.md

@@ -269,32 +269,26 @@ Tau keeps tool and lifecycle chatter quiet. It sends committed assistant text, i
 
 ## Attachments and audio
 
-Telegram accepts photos and documents identified as images, PDF, `.txt`, `.md`, `.json`, `.csv`, `.yaml`, or `.yml`. It downloads supported attachments immediately to a runner temporary directory and gives the agent the local path, MIME type, byte size, and caption.
+Telegram accepts arbitrary documents, photos, videos, animations, video notes, audio files, and voice notes. The agent receives execution-environment paths, sanitized filenames, MIME types, byte sizes, and captions. Receipt never executes files.
 
-Limits are:
+Limits are 32 attachments per turn, 20 MiB per file, and 100 MiB per turn. Downloads are bounded even without size metadata. Oversized or failed transfers are skipped with a warning.
 
-- at most 32 attachments per turn
-- at most 20 MiB per file
-- at most 100 MiB total per turn
+Uploads only queue files; captions are metadata. The next text or voice transcript submits or steers with one hidden `<system>` block per attachment. Only voice notes are automatically transcribed, up to 20 minutes. Audio-player files and audio documents arrive without transcription. Voice originals remain available even if transcription fails. DM and bot-triggering group transcripts are echoed before submission.
 
-An attachment-only DM queues files for the next text or voice/audio turn; it does not run the agent by itself. Unsupported, oversized, or failed downloads are skipped with a warning. Treat every attachment as untrusted input even though its path is local.
-
-The local execution environment can access these runner temporary paths. Pending attachments and group context are not snapshot data. Shutdown clears their queues, and startup removes stale `tau-telegram-attachments-*` directories.
-
-Telegram `voice` and `audio` messages up to 20 minutes long are downloaded and transcribed. Direct DM turns and bot-triggering group turns echo the transcript to the chat before submission so the sender can verify it.
+Files use native temporary directories in the execution environment; OS cleanup can invalidate their paths. Pending queues and group context clear on shutdown. File contents and metadata are untrusted.
 
 OpenAI is the default. Select Gemini with `speechToText.provider`. Runner credentials are:
 
 - Gemini: `GEMINI_API_KEY`, then `apiKeys.google`
 - OpenAI: `OPENAI_API_KEY`, then `apiKeys.openai`
 
-Gemini extracts spelling hints with Gemini 3.8 Flash and transcribes uploaded audio verbatim with `gemini-3.5-transcribe`, using English (`en-US`) and Finnish (`fi-FI`) hints. It attempts remote file deletion afterward. OpenAI normalizes downloaded audio with runner-side `ffmpeg` and uploads it to `gpt-transcribe`. Missing keys reject the audio. See [credentials](credentials.md).
+Gemini extracts spelling hints with Gemini 3.8 Flash and transcribes uploaded audio verbatim with `gemini-3.5-transcribe`, using English (`en-US`) and Finnish (`fi-FI`) hints. It attempts remote file deletion afterward. OpenAI normalizes downloaded audio with runner-side `ffmpeg` and uploads it to `gpt-transcribe`. Missing keys prevent transcription, not file delivery. See [credentials](credentials.md).
 
 `/tts_on` uses `gemini-3.8-flash`, `gemini-3.1-flash-tts-preview`, Despina, the Google key, and runner `ffmpeg` with Opus. Source and rewritten text each allow 10,000 Unicode characters; audio allows 32 MiB. Rewrite and job timeouts are one and five minutes. Jobs are ephemeral. Failure sends `voice response failed. please try again.` without affecting text; details stay in logs.
 
 ## Command client tools
 
-Telegram provides [`send_image`](tools.md#sending-images-to-telegram) for original PNG/JPEG delivery, plus workspace-selected command tools. Global `clientTools` definitions provide executables; the workspace's nearest `enabledClientTools` selects an exact subset. An empty list disables configured tools, not `send_image`.
+Telegram provides [photo, video, audio, and document delivery tools](tools.md#sending-files-to-telegram), plus workspace-selected command tools. Global `clientTools` definitions provide executables; the workspace's nearest `enabledClientTools` selects an exact subset. An empty list disables configured tools, not built-in delivery tools.
 
 These command processes run on the Telegram runner machine with the runner process environment. They can reach the session workspace only through their explicit execution-environment facade, despite physical co-location. Telegram does not advertise TUI-only `diff_review` or `prefill_input` tools.
 

package/dist/core/static/tau_docs/tools.md

@@ -69,7 +69,7 @@ The tool accepts one exact flat Markdown path. It has no search or list operatio
 index.md
 ```
 
-Then follow paths linked by that page. Unknown paths are rejected. The corpus describes supported Tau contracts, not the current effective configuration of a particular session, so use configuration inspection or debug output when the answer depends on local state.
+Then follow paths linked by that page. The tool description also advertises built-in command-line tools with capability summaries and dedicated documentation paths. Agents may read those pages directly when a tool is relevant, without first loading the index. Use of these utilities is optional and depends on the execution environment's prerequisites. Unknown paths are rejected. The corpus describes supported Tau contracts, not the current effective configuration of a particular session, so use configuration inspection or debug output when the answer depends on local state.
 
 ## Main-session goal tools
 
@@ -133,27 +133,17 @@ Neither tool provides a general read operation. Use a scoped non-interactive Bas
 
 The supported formats are JPEG, PNG, and WebP. Source reads are capped at 50 MiB. Images larger than 2,000 pixels in either dimension or 2.5 MiB of model payload are resized or re-encoded while preserving aspect ratio. If Tau cannot reduce a valid image below the model payload limit, the call fails. Relative paths resolve from the execution-environment working directory.
 
-## Unpack a PDF from the command line
+## Command-line tools
 
-`tau tool pdf-unpack` is a standalone utility for turning a PDF into OCR Markdown and page-image patches. It is not an agent-callable host tool. Run it from the machine that owns the input file:
+`tau tool` provides standalone utilities for people and agents, usable from a shell inside or outside Tau. Files, configuration, credentials, and required executables must be available on the machine running the command. Agents can invoke these commands through Bash.
 
-```bash
-tau tool pdf-unpack ./docs/architecture.pdf
-```
-
-The path is resolved from the command's current working directory and must name a readable file. The command requires `pdftoppm` from Poppler on `PATH`. On macOS, install it with `brew install poppler`; Debian-based Linux distributions provide it through `apt install poppler-utils`.
-
-PDF OCR requires `MISTRAL_API_KEY` or `apiKeys.mistral`, with the environment variable taking precedence. Tau loads configuration for the command's current working directory. The credential and local executable therefore belong to the process running `tau tool`, not to an attached TUI, remote host, or session execution environment unless that is where the command itself runs. See [credentials](credentials.md) for credential ownership.
-
-On success, Tau prints the persistent temporary output directory and a complete artifact list. The directory contains:
-
-- `document.md`, the complete OCR document with recognized tables inlined;
-- `pages/page-0001.md` and later numbered files, one Markdown file per PDF page; and
-- `images/page-0001/patch-0001.png` and later numbered patches for visual verification.
+| Command | Purpose | Guide |
+| --- | --- | --- |
+| `tau tool pdf-unpack` | Extract OCR Markdown and page-image patches from a PDF using Mistral. | [PDF unpacking](pdf-unpacking.md) |
+| `tau tool image-generate` | Generate or edit an image using Google or OpenAI. | [Image generation](image-generation.md) |
+| `tau tool speech-generate` | Generate narration or dialogue using ElevenLabs and assemble a WAV. | [Speech generation](speech-generation.md) |
 
-OCR text can contain recognition mistakes. Embedded visuals that are not represented in Markdown are marked with placeholders pointing to the corresponding page patches. Read `document.md` for the whole document, use `pages/` for page-level work, and inspect `images/` before trusting or correcting uncertain OCR.
-
-The command uploads the PDF to Mistral and attempts to delete the remote upload after OCR. A deletion failure is reported in the command output. Successful local artifacts remain on disk for follow-up use; delete them when they are no longer needed. If processing fails, Tau attempts to remove the partial local output directory. Do not use the command for a sensitive document unless sending it to Mistral and retaining derived local artifacts are both permitted.
+Each guide covers setup, input, examples, outputs, and failure behavior. Use `tau tool --help` to list commands or `tau tool <command> --help` for command-specific help. The same guides are packaged for `tau_docs` and linked from its `index.md`.
 
 ## Code-mode service tools
 
@@ -207,13 +197,24 @@ The TUI advertises `diff_review` and `prefill_input` unless client tools are dis
 
 Configured command client tools are also client-owned. They can run local client processes and use a bounded execution-environment facade when work belongs on the session machine. Remote attachment makes this distinction visible: the TUI process and its tools may be on a laptop while the host and execution environment are elsewhere. See [client tools](client-tools.md) for configuration, protocol helpers, and limits, and [TUI](tui.md) for diff-tool configuration.
 
-### Sending images to Telegram
+### Sending files to Telegram
+
+Telegram provides four client tools, each accepting `{ path, caption? }` and delivering to the current chat:
+
+| Tool | Delivery | Supported files | Maximum size |
+| --- | --- | --- | --- |
+| `send_photo_to_telegram` | In-chat photo | JPEG/PNG; width plus height at most 10,000 pixels, aspect ratio at most 20 | 10,000,000 bytes |
+| `send_video_to_telegram` | Playable video | MPEG4 | 50,000,000 bytes |
+| `send_audio_to_telegram` | Audio player | MP3/M4A | 50,000,000 bytes |
+| `send_document_to_telegram` | Original file attachment | Any type, including PDF, CSV, ZIP, WAV, and original-quality images | 50,000,000 bytes |
+
+Paths may be absolute or relative to the session working directory. Captions are plain text, limited to 1,024 UTF-16 code units. Document delivery preserves original bytes and filenames; photo delivery uses Telegram's native photo processing. File bytes do not enter model context or session history.
 
-The Telegram-only `send_image({ path, caption? })` client tool delivers a PNG or JPEG from the session execution environment to the current chat as a Telegram document. It preserves the original bytes and filename without resizing or recompression. Paths may be absolute or relative to the session working directory; captions are plain text, limited to 1,024 UTF-16 code units.
+Transfers read at most 8,000,000 bytes per execution-environment request and reject non-regular, empty, oversized, or changing files. Tools do not inspect codecs, convert files, or fall back to another delivery method. Telegram validates media compatibility, including photo dimensions. Unsupported media can be explicitly converted before sending or delivered as a document instead.
 
-Images may be up to 50,000,000 bytes. Transfers read at most 8,000,000 bytes per execution-environment request and reject non-regular, empty, oversized, or changing files. Format is detected from file content rather than the extension. Image bytes do not enter model context or session history. Reading, validation, cancellation, and Telegram delivery failures fail the tool call. Calls allow five minutes and are not automatically retried, since an interrupted upload may already have reached Telegram.
+Reading, cancellation, and Telegram delivery failures fail the tool call. Calls allow five minutes and are not automatically retried, since an interrupted upload may already have reached Telegram.
 
-The runner owns upload credentials and binds the recipient to the session's bot and chat; the agent cannot select a recipient. This built-in tool is independent of configured command-tool selection and does not appear in TUI sessions or subagents. Inspecting an image with `view_image` does not send it to Telegram.
+The runner owns upload credentials and binds the recipient to the session's bot and chat; the agent cannot select a recipient. These built-in tools are independent of configured command-tool selection and do not appear in TUI sessions or subagents. Inspecting an image with `view_image` does not send it to Telegram. Automatic voice responses through `/tts_on` are separate from these tools.
 
 ## When a tool is missing or fails