retiam 2.0.2 → 2.0.3

package/agent/AGENTS.md

@@ -32,7 +32,10 @@ and WebGPU APIs.
 - Use Retiam's built-in libraries for their supported features. For generated
   narration, use `retiam/tts`; read the skill's `references/tts.md` and
   `examples/NarratedMotion.ts`. Use another speech generator only when the user
-  requests it. Diagnose native TTS failures instead of silently installing Python
+  requests it. ElevenLabs is a native provider: pass its key to provider.init(),
+  direct expression with model-compatible controls, and keep captions clean.
+  Follow the TTS reference; no ElevenLabs Vite setup is needed. Diagnose failures
+  instead of silently installing Python
   TTS tools or replacing the narration pipeline.
 - Use `retiam/explainer` for prepared speech, named cues, explicit silent pauses
   and synchronized beats. `npm run lesson:check` validates the contract;

package/agent/skill/SKILL.md

@@ -57,10 +57,13 @@ and `retiam/rough` for sketch styling.
 For voiceover, speech, or narration, read [references/tts.md](references/tts.md)
 and [examples/NarratedMotion.ts](examples/NarratedMotion.ts) before authoring.
 That example isolates speech integration; it is not a finished video's visual standard.
-Use `retiam/tts` for synthesis, timing, subtitles, and the scene audio bus. Use a
-different speech generator only when the user requests it. If Retiam TTS fails,
-check its Vite middleware, endpoint, network access, and audio unlock; report the
-actual blocker instead of silently installing Python TTS tools or switching providers.
+Use `retiam/tts` for synthesis, timing, subtitles, and the scene audio bus. Edge
+is the default; when the user chooses ElevenLabs, initialize its native provider
+with the caller's key directly, without Vite configuration. Direct the chosen
+voice for natural questions, discoveries and resolution; keep delivery markup
+out of captions and prepare audio before locking cues. The TTS reference covers
+model-specific expression and prepared assets. Diagnose the selected provider's
+actual failure instead of silently installing Python tools or switching providers.
 
 ## Author for the requested result
 

package/agent/skill/examples/NarratedMotion.ts

@@ -1,7 +1,15 @@
 import {
   CENTER, Circle, Concurrent, Create, CustomAnimation, Line, Scene, Text, Write, hex,
 } from 'retiam';
-import { Speak, Subtitles } from 'retiam/tts';
+import { ElevenLabsTTSProvider, Speak, Subtitles, TTS } from 'retiam/tts';
+
+// Optional application setup before starting the scene. Credentials come from
+// the caller; browser-supplied keys are visible in that browser. No Vite changes.
+export async function initializeElevenLabs(apiKey: string, voiceId: string) {
+  const provider = new ElevenLabsTTSProvider();
+  await provider.init({ apiKey, voiceId, modelId: 'eleven_v3', voiceSettings: { stability: 0.5 } });
+  TTS.setProvider(provider);
+}
 
 export default class NarratedMotion extends Scene {
   async run() {
@@ -19,7 +27,7 @@ export default class NarratedMotion extends Scene {
     this.camera.frame([[-4.5, -2.5, 0], [4.5, 3, 0]], { padding: 0.25 });
 
     const narration = new Speak('Watch the marker move from left to right.', {
-      voice: 'en-US-JennyNeural', speed: 1, subtitles, entity: marker,
+      subtitles, entity: marker,
       pulseAmplitude: 0, volume: 1,
     });
     // Headless tools must wait for synthesis, not a promise requiring timeline steps.

package/agent/skill/references/api.md

@@ -72,6 +72,7 @@ Prefer absolute values derived from progress over accumulating per-frame offsets
 | `retiam/graphs` | Graph and automata entities |
 | `retiam/physics` | Physics worlds, bodies, joints, backend adapters |
 | `retiam/tts` | Speech, narration, and subtitles |
+| `retiam/tts/node` | Renderer-free speech synthesis for Node preparation scripts |
 | `retiam/rough` | Rough styling utilities |
 | `retiam/vite` | Node-side Vite middleware for Retiam TTS; import in Vite config |
 
@@ -79,8 +80,9 @@ Follow the installed sublibrary's index declarations to its options and examples
 before using it. Optional features can have their own asset or environment needs.
 
 For narration, follow [tts.md](tts.md) and the checked
-[NarratedMotion.ts](../examples/NarratedMotion.ts) example. The native TTS route is
-already enabled by new project Vite configurations. Existing projects can add
+[NarratedMotion.ts](../examples/NarratedMotion.ts) example. ElevenLabs initializes
+directly with a key on `ElevenLabsTTSProvider`, without Vite changes. The Edge TTS
+route is already enabled by new project Vite configurations. Existing projects can add
 `msEdgeTTSPlugin` from `retiam/vite`; see the TTS guide for the configuration and
 failure checks. Never import the Vite plugin in browser scene code.
 

package/agent/skill/references/explainers.md

@@ -57,8 +57,10 @@ Do not expose the candidate or subtitle answer during that prediction pause.
 Keep performance notes out of spoken text. A narration segment has an `id`,
 `text`, and optional `pauseAfterMs`. Segments need not correspond one-to-one to
 visual actions. Split at purposeful thinking pauses, retaining coherent speech
-between them. The built-in voice remains the default. Change a provider only when
-requested; additional providers may normalize recordings into `PreparedSpeech`.
+between them. Edge remains the default. When ElevenLabs is selected, use its
+native provider and the initialization/performance guidance in [tts.md](tts.md).
+Keep caption/cue `text` clean; put delivery markup in a segment's `speechText`.
+Other generators may normalize recordings into `PreparedSpeech`.
 Include narration in an explainer unless the user requests a silent treatment.
 The sample's disabled narration is an offline-preview convenience, not a reason
 to omit the requested video's spoken explanation.

package/agent/skill/references/quality.md

@@ -28,6 +28,9 @@ style or palette overrides this default.
   `retiam/tts`. Use conversational questions and precise language, synchronize
   speech with visible events, and allow meaningful silent pauses. Keep requested
   subtitles unobtrusive so they leave room for the mathematical construction.
+  Match vocal delivery to the beat's question, discovery, or resolution. For
+  expressive ElevenLabs delivery follow [tts.md](tts.md): choose model-compatible
+  cues, listen to a representative sample, and preserve clean caption text.
 
 The goal is a clear, satisfying visual argument. Choose 2D or 3D according to
 the concept. Use a slide, dashboard, or another aesthetic when requested.

package/agent/skill/references/tts.md

@@ -1,72 +1,166 @@
-# Native Retiam narration
+# Retiam narration: synthesis and performance
 
-Generate speech for Retiam scenes through `retiam/tts`. It supplies synthesis,
-timing, subtitle synchronization, and playback through the scene's voiceover bus.
-The built-in provider uses Microsoft Edge neural speech over the network. It
-does not require installing Python TTS packages.
+Use `retiam/tts` for speech, timing, subtitles, and the scene voiceover bus.
+Microsoft Edge is the default. Use the native `ElevenLabsTTSProvider` when the
+user selects ElevenLabs and supplies credentials. Preserve the chosen provider;
+diagnose failures rather than silently switching generators or installing Python
+TTS tools. Read the installed declarations before using newer options.
 
-Use [NarratedMotion.ts](../examples/NarratedMotion.ts) as the executable recipe.
-Copy it into `src/scenes/` to preview or render it. Adapt the text, voice, and
-motion to the brief. Use a different speech generator only if the user requests
-one; diagnose failures in this path instead of silently replacing it.
+## ElevenLabs: initialize the provider directly
 
-## Browser and renderer setup
+No ElevenLabs Vite plugin or Vite configuration is needed. For a self-contained
+scene example, put `new ElevenLabsTTSProvider()`, `await provider.init({ apiKey,
+voiceId, ... })`, and `await provider.synthesize(...)` inside `Scene.run()`.
+Pass the key directly to initialization. A `.env`, `process.env`, a preparation
+script, and pre-generated audio are optional workflows, not setup requirements;
+do not introduce them when the user requests a simple in-scene example.
+Register synthesis in `composer.pendingTasks` before awaiting it for headless
+rendering. A provider instance can be used directly; selecting it globally is
+only needed when using the `TTS`, `Speak`, or `speak()` facades.
 
-New projects enable the native proxy in their Vite configuration. For an existing
-project using an engine release that provides `retiam/vite`, preserve its other
-plugins and add:
+To select an initialized provider globally:
 
 ```ts
-import { defineConfig } from 'vite';
-import { msEdgeTTSPlugin } from 'retiam/vite';
+import { ElevenLabsTTSProvider, TTS } from 'retiam/tts';
 
-export default defineConfig({ plugins: [msEdgeTTSPlugin()] });
+export async function useExpressiveVoice(apiKey: string, voiceId: string) {
+  const provider = new ElevenLabsTTSProvider();
+  await provider.init({
+    apiKey, voiceId,
+    modelId: 'eleven_v3',
+    voiceSettings: { stability: 0.5 },
+  });
+  TTS.setProvider(provider);
+  return provider;
+}
 ```
 
-This is a **server-side** import. Scene code imports speech APIs from `retiam/tts`.
-The middleware serves `/api/tts/msedge/synthesize` in Vite development and preview
-servers, including the Vite server started by Retiam's render/snapshot commands.
-A static production host needs an equivalent server endpoint or pre-generated
-audio; bundling the page alone does not host speech synthesis.
+`apiKey` and `voiceId` come from the caller's setup, not literal credentials
+committed to a scene. Browser initialization uses the caller's key directly and
+makes it accessible to that browser. For private preparation use Node's
+`retiam/tts/node` entry point, with the same provider API, and ship the generated
+audio. Do not add ElevenLabs credentials or middleware to `vite.config.ts`.
+Initialization validates local settings without generating audio or fetching
+voices. Use `await provider.fetchVoices()` when choosing a voice;
+`provider.getVoices()` and `TTS.findVoice()` then use that fetched catalog.
+Require a real ElevenLabs voice ID, not an Edge ID or an invented display name.
+
+The default model is `eleven_multilingual_v2`. Select an available expressive
+model such as `eleven_v3` or `eleven_v4` deliberately when directing with audio
+tags. Keep a compatible model/voice pairing throughout a speaker's performance.
+The provider accepts open model IDs, MP3 output formats, model-specific voice
+settings, language, seed, pronunciation dictionaries, and neighboring-text context.
+Per-line overrides go in `elevenlabs`; `speed` can also be supplied at the top
+level. Omitted voice/rate settings inherit provider defaults through `Speak`,
+`speak()` and `TTSNarrationComponent`. `TTS.init()` preserves the selected provider;
+`TTS.configure()` explicitly restores the default Edge client.
+
+## Direct a natural explanation
+
+Choose the emotional action of a beat: inviting a prediction, weighing a
+possibility, making a discovery, or calmly resolving the explanation. Let the
+voice express that action. A constant promotional excitement or a cue on every
+sentence makes an explanation harder to follow. Keep the speaker recognizable
+across beats, with a voice suited to the requested language and audience.
+
+Write speech for listening: concrete questions, contractions, varied sentence
+lengths, and emphasis on the relationship changing on screen. Read technical
+symbols as intended speech; keep formulas and concise notation in the visuals.
+Prefer phrase or paragraph performances to synthesizing isolated words, so the
+model can carry an idea through to its conclusion. Split at genuine explanatory
+boundaries, preserving neighboring context when needed.
+
+Model-specific controls:
+
+- On v3/v4, use occasional delivery tags such as `[curious]` or `[excited]` for
+  a meaningful change in delivery. These models do not support SSML break tags;
+  use punctuation and phrasing for vocal pauses. Test the chosen voice's delivery.
+- On v2 models, use natural punctuation and supported SSML breaks sparingly.
+  Do not assume v3 audio tags work on v2 or Edge. Edge supports its own `pitch`
+  prosody; ElevenLabs rejects nonzero Edge pitch settings.
+- Lower stability can increase variation; higher stability can reduce it.
+  Treat settings as voice/model-dependent. Adjust a short representative sample
+  after listening, rather than maximizing style or copying one preset everywhere.
+- Supported speed is 0.7–1.2. Start close to the voice's natural pace and use
+  explicit scene holds for prediction time. A seed is best-effort, not guaranteed
+  repeatability; approved local audio provides repeatable rendering.
+
+Keep performance directions out of captions. Supply clean `displayText` when
+synthesis text contains delivery or pronunciation markup:
+
+```ts
+import { Speak } from 'retiam/tts';
+
+export function directedLine() {
+  return new Speak('[curious] What changes if we move this point?', {
+    displayText: 'What changes if we move this point?',
+    elevenlabs: { modelId: 'eleven_v3', voiceSettings: { stability: 0.5 } },
+  });
+}
+```
+
+Listen for pronunciation, expression, unwanted vocalizations, rushed endings,
+and pauses before approving a voice. Regenerate only the lines that need it.
+Keep approved audio across visual revisions; settings alone cannot lock a
+performance. Do not spend credits merely to test whether initialization works.
+See the model-specific [ElevenLabs delivery guide](https://elevenlabs.io/docs/overview/capabilities/text-to-speech/best-practices)
+when using an unfamiliar model or control.
+
+## Lessons and prepared speech
+
+For ElevenLabs lessons set `narration.provider` to `elevenlabs`, set `voice` to
+the chosen voice ID, and put model/delivery settings under `narration.elevenlabs`.
+Keep each segment's `text` clean for captions and cue phrases. Optional
+`speechText` contains the exact synthesis script, including delivery tags.
+Keep actual prediction/reveal gaps in `pauseAfterMs`, separate from vocal rhythm.
+
+`retiam lesson prepare` supports both native providers. For ElevenLabs the CLI
+reads `ELEVENLABS_API_KEY` from its environment and passes it to provider.init().
+The key does not go in the lesson JSON. It caches audio and word timings and
+writes `lesson/prepared.json` plus `public/narration/*.mp3`. Reuse those assets
+for subsequent preview and render. Changing delivery text or voice settings
+invalidates the corresponding prepared speech. The runtime can also synthesize
+using an initialized matching global provider or an explicit synthesis adapter.
+
+## Edge browser and renderer setup
+
+Only Edge uses the existing Vite proxy. New projects already enable it. For an
+older project, preserve other plugins and add `msEdgeTTSPlugin` from
+`retiam/vite` on the server side. Its route is `/api/tts/msedge/synthesize` in
+Vite development/preview and Retiam's renderer. Static hosts need that server
+route or prepared audio. Never import the Vite plugin into scene code.
 
 ## Timeline recipe
 
-- Use `Speak` as an animation for simultaneous narration and motion. Set `entity`
-  to an entity already added to the scene so speech uses that scene's audio bus.
-  Call `prepareAudio()` first; its result sets the animation's exact duration.
-- Register the **synthesis promise** in `this.composer.pendingTasks` before
-  awaiting it. Headless tools can then wait for network speech generation without
-  advancing the scene past the narration. Do not register `this.play(...)` or
-  `speak(...)` promises there: they need timeline updates and would deadlock.
-- Set accompanying motion duration from the prepared `Speak.duration`. Avoid
-  guessing speech duration from word counts. Also pass that duration explicitly
-  to `Concurrent` so the group's defaults do not rescale the prepared narration.
-- For sequential narration, `await speak(this, text, options)` synthesizes,
-  plays, synchronizes optional subtitles, and waits for the utterance duration.
-  Pre-synthesize that text with `TTS.synthesize(text, options)` and register only
-  its synthesis promise before headless rendering; the default provider caches
-  the result in memory.
-- `TTS.speak(...)` is a different, lower-level client facade. Its playback result
-  does not itself provide the scene's `wait(...)` pacing.
-- Add `new Subtitles({ fontSize, bottomOffset, maxWidth, ... })` to the scene, and
-  pass it as `subtitles`. Keep the caption area within the shot. Caption timing
-  uses the generated audio metadata.
-- `voice` is a voice ID such as `en-US-JennyNeural`; `speed` is a multiplier
-  such as `1.0`, `pitch` accepts values such as `'+0Hz'`, and playback `volume`
-  is 0–1. Follow installed types for more options.
-
-## Diagnose native TTS failures
-
-1. A missing `retiam/vite` import means the installed engine predates this setup
-   or its build is incomplete. Check `retiam context --json` and the installed
-   package version before changing providers.
-2. HTTP 404 at `/api/tts/msedge/synthesize` means the middleware is absent or the
-   configured endpoint differs. Fix the Vite configuration and restart the server.
-3. HTTP 502 means the proxy ran but synthesis failed. Read the server error;
-   check network access and the requested voice. Do not present silent audio as
-   a successful narration render.
-4. Browser autoplay may require a click or keypress to unlock audio in Studio.
-   The headless CLI supplies its autoplay flag.
-5. For exported video, verify an audio stream exists and listen to the result.
-   Snapshots verify visuals only. Speech generation requires network access on a
-   cache miss; the middleware caches audio in `node_modules/.cache/msedge-tts`.
+Use [NarratedMotion.ts](../examples/NarratedMotion.ts) as the executable
+integration recipe, adapting its speech and visual action to the brief.
+
+- Use `Speak` for simultaneous narration and motion. Set `entity` to an entity
+  already added to the scene, so playback uses that scene's audio bus.
+- Call `prepareAudio()` before locking timings. Register the synthesis promise
+  in `this.composer.pendingTasks` before awaiting it. Do not register playback
+  promises or `this.play(...)`: those require timeline steps and would deadlock.
+- Derive motion duration from prepared `Speak.duration`. Pass that duration to
+  `Concurrent` too, so its defaults do not rescale the narration.
+- `await speak(this, text, options)` handles playback, subtitles, ducking and
+  scene pacing. For headless use, pre-synthesize the same text/options and
+  register only synthesis as pending work. The provider caches that result.
+- `TTS.speak(...)` is a lower-level client facade; it does not pace the scene.
+- Add `Subtitles` and pass it to the narration. Caption text uses the generated
+  audio's `text`; ElevenLabs character alignment is converted to word timings
+  for lesson cues. Missing timing metadata is not a license to guess cue times.
+- `stream()` yields sentence-sized complete audio results, not transport byte
+  chunks. A directed line with `displayText` stays one performance.
+
+## Diagnose and verify
+
+For ElevenLabs, check initialization, voice/model compatibility, API permissions,
+quota/rate-limit errors, and network access. Errors surface without automatic
+paid retries. After disposal the provider requires initialization again. For
+Edge, a 404 means missing middleware/route; a 502 means upstream synthesis
+failed. Inspect the actual error before changing providers.
+
+Browser autoplay may require a user gesture; headless tools supply their own
+autoplay flag. Verify that exported video contains non-silent audio, listen to
+the performance, and check synchronization. A type check or visual snapshot
+does not establish speech quality.

package/bin/lesson.js

@@ -15,6 +15,10 @@ const help = `Usage: retiam lesson <check|prepare|timeline|captions|styles|style
   --out PATH Override the prepared JSON or SRT destination.
   --json     Emit one machine-readable result; diagnostics go to stderr.
 
+ElevenLabs: set narration.provider to "elevenlabs", choose narration.voice,
+and supply ELEVENLABS_API_KEY in the command environment. The CLI passes it
+directly to provider.init(); no Vite configuration is required.
+
 Examples:
   retiam lesson style excalidraw
   retiam lesson check --json
@@ -79,22 +83,45 @@ export async function runLessonCLI(args) {
     if (command === 'prepare') {
         if (!plan.narration.enabled)
             throw Error('Narration is disabled. Set narration.enabled to true before preparing speech.');
-        if ((plan.narration.provider ?? 'msedge') !== 'msedge')
-            throw Error('This CLI prepares the built-in msedge voice. Import a PreparedSpeech manifest for another provider, or use prepareLesson with a synthesize adapter.');
-        const { createServer } = await import('vite'), { msEdgeTTSPlugin } = await import('../dist/libs/vite/index.js');
-        const cache = path.resolve('node_modules/.cache/msedge-tts'), assets = path.resolve('public/narration');
+        const providerName = plan.narration.provider ?? 'msedge';
+        if (!['msedge', 'elevenlabs'].includes(providerName))
+            throw Error('Import a PreparedSpeech manifest for this provider, or use prepareLesson with a synthesize adapter.');
+        const cache = path.resolve(`node_modules/.cache/${providerName}-tts`), assets = path.resolve('public/narration');
         fs.mkdirSync(assets, { recursive: true });
-        const server = await createServer({ configFile: false, logLevel: 'error', plugins: [msEdgeTTSPlugin({ cacheDir: cache, verbose: false })], server: { host: '127.0.0.1', port: 0, open: false, hmr: false, watch: null }, optimizeDeps: { noDiscovery: true } });
+        let server, speechProvider, port;
+        if (providerName === 'elevenlabs') {
+            const { ElevenLabsTTSProvider } = await import('../dist/libs/tts/node/index.js');
+            speechProvider = new ElevenLabsTTSProvider();
+            await speechProvider.init({ ...plan.narration.elevenlabs,
+                apiKey: process.env.ELEVENLABS_API_KEY, voiceId: plan.narration.voice });
+            fs.mkdirSync(cache, { recursive: true });
+        } else {
+            const { createServer } = await import('vite'), { msEdgeTTSPlugin } = await import('../dist/libs/vite/index.js');
+            server = await createServer({ configFile: false, logLevel: 'error', plugins: [msEdgeTTSPlugin({ cacheDir: cache, verbose: false })], server: { host: '127.0.0.1', port: 0, open: false, hmr: false, watch: null }, optimizeDeps: { noDiscovery: true } });
+        }
         prepared = { version: 1, voiceKey: voiceKey(plan), assets: {} };
         try {
-            await server.listen();
-            const port = server.httpServer.address().port;
+            if (server) { await server.listen(); port = server.httpServer.address().port; }
             // Sequential generation avoids provider throttling; cached requests return immediately.
             for (const beat of plan.beats)
                 for (const segment of beat.narration) {
-                    const request = { text: segment.text, voice: plan.narration.voice ?? 'en-US-AndrewNeural', speed: plan.narration.speed ?? 1, pitch: plan.narration.pitch ?? '+0Hz', volume: 1 };
+                    const request = providerName === 'msedge'
+                        ? { text: segment.speechText ?? segment.text, voice: plan.narration.voice ?? 'en-US-AndrewNeural', speed: plan.narration.speed ?? 1, pitch: plan.narration.pitch ?? '+0Hz', volume: 1 }
+                        : { text: segment.speechText ?? segment.text, displayText: segment.text, voice: plan.narration.voice, speed: plan.narration.speed, pitch: plan.narration.pitch, elevenlabs: plan.narration.elevenlabs };
+                    const hash = crypto.createHash('sha256').update(JSON.stringify({ provider: providerName, request })).digest('hex');
                     let result;
-                    for (let attempt = 0; attempt < 3; attempt++) {
+                    if (speechProvider) {
+                        const audioFile = path.join(cache, hash + '.mp3'), metadataFile = path.join(cache, hash + '.json');
+                        if (fs.existsSync(audioFile) && fs.existsSync(metadataFile)) {
+                            result = { ...JSON.parse(fs.readFileSync(metadataFile, 'utf8')), bytes: fs.readFileSync(audioFile) };
+                        } else {
+                            const audio = await speechProvider.synthesize(request.text, request);
+                            result = { bytes: Buffer.from(audio.rawBuffer), durationMs: audio.durationMs, words: audio.wordBoundaries ?? [] };
+                            if (!result.words.length) throw Error('ElevenLabs returned no word timings; narration was not prepared.');
+                            fs.writeFileSync(audioFile, result.bytes);
+                            fs.writeFileSync(metadataFile, JSON.stringify({ durationMs: result.durationMs, words: result.words }));
+                        }
+                    } else for (let attempt = 0; attempt < 3; attempt++) {
                         try {
                             const res = await fetch(`http://127.0.0.1:${port}/api/tts/msedge/synthesize`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(request), signal: AbortSignal.timeout(90000) });
                             if (!res.ok)
@@ -111,13 +138,13 @@ export async function runLessonCLI(args) {
                             console.error(`Retrying ${beat.id}/${segment.id}: ${error.message}`);
                         }
                     }
-                    const hash = crypto.createHash('sha256').update(JSON.stringify(request)).digest('hex');
                     fs.writeFileSync(path.join(assets, hash + '.mp3'), result.bytes);
-                    prepared.assets[speechKey(beat.id, segment.id)] = { text: segment.text, durationMs: result.durationMs, words: result.words, url: `/narration/${hash}.mp3` };
+                    prepared.assets[speechKey(beat.id, segment.id)] = { text: segment.text, speechText: segment.speechText, durationMs: result.durationMs, words: result.words, url: `/narration/${hash}.mp3` };
                 }
         }
         finally {
-            await server.close();
+            await server?.close();
+            await speechProvider?.dispose();
         }
         compileLesson(plan, prepared);
         const dest = path.resolve(out ?? preparedFile);