@startsimpli/llm 0.4.29 → 0.4.30

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -181,9 +181,9 @@ Per CLAUDE.md **rule 9**, LLM client wiring never belongs in app `src/`. If you
181
181
 
182
182
  ## Speech (`@startsimpli/llm/speech`)
183
183
 
184
- Pluggable speech-to-text, same provider pattern as the text service. **It picks the
185
- cheapest vendor that has a key** — each provider states its list price — and falls
186
- through to the next cheapest on failure.
184
+ Pluggable speech-to-text and text-to-speech, same provider pattern as the text service.
185
+ **Each direction picks the cheapest vendor that has a key** — each provider states its
186
+ list price — and falls through to the next cheapest on failure.
187
187
 
188
188
  ```ts
189
189
  import { SpeechService } from '@startsimpli/llm/speech';
@@ -196,11 +196,39 @@ const { text, words } = await speech.transcribe({
196
196
 
197
197
  | Env | Provider | List price |
198
198
  | --- | --- | --- |
199
- | `OPENAI_API_KEY` | OpenAI `gpt-4o-mini-transcribe` (default; `gpt-4o-transcribe` / `whisper-1` cost 2x) | $0.18 / hour |
200
- | `ELEVENLABS_API_KEY` | ElevenLabs Scribe (keyterm hints are a billed add-on, so off unless `sendKeyterms`) | $0.22 / hour |
199
+ | `OPENAI_API_KEY` | OpenAI TTS (`gpt-4o-mini-tts`, no alignment) and `gpt-4o-mini-transcribe` (default; `gpt-4o-transcribe` / `whisper-1` cost 2x) | $0.18 / hour |
200
+ | `ELEVENLABS_API_KEY` | ElevenLabs Scribe + text-to-speech with alignment (`ELEVENLABS_VOICE_ID` = default voice); Scribe (keyterm hints are a billed add-on, so off unless `sendKeyterms`) | $0.22 / hour |
201
201
  | `SPEECH_PROVIDER=mock` | deterministic, for tests | — |
202
202
  | `SPEECH_PROVIDER` / `SPEECH_LANGUAGE` | force a vendor / pin the language (e.g. `en`) so short clips don't mis-detect | — |
203
203
 
204
+ **Text to speech**
205
+
206
+ ```ts
207
+ const { audio, mimeType, words, characters, durationSec, usage } = await speech.synthesize({
208
+ text: 'Plan mode changes that.',
209
+ voiceId, model: 'eleven_v3', outputFormat: 'mp3_44100_128', // voiceId defaults to ELEVENLABS_VOICE_ID
210
+ style: { tags: ['excited'], speed: 1.05, stability: 0.4 }, // tags only on models that read them
211
+ });
212
+ // usage = { characters, costUsd } at the model's list rate
213
+ ```
214
+
215
+ `words` come from the vendor's character alignment (absent when it gives none, as OpenAI's doesn't).
216
+ `[tag]` audio markup is not a word: it is dropped from `words` with the timing offsets kept; `characters`
217
+ is the raw alignment and `keepTags: true` keeps tags in `words`. `voiceSettings` passes raw vendor
218
+ settings and wins over `style`.
219
+
220
+ **CLI for non-TS callers** — `startsimpli-speech` (JSON in via `--in` or stdin, JSON on stdout, audio to a file;
221
+ keys from env only; exit 0 ok, 1 provider error as `{"error":{code,retryable,...}}`, 2 bad command line):
222
+
223
+ ```bash
224
+ echo '{"text":"[excited] Hello.","model":"eleven_v3","style":{"speed":1.0}}' \
225
+ | startsimpli-speech synthesize --out hello.mp3 [--voice ID --model ID --format pcm_22050 --voice-settings '{"stability":0.3}']
226
+ # -> {"mimeType":"audio/mpeg","audioPath":"hello.mp3","bytes":...,"words":[{"text":"Hello.","start":..,"end":..}],"characters":[...],"usage":{"characters":..,"costUsd":..},...}
227
+ echo '{"audioPath":"hello.mp3"}' | startsimpli-speech transcribe # -> {"text":..,"words":[..],"usage":{..}}
228
+ ```
229
+
230
+ It runs the tsup bundle when built (`pnpm --filter @startsimpli/llm build`), else the source via `tsx`.
231
+
204
232
  Retryable errors back off and retry; a failure that survives falls through to the next
205
233
  vendor (`fallback: false` to pin one). Bad audio (`INVALID_REQUEST`) never falls through.
206
234
 
@@ -213,6 +241,7 @@ registerSpeechProvider('mine', () => (process.env.MINE_KEY ? new MyProvider(proc
213
241
  ```
214
242
 
215
243
  `registerLLMProvider(name, factory)` does the same for text. Both sit on `ProviderRegistry`,
244
+ <<<<<<< HEAD
216
245
  so a new modality (TTS, embeddings) reuses it rather than hard-coding a provider list.
217
246
 
218
247
  ## Links
@@ -220,3 +249,6 @@ so a new modality (TTS, embeddings) reuses it rather than hard-coding a provider
220
249
  - Source: https://github.com/qosha1/start-simpli/tree/main/packages/llm
221
250
  - Homepage: https://startsimpli.com
222
251
  - License: MIT, see [LICENSE](./LICENSE)
252
+ =======
253
+ so a new modality (embeddings) reuses it rather than hard-coding a provider list.
254
+ >>>>>>> eda91372 (speech: synthesize() with alignment, ElevenLabs TTS, tags and style, startsimpli-speech CLI)
@@ -0,0 +1,22 @@
1
+ #!/usr/bin/env node
2
+ // Launcher for the speech CLI. Packages here publish TypeScript source, so this
3
+ // runs the tsup bundle (`pnpm build`) when it exists, else the source via tsx.
4
+ const { existsSync } = require('node:fs');
5
+ const { join } = require('node:path');
6
+ const { spawnSync } = require('node:child_process');
7
+
8
+ const root = join(__dirname, '..');
9
+ const bundle = join(root, 'dist', 'speech', 'bin.js');
10
+ const localTsx = join(root, 'node_modules', '.bin', 'tsx');
11
+ const args = process.argv.slice(2);
12
+ if (existsSync(bundle)) {
13
+ process.argv.splice(1, 1, bundle);
14
+ require(bundle);
15
+ } else {
16
+ const tsx = spawnSync(existsSync(localTsx) ? localTsx : 'tsx', [join(root, 'src', 'speech', 'bin.ts'), ...args], { stdio: 'inherit' });
17
+ if (tsx.error) {
18
+ process.stdout.write(JSON.stringify({ error: { code: 'USAGE', message: 'startsimpli-speech needs a build (pnpm --filter @startsimpli/llm build) or tsx on PATH' } }) + '\n');
19
+ process.exit(2);
20
+ }
21
+ process.exit(tsx.status ?? 1);
22
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@startsimpli/llm",
3
- "version": "0.4.29",
3
+ "version": "0.4.30",
4
4
  "description": "Generic LLM service infrastructure — provider abstraction, retry logic, JSON extraction",
5
5
  "license": "MIT",
6
6
  "homepage": "https://startsimpli.com",
@@ -21,6 +21,7 @@
21
21
  },
22
22
  "files": [
23
23
  "src",
24
+ "bin",
24
25
  "README.md",
25
26
  "LICENSE"
26
27
  ],
@@ -29,7 +30,7 @@
29
30
  },
30
31
  "dependencies": {
31
32
  "zod": "^4.3.6",
32
- "@startsimpli/i18n": "^0.1.4"
33
+ "@startsimpli/i18n": "^0.1.3"
33
34
  },
34
35
  "devDependencies": {
35
36
  "@types/node": "^20.19.39",
@@ -37,6 +38,9 @@
37
38
  "typescript": "^6.0.3",
38
39
  "vitest": "^4.1.5"
39
40
  },
41
+ "bin": {
42
+ "startsimpli-speech": "./bin/startsimpli-speech.js"
43
+ },
40
44
  "scripts": {
41
45
  "build": "tsup",
42
46
  "dev": "tsup --watch",
@@ -0,0 +1 @@
1
+ {"text": "[excited] Most people use Claude Code li", "model": "eleven_v3", "alignment": {"characters": ["[", "e", "x", "c", "i", "t", "e", "d", "]", " ", "M", "o", "s", "t", " ", "p", "e", "o", "p", "l", "e", " ", "u", "s", "e", " ", "C", "l", "a", "u", "d", "e", " ", "C", "o", "d", "e", " ", "l", "i"], "character_start_times_seconds": [0.0, 0.048, 0.064, 0.08, 0.096, 0.108, 0.12, 0.132, 0.144, 0.192, 0.202, 0.211, 0.221, 0.23, 0.24, 0.297, 0.354, 0.411, 0.469, 0.526, 0.583, 0.64, 0.72, 0.8, 0.88, 0.96, 1.017, 1.074, 1.131, 1.189, 1.246, 1.303, 1.36, 1.44, 1.52, 1.6, 1.68, 1.76, 1.84, 1.92], "character_end_times_seconds": [0.048, 0.064, 0.08, 0.096, 0.108, 0.12, 0.132, 0.144, 0.192, 0.202, 0.211, 0.221, 0.23, 0.24, 0.297, 0.354, 0.411, 0.469, 0.526, 0.583, 0.64, 0.72, 0.8, 0.88, 0.96, 1.017, 1.074, 1.131, 1.189, 1.246, 1.303, 1.36, 1.44, 1.52, 1.6, 1.68, 1.76, 1.84, 1.92, 2.0]}}
@@ -0,0 +1,67 @@
1
+ import { mkdtemp, readFile, writeFile } from 'node:fs/promises';
2
+ import { tmpdir } from 'node:os';
3
+ import { join } from 'node:path';
4
+ import { afterEach, describe, expect, it, vi } from 'vitest';
5
+
6
+ import { resetMockSpeechBehavior, setMockSpeechBehavior } from '../speech';
7
+ import { runCli } from '../speech/cli';
8
+
9
+ afterEach(() => { resetMockSpeechBehavior(); vi.unstubAllEnvs(); });
10
+
11
+ const run = (argv: string[], stdin = '{}') => {
12
+ vi.stubEnv('SPEECH_PROVIDER', 'mock');
13
+ return runCli(argv, { readStdin: async () => stdin });
14
+ };
15
+ const tmp = () => mkdtemp(join(tmpdir(), 'speech-cli-'));
16
+
17
+ describe('startsimpli-speech', () => {
18
+ it('synthesize: JSON in on stdin, audio to --out, JSON result on stdout', async () => {
19
+ const dir = await tmp();
20
+ const out = join(dir, 'a.mp3');
21
+ const res = await run(['synthesize', '--out', out], JSON.stringify({ text: '[calm] hi you', voiceId: 'v' }));
22
+ expect(res.exitCode).toBe(0);
23
+ expect(res.output).toMatchObject({ audioPath: out, mimeType: 'audio/mpeg', provider: 'mock', usage: { characters: 13 } });
24
+ expect((res.output as any).words.map((w: any) => w.text)).toEqual(['hi', 'you']);
25
+ expect((res.output as any).audio).toBeUndefined();
26
+ expect((await readFile(out)).byteLength).toBe((res.output as any).bytes);
27
+ });
28
+
29
+ it('synthesize: --in file and flags override the request', async () => {
30
+ const dir = await tmp();
31
+ const input = join(dir, 'req.json');
32
+ await writeFile(input, JSON.stringify({ text: 'hi', voiceId: 'from-file' }));
33
+ const res = await run(['synthesize', '--in', input, '--out', join(dir, 'o.mp3'), '--voice', 'from-flag', '--voice-settings', '{"stability":0.2}']);
34
+ expect(res.exitCode).toBe(0);
35
+ expect((res.output as any).voiceId).toBe('from-flag');
36
+ });
37
+
38
+ it('transcribe: reads audioPath, infers the mime type, returns the transcript', async () => {
39
+ const dir = await tmp();
40
+ const audio = join(dir, 'n.m4a');
41
+ await writeFile(audio, new Uint8Array([1, 2, 3]));
42
+ setMockSpeechBehavior({ text: 'hello' });
43
+ const res = await run(['transcribe'], JSON.stringify({ audioPath: audio }));
44
+ expect(res).toMatchObject({ exitCode: 0, output: { text: 'hello', provider: 'mock' } });
45
+ });
46
+
47
+ it('a provider failure is JSON on stdout with exit 1 and the typed code', async () => {
48
+ setMockSpeechBehavior({ authError: true });
49
+ const res = await run(['synthesize', '--out', join(await tmp(), 'x.mp3')], '{"text":"hi"}');
50
+ expect(res).toMatchObject({ exitCode: 1, output: { error: { code: 'AUTHENTICATION_ERROR', retryable: false, statusCode: 401 } } });
51
+ });
52
+
53
+ it.each([
54
+ [[], '{}', /usage/],
55
+ [['sing'], '{}', /usage/],
56
+ [['synthesize'], '{"text":"hi"}', /--out/],
57
+ [['synthesize', '--out'], '{}', /needs a value/],
58
+ [['synthesize', '--bogus', 'x'], '{}', /unknown argument/],
59
+ [['synthesize', '--out', 'x'], 'not json', /JSON object/],
60
+ [['transcribe'], '{}', /audioPath/],
61
+ [['synthesize', '--out', 'x', '--voice-settings', '[1]'], '{"text":"hi"}', /voice-settings/],
62
+ ])('malformed command %j exits 2 with a message', async (argv, stdin, message) => {
63
+ const res = await run(argv as string[], stdin);
64
+ expect(res.exitCode).toBe(2);
65
+ expect((res.output as any).error.message).toMatch(message);
66
+ });
67
+ });
@@ -0,0 +1,277 @@
1
+ import { readFileSync } from 'node:fs';
2
+ import { join } from 'node:path';
3
+ import { afterEach, describe, expect, it, vi } from 'vitest';
4
+
5
+ import { LLMProviderError } from '../errors';
6
+ import {
7
+ ElevenLabsSpeechProvider, MockSpeechProvider, OpenAISpeechProvider, SpeechService,
8
+ createElevenLabsSpeechProvider, resetMockSpeechBehavior, setMockSpeechBehavior, wordsFromCharacters,
9
+ } from '../speech';
10
+ import type { ISpeechProvider, SynthesizeRequest } from '../speech';
11
+
12
+ afterEach(() => { resetMockSpeechBehavior(); vi.unstubAllEnvs(); });
13
+
14
+ const req: SynthesizeRequest = { text: 'hi you', voiceId: 'v1' };
15
+ const AUDIO = new Uint8Array([9, 8, 7, 6]);
16
+ const b64 = Buffer.from(AUDIO).toString('base64');
17
+
18
+ /** ElevenLabs with-timestamps body for `hi you`. */
19
+ const elevenBody = (text = 'hi you') => {
20
+ const chars = [...text];
21
+ return {
22
+ audio_base64: b64,
23
+ alignment: {
24
+ characters: chars,
25
+ character_start_times_seconds: chars.map((_, i) => i * 0.1),
26
+ character_end_times_seconds: chars.map((_, i) => (i + 1) * 0.1),
27
+ },
28
+ };
29
+ };
30
+
31
+ const respond = (body: unknown | Uint8Array, status = 200) =>
32
+ vi.fn(async (_u: unknown, _i?: RequestInit) =>
33
+ new Response(body instanceof Uint8Array ? body : typeof body === 'string' ? body : JSON.stringify(body), { status }));
34
+
35
+ const wires: Array<{ name: string; make: (f: typeof fetch) => ISpeechProvider; ok: unknown; aligned: boolean; mime: string }> = [
36
+ { name: 'elevenlabs', make: (fetchImpl) => new ElevenLabsSpeechProvider({ apiKey: 'k', fetchImpl }), ok: elevenBody(), aligned: true, mime: 'audio/mpeg' },
37
+ { name: 'openai', make: (fetchImpl) => new OpenAISpeechProvider({ apiKey: 'k', fetchImpl }), ok: AUDIO, aligned: false, mime: 'audio/mpeg' },
38
+ ];
39
+
40
+ describe.each(wires)('synthesize contract: $name', ({ name, make, ok, aligned, mime }) => {
41
+ it('advertises the capability', () => {
42
+ expect(make(fetch).capabilities.synthesize).toBe(true);
43
+ });
44
+
45
+ it('returns audio bytes, mime, provider, model and character usage', async () => {
46
+ const out = await make(respond(ok) as never).synthesize!(req);
47
+ expect([...out.audio]).toEqual([...AUDIO]);
48
+ expect(out.mimeType).toBe(mime);
49
+ expect(out).toMatchObject({ provider: name });
50
+ expect(out.model).toBeTruthy();
51
+ expect(out.usage.characters).toBe(req.text.length);
52
+ });
53
+
54
+ it(aligned ? 'returns ordered, non-overlapping word timings' : 'has no alignment (the vendor gives none)', async () => {
55
+ const out = await make(respond(ok) as never).synthesize!(req);
56
+ if (!aligned) { expect(out.words).toBeUndefined(); return; }
57
+ expect(out.words?.map((w) => w.text)).toEqual(['hi', 'you']);
58
+ const [a, b] = out.words!;
59
+ expect(a.end).toBeLessThanOrEqual(b.start);
60
+ expect(out.durationSec).toBeGreaterThan(0);
61
+ });
62
+
63
+ it.each([[401, 'AUTHENTICATION_ERROR', false], [429, 'RATE_LIMITED', true], [503, 'SERVICE_UNAVAILABLE', true], [400, 'INVALID_REQUEST', false]])(
64
+ 'classifies HTTP %i as %s (retryable=%s)', async (status, code, retryable) => {
65
+ const err = await make(respond('boom', status as number) as never).synthesize!(req).catch((e) => e);
66
+ expect(err).toBeInstanceOf(LLMProviderError);
67
+ expect(err).toMatchObject({ code, retryable, provider: name });
68
+ expect(err.message).not.toMatch(/transcription/);
69
+ });
70
+
71
+ it('refuses empty text without calling the vendor', async () => {
72
+ const f = respond(ok);
73
+ await expect(make(f as never).synthesize!({ ...req, text: ' ' })).rejects.toMatchObject({ code: 'INVALID_REQUEST' });
74
+ expect(f).not.toHaveBeenCalled();
75
+ });
76
+
77
+ it('turns a network failure into a retryable error', async () => {
78
+ const f = vi.fn(async () => { throw new TypeError('fetch failed'); });
79
+ await expect(make(f as never).synthesize!(req)).rejects.toMatchObject({ retryable: true, code: 'SERVICE_UNAVAILABLE' });
80
+ });
81
+ });
82
+
83
+ describe('elevenlabs synthesize specifics', () => {
84
+ const run = async (patch: Partial<SynthesizeRequest> = {}, body: unknown = elevenBody()) => {
85
+ const f = respond(body);
86
+ const out = await new ElevenLabsSpeechProvider({ apiKey: 'sekret', fetchImpl: f as never }).synthesize({ ...req, ...patch });
87
+ const [url, init] = f.mock.calls[0] as unknown as [string, RequestInit];
88
+ return { out, url, init, sent: JSON.parse(init.body as string) };
89
+ };
90
+
91
+ it('posts to with-timestamps with the key header and voice in the path', async () => {
92
+ const { url, init } = await run();
93
+ expect(url).toBe('https://api.elevenlabs.io/v1/text-to-speech/v1/with-timestamps?output_format=mp3_44100_128');
94
+ expect((init.headers as Record<string, string>)['xi-api-key']).toBe('sekret');
95
+ });
96
+
97
+ it('takes the voice from the call, else ELEVENLABS_VOICE_ID, else refuses', async () => {
98
+ const p = createElevenLabsSpeechProvider({ ELEVENLABS_API_KEY: 'k', ELEVENLABS_VOICE_ID: 'envvoice' })!;
99
+ const f = respond(elevenBody());
100
+ (p as any).fetchImpl = f;
101
+ await p.synthesize({ text: 'hi you' });
102
+ expect((f.mock.calls[0] as unknown as [string])[0]).toContain('/text-to-speech/envvoice/with-timestamps');
103
+ const bare = new ElevenLabsSpeechProvider({ apiKey: 'k', fetchImpl: respond(elevenBody()) as never });
104
+ await expect(bare.synthesize({ text: 'hi' })).rejects.toMatchObject({ code: 'INVALID_REQUEST' });
105
+ });
106
+
107
+ it('maps outputFormat to the mime type', async () => {
108
+ expect((await run({ outputFormat: 'pcm_22050' })).out.mimeType).toBe('audio/pcm');
109
+ expect((await run({ outputFormat: 'pcm_22050' })).url).toContain('output_format=pcm_22050');
110
+ });
111
+
112
+ it('passes model, language and voiceSettings through', async () => {
113
+ const { sent } = await run({ model: 'eleven_v3', language: 'en', voiceSettings: { stability: 0.5, similarity_boost: 0.7 } });
114
+ expect(sent).toMatchObject({ text: 'hi you', model_id: 'eleven_v3', language_code: 'en', voice_settings: { stability: 0.5, similarity_boost: 0.7 } });
115
+ });
116
+
117
+ it('typed style speed/stability become voice_settings; explicit voiceSettings win', async () => {
118
+ const { sent } = await run({ style: { speed: 1.1, stability: 0.3 }, voiceSettings: { stability: 0.9 } });
119
+ expect(sent.voice_settings).toEqual({ speed: 1.1, stability: 0.9 });
120
+ });
121
+
122
+ it('style tags are prefixed on models that read them', async () => {
123
+ const { sent, out } = await run({ model: 'eleven_v3', style: { tags: ['excited', 'whispers'] } },
124
+ elevenBody('[excited][whispers] hi you'));
125
+ expect(sent.text).toBe('[excited][whispers] hi you');
126
+ expect(out.usage.characters).toBe('[excited][whispers] hi you'.length);
127
+ });
128
+
129
+ it('style tags are dropped on models that would speak them aloud', async () => {
130
+ const { sent } = await run({ model: 'eleven_multilingual_v2', style: { tags: ['excited'] } });
131
+ expect(sent.text).toBe('hi you');
132
+ });
133
+
134
+ it('audio tags are not words: stripped from the words, offsets kept, characters untouched', async () => {
135
+ const text = '[excited] hi you';
136
+ const { out } = await run({ model: 'eleven_v3', text }, elevenBody(text));
137
+ expect(out.words?.map((w) => w.text)).toEqual(['hi', 'you']);
138
+ expect(out.words![0].start).toBeCloseTo(1.0); // "[excited] " is 10 chars at 0.1s
139
+ expect(out.characters?.length).toBe(text.length);
140
+ });
141
+
142
+ it('keepTags:true leaves tags in the words', async () => {
143
+ const text = '[excited] hi you';
144
+ const { out } = await run({ model: 'eleven_v3', text, keepTags: true }, elevenBody(text));
145
+ expect(out.words?.map((w) => w.text)).toEqual(['[excited]', 'hi', 'you']);
146
+ });
147
+
148
+ it('cost follows the model rate table (USD per 1k characters)', async () => {
149
+ const text = 'x'.repeat(1000);
150
+ expect((await run({ model: 'eleven_v3', text }, elevenBody(text))).out.usage.costUsd).toBeCloseTo(0.08);
151
+ expect((await run({ model: 'eleven_multilingual_v2', text }, elevenBody(text))).out.usage.costUsd).toBeCloseTo(0.08);
152
+ expect((await run({ model: 'eleven_flash_v2_5', text }, elevenBody(text))).out.usage.costUsd).toBeCloseTo(0.04);
153
+ expect((await run({ model: 'some_future_model', text }, elevenBody(text))).out.usage.costUsd).toBeCloseTo(0.10);
154
+ });
155
+
156
+ it('a response with no alignment still returns audio', async () => {
157
+ const { out } = await run({}, { audio_base64: b64 });
158
+ expect(out.words).toBeUndefined();
159
+ expect([...out.audio]).toEqual([...AUDIO]);
160
+ });
161
+
162
+ it('openai: posts the voice and format to /v1/audio/speech', async () => {
163
+ const f = respond(AUDIO);
164
+ await new OpenAISpeechProvider({ apiKey: 'k', fetchImpl: f as never }).synthesize({ text: 'hi' });
165
+ const [url, init] = f.mock.calls[0] as unknown as [string, RequestInit];
166
+ expect(url).toBe('https://api.openai.com/v1/audio/speech');
167
+ expect(JSON.parse(init.body as string)).toMatchObject({ input: 'hi', response_format: 'mp3' });
168
+ });
169
+ });
170
+
171
+ describe('wordsFromCharacters', () => {
172
+ it('splits on whitespace and skips bracket tags by default', () => {
173
+ const chars = [...'[calm] a b'].map((text, i) => ({ text, start: i, end: i + 1 }));
174
+ expect(wordsFromCharacters(chars).map((w) => [w.text, w.start])).toEqual([['a', 7], ['b', 9]]);
175
+ expect(wordsFromCharacters(chars, { keepTags: true }).map((w) => w.text)).toEqual(['[calm]', 'a', 'b']);
176
+ });
177
+ it('leaves an unclosed bracket as ordinary text', () => {
178
+ const chars = [...'a [b'].map((text, i) => ({ text, start: i, end: i + 1 }));
179
+ expect(wordsFromCharacters(chars).map((w) => w.text)).toEqual(['a', '[b']);
180
+ });
181
+ });
182
+
183
+ describe('real v3 alignment fixture', () => {
184
+ it('drops the [excited] tag from the words but keeps the true offsets', () => {
185
+ const fx = JSON.parse(readFileSync(join(__dirname, 'fixtures', 'eleven-v3-tagged-alignment.json'), 'utf8'));
186
+ const chars = fx.alignment.characters.map((text: string, i: number) => ({
187
+ text, start: fx.alignment.character_start_times_seconds[i], end: fx.alignment.character_end_times_seconds[i] }));
188
+ const words = wordsFromCharacters(chars);
189
+ expect(words[0].text).toBe('Most');
190
+ expect(words[0].start).toBeCloseTo(0.202, 2); // the tag took the first ~0.2s
191
+ expect(wordsFromCharacters(chars, { keepTags: true })[0].text).toBe('[excited]');
192
+ });
193
+ });
194
+
195
+ describe('mock synthesize', () => {
196
+ it('is deterministic with aligned words and usage', async () => {
197
+ const out = await new MockSpeechProvider().synthesize(req);
198
+ expect(out.audio.byteLength).toBeGreaterThan(0);
199
+ expect(out.words?.map((w) => w.text)).toEqual(['hi', 'you']);
200
+ expect(out.usage.characters).toBe(6);
201
+ expect(out.provider).toBe('mock');
202
+ });
203
+ it('refuses empty text and simulates auth and outage failures', async () => {
204
+ const mock = new MockSpeechProvider();
205
+ await expect(mock.synthesize({ text: '' })).rejects.toMatchObject({ code: 'INVALID_REQUEST' });
206
+ setMockSpeechBehavior({ authError: true });
207
+ await expect(mock.synthesize(req)).rejects.toMatchObject({ code: 'AUTHENTICATION_ERROR', retryable: false });
208
+ setMockSpeechBehavior({ failTimes: 1 });
209
+ await expect(mock.synthesize(req)).rejects.toMatchObject({ code: 'SERVICE_UNAVAILABLE', retryable: true });
210
+ });
211
+ });
212
+
213
+ describe('SpeechService.synthesize', () => {
214
+ const fake = (name: string, impl: () => Promise<any>, synth = true): ISpeechProvider => ({
215
+ name, capabilities: { transcribe: true, synthesize: synth }, defaultModel: 'm', isAvailable: () => true,
216
+ transcribe: vi.fn() as never, synthesize: vi.fn(impl) as never,
217
+ });
218
+ const withProviders = (service: SpeechService, ...ps: ISpeechProvider[]) => {
219
+ (service as any).providers = new Map(ps.map((p) => [p.name, p]));
220
+ return service;
221
+ };
222
+ const ok = (provider: string) => async () => ({ audio: AUDIO, mimeType: 'audio/mpeg', model: 'm', provider, usage: { characters: 6 } });
223
+ const outage = async () => { throw new LLMProviderError('down', 'SERVICE_UNAVAILABLE', true, 503); };
224
+
225
+ it('only considers providers that can synthesize', async () => {
226
+ const s = withProviders(new SpeechService(), fake('elevenlabs', ok('elevenlabs'), false), fake('openai', ok('openai')));
227
+ expect((await s.synthesize(req)).provider).toBe('openai');
228
+ expect(s.getAvailableProviders('synthesize')).toEqual(['openai']);
229
+ });
230
+ it('picks the cheapest vendor that states a price, then falls back to the dearer one', async () => {
231
+ const priced = (name: string, price: number, impl: () => Promise<any>) => ({ ...fake(name, impl), costPer1kCharsUsd: () => price });
232
+ const s = withProviders(new SpeechService({ retryDelayMs: 1, maxRetries: 0 }),
233
+ priced('elevenlabs', 0.08, ok('elevenlabs')), priced('openai', 0.015, outage));
234
+ expect((await s.synthesize(req)).provider).toBe('elevenlabs'); // openai (cheapest) failed, fell through
235
+ const cheap = withProviders(new SpeechService(), priced('elevenlabs', 0.08, ok('elevenlabs')), priced('openai', 0.015, ok('openai')));
236
+ expect((await cheap.synthesize(req)).provider).toBe('openai');
237
+ const pinned = withProviders(new SpeechService({ preferredProvider: 'elevenlabs' }), priced('elevenlabs', 0.08, ok('elevenlabs')), priced('openai', 0.015, ok('openai')));
238
+ expect((await pinned.synthesize(req)).provider).toBe('elevenlabs');
239
+ });
240
+ it('a provider without a key is unavailable, not an error', () => {
241
+ const none = new OpenAISpeechProvider({ apiKey: '' });
242
+ expect(none.isAvailable()).toBe(false);
243
+ expect(withProviders(new SpeechService(), none).getAvailableProviders('synthesize')).toEqual([]);
244
+ });
245
+ it('retries a retryable failure, then falls through', async () => {
246
+ const primary = fake('elevenlabs', outage);
247
+ const s = withProviders(new SpeechService({ retryDelayMs: 1, maxRetries: 2 }), primary, fake('openai', ok('openai')));
248
+ expect((await s.synthesize(req)).provider).toBe('openai');
249
+ expect(primary.synthesize).toHaveBeenCalledTimes(3);
250
+ });
251
+ it('never retries an auth error', async () => {
252
+ const bad = fake('elevenlabs', async () => { throw new LLMProviderError('no', 'AUTHENTICATION_ERROR', false, 401); });
253
+ const s = withProviders(new SpeechService({ retryDelayMs: 1 }), bad, fake('openai', ok('openai')));
254
+ await s.synthesize(req);
255
+ expect(bad.synthesize).toHaveBeenCalledTimes(1);
256
+ });
257
+ it('does not fall through on the caller\'s bad input', async () => {
258
+ const bad = fake('elevenlabs', async () => { throw new LLMProviderError('bad', 'INVALID_REQUEST', false, 400); });
259
+ const next = fake('openai', ok('openai'));
260
+ await expect(withProviders(new SpeechService(), bad, next).synthesize(req)).rejects.toMatchObject({ code: 'INVALID_REQUEST' });
261
+ expect(next.synthesize).not.toHaveBeenCalled();
262
+ });
263
+ it('says what to set when nothing can synthesize', async () => {
264
+ await expect(withProviders(new SpeechService()).synthesize(req)).rejects.toThrow(/ELEVENLABS_API_KEY/);
265
+ });
266
+ it('SPEECH_PROVIDER=mock synthesizes, and survives a cleared outage', async () => {
267
+ vi.stubEnv('SPEECH_PROVIDER', 'mock');
268
+ setMockSpeechBehavior({ failTimes: 2 });
269
+ const out = await new SpeechService({ retryDelayMs: 1 }).synthesize(req);
270
+ expect(out.provider).toBe('mock');
271
+ });
272
+ it('transcribe is untouched by synthesize-only providers', async () => {
273
+ const synthOnly = { ...fake('elevenlabs', ok('elevenlabs')), capabilities: { transcribe: false, synthesize: true } } as ISpeechProvider;
274
+ const s = withProviders(new SpeechService(), synthOnly);
275
+ await expect(s.transcribe({ audio: AUDIO, mimeType: 'audio/mp4' })).rejects.toThrow(/No speech providers/);
276
+ });
277
+ });
@@ -0,0 +1,48 @@
1
+ import type { AlignedCharacter, TranscriptWord } from './types';
2
+
3
+ /**
4
+ * Character alignment -> word timings, split on whitespace.
5
+ *
6
+ * `[tag]` spans (v3-style audio tags such as `[excited]`) are delivery
7
+ * directions the vendor aligns like speech but nobody hears as words, so they
8
+ * are skipped by default. Skipping removes the characters, not the time: the
9
+ * next word keeps its true offset, so a ~0.2s tag leaves a gap rather than
10
+ * shifting everything left. A `[` with no closing `]` is ordinary text.
11
+ */
12
+ export function wordsFromCharacters(
13
+ characters: AlignedCharacter[],
14
+ options: { keepTags?: boolean } = {},
15
+ ): TranscriptWord[] {
16
+ const words: TranscriptWord[] = [];
17
+ let text = '';
18
+ let start = 0;
19
+ let end = 0;
20
+ const flush = () => {
21
+ if (text) words.push({ text, start: round(start), end: round(end) });
22
+ text = '';
23
+ };
24
+ for (let i = 0; i < characters.length; i += 1) {
25
+ const c = characters[i];
26
+ if (!options.keepTags && c.text === '[' && !text) {
27
+ const close = closingBracket(characters, i);
28
+ if (close > 0) { i = close; continue; }
29
+ }
30
+ if (/\s/.test(c.text)) { flush(); continue; }
31
+ if (!text) start = c.start;
32
+ text += c.text;
33
+ end = c.end;
34
+ }
35
+ flush();
36
+ return words;
37
+ }
38
+
39
+ /** Index of the `]` closing a tag opened at `open`, or -1 (no close, or whitespace-only run is not a tag). */
40
+ function closingBracket(characters: AlignedCharacter[], open: number): number {
41
+ for (let j = open + 1; j < characters.length; j += 1) {
42
+ if (characters[j].text === ']') return j > open + 1 ? j : -1;
43
+ if (characters[j].text === '[') return -1;
44
+ }
45
+ return -1;
46
+ }
47
+
48
+ const round = (n: number) => Math.round(n * 1000) / 1000;
@@ -0,0 +1,12 @@
1
+ import { runCli } from './cli';
2
+
3
+ async function readStdin(): Promise<string> {
4
+ const chunks: Buffer[] = [];
5
+ for await (const chunk of process.stdin) chunks.push(Buffer.from(chunk));
6
+ return Buffer.concat(chunks).toString('utf8');
7
+ }
8
+
9
+ runCli(process.argv.slice(2), { readStdin }).then(({ exitCode, output }) => {
10
+ process.stdout.write(`${JSON.stringify(output)}\n`);
11
+ process.exitCode = exitCode;
12
+ });
@@ -0,0 +1,129 @@
1
+ /**
2
+ * `startsimpli-speech` — the speech service for callers that are not TypeScript.
3
+ *
4
+ * startsimpli-speech synthesize --in request.json --out speech.mp3 > result.json
5
+ * startsimpli-speech transcribe --in request.json > result.json
6
+ *
7
+ * JSON in (a file with --in, else stdin), JSON out on stdout, audio as a file.
8
+ * Keys come from the environment only (ELEVENLABS_API_KEY, ELEVENLABS_VOICE_ID,
9
+ * OPENAI_API_KEY, SPEECH_PROVIDER). Nothing here knows who is calling.
10
+ *
11
+ * Failures print `{"error": {code, message, retryable, statusCode, provider}}`
12
+ * and exit 1; a malformed command line exits 2. Success exits 0.
13
+ */
14
+ import { readFile, writeFile } from 'node:fs/promises';
15
+ import { extname } from 'node:path';
16
+
17
+ import { LLMProviderError } from '../errors';
18
+ import { SpeechService } from './service';
19
+ import type { SynthesizeRequest, TranscribeRequest } from './types';
20
+
21
+ export interface CliIO {
22
+ /** Reads all of stdin; only called when --in is absent. */
23
+ readStdin: () => Promise<string>;
24
+ /** Builds the service; a seam for tests. */
25
+ makeService?: () => SpeechService;
26
+ }
27
+
28
+ export interface CliOutcome {
29
+ exitCode: 0 | 1 | 2;
30
+ /** JSON document for stdout. */
31
+ output: unknown;
32
+ }
33
+
34
+ const MIME_BY_EXT: Record<string, string> = {
35
+ '.webm': 'audio/webm', '.ogg': 'audio/ogg', '.mp4': 'audio/mp4', '.m4a': 'audio/mp4', '.mp3': 'audio/mpeg',
36
+ '.wav': 'audio/wav', '.flac': 'audio/flac',
37
+ };
38
+
39
+ const FLAGS = new Set(['in', 'out', 'provider', 'model', 'voice', 'format', 'voice-settings', 'language']);
40
+
41
+ const USAGE = 'usage: startsimpli-speech <synthesize|transcribe> [--in request.json] [--out audio-file] '
42
+ + '[--provider NAME] [--model ID] [--voice ID] [--format FMT] [--voice-settings JSON] [--language CODE]';
43
+
44
+ class UsageError extends Error {}
45
+
46
+ function parseFlags(args: string[]): Record<string, string> {
47
+ const flags: Record<string, string> = {};
48
+ for (let i = 0; i < args.length; i += 1) {
49
+ const name = args[i].startsWith('--') ? args[i].slice(2) : '';
50
+ if (!FLAGS.has(name)) throw new UsageError(`unknown argument "${args[i]}"`);
51
+ const value = args[i + 1];
52
+ if (value === undefined) throw new UsageError(`--${name} needs a value`);
53
+ flags[name] = value;
54
+ i += 1;
55
+ }
56
+ return flags;
57
+ }
58
+
59
+ async function readRequest(flags: Record<string, string>, io: CliIO): Promise<Record<string, any>> {
60
+ const raw = flags.in ? await readFile(flags.in, 'utf8') : await io.readStdin();
61
+ try {
62
+ const parsed = JSON.parse(raw || '{}');
63
+ if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) throw new Error('not an object');
64
+ return parsed;
65
+ } catch (error) {
66
+ throw new UsageError(`request is not a JSON object: ${(error as Error).message}`);
67
+ }
68
+ }
69
+
70
+ export async function runCli(argv: string[], io: CliIO): Promise<CliOutcome> {
71
+ try {
72
+ const [command, ...rest] = argv;
73
+ if (command !== 'synthesize' && command !== 'transcribe') throw new UsageError(USAGE);
74
+ const flags = parseFlags(rest);
75
+ const body = await readRequest(flags, io);
76
+ const service = io.makeService ? io.makeService() : new SpeechService(flags.provider ? { preferredProvider: flags.provider } : {});
77
+ return { exitCode: 0, output: command === 'synthesize' ? await synthesize(service, body, flags) : await transcribe(service, body, flags) };
78
+ } catch (error) {
79
+ if (error instanceof UsageError) return { exitCode: 2, output: { error: { code: 'USAGE', message: error.message } } };
80
+ if (error instanceof LLMProviderError) {
81
+ const { code, message, retryable, statusCode, provider } = error;
82
+ return { exitCode: 1, output: { error: { code, message, retryable, statusCode, provider } } };
83
+ }
84
+ return { exitCode: 1, output: { error: { code: 'UNKNOWN_ERROR', message: (error as Error)?.message ?? String(error), retryable: false } } };
85
+ }
86
+ }
87
+
88
+ async function synthesize(service: SpeechService, body: Record<string, any>, flags: Record<string, string>) {
89
+ const out: string | undefined = flags.out ?? body.out;
90
+ if (!out) throw new UsageError('synthesize needs --out <audio file> (or "out" in the request)');
91
+ const request: SynthesizeRequest = {
92
+ ...body,
93
+ ...(flags.model ? { model: flags.model } : {}),
94
+ ...(flags.voice ? { voiceId: flags.voice } : {}),
95
+ ...(flags.format ? { outputFormat: flags.format } : {}),
96
+ ...(flags.language ? { language: flags.language } : {}),
97
+ ...(flags['voice-settings'] ? { voiceSettings: parseSettings(flags['voice-settings']) } : {}),
98
+ } as SynthesizeRequest;
99
+ delete (request as any).out;
100
+ delete (request as any).provider;
101
+ const result = await service.synthesize(request);
102
+ await writeFile(out, result.audio);
103
+ const { audio, ...rest } = result;
104
+ return { ...rest, audioPath: out, bytes: audio.byteLength };
105
+ }
106
+
107
+ function parseSettings(text: string): Record<string, number | boolean> {
108
+ try {
109
+ const parsed = JSON.parse(text);
110
+ if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) return parsed;
111
+ } catch { /* fall through */ }
112
+ throw new UsageError('--voice-settings must be a JSON object, e.g. \'{"stability":0.4}\'');
113
+ }
114
+
115
+ async function transcribe(service: SpeechService, body: Record<string, any>, flags: Record<string, string>) {
116
+ const audioPath: string | undefined = body.audioPath;
117
+ if (!audioPath) throw new UsageError('transcribe needs "audioPath" in the request');
118
+ const audio = new Uint8Array(await readFile(audioPath));
119
+ const { audioPath: _drop, provider: _p, ...rest } = body;
120
+ const request: TranscribeRequest = {
121
+ timestamps: 'word',
122
+ ...rest,
123
+ audio,
124
+ mimeType: body.mimeType ?? MIME_BY_EXT[extname(audioPath).toLowerCase()] ?? 'audio/mpeg',
125
+ ...(flags.model ? { model: flags.model } : {}),
126
+ ...(flags.language ? { language: flags.language } : {}),
127
+ };
128
+ return service.transcribe(request);
129
+ }
@@ -1,6 +1,6 @@
1
1
  import { LLMProviderError } from '../errors';
2
2
  import type { LLMErrorCode } from '../errors';
3
- import type { TranscribeRequest } from './types';
3
+ import type { SynthesizeRequest, TranscribeRequest } from './types';
4
4
 
5
5
  /** Container extension a vendor can recognise, from the recorder's MIME type. */
6
6
  const EXTENSIONS: Record<string, string> = {
@@ -34,8 +34,14 @@ export function assertAudio(request: Pick<TranscribeRequest, 'audio'>, provider:
34
34
  }
35
35
  }
36
36
 
37
+ export function assertText(request: Pick<SynthesizeRequest, 'text'>, provider: string): void {
38
+ if (!request.text?.trim()) {
39
+ throw new LLMProviderError('There is no text to speak.', 'INVALID_REQUEST', false, undefined, provider);
40
+ }
41
+ }
42
+
37
43
  /** One HTTP-status -> error-code mapping, so every vendor classifies failures the same way. */
38
- export function speechHttpError(status: number, body: string, provider: string): LLMProviderError {
44
+ export function speechHttpError(status: number, body: string, provider: string, operation = 'transcription'): LLMProviderError {
39
45
  const code: LLMErrorCode =
40
46
  status === 401 || status === 403 ? 'AUTHENTICATION_ERROR'
41
47
  : status === 429 ? 'RATE_LIMITED'
@@ -45,7 +51,7 @@ export function speechHttpError(status: number, body: string, provider: string):
45
51
  : 'UNKNOWN_ERROR';
46
52
  const retryable = code === 'RATE_LIMITED' || code === 'SERVICE_UNAVAILABLE' || code === 'TIMEOUT';
47
53
  return new LLMProviderError(
48
- `${provider} transcription failed (${status}): ${body.slice(0, 300)}`,
54
+ `${provider} ${operation} failed (${status}): ${body.slice(0, 300)}`,
49
55
  code, retryable, status, provider,
50
56
  );
51
57
  }
@@ -57,10 +63,11 @@ export async function speechFetch(
57
63
  init: RequestInit,
58
64
  timeoutMs = 240_000,
59
65
  fetchImpl: typeof fetch = fetch,
66
+ operation = 'transcription',
60
67
  ): Promise<Response> {
61
68
  try {
62
69
  const response = await fetchImpl(url, { ...init, signal: AbortSignal.timeout(timeoutMs) });
63
- if (!response.ok) throw speechHttpError(response.status, await response.text().catch(() => ''), provider);
70
+ if (!response.ok) throw speechHttpError(response.status, await response.text().catch(() => ''), provider, operation);
64
71
  return response;
65
72
  } catch (error) {
66
73
  if (error instanceof LLMProviderError) throw error;
@@ -1,18 +1,22 @@
1
1
  /**
2
- * `@startsimpli/llm/speech` — pluggable speech-to-text.
2
+ * `@startsimpli/llm/speech` — pluggable speech-to-text and text-to-speech.
3
3
  *
4
4
  * const speech = new SpeechService(); // SPEECH_PROVIDER, else the cheapest vendor with a key
5
5
  * const { text } = await speech.transcribe({ audio, mimeType: 'audio/webm' });
6
6
  *
7
+ * const { audio, words } = await speech.synthesize({ text: 'Hello there.' });
8
+ *
7
9
  * A new vendor: implement `ISpeechProvider`, then `registerSpeechProvider(name, factory)`.
8
10
  */
9
11
  export { SpeechService, registerSpeechProvider, speechProviders } from './service';
10
12
  export type { ISpeechProvider } from './interface';
11
13
  export type {
12
- SpeechServiceConfig, SpeechUsage, TranscribeRequest, TranscriptResult, TranscriptWord,
14
+ AlignedCharacter, SpeechServiceConfig, SpeechStyle, SpeechUsage, SynthesizeRequest, SynthesizeResult,
15
+ TranscribeRequest, TranscriptResult, TranscriptWord,
13
16
  } from './types';
14
17
  export { ElevenLabsSpeechProvider, createElevenLabsSpeechProvider } from './providers/elevenlabs';
15
18
  export { OpenAISpeechProvider, createOpenAISpeechProvider } from './providers/openai';
16
19
  export type { OpenAISpeechOptions } from './providers/openai';
17
20
  export { MockSpeechProvider, setMockSpeechBehavior, resetMockSpeechBehavior } from './providers/mock';
18
21
  export { audioExtension, speechHttpError } from './http';
22
+ export { wordsFromCharacters } from './alignment';
@@ -1,5 +1,5 @@
1
1
  import type { Selectable } from '../providers/registry';
2
- import type { TranscribeRequest, TranscriptResult } from './types';
2
+ import type { SynthesizeRequest, SynthesizeResult, TranscribeRequest, TranscriptResult } from './types';
3
3
 
4
4
  /**
5
5
  * What a speech vendor implements. A provider supports whichever directions its
@@ -8,12 +8,11 @@ import type { TranscribeRequest, TranscriptResult } from './types';
8
8
  *
9
9
  * ADDING A VENDOR: implement this, then
10
10
  * `registerSpeechProvider('name', () => new Mine(process.env.MINE_KEY) )` — no
11
- * edit to `SpeechService`. Text-to-speech joins here as an optional
12
- * `synthesize` alongside `transcribe` (bd startsim-2lnyg); nothing in the
13
- * service needs to change for it either.
11
+ * edit to `SpeechService`. Each direction is an optional method gated by
12
+ * `capabilities`.
14
13
  */
15
14
  export interface ISpeechProvider extends Selectable {
16
- readonly capabilities: { transcribe: boolean };
15
+ readonly capabilities: { transcribe: boolean; synthesize?: boolean };
17
16
  readonly defaultModel: string;
18
17
  /**
19
18
  * List price per audio minute, USD, for `model` (default model when omitted).
@@ -21,5 +20,8 @@ export interface ISpeechProvider extends Selectable {
21
20
  * that does not state a price is tried after every provider that does.
22
21
  */
23
22
  costPerMinuteUsd?(model?: string): number;
23
+ /** List price per 1,000 synthesized characters, USD. `synthesize` prefers the cheapest. */
24
+ costPer1kCharsUsd?(model?: string): number;
24
25
  transcribe?(request: TranscribeRequest): Promise<TranscriptResult>;
26
+ synthesize?(request: SynthesizeRequest): Promise<SynthesizeResult>;
25
27
  }
@@ -0,0 +1,34 @@
1
+ /**
2
+ * What ElevenLabs' models can do and cost. Data, not logic: a new model is a
3
+ * row here. USD per 1,000 characters, elevenlabs.io/pricing/api.
4
+ */
5
+ const RATE_PER_1K_CHARS: Record<string, number> = {
6
+ eleven_v3: 0.08,
7
+ eleven_multilingual_v2: 0.08,
8
+ eleven_flash_v2_5: 0.04,
9
+ eleven_v4_turbo: 0.04,
10
+ };
11
+ /** Rate for a model not in the table: the top published tier, so spend is never under-reported. */
12
+ const DEFAULT_RATE_PER_1K_CHARS = 0.10;
13
+
14
+ /** Models that read `[excited]`-style audio tags; the rest would speak them aloud. */
15
+ const AUDIO_TAG_MODELS = new Set(['eleven_v3', 'eleven_v4', 'eleven_v4_turbo', 'eleven_v3_conversational']);
16
+
17
+ export const DEFAULT_TTS_MODEL = 'eleven_multilingual_v2';
18
+
19
+ export function supportsAudioTags(model: string): boolean {
20
+ return AUDIO_TAG_MODELS.has(model);
21
+ }
22
+
23
+ export function costUsd(model: string, characters: number): number {
24
+ return ((RATE_PER_1K_CHARS[model] ?? DEFAULT_RATE_PER_1K_CHARS) * characters) / 1000;
25
+ }
26
+
27
+ export const DEFAULT_OUTPUT_FORMAT = 'mp3_44100_128';
28
+
29
+ /** Output formats are `<codec>_<rate>[_<bitrate>]`. */
30
+ export function mimeForFormat(format: string): string {
31
+ const codec = format.split('_')[0];
32
+ return ({ mp3: 'audio/mpeg', pcm: 'audio/pcm', ulaw: 'audio/basic', alaw: 'audio/alaw', opus: 'audio/ogg' } as Record<string, string>)[codec]
33
+ ?? 'application/octet-stream';
34
+ }
@@ -0,0 +1,88 @@
1
+ /**
2
+ * ElevenLabs text-to-speech with timestamps. Split from the provider so the
3
+ * request/response mapping reads on its own.
4
+ * https://elevenlabs.io/docs/api-reference/text-to-speech/convert-with-timestamps
5
+ */
6
+ import { wordsFromCharacters } from '../alignment';
7
+ import { LLMProviderError } from '../../errors';
8
+ import { assertText, speechFetch } from '../http';
9
+ import type { AlignedCharacter, SynthesizeRequest, SynthesizeResult } from '../types';
10
+ import {
11
+ DEFAULT_OUTPUT_FORMAT, DEFAULT_TTS_MODEL, costUsd, mimeForFormat, supportsAudioTags,
12
+ } from './elevenlabs-models';
13
+
14
+ interface Alignment {
15
+ characters?: string[];
16
+ character_start_times_seconds?: number[];
17
+ character_end_times_seconds?: number[];
18
+ }
19
+
20
+ export interface TtsContext {
21
+ provider: string;
22
+ apiKey: string;
23
+ baseUrl: string;
24
+ defaultVoiceId?: string;
25
+ fetchImpl?: typeof fetch;
26
+ }
27
+
28
+ /** Text as sent: style tags go in front, but only on models that read them. */
29
+ export function spokenText(request: SynthesizeRequest, model: string): string {
30
+ const tags = supportsAudioTags(model) ? (request.style?.tags ?? []) : [];
31
+ return tags.map((t) => `[${t.replace(/^\[|\]$/g, '')}]`).join('') + (tags.length ? ' ' : '') + request.text;
32
+ }
33
+
34
+ function voiceSettings(request: SynthesizeRequest): Record<string, number | boolean> | undefined {
35
+ const { speed, stability } = request.style ?? {};
36
+ const merged = {
37
+ ...(speed !== undefined ? { speed } : {}),
38
+ ...(stability !== undefined ? { stability } : {}),
39
+ ...request.voiceSettings,
40
+ };
41
+ return Object.keys(merged).length ? merged : undefined;
42
+ }
43
+
44
+ export async function elevenLabsSynthesize(ctx: TtsContext, request: SynthesizeRequest): Promise<SynthesizeResult> {
45
+ assertText(request, ctx.provider);
46
+ const voiceId = request.voiceId ?? ctx.defaultVoiceId;
47
+ if (!voiceId) {
48
+ throw new LLMProviderError('No voice: pass voiceId or set ELEVENLABS_VOICE_ID.', 'INVALID_REQUEST', false, undefined, ctx.provider);
49
+ }
50
+ const model = request.model ?? DEFAULT_TTS_MODEL;
51
+ const format = request.outputFormat ?? DEFAULT_OUTPUT_FORMAT;
52
+ const text = spokenText(request, model);
53
+ const settings = voiceSettings(request);
54
+
55
+ const response = await speechFetch(
56
+ ctx.provider,
57
+ `${ctx.baseUrl}/v1/text-to-speech/${encodeURIComponent(voiceId)}/with-timestamps?output_format=${encodeURIComponent(format)}`,
58
+ {
59
+ method: 'POST',
60
+ headers: { 'xi-api-key': ctx.apiKey, 'content-type': 'application/json' },
61
+ body: JSON.stringify({
62
+ text, model_id: model,
63
+ ...(request.language ? { language_code: request.language } : {}),
64
+ ...(settings ? { voice_settings: settings } : {}),
65
+ }),
66
+ },
67
+ request.timeoutMs, ctx.fetchImpl, 'synthesis',
68
+ );
69
+ const json = (await response.json()) as { audio_base64?: string; alignment?: Alignment | null };
70
+ const audio = new Uint8Array(Buffer.from(json.audio_base64 ?? '', 'base64'));
71
+ const characters = alignedCharacters(json.alignment);
72
+ const words = characters ? wordsFromCharacters(characters, { keepTags: request.keepTags }) : undefined;
73
+ return {
74
+ audio, mimeType: mimeForFormat(format),
75
+ ...(characters ? { characters, durationSec: characters[characters.length - 1].end } : {}),
76
+ ...(words ? { words } : {}),
77
+ voiceId, model, provider: ctx.provider,
78
+ usage: { characters: text.length, costUsd: costUsd(model, text.length) },
79
+ };
80
+ }
81
+
82
+ function alignedCharacters(a?: Alignment | null): AlignedCharacter[] | undefined {
83
+ const chars = a?.characters;
84
+ const starts = a?.character_start_times_seconds;
85
+ const ends = a?.character_end_times_seconds;
86
+ if (!chars?.length || !starts || !ends) return undefined;
87
+ return chars.map((text, i) => ({ text, start: starts[i], end: ends[i] }));
88
+ }
@@ -1,10 +1,12 @@
1
1
  /**
2
- * ElevenLabs Scribe speech-to-text. https://elevenlabs.io/docs/api-reference/speech-to-text
3
- * Env: ELEVENLABS_API_KEY.
2
+ * ElevenLabs Scribe speech-to-text and text-to-speech. https://elevenlabs.io/docs/api-reference/speech-to-text
3
+ * Env: ELEVENLABS_API_KEY; ELEVENLABS_VOICE_ID is the default voice.
4
4
  */
5
5
  import type { ISpeechProvider } from '../interface';
6
6
  import { assertAudio, audioBlob, speechFetch } from '../http';
7
- import type { TranscribeRequest, TranscriptResult, TranscriptWord } from '../types';
7
+ import type { SynthesizeRequest, SynthesizeResult, TranscribeRequest, TranscriptResult, TranscriptWord } from '../types';
8
+ import { elevenLabsSynthesize } from './elevenlabs-tts';
9
+ import { DEFAULT_TTS_MODEL, costUsd } from './elevenlabs-models';
8
10
 
9
11
  export interface ElevenLabsOptions {
10
12
  /**
@@ -14,6 +16,8 @@ export interface ElevenLabsOptions {
14
16
  */
15
17
  sendKeyterms?: boolean;
16
18
  apiKey?: string;
19
+ /** Voice used when a call names none. */
20
+ voiceId?: string;
17
21
  baseUrl?: string;
18
22
  fetchImpl?: typeof fetch;
19
23
  }
@@ -22,10 +26,11 @@ interface ScribeWord { text: string; start?: number; end?: number; type?: string
22
26
 
23
27
  export class ElevenLabsSpeechProvider implements ISpeechProvider {
24
28
  readonly name = 'elevenlabs';
25
- readonly capabilities = { transcribe: true };
29
+ readonly capabilities = { transcribe: true, synthesize: true };
26
30
  readonly defaultModel = 'scribe_v1';
27
31
  private readonly apiKey: string;
28
32
  private readonly baseUrl: string;
33
+ private readonly voiceId?: string;
29
34
  private readonly fetchImpl?: typeof fetch;
30
35
  private readonly sendKeyterms: boolean;
31
36
 
@@ -33,6 +38,7 @@ export class ElevenLabsSpeechProvider implements ISpeechProvider {
33
38
  this.sendKeyterms = options.sendKeyterms ?? false;
34
39
  this.apiKey = options.apiKey ?? '';
35
40
  this.baseUrl = (options.baseUrl ?? 'https://api.elevenlabs.io').replace(/\/+$/, '');
41
+ this.voiceId = options.voiceId;
36
42
  this.fetchImpl = options.fetchImpl;
37
43
  }
38
44
 
@@ -45,6 +51,18 @@ export class ElevenLabsSpeechProvider implements ISpeechProvider {
45
51
  return 0.22 / 60;
46
52
  }
47
53
 
54
+ /** Cheapest TTS tier per 1,000 characters (eleven_flash_v2_5). */
55
+ costPer1kCharsUsd(model?: string): number {
56
+ return costUsd(model ?? DEFAULT_TTS_MODEL, 1000);
57
+ }
58
+
59
+ synthesize(request: SynthesizeRequest): Promise<SynthesizeResult> {
60
+ return elevenLabsSynthesize(
61
+ { provider: this.name, apiKey: this.apiKey, baseUrl: this.baseUrl, defaultVoiceId: this.voiceId, fetchImpl: this.fetchImpl },
62
+ request,
63
+ );
64
+ }
65
+
48
66
  async transcribe(request: TranscribeRequest): Promise<TranscriptResult> {
49
67
  assertAudio(request, this.name);
50
68
  const model = request.model ?? this.defaultModel;
@@ -81,5 +99,5 @@ export class ElevenLabsSpeechProvider implements ISpeechProvider {
81
99
  export function createElevenLabsSpeechProvider(
82
100
  env: Record<string, string | undefined> = typeof process !== 'undefined' ? process.env : {},
83
101
  ): ElevenLabsSpeechProvider | null {
84
- return env.ELEVENLABS_API_KEY ? new ElevenLabsSpeechProvider({ apiKey: env.ELEVENLABS_API_KEY }) : null;
102
+ return env.ELEVENLABS_API_KEY ? new ElevenLabsSpeechProvider({ apiKey: env.ELEVENLABS_API_KEY, voiceId: env.ELEVENLABS_VOICE_ID || undefined }) : null;
85
103
  }
@@ -1,6 +1,8 @@
1
1
  import { LLMProviderError } from '../../errors';
2
2
  import type { ISpeechProvider } from '../interface';
3
- import type { TranscribeRequest, TranscriptResult } from '../types';
3
+ import { wordsFromCharacters } from '../alignment';
4
+ import { assertText } from '../http';
5
+ import type { SynthesizeRequest, SynthesizeResult, TranscribeRequest, TranscriptResult } from '../types';
4
6
 
5
7
  export interface MockSpeechBehavior {
6
8
  text?: string;
@@ -24,15 +26,33 @@ export function resetMockSpeechBehavior() {
24
26
  /** Deterministic transcription for tests — selected with `SPEECH_PROVIDER=mock`. */
25
27
  export class MockSpeechProvider implements ISpeechProvider {
26
28
  readonly name = 'mock';
27
- readonly capabilities = { transcribe: true };
29
+ readonly capabilities = { transcribe: true, synthesize: true };
28
30
  readonly defaultModel = 'mock-transcribe';
29
31
  isAvailable() { return true; }
30
- async transcribe(request: TranscribeRequest): Promise<TranscriptResult> {
32
+ private maybeFail() {
31
33
  if (behavior.authError) throw new LLMProviderError('mock auth failure', 'AUTHENTICATION_ERROR', false, 401, this.name);
32
34
  if (remainingFailures > 0) {
33
35
  remainingFailures -= 1;
34
36
  throw new LLMProviderError('mock outage', 'SERVICE_UNAVAILABLE', true, 503, this.name);
35
37
  }
38
+ }
39
+
40
+ /** Fake audio (one byte per character) with 50ms-per-character alignment. */
41
+ async synthesize(request: SynthesizeRequest): Promise<SynthesizeResult> {
42
+ assertText(request, this.name);
43
+ this.maybeFail();
44
+ const characters = [...request.text].map((text, i) => ({ text, start: Math.round(i * 50) / 1000, end: Math.round((i + 1) * 50) / 1000 }));
45
+ return {
46
+ audio: new Uint8Array(characters.length).fill(1), mimeType: 'audio/mpeg',
47
+ characters, words: wordsFromCharacters(characters, { keepTags: request.keepTags }),
48
+ durationSec: characters[characters.length - 1].end,
49
+ voiceId: request.voiceId ?? 'mock-voice', model: 'mock-synthesize', provider: this.name,
50
+ usage: { characters: request.text.length, costUsd: 0 },
51
+ };
52
+ }
53
+
54
+ async transcribe(request: TranscribeRequest): Promise<TranscriptResult> {
55
+ this.maybeFail();
36
56
  return {
37
57
  text: behavior.text ?? `mock transcript of ${request.audio.byteLength} bytes`,
38
58
  model: this.defaultModel, provider: this.name, usage: { audioSeconds: 1 },
@@ -9,8 +9,8 @@
9
9
  * URL, default model, word-timing model and price table.
10
10
  */
11
11
  import type { ISpeechProvider } from '../interface';
12
- import { assertAudio, audioBlob, speechFetch } from '../http';
13
- import type { TranscribeRequest, TranscriptResult, TranscriptWord } from '../types';
12
+ import { assertAudio, assertText, audioBlob, speechFetch } from '../http';
13
+ import type { SynthesizeRequest, SynthesizeResult, TranscribeRequest, TranscriptResult, TranscriptWord } from '../types';
14
14
 
15
15
  export interface OpenAISpeechOptions {
16
16
  apiKey?: string;
@@ -34,7 +34,7 @@ const OPENAI_PRICES: Record<string, number> = {
34
34
 
35
35
  export class OpenAISpeechProvider implements ISpeechProvider {
36
36
  readonly name: string;
37
- readonly capabilities = { transcribe: true };
37
+ readonly capabilities = { transcribe: true, synthesize: true };
38
38
  readonly defaultModel: string;
39
39
  private readonly apiKey: string;
40
40
  private readonly baseUrl: string;
@@ -62,6 +62,37 @@ export class OpenAISpeechProvider implements ISpeechProvider {
62
62
  return this.prices[model ?? this.defaultModel] ?? this.fallbackPrice;
63
63
  }
64
64
 
65
+ /** TTS list price per 1,000 characters (tts-1 $15/M; gpt-4o-mini-tts bills about the same per spoken minute). */
66
+ costPer1kCharsUsd(): number {
67
+ return 0.015;
68
+ }
69
+
70
+ /** `/v1/audio/speech`. OpenAI returns bare audio: no alignment, so no word timings. */
71
+ async synthesize(request: SynthesizeRequest): Promise<SynthesizeResult> {
72
+ assertText(request, this.name);
73
+ const model = request.model ?? 'gpt-4o-mini-tts';
74
+ const voice = request.voiceId ?? 'alloy';
75
+ const format = request.outputFormat ?? 'mp3';
76
+ const response = await speechFetch(
77
+ this.name, `${this.baseUrl}/v1/audio/speech`,
78
+ {
79
+ method: 'POST',
80
+ headers: { authorization: `Bearer ${this.apiKey}`, 'content-type': 'application/json' },
81
+ body: JSON.stringify({
82
+ model, voice, input: request.text, response_format: format,
83
+ ...(request.style?.speed !== undefined ? { speed: request.style.speed } : {}),
84
+ }),
85
+ },
86
+ request.timeoutMs, this.fetchImpl, 'synthesis',
87
+ );
88
+ const mimeType = ({ mp3: 'audio/mpeg', wav: 'audio/wav', opus: 'audio/ogg', aac: 'audio/aac', flac: 'audio/flac', pcm: 'audio/pcm' } as Record<string, string>)[format] ?? 'application/octet-stream';
89
+ return {
90
+ audio: new Uint8Array(await response.arrayBuffer()), mimeType,
91
+ voiceId: voice, model, provider: this.name,
92
+ usage: { characters: request.text.length, costUsd: (request.text.length * this.costPer1kCharsUsd()) / 1000 },
93
+ };
94
+ }
95
+
65
96
  async transcribe(request: TranscribeRequest): Promise<TranscriptResult> {
66
97
  assertAudio(request, this.name);
67
98
  const wantWords = request.timestamps === 'word';
@@ -4,7 +4,11 @@ import type { ISpeechProvider } from './interface';
4
4
  import { createElevenLabsSpeechProvider } from './providers/elevenlabs';
5
5
  import { MockSpeechProvider } from './providers/mock';
6
6
  import { createOpenAISpeechProvider } from './providers/openai';
7
- import type { SpeechServiceConfig, TranscribeRequest, TranscriptResult } from './types';
7
+ import type {
8
+ SpeechServiceConfig, SynthesizeRequest, SynthesizeResult, TranscribeRequest, TranscriptResult,
9
+ } from './types';
10
+
11
+ type Direction = 'transcribe' | 'synthesize';
8
12
 
9
13
  /**
10
14
  * The speech registry. Built-ins register here; a new vendor is
@@ -48,14 +52,15 @@ export class SpeechService {
48
52
  return this.getAvailableProviders().length > 0;
49
53
  }
50
54
 
51
- getAvailableProviders(): string[] {
52
- return [...this.providers].filter(([, p]) => p.isAvailable() && p.capabilities.transcribe).map(([n]) => n);
55
+ getAvailableProviders(direction: Direction = 'transcribe'): string[] {
56
+ return [...this.providers].filter(([, p]) => p.isAvailable() && p.capabilities[direction]).map(([n]) => n);
53
57
  }
54
58
 
55
59
  /** Providers in the order they will be tried: the preferred one (if any), then cheapest first. */
56
- private order(): ISpeechProvider[] {
57
- const usable = [...this.providers.values()].filter((p) => p.isAvailable() && p.capabilities.transcribe);
58
- const price = (p: ISpeechProvider) => p.costPerMinuteUsd?.() ?? Number.POSITIVE_INFINITY;
60
+ private order(direction: Direction): ISpeechProvider[] {
61
+ const usable = [...this.providers.values()].filter((p) => p.isAvailable() && p.capabilities[direction]);
62
+ const price = (p: ISpeechProvider) =>
63
+ (direction === 'synthesize' ? p.costPer1kCharsUsd?.() : p.costPerMinuteUsd?.()) ?? Number.POSITIVE_INFINITY;
59
64
  // Array.sort is stable, so equal/unknown prices keep registration order.
60
65
  const byPrice = [...usable].sort((a, b) => price(a) - price(b));
61
66
  const wanted = this.config.preferredProvider
@@ -71,18 +76,24 @@ export class SpeechService {
71
76
  // (an English note came back in French once). A per-call `language` wins.
72
77
  const env: Record<string, string | undefined> = typeof process !== 'undefined' ? process.env : {};
73
78
  const request = input.language || !env.SPEECH_LANGUAGE ? input : { ...input, language: env.SPEECH_LANGUAGE };
74
- const order = this.order();
75
- if (!order.length) {
76
- throw new LLMProviderError(
77
- 'No speech providers available. Set OPENAI_API_KEY or ELEVENLABS_API_KEY.',
78
- 'AUTHENTICATION_ERROR', false,
79
- );
80
- }
79
+ return this.run('transcribe', 'No speech providers available. Set OPENAI_API_KEY or ELEVENLABS_API_KEY.',
80
+ (provider) => provider.transcribe!(request));
81
+ }
82
+
83
+ /** Text to speech, with the same retry, backoff and fallback as `transcribe`. */
84
+ async synthesize(request: SynthesizeRequest): Promise<SynthesizeResult> {
85
+ return this.run('synthesize', 'No text-to-speech provider available. Set OPENAI_API_KEY or ELEVENLABS_API_KEY.',
86
+ (provider) => provider.synthesize!(request));
87
+ }
88
+
89
+ private async run<T>(direction: Direction, none: string, call: (provider: ISpeechProvider) => Promise<T>): Promise<T> {
90
+ const order = this.order(direction);
91
+ if (!order.length) throw new LLMProviderError(none, 'AUTHENTICATION_ERROR', false);
81
92
  let last: unknown;
82
93
  for (const provider of order) {
83
94
  for (let attempt = 0; ; attempt += 1) {
84
95
  try {
85
- return await provider.transcribe!(request);
96
+ return await call(provider);
86
97
  } catch (error) {
87
98
  last = error;
88
99
  const typed = error instanceof LLMProviderError ? error : null;
@@ -14,8 +14,10 @@ export interface TranscriptWord {
14
14
  export interface SpeechUsage {
15
15
  /** Billable input, when the provider reports it; else derived from word timings. */
16
16
  audioSeconds?: number;
17
- /** Characters synthesized (TTS). */
17
+ /** Characters synthesized (TTS), as billed: tags included. */
18
18
  characters?: number;
19
+ /** Cost at the vendor's list rate, when the provider knows it. */
20
+ costUsd?: number;
19
21
  }
20
22
 
21
23
  export interface TranscribeRequest {
@@ -47,6 +49,58 @@ export interface TranscriptResult {
47
49
  usage: SpeechUsage;
48
50
  }
49
51
 
52
+ /** One aligned unit (a character) with its time span, in seconds from the start of the audio. */
53
+ export interface AlignedCharacter {
54
+ text: string;
55
+ start: number;
56
+ end: number;
57
+ }
58
+
59
+ /**
60
+ * Per-call delivery. Providers apply what their model supports and ignore the
61
+ * rest — a caller never needs to know which model reads which knob.
62
+ */
63
+ export interface SpeechStyle {
64
+ /** Audio tags (`excited`, `whispers`), written as `[tag]` ahead of the text on models that read them. */
65
+ tags?: string[];
66
+ speed?: number;
67
+ /** Lower is more expressive. */
68
+ stability?: number;
69
+ }
70
+
71
+ export interface SynthesizeRequest {
72
+ text: string;
73
+ /** Provider voice id. Omit to use the provider's configured default (ELEVENLABS_VOICE_ID). */
74
+ voiceId?: string;
75
+ /** Provider-specific model override. */
76
+ model?: string;
77
+ /** ISO-639 hint. */
78
+ language?: string;
79
+ /** Vendor output format (`mp3_44100_128`, `pcm_22050`). Default: mp3. */
80
+ outputFormat?: string;
81
+ style?: SpeechStyle;
82
+ /** Raw vendor voice settings; wins over `style` where both set a key. */
83
+ voiceSettings?: Record<string, number | boolean>;
84
+ /** Keep `[tag]` markup in the returned words. Default false: tags are delivery, not speech. */
85
+ keepTags?: boolean;
86
+ timeoutMs?: number;
87
+ }
88
+
89
+ export interface SynthesizeResult {
90
+ audio: Uint8Array;
91
+ mimeType: string;
92
+ /** Per-character timings, exactly as the vendor aligned the text it was sent. Absent when the vendor gives none. */
93
+ characters?: AlignedCharacter[];
94
+ /** Word timings derived from `characters`; audio tags removed unless `keepTags`. */
95
+ words?: TranscriptWord[];
96
+ /** End of the last aligned character; absent without alignment. */
97
+ durationSec?: number;
98
+ voiceId?: string;
99
+ model: string;
100
+ provider: string;
101
+ usage: SpeechUsage;
102
+ }
103
+
50
104
  export interface SpeechServiceConfig {
51
105
  /** Provider tried first. Default: `SPEECH_PROVIDER`, else the cheapest vendor with a key. */
52
106
  preferredProvider?: string;