@happyvertical/speech 0.96.0 → 0.97.0

package/AGENT.md

@@ -27,7 +27,7 @@ pnpm --filter @happyvertical/speech clean
 
 ## Ecosystem Relationships
 - Provides: Speech provider abstraction for STT and TTS backends
-- Implements: Studio Server STT, Studio Server TTS, Qwen3 TTS, OpenAI-compatible TTS
+- Implements: Studio Server STT, OpenAI-compatible STT, Studio Server TTS, Qwen3 TTS, OpenAI-compatible TTS
 - Requires: none
 - Stability: experimental (Marked as preview or experimental in package guidance.)
 <!-- END AGENT:GENERATED -->
@@ -46,6 +46,19 @@ Adapter constructors are internal implementation details. Keep new backends behi
 ## Adapters
 
 - Studio Server STT (`type: 'studio-server'`) posts multipart audio to `/v1/transcribe`.
+- OpenAI-compatible STT (`type: 'openai-compatible'`) posts multipart `file`/`model` to `<base>/audio/transcriptions`.
 - Studio Server TTS (`type: 'studio-server'`) posts multipart form data to `/v1/tts/synthesize`.
 - Qwen3 TTS (`type: 'qwen3-tts'`) posts multipart form data to `/v1/audio/speech`.
 - OpenAI-compatible TTS (`type: 'openai-compatible'`) posts OpenAI-shaped JSON to `/v1/audio/speech`.
+
+## Shared Building Blocks
+
+New adapters (including streaming/realtime transcribers) should reuse these modules in `src/shared/` instead of re-implementing them:
+
+- `usage.ts`: `SpeechUsage`, `SpeechUsageCallback`, `reportSpeechUsage()` (adapter `onUsage` first, then request `onUsage`; callback errors propagate), and `audioSecondsFromProviderUsage()`.
+- `env.ts`: `resolveTranscriberConfig(options, context)` reads `HAVE_SPEECH_TRANSCRIBER_*` (then legacy `HAVE_SPEECH_STT_*`/`STT_*`) with explicit options winning; headers merge env → context → options via `mergeHeaderInits()`.
+- `audio.ts`: `normalizeAudioInput()` accepts `AudioInput` or a bare Blob/Buffer/Uint8Array/ArrayBuffer/ReadableStream, enforces `maxBytes` while buffering, and derives the multipart filename through `mimeTypeToAudioExtension()`.
+- `retry.ts`: `withSpeechRetry()` retries 429/5xx `SpeechProviderError`s with exponential backoff, honours `Retry-After` (`retryAfterMs`), and stops on abort.
+- `http.ts`: `HttpSpeechAdapter.post(..., retry)` sends auth/extra headers (`headers` option plus per-request `headers`), applies `timeoutMs` per attempt, and redacts the API key from provider error bodies; `resolveOpenAICompatibleUrl()` normalises OpenAI-style base URLs.
+
+The OpenAI-compatible transcriber holds an API key and is server-side only.

package/README.md

@@ -46,14 +46,71 @@ const spoken = await speech.synthesize({
 | Provider | Type | Method | Path | Encoding |
 | --- | --- | --- | --- | --- |
 | Studio Server STT | `studio-server` | `POST` | `/v1/transcribe` | Multipart (`audio`) |
+| OpenAI-compatible STT | `openai-compatible` | `POST` | `<base>/audio/transcriptions` | Multipart (`file`) |
 | Studio Server TTS | `studio-server` | `POST` | `/v1/tts/synthesize` | Multipart |
 | Qwen3 TTS | `qwen3-tts` | `POST` | `/v1/audio/speech` | Multipart |
 | OpenAI-compatible TTS | `openai-compatible` | `POST` | `/v1/audio/speech` | JSON |
 
 Studio Server and Qwen3 accept pre-extracted provider voice prompts through `SpeechVoice.prompt`; the adapters forward these as the multipart `voice_prompt` field.
 
+## OpenAI-compatible Transcription
+
+`type: 'openai-compatible'` speaks the OpenAI audio API, so one adapter covers OpenAI, Groq, Fireworks, LiteLLM and Bifrost gateways, vLLM, speaches/faster-whisper-server, and whisper.cpp server.
+
+> **Server-side only.** The adapter holds the provider API key. Never construct it in a browser. Browsers should record audio (for example with `MediaRecorder`) and post it to your own server route, which calls the transcriber.
+
+```typescript
+import { getTranscriber } from '@happyvertical/speech';
+
+const transcriber = await getTranscriber({
+  type: 'openai-compatible',
+  baseUrl: 'http://bifrost:8080/openai/v1', // or https://api.openai.com
+  apiKey: process.env.OPENAI_API_KEY,
+  model: 'whisper-1',
+  headers: { 'x-bf-vk': virtualKey }, // gateway routing / org / project headers
+  onUsage: (usage) => recordUsage(tenantId, usage),
+});
+
+// Inside a server route that received the browser upload:
+const result = await transcriber.transcribe({
+  audio: request.body, // Blob | Buffer | Uint8Array | ArrayBuffer | ReadableStream
+  mimeType: 'audio/webm;codecs=opus', // Safari sends audio/mp4
+  language: 'en',
+  timestampGranularities: ['word', 'segment'],
+  headers: { 'x-bf-vk': tenantVirtualKey }, // per-request headers win
+  signal: abortController.signal,
+});
+
+result.text;
+result.segments; // present when the server returns verbose_json
+result.words; // present when word timestamps were requested and supported
+result.usage; // { operation, provider, model, audioSeconds, bytes, providerUsage }
+```
+
+- **Base URL.** A server root gets `/v1/audio/transcriptions`; a base ending in a version segment (`http://gateway/stt/v1`) gets `/audio/transcriptions`; a URL already ending in `/audio/transcriptions` is used as-is. Any query string (for example `?api-version=…`) is preserved and a fragment is dropped.
+- **Audio formats.** The multipart filename extension comes from the MIME type (codec parameters are ignored): `webm`, `m4a` (`audio/mp4`), `mp4` (`video/mp4`), `wav`, `ogg`, `mp3`, `flac`. Pass `audio: { data, filename }` to choose the name yourself.
+- **Limits.** `maxBytes` (adapter or request) defaults to 25 MB, the OpenAI upload cap. Oversized or empty input fails with `SpeechConfigurationError` before any request; streams are cancelled as soon as they exceed the limit.
+- **Response formats.** Defaults to `verbose_json` so segments, duration, and (with `timestampGranularities`) word timings map onto `TranscriptResult`. Models that only return `json`/`text` (`gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, including gateway-prefixed names) are sent `json` without timestamps. If any other server rejects `verbose_json` or `timestamp_granularities[]` with a 400/422, the adapter retries once with plain `json`.
+- **Retries.** 429 and 5xx responses are retried with exponential backoff (default 2 retries, 500 ms initial delay, 30 s cap), waiting at least `Retry-After`. A `Retry-After` longer than `maxDelayMs` ends retrying. Configure with `retry: { maxRetries, initialDelayMs, maxDelayMs }` or disable with `retry: false`; non-finite values (`NaN`, `Infinity`) throw `SpeechConfigurationError`. `timeoutMs` applies per attempt; `signal` cancels the request and any backoff wait.
+- **Usage.** `result.usage` and `onUsage` (adapter-level, then request-level) report the model, audio seconds (provider duration, then `usage.seconds`, then `AudioInput.durationSeconds`), uploaded bytes, and the provider `usage` block untouched. Errors thrown by `onUsage` propagate.
+- **Errors.** HTTP failures throw `SpeechProviderError` with `status`, `responseBody`, and `retryAfterMs`. The API key never appears in error messages and is redacted from response bodies.
+
 ## Environment Configuration
 
+Transcriber settings switch between cloud and self-hosted models without code changes. Explicit options win over the environment:
+
+```bash
+HAVE_SPEECH_TRANSCRIBER_TYPE=openai-compatible
+HAVE_SPEECH_TRANSCRIBER_BASE_URL=http://bifrost:8080/openai/v1
+HAVE_SPEECH_TRANSCRIBER_MODEL=whisper-1
+HAVE_SPEECH_TRANSCRIBER_API_KEY=sk-...
+HAVE_SPEECH_TRANSCRIBER_TIMEOUT=60000          # milliseconds, per attempt
+HAVE_SPEECH_TRANSCRIBER_MAX_BYTES=26214400     # optional
+HAVE_SPEECH_TRANSCRIBER_HEADERS='{"x-bf-vk":"vk-..."}'  # optional JSON object
+```
+
+Headers merge in order environment, factory context, then explicit options (later wins per header). The `HAVE_SPEECH_STT_*` names below remain supported as fallbacks.
+
 SDK-style names are preferred:
 
 ```bash
@@ -74,11 +131,11 @@ TTS_ADAPTER=qwen3-tts
 TTS_BASE_URL=http://qwen3-tts.qwen3-tts.svc.cluster.local
 ```
 
-Optional overrides include `STT_PATH`, `TTS_PATH`, `STT_API_KEY`, `TTS_API_KEY`, `SPEECH_API_KEY`, `STT_TIMEOUT_MS`, and `TTS_TIMEOUT_MS`.
+Optional overrides include `HAVE_SPEECH_STT_MODEL`, `STT_MODEL`, `STT_PATH`, `TTS_PATH`, `STT_API_KEY`, `TTS_API_KEY`, `SPEECH_API_KEY`, `STT_TIMEOUT_MS`, and `TTS_TIMEOUT_MS`.
 
 ## Testing
 
-Default tests use tiny in-process HTTP fixture services that mirror the Studio Server, Qwen3, and OpenAI-compatible request shapes. They validate field names, encodings, and response normalization without downloading production-scale models.
+Default tests use tiny in-process HTTP fixture services or an injected `fetch` that mirror the Studio Server, Qwen3, and OpenAI-compatible request shapes. They validate field names, encodings, and response normalization without downloading production-scale models.
 
 Docker or Testcontainers integration suites should run the fixture services or Studio Server with mock backends. Model-backed tests must remain opt-in, for example behind `HV_SPEECH_MODEL_TESTS=1`, so the normal SDK suite never downloads model weights.
 

package/dist/adapters/openai-compatible-transcriber.d.ts

@@ -0,0 +1,23 @@
+import { HttpSpeechAdapter } from '../shared/http.js';
+import { OpenAICompatibleTranscriberOptions, Transcriber, TranscriptionRequest, TranscriptResult } from '../shared/types.js';
+export declare function isJsonOnlyTranscriptionModel(model: string): boolean;
+/**
+ * OpenAI-compatible speech-to-text adapter (`POST <base>/audio/transcriptions`).
+ * Works with OpenAI, Groq, Fireworks, LiteLLM/Bifrost gateways, vLLM,
+ * speaches/faster-whisper-server, and whisper.cpp server.
+ *
+ * Server-side only: the adapter holds the API key.
+ */
+export declare class OpenAICompatibleTranscriber extends HttpSpeechAdapter implements Transcriber {
+    readonly type: "openai-compatible";
+    private readonly endpoint;
+    private readonly model;
+    private readonly responseFormat?;
+    private readonly maxBytes;
+    private readonly retry;
+    private readonly onUsage?;
+    constructor(options: OpenAICompatibleTranscriberOptions);
+    transcribe(request: TranscriptionRequest): Promise<TranscriptResult>;
+    private resolveWireOptions;
+}
+//# sourceMappingURL=openai-compatible-transcriber.d.ts.map

package/dist/adapters/openai-compatible-transcriber.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"openai-compatible-transcriber.d.ts","sourceRoot":"","sources":["../../src/adapters/openai-compatible-transcriber.ts"],"names":[],"mappings":"AAKA,OAAO,EAEL,iBAAiB,EAGlB,MAAM,mBAAmB,CAAC;AAK3B,OAAO,KAAK,EACV,kCAAkC,EAElC,WAAW,EACX,oBAAoB,EAEpB,gBAAgB,EACjB,MAAM,oBAAoB,CAAC;AAsB5B,wBAAgB,4BAA4B,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAEnE;AAQD;;;;;;GAMG;AACH,qBAAa,2BACX,SAAQ,iBACR,YAAW,WAAW;IAEtB,QAAQ,CAAC,IAAI,EAAG,mBAAmB,CAAU;IAE7C,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAS;IAC/B,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAC,CAA8B;IAC9D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAuC;IAC7D,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAsB;gBAEnC,OAAO,EAAE,kCAAkC;IAajD,UAAU,CAAC,OAAO,EAAE,oBAAoB,GAAG,OAAO,CAAC,gBAAgB,CAAC;IA0E1E,OAAO,CAAC,kBAAkB;CAsB3B"}

package/dist/adapters/studio-server.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"studio-server.d.ts","sourceRoot":"","sources":["../../src/adapters/studio-server.ts"],"names":[],"mappings":"AAAA,OAAO,EAIL,iBAAiB,EAGlB,MAAM,mBAAmB,CAAC;AAC3B,OAAO,KAAK,EACV,iBAAiB,EACjB,oCAAoC,EACpC,8BAA8B,EAC9B,gBAAgB,EAChB,iBAAiB,EACjB,WAAW,EACX,oBAAoB,EACpB,gBAAgB,EACjB,MAAM,oBAAoB,CAAC;AAE5B,qBAAa,uBACX,SAAQ,iBACR,YAAW,WAAW;IAEtB,QAAQ,CAAC,IAAI,EAAG,eAAe,CAAU;IACzC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAS;gBAE5B,OAAO,EAAE,8BAA8B;IAK7C,UAAU,CAAC,OAAO,EAAE,oBAAoB,GAAG,OAAO,CAAC,gBAAgB,CAAC;CAe3E;AAED,qBAAa,6BACX,SAAQ,iBACR,YAAW,iBAAiB;IAE5B,QAAQ,CAAC,IAAI,EAAG,eAAe,CAAU;IACzC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAS;IACxC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAS;gBAE3B,OAAO,EAAE,oCAAoC;IAMnD,UAAU,CAAC,OAAO,EAAE,gBAAgB,GAAG,OAAO,CAAC,iBAAiB,CAAC;CAaxE"}
+{"version":3,"file":"studio-server.d.ts","sourceRoot":"","sources":["../../src/adapters/studio-server.ts"],"names":[],"mappings":"AACA,OAAO,EAGL,iBAAiB,EAGlB,MAAM,mBAAmB,CAAC;AAC3B,OAAO,KAAK,EACV,iBAAiB,EACjB,oCAAoC,EACpC,8BAA8B,EAC9B,gBAAgB,EAChB,iBAAiB,EACjB,WAAW,EACX,oBAAoB,EACpB,gBAAgB,EACjB,MAAM,oBAAoB,CAAC;AAE5B,qBAAa,uBACX,SAAQ,iBACR,YAAW,WAAW;IAEtB,QAAQ,CAAC,IAAI,EAAG,eAAe,CAAU;IACzC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAS;gBAE5B,OAAO,EAAE,8BAA8B;IAK7C,UAAU,CAAC,OAAO,EAAE,oBAAoB,GAAG,OAAO,CAAC,gBAAgB,CAAC;CAuB3E;AAED,qBAAa,6BACX,SAAQ,iBACR,YAAW,iBAAiB;IAE5B,QAAQ,CAAC,IAAI,EAAG,eAAe,CAAU;IACzC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAS;IACxC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAS;gBAE3B,OAAO,EAAE,oCAAoC;IAMnD,UAAU,CAAC,OAAO,EAAE,gBAAgB,GAAG,OAAO,CAAC,iBAAiB,CAAC;CAaxE"}

package/dist/index.d.ts

@@ -21,7 +21,9 @@
  *
  * @packageDocumentation
  */
+export { DEFAULT_MAX_AUDIO_BYTES, mimeTypeToAudioExtension, } from './shared/audio.js';
+export { type SpeechEnv, TRANSCRIBER_ENV_KEYS, } from './shared/env.js';
 export * from './shared/errors.js';
 export { getAvailableSpeechAdapters, getSpeech, getSpeechSynthesizer, getTranscriber, type SpeechFactoryContext, } from './shared/factory.js';
-export type { AudioBytes, AudioInput, GetSpeechOptions, GetSpeechSynthesizerOptions, GetTranscriberOptions, OpenAICompatibleSpeechSynthesizerOptions, Qwen3SpeechSynthesizerOptions, Speech, SpeechAdapterAvailability, SpeechAdapterType, SpeechFetch, SpeechSynthesizer, SpeechSynthesizerType, SpeechVoice, SpeechVoiceInput, StudioServerSpeechSynthesizerOptions, StudioServerTranscriberOptions, SynthesisRequest, SynthesizedSpeech, Transcriber, TranscriberType, TranscriptionRequest, TranscriptResult, TranscriptSegment, WordTiming, } from './shared/types.js';
+export type { AudioBytes, AudioInput, AudioSource, GetSpeechOptions, GetSpeechSynthesizerOptions, GetTranscriberOptions, OpenAICompatibleSpeechSynthesizerOptions, OpenAICompatibleTranscriberOptions, Qwen3SpeechSynthesizerOptions, Speech, SpeechAdapterAvailability, SpeechAdapterType, SpeechFetch, SpeechOperation, SpeechRetryOptions, SpeechSynthesizer, SpeechSynthesizerType, SpeechUsage, SpeechUsageCallback, SpeechVoice, SpeechVoiceInput, StudioServerSpeechSynthesizerOptions, StudioServerTranscriberOptions, SynthesisRequest, SynthesizedSpeech, TimestampGranularity, Transcriber, TranscriberType, TranscriptionRequest, TranscriptionResponseFormat, TranscriptResult, TranscriptSegment, WordTiming, } from './shared/types.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,cAAc,oBAAoB,CAAC;AACnC,OAAO,EACL,0BAA0B,EAC1B,SAAS,EACT,oBAAoB,EACpB,cAAc,EACd,KAAK,oBAAoB,GAC1B,MAAM,qBAAqB,CAAC;AAC7B,YAAY,EACV,UAAU,EACV,UAAU,EACV,gBAAgB,EAChB,2BAA2B,EAC3B,qBAAqB,EACrB,wCAAwC,EACxC,6BAA6B,EAC7B,MAAM,EACN,yBAAyB,EACzB,iBAAiB,EACjB,WAAW,EACX,iBAAiB,EACjB,qBAAqB,EACrB,WAAW,EACX,gBAAgB,EAChB,oCAAoC,EACpC,8BAA8B,EAC9B,gBAAgB,EAChB,iBAAiB,EACjB,WAAW,EACX,eAAe,EACf,oBAAoB,EACpB,gBAAgB,EAChB,iBAAiB,EACjB,UAAU,GACX,MAAM,mBAAmB,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,OAAO,EACL,uBAAuB,EACvB,wBAAwB,GACzB,MAAM,mBAAmB,CAAC;AAC3B,OAAO,EACL,KAAK,SAAS,EACd,oBAAoB,GACrB,MAAM,iBAAiB,CAAC;AACzB,cAAc,oBAAoB,CAAC;AACnC,OAAO,EACL,0BAA0B,EAC1B,SAAS,EACT,oBAAoB,EACpB,cAAc,EACd,KAAK,oBAAoB,GAC1B,MAAM,qBAAqB,CAAC;AAC7B,YAAY,EACV,UAAU,EACV,UAAU,EACV,WAAW,EACX,gBAAgB,EAChB,2BAA2B,EAC3B,qBAAqB,EACrB,wCAAwC,EACxC,kCAAkC,EAClC,6BAA6B,EAC7B,MAAM,EACN,yBAAyB,EACzB,iBAAiB,EACjB,WAAW,EACX,eAAe,EACf,kBAAkB,EAClB,iBAAiB,EACjB,qBAAqB,EACrB,WAAW,EACX,mBAAmB,EACnB,WAAW,EACX,gBAAgB,EAChB,oCAAoC,EACpC,8BAA8B,EAC9B,gBAAgB,EAChB,iBAAiB,EACjB,oBAAoB,EACpB,WAAW,EACX,eAAe,EACf,oBAAoB,EACpB,2BAA2B,EAC3B,gBAAgB,EAChB,iBAAiB,EACjB,UAAU,GACX,MAAM,mBAAmB,CAAC"}