nixamp 0.25.0 → 0.26.0

package/README.md

@@ -421,11 +421,18 @@ nixamp.com instead. `NIXAMP_STT_MODEL` picks another Whisper
 takes twice as long), `NIXAMP_STT_CACHE` says where its files are kept, and
 `NIXAMP_STT=off` leaves the ear out of a deployment altogether.
 
-The ear tells which language it heard: one pass over the first thirty
-seconds, the way whisper.cpp does it, before the words are read. Without
-that a Swedish channel came back as three English words repeated to the end
-of the window. A captioner learns the language from its first line and says
-it on every ask after that.
+Captions default to **Original (auto-detect)**. Each audio window detects its
+own language and explicitly transcribes it. A language selected in the menu
+only affects translation; neither it nor a cached transcript can force the
+recognizer into English. Short windows have a decoding limit, and silent
+audio and repetitive hallucinations are discarded. Language is stored on
+each line, so an interview can switch languages. Legacy live-caption cache
+entries are heard again instead of replaying their incorrect words.
+
+At most four channels are captioned per server, with two recognition requests
+per channel in flight. Live work expires after twelve seconds. Translation
+keeps one active request and the latest pending line per target; joining a
+live reads cached translations without starting a whole-transcript job.
 
 ### Kept: written down once, for everybody
 
@@ -523,8 +530,89 @@ The page has the same choice beside the Captions switch, remembered per
 device; a translated line is marked with its language and shows what was
 heard under the pointer. An agent has `translate_text`. `NIXAMP_MT_WARM`
 names pairs to load at boot (`en-de,en-sv`), `NIXAMP_MT=off` leaves
-translation out, and the Docker image bakes the ear and the German and
-Swedish pairs in so a deploy never downloads them again.
+translation out. The Docker image includes the ear, German/Swedish pairs
+with English, and Spanish pairs with English and German. Spanish-to-German
+uses its direct model; it does not first translate the audio into English.
+
+
+### Hear it in your language
+
+The browser's Transcript panel works with movies, shows, sports, courses,
+podcasts, live channels, and rooms. Choose a language and enable **Play
+translated audio**. This is a listener preference: the video and the room's
+shared playback clock keep running. Turn it off to restore the original sound.
+**Start player captions** transcribes other playback in its detected source
+language. Files are played locally; these explicit controls opt into sending
+short audio clips for processing. Ordinary playback uploads no audio.
+
+**Translate another tab** opens the browser's audio-sharing chooser, so a
+watch party or video hosted on another site can be interpreted too. Select a
+browser tab with **Share audio** enabled; **Stop listening** releases sharing.
+Only audio is sent, even though the browser requires a video track to select
+its source. Sharing support depends on the browser and the source's permissions;
+protected media and sources whose audio cannot be captured remain unavailable.
+The control requests suppression of the source tab's local sound. If a browser
+ignores that option, mute the source tab to avoid hearing both languages.
+
+Native captions use local Whisper Base through Transformers.js. Optional
+speaker-aware audio uses ElevenLabs **Scribe v2** for native transcription with
+speaker turns, local **OPUS-MT** for the selected translation, and ElevenLabs
+**Flash v2.5** HTTP streaming for natural stock voices. The application sends
+short audio clips and translated text to ElevenLabs only for this optional
+feature. This uses the direct API; it does not need an MCP server or clone voices.
+Supported translation pairs come from `/api/v1/translate`; voice languages are
+also checked before enabling the audio toggle.
+
+A rolling 15-second audio window advances every 5 seconds. Speaker labels are
+reconciled using overlapping timestamps, with different voices assigned to
+separate speakers. Lower/higher pitch suggests a male/female stock voice;
+ambiguous audio uses a default. Each speaker's voice can be changed in the
+panel. Pitch is not gender identity, and a speaker returning after leaving the
+rolling context may receive a new label. Simultaneous speech and noisy crowds
+can still confuse recognition. Native captions never translate to English as
+an intermediate recognition step.
+
+Processing has one active request and only the latest pending window per
+listener; speech queues and response sizes are bounded. Old transcript history
+is never spoken. Pause, seek, source changes, and disabling the feature cancel
+queued speech; errors restore the original audio. This is a delayed live
+interpreter, not a promise of exact lip sync or background-music separation.
+
+The account server needs `ELEVENLABS_API_KEY`; `NIXAMP_DUBBING=off` disables
+this feature. The key stays on the server. Sign-in is required for speaker
+analysis, voice selection, and short-lived playback grants. Grants expire after
+90 seconds, authorize at most 2,000 characters, and are scoped to one playback
+session or channel. Provider requests also have account/IP throttles, concurrency
+limits, and cached duplicate voice generation. Native speech and local
+translation retain their existing account and queue limits.
+
+Postgres stores atomic usage reservations and hashed grants, so the feature's
+budgets survive restarts and are shared between replicas. Provider failures
+still consume reservations conservatively. The configurable daily limits are:
+
+| Setting | Default | Counts |
+| --- | ---: | --- |
+| `NIXAMP_DUB_DAILY_CHARS` | 200,000 | New voice characters across this server |
+| `NIXAMP_DUB_USER_DAILY_CHARS` | 120,000 | New voice characters per account |
+| `NIXAMP_DUB_DAILY_AUDIO_SECONDS` | 86,400 | Scribe audio seconds across this server |
+| `NIXAMP_DUB_USER_DAILY_AUDIO_SECONDS` | 43,200 | Scribe audio seconds per account |
+
+Audio limits count overlapping context too: a continuous hour of speaker-aware
+listening submits about three hours of Scribe audio. At the [published API
+rates](https://elevenlabs.io/pricing/api) of $0.05 per 1,000 Flash characters and
+$0.22 per Scribe audio hour, a listener producing 1,000 translated characters
+per minute costs about $3.66/hour, before plan minimums or discounts. These default
+server quotas limit this feature to about $15.28/day at those rates; they do not
+cover other applications using the same provider key. Set a daily limit to zero
+to block new use of that resource. Limits return 429 and never trigger an
+unlimited fallback provider.
+
+```
+GET  /api/v1/speech/voices       authenticated stock voices and supported audio languages
+POST /api/v1/speech/speakers     authenticated, bounded mono 16 kHz WAV -> native speaker turns
+POST /api/v1/speech/grant        authenticated {channel: playbackScope} -> short-lived grant
+POST /api/v1/speech/synthesize   scoped grant + {channel, text, language, voice, profile} -> streaming PCM
+```
 
 ## Several streams at once
 
@@ -908,3 +996,16 @@ make the name honest.
 ## Licence
 
 MIT
+
+### Accessibility and interaction
+
+Keyboard access, named controls, headings, skip links, visible focus, descriptive
+slider values, reduced motion, and concise screen-reader status announcements
+are built into the web player. Incoming transcripts, chat, and playback updates
+preserve scrolling, focus, caret, and the browsing page. The
+[UX and accessibility baseline](docs/ux.md) applies to all interface changes.
+
+When an older broadcaster still provides unversioned English-first captions,
+a signed-in listener uses native-language recognition of the playing audio
+instead of that stale transcript cache. Update the broadcaster for shared native
+captions; translated audio uses the listener's current audio and selected language.

package/dist/captions.d.ts

@@ -12,6 +12,8 @@ export interface CaptionLine {
     language?: string;
     /** What was heard, when this line is a translation of it. */
     original?: string;
+    sourceLanguage?: string;
+    voiceProfile?: "lower" | "higher" | "unknown";
 }
 /** What turns a channel's bytes into 16 kHz mono 16-bit PCM. ffmpeg, or a test's stand-in. */
 export interface Decoder {
@@ -71,6 +73,8 @@ export declare const IDLE_MS = 60000;
 export declare const QUIET = 0.004;
 /** Windows waiting on the ear at once. Past this the sound is dropped, not queued: late words are worse than none. */
 export declare const IN_FLIGHT = 2;
+export declare const LIVE_DEADLINE_MS = 12000;
+export declare const MAX_CAPTIONERS = 4;
 /** Heard lines wait this long, at most, before they are kept. */
 export declare const FLUSH_MS = 20000;
 /** Or this many. */
@@ -118,6 +122,9 @@ export declare class Captions {
     constructor(options: CaptionsOptions);
     /** Whether this server can caption at all: it has to be signed in for the ear to answer it. */
     available(): boolean;
+    capacity(id: string): boolean;
+    /** Voice requests are constrained to captions this channel actually produced. */
+    voiceRequest(id: string, at: number | null, language: string, voice: string, signal: AbortSignal, grant?: string): Promise<Response>;
     /**
      * Lines for a channel as they are heard, starting the captioner if it is
      * not running. Null when there is no such channel. The returned function

package/dist/captions.js

@@ -31,7 +31,8 @@
  * handed to whoever wanted that language, and kept beside the original.
  */
 import { spawn } from "node:child_process";
-import { RATE } from "./speech.js";
+import { NATIVE_REVISION, RATE, SpeechError, reliableText } from "./speech.js";
+import { voiceProfile } from "./voice-profile.js";
 import { fetchTranscript, keepLines, keepMedia, translateTexts } from "./transcript-client.js";
 import { covered, lineAt, transcriptIdOf } from "./transcripts.js";
 export const WINDOW_MS = 5000;
@@ -42,6 +43,8 @@ export const IDLE_MS = 60_000;
 export const QUIET = 0.004;
 /** Windows waiting on the ear at once. Past this the sound is dropped, not queued: late words are worse than none. */
 export const IN_FLIGHT = 2;
+export const LIVE_DEADLINE_MS = 12_000;
+export const MAX_CAPTIONERS = 4;
 /** Heard lines wait this long, at most, before they are kept. */
 export const FLUSH_MS = 20_000;
 /** Or this many. */
@@ -174,6 +177,9 @@ class Captioner {
     stopped = false;
     complainedAt = 0;
     windows = 0;
+    audioUntil = 0;
+    lastEmittedAt = -Infinity;
+    requests = new Set();
     /** What the channel is playing, when the store is to be told. */
     media;
     transcriptId;
@@ -186,6 +192,8 @@ class Captioner {
     flush = null;
     /** Translations in order, per language: a slow one must not overtake the next. */
     chains = new Map();
+    translating = new Set();
+    nextTranslation = new Map();
     constructor(id, options, onStop) {
         this.id = id;
         this.options = options;
@@ -248,7 +256,7 @@ class Captioner {
         const session = this.options.session();
         if (session === null)
             return;
-        const got = await fetchTranscript(session, this.transcriptId, language, this.options.fetcher ?? fetch);
+        const got = await fetchTranscript(session, this.transcriptId, language, this.options.fetcher ?? fetch, true);
         if (this.stopped)
             return;
         if (!got.ok) {
@@ -256,11 +264,12 @@ class Captioner {
                 this.complain(`the store did not answer: ${got.error}`);
             return;
         }
-        this.known.set(language, got.body.lines);
-        if (language === "" && this.language === "" && got.body.language)
-            this.language = got.body.language;
-        if (got.body.lines.length > 0) {
-            this.options.onEvent?.(`captions for "${this.id}": the store knows ${got.body.lines.length} lines of this${language ? ` in ${language}` : ""}`);
+        // A row's model/language could have been updated while its old bad lines remained.
+        // Trust individual lines made with the corrected native pipeline only.
+        const usable = got.body.lines.filter((line) => line.revision === NATIVE_REVISION && reliableText(line.text, line.end - line.start));
+        this.known.set(language, usable);
+        if (usable.length > 0) {
+            this.options.onEvent?.(`captions for "${this.id}": the store knows ${usable.length} lines of this${language ? ` in ${language}` : ""}`);
         }
     }
     onPcm(pcm) {
@@ -275,7 +284,9 @@ class Captioner {
             const rest = all.subarray(size);
             this.pending = rest.length > 0 ? [Buffer.from(rest)] : [];
             this.pendingBytes = rest.length;
-            const until = this.now();
+            const duration = this.options.windowMs ?? WINDOW_MS;
+            const until = Math.max(this.audioUntil + duration, this.now() - rest.length / (RATE * 2) * 1000);
+            this.audioUntil = until;
             const index = this.windows;
             this.windows += 1;
             void this.hear(Buffer.from(window), until - (this.options.windowMs ?? WINDOW_MS), until, index);
@@ -294,7 +305,8 @@ class Captioner {
                         continue;
                     this.readOut.add(line.start);
                     const lineAt = at + (line.start - span.start) * 1000;
-                    this.emit({ channel: this.id, at: lineAt, until: lineAt + (line.end - line.start) * 1000, text: line.text, ...(this.language ? { language: this.language } : {}) }, line.start);
+                    this.emit({ channel: this.id, at: lineAt, until: lineAt + (line.end - line.start) * 1000, text: line.text,
+                        ...(line.language ? { language: line.language } : {}), ...(line.voiceProfile ? { voiceProfile: line.voiceProfile } : {}) }, line.start);
                 }
                 return;
             }
@@ -307,43 +319,52 @@ class Captioner {
             return;
         }
         this.inFlight += 1;
+        const controller = new AbortController();
+        this.requests.add(controller);
+        const timeout = setTimeout(() => controller.abort(), LIVE_DEADLINE_MS);
         try {
             const wav = wavAround(pcm);
             const url = new URL(`${session.site.replace(/\/+$/, "")}/api/v1/speech/transcribe`);
-            if (this.language)
-                url.searchParams.set("language", this.language);
+            // Let each audio window detect its own language. Cached text and a viewer's
+            // translation selection must never constrain the recognizer.
+            url.searchParams.set("live", "1");
             const response = await (this.options.fetcher ?? fetch)(url.toString(), {
                 method: "POST",
                 headers: { authorization: `Bearer ${session.token}`, "content-type": "audio/wav" },
                 body: new Blob([wav.buffer.slice(wav.byteOffset, wav.byteOffset + wav.byteLength)]),
+                signal: controller.signal,
             });
             const body = (await response.json().catch(() => ({})));
             if (!response.ok) {
                 this.complain(body.error ?? `nixamp.com answered ${response.status}`);
                 return;
             }
-            const text = (body.text ?? "").trim();
-            if (text === "" || this.stopped)
+            const text = reliableText(body.text ?? "", (until - at) / 1000);
+            if (text === "" || this.stopped || controller.signal.aborted || this.now() - until > LIVE_DEADLINE_MS)
                 return;
             this.error = "";
-            if (body.language && this.language === "")
-                this.language = body.language;
             if (body.model)
                 this.model = body.model;
-            const line = { channel: this.id, at, until, text, ...(this.language ? { language: this.language } : {}) };
+            const line = { channel: this.id, at, until, text, ...(body.language ? { language: body.language } : {}), voiceProfile: voiceProfile(pcm) };
             this.emit(line, span?.start ?? null);
             if (span)
-                this.keep("", { start: span.start, end: span.end, text });
+                this.keep("", { start: span.start, end: span.end, text, language: line.language, voiceProfile: line.voiceProfile, revision: NATIVE_REVISION });
         }
         catch (error) {
             this.complain(`could not reach the ear: ${error.message}`);
         }
         finally {
+            clearTimeout(timeout);
+            this.requests.delete(controller);
             this.inFlight -= 1;
         }
     }
     /** A line as heard, to whoever wants the original, and translated to whoever wants another language. */
     emit(line, mediaStart) {
+        if (line.at <= this.lastEmittedAt)
+            return;
+        this.lastEmittedAt = line.at;
+        this.language = line.language ?? "";
         this.lines.push(line);
         while (this.lines.length > KEEP)
             this.lines.shift();
@@ -372,32 +393,41 @@ class Captioner {
     }
     /** The line in another language: from the store when it has been through this moment, from nixamp.com otherwise. */
     translated(language, line, mediaStart) {
-        const chain = (this.chains.get(language) ?? Promise.resolve()).then(async () => {
-            if (this.stopped)
+        // One active request and the latest pending line per target, never a promise
+        // chain containing minutes of stale commentary.
+        if (this.translating.has(language)) {
+            this.nextTranslation.set(language, { line, mediaStart });
+            return;
+        }
+        this.translating.add(language);
+        const chain = Promise.resolve().then(async () => {
+            if (this.stopped || !this.wanted().has(language) || this.now() - line.until > LIVE_DEADLINE_MS)
                 return;
             let text = "";
             const stored = mediaStart === null ? null : lineAt(this.known.get(language) ?? [], mediaStart);
-            if (stored) {
+            if (stored && stored.original === line.text) {
                 text = stored.text;
             }
             else {
                 const session = this.options.session();
                 if (session === null)
                     return;
-                const got = await translateTexts(session, [line.text], this.language, language, this.options.fetcher ?? fetch);
+                if (!line.language)
+                    return;
+                const got = await translateTexts(session, [line.text], line.language, language, this.options.fetcher ?? fetch, AbortSignal.timeout(LIVE_DEADLINE_MS));
                 if (!got.ok) {
                     this.complain(`could not translate to ${language}: ${got.error}`);
                     return;
                 }
-                text = (got.body.texts[0] ?? "").trim();
+                text = reliableText(got.body.texts[0] ?? "", (line.until - line.at) / 1000);
                 if (text === "")
                     return;
                 if (mediaStart !== null)
-                    this.keep(language, { start: mediaStart, end: round(mediaStart + (line.until - line.at) / 1000), text });
+                    this.keep(language, { start: mediaStart, end: round(mediaStart + (line.until - line.at) / 1000), text, original: line.text, revision: NATIVE_REVISION });
             }
-            if (this.stopped)
+            if (this.stopped || !this.wanted().has(language) || this.now() - line.until > LIVE_DEADLINE_MS)
                 return;
-            const said = { ...line, text, language, original: line.text };
+            const said = { ...line, text, language, sourceLanguage: line.language, original: line.text };
             const lines = this.linesBy.get(language) ?? [];
             lines.push(said);
             while (lines.length > KEEP)
@@ -407,7 +437,13 @@ class Captioner {
                 if (wanted === language)
                     this.tell(subscriber, said);
         });
-        this.chains.set(language, chain.catch(() => undefined));
+        this.chains.set(language, chain.catch(() => undefined).finally(() => {
+            this.translating.delete(language);
+            const next = this.nextTranslation.get(language);
+            this.nextTranslation.delete(language);
+            if (next && !this.stopped)
+                this.translated(language, next.line, next.mediaStart);
+        }));
     }
     /** A line for the store, kept with the others of its language until the next flush. */
     keep(language, line) {
@@ -441,7 +477,7 @@ class Captioner {
             const got = await keepLines(session, this.transcriptId, {
                 media: this.media.media,
                 title: this.media.title,
-                language: language === "" ? this.language : language,
+                language,
                 ...(language === "" ? {} : { translatedFrom: this.language }),
                 ...(this.model ? { model: this.model } : {}),
                 lines,
@@ -478,7 +514,7 @@ class Captioner {
         };
     }
     recent(after, language = "") {
-        const lines = language === "" || language === this.language ? this.lines : (this.linesBy.get(language) ?? []);
+        const lines = language === "" ? this.lines : [...this.lines.filter((line) => line.language === language), ...(this.linesBy.get(language) ?? [])].sort((a, b) => a.at - b.at);
         return after > 0 ? lines.filter((line) => line.at > after) : [...lines];
     }
     status() {
@@ -496,6 +532,10 @@ class Captioner {
         if (this.stopped)
             return;
         this.stopped = true;
+        for (const request of this.requests)
+            request.abort();
+        this.requests.clear();
+        this.nextTranslation.clear();
         if (this.idle)
             clearTimeout(this.idle);
         this.idle = null;
@@ -520,6 +560,28 @@ export class Captions {
     available() {
         return this.options.session() !== null;
     }
+    capacity(id) { return this.running.has(id) || this.running.size < MAX_CAPTIONERS; }
+    /** Voice requests are constrained to captions this channel actually produced. */
+    async voiceRequest(id, at, language, voice, signal, grant = "") {
+        const session = this.options.session();
+        if (!session)
+            throw new SpeechError("this server must sign in to use translated audio", 503);
+        if (at === null)
+            return (this.options.fetcher ?? fetch)(`${session.site.replace(/\/+$/, "")}/api/v1/speech/voices`, {
+                headers: { authorization: `Bearer ${session.token}` }, signal,
+            });
+        if (!language)
+            throw new SpeechError("choose a translation language first", 400);
+        if (!/^nxd_[A-Za-z0-9_-]{43}$/.test(grant))
+            throw new SpeechError("sign in to enable translated audio", 401);
+        const line = this.recent(id, 0, language).find(line => line.at === at);
+        if (!line || (this.options.now ?? Date.now)() - line.until > 30_000)
+            throw new SpeechError("that live caption is no longer available for audio", 404);
+        return (this.options.fetcher ?? fetch)(`${session.site.replace(/\/+$/, "")}/api/v1/speech/synthesize`, {
+            method: "POST", headers: { authorization: `Bearer ${grant}`, "content-type": "application/json" },
+            body: JSON.stringify({ text: line.text, language, voice, profile: line.voiceProfile, channel: id }), signal,
+        });
+    }
     /**
      * Lines for a channel as they are heard, starting the captioner if it is
      * not running. Null when there is no such channel. The returned function
@@ -528,6 +590,8 @@ export class Captions {
      * "" is the original.
      */
     subscribe(id, subscriber, language = "") {
+        if (!this.capacity(id))
+            return null;
         let captioner = this.running.get(id);
         if (!captioner) {
             const made = new Captioner(id, this.options, () => {

package/dist/live-voice.d.ts

@@ -0,0 +1,70 @@
+import { type SpeakerTranscript } from "./speaker-turns.ts";
+import type { VoiceProfile } from "./voice-profile.ts";
+import type { Queryable } from "./follows.ts";
+export declare const LIVE_VOICE_MODEL = "eleven_flash_v2_5";
+export declare const LIVE_VOICE_RATE = 16000;
+export declare const LIVE_VOICE_LANGUAGES: Set<string>;
+export interface LiveVoiceChoice {
+    id: string;
+    name: string;
+    gender: string;
+    language: string;
+}
+export interface VoiceRequest {
+    text: string;
+    language: string;
+    voice?: string;
+    profile?: VoiceProfile;
+    channel?: string;
+}
+export declare class LiveVoice {
+    private readonly key;
+    private readonly fetcher;
+    private readonly now;
+    private catalog;
+    private catalogUntil;
+    private readonly cache;
+    private readonly pending;
+    private readonly usage;
+    private readonly charsPerMinute;
+    private readonly requests;
+    private readonly grants;
+    private readonly activeBy;
+    private readonly db?;
+    private schema;
+    private readonly dailyChars;
+    private cleanupAt;
+    private readonly hearing;
+    private readonly dailyAudioSeconds;
+    private readonly userDailyChars;
+    private readonly userDailyAudioSeconds;
+    constructor(options?: {
+        apiKey?: string;
+        fetcher?: typeof fetch;
+        now?: () => number;
+        charsPerMinute?: number;
+        dailyChars?: number;
+        dailyAudioSeconds?: number;
+        userDailyChars?: number;
+        userDailyAudioSeconds?: number;
+        db?: Queryable;
+    });
+    available(): boolean;
+    /** Optional diarization, billed only while a signed-in listener requests it.
+     * Rolling audio is bounded to 15 seconds, including overlap. Every second
+     * submitted (also repeated context) consumes the persistent provider budget. */
+    hear(bytes: Uint8Array, by: string, signal?: AbortSignal): Promise<SpeakerTranscript>;
+    voices(): Promise<LiveVoiceChoice[]>;
+    /** A 90-second capability for one channel, never the viewer's account credential. */
+    grant(by: string, channel: string): Promise<{
+        token: string;
+        expires: number;
+    }>;
+    authorize(token: string, channel: string, chars: number): Promise<string>;
+    private ensure;
+    private reserve;
+    private charge;
+    private checkRequest;
+    /** Stream the first request immediately; concurrent listeners share its cached result. */
+    stream(ask: VoiceRequest, by: string, signal?: AbortSignal): Promise<Response>;
+}