echogarden 1.4.4 → 1.5.0

package/docs/API.md

@@ -72,10 +72,13 @@ const { audio } = await Echogarden.synthesize("Hello World!", { engine: 'espeak'
 {
 	index: number              // Index of part
 	total: number              // Total number of parts
+
 	audio: RawAudio | Buffer   // Audio for part
+
 	timeline: Timeline         // Timeline for part
 	transcript: string         // Transcript for part
 	language: string           // Language for part
+
 	peakDecibelsSoFar: number  // Peak decibels measured for all synthesized audio, so far
 }
 ```
@@ -109,9 +112,15 @@ Applies speech recognition to the input.
 ```ts
 {
 	transcript: string
+
 	timeline: Timeline
 	wordTimeline: Timeline
+
 	language: string
+
+	inputRawAudio: RawAudio
+	isolatedRawAudio?: RawAudio
+	backgroundRawAudio?: RawAudio
 }
 ```
 
@@ -131,8 +140,13 @@ Aligns input audio with the given transcript.
 {
 	timeline: Timeline
 	wordTimeline: Timeline
+
 	transcript: string
 	language: string
+
+	inputRawAudio: RawAudio
+	isolatedRawAudio?: RawAudio
+	backgroundRawAudio?: RawAudio
 }
 ```
 
@@ -140,7 +154,7 @@ Aligns input audio with the given transcript.
 
 ### `translateSpeech(input, options)`
 
-Translates speech audio directly to English text.
+Translates speech audio directly to a transcript in a different language (only English is currently supported).
 
 * `input`: can be an audio file path (`string`), encoded audio (`Buffer` or `Uint8array`) or a raw audio object (`RawAudio`)
 * `options`: speech translation options object
@@ -150,9 +164,14 @@ Translates speech audio directly to English text.
 {
 	transcript: string
 	timeline: Timeline
-	wordTimeline: Timeline
+	wordTimeline?: Timeline
+
 	sourceLanguage: string
 	targetLanguage: string
+
+	inputRawAudio: RawAudio
+	isolatedRawAudio?: RawAudio
+	backgroundRawAudio?: RawAudio
 }
 ```
 
@@ -172,8 +191,67 @@ Aligns input audio with the given translated transcript.
 {
 	timeline: Timeline
 	wordTimeline: Timeline
+
+	translatedTranscript: string
+	sourceLanguage: string
+	targetLanguage: string
+
+	inputRawAudio: RawAudio
+	isolatedRawAudio?: RawAudio
+	backgroundRawAudio?: RawAudio
+}
+```
+
+### `alignTranscriptAndTranslation(input, transcript, translatedTranscript, options)`
+
+Aligns input audio to both the native language transcript a translated one.
+
+* `input`: can be an audio file path (`string`), encoded audio (`Buffer` or `Uint8array`) or a raw audio object (`RawAudio`)
+* `transcript`: the transcript to align to, in the native speech language
+* `translatedTranscript`: the translated transcript to align to
+* `options`: transcript and translation alignment options object
+
+#### Returns (via promise):
+
+```ts
+{
+	timeline: Timeline
+	wordTimeline: Timeline
+
+	translatedTimeline: Timeline
+	translatedWordTimeline: Timeline
+
 	transcript: string
-	language: string
+	translatedTranscript: string
+
+	sourceLanguage: string
+	targetLanguage: string
+
+	inputRawAudio: RawAudio
+	isolatedRawAudio?: RawAudio
+	backgroundRawAudio?: RawAudio
+}
+```
+
+### `alignTimelineTranslation(inputTimeline, translatedTranscript, options)`
+
+Aligns given timeline with its translated transcript.
+
+* `inputTimeline`: input timeline in the native language
+* `translatedTranscript`: the translated transcript to align to
+* `options`: timeline translation alignment options object
+
+#### Returns (via promise):
+
+```ts
+{
+	timeline: Timeline
+	wordTimeline: Timeline
+
+	sourceLanguage?: string
+	targetLanguage: string
+
+	rawAudio?: RawAudio
 }
 ```
 

package/docs/CLI.md

@@ -106,7 +106,8 @@ echogarden translate-speech speech.mp3 translation.txt --sourceLanguage=es --tar
 
 ## Speech-to-translated-transcript alignment
 
-**Task**: Given an audio file and its translated transcript, try to approximate the timing of the start and end of each translated word.
+### Direct alignment (English target only)
+**Task**: Given a spoken audio file and its English translated transcript, try to approximate the timing of the start and end of each translated word.
 
 This would align the audio file `dutch-speech.mp3` with the translated transcript provided in `english-translation.txt`, and would play the synchronized result in the terminal:
 ```bash
@@ -118,6 +119,55 @@ This would align the audio file `dutch-speech.mp3` with the translated transcrip
 echogarden align-translation dutch-speech.mp3 english-translation.txt result.srt result.json
 ```
 
+### Two-stage alignment (any of 96 source and target languages, combined stages)
+**Task**: Given a spoken audio file, its transcript, and its translated transcript, try to approximate the timing of the start and end of each translated word.
+
+This would align the audio file `dutch-speech.mp3` with the Dutch (native language) transcript provided in `dutch-transcript.txt` and the translated transcript provided in `russian-translation.txt`, and would play the synchronized result in the terminal:
+
+```bash
+echogarden align-transcript-and-translation dutch-speech.mp3 dutch-transcript.txt russian-translation.txt
+```
+
+This would perform the same operation but write the results to disk:
+
+```bash
+echogarden align-transcript-and-translation dutch-speech.mp3 dutch-transcript.txt russian-translation.txt out.json out.srt
+```
+
+The output would include separate files for the native language and the translation language:
+
+```
+out.json
+out.srt
+
+out.translated.json
+out.translated.srt
+```
+
+### Two-stage alignment (any of 96 source and target languages, separate stages)
+
+**Task**: Given an audio file, its transcript, and its translated transcript, try to approximate the timing of the start and end of each translated word. Do this in two, separate stages.
+
+This manual two-step approach allows to reuse the already-aligned transcript in the next stage, possibly for several different translation languages. The method used for alignment is otherwise identical to `align-transcript-and-translation`.
+
+**Stage 1**:
+
+Align the audio with its native language transcript, to produce a timeline in the native language:
+```bash
+echogarden align dutch-speech.mp3 dutch-transcript.txt dutch-timeline.json
+```
+
+**Stage 2**:
+
+Align the resulting timeline with the target translation, and play the synchronized result in the terminal.
+
+```bash
+echogarden align-timeline-translation dutch-timeline.json russian-transcript.txt --audio=dutch-speech.mp3
+```
+
+(`--audio` is only used for previewing the result in the terminal. Otherwise, it is not necessary)
+
+
 ## Language detection
 
 **Task**: Given audio or textual input, try to identify which language it is spoken or written in.

package/docs/Engines.md

@@ -84,6 +84,30 @@ The goal here is to match (or "align") a given spoken recording in one language,
 
 * `whisper`: given a spoken recording in any of the [98 languages](https://platform.openai.com/docs/guides/speech-to-text/supported-languages) supported by Whisper, and an English translation of its transcript, the translated transcript is tokenized and then decoded, in order, using a guided approach, with any multilingual Whisper model, set to its `translate` task mode. In this way, the approximate mapping between the spoken audio and each word of the translation is estimated
 
+## Speech-to-transcript-and-translation alignment
+
+This is a two-stage approach for translation alignment, that can accept about 100 source and target languages:
+
+1. First, the spoken audio is aligned with the native language transcript
+2. Then, the resulting timeline is aligned with the translated text using semantic text-to-text alignment
+
+This approach is potentially faster than the single-stage one, because the first stage can use any alignment approach, including the default synthesis-based `dtw` engine, which is much faster than running a full speech recognition engine like Whisper.
+
+The second stage uses a multilingual text embedding model (currently defaults to [multilingual E5](https://huggingface.co/intfloat/multilingual-e5-base)) to produce a vector representation of both the transcript and the translated text tokens. Then, it applies DTW over the two vector sequences to align the tokens of the two languages. Since this stage works on tokens only (the audio is not involved), it is generally fast.
+
+In terms of accuracy, the two-stage approach can be more accurate than the alignment derived from the Whisper model in the one-stage approach, especially when compared to using the smaller Whisper models like `tiny` or `base`, and for source languages that Whisper generally isn't very good at, like Chinese, Japanese, and less common languages.
+
+For a non-English target language, this approach is currently the only one that can be used.
+
+## Timeline-to-translation alignment
+
+Aligns a timeline with a translation of its text. Does not involve the spoken audio itself.
+
+This is used for the second stage of the two-stage approach described above. It can also be used independently. It allows to reuse the same timeline to align with multiple translations, in different languages, without needing to redo the native-language alignment each time.
+
+Another use case it to take a timeline produced as part of synthesized or recognized speech and then align it with one or more translations of its text.
+
+* [`E5`](https://huggingface.co/intfloat/multilingual-e5-base) (`e5`): E5 is a multilingual text embedding model by Microsoft, supporting 100 languages. This model encodes sequences of text tokens to vectors, in such a way that words with similar meanings in different languages are mapped to similar vectors. This cross-language semantic vector encoding is then used for aligning the tokens between the timeline and the translation. Then, the translated words are mapped to the timestamps of the corresponding words in the timeline.
 
 ## Language detection
 

package/docs/Options.md

@@ -271,7 +271,8 @@ Applies to CLI operation: `align-translation`, API method: `alignTranslation`
 
 **General**:
 * `engine`: alignment algorithm to use, can only be `whisper`. Defaults to `whisper`
-* `language`: language code for the source audio ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), like `en`, `fr`, `zh`, etc. Auto-detected from audio if not set
+* `sourceLanguage`: language code for the source audio ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), like `en`, `fr`, `zh`, etc. Auto-detected from audio if not set
+* `targetLanguage`: language code for the translated transcript. Can only be `en` for now. Defaults to `en`
 * `crop`: crop to active parts using voice activity detection before starting. Defaults to `true`
 * `isolate`: apply source separation to isolate voice before starting alignment. Defaults to `false`
 * `subtitles`: prefix to provide options for subtitles. Options detailed in section for subtitles
@@ -284,6 +285,37 @@ Applies to CLI operation: `align-translation`, API method: `alignTranslation`
 * `whisper.encoderProvider`: encoder ONNX execution provider. See details in recognition section above
 * `whisper.decoderProvider`: decoder ONNX execution provider. See details in recognition section above
 
+## Speech-to-transcript-and-translation alignment
+
+Applies to CLI operation: `align-transcript-and-translation`, API method: `alignTranscriptAndTranslation`
+
+**General**:
+* `engine`: can only be `two-stage`. Defaults to `two-stage`
+* `sourceLanguage`: language code for the source audio ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), like `en`, `fr`, `zh`, etc. Auto-detected from audio if not set
+* `targetLanguage`: language code for the translated transcript. Can only be `en` for now. Defaults to `en`
+* `crop`: crop to active parts using voice activity detection before starting. Defaults to `true`
+* `isolate`: apply source separation to isolate voice before starting alignment. Defaults to `false`
+* `alignment`: prefix to provide options for alignment. Options detailed in section for alignment
+* `timelineAlignment`: prefix to provide options for timeline alignment. Options detailed in section for timeline alignment
+* `vad`: prefix to provide options for voice activity detection when `crop` is set to `true`. Options detailed in section for voice activity detection
+* `sourceSeparation`: prefix to provide options for source separation when `isolate` is set to `true`. Options detailed in section for source separation
+* `subtitles`: prefix to provide options for subtitles. Options detailed in section for subtitles
+
+## Timeline-to-translated-text alignment
+
+Applies to CLI operation: `align-timeline-translation`, API method: `alignTimelineTranslation`
+
+**General**:
+* `engine`: alignment engine to use. Can only be `e5`. Defaults to `e5`
+* `sourceLanguage`: language code for the source timeline. Auto-detected from timeline if not set
+* `targetLanguage`: language code for the translated transcript. Auto-detected if not set
+* `audio`: spoken audio to play when previewing the result in the CLI (not required or used by the alignment itself). Optional
+* `languageDetection`: prefix to provide options for language detection. Options detailed in section for text language detection
+* `subtitles`: prefix to provide options for subtitles. Options detailed in section for subtitles
+
+**E5**:
+* `e5.model`: E5 model to use. Defaults to `e5-small-fp16` (support for additional models will be added in the future)
+
 ## Language detection
 
 ### Speech language detection

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "echogarden",
-	"version": "1.4.4",
+	"version": "1.5.0",
 	"description": "An easy-to-use speech toolset. Includes tools for synthesis, recognition, alignment, speech translation, language detection, source separation and more.",
 	"author": "Rotem Dan",
 	"license": "GPL-3.0",
@@ -55,8 +55,8 @@
 		"echogarden": "./dist/cli/CLILauncher.js"
 	},
 	"dependencies": {
-		"@aws-sdk/client-polly": "^3.576.0",
-		"@aws-sdk/client-transcribe-streaming": "^3.576.0",
+		"@aws-sdk/client-polly": "^3.583.0",
+		"@aws-sdk/client-transcribe-streaming": "^3.583.0",
 		"@echogarden/espeak-ng-emscripten": "^0.1.2",
 		"@echogarden/fasttext-wasm": "^0.1.0",
 		"@echogarden/flite-wasi": "^0.1.1",
@@ -76,19 +76,19 @@
 		"command-exists": "^1.2.9",
 		"compromise": "^14.13.0",
 		"fs-extra": "^11.2.0",
-		"gaxios": "^6.5.0",
+		"gaxios": "^6.6.0",
 		"graceful-fs": "^4.2.11",
 		"html-escaper": "^3.0.3",
 		"html-to-text": "^9.0.5",
 		"import-meta-resolve": "^4.1.0",
-		"jieba-wasm": "^0.0.2",
-		"jsdom": "^24.0.0",
+		"jieba-wasm": "^1.0.0",
+		"jsdom": "^24.1.0",
 		"json5": "^2.2.3",
 		"kuromoji": "^0.1.2",
 		"microsoft-cognitiveservices-speech-sdk": "^1.36.0",
 		"moving-median": "^1.0.0",
 		"msgpack-lite": "^0.1.26",
-		"onnxruntime-node": "^1.17.3",
+		"onnxruntime-node": "^1.18.0",
 		"openai": "^4.47.1",
 		"sam-js": "^0.2.1",
 		"strip-ansi": "^7.1.0",
@@ -124,7 +124,7 @@
 		"@types/recursive-readdir": "^2.2.4",
 		"@types/tar": "^6.1.13",
 		"@types/ws": "^8.5.10",
-		"ts-json-schema-generator": "^2.1.2-next.1",
+		"ts-json-schema-generator": "^2.2.0",
 		"typescript": "^5.4.5"
 	}
 }