echogarden 0.6.6 → 0.6.8

package/docs/Options.md

@@ -2,7 +2,7 @@
 
 Here is a detailed reference for the options accepted by the Echogarden API and CLI.
 
-Related resources:
+**Related resources**:
 * [A comprehensive list of all supported engines](Engines.md)
 * [A quick guide for using the command line interface](CLI.md)
 
@@ -10,7 +10,7 @@ Related resources:
 
 Applicable to CLI commands: `speak`, `speak-file`, `speak-url`, `speak-wikipedia`.
 
-General:
+**General**:
 * `engine`: identifier of the synthesis engine to use, such as `espeak` or `vits`.
 * `language`: language code ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected if not set
 * `voice`: name of the voice to use. Can be a search string. Optional
@@ -23,11 +23,11 @@ General:
 * `sentenceEndPause`: pause duration (seconds) at end of sentence. Defaults to `0.75`
 * `segmentEndPause`: pause duration (seconds) at end of segment. Defaults to `1.0`
 
-Plain text processing:
+**Plain text processing**:
 * `plainText.paragraphBreaks`: split to paragraphs based on single (`single`), or double (`double`) line breaks. Defaults to `double`
 * `plainText.whitespace`: determines how to process whitespace within paragraphs. Can be `preserve` (leave as is), `removeLineBreaks` (convert line breaks to spaces) or `collapse` (convert runs of whitespace characters, including line breaks, to a single space character). Defaults to `collapse`
 
-Post-processing:
+**Post-processing**:
 * `postProcessing.normalizeAudio`: should normalize output audio. Defaults to `true`
 * `postProcessing.targetPeakDb`: target peak (decibels) for normalization. Defaults to `-3`
 * `postProcessing.maxIncreaseDb`: max gain increase (decibels) when performing normalization. Defaults to `30`
@@ -36,62 +36,62 @@ Post-processing:
 * `postProcessing.timePitchShiftingMethod`: method for time and pitch shifting. Can be `sonic` or `rubberband`. Defaults to `sonic`
 * `postProcessing.rubberband`: prefix for RubberBand options (TODO)
 
-Language detection:
+**Language detection**:
 * `languageDetection`: prefix to provide custom options for language detection. Same options as detailed for text language detection, down below
 
-VITS:
+**VITS**:
 * `vits.speakerId`: speaker ID, for VITS models that support multiple speakers. Optional
 * `vits.customLexiconPaths`: an array of custom lexicon file paths. Optional
 
-eSpeak-ng:
+**eSpeak-ng**:
 * `espeak.rate`: speech rate, in eSpeak units. Overrides `speed` when set
 * `espeak.pitch`: pitch, in eSpeak units. Overrides `pitch` when set
 * `espeak.pitchRange`: pitch range, in eSpeak units. Overrides `pitchVariation` when set
 
-SAM:
+**SAM**:
 * `sam.pitch`: pitch value, between `0`..`255`. Overrides `pitch` when set
 * `sam.speed`: speed value, between `0`..`255`. Overrides `speed` when set
 * `sam.mouth`: mouth value, between `0`..`255` (defaults to `128`)
 * `sam.throat`: throat value, between `0`..`255` (defaults to `128`)
 
-SAPI:
+**SAPI**:
 * `sapi.rate`: SAPI speech rate, in its native units. An integer number between `-10` and `10`. Setting `speed` would apply time stretching instead. The two options can be used together
 
-Microsoft Speech Platform:
+**Microsoft Speech Platform**:
 * `msspeech.rate`: same  units and effects as the SAPI speech rate
 
-Coqui Server:
+**Coqui Server**:
 * `coquiServer.serverUrl`: server URL
 * `coquiServer.speakerId`: speaker ID (if applicable)
 
-Google Cloud:
+**Google Cloud**:
 * `googleCloud.apiKey`: API key (required)
 * `googleCloud.pitchDeltaSemitones`: pitch delta in semitones. Overrides `pitch` when set
 * `googleCloud.customVoice.model`: name of custom voice
 * `googleCloud.customVoice.reportedUsage`: reported usage of custom voice
 
-Azure Cognitive Services:
+**Azure Cognitive Services**:
 * `microsoftAzure.subscriptionKey`: subscription key (required)
 * `microsoftAzure.serviceRegion`: service region (required)
 * `microsoftAzure.pitchDeltaHz`: pitch delta in Hz. Overrides `pitch` when set
 
-Amazon Polly:
+**Amazon Polly**:
 * `amazonPolly.region`: region (required)
 * `amazonPolly.accessKeyId`: access key ID (required)
 * `amazonPolly.secretAccessKey`: secret access key (required)
 * `amazonPolly.pollyEngine`: Amazon Polly engine kind, can be `standard` or `neural`. Defaults to `neural`
 * `amazonPolly.lexiconNames`: An array of lexicon names. Optional
 
-Elevenlabs:
+**Elevenlabs**:
 * `elevenLabs.apiKey`: API key (required)
 * `elevenLabs.modelId`: Model identifier. Defaults to `eleven_monolingual_v1`
 * `elevenLabs.stability`: stability. Defaults to `0.5`
 * `elevenLabs.similarityBoost`: similarity boost. Defaults to `0.5`
 
-Google Translate:
+**Google Translate**:
 * `googleTranslate.tld`: top level domain to to connect to. Can change the dialect for a small number or voices. For example `us` gives American English for `en`, while `com` gives British English for `en`. Defaults to `us`
 
-Microsoft Edge:
+**Microsoft Edge**:
 * `microsoftEdge.trustedClientToken`: trusted client token (required). A special token required to use the service
 * `microsoftEdge.pitchDeltaHz`: pitch delta in Hz. Overrides `pitch` when set
 
@@ -99,31 +99,31 @@ Microsoft Edge:
 
 Applicable to CLI command: `transcribe`.
 
-General:
+**General**:
 * `engine`: identifier of the recognition engine to use, such as `whisper` or `vosk`
 * `language`: language code ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)) for the audio, like `en`, `fr`, `de`. Auto-detected if not set
 
-Whisper:
+**Whisper**:
 * `whisper.model`: selects which Whisper model to use. Can be `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large` (same as `large-v2`), `large-v1`, `large-v2`. Defaults to `tiny`
 
-Vosk:
+**Vosk**:
 * `vosk.modelPath`: path to the Vosk model to be used
 
-Silero:
+**Silero**:
 * `silero.modelPath`: path to a Silero model. Note that latest `en`, `de`, `fr` and `uk` models are automatically installed when needed based on the selected language. This should only be used to manually specify a different model, otherwise specify `language` instead
 
-Google Cloud:
+**Google Cloud**:
 * `googleCloud.apiKey`: Google Cloud API key (required)
 * `googleCloud.alternativeLanguageCodes`: An array of alternative language codes. Optional
 * `googleCloud.profanityFilter`: censor profanity. Defaults to `false`
 * `googleCloud.autoPunctuation`: add punctuation automatically. Defaults to `true`
 * `googleCloud.useEnhancedModel`: use enhanced model. Defaults to `true`
 
-Azure Cognitive Services:
+**Azure Cognitive Services**:
 * `azureCognitiveServices.subscriptionKey`: subscription key (required)
 * `azureCognitiveServices.serviceRegion`: service region (required)
 
-Amazon Transcribe:
+**Amazon Transcribe**:
 * `amazonTranscribe.region`: region (required)
 * `amazonTranscribe.accessKeyId`: access key ID (required)
 * `amazonTranscribe.secretAccessKey`: secret access key (required)
@@ -132,14 +132,14 @@ Amazon Transcribe:
 
 Applicable to CLI command: `align`.
 
-General:
+**General**:
 * `method`: what alignment algorithm to use, can be `dtw`, `dtw-ra` or `whisper`. Defaults to `dtw`
 * `language`: language code for the audio and transcript ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected from transcript if not set
 
-DTW:
+**DTW**:
 * `dtw.windowDuration`: time duration (in seconds) of the Sakoe-Chiba window when performing DTW alignment. Defaults to `120`. If your audio is longer than two minutes, consider increasing this value for better results. Note that a higher value would consume quadratically larger amounts of memory. A value of `600` (10 minutes) would already require several Gigabytes of memory when the audio duration is 10 minutes or greater.
 
-DTW-RA only:
+**DTW-RA only**:
 * `dtw.recognition`: prefix for providing custom recognition options when using `dtw-ra` method, for example: setting `dtw.recognition.engine = silero`
 * `dtw.phoneAlignmentMethod`: algorithm to use when aligning phones: can either be set to `dtw` or `interpolate`. Defaults to `dtw`
 
@@ -147,12 +147,12 @@ DTW-RA only:
 
 Applicable to CLI command: `translate-speech`.
 
-General:
+**General**:
 * `engine`: only `whisper` supported
 * `sourceLanguage`: the source language code for the input speech. Auto-detected if not set
 * `targetLanguage`: the target language code for the output speech. Only `en` supported at this time.
 
-Whisper:
+**Whisper**:
 * `whisper.model`: Whisper model to use (multilingual engines only). Defaults to `tiny`
 
 ## Language detection
@@ -176,25 +176,25 @@ Applicable to CLI command: `detect-text-langauge`.
 
 Applicable to CLI command: `detect-voice-activity`.
 
-General:
+**General**:
 * `engine`: VAD engine to use. Can be `webrtc`, `silero` or `rnnoise`. Defaults to `webrtc`
 * `activityThreshold`: minimum predicted probability for determining a frame as having speech activity. Defaults to `0.5`
 
-WebRTC:
+**WebRTC**:
 * `webrtc.frameDuration`: WebRTC frame duration (ms). Can be `10`, `20` or `30`. Defaults to `30`
 * `webrtc.mode`: WebRTC mode (aggressiveness). Can be `0`, `1`, `2` or `3`. Defaults to `1`
 
-Silero:
+**Silero**:
 * `silero.frameDuration`: Silero frame duration (ms). Can be `30`, `60` or `90`. Defaults to `90`
 
 ## Speech denoising
 
 Applicable to CLI command: `denoise`.
 
-General:
+**General**:
 * `engine`: can only be `rnnoise`
 
-Postprocessing:
+**Postprocessing**:
 * `postProcessing.normalizeAudio`: should normalize output audio. Defaults to `false`
 * `postProcessing.targetPeakDb`: target peak (decibels) for normalization. Defaults to `-3`
 * `postProcessing.maxIncreaseDb`: max gain increase (decibels) when performing normalization. Defaults to `30`
@@ -204,6 +204,13 @@ Postprocessing:
 
 Applicable to CLI command: `list-voices`.
 
+**General**
 * `language`: language code to filter by (optional)
 * `voice`: name or name pattern to filter by (optional)
 * `voiceGender`: gender to filter by (optional)
+
+Also accepted are engine-specific options that may be required in order to retrieve the voice list, especially for cloud engines. Examples:
+* `googleCloud.apiKey`
+* `microsoftAzure.subscriptionKey`, `microsoftAzure.serviceRegion`
+* `amazonPolly.region`, `amazonPolly.accessKeyId`, `amazonPolly.secretAccessKey`
+* `elevenLabs.apiKey`, `elevenLabs.modelId`

package/docs/Tasklist.md

@@ -42,14 +42,13 @@
 * Minimum size when iterating text nodes to get handle
 
 ### Worker
-* Optionally omit unnecessary data from the response (decoded input, segment data, etc.)
-* Support compressed audio in response
 * Add cancelation checks in more operations
+* Optionally omit unnecessary data from the response (decoded input, segment data, etc.)
 * Support more operations
+* Support compressed audio in response
 
 ### CLI
 * Colors in log messages
-* Find a way to ensure that a user who typed `align audio.mp3 transcript.txt` and then changed to `transcribe audio.mp3 transcript.txt` won't accidently overwrite their transcript file. Simple solution, but possibly not the best solution: `align audio.mp3 --reference=transcript.txt`. Other solution: on `transcribe` and `translate-speech`, ask if output file already exist or require an `--overwrite` flag to ensure that the user intended to overwrite the existing file.
 * Restrict input media file extensions to a set list to avoid cases where an output media file would be overwritten due to user error
 * Mode to print IPA words when speaking
 * Show a message when a new version is available
@@ -58,7 +57,7 @@
 * Option to set audio output device
 * Print available synthesis voices when no voice matches (or suggest near matches)
 * `transcribe` may also accept `http://` and `https://` URLs and pull the remote media file
-* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help giving better error messages when the given file type is wrong.
+* Make enum options case-insensitive if possible
 * Consider adding the input text offset to each segment, sentence and word in the resulting timeline with respect to the original file (even if it is, say, an HTML or captions file)
 * Add phone playback support
 * More fine-grained intermediate progress report for operations
@@ -66,7 +65,7 @@
 * Multiple configuration files in `--config=..` taking precedence by order
 * Support comments in the JSON configuration file
 * Generate JSON configuration file schema
-* Make enum options case-insensitive if possible
+* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help giving better error messages when the given file type is wrong.
 
 ### CLI / `speak`
 * Add support for sentence templates, like `echogarden speak-file text.txt /parts/[sentence].wav`.
@@ -85,7 +84,7 @@
 * Support filters
 
 ### CLI / New commands
-* `list-engines`: List available engines for a particular command, like `list-engines speak`
+* `speak-youtube`: To speak the subtitles of a YouTube video
 * `play-with-captions`: Preview captions in terminal
 * `play-with-timeline`: Preview timeline in terminal
 * `captions-to-text`, `captions-to-timeline`, `srt-to-vtt`, `vtt-to-srt`
@@ -94,22 +93,23 @@
 * `phonemize-text`
 * `normalize-text`
 * `remove-nonspeech`
-* `speak-youtube`: To speak the subtitles of a YouTube video
 
 ### API
 * Option to control logging verbosity
-* Accept full language names as language identifiers
 * Add support to accept caption options in API and CLI
 * Retry on error when connecting to cloud providers, including WebSocket disconnection with `microsoft-edge` (already supported by `gaxios`, not sure about `ws` - decide on default setting)
+* Accept full language names as language identifiers
 * Validate timelines to ensure timestamps are always increasing, no -1 timestamps or timestamps over the time of the audio, no sentences without words, etc. and correct if needed
-* Time/pitch shifting for recognition and alignment results
 * Add support for phrases in timelines
+* Time/pitch shifting for recognition and alignment results
 * Accept voice list caching options in `SynthesisOptions`
 
-### Language detection
-* Deploy and add the new language detection model
+### Speech Language detection
 * When using Whisper for language detection of speech, apply it to the entire audio, not just the first 30 seconds
 
+### Text Language detection
+* Deploy and add the new language detection model
+
 ### Segmentation
 * Split long words
 * See if it's possible to reliably use eSpeak as a segmentation engine
@@ -130,14 +130,12 @@
 * Ensure abbreviations like "Ph.d" or names like are segmented and read correctly (why doesn't `cldr` treat it as a word? Maybe it's not getting the right parameters, or it's not included in the list?) and "C#"
 * Find way to manually reset voice list cache
 * When synthesized text isn't pre-split to sentences, apply sentence splits by using the existing method to convert the output of word timelines to sentence/segment timelines
-* Log full language of selected voice (it may have a different dialect than expected)
-* Add partial SSML support for all engines. In particular, allow changing language or voice using the `<voice>` and `<lang>` tags, `<say-as>` and `<phoneme>` where possible.
 * Some `sapi` voices and `msspeech` languages output phones that are converted to Microsoft alphabet, not IPA symbols. Try to see if these can be translated to IPA
 * Decide whether asterisk `*` should be spoken when using `speak-url` or `speak-wikipedia`
-* Decide what to do with `«` and `»` punctuation characters (guillemets) when parsing and playing
+* Add partial SSML support for all engines. In particular, allow changing language or voice using the `<voice>` and `<lang>` tags, `<say-as>` and `<phoneme>` where possible.
 * Try to remove reliance on `()` after `.` character hack in `EspeakTTS.synthesizeFragments`.
 * eSpeak IPA output puts stress marks on vowels, not syllables - which is the standard for IPA. Consider how to make a conversion to and from these two approaches (possibly detect it automatically).
-* Investigate if `espeak` can be made to correctly support phonemizing and pronouncing the dot character like in `object.key`
+* Investigate if eSpeak can be made to correctly support phonemizing and pronouncing the dot character like in `object.key`
 * Speaker-specific voice option
 * Decide if `msspeech` engine should be selected if available. This would require attempting to load a matching voice, and falling back if it is not installed
 * Option to disable alignment
@@ -156,12 +154,15 @@
 * Try to use entity recognition to detect years, dates, currencies etc., which would disambiguate cases where it is not clear, like "in 1993" in "She was born in 1993" and "It searched in 1993 websites"
 * Option to add POS tags to timeline, if available
 
-### VITS
+### Synthesis / VITS
 * Allow to limit how many models are cached in memory
 * Custom model paths (decide how to implement)
 * Pull voice list from JSON file, or based on URL? Is that a good idea?
 * Add speaker names to voice list somehow
 
+### Synthesis / Azure Cognitive Services
+* Currently, when input is set to be SSML, it is wrapped in a `<speak>` tag. Handle the case where the user made their own document wrapped with a `<speak>` tag as well. Currently it may send invalid input to Azure
+
 ### Recognition
 * Add confidence to each recognized word, if available
 * Show alternatives when playing in the CLI. Clear current line and rewrite already printed text for alternatives during the speech recognition process
@@ -171,13 +172,15 @@
 ### Recognition / Whisper
 * When using `dtw-ra` alignment, pass the transcript as a prompt. Remove some of the initial transcript based on what has been detected (try to find the best matching initial segment between the transcript and recognized text, and remove it at each recognition window).
 * During language detection, if file is more than 30s, run the detection over all the segments and average the resulting probability distributions, consider how to handle very short segments
+* Log individual tokens to the terminal as they are being decoded from the model
+* Add sampling and temperature support to decoder
 * Timestamps extracted from cross-attention are still not as accurate as what the official Python implementation gets. Try to see if you can make them better.
 * Cache last model
 * Integrate speech language detection into the recognition itself, so it is done efficiently when the language is not known
 * Bring back option to use eSpeak DTW based alignment on segments, as an alternative approach
 * The segment output can be use to split to segment files, otherwise it is possible to try to guess using the pause lengths or voice activity detection
 * Way to specify model size, such that the English-only/multilingual would be auto selection for sizes other than `tiny`?
-* Accept custom prompt as through an option
+* Accept custom prompt through an option
 
 ### Alignment
 * Warn when input is larger than DTW window (this can also happen when synthesizing SSML, which can't be split to segments)
@@ -240,7 +243,7 @@
 * Live vosk alternatives events
 * Implement beam search for Whisper decoder
 * Implement beam search for Silero decoder
-* Investigate exporting Whisper models to 16-bit and 8-bit quantized ONNX
+* Investigate exporting Whisper models to 16-bit quantized ONNX or a mix of 16-bit and 32-bits
 
 ### Web
 * Web based frontend UI to the server

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "echogarden",
-	"version": "0.6.6",
+	"version": "0.6.8",
 	"description": "An integrated speech system, providing a range of synthesis, recognition and other processing tools designed to be directly accessible to end-users.",
 	"author": "Rotem Dan",
 	"license": "GPL-3.0-only",
@@ -92,6 +92,7 @@
 		"ndarray-ops": "^1.2.2",
 		"onnxruntime-node": "^1.15.1",
 		"sam-js": "^0.2.1",
+		"strip-ansi": "^7.1.0",
 		"tar": "^6.1.15",
 		"tinyld": "^1.3.4",
 		"ws": "^8.13.0",