echogarden 1.0.4 → 1.1.0

package/docs/Engines.md

@@ -5,17 +5,17 @@
 
 **Offline**:
 
-* [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Currently, there are 117 voices, in a range of languages, including English (US, UK), Spanish (ES, MX), Portuguese (PT, BR), Italian, French, German, Dutch (NL, BE), Swedish, Norwegian, Danish, Finnish, Polish, Greek, Romanian, Serbian, Czech, Hungarian, Slovak, Slovenian, Turkish, Arabic, Farsi, Russian, Ukrainian, Catalan, Luxembourgish, Icelandic, Swahili, Kazakh, Georgian, Nepali, Vietnamese and Chinese. You can listen to audio samples of all voices and languages in [Piper's samples page](https://rhasspy.github.io/piper-samples/).
-* [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a legacy diphone-based synthesis engine. Supports English (US, UK), Spanish, Italian, French, and German.
-* [Flite](https://github.com/festvox/flite) (`flite`): a legacy diphone-based synthesis engine. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi.
-* [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a lightweight "robot" sounding formant-based synthesizer. Supports 100+ languages. Extensively used internally for speech alignment, phonemization, and other internal tasks.
-* [SAM (Software Automatic Mouth)](https://github.com/discordier/sam) (`sam`): a classic "robot" speech synthesizer from 1982. English only.
+* [VITS](https://github.com/jaywalnut310/vits) (`vits`): end-to-end neural speech synthesis architecture. Available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Currently, there are 117 voices, in a range of languages, including English (US, UK), Spanish (ES, MX), Portuguese (PT, BR), Italian, French, German, Dutch (NL, BE), Swedish, Norwegian, Danish, Finnish, Polish, Greek, Romanian, Serbian, Czech, Hungarian, Slovak, Slovenian, Turkish, Arabic, Farsi, Russian, Ukrainian, Catalan, Luxembourgish, Icelandic, Swahili, Kazakh, Georgian, Nepali, Vietnamese and Chinese. You can listen to audio samples of all voices and languages in [Piper's samples page](https://rhasspy.github.io/piper-samples/)
+* [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a legacy diphone-based synthesis engine. Supports English (US, UK), Spanish, Italian, French, and German
+* [Flite](https://github.com/festvox/flite) (`flite`): a legacy diphone-based synthesis engine. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi
+* [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a lightweight "robot" sounding formant-based synthesizer. Supports 100+ languages. Used internally for speech alignment, phonemization, and other internal tasks
+* [SAM (Software Automatic Mouth)](https://github.com/discordier/sam) (`sam`): a classic "robot" speech synthesizer from 1982. English only
 
 **Offline, Windows only**:
 
-* [SAPI](https://en.wikipedia.org/wiki/Microsoft_Speech_API) (`sapi`): Microsoft Speech API. Supports the system's language voices, as well as legacy voices produced by third-party vendors, like Ivona, NeoSpeech, Acapela, Cepstral, CereProc, Nuance, AT&T, Loquendo, ScanSoft and others (note that only 64-bit SAPI voices are supported, which makes it incompatible with a significant portion of older voices).
+* [SAPI](https://en.wikipedia.org/wiki/Microsoft_Speech_API) (`sapi`): Microsoft Speech API. Supports the system's language voices, as well as legacy voices produced by third-party vendors, like Ivona, NeoSpeech, Acapela, Cepstral, CereProc, Nuance, AT&T, Loquendo, ScanSoft and others (note that only 64-bit SAPI voices are supported, which makes it incompatible with a significant portion of older voices)
 
-* [Microsoft Speech Platform](https://www.microsoft.com/en-us/download/details.aspx?id=27225) (`msspeech`): Microsoft Server Speech API. Requires [installing a runtime (2.6MB)](https://www.microsoft.com/en-us/download/details.aspx?id=27225). Supports 28 dialects, which can be individually downloaded via [freely available installers](https://www.microsoft.com/en-us/download/details.aspx?id=27224), or, for convenience, bundled as [a single 358MB zip file](https://drive.google.com/u/0/uc?id=1uQdFNxLzUxpaEwVVKhMawys8cIh3F21T&export=download). Has voices for English (US, UK, AU, CA), Spanish (ES, MX), Portuguese (BR, PT), German, French (FR, CA), Italian, Norwegian, Dutch, Russian, Swedish, Danish, Catalan, Finnish, Japanese, Korean and Chinese (ZH, HK, TW). All voices are female.
+* [Microsoft Speech Platform](https://www.microsoft.com/en-us/download/details.aspx?id=27225) (`msspeech`): Microsoft Server Speech API. Requires [installing a runtime (2.6MB)](https://www.microsoft.com/en-us/download/details.aspx?id=27225). Supports 28 dialects, which can be individually downloaded via [freely available installers](https://www.microsoft.com/en-us/download/details.aspx?id=27224), or, for convenience, bundled as [a single 358MB zip file](https://drive.google.com/u/0/uc?id=1uQdFNxLzUxpaEwVVKhMawys8cIh3F21T&export=download). Has voices for English (US, UK, AU, CA), Spanish (ES, MX), Portuguese (BR, PT), German, French (FR, CA), Italian, Norwegian, Dutch, Russian, Swedish, Danish, Catalan, Finnish, Japanese, Korean and Chinese (ZH, HK, TW). All voices are female
 
 **Note**: both these engines require manually installing the [`winax` npm package](https://www.npmjs.com/package/winax) by running `npm install winax -g`.
 
@@ -30,71 +30,82 @@
 These are commercial services that require a subscription and an API key to use:
 
 * [Google Cloud](https://cloud.google.com/text-to-speech) (`google-cloud`)
-* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/text-to-speech/) (`microsoft-azure`)
+* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/ai-services/text-to-speech/) (`microsoft-azure`)
 * [Amazon Polly](https://aws.amazon.com/polly/) (`amazon-polly`)
-* [OpenAI Cloud](https://platform.openai.com/) (`openai-cloud`)
+* [OpenAI Cloud Platform](https://platform.openai.com/) (`openai-cloud`)
 * [Elevenlabs](https://elevenlabs.io/) (`elevenlabs`)
 
 **Cloud services (unofficial)**:
 
 These cloud-based engines connect to public cloud APIs that are not officially publicized by their operators. They are included for educational purposes only, and may be removed in the future:
 
-* Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice for each language (usually female).
-* Microsoft Edge (`microsoft-edge`): subset of the Azure Cognitive Services cloud TTS API used by the Microsoft Edge browser as part of its support for the [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API) and its [Read Aloud](https://www.microsoft.com/en-us/edge/features/read-aloud?form=MT00D8) feature. Using this engine requires a special token, which should be passed via the `microsoftEdge.trustedClientToken` option.
-* Streamlabs Polly (`streamlabs-polly`): a public REST API by Streamlabs, primarily intended for generating speech for TTS donations. It includes a few English (US, UK, AU, IN) voices, which are similar to some of the non-neural (Ivona-based) voices offered by Amazon Polly (**Note**: as of April 2024, the public Streamlabs Polly REST API doesn't seem to be accessible anymore).
+* Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice for each language (usually female)
+* Microsoft Edge (`microsoft-edge`): subset of the Azure Cognitive Services cloud TTS API used by the Microsoft Edge browser as part of its support for the [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API) and its [Read Aloud](https://www.microsoft.com/en-us/edge/features/read-aloud?form=MT00D8) feature. Using this engine requires a special token, which should be passed via the `microsoftEdge.trustedClientToken` option
+* Streamlabs Polly (`streamlabs-polly`): a public REST API by Streamlabs, primarily intended for generating speech for TTS donations. It includes a few English (US, UK, AU, IN) voices, which are similar to some of the non-neural (Ivona-based) voices offered by Amazon Polly (**Note**: as of April 2024, the public Streamlabs Polly REST API doesn't seem to be accessible anymore)
 
 ## Speech-to-text
 
 **Offline**:
-* [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high accuracy transformer-based speech recognition architecture. Supports 99 languages. There are several models of different sizes, some are multilingual, and some are English only (`.en`): `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v1` and `large-v2`, `large-v3`. **Note**: large models are not currently supported by `onnxruntime-node` due to model size restrictions.
-* [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) (`whisper.cpp`): a port of the Whisper architecture to C++, by Georgi Gerganov. Supports all Whisper models, including several quantized ones (see full list in the options page). Has various different builds, including CUDA and OpenCL for GPU support.
-* [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for 25+ languages. **Note**: the Vosk package is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. Then, you'll need to manually [download a model](https://alphacephei.com/vosk/models) and specify its directory path via the `vosk.modelPath` option.
-* [Silero](https://github.com/snakers4/silero-models) (`silero`): models available for English, Spanish, German and Ukrainian. For [non-commercial use only](https://github.com/snakers4/silero-models/blob/master/LICENSE).
+* [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high-accuracy transformer-based speech recognition architecture. TypeScript implementation, with inference done via the [ONNX runtime](https://onnxruntime.ai/). Supports [98 languages](https://platform.openai.com/docs/guides/speech-to-text/supported-languages). There are several models of different sizes, some are multilingual, and some are English only: `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v1` and `large-v2`, `large-v3`. **Note**: large models are not currently supported by `onnxruntime-node` due to model size restrictions
+* [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) (`whisper.cpp`): a C++ port of the Whisper architecture by Georgi Gerganov. Supports all Whisper models, including several quantized ones (see full model list in the [options reference](docs/Options.md)). Has various builds, including CUDA and OpenCL for GPU support
+* [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for 25+ languages. **Note**: the Vosk package is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. Then, you'll need to manually [download a model](https://alphacephei.com/vosk/models) and specify its directory path via the `vosk.modelPath` option
+* [Silero](https://github.com/snakers4/silero-models) (`silero`): models available for English, Spanish, German and Ukrainian. For [non-commercial use only](https://github.com/snakers4/silero-models/blob/master/LICENSE)
 
 **Cloud services**:
 
 These are commercial services that require a subscription and an API key to use:
 
 * [Google Cloud](https://cloud.google.com/speech-to-text) (`google-cloud`)
-* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/speech-to-text/) (`microsoft-azure`)
+* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/ai-services/speech-to-text/) (`microsoft-azure`)
 * [Amazon Transcribe](https://aws.amazon.com/transcribe/) (`amazon-transcribe`)
-* [OpenAI Cloud](https://platform.openai.com/) (`openai-cloud`)
+* [OpenAI Cloud Platform](https://platform.openai.com/) (`openai-cloud`): runs the `large-v2` Whisper model on the cloud
 
 ## Speech-to-transcript alignment
 
 These engines' goal is to match (or "align") a given spoken recording with a given transcript as closely as possible. They will annotate each word in the transcript with approximate start and end timestamps:
 
-* Dynamic Time Warping (`dtw`): transcript is first synthesized using the eSpeak engine, then the [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) alignment algorithm is applied to find the best mapping between the synthesized and original audio frames.
-* Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using eSpeak. Then, the best mapping is found between the two synthesized waveforms, and the result is mapped back to the original audio using the timing information produced by the recognizer.
-* Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model. **Note**: currently, only supports audio inputs that are 30 seconds or less. Some words or special characters may fail to tokenize due to limitations of the tokenizer used by Whisper.
+* Dynamic Time Warping (`dtw`): transcript is first synthesized using the eSpeak engine, then the [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) sequence alignment algorithm is applied to find the best mapping between the synthesized and original audio frames
+* Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using eSpeak. Then, the best mapping is found between the two synthesized waveforms, using the DTW algorithm, and the result is remapped back to the original audio using the timing information produced by the recognizer
+* Whisper-based alignment (`whisper`): transcript is first tokenized, then, its tokens are decoded, in order, with a guided approach, using the Whisper model. The resulting token timestamps are then used to derive the timing for each word
+
 
 ## Speech-to-text translation
 
-* [Whisper](https://github.com/openai/whisper) (`whisper`): the Whisper model can recognize speech in any one of its supported languages and output a transcript directly translated to English. Other languages are not supported as targets.
+**Offline**:
+* [Whisper](https://github.com/openai/whisper) (`whisper`): the Whisper model can recognize speech in any one of its supported languages and output a transcript directly translated to English. Other languages are not supported as targets
 * [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) (`whisper.cpp`): supports translation to English
-* [OpenAI Cloud](https://platform.openai.com/) (`openai-cloud`): OpenAI cloud service. Only translates to English.
+
+**Cloud services**:
+* [OpenAI Cloud Platform](https://platform.openai.com/) (`openai-cloud`): runs the `large-v2` Whisper model on the cloud. Only supports English as target
+
+## Speech-to-translated-transcript alignment
+
+These goal here is to match (or "align") a given spoken recording in one language, with a given translated transcript in a different language, as closely as possible.
+
+* `whisper`: given a spoken recording in any of the [98 languages](https://platform.openai.com/docs/guides/speech-to-text/supported-languages) supported by Whisper, and an English translation of its transcript, the translated transcript is tokenized and then decoded, in order, using a guided approach, with any multilingual Whisper model, set to its `translate` task mode. In this way, the approximate mapping between the spoken audio and each word of the translation is estimated
+
 
 ## Language detection
 
 **Spoken language detection**:
-* [Silero Language Classifier](https://github.com/snakers4/silero-vad/wiki/Other-Models) (`silero`): a speech language classification model by Silero.
-* [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the 99 languages it has been trained on.
+* [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the [98 languages](https://platform.openai.com/docs/guides/speech-to-text/supported-languages) it has been trained on
+* [Silero Language Classifier](https://github.com/snakers4/silero-vad/wiki/Other-Models) (`silero`): a speech language classification model by Silero
 
 **Text language detection**:
-* [TinyLD](https://www.npmjs.com/package/tinyld) (`tinyld`): a simple language detection library.
-* [FastText](https://github.com/facebookresearch/fastText) (`fasttext`): a library for word representations and sentence classification by Facebook research.
+* [TinyLD](https://www.npmjs.com/package/tinyld) (`tinyld`): a simple language detection library
+* [FastText](https://github.com/facebookresearch/fastText) (`fasttext`): a library for word representations and sentence classification by Facebook research
 
 ## Voice activity detection
 
-* [WebRTC VAD](https://github.com/dpirch/libfvad) (`webrtc`): a voice activity detector. Originally from the Chromium browser source code.
+* [WebRTC VAD](https://github.com/dpirch/libfvad) (`webrtc`): a voice activity detector. Originally from the Chromium browser source code
 * [Silero VAD](https://github.com/snakers4/silero-vad) (`silero`): a voice activity detection model by Silero.
-* [RNNoise](https://github.com/xiph/rnnoise) (`rnnoise`): uses RNNoise's speech probabilities output for each audio frame as a VAD metric.
-* Adaptive Gate (`adaptive-gate`): uses a band-limited adaptive gate to identify activity in the lower voice frequencies. Reliable, but will often pass non-vocal sounds if they are loud enough. Good for clean speech and a cappella singing, where most non-vocal segments are quiet.
+* [RNNoise](https://github.com/xiph/rnnoise) (`rnnoise`): uses RNNoise's speech probabilities output for each audio frame as a VAD metric
+* Adaptive Gate (`adaptive-gate`): uses a band-limited adaptive gate to identify activity in the lower voice frequencies. Reliable, but will often pass non-vocal sounds if they are loud enough. Good for clean speech and a cappella singing, where most non-vocal segments are quiet
 
 ## Speech denoising
 
-* [RNNoise](https://github.com/xiph/rnnoise) (`rnnoise`): a noise suppression library based on a recurrent neural network.
+* [RNNoise](https://github.com/xiph/rnnoise) (`rnnoise`): a noise suppression library based on a recurrent neural network
 
 ## Source separation
 
-* [MDX-NET](https://github.com/kuielab/mdx-net) (`mdx-net`): Deep learning source separation architecture by [KUIELAB (Korea University)](https://kuielab.github.io/).
+* [MDX-NET](https://github.com/kuielab/mdx-net) (`mdx-net`): deep learning source separation architecture by [KUIELAB (Korea University)](https://kuielab.github.io/)

package/docs/Licenses.md

@@ -1,9 +1,8 @@
 # Echogarden components licensing
 
-## Engines and libraries:
+## Engines and libraries
 
 * `onnxruntime-node`: [MIT License](https://github.com/microsoft/onnxruntime/blob/main/LICENSE)
-* `whisper.cpp`: [MIT License](https://github.com/ggerganov/whisper.cpp/blob/master/LICENSE)
 * `espeak`: [GNU GPL v3](https://github.com/espeak-ng/espeak-ng/blob/master/COPYING)
 * `flite`: [BSD License](https://github.com/festvox/flite/blob/master/COPYING)
 * `pico`: [Apache License 2.0](https://github.com/gmn/nanotts/blob/master/LICENSE)
@@ -31,11 +30,11 @@ All are freely distributable, with varying licenses:
 * SVOX Pico resources (`pico-`): [Apache License 2.0](https://github.com/gmn/nanotts/blob/master/LICENSE)
 * Silero VAD (`silero-vad`) and Silero language classifier (`silero-lang-classifier-95`): [MIT License](https://github.com/snakers4/silero-vad/blob/master/LICENSE)
 * Silero speech recognition models (`silero-en-`, `silero-de-`, `silero-es-`, `silero-ua-`): [BY-NC-SA](https://github.com/snakers4/silero-models/blob/master/LICENSE)
-* VITS pre-trained models (`vits-`): licensed under various creative commons licenses: [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/), [CC-BY](https://creativecommons.org/licenses/by/4.0/) and [BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/), and few are public domain (you can view the individual license for each model in the model cards on the [samples page](https://rhasspy.github.io/piper-samples/)). The [Piper system](https://github.com/rhasspy/piper) itself is published under the [MIT License](https://github.com/rhasspy/piper/blob/master/LICENSE.md)
+* VITS pre-trained models (`vits-`): licensed under various creative commons licenses: [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/), [CC-BY](https://creativecommons.org/licenses/by/4.0/) and [BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/), and few are public domain. You can view the individual license for each model in the model cards on the [Piper samples page](https://rhasspy.github.io/piper-samples/)
 * Whisper pre-trained models (`whisper-`): [MIT License](https://github.com/openai/whisper/blob/main/LICENSE)
 * MDX-NET source separation models (`mdxnet-`): [MIT License](https://github.com/kuielab/mdx-net/blob/main/LICENSE)
 
-Tool binary distributions:
+Tool binary distributions
 * FFmpeg: [LGPL, GPL v2 and GPL v3 Licenses](https://github.com/FFmpeg/FFmpeg)
 * SoX: [GPL v2 License](https://github.com/chirlu/sox/blob/master/LICENSE.GPL)
 * whisper.cpp: [MIT License](https://github.com/ggerganov/whisper.cpp/blob/master/LICENSE)

package/docs/Options.md

@@ -4,7 +4,7 @@ Here's a detailed reference for all the options accepted by the Echogarden CLI a
 
 **Related pages**:
 * [List of all supported engines](Engines.md)
-* [Quick guide for the command line interface](CLI.md)
+* [Quick guide to the command line interface](CLI.md)
 * [Node.js API reference](API.md)
 
 ## Text-to-speech
@@ -46,7 +46,8 @@ Applies to CLI operations: `speak`, `speak-file`, `speak-url`, `speak-wikipedia`
 * `outputAudioFormat.bitrate`: Custom bitrate for encoding, applies only to  `mp3`, `opus`, `m4a`, `ogg`. By default, bitrates are selected between 48Kbps and 64Kbps, to provide a good speech quality while minimizing file size. Optional
 
 **VITS**:
-* `vits.speakerId`: speaker ID, for VITS models that support multiple speakers. Optional
+* `vits.speakerId`: speaker ID, for VITS models that support multiple speakers. Defaults to `0`
+* `vits.provider`: ONNX execution provider to use. Can be `cpu` or `dml` (https://microsoft.github.io/DirectML/)-based GPU acceleration - Windows only). Using GPU acceleration for VITS may or may not be faster than CPU, depending on your hardware. Defaults to `cpu`
 
 **eSpeak**:
 * `espeak.rate`: speech rate, in eSpeak units. Overrides `speed` when set
@@ -108,7 +109,7 @@ Applies to CLI operations: `speak`, `speak-file`, `speak-url`, `speak-wikipedia`
 * `microsoftEdge.trustedClientToken`: trusted client token (required). A special token required to use the service
 * `microsoftEdge.pitchDeltaHz`: pitch delta in Hz. Overrides `pitch` when set
 
-## Voice list request
+### Voice list request
 
 Applies to CLI operation: `list-voices`, API method: `requestVoiceList`
 
@@ -146,9 +147,11 @@ Applies to CLI operation: `transcribe`, API method: `recognize`
 * `whisper.topCandidateCount`: the number of top candidate tokens to consider. Defaults to `5`
 * `whisper.punctuationThreshold`: the minimal probability for a punctuation token, included in the top candidates, to be chosen unconditionally. A lower threshold encourages the model to output more punctuation symbols. Defaults to `0.2`
 * `whisper.autoPromptParts`: use previous part's recognized text as prompt for the next part. Disabling this may help to prevent repetition carrying over between parts, in some cases. Defaults to `true`
-* `whisper.maxTokensPerPart`: maximum number of tokens to decode for each 30 second audio part. Defaults to `250`
-* `whisper.suppressRepetition`: attempt to suppress decoding repeating token patterns. Defaults to `true`
-* `whisper.decodeTimestampTokens`: enable/disable decoding of timestamp tokens, since more accurate timing is already extracted via cross-attention weight alignment. For unclear reasons, setting to `false` can significantly reduce the occurrence of hallucinations and token repetition loops, and increases word timestamp accuracy. However, there are cases where this can cause the model to end a part prematurely, especially in singing and less speech-like voice segments, or when there are multiple speakers. Defaults to `true`
+* `whisper.maxTokensPerPart`: maximum number of tokens to decode for each audio part. Defaults to `250`
+* `whisper.suppressRepetition`: attempt to suppress decoding of repeating token patterns. Defaults to `true`
+* `whisper.decodeTimestampTokens`: enable/disable decoding of timestamp tokens. Setting to `false` can reduce the occurrence of hallucinations and token repetition loops, possibly due to the overall reduction in the number of tokens decoded. This has no impact on the accuracy of timestamps, since they are derived independently using cross-attention weights. However, there are cases where this can cause the model to end a part prematurely, especially in singing and less speech-like voice segments, or when there are multiple speakers. Defaults to `true`
+* `whisper.encoderProvider`: identifier for the ONNX execution provider to use with the encoder model. Can be `cpu` or `dml` ([DirectML](https://microsoft.github.io/DirectML/)-based GPU acceleration - Windows only). In general, GPU-based encoding should be significantly faster. Defaults to `cpu`, or `dml` if available
+* `whisper.decoderProvider`: identifier for the ONNX execution provider to use with the decoder model. Can be `cpu` or `dml` (Windows only). Using GPU acceleration for the decoder may be faster than CPU, especially for larger models, but that depends on your particular combination of CPU and GPU. Defaults to `cpu`
 * `whisper.seed`: provide a custom random seed for token selection when temperature is greater than 0. Uses a constant seed by default to ensure reproducibility
 
 **Whisper.cpp**:
@@ -157,7 +160,7 @@ Applies to CLI operation: `transcribe`, API method: `recognize`
 * `whisperCpp.build`: type of `whisper.cpp` build to use. Can be set `cpu`, `cublas-11.8.0`, `cublas-12.4.0`. By default, builds are auto-selected and downloaded for Windows x64 (`cpu`, `cublas-11.8.0`, `cublas-12.4.0`) and Linux x64 (`cpu`). Using other builds requires providing a custom `executablePath`
 * `whisperCpp.threadCount`: number of threads to use, defaults to `4`
 * `whisperCpp.splitCount`: number of splits of the audio data to process in parallel (called `--processors` in the `whisper.cpp` CLI). A value greater than `1` can increase memory use significantly, reduce timing accuracy, and slow down execution in some cases. Defaults to `1` (highly recommended)
-* `whisperCpp.enableGPU`: enable GPU processing. Defaults to `true` on CUDA-enabled builds, otherwise `false`
+* `whisperCpp.enableGPU`: enable GPU processing. Setting to `true` will try to use a CUDA build, if available for your system. Defaults to `true` when a CUDA-enabled build is selected via `whisperCpp.build`, otherwise `false`
 * `whisperCpp.topCandidateCount`: the number of top candidate tokens to consider. Defaults to `5`
 * `whisperCpp.beamCount`: the number of decoding paths to use during beam search. Defaults to `5`
 * `whisperCpp.repetitionThreshold`: minimal repetition / compressibility score to cause a decoded segment to be discarded. Defaults to `2.4`
@@ -170,6 +173,7 @@ Applies to CLI operation: `transcribe`, API method: `recognize`
 
 **Silero**:
 * `silero.modelPath`: path to a Silero model. Note that latest `en`, `de`, `fr` and `uk` models are automatically installed when needed based on the selected language. This should only be used to manually specify a different model, otherwise specify `language` instead
+* `silero.provider`: ONNX execution provider to use. Can be `cpu` or `dml` (Windows only). Defaults to `cpu`, or `dml` if available
 
 **Google Cloud**:
 * `googleCloud.apiKey`: Google Cloud API key (required)
@@ -192,7 +196,7 @@ Applies to CLI operation: `transcribe`, API method: `recognize`
 * `openAICloud.model`: model to use. Can only be `whisper-1`
 * `openAICloud.organization`: organization identifier. Optional
 * `openAICloud.baseURL`: override the default base URL used by the API. Optional
-* `openAICloud.temperature`: temperature. Choosing `0` uses a dynamic temperature approach. Defaults to `0.0`
+* `openAICloud.temperature`: temperature. Choosing `0` uses a dynamic temperature approach. Defaults to `0`
 * `openAICloud.prompt`: initial prompt for the model. Optional
 * `openAICloud.timeout`: request timeout. Optional
 * `openAICloud.maxRetries`: maximum retries on failure. Defaults to 10
@@ -220,12 +224,18 @@ Applies to CLI operation: `align`, API method: `align`
 * `dtw.granularity`: adjusts the MFCC frame width and hop size based on the profile selected. Can be set to either `auto` (auto-selected based on audio duration and task), `xx-low` (400ms width, 160ms hop), `x-low` (200ms width, 80ms hop), `low` (100ms width, 40ms hop), `medium` (50ms width, 20ms hop), `high` (25ms width, 10ms hop), `x-high` (20ms width, 5ms hop). For multi-pass processing, multiple granularities can be provided, like `dtw.granularity=['low','high']`. Defaults to `auto`.
 * `dtw.windowDuration`: maximum duration (in seconds) of the Sakoe-Chiba window when performing DTW alignment. Higher values consume quadratically larger amounts of memory. The estimated memory requirement is shown in the log before alignment starts. Recommended to be set to at least 10% - 20% of total audio duration. For multi-pass processing, multiple durations can be provided, like `dtw.windowDuration=[240,20]`. Auto-selected by default
 
-**DTW-RA only**:
+**DTW-RA**:
 * `recognition`: prefix to provide recognition options when using `dtw-ra` method, for example: setting `recognition.engine = whisper` and `recognition.whisper.model = base.en`
 * `dtw.phoneAlignmentMethod`: algorithm to use when aligning phones: can either be set to `dtw` or `interpolation`. Defaults to `dtw`
 
-**Whisper alignment only**:
-* `whisper`: prefix to provide Whisper options when the `whisper` alignment engine is used (does not apply to `dtw-ra` when `whisper` engine is used, for that use `recognition.whisper` prefix instead).
+**Whisper**:
+
+Applies to the `whisper` engine only. To provide Whisper options for `dtw-ra`, use `recognition.whisper` instead.
+
+* `whisper.model`: Whisper model to use. Defaults to `tiny` or `tiny.en`
+* `whisper.endTokenThreshold`: minimal probability to accept an end-of-text token for a recognized part. The probability is measured via the softmax between the end-of-text token's logit and the second highest logit. You can try to adjust this threshold in cases the model is ending a part with too few, or many tokens decoded. Defaults to `0.9`. On the last audio part, it is always effectively set to `Infinity`, to ensure the remaining transcript tokens are decoded in full
+* `whisper.encoderProvider`: encoder ONNX provider. See details in recognition section above
+* `whisper.decoderProvider`: decoder ONNX provider. See details in recognition section above
 
 
 ## Speech-to-text translation
@@ -254,6 +264,25 @@ Applies to CLI operation: `translate-speech`, API method: `translateSpeech`
 
 * `openAICloud`: prefix to provide options for OpenAI cloud. Same options as detailed in the recognition section above
 
+## Speech-to-translated-transcript alignment
+
+Applies to CLI operation: `align-translation`, API method: `alignTranslation`
+
+**General**:
+* `engine`: alignment algorithm to use, can only be `whisper`. Defaults to `whisper`
+* `language`: language code for the source audio ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), like `en`, `fr`, `zh`, etc. Auto-detected from audio if not set
+* `crop`: crop to active parts using voice activity detection before starting. Defaults to `true`
+* `isolate`: apply source separation to isolate voice before starting alignment. Defaults to `false`
+* `subtitles`: prefix to provide options for subtitles. Options detailed in section for subtitles
+* `vad`: prefix to provide options for voice activity detection when `crop` is set to `true`. Options detailed in section for voice activity detection
+* `sourceSeparation`: prefix to provide options for source separation when `isolate` is set to `true`. Options detailed in section for source separation
+
+**Whisper**:
+* `whisper.model`: Whisper model to use. Only multilingual models can be used. Defaults to `tiny`
+* `whisper.endTokenThreshold`: see details in the alignment section above
+* `whisper.encoderProvider`: encoder ONNX execution provider. See details in recognition section above
+* `whisper.decoderProvider`: decoder ONNX execution provider. See details in recognition section above
+
 ## Language detection
 
 ### Speech language detection
@@ -270,6 +299,11 @@ Applies to CLI operation: `detect-speech-langauge`, API method: `detectSpeechLan
 **Whisper**:
 * `whisper.model`: Whisper model to use. See model list in the recognition section
 * `whisper.temperature`: impacts the distribution of candidate languages when applying the softmax function to compute language probabilities over the model output. Higher temperature causes the distribution to be more uniform, while lower temperature causes it to be more strongly weighted towards the best scoring candidates. Defaults to `1.0`
+* `whisper.encoderProvider`: encoder ONNX execution provider. See details in recognition section above
+* `whisper.decoderProvider`: decoder ONNX execution provider. See details in recognition section above
+
+**Silero**:
+* `silero.provider`: ONNX execution provider to use. Can be `cpu` or `dml` (Windows only). Using GPU may be faster, but the initialization overhead is larger. **Note**: `dml` provider seems to be unstable at the moment for this model. Defaults to `cpu`
 
 ### Text language detection
 
@@ -294,6 +328,7 @@ Applies to CLI operation: `detect-voice-activity`, API method: `detectVoiceActiv
 
 **Silero**:
 * `silero.frameDuration`: Silero frame duration (ms). Can be `30`, `60` or `90`. Defaults to `90`
+* `silero.provider`: ONNX provider to use. Can be `cpu` or `dml` (Windows only). Using GPU is likely to be slower than CPU due to inference being independently executed on each audio frame. Defaults to `cpu` (recommended)
 
 ## Speech denoising
 
@@ -319,6 +354,7 @@ Applies to CLI operation: `isolate`, API method: `isolate`
 **MDX-NET**:
 
 * `mdxNet.model`: model to use. Currently available models are `UVR_MDXNET_1_9703`, `UVR_MDXNET_2_9682`, `UVR_MDXNET_3_9662`, `UVR_MDXNET_KARA`. Defaults to `UVR_MDXNET_1_9703`
+* `mdxNet.provider`: ONNX execution provider to use. Can be `cpu` or `dml` ([DirectML](https://microsoft.github.io/DirectML/), Windows only). **Note**: `dml` provider seems to be unstable with MDX-NET models at the moment. Defaults to `cpu`
 
 # Common options
 

package/docs/Releases.md

@@ -1,5 +1,9 @@
 # Release notes
 
+## Releases after `1.0.0`
+
+For releases after `1.0.0`, see the [GitHub releases page](https://github.com/echogarden-project/echogarden/releases).
+
 ## `1.0.0` (April 12, 2024)
 
 **New features**:

package/docs/Server.md

@@ -1,6 +1,8 @@
-# Starting and interfacing with the WebSocket server
+# WebSocket server API reference
 
-This is a guide for the WebSocket server protocol. The protocol is still in an early development stage and may change on future releases.
+This is a guide to the WebSocket server protocol.
+
+**Note**: The protocol is still in early development and may change in future releases. Many features are currently missing, and the server hasn't been thoroughly tested.
 
 ## Starting the server
 
@@ -36,10 +38,10 @@ ws.on("open", async () => {
 })
 ```
 
-In the future, this module may be separated to an independent lightweight package.
-
-**TODO**: Document using the client class with a background worker.
-**TODO**: Add support for cancellation signals in the client class.
+**TODO**:
+* Separate the client to an independent, lightweight, Node.js package, with browser compatibility
+* Add support for cancellation signals
+* Document how to use with a background worker
 
 ## Protocol details
 

package/docs/Tasklist.md

@@ -4,13 +4,13 @@
 
 ### Alignment
 
-* In DTW-RA, recognition transcript including something like "Question 2.What does Juan", where "2.What" has a point in the middle, is breaking playback of the timeline.
-* DTW-RA will not work correctly with Polish language texts, due to issues with the eSpeak engine pronouncing `|` characters, which are intended to be used as separators and ignored by all other eSpeak languages.
+* In DTW-RA, recognition transcript including something like "Question 2.What does Juan", where "2.What" has a point in the middle, is breaking playback of the timeline
+* DTW-RA will not work correctly with Polish language texts, due to issues with the eSpeak engine pronouncing `|` characters, which are intended to be used as separators and ignored by all other eSpeak languages
 
 ### Synthesis
 
 ### Phoneme processing
-* IPA -> Kirshenbaum translation is still not completely similar to what is output by eSpeak. Also, in rare situations, it outputs characters that are not accepted by eSpeak and eSpeak errors. Investigate when that happens and how to improve on this.
+* IPA -> Kirshenbaum translation is still not completely similar to what is output by eSpeak. Also, in rare situations, it outputs characters that are not accepted by eSpeak and eSpeak errors. Investigate when that happens and how to improve on this
 
 ### Browser extension
 * Investigate why WebSpeech events sometimes completely stop working in the middle of an utterance for no apparent reason. Sometimes this is permanent, until the extension is restarted. Is this a browser issue?
@@ -34,17 +34,9 @@
 
 ## Features and enhancements
 
-### Server
-* Option to allow or disallow local file paths as arguments to API methods (as a security safeguard)
-
-### Worker
-* Add cancellation checks in more operations
-* Support more operations
-
 ### CLI
 * Show names of files written do disk. This is useful for cases where a file is auto-renamed to prevent overwriting existing data
 * Restrict input media file extensions to ensure that invalid files are not passed to FFmpeg
-* Mode to print IPA words when speaking
 * Consider what to do with non-supported templates like `[hello]`
 * Show a message when a new version is available
 * Figure out which terminal outputs should go to stdout, or if that's a good idea at all
@@ -55,7 +47,8 @@
 * Suggest possible correction on the error of not using `=`, e.g. `speed 0.9` instead of `speed=0.9`
 * Multiple configuration files in `--config=..` taking precedence by order
 * Generate JSON configuration file schema
-* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help to give better error messages when the given file type is wrong.
+* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help to give better error messages when the given file type is wrong
+* Mode to print IPA words when speaking
 
 ### CLI / playback
 * Option to set audio output device for playback
@@ -64,7 +57,7 @@
 * Add phone playback support
 
 ### CLI / `speak`
-* Add support for sentence templates, like `echogarden speak-file text.txt /parts/[sentence].wav`.
+* Add support for sentence templates, like `echogarden speak-file text.txt /parts/[sentence].wav`
 
 ### CLI / `speak-wikipedia`
 * Correctly detect language when a Wikipedia URL is passed instead of an article name
@@ -84,7 +77,7 @@
 * `play-with-timeline`: Preview timeline in terminal
 * `subtitles-to-text`, `subtitles-to-timeline`, `srt-to-vtt`, `vtt-to-srt`
 * `text-to-ipa`, `arpabet-to-ipa`, `ipa-to-arpabet`
-* `phonemize-text`
+* `phonemize`
 * `normalize-text`
 * `transcribe-youtube`: Transcribe the audio in a YouTube video (requires fetching the audio somehow - which can't be done using the normal YouTube API)
 * `speak-youtube-subtitles`: To speak the subtitles of a YouTube video
@@ -98,7 +91,7 @@
 * Accept voice list caching options in `SynthesisOptions`
 
 ### Package manager
-* Better error message when package is not found remotely. Currently, it just gives a `404 not found` without any other information.
+* Better error message when package is not found remotely. Currently, it just gives a `404 not found` without any other information
 * Retry on network failure
 
 ### Speech language detection
@@ -111,7 +104,7 @@
 * See if it's possible to reliably use eSpeak as a segmentation engine
 
 ### Subtitles
-* Split long words if needed. This is especially important for Chinese
+* Split long words if needed. This is especially important for Chinese and Japanese
 * If a subtitle is too short and at the end of the audio, try to extend it back if possible (for example, if the previous subtitle is already extended, take back from it)
 * Decide how many punctuation characters to allow before breaking to a new line (currently it's infinite)
 * Add more clause separators, for even more special cases
@@ -119,26 +112,26 @@
 * Parse VTT's language
 
 ### Synthesis
-* Option to disable alignment (only for some engines). Alternative: use a low granularity setting that is very fast to compute
+* Option to disable alignment (only for some engines). Alternative: use a low granularity DTW setting that is very fast to compute
 * Find places to add commas (",") to improve speech fluency. VITS voices don't normally add speech breaks if there is no punctuation
-* An isolated dash " - " can be converted to a " , " to ensure there's a break in the speech.
+* An isolated dash " - " can be converted to a " , " to ensure there's a break in the speech
 * Ensure abbreviations like "Ph.d" or names like are segmented and read correctly (does `cldr` treat it as a word? Maybe eSpeak doesn't recognize it as a word). "C#" and ".NET" as well
 * Find way to manually reset voice list cache
 * When synthesized text isn't pre-split to sentences, apply sentence splits by using the existing method to convert the output of word timelines to sentence/segment timelines
 * Some `sapi` voices and `msspeech` languages output phones that are converted to Microsoft alphabet, not IPA symbols. Try to see if these can be translated to IPA
 * Decide whether asterisk `*` should be spoken when using `speak-url` or `speak-wikipedia`
-* Add partial SSML support for all engines. In particular, allow changing language or voice using the `<voice>` and `<lang>` tags, `<say-as>` and `<phoneme>` where possible.
-* Try to remove reliance on `()` after `.` character hack in `EspeakTTS.synthesizeFragments`.
-* eSpeak IPA output puts stress marks on vowels, not syllables - which is the standard for IPA. Consider how to make a conversion to and from these two approaches (possibly detect it automatically).
+* Add partial SSML support for all engines. In particular, allow changing language or voice using the `<voice>` and `<lang>` tags, `<say-as>` and `<phoneme>` where possible
+* Try to remove reliance on `()` after `.` character hack in `EspeakTTS.synthesizeFragments`
+* eSpeak IPA output puts stress marks on vowels, not syllables - which is the standard for IPA. Consider how to make a conversion to and from these two approaches (possibly detect it automatically)
 * Decide if `msspeech` engine should be selected if available. This would require attempting to load a matching voice, and falling back if it is not installed
 * Speaker-specific voice option
 * Use VAD on the synthesized audio file to get more accurate sentence or word segmentation
-* When `splitToSentences` is set to `false`, the timeline doesn't include proper sentences. Find a way to pass larger sections to the TTS, but still have proper sentences in the timeline.
+* When `splitToSentences` is set to `false`, the timeline doesn't include proper sentences. Find a way to pass larger sections to the TTS, but still have proper sentences in the timeline
 
 ### Synthesis / preprocessing
 * Extend the heteronyms JSON document with additional words like "conducts", "survey", "protest", "transport", "abuse", "combat", "combats", "affect", "contest", "detail", "marked", "contrast", "construct", "constructs", "console", "recall", "permit", "permits", "prospect", "prospects", "proceed", "proceeds", "invite", "reject", "deserts", "transcript", "transcripts", "compact", "impact", "impacts"
 * Full date normalization (e.g. `21 August 2023`, `21 Aug 2023`, `August 21, 2023`)
-* Add support for capitalized-only rules, and possibly also all uppercase / all lowercase rules.
+* Add support for capitalized-only rules, and possibly also all uppercase / all lowercase rules
 * Add support for multiple words in `precededBy` and `succeededBy`
 * Support substituting to graphemes in lexicons, not only phonemes
 * Cache lexicons to avoid parsing the JSON each time it is loaded (this may not be needed for if the file is relatively small)
@@ -160,30 +153,33 @@
 ### Recognition
 * Recognized word entries that span VAD segment boundaries can be split
 * Show alternatives when playing in the CLI. Clear current line and rewrite already printed text for alternatives during the speech recognition process
-* Option to split recognized audio to segments or sentences, as is done with synthesized audio
-* Try to exclude the timing for trailing punctuation tokens in words that contain them. This can help narrow down the end timestamp to cover the word more tightly
 
 ### Recognition / Whisper
-* May get stuck in a token repeat loop when silence or non-speech segment encountered in audio. Decide what to do
+* Whisper's Chinese and Japanese output can be split to words in a more accurate way. Consider using a dedicated segmentation library to perform the segmentation in character sequences that have no spaces within them
 * Automatically disable using previous section recognized transcript as prompt for the next section when lots of repetition occurred in previous section
 * Cache last model (if enough memory available)
-* Bring back option to use eSpeak DTW based alignment on segments, as an alternative approach
 * The segment output can be used to split to segments, otherwise it is possible to try to guess using pause lengths or voice activity detection
-* Use compression ratios on the decoded tokens of individual segments and discard if too much repetition detected
-* Way to specify model size only, such that the English-only/multilingual variant would be automatically selected for sizes other than `tiny`?
-* Timestamps extracted from cross-attention are still not as accurate as what the official Python implementation gets. Try to see if you can make them better
-* Whisper's Chinese output can be split to words in a more accurate way. Consider using a dedicated segmentation library to perform the segmentation in character sequences that have no spaces within them
+* Bring back option to use eSpeak DTW based alignment on segments, as an alternative approach
 
 ### Alignment
 
 * Aligned words entries that span VAD boundaries may be split
 
 ### Alignment / DTW-RA
-* Remove emojis and other special characters, that are not likely to be pronounced in the speech, from the transcript timeline before it is synthesized. For example Whisper may produce 'note' emojis when it detects singing or music. Pronouncing them reduces the accuracy of the alignment
-* Optional mode to pass Whisper a special vocabulary of tokens that can appear in the transcript. All other tokens would be suppressed
 
 ### Alignment / Whisper
-* New mode to decode the transcript tokens in order using a more standard decoding approach (updating the KV cache at each step). This would allow audio inputs longer than 30 seconds. See if this produces better results
+
+### Source separation / MDX-NET
+* Since MDX-NET requires FFT with large window sizes, the FFT computation overhead currently acts as a bottleneck, especially when GPU is used for inference. Currently it uses a WASM port of KissFFT, running on a single thread, which is still relatively fast. To get higher performance, try to (optionally) use native, SIMD optimized FFT like FFTW3 via a NAPI addon, with multi-threading enabled
+* Option to customize overlap
+* Add more models
+
+### Server
+* Option to allow or disallow local file paths as arguments to API methods (as a security safeguard)
+
+### Worker
+* Add cancellation checks in more operations
+* Support more operations
 
 ### Browser extension
 * Options UI
@@ -212,7 +208,7 @@
 * See if the installation of `winax` can be automated and only initiate if it is in a Windows environment
 * Ensure that all modules have no internal state other than caching
 * Start thinking about some modules being available in the browser. Which node core APIs the use? Which of them can be polyfilled, and which cannot?
-* Change all the Emscripten WASM modules to use the `EXPORT_ES6=1` flag and rebuild them. Support for node.js modules was only added in September 2022 (https://github.com/emscripten-core/emscripten/pull/17915).
+* Change all the Emscripten WASM modules to use the `EXPORT_ES6=1` flag and rebuild them. Support for node.js modules was added in September 2022 (https://github.com/emscripten-core/emscripten/pull/17915)
 * Remove built-in voices from `flite` to reduce size?
 * Slim down `kuromoji` package to reduce base installation size
 
@@ -224,7 +220,6 @@
 * Test everything's fine on macOS
 * Test that cloud services all still work correctly, especially with SSML inputs
 
-
 ## Future features and enhancements
 
 ### CLI
@@ -259,18 +254,10 @@
 * Investigate exporting Whisper models to 16-bit quantized ONNX or a mix of 16-bit and 32-bit
 
 ### Alignment
-* Implement alignment with speech-to-text translation assistance, which would enable multilingual subtitle replacement for translated subtitles
 * Method to align audio file to audio file
-* Make `dtw` mode work with more speech synthesizers to produce its reference
-* Predict timing for individual letters (graphemes) based on phoneme timestamps
+* Allow `dtw` mode work with more speech synthesizers to produce its reference
+* Predict timing for individual letters (graphemes) based on phoneme timestamps (especially useful for Chinese and Japanese)
 
-### Voice activity detection
-
-* Whisper-based VAD. Use Whisper's 'no speech' token to determine if the audio contains speech
-
-### Source separation
-* Option to customize overlap
-* Add more MDX-NET models
 
 ## Possible new engines or platforms
 
@@ -280,13 +267,13 @@
 * Coqui STT server connection
 * [MarbleNet VAD](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Online_Offline_Microphone_VAD_Demo.ipynb), included of the NVIDIA NeMo framework, can be exported to ONNX
 * Silero text enhancement engine can be ported to ONNX
-* See what can be done on supporting WinRT speech: in particular `windows.media.speechsynthesis` and `windows.media.speechrecognition` support, possibly using NodeRT or some other method.
-* Figure out how to support `julius` speech recognition via WASM.
+* See what can be done for supporting WinRT speech: in particular `windows.media.speechsynthesis` and `windows.media.speechrecognition` support, possibly using NodeRT or some other method
+* Figure out how to support `julius` speech recognition via WASM
 * Any way to support RHVoice?
 
 ## Maybe?
 
-* Using a machine translation model to provide speech translation to languages other than English?
+* Using a machine translation model to provide speech translation to languages other than English? How would the timing be determined?
 * Is it possible to get sentence boundaries without punctuation using NLP techniques like part of speech tagging?
 
 ## May or may not be good ideas
@@ -298,10 +285,10 @@
 
 * Support alignment of EPUB 3 eBooks with corresponding audiobook
 * Voice cloning
-* Speech to speech voice conversion
-* Speech-to-speech translation (need to find a good model)
+* Speech-to-speech voice conversion
+* Speech-to-speech translation
 * HTML generator, that includes text and audio, with playback and word highlighting
 * Video generator
 * Desktop app that uses the tool to transcribe the PC audio output
-* Special method to use time stretching to project between different utterances of the same text
+* Special method to use time stretching to project between different aligned utterances of the same text
 * Is it possible to combine the Silero speech recognizer and a language model and try to perform Viterbi decoding to find alignments?

package/docs/Technical.md

@@ -26,7 +26,7 @@ The base installed (uncompressed) size, including dependencies, is around 270MB.
 
 Currently, the largest contributors to the size are:
 
-* `onnxruntime-node` (NAPI): 92MB
+* `onnxruntime-node` (NAPI): 133MB
 * `kuromoji` (JavaScript) 40MB
 * `flite-wasi` (WASI): 20MB
 * `espeak-ng-emscripten` (WASM): 18MB