echogarden 0.0.1 → 0.1.0

package/dist/voice-activity-detection/WebRtcVAD.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"WebRtcVAD.js","sourceRoot":"","sources":["../../src/voice-activity-detection/WebRtcVAD.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,qBAAqB,MAAM,mCAAmC,CAAA;AAE1E,OAAO,EAAE,iBAAiB,EAAE,MAAM,mCAAmC,CAAA;AAErE,MAAM,CAAC,KAAK,UAAU,mBAAmB,CAAC,QAAkB,EAAE,gBAA8B,EAAE,EAAE,OAAsB,CAAC;IACtH,IAAI,QAAQ,CAAC,UAAU,IAAI,KAAK,EAAE;QACjC,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAA;KAClD;IAED,MAAM,oBAAoB,GAAc,MAAM,IAAI,CAAC,qBAAqB,CAAC,iBAAiB,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAC,CAAC,CAAC,EAAE,QAAQ,CAAC,UAAU,EAAE,aAAa,EAAE,IAAI,CAAC,CAAA;IAEhK,OAAO,oBAAoB,CAAA;AAC5B,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,IAAI,CAAC,OAAmB,EAAE,UAAkB,EAAE,aAA2B,EAAE,IAAmB;IACnH,MAAM,CAAC,GAAG,MAAM,eAAe,EAAE,CAAA;IAEjC,MAAM,UAAU,GAAG,IAAI,iBAAiB,CAAC,CAAC,CAAC,CAAA;IAE3C,MAAM,QAAQ,GAAG,CAAC,CAAC,SAAS,CAAA;IAC5B,MAAM,SAAS,GAAG,CAAC,CAAC,UAAU,CAAA;IAC9B,MAAM,UAAU,GAAG,CAAC,CAAC,WAAW,CAAA;IAChC,MAAM,aAAa,GAAG,CAAC,CAAC,cAAc,CAAA;IACtC,MAAM,oBAAoB,GAAG,CAAC,CAAC,qBAAqB,CAAA;IACpD,MAAM,YAAY,GAAG,CAAC,CAAC,aAAa,CAAA;IAEpC,MAAM,WAAW,GAAG,QAAQ,EAAE,CAAA;IAE9B,MAAM,eAAe,GAAG,oBAAoB,CAAC,WAAW,EAAE,UAAU,CAAC,CAAA;IAErE,IAAI,eAAe,IAAI,CAAC,EAAE;QACzB,MAAM,IAAI,KAAK,CAAC,yCAAyC,UAAU,EAAE,CAAC,CAAA;KACtE;IAED,MAAM,SAAS,GAAG,aAAa,CAAC,WAAW,EAAE,IAAI,CAAC,CAAA;IAElD,IAAI,SAAS,IAAI,CAAC,EAAE;QACnB,MAAM,IAAI,KAAK,CAAC,iCAAiC,IAAI,EAAE,CAAC,CAAA;KACxD;IAED,MAAM,gBAAgB,GAAG,IAAI,CAAC,KAAK,CAAC,UAAU,GAAG,CAAC,aAAa,GAAG,IAAI,CAAC,CAAC,CAAA;IACxE,MAAM,eAAe,GAAG,UAAU,CAAC,eAAe,CAAC,gBAAgB,CAAC,CAAA;IAEpE,MAAM,MAAM,GAAG,EAAE,CAAA;IAEjB,KAAK,IAAI,YAAY,GAAG,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,MAAM,EAAE,YAAY,IAAI,gBAAgB,EAAE;QAC3F,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,CAAC,YAAY,EAAE,YAAY,GAAG,gBAAgB,CAAC,CAAA;QAE7E,eAAe,CAAC,KAAK,EAAE,CAAA;QACvB,eAAe,CAAC,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,CAAA;QAE/B,MAAM,UAAU,GAAG,YAAY,CAAC,WAAW,EAAE,eAAe,CAAC,OAAO,EAAE,gBAAgB,CAAC,CAAA;QAEvF,IAAI,UAAU,IAAI,CAAC,CAAC,EAAE;YACrB,MAAM,IAAI,KAAK,CAAC,qBAAqB,CAAC,CAAA;SACtC;QAED,MAAM,CAAC,IAAI,CAAC,UAAU,CAAC,CAAA;KACvB;IAED,SAAS,CAAC,WAAW,CAAC,CAAA;IACtB,UAAU,CAAC,OAAO,EAAE,CAAA;IAEpB,OAAO,MAAM,CAAA;AACd,CAAC;AAED,IAAI,YAAiB,CAAA;AACrB,KAAK,UAAU,eAAe;IAC7B,IAAI,CAAC,YAAY,EAAE;QAClB,MAAM,EAAE,OAAO,EAAE,eAAe,EAAE,GAAG,MAAM,MAAM,CAAC,uBAAuB,CAAC,CAAA;QAE1E,YAAY,GAAG,MAAM,eAAe,EAAE,CAAA;KACtC;IAED,OAAO,YAAY,CAAA;AACpB,CAAC"}

package/docs/CLI.md

@@ -0,0 +1,234 @@
+
+# Using the command line interface
+
+All CLI command lines have the general structure:
+
+```bash
+echogarden [command] [one or more inputs..] [one or more outputs...] [options...]
+```
+
+Here's a quick tour of the main operations available via the CLI.
+
+Each command accepts many options, in the form `--[optionName]=[value]` (The `=` is required). A detailed reference of all the options can be found [here](Options.md).
+
+## Text to speech
+
+**Task**: given a text file, synthesize spoken audio for it.
+
+
+This would synthesize "Hello World" and play the result in the terminal:
+```bash
+echogarden speak "Hello world!"
+```
+
+You can add a file output instead, which would cause the audio to be stored in `result.mp3`:
+```bash
+echogarden speak "Hello world!" result.mp3
+```
+
+This would synthesize a text file `text.txt`, and have the resulting audio stored in `result.mp3`. The `--engine=pico` would set the synthesis engine to `pico` (SVOX Pico):
+```bash
+echogarden speak-file text.txt result.mp3 --engine=pico
+```
+
+This would synthesize a text file, and have the result stored in both `result.mp3` and `result.wav`, as well as a subtitle file in `result.srt`:
+```bash
+echogarden speak-file text.txt result.mp3 result.wav result.srt --engine=vits --speed=1.1
+```
+
+Synthesize a web page (will try to extract its main article parts and omit the rest):
+```bash
+echogarden speak-url http://example.com/hola
+```
+
+Synthesize a Wikipedia article in any of its language editions:
+```bash
+echogarden speak-wikipedia "Psychologie" --language=fr
+```
+
+## Speech to text
+
+**Task**: given an audio recording containing speech, find a text transcription that best matches it.
+
+This would transcribe the audio file `speech.mp3`, and then play the audio, along with the recognized text, in the terminal:
+```bash
+echogarden transcribe speech.mp3
+```
+
+This would transcribe the audio file `speech.mp3` and store the resulting transcription in `result.txt`, subtitles in `result.srt`, and a full timeline tree in `result.json`:
+```bash
+echogarden transcribe speech.mp3 result.txt result.srt result.json
+```
+
+## Speech to transcript alignment
+
+**Task**: given an audio file and its transcript, try to approximate the timing of the start and end of each spoken word (and its subparts).
+
+This would align the audio file `speech.mp3` with the transcript provided in `transcript.txt`, and would play the synchronized result in the terminal:
+```bash
+echogarden align speech.mp3 transcript.txt
+```
+
+This would align the audio file `speech.mp3` with the transcript provided in `transcript.txt`, and store the resulting timeline tree in `result.json` and subtitles in `result.srt`:
+```bash
+echogarden speak-file text.txt transcript.txt result.json result.srt
+```
+
+## Using output file templates to split the outputs to multiple parts
+
+Echogarden can split the audio to multiple parts based on the segment boundaries detected. For example:
+
+```bash
+echogarden speak text.txt parts/[segment].opus
+```
+
+The `[segment]` pattern would cause multiple files to be created, one for each text segment (segments would be determined according to paragraph or line breaks, in this case). The pattern would be replaced by the index and initial text of the segment. For example by `parts/01 Hello world how are you doing ... .opus`.
+
+Templates can also be used in multiple outputs. For instance, the following would align `speech.mp3` with `transcript.txt` and then split the audio by the segments found in the transcript, and store separate audio and timeline files for each part.
+
+```bash
+echogarden align speech.mp3 transcript.txt parts/[segment].m4a parts/[segment].json
+```
+
+## Loading configuration from a file
+
+Since there are many possible configuration options, it may be more convenient to store them in a configuration file.
+
+When a file named `echogarden.config` is found at the current directory, it will be loaded automatically and its content would be used as default options. You can also specify a particular configuration file path with the option `--config=path/to/your-config-file.config`.
+
+The configuration file format is simple and has a dedicated section for each command (all `speak-` commands are grouped together under `speak`):
+
+```conf
+[speak]
+
+# Engine for synthesis:
+engine = sapi
+
+# Voice for synthesis (case-insensitive, can be a search pattern):
+voice = zira
+
+[transcribe]
+
+# Engine for recognition:
+engine = whisper
+
+# Whisper model to use:
+whisper.model = tiny
+```
+
+### JSON configuration file
+
+You can also use a JSON configuration file format instead, if preferred.
+
+Name your file `echogarden.config.json`:
+```json
+{
+	"speak": {
+		"engine": "sapi",
+		"voice": "zira"
+	},
+
+	"transcribe": {
+		"engine": "whisper",
+		"whisper": {
+			"model": "tiny"
+		}
+	}
+}
+```
+
+Flattened property names are also accepted:
+```json
+{
+	"transcribe": {
+		"engine": "whisper",
+		"whisper.model": "tiny"
+	}
+}
+```
+
+## Other operations
+
+### Speech translation
+
+**Task**: given an audio file containing speech in one language, transcribe it in a second language. The translated transcript should be generated directly from the speech itself, without an intermediate translation step.
+
+This will detect the spoken language, apply speech translation to English, and play the original audio, synced with the translated transcript:
+```bash
+echogarden translate speech.mp3
+```
+
+To specify the source and target languages explicitly, use the `sourceLanguage` and `targetLanguage` options:
+```bash
+echogarden translate speech.mp3 transcript.txt --sourceLanguage=es --targetLanguage=en
+```
+
+**Note**: currently, only English is supported as target language. This is a limitation of the `whisper` Engine, which is the only one used for speech translation, at this time.
+
+### Language detection
+
+**Task**: given audio or textual input, try to identify which language it is spoken or written in.
+
+Try to identify the language of an audio file containing speech, and print the probabilities to the terminal:
+```bash
+echogarden detect-speech-language speech.mp3
+```
+
+Try to identify the language of a text file, and print the probabilities to the terminal:
+```bash
+echogarden detect-text-language story.txt
+```
+
+Try to identify the language of a text file, and store the probabilities in a JSON file:
+```bash
+echogarden detect-text-language story.txt detection-results.json
+```
+
+### Voice activity detection
+
+**Task**: given an audio file, try to classify which parts of the audio contain speech, and which don't.
+
+This would apply VAD and play the audio, synchronized with `speech` and `nonspeech` indicators, printed to the terminal.
+```bash
+echogarden detect-voice-activity speech.mp3
+```
+
+This would apply VAD and store the results in a timeline JSON file.
+```bash
+echogarden detect-voice-activity speech.mp3 timeline.json
+```
+
+### Speech denoising
+
+**Task**: try to reduce the amount of background noise in a spoken recording.
+
+This would apply denoising and play the denoised audio:
+```bash
+echogarden denoise speech.mp3
+```
+
+This would apply denoising, and save the denoised audio to a file:
+```bash
+echogarden denoise speech.mp3 denoised-speech.mp3
+```
+
+### Misc.
+
+#### `list-tts-voices`
+
+Shows a list of TTS voices for a given engine:
+
+```bash
+echogarden list-tts-voices google-cloud
+```
+
+Stores the voice list in a JSON file:
+```bash
+echogarden list-tts-voices google-cloud google-cloud-voices.json
+```
+
+#### `install`, `uninstall` and `list-packages`
+
+* `install`: install one or more expansion packages
+* `uninstall`: uninstall one or more expansion packages
+* `list-packages`: list installed packages

package/docs/Development.md

@@ -0,0 +1,3 @@
+# How to help
+
+TODO

package/docs/Engines.md

@@ -0,0 +1,80 @@
+
+# Supported engines
+
+## Text to speech
+
+**Offline**:
+* [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis model. Currently available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Supports English (US, UK), Spanish, Brazilian Portuguese, Italian, French, German, Dutch, Norwegian, Danish, Finnish, Polish, Ukrainian, Catalan, Kazakh, Nepali, Vietnamese and Chinese. You can listen to audio examples of all voices and languages on [Piper's samples page](https://rhasspy.github.io/piper-samples/).
+* [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a diphone-based synthesizer. Supports English (US, UK), Spanish, Italian, French, and German.
+* [Flite](https://github.com/festvox/flite) (`flite`): a diphone-based synthesizer. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi.
+* [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a formant-based synthesizer. Supports 100+ languages.
+* [SAM (Software Automatic Mouth)](https://github.com/discordier/sam) (`sam`): a classic "robot" speech synthesizer from 1982. English only.
+
+**Offline, Windows only**:
+
+_Note_: the SAPI-based engines require manually installing the [`winax` npm package](https://www.npmjs.com/package/winax) by running `npm install winax -g`.
+
+`winax` is a native module which requires the Node.js Windows build tools to successfully install. If you have issues installing this package, please ensure that you've checked the "install necessary tools" checkbox during the installation of node.js.
+
+* [SAPI](https://en.wikipedia.org/wiki/Microsoft_Speech_API) (`sapi`): Microsoft Speech API. Supports the system's language voices, as well as voices produced by third-party vendors, like Ivona, NeoSpeech, Acapela, Cepstral, CereProc, Nuance, AT&T, Loquendo, ScanSoft and others (note that only 64-bit voices are supported, which makes it incompatible with a significant portion of older voices).
+
+* [Microsoft Speech Platform](https://www.microsoft.com/en-us/download/details.aspx?id=27225) (`msspeech`): Microsoft Server Speech API. Requires [installing a runtime (2.6MB)](https://www.microsoft.com/en-us/download/details.aspx?id=27225). Supports 28 dialects, which can be individually downloaded via [freely available installers](https://www.microsoft.com/en-us/download/details.aspx?id=27224), or, for convenience, bundled as [a single 358MB zip file](https://drive.google.com/u/0/uc?id=1uQdFNxLzUxpaEwVVKhMawys8cIh3F21T&export=download). Has voices for English (US, UK, AU, CA), Spanish (ES, MX), Portuguese (BR, PT), German, French (FR, CA), Italian, Norwegian, Dutch, Russian, Swedish, Danish, Catalan, Finnish, Japanese, Korean and Chinese (ZH, HK, TW). All voices are female.
+
+**Remote or self-hosted servers**:
+* [Coqui TTS](https://github.com/coqui-ai/TTS) server (`coqui-server`)
+
+**Cloud-based**:
+* [Google Cloud](https://cloud.google.com/text-to-speech) (`google-cloud`)
+* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/text-to-speech/) (`microsoft-azure`)
+* [Amazon Polly](https://aws.amazon.com/polly/) (`amazon-polly`)
+* [Elevenlabs](https://beta.elevenlabs.io/) (`elevenlabs`)
+
+**Cloud-based (unofficial)**:
+
+These cloud-based engines connect to public-facing cloud APIs that are not officially publicized by their operators. They are included for experimental purposes only, and may be removed in the future:
+
+* Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice (usually female) for each language.
+* Microsoft Edge (`microsoft-edge`): subset of the Azure Cognitive Services cloud TTS API used by the Microsoft Edge browser as part of its support for the [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API) and its [Read Aloud](https://www.microsoft.com/en-us/edge/features/read-aloud?form=MT00D8) feature. Using this engine requires a special token, which should be passed via the `microsoftEdge.trustedClientToken` option.
+* Streamlabs Polly (`streamlabs-polly`): a public REST API by Streamlabs, primarily intended for generating speech for TTS donations. It includes a few English (US, UK, AU, IN) voices, which are similar to some of the non-neural (Ivona-based) voices offered by Amazon Polly.
+
+## Speech to text
+
+**Offline**:
+* [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high accuracy transformer-based model. Supports 99 languages. There are several models of different sizes, some are multilingual, and some are English only (`.en`): `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v1` and `large-v2`. _Note_: large models are not currently auto-downloaded as Echogarden packages, but may become available in the future.
+* [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for [25+ languages](https://alphacephei.com/vosk/models). _Note_: Vosk is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. You'll need to download a model yourself and specify its directory path via the `vosk.modelPath` option.
+* [Silero](https://github.com/snakers4/silero-models) (`silero`): models available for English, Spanish, German and Ukrainian. For [non-commercial use only](https://github.com/snakers4/silero-models/blob/master/LICENSE).
+
+**Cloud based**:
+* [Google Cloud](https://cloud.google.com/speech-to-text) (`google-cloud`)
+* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/speech-to-text/) (`microsoft-azure`)
+* [Amazon Transcribe](https://aws.amazon.com/transcribe/) (`amazon-transcribe`)
+
+## Speech to transcript alignment
+
+* Dynamic Time Warping (`dtw`): transcript is first synthesized using the `espeak` engine, then [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) is applied to find the best mapping between the synthesized audio and the original audio.
+* Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using `espeak`. Then, the best mapping is found between the two synthesized audio sequences, and the result is mapped back to the original audio using the timing information produced by the recognizer.
+* Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model (only supports audio inputs that are 30 seconds or less).
+
+## Speech translation
+
+* [Whisper](https://github.com/openai/whisper) (`whisper`): the Whisper model can transcribe speech in any one of its supported languages and output a transcript directly translated to English. Other languages are not supported as targets.
+
+## Language detection
+
+**Spoken language detection**:
+* [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the 99 languages it has been trained on.
+* [Silero Language Classifier](https://github.com/snakers4/silero-vad/wiki/Other-Models) (`silero`): a speech language classification model by Silero.
+
+**Written language detection**:
+* [TinyLD](https://www.npmjs.com/package/tinyld) (`tinyld`): a simple language detection library.
+* [FastText](https://github.com/facebookresearch/fastText) (`fasttext`): a library for word representations and sentence classification by Facebook research.
+
+## Voice activity detection
+
+* [WebRTC VAD](https://github.com/dpirch/libfvad) (`webrtc`): a voice activity detector. Originally from the Chromium source code.
+* [Silero VAD](https://github.com/snakers4/silero-vad) (`silero`): a voice activity detection model by Silero.
+* [RNNoise](https://github.com/xiph/rnnoise) (`rnnoise`): uses RNNoise's speech probabilities output for each audio frame as a VAD metric.
+
+## Speech denoising
+
+* [RNNoise](https://github.com/xiph/rnnoise) (`rnnoise`): a noise suppression library based on a recurrent neural network.

package/docs/Licenses.md

@@ -0,0 +1,37 @@
+# Echogarden components licensing
+
+## Engines and libraries:
+
+* `onnxruntime-node`: [MIT License](https://github.com/microsoft/onnxruntime/blob/main/LICENSE)
+* `espeak`: [GNU GPL v3](https://github.com/espeak-ng/espeak-ng/blob/master/COPYING)
+* `flite`: [BSD License](https://github.com/festvox/flite/blob/master/COPYING)
+* `pico`: [Apache License 2.0](https://github.com/gmn/nanotts/blob/master/LICENSE)
+* `sam`: [Abandonware / fair-use / unknown](https://github.com/discordier/sam#license)
+* `vosk` [Apache License 2.0](https://github.com/alphacep/vosk-api/blob/master/COPYING)
+* `tinyld` [MIT License](https://github.com/komodojp/tinyld/blob/develop/license)
+* `fasttext`: [MIT License](https://github.com/loretoparisi/fasttext.js/blob/master/LICENSE)
+* `fvad` [BSD-3-Clause](https://github.com/cpuimage/WebRTC_VAD/blob/master/LICENSE)
+* `kissfft`: [BSD-3-Clause](https://github.com/mborgerding/kissfft/blob/master/COPYING)
+* `rnnoise`: [BSD-3-Clause](https://github.com/xiph/rnnoise/blob/master/COPYING)
+* `sonic`: [Apache License 2.0](https://github.com/waywardgeek/sonic/blob/master/LICENSE)
+* `speex-resampler`: [BSD-3-Clause](https://github.com/xiph/speexdsp/blob/master/COPYING)
+* `rubberband`: [GNU GPL v2](https://github.com/breakfastquay/rubberband/blob/default/COPYING)
+* `compromise`: [MIT License](https://github.com/spencermountain/compromise/blob/master/LICENSE)
+* `cldr-segmentation`: [MIT License](https://github.com/camertron/cldr-segmentation.js/blob/master/LICENSE)
+* `jieba-wasm`: [MIT License](https://github.com/fengkx/jieba-wasm/blob/master/LICENSE)
+
+## Package repository
+
+There is a variety of voices, models and binaries served from the repository.
+
+All are freely distributable, with varying licenses:
+* Flite voices (`flite-`): [BSD License](https://github.com/festvox/flite/blob/master/COPYING)
+* SVOX Pico resources (`pico-`): [Apache License 2.0](https://github.com/gmn/nanotts/blob/master/LICENSE)
+* Silero VAD (`silero-vad`) and Silero language classifier  (`silero-lang-classifier-95`): [MIT License](https://github.com/snakers4/silero-vad/blob/master/LICENSE)
+* Silero speech recognition models (`silero-en-`, `silero-de-`, `silero-es-`, `silero-ua-`): [BY-NC-SA](https://github.com/snakers4/silero-models/blob/master/LICENSE)
+* VITS pre-trained models (`vits-`): licensed under various creative commons licenses: [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/), [CC-BY](https://creativecommons.org/licenses/by/4.0/) and [BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/), and few are public domain (you can view individual license for each model in the model cards on the [samples page](https://rhasspy.github.io/piper-samples/)). The Piper application itself is published under the [MIT License](https://github.com/rhasspy/piper/blob/master/LICENSE.md)
+* Whisper pre-trained models (`whisper-`): [MIT License](https://github.com/openai/whisper/blob/main/LICENSE)
+
+Tool binary distributions:
+* FFMpeg: [LGPL, GPL v2 and GPL v3 Licenses](https://github.com/FFmpeg/FFmpeg)
+* SoX: [GPL v2 License]()

package/docs/Options.md

@@ -0,0 +1,192 @@
+# Configuration options reference
+
+For a comprehensive list of all supported engines: see [this page](Engines.md).
+
+## Synthesis
+
+Applicable to CLI commands: `speak`, `speak-file`, `speak-url`, `speak-wikipedia`.
+
+General:
+* `engine`: identifier of the synthesis engine to use, such as `espeak` or `vits`.
+* `language`: language code, like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected if not set
+* `voice`: name of the voice to use. Optional
+* `voiceGender`: gender of the voice to use. Optional
+* `speed`: speech rate factor, relative to default. In the range `0.1`..`10.0`. Defaults to `1.0`
+* `pitch`: pitch factor, relative to default. In the range `0.1`..`10.0`. Defaults to `1.0`
+* `pitchVariation`: pitch variation factor. In the range `0.1`..`10.0`. Defaults to `1.0`
+* `ssml`: the input is SSML. Defaults to `false`
+* `sentenceEndPause`: pause duration (seconds) at end of sentence. Defaults to `0.75`
+* `segmentEndPause` pause duration (seconds) at end of segment. Defaults to `1.0`
+
+Post-processing:
+* `postProcessing.normalizeAudio` should normalize output audio. Defaults to `true`
+* `postProcessing.targetPeakDb` target peak (decibels) for normalization. Defaults to `-3`
+* `postProcessing.maxIncreaseDb` max gain increase (decibels) when performing normalization. Defaults to `30`
+* `postProcessing.speed`: target speed for time stretching. Defaults to `1.0`
+* `postProcessing.pitch`: target pitch for pitch shifting. Defaults to `1.0`
+* `postProcessing.timePitchShiftingMethod` method for time and pitch shifting. Can be `sonic` or `rubberband`. Defaults to `sonic`
+* `postProcessing.rubberband`: prefix for RubberBand options (TODO)
+
+VITS:
+* `vits.speakerId`: speaker ID, for VITS models that support multiple speakers
+
+eSpeak-ng:
+* `espeak.rate`: speech rate, in eSpeak units. Overrides `speed` when set
+* `espeak.pitch`: pitch, in eSpeak units. Overrides `pitch` when set
+* `espeak.pitchRange`: pitch range, in eSpeak units. Overrides `pitchVariation` when set
+
+SAM:
+* `sam.pitch`: pitch value, between `0`..`255`, optional. Overrides `pitch` when set
+* `sam.speed`: speed value, between `0`..`255`, optional. Overrides `speed` when set
+* `sam.mouth`: mouth value, between `0`..`255` (defaults to `128`)
+* `sam.throat`: throat value, between `0`..`255` (defaults to `128`)
+
+SAPI:
+* `sapi.rate`: SAPI speech rate, in its native units. An integer number between `-10` and `10`. Setting `speed` would apply time stretching instead. The two options can be used together
+
+Microsoft Speech Platform:
+* `msspeech.rate`: equivalent units and effect to the SAPI speech rate
+
+Coqui Server:
+* `coquiServer.serverUrl`: server URL
+* `coquiServer.speakerId`: speaker ID (if applicable)
+
+Google Cloud:
+* `googleCloud.apiKey`: API key (required)
+* `googleCloud.pitchDeltaSemitones`: pitch delta in semitones. Overrides `pitch` when set
+* `googleCloud.customVoice.model`: name of custom voice
+* `googleCloud.customVoice.reportedUsage`: reported usage of custom voice
+
+Azure Cognitive Services:
+* `microsoftAzure.subscriptionKey`: subscription key (required)
+* `microsoftAzure.serviceRegion`: service region (required)
+* `microsoftAzure.pitchDeltaHz`: pitch delta in Hz. Overrides `pitch` when set
+
+Amazon Polly:
+* `amazonPolly.region`: region (required)
+* `amazonPolly.accessKeyId`: access key ID (required)
+* `amazonPolly.secretAccessKey`: secret access key (required)
+* `amazonPolly.pollyEngine`: Amazon Polly engine kind, can be `standard` or `neural`. Defaults to `neural`
+* `amazonPolly.lexiconNames`: TODO
+
+Elevenlabs:
+* `elevenLabs.apiKey`: API key (required)
+* `elevenLabs.stability`: stability
+* `elevenLabs.similarityBoost`: similarity boost
+
+Google Translate:
+* `googleTranslate.tld`: top level domain to to connect to. Can change the dialect of the voices for a small number or voices. For example `us` gives American English for `en`, while `com` gives British English for `en`. Defaults to `us`
+
+Microsoft Edge:
+* `microsoftEdge.trustedClientToken`: trusted client token (required). A special token required to use the service
+* `microsoftEdge.pitchDeltaHz`: pitch delta in Hz. Overrides `pitch` when set
+
+## Recognition
+
+Applicable to CLI command: `transcribe`.
+
+General:
+* `engine`: identifier of the recognition engine to use, such as `whisper` or `vosk`
+* `language`: language code for the audio, like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected if not set
+
+Whisper:
+* `whisper.model`: selects which Whisper model to use. Can be `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large` (same as `large-v2`), `large-v1`, `large-v2`. Defaults to `tiny`
+
+Vosk:
+* `vosk.modelPath`: path to the Vosk model to be used
+
+Silero:
+* `silero.modelPath`: path to a Silero model. Note that latest `en`, `de`, `fr` and `uk` models are automatically installed when needed based on the selected language. This should only be used to manually specify a different model, otherwise specify `language` instead
+
+Google Cloud:
+* `googleCloud.apiKey`: Google Cloud API key (required)
+* `googleCloud.alternativeLanguageCodes`: TODO
+* `googleCloud.profanityFilter`: censor profanity. Defaults to `false`
+* `googleCloud.autoPunctuation`: add punctuation automatically. Defaults to `true`
+* `googleCloud.useEnhancedModel`: use enhanced model. Defaults to `true`
+
+Azure Cognitive Services:
+* `azureCognitiveServices.subscriptionKey`: subscription key (required)
+* `azureCognitiveServices.serviceRegion`: service region (required)
+
+Amazon Transcribe:
+* `amazonTranscribe.region`: region (required)
+* `amazonTranscribe.accessKeyId`: access key ID (required)
+* `amazonTranscribe.secretAccessKey`: secret access key (required)
+
+## Alignment
+
+Applicable to CLI command: `align`.
+
+General:
+* `method`: what alignment algorithm to use, can be `dtw`, `dtw-ra` or `whisper`. Defaults to `dtw`
+* `language`: language code for the audio and transcript, like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected if not set
+
+DTW:
+* `dtw.windowDuration`: time duration (in seconds) of the Sakoe-Chiba window when performing DTW alignment. Defaults to `120`. If your audio is longer than two minutes, consider increasing this value for better results. Note that a higher value would consume quadratically larger amounts of memory. A value of `600` (ten minutes) would already require several Gigabytes of memory when the audio duration is 10 minutes or greater.
+
+DTW-RA only:
+* `dtw.recognition`: prefix for providing custom recognition options when using `dtw-ra` method, for example: setting `dtw.recognition.engine = silero`
+* `dtw.phoneAlignmentMethod`: algorithm to use when aligning phones: can either be set to `dtw` or `interpolate`. Defaults to `dtw`
+
+## Speech translation
+
+Applicable to CLI command: `translate-speech`.
+
+General:
+* `engine`: only `whisper` supported
+
+Whisper:
+* `whisper.engine`: Whisper engine to use (multilingual engines only). Defaults to `tiny`
+
+## Language detection
+
+
+### Spoken language detection
+
+Applicable to CLI command: `detect-speech-langauge`.
+
+* `engine`: `silero` or `whisper`. Defaults to `silero`
+* `whisper`: whisper options prefix, can be used like `whisper.model = base` to set options for the Whisper engine. See Whisper options on the recognition section
+
+### Written language detection
+
+Applicable to CLI command: `detect-text-langauge`.
+
+* `engine`: `tinyld` or `fasttext`. Defaults to `tinyld`
+
+## Voice activity detection
+
+Applicable to CLI command: `detect-voice-activity`.
+
+General:
+* `engine`: VAD engine to use. Can be `webrtc`, `silero` or `rnnoise`. Defaults to `webrtc`
+* `activityThreshold`: minimum predicted probability for determining a frame as having speech activity. Defaults to `0.5`
+
+WebRTC:
+* `webrtc.frameDuration`: WebRTC frame duration (ms). Can be `10`, `20` or `30`. Defaults to `30`
+* `webrtc.mode`: WebRTC mode (aggressiveness). Can be `0`, `1`, `2` or `3`. Defaults to `1`
+
+Silero:
+* `silero.frameDuration`: Silero frame duration (ms). Can be `30`, `60` or `90`. Defaults to `90`
+
+## Speech denoising
+
+Applicable to CLI command: `denoise`.
+
+General:
+* `engine`: can only be `rnnoise`
+
+Postprocessing:
+* `postProcessing.normalizeAudio` should normalize output audio. Defaults to `false`
+* `postProcessing.targetPeakDb` target peak (decibels) for normalization. Defaults to `-3`
+* `postProcessing.maxIncreaseDb` max gain increase (decibels) when performing normalization. Defaults to `30`
+* `postProcessing.dryMixGainDb` gain (decibels) of dry (original) signal to mix back to the denoised output. Defaults to `-20`
+
+## Voice list request
+
+Applicable to CLI command: `list-voices`.
+
+* `language`: language code to filter by (optional)
+* `voice`: name or name pattern to filter by (optional)
+* `voiceGender`: gender to filter by (optional)

package/docs/Roadmap.md

@@ -0,0 +1,16 @@
+# Roadmap
+
+1. Publish a working command line application
+1. Smooth out any significant bugs
+1. Ensure platform support
+1. Add missing features, options or commands to the CLI
+1. Stabilize the programmatic API
+1. Open up the programmatic API. Add any extra methods needed for that to happen
+1. Develop a WebSocket server that closely mirrors the API
+1. Develop a basic Web-based client app
+1. Modify the CLI app to internally launch and communicate with a WebSocket service
+1. Add parallelism to the CLI and WebSocket server
+1. Further improve the client app
+1. Start porting small subsets of the code to run in the browser
+1. Port some of the speech API to the browser
+

package/docs/Technical.md

@@ -0,0 +1,72 @@
+# Technical overview and Q&A
+
+* Echogarden is written in TypeScript and targets the Node.js platform
+* It uses ESM modules and latest ECMAScript and TypeScript features
+* It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime (with some exceptions: the CLI does use executables, loaded from an expansion package, for the SoX tool [and `ffmpeg` as well], which is used only for audio playback and recording. This simplifies the installation and ensures a non-buggy version is used. Since SoX `v14.4.2` is broken on Windows, it bundles `v14.4.1`).
+* It does not depend on essential native modules requiring compilation with `node-gyp`. This greatly simplifies and ensures the success of the installation for end-users (the ONNX runtime bundles precompiled NAPI modules for all supported platforms - it doesn't require any compilation during installation)
+
+## Package system
+
+Echogarden uses its own package system to download and install various components as needed. These components can be TTS voices, STT models, or other types of models and data.
+
+Packages are downloaded as `.tar.gz` files, and are extracted to `[data-folder]/packages/[package-name]`. Each package has its own subdirectory.
+
+`[data-folder]` is located at:
+* `%AppData%\Local\echogarden` on Windows
+* `Users/User/Library/Application Support/echogarden` on macOS
+* `/home/user/.local/share/echogarden` on Linux
+
+By using downloadable packages, the installed size is made significantly smaller and the installation faster. The total size of all available packages is currently about 11.4GB (6.8GB compressed). Another advantage of using a data folder is that the data does not need to be re-downloaded when the program is updated.
+
+## Can the base install size be made smaller?
+
+The installed (uncompressed) size, including dependencies, is around 350MB. This is without any models or voices, which are downloaded as needed.
+
+Currently, the biggest contributors to the size are:
+
+* `onnxruntime-node` (NAPI): 92MB
+* `kuromoji` (JavaScript) 40MB
+* `ffmpeg` (WASM): 24MB
+* `flite-wasi` (WASI): 20MB
+* `espeak-ng-emscripten` (WASM): 18MB
+
+`onnxruntime-node` is big because it bundles pre-compiled binaries for multiple platforms. `kuromoji` is large because of its dictionary files and some unessential test code it bundles. The other three packages include large WASM binaries.
+
+So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies, as npm packages, only when needed.
+
+## Since the code is almost all JavaScript and WASM, why can't it just run in a web browser?
+
+It is technically possible, overall, since its core components: `espeak-ng` and `onnxruntime` both have WASM ports. Actually, `onnxruntime-web`, unlike `onnxruntime-node` can also make use of the GPU via WebGL, and in the future, [it will support WebGPU](https://github.com/microsoft/onnxruntime/issues/11695), which should give a performance boost.
+
+However, it is a lot of work, and only a subset of the engines can be supported (no cloud engines, in particular). There are several reasons why the web may not be the most effective platform:
+
+* Significantly slower inference when running on CPU
+* No cross-domain network connectivity - can't connect to Google Cloud, Microsoft, Amazon etc. without a proxy
+* Large initial download size would make it too heavy and slow to load as part of a standard web page directly
+* Will require a virtual file system to store models and make use of downloadable packages
+* Requires duplicating a lot of prior work and increasing code complexity
+* Possibly lots of issues with inconsistent browser support and browser security constraints
+* Not future-proof. Due to changing restrictions of browsers, the runtime environment is not guaranteed be reliably reproducible in the future, meaning that it may need continuous maintenance to ensure it keeps working on the newest browsers
+
+It remains to be seen what type of use cases would justify this kind of endeavor. I designed the tool to make the local installation extremely easy and issue-free. I guess it could look "impressive" to be able to run it in a browser, and may be come as a nice "toy" or "technology demonstration", and get some attention, but it may eventually turn out to be simpler and more practical to just install a local instance and connect to it from the browser via a WebSocket.
+
+Another option is to include some of the TTS modules in a browser extension and register TTS voices on the browser's Web Speech API using the [`chrome.ttsEngine`](https://developer.chrome.com/docs/extensions/reference/ttsEngine/) extension API. That sounds reasonable, but due to the size and complexity of such an extension, it is not certain it would be approved by the Chrome Web Store. Also, `chrome.ttsEngine` is currently only supported on Chromium browsers, but not on Firefox.
+
+## Why can't I use `stdin` and `stdout` to pipe into and out from the CLI app?
+
+I don't think that `stdin` and `stdout` are able to capture the type of complex, multi-message, bidirectional communication that is needed for a full speech processing service. When a WebSocket server is implemented, it will fulfill all these needs, and more. It would provide a uniform interface for all external clients, and would also be launched and used internally by the CLI itself. It would enable the CLI to do complex asynchronous and parallel operations it can't currently do, like transcribing a live input while text is written to the terminal and audio is played, processed and transmitted, all at the same time.
+
+## Why does the CLI use `--option=value` and not `--option value` syntax?
+
+I would have allowed `--option value` if I could. The reason `--option=value` syntax is used is because the options are defined using an auto-generated schema, which is different for each command. This means I don't know ahead of time which options require an extra value, like `--voice Bob`, and which don't, like `--play`, at the time the command line is parsed. This makes it difficult to intersperse switches and inputs/outputs like `echogarden speak "Hello" --play output.mp3 --voice Bob`.
+
+In order to support a syntax like `--option value` I will need to parse the command line separately for each command, but then it would be difficult to share common options between all of them.
+
+Also, `option=value` is more similar to the syntax used in the configuration file, which makes it more consistent, and since the CLI accepts arbitrary numbers of free arguments, it help prevent confusion on whether an argument is related to an option or is it a free one.
+
+## Code organization
+
+* `src`: TypeScript source code
+* `dist`: compiled JavaScript modules
+* `data`: various data files, including phonetic lexicons and conversion tables. `data/schemas` stores JSON schemas for all configuration options, auto-generated using [`ts-json-schema-generator`](https://github.com/vega/ts-json-schema-generator) directly from the TypeScript code, and used by the CLI to parse and validate the types of options provided
+* `docs`: documentation