echogarden 0.4.2 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/docs/CLI.md CHANGED
@@ -8,36 +8,45 @@ echogarden [command] [one or more inputs..] [one or more outputs...] [options...
8
8
 
9
9
  Here's a quick tour of the main operations available via the CLI.
10
10
 
11
- Each command accepts many options, in the form `--[optionName]=[value]` (The `=` is required). A detailed reference of all the options can be found [here](Options.md).
11
+ Each command can accepts one or more options, in the form `--[optionName]=[value]` (The `=` is required). A detailed reference of all the available options can be found [here](Options.md).
12
12
 
13
13
  ## Text to speech
14
14
 
15
15
  **Task**: given a text file, synthesize spoken audio for it.
16
16
 
17
-
18
17
  This would synthesize "Hello World" and play the result in the terminal:
19
18
  ```bash
20
19
  echogarden speak "Hello world!"
21
20
  ```
22
21
 
23
- You can add a file output instead, which would cause the audio to be stored in `result.mp3`:
22
+ If no language is specified, it would attempt to detect it. This usually works better for longer texts, and may misidentify shorter ones. To ensure the right language is selected, you can specify the language explicitly:
24
23
  ```bash
25
- echogarden speak "Hello world!" result.mp3
24
+ echogarden speak "Hello world!" --language=en
26
25
  ```
27
26
 
28
- This would synthesize a text file `text.txt`, and have the resulting audio stored in `result.mp3`. The `--engine=pico` would set the synthesis engine to `pico` (SVOX Pico):
27
+ This would save the resulting audio to `result.mp3`:
29
28
  ```bash
30
- echogarden speak-file text.txt result.mp3 --engine=pico
29
+ echogarden speak "Hello world!" result.mp3 --language=en
31
30
  ```
32
31
 
33
- This would synthesize a text file, and have the result stored in both `result.mp3` and `result.wav`, as well as a subtitle file in `result.srt`:
32
+ `speak-file` synthesizes text loaded from a textual file, which can be either `.txt`, `.html`, `.srt`, `.vtt`:
33
+ ```bash
34
+ echogarden speak-file text.txt result.mp3 --language=en
35
+ ```
36
+
37
+ You can specify an engine using the `--engine` option (a full list of engines can be found [here](Engines.md)). This would set the synthesis engine to `pico` (SVOX Pico):
38
+ ```bash
39
+ echogarden speak-file text.txt result.mp3 --language=en --engine=pico
40
+ ```
41
+
42
+ The CLI supports multiple output files. This would synthesize a text file, and save the resulting audio in both `result.mp3` and `result.wav`, as well as subtitles in `result.srt`:
34
43
  ```bash
35
44
  echogarden speak-file text.txt result.mp3 result.wav result.srt --engine=vits --speed=1.1
36
45
  ```
37
46
 
38
47
  Synthesize a web page (will try to extract its main article parts and omit the rest):
39
48
  ```bash
40
- echogarden speak-url http://example.com/hola
49
+ echogarden speak-url https://example.com/hola
41
50
  ```
42
51
 
43
52
  Synthesize a Wikipedia article in any of its language editions:
@@ -45,9 +54,20 @@ Synthesize a Wikipedia article in any of its language editions:
45
54
  echogarden speak-wikipedia "Psychologie" --language=fr
46
55
  ```
47
56
 
57
+ By default, audio isn't played in the terminal when an output file is specified, you can override this behavior by adding `--play`
58
+
59
+ ```bash
60
+ echogarden speak-file text.txt result.mp3 --engine=vits --play
61
+ ```
62
+
63
+ Or similarly prevent playback using `--no-play`:
64
+ ```bash
65
+ echogarden speak "Hello world!" --language=en --no-play
66
+ ```
67
+
48
68
  ## Speech to text
49
69
 
50
- **Task**: given an audio recording containing speech, find a text transcription that best matches it.
70
+ **Task**: given an audio recording containing speech, find a textual transcription that best matches it.
51
71
 
52
72
  This would transcribe the audio file `speech.mp3`, and then play the audio, along with the recognized text, in the terminal:
53
73
  ```bash
@@ -81,9 +101,9 @@ Echogarden can split the audio to multiple parts based on the segment boundaries
81
101
  echogarden speak text.txt parts/[segment].opus
82
102
  ```
83
103
 
84
- The `[segment]` pattern would cause multiple files to be created, one for each text segment (segments would be determined according to paragraph or line breaks, in this case). The pattern would be replaced by the index and initial text of the segment. For example by `parts/01 Hello world how are you doing ... .opus`.
104
+ The `[segment]` placeholder would cause multiple files to be created, one for each text segment (segments would be determined according to paragraph or line breaks, in this case). The placeholder would be replaced by the index and initial text of the segment, producing an output file with a name like `parts/001 Hello world how are you doing ... .opus`.
85
105
 
86
- Templates can also be used in multiple outputs. For instance, the following would align `speech.mp3` with `transcript.txt` and then split the audio by the segments found in the transcript, and store separate audio and timeline files for each part.
106
+ Templates can also be used in multiple outputs. For instance, the following would align `speech.mp3` with `transcript.txt` and then split the audio according to the segments found in the transcript, and store separate audio and timeline files for each part.
87
107
 
88
108
  ```bash
89
109
  echogarden align speech.mp3 transcript.txt parts/[segment].m4a parts/[segment].json
@@ -215,19 +235,21 @@ echogarden denoise speech.mp3 denoised-speech.mp3
215
235
 
216
236
  #### `list-tts-voices`
217
237
 
218
- Shows a list of TTS voices for a given engine:
238
+ Shows a list of available TTS voices for a given engine:
219
239
 
220
240
  ```bash
221
241
  echogarden list-tts-voices google-cloud
222
242
  ```
223
243
 
224
- Stores the voice list in a JSON file:
244
+ Saves the voice list in a JSON file:
225
245
  ```bash
226
246
  echogarden list-tts-voices google-cloud google-cloud-voices.json
227
247
  ```
228
248
 
229
249
  #### `install`, `uninstall` and `list-packages`
230
250
 
251
+ Manage the Echogarden packages that are locally installed:
252
+
231
253
  * `install`: install one or more expansion packages
232
254
  * `uninstall`: uninstall one or more expansion packages
233
255
  * `list-packages`: list installed packages
@@ -1,3 +1,9 @@
1
1
  # How to help
2
2
 
3
- TODO
3
+ So far, this project has been the solo work of a single person (yours truly).
4
+
5
+ However, there are many areas where contributions can be made.
6
+
7
+ For instance, if you are proficient in one or more languages, other than English, you could help a lot by extending the pronunciations lexicons to help correct errors in the default phonemizations produced by the eSpeak engine, or add rules to help resolve the pronunciations of heteronyms (words that are written the same but can be pronounced in different ways depending on their context) based on their context.
8
+
9
+ ... TODO ...
package/docs/Engines.md CHANGED
@@ -4,56 +4,64 @@
4
4
  ## Text to speech
5
5
 
6
6
  **Offline**:
7
- * [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Currently available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Supports English (US, UK), Spanish, Brazilian Portuguese, Italian, French, German, Dutch, Swedish, Norwegian, Danish, Finnish, Polish, Greek, Russian, Ukrainian, Catalan, Icelandic, Kazakh, Nepali, Vietnamese and Chinese. You can listen to audio examples of all voices and languages on [Piper's samples page](https://rhasspy.github.io/piper-samples/).
8
- * [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a diphone-based synthesizer. Supports English (US, UK), Spanish, Italian, French, and German.
9
- * [Flite](https://github.com/festvox/flite) (`flite`): a diphone-based synthesizer. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi.
10
- * [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a formant-based synthesizer. Supports 100+ languages.
7
+ * [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Currently available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Currently there are 83 voices, in a range of languages, including English (US, UK), Spanish (ES, MX), Brazilian Portuguese, Italian, French, German, Dutch (NL, BE), Swedish, Norwegian, Danish, Finnish, Polish, Greek, Russian, Ukrainian, Catalan, Icelandic, Swahili, Kazakh, Georgian, Nepali, Vietnamese and Chinese. You can listen to audio samples of all voices and languages in [Piper's samples page](https://rhasspy.github.io/piper-samples/).
8
+ * [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a legacy diphone-based synthesis engine. Supports English (US, UK), Spanish, Italian, French, and German.
9
+ * [Flite](https://github.com/festvox/flite) (`flite`): a legacy diphone-based synthesis engine. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi.
10
+ * [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a lightweight "robot" sounding formant-based synthesizer. Supports 100+ languages. Extensively used internally for speech alignment, phonemization, and other internal tasks.
11
11
  * [SAM (Software Automatic Mouth)](https://github.com/discordier/sam) (`sam`): a classic "robot" speech synthesizer from 1982. English only.
12
12
 
13
13
  **Offline, Windows only**:
14
14
 
15
- _Note_: the SAPI-based engines require manually installing the [`winax` npm package](https://www.npmjs.com/package/winax) by running `npm install winax -g`.
15
+ * [SAPI](https://en.wikipedia.org/wiki/Microsoft_Speech_API) (`sapi`): Microsoft Speech API. Supports the system's language voices, as well as legacy voices produced by third-party vendors, like Ivona, NeoSpeech, Acapela, Cepstral, CereProc, Nuance, AT&T, Loquendo, ScanSoft and others (note that only 64-bit SAPI voices are supported, which makes it incompatible with a significant portion of older voices).
16
16
 
17
- `winax` is a native module which requires the Node.js Windows build tools to successfully install. If you have issues installing this package, please ensure that you've checked the "install necessary tools" checkbox during the installation of node.js.
17
+ * [Microsoft Speech Platform](https://www.microsoft.com/en-us/download/details.aspx?id=27225) (`msspeech`): Microsoft Server Speech API. Requires [installing a runtime (2.6MB)](https://www.microsoft.com/en-us/download/details.aspx?id=27225). Supports 28 dialects, which can be individually downloaded via [freely available installers](https://www.microsoft.com/en-us/download/details.aspx?id=27224), or, for convenience, bundled as [a single 358MB zip file](https://drive.google.com/u/0/uc?id=1uQdFNxLzUxpaEwVVKhMawys8cIh3F21T&export=download). Has voices for English (US, UK, AU, CA), Spanish (ES, MX), Portuguese (BR, PT), German, French (FR, CA), Italian, Norwegian, Dutch, Russian, Swedish, Danish, Catalan, Finnish, Japanese, Korean and Chinese (ZH, HK, TW). All voices are female.
18
18
 
19
- * [SAPI](https://en.wikipedia.org/wiki/Microsoft_Speech_API) (`sapi`): Microsoft Speech API. Supports the system's language voices, as well as voices produced by third-party vendors, like Ivona, NeoSpeech, Acapela, Cepstral, CereProc, Nuance, AT&T, Loquendo, ScanSoft and others (note that only 64-bit voices are supported, which makes it incompatible with a significant portion of older voices).
19
+ _Note_: both these engines require manually installing the [`winax` npm package](https://www.npmjs.com/package/winax) by running `npm install winax -g`.
20
20
 
21
- * [Microsoft Speech Platform](https://www.microsoft.com/en-us/download/details.aspx?id=27225) (`msspeech`): Microsoft Server Speech API. Requires [installing a runtime (2.6MB)](https://www.microsoft.com/en-us/download/details.aspx?id=27225). Supports 28 dialects, which can be individually downloaded via [freely available installers](https://www.microsoft.com/en-us/download/details.aspx?id=27224), or, for convenience, bundled as [a single 358MB zip file](https://drive.google.com/u/0/uc?id=1uQdFNxLzUxpaEwVVKhMawys8cIh3F21T&export=download). Has voices for English (US, UK, AU, CA), Spanish (ES, MX), Portuguese (BR, PT), German, French (FR, CA), Italian, Norwegian, Dutch, Russian, Swedish, Danish, Catalan, Finnish, Japanese, Korean and Chinese (ZH, HK, TW). All voices are female.
21
+ `winax` is a native module which requires the Node.js Windows build tools to successfully install. If you have issues installing this package, please ensure that you've checked the "install necessary tools" checkbox during the installation of node.js.
22
22
 
23
- **Remote or self-hosted servers**:
23
+ **Client for remote or self-hosted servers**:
24
24
  * [Coqui TTS](https://github.com/coqui-ai/TTS) server (`coqui-server`)
25
25
 
26
- **Cloud-based**:
26
+ **Cloud services**:
27
+
28
+ These are commercial services that require a subscription and an API key to use:
29
+
27
30
  * [Google Cloud](https://cloud.google.com/text-to-speech) (`google-cloud`)
28
31
  * [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/text-to-speech/) (`microsoft-azure`)
29
32
  * [Amazon Polly](https://aws.amazon.com/polly/) (`amazon-polly`)
30
33
  * [Elevenlabs](https://beta.elevenlabs.io/) (`elevenlabs`)
31
34
 
32
- **Cloud-based (unofficial)**:
35
+ **Cloud services (unofficial)**:
33
36
 
34
- These cloud-based engines connect to public-facing cloud APIs that are not officially publicized by their operators. They are included for experimental purposes only, and may be removed in the future:
37
+ These cloud-based engines connect to public-facing cloud APIs that are not officially publicized by their operators. They are included for educational purposes only, and may be removed in the future:
35
38
 
36
- * Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice (usually female) for each language.
39
+ * Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice for each language (usually female).
37
40
  * Microsoft Edge (`microsoft-edge`): subset of the Azure Cognitive Services cloud TTS API used by the Microsoft Edge browser as part of its support for the [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API) and its [Read Aloud](https://www.microsoft.com/en-us/edge/features/read-aloud?form=MT00D8) feature. Using this engine requires a special token, which should be passed via the `microsoftEdge.trustedClientToken` option.
38
41
  * Streamlabs Polly (`streamlabs-polly`): a public REST API by Streamlabs, primarily intended for generating speech for TTS donations. It includes a few English (US, UK, AU, IN) voices, which are similar to some of the non-neural (Ivona-based) voices offered by Amazon Polly.
39
42
 
40
43
  ## Speech to text
41
44
 
42
45
  **Offline**:
43
- * [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high accuracy transformer-based model. Supports 99 languages. There are several models of different sizes, some are multilingual, and some are English only (`.en`): `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v1` and `large-v2`. _Note_: large models are not currently auto-downloaded as Echogarden packages, but may become available in the future.
44
- * [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for 25+ languages. _Note_: Vosk is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. You'll need to manually [download a model](https://alphacephei.com/vosk/models) and specify its directory path via the `vosk.modelPath` option.
46
+ * [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high accuracy transformer-based architecture. Supports 99 languages. There are several models of different sizes, some are multilingual, and some are English only (`.en`): `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v1` and `large-v2`. _Note_: large models are not currently auto-downloaded as Echogarden packages, but may become available in the future.
47
+ * [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for 25+ languages. _Note_: the Vosk package is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. You'll need to manually [download a model](https://alphacephei.com/vosk/models) and specify its directory path via the `vosk.modelPath` option.
45
48
  * [Silero](https://github.com/snakers4/silero-models) (`silero`): models available for English, Spanish, German and Ukrainian. For [non-commercial use only](https://github.com/snakers4/silero-models/blob/master/LICENSE).
46
49
 
47
- **Cloud based**:
50
+ **Cloud services**:
51
+
52
+ These are commercial services that require a subscription and an API key to use:
53
+
48
54
  * [Google Cloud](https://cloud.google.com/speech-to-text) (`google-cloud`)
49
55
  * [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/speech-to-text/) (`microsoft-azure`)
50
56
  * [Amazon Transcribe](https://aws.amazon.com/transcribe/) (`amazon-transcribe`)
51
57
 
52
58
  ## Speech to transcript alignment
53
59
 
60
+ These engines' goal is to match (or "align") a given spoken recording with a given transcript as closely as possible. They will annotate each word in the transcript with approximate start and end timestamps:
61
+
54
62
  * Dynamic Time Warping (`dtw`): transcript is first synthesized using the `espeak` engine, then [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) is applied to find the best mapping between the synthesized audio and the original audio.
55
63
  * Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using `espeak`. Then, the best mapping is found between the two synthesized audio sequences, and the result is mapped back to the original audio using the timing information produced by the recognizer.
56
- * Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model (only supports audio inputs that are 30 seconds or less).
64
+ * Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model (_note_: currently, only supports audio inputs that are 30 seconds or less).
57
65
 
58
66
  ## Speech translation
59
67
 
@@ -65,7 +73,7 @@ These cloud-based engines connect to public-facing cloud APIs that are not offic
65
73
  * [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the 99 languages it has been trained on.
66
74
  * [Silero Language Classifier](https://github.com/snakers4/silero-vad/wiki/Other-Models) (`silero`): a speech language classification model by Silero.
67
75
 
68
- **Written language detection**:
76
+ **Text language detection**:
69
77
  * [TinyLD](https://www.npmjs.com/package/tinyld) (`tinyld`): a simple language detection library.
70
78
  * [FastText](https://github.com/facebookresearch/fastText) (`fasttext`): a library for word representations and sentence classification by Facebook research.
71
79
 
package/docs/Licenses.md CHANGED
@@ -23,12 +23,12 @@
23
23
 
24
24
  ## Package repository
25
25
 
26
- There is a variety of voices, models and binaries served from the repository.
26
+ A large variety of voices, models and binaries are served from the repository.
27
27
 
28
28
  All are freely distributable, with varying licenses:
29
29
  * Flite voices (`flite-`): [BSD License](https://github.com/festvox/flite/blob/master/COPYING)
30
30
  * SVOX Pico resources (`pico-`): [Apache License 2.0](https://github.com/gmn/nanotts/blob/master/LICENSE)
31
- * Silero VAD (`silero-vad`) and Silero language classifier (`silero-lang-classifier-95`): [MIT License](https://github.com/snakers4/silero-vad/blob/master/LICENSE)
31
+ * Silero VAD (`silero-vad`) and Silero language classifier (`silero-lang-classifier-95`): [MIT License](https://github.com/snakers4/silero-vad/blob/master/LICENSE)
32
32
  * Silero speech recognition models (`silero-en-`, `silero-de-`, `silero-es-`, `silero-ua-`): [BY-NC-SA](https://github.com/snakers4/silero-models/blob/master/LICENSE)
33
33
  * VITS pre-trained models (`vits-`): licensed under various creative commons licenses: [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/), [CC-BY](https://creativecommons.org/licenses/by/4.0/) and [BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/), and few are public domain (you can view the individual license for each model in the model cards on the [samples page](https://rhasspy.github.io/piper-samples/)). The [Piper system](https://github.com/rhasspy/piper) itself is published under the [MIT License](https://github.com/rhasspy/piper/blob/master/LICENSE.md)
34
34
  * Whisper pre-trained models (`whisper-`): [MIT License](https://github.com/openai/whisper/blob/main/LICENSE)
package/docs/Options.md CHANGED
@@ -9,7 +9,7 @@ Applicable to CLI commands: `speak`, `speak-file`, `speak-url`, `speak-wikipedia
9
9
  General:
10
10
  * `engine`: identifier of the synthesis engine to use, such as `espeak` or `vits`.
11
11
  * `language`: language code, like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected if not set
12
- * `voice`: name of the voice to use. Optional
12
+ * `voice`: name of the voice to use. Can be a search string. Optional
13
13
  * `voiceGender`: gender of the voice to use. Optional
14
14
  * `speed`: speech rate factor, relative to default. In the range `0.1`..`10.0`. Defaults to `1.0`
15
15
  * `pitch`: pitch factor, relative to default. In the range `0.1`..`10.0`. Defaults to `1.0`
@@ -75,7 +75,7 @@ Elevenlabs:
75
75
  * `elevenLabs.similarityBoost`: similarity boost
76
76
 
77
77
  Google Translate:
78
- * `googleTranslate.tld`: top level domain to to connect to. Can change the dialect of the voices for a small number or voices. For example `us` gives American English for `en`, while `com` gives British English for `en`. Defaults to `us`
78
+ * `googleTranslate.tld`: top level domain to to connect to. Can change the dialect for a small number or voices. For example `us` gives American English for `en`, while `com` gives British English for `en`. Defaults to `us`
79
79
 
80
80
  Microsoft Edge:
81
81
  * `microsoftEdge.trustedClientToken`: trusted client token (required). A special token required to use the service
package/docs/Roadmap.md CHANGED
@@ -1,14 +1,15 @@
1
1
  # Roadmap
2
2
 
3
- 1. Publish a working command line application
4
- 1. Smooth out any significant bugs
5
- 1. Ensure platform support
6
- 1. Add missing features, options or commands to the CLI
7
- 1. Stabilize the library API
8
- 1. Open up the library API. Add any extra methods or options needed for that to happen
9
- 1. Develop a worker API that closely mirrors the library API
10
- 1. Develop a WebSocket server based on the worker API
11
- 1. Develop a basic Web-based client app, talking the WebSocket server
12
- 1. Modify the CLI app to internally use the background worker
13
- 1. Further improve the client app
14
- 1. Port the offline speech synthesis API to the browser (in particular the `vits`, `pico`, `flite`, `eSpeak` engines)
3
+ - [x] Publish a working command line application
4
+ - [x] Smooth out any significant bugs
5
+ - [x] Ensure platform support
6
+ - [x] Add missing features, options or commands to the CLI
7
+ - [x] Design a message-based worker API
8
+ - [x] Implement the worker API
9
+ - [x] Implement a WebSocket server that implements the worker API
10
+ - [ ] Publish the worker API. Add any extra methods or options needed for that to happen
11
+ - [ ] Develop a browser extension, talking the WebSocket server
12
+ - [ ] Develop a web-based client app, talking to, and hosted on the server
13
+ - [ ] Modify the CLI app to internally use the background worker for some tasks
14
+ - [ ] Further improve the client app
15
+ - [ ] Port some of the offline speech synthesis API to the browser (in particular the `vits`, `pico`, `flite`, `eSpeak` engines)
package/docs/Technical.md CHANGED
@@ -2,8 +2,8 @@
2
2
 
3
3
  * Echogarden is written in TypeScript and targets the Node.js platform.
4
4
  * It uses ESM modules and latest ECMAScript and TypeScript features.
5
- * It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime (with some exceptions: the CLI does use executables, loaded from an expansion package, for the SoX tool [and `ffmpeg` as well], which is used only for audio playback and recording. This simplifies the installation and ensures a non-buggy version is used. Since SoX `v14.4.2` is broken on Windows, it bundles `v14.4.1`).
6
- * It does not depend on essential native modules requiring compilation with `node-gyp`. This greatly simplifies and ensures the success of the installation for end-users (the ONNX runtime bundles precompiled NAPI modules for all supported platforms - it doesn't require any compilation during installation).
5
+ * It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime (with some exceptions: the CLI does invoke a few binary executables, loaded from expansion packages, for the `SoX` and `ffmpeg` tools.Using expansion packages simplifies the installation and ensures non-buggy version are used. Since SoX `v14.4.2` is broken on Windows, it bundles `v14.4.1`).
6
+ * It does not depend on essential native node.js modules requiring compilation with `node-gyp`. This greatly simplifies the installation experience for end-users (the ONNX runtime bundles precompiled NAPI modules for all supported platforms - it doesn't require any compilation during its installation).
7
7
 
8
8
  ## Package system
9
9
 
@@ -16,13 +16,13 @@ Packages are downloaded as `.tar.gz` files, and are extracted to `[data-folder]/
16
16
  * `Users/User/Library/Application Support/echogarden` on macOS
17
17
  * `/home/user/.local/share/echogarden` on Linux
18
18
 
19
- By using downloadable packages, the installed size is made significantly smaller and the installation faster. The total size of all available packages is currently about 13GB (8.45GB compressed).
19
+ By using downloadable packages, the installed size is made significantly smaller and the installation faster. The total size of all available packages is currently about 15.4GB (10.6GB compressed).
20
20
 
21
- Another advantage of using a data folder is that the data does not need to be re-downloaded when the program is updated.
21
+ The packages are currently hosted and downloaded from a dedicated [Hugging Face repository](https://huggingface.co/echogarden/echogarden-packages).
22
22
 
23
23
  ## Can the base install size be made smaller?
24
24
 
25
- The installed (uncompressed) size, including dependencies, is around 270MB. This is without any models or voices, which are downloaded as needed.
25
+ The base installed (uncompressed) size, including dependencies, is around 270MB. This is without any models or voices, which are downloaded as needed.
26
26
 
27
27
  Currently, the biggest contributors to the size are:
28
28
 
@@ -34,29 +34,31 @@ Currently, the biggest contributors to the size are:
34
34
 
35
35
  `onnxruntime-node` is big because it bundles pre-compiled binaries for multiple platforms. `kuromoji` is large because of its dictionary files and some unessential test code it bundles. The other three packages include large WASM binaries.
36
36
 
37
- So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies, as npm packages, only when needed.
37
+ So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies, or using modified, "slimmed-down" custom versions.
38
38
 
39
39
  ## Since the code is almost all JavaScript and WASM, why can't it just run in a web browser?
40
40
 
41
41
  It is technically possible, overall, since its core components: `espeak-ng` and `onnxruntime` both have WASM ports. Actually, `onnxruntime-web`, unlike `onnxruntime-node` can also make use of the GPU via WebGL, and in the future, [it will support WebGPU](https://github.com/microsoft/onnxruntime/issues/11695), which should give a performance boost.
42
42
 
43
- However, it is a lot of work, and only a subset of the engines can be supported (no cloud engines, in particular). There are several reasons why the web may not be the most effective platform:
43
+ However, it is a lot of work, and only a subset of the engines can be supported (no cloud engines, in particular). There are several reasons why the web may not be the most effective platform for Echogarden:
44
44
 
45
- * Significantly slower inference when running on CPU
45
+ * Significantly slower inference when using CPU for ONNX models
46
46
  * No cross-domain network connectivity - can't connect to Google Cloud, Microsoft, Amazon etc. without a proxy
47
47
  * Large initial download size would make it too heavy and slow to load as part of a standard web page directly
48
+ * Large memory requirement for the VITS models, starting at about 800MB - 1GB, which is a bit too much for a browser
49
+ * Due to the high code complexity, data size, and memory consumption, it is unlikely that a browser extension, internally bundling some of the models, would be accepted to the Chrome and Firefox web stores
48
50
  * Will require a virtual file system to store models and make use of downloadable packages
49
- * Requires duplicating a lot of prior work and increasing code complexity
51
+ * Requires duplicating a lot of prior work, porting many node.js-only APIs, and increasing code complexity
50
52
  * Possibly lots of issues with inconsistent browser support and browser security constraints
51
53
  * Not future-proof. Due to changing restrictions of browsers, the runtime environment is not guaranteed be reliably reproducible in the future, meaning that it may need continuous maintenance to ensure it keeps working on the newest browsers
52
54
 
53
- It remains to be seen what type of use cases would justify this kind of endeavor. I designed the tool to make the local installation extremely easy and issue-free. I guess it could look "impressive" to be able to run it in a browser, and may be come as a nice "toy" or "technology demonstration", and get some attention, but it may eventually turn out to be simpler and more practical to just install a local instance and connect to it from the browser via a WebSocket.
55
+ It remains to be seen if this sort of work would feel justified somehow. I designed the tool to make the local installation extremely easy and issue-free. I guess it could look "impressive" to be able to run it in a browser, and may be come as a nice "toy" or "tech-demo", and could get some attention, but it may eventually turn out to be simpler and more practical to just install a local instance and connect to it from the browser via a WebSocket API (which is already working and running, but at a development stage).
54
56
 
55
- Another option is to include some of the TTS modules in a browser extension and register TTS voices on the browser's Web Speech API using the [`chrome.ttsEngine`](https://developer.chrome.com/docs/extensions/reference/ttsEngine/) extension API. That sounds reasonable, but due to the size and complexity of such an extension, it is not certain it would be approved by the Chrome Web Store. Also, `chrome.ttsEngine` is currently only supported on Chromium browsers, but not on Firefox.
57
+ A TTS-only browser extension is in development. It registers Echogarden's voices on the browser's Web Speech API using the [`chrome.ttsEngine`](https://developer.chrome.com/docs/extensions/reference/ttsEngine/) extension API and communicates with it using the WebSocket API
56
58
 
57
59
  ## Why can't I use `stdin` and `stdout` to pipe into and out from the CLI app?
58
60
 
59
- I don't think that `stdin` and `stdout` are able to capture the type of complex, multi-message, bidirectional communication that is needed for a full speech processing service. When a WebSocket server is implemented, it will fulfill all these needs, and more. It would provide a uniform interface for all external clients, and would also be launched and used internally by the CLI itself. It would enable the CLI to do complex asynchronous and parallel operations it can't currently do, like transcribing a live input while text is written to the terminal and audio is played, processed and transmitted, all at the same time.
61
+ I don't think that `stdin` and `stdout` are able to capture the type of complex, multi-message, bidirectional communication that is needed for a full speech processing service. When the WebSocket server is released, it will fulfill all these needs, and more. It would provide a uniform interface for all external clients, and would also be launched and used internally by the CLI itself. It would enable the CLI to do complex asynchronous and parallel operations it can't currently do, like transcribing a live input while text is written to the terminal and audio is played, processed and transmitted, all at the same time.
60
62
 
61
63
  ## Why does the CLI use `--option=value` and not `--option value` syntax?
62
64
 
@@ -64,11 +66,11 @@ I would have allowed `--option value` if I could. The reason `--option=value` sy
64
66
 
65
67
  In order to support a syntax like `--option value` I will need to parse the command line separately for each command, but then it would be difficult to share common options between all of them.
66
68
 
67
- Also, `option=value` is more similar to the syntax used in the configuration file, which makes it more consistent, and since the CLI accepts arbitrary numbers of free arguments, it help prevent confusion on whether an argument is related to an option or is it a free one.
69
+ Also, `option=value` is more similar to the syntax used in the configuration file, which makes it more consistent, and since the CLI accepts arbitrary numbers of free arguments, it helps prevent confusion on whether an argument is related to an option or a free one.
68
70
 
69
71
  ## Code organization
70
72
 
71
73
  * `src`: TypeScript source code
72
74
  * `dist`: compiled JavaScript modules
73
- * `data`: various data files, including phonetic lexicons and conversion tables. `data/schemas` stores JSON schemas for all configuration options, auto-generated using [`ts-json-schema-generator`](https://github.com/vega/ts-json-schema-generator) directly from the TypeScript code, and used by the CLI to parse and validate the types of options provided
75
+ * `data`: various data files, including phonetic lexicons and language code conversion tables. `data/schemas` stores JSON schemas for all configuration options, auto-generated using [`ts-json-schema-generator`](https://github.com/vega/ts-json-schema-generator) directly from the TypeScript code, and used by the CLI to parse and validate the options provided
74
76
  * `docs`: documentation
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "echogarden",
3
- "version": "0.4.2",
4
- "description": "An integrated speech toolbox designed with end-users in mind.",
3
+ "version": "0.5.0",
4
+ "description": "A fully open-source speech system designed with end-users in mind.",
5
5
  "author": "Rotem Dan",
6
6
  "license": "GPL-3.0-only",
7
7
  "keywords": [
@@ -55,8 +55,8 @@
55
55
  "echogarden": "./dist/cli/CLILauncher.js"
56
56
  },
57
57
  "dependencies": {
58
- "@aws-sdk/client-polly": "^3.363.0",
59
- "@aws-sdk/client-transcribe-streaming": "^3.363.0",
58
+ "@aws-sdk/client-polly": "^3.370.0",
59
+ "@aws-sdk/client-transcribe-streaming": "^3.370.0",
60
60
  "@echogarden/espeak-ng-emscripten": "^0.1.2",
61
61
  "@echogarden/fasttext-wasm": "^0.1.0",
62
62
  "@echogarden/flite-wasi": "^0.1.1",
@@ -122,13 +122,13 @@
122
122
  "@types/msgpack-lite": "^0.1.8",
123
123
  "@types/ndarray": "^1.0.11",
124
124
  "@types/ndarray-ops": "^1.2.4",
125
- "@types/node": "^20.4.1",
125
+ "@types/node": "^20.4.2",
126
126
  "@types/recursive-readdir": "^2.2.1",
127
127
  "@types/tar": "^6.1.5",
128
128
  "@types/ws": "^8.5.5",
129
- "@typescript-eslint/eslint-plugin": "^5.61.0",
130
- "@typescript-eslint/parser": "^5.61.0",
131
- "eslint": "^8.44.0",
129
+ "@typescript-eslint/eslint-plugin": "^6.1.0",
130
+ "@typescript-eslint/parser": "^6.1.0",
131
+ "eslint": "^8.45.0",
132
132
  "ts-json-schema-generator": "^1.2.0",
133
133
  "typescript": "^5.1.6"
134
134
  }