echogarden 0.4.2 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -14
- package/data/lexicons/heteronyms.en.json +254 -138
- package/dist/cli/CLI.js +22 -6
- package/dist/cli/CLI.js.map +1 -1
- package/dist/nlp/TextNormalizer.js +1 -1
- package/dist/nlp/TextNormalizer.js.map +1 -1
- package/dist/synthesis/VitsTTS.js +230 -65
- package/dist/synthesis/VitsTTS.js.map +1 -1
- package/dist/utilities/PackageManager.d.ts +3 -1
- package/dist/utilities/PackageManager.js +20 -16
- package/dist/utilities/PackageManager.js.map +1 -1
- package/docs/CLI.md +35 -13
- package/docs/Development.md +7 -1
- package/docs/Engines.md +26 -18
- package/docs/Licenses.md +2 -2
- package/docs/Options.md +2 -2
- package/docs/Roadmap.md +13 -12
- package/docs/Technical.md +16 -14
- package/package.json +8 -8
package/docs/CLI.md
CHANGED
|
@@ -8,36 +8,45 @@ echogarden [command] [one or more inputs..] [one or more outputs...] [options...
|
|
|
8
8
|
|
|
9
9
|
Here's a quick tour of the main operations available via the CLI.
|
|
10
10
|
|
|
11
|
-
Each command accepts
|
|
11
|
+
Each command can accepts one or more options, in the form `--[optionName]=[value]` (The `=` is required). A detailed reference of all the available options can be found [here](Options.md).
|
|
12
12
|
|
|
13
13
|
## Text to speech
|
|
14
14
|
|
|
15
15
|
**Task**: given a text file, synthesize spoken audio for it.
|
|
16
16
|
|
|
17
|
-
|
|
18
17
|
This would synthesize "Hello World" and play the result in the terminal:
|
|
19
18
|
```bash
|
|
20
19
|
echogarden speak "Hello world!"
|
|
21
20
|
```
|
|
22
21
|
|
|
23
|
-
|
|
22
|
+
If no language is specified, it would attempt to detect it. This usually works better for longer texts, and may misidentify shorter ones. To ensure the right language is selected, you can specify the language explicitly:
|
|
24
23
|
```bash
|
|
25
|
-
echogarden speak "Hello world!"
|
|
24
|
+
echogarden speak "Hello world!" --language=en
|
|
26
25
|
```
|
|
27
26
|
|
|
28
|
-
This would
|
|
27
|
+
This would save the resulting audio to `result.mp3`:
|
|
29
28
|
```bash
|
|
30
|
-
echogarden speak
|
|
29
|
+
echogarden speak "Hello world!" result.mp3 --language=en
|
|
31
30
|
```
|
|
32
31
|
|
|
33
|
-
|
|
32
|
+
`speak-file` synthesizes text loaded from a textual file, which can be either `.txt`, `.html`, `.srt`, `.vtt`:
|
|
33
|
+
```bash
|
|
34
|
+
echogarden speak-file text.txt result.mp3 --language=en
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
You can specify an engine using the `--engine` option (a full list of engines can be found [here](Engines.md)). This would set the synthesis engine to `pico` (SVOX Pico):
|
|
38
|
+
```bash
|
|
39
|
+
echogarden speak-file text.txt result.mp3 --language=en --engine=pico
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The CLI supports multiple output files. This would synthesize a text file, and save the resulting audio in both `result.mp3` and `result.wav`, as well as subtitles in `result.srt`:
|
|
34
43
|
```bash
|
|
35
44
|
echogarden speak-file text.txt result.mp3 result.wav result.srt --engine=vits --speed=1.1
|
|
36
45
|
```
|
|
37
46
|
|
|
38
47
|
Synthesize a web page (will try to extract its main article parts and omit the rest):
|
|
39
48
|
```bash
|
|
40
|
-
echogarden speak-url
|
|
49
|
+
echogarden speak-url https://example.com/hola
|
|
41
50
|
```
|
|
42
51
|
|
|
43
52
|
Synthesize a Wikipedia article in any of its language editions:
|
|
@@ -45,9 +54,20 @@ Synthesize a Wikipedia article in any of its language editions:
|
|
|
45
54
|
echogarden speak-wikipedia "Psychologie" --language=fr
|
|
46
55
|
```
|
|
47
56
|
|
|
57
|
+
By default, audio isn't played in the terminal when an output file is specified, you can override this behavior by adding `--play`
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
echogarden speak-file text.txt result.mp3 --engine=vits --play
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Or similarly prevent playback using `--no-play`:
|
|
64
|
+
```bash
|
|
65
|
+
echogarden speak "Hello world!" --language=en --no-play
|
|
66
|
+
```
|
|
67
|
+
|
|
48
68
|
## Speech to text
|
|
49
69
|
|
|
50
|
-
**Task**: given an audio recording containing speech, find a
|
|
70
|
+
**Task**: given an audio recording containing speech, find a textual transcription that best matches it.
|
|
51
71
|
|
|
52
72
|
This would transcribe the audio file `speech.mp3`, and then play the audio, along with the recognized text, in the terminal:
|
|
53
73
|
```bash
|
|
@@ -81,9 +101,9 @@ Echogarden can split the audio to multiple parts based on the segment boundaries
|
|
|
81
101
|
echogarden speak text.txt parts/[segment].opus
|
|
82
102
|
```
|
|
83
103
|
|
|
84
|
-
The `[segment]`
|
|
104
|
+
The `[segment]` placeholder would cause multiple files to be created, one for each text segment (segments would be determined according to paragraph or line breaks, in this case). The placeholder would be replaced by the index and initial text of the segment, producing an output file with a name like `parts/001 Hello world how are you doing ... .opus`.
|
|
85
105
|
|
|
86
|
-
Templates can also be used in multiple outputs. For instance, the following would align `speech.mp3` with `transcript.txt` and then split the audio
|
|
106
|
+
Templates can also be used in multiple outputs. For instance, the following would align `speech.mp3` with `transcript.txt` and then split the audio according to the segments found in the transcript, and store separate audio and timeline files for each part.
|
|
87
107
|
|
|
88
108
|
```bash
|
|
89
109
|
echogarden align speech.mp3 transcript.txt parts/[segment].m4a parts/[segment].json
|
|
@@ -215,19 +235,21 @@ echogarden denoise speech.mp3 denoised-speech.mp3
|
|
|
215
235
|
|
|
216
236
|
#### `list-tts-voices`
|
|
217
237
|
|
|
218
|
-
Shows a list of TTS voices for a given engine:
|
|
238
|
+
Shows a list of available TTS voices for a given engine:
|
|
219
239
|
|
|
220
240
|
```bash
|
|
221
241
|
echogarden list-tts-voices google-cloud
|
|
222
242
|
```
|
|
223
243
|
|
|
224
|
-
|
|
244
|
+
Saves the voice list in a JSON file:
|
|
225
245
|
```bash
|
|
226
246
|
echogarden list-tts-voices google-cloud google-cloud-voices.json
|
|
227
247
|
```
|
|
228
248
|
|
|
229
249
|
#### `install`, `uninstall` and `list-packages`
|
|
230
250
|
|
|
251
|
+
Manage the Echogarden packages that are locally installed:
|
|
252
|
+
|
|
231
253
|
* `install`: install one or more expansion packages
|
|
232
254
|
* `uninstall`: uninstall one or more expansion packages
|
|
233
255
|
* `list-packages`: list installed packages
|
package/docs/Development.md
CHANGED
|
@@ -1,3 +1,9 @@
|
|
|
1
1
|
# How to help
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
So far, this project has been the solo work of a single person (yours truly).
|
|
4
|
+
|
|
5
|
+
However, there are many areas where contributions can be made.
|
|
6
|
+
|
|
7
|
+
For instance, if you are proficient in one or more languages, other than English, you could help a lot by extending the pronunciations lexicons to help correct errors in the default phonemizations produced by the eSpeak engine, or add rules to help resolve the pronunciations of heteronyms (words that are written the same but can be pronounced in different ways depending on their context) based on their context.
|
|
8
|
+
|
|
9
|
+
... TODO ...
|
package/docs/Engines.md
CHANGED
|
@@ -4,56 +4,64 @@
|
|
|
4
4
|
## Text to speech
|
|
5
5
|
|
|
6
6
|
**Offline**:
|
|
7
|
-
* [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Currently available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper).
|
|
8
|
-
* [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a diphone-based
|
|
9
|
-
* [Flite](https://github.com/festvox/flite) (`flite`): a diphone-based
|
|
10
|
-
* [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a formant-based synthesizer. Supports 100+ languages.
|
|
7
|
+
* [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Currently available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Currently there are 83 voices, in a range of languages, including English (US, UK), Spanish (ES, MX), Brazilian Portuguese, Italian, French, German, Dutch (NL, BE), Swedish, Norwegian, Danish, Finnish, Polish, Greek, Russian, Ukrainian, Catalan, Icelandic, Swahili, Kazakh, Georgian, Nepali, Vietnamese and Chinese. You can listen to audio samples of all voices and languages in [Piper's samples page](https://rhasspy.github.io/piper-samples/).
|
|
8
|
+
* [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a legacy diphone-based synthesis engine. Supports English (US, UK), Spanish, Italian, French, and German.
|
|
9
|
+
* [Flite](https://github.com/festvox/flite) (`flite`): a legacy diphone-based synthesis engine. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi.
|
|
10
|
+
* [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a lightweight "robot" sounding formant-based synthesizer. Supports 100+ languages. Extensively used internally for speech alignment, phonemization, and other internal tasks.
|
|
11
11
|
* [SAM (Software Automatic Mouth)](https://github.com/discordier/sam) (`sam`): a classic "robot" speech synthesizer from 1982. English only.
|
|
12
12
|
|
|
13
13
|
**Offline, Windows only**:
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
* [SAPI](https://en.wikipedia.org/wiki/Microsoft_Speech_API) (`sapi`): Microsoft Speech API. Supports the system's language voices, as well as legacy voices produced by third-party vendors, like Ivona, NeoSpeech, Acapela, Cepstral, CereProc, Nuance, AT&T, Loquendo, ScanSoft and others (note that only 64-bit SAPI voices are supported, which makes it incompatible with a significant portion of older voices).
|
|
16
16
|
|
|
17
|
-
`
|
|
17
|
+
* [Microsoft Speech Platform](https://www.microsoft.com/en-us/download/details.aspx?id=27225) (`msspeech`): Microsoft Server Speech API. Requires [installing a runtime (2.6MB)](https://www.microsoft.com/en-us/download/details.aspx?id=27225). Supports 28 dialects, which can be individually downloaded via [freely available installers](https://www.microsoft.com/en-us/download/details.aspx?id=27224), or, for convenience, bundled as [a single 358MB zip file](https://drive.google.com/u/0/uc?id=1uQdFNxLzUxpaEwVVKhMawys8cIh3F21T&export=download). Has voices for English (US, UK, AU, CA), Spanish (ES, MX), Portuguese (BR, PT), German, French (FR, CA), Italian, Norwegian, Dutch, Russian, Swedish, Danish, Catalan, Finnish, Japanese, Korean and Chinese (ZH, HK, TW). All voices are female.
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
_Note_: both these engines require manually installing the [`winax` npm package](https://www.npmjs.com/package/winax) by running `npm install winax -g`.
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
`winax` is a native module which requires the Node.js Windows build tools to successfully install. If you have issues installing this package, please ensure that you've checked the "install necessary tools" checkbox during the installation of node.js.
|
|
22
22
|
|
|
23
|
-
**
|
|
23
|
+
**Client for remote or self-hosted servers**:
|
|
24
24
|
* [Coqui TTS](https://github.com/coqui-ai/TTS) server (`coqui-server`)
|
|
25
25
|
|
|
26
|
-
**Cloud
|
|
26
|
+
**Cloud services**:
|
|
27
|
+
|
|
28
|
+
These are commercial services that require a subscription and an API key to use:
|
|
29
|
+
|
|
27
30
|
* [Google Cloud](https://cloud.google.com/text-to-speech) (`google-cloud`)
|
|
28
31
|
* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/text-to-speech/) (`microsoft-azure`)
|
|
29
32
|
* [Amazon Polly](https://aws.amazon.com/polly/) (`amazon-polly`)
|
|
30
33
|
* [Elevenlabs](https://beta.elevenlabs.io/) (`elevenlabs`)
|
|
31
34
|
|
|
32
|
-
**Cloud
|
|
35
|
+
**Cloud services (unofficial)**:
|
|
33
36
|
|
|
34
|
-
These cloud-based engines connect to public-facing cloud APIs that are not officially publicized by their operators. They are included for
|
|
37
|
+
These cloud-based engines connect to public-facing cloud APIs that are not officially publicized by their operators. They are included for educational purposes only, and may be removed in the future:
|
|
35
38
|
|
|
36
|
-
* Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice (usually female)
|
|
39
|
+
* Google Translate (`google-translate`): used by the [Google Translate web UI](https://translate.google.com/) to speak written text in any one of its supported languages. Offers a single voice for each language (usually female).
|
|
37
40
|
* Microsoft Edge (`microsoft-edge`): subset of the Azure Cognitive Services cloud TTS API used by the Microsoft Edge browser as part of its support for the [Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API) and its [Read Aloud](https://www.microsoft.com/en-us/edge/features/read-aloud?form=MT00D8) feature. Using this engine requires a special token, which should be passed via the `microsoftEdge.trustedClientToken` option.
|
|
38
41
|
* Streamlabs Polly (`streamlabs-polly`): a public REST API by Streamlabs, primarily intended for generating speech for TTS donations. It includes a few English (US, UK, AU, IN) voices, which are similar to some of the non-neural (Ivona-based) voices offered by Amazon Polly.
|
|
39
42
|
|
|
40
43
|
## Speech to text
|
|
41
44
|
|
|
42
45
|
**Offline**:
|
|
43
|
-
* [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high accuracy transformer-based
|
|
44
|
-
* [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for 25+ languages. _Note_: Vosk is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. You'll need to manually [download a model](https://alphacephei.com/vosk/models) and specify its directory path via the `vosk.modelPath` option.
|
|
46
|
+
* [OpenAI Whisper](https://github.com/openai/whisper) (`whisper`): high accuracy transformer-based architecture. Supports 99 languages. There are several models of different sizes, some are multilingual, and some are English only (`.en`): `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v1` and `large-v2`. _Note_: large models are not currently auto-downloaded as Echogarden packages, but may become available in the future.
|
|
47
|
+
* [Vosk](https://github.com/alphacep/vosk-api) (`vosk`): models available for 25+ languages. _Note_: the Vosk package is not included in the default installation, but you can add support for it using `npm install @echogarden/vosk -g`. You'll need to manually [download a model](https://alphacephei.com/vosk/models) and specify its directory path via the `vosk.modelPath` option.
|
|
45
48
|
* [Silero](https://github.com/snakers4/silero-models) (`silero`): models available for English, Spanish, German and Ukrainian. For [non-commercial use only](https://github.com/snakers4/silero-models/blob/master/LICENSE).
|
|
46
49
|
|
|
47
|
-
**Cloud
|
|
50
|
+
**Cloud services**:
|
|
51
|
+
|
|
52
|
+
These are commercial services that require a subscription and an API key to use:
|
|
53
|
+
|
|
48
54
|
* [Google Cloud](https://cloud.google.com/speech-to-text) (`google-cloud`)
|
|
49
55
|
* [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/cognitive-services/speech-to-text/) (`microsoft-azure`)
|
|
50
56
|
* [Amazon Transcribe](https://aws.amazon.com/transcribe/) (`amazon-transcribe`)
|
|
51
57
|
|
|
52
58
|
## Speech to transcript alignment
|
|
53
59
|
|
|
60
|
+
These engines' goal is to match (or "align") a given spoken recording with a given transcript as closely as possible. They will annotate each word in the transcript with approximate start and end timestamps:
|
|
61
|
+
|
|
54
62
|
* Dynamic Time Warping (`dtw`): transcript is first synthesized using the `espeak` engine, then [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) is applied to find the best mapping between the synthesized audio and the original audio.
|
|
55
63
|
* Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using `espeak`. Then, the best mapping is found between the two synthesized audio sequences, and the result is mapped back to the original audio using the timing information produced by the recognizer.
|
|
56
|
-
* Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model (only supports audio inputs that are 30 seconds or less).
|
|
64
|
+
* Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model (_note_: currently, only supports audio inputs that are 30 seconds or less).
|
|
57
65
|
|
|
58
66
|
## Speech translation
|
|
59
67
|
|
|
@@ -65,7 +73,7 @@ These cloud-based engines connect to public-facing cloud APIs that are not offic
|
|
|
65
73
|
* [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the 99 languages it has been trained on.
|
|
66
74
|
* [Silero Language Classifier](https://github.com/snakers4/silero-vad/wiki/Other-Models) (`silero`): a speech language classification model by Silero.
|
|
67
75
|
|
|
68
|
-
**
|
|
76
|
+
**Text language detection**:
|
|
69
77
|
* [TinyLD](https://www.npmjs.com/package/tinyld) (`tinyld`): a simple language detection library.
|
|
70
78
|
* [FastText](https://github.com/facebookresearch/fastText) (`fasttext`): a library for word representations and sentence classification by Facebook research.
|
|
71
79
|
|
package/docs/Licenses.md
CHANGED
|
@@ -23,12 +23,12 @@
|
|
|
23
23
|
|
|
24
24
|
## Package repository
|
|
25
25
|
|
|
26
|
-
|
|
26
|
+
A large variety of voices, models and binaries are served from the repository.
|
|
27
27
|
|
|
28
28
|
All are freely distributable, with varying licenses:
|
|
29
29
|
* Flite voices (`flite-`): [BSD License](https://github.com/festvox/flite/blob/master/COPYING)
|
|
30
30
|
* SVOX Pico resources (`pico-`): [Apache License 2.0](https://github.com/gmn/nanotts/blob/master/LICENSE)
|
|
31
|
-
* Silero VAD (`silero-vad`) and Silero language classifier
|
|
31
|
+
* Silero VAD (`silero-vad`) and Silero language classifier (`silero-lang-classifier-95`): [MIT License](https://github.com/snakers4/silero-vad/blob/master/LICENSE)
|
|
32
32
|
* Silero speech recognition models (`silero-en-`, `silero-de-`, `silero-es-`, `silero-ua-`): [BY-NC-SA](https://github.com/snakers4/silero-models/blob/master/LICENSE)
|
|
33
33
|
* VITS pre-trained models (`vits-`): licensed under various creative commons licenses: [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/), [CC-BY](https://creativecommons.org/licenses/by/4.0/) and [BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/), and few are public domain (you can view the individual license for each model in the model cards on the [samples page](https://rhasspy.github.io/piper-samples/)). The [Piper system](https://github.com/rhasspy/piper) itself is published under the [MIT License](https://github.com/rhasspy/piper/blob/master/LICENSE.md)
|
|
34
34
|
* Whisper pre-trained models (`whisper-`): [MIT License](https://github.com/openai/whisper/blob/main/LICENSE)
|
package/docs/Options.md
CHANGED
|
@@ -9,7 +9,7 @@ Applicable to CLI commands: `speak`, `speak-file`, `speak-url`, `speak-wikipedia
|
|
|
9
9
|
General:
|
|
10
10
|
* `engine`: identifier of the synthesis engine to use, such as `espeak` or `vits`.
|
|
11
11
|
* `language`: language code, like `en`, `fr`, `en-US`, `pt-BR`. Auto-detected if not set
|
|
12
|
-
* `voice`: name of the voice to use. Optional
|
|
12
|
+
* `voice`: name of the voice to use. Can be a search string. Optional
|
|
13
13
|
* `voiceGender`: gender of the voice to use. Optional
|
|
14
14
|
* `speed`: speech rate factor, relative to default. In the range `0.1`..`10.0`. Defaults to `1.0`
|
|
15
15
|
* `pitch`: pitch factor, relative to default. In the range `0.1`..`10.0`. Defaults to `1.0`
|
|
@@ -75,7 +75,7 @@ Elevenlabs:
|
|
|
75
75
|
* `elevenLabs.similarityBoost`: similarity boost
|
|
76
76
|
|
|
77
77
|
Google Translate:
|
|
78
|
-
* `googleTranslate.tld`: top level domain to to connect to. Can change the dialect
|
|
78
|
+
* `googleTranslate.tld`: top level domain to to connect to. Can change the dialect for a small number or voices. For example `us` gives American English for `en`, while `com` gives British English for `en`. Defaults to `us`
|
|
79
79
|
|
|
80
80
|
Microsoft Edge:
|
|
81
81
|
* `microsoftEdge.trustedClientToken`: trusted client token (required). A special token required to use the service
|
package/docs/Roadmap.md
CHANGED
|
@@ -1,14 +1,15 @@
|
|
|
1
1
|
# Roadmap
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
3
|
+
- [x] Publish a working command line application
|
|
4
|
+
- [x] Smooth out any significant bugs
|
|
5
|
+
- [x] Ensure platform support
|
|
6
|
+
- [x] Add missing features, options or commands to the CLI
|
|
7
|
+
- [x] Design a message-based worker API
|
|
8
|
+
- [x] Implement the worker API
|
|
9
|
+
- [x] Implement a WebSocket server that implements the worker API
|
|
10
|
+
- [ ] Publish the worker API. Add any extra methods or options needed for that to happen
|
|
11
|
+
- [ ] Develop a browser extension, talking the WebSocket server
|
|
12
|
+
- [ ] Develop a web-based client app, talking to, and hosted on the server
|
|
13
|
+
- [ ] Modify the CLI app to internally use the background worker for some tasks
|
|
14
|
+
- [ ] Further improve the client app
|
|
15
|
+
- [ ] Port some of the offline speech synthesis API to the browser (in particular the `vits`, `pico`, `flite`, `eSpeak` engines)
|
package/docs/Technical.md
CHANGED
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
* Echogarden is written in TypeScript and targets the Node.js platform.
|
|
4
4
|
* It uses ESM modules and latest ECMAScript and TypeScript features.
|
|
5
|
-
* It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime (with some exceptions: the CLI does
|
|
6
|
-
* It does not depend on essential native modules requiring compilation with `node-gyp`. This greatly simplifies
|
|
5
|
+
* It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime (with some exceptions: the CLI does invoke a few binary executables, loaded from expansion packages, for the `SoX` and `ffmpeg` tools.Using expansion packages simplifies the installation and ensures non-buggy version are used. Since SoX `v14.4.2` is broken on Windows, it bundles `v14.4.1`).
|
|
6
|
+
* It does not depend on essential native node.js modules requiring compilation with `node-gyp`. This greatly simplifies the installation experience for end-users (the ONNX runtime bundles precompiled NAPI modules for all supported platforms - it doesn't require any compilation during its installation).
|
|
7
7
|
|
|
8
8
|
## Package system
|
|
9
9
|
|
|
@@ -16,13 +16,13 @@ Packages are downloaded as `.tar.gz` files, and are extracted to `[data-folder]/
|
|
|
16
16
|
* `Users/User/Library/Application Support/echogarden` on macOS
|
|
17
17
|
* `/home/user/.local/share/echogarden` on Linux
|
|
18
18
|
|
|
19
|
-
By using downloadable packages, the installed size is made significantly smaller and the installation faster. The total size of all available packages is currently about
|
|
19
|
+
By using downloadable packages, the installed size is made significantly smaller and the installation faster. The total size of all available packages is currently about 15.4GB (10.6GB compressed).
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
The packages are currently hosted and downloaded from a dedicated [Hugging Face repository](https://huggingface.co/echogarden/echogarden-packages).
|
|
22
22
|
|
|
23
23
|
## Can the base install size be made smaller?
|
|
24
24
|
|
|
25
|
-
The installed (uncompressed) size, including dependencies, is around 270MB. This is without any models or voices, which are downloaded as needed.
|
|
25
|
+
The base installed (uncompressed) size, including dependencies, is around 270MB. This is without any models or voices, which are downloaded as needed.
|
|
26
26
|
|
|
27
27
|
Currently, the biggest contributors to the size are:
|
|
28
28
|
|
|
@@ -34,29 +34,31 @@ Currently, the biggest contributors to the size are:
|
|
|
34
34
|
|
|
35
35
|
`onnxruntime-node` is big because it bundles pre-compiled binaries for multiple platforms. `kuromoji` is large because of its dictionary files and some unessential test code it bundles. The other three packages include large WASM binaries.
|
|
36
36
|
|
|
37
|
-
So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies,
|
|
37
|
+
So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies, or using modified, "slimmed-down" custom versions.
|
|
38
38
|
|
|
39
39
|
## Since the code is almost all JavaScript and WASM, why can't it just run in a web browser?
|
|
40
40
|
|
|
41
41
|
It is technically possible, overall, since its core components: `espeak-ng` and `onnxruntime` both have WASM ports. Actually, `onnxruntime-web`, unlike `onnxruntime-node` can also make use of the GPU via WebGL, and in the future, [it will support WebGPU](https://github.com/microsoft/onnxruntime/issues/11695), which should give a performance boost.
|
|
42
42
|
|
|
43
|
-
However, it is a lot of work, and only a subset of the engines can be supported (no cloud engines, in particular). There are several reasons why the web may not be the most effective platform:
|
|
43
|
+
However, it is a lot of work, and only a subset of the engines can be supported (no cloud engines, in particular). There are several reasons why the web may not be the most effective platform for Echogarden:
|
|
44
44
|
|
|
45
|
-
* Significantly slower inference when
|
|
45
|
+
* Significantly slower inference when using CPU for ONNX models
|
|
46
46
|
* No cross-domain network connectivity - can't connect to Google Cloud, Microsoft, Amazon etc. without a proxy
|
|
47
47
|
* Large initial download size would make it too heavy and slow to load as part of a standard web page directly
|
|
48
|
+
* Large memory requirement for the VITS models, starting at about 800MB - 1GB, which is a bit too much for a browser
|
|
49
|
+
* Due to the high code complexity, data size, and memory consumption, it is unlikely that a browser extension, internally bundling some of the models, would be accepted to the Chrome and Firefox web stores
|
|
48
50
|
* Will require a virtual file system to store models and make use of downloadable packages
|
|
49
|
-
* Requires duplicating a lot of prior work and increasing code complexity
|
|
51
|
+
* Requires duplicating a lot of prior work, porting many node.js-only APIs, and increasing code complexity
|
|
50
52
|
* Possibly lots of issues with inconsistent browser support and browser security constraints
|
|
51
53
|
* Not future-proof. Due to changing restrictions of browsers, the runtime environment is not guaranteed be reliably reproducible in the future, meaning that it may need continuous maintenance to ensure it keeps working on the newest browsers
|
|
52
54
|
|
|
53
|
-
It remains to be seen
|
|
55
|
+
It remains to be seen if this sort of work would feel justified somehow. I designed the tool to make the local installation extremely easy and issue-free. I guess it could look "impressive" to be able to run it in a browser, and may be come as a nice "toy" or "tech-demo", and could get some attention, but it may eventually turn out to be simpler and more practical to just install a local instance and connect to it from the browser via a WebSocket API (which is already working and running, but at a development stage).
|
|
54
56
|
|
|
55
|
-
|
|
57
|
+
A TTS-only browser extension is in development. It registers Echogarden's voices on the browser's Web Speech API using the [`chrome.ttsEngine`](https://developer.chrome.com/docs/extensions/reference/ttsEngine/) extension API and communicates with it using the WebSocket API
|
|
56
58
|
|
|
57
59
|
## Why can't I use `stdin` and `stdout` to pipe into and out from the CLI app?
|
|
58
60
|
|
|
59
|
-
I don't think that `stdin` and `stdout` are able to capture the type of complex, multi-message, bidirectional communication that is needed for a full speech processing service. When
|
|
61
|
+
I don't think that `stdin` and `stdout` are able to capture the type of complex, multi-message, bidirectional communication that is needed for a full speech processing service. When the WebSocket server is released, it will fulfill all these needs, and more. It would provide a uniform interface for all external clients, and would also be launched and used internally by the CLI itself. It would enable the CLI to do complex asynchronous and parallel operations it can't currently do, like transcribing a live input while text is written to the terminal and audio is played, processed and transmitted, all at the same time.
|
|
60
62
|
|
|
61
63
|
## Why does the CLI use `--option=value` and not `--option value` syntax?
|
|
62
64
|
|
|
@@ -64,11 +66,11 @@ I would have allowed `--option value` if I could. The reason `--option=value` sy
|
|
|
64
66
|
|
|
65
67
|
In order to support a syntax like `--option value` I will need to parse the command line separately for each command, but then it would be difficult to share common options between all of them.
|
|
66
68
|
|
|
67
|
-
Also, `option=value` is more similar to the syntax used in the configuration file, which makes it more consistent, and since the CLI accepts arbitrary numbers of free arguments, it
|
|
69
|
+
Also, `option=value` is more similar to the syntax used in the configuration file, which makes it more consistent, and since the CLI accepts arbitrary numbers of free arguments, it helps prevent confusion on whether an argument is related to an option or a free one.
|
|
68
70
|
|
|
69
71
|
## Code organization
|
|
70
72
|
|
|
71
73
|
* `src`: TypeScript source code
|
|
72
74
|
* `dist`: compiled JavaScript modules
|
|
73
|
-
* `data`: various data files, including phonetic lexicons and conversion tables. `data/schemas` stores JSON schemas for all configuration options, auto-generated using [`ts-json-schema-generator`](https://github.com/vega/ts-json-schema-generator) directly from the TypeScript code, and used by the CLI to parse and validate the
|
|
75
|
+
* `data`: various data files, including phonetic lexicons and language code conversion tables. `data/schemas` stores JSON schemas for all configuration options, auto-generated using [`ts-json-schema-generator`](https://github.com/vega/ts-json-schema-generator) directly from the TypeScript code, and used by the CLI to parse and validate the options provided
|
|
74
76
|
* `docs`: documentation
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "echogarden",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "0.5.0",
|
|
4
|
+
"description": "A fully open-source speech system designed with end-users in mind.",
|
|
5
5
|
"author": "Rotem Dan",
|
|
6
6
|
"license": "GPL-3.0-only",
|
|
7
7
|
"keywords": [
|
|
@@ -55,8 +55,8 @@
|
|
|
55
55
|
"echogarden": "./dist/cli/CLILauncher.js"
|
|
56
56
|
},
|
|
57
57
|
"dependencies": {
|
|
58
|
-
"@aws-sdk/client-polly": "^3.
|
|
59
|
-
"@aws-sdk/client-transcribe-streaming": "^3.
|
|
58
|
+
"@aws-sdk/client-polly": "^3.370.0",
|
|
59
|
+
"@aws-sdk/client-transcribe-streaming": "^3.370.0",
|
|
60
60
|
"@echogarden/espeak-ng-emscripten": "^0.1.2",
|
|
61
61
|
"@echogarden/fasttext-wasm": "^0.1.0",
|
|
62
62
|
"@echogarden/flite-wasi": "^0.1.1",
|
|
@@ -122,13 +122,13 @@
|
|
|
122
122
|
"@types/msgpack-lite": "^0.1.8",
|
|
123
123
|
"@types/ndarray": "^1.0.11",
|
|
124
124
|
"@types/ndarray-ops": "^1.2.4",
|
|
125
|
-
"@types/node": "^20.4.
|
|
125
|
+
"@types/node": "^20.4.2",
|
|
126
126
|
"@types/recursive-readdir": "^2.2.1",
|
|
127
127
|
"@types/tar": "^6.1.5",
|
|
128
128
|
"@types/ws": "^8.5.5",
|
|
129
|
-
"@typescript-eslint/eslint-plugin": "^
|
|
130
|
-
"@typescript-eslint/parser": "^
|
|
131
|
-
"eslint": "^8.
|
|
129
|
+
"@typescript-eslint/eslint-plugin": "^6.1.0",
|
|
130
|
+
"@typescript-eslint/parser": "^6.1.0",
|
|
131
|
+
"eslint": "^8.45.0",
|
|
132
132
|
"ts-json-schema-generator": "^1.2.0",
|
|
133
133
|
"typescript": "^5.1.6"
|
|
134
134
|
}
|